From patchwork Tue Aug 1 13:24:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 129304 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp2677827vqg; Tue, 1 Aug 2023 06:43:15 -0700 (PDT) X-Google-Smtp-Source: APBJJlHY7wBgf23ZrsZKZRWirCtvMTGjPs37V0ty+oeUM5vHOb753uxkZmstXXLxDLcqqB7/kXgP X-Received: by 2002:a17:902:ed8c:b0:1b8:28f6:20e6 with SMTP id e12-20020a170902ed8c00b001b828f620e6mr11022002plj.34.1690897394850; Tue, 01 Aug 2023 06:43:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690897394; cv=none; d=google.com; s=arc-20160816; b=SbxzRpzYKqOed4hWmunFq81ErNoS5vOkfdPXmO2bvRZ8CEMPChFetygUn8KI+whBtR Xs5nVnqaXKO9dGJbOmdl+NBX7QdElY5e1EnG96PWBkodfxD/gRY9KdD60Hdu5EQk+a7K TdXhI08xP2UnSivgVG5VWog45RKCWt6wY0RoyuyFs2tzRJw3Lg8Hgctb3IvXvvYdhn3x zg/dWShx+aZJ1AO6JvHHPG0ANq0hMAOIpZI1P9DbnSGvl+4ccqrIYLuouss54BIydrOb z0MxUvJYg6fdpvMpzV5yZR4pxnJ0v8vs4Xes0WyI+0/7FY+cADDoxSZC5jvR8VktJZhd FSUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Vjm5rqPvdxUy7Z6aUM6JkGwm74Jj3I1LYFhNzNy9bzo=; fh=JC0FP1KBPXa+8NkvIvZoCUIIa+zGdCFP1xCgvJrJBSE=; b=j9Y5oH/izDqsnslTxeDoKj9G4UTUK3BFMl8Oi2UxDrdA5xM6kQm82ocHmlkMxZAdlb 5UiTOhlPiu36i2b1P8cdy6cCndyfse9I2d+wLOUMxa5HpX7DDdhqla7DiyTWbmimzLHK 3veiMTTZ/dUoNri6+kO0VSNp1pZEcUcsTYMhwuufZxx/IhTWDYdDrVtS25kvGUwUpwn8 30rWmCgqme/w1qJlMGXRr9g6m4EJI5upT7ZS2rt5Dxy+aSFBeSzZRAEPd/uPi37k8AzI 6oxSuayS1BOqk7ZmSJLYxyQ/pxV8claGMLLJ0iJsr9Rk8klIxOMceoHSl5buPbO2QYAL x/5Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=oUheRgQ6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n10-20020a170902d0ca00b001b20e2ba8e1si5251398pln.23.2023.08.01.06.43.01; Tue, 01 Aug 2023 06:43:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=oUheRgQ6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234093AbjHANYy (ORCPT + 99 others); Tue, 1 Aug 2023 09:24:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36812 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230315AbjHANYw (ORCPT ); Tue, 1 Aug 2023 09:24:52 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C1841985 for ; Tue, 1 Aug 2023 06:24:51 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CE895615A4 for ; Tue, 1 Aug 2023 13:24:50 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4BDFBC433C7; Tue, 1 Aug 2023 13:24:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690896290; bh=OCbZZEYY6XG+Gj++glqCcqULb++oCr0LofyrtVtyhEc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oUheRgQ6e3jTLVKzFRNeivu8xi+wx3bSs7luMToq7MPUyeMEOdFV9Z10Xwj52jGBl TsBzJKvb9b+/lHy5AF7lkLk+ev1RoQAc/mtDMsEnAswNdWKiH79GfrtjsA5DmDxD8M GFiG1ohj3mNsLa4alssdvt9LGyjknXtmPAD/p8AhgFnq34OEh2S6c0lToN5VK9gU5j kA/v7zsvQ5CTF3dGiW/7YcQBuJqvH75sOtwB2Kultu/XQULLc5jC2TOqCrm5klZUOs /brfyl+A+H/TH738p7IqIOp+65VenPfrbLtrAiSiK77m2Oh3wDfMsnvm2MzM8Luh4+ u7PdXIokymRMw== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Sebastian Andrzej Siewior , Peter Zijlstra , Thomas Gleixner , "Paul E . McKenney" , Linus Torvalds , Anna-Maria Behnsen , Eric Dumazet Subject: [RFC PATCH 1/6] softirq: Turn set_softirq_pending() to reset_softirq_pending() Date: Tue, 1 Aug 2023 15:24:36 +0200 Message-Id: <20230801132441.559222-2-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230801132441.559222-1-frederic@kernel.org> References: <20230801132441.559222-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773034426743812526 X-GMAIL-MSGID: 1773034426743812526 set_softirq_pending() is only ever used to reset the pending vector's mask to 0. Make the function more specialized for that very purpose and rename it accordingly. Signed-off-by: Frederic Weisbecker --- include/linux/interrupt.h | 2 +- kernel/softirq.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h index bf82980f569d..2099fe3980bc 100644 --- a/include/linux/interrupt.h +++ b/include/linux/interrupt.h @@ -524,7 +524,7 @@ DECLARE_STATIC_KEY_FALSE(force_irqthreads_key); #endif #define local_softirq_pending() (__this_cpu_read(local_softirq_pending_ref)) -#define set_softirq_pending(x) (__this_cpu_write(local_softirq_pending_ref, (x))) +#define reset_softirq_pending() (__this_cpu_write(local_softirq_pending_ref, 0)) #define or_softirq_pending(x) (__this_cpu_or(local_softirq_pending_ref, (x))) #endif /* local_softirq_pending */ diff --git a/kernel/softirq.c b/kernel/softirq.c index 623985f18833..1a3c3fe341ea 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -532,7 +532,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) restart: /* Reset the pending bitmask before enabling irqs */ - set_softirq_pending(0); + reset_softirq_pending(); local_irq_enable(); From patchwork Tue Aug 1 13:24:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 129299 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp2673540vqg; Tue, 1 Aug 2023 06:35:35 -0700 (PDT) X-Google-Smtp-Source: APBJJlHa5cGao7/FuMkY6DlxOYxxMzqeXDzP8SsEGlpXskXnTLnW919p5UXKWg5FuSd2Q3GVrFPK X-Received: by 2002:a05:6402:2054:b0:522:af5c:ad20 with SMTP id bc20-20020a056402205400b00522af5cad20mr2748337edb.1.1690896935285; Tue, 01 Aug 2023 06:35:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690896935; cv=none; d=google.com; s=arc-20160816; b=EOHTwYIYdI/Hw7djv0tMDQczqOXDlieBntYLHWNldLxwrJl26gRrmvaecgT6q6fnsO DSzYoaJhbJIpDmgmL9+zvEBZNO5FSu6xDvtmcl8ENs7K8KApZ8MOJ18MzsAnXhzx7YQB QUc/Flp9tItS/du6YCvFRftPCDnUXv9WAz04xP5mITqP8Qny8shc11pHlt9uwSQC9uin UKtXDxPhNrQsZsLzVAUWrdb/3hjYVDMGGxaKCFSiTXFlQx4a16c/z/ZnkxQsoUhCgvFY sgf7r7+WZErb1KyAy5i/Mjcl4+Ygq0if7dLT8H8bZIe6+iARNSu/h+hmrpCHOZwU/eQw +hKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5b3us94WbDx9mKfVF8QwSdN8s8lw+h1JzqD7Qo/ycak=; fh=JC0FP1KBPXa+8NkvIvZoCUIIa+zGdCFP1xCgvJrJBSE=; b=wXl0MN9iI1NjnVtmwzN37QxykQU3xdU+WFIr4Wd1UVf7HlOWR5fquv0UHJSAjz1sRl wdq/3zm1Z+BTTkDqEjEDpLzc84qUu448BacG8qXVqwFEuiH66I6sYh1yROzCvAQZqC95 gyzr9TFJpSmbY2kRlhu79S9GEdBuE/WoLXy7gZ/5qtWFqZ6BRUxCA91eQ2vOG9hYk/iB 52gwkIHhme+kzGBZya1LDjo8pwdA95leOdUfWS0e7c4xyPE3CKJGn4FVol992xqQmAaq Xss3opx+Bd7lVdewEhOk5kE3Ba7YL4h9XbvxpG2JWnnkv318HnsGmqorNQJD+mXMs2f8 qH/g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=DivxzMVQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j6-20020aa7c0c6000000b0052280c939fasi8705007edp.481.2023.08.01.06.35.10; Tue, 01 Aug 2023 06:35:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=DivxzMVQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234111AbjHANY5 (ORCPT + 99 others); Tue, 1 Aug 2023 09:24:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36826 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234085AbjHANYz (ORCPT ); Tue, 1 Aug 2023 09:24:55 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E5EE198D for ; Tue, 1 Aug 2023 06:24:53 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3CD1661594 for ; Tue, 1 Aug 2023 13:24:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AEFEDC433CA; Tue, 1 Aug 2023 13:24:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690896292; bh=vb0eW648+k9SO6deOUTYMYcSVwD0/SVM0GpxXUYmFVs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DivxzMVQpdG0eYF91RGmaydcwUI6rJqX/Eh1Q2gJdYvQasfpLmBSaExyfOVmDFC/b vTywotqFfXyX6HJyrvhKIrH95pEym5ZDYyujswn+zCZqtV5e726DqIs13NQuU9VAOK RYjntqOs3CqvUP6yDz2oFWbAXSIXBZxtmFyYBo0rzMJVA+o5KikYNYm7fiqfIiU6sG Nai3Tmk+XjdQzGejIVIu6VA3m/Yw1Q0IcKgWx7LW8Ncm9xcaUY5LtB5jZnzpk9h30q ruMyRtUdRgRV0vDp2UaVRvcO67ZkPCdSnxa1CZJ24yPF9vupAlpCkmmqa6zDmX991f yw3xq1ru+dgBQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Sebastian Andrzej Siewior , Peter Zijlstra , Thomas Gleixner , "Paul E . McKenney" , Linus Torvalds , Anna-Maria Behnsen , Eric Dumazet Subject: [RFC PATCH 2/6] softirq: Make softirq handling entry/exit generally available Date: Tue, 1 Aug 2023 15:24:37 +0200 Message-Id: <20230801132441.559222-3-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230801132441.559222-1-frederic@kernel.org> References: <20230801132441.559222-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773033944777229915 X-GMAIL-MSGID: 1773033944777229915 In order to prepare for re-enabling softirqs from vector callbacks that are known safe, make the code incrementing the preempt count while servicing softirqs more generally available. No intended behaviour change. Signed-off-by: Frederic Weisbecker --- include/linux/bottom_half.h | 7 +++++++ kernel/softirq.c | 22 ++++++++++++++++------ 2 files changed, 23 insertions(+), 6 deletions(-) diff --git a/include/linux/bottom_half.h b/include/linux/bottom_half.h index fc53e0ad56d9..2243c7de4917 100644 --- a/include/linux/bottom_half.h +++ b/include/linux/bottom_half.h @@ -33,6 +33,13 @@ static inline void local_bh_enable(void) __local_bh_enable_ip(_THIS_IP_, SOFTIRQ_DISABLE_OFFSET); } +static inline void local_bh_enter(void) +{ + __local_bh_disable_ip(_RET_IP_, SOFTIRQ_OFFSET); +} + +extern void local_bh_exit(void); + #ifdef CONFIG_PREEMPT_RT extern bool local_bh_blocked(void); #else diff --git a/kernel/softirq.c b/kernel/softirq.c index 1a3c3fe341ea..ba998d572ef4 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -247,21 +247,26 @@ void __local_bh_enable_ip(unsigned long ip, unsigned int cnt) } EXPORT_SYMBOL(__local_bh_enable_ip); +inline void local_bh_exit(void) +{ + __local_bh_enable(SOFTIRQ_OFFSET, true); + WARN_ON_ONCE(in_interrupt()); +} + /* * Invoked from ksoftirqd_run() outside of the interrupt disabled section * to acquire the per CPU local lock for reentrancy protection. */ static inline void ksoftirqd_run_begin(void) { - __local_bh_disable_ip(_RET_IP_, SOFTIRQ_OFFSET); + local_bh_enter(); local_irq_disable(); } /* Counterpart to ksoftirqd_run_begin() */ static inline void ksoftirqd_run_end(void) { - __local_bh_enable(SOFTIRQ_OFFSET, true); - WARN_ON_ONCE(in_interrupt()); + local_bh_exit(); local_irq_enable(); } @@ -389,15 +394,20 @@ void __local_bh_enable_ip(unsigned long ip, unsigned int cnt) } EXPORT_SYMBOL(__local_bh_enable_ip); +inline void local_bh_exit(void) +{ + __local_bh_enable(SOFTIRQ_OFFSET); + WARN_ON_ONCE(in_interrupt()); +} + static inline void softirq_handle_begin(void) { - __local_bh_disable_ip(_RET_IP_, SOFTIRQ_OFFSET); + local_bh_enter(); } static inline void softirq_handle_end(void) { - __local_bh_enable(SOFTIRQ_OFFSET); - WARN_ON_ONCE(in_interrupt()); + local_bh_exit(); } static inline void ksoftirqd_run_begin(void) From patchwork Tue Aug 1 13:24:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 129318 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp2685783vqg; Tue, 1 Aug 2023 06:58:25 -0700 (PDT) X-Google-Smtp-Source: APBJJlEyd4PU48i5gTQJ/1yd8NyOsIigGTUevIzN4udOYlw5SOurwc89uVSKYjLjz0TKAC7Z5ZNt X-Received: by 2002:a2e:9b09:0:b0:2b9:e15f:e780 with SMTP id u9-20020a2e9b09000000b002b9e15fe780mr2498377lji.26.1690898305259; Tue, 01 Aug 2023 06:58:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690898305; cv=none; d=google.com; s=arc-20160816; b=vdovm1FLHVXnnKhgZnCcy0WD4qlj+Q+L7jxnqgLd0WP68ga14f7jTiatG9ak2ubDK7 TWlOTxNV8UG8oed4NMT28ztZjoGaX2FAHI8kjGZrGbSzHSg29XIfxLKzzeog8LT8Tcbs c2bid9649WD3oo1mxQ9iOse25aQ8ycMH3SCStbe/JgAvZ3Y948BbkkinNXU/bD1BM5xe v4fIiq1v6A6abvkkgNAu083keWKGPa8vYl7piwxTaXNuP90nv/WkPKE8owFuqZ7a/65d lhfZlR0E2tQdk4yorvtImyI2JajPN2eLV/r9DzX1WNuCfUcPjkWNrFxvhQC2HxuZ/UXj Ra9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3e5bEHj6eCalFVH5s6LckEauR/VK5dtIa3XF/uNrtMs=; fh=JC0FP1KBPXa+8NkvIvZoCUIIa+zGdCFP1xCgvJrJBSE=; b=iPGIBoyUJxxbF+ek9a07N/yvsknIiiG3/huBIKo1/NomZZdPuMBQXJkHdoNWy4W6fU SFmMGsz5hFRAqbuy7da1qzBv4b5LQ/BGDBXYWcu0EbaWHABgZ28rPFm7MsBBAMCeflBJ 1Ywc/AURjMFkApRxyMKEu05S4EUShYedMwlHulklb0c/dmrGep08ir9kglnA9lzvmalT xAd2HMwVo6IimAUwqbpKawE1JgE2JhREUWMKddfVm7PIZdVnx3VYoyO88IVyoxURCzfn gkeMRhnRovLLJsamd1hpRkUhgIUQMxfgbG0kwT0e5GfeADGnwnRVaekMoSWgSURQo9JN VAvg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=hJ1Vvodx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i3-20020a170906850300b0099bcbb766dbsi4301304ejx.780.2023.08.01.06.58.01; Tue, 01 Aug 2023 06:58:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=hJ1Vvodx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234167AbjHANZF (ORCPT + 99 others); Tue, 1 Aug 2023 09:25:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36878 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234114AbjHANY5 (ORCPT ); Tue, 1 Aug 2023 09:24:57 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0AF091985 for ; Tue, 1 Aug 2023 06:24:56 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9DD4F61586 for ; Tue, 1 Aug 2023 13:24:55 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1B81DC433C8; Tue, 1 Aug 2023 13:24:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690896295; bh=feqQBtIQhDcLNaK5yV5c0SoBbM4IgwK6op+QnaFp2wY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hJ1VvodxVi5EP11Bjtgw6Oj0x5oqLYcCDP6Jfz/HKcjqeloi1SlufqjWZ14f0Wpco 1AUGhZ2yRxdEFjieO2AJkIxl1upO9jbBluXpIcvFnyNSe3D1sPqeiVTFtlraJPeRFj UYCyv4B8zEjo13yezuB34uFq8Q+CNAtxh4bV104867QpLikgOnv2FzxpKbB/vYIg7j R2Hez7VsHhP1DtV/hANNbPwxjqTvqa+nse/MjQWu1+GMAMSF1Ta8dARfDl2oNBoLdv ZTEEPOXW0M/ww/bfXlau7+mEp6IPb/yVTo8/xIQlA78omsqY7eZBrhhWolq81ik3rN pIXQiO8N55kVA== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Sebastian Andrzej Siewior , Peter Zijlstra , Thomas Gleixner , "Paul E . McKenney" , Linus Torvalds , Anna-Maria Behnsen , Eric Dumazet Subject: [RFC PATCH 3/6] softirq: Introduce softirq disabled mask Date: Tue, 1 Aug 2023 15:24:38 +0200 Message-Id: <20230801132441.559222-4-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230801132441.559222-1-frederic@kernel.org> References: <20230801132441.559222-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773035381362244986 X-GMAIL-MSGID: 1773035381362244986 (DISCLAIMER: contains -RT bits) When softirq vectors will be able to re-enable softirqs when deemed safe, for example when a timer callback is tagged as soft-interruptible by other softirq vectors, care must be taken to ensure a given vector is not re-entrant. Ie: a vector can be interrupted by others but not by itself. In order to prepare for this, introduce a softirq disabled mask so that vectors can disable themselves before re-enabling softirqs. Signed-off-by: Frederic Weisbecker --- arch/Kconfig | 3 +++ include/linux/bottom_half.h | 2 ++ include/linux/interrupt.h | 15 ++++++++++++--- kernel/softirq.c | 16 +++++++++++++++- 4 files changed, 32 insertions(+), 4 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 205fd23e0cad..d23968860ddf 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1358,6 +1358,9 @@ config RELR config ARCH_HAS_MEM_ENCRYPT bool +config ARCH_HAS_SOFTIRQ_DISABLED_MASK + bool + config ARCH_HAS_CC_PLATFORM bool diff --git a/include/linux/bottom_half.h b/include/linux/bottom_half.h index 2243c7de4917..d5b37b580c79 100644 --- a/include/linux/bottom_half.h +++ b/include/linux/bottom_half.h @@ -42,6 +42,8 @@ extern void local_bh_exit(void); #ifdef CONFIG_PREEMPT_RT extern bool local_bh_blocked(void); +extern void local_bh_vec_enable(int vec); +extern void local_bh_vec_disable(int vec); #else static inline bool local_bh_blocked(void) { return false; } #endif diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h index 2099fe3980bc..7819d16d8d6f 100644 --- a/include/linux/interrupt.h +++ b/include/linux/interrupt.h @@ -523,8 +523,16 @@ DECLARE_STATIC_KEY_FALSE(force_irqthreads_key); #define local_softirq_pending_ref irq_stat.__softirq_pending #endif -#define local_softirq_pending() (__this_cpu_read(local_softirq_pending_ref)) -#define reset_softirq_pending() (__this_cpu_write(local_softirq_pending_ref, 0)) +#if defined(CONFIG_PREEMPT_RT) && defined(CONFIG_ARCH_HAS_SOFTIRQ_DISABLED_MASK) +#define local_softirq_disabled() (__this_cpu_read(local_softirq_disabled_ref)) +#else +#define local_softirq_disabled() (0) +#endif + +#define local_softirq_pending() (__this_cpu_read(local_softirq_pending_ref) & \ + ~local_softirq_disabled()) +#define reset_softirq_pending() (__this_cpu_and(local_softirq_pending_ref, \ + local_softirq_disabled())) #define or_softirq_pending(x) (__this_cpu_or(local_softirq_pending_ref, (x))) #endif /* local_softirq_pending */ @@ -614,7 +622,8 @@ extern void raise_hrtimer_softirq(void); static inline unsigned int local_pending_timers(void) { - return __this_cpu_read(pending_timer_softirq); + return __this_cpu_read(pending_timer_softirq) & + ~local_softirq_disabled(); } #else diff --git a/kernel/softirq.c b/kernel/softirq.c index ba998d572ef4..a394f78de627 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -297,6 +297,18 @@ void do_softirq_post_smp_call_flush(unsigned int was_pending) invoke_softirq(); } +#ifdef CONFIG_ARCH_HAS_SOFTIRQ_DISABLED_MASK +void local_bh_vec_enable(int vec) +{ + __this_cpu_and(local_softirq_disabled_ref, ~vec); +} + +void local_bh_vec_disable(int vec) +{ + __this_cpu_or(local_softirq_disabled_ref, vec); +} +#endif + #else /* CONFIG_PREEMPT_RT */ /* @@ -1009,11 +1021,13 @@ static int timersd_should_run(unsigned int cpu) static void run_timersd(unsigned int cpu) { unsigned int timer_si; + unsigned long timersd_vecs = (1 << TIMER_SOFTIRQ) | (1 << HRTIMER_SOFTIRQ); ksoftirqd_run_begin(); timer_si = local_pending_timers(); - __this_cpu_write(pending_timer_softirq, 0); + __this_cpu_and(pending_timer_softirq, + local_softirq_disabled() & timersd_vecs); or_softirq_pending(timer_si); __do_softirq(); From patchwork Tue Aug 1 13:24:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 129300 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp2673637vqg; Tue, 1 Aug 2023 06:35:45 -0700 (PDT) X-Google-Smtp-Source: APBJJlEvuIVI3J9lcC5wN/i4r1zmdqKF9yE/+9CAMXlwvQWXBTx9rVtdwaaRd+C6/66k+XIg8fuf X-Received: by 2002:a05:6512:2388:b0:4fe:676:8c0b with SMTP id c8-20020a056512238800b004fe06768c0bmr3029441lfv.11.1690896944651; Tue, 01 Aug 2023 06:35:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690896944; cv=none; d=google.com; s=arc-20160816; b=ZQjNfiDvPSi6cXaKyWAp2c+yGYJg9SJU77Mpj5j7hpdaYF9N/D7G4MxLK4Pqawt0bF vIcNt9kDum8pZnLj/9ELhO+lDQk8XN1K2vEVArYoYZatiENk9rmcnPcAarrC73a9S+K4 mxqD8ZbeAYE0ITgVQCpQqM8AHK8BuztpJR2yMQAoy+k/tuGCsFGrf8lwMcQat7g7xW2j QsumyBe2GUFf9ZEfUKiLxTzUOkU9JzoVfUmhmCVl3PN0+FpOsqOoxrpzViwlOm8UO5gs MMryujadUGIfOneYhrHHFmtlq6URjQztTU6+48K+qoz+8OZhTeft7Ch88Wc+WH0apJU3 EwgA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+QIYYVXUz47R2jxU+L2GU2guhnsqHmrarUFNOO0FiSI=; fh=JC0FP1KBPXa+8NkvIvZoCUIIa+zGdCFP1xCgvJrJBSE=; b=zGjTissa5v+xXoilRFlHIMuBZjyPxsIrxEH0opI4dowd7G3U0/9dP48pCbQqjHnyDq KOmcevrT76e3BLTcVi16CEo7fSjBf7I1bi7jVgSPiTb3T7ClXWbLgH8i3vxsH702tZiU KG1g3rLe8tFpuxdqwCT0bkm1H4oJI9xPhO8iWbqQQKIaIzvSaIAa4Uj1finyM5HtGDfG LzMexRhq2w7Iz0OE3frQplHTDd9dLuRYJHOQX7uoIe+VyERTX5X4URihoUt+qZve2JNF BTOcB/N5unhtf++7+hPGZPMHjE2zxRpOj+SmIhbDRAVJ9IDcwYyuEGPsKErQWZvDVj8o wEyQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=WQ8ryfpI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q22-20020a17090622d600b00992d262fb29si8980253eja.992.2023.08.01.06.35.20; Tue, 01 Aug 2023 06:35:44 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=WQ8ryfpI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234199AbjHANZJ (ORCPT + 99 others); Tue, 1 Aug 2023 09:25:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37012 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234108AbjHANZD (ORCPT ); Tue, 1 Aug 2023 09:25:03 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B379C2116 for ; Tue, 1 Aug 2023 06:24:58 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 0E5E8615A3 for ; Tue, 1 Aug 2023 13:24:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7FF58C433C7; Tue, 1 Aug 2023 13:24:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690896297; bh=kRD+z973ymfN8CUInsVsbutA6ADoO6ucBF0+2kGIUy4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WQ8ryfpIqxjOXfYyBJzKG4qNR6jyMndhOItuP0gL6Zbgx8+366OyCfJMkg5+BZMFU auaat53rdg3hW7VpAIdGX1pYYWZ4yNc/ppnGpvrqLn4F4ZP4fD1FYymoXgwah8cSMT uYWt+A+CW+hwn1wiSqnpb5qBSDAv+Ibjqz2u5kSPq5VV/H2Gl6jL7YPMeV/6AkfGuA 9XuN+ZIvzLOaoddC29YcQln5X8lCsIrY3xThY7Dd6O09l0/a3Ley0FzhflcQBXdx47 eWINZFRzrKs6nKk3kKDM9viCOE2mroMnZiqreHOWvNJwXxhQX4BTbr6y6LOTwocsVN oUDSA7fzB3cWw== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Sebastian Andrzej Siewior , Peter Zijlstra , Thomas Gleixner , "Paul E . McKenney" , Linus Torvalds , Anna-Maria Behnsen , Eric Dumazet Subject: [RFC PATCH 4/6] x86/softirq: Support softirq disabled mask Date: Tue, 1 Aug 2023 15:24:39 +0200 Message-Id: <20230801132441.559222-5-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230801132441.559222-1-frederic@kernel.org> References: <20230801132441.559222-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773033954428291368 X-GMAIL-MSGID: 1773033954428291368 Support the new softirq disabled vectors mask and put in the per-cpu hot structure along with the pending vectors mask as both are used closely together. Signed-off-by: Frederic Weisbecker --- arch/x86/Kconfig | 1 + arch/x86/include/asm/current.h | 1 + arch/x86/include/asm/hardirq.h | 1 + 3 files changed, 3 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index a2a410d13e39..f1cd68b672dc 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -93,6 +93,7 @@ config X86 select ARCH_HAS_COPY_MC if X86_64 select ARCH_HAS_SET_MEMORY select ARCH_HAS_SET_DIRECT_MAP + select ARCH_HAS_SOFTIRQ_DISABLED_MASK select ARCH_HAS_STRICT_KERNEL_RWX select ARCH_HAS_STRICT_MODULE_RWX select ARCH_HAS_SYNC_CORE_BEFORE_USERMODE diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index a1168e7b69e5..b86ca9c0ddc2 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -23,6 +23,7 @@ struct pcpu_hot { unsigned long top_of_stack; void *hardirq_stack_ptr; u16 softirq_pending; + u16 softirq_disabled; #ifdef CONFIG_X86_64 bool hardirq_stack_inuse; #else diff --git a/arch/x86/include/asm/hardirq.h b/arch/x86/include/asm/hardirq.h index 66837b8c67f1..933cf05e5738 100644 --- a/arch/x86/include/asm/hardirq.h +++ b/arch/x86/include/asm/hardirq.h @@ -61,6 +61,7 @@ extern u64 arch_irq_stat(void); #define arch_irq_stat arch_irq_stat #define local_softirq_pending_ref pcpu_hot.softirq_pending +#define local_softirq_disabled_ref pcpu_hot.softirq_disabled #if IS_ENABLED(CONFIG_KVM_INTEL) static inline void kvm_set_cpu_l1tf_flush_l1d(void) From patchwork Tue Aug 1 13:24:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 129302 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp2673748vqg; Tue, 1 Aug 2023 06:35:56 -0700 (PDT) X-Google-Smtp-Source: APBJJlGW0uLioz7WPuXM/4sh7KkJCJbz/T5kT7LT8LJ8ds+4VMtXMUhuMSv6P8gj6muxLjIP+jB5 X-Received: by 2002:a17:907:6d15:b0:99b:cadd:c2ee with SMTP id sa21-20020a1709076d1500b0099bcaddc2eemr3465552ejc.29.1690896956539; Tue, 01 Aug 2023 06:35:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690896956; cv=none; d=google.com; s=arc-20160816; b=RTAt7HFb88RqMoz0YWeKfClypuR/BfXhhbo2zJMJIGtvYVt+VYevSfuHVRDW4pWohQ bqKMvOlIkYIgRhcTtz9CTrmeCQd5RwlxDnkFA2BC7XKFpgp+njHswtOc0KSo57rxbp54 zowYNCQXntlvXDajh+nteUH0Gcb5bzC2jfEVmoXGTFd0UeWa1SOmkDVfOdmBdY58GSgP bERnoMxH1aYWGLUg07VfllFKgAg/TNKYTJIUfQpaehVcYZUtLahAhgymYY7FsPwrKsYW g6qxcq3SN3ZaPWTetqY1pOqGnbXqak03vUDB5CXBlCSxkNHimlDZcFdIYxdehJmbjVj/ Ur/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=shsHwMhe8RTZPnVpzPVtiTq2YzcNIGu9Lh7SIi2A7uQ=; fh=JC0FP1KBPXa+8NkvIvZoCUIIa+zGdCFP1xCgvJrJBSE=; b=rdnM6g/VJ7KFC4Brovc08TLT+BwdGUfomih+y1IbKXEyjBcK7RnU3RiyYRomQimzlh uN3e/3JhlW6zt52bPXCK8y9R9BqUYrxVQWHiwrUUFUriddYRXL+IAMl0OuedjK2aJalg DQX27j+WDvMestRV+cmy4/x5iVQNrHCZgG3HLiJxznBIqLq8K2B9On9cO0U7cM/FxKUe RAA6cWLUapjjzC85PE+5FscJJwso7+aePIzHKGZJ/jdxFTGBah3IRNaLaH6Bi9kAhwtT Qe02qPUYwTL0Msr1gQah9tWNLmZ7qb/slqdxHjbARzWs9YlZkARX9QAABT8Xo21UlB3a D+rQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=eg8Wsrw5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f9-20020a170906138900b00988c92d2861si9313754ejc.397.2023.08.01.06.35.32; Tue, 01 Aug 2023 06:35:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=eg8Wsrw5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234161AbjHANZM (ORCPT + 99 others); Tue, 1 Aug 2023 09:25:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37072 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233096AbjHANZF (ORCPT ); Tue, 1 Aug 2023 09:25:05 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C036A2683 for ; Tue, 1 Aug 2023 06:25:01 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6F61961594 for ; Tue, 1 Aug 2023 13:25:00 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E4767C433C9; Tue, 1 Aug 2023 13:24:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690896299; bh=rdh2v8ubgWirANjBqQeINfRDtrMu7AmbvfgkqLxk0ZQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eg8Wsrw5hhHyNuQV0Y6edeuyltWCE4/vwspiKaXOia1pjtO644tTGTaOJ4UmbAXoH rhQfXc0vC+L0z6dqLCXrZKJb5TyV0VDLXhQCSqPKPNBhlYp8od3LSXJ7ct6zYYTS+1 bMqrSBbA2Reu++Eulkkw+tVEmB0Xqdhw34BP4GGdfuO5vCW7lmb+XPjShz1/sva7Q6 ESsceLV2WEluWAt56iBfc68isurryOxCW1erhZ+g9K9C9aRh5cSAOTNBRhpKbhrH81 Na3OoRxbeq2wS9Fi3g8nX2KGaI0N8gZuSPBnVkrfDcQjgOaVrOsTq2CvukqOMM1BZh E8dBg1bEfGepQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Sebastian Andrzej Siewior , Peter Zijlstra , Thomas Gleixner , "Paul E . McKenney" , Linus Torvalds , Anna-Maria Behnsen , Eric Dumazet Subject: [RFC PATCH 5/6] timers: Introduce soft-interruptible timers Date: Tue, 1 Aug 2023 15:24:40 +0200 Message-Id: <20230801132441.559222-6-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230801132441.559222-1-frederic@kernel.org> References: <20230801132441.559222-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773033966982446946 X-GMAIL-MSGID: 1773033966982446946 Most timers are unrelated to networking or other softirq vectors (RCU, NET_*, HRTIMER, TASKLET, ...). Yet when a timer batch is executing, other softirq vectors have to wait for the timers batch completion even though there is nothing to synchronize against most callbacks. However there is no automatic way to determine if a timer callback is safely soft-interruptible by other vectors. So the only long term viable approach to solve this is to adopt a progressive push down solution similar to the one used for getting rid of the big kernel lock. Introduce a new TIMER_SOFTINTERRUPTIBLE flag which tells the timer subsystem that a callback is safely soft-interruptible by other vectors, either because it's completely unrelated to them or because it uses the appropriate local_bh_disable()/spin_lock_bh() on narrowed-down regions. Once all timers are dealt with after a few years, it will become possible to run timers out of the softirqs processing. It's worth noting though that if the softirq infrastructure supports soft-interruption of a TIMER_SOFTINTERRUPTIBLE timer, it doesn't allow yet a TIMER_SOFTINTERRUPTIBLE timer to soft-interrupt other vectors, even though nothing prevents from it to happen from a correctness point of view, more tweaks are needed to support that. Signed-off-by: Frederic Weisbecker --- include/linux/timer.h | 5 +++-- kernel/time/timer.c | 18 ++++++++++++++++++ 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/include/linux/timer.h b/include/linux/timer.h index 9162f275819a..fbe40bacc8c3 100644 --- a/include/linux/timer.h +++ b/include/linux/timer.h @@ -61,13 +61,14 @@ struct timer_list { * should be placed on a particular CPU, then add_timer_on() has to be * used. */ -#define TIMER_CPUMASK 0x0003FFFF +#define TIMER_CPUMASK 0x0001FFFF /* If 1 more bit is needed, flags must be 64 */ +#define TIMER_SOFTINTERRUPTIBLE 0x00020000 #define TIMER_MIGRATING 0x00040000 #define TIMER_BASEMASK (TIMER_CPUMASK | TIMER_MIGRATING) #define TIMER_DEFERRABLE 0x00080000 #define TIMER_PINNED 0x00100000 #define TIMER_IRQSAFE 0x00200000 -#define TIMER_INIT_FLAGS (TIMER_DEFERRABLE | TIMER_PINNED | TIMER_IRQSAFE) +#define TIMER_INIT_FLAGS (TIMER_SOFTINTERRUPTIBLE | TIMER_DEFERRABLE | TIMER_PINNED | TIMER_IRQSAFE) #define TIMER_ARRAYSHIFT 22 #define TIMER_ARRAYMASK 0xFFC00000 diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 7cad6fe3c035..1e43f54def0e 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -1676,6 +1676,7 @@ static void call_timer_fn(struct timer_list *timer, unsigned long baseclk) { int count = preempt_count(); + bool softinterruptible = false; #ifdef CONFIG_LOCKDEP /* @@ -1689,6 +1690,17 @@ static void call_timer_fn(struct timer_list *timer, lockdep_copy_map(&lockdep_map, &timer->lockdep_map); #endif + + if (IS_ENABLED(CONFIG_PREEMPT_RT) && + IS_ENABLED(CONFIG_ARCH_HAS_SOFTIRQ_DISABLED_MASK) && + timer->flags & TIMER_SOFTINTERRUPTIBLE) + softinterruptible = true; + + if (softinterruptible) { + local_bh_vec_disable(1 << TIMER_SOFTIRQ); + local_bh_exit(); + } + /* * Couple the lock chain with the lock chain at * timer_delete_sync() by acquiring the lock_map around the fn() @@ -1702,6 +1714,12 @@ static void call_timer_fn(struct timer_list *timer, lock_map_release(&lockdep_map); + if (softinterruptible) { + local_bh_enter(); + local_bh_vec_enable(1 << TIMER_SOFTIRQ); + } + + if (count != preempt_count()) { WARN_ONCE(1, "timer: %pS preempt leak: %08x -> %08x\n", fn, count, preempt_count()); From patchwork Tue Aug 1 13:24:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 129301 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp2673749vqg; Tue, 1 Aug 2023 06:35:57 -0700 (PDT) X-Google-Smtp-Source: APBJJlEgzpuaTuxxGmChCWsKAlsmne/qt9EVX38G/krecvEf7ZluYt1j4CpT807T4Gi/o1VuavvV X-Received: by 2002:aa7:c753:0:b0:522:1eab:e466 with SMTP id c19-20020aa7c753000000b005221eabe466mr2174113eds.28.1690896956693; Tue, 01 Aug 2023 06:35:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690896956; cv=none; d=google.com; s=arc-20160816; b=m+1qWw9VtQfpZthyxtuOznANJtEllnBzIj+gsCn8n4bSLDEC8DXeKHSjnHt/InXxRF gBHgIPN+cZlLQYgwFL4fE1dDwXNpgyn9ZaHMz8WipwVaKL6I/YavxydGpXd1K+5n4/MG Vb6io1OvNdTNJqAg/DQC9+S7iDbzwB0urbcLA1M/+4BEbU8Q6UfvLIBk/eE7QeerJeW/ HQZBMG9XxmWyZcerfgeJaVrDN6jFys0DlzWYEdh3Ror7KvSEBLr9Q/TCcdBu0w6YA4yF 04jv8Hdz/L65UW8uu2MNLimNxmfoOCWWWfzkR50LAUAZoPraWKfafes7m6NmnLd0kHfs vsqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=AihPiO106GfLqvjzfFAw8wCMzrvalBb7/AaeBIaji7c=; fh=JC0FP1KBPXa+8NkvIvZoCUIIa+zGdCFP1xCgvJrJBSE=; b=HZrkL6XMr85RICmyzUXfKXqOdQDx2KDeOyUSOcbJwG80Q2JQp1Q1r7rQvXWfkRBKao hizSF4zGHBcHjxC7pxgz/V4try8+RKQrhZp7TapuZTStKMvJDRZpvxmyqTaLO0ZD8WCv sNsTDbv+0YSQN3WmDzzza+uHKCHG/Zm/ZwqixMX6WpDIuF1EbsIPUwRVJ7GmaMxhzOsh iIruwj06BsE8/Egkruswc5/VaKoRd5362U1hydnejqz9S87gcLsYLzzdPbD8fIEKvI6m YMShcUL1VpdKS+C0AqOYDe0W6cz+F3UmxlnkPoOweBZKvYEFNvOFEMJfxKDKlCdrnciD kwkQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=lnp7XiqA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k24-20020a05640212d800b00522ba992d67si4490737edx.30.2023.08.01.06.35.32; Tue, 01 Aug 2023 06:35:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=lnp7XiqA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234235AbjHANZT (ORCPT + 99 others); Tue, 1 Aug 2023 09:25:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37154 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234187AbjHANZI (ORCPT ); Tue, 1 Aug 2023 09:25:08 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 36825269F for ; Tue, 1 Aug 2023 06:25:03 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id E1624615A3 for ; Tue, 1 Aug 2023 13:25:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 509D6C433C8; Tue, 1 Aug 2023 13:25:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1690896302; bh=rciJFUiyVsvhXbHYtyfujY9ldyQ8u3jv3Unlkm5oJ8Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lnp7XiqARKpzKndW55oU0pFk++KVe7qMUYvK5aKtl891T2uF5+HHyVOFY++Pfj7cb 9hIF71vVnTxKoWv8urpMEhOPN9X+JwnUUTcJ0iKi2dnt6OrQecFIZq/jtkqDOHeIRx 0B6hJ1R3KNBOfN8wHxkzYjsgIXm2rGyxDV2qS5hr6qHZ6cn+mg7ijE4vbVyFziZpGJ 63bdbmru35c+St1REUh6OG8y0Z4iaOlXWpJvmsgypEio81WHULIM+b0B8qX94ygl7H 2ekN/heycVbwDUdNup6qsP/UYHcyO3HEwuovJ3oE71r8F4iINoP3dB1axgSedSDqPd usHOdgqfR2MhA== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Sebastian Andrzej Siewior , Peter Zijlstra , Thomas Gleixner , "Paul E . McKenney" , Linus Torvalds , Anna-Maria Behnsen , Eric Dumazet Subject: [RFC PATCH 6/6] timers: Make process_timeout() soft-interruptible Date: Tue, 1 Aug 2023 15:24:41 +0200 Message-Id: <20230801132441.559222-7-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230801132441.559222-1-frederic@kernel.org> References: <20230801132441.559222-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773033967094323538 X-GMAIL-MSGID: 1773033967094323538 The most frequent timer can be safely interrupted by other vectors as it's only performing a wake-up. Signed-off-by: Frederic Weisbecker --- kernel/time/timer.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/time/timer.c b/kernel/time/timer.c index 1e43f54def0e..0f6eb9c0901c 100644 --- a/kernel/time/timer.c +++ b/kernel/time/timer.c @@ -2180,7 +2180,7 @@ signed long __sched schedule_timeout(signed long timeout) expire = timeout + jiffies; timer.task = current; - timer_setup_on_stack(&timer.timer, process_timeout, 0); + timer_setup_on_stack(&timer.timer, process_timeout, TIMER_SOFTINTERRUPTIBLE); __mod_timer(&timer.timer, expire, MOD_TIMER_NOTPENDING); schedule(); del_timer_sync(&timer.timer);