From patchwork Wed May 10 16:58:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 92226 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3781566vqo; Wed, 10 May 2023 10:07:00 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5SXadOCeHXJM7ry+fghi3uLMZmHcdlgU64uj1DlboMgNVaBHINZTr+1gkFnbOR5rdD8Mkl X-Received: by 2002:a17:903:2782:b0:19f:8ad5:4331 with SMTP id jw2-20020a170903278200b0019f8ad54331mr17476981plb.38.1683738420645; Wed, 10 May 2023 10:07:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683738420; cv=none; d=google.com; s=arc-20160816; b=fDnv/oJ7AtiKciQE6DfSuAq4tCVUMEb5azs3rVeJUiGIk/Ej44Kf1aigbb976//phG a7RdlBsTPlAn35wswcvrq9NYEaxhtHfz/2Qvh04ieu0mOT2PUVJF+6k6q9ksSp8UuqDh 2/9vhpmRPqN0dvF056DmIjXurAAo9V8gQJFNS0MoeX/kk3VD6MkVcy+V2GzJpsd0BjSt 8Pwz+X9ucMMn9qh9XDmMgYF768TulSURWv+8riGfUfaTMZ85xb4UPIZHYDI6DyfAlYhB wNYhpusS+4f6RFfYX0Vuzmv0BK2Q8KVnP4/KgTWGumdN7Ai+6dl0aVpD0Azp9xFixgym PLrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=wbD3FdmvSLLwy2D44F6VTd7R/K8bDFpWHRcZ6mMTOmg=; b=omis9+0vCBxtaqke/sfou5mnlTnOSYL/1ox0JGDnXr1ScwfQzr+O/O2QRESNU7sY69 ugaZ4yAMe0J9GttJ3u0QeYsP8RGpJyZdCabuIzwgMm6A7oYg0nIWZA0XKLiuxuz+7SU9 B4wstQLDPDot6L4cTjmMvaemVOfweknY41DwPa77X0U1FbM3DraYK/Hqo6acsNJ63g9X gQegIX6/M2/cNZd+PQN05LmrW+ftkAfoM7WVMKW7FSpZg4QM/ufMOKT9xp64Xcoh/zM+ 3p2nhlNOyN+5VS6hHjXwWJuI6znNOWLpWFlyOWG5RU5dtab8lacISWpuFwzzpH4MPixf favw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=K4I+TNIM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g22-20020a1709029f9600b001a1f70cc7dbsi4193867plq.562.2023.05.10.10.06.31; Wed, 10 May 2023 10:07:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=K4I+TNIM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236072AbjEJQ6h (ORCPT + 99 others); Wed, 10 May 2023 12:58:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236017AbjEJQ6f (ORCPT ); Wed, 10 May 2023 12:58:35 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2EEB9C; Wed, 10 May 2023 09:58:34 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6E92F63F2F; Wed, 10 May 2023 16:58:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CDFD6C433EF; Wed, 10 May 2023 16:58:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683737913; bh=w0ne2UEEwb1HXO/hog3r2ij7PcPs9e+mZYY78+weSf8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=K4I+TNIM/LFj0At87OTvDGe8F5jYXUYn3932b/WhEKaSwLLJdqJclULahRC58qInM G8Qns7WhbE1BHPRwTj/PI+S86sIp/Q1bS+us7JGUMvep2l0JLsg3ls0C4pcDphzv5N 4OVCT+Tf974fkEOcl+fU8awmV0wtt/X6c7p0N9/PVF85nvGLRh7VT5T0CUYqQmI64p 7X8Sd5+Xl5ijfAXbNWvul8RiB0rfQgaAe/RiOhgsV7RUnQBTDx/xR/9mXxB+jK61c5 gbZmFZhon1pUyNPZNNXp4chw8+dZuc+Fpwz9uEBhqm9otPb3STjn6fr1wvkkuZRxjE dOsnidVPjUMOA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8856CCE126E; Wed, 10 May 2023 09:58:33 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Christoph Hellwig , Sachin Sant , "Zhang, Qiang1" Subject: [PATCH rcu 1/8] srcu: Remove extraneous parentheses from srcu_read_lock() etc. Date: Wed, 10 May 2023 09:58:25 -0700 Message-Id: <20230510165832.2187453-1-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765527698044161835?= X-GMAIL-MSGID: =?utf-8?q?1765527698044161835?= This commit removes extraneous parentheses from srcu_read_lock(), srcu_read_lock_nmisafe(), srcu_read_unlock(), and srcu_read_unlock_nmisafe(). Looks like someone was once a macro. Cc: Christoph Hellwig Tested-by: Sachin Sant Tested-by: "Zhang, Qiang1" Signed-off-by: Paul E. McKenney --- include/linux/srcu.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 41c4b26fb1c1..eb92a50a4599 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -212,7 +212,7 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp) srcu_check_nmi_safety(ssp, false); retval = __srcu_read_lock(ssp); - srcu_lock_acquire(&(ssp)->dep_map); + srcu_lock_acquire(&ssp->dep_map); return retval; } @@ -229,7 +229,7 @@ static inline int srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp srcu_check_nmi_safety(ssp, true); retval = __srcu_read_lock_nmisafe(ssp); - rcu_lock_acquire(&(ssp)->dep_map); + rcu_lock_acquire(&ssp->dep_map); return retval; } @@ -284,7 +284,7 @@ static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx) { WARN_ON_ONCE(idx & ~0x1); srcu_check_nmi_safety(ssp, false); - srcu_lock_release(&(ssp)->dep_map); + srcu_lock_release(&ssp->dep_map); __srcu_read_unlock(ssp, idx); } @@ -300,7 +300,7 @@ static inline void srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) { WARN_ON_ONCE(idx & ~0x1); srcu_check_nmi_safety(ssp, true); - rcu_lock_release(&(ssp)->dep_map); + rcu_lock_release(&ssp->dep_map); __srcu_read_unlock_nmisafe(ssp, idx); } From patchwork Wed May 10 16:58:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 92221 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3775145vqo; Wed, 10 May 2023 10:00:01 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5/YkpxpUfa+wfZHb3aLcJ/FsO1yUDC77nOYZ/8mn6248sCegmp9cXHqovq5FoHuYHlq6fQ X-Received: by 2002:a05:6a20:8412:b0:101:7e8e:445d with SMTP id c18-20020a056a20841200b001017e8e445dmr6940362pzd.19.1683738001424; Wed, 10 May 2023 10:00:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683738001; cv=none; d=google.com; s=arc-20160816; b=ok1kmqfe8bFnFvBffXS59R3NWq0EXXwUjKr09FxyC+CuU/R201xtejkMhgmRHNrXw8 udO27RdUvTvLZRiuhGkrCmiOmwbJMDDnj4SlARxJ+WAch3+C8+MTTBBwG/lFMK4TRSjG wo4JEJwFZaARhlRoB/Q6Ve6o7zidpz+n8og/LY0U6SFE97aWjbieqvbyMkOcSv8csqh9 mf30ABf2+swSY8lnIq2WzCruFGQ3tBBZXuKyrlw1ehh9Xbfn7X86nR8JvWvyvJxXrquY ypOpO007uD6b5bHe1GKb+lEWuKL7231T0vLwwos1/92Eh1f1rMUAM0AnZK+63OfwOBuq s14w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=dWt94A0aN8erkD5V9cBUYF7l5i9gxyZwKWXzYj7Ae58=; b=g7USxIRrjkHlzhWXGPXKd4Bgk69WLj9Dj6R037aDWzYelllKncpBUX2kOVXmAmTOuO bwtgYDUb/4e+DHhcT1RuktokShcmMMNnHjOIg0I01wZae+WL/2AmWA5EfCuF7BKDh55e /ppRZejVBUo513u1JjZIEgSt4Eaa+RgIqAPaF2DBOEatLVfxyPxuX0v0ZT9E+gX9++Vh dzkdKkaKRgAf7WyeeMqGGiRil7aw84kSVXU4hD+cRl6YoMY5Dk5IPUMF6vcSrto04RCs UOmzvmAu4UyQmBLG25uI8qdztDsh7NKeV8LIcYBEq+iCev4DpUixIh9xyhh3amZaSNCa MzWw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=QbxQMV2l; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l72-20020a63914b000000b0052c5e4694d1si4450586pge.439.2023.05.10.09.59.48; Wed, 10 May 2023 10:00:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=QbxQMV2l; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229732AbjEJQ6m (ORCPT + 99 others); Wed, 10 May 2023 12:58:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39830 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236042AbjEJQ6g (ORCPT ); Wed, 10 May 2023 12:58:36 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 00B67DF; Wed, 10 May 2023 09:58:34 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8C4D963F3B; Wed, 10 May 2023 16:58:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EB841C4339B; Wed, 10 May 2023 16:58:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683737914; bh=DvqudYdWdeAAC6Fv9v/alqS+NZUrTSGbZqd+38O62+I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QbxQMV2lIShPlKpUOibNRU79u56DsG+OcHqVlC5JEv3Nj/5G/sif5bPc20LyOSxTx iFyGnsNO5VKyL26ZhdhvptqHTR718DbH0DUxgbw6/0ECzpuBgbmCgyQEoHhDvLbEFJ pgcX/PxCGB3uTc3K8SdOUA/Bu3N9Vwr95FaBcPrLRGt0Qh/9OFPzDqs9qNGxiVWxVK omaJ8XPmRAPmWpQ5kNPvXsLtZWu18v74M3sFJYFTjtbsw9lXnpMGbPG12j2HFL2kW0 Ea/VhYsq+yZrFoh1sexh0ZOKx4DawQEhYUpZxp0CR2DaUHxm9/fRU5lX+imgz7FWR9 541j+E34bcF7A== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8B0F2CE126D; Wed, 10 May 2023 09:58:33 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Peter Zijlstra , Mark Rutland , Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH rcu 2/8] rcu: Remove RCU_NONIDLE() Date: Wed, 10 May 2023 09:58:26 -0700 Message-Id: <20230510165832.2187453-2-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765527258203297644?= X-GMAIL-MSGID: =?utf-8?q?1765527258203297644?= From: Peter Zijlstra Since there are now exactly _zero_ users of RCU_NONIDLE(), make it go away before someone else decides to (ab)use it. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Mark Rutland Acked-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- .../RCU/Design/Requirements/Requirements.rst | 36 +------------------ Documentation/RCU/whatisRCU.rst | 1 - include/linux/rcupdate.h | 25 ------------- 3 files changed, 1 insertion(+), 61 deletions(-) diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Documentation/RCU/Design/Requirements/Requirements.rst index 49387d823619..77155b51d4c2 100644 --- a/Documentation/RCU/Design/Requirements/Requirements.rst +++ b/Documentation/RCU/Design/Requirements/Requirements.rst @@ -2071,41 +2071,7 @@ call. Because RCU avoids interrupting idle CPUs, it is illegal to execute an RCU read-side critical section on an idle CPU. (Kernels built with -``CONFIG_PROVE_RCU=y`` will splat if you try it.) The RCU_NONIDLE() -macro and ``_rcuidle`` event tracing is provided to work around this -restriction. In addition, rcu_is_watching() may be used to test -whether or not it is currently legal to run RCU read-side critical -sections on this CPU. I learned of the need for diagnostics on the one -hand and RCU_NONIDLE() on the other while inspecting idle-loop code. -Steven Rostedt supplied ``_rcuidle`` event tracing, which is used quite -heavily in the idle loop. However, there are some restrictions on the -code placed within RCU_NONIDLE(): - -#. Blocking is prohibited. In practice, this is not a serious - restriction given that idle tasks are prohibited from blocking to - begin with. -#. Although nesting RCU_NONIDLE() is permitted, they cannot nest - indefinitely deeply. However, given that they can be nested on the - order of a million deep, even on 32-bit systems, this should not be a - serious restriction. This nesting limit would probably be reached - long after the compiler OOMed or the stack overflowed. -#. Any code path that enters RCU_NONIDLE() must sequence out of that - same RCU_NONIDLE(). For example, the following is grossly - illegal: - - :: - - 1 RCU_NONIDLE({ - 2 do_something(); - 3 goto bad_idea; /* BUG!!! */ - 4 do_something_else();}); - 5 bad_idea: - - - It is just as illegal to transfer control into the middle of - RCU_NONIDLE()'s argument. Yes, in theory, you could transfer in - as long as you also transferred out, but in practice you could also - expect to get sharply worded review comments. +``CONFIG_PROVE_RCU=y`` will splat if you try it.) It is similarly socially unacceptable to interrupt an ``nohz_full`` CPU running in userspace. RCU must therefore track ``nohz_full`` userspace diff --git a/Documentation/RCU/whatisRCU.rst b/Documentation/RCU/whatisRCU.rst index 8eddef28d3a1..e488c8e557a9 100644 --- a/Documentation/RCU/whatisRCU.rst +++ b/Documentation/RCU/whatisRCU.rst @@ -1117,7 +1117,6 @@ All: lockdep-checked RCU utility APIs:: RCU_LOCKDEP_WARN rcu_sleep_check - RCU_NONIDLE All: Unchecked RCU-protected pointer access:: diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index dcd2cf1e8326..aae31a3e28dd 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -156,31 +156,6 @@ static inline int rcu_nocb_cpu_deoffload(int cpu) { return 0; } static inline void rcu_nocb_flush_deferred_wakeup(void) { } #endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */ -/** - * RCU_NONIDLE - Indicate idle-loop code that needs RCU readers - * @a: Code that RCU needs to pay attention to. - * - * RCU read-side critical sections are forbidden in the inner idle loop, - * that is, between the ct_idle_enter() and the ct_idle_exit() -- RCU - * will happily ignore any such read-side critical sections. However, - * things like powertop need tracepoints in the inner idle loop. - * - * This macro provides the way out: RCU_NONIDLE(do_something_with_RCU()) - * will tell RCU that it needs to pay attention, invoke its argument - * (in this example, calling the do_something_with_RCU() function), - * and then tell RCU to go back to ignoring this CPU. It is permissible - * to nest RCU_NONIDLE() wrappers, but not indefinitely (but the limit is - * on the order of a million or so, even on 32-bit systems). It is - * not legal to block within RCU_NONIDLE(), nor is it permissible to - * transfer control either into or out of RCU_NONIDLE()'s statement. - */ -#define RCU_NONIDLE(a) \ - do { \ - ct_irq_enter_irqson(); \ - do { a; } while (0); \ - ct_irq_exit_irqson(); \ - } while (0) - /* * Note a quasi-voluntary context switch for RCU-tasks's benefit. * This is a macro rather than an inline function to avoid #include hell. From patchwork Wed May 10 16:58:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 92220 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3775131vqo; Wed, 10 May 2023 10:00:01 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5iaYPYCxbHX16j7FSkMT5ertu0aGKGncliGej7dyqYXLyaN47aS5pWMni+SoZHjZ3u6npG X-Received: by 2002:a17:903:32c7:b0:1ac:820e:c348 with SMTP id i7-20020a17090332c700b001ac820ec348mr13709197plr.0.1683738000731; Wed, 10 May 2023 10:00:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683738000; cv=none; d=google.com; s=arc-20160816; b=JpCdz8D7Sn1W33+Cpq3ppx61flCgNubfl60rqWYpNevWP6F+3LOs2CsjNO81Npz0rw +Vw7ygxZuWAmdJTTEwh9S3tRnJE8Ydw77qLfk68sPEnOwVa3jywosiwc0BzstORKagM4 4TDoI2nV5sXcsqdl0HznrIKttLH/wKlOxX1ebYVYGbzgqK1zCc4wWMRx435Zv6bo+LjU xzlhkWvBCBAPVfaiG/w+qQray4si/d8SpEkcPltRDD4RqrhMydu5Wh5OWZ83+fpm1SIi cBQhYU2BZEwqu2+C4hGVal1av7doxppe4P7SHUNeGDyCbWjedYdIMm/JRIgERUnONMtz Y3vA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=VPFs5Jy/DwCLiaJuvSjEb1AbxP9bne6MUstKvT+nPD4=; b=r9RKx9bD0gErFK/smMEMs2GYI8KANYwNpTyR1KAyrHkZErwEUrH6iC0q9LGlrMcSyX PtSjwU2QcL2Id+mNzWaOZjY4glXNEE6oTTo0dCQ8ha7hmvtbvCiWO0/2cRbmhNOIWR11 TdmvGsrICHNfSIk9NXhgTzm11qYDc/3wNTfl+SWeGZfaBHxa8VO21JYOhVReLCL6EKYZ PVbLUHhGbzqX6gqvNVDQllpC6I9OVCfOBmvl1PRZGTHy5Eqden3ikVSq02b7L2tKYoeA 29semgrBTMMjSB8Kutdd3+//1EAJEpk7XCqUVxV8+ywPBcONw3lZfRbTtqHnONSXsmPi fegg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=arJy9OLk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ji19-20020a170903325300b001a96d2430a1si4233507plb.309.2023.05.10.09.59.46; Wed, 10 May 2023 10:00:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=arJy9OLk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235407AbjEJQ6k (ORCPT + 99 others); Wed, 10 May 2023 12:58:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236036AbjEJQ6g (ORCPT ); Wed, 10 May 2023 12:58:36 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 00A0EA0; Wed, 10 May 2023 09:58:34 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9186863F40; Wed, 10 May 2023 16:58:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F1742C433D2; Wed, 10 May 2023 16:58:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683737914; bh=lQMcRttOrnsmeuCGpfDl3MfHTsoNHuhuIBx3bHzbYGs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=arJy9OLkowmfnbg7ezlR1oAzrgv3Gvd39Xj86Spc7DcLNl2FnMkaD+uPlspX0KAne laMlsxdoHplNCpx+nhiBjQk/hXj/1mHa674d04JdOismzbcGwK1E6zQj3hNLfZwKMr aMtFQfO59H5+C6S66KeE3CwSpzR088PCSSQZeEFFLh6zm/28HJ7WRM0kKjzCmIsT1r h6aPmIi/9LWY55genLWYgtynRSQLkgaGDrc3my86jYSLGGQvwbzg+y0uKzcIi5f5r/ 1BLqoWek4dw4j1nsp0NPfptlMpTaSgW5k4PD6Da7FZ2SPZed5JSoOqx0ylp/9oxKBq 3plhQ4BeJAtrg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8D68ECE1279; Wed, 10 May 2023 09:58:33 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 3/8] rcu: Check callback-invocation time limit for rcuc kthreads Date: Wed, 10 May 2023 09:58:27 -0700 Message-Id: <20230510165832.2187453-3-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765527258008483258?= X-GMAIL-MSGID: =?utf-8?q?1765527258008483258?= Currently, a callback-invocation time limit is enforced only for callbacks invoked from the softirq environment, the rationale being that when callbacks are instead invoked from rcuc and rcuoc kthreads, these callbacks cannot be holding up other softirq vectors. Which is in fact true. However, if an rcuc kthread spends too much time invoking callbacks, it can delay quiescent-state reports from its CPU, which can also be a problem. This commit therefore applies the callback-invocation time limit to callback invocation from the rcuc kthreads as well as from softirq. Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index f52ff7241041..9a5c160186d1 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2046,6 +2046,13 @@ rcu_check_quiescent_state(struct rcu_data *rdp) rcu_report_qs_rdp(rdp); } +/* Return true if callback-invocation time limit exceeded. */ +static bool rcu_do_batch_check_time(long count, long tlimit) +{ + // Invoke local_clock() only once per 32 consecutive callbacks. + return unlikely(tlimit) && !likely(count & 31) && local_clock() >= tlimit; +} + /* * Invoke any RCU callbacks that have made it to the end of their grace * period. Throttle as specified by rdp->blimit. @@ -2082,7 +2089,8 @@ static void rcu_do_batch(struct rcu_data *rdp) div = READ_ONCE(rcu_divisor); div = div < 0 ? 7 : div > sizeof(long) * 8 - 2 ? sizeof(long) * 8 - 2 : div; bl = max(rdp->blimit, pending >> div); - if (in_serving_softirq() && unlikely(bl > 100)) { + if ((in_serving_softirq() || rdp->rcu_cpu_kthread_status == RCU_KTHREAD_RUNNING) && + unlikely(bl > 100)) { long rrn = READ_ONCE(rcu_resched_ns); rrn = rrn < NSEC_PER_MSEC ? NSEC_PER_MSEC : rrn > NSEC_PER_SEC ? NSEC_PER_SEC : rrn; @@ -2126,21 +2134,23 @@ static void rcu_do_batch(struct rcu_data *rdp) * Make sure we don't spend too much time here and deprive other * softirq vectors of CPU cycles. */ - if (unlikely(tlimit)) { - /* only call local_clock() every 32 callbacks */ - if (likely((count & 31) || local_clock() < tlimit)) - continue; - /* Exceeded the time limit, so leave. */ + if (rcu_do_batch_check_time(count, tlimit)) break; - } } else { - // In rcuoc context, so no worries about depriving - // other softirq vectors of CPU cycles. + // In rcuc/rcuoc context, so no worries about + // depriving other softirq vectors of CPU cycles. local_bh_enable(); lockdep_assert_irqs_enabled(); cond_resched_tasks_rcu_qs(); lockdep_assert_irqs_enabled(); local_bh_disable(); + // But rcuc kthreads can delay quiescent-state + // reporting, so check time limits for them. + if (rdp->rcu_cpu_kthread_status == RCU_KTHREAD_RUNNING && + rcu_do_batch_check_time(count, tlimit)) { + rdp->rcu_cpu_has_work = 1; + break; + } } } From patchwork Wed May 10 16:58:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 92222 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3775596vqo; Wed, 10 May 2023 10:00:28 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6nlsGlgpbGC0xTrDfZNluLGsryg0X5vbAIGHjhK5UVFtbhL3Is6E313b9cU95E/xyXbo1d X-Received: by 2002:a17:902:e810:b0:1a1:f5dd:2dce with SMTP id u16-20020a170902e81000b001a1f5dd2dcemr23144133plg.6.1683738028228; Wed, 10 May 2023 10:00:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683738028; cv=none; d=google.com; s=arc-20160816; b=mH9vwqvqmiAT5vkeQ406budwOwnoAuNMX6NIZsWjRMdvgofkq94wqiS/E3d+MgVqcw WGuf0QgHvL2yoBP6aFBnkxdPXNbrxdTZLsXM74WIK4GdSgNHbXxeP3pp3/3kfw3je1G+ 5lYp4Eq48CuPjKkoMZPtdsUczHTK4/YgAxOP91R90D/JAhkzYKGoYTSj6vzWMFQv6g64 jAEm2jKxuhaHQTQ82SvNDQdnhb6V7E/bs3jW1uruI3WdLQmGD54RERLTVQDRJlBz9Cz4 iamzo/146YJjguP9fLq9dXVlBZPVia18CBhsFyIX5WHknsgPzvW/3H/RkhrXbGuGqbFU G+vw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2wH4zJ9tuy6Ga5c9SPHlQmvGPBReKvg5idzafK4FCqY=; b=0df+DZH3CtbmEum2tGobtduv6ECJBtLKt73gxm4WMoOcu/vDexmi0POQ75iPVEDE3p /+mdEwxDTFiyk/Pw1E7MqxmNoU4XAPDiH/Of1srUFn9H8QvNLMStCxo0g7p2Dml7+8/9 Avowx4zEgzEXWyi5uFz27Lj/yit7AvVBnKCUVUCymEIaIjDV9HEfVPuUgBGNE5VUGMuI NRsaqPE4TFZIlRQNfWl+lupQPEHIzlkYWNjOnwyu8k5vjtVZRJ335yGe+eGKjdqrChna g5jZW9lYSiKbMyA0d8KdNGfva7ZvmQlgUW99DEsZnVtpGDuMsOF4Sj3dV0xUSVy3PYBd 3QyA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=U1TdPulp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t62-20020a638141000000b00517a4a75528si4466611pgd.162.2023.05.10.10.00.13; Wed, 10 May 2023 10:00:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=U1TdPulp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236142AbjEJQ6q (ORCPT + 99 others); Wed, 10 May 2023 12:58:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236052AbjEJQ6g (ORCPT ); Wed, 10 May 2023 12:58:36 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1013A11D; Wed, 10 May 2023 09:58:35 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 993D763F32; Wed, 10 May 2023 16:58:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 058B1C4339E; Wed, 10 May 2023 16:58:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683737914; bh=Me3Kqr9uGd2TfIIIWzyQD74ZVyYw70Fp/6TxMe8zjFI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=U1TdPulpc0Fay6L9gjHHhILfIAdAL8QPi97Xqr0spylCPVSnbwPeaJ2NePxbA+THP eXEWdHZsiKXDszihD5HV88r9S3yJ4gsljgDWiG7hBZiKF6on2WarmEWE4kI/2fctMb obqNNDLnGQZgi+p0MjmCCca3A3cVOOcwj66vNqR0LSXhin0tInFAsEYAWtn0QVRF67 r94nOb7T+TVwXEP/xlikS77r97wi/P/1pzPb3PA6woNG0+Ef3Vy76EZ8HSjQ7ILWb6 gXOsOswLlUkisiCuEhIjtBhvp/ugO4IRJ1CEyDH0ohpOsU0ksZfgHmoLwpWXa2VqHn Es55T/aDjXDZw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8FBE7CE127F; Wed, 10 May 2023 09:58:33 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Domas Mituzas Subject: [PATCH rcu 4/8] rcu: Employ jiffies-based backstop to callback time limit Date: Wed, 10 May 2023 09:58:28 -0700 Message-Id: <20230510165832.2187453-4-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765527286753619975?= X-GMAIL-MSGID: =?utf-8?q?1765527286753619975?= Currently, if there are more than 100 ready-to-invoke RCU callbacks queued on a given CPU, the rcu_do_batch() function sets a timeout for invocation of the series. This timeout defaulting to three milliseconds, and may be adjusted using the rcutree.rcu_resched_ns kernel boot parameter. This timeout is checked using local_clock(), but the overhead of this function combined with the common-case very small callback-invocation overhead means that local_clock() is checked every 32nd invocation. This works well except for longer-than average callbacks. For example, a series of 500-microsecond-duration callbacks means that local_clock() is checked only once every 16 milliseconds, which makes it difficult to enforce a three-millisecond timeout. This commit therefore adds a Kconfig option RCU_DOUBLE_CHECK_CB_TIME that enables backup timeout checking using the coarser grained but lighter weight jiffies. If the jiffies counter detects a timeout, then local_clock() is consulted even if this is not the 32nd callback. This prevents the aforementioned 16-millisecond latency blow. Reported-by: Domas Mituzas Signed-off-by: Paul E. McKenney --- kernel/rcu/Kconfig | 18 ++++++++++++++++++ kernel/rcu/tree.c | 28 ++++++++++++++++++++-------- 2 files changed, 38 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index 9071182b1284..bdd7eadb33d8 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -314,4 +314,22 @@ config RCU_LAZY To save power, batch RCU callbacks and flush after delay, memory pressure, or callback list growing too big. +config RCU_DOUBLE_CHECK_CB_TIME + bool "RCU callback-batch backup time check" + depends on RCU_EXPERT + default n + help + Use this option to provide more precise enforcement of the + rcutree.rcu_resched_ns module parameter in situations where + a single RCU callback might run for hundreds of microseconds, + thus defeating the 32-callback batching used to amortize the + cost of the fine-grained but expensive local_clock() function. + + This option rounds rcutree.rcu_resched_ns up to the next + jiffy, and overrides the 32-callback batching if this limit + is exceeded. + + Say Y here if you need tighter callback-limit enforcement. + Say N here if you are unsure. + endmenu # "RCU Subsystem" diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 9a5c160186d1..e2dbea6cee4b 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2047,10 +2047,15 @@ rcu_check_quiescent_state(struct rcu_data *rdp) } /* Return true if callback-invocation time limit exceeded. */ -static bool rcu_do_batch_check_time(long count, long tlimit) +static bool rcu_do_batch_check_time(long count, long tlimit, + bool jlimit_check, unsigned long jlimit) { // Invoke local_clock() only once per 32 consecutive callbacks. - return unlikely(tlimit) && !likely(count & 31) && local_clock() >= tlimit; + return unlikely(tlimit) && + (!likely(count & 31) || + (IS_ENABLED(CONFIG_RCU_DOUBLE_CHECK_CB_TIME) && + jlimit_check && time_after(jiffies, jlimit))) && + local_clock() >= tlimit; } /* @@ -2059,13 +2064,17 @@ static bool rcu_do_batch_check_time(long count, long tlimit) */ static void rcu_do_batch(struct rcu_data *rdp) { + long bl; + long count = 0; int div; bool __maybe_unused empty; unsigned long flags; - struct rcu_head *rhp; + unsigned long jlimit; + bool jlimit_check = false; + long pending; struct rcu_cblist rcl = RCU_CBLIST_INITIALIZER(rcl); - long bl, count = 0; - long pending, tlimit = 0; + struct rcu_head *rhp; + long tlimit = 0; /* If no callbacks are ready, just return. */ if (!rcu_segcblist_ready_cbs(&rdp->cblist)) { @@ -2090,11 +2099,14 @@ static void rcu_do_batch(struct rcu_data *rdp) div = div < 0 ? 7 : div > sizeof(long) * 8 - 2 ? sizeof(long) * 8 - 2 : div; bl = max(rdp->blimit, pending >> div); if ((in_serving_softirq() || rdp->rcu_cpu_kthread_status == RCU_KTHREAD_RUNNING) && - unlikely(bl > 100)) { + (IS_ENABLED(CONFIG_RCU_DOUBLE_CHECK_CB_TIME) || unlikely(bl > 100))) { + const long npj = NSEC_PER_SEC / HZ; long rrn = READ_ONCE(rcu_resched_ns); rrn = rrn < NSEC_PER_MSEC ? NSEC_PER_MSEC : rrn > NSEC_PER_SEC ? NSEC_PER_SEC : rrn; tlimit = local_clock() + rrn; + jlimit = jiffies + (rrn + npj + 1) / npj; + jlimit_check = true; } trace_rcu_batch_start(rcu_state.name, rcu_segcblist_n_cbs(&rdp->cblist), bl); @@ -2134,7 +2146,7 @@ static void rcu_do_batch(struct rcu_data *rdp) * Make sure we don't spend too much time here and deprive other * softirq vectors of CPU cycles. */ - if (rcu_do_batch_check_time(count, tlimit)) + if (rcu_do_batch_check_time(count, tlimit, jlimit_check, jlimit)) break; } else { // In rcuc/rcuoc context, so no worries about @@ -2147,7 +2159,7 @@ static void rcu_do_batch(struct rcu_data *rdp) // But rcuc kthreads can delay quiescent-state // reporting, so check time limits for them. if (rdp->rcu_cpu_kthread_status == RCU_KTHREAD_RUNNING && - rcu_do_batch_check_time(count, tlimit)) { + rcu_do_batch_check_time(count, tlimit, jlimit_check, jlimit)) { rdp->rcu_cpu_has_work = 1; break; } From patchwork Wed May 10 16:58:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 92223 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3776206vqo; Wed, 10 May 2023 10:01:02 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6BRfxOD/+snS70sj25jNpFGSe57pKI9U8U9blMnEM1f7IGoyJ8KvXb6Qnivr7471OlYEEd X-Received: by 2002:a05:6a00:13a1:b0:643:440b:1af5 with SMTP id t33-20020a056a0013a100b00643440b1af5mr27343514pfg.16.1683738062094; Wed, 10 May 2023 10:01:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683738062; cv=none; d=google.com; s=arc-20160816; b=I1cnXOzWP6pAShV3zWRhvcn/J37MuziFCc9SM7ADoupsdRu7eEvtLuLXrC4EZti+NS P04+UB/5YsnAIBjKczfEZkkAdkoPQdiFMIn3tKBEebVG1yJ21RolTV9lH7/GyNdVYfJH sdpQVIiUqxyclBYWAQStPFNtAVv8+nHmsrQvTNqPCTZDQnh/nr3xKrqHf212LVwfMLI+ goa8GCluZd18NelN8tAExY7JZe+xjh/J4ajurxjQ0EQIb7obhW6lPkx3dxIYR0lvVNlk ZSOZJd6JM+RsrI/gjgJ8KPlgiqJbbVAe1ZiPB96nJ1wQAhdwHtNMXUXvLG7zTk5xHHLj A1Hw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/jwXgnzyg3TI5PNypIg91qZ7S7n9963pg09DyiCtpEE=; b=0nUSj2yhxESlp/utYNtNx1jNXzm9kPy42Nmcn/fwnTMsXGH97B350fFTs6AzvnSSDt o2S2UK8ewMjkwQw3evYES7Xk7GRDDDNb0s96NgOXRT8OkIkSk0QwYIhLQArPgL9UHaTL Mpv+D9iQ8s56sA4bpTFn0QTPdUAcB0MNRGLmRjJnVqLaJU7mecD6r1AJGe6rwTw7KRyU kqpVtk/0H5jtmLvEXDJ4hbcd/syec+XHbQ+DWb2XKEI90Dpy0Cl0SWB7wkKt+rRQ1Ws1 g8CdpRdKGns89kOBobceLeKy0oOB3nku52tTfOX1MTvrIq+BaJTErt5cTadbJbZzWKhU C8CA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=aOByI0GY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x21-20020aa78f15000000b00622ae306420si5266537pfr.279.2023.05.10.10.00.22; Wed, 10 May 2023 10:01:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=aOByI0GY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236145AbjEJQ6u (ORCPT + 99 others); Wed, 10 May 2023 12:58:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236049AbjEJQ6g (ORCPT ); Wed, 10 May 2023 12:58:36 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35D4519BA; Wed, 10 May 2023 09:58:35 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A9CD963F43; Wed, 10 May 2023 16:58:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1B00DC4339C; Wed, 10 May 2023 16:58:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683737914; bh=cpG57oSoRSWjavlSva8U9oIMQoMhIp2jDOdopl67PsU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aOByI0GYtGVbtiPinrjCJ4tmQ1TKhAgSY4hoTekZ4hTBOsW35VcSF8+31FJpaXVnj kImZ4LqpeYPCqnq/PAVFdxCL6yejhG7ooWhMNToJJJ+J5urMT4lo32GCGxxd+rlFmv stCglK78OegDBJmt8JUN4AtOxSPAmib3SOP0edoMI6fejv5lh5B+ef84g6X1WMtUKu NJD+GhCDfPV6IhGUF19XD4ayOUUnhHoMHz5k3B6mk4TufPicFarG+ovemXnvQ8lsVM IR3RorwH48ToCKMCn6Zdm93MTkG6xmC1KRXTlYjx7gZl/i6CcFoJvlmzJjrKHqvQM7 D5G0/kSg11S2g== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 91EEDCE12D5; Wed, 10 May 2023 09:58:33 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 5/8] rcu: Mark additional concurrent load from ->cpu_no_qs.b.exp Date: Wed, 10 May 2023 09:58:29 -0700 Message-Id: <20230510165832.2187453-5-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765527322181043767?= X-GMAIL-MSGID: =?utf-8?q?1765527322181043767?= The per-CPU rcu_data structure's ->cpu_no_qs.b.exp field is updated only on the instance corresponding to the current CPU, but can be read more widely. Unmarked accesses are OK from the corresponding CPU, but only if interrupts are disabled, given that interrupt handlers can and do modify this field. Unfortunately, although the load from rcu_preempt_deferred_qs() is always carried out from the corresponding CPU, interrupts are not necessarily disabled. This commit therefore upgrades this load to READ_ONCE. Similarly, the diagnostic access from synchronize_rcu_expedited_wait() might run with interrupts disabled and from some other CPU. This commit therefore marks this load with data_race(). Finally, the C-language access in rcu_preempt_ctxt_queue() is OK as is because interrupts are disabled and this load is always from the corresponding CPU. This commit adds a comment giving the rationale for this access being safe. This data race was reported by KCSAN. Not appropriate for backporting due to failure being unlikely. Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_exp.h | 2 +- kernel/rcu/tree_plugin.h | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 3b7abb58157d..8239b39d945b 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -643,7 +643,7 @@ static void synchronize_rcu_expedited_wait(void) "O."[!!cpu_online(cpu)], "o."[!!(rdp->grpmask & rnp->expmaskinit)], "N."[!!(rdp->grpmask & rnp->expmaskinitnext)], - "D."[!!(rdp->cpu_no_qs.b.exp)]); + "D."[!!data_race(rdp->cpu_no_qs.b.exp)]); } } pr_cont(" } %lu jiffies s: %lu root: %#lx/%c\n", diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 7b0fe741a088..41021080ad25 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -257,6 +257,8 @@ static void rcu_preempt_ctxt_queue(struct rcu_node *rnp, struct rcu_data *rdp) * GP should not be able to end until we report, so there should be * no need to check for a subsequent expedited GP. (Though we are * still in a quiescent state in any case.) + * + * Interrupts are disabled, so ->cpu_no_qs.b.exp cannot change. */ if (blkd_state & RCU_EXP_BLKD && rdp->cpu_no_qs.b.exp) rcu_report_exp_rdp(rdp); @@ -941,7 +943,7 @@ notrace void rcu_preempt_deferred_qs(struct task_struct *t) { struct rcu_data *rdp = this_cpu_ptr(&rcu_data); - if (rdp->cpu_no_qs.b.exp) + if (READ_ONCE(rdp->cpu_no_qs.b.exp)) rcu_report_exp_rdp(rdp); } From patchwork Wed May 10 16:58:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 92224 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3776239vqo; Wed, 10 May 2023 10:01:04 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4E46uZdfz1P2Pq3kpuwfJUpLosoEsTROTQkcucwl5gGLurTuNTTAJ7bqBZ6JyFddugaoJA X-Received: by 2002:a05:6a00:88f:b0:648:24c1:fe5a with SMTP id q15-20020a056a00088f00b0064824c1fe5amr4452119pfj.16.1683738063860; Wed, 10 May 2023 10:01:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683738063; cv=none; d=google.com; s=arc-20160816; b=dqTXzQzin3KE43y//tARS86IIFhiy6NVitdLSKno/sLNKahe95s7cVVjNZjca3Br6N U4uLzUgXG8Kvbl0E5tRM8V2BbFbAVr4crfxn49RoHSwJ1nyMA07SEGhbGRvC0yARD8ma 96BTQi4B5EQ2V3LENuR87xvByZm/Lrj04DpzH0cBd69b9UgMajNtBbgkZWiQkmM2KhuD b2rfYW51rScLy/DmV1Ebqwbfg0G5o1MRYJ+VaWU7Ccgyc5JpG8w81ryAWIqbmLNIYlPh VGkqFL6GTMdNFFlZMzy8Cd4fZ57GE5mFZDMvYxMIji7BhZwJv2cUtodO7aYeR15shBFj WwXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=SGLk/CV8Lc7jIQCwE4B21z6LvLrby33yv4eFpmFIlfE=; b=b5xOHeqKCdcF4RATetQmuB6ezcyXsYr8p0GVQEDUpjY3WtnjUuv/WHSLNEIarYvYjy BWIbRRL2al9uOgthYM5kUVotemOFIqjMhoOBrUlGhg4SB9E2pwzLgt+jk1qptemDli7r CNSG50DvN51yjvJYNJXcyuiqS2Jl/1S+CPxNY6QforgLykzmMOx0arG4mdBMomhI44Hs 4rY/Zb6snr6NIDqkCY5ffDFOfBhDBSFNWOZAjwsFNrg04zFPcUDmW9mMTRbvbAFF3JEI SpmTPwjVr5Em7yWZxJDYAeh1XaSFUzgjvsT351pA8+1tw94BjCQhPaymjmurAjtge1Bk /JVA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=o3CgNUuS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c7-20020aa79527000000b00643c4d0d0f5si5430657pfp.39.2023.05.10.10.00.38; Wed, 10 May 2023 10:01:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=o3CgNUuS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236162AbjEJQ6z (ORCPT + 99 others); Wed, 10 May 2023 12:58:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39850 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236062AbjEJQ6h (ORCPT ); Wed, 10 May 2023 12:58:37 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3252E35B8; Wed, 10 May 2023 09:58:36 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 09AA363F4F; Wed, 10 May 2023 16:58:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3822CC433A1; Wed, 10 May 2023 16:58:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683737914; bh=w8hk0qAptQ2qHu/gba7mNofKjO7C3JHylCL5lamAkpo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=o3CgNUuSMI3EHQX0bI4HWZx5/dz0TI/DUqH/nfyFIvHyEgqYjfggh2e0nUVeVKrZG q3NzT7ctcib7GZZs5pBWnTb7n/MTTKalvxhnQQCdz4QGmaw+ma5kqhn0k4FbKo59hc vzRd7CNhmDScyON9rgjzQKTR5hYHj5vLuzR6Vu4l6vhU1SKUZG9wWszVCEJL8CdHEJ 6RPQ1Vh9T7QEdaAuUVq5U0/YTJLmyfPDrIwjy41vcO7zjuppfEZgYrAqDn0TDa9iUV Cku8FGnO1NFnmpCdn1yC9YyRZvavnJwvFWT+cHd7ZFnlIzLuNEp4JpiAkVsS6IcF7N lQILYW7YNROPg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 941C0CE1348; Wed, 10 May 2023 09:58:33 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 6/8] rcu: Mark rcu_cpu_kthread() accesses to ->rcu_cpu_has_work Date: Wed, 10 May 2023 09:58:30 -0700 Message-Id: <20230510165832.2187453-6-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765527323851748099?= X-GMAIL-MSGID: =?utf-8?q?1765527323851748099?= The rcu_data structure's ->rcu_cpu_has_work field can be modified by any CPU attempting to wake up the rcuc kthread. Therefore, this commit marks accesses to this field from the rcu_cpu_kthread() function. This data race was reported by KCSAN. Not appropriate for backporting due to failure being unlikely. Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index e2dbea6cee4b..faa1c01d2917 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2481,12 +2481,12 @@ static void rcu_cpu_kthread(unsigned int cpu) *statusp = RCU_KTHREAD_RUNNING; local_irq_disable(); work = *workp; - *workp = 0; + WRITE_ONCE(*workp, 0); local_irq_enable(); if (work) rcu_core(); local_bh_enable(); - if (*workp == 0) { + if (!READ_ONCE(*workp)) { trace_rcu_utilization(TPS("End CPU kthread@rcu_wait")); *statusp = RCU_KTHREAD_WAITING; return; From patchwork Wed May 10 16:58:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 92225 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3776671vqo; Wed, 10 May 2023 10:01:32 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7er4gRlxY1cIBGzREYQkGHy7ZSm1k4Xlou7O8j9IFphxyb7Djq/Iii9m5ILucGDdyE/XYu X-Received: by 2002:a17:902:c102:b0:1ac:4735:45d7 with SMTP id 2-20020a170902c10200b001ac473545d7mr18165642pli.6.1683738091785; Wed, 10 May 2023 10:01:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683738091; cv=none; d=google.com; s=arc-20160816; b=kzzHxQcvhvfITchaS03kCmQPDN5yybLtVM/YuApzytQgcSPbchRMGlxk+G4EG02FUB 6GssNnTaPWbr/V/33kejE6YHvvTG23QZnrVFN+CGql8mzLAfPxkLlZNEyZcQisMlWT0r j4FRXoNo+yfjZpCJHTnBVFlNaPizF37IfteMsmVAEQiuVWc8YF5mQhfqm1anF3bIULY9 cRBDepx2/yLnvZmp7pk5lex0jSxbt8vGcb2Rrasatw8c1icyQ+jVcM4O1m4x2DsG5o8p buhqkIWlWA49DmnGU+0zHedGw3fktBDzobtv5a/EpPztJa8YuttAwH6WJ6uYFoJBFWwr PFBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Q2dR+WNvS+KAb3HHTAqFYeWsyfVleaofwCS2Qolofes=; b=sr0stvDjnv4Y06jnofBiBG+kwYs6x3i+MMb2USkEJRsx6mK3WiWOseMfN755LajiD9 ixznqjHprXcfZdX2A9xIa5xKFUA4RDlCCii/+fvlo9/FzcWEzrutW+Tl8bQNthpUsDAs /WsXCTmCIoIf75HLnscacKE+2cFtxqIPFe9vlruwaZbkPqmvIFwX8kkVEjiD6Or7Q7qH qdV1OCULWv6fWufabijo0nGELCUoSC40FDWSMsuaYm05HDjgNF7vpqYp84TWboA/kPQO g7vrLgNfbLe5TLVqU3KivIlw5pX3OqkG1tfid3bC64SNudE3elYyQJFA5/oSmiyyl072 F6ig== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="TCfVPN/u"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ji19-20020a170903325300b001a96d2430a1si4233507plb.309.2023.05.10.10.01.09; Wed, 10 May 2023 10:01:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="TCfVPN/u"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236150AbjEJQ6x (ORCPT + 99 others); Wed, 10 May 2023 12:58:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236057AbjEJQ6h (ORCPT ); Wed, 10 May 2023 12:58:37 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D20035A9; Wed, 10 May 2023 09:58:36 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 0F3F363F51; Wed, 10 May 2023 16:58:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4AF5FC433A4; Wed, 10 May 2023 16:58:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683737914; bh=uNamI0jeV30aO15+YqMLedWVb3YyLF5CZ+DX3DeiM/8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TCfVPN/uCdKj5sO95xFJZhTLccdk99nLvQzoYglN/gBpmK/C7h12YjMim6mufWv04 wNBGbuPMWIIQaxKzz3SmVD2PUf1xZMEwJy+n+HVZTITmy+f+iSpDJfiC+dKQk5a8Jo vJNTqPyxNZKFq3VULjT9pymdIyltTp77pUEffB2Vgw+GPNw6FLwz3Qyb/B/tdwipMq 7p3KqswaQAuk5QNkaNATaXPsr6xP4uE1za5xKCgM3FjEOZY9x1y8gI/Qna2Ohl5pwl Ft0fpcJRB+iEsNGjWcjOm7dz8VN0FMPsCjfpk99ny8VsShhHIoR9KaxAefelnY+JSR ZM2JNBf/TcEIA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 967FCCE1349; Wed, 10 May 2023 09:58:33 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 7/8] rcu: Make rcu_cpu_starting() rely on interrupts being disabled Date: Wed, 10 May 2023 09:58:31 -0700 Message-Id: <20230510165832.2187453-7-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765527353458004244?= X-GMAIL-MSGID: =?utf-8?q?1765527353458004244?= Currently, rcu_cpu_starting() is written so that it might be invoked with interrupts enabled. However, it is always called when interrupts are disabled, either by rcu_init(), notify_cpu_starting(), or from a call point prior to the call to notify_cpu_starting(). But why bother requiring that interrupts be disabled? The purpose is to allow the rcu_data structure's ->beenonline flag to be set after all early processing has completed for the incoming CPU, thus allowing this flag to be used to determine when workqueues have been set up for the incoming CPU, while still allowing this flag to be used as a diagnostic within rcu_core(). This commit therefore makes rcu_cpu_starting() rely on interrupts being disabled. Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index faa1c01d2917..23685407dcf6 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4390,15 +4390,16 @@ int rcutree_offline_cpu(unsigned int cpu) * Note that this function is special in that it is invoked directly * from the incoming CPU rather than from the cpuhp_step mechanism. * This is because this function must be invoked at a precise location. + * This incoming CPU must not have enabled interrupts yet. */ void rcu_cpu_starting(unsigned int cpu) { - unsigned long flags; unsigned long mask; struct rcu_data *rdp; struct rcu_node *rnp; bool newcpu; + lockdep_assert_irqs_disabled(); rdp = per_cpu_ptr(&rcu_data, cpu); if (rdp->cpu_started) return; @@ -4406,7 +4407,6 @@ void rcu_cpu_starting(unsigned int cpu) rnp = rdp->mynode; mask = rdp->grpmask; - local_irq_save(flags); arch_spin_lock(&rcu_state.ofl_lock); rcu_dynticks_eqs_online(); raw_spin_lock(&rcu_state.barrier_lock); @@ -4425,17 +4425,16 @@ void rcu_cpu_starting(unsigned int cpu) /* An incoming CPU should never be blocking a grace period. */ if (WARN_ON_ONCE(rnp->qsmask & mask)) { /* RCU waiting on incoming CPU? */ /* rcu_report_qs_rnp() *really* wants some flags to restore */ - unsigned long flags2; + unsigned long flags; - local_irq_save(flags2); + local_irq_save(flags); rcu_disable_urgency_upon_qs(rdp); /* Report QS -after- changing ->qsmaskinitnext! */ - rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags2); + rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags); } else { raw_spin_unlock_rcu_node(rnp); } arch_spin_unlock(&rcu_state.ofl_lock); - local_irq_restore(flags); smp_mb(); /* Ensure RCU read-side usage follows above initialization. */ } From patchwork Wed May 10 16:58:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 92232 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3784319vqo; Wed, 10 May 2023 10:10:23 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7CcY/Pb/JZRIMxCPZd1KTQn5WkwNdEAj3OFF5ShdPmFsxDdMXdcR7VtvTaF0mMrXIB5Nj9 X-Received: by 2002:a05:6a20:8f07:b0:ef:70b:42fc with SMTP id b7-20020a056a208f0700b000ef070b42fcmr23428705pzk.38.1683738622748; Wed, 10 May 2023 10:10:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683738622; cv=none; d=google.com; s=arc-20160816; b=HuuUGdmC29LCDKykArFLbWkUwLca5jKyjkpkAYjyzfkktXSs83nQOyW/Pvbn+e/ist 15LdPUuUnamizikZVobLpZIYpYt3DiTmfYHF6LTlsy2zF2LDAsrko0dycrw+Qr77lSZo gQlQOhbhQy3knHltHhYT1cxqc0XSTY7Zkp3gdsJkvCD0ryGTUNyTVIplcL+xmV40XV8l eWvskqmS1/MlIheZg1xhNTIwJEmobgIZcCUr1b/wgVEJTIUDk2gn4YDI/6JGbnbkZQxu TItFfT+FDEL6cLrxjhYlcOnir96JQMUZ8ll6ZCUf9s8PIcoslYY01O/HngNeaDNLmAIf 5rLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0c2thBdB4+20/9nKI2PzvSOFpWTTDqSLnluA9j6ahAE=; b=PXNa+2ZElM6k3b79YPQwU4/bWdmwwZiVPpuW7/ZnoTC+TS9MGc67c9jV3Ng5bw5M9k h2N79OUABt66iypVCzUmOvP6TwqwJBzMKlj/6Xi8xR7kPO7jSL4d78V5KgKTmbF0b218 x56vSJncu1DCu+jbPtgKYBPq/BJAy/NY46sDTbqbAHz1cyVFPeL/GnFwOvisRo9Rvxi0 nkctgPwJx62cv+hCfzlojeu1qhUEGQMsOYfYraP7urnRNvY+GrX/3+OHHWyaB5/md60N CyVs2Shtbtn35ODsrvfTEEktIZD/S7gk54kji0K1+x07IJREW7LF1NSnaYc3CGYyN97I Hf2A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=BD+k8SWq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w21-20020a63af15000000b0053039ec2b61si4929751pge.18.2023.05.10.10.10.07; Wed, 10 May 2023 10:10:22 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=BD+k8SWq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236189AbjEJQ66 (ORCPT + 99 others); Wed, 10 May 2023 12:58:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39876 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235430AbjEJQ6h (ORCPT ); Wed, 10 May 2023 12:58:37 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0940A0; Wed, 10 May 2023 09:58:36 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 244C6648A5; Wed, 10 May 2023 16:58:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 50CA7C433A0; Wed, 10 May 2023 16:58:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1683737914; bh=/JbkLgwzzOQw92qBmmq/yGo6ZclGT5J4rQaInPIkh3E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BD+k8SWq90YgLMtHTS24acU0iCn9eU2nCp4Z4aNrQGfhlQF1IN2hBetV/VGpBm+uZ Lt+p/aOzggTlzacrVMzwAbdiTJ46iVS+FkazkI2ynZR4u1ytOywQK46oCLPJAAnDNP 6tvcOI8UZ1tcCuxRcLTzvzmH0Bh52Jvg2uaJlytq7WdfBF37Al6bfcqTONfyhIZ5k2 1ul2MImZG8wTzQ4ORh3/EWyCQzhQT5cfe9UJYClX0z0/c190K4ZGucjlmUwnQCEHVt BZqo1MLet+fhu90ivqHlblDEyrWIYPG2fMVwVPDjEuXnzX8xa/pOdiqNfovopk/ffi BNGwyMH9dtevw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 98EF1CE134A; Wed, 10 May 2023 09:58:33 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Tejun Heo Subject: [PATCH rcu 8/8] rcu-tasks: Stop rcu_tasks_invoke_cbs() from using never-onlined CPUs Date: Wed, 10 May 2023 09:58:32 -0700 Message-Id: <20230510165832.2187453-8-paulmck@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765527909940035237?= X-GMAIL-MSGID: =?utf-8?q?1765527909940035237?= The rcu_tasks_invoke_cbs() function relies on queue_work_on() to silently fall back to WORK_CPU_UNBOUND when the specified CPU is offline. However, the queue_work_on() function's silent fallback mechanism relies on that CPU having been online at some time in the past. When queue_work_on() is passed a CPU that has never been online, workqueue lockups ensue, which can be bad for your kernel's general health and well-being. This commit therefore checks whether a given CPU has ever been online, and, if not substitutes WORK_CPU_UNBOUND in the subsequent call to queue_work_on(). Why not simply omit the queue_work_on() call entirely? Because this function is flooding callback-invocation notifications to all CPUs, and must deal with possibilities that include a sparse cpu_possible_mask. This commit also moves the setting of the rcu_data structure's ->beenonline field to rcu_cpu_starting(), which executes on the incoming CPU before that CPU has ever enabled interrupts. This ensures that the required workqueues are present. In addition, because the incoming CPU has not yet enabled its interrupts, there cannot yet have been any softirq handlers running on this CPU, which means that the WARN_ON_ONCE(!rdp->beenonline) within the RCU_SOFTIRQ handler cannot have triggered yet. Fixes: d363f833c6d88 rcu-tasks: Use workqueues for multiple rcu_tasks_invoke_cbs() invocations Reported-by: Tejun Heo Signed-off-by: Paul E. McKenney --- kernel/rcu/rcu.h | 6 ++++++ kernel/rcu/tasks.h | 7 +++++-- kernel/rcu/tree.c | 12 +++++++++++- 3 files changed, 22 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index 4a1b9622598b..98c1544cf572 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -642,4 +642,10 @@ void show_rcu_tasks_trace_gp_kthread(void); static inline void show_rcu_tasks_trace_gp_kthread(void) {} #endif +#ifdef CONFIG_TINY_RCU +static inline bool rcu_cpu_beenfullyonline(int cpu) { return true; } +#else +bool rcu_cpu_beenfullyonline(int cpu); +#endif + #endif /* __LINUX_RCU_H */ diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 5f4fc8184dd0..8f08c087142b 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -463,6 +463,7 @@ static void rcu_tasks_invoke_cbs(struct rcu_tasks *rtp, struct rcu_tasks_percpu { int cpu; int cpunext; + int cpuwq; unsigned long flags; int len; struct rcu_head *rhp; @@ -473,11 +474,13 @@ static void rcu_tasks_invoke_cbs(struct rcu_tasks *rtp, struct rcu_tasks_percpu cpunext = cpu * 2 + 1; if (cpunext < smp_load_acquire(&rtp->percpu_dequeue_lim)) { rtpcp_next = per_cpu_ptr(rtp->rtpcpu, cpunext); - queue_work_on(cpunext, system_wq, &rtpcp_next->rtp_work); + cpuwq = rcu_cpu_beenfullyonline(cpunext) ? cpunext : WORK_CPU_UNBOUND; + queue_work_on(cpuwq, system_wq, &rtpcp_next->rtp_work); cpunext++; if (cpunext < smp_load_acquire(&rtp->percpu_dequeue_lim)) { rtpcp_next = per_cpu_ptr(rtp->rtpcpu, cpunext); - queue_work_on(cpunext, system_wq, &rtpcp_next->rtp_work); + cpuwq = rcu_cpu_beenfullyonline(cpunext) ? cpunext : WORK_CPU_UNBOUND; + queue_work_on(cpuwq, system_wq, &rtpcp_next->rtp_work); } } diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 23685407dcf6..54963f8c244c 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4305,7 +4305,6 @@ int rcutree_prepare_cpu(unsigned int cpu) */ rnp = rdp->mynode; raw_spin_lock_rcu_node(rnp); /* irqs already disabled. */ - rdp->beenonline = true; /* We have now been online. */ rdp->gp_seq = READ_ONCE(rnp->gp_seq); rdp->gp_seq_needed = rdp->gp_seq; rdp->cpu_no_qs.b.norm = true; @@ -4332,6 +4331,16 @@ static void rcutree_affinity_setting(unsigned int cpu, int outgoing) rcu_boost_kthread_setaffinity(rdp->mynode, outgoing); } +/* + * Has the specified (known valid) CPU ever been fully online? + */ +bool rcu_cpu_beenfullyonline(int cpu) +{ + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + + return smp_load_acquire(&rdp->beenonline); +} + /* * Near the end of the CPU-online process. Pretty much all services * enabled, and the CPU is now very much alive. @@ -4435,6 +4444,7 @@ void rcu_cpu_starting(unsigned int cpu) raw_spin_unlock_rcu_node(rnp); } arch_spin_unlock(&rcu_state.ofl_lock); + smp_store_release(&rdp->beenonline, true); smp_mb(); /* Ensure RCU read-side usage follows above initialization. */ }