From patchwork Fri Dec 8 22:05:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 176064 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5746047vqy; Fri, 8 Dec 2023 14:06:44 -0800 (PST) X-Google-Smtp-Source: AGHT+IF04zHlVFYpEW88kRa6hCJctbthPJa6glrG5xXlHLzhv15DtIaPvw8DnyXv+vIdEA4nZONN X-Received: by 2002:a05:6e02:218a:b0:35d:7acc:c455 with SMTP id j10-20020a056e02218a00b0035d7accc455mr1042746ila.64.1702073203733; Fri, 08 Dec 2023 14:06:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702073203; cv=none; d=google.com; s=arc-20160816; b=p975KDnNu8DsVy7miQqyV8OPpzvBKmzggs99iFbgiY3GpS5fUolagOzIn0ru6B+k++ Uq85/iHlOoi6pQXTcL5okawXdc6wAo5Xm7ELg1hSep6FclqJy8hGAAt2N0h+dwOY3OIW zcu7Va3w4hiJjfBBlVR94CRwQht41HtoYzhR6ZW4McRZfDDK+UbdAxKdj11nye2D9/nm Ejdtg3EmEH1JhWdSo+lGicshtv8J+rddxgUzaCfVCOhv6wZWo+TwYXhy5gi+FUXrvEL0 OVYLnP4p2jkPnykse1yrdJBXMv1etvJRmME4PGcec91Pm6r743JN6+wVxYS8Zt9380dX JI+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=PqUjEJ/ZReyjT9D828gjncP8CCDG3l+xIEkxi4apZJY=; fh=ruOaVEkAbEh3t4sLFZXGQmBwM4c1/V/LLnMggyZGb4c=; b=X10NnQCFU+u5hCA9EapLn6Y1hhunLAu2mxwuoHVEZ1oy3t55ej15EhwncKR/OF+9+3 Kov9Pb84tU0ROZJwXTTk5hR1f2zlZRaQWXhU+IGVtGYzpdX7Ln7n8A0YJVVAPAnCfHBt O3hMSQXnCPDSARrlzH67IaaWhGkN4cmJl6z5l5vUSYrMx8c4unRgGYnzFD3+zNvTM1o9 wP/PFIivjcQra41zfEan8eIVcRQh4rF978AYorDBGqrcY28bxMLPyTLpwyQm4PvNMOFz 8F0eQ0mNewljqK2QoYmIENOs+wlVobYJmHTUHVq1SO3ABB49a9OINuBX/dXmrZn+anky SFhA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Yi1AB688; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id e20-20020a656794000000b005c620f350fdsi2095099pgr.341.2023.12.08.14.06.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 14:06:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Yi1AB688; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 10B4F806A136; Fri, 8 Dec 2023 14:06:22 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1574850AbjLHWGG (ORCPT + 99 others); Fri, 8 Dec 2023 17:06:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37046 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1574865AbjLHWFt (ORCPT ); Fri, 8 Dec 2023 17:05:49 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE81210E0 for ; Fri, 8 Dec 2023 14:05:55 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 402D3C433C8; Fri, 8 Dec 2023 22:05:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702073155; bh=m4pjfWFUMrRQntPdsKm0V/mNKmveqDHFHRRYTbCkI3Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Yi1AB688dExVB+lGyES6dYBZycKdaNfH9mQxxVbWHG3r6+uUpAU9yspsSo31ixBEV +4ERDwEsw4h6MSniMHhEw25JB2SdUs/KFltswE5fIsuVcB6YLru30dWGRoYLn8hwvv sl617tcUAv0vuMIbT5GGnJnyr4T1hEnx/48QZ3aPMxjGgoU1ZdpcmBaxEpAtY6rCx2 fei+kMOiFFCG3xd460V4uj5HYHjnpIYlGnE8RPYFayBt+/WXN9ih3mRaJn4GtRXdbz mNJBxRFEEreHngtIWcBi+fZ2FUs6WXYOBl6lfsh8TXC2Dfy3PdMJiOLtwCL09N907X l66Ki9++fNI+Q== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Boqun Feng , Joel Fernandes , Neeraj Upadhyay , "Paul E . McKenney" , Uladzislau Rezki , Zqiang , rcu Subject: [PATCH 1/8] rcu/nocb: Make IRQs disablement symetric Date: Fri, 8 Dec 2023 23:05:38 +0100 Message-ID: <20231208220545.7452-2-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231208220545.7452-1-frederic@kernel.org> References: <20231208220545.7452-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 08 Dec 2023 14:06:22 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784753111428101222 X-GMAIL-MSGID: 1784753111428101222 Currently IRQs are disabled on call_rcu() and then depending on the context: * If the CPU is in nocb mode: - If the callback is enqueued in the bypass list, IRQs are re-enabled implictly by rcu_nocb_try_bypass() - If the callback is enqueued in the normal list, IRQs are re-enabled implicitly by __call_rcu_nocb_wake() * If the CPU is NOT in nocb mode, IRQs are reenabled explicitly from call_rcu() This makes the code a bit hard to follow, especially as it interleaves with nocb locking. To make the IRQ flags coverage clearer and also in order to prepare for moving all the nocb enqueue code to its own function, always re-enable the IRQ flags explicitly from call_rcu(). Signed-off-by: Frederic Weisbecker Reviewed-by: Neeraj Upadhyay (AMD) --- kernel/rcu/tree.c | 9 ++++++--- kernel/rcu/tree_nocb.h | 20 +++++++++----------- 2 files changed, 15 insertions(+), 14 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 52c2fdbc6363..74159c6d3bdf 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2697,8 +2697,10 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) } check_cb_ovld(rdp); - if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) + if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) { + local_irq_restore(flags); return; // Enqueued onto ->nocb_bypass, so just leave. + } // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. rcu_segcblist_enqueue(&rdp->cblist, head); if (__is_kvfree_rcu_offset((unsigned long)func)) @@ -2716,8 +2718,8 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) __call_rcu_nocb_wake(rdp, was_alldone, flags); /* unlocks */ } else { __call_rcu_core(rdp, head, flags); - local_irq_restore(flags); } + local_irq_restore(flags); } #ifdef CONFIG_RCU_LAZY @@ -4615,8 +4617,9 @@ void rcutree_migrate_callbacks(int cpu) __call_rcu_nocb_wake(my_rdp, true, flags); } else { rcu_nocb_unlock(my_rdp); /* irqs remain disabled. */ - raw_spin_unlock_irqrestore_rcu_node(my_rnp, flags); + raw_spin_unlock_rcu_node(my_rnp); /* irqs remain disabled. */ } + local_irq_restore(flags); if (needwake) rcu_gp_kthread_wake(); lockdep_assert_irqs_enabled(); diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 5598212d1f27..3f70fd0a2db4 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -532,9 +532,7 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, // 2. Both of these conditions are met: // a. The bypass list previously had only lazy CBs, and: // b. The new CB is non-lazy. - if (ncbs && (!bypass_is_lazy || lazy)) { - local_irq_restore(flags); - } else { + if (!ncbs || (bypass_is_lazy && !lazy)) { // No-CBs GP kthread might be indefinitely asleep, if so, wake. rcu_nocb_lock(rdp); // Rare during call_rcu() flood. if (!rcu_segcblist_pend_cbs(&rdp->cblist)) { @@ -544,7 +542,7 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, } else { trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstBQnoWake")); - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); } } return true; // Callback already enqueued. @@ -570,7 +568,7 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, // If we are being polled or there is no kthread, just leave. t = READ_ONCE(rdp->nocb_gp_kthread); if (rcu_nocb_poll || !t) { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeNotPoll")); return; @@ -583,17 +581,17 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, rdp->qlen_last_fqs_check = len; // Only lazy CBs in bypass list if (lazy_len && bypass_len == lazy_len) { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_LAZY, TPS("WakeLazy")); } else if (!irqs_disabled_flags(flags)) { /* ... if queue was empty ... */ - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); wake_nocb_gp(rdp, false); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeEmpty")); } else { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE, TPS("WakeEmptyIsDeferred")); } @@ -611,15 +609,15 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, if ((rdp->nocb_cb_sleep || !rcu_segcblist_ready_cbs(&rdp->cblist)) && !timer_pending(&rdp->nocb_timer)) { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_FORCE, TPS("WakeOvfIsDeferred")); } else { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeNot")); } } else { - rcu_nocb_unlock_irqrestore(rdp, flags); + rcu_nocb_unlock(rdp); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WakeNot")); } } From patchwork Fri Dec 8 22:05:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 176063 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5746046vqy; Fri, 8 Dec 2023 14:06:43 -0800 (PST) X-Google-Smtp-Source: AGHT+IFHD/0Atzy1MJw1xF5n/c3J8ciqGQun0eCVHd56wQeqI0RSQ84cJaEBROFkJuDPKfxBaIXi X-Received: by 2002:a05:6e02:19ce:b0:35d:59a2:3323 with SMTP id r14-20020a056e0219ce00b0035d59a23323mr1051237ill.39.1702073203740; Fri, 08 Dec 2023 14:06:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702073203; cv=none; d=google.com; s=arc-20160816; b=cJH/yTuWa85FWJ+Cbr5Am6uzCsq18deqBEu/FR/p8V43Eu1vU86mugCmzovWnY0mt3 m3rHPjUJm+xAAYEKBkZWSijeWuAuTvyTxDmnzPLduifFAmcWJU0x+in4oBboYAsRN3bl gSKa50BTrCyZefCoHUtzomo9Sqx9I9I4UaZySN4K15Yufm2WRrwu5L2J6Aedb5bl2Nb+ upO4mVYhx0itnBjapxjKKAHCrhGEjUe8ARhNL0EV9/omw53db2NcDtJmEfaJ2w6ThPew ao6ZUKwVHyp7andaLF3n9lP7GEtAejqFZY9qACE0AFpryC7VrvqYn++syOzWCv5e9Gag Vdyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=FDr+17Y2V04UNhgKomw+j1ldjACcpn9xX1Zj7UWo4nE=; fh=ruOaVEkAbEh3t4sLFZXGQmBwM4c1/V/LLnMggyZGb4c=; b=FRM/gZ2azKgFZCxT165iTZ2OckyoLPjKLG0VdjwssCioDI5sEcCXK4O0/GmRoO0KJY bsTwFLaGdTmDLmq4OXnPW38DrCay5CtD2/kxc59ZT5pe/1v5HdHnFY2N5ejTWf/Hz3rX 3zC+KukGwu8uHD18D5ZvMIFI1aUHBesc5Edsy10dhFJqcz869HIhy75qpS7PgE9FnVH2 id2E/CQF3hnxjSA6vwM96qdrwN7qq2pTCit36T+RERAXfAU1J3Aqw8k5NmGdmofE2qBv G6KuYKTnpD7YdQyymCmX8mc3Fncp71QYPqOyXKSOIPps+iFuTHRXzzPJsjD6g+Lk6Vrs V5iw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=LM2juQ6E; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id b12-20020a631b4c000000b005c688ff346bsi2132746pgm.589.2023.12.08.14.06.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 14:06:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=LM2juQ6E; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 22B2A80657EE; Fri, 8 Dec 2023 14:06:22 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1574860AbjLHWGL (ORCPT + 99 others); Fri, 8 Dec 2023 17:06:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41424 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1574872AbjLHWFw (ORCPT ); Fri, 8 Dec 2023 17:05:52 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9FB2210E0 for ; Fri, 8 Dec 2023 14:05:58 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 18B5AC433CA; Fri, 8 Dec 2023 22:05:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702073158; bh=qpg9fNbGNVyk64uMzemypMgJeLqksx8712MtUIEzckQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LM2juQ6EmRrOhvmyxHcXE3u2bwlmf2LzZdDUGrE5xSqLOVXm54u1x1In7oUrGl6oT c9G0m8dOHtHGQDbm0FUaYZQp8gANhIj15K6wJzxHBiCj4kYxvfHGQZA66ALO4rfqdf XXD8u+yq8Wd4uuahIMkSI8AZPHQHzP3sfI0wHaW8ne8VD7r7rvFqPITcqs1NCsnpe9 z4Sy9vQjMsCAyZqrPaor5wHUA4J6uduTMXEACDNVy7qYIBIdUD7evm9w5PMFuk4EaX ry/xrNwEj7c0JQgnfWcEE5EeJ0TS+Z0aDZx/OtcVjhH8XevhLnsXLacLB02Tv/YCVM 9/P2mxaJgYV/Q== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Boqun Feng , Joel Fernandes , Neeraj Upadhyay , "Paul E . McKenney" , Uladzislau Rezki , Zqiang , rcu Subject: [PATCH 2/8] rcu/nocb: Re-arrange call_rcu() NOCB specific code Date: Fri, 8 Dec 2023 23:05:39 +0100 Message-ID: <20231208220545.7452-3-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231208220545.7452-1-frederic@kernel.org> References: <20231208220545.7452-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 08 Dec 2023 14:06:22 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784753111177769642 X-GMAIL-MSGID: 1784753111177769642 Currently the call_rcu() function interleaves NOCB and !NOCB enqueue code in a complicated way such that: * The bypass enqueue code may or may not have enqueued and may or may not have locked the ->nocb_lock. Everything that follows is in a Schrödinger locking state for the unwary reviewer's eyes. * The was_alldone is always set but only used in NOCB related code. * The NOCB wake up is distantly related to the locking hopefully performed by the bypass enqueue code that did not enqueue on the bypass list. Unconfuse the whole and gather NOCB and !NOCB specific enqueue code to their own functions. Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree.c | 44 +++++++++++++++++++----------------------- kernel/rcu/tree.h | 5 ++--- kernel/rcu/tree_nocb.h | 18 ++++++++++++++--- 3 files changed, 37 insertions(+), 30 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 74159c6d3bdf..82f8130d3fe3 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2559,12 +2559,26 @@ static int __init rcu_spawn_core_kthreads(void) return 0; } +static void rcutree_enqueue(struct rcu_data *rdp, struct rcu_head *head, rcu_callback_t func) +{ + rcu_segcblist_enqueue(&rdp->cblist, head); + if (__is_kvfree_rcu_offset((unsigned long)func)) + trace_rcu_kvfree_callback(rcu_state.name, head, + (unsigned long)func, + rcu_segcblist_n_cbs(&rdp->cblist)); + else + trace_rcu_callback(rcu_state.name, head, + rcu_segcblist_n_cbs(&rdp->cblist)); + trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCBQueued")); +} + /* * Handle any core-RCU processing required by a call_rcu() invocation. */ -static void __call_rcu_core(struct rcu_data *rdp, struct rcu_head *head, - unsigned long flags) +static void call_rcu_core(struct rcu_data *rdp, struct rcu_head *head, + rcu_callback_t func, unsigned long flags) { + rcutree_enqueue(rdp, head, func); /* * If called from an extended quiescent state, invoke the RCU * core in order to force a re-evaluation of RCU's idleness. @@ -2660,7 +2674,6 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) unsigned long flags; bool lazy; struct rcu_data *rdp; - bool was_alldone; /* Misaligned rcu_head! */ WARN_ON_ONCE((unsigned long)head & (sizeof(void *) - 1)); @@ -2697,28 +2710,11 @@ __call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy_in) } check_cb_ovld(rdp); - if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) { - local_irq_restore(flags); - return; // Enqueued onto ->nocb_bypass, so just leave. - } - // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. - rcu_segcblist_enqueue(&rdp->cblist, head); - if (__is_kvfree_rcu_offset((unsigned long)func)) - trace_rcu_kvfree_callback(rcu_state.name, head, - (unsigned long)func, - rcu_segcblist_n_cbs(&rdp->cblist)); + + if (unlikely(rcu_rdp_is_offloaded(rdp))) + call_rcu_nocb(rdp, head, func, flags, lazy); else - trace_rcu_callback(rcu_state.name, head, - rcu_segcblist_n_cbs(&rdp->cblist)); - - trace_rcu_segcb_stats(&rdp->cblist, TPS("SegCBQueued")); - - /* Go handle any RCU core processing required. */ - if (unlikely(rcu_rdp_is_offloaded(rdp))) { - __call_rcu_nocb_wake(rdp, was_alldone, flags); /* unlocks */ - } else { - __call_rcu_core(rdp, head, flags); - } + call_rcu_core(rdp, head, func, flags); local_irq_restore(flags); } diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index e9821a8422db..e0e70b663cbf 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -467,9 +467,8 @@ static void rcu_init_one_nocb(struct rcu_node *rnp); static bool wake_nocb_gp(struct rcu_data *rdp, bool force); static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, unsigned long j, bool lazy); -static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags, - bool lazy); +static void call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *head, + rcu_callback_t func, unsigned long flags, bool lazy); static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty, unsigned long flags); static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level); diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 3f70fd0a2db4..868063621c2f 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -622,6 +622,18 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, } } +static void call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *head, + rcu_callback_t func, unsigned long flags, bool lazy) +{ + bool was_alldone; + + if (!rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) { + /* Not enqueued on bypass but locked, do regular enqueue */ + rcutree_enqueue(rdp, head, func); + __call_rcu_nocb_wake(rdp, was_alldone, flags); /* unlocks */ + } +} + static int nocb_gp_toggle_rdp(struct rcu_data *rdp, bool *wake_state) { @@ -1765,10 +1777,10 @@ static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, return true; } -static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags, bool lazy) +static void call_rcu_nocb(struct rcu_data *rdp, struct rcu_head *head, + rcu_callback_t func, unsigned long flags, bool lazy) { - return false; + WARN_ON_ONCE(1); /* Should be dead code! */ } static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty, From patchwork Fri Dec 8 22:05:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 176065 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5746054vqy; Fri, 8 Dec 2023 14:06:44 -0800 (PST) X-Google-Smtp-Source: AGHT+IFTDWLxpHRBMZF+m2QRKJ3oZEXb7qLvZ8BJPVlLqNvyid0uBocHNp5FTpa++MWKch2qJoCr X-Received: by 2002:a05:6a20:7f83:b0:18b:594a:284c with SMTP id d3-20020a056a207f8300b0018b594a284cmr1935788pzj.15.1702073204408; Fri, 08 Dec 2023 14:06:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702073204; cv=none; d=google.com; s=arc-20160816; b=MrJS2djP+VVOLk+iIJlOndnbBEc/5eXFnujQ5+QJ4fzZlS8B/WGxQmU6fadvpCkkKd 92naq/Vh/KY97m8MZfjvz9pArpR5e0ZmusHqqnVM5x8qqm4REhxyhQsOopKCn+TLz79R nFUgD6mjxPLzfWqUD4BONHeBZaBTc9KsYrv5qvqYAu26vW1SHYgJTBD8TLGac1YF5a9Y CL+ncJWk0ANNps68cKYZhVx+KVioriCHkF9QvWizs1MssjluGXFYRA35cpB50fIu6/EW Jo5dfgNMan8eyZwmaR2brEgJkWwItrp2DgAo5uWjd/9gQCp4cJwUL2ESh7Cp5dgVSmY4 cZ/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Uh5OANqX7FMWY3V9rBp+IcxRcWoBA18zMMXCxoygkls=; fh=z8Ne2JrnnXKUY28f9pwEdqls7nShSFwQTzvJMmkinP0=; b=DW/f9UDOfL2hAJZ3Luoxejq05zcwnqpWws1aZEUhdwtuyzyrHqlWb8me30ugjIbmI1 YLlhhQgmRhJMJFkxfthW9n4IEwzQuv8YEB4YBnPVDpIR/BfhsU3IdxUIEg0Lz/EOpXZn rFORuU/fGUeUK9efHcSCdskhWyljoLRq7K1PrBd4IKZVyehtKvWP+NGVM9Fo48eXaib6 xvVxbnoPvXqNHb5Q0A93861cplCRCSlTWirx98nCPN3/e62g39jg8uCzCVCFYmdJQOEh OxwvEllXRSa9KayAuG/Xa24OUJPxE5mdIZ1avKsvYyuSH0HC6EeHeesUQ1CCrcrYWx45 BeuQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=VXojr6rg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id m128-20020a632686000000b005c66e00ffccsi2133441pgm.260.2023.12.08.14.06.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 14:06:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=VXojr6rg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 7D351802FD33; Fri, 8 Dec 2023 14:06:26 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1574867AbjLHWGQ (ORCPT + 99 others); Fri, 8 Dec 2023 17:06:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41444 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1574886AbjLHWFz (ORCPT ); Fri, 8 Dec 2023 17:05:55 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A980E1712 for ; Fri, 8 Dec 2023 14:06:01 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B8C95C433C7; Fri, 8 Dec 2023 22:05:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702073161; bh=6Rj2/Ew9i8og8UV1rs/YRIIkk8Z14PNfosvKFa7ip4I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VXojr6rgF4CVtj1/yBPc4JDT25DImi7RK6bjPnZ/E9+OwRt5xlCa7VBBdmwAuZRQk aSMyIc9cocqzJ7LMudOAEPf9o4ryZal2Oa/HV7IOcfmPXz4JSTHA6hflKzWJ/j5U04 47RHygrGazdzJEjuHxp0zlfgxtpGTF+qVfmMzN8SY2ZLR55GpJobnvyxwHk7UYALl7 4STCMLMjN779SK0D/wWSsCw1K+XE/WQfSuIe67z0e0HnM3EUh57lV7WsdRJ2iUezAj fb1y3eftmg/tb2vDlQw7dP4H+Gb2wIPAd2efJnXF2+n9aW8kKJTT1tmwcbg5O9rDhc iINxxKaf0URqg== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Boqun Feng , Joel Fernandes , Neeraj Upadhyay , "Paul E . McKenney" , Uladzislau Rezki , Zqiang , rcu , Anna-Maria Behnsen , Thomas Gleixner , Kalesh Singh Subject: [PATCH 3/8] rcu/exp: Fix RCU expedited parallel grace period kworker allocation failure recovery Date: Fri, 8 Dec 2023 23:05:40 +0100 Message-ID: <20231208220545.7452-4-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231208220545.7452-1-frederic@kernel.org> References: <20231208220545.7452-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 08 Dec 2023 14:06:26 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784753112570783747 X-GMAIL-MSGID: 1784753112570783747 Under CONFIG_RCU_EXP_KTHREAD=y, the nodes initialization for expedited grace periods is queued to a kworker. However if the allocation of that kworker failed, the nodes initialization is performed synchronously by the caller instead. Now the check for kworker initialization failure relies on the kworker pointer to be NULL while its value might actually encapsulate an allocation failure error. Make sure to handle this case. Fixes: 9621fbee44df ("rcu: Move expedited grace period (GP) work to RT kthread_worker") Cc: Kalesh Singh Signed-off-by: Frederic Weisbecker Reviewed-by: Kalesh Singh --- kernel/rcu/tree.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 82f8130d3fe3..055f4817bc70 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4723,6 +4723,7 @@ static void __init rcu_start_exp_gp_kworkers(void) rcu_exp_par_gp_kworker = kthread_create_worker(0, par_gp_kworker_name); if (IS_ERR_OR_NULL(rcu_exp_par_gp_kworker)) { pr_err("Failed to create %s!\n", par_gp_kworker_name); + rcu_exp_par_gp_kworker = NULL; kthread_destroy_worker(rcu_exp_gp_kworker); return; } From patchwork Fri Dec 8 22:05:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 176062 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5745996vqy; Fri, 8 Dec 2023 14:06:36 -0800 (PST) X-Google-Smtp-Source: AGHT+IGwFJDKJ/AgJE1W6D1quMhapaW+/5eCAQgawrKlNrk5rWSWftsBGgRSl1ebijO+u7QRz42s X-Received: by 2002:a17:90b:87:b0:286:9bc6:c95b with SMTP id bb7-20020a17090b008700b002869bc6c95bmr736139pjb.2.1702073196277; Fri, 08 Dec 2023 14:06:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702073196; cv=none; d=google.com; s=arc-20160816; b=kAFF8pV94eONIIbn3bsSskaSZHXHbYeNvXqJYdA0Lf0zLLuubrHKSffhUqlbtSmyNL dbWfjMfRyu8YX01j1O0vAlnub3weMfITYU9QVz0rWDi+WUws8qlctHcihLPGUWgOrs82 G95q8RF/LnYiskiwsc++f59fIfnzI3WeXKtdPt9SEB/aIu5YxNPvI7f8eOBVwlC5wstf do7yRELlPcFVtOc8/UQuMKIs1lo/Ck0jCm5Jq8lV0yQF2fF7EmJrqk/uRp9NhSeHgj8+ scSZUmRIzTKt7iyQ7RcY51SWbZRB3gnbgHhVjiFQM5usvfT1iObJC8JySsBspqvib/oq AVcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=EcyFKGCWlUpX9+xfMmQGnKBWo1Tws8oPuAWQpu+6bM4=; fh=z8Ne2JrnnXKUY28f9pwEdqls7nShSFwQTzvJMmkinP0=; b=emNxlzeaq0o2yJ20oGNH8SIZZvMqTy2Lq9902XDRRFfPDHLYapos0JZoLV4ZVgzP/T k9KxXPZDEWMySDtS3SmhsWoTCkwsIDMFHlmFBWLsti62gTF9I0Cy8s8tGLFG5Jic6mbP B9cbLAMU7jkBvOXviytb34RAcQje/ZOR+2zDQQF6T+JAJvBnoX2SVpRRbQSdX46umeBZ sRUtjGehHgewwBPjI7NBt0rXoiiJIMVxlzTc/UBrmGfyMu9ga+cJg5HzGTketp+MGD6D hBS2S99+XvMCVMr5z9ktkPpzTKNDnNSiTOj03kKdJFsdoHP+kt28w6SYyfs0FxFiZEdM qQZw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=sfV0bAA3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id o12-20020a17090ac08c00b0028688bb3d17si2191858pjs.159.2023.12.08.14.06.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 14:06:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=sfV0bAA3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id D69F38112A8A; Fri, 8 Dec 2023 14:06:31 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1574904AbjLHWGU (ORCPT + 99 others); Fri, 8 Dec 2023 17:06:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1574896AbjLHWF6 (ORCPT ); Fri, 8 Dec 2023 17:05:58 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE35610E0 for ; Fri, 8 Dec 2023 14:06:04 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C2059C433CB; Fri, 8 Dec 2023 22:06:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702073164; bh=2VSL0I6CDQD3On48r7ZZl2KNfsOXx4qI+XGTjXvz4To=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sfV0bAA3jMUaWQ7vGD0VMm4RIRuqF0R57O6AyLWLuIz64dUmxfQpa8c1ga8kShpyW w/tmMOwatv3tjWq6lRY5NzU3lVvcHjTa6NLBBeUgAHOR9YdENAOSGYTgK3vs6pystc e+GqnbPLuT7sFWpY1mrdr55j099gb7C2lTMluQur4lQOFYzcWPm3r33wagwqn7AYZ+ e/T7zYVHXergvzndqp/awswOBzn26JxZhhw01hSmeGlJSJdxi6vwEHnWJaXdDV+f2f pSnEbC4NZlMyi2DCb3fYtdjlnztzoxJiLz6/recBTwHZpXndpdXAXKsgYKyHdhqKKM DQ/9wRlfXoBUQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Boqun Feng , Joel Fernandes , Neeraj Upadhyay , "Paul E . McKenney" , Uladzislau Rezki , Zqiang , rcu , Anna-Maria Behnsen , Thomas Gleixner , Kalesh Singh Subject: [PATCH 4/8] rcu/exp: Handle RCU expedited grace period kworker allocation failure Date: Fri, 8 Dec 2023 23:05:41 +0100 Message-ID: <20231208220545.7452-5-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231208220545.7452-1-frederic@kernel.org> References: <20231208220545.7452-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 08 Dec 2023 14:06:32 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784753104038389591 X-GMAIL-MSGID: 1784753104038389591 Just like is done for the kworker performing nodes initialization, gracefully handle the possible allocation failure of the RCU expedited grace period main kworker. While at it perform a rename of the related checking functions to better reflect the expedited specifics. Fixes: 9621fbee44df ("rcu: Move expedited grace period (GP) work to RT kthread_worker") Cc: Kalesh Singh Signed-off-by: Frederic Weisbecker Reviewed-by: Kalesh Singh --- kernel/rcu/tree.c | 2 ++ kernel/rcu/tree_exp.h | 25 +++++++++++++++++++------ 2 files changed, 21 insertions(+), 6 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 055f4817bc70..39679cf66c3a 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4717,6 +4717,7 @@ static void __init rcu_start_exp_gp_kworkers(void) rcu_exp_gp_kworker = kthread_create_worker(0, gp_kworker_name); if (IS_ERR_OR_NULL(rcu_exp_gp_kworker)) { pr_err("Failed to create %s!\n", gp_kworker_name); + rcu_exp_gp_kworker = NULL; return; } @@ -4725,6 +4726,7 @@ static void __init rcu_start_exp_gp_kworkers(void) pr_err("Failed to create %s!\n", par_gp_kworker_name); rcu_exp_par_gp_kworker = NULL; kthread_destroy_worker(rcu_exp_gp_kworker); + rcu_exp_gp_kworker = NULL; return; } diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 6d7cea5d591f..cb31f4fb4b36 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -429,7 +429,12 @@ static void sync_rcu_exp_select_node_cpus(struct kthread_work *wp) __sync_rcu_exp_select_node_cpus(rewp); } -static inline bool rcu_gp_par_worker_started(void) +static inline bool rcu_exp_worker_started(void) +{ + return !!READ_ONCE(rcu_exp_gp_kworker); +} + +static inline bool rcu_exp_par_worker_started(void) { return !!READ_ONCE(rcu_exp_par_gp_kworker); } @@ -479,7 +484,12 @@ static void sync_rcu_exp_select_node_cpus(struct work_struct *wp) __sync_rcu_exp_select_node_cpus(rewp); } -static inline bool rcu_gp_par_worker_started(void) +static inline bool rcu_exp_worker_started(void) +{ + return !!READ_ONCE(rcu_gp_wq); +} + +static inline bool rcu_exp_par_worker_started(void) { return !!READ_ONCE(rcu_par_gp_wq); } @@ -542,7 +552,7 @@ static void sync_rcu_exp_select_cpus(void) rnp->exp_need_flush = false; if (!READ_ONCE(rnp->expmask)) continue; /* Avoid early boot non-existent wq. */ - if (!rcu_gp_par_worker_started() || + if (!rcu_exp_par_worker_started() || rcu_scheduler_active != RCU_SCHEDULER_RUNNING || rcu_is_last_leaf_node(rnp)) { /* No worker started yet or last leaf, do direct call. */ @@ -957,7 +967,7 @@ static void rcu_exp_print_detail_task_stall_rnp(struct rcu_node *rnp) */ void synchronize_rcu_expedited(void) { - bool boottime = (rcu_scheduler_active == RCU_SCHEDULER_INIT); + bool can_queue; unsigned long flags; struct rcu_exp_work rew; struct rcu_node *rnp; @@ -968,6 +978,9 @@ void synchronize_rcu_expedited(void) lock_is_held(&rcu_sched_lock_map), "Illegal synchronize_rcu_expedited() in RCU read-side critical section"); + can_queue = (rcu_scheduler_active != RCU_SCHEDULER_INIT) && + rcu_exp_worker_started(); + /* Is the state is such that the call is a grace period? */ if (rcu_blocking_is_gp()) { // Note well that this code runs with !PREEMPT && !SMP. @@ -997,7 +1010,7 @@ void synchronize_rcu_expedited(void) return; /* Someone else did our work for us. */ /* Ensure that load happens before action based on it. */ - if (unlikely(boottime)) { + if (unlikely(!can_queue)) { /* Direct call during scheduler init and early_initcalls(). */ rcu_exp_sel_wait_wake(s); } else { @@ -1015,7 +1028,7 @@ void synchronize_rcu_expedited(void) /* Let the next expedited grace period start. */ mutex_unlock(&rcu_state.exp_mutex); - if (likely(!boottime)) + if (likely(can_queue)) synchronize_rcu_expedited_destroy_work(&rew); } EXPORT_SYMBOL_GPL(synchronize_rcu_expedited); From patchwork Fri Dec 8 22:05:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 176068 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5746139vqy; Fri, 8 Dec 2023 14:06:53 -0800 (PST) X-Google-Smtp-Source: AGHT+IE5oFQ2FEcLt50eN6bGrRXNxTZ3A3tID0HUnitwY91cgSYbuorhPLPbHgfUWd6bGCOnhRjC X-Received: by 2002:a05:6a20:7fa6:b0:18b:d207:d84d with SMTP id d38-20020a056a207fa600b0018bd207d84dmr909935pzj.0.1702073212883; Fri, 08 Dec 2023 14:06:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702073212; cv=none; d=google.com; s=arc-20160816; b=PwQZhThiG78L9oJ/entgz02typ5IozYao7dYV62SK+AK4pqHqvCopNax2y7Rdfw8KO XgWIUJSyPc9sEA6TSOjNIMMkwhgqoIV+N2l+O8uNfJK/P8+L0WOET3bKGqBrquO3nKRV KF6ZDdXk7bVYcmEl8ykpCZMylcFPhpsmGCuyNGFxumeymifuTsReVtaJsI1eVmpB5Luq bX6o7xD4NOGkeUL0Isph8L381vBt/c7NqMwqPcQH/NrhDujLa1FWpYRAm8TE5IPeIZcs 6kewcMsR14Y6B00h1zv2M4Yjjh9BlqJLCaqAcPjZZX2vr6upxNl4AWoKb4eSvWicoiVS Amag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Y2sZ+Nzc5TiMVmg3ajbIW4sHf7mTTrsafqJY8u9+TlY=; fh=oQYyamNNPu+olgCSRq+hbX8YB7X4Nkc8CF7lh4c2TlE=; b=p8+eqZTotlnJpqCcrd6CO1PW+xT2FFv93ZaPjvFcUfqmojRjUmj2mJnMiyIDjWALPr tMz8TznjEHFb1lVT3P+SkXZXLzuEaU9K9MKJ8Nk+es6M30/dNi18tmFYawCpyj1c+QvB gJF0rd2xVc2DzEpvza/4jVOwWyW12z9D3IHvuUVKCGmG8UjrmQIVCzYOqk7E0DqZlpos WsHKFhWDd6c2FTv0DDZpRDOq73sRFViisY8JtIsB3zRDPMZQCrjZ6VgC5ueH5GpD/oxH NT5Jd8A2L7tA9C/xdGNYGknBbCnTbb01iO92+Jztr98B1wrH/qBy3qMKkPIT+Ct/UbMv oNsA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=p9mX3ajP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id b20-20020a6567d4000000b005c685a2d34esi2072545pgs.321.2023.12.08.14.06.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 14:06:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=p9mX3ajP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id BA92884D5EA7; Fri, 8 Dec 2023 14:06:44 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1574911AbjLHWGY (ORCPT + 99 others); Fri, 8 Dec 2023 17:06:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41454 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1574908AbjLHWGB (ORCPT ); Fri, 8 Dec 2023 17:06:01 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 839D210E0 for ; Fri, 8 Dec 2023 14:06:07 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C9066C433CA; Fri, 8 Dec 2023 22:06:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702073167; bh=yWebujIG4HSorgdJi4M3lUUuBe+hsvsBv+UYBWpi03o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=p9mX3ajPq8YULvimHUvR6KN5ezjGUnX5P3BFYLS3YyCehL1VtWGV705VErfOiqhdj 83TJ5wXcahka4lu7FykluPIc2eMRrgnuA4nfuXuam3jdhKyXOcg4PO7LYNtJF2N0hG 84uR1qF51GWYppty8R1UFYywgyxgWON2AnUWu5HjdqI96nMlCsrBZn5REKeriurL2p 84Rhf5qVjgnnIiynDaMaM6kAiE5hQlHumfu7vaYRKcgu04PV+USxKeWNemMDRCBhpz 8L12JkJGnXg/O9cPQTedeJFC8ymzxvtPSDWPCO8LEUTn0v+409vJFdc/9SVj298Czh ac7QIUKTOUMpg== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Boqun Feng , Joel Fernandes , Neeraj Upadhyay , "Paul E . McKenney" , Uladzislau Rezki , Zqiang , rcu , Anna-Maria Behnsen , Thomas Gleixner Subject: [PATCH 5/8] rcu: s/boost_kthread_mutex/kthread_mutex Date: Fri, 8 Dec 2023 23:05:42 +0100 Message-ID: <20231208220545.7452-6-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231208220545.7452-1-frederic@kernel.org> References: <20231208220545.7452-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Fri, 08 Dec 2023 14:06:44 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784753120970037163 X-GMAIL-MSGID: 1784753120970037163 This mutex currently protecting per node boost kthreads creation and affinity setting across CPU hotplug operations will soon be used also for expedited kworkers. Generalize its name to prepare for that. Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree.c | 2 +- kernel/rcu/tree.h | 2 +- kernel/rcu/tree_plugin.h | 10 +++++----- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 39679cf66c3a..996efaded5bf 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4892,7 +4892,7 @@ static void __init rcu_init_one(void) init_waitqueue_head(&rnp->exp_wq[2]); init_waitqueue_head(&rnp->exp_wq[3]); spin_lock_init(&rnp->exp_lock); - mutex_init(&rnp->boost_kthread_mutex); + mutex_init(&rnp->kthread_mutex); raw_spin_lock_init(&rnp->exp_poll_lock); rnp->exp_seq_poll_rq = RCU_GET_STATE_COMPLETED; INIT_WORK(&rnp->exp_poll_wq, sync_rcu_do_polled_gp); diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index e0e70b663cbf..f98a245e5f32 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -113,7 +113,7 @@ struct rcu_node { /* side effect, not as a lock. */ unsigned long boost_time; /* When to start boosting (jiffies). */ - struct mutex boost_kthread_mutex; + struct mutex kthread_mutex; /* Exclusion for thread spawning and affinity */ /* manipulation. */ struct task_struct *boost_kthread_task; diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 41021080ad25..0d307674915c 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -1195,7 +1195,7 @@ static void rcu_spawn_one_boost_kthread(struct rcu_node *rnp) struct sched_param sp; struct task_struct *t; - mutex_lock(&rnp->boost_kthread_mutex); + mutex_lock(&rnp->kthread_mutex); if (rnp->boost_kthread_task || !rcu_scheduler_fully_active) goto out; @@ -1212,7 +1212,7 @@ static void rcu_spawn_one_boost_kthread(struct rcu_node *rnp) wake_up_process(t); /* get to TASK_INTERRUPTIBLE quickly. */ out: - mutex_unlock(&rnp->boost_kthread_mutex); + mutex_unlock(&rnp->kthread_mutex); } /* @@ -1224,7 +1224,7 @@ static void rcu_spawn_one_boost_kthread(struct rcu_node *rnp) * no outgoing CPU. If there are no CPUs left in the affinity set, * this function allows the kthread to execute on any CPU. * - * Any future concurrent calls are serialized via ->boost_kthread_mutex. + * Any future concurrent calls are serialized via ->kthread_mutex. */ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu) { @@ -1237,7 +1237,7 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu) return; if (!zalloc_cpumask_var(&cm, GFP_KERNEL)) return; - mutex_lock(&rnp->boost_kthread_mutex); + mutex_lock(&rnp->kthread_mutex); mask = rcu_rnp_online_cpus(rnp); for_each_leaf_node_possible_cpu(rnp, cpu) if ((mask & leaf_node_cpu_bit(rnp, cpu)) && @@ -1250,7 +1250,7 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu) cpumask_clear_cpu(outgoingcpu, cm); } set_cpus_allowed_ptr(t, cm); - mutex_unlock(&rnp->boost_kthread_mutex); + mutex_unlock(&rnp->kthread_mutex); free_cpumask_var(cm); } From patchwork Fri Dec 8 22:05:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 176066 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5746087vqy; Fri, 8 Dec 2023 14:06:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IEUhrOojgaw014BPR9WHoE9x7HmVt1rADSc+qpkG7upA96xmxVf+sAZjYTyPHiaCoUhkfxz X-Received: by 2002:a17:90a:7087:b0:286:4f6f:76ee with SMTP id g7-20020a17090a708700b002864f6f76eemr755785pjk.34.1702073208733; Fri, 08 Dec 2023 14:06:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702073208; cv=none; d=google.com; s=arc-20160816; b=fcPP5rYKGstquAWd4F09vBE7ZKtu6iOJ4KgJAQVld4MFjUnWwSb1iG6G7Xby7Ghn5J o6Pq7jE28FFShKgeGBroDt31GZB+FRySMqB/QCYwBlUqgQ5aZYQPmWhJnQz6IhXBu6Dc O+68q5GkFPMV2AokoFkne6sjc9docJ+OeL0pNiaD6Ja4YXCEnOqk196wB2vX0TQBHlEf AkFcigu4dDkBOOM1X/Z+YqEWOK6l0HQKbwtgreHF4CcYxUs377NQajEHYrpfDGlU3y2K JOP4cTzfwyYqUSMuwvb5w+ujefOgul0coWx8ohHU2cdQoGoqj7nqVO+RN4ibSvr94+eU sgjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=N/HsInI3Luy0lOSwRxsRAMU11InyzSl+BMWaozNSKlo=; fh=oQYyamNNPu+olgCSRq+hbX8YB7X4Nkc8CF7lh4c2TlE=; b=o7a53X46Amxp2HGznc30c8D81m76dyHnEkwpSyx8+Mh7kVS/3y9m582mmaBjV6ugCd 0qyiBch0946y1wj6pDGSMHb7r68Ev/o8B9jXVO17YBnrD2euhvtQ17j1ZSz4PQeynbgZ +Ic+sHl+48TPW8reRfxekQF4AumQ2uEPKVXrs+tEUuFkiheTERDENVitU8YFTjJ7fvYm +OY4/5P91KrJ3cBnLC8ITnva17n5zNK1eu3Wwdgbx7hVFkVwzggRPeS0bvvCiXoUNQQY On8Wmj6pkw7Tmh/2e+QqdhMcHeBfrt0s72tkVT4U/I68sEwuNrCmLs7henpqr8gXv5ov VYtA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=PoV0+E8V; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id y23-20020a17090ad71700b0028881bd2486si2224789pju.182.2023.12.08.14.06.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 14:06:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=PoV0+E8V; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 07E7281CA3EA; Fri, 8 Dec 2023 14:06:44 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1574873AbjLHWGa (ORCPT + 99 others); Fri, 8 Dec 2023 17:06:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56230 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1574829AbjLHWGE (ORCPT ); Fri, 8 Dec 2023 17:06:04 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 83B4F10E0 for ; Fri, 8 Dec 2023 14:06:10 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9B2D1C433C7; Fri, 8 Dec 2023 22:06:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702073170; bh=JtB1Va/I7eoL4//W12pPheH2AoGkgzH2ZAYK2M9oTq4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PoV0+E8VVFEKUVt5AZIOWstBTwrAbVB6slG/RhF24YSVKUL/3Gp5XPFP4i0aAVZCT 4dQdaGNGBxKydG/3VP2RRwhXfRmU4RyYwhlYPsh+NCat6iWiRESLLow49iOqzFue2u 2JWyGZI7uuDBZJvB+OfeTLHbBlgTgL6X6gGL2cTSbO0IPgb9O3NpvaE8Dz0KUNQIqI zZMJ7Xk0kJwu0ub2mqqHGrgNBJaCmMAdW5FYKE9aLwWR3FflLYfYc96YmN8DWOYP6e 492k0bPrPheJVyywybQpeDZx+OOn9NKAdFJy2o214lVKvxkS4vT4I1tMro/zmEbkrD D07scZVi+m2Eg== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Boqun Feng , Joel Fernandes , Neeraj Upadhyay , "Paul E . McKenney" , Uladzislau Rezki , Zqiang , rcu , Anna-Maria Behnsen , Thomas Gleixner Subject: [PATCH 6/8] rcu/exp: Make parallel exp gp kworker per rcu node Date: Fri, 8 Dec 2023 23:05:43 +0100 Message-ID: <20231208220545.7452-7-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231208220545.7452-1-frederic@kernel.org> References: <20231208220545.7452-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 08 Dec 2023 14:06:44 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784753116633891186 X-GMAIL-MSGID: 1784753116633891186 When CONFIG_RCU_EXP_KTHREAD=n, the expedited grace period per node initialization is performed in parallel via workqueues (one work per node). However in CONFIG_RCU_EXP_KTHREAD=y, this per node initialization is performed by a single kworker serializing each node initialization (one work for all nodes). The second part is certainly less scalable and efficient beyond a single leaf node. To improve this, expand this single kworker into per-node kworkers. This new layout is eventually intended to remove the workqueues based implementation since it will essentially now become duplicate code. Signed-off-by: Frederic Weisbecker --- kernel/rcu/rcu.h | 1 - kernel/rcu/tree.c | 123 +++++++++++++++++++++++---------------- kernel/rcu/tree.h | 1 + kernel/rcu/tree_exp.h | 10 ++-- kernel/rcu/tree_plugin.h | 10 +--- 5 files changed, 81 insertions(+), 64 deletions(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index f94f65877f2b..6beaf70d629f 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -625,7 +625,6 @@ void rcu_force_quiescent_state(void); extern struct workqueue_struct *rcu_gp_wq; #ifdef CONFIG_RCU_EXP_KTHREAD extern struct kthread_worker *rcu_exp_gp_kworker; -extern struct kthread_worker *rcu_exp_par_gp_kworker; #else /* !CONFIG_RCU_EXP_KTHREAD */ extern struct workqueue_struct *rcu_par_gp_wq; #endif /* CONFIG_RCU_EXP_KTHREAD */ diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 996efaded5bf..060d418c2b44 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4367,6 +4367,75 @@ rcu_boot_init_percpu_data(int cpu) rcu_boot_init_nocb_percpu_data(rdp); } +#ifdef CONFIG_RCU_EXP_KTHREAD +struct kthread_worker *rcu_exp_gp_kworker; + +static void rcu_spawn_exp_par_gp_kworker(struct rcu_node *rnp) +{ + struct kthread_worker *kworker; + const char *name = "rcu_exp_par_gp_kthread_worker/%d"; + struct sched_param param = { .sched_priority = kthread_prio }; + int rnp_index = rnp - rcu_get_root(); + + if (rnp->exp_kworker) + return; + + kworker = kthread_create_worker(0, name, rnp_index); + if (IS_ERR_OR_NULL(kworker)) { + pr_err("Failed to create par gp kworker on %d/%d\n", + rnp->grplo, rnp->grphi); + return; + } + WRITE_ONCE(rnp->exp_kworker, kworker); + sched_setscheduler_nocheck(kworker->task, SCHED_FIFO, ¶m); +} + +static void __init rcu_start_exp_gp_kworker(void) +{ + const char *name = "rcu_exp_gp_kthread_worker"; + struct sched_param param = { .sched_priority = kthread_prio }; + + rcu_exp_gp_kworker = kthread_create_worker(0, name); + if (IS_ERR_OR_NULL(rcu_exp_gp_kworker)) { + pr_err("Failed to create %s!\n", name); + rcu_exp_gp_kworker = NULL; + return; + } + sched_setscheduler_nocheck(rcu_exp_gp_kworker->task, SCHED_FIFO, ¶m); +} + +static inline void rcu_alloc_par_gp_wq(void) +{ +} +#else /* !CONFIG_RCU_EXP_KTHREAD */ +struct workqueue_struct *rcu_par_gp_wq; + +static void rcu_spawn_exp_par_gp_kworker(struct rcu_node *rnp) +{ +} + +static void __init rcu_start_exp_gp_kworker(void) +{ +} + +static inline void rcu_alloc_par_gp_wq(void) +{ + rcu_par_gp_wq = alloc_workqueue("rcu_par_gp", WQ_MEM_RECLAIM, 0); + WARN_ON(!rcu_par_gp_wq); +} +#endif /* CONFIG_RCU_EXP_KTHREAD */ + +static void rcu_spawn_rnp_kthreads(struct rcu_node *rnp) +{ + if ((IS_ENABLED(CONFIG_RCU_EXP_KTHREAD) || + IS_ENABLED(CONFIG_RCU_BOOST)) && rcu_scheduler_fully_active) { + mutex_lock(&rnp->kthread_mutex); + rcu_spawn_one_boost_kthread(rnp); + rcu_spawn_exp_par_gp_kworker(rnp); + mutex_unlock(&rnp->kthread_mutex); + } +} + /* * Invoked early in the CPU-online process, when pretty much all services * are available. The incoming CPU is not present. @@ -4415,7 +4484,7 @@ int rcutree_prepare_cpu(unsigned int cpu) rdp->rcu_iw_gp_seq = rdp->gp_seq - 1; trace_rcu_grace_period(rcu_state.name, rdp->gp_seq, TPS("cpuonl")); raw_spin_unlock_irqrestore_rcu_node(rnp, flags); - rcu_spawn_one_boost_kthread(rnp); + rcu_spawn_rnp_kthreads(rnp); rcu_spawn_cpu_nocb_kthread(cpu); WRITE_ONCE(rcu_state.n_online_cpus, rcu_state.n_online_cpus + 1); @@ -4704,54 +4773,6 @@ static int rcu_pm_notify(struct notifier_block *self, return NOTIFY_OK; } -#ifdef CONFIG_RCU_EXP_KTHREAD -struct kthread_worker *rcu_exp_gp_kworker; -struct kthread_worker *rcu_exp_par_gp_kworker; - -static void __init rcu_start_exp_gp_kworkers(void) -{ - const char *par_gp_kworker_name = "rcu_exp_par_gp_kthread_worker"; - const char *gp_kworker_name = "rcu_exp_gp_kthread_worker"; - struct sched_param param = { .sched_priority = kthread_prio }; - - rcu_exp_gp_kworker = kthread_create_worker(0, gp_kworker_name); - if (IS_ERR_OR_NULL(rcu_exp_gp_kworker)) { - pr_err("Failed to create %s!\n", gp_kworker_name); - rcu_exp_gp_kworker = NULL; - return; - } - - rcu_exp_par_gp_kworker = kthread_create_worker(0, par_gp_kworker_name); - if (IS_ERR_OR_NULL(rcu_exp_par_gp_kworker)) { - pr_err("Failed to create %s!\n", par_gp_kworker_name); - rcu_exp_par_gp_kworker = NULL; - kthread_destroy_worker(rcu_exp_gp_kworker); - rcu_exp_gp_kworker = NULL; - return; - } - - sched_setscheduler_nocheck(rcu_exp_gp_kworker->task, SCHED_FIFO, ¶m); - sched_setscheduler_nocheck(rcu_exp_par_gp_kworker->task, SCHED_FIFO, - ¶m); -} - -static inline void rcu_alloc_par_gp_wq(void) -{ -} -#else /* !CONFIG_RCU_EXP_KTHREAD */ -struct workqueue_struct *rcu_par_gp_wq; - -static void __init rcu_start_exp_gp_kworkers(void) -{ -} - -static inline void rcu_alloc_par_gp_wq(void) -{ - rcu_par_gp_wq = alloc_workqueue("rcu_par_gp", WQ_MEM_RECLAIM, 0); - WARN_ON(!rcu_par_gp_wq); -} -#endif /* CONFIG_RCU_EXP_KTHREAD */ - /* * Spawn the kthreads that handle RCU's grace periods. */ @@ -4786,10 +4807,10 @@ static int __init rcu_spawn_gp_kthread(void) * due to rcu_scheduler_fully_active. */ rcu_spawn_cpu_nocb_kthread(smp_processor_id()); - rcu_spawn_one_boost_kthread(rdp->mynode); + rcu_spawn_rnp_kthreads(rdp->mynode); rcu_spawn_core_kthreads(); /* Create kthread worker for expedited GPs */ - rcu_start_exp_gp_kworkers(); + rcu_start_exp_gp_kworker(); return 0; } early_initcall(rcu_spawn_gp_kthread); diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index f98a245e5f32..ef3d3385063f 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -72,6 +72,7 @@ struct rcu_node { /* Online CPUs for next expedited GP. */ /* Any CPU that has ever been online will */ /* have its bit set. */ + struct kthread_worker *exp_kworker; unsigned long cbovldmask; /* CPUs experiencing callback overload. */ unsigned long ffmask; /* Fully functional CPUs. */ diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index cb31f4fb4b36..744d6acf5553 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -434,9 +434,9 @@ static inline bool rcu_exp_worker_started(void) return !!READ_ONCE(rcu_exp_gp_kworker); } -static inline bool rcu_exp_par_worker_started(void) +static inline bool rcu_exp_par_worker_started(struct rcu_node *rnp) { - return !!READ_ONCE(rcu_exp_par_gp_kworker); + return !!READ_ONCE(rnp->exp_kworker); } static inline void sync_rcu_exp_select_cpus_queue_work(struct rcu_node *rnp) @@ -447,7 +447,7 @@ static inline void sync_rcu_exp_select_cpus_queue_work(struct rcu_node *rnp) * another work item on the same kthread worker can result in * deadlock. */ - kthread_queue_work(rcu_exp_par_gp_kworker, &rnp->rew.rew_work); + kthread_queue_work(READ_ONCE(rnp->exp_kworker), &rnp->rew.rew_work); } static inline void sync_rcu_exp_select_cpus_flush_work(struct rcu_node *rnp) @@ -489,7 +489,7 @@ static inline bool rcu_exp_worker_started(void) return !!READ_ONCE(rcu_gp_wq); } -static inline bool rcu_exp_par_worker_started(void) +static inline bool rcu_exp_par_worker_started(struct rcu_node *rnp) { return !!READ_ONCE(rcu_par_gp_wq); } @@ -552,7 +552,7 @@ static void sync_rcu_exp_select_cpus(void) rnp->exp_need_flush = false; if (!READ_ONCE(rnp->expmask)) continue; /* Avoid early boot non-existent wq. */ - if (!rcu_exp_par_worker_started() || + if (!rcu_exp_par_worker_started(rnp) || rcu_scheduler_active != RCU_SCHEDULER_RUNNING || rcu_is_last_leaf_node(rnp)) { /* No worker started yet or last leaf, do direct call. */ diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 0d307674915c..09bdd36ca9ff 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -1195,14 +1195,13 @@ static void rcu_spawn_one_boost_kthread(struct rcu_node *rnp) struct sched_param sp; struct task_struct *t; - mutex_lock(&rnp->kthread_mutex); - if (rnp->boost_kthread_task || !rcu_scheduler_fully_active) - goto out; + if (rnp->boost_kthread_task) + return; t = kthread_create(rcu_boost_kthread, (void *)rnp, "rcub/%d", rnp_index); if (WARN_ON_ONCE(IS_ERR(t))) - goto out; + return; raw_spin_lock_irqsave_rcu_node(rnp, flags); rnp->boost_kthread_task = t; @@ -1210,9 +1209,6 @@ static void rcu_spawn_one_boost_kthread(struct rcu_node *rnp) sp.sched_priority = kthread_prio; sched_setscheduler_nocheck(t, SCHED_FIFO, &sp); wake_up_process(t); /* get to TASK_INTERRUPTIBLE quickly. */ - - out: - mutex_unlock(&rnp->kthread_mutex); } /* From patchwork Fri Dec 8 22:05:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 176067 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5746100vqy; Fri, 8 Dec 2023 14:06:49 -0800 (PST) X-Google-Smtp-Source: AGHT+IF0PVg5I9dF5+T1zslYeZMS7jN9ep7HCSF3V+ry/JE+a9Uqrdp/INKtrXyuGguIMXegvez1 X-Received: by 2002:a17:90a:6e41:b0:285:c1c0:ffb4 with SMTP id s1-20020a17090a6e4100b00285c1c0ffb4mr638120pjm.37.1702073209467; Fri, 08 Dec 2023 14:06:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702073209; cv=none; d=google.com; s=arc-20160816; b=y9HLhNY4H3CsRLaS8RYlPMXY+ULXK6p0RACN0pF/qTLe9lGfvUf3vQ2r0p7VVN4AYC 6+bm/oR7VDHaUyKlbKEbPMMz+i8/csc1YN6WHGqRyWqO5sxto11FaEpI3uqDxXlL1wm4 lh2ByiopbaXx+3vOod1Hhu6Fex51xshPa6M/3d7tZUuFYVDO44WM0jNI8wnbrhrsX5IP 5u3G2B9Bwv/d7JgBp22yCR56Y0BudSQ2ErzK005R2xCqZtQDDsKfjBrfYx0qlmDhlXs6 kn79hPYDu+U/D1DPWxgjiT6jdAkoLBSKZ/2TA49Mq1LCu+a+Gx6ofUDbVu1nSydS8dG7 9S2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Uo48vdKLB8V9p+6GBRzUofGoceIZvcahuLkFvmcw7Pk=; fh=oQYyamNNPu+olgCSRq+hbX8YB7X4Nkc8CF7lh4c2TlE=; b=FfjTHRq5OdOCjRWEd3J7M1zGuzZR7RlAMvXFIg9gDoJHrNUD1NsqnIiRcCYf2rfFbW Rfjguhbks3WP3poghDOCRaNwERoTAKfj3EK9ddBmqD78lDyUzX2SiRgvCjRhNz+nLQ1r 3mdcfBizGI9FBiXVQS6OlHco5drT9dZcWBRe8MbM0ci7tKbFqGzpu6Ux0IrdD+w610Av Ii0YqQw9Q0Nbufa4LbKMAJ27URri8ibf6JLQtKnzduDBhGu1o2KQp+NBVWinDISY7271 cURicgvwB1FNrRS+cGAirQdnQjR0jNvGJxbdvjTsgSemZL5zsHpP4BSbQvNZtTGnrQpG 0n8w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=tmXWY8s5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id q14-20020a170902eb8e00b001d07603a4d8si2100943plg.653.2023.12.08.14.06.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 14:06:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=tmXWY8s5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 561BF831359E; Fri, 8 Dec 2023 14:06:43 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1574932AbjLHWGd (ORCPT + 99 others); Fri, 8 Dec 2023 17:06:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56246 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1574854AbjLHWGH (ORCPT ); Fri, 8 Dec 2023 17:06:07 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8973B1706 for ; Fri, 8 Dec 2023 14:06:13 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9FD22C433C9; Fri, 8 Dec 2023 22:06:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702073173; bh=AcdnP+nheXUu6zmk19wWpBUVB1bhfK7chHBbiVuRGSY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tmXWY8s5C92lss79645hQK/enaj+GdOHvU0t3O5h5jPdFzYkc8EAJjAXevCXfzGwn fXIVHh2uloQjEqyXxhf14PcM3xq1rHBwoO3xqHdXkXWprdRbzGk7nX9eC+Mr5UV2Ys mLXeDTSBkQjNEPN8VgaHhDxCB/VzBQW1yZdRGEyP5AO22SWK8aWjUKYVvPmMD3Wbbj zCe212qGBmpuQFj+PB+I7erLlH3BXgX4rRmK7ig36md7k/1820PDsZPF1sp1MmB9eN AFM1r9CcmCvJsGfk/vKsEzYEVbg5Tpp0bPGvvECdpVPyBdlWyN8BaGvNpV24JM+9pS ndRJDa6HYI2ig== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Boqun Feng , Joel Fernandes , Neeraj Upadhyay , "Paul E . McKenney" , Uladzislau Rezki , Zqiang , rcu , Anna-Maria Behnsen , Thomas Gleixner Subject: [PATCH 7/8] rcu/exp: Handle parallel exp gp kworkers affinity Date: Fri, 8 Dec 2023 23:05:44 +0100 Message-ID: <20231208220545.7452-8-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231208220545.7452-1-frederic@kernel.org> References: <20231208220545.7452-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 08 Dec 2023 14:06:43 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784753117694820021 X-GMAIL-MSGID: 1784753117694820021 Affine the parallel expedited gp kworkers to their respective RCU node in order to make them close to the cache their are playing with. This reuses the boost kthreads machinery that probe into CPU hotplug operations such that the kthreads become/stay affine to their respective node as soon/long as they contain online CPUs. Otherwise and if the current CPU going down was the last online on the leaf node, the related kthread is affine to the housekeeping CPUs. In the long run, this affinity VS CPU hotplug operation game should probably be implemented at the generic kthread level. Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree.c | 79 +++++++++++++++++++++++++++++++++++++--- kernel/rcu/tree_plugin.h | 42 ++------------------- 2 files changed, 78 insertions(+), 43 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 060d418c2b44..e75ddf42e9b1 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -145,7 +145,7 @@ static int rcu_scheduler_fully_active __read_mostly; static void rcu_report_qs_rnp(unsigned long mask, struct rcu_node *rnp, unsigned long gps, unsigned long flags); -static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu); +static struct task_struct *rcu_boost_task(struct rcu_node *rnp); static void invoke_rcu_core(void); static void rcu_report_exp_rdp(struct rcu_data *rdp); static void sync_sched_exp_online_cleanup(int cpu); @@ -4390,6 +4390,16 @@ static void rcu_spawn_exp_par_gp_kworker(struct rcu_node *rnp) sched_setscheduler_nocheck(kworker->task, SCHED_FIFO, ¶m); } +static struct task_struct *rcu_exp_par_gp_task(struct rcu_node *rnp) +{ + struct kthread_worker *kworker = READ_ONCE(rnp->exp_kworker); + + if (!kworker) + return NULL; + + return kworker->task; +} + static void __init rcu_start_exp_gp_kworker(void) { const char *name = "rcu_exp_gp_kthread_worker"; @@ -4414,6 +4424,11 @@ static void rcu_spawn_exp_par_gp_kworker(struct rcu_node *rnp) { } +static struct task_struct *rcu_exp_par_gp_task(struct rcu_node *rnp) +{ + return NULL; +} + static void __init rcu_start_exp_gp_kworker(void) { } @@ -4492,13 +4507,67 @@ int rcutree_prepare_cpu(unsigned int cpu) } /* - * Update RCU priority boot kthread affinity for CPU-hotplug changes. + * Update kthreads affinity during CPU-hotplug changes. + * + * Set the per-rcu_node kthread's affinity to cover all CPUs that are + * served by the rcu_node in question. The CPU hotplug lock is still + * held, so the value of rnp->qsmaskinit will be stable. + * + * We don't include outgoingcpu in the affinity set, use -1 if there is + * no outgoing CPU. If there are no CPUs left in the affinity set, + * this function allows the kthread to execute on any CPU. + * + * Any future concurrent calls are serialized via ->kthread_mutex. */ -static void rcutree_affinity_setting(unsigned int cpu, int outgoing) +static void rcutree_affinity_setting(unsigned int cpu, int outgoingcpu) { - struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + cpumask_var_t cm; + unsigned long mask; + struct rcu_data *rdp; + struct rcu_node *rnp; + struct task_struct *task_boost, *task_exp; - rcu_boost_kthread_setaffinity(rdp->mynode, outgoing); + if (!IS_ENABLED(CONFIG_RCU_EXP_KTHREAD) && !IS_ENABLED(CONFIG_RCU_BOOST)) + return; + + rdp = per_cpu_ptr(&rcu_data, cpu); + rnp = rdp->mynode; + + task_boost = rcu_boost_task(rnp); + task_exp = rcu_exp_par_gp_task(rnp); + + /* + * If CPU is the boot one, those tasks are created later from early + * initcall since kthreadd must be created first. + */ + if (!task_boost && !task_exp) + return; + + if (!zalloc_cpumask_var(&cm, GFP_KERNEL)) + return; + + mutex_lock(&rnp->kthread_mutex); + mask = rcu_rnp_online_cpus(rnp); + for_each_leaf_node_possible_cpu(rnp, cpu) + if ((mask & leaf_node_cpu_bit(rnp, cpu)) && + cpu != outgoingcpu) + cpumask_set_cpu(cpu, cm); + cpumask_and(cm, cm, housekeeping_cpumask(HK_TYPE_RCU)); + if (cpumask_empty(cm)) { + cpumask_copy(cm, housekeeping_cpumask(HK_TYPE_RCU)); + if (outgoingcpu >= 0) + cpumask_clear_cpu(outgoingcpu, cm); + } + + if (task_exp) + set_cpus_allowed_ptr(task_exp, cm); + + if (task_boost) + set_cpus_allowed_ptr(task_boost, cm); + + mutex_unlock(&rnp->kthread_mutex); + + free_cpumask_var(cm); } /* diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 09bdd36ca9ff..08246cca663f 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -1211,43 +1211,9 @@ static void rcu_spawn_one_boost_kthread(struct rcu_node *rnp) wake_up_process(t); /* get to TASK_INTERRUPTIBLE quickly. */ } -/* - * Set the per-rcu_node kthread's affinity to cover all CPUs that are - * served by the rcu_node in question. The CPU hotplug lock is still - * held, so the value of rnp->qsmaskinit will be stable. - * - * We don't include outgoingcpu in the affinity set, use -1 if there is - * no outgoing CPU. If there are no CPUs left in the affinity set, - * this function allows the kthread to execute on any CPU. - * - * Any future concurrent calls are serialized via ->kthread_mutex. - */ -static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu) +static struct task_struct *rcu_boost_task(struct rcu_node *rnp) { - struct task_struct *t = rnp->boost_kthread_task; - unsigned long mask; - cpumask_var_t cm; - int cpu; - - if (!t) - return; - if (!zalloc_cpumask_var(&cm, GFP_KERNEL)) - return; - mutex_lock(&rnp->kthread_mutex); - mask = rcu_rnp_online_cpus(rnp); - for_each_leaf_node_possible_cpu(rnp, cpu) - if ((mask & leaf_node_cpu_bit(rnp, cpu)) && - cpu != outgoingcpu) - cpumask_set_cpu(cpu, cm); - cpumask_and(cm, cm, housekeeping_cpumask(HK_TYPE_RCU)); - if (cpumask_empty(cm)) { - cpumask_copy(cm, housekeeping_cpumask(HK_TYPE_RCU)); - if (outgoingcpu >= 0) - cpumask_clear_cpu(outgoingcpu, cm); - } - set_cpus_allowed_ptr(t, cm); - mutex_unlock(&rnp->kthread_mutex); - free_cpumask_var(cm); + return READ_ONCE(rnp->boost_kthread_task); } #else /* #ifdef CONFIG_RCU_BOOST */ @@ -1266,10 +1232,10 @@ static void rcu_spawn_one_boost_kthread(struct rcu_node *rnp) { } -static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu) +static struct task_struct * rcu_boost_task(struct rcu_node *rnp) { + return NULL; } - #endif /* #else #ifdef CONFIG_RCU_BOOST */ /* From patchwork Fri Dec 8 22:05:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 176069 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5746147vqy; Fri, 8 Dec 2023 14:06:54 -0800 (PST) X-Google-Smtp-Source: AGHT+IFbvfhk+NJYKoJvx43YRPhrfTxjME6Pz8VbmEqjZfVJbAmBzyySj2P1dg9SNcGrH5JKdLwg X-Received: by 2002:a05:6a20:1445:b0:190:2036:fc4f with SMTP id a5-20020a056a20144500b001902036fc4fmr951064pzi.33.1702073213728; Fri, 08 Dec 2023 14:06:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702073213; cv=none; d=google.com; s=arc-20160816; b=IqWyrE/wgeLIBVume646itn8s+4loJGwF5kYQ4+WlAL/w2nJkMrAjgQzFH/C5GWjnk g7YJdpCunLaEzpXjGo6UF9cdo9mzy7F8fyPb1Mq1rB4XiRPE+c5WZgsnK7es4mtncUeV HwcFShdlRrlG4B5cQAf9XSthYFxvAv1OZQMbJj4Uleg90WDwgaJl4s6DI9utRRiD3xT4 hEM/X/ACpWwV5LjWQcdTnQR5p0pQ5iB2tNMtmgsBLm/98jvoS3zsTX7/uHQgfyQ+z47z jmUyFxNz6gJfkCZgEse9xgiNHykHbWDw+1thsUtSvXTwZ62b4jLWfRM/9abIgc7w0Ve3 vV5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ZT00O8Vo096jz01k+sOGLtlY1MvLTUlQ5zHl4Q04JXU=; fh=nQGE9Ev5fZ1wOEfZyjbmCcB4dQE4zTJGbgPJrgjfu7k=; b=L6TIpR0HXrZsCM7Pk2dG+026woXf8W6tnPOzBYvefMgDwPs6/4PVrjhs4oaDIebw/l 6dA3hwwV5FQ+WM1b4rU2MoE9Pd0Vxxj/NWmitg8lrtcLXoMEtx8tDqWkQodVvc50WMgz aJ2+T6T6+reFVUCKGuAybUrzZguDXEy4flj0W42YC5WS6V5cP5K+z6QnlhMWYEAkbNWM ueV/b8c49eUWO7r7iapG27hDwGgo3wcDtZn882jykQOGQ0Vuthq+TDcpNPGbpZfyCNSt PzUd1qGEE9PKnJJSEFtayeJcPs/9vwICQIyI+3iNs3VzxeHbmZzUdz6yapGbjWWWhQnp jbVA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=N7ajCbOy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id g21-20020aa78195000000b006ce7f143603si2068677pfi.184.2023.12.08.14.06.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Dec 2023 14:06:53 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=N7ajCbOy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 01F1881CA3E2; Fri, 8 Dec 2023 14:06:49 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1574938AbjLHWGh (ORCPT + 99 others); Fri, 8 Dec 2023 17:06:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1574868AbjLHWGN (ORCPT ); Fri, 8 Dec 2023 17:06:13 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BED04172A for ; Fri, 8 Dec 2023 14:06:16 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A6F3DC433C7; Fri, 8 Dec 2023 22:06:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1702073176; bh=JLGAnxgy6nMvCAyCTe/HdN56Qsu1Gnt8OmKPAxvGxvo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=N7ajCbOy9LlP59Yg0Xo3AzLcwsyZc0S0S2qpauxE4tjdhJgOZWjBfBSnmxCsRX40B 39C1NreJGUtnKIw5QIjWyMIH/c2/auPtH8rmtB9WgobxtqUd7hU4jZVQhhVh6S2KCp Bo0Vs1Kv3ZAtjG2/6SzfBAkVmV7Be4cnc1Tb9FkNpKscIEgab7Cfu/rSoF29CyTqgr 5OtUd2/ThgK7newWTCY06JwbRc5Rdwyumd5xH0xi7Kn9LkiHwh0STVkYeB7c00JuuS DIj/JSf2kXsEKAB6DdmQ/ajkBNQpJQ7gYryshlUO8m8DzaeZ/EnY587PfblOytxp0r IxkoHLVG8girA== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Boqun Feng , Joel Fernandes , Neeraj Upadhyay , "Paul E . McKenney" , Uladzislau Rezki , Zqiang , rcu , Anna-Maria Behnsen , Thomas Gleixner , Neeraj upadhyay Subject: [PATCH 8/8] rcu/exp: Remove rcu_par_gp_wq Date: Fri, 8 Dec 2023 23:05:45 +0100 Message-ID: <20231208220545.7452-9-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231208220545.7452-1-frederic@kernel.org> References: <20231208220545.7452-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 08 Dec 2023 14:06:49 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784753122384200587 X-GMAIL-MSGID: 1784753122384200587 TREE04 running on short iterations can produce writer stalls of the following kind: ??? Writer stall state RTWS_EXP_SYNC(4) g3968 f0x0 ->state 0x2 cpu 0 task:rcu_torture_wri state:D stack:14568 pid:83 ppid:2 flags:0x00004000 Call Trace: __schedule+0x2de/0x850 ? trace_event_raw_event_rcu_exp_funnel_lock+0x6d/0xb0 schedule+0x4f/0x90 synchronize_rcu_expedited+0x430/0x670 ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_synchronize_rcu_expedited+0x10/0x10 do_rtws_sync.constprop.0+0xde/0x230 rcu_torture_writer+0x4b4/0xcd0 ? __pfx_rcu_torture_writer+0x10/0x10 kthread+0xc7/0xf0 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x2f/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1b/0x30 Waiting for an expedited grace period and polling for an expedited grace period both are operations that internally rely on the same workqueue performing necessary asynchronous work. However, a dependency chain is involved between those two operations, as depicted below: ====== CPU 0 ======= ====== CPU 1 ======= synchronize_rcu_expedited() exp_funnel_lock() mutex_lock(&rcu_state.exp_mutex); start_poll_synchronize_rcu_expedited queue_work(rcu_gp_wq, &rnp->exp_poll_wq); synchronize_rcu_expedited_queue_work() queue_work(rcu_gp_wq, &rew->rew_work); wait_event() // A, wait for &rew->rew_work completion mutex_unlock() // B //======> switch to kworker sync_rcu_do_polled_gp() { synchronize_rcu_expedited() exp_funnel_lock() mutex_lock(&rcu_state.exp_mutex); // C, wait B .... } // D Since workqueues are usually implemented on top of several kworkers handling the queue concurrently, the above situation wouldn't deadlock most of the time because A then doesn't depend on D. But in case of memory stress, a single kworker may end up handling alone all the works in a serialized way. In that case the above layout becomes a problem because A then waits for D, closing a circular dependency: A -> D -> C -> B -> A This however only happens when CONFIG_RCU_EXP_KTHREAD=n. Indeed synchronize_rcu_expedited() is otherwise implemented on top of a kthread worker while polling still relies on rcu_gp_wq workqueue, breaking the above circular dependency chain. Fix this with making expedited grace period to always rely on kthread worker. The workqueue based implementation is essentially a duplicate anyway now that the per-node initialization is performed by per-node kthread workers. Meanwhile the CONFIG_RCU_EXP_KTHREAD switch is still kept around to manage the scheduler policy of these kthread workers. Reported-by: Anna-Maria Behnsen Reported-by: Thomas Gleixner Suggested-by: Joel Fernandes Suggested-by: Paul E. McKenney Suggested-by: Neeraj upadhyay Signed-off-by: Frederic Weisbecker --- kernel/rcu/rcu.h | 4 --- kernel/rcu/tree.c | 40 ++++--------------------- kernel/rcu/tree.h | 4 --- kernel/rcu/tree_exp.h | 70 +------------------------------------------ 4 files changed, 7 insertions(+), 111 deletions(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index 6beaf70d629f..99032b9cb667 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -623,11 +623,7 @@ int rcu_get_gp_kthreads_prio(void); void rcu_fwd_progress_check(unsigned long j); void rcu_force_quiescent_state(void); extern struct workqueue_struct *rcu_gp_wq; -#ifdef CONFIG_RCU_EXP_KTHREAD extern struct kthread_worker *rcu_exp_gp_kworker; -#else /* !CONFIG_RCU_EXP_KTHREAD */ -extern struct workqueue_struct *rcu_par_gp_wq; -#endif /* CONFIG_RCU_EXP_KTHREAD */ void rcu_gp_slow_register(atomic_t *rgssp); void rcu_gp_slow_unregister(atomic_t *rgssp); #endif /* #else #ifdef CONFIG_TINY_RCU */ diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index e75ddf42e9b1..0c28adb56ad4 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4367,7 +4367,6 @@ rcu_boot_init_percpu_data(int cpu) rcu_boot_init_nocb_percpu_data(rdp); } -#ifdef CONFIG_RCU_EXP_KTHREAD struct kthread_worker *rcu_exp_gp_kworker; static void rcu_spawn_exp_par_gp_kworker(struct rcu_node *rnp) @@ -4387,7 +4386,9 @@ static void rcu_spawn_exp_par_gp_kworker(struct rcu_node *rnp) return; } WRITE_ONCE(rnp->exp_kworker, kworker); - sched_setscheduler_nocheck(kworker->task, SCHED_FIFO, ¶m); + + if (IS_ENABLED(CONFIG_RCU_EXP_KTHREAD)) + sched_setscheduler_nocheck(kworker->task, SCHED_FIFO, ¶m); } static struct task_struct *rcu_exp_par_gp_task(struct rcu_node *rnp) @@ -4411,39 +4412,14 @@ static void __init rcu_start_exp_gp_kworker(void) rcu_exp_gp_kworker = NULL; return; } - sched_setscheduler_nocheck(rcu_exp_gp_kworker->task, SCHED_FIFO, ¶m); -} - -static inline void rcu_alloc_par_gp_wq(void) -{ -} -#else /* !CONFIG_RCU_EXP_KTHREAD */ -struct workqueue_struct *rcu_par_gp_wq; - -static void rcu_spawn_exp_par_gp_kworker(struct rcu_node *rnp) -{ -} - -static struct task_struct *rcu_exp_par_gp_task(struct rcu_node *rnp) -{ - return NULL; -} - -static void __init rcu_start_exp_gp_kworker(void) -{ -} -static inline void rcu_alloc_par_gp_wq(void) -{ - rcu_par_gp_wq = alloc_workqueue("rcu_par_gp", WQ_MEM_RECLAIM, 0); - WARN_ON(!rcu_par_gp_wq); + if (IS_ENABLED(CONFIG_RCU_EXP_KTHREAD)) + sched_setscheduler_nocheck(rcu_exp_gp_kworker->task, SCHED_FIFO, ¶m); } -#endif /* CONFIG_RCU_EXP_KTHREAD */ static void rcu_spawn_rnp_kthreads(struct rcu_node *rnp) { - if ((IS_ENABLED(CONFIG_RCU_EXP_KTHREAD) || - IS_ENABLED(CONFIG_RCU_BOOST)) && rcu_scheduler_fully_active) { + if (rcu_scheduler_fully_active) { mutex_lock(&rnp->kthread_mutex); rcu_spawn_one_boost_kthread(rnp); rcu_spawn_exp_par_gp_kworker(rnp); @@ -4527,9 +4503,6 @@ static void rcutree_affinity_setting(unsigned int cpu, int outgoingcpu) struct rcu_node *rnp; struct task_struct *task_boost, *task_exp; - if (!IS_ENABLED(CONFIG_RCU_EXP_KTHREAD) && !IS_ENABLED(CONFIG_RCU_BOOST)) - return; - rdp = per_cpu_ptr(&rcu_data, cpu); rnp = rdp->mynode; @@ -5209,7 +5182,6 @@ void __init rcu_init(void) /* Create workqueue for Tree SRCU and for expedited GPs. */ rcu_gp_wq = alloc_workqueue("rcu_gp", WQ_MEM_RECLAIM, 0); WARN_ON(!rcu_gp_wq); - rcu_alloc_par_gp_wq(); /* Fill in default value for rcutree.qovld boot parameter. */ /* -After- the rcu_node ->lock fields are initialized! */ diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index ef3d3385063f..35f7af331e6c 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -24,11 +24,7 @@ /* Communicate arguments to a workqueue handler. */ struct rcu_exp_work { unsigned long rew_s; -#ifdef CONFIG_RCU_EXP_KTHREAD struct kthread_work rew_work; -#else - struct work_struct rew_work; -#endif /* CONFIG_RCU_EXP_KTHREAD */ }; /* RCU's kthread states for tracing. */ diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 744d6acf5553..dd33948ab80f 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -420,7 +420,6 @@ static void __sync_rcu_exp_select_node_cpus(struct rcu_exp_work *rewp) static void rcu_exp_sel_wait_wake(unsigned long s); -#ifdef CONFIG_RCU_EXP_KTHREAD static void sync_rcu_exp_select_node_cpus(struct kthread_work *wp) { struct rcu_exp_work *rewp = @@ -472,69 +471,6 @@ static inline void synchronize_rcu_expedited_queue_work(struct rcu_exp_work *rew kthread_queue_work(rcu_exp_gp_kworker, &rew->rew_work); } -static inline void synchronize_rcu_expedited_destroy_work(struct rcu_exp_work *rew) -{ -} -#else /* !CONFIG_RCU_EXP_KTHREAD */ -static void sync_rcu_exp_select_node_cpus(struct work_struct *wp) -{ - struct rcu_exp_work *rewp = - container_of(wp, struct rcu_exp_work, rew_work); - - __sync_rcu_exp_select_node_cpus(rewp); -} - -static inline bool rcu_exp_worker_started(void) -{ - return !!READ_ONCE(rcu_gp_wq); -} - -static inline bool rcu_exp_par_worker_started(struct rcu_node *rnp) -{ - return !!READ_ONCE(rcu_par_gp_wq); -} - -static inline void sync_rcu_exp_select_cpus_queue_work(struct rcu_node *rnp) -{ - int cpu = find_next_bit(&rnp->ffmask, BITS_PER_LONG, -1); - - INIT_WORK(&rnp->rew.rew_work, sync_rcu_exp_select_node_cpus); - /* If all offline, queue the work on an unbound CPU. */ - if (unlikely(cpu > rnp->grphi - rnp->grplo)) - cpu = WORK_CPU_UNBOUND; - else - cpu += rnp->grplo; - queue_work_on(cpu, rcu_par_gp_wq, &rnp->rew.rew_work); -} - -static inline void sync_rcu_exp_select_cpus_flush_work(struct rcu_node *rnp) -{ - flush_work(&rnp->rew.rew_work); -} - -/* - * Work-queue handler to drive an expedited grace period forward. - */ -static void wait_rcu_exp_gp(struct work_struct *wp) -{ - struct rcu_exp_work *rewp; - - rewp = container_of(wp, struct rcu_exp_work, rew_work); - rcu_exp_sel_wait_wake(rewp->rew_s); -} - -static inline void synchronize_rcu_expedited_queue_work(struct rcu_exp_work *rew) -{ - INIT_WORK_ONSTACK(&rew->rew_work, wait_rcu_exp_gp); - queue_work(rcu_gp_wq, &rew->rew_work); -} - -static inline void synchronize_rcu_expedited_destroy_work(struct rcu_exp_work *rew) -{ - destroy_work_on_stack(&rew->rew_work); -} -#endif /* CONFIG_RCU_EXP_KTHREAD */ - /* * Select the nodes that the upcoming expedited grace period needs * to wait for. @@ -978,8 +914,7 @@ void synchronize_rcu_expedited(void) lock_is_held(&rcu_sched_lock_map), "Illegal synchronize_rcu_expedited() in RCU read-side critical section"); - can_queue = (rcu_scheduler_active != RCU_SCHEDULER_INIT) && - rcu_exp_worker_started(); + can_queue = (rcu_scheduler_active != RCU_SCHEDULER_INIT) && rcu_exp_worker_started(); /* Is the state is such that the call is a grace period? */ if (rcu_blocking_is_gp()) { @@ -1027,9 +962,6 @@ void synchronize_rcu_expedited(void) /* Let the next expedited grace period start. */ mutex_unlock(&rcu_state.exp_mutex); - - if (likely(can_queue)) - synchronize_rcu_expedited_destroy_work(&rew); } EXPORT_SYMBOL_GPL(synchronize_rcu_expedited);