From patchwork Wed Nov 30 18:13:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 27939 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1078109wrr; Wed, 30 Nov 2022 10:15:31 -0800 (PST) X-Google-Smtp-Source: AA0mqf4FQpdafUFkeYkd5MKo4/sWLosH/YujNmCtF5QV5CmHawN4fNnKZTHCvcLu4JriXO2UJ2/4 X-Received: by 2002:a05:6402:3644:b0:45f:c7f2:297d with SMTP id em4-20020a056402364400b0045fc7f2297dmr57880697edb.266.1669832131509; Wed, 30 Nov 2022 10:15:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669832131; cv=none; d=google.com; s=arc-20160816; b=nLHkB6/RJ9suMeLACF3Q1vQRvDYtKsDkhrgsipVdFYAorh0W3vNMLtmm3HYkm59PH2 RzkO+Vsmsu5+lxbXVEa09JsWeuwjt67fBZtQFq1Ae82lfLofc1Fi+mmyR6UTOifMNwbm D0YGMKCUmzWBcUlkC/pTTPJChnvVSoQG8+OjTZFGfBVcaHs4WppabO+kjLvNh55k1sML 6N5WM16FPhJ2uiVyviav8pW+0mbIJoi3Fgbh+2v+RwhjdNX98M5s2xDsMnhljizBUVCc Gh1DZQV2Rz5/53P5HswFjIScHM4pkps16xLrIPBZ+EiDAYSbQ5h/AfqGJO7h5A68DXre JLDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=uuQBoT0/9VgjI8k7NsHO7zPPIVhrYWCEsChCDjUY0bA=; b=qrZyl3nU4DRrkb5gkFTVjw/TSO2EhWb/VafNPnDJITOMsIyOwKQxmzbpPHUa0Bw6kQ IG4quq7tJCMgpjzmsUeeJ8/xn7TkA+KepkqiuaM8xsHM3jFwerxXVvCYMXQK27iQ6qD9 2+eoekqZH0no7N3jFfGlECoKubSGRMxXieySxEu6a2M98wtDh2MxCXoPpKYPHwi21CV5 7651wsdh9YobaKXm3XX/fIBd+LxCmuSA8rjue3vaEjMT6lGh8F7pYC/wWNCl+ad0HQLd noh0CeKLado32xo5FgX4SwOWzWMNDtMFSwkf9VmHsF7+QgLNDEQVA98pNh6XZj/+yPDX SA8g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=W8Ib8dqo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ss28-20020a170907c01c00b00780a3097d20si1507451ejc.180.2022.11.30.10.15.07; Wed, 30 Nov 2022 10:15:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=W8Ib8dqo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230232AbiK3SOO (ORCPT + 99 others); Wed, 30 Nov 2022 13:14:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39054 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230120AbiK3SNb (ORCPT ); Wed, 30 Nov 2022 13:13:31 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E66DD862D4; Wed, 30 Nov 2022 10:13:30 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 5F407CE1ACC; Wed, 30 Nov 2022 18:13:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9BAC0C433D6; Wed, 30 Nov 2022 18:13:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669832007; bh=KTUh/BBOVf/jDi48jtrSyvr3uGxVW8ae61JQrk9NyJc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=W8Ib8dqo0tM2ZZX1nN5aPSqBIQqvc+AFNKQ9RXDNJHY7jdtD5p70hLzIGA+ZplQn9 yNjVd3NK9aPO965vSs9Vxf2u7XONZv0fe7n09XoPaKp9RIKv91a7/I0AQ7kSthytkW Tr4sLMqh6hy03j1NuMEVmSvUV4p4Aw6Ers1Xb9d3m4edfqbeNf2K1i1k8peYxz+01m jFywHwADwCLsnoEfQDXsnQSZERbTmMKe26WKNqh7Ivr5ZSmu4LY07LmV0OSh36uYm2 5yRt7Xd0DlSZXIHyPtgMwz20ir1LCys0eIY5urJHAQqB9g80LnsUGhlFagIHmTJKtZ eZ52DRv0AsnUg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 558095C051C; Wed, 30 Nov 2022 10:13:27 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Zhen Lei , Joel Fernandes , Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH rcu 01/16] rcu: Simplify rcu_init_nohz() cpumask handling Date: Wed, 30 Nov 2022 10:13:10 -0800 Message-Id: <20221130181325.1012760-1-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> References: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750945897016496255?= X-GMAIL-MSGID: =?utf-8?q?1750945897016496255?= From: Zhen Lei In kernels built with either CONFIG_RCU_NOCB_CPU_DEFAULT_ALL=y or CONFIG_NO_HZ_FULL=y, additional CPUs must be added to rcu_nocb_mask. Except that kernels booted without the rcu_nocbs= will not have allocated rcu_nocb_mask. And the current rcu_init_nohz() function uses its need_rcu_nocb_mask and offload_all local variables to track the rcu_nocb and nohz_full state. But there is a much simpler approach, namely creating a cpumask pointer to track the default and then using cpumask_available() to check the rcu_nocb_mask state. This commit takes this approach, thereby simplifying and shortening the rcu_init_nohz() function. Signed-off-by: Zhen Lei Reviewed-by: Joel Fernandes (Google) Acked-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 34 +++++++++++----------------------- 1 file changed, 11 insertions(+), 23 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 0a5f0ef414845..ce526cc2791ca 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1210,45 +1210,33 @@ EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload); void __init rcu_init_nohz(void) { int cpu; - bool need_rcu_nocb_mask = false; - bool offload_all = false; struct rcu_data *rdp; - -#if defined(CONFIG_RCU_NOCB_CPU_DEFAULT_ALL) - if (!rcu_state.nocb_is_setup) { - need_rcu_nocb_mask = true; - offload_all = true; - } -#endif /* #if defined(CONFIG_RCU_NOCB_CPU_DEFAULT_ALL) */ + const struct cpumask *cpumask = NULL; #if defined(CONFIG_NO_HZ_FULL) - if (tick_nohz_full_running && !cpumask_empty(tick_nohz_full_mask)) { - need_rcu_nocb_mask = true; - offload_all = false; /* NO_HZ_FULL has its own mask. */ - } -#endif /* #if defined(CONFIG_NO_HZ_FULL) */ + if (tick_nohz_full_running && !cpumask_empty(tick_nohz_full_mask)) + cpumask = tick_nohz_full_mask; +#endif + + if (IS_ENABLED(CONFIG_RCU_NOCB_CPU_DEFAULT_ALL) && + !rcu_state.nocb_is_setup && !cpumask) + cpumask = cpu_possible_mask; - if (need_rcu_nocb_mask) { + if (cpumask) { if (!cpumask_available(rcu_nocb_mask)) { if (!zalloc_cpumask_var(&rcu_nocb_mask, GFP_KERNEL)) { pr_info("rcu_nocb_mask allocation failed, callback offloading disabled.\n"); return; } } + + cpumask_or(rcu_nocb_mask, rcu_nocb_mask, cpumask); rcu_state.nocb_is_setup = true; } if (!rcu_state.nocb_is_setup) return; -#if defined(CONFIG_NO_HZ_FULL) - if (tick_nohz_full_running) - cpumask_or(rcu_nocb_mask, rcu_nocb_mask, tick_nohz_full_mask); -#endif /* #if defined(CONFIG_NO_HZ_FULL) */ - - if (offload_all) - cpumask_setall(rcu_nocb_mask); - if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) { pr_info("\tNote: kernel parameter 'rcu_nocbs=', 'nohz_full', or 'isolcpus=' contains nonexistent CPUs.\n"); cpumask_and(rcu_nocb_mask, cpu_possible_mask, From patchwork Wed Nov 30 18:13:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 27933 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1077820wrr; Wed, 30 Nov 2022 10:15:01 -0800 (PST) X-Google-Smtp-Source: AA0mqf7gvE04sJuXP4+DmpjnWLhmemvHZkoB4CLRb+1QJNj77ScuqrdT9NJuz58BZTQ10Tx3mXkY X-Received: by 2002:a17:907:8749:b0:7c0:7c6c:d484 with SMTP id qo9-20020a170907874900b007c07c6cd484mr3268572ejc.752.1669832101324; Wed, 30 Nov 2022 10:15:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669832101; cv=none; d=google.com; s=arc-20160816; b=za9fZ3hU6RmuvvME0DqwZWt1Ww5cOejAYcVy9Sb0o+rlGlqqRWCB6/VwiygLdz9zd+ nlGZpT66wHXBZSqOVdivVIGuSDPg+GqC79vlIPRbOQkcIpCA4XCrehy/HWYlzdljzkw1 HDbVaQzpvL60867gyPrOcldyZjbCl/py7Yjh/j6F/vd+kORijFL5M9RfvfOW5r2dDfXz KkXMWog/rQdDnjkEaBQ/MBZ5s+SGBEnJFi/1RKMrGwlvInvE4poKn6o6Vo8u8KxTkMVG GlE5GjnVYU9cula4eLPO0xRDi1SGtGHZ0ccpWcP0cNWtghO4mMPKV9SbtXyoRfAliUlE whZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=U4oVFdZuYHclhhiYM3y5LbmYNw6QTk652e3IwS9BkbM=; b=vaT7NrvVQh6i476+hrKJ0qsHT/A0vETxcQU+QBM+5RWfXuVG68h6r/w9Yf+YVcUt9k fjUNYbpWcw2BqdbeITVNNUZ22BzKLwjTc3ZHfsg5l7nDM8Srp0qDvowAspwC+9Y/B/iR RESq/hRkRToVcQDn+eGBKGkoC3/DH4tgb2dmoiGAdn4avejGi0p42gBs0GFI6eEYFwbc kZaszdIFkEgoluUh6z+H9EiwP+UsChAosVGfWtl3IBbQ9tZ5GU97qDvFNCm3HyJKLeSl me/7N22anpmjosS5WbRmGh8sm9k6I48nOW5WxHwkSnE2fD9BunbTtdGmpuZm2Izt67E4 IU0g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="b3goJ/CR"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gs8-20020a1709072d0800b007b273d1f664si2108545ejc.128.2022.11.30.10.14.37; Wed, 30 Nov 2022 10:15:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="b3goJ/CR"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230046AbiK3SNq (ORCPT + 99 others); Wed, 30 Nov 2022 13:13:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229964AbiK3SNa (ORCPT ); Wed, 30 Nov 2022 13:13:30 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE198880CF; Wed, 30 Nov 2022 10:13:28 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4C82F61D4F; Wed, 30 Nov 2022 18:13:28 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9EDB2C433C1; Wed, 30 Nov 2022 18:13:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669832007; bh=cd9rgMHV5ytGRmwwPo9gYzUKZlyUq56zzJaeb9ZJrZY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=b3goJ/CRk/LONtMivA9/iHC0U6MW8urB5Fq8uY3sZ97b73caLX8TumVEsjYST8KDo cRdgYlmyabIaKt4Wfuh+vJryxEIvJGP2cQTBqrxBjJXfWgJ6NkceSrVHXdMqY5MrBf p44wjuDuJahQZzKJfFcYTxh5mlrotKI56/xUXDR+yWzlRvcHwZuycMAOCPY4rpc1/y Aop9z9lMb2KCcPjrg65ItUWfs//IyOhclfAd6JRMExQncO2iM33DG2z3O8ODcoiZ7o zsjiQvq1Q6Otv8TnFf5jaulAWIk7co1Au/2zGaCUEMNWFXxVbXpFhcYtfg1p2Hm+yN 1EHt7xCZENs2Q== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 57C6C5C0531; Wed, 30 Nov 2022 10:13:27 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH rcu 02/16] rcu: Fix late wakeup when flush of bypass cblist happens Date: Wed, 30 Nov 2022 10:13:11 -0800 Message-Id: <20221130181325.1012760-2-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> References: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750945865523797920?= X-GMAIL-MSGID: =?utf-8?q?1750945865523797920?= From: "Joel Fernandes (Google)" When the bypass cblist gets too big or its timeout has occurred, it is flushed into the main cblist. However, the bypass timer is still running and the behavior is that it would eventually expire and wake the GP thread. Since we are going to use the bypass cblist for lazy CBs, do the wakeup soon as the flush for "too big or too long" bypass list happens. Otherwise, long delays can happen for callbacks which get promoted from lazy to non-lazy. This is a good thing to do anyway (regardless of future lazy patches), since it makes the behavior consistent with behavior of other code paths where flushing into the ->cblist makes the GP kthread into a non-sleeping state quickly. [ Frederic Weisbecker: Changes to avoid unnecessary GP-thread wakeups plus comment changes. ] Reviewed-by: Frederic Weisbecker Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index ce526cc2791ca..f77a6d7e13564 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -433,8 +433,9 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, if ((ncbs && j != READ_ONCE(rdp->nocb_bypass_first)) || ncbs >= qhimark) { rcu_nocb_lock(rdp); + *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); + if (!rcu_nocb_flush_bypass(rdp, rhp, j)) { - *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstQ")); @@ -447,7 +448,12 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, rcu_advance_cbs_nowake(rdp->mynode, rdp); rdp->nocb_gp_adv_time = j; } - rcu_nocb_unlock_irqrestore(rdp, flags); + + // The flush succeeded and we moved CBs into the regular list. + // Don't wait for the wake up timer as it may be too far ahead. + // Wake up the GP thread now instead, if the cblist was empty. + __call_rcu_nocb_wake(rdp, *was_alldone, flags); + return true; // Callback already enqueued. } From patchwork Wed Nov 30 18:13:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 27936 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1077945wrr; Wed, 30 Nov 2022 10:15:13 -0800 (PST) X-Google-Smtp-Source: AA0mqf5LPL98zQ3IT79vICleWcS3W6jBRfSW2+PqKRm7yOJ0y6TLhp4SP9wyYTqOeAMXDIZOElSN X-Received: by 2002:a17:906:3954:b0:7bf:852b:e23c with SMTP id g20-20020a170906395400b007bf852be23cmr13929608eje.614.1669832113298; Wed, 30 Nov 2022 10:15:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669832113; cv=none; d=google.com; s=arc-20160816; b=Y67/N41uh2Z91+1Be0HKt50lybIzfhuxJv9JkCUl5iIITYXKN3bB/zH2cRf5i9I2yf 4wE1ZNjnXrUnx0/s+4jbnCbmYmf2mxhZZQY2NiyihM7BVJxTSUkS7yGn1CWidT/a2rEX YK2J7rXJOiEZIR8SbfBuUHKBeHB4gT6Xw1zTtDyKqHcIfxVl7C8T4UI6I7M684odPuSY Gq2Z0x3LwZ+v0ChYdSycKu7NFmUJqWJb5Csfgzyq7hONVXYgttOfjLOh/Asg6yMRVCV+ i9UK0Y6HhrBnbjF5XvEE9iJy8T9pSu/ZCGsr9KPLw3fiV1VIn2zQesYFAHmbBqZD/04q 7mWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ozKhBfyN+FxyuLIwEkPQLFxihGTESoRyKpstG62wPHM=; b=a6uSfbQVxYsi0B/4Sd3F/7r3MYGAfvm3Hz8VCVh6Qv+8DF4LNmeLeFCW0lNIX2fpe2 3Hee1UjCAIJtwSdtjdwgWDMHCxO/AK/Nz7meNAf7s8aXl7DDaDhS2EmRSsdSoT9SfWyU lI4enZnqIEzmbs0jlWx4c6G62nG+97t4m1Olsmw4fs1fpA+/K9h790RI9G/hIm25vB/3 q4En8DqW9XJUBtaqOlimzOBbn509EIpuFIOcr5RhtArSqCXlqR14o+EtBoPMJTLXfjTa n6WT0Fi4MnsaB4sUPklr2dJpjtFEElq7WJB4s32fQkl7+cJw8OHDqTW8t7wRBfSKLwjW 7UOQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=oZ4GteHy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hb33-20020a170907162100b00782b24c1ed8si1896095ejc.368.2022.11.30.10.14.48; Wed, 30 Nov 2022 10:15:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=oZ4GteHy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230203AbiK3SOB (ORCPT + 99 others); Wed, 30 Nov 2022 13:14:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38422 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230116AbiK3SNb (ORCPT ); Wed, 30 Nov 2022 13:13:31 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 57740862D2; Wed, 30 Nov 2022 10:13:30 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 03D24B81C97; Wed, 30 Nov 2022 18:13:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A9F05C433B5; Wed, 30 Nov 2022 18:13:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669832007; bh=JgCYhsIq4xhCHA9e/N+ivpZlC63szLfECFNNQENWSoc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oZ4GteHyu0gnGxMqABOVcD9Dxi6EMrRfdI/0I+c/8WyoaHzOcbNuRXFJercOt1gc5 DR2zF2t2Ypo/uoW4J3F9Ugza4XapYc8PzYoLGvRx35iR/YPJEIfMdtbT6JhOA/0C4p +4yYw6R0d+oZL57J4aUMt+1BSDQIUVvchk6C6SiUlSSQaA8qMhPZTR5ttBhLXO2Ity F8H7YhHPEywYMXW8xmoYYAwbOsggnSs0NCj5JTKqRIUpie6OMDOtE6bWlyFhUBF80G Y3oGpYjoZG8xvSVwnv6stpFfPc3dCDqTTHocOKrlg3T9NW38YNhnqsgfS1wNMxJLfn +OQFbdWwP8ECg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 59E9E5C0863; Wed, 30 Nov 2022 10:13:27 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Frederic Weisbecker , Joel Fernandes , "Paul E . McKenney" Subject: [PATCH rcu 03/16] rcu: Fix missing nocb gp wake on rcu_barrier() Date: Wed, 30 Nov 2022 10:13:12 -0800 Message-Id: <20221130181325.1012760-3-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> References: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750945878051343248?= X-GMAIL-MSGID: =?utf-8?q?1750945878051343248?= From: Frederic Weisbecker In preparation for RCU lazy changes, wake up the RCU nocb gp thread if needed after an entrain. This change prevents the RCU barrier callback from waiting in the queue for several seconds before the lazy callbacks in front of it are serviced. Reported-by: Joel Fernandes (Google) Signed-off-by: Frederic Weisbecker Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 11 +++++++++++ kernel/rcu/tree.h | 1 + kernel/rcu/tree_nocb.h | 5 +++++ 3 files changed, 17 insertions(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 6bb8e72bc8151..fb7a1b95af71e 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3894,6 +3894,8 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) { unsigned long gseq = READ_ONCE(rcu_state.barrier_sequence); unsigned long lseq = READ_ONCE(rdp->barrier_seq_snap); + bool wake_nocb = false; + bool was_alldone = false; lockdep_assert_held(&rcu_state.barrier_lock); if (rcu_seq_state(lseq) || !rcu_seq_state(gseq) || rcu_seq_ctr(lseq) != rcu_seq_ctr(gseq)) @@ -3902,7 +3904,14 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) rdp->barrier_head.func = rcu_barrier_callback; debug_rcu_head_queue(&rdp->barrier_head); rcu_nocb_lock(rdp); + /* + * Flush bypass and wakeup rcuog if we add callbacks to an empty regular + * queue. This way we don't wait for bypass timer that can reach seconds + * if it's fully lazy. + */ + was_alldone = rcu_rdp_is_offloaded(rdp) && !rcu_segcblist_pend_cbs(&rdp->cblist); WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies)); + wake_nocb = was_alldone && rcu_segcblist_pend_cbs(&rdp->cblist); if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head)) { atomic_inc(&rcu_state.barrier_cpu_count); } else { @@ -3910,6 +3919,8 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) rcu_barrier_trace(TPS("IRQNQ"), -1, rcu_state.barrier_sequence); } rcu_nocb_unlock(rdp); + if (wake_nocb) + wake_nocb_gp(rdp, false); smp_store_release(&rdp->barrier_seq_snap, gseq); } diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index d4a97e40ea9c3..925dd98f8b23b 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -439,6 +439,7 @@ static void zero_cpu_stall_ticks(struct rcu_data *rdp); static struct swait_queue_head *rcu_nocb_gp_get(struct rcu_node *rnp); static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq); static void rcu_init_one_nocb(struct rcu_node *rnp); +static bool wake_nocb_gp(struct rcu_data *rdp, bool force); static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, unsigned long j); static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index f77a6d7e13564..094fd454b6c38 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1558,6 +1558,11 @@ static void rcu_init_one_nocb(struct rcu_node *rnp) { } +static bool wake_nocb_gp(struct rcu_data *rdp, bool force) +{ + return false; +} + static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, unsigned long j) { From patchwork Wed Nov 30 18:13:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 27945 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1078758wrr; Wed, 30 Nov 2022 10:16:51 -0800 (PST) X-Google-Smtp-Source: AA0mqf6bErhGeNtc7vRg2oNe5qmihxFau5EqKkBkXUo/Bt4tlN+67exZiSdwnDCvf0Bvz6CZ/hur X-Received: by 2002:a05:6402:294b:b0:458:cd82:91a with SMTP id ed11-20020a056402294b00b00458cd82091amr56492436edb.15.1669832210836; Wed, 30 Nov 2022 10:16:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669832210; cv=none; d=google.com; s=arc-20160816; b=IU2hcZg937n87h/qCqUDeU6dHZsBlIgQgm1klZ7cIATiCT7mMAGK1N/kdWIsyf+5AZ ak7zYyp+TIpOyUQUUreBGSwEcs1wNu7+kixQfX+UocDBBCSpegblS97Z6KJ1aUve39xc qNEIXx7BF/oC9wV740hxQLGj50r4pFpy414fnZaXaUEKH27stQiGnnpW1lEgEtUhlGbP m9XqkQwIokOz47jMQEZtz71K05MOEpR3GOHGFcqi2VZsaNNXhLP0jFdaXQSw9i6lY1/X CV9iAMcUBs2Fx4kKrohVqKva9gIslFZTcPA6t5upRi83EM51F9yAF5RocWsizRLumwTn imcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=6keSclMWsJla3D76FB4EPTJc/HXtFw3/QSEjycKvbmo=; b=J7ESZW+XZ9uWw1dhV1bCA07+F/bAaifaR/dy6FslOkCnRl7cPSg0kBVBNmuGNiJotE mDOFZ1J2O4DgFZT/+qp3n7CZoi0DShZjyZgJKNsx1kWg/OvWINeiduTHpwRzKLbMdF3u I8VTQjTfrwuX315v+DhUK04ptXcF6chVQvMyyGtaxsVGU4rv66YGM185AlcrH4qG3fiQ IDygE8qQoRCP+gAIQBmB5JnXLtNZsmeHrLIFw1GT2XkcZFNTKSdH/gUpUgDXBAMh00MU 5EtdhEPmsyZDeSUFEh54ECzWHFW0UuHI78D3p+vGOPYqtQLVLEiWMDmBrxH0Da71xz3N OCWA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=pGimi8bT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id wy1-20020a170906fe0100b007940226fd21si1276814ejb.903.2022.11.30.10.16.25; Wed, 30 Nov 2022 10:16:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=pGimi8bT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230295AbiK3SOn (ORCPT + 99 others); Wed, 30 Nov 2022 13:14:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39150 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229854AbiK3SNd (ORCPT ); Wed, 30 Nov 2022 13:13:33 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 36685862D1; Wed, 30 Nov 2022 10:13:31 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 7A415CE1AA7; Wed, 30 Nov 2022 18:13:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AC43DC43470; Wed, 30 Nov 2022 18:13:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669832007; bh=to/XPhqpQV+NsRdl8wEReb+/zpVlXhyvzaZjN9tSi1k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pGimi8bTo2D351Uxe11qFg49gs4b3D6WviGX1+grbppC2D3Yakhs2zLZ3K5Hf/oaK yi6QEl2j0Ssqsg4RHi6zpE+YHDIu81rMITEBzK+0Qke/Q+MZATuFgdWOw+IF/pN2uS vsPfQ46fgPp4IE7B6Xqilbywm8CfpsbF5jTw2RIf7tiufagzO/me7CxjOBK2QNmyvr MFMv3Cl4K5LgUgHMBuQ/lldj9XXOLlDhPZjOWksBrDQzSnNPWBYjvaS2Xz3EoYR6j1 EH20TYjGGNOkt9fpRdS6tfbdOW5xLm2aINmhXSsI3XnFLYxsmuSCsRzYuAgWp91hoa SmtvdHYRK4nDA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 5C98C5C09C9; Wed, 30 Nov 2022 10:13:27 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , Paul McKenney , Frederic Weisbecker Subject: [PATCH rcu 04/16] rcu: Make call_rcu() lazy to save power Date: Wed, 30 Nov 2022 10:13:13 -0800 Message-Id: <20221130181325.1012760-4-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> References: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750945980412326820?= X-GMAIL-MSGID: =?utf-8?q?1750945980412326820?= From: "Joel Fernandes (Google)" Implement timer-based RCU callback batching (also known as lazy callbacks). With this we save about 5-10% of power consumed due to RCU requests that happen when system is lightly loaded or idle. By default, all async callbacks (queued via call_rcu) are marked lazy. An alternate API call_rcu_hurry() is provided for the few users, for example synchronize_rcu(), that need the old behavior. The batch is flushed whenever a certain amount of time has passed, or the batch on a particular CPU grows too big. Also memory pressure will flush it in a future patch. To handle several corner cases automagically (such as rcu_barrier() and hotplug), we re-use bypass lists which were originally introduced to address lock contention, to handle lazy CBs as well. The bypass list length has the lazy CB length included in it. A separate lazy CB length counter is also introduced to keep track of the number of lazy CBs. [ paulmck: Fix formatting of inline call_rcu_lazy() definition. ] [ paulmck: Apply Zqiang feedback. ] [ paulmck: Apply s/call_rcu_flush/call_rcu_hurry/ feedback from Tejun Heo. ] Suggested-by: Paul McKenney Acked-by: Frederic Weisbecker Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- include/linux/rcupdate.h | 9 +++ kernel/rcu/Kconfig | 8 ++ kernel/rcu/rcu.h | 8 ++ kernel/rcu/tiny.c | 2 +- kernel/rcu/tree.c | 129 ++++++++++++++++++++----------- kernel/rcu/tree.h | 11 ++- kernel/rcu/tree_exp.h | 2 +- kernel/rcu/tree_nocb.h | 159 +++++++++++++++++++++++++++++++-------- 8 files changed, 246 insertions(+), 82 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 08605ce7379d7..611c11383d236 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -108,6 +108,15 @@ static inline int rcu_preempt_depth(void) #endif /* #else #ifdef CONFIG_PREEMPT_RCU */ +#ifdef CONFIG_RCU_LAZY +void call_rcu_hurry(struct rcu_head *head, rcu_callback_t func); +#else +static inline void call_rcu_hurry(struct rcu_head *head, rcu_callback_t func) +{ + call_rcu(head, func); +} +#endif + /* Internal to kernel */ void rcu_init(void); extern int rcu_scheduler_active; diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index d471d22a5e21b..d78f6181c8aad 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -311,4 +311,12 @@ config TASKS_TRACE_RCU_READ_MB Say N here if you hate read-side memory barriers. Take the default if you are unsure. +config RCU_LAZY + bool "RCU callback lazy invocation functionality" + depends on RCU_NOCB_CPU + default n + help + To save power, batch RCU callbacks and flush after delay, memory + pressure, or callback list growing too big. + endmenu # "RCU Subsystem" diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index be5979da07f59..65704cbc9df7b 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -474,6 +474,14 @@ enum rcutorture_type { INVALID_RCU_FLAVOR }; +#if defined(CONFIG_RCU_LAZY) +unsigned long rcu_lazy_get_jiffies_till_flush(void); +void rcu_lazy_set_jiffies_till_flush(unsigned long j); +#else +static inline unsigned long rcu_lazy_get_jiffies_till_flush(void) { return 0; } +static inline void rcu_lazy_set_jiffies_till_flush(unsigned long j) { } +#endif + #if defined(CONFIG_TREE_RCU) void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags, unsigned long *gp_seq); diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index a33a8d4942c37..72913ce21258b 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -44,7 +44,7 @@ static struct rcu_ctrlblk rcu_ctrlblk = { void rcu_barrier(void) { - wait_rcu_gp(call_rcu); + wait_rcu_gp(call_rcu_hurry); } EXPORT_SYMBOL(rcu_barrier); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index fb7a1b95af71e..4b68e50312d95 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2728,47 +2728,8 @@ static void check_cb_ovld(struct rcu_data *rdp) raw_spin_unlock_rcu_node(rnp); } -/** - * call_rcu() - Queue an RCU callback for invocation after a grace period. - * @head: structure to be used for queueing the RCU updates. - * @func: actual callback function to be invoked after the grace period - * - * The callback function will be invoked some time after a full grace - * period elapses, in other words after all pre-existing RCU read-side - * critical sections have completed. However, the callback function - * might well execute concurrently with RCU read-side critical sections - * that started after call_rcu() was invoked. - * - * RCU read-side critical sections are delimited by rcu_read_lock() - * and rcu_read_unlock(), and may be nested. In addition, but only in - * v5.0 and later, regions of code across which interrupts, preemption, - * or softirqs have been disabled also serve as RCU read-side critical - * sections. This includes hardware interrupt handlers, softirq handlers, - * and NMI handlers. - * - * Note that all CPUs must agree that the grace period extended beyond - * all pre-existing RCU read-side critical section. On systems with more - * than one CPU, this means that when "func()" is invoked, each CPU is - * guaranteed to have executed a full memory barrier since the end of its - * last RCU read-side critical section whose beginning preceded the call - * to call_rcu(). It also means that each CPU executing an RCU read-side - * critical section that continues beyond the start of "func()" must have - * executed a memory barrier after the call_rcu() but before the beginning - * of that RCU read-side critical section. Note that these guarantees - * include CPUs that are offline, idle, or executing in user mode, as - * well as CPUs that are executing in the kernel. - * - * Furthermore, if CPU A invoked call_rcu() and CPU B invoked the - * resulting RCU callback function "func()", then both CPU A and CPU B are - * guaranteed to execute a full memory barrier during the time interval - * between the call to call_rcu() and the invocation of "func()" -- even - * if CPU A and CPU B are the same CPU (but again only if the system has - * more than one CPU). - * - * Implementation of these memory-ordering guarantees is described here: - * Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst. - */ -void call_rcu(struct rcu_head *head, rcu_callback_t func) +static void +__call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy) { static atomic_t doublefrees; unsigned long flags; @@ -2809,7 +2770,7 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) } check_cb_ovld(rdp); - if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags)) + if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) return; // Enqueued onto ->nocb_bypass, so just leave. // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. rcu_segcblist_enqueue(&rdp->cblist, head); @@ -2831,8 +2792,84 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) local_irq_restore(flags); } } -EXPORT_SYMBOL_GPL(call_rcu); +#ifdef CONFIG_RCU_LAZY +/** + * call_rcu_hurry() - Queue RCU callback for invocation after grace period, and + * flush all lazy callbacks (including the new one) to the main ->cblist while + * doing so. + * + * @head: structure to be used for queueing the RCU updates. + * @func: actual callback function to be invoked after the grace period + * + * The callback function will be invoked some time after a full grace + * period elapses, in other words after all pre-existing RCU read-side + * critical sections have completed. + * + * Use this API instead of call_rcu() if you don't want the callback to be + * invoked after very long periods of time, which can happen on systems without + * memory pressure and on systems which are lightly loaded or mostly idle. + * This function will cause callbacks to be invoked sooner than later at the + * expense of extra power. Other than that, this function is identical to, and + * reuses call_rcu()'s logic. Refer to call_rcu() for more details about memory + * ordering and other functionality. + */ +void call_rcu_hurry(struct rcu_head *head, rcu_callback_t func) +{ + return __call_rcu_common(head, func, false); +} +EXPORT_SYMBOL_GPL(call_rcu_hurry); +#endif + +/** + * call_rcu() - Queue an RCU callback for invocation after a grace period. + * By default the callbacks are 'lazy' and are kept hidden from the main + * ->cblist to prevent starting of grace periods too soon. + * If you desire grace periods to start very soon, use call_rcu_hurry(). + * + * @head: structure to be used for queueing the RCU updates. + * @func: actual callback function to be invoked after the grace period + * + * The callback function will be invoked some time after a full grace + * period elapses, in other words after all pre-existing RCU read-side + * critical sections have completed. However, the callback function + * might well execute concurrently with RCU read-side critical sections + * that started after call_rcu() was invoked. + * + * RCU read-side critical sections are delimited by rcu_read_lock() + * and rcu_read_unlock(), and may be nested. In addition, but only in + * v5.0 and later, regions of code across which interrupts, preemption, + * or softirqs have been disabled also serve as RCU read-side critical + * sections. This includes hardware interrupt handlers, softirq handlers, + * and NMI handlers. + * + * Note that all CPUs must agree that the grace period extended beyond + * all pre-existing RCU read-side critical section. On systems with more + * than one CPU, this means that when "func()" is invoked, each CPU is + * guaranteed to have executed a full memory barrier since the end of its + * last RCU read-side critical section whose beginning preceded the call + * to call_rcu(). It also means that each CPU executing an RCU read-side + * critical section that continues beyond the start of "func()" must have + * executed a memory barrier after the call_rcu() but before the beginning + * of that RCU read-side critical section. Note that these guarantees + * include CPUs that are offline, idle, or executing in user mode, as + * well as CPUs that are executing in the kernel. + * + * Furthermore, if CPU A invoked call_rcu() and CPU B invoked the + * resulting RCU callback function "func()", then both CPU A and CPU B are + * guaranteed to execute a full memory barrier during the time interval + * between the call to call_rcu() and the invocation of "func()" -- even + * if CPU A and CPU B are the same CPU (but again only if the system has + * more than one CPU). + * + * Implementation of these memory-ordering guarantees is described here: + * Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst. + */ +void call_rcu(struct rcu_head *head, rcu_callback_t func) +{ + return __call_rcu_common(head, func, IS_ENABLED(CONFIG_RCU_LAZY)); +} +EXPORT_SYMBOL_GPL(call_rcu); /* Maximum number of jiffies to wait before draining a batch. */ #define KFREE_DRAIN_JIFFIES (5 * HZ) @@ -3507,7 +3544,7 @@ void synchronize_rcu(void) if (rcu_gp_is_expedited()) synchronize_rcu_expedited(); else - wait_rcu_gp(call_rcu); + wait_rcu_gp(call_rcu_hurry); return; } @@ -3910,7 +3947,7 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) * if it's fully lazy. */ was_alldone = rcu_rdp_is_offloaded(rdp) && !rcu_segcblist_pend_cbs(&rdp->cblist); - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); wake_nocb = was_alldone && rcu_segcblist_pend_cbs(&rdp->cblist); if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head)) { atomic_inc(&rcu_state.barrier_cpu_count); @@ -4336,7 +4373,7 @@ void rcutree_migrate_callbacks(int cpu) my_rdp = this_cpu_ptr(&rcu_data); my_rnp = my_rdp->mynode; rcu_nocb_lock(my_rdp); /* irqs already disabled. */ - WARN_ON_ONCE(!rcu_nocb_flush_bypass(my_rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(my_rdp, NULL, jiffies, false)); raw_spin_lock_rcu_node(my_rnp); /* irqs already disabled. */ /* Leverage recent GPs and set GP for new callbacks. */ needwake = rcu_advance_cbs(my_rnp, rdp) || diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 925dd98f8b23b..fcb5d696eb170 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -263,14 +263,16 @@ struct rcu_data { unsigned long last_fqs_resched; /* Time of last rcu_resched(). */ unsigned long last_sched_clock; /* Jiffies of last rcu_sched_clock_irq(). */ + long lazy_len; /* Length of buffered lazy callbacks. */ int cpu; }; /* Values for nocb_defer_wakeup field in struct rcu_data. */ #define RCU_NOCB_WAKE_NOT 0 #define RCU_NOCB_WAKE_BYPASS 1 -#define RCU_NOCB_WAKE 2 -#define RCU_NOCB_WAKE_FORCE 3 +#define RCU_NOCB_WAKE_LAZY 2 +#define RCU_NOCB_WAKE 3 +#define RCU_NOCB_WAKE_FORCE 4 #define RCU_JIFFIES_TILL_FORCE_QS (1 + (HZ > 250) + (HZ > 500)) /* For jiffies_till_first_fqs and */ @@ -441,9 +443,10 @@ static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq); static void rcu_init_one_nocb(struct rcu_node *rnp); static bool wake_nocb_gp(struct rcu_data *rdp, bool force); static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - unsigned long j); + unsigned long j, bool lazy); static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags); + bool *was_alldone, unsigned long flags, + bool lazy); static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty, unsigned long flags); static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level); diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 18e9b4cd78ef8..ed6c3cce28f23 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -937,7 +937,7 @@ void synchronize_rcu_expedited(void) /* If expedited grace periods are prohibited, fall back to normal. */ if (rcu_gp_is_normal()) { - wait_rcu_gp(call_rcu); + wait_rcu_gp(call_rcu_hurry); return; } diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 094fd454b6c38..d6e4c076b0515 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -256,6 +256,31 @@ static bool wake_nocb_gp(struct rcu_data *rdp, bool force) return __wake_nocb_gp(rdp_gp, rdp, force, flags); } +/* + * LAZY_FLUSH_JIFFIES decides the maximum amount of time that + * can elapse before lazy callbacks are flushed. Lazy callbacks + * could be flushed much earlier for a number of other reasons + * however, LAZY_FLUSH_JIFFIES will ensure no lazy callbacks are + * left unsubmitted to RCU after those many jiffies. + */ +#define LAZY_FLUSH_JIFFIES (10 * HZ) +static unsigned long jiffies_till_flush = LAZY_FLUSH_JIFFIES; + +#ifdef CONFIG_RCU_LAZY +// To be called only from test code. +void rcu_lazy_set_jiffies_till_flush(unsigned long jif) +{ + jiffies_till_flush = jif; +} +EXPORT_SYMBOL(rcu_lazy_set_jiffies_till_flush); + +unsigned long rcu_lazy_get_jiffies_till_flush(void) +{ + return jiffies_till_flush; +} +EXPORT_SYMBOL(rcu_lazy_get_jiffies_till_flush); +#endif + /* * Arrange to wake the GP kthread for this NOCB group at some future * time when it is safe to do so. @@ -269,10 +294,14 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, raw_spin_lock_irqsave(&rdp_gp->nocb_gp_lock, flags); /* - * Bypass wakeup overrides previous deferments. In case - * of callback storm, no need to wake up too early. + * Bypass wakeup overrides previous deferments. In case of + * callback storms, no need to wake up too early. */ - if (waketype == RCU_NOCB_WAKE_BYPASS) { + if (waketype == RCU_NOCB_WAKE_LAZY && + rdp->nocb_defer_wakeup == RCU_NOCB_WAKE_NOT) { + mod_timer(&rdp_gp->nocb_timer, jiffies + jiffies_till_flush); + WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); + } else if (waketype == RCU_NOCB_WAKE_BYPASS) { mod_timer(&rdp_gp->nocb_timer, jiffies + 2); WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); } else { @@ -293,10 +322,13 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, * proves to be initially empty, just return false because the no-CB GP * kthread may need to be awakened in this case. * + * Return true if there was something to be flushed and it succeeded, otherwise + * false. + * * Note that this function always returns true if rhp is NULL. */ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - unsigned long j) + unsigned long j, bool lazy) { struct rcu_cblist rcl; @@ -310,7 +342,20 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, /* Note: ->cblist.len already accounts for ->nocb_bypass contents. */ if (rhp) rcu_segcblist_inc_len(&rdp->cblist); /* Must precede enqueue. */ - rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); + + /* + * If the new CB requested was a lazy one, queue it onto the main + * ->cblist so we can take advantage of a sooner grade period. + */ + if (lazy && rhp) { + rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, NULL); + rcu_cblist_enqueue(&rcl, rhp); + WRITE_ONCE(rdp->lazy_len, 0); + } else { + rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); + WRITE_ONCE(rdp->lazy_len, 0); + } + rcu_segcblist_insert_pend_cbs(&rdp->cblist, &rcl); WRITE_ONCE(rdp->nocb_bypass_first, j); rcu_nocb_bypass_unlock(rdp); @@ -326,13 +371,13 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, * Note that this function always returns true if rhp is NULL. */ static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - unsigned long j) + unsigned long j, bool lazy) { if (!rcu_rdp_is_offloaded(rdp)) return true; rcu_lockdep_assert_cblist_protected(rdp); rcu_nocb_bypass_lock(rdp); - return rcu_nocb_do_flush_bypass(rdp, rhp, j); + return rcu_nocb_do_flush_bypass(rdp, rhp, j, lazy); } /* @@ -345,7 +390,7 @@ static void rcu_nocb_try_flush_bypass(struct rcu_data *rdp, unsigned long j) if (!rcu_rdp_is_offloaded(rdp) || !rcu_nocb_bypass_trylock(rdp)) return; - WARN_ON_ONCE(!rcu_nocb_do_flush_bypass(rdp, NULL, j)); + WARN_ON_ONCE(!rcu_nocb_do_flush_bypass(rdp, NULL, j, false)); } /* @@ -367,12 +412,14 @@ static void rcu_nocb_try_flush_bypass(struct rcu_data *rdp, unsigned long j) * there is only one CPU in operation. */ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags) + bool *was_alldone, unsigned long flags, + bool lazy) { unsigned long c; unsigned long cur_gp_seq; unsigned long j = jiffies; long ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + bool bypass_is_lazy = (ncbs == READ_ONCE(rdp->lazy_len)); lockdep_assert_irqs_disabled(); @@ -417,25 +464,29 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, // If there hasn't yet been all that many ->cblist enqueues // this jiffy, tell the caller to enqueue onto ->cblist. But flush // ->nocb_bypass first. - if (rdp->nocb_nobypass_count < nocb_nobypass_lim_per_jiffy) { + // Lazy CBs throttle this back and do immediate bypass queuing. + if (rdp->nocb_nobypass_count < nocb_nobypass_lim_per_jiffy && !lazy) { rcu_nocb_lock(rdp); *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstQ")); - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, j)); + + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, j, false)); WARN_ON_ONCE(rcu_cblist_n_cbs(&rdp->nocb_bypass)); return false; // Caller must enqueue the callback. } // If ->nocb_bypass has been used too long or is too full, // flush ->nocb_bypass to ->cblist. - if ((ncbs && j != READ_ONCE(rdp->nocb_bypass_first)) || + if ((ncbs && !bypass_is_lazy && j != READ_ONCE(rdp->nocb_bypass_first)) || + (ncbs && bypass_is_lazy && + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush))) || ncbs >= qhimark) { rcu_nocb_lock(rdp); *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); - if (!rcu_nocb_flush_bypass(rdp, rhp, j)) { + if (!rcu_nocb_flush_bypass(rdp, rhp, j, lazy)) { if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstQ")); @@ -463,13 +514,24 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); rcu_segcblist_inc_len(&rdp->cblist); /* Must precede enqueue. */ rcu_cblist_enqueue(&rdp->nocb_bypass, rhp); + + if (lazy) + WRITE_ONCE(rdp->lazy_len, rdp->lazy_len + 1); + if (!ncbs) { WRITE_ONCE(rdp->nocb_bypass_first, j); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstBQ")); } rcu_nocb_bypass_unlock(rdp); smp_mb(); /* Order enqueue before wake. */ - if (ncbs) { + // A wake up of the grace period kthread or timer adjustment + // needs to be done only if: + // 1. Bypass list was fully empty before (this is the first + // bypass list entry), or: + // 2. Both of these conditions are met: + // a. The bypass list previously had only lazy CBs, and: + // b. The new CB is non-lazy. + if (ncbs && (!bypass_is_lazy || lazy)) { local_irq_restore(flags); } else { // No-CBs GP kthread might be indefinitely asleep, if so, wake. @@ -497,8 +559,10 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, unsigned long flags) __releases(rdp->nocb_lock) { + long bypass_len; unsigned long cur_gp_seq; unsigned long j; + long lazy_len; long len; struct task_struct *t; @@ -512,9 +576,16 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, } // Need to actually to a wakeup. len = rcu_segcblist_n_cbs(&rdp->cblist); + bypass_len = rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_len = READ_ONCE(rdp->lazy_len); if (was_alldone) { rdp->qlen_last_fqs_check = len; - if (!irqs_disabled_flags(flags)) { + // Only lazy CBs in bypass list + if (lazy_len && bypass_len == lazy_len) { + rcu_nocb_unlock_irqrestore(rdp, flags); + wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_LAZY, + TPS("WakeLazy")); + } else if (!irqs_disabled_flags(flags)) { /* ... if queue was empty ... */ rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); @@ -605,12 +676,12 @@ static void nocb_gp_sleep(struct rcu_data *my_rdp, int cpu) static void nocb_gp_wait(struct rcu_data *my_rdp) { bool bypass = false; - long bypass_ncbs; int __maybe_unused cpu = my_rdp->cpu; unsigned long cur_gp_seq; unsigned long flags; bool gotcbs = false; unsigned long j = jiffies; + bool lazy = false; bool needwait_gp = false; // This prevents actual uninitialized use. bool needwake; bool needwake_gp; @@ -640,24 +711,43 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) * won't be ignored for long. */ list_for_each_entry(rdp, &my_rdp->nocb_head_rdp, nocb_entry_rdp) { + long bypass_ncbs; + bool flush_bypass = false; + long lazy_ncbs; + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check")); rcu_nocb_lock_irqsave(rdp, flags); lockdep_assert_held(&rdp->nocb_lock); bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); - if (bypass_ncbs && + lazy_ncbs = READ_ONCE(rdp->lazy_len); + + if (bypass_ncbs && (lazy_ncbs == bypass_ncbs) && + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush) || + bypass_ncbs > 2 * qhimark)) { + flush_bypass = true; + } else if (bypass_ncbs && (lazy_ncbs != bypass_ncbs) && (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) || bypass_ncbs > 2 * qhimark)) { - // Bypass full or old, so flush it. - (void)rcu_nocb_try_flush_bypass(rdp, j); - bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + flush_bypass = true; } else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) { rcu_nocb_unlock_irqrestore(rdp, flags); continue; /* No callbacks here, try next. */ } + + if (flush_bypass) { + // Bypass full or old, so flush it. + (void)rcu_nocb_try_flush_bypass(rdp, j); + bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_ncbs = READ_ONCE(rdp->lazy_len); + } + if (bypass_ncbs) { trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, - TPS("Bypass")); - bypass = true; + bypass_ncbs == lazy_ncbs ? TPS("Lazy") : TPS("Bypass")); + if (bypass_ncbs == lazy_ncbs) + lazy = true; + else + bypass = true; } rnp = rdp->mynode; @@ -705,12 +795,20 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) my_rdp->nocb_gp_gp = needwait_gp; my_rdp->nocb_gp_seq = needwait_gp ? wait_gp_seq : 0; - if (bypass && !rcu_nocb_poll) { - // At least one child with non-empty ->nocb_bypass, so set - // timer in order to avoid stranding its callbacks. - wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_BYPASS, - TPS("WakeBypassIsDeferred")); + // At least one child with non-empty ->nocb_bypass, so set + // timer in order to avoid stranding its callbacks. + if (!rcu_nocb_poll) { + // If bypass list only has lazy CBs. Add a deferred lazy wake up. + if (lazy && !bypass) { + wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_LAZY, + TPS("WakeLazyIsDeferred")); + // Otherwise add a deferred bypass wake up. + } else if (bypass) { + wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_BYPASS, + TPS("WakeBypassIsDeferred")); + } } + if (rcu_nocb_poll) { /* Polling, so trace if first poll in the series. */ if (gotcbs) @@ -1036,7 +1134,7 @@ static long rcu_nocb_rdp_deoffload(void *arg) * return false, which means that future calls to rcu_nocb_try_bypass() * will refuse to put anything into the bypass. */ - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); /* * Start with invoking rcu_core() early. This way if the current thread * happens to preempt an ongoing call to rcu_core() in the middle, @@ -1278,6 +1376,7 @@ static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp) raw_spin_lock_init(&rdp->nocb_gp_lock); timer_setup(&rdp->nocb_timer, do_nocb_deferred_wakeup_timer, 0); rcu_cblist_init(&rdp->nocb_bypass); + WRITE_ONCE(rdp->lazy_len, 0); mutex_init(&rdp->nocb_gp_kthread_mutex); } @@ -1564,13 +1663,13 @@ static bool wake_nocb_gp(struct rcu_data *rdp, bool force) } static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - unsigned long j) + unsigned long j, bool lazy) { return true; } static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags) + bool *was_alldone, unsigned long flags, bool lazy) { return false; } From patchwork Wed Nov 30 18:13:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 27942 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1078400wrr; Wed, 30 Nov 2022 10:16:02 -0800 (PST) X-Google-Smtp-Source: AA0mqf5H3xImBtmjpPzycZ1gmhZ/u03frBDLmoPbF0TJ+G8dwLVsKlB5DVVcJ81mr4hL2xeP9i4j X-Received: by 2002:a05:6402:5305:b0:467:69e3:c25b with SMTP id eo5-20020a056402530500b0046769e3c25bmr56245432edb.3.1669832162635; Wed, 30 Nov 2022 10:16:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669832162; cv=none; d=google.com; s=arc-20160816; b=JIi2OChZ2yWQkMNxEz1Tlgl/LYHE15WMBYDNQloA1O2eKn8cmWmBnB95EOuyXfIgtl qDHFfMwG+/5/5fLnGWSN2mcFYhVF79OdWVgW/ZqyoP9oobczP0H/g36pgXoRI3iFK5lX R4yJytk9WHGlmupJcpyBt//J4omlMK/Q4Q4367VW/HTmxGjboj+WofoyWbOuO6hwpbKT jD0RpGVMkZY/XhcScQDX8VwubTtlOgTk6wSllNDvFODm5FGMOJaIi5BVWQ9Cl8dtaYfr C1Qt+CxECkbys1Vs1qN1GPk9GB7yDEtEIbyA8AgGOE5n+jYGl5kUiFI+XepiwfTbQs84 jo0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=R3uaRFK395oD/p+1i+YQ/VTBTHwx4oJTVy8oJbcqVmw=; b=QvHW2iUMrll86DDDnHlYEGm4QNaayl5ghMLxUm/26qxPwshuUQnmzOZZ+hLRgb5aVu D+7KoH4BXosMhwWV6cbHbyrrLN+JgRvDK3b3eppSPlIolZLfEV0/3aUKkJJehWp00mpC gWlZNvTR8HVOP4iaUg5Zk1tJWy9Dd7bDYly4Stl5Cikye3eJp+DXm2JB7cwTOMTORuvg 2VoovzShNsu/bVkFBxtKVTxbIRDgxOuG+ZT4tG6gcsfEgy74AZ2RehyjttbOO8zjWJ2+ LRuc8XYxt+xZ2iANz82tc1DL/3wq1+v0bmChDMSj9xSIyV6BC5HT6lKC4YRkP/CO8atz Za7g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=pGXc5mgC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id di10-20020a170906730a00b007aea0bbe2e2si2199927ejc.394.2022.11.30.10.15.37; Wed, 30 Nov 2022 10:16:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=pGXc5mgC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230259AbiK3SOY (ORCPT + 99 others); Wed, 30 Nov 2022 13:14:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39052 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230117AbiK3SNb (ORCPT ); Wed, 30 Nov 2022 13:13:31 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 63B29862D3; Wed, 30 Nov 2022 10:13:30 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 1430BB81C9B; Wed, 30 Nov 2022 18:13:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A78D8C433D7; Wed, 30 Nov 2022 18:13:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669832007; bh=PS6L08FWflyTwRyMdRxiUT1KwyXbAuVOfA4oJ9gzHj4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pGXc5mgCEhUZNWYca5bydsevRHCbD4DO79w1BS8KcYR7QYHB9zGV0G0A4fauxBfHh 5I5ab21+L/vX8s6hvLXGlgwd0x6QD9xFKdj32OX3EmAH2dB0OubMvStmc2il1CLL5/ 5kNGkiri929t2+8e/nJY9xRKK/CQnnKPpo7JIagwvvPV1Sg5h11usqvCBSoXx4QTT8 Mk17SV3zMXp0jSFua5jMhHhUeD/tfP9/ML1pXsyw+1yPCPGlkt7Hcdtcg69ow3yryx ZVg/uvhtcNf9DfyZl6lNLCGc6Vz0N77HpD6iRbJquWS+tUsqHYZzjYdwAi4sXUXzTw xAnA0LIGox9eA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 5E8AC5C0BA0; Wed, 30 Nov 2022 10:13:27 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , "Paul E . McKenney" , Frederic Weisbecker Subject: [PATCH rcu 05/16] rcu: Refactor code a bit in rcu_nocb_do_flush_bypass() Date: Wed, 30 Nov 2022 10:13:14 -0800 Message-Id: <20221130181325.1012760-5-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> References: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750945929427124121?= X-GMAIL-MSGID: =?utf-8?q?1750945929427124121?= From: "Joel Fernandes (Google)" This consolidates the code a bit and makes it cleaner. Functionally it is the same. Reported-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) Reviewed-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index d6e4c076b0515..213daf81c057f 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -327,10 +327,11 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, * * Note that this function always returns true if rhp is NULL. */ -static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, +static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp_in, unsigned long j, bool lazy) { struct rcu_cblist rcl; + struct rcu_head *rhp = rhp_in; WARN_ON_ONCE(!rcu_rdp_is_offloaded(rdp)); rcu_lockdep_assert_cblist_protected(rdp); @@ -345,16 +346,16 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, /* * If the new CB requested was a lazy one, queue it onto the main - * ->cblist so we can take advantage of a sooner grade period. + * ->cblist so that we can take advantage of the grace-period that will + * happen regardless. But queue it onto the bypass list first so that + * the lazy CB is ordered with the existing CBs in the bypass list. */ if (lazy && rhp) { - rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, NULL); - rcu_cblist_enqueue(&rcl, rhp); - WRITE_ONCE(rdp->lazy_len, 0); - } else { - rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); - WRITE_ONCE(rdp->lazy_len, 0); + rcu_cblist_enqueue(&rdp->nocb_bypass, rhp); + rhp = NULL; } + rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); + WRITE_ONCE(rdp->lazy_len, 0); rcu_segcblist_insert_pend_cbs(&rdp->cblist, &rcl); WRITE_ONCE(rdp->nocb_bypass_first, j); From patchwork Wed Nov 30 18:13:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 27934 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1077858wrr; Wed, 30 Nov 2022 10:15:04 -0800 (PST) X-Google-Smtp-Source: AA0mqf7Z5TQ9z1hnIN5nv1iqwRpoHW1L05vF+20IrKUBhpIFDk1hCeqpBUIwPi4fykLFmzcBQM2f X-Received: by 2002:a17:907:d047:b0:7ba:530:223a with SMTP id vb7-20020a170907d04700b007ba0530223amr30642415ejc.215.1669832104280; Wed, 30 Nov 2022 10:15:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669832104; cv=none; d=google.com; s=arc-20160816; b=VIN1FQfIEqxzYAPNS09RJou1Mfb3gmJgLC78VYWoGorfibFgpIOKTIqnK4D3Y1pU/o 35c9zyrz/r2t33BBwAjkL1fz2+85ibkSkLMDPmEjbgTacFuiz0diVQMPcUuqvTxZ0smq FXYjYspTXwgoAtDwu7s/T/+W9fyZN/x9zt3kT9MEB+yG2n/2j9jH/IAESHEvMHVKg8Ye Qx0vOrYUxVAivelFuwrYUxGtJiKU5g1Pc4Eo9LdyFX91WL7pUiE6HtrTzGiD43DhD+Pg 6Yv+mqvRfiXovPyJMT3H43QJA7KPq7duvsEqSKauuWLmISbQkclcXiLwU+6/kT5k9Dyv JOuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KpkkHRs6NSlaA30zXYSvuq9VaM3mDcAD5+jXqlQvvlo=; b=bsMmOLLnCss2uHInWxPEYw7IEd3QrlThUl0onISHPLGalpUG+ZsL6//7wkoPWl08B1 IjPagJemUE96ul0XID0NmHn+WGxPs2e45MyMphuC/4qL/DO9wr6ekS8gBY/Z/EmdC+Jc 4LHZNn89oxc973Tk3cKVtXxlGIVSUcEHxdcyZcrFx8+uoDJGg6oEz/5HXKPFdt2N6rOM GNqsFm8/ln90ft+VvjGDitkSlrmlYXfKPPraEYPmb9IJv9UuPGpYq6+aS+E3aI5AlRU4 faVG20AAo3k1JEkKE2uU4xZOQsHe0nPZf2VQCJhWmADRB80yWswJyVJ1t9vGiL4gKMST 2NmA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ZcqbaoCw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gt11-20020a1709072d8b00b007a20c586f4esi2032590ejc.876.2022.11.30.10.14.39; Wed, 30 Nov 2022 10:15:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ZcqbaoCw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230103AbiK3SNv (ORCPT + 99 others); Wed, 30 Nov 2022 13:13:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229974AbiK3SNa (ORCPT ); Wed, 30 Nov 2022 13:13:30 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F20C880D5; Wed, 30 Nov 2022 10:13:29 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id DDDCC61D59; Wed, 30 Nov 2022 18:13:28 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EF12BC43150; Wed, 30 Nov 2022 18:13:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669832008; bh=hIFNQHcIbY8YtBkmBJx5icL1buQHNTtIxJeozRTrMW8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZcqbaoCwWgpfTBjbKPRFJJwRPJc5Ejc/7PIei5D9g6OZD+vcLZL44CqUzletkgDmn kmTWhtbegh8kbXGBylPtd1ALYn4P73Qd09zqgGld2TDoc3I4TMgmOxh0cyfMkOqOpu ZWtPl4JshImqKuU1LHowxQdfmR5kp6HqByOBJIq/kv/AgQydFU4yx4gFRqF2tzUT2X N+Ri6k718HDvY4R5aByGoXEDtUeJQU2zz9Dsf/P3gUJVhUqW20mGH2Oo6BBpPsftsa vNrtqqiRXNPbtUdm21PaHp2OTpsVqrYRya38dDVLf1ceFxbB7PXX1TQT6xSqUBU2gu 0mECVh3dOkCwQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 607D65C0F36; Wed, 30 Nov 2022 10:13:27 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Vineeth Pillai , Joel Fernandes , "Paul E . McKenney" Subject: [PATCH rcu 06/16] rcu: Shrinker for lazy rcu Date: Wed, 30 Nov 2022 10:13:15 -0800 Message-Id: <20221130181325.1012760-6-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> References: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750945868541366097?= X-GMAIL-MSGID: =?utf-8?q?1750945868541366097?= From: Vineeth Pillai The shrinker is used to speed up the free'ing of memory potentially held by RCU lazy callbacks. RCU kernel module test cases show this to be effective. Test is introduced in a later patch. Signed-off-by: Vineeth Pillai Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 52 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 213daf81c057f..9e1c8caec5ceb 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1312,6 +1312,55 @@ int rcu_nocb_cpu_offload(int cpu) } EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload); +static unsigned long +lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + + count += READ_ONCE(rdp->lazy_len); + } + + return count ? count : SHRINK_EMPTY; +} + +static unsigned long +lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu; + unsigned long flags; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + int _count = READ_ONCE(rdp->lazy_len); + + if (_count == 0) + continue; + rcu_nocb_lock_irqsave(rdp, flags); + WRITE_ONCE(rdp->lazy_len, 0); + rcu_nocb_unlock_irqrestore(rdp, flags); + wake_nocb_gp(rdp, false); + sc->nr_to_scan -= _count; + count += _count; + if (sc->nr_to_scan <= 0) + break; + } + return count ? count : SHRINK_STOP; +} + +static struct shrinker lazy_rcu_shrinker = { + .count_objects = lazy_rcu_shrink_count, + .scan_objects = lazy_rcu_shrink_scan, + .batch = 0, + .seeks = DEFAULT_SEEKS, +}; + void __init rcu_init_nohz(void) { int cpu; @@ -1342,6 +1391,9 @@ void __init rcu_init_nohz(void) if (!rcu_state.nocb_is_setup) return; + if (register_shrinker(&lazy_rcu_shrinker, "rcu-lazy")) + pr_err("Failed to register lazy_rcu shrinker!\n"); + if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) { pr_info("\tNote: kernel parameter 'rcu_nocbs=', 'nohz_full', or 'isolcpus=' contains nonexistent CPUs.\n"); cpumask_and(rcu_nocb_mask, cpu_possible_mask, From patchwork Wed Nov 30 18:13:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 27935 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1077910wrr; Wed, 30 Nov 2022 10:15:11 -0800 (PST) X-Google-Smtp-Source: AA0mqf48GoihaBVanZGElTOsPTXHOFGkvnGwho9IGUfJ9RHM6M2xKSK+yjS4mpCKHC11IH/d8bxg X-Received: by 2002:a17:906:1695:b0:7aa:493b:679 with SMTP id s21-20020a170906169500b007aa493b0679mr37087919ejd.320.1669832111167; Wed, 30 Nov 2022 10:15:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669832111; cv=none; d=google.com; s=arc-20160816; b=1DbTPXUq+l2xlvhO4pyf77dzo1LpjdcCKValUyMWLFh7ExEYIN/+MJzkLjEdxZNYTO 9GEzxbAqpZz27ynzW6pXQWuAXLkaXPsU3ya2vIKi7y7Pj0juOfKR7xdK9SsYWJZ+Osgl z5I2KkXBw+DDuM7SHE2iGUO39GwaqUlMfAefCPuDBN2Bk3132Q3Ad7LAx0QQCxaZic1K oxOX7WhS9Y4TK0UnwlM3E/oH0GraUMBH22GIlI8zZRKaLy/4RZhQ285DAmluGMgAV+sk ynkTr0hUeb/dYLxur2BNjYMkkNHiPV3pX81IOcmdIhM+cYGRvJ4pVh65tZt4J03SIx+A RnRQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ZHuFqUUNuNBBbGtm2obtlgmElzjDDyVm88R/Db4kxYU=; b=Bm+Xi4sZzNGZaUGfeyZAF3CvtPF+F8A3TIiKCVQV4A08hxDG7xERwkZEfT1LoMfqq+ Ct+ZeorI8H4vXw4F32u0ojiRlxWC5JifraPAVq29r8dFn5ybt0tGGE8oNkgKjYpB973w J5OY8jkqV0psvy0SBo8xtTXdb4nb+TTchbf2uEEQE7M6HnECkdh6lWZqofk82YLWOy4C qiCRtyKLYskq+zbpLOfhaVjEQdiSx3j3kZfASnAISSGj0B8uwJN+6AvFiRIcBMqIXqp1 jqh25Rs2MJKU/3vyTwnLHKBNiIn19nFhMKfis8qD8STzk9e020hblkPQT4zp6itpxB7M gvTg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=mfxSe3go; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y17-20020a056402271100b004612505ece4si2022999edd.483.2022.11.30.10.14.46; Wed, 30 Nov 2022 10:15:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=mfxSe3go; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230171AbiK3SN6 (ORCPT + 99 others); Wed, 30 Nov 2022 13:13:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229995AbiK3SNa (ORCPT ); Wed, 30 Nov 2022 13:13:30 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80D2C862C9; Wed, 30 Nov 2022 10:13:29 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 17FDF61D5C; Wed, 30 Nov 2022 18:13:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F3625C43151; Wed, 30 Nov 2022 18:13:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669832008; bh=B649m1ZX6zSaxE2rwxoHbFbQjFjTg7X6m3NxA9r7zOA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mfxSe3go/rNNMfTBQnQyJaJaHF/0vxwxJugp+gTMgidOOl7XzZFzW6lt6iAxEfszC CTZOz52+VrmRWHwuVmNzFguPfy/ApYLBKyZjKAscf+hX6p1B0vFqWmwtj3/dhs53dv XK+rk7SMsaplB3m/ektanHjB+/ZngnNe+gI8IEXIfg3uUBxKlFnnIwNtT8NDczzvUi rvci2gYKPco/tNgIKh2mcT4wula/mw7MDkPQ6tjzwPdffk2XW+atTYbj2QUi6cih7Q O0Rskjv3AT7ppaFhK45PUYI57MC6g1o8TElFYKmT+YfUjP3paDLa5Oc2azScYCPh2h 2qTBaSL/gq46Q== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 624275C14A0; Wed, 30 Nov 2022 10:13:27 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , "Paul E . McKenney" Subject: [PATCH rcu 07/16] rcuscale: Add laziness and kfree tests Date: Wed, 30 Nov 2022 10:13:16 -0800 Message-Id: <20221130181325.1012760-7-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> References: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750945876169524722?= X-GMAIL-MSGID: =?utf-8?q?1750945876169524722?= From: "Joel Fernandes (Google)" This commit adds 2 tests to rcuscale. The first one is a startup test to check whether we are not too lazy or too hard working. The second one causes kfree_rcu() itself to use call_rcu() and checks memory pressure. Testing indicates that the new call_rcu() keeps memory pressure under control roughly as well as does kfree_rcu(). [ paulmck: Apply checkpatch feedback. ] Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/rcuscale.c | 67 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 65 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 3ef02d4a81085..3baded807a616 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -95,6 +95,7 @@ torture_param(int, verbose, 1, "Enable verbose debugging printk()s"); torture_param(int, writer_holdoff, 0, "Holdoff (us) between GPs, zero to disable"); torture_param(int, kfree_rcu_test, 0, "Do we run a kfree_rcu() scale test?"); torture_param(int, kfree_mult, 1, "Multiple of kfree_obj size to allocate."); +torture_param(int, kfree_by_call_rcu, 0, "Use call_rcu() to emulate kfree_rcu()?"); static char *scale_type = "rcu"; module_param(scale_type, charp, 0444); @@ -659,6 +660,14 @@ struct kfree_obj { struct rcu_head rh; }; +/* Used if doing RCU-kfree'ing via call_rcu(). */ +static void kfree_call_rcu(struct rcu_head *rh) +{ + struct kfree_obj *obj = container_of(rh, struct kfree_obj, rh); + + kfree(obj); +} + static int kfree_scale_thread(void *arg) { @@ -696,6 +705,11 @@ kfree_scale_thread(void *arg) if (!alloc_ptr) return -ENOMEM; + if (kfree_by_call_rcu) { + call_rcu(&(alloc_ptr->rh), kfree_call_rcu); + continue; + } + // By default kfree_rcu_test_single and kfree_rcu_test_double are // initialized to false. If both have the same value (false or true) // both are randomly tested, otherwise only the one with value true @@ -767,11 +781,58 @@ kfree_scale_shutdown(void *arg) return -EINVAL; } +// Used if doing RCU-kfree'ing via call_rcu(). +static unsigned long jiffies_at_lazy_cb; +static struct rcu_head lazy_test1_rh; +static int rcu_lazy_test1_cb_called; +static void call_rcu_lazy_test1(struct rcu_head *rh) +{ + jiffies_at_lazy_cb = jiffies; + WRITE_ONCE(rcu_lazy_test1_cb_called, 1); +} + static int __init kfree_scale_init(void) { - long i; int firsterr = 0; + long i; + unsigned long jif_start; + unsigned long orig_jif; + + // Also, do a quick self-test to ensure laziness is as much as + // expected. + if (kfree_by_call_rcu && !IS_ENABLED(CONFIG_RCU_LAZY)) { + pr_alert("CONFIG_RCU_LAZY is disabled, falling back to kfree_rcu() for delayed RCU kfree'ing\n"); + kfree_by_call_rcu = 0; + } + + if (kfree_by_call_rcu) { + /* do a test to check the timeout. */ + orig_jif = rcu_lazy_get_jiffies_till_flush(); + + rcu_lazy_set_jiffies_till_flush(2 * HZ); + rcu_barrier(); + + jif_start = jiffies; + jiffies_at_lazy_cb = 0; + call_rcu(&lazy_test1_rh, call_rcu_lazy_test1); + + smp_cond_load_relaxed(&rcu_lazy_test1_cb_called, VAL == 1); + + rcu_lazy_set_jiffies_till_flush(orig_jif); + + if (WARN_ON_ONCE(jiffies_at_lazy_cb - jif_start < 2 * HZ)) { + pr_alert("ERROR: call_rcu() CBs are not being lazy as expected!\n"); + WARN_ON_ONCE(1); + return -1; + } + + if (WARN_ON_ONCE(jiffies_at_lazy_cb - jif_start > 3 * HZ)) { + pr_alert("ERROR: call_rcu() CBs are being too lazy!\n"); + WARN_ON_ONCE(1); + return -1; + } + } kfree_nrealthreads = compute_real(kfree_nthreads); /* Start up the kthreads. */ @@ -784,7 +845,9 @@ kfree_scale_init(void) schedule_timeout_uninterruptible(1); } - pr_alert("kfree object size=%zu\n", kfree_mult * sizeof(struct kfree_obj)); + pr_alert("kfree object size=%zu, kfree_by_call_rcu=%d\n", + kfree_mult * sizeof(struct kfree_obj), + kfree_by_call_rcu); kfree_reader_tasks = kcalloc(kfree_nrealthreads, sizeof(kfree_reader_tasks[0]), GFP_KERNEL); From patchwork Wed Nov 30 18:13:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 27944 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1078423wrr; Wed, 30 Nov 2022 10:16:05 -0800 (PST) X-Google-Smtp-Source: AA0mqf54lY05KtP3YcSbi0Ms4ZsLo/QpC4Aj8GvwhfvrvJCfb7DAiOhQnidMX8Xn9F0M4lt9b/wb X-Received: by 2002:a05:6402:28c9:b0:469:ba7c:bc75 with SMTP id ef9-20020a05640228c900b00469ba7cbc75mr1547088edb.286.1669832165293; Wed, 30 Nov 2022 10:16:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669832165; cv=none; d=google.com; s=arc-20160816; b=y/0ty8LJ5YZ99LdjKHDokB2lMvcv+gBOwoCEzX1oIYURAIdafPDmEx3nxd1Da84aiw 3og8MIOtoAvIfUqj0p7LvMqC271HvZVhT3YexLtLY7Mrvh4OWKqUdqd/kCCnDXIzWOzn sP8AAVVCuNuVLaQJfUS1GG0xQ9bzWNKMWkcWcc1ZAu1Ou/1NvWUwM0eQTbkBXoHhW6zF Eq/sqynTYa2LlWsJXHOA7oVDk2z3qnK2IVAMHt8IubXtQBUcVKBi/F5hYNtoaB8hnc6I IFhlPLbh5sMq49th70YNAwRFpv3sTUD6Vao6A8IgGF08W6rCicfAaQ9D+rkqiRHoxxeC 0W0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Jcrz8UWs1g+pcWCvyNn4dQJdOvSc+3K6bxQqvbL4diI=; b=I174c85YnusQ6QTak3yBnulkPujA4jMLJB55YS1YOfSqmJSk9knJpdWeyLgrsdxHW3 yk/gLSNdGZRgsy4w+Uf5Xb5E24EcGtD/OyISd3doIj3hU9Y8T2F833+Z30KrqGK14REo EqhykMlMworeWcureYY/I8Meu4QiugKh7GxKoKyg/B9H6/AvNIbbFgS3GB8HfSkd+j8a b7hd/6mT7p91tgB+a5VZ9mgGOhlalN9dCZ02kk632N3PsC7f3caFatxeAWny2xB1xEjO S3Hv6zTjtfjpFJdQk4zK6M58O92M7dTNl9q9jtfa4mcNRMKpYgsHtGP29xZrrPpGf3JT jwrA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=FjfU2klt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fe17-20020a1709072a5100b007adffbaa908si1295347ejc.916.2022.11.30.10.15.41; Wed, 30 Nov 2022 10:16:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=FjfU2klt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230266AbiK3SOe (ORCPT + 99 others); Wed, 30 Nov 2022 13:14:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229935AbiK3SNc (ORCPT ); Wed, 30 Nov 2022 13:13:32 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AAF24880E6; Wed, 30 Nov 2022 10:13:31 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 0CC9ECE1AD0; Wed, 30 Nov 2022 18:13:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EEEA6C4314D; Wed, 30 Nov 2022 18:13:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669832008; bh=MK8lzAbrXYSo4oOc8yMliggTzQ2OiQszviHhFcYo3R0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FjfU2kltztzGEEUXpT6+A2WCfTFbhIFo9O3pV8l+Ck6OClIeYBR8pxnnK7hu8gWgy rioIlnCjCJP9JdqMpzTz41HVKQWT/xosRznfzCu6P1//+IAtZrjDhSYUZoHCNrLsDi LTmAE61tiDqfFpQfQlPD14AYIrt/oMrqt8fTYdNITNJog2uWgxA1aATWO9MX5u31tZ Lit0BwOw+OzHWwRMlNZBCU83hD4khg4XDZC2KxSvQ85bUzremDXiPde7Pzpwzpzs0c 7SN1iHzydYSrmNjtY7UvAsNuiyc7FbcqK9nhpjYCIIs5/4KGrz55k15Ru/9BY3ogon 4LhQG/tfGAwVw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 63FAA5C14B8; Wed, 30 Nov 2022 10:13:27 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , "Paul E . McKenney" Subject: [PATCH rcu 08/16] rcu/sync: Use call_rcu_hurry() instead of call_rcu Date: Wed, 30 Nov 2022 10:13:17 -0800 Message-Id: <20221130181325.1012760-8-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> References: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750945932477722526?= X-GMAIL-MSGID: =?utf-8?q?1750945932477722526?= From: "Joel Fernandes (Google)" call_rcu() changes to save power will slow down rcu sync. Use the call_rcu_hurry() API instead which reverts to the old behavior. [ paulmck: Apply s/call_rcu_flush/call_rcu_hurry/ feedback from Tejun Heo. ] Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/sync.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/sync.c b/kernel/rcu/sync.c index 5cefc702158fe..e550f97779b8d 100644 --- a/kernel/rcu/sync.c +++ b/kernel/rcu/sync.c @@ -44,7 +44,7 @@ static void rcu_sync_func(struct rcu_head *rhp); static void rcu_sync_call(struct rcu_sync *rsp) { - call_rcu(&rsp->cb_head, rcu_sync_func); + call_rcu_hurry(&rsp->cb_head, rcu_sync_func); } /** From patchwork Wed Nov 30 18:13:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 27937 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1078046wrr; Wed, 30 Nov 2022 10:15:24 -0800 (PST) X-Google-Smtp-Source: AA0mqf5GssL/kBhSd+DxAIa8uEdPiDniqP+OfzWIlA4N2QPdEFvE3mZ2oxpiOi1j1+hh1V9kfg04 X-Received: by 2002:a05:6402:1f03:b0:468:7be6:55e7 with SMTP id b3-20020a0564021f0300b004687be655e7mr56789270edb.345.1669832124164; Wed, 30 Nov 2022 10:15:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669832124; cv=none; d=google.com; s=arc-20160816; b=rsT0mu1bEf47mwXygDA0TeSN59X5EAQbayJMg+Mbe9VUQTnkSmGsk4atEmyarNrM/K PS83NrfjxnJACDSMdoYXkYghnETXdCmJYCmX5Gfl6t8PUQRbPVahoqwu4tt6JTlR2MEQ o4BJfFvNTzvI+oRPlZ1sVpCBimwTghet6/UOkYDJIKI9iBHBliwO7Zaho3SjRjFN2LRC 2e6iJdDrwfaZMeBRj5Io/xEpQGiMRU377wHzTqbW8oTXw5+RrZtPCFqnnIZ1OH5zh93U ZkqBDC4Ndlc9zhcjxqlaPOFqjq9Zs/JEUQ7dLcr6y1SoPmqGGZjJoKgOGne+Nk4MRtvt 5viA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Zig+YJsTEZ14XKgwC2gAUteOOpWn4KzNLVDHPJ5f60U=; b=n+RuxzJRNYsLQ7bZfpFWVBZH9aN4WhVyY5mr02FVxwoJ6NQA1KPkjT5JWQUJJ9whDL LGJLRY7XHTOYQC6mXztSyudQoPkTsBDvj8w+PWxBMl4YK6gLc3XTJSvgYpq8xkhXywwH OeBawUEutE9coWD0ON0RqS5vPFaMo9zIvUOyi2LJGekiepG4W5EgMYALxWh9CqS+t+N0 N8LvPTa3RCoz8V01pWph3ntXiWX0z/aVpz57hcfUu1kgY85dXMwUpyipzQ1pv6qZ9QGz Eq/UqTHnweESORRq7Kb+Tfh0CQ+iRVIJ+xC0RTBynPE9/xmqMk587XBf/fB0uGEOcwuH tfOg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=RrdwL+VE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b20-20020a056402279400b00468b9d932c6si1854410ede.258.2022.11.30.10.14.51; Wed, 30 Nov 2022 10:15:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=RrdwL+VE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230210AbiK3SOF (ORCPT + 99 others); Wed, 30 Nov 2022 13:14:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229998AbiK3SNb (ORCPT ); Wed, 30 Nov 2022 13:13:31 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB69C880E0; Wed, 30 Nov 2022 10:13:29 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 464DF61D61; Wed, 30 Nov 2022 18:13:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0E148C43153; Wed, 30 Nov 2022 18:13:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669832008; bh=NFAW1d1m+42EgepEaaKguo2hXsVIBXe6LMvkxK6/6ng=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RrdwL+VEqHefMoZWMlt0MZVQrCX0hEEnO1UrqAtETnTeRw4rvuEZwfy/oai95PA2+ w0YezOcB9Cjpxr+sGIwHjhI8eLE0QvwV45qGjqZFH8lPxtmaa5QVtGqedkfjk3pvU9 1eysAwZV7m1EnR7tq7y463fBh+pZC3J6vwUcQ7ZAYAjDx2CrObhCSbr9ERPcZ6PUcT 0lF0Oukih3H/9iX3Qt3h/iLuTOkw/TQPcsw+f5qfX/UetXEV42MG9xaB921OR770yq 476dl+YbpwQOMrgDhR5g3uReIo1jn+IvkCBHnZP0Z+zpnqvLEJiomT3rTpc+fZy5jM SlCp+SoLxUY9A== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 65B725C14C2; Wed, 30 Nov 2022 10:13:27 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , "Paul E . McKenney" Subject: [PATCH rcu 09/16] rcu/rcuscale: Use call_rcu_hurry() for async reader test Date: Wed, 30 Nov 2022 10:13:18 -0800 Message-Id: <20221130181325.1012760-9-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> References: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750945889334603361?= X-GMAIL-MSGID: =?utf-8?q?1750945889334603361?= From: "Joel Fernandes (Google)" rcuscale uses call_rcu() to queue async readers. With recent changes to save power, the test will have fewer async readers in flight. Use the call_rcu_hurry() API instead to revert to the old behavior. [ paulmck: Apply s/call_rcu_flush/call_rcu_hurry/ feedback from Tejun Heo. ] Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/rcuscale.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 3baded807a616..91fb5905a008f 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -176,7 +176,7 @@ static struct rcu_scale_ops rcu_ops = { .get_gp_seq = rcu_get_gp_seq, .gp_diff = rcu_seq_diff, .exp_completed = rcu_exp_batches_completed, - .async = call_rcu, + .async = call_rcu_hurry, .gp_barrier = rcu_barrier, .sync = synchronize_rcu, .exp_sync = synchronize_rcu_expedited, From patchwork Wed Nov 30 18:13:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 27943 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1078401wrr; Wed, 30 Nov 2022 10:16:03 -0800 (PST) X-Google-Smtp-Source: AA0mqf5YF33d50B5veHohAxVNA3qtm4L73Wtk5Cmu27q6jRc62cDhcLGCCgsPlXKF+eZTAvuGr8J X-Received: by 2002:a17:906:a1a:b0:79e:9aea:7b60 with SMTP id w26-20020a1709060a1a00b0079e9aea7b60mr54059341ejf.444.1669832162964; Wed, 30 Nov 2022 10:16:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669832162; cv=none; d=google.com; s=arc-20160816; b=k/hjMb2e09R2NZEPpwfx3Sur/E/xXVdokcaehs8FbIefTgiW+pQ2zXFyJMm/aQn+YP rItMwB/UuU+aH3EYzDE+e/taELH+ZL/1qaxMSKQUWHH9DAN/J4GLp7UFkb0396FoTy+c lkqsBZH7diMlQa/6HkoH8s4bA4keEqPcXqcjHT6wUnMm5jojFKD9LLSQrBu6EHNk5kAj g6F6K4EEL5iaVl5s1Zepr5hwC5BdiI/ThjgA923z/OpFLKSgIPTw8iYMrEucDLI0s9iN t4AoNLP9R9DUI/Y/CwNQ8zd2489e5YLRwrBjcambPT7hTLA52WSBhFJbZqG47yvADvIj Vb7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jdYWWKEzPd8HqynMUrAHwuF5ukCqRckUrNNkQCtQfq4=; b=GKVyQaJaIDnpvB280s/7IpoQ+pNXUxWkh9fv9xBNvk4xCfEBFYXru9XM9nWd/gOS3H m6XTKDeiETmFCQfDykqE4V8xovA9mdW+zTjebXohqKs68NNP/JH3vODucJelTKg/JcN0 Gqh4FqpYwk1AYTZurFdCX867pMmtp26KAszkd0nRND1fsfhN/yLA5x68tCeZEr0l00do ETu5maGz8mGgM6tQsffUBgYHO9GMFwMS2i1++Bq+016YpbU1Vi/PyGVy1MUOWEKAkcOt U9ZTy0CPjB5IWC1uFrY8WbUun8iBXYIfGMILbZNhg4Y3MYw3ESTT4v0qFATDAikl3tkH RomQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ujwYutX6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gt7-20020a1709072d8700b007309e8d567fsi2203431ejc.145.2022.11.30.10.15.38; Wed, 30 Nov 2022 10:16:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ujwYutX6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229866AbiK3SOa (ORCPT + 99 others); Wed, 30 Nov 2022 13:14:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37576 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229899AbiK3SNc (ORCPT ); Wed, 30 Nov 2022 13:13:32 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6763862DA; Wed, 30 Nov 2022 10:13:31 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 6195CB81C9C; Wed, 30 Nov 2022 18:13:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 10CB6C43154; Wed, 30 Nov 2022 18:13:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669832008; bh=bN1R1Byx9JRzp9KLFli6bl4Fbxjs+KDfwn4AF0kT8Bw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ujwYutX6/+BLv6Om3JgkjH/l6Pfi1kGfAwpi0Jrm78iqr1Ko5dBems6PV/sGMeL20 OWevhyCULn9ND6xnLOZim1GIbUQ35Col1jDlXkYyDtD42jK/dmDwznsmBIQcB4ZwJG TpVJsiu7Xsl7qArx58yTBUKcIiM9XNkGwT+9xcJA8SwZR6isg028Vimh8eFj0HhQzL ow4K4Co1ewczqUMXI2j4Gcwwz8cIHZ6XirPSChEct56uEJCyYzCsQ7kU5p4jec7NIq jvMm41apnBYIylB1VNTrBS0APxfrOhCccrIiAE36kBJIvgjnIHSqtz6VXzSXDWJVa7 TJMMFS4y19Scg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 67C6E5C16C5; Wed, 30 Nov 2022 10:13:27 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , "Paul E . McKenney" Subject: [PATCH rcu 10/16] rcu/rcutorture: Use call_rcu_hurry() where needed Date: Wed, 30 Nov 2022 10:13:19 -0800 Message-Id: <20221130181325.1012760-10-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> References: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750945930479241204?= X-GMAIL-MSGID: =?utf-8?q?1750945930479241204?= From: "Joel Fernandes (Google)" call_rcu() changes to save power will change the behavior of rcutorture tests. Use the call_rcu_hurry() API instead which reverts to the old behavior. [ paulmck: Apply s/call_rcu_flush/call_rcu_hurry/ feedback from Tejun Heo. ] Reported-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/rcutorture.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 503c2aa845a4a..2226f86f54f78 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -510,7 +510,7 @@ static unsigned long rcu_no_completed(void) static void rcu_torture_deferred_free(struct rcu_torture *p) { - call_rcu(&p->rtort_rcu, rcu_torture_cb); + call_rcu_hurry(&p->rtort_rcu, rcu_torture_cb); } static void rcu_sync_torture_init(void) @@ -551,7 +551,7 @@ static struct rcu_torture_ops rcu_ops = { .start_gp_poll_exp_full = start_poll_synchronize_rcu_expedited_full, .poll_gp_state_exp = poll_state_synchronize_rcu, .cond_sync_exp = cond_synchronize_rcu_expedited, - .call = call_rcu, + .call = call_rcu_hurry, .cb_barrier = rcu_barrier, .fqs = rcu_force_quiescent_state, .stats = NULL, @@ -848,7 +848,7 @@ static void rcu_tasks_torture_deferred_free(struct rcu_torture *p) static void synchronize_rcu_mult_test(void) { - synchronize_rcu_mult(call_rcu_tasks, call_rcu); + synchronize_rcu_mult(call_rcu_tasks, call_rcu_hurry); } static struct rcu_torture_ops tasks_ops = { @@ -3388,13 +3388,13 @@ static void rcu_test_debug_objects(void) /* Try to queue the rh2 pair of callbacks for the same grace period. */ preempt_disable(); /* Prevent preemption from interrupting test. */ rcu_read_lock(); /* Make it impossible to finish a grace period. */ - call_rcu(&rh1, rcu_torture_leak_cb); /* Start grace period. */ + call_rcu_hurry(&rh1, rcu_torture_leak_cb); /* Start grace period. */ local_irq_disable(); /* Make it harder to start a new grace period. */ - call_rcu(&rh2, rcu_torture_leak_cb); - call_rcu(&rh2, rcu_torture_err_cb); /* Duplicate callback. */ + call_rcu_hurry(&rh2, rcu_torture_leak_cb); + call_rcu_hurry(&rh2, rcu_torture_err_cb); /* Duplicate callback. */ if (rhp) { - call_rcu(rhp, rcu_torture_leak_cb); - call_rcu(rhp, rcu_torture_err_cb); /* Another duplicate callback. */ + call_rcu_hurry(rhp, rcu_torture_leak_cb); + call_rcu_hurry(rhp, rcu_torture_err_cb); /* Another duplicate callback. */ } local_irq_enable(); rcu_read_unlock(); From patchwork Wed Nov 30 18:13:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 27946 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1078870wrr; Wed, 30 Nov 2022 10:17:05 -0800 (PST) X-Google-Smtp-Source: AA0mqf6Hp5url7RbVRJYvzSLSA5i+hwbswrV53TDtPeLPAYq0ZoWABbJFOMCTSV8RI/Gf+xtyRSw X-Received: by 2002:aa7:d5d4:0:b0:46a:96b3:57bc with SMTP id d20-20020aa7d5d4000000b0046a96b357bcmr27614795eds.231.1669832225448; Wed, 30 Nov 2022 10:17:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669832225; cv=none; d=google.com; s=arc-20160816; b=GHMrSIkR+P8nZSgDa5B3nwwAgw4798pqXBhdSiG5UiQqZAuU7ehVfwfXH+MFnZte7I 5RPv5NSdIiKkLV2mUXZE6SU/GTjHYjbwSbMpuyNv2hra4gHikC9KmxoNdGg21AJRQT5T E/mo7EU2ZhCCvyPqpwSpc058LaR2aBG9mZWld3MENNo4YVhky3IPdbjNEsLdIQwbqh2Z g2Gqdsg+UA2l8dhXnGXfobQlPgrwm0VZaFOHMgQJKojUMLgBSEW59Nc6KbqnADfLMiBJ Znjy3ar6tppM+7N3+dVdj6OfGGey68M3aw4h6r5DGa16C2yhBy8P0FlPJ1LEH9osD+0l JKUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+8b5BxL1I/bbJYi6TQ9l3aZ9C+cMD/Q2idftaOg2Ai8=; b=p19zCMWgqET4IV1AdcEnrCLUHTdytF7onDuGIRQ6qpN2MPmMcwANzFLLf38CACHlrW ld+y1kDb7guUp9JZuFC2wpx+To3yAroxy+K3c8QjG0mXhf8lA35V/rFDe8m7RyICt2CX 7M6yFQwCKwc7lDvytTIdgN+NuuVajXcPDxJQdemHflFsluPvsF2wZBhEgmV2SjauY9XD adBKuq//t/bO7wMXL9RD6n15lIvRQoSfMxMCWiJjOcY5BCNW8gKzlCyq5/QLEa6oOHEA 7kKbvn1FqO2/25wSzT8SFNtJ2MdoYkPFo36AGWYaQQkXXVLnKSfVpSmA7fbgsxSVMGXL hToA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="g/yaPLuZ"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cy22-20020a0564021c9600b0045d9ceae669si1876082edb.404.2022.11.30.10.16.42; Wed, 30 Nov 2022 10:17:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="g/yaPLuZ"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230303AbiK3SOr (ORCPT + 99 others); Wed, 30 Nov 2022 13:14:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39154 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230003AbiK3SNd (ORCPT ); Wed, 30 Nov 2022 13:13:33 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D8772B609; Wed, 30 Nov 2022 10:13:32 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id B3B28CE1ACE; Wed, 30 Nov 2022 18:13:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0DB98C4314F; Wed, 30 Nov 2022 18:13:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669832008; bh=JWSyRakhy+dkVAOMqz8EUh288eeQKUxl8b8ecP4VVOs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=g/yaPLuZPzWktPAHagD6UFkCYaDu8tya1qQO49HjS1nedSCUGvwEWFpmtPHku1wc2 WKZPghDQCNGvnOhydYxN1R0NSOOckw+eT5aU1qcoSTT05VhnZRw3wu/E90O08NPgpU NX9XrGUbGKD4FzAFyCc7HuXqlG/Mi0SITZt9VByrOA+0i6cqH8+yLOjto2RyAOgS1s eGSk6cgv9sxqd5ksp9JSLSqbauqpUvfyqNwsOqo0Xf1gJ+kOiWGIZZacgEZB1NekX6 stuVJWxwItCP9uutcpBfLHSIkmR6iBv9E4bASwgu7M8bj+7KpkUbx6s60il2mpyLxZ S1HtBhZ/Neqpw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 69A615C1722; Wed, 30 Nov 2022 10:13:27 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Uladzislau Rezki , Joel Fernandes , "James E.J. Bottomley" , linux-scsi@vger.kernel.org, Bart Van Assche , "Martin K . Petersen" , "Paul E . McKenney" Subject: [PATCH rcu 11/16] scsi/scsi_error: Use call_rcu_hurry() instead of call_rcu() Date: Wed, 30 Nov 2022 10:13:20 -0800 Message-Id: <20221130181325.1012760-11-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> References: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750945995724166511?= X-GMAIL-MSGID: =?utf-8?q?1750945995724166511?= From: Uladzislau Rezki Earlier commits in this series allow battery-powered systems to build their kernels with the default-disabled CONFIG_RCU_LAZY=y Kconfig option. This Kconfig option causes call_rcu() to delay its callbacks in order to batch them. This means that a given RCU grace period covers more callbacks, thus reducing the number of grace periods, in turn reducing the amount of energy consumed, which increases battery lifetime which can be a very good thing. This is not a subtle effect: In some important use cases, the battery lifetime is increased by more than 10%. This CONFIG_RCU_LAZY=y option is available only for CPUs that offload callbacks, for example, CPUs mentioned in the rcu_nocbs kernel boot parameter passed to kernels built with CONFIG_RCU_NOCB_CPU=y. Delaying callbacks is normally not a problem because most callbacks do nothing but free memory. If the system is short on memory, a shrinker will kick all currently queued lazy callbacks out of their laziness, thus freeing their memory in short order. Similarly, the rcu_barrier() function, which blocks until all currently queued callbacks are invoked, will also kick lazy callbacks, thus enabling rcu_barrier() to complete in a timely manner. However, there are some cases where laziness is not a good option. For example, synchronize_rcu() invokes call_rcu(), and blocks until the newly queued callback is invoked. It would not be a good for synchronize_rcu() to block for ten seconds, even on an idle system. Therefore, synchronize_rcu() invokes call_rcu_hurry() instead of call_rcu(). The arrival of a non-lazy call_rcu_hurry() callback on a given CPU kicks any lazy callbacks that might be already queued on that CPU. After all, if there is going to be a grace period, all callbacks might as well get full benefit from it. Yes, this could be done the other way around by creating a call_rcu_lazy(), but earlier experience with this approach and feedback at the 2022 Linux Plumbers Conference shifted the approach to call_rcu() being lazy with call_rcu_hurry() for the few places where laziness is inappropriate. And another call_rcu() instance that cannot be lazy is the one in the scsi_eh_scmd_add() function. Leaving this instance lazy results in unacceptably slow boot times. Therefore, make scsi_eh_scmd_add() use call_rcu_hurry() in order to revert to the old behavior. [ paulmck: Apply s/call_rcu_flush/call_rcu_hurry/ feedback from Tejun Heo. ] Tested-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki Signed-off-by: Joel Fernandes (Google) Cc: "James E.J. Bottomley" Cc: Reviewed-by: Bart Van Assche Acked-by: Martin K. Petersen Signed-off-by: Paul E. McKenney --- drivers/scsi/scsi_error.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c index 6995c89792300..ac5ff0783b4f0 100644 --- a/drivers/scsi/scsi_error.c +++ b/drivers/scsi/scsi_error.c @@ -312,7 +312,7 @@ void scsi_eh_scmd_add(struct scsi_cmnd *scmd) * Ensure that all tasks observe the host state change before the * host_failed change. */ - call_rcu(&scmd->rcu, scsi_eh_inc_host_failed); + call_rcu_hurry(&scmd->rcu, scsi_eh_inc_host_failed); } /** From patchwork Wed Nov 30 18:13:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 27947 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1078883wrr; Wed, 30 Nov 2022 10:17:07 -0800 (PST) X-Google-Smtp-Source: AA0mqf50eL4CNVzG91mHqerqRLr3fCms+XgvyEYtQjAFO9td1lO649ru1NKT/eYcW7RiyV3tqh9u X-Received: by 2002:a17:906:86c9:b0:78d:9324:6f18 with SMTP id j9-20020a17090686c900b0078d93246f18mr54214719ejy.664.1669832227160; Wed, 30 Nov 2022 10:17:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669832227; cv=none; d=google.com; s=arc-20160816; b=xYe3VETJLW/MHJyoz5p+5CNqNLEBIf5idNPyYYBHdykeSURZi79ibvMbBCADqmWr// cJ/T5Sw0HplzoGrFiNWbmMLAHVJMo3KoI3Mt+kQFFGgEhsas6IUdk7ORYz8Pqer9lquq N9rBokhizYLIIgh/26U2WQzHES9Kj/k6ELdqw8vZsaDFMZHEHSBXnVw2izb9apdPyFT2 KHHQNeRgEev5hQrSJpXWkWxiJFfoZvFLNcBdMuvfaO0TGYPcZvcQmngdgPkkeupXkAIz 8Pgjcg6nZLxGExy3g4o+nrIuXHIS3vUdfLplqL8tCETqhsMpJarCABF2+RzIUGHk/1N4 casA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=opcgk9bN3S6Ta99hn9GnQg/Lq50u2zXrg/hR2h12HyA=; b=xoq0cE/rGWIwr6OKAcuhb38yM7gvuBzhWm6CXeRyJcne4NmWV5+AqoD0TCrc+RCDTV PS3+zFdKXzPNmcEZ544O+glm/Av2HkLwJ/jOTojxdnaTrhPTU5Bf+d0g4IhsUbxSrt+9 ZpnWRZXzCBFxnH65Rhy+bx4H35bts+zrFRS+eFt/aZidTODMfyVwEkXoJZera1vq0B5d 9z1vletOB+/XgTl+mKixmSic1MMlxHx1LScjDhe7EBMLe1iZHqYx7pQhWMenpo9IBzuN L7K6Wsq51iz4OnM5Kl1DsAYvyRJo3W2eUXYQuQce3iDAY6SokVSZpMQMceH0CZO3BXF+ dsvA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=hkJumWSp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id he18-20020a1709073d9200b0078dc5c888f1si2127672ejc.135.2022.11.30.10.16.26; Wed, 30 Nov 2022 10:17:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=hkJumWSp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230244AbiK3SOQ (ORCPT + 99 others); Wed, 30 Nov 2022 13:14:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38440 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229850AbiK3SNc (ORCPT ); Wed, 30 Nov 2022 13:13:32 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4CE32862D5; Wed, 30 Nov 2022 10:13:31 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B6B1161D64; Wed, 30 Nov 2022 18:13:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 16823C43155; Wed, 30 Nov 2022 18:13:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669832008; bh=uYoVp3wfqf3fICtozjC2ekQQHH5L+XdF7tRF4atTO+Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hkJumWSp1ikZCpXNqPUSCDnpFhz551D4U9Kk4eMWYUwEh5UqIy30sMH2g8L1UtMiy 7FphIoNYbtfdlUNTFX/BCIMtzBPo4pMetUOH2bKJ9hmmYGktR/W7is9KKoKtr/HkH2 mzVf0qcewBBpPecHuJ9AhrGl6qsru3qJlSA3NL4ZOp/izhZi3AgXH5K1Wx+oKjPNLR 92+bWi0y2zwX2rkYwjjEf0tEKKK2QtPxNd7VwnNiFKi+bvrH2EjtnXgHuuXfoHnSHw gjC8esHljhMc+IAiMsNlhd5C9exE4K3RwPYYsJA1Yp4O7P8OLU+tVMNi7zluyW8m5f tuenjVMK8X+Kw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 6BEBA5C1727; Wed, 30 Nov 2022 10:13:27 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , "Paul E . McKenney" , Dennis Zhou , Tejun Heo , Christoph Lameter , linux-mm@kvack.org Subject: [PATCH rcu 12/16] percpu-refcount: Use call_rcu_hurry() for atomic switch Date: Wed, 30 Nov 2022 10:13:21 -0800 Message-Id: <20221130181325.1012760-12-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> References: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750945997531312367?= X-GMAIL-MSGID: =?utf-8?q?1750945997531312367?= From: "Joel Fernandes (Google)" Earlier commits in this series allow battery-powered systems to build their kernels with the default-disabled CONFIG_RCU_LAZY=y Kconfig option. This Kconfig option causes call_rcu() to delay its callbacks in order to batch callbacks. This means that a given RCU grace period covers more callbacks, thus reducing the number of grace periods, in turn reducing the amount of energy consumed, which increases battery lifetime which can be a very good thing. This is not a subtle effect: In some important use cases, the battery lifetime is increased by more than 10%. This CONFIG_RCU_LAZY=y option is available only for CPUs that offload callbacks, for example, CPUs mentioned in the rcu_nocbs kernel boot parameter passed to kernels built with CONFIG_RCU_NOCB_CPU=y. Delaying callbacks is normally not a problem because most callbacks do nothing but free memory. If the system is short on memory, a shrinker will kick all currently queued lazy callbacks out of their laziness, thus freeing their memory in short order. Similarly, the rcu_barrier() function, which blocks until all currently queued callbacks are invoked, will also kick lazy callbacks, thus enabling rcu_barrier() to complete in a timely manner. However, there are some cases where laziness is not a good option. For example, synchronize_rcu() invokes call_rcu(), and blocks until the newly queued callback is invoked. It would not be a good for synchronize_rcu() to block for ten seconds, even on an idle system. Therefore, synchronize_rcu() invokes call_rcu_hurry() instead of call_rcu(). The arrival of a non-lazy call_rcu_hurry() callback on a given CPU kicks any lazy callbacks that might be already queued on that CPU. After all, if there is going to be a grace period, all callbacks might as well get full benefit from it. Yes, this could be done the other way around by creating a call_rcu_lazy(), but earlier experience with this approach and feedback at the 2022 Linux Plumbers Conference shifted the approach to call_rcu() being lazy with call_rcu_hurry() for the few places where laziness is inappropriate. And another call_rcu() instance that cannot be lazy is the one on the percpu refcounter's "per-CPU to atomic switch" code path, which uses RCU when switching to atomic mode. The enqueued callback wakes up waiters waiting in the percpu_ref_switch_waitq. Allowing this callback to be lazy would result in unacceptable slowdowns for users of per-CPU refcounts, such as blk_pre_runtime_suspend(). Therefore, make __percpu_ref_switch_to_atomic() use call_rcu_hurry() in order to revert to the old behavior. [ paulmck: Apply s/call_rcu_flush/call_rcu_hurry/ feedback from Tejun Heo. ] Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: Acked-by: Tejun Heo --- lib/percpu-refcount.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/percpu-refcount.c b/lib/percpu-refcount.c index e5c5315da2741..668f6aa6a75de 100644 --- a/lib/percpu-refcount.c +++ b/lib/percpu-refcount.c @@ -230,7 +230,8 @@ static void __percpu_ref_switch_to_atomic(struct percpu_ref *ref, percpu_ref_noop_confirm_switch; percpu_ref_get(ref); /* put after confirmation */ - call_rcu(&ref->data->rcu, percpu_ref_switch_to_atomic_rcu); + call_rcu_hurry(&ref->data->rcu, + percpu_ref_switch_to_atomic_rcu); } static void __percpu_ref_switch_to_percpu(struct percpu_ref *ref) From patchwork Wed Nov 30 18:13:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 27949 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1079073wrr; Wed, 30 Nov 2022 10:17:36 -0800 (PST) X-Google-Smtp-Source: AA0mqf5UyWr6GDAi5f7XjFkKqUmvU4QAW7PEwnoCTTJMMdDuXGgiKJv3EKwEskEMstY9mJgNDzeN X-Received: by 2002:a50:ff0d:0:b0:461:c6e8:452e with SMTP id a13-20020a50ff0d000000b00461c6e8452emr46572014edu.298.1669832256148; Wed, 30 Nov 2022 10:17:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669832256; cv=none; d=google.com; s=arc-20160816; b=Qt0wQpqsOWjVk6gdCK+PuE22skLArgGfa9N1sOUyxJFrDq+UzmL6WArnVwv4OkIoFJ 9RLH0fyD8kAlTjLB0BKVN4TCHjGYSGu/4n4gItupXijT6MNJPzQ9W0uqsJAGpUqar1C0 afp7OGrHCFL/MIZtbTtrGh1VK+xG8UUFUczHsIAfkQ/stgQC/sdDshYRJ7D/o+rJPrAY KhzBQDKYOZ8RG8pipvHBGx+KlStyJTtOdc8u6X4Tm4xOFjeEQVxuYe+f50vRNZg3OTXn tR8QQq5hXfcA6cGVSWGdwvjzOWeXNcY1GwYrhCa3PPUxQEKohNx7ULvKsbv/PcRjDsqw QMZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XBexMMUbu3Nz6j4FkYi/r25bCvhcwYCW76m8WbqGAmI=; b=gQDSK3TstbactiE4Aj+1YOcKICdIqrOwCMsBQaWxHVOtW0i7TBQUW4qRpwGEKlqlqk xTHHQKieBxgJW1VSvQVYvarauS64oIYg6Xm7evLYing/NUg3ZlP0Pqmu4BzAAsGVWzbw X6iLBO/DfPb117THcrUefbm8S/ma8AJxKrtTg3FZ4H2fT6Ux3fXZPFpI2w1Vhd2WXl0D /kBInd0N/5b82YLNHqB4QTGMIiV4HFe5Iha26D+RBrfCjj/+ald1Qur7REe1SU9VhRgZ f3V34VTNOGXLnedKYz9PQiuBy5jtE9RyVBzJvt4PF75Q2dwlalb9mKYDoUZ8QKbPrLJw Zl5w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=NJNSrIdT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ds15-20020a170907724f00b007bfdadb58dfsi2017535ejc.208.2022.11.30.10.17.12; Wed, 30 Nov 2022 10:17:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=NJNSrIdT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230280AbiK3SOh (ORCPT + 99 others); Wed, 30 Nov 2022 13:14:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37422 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229876AbiK3SNc (ORCPT ); Wed, 30 Nov 2022 13:13:32 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E30A862D8; Wed, 30 Nov 2022 10:13:31 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C08B261D66; Wed, 30 Nov 2022 18:13:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 16D6CC43156; Wed, 30 Nov 2022 18:13:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669832008; bh=bV4YKsjlHn2g0UUJMN8cc0Rz3CedXb1hl5bvrhhBb38=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NJNSrIdTEWWdXI6msK7DGfG0coc2GIOoXRiT7vsENQXHEZI2bVDZeOcccpqtp5dTz 0vv/2fHYDXpqwFQ30sKYzj5tMnnXjk1pvWM3I/TDg2T4qjejKhXOluUuQvCoI5xT6Z Kic99ZMzMu3X9u1aef5hYkvYTVGN1UuhbM2GCxEsLCmOnOZcyze3d/6AqfGgD+bHe7 qcOybBwGDxUtIFY+pDqTTCDuL81vldfpYkdVpMHxwJM53hxbgrCLcZ1sT6t0KUQH3P JiVr5JoLHuADj83l9WubCtUpgIBtCVYu4fxA2yTjXIZKGa8WuXzBaTYp5vruZlY1Ee NXRsYsEbiceGw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 6DBFE5C1799; Wed, 30 Nov 2022 10:13:27 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Uladzislau Rezki , Joel Fernandes , Tejun Heo , Lai Jiangshan , "Paul E . McKenney" Subject: [PATCH rcu 13/16] workqueue: Make queue_rcu_work() use call_rcu_hurry() Date: Wed, 30 Nov 2022 10:13:22 -0800 Message-Id: <20221130181325.1012760-13-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> References: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750946028109317499?= X-GMAIL-MSGID: =?utf-8?q?1750946028109317499?= From: Uladzislau Rezki Earlier commits in this series allow battery-powered systems to build their kernels with the default-disabled CONFIG_RCU_LAZY=y Kconfig option. This Kconfig option causes call_rcu() to delay its callbacks in order to batch them. This means that a given RCU grace period covers more callbacks, thus reducing the number of grace periods, in turn reducing the amount of energy consumed, which increases battery lifetime which can be a very good thing. This is not a subtle effect: In some important use cases, the battery lifetime is increased by more than 10%. This CONFIG_RCU_LAZY=y option is available only for CPUs that offload callbacks, for example, CPUs mentioned in the rcu_nocbs kernel boot parameter passed to kernels built with CONFIG_RCU_NOCB_CPU=y. Delaying callbacks is normally not a problem because most callbacks do nothing but free memory. If the system is short on memory, a shrinker will kick all currently queued lazy callbacks out of their laziness, thus freeing their memory in short order. Similarly, the rcu_barrier() function, which blocks until all currently queued callbacks are invoked, will also kick lazy callbacks, thus enabling rcu_barrier() to complete in a timely manner. However, there are some cases where laziness is not a good option. For example, synchronize_rcu() invokes call_rcu(), and blocks until the newly queued callback is invoked. It would not be a good for synchronize_rcu() to block for ten seconds, even on an idle system. Therefore, synchronize_rcu() invokes call_rcu_hurry() instead of call_rcu(). The arrival of a non-lazy call_rcu_hurry() callback on a given CPU kicks any lazy callbacks that might be already queued on that CPU. After all, if there is going to be a grace period, all callbacks might as well get full benefit from it. Yes, this could be done the other way around by creating a call_rcu_lazy(), but earlier experience with this approach and feedback at the 2022 Linux Plumbers Conference shifted the approach to call_rcu() being lazy with call_rcu_hurry() for the few places where laziness is inappropriate. And another call_rcu() instance that cannot be lazy is the one in queue_rcu_work(), given that callers to queue_rcu_work() are not necessarily OK with long delays. Therefore, make queue_rcu_work() use call_rcu_hurry() in order to revert to the old behavior. [ paulmck: Apply s/call_rcu_flush/call_rcu_hurry/ feedback from Tejun Heo. ] Signed-off-by: Uladzislau Rezki Signed-off-by: Joel Fernandes (Google) Cc: Tejun Heo Cc: Lai Jiangshan Signed-off-by: Paul E. McKenney Acked-by: Tejun Heo --- kernel/workqueue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 7cd5f5e7e0a1b..07895deca2711 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -1771,7 +1771,7 @@ bool queue_rcu_work(struct workqueue_struct *wq, struct rcu_work *rwork) if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))) { rwork->wq = wq; - call_rcu(&rwork->rcu, rcu_work_rcufn); + call_rcu_hurry(&rwork->rcu, rcu_work_rcufn); return true; } From patchwork Wed Nov 30 18:13:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 27940 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1078236wrr; Wed, 30 Nov 2022 10:15:46 -0800 (PST) X-Google-Smtp-Source: AA0mqf7duUNsIgPwKqDQ4VL3vMlJnLYb+4NH1TKGnSTkF+7uS1zciDXf/fwkEwN1yjdnAVZrTh7d X-Received: by 2002:a17:907:11c5:b0:7b1:316c:38f7 with SMTP id va5-20020a17090711c500b007b1316c38f7mr54386292ejb.387.1669832145983; Wed, 30 Nov 2022 10:15:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669832145; cv=none; d=google.com; s=arc-20160816; b=bPFHymwmkqfaXCYB0HQMVUO3E9/7hFLdgWT7jPJ9NERA9G1Oo/pE6ZNUiG1H+NX5Gr 8CNyJRY6DY7jOd4p2L8KT+D54fzWKlWZpmWDeteKn2OH3fLk+BYsXeKIqWm1ODZkx5QV ydmOwIYd4kJ1hiLwpdZNUR7EIhlG9c0Wrr8A7NIraP+ZFep79hnljXtLtcO3XSHqlWIc iqRzgzSInVV/LvkDd0XMMgq3xRPxnaA3DCF8GGlfNW96oboKIa5CUISIyuVY0N71E/dk x+y4KsR5Mq32KbmQG2gMvq/cRf36v8AOeIyfbdq9z2z/HWzT9DnDMWnQoLG7Sa8IsRUn dL8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1Btgh5/dswv9zbVo1Quy9hLPmdL09vVBXC0pZIn22Ho=; b=f+yOi6/9s09+mJLubHGPgPSPR62OQe/jP6ZHADQtUkbFpRNKeqHjMTSCTcpzUDgZWN h9v2Y4do0YmDZXNcu20U0rK1lRNgtkAu0lwBdrMqS5eBn+ZxvLrW7pI7rrP23vttLKZq Pai9MutplVctVtNF7BOIB8aMYYRMzdbx3u4e8mzxXpdiBJn9jFfZC34WW0S2EXvNHtfE hbNuy4x7c9p/H68ntUFYNzNAOsxRQrVeiZLicZX7r8rfgyhVcsu2DfvMZLXyd8Rsk3TL Sn7BtJ0db5NL8hkP6u3A6qXSJmy35oEaO4ZTUXpO9rDNnJRLrOYf+H50QccS2GGNIWXf E2ow== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=pN9sTHal; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t18-20020a1709063e5200b007ae1e519db0si1466802eji.220.2022.11.30.10.15.21; Wed, 30 Nov 2022 10:15:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=pN9sTHal; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230251AbiK3SOU (ORCPT + 99 others); Wed, 30 Nov 2022 13:14:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38140 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229852AbiK3SNc (ORCPT ); Wed, 30 Nov 2022 13:13:32 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E109862D7; Wed, 30 Nov 2022 10:13:31 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C618961D74; Wed, 30 Nov 2022 18:13:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1DD0AC43152; Wed, 30 Nov 2022 18:13:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669832008; bh=O3sihN19Kz2sN5DYDWZyYgF1v27O1pMTYDT2Vmf9Dd4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pN9sTHaln9m6EGe3r9Eb2mbBcomn8O7ZzzGuAofcxjdtAaYBBhuhEjCdEYKuuSr06 v0BxvoaljMSpxX4FpfkPcIJSQGSpia3bFMMSWxo9Hw1m5HwSGwMZFeJpkfP4YAf9ZR xYFs1/y0W4mmseFXE/jsG/vHS0wbW4TNkTZnkww03YP1y/+SXiuDlCBqDWwywzHvf6 UVnawFrUxFkSAl1hhOSYWBxQF7dkCz4UCuF5KH2hARoBTkQY1Nu5lTOiIJ4fqmvdez t973eHlre30kymo32erxf/RHRtlVEcgD/glMh8n2Zh+g+Qak1llmk5ftZmJKfsUce4 JU7NjJFwIrRmw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 7006E5C196D; Wed, 30 Nov 2022 10:13:27 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , David Howells , Marc Dionne , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , linux-afs@lists.infradead.org, netdev@vger.kernel.org, "Paul E . McKenney" Subject: [PATCH rcu 14/16] rxrpc: Use call_rcu_hurry() instead of call_rcu() Date: Wed, 30 Nov 2022 10:13:23 -0800 Message-Id: <20221130181325.1012760-14-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> References: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750945912765080305?= X-GMAIL-MSGID: =?utf-8?q?1750945912765080305?= From: "Joel Fernandes (Google)" Earlier commits in this series allow battery-powered systems to build their kernels with the default-disabled CONFIG_RCU_LAZY=y Kconfig option. This Kconfig option causes call_rcu() to delay its callbacks in order to batch them. This means that a given RCU grace period covers more callbacks, thus reducing the number of grace periods, in turn reducing the amount of energy consumed, which increases battery lifetime which can be a very good thing. This is not a subtle effect: In some important use cases, the battery lifetime is increased by more than 10%. This CONFIG_RCU_LAZY=y option is available only for CPUs that offload callbacks, for example, CPUs mentioned in the rcu_nocbs kernel boot parameter passed to kernels built with CONFIG_RCU_NOCB_CPU=y. Delaying callbacks is normally not a problem because most callbacks do nothing but free memory. If the system is short on memory, a shrinker will kick all currently queued lazy callbacks out of their laziness, thus freeing their memory in short order. Similarly, the rcu_barrier() function, which blocks until all currently queued callbacks are invoked, will also kick lazy callbacks, thus enabling rcu_barrier() to complete in a timely manner. However, there are some cases where laziness is not a good option. For example, synchronize_rcu() invokes call_rcu(), and blocks until the newly queued callback is invoked. It would not be a good for synchronize_rcu() to block for ten seconds, even on an idle system. Therefore, synchronize_rcu() invokes call_rcu_hurry() instead of call_rcu(). The arrival of a non-lazy call_rcu_hurry() callback on a given CPU kicks any lazy callbacks that might be already queued on that CPU. After all, if there is going to be a grace period, all callbacks might as well get full benefit from it. Yes, this could be done the other way around by creating a call_rcu_lazy(), but earlier experience with this approach and feedback at the 2022 Linux Plumbers Conference shifted the approach to call_rcu() being lazy with call_rcu_hurry() for the few places where laziness is inappropriate. And another call_rcu() instance that cannot be lazy is the one in rxrpc_kill_connection(), which sometimes does a wakeup that should not be unduly delayed. Therefore, make rxrpc_kill_connection() use call_rcu_hurry() in order to revert to the old behavior. [ paulmck: Apply s/call_rcu_flush/call_rcu_hurry/ feedback from Tejun Heo. ] Signed-off-by: Joel Fernandes (Google) Cc: David Howells Cc: Marc Dionne Cc: "David S. Miller" Cc: Eric Dumazet Cc: Jakub Kicinski Cc: Paolo Abeni Cc: Cc: Signed-off-by: Paul E. McKenney Reviewed-by: Eric Dumazet --- net/rxrpc/conn_object.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/rxrpc/conn_object.c b/net/rxrpc/conn_object.c index 22089e37e97f0..9c5fae9ca106c 100644 --- a/net/rxrpc/conn_object.c +++ b/net/rxrpc/conn_object.c @@ -253,7 +253,7 @@ void rxrpc_kill_connection(struct rxrpc_connection *conn) * must carry a ref on the connection to prevent us getting here whilst * it is queued or running. */ - call_rcu(&conn->rcu, rxrpc_destroy_connection); + call_rcu_hurry(&conn->rcu, rxrpc_destroy_connection); } /* From patchwork Wed Nov 30 18:13:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 27950 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1079457wrr; Wed, 30 Nov 2022 10:18:22 -0800 (PST) X-Google-Smtp-Source: AA0mqf7AafMd7k4yvG9VXcDHpCmQCpoRWCCuEmA2fu2nJ06dpsXloSWeZqAI3Oe93/MOx7trfrvO X-Received: by 2002:a05:6402:5289:b0:462:70ee:fdb8 with SMTP id en9-20020a056402528900b0046270eefdb8mr22393174edb.66.1669832302807; Wed, 30 Nov 2022 10:18:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669832302; cv=none; d=google.com; s=arc-20160816; b=qOuszZmOYv5M/Nz6sqQ+EwnIYH6naBUvdJfJlgDE/Q+0fR/yKmceDRcld3I0eriOZJ kD/BfaD7z5uuG2KeOoLegtbk6GbxGxO1I2pds1b4QdXPFV+q+q6Aai94O7hM81GpB07X o+eDVFbsqLzhWRcMxtLrHlL4z9AD+UguuIECdLNW7e5TDbM3W3CPQfJVNRnTshJajhML I5AVL7NmkyEZPT/xgHj1VA667YVgCTlhlkA2I+RbQpvAySEo1KA0h2W7tt2VRKxCABpz EMx8Jv9y+nYCOFogymCzGKUDAU4/z+xLQ1hmwIuuPZjHPHem0SxLw6dejTpKwuURPqng wdbg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1zXdvyfc1MP3KLtbXijQXOY0+CV9MXBH71IhPP6Bqec=; b=yYf9h3PdCxJkPhqAumguJbAAGWskn2Nuev47VObYQNndcTrq0QW7hYmnn6RkD1ntj4 wQCBkNbehx2a4B8bWPekC8SI13zld3MsCRqnse02DtVm1j6DpUN51MpwbI36YfLBfNFg IBR531z7T6YgMuky5ODZSisK1eIsqHidARUFQ9UXMlJQe2gzqBmPP+E9D1vo3oLnspLU OmfAL2vi30UXM7IySroA33pvKh5H8fpj+2tdV+LjgTScgvAg37pTrIaQZpI9W3mtUPkL 4CmJxkggaWjdKBZd4cvFoGe0RRrJIPI/zfrZprQFAzw10hOmD0TDiEN6eeHnWC0bDQ17 oQXA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="Qn/G0f3u"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id go36-20020a1709070da400b0078d00203ab2si2068730ejc.41.2022.11.30.10.17.56; Wed, 30 Nov 2022 10:18:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="Qn/G0f3u"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230285AbiK3SOk (ORCPT + 99 others); Wed, 30 Nov 2022 13:14:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38484 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230001AbiK3SNd (ORCPT ); Wed, 30 Nov 2022 13:13:33 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4D0E862E7; Wed, 30 Nov 2022 10:13:31 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 8630DB81CA1; Wed, 30 Nov 2022 18:13:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2187AC4315A; Wed, 30 Nov 2022 18:13:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669832008; bh=tLEwwsXJ50F5WMjv+BokwLjrPrLGPh46UmjaqeJOMNw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Qn/G0f3uTFbWSF42AqjtciE1KrJdOMt55GTNskCSfXG+ehVdvjQo7PT/4uic/Xzg1 Wi/E5KetpU6444YTs4tO67fBw4KORpTJ0BKVkR/Z3EMAh6cV20/1KWN/BxoKaOaAAN 4IPkSER2QT2w1PfvXv0JlBCGSXN2yG7XA0W899GLRdiFLJ5uYDWtNbzwrVYZ5jU9iz hauOlXc4XrtNkMQU9gYEnLcdZLg3winmFw/9vpHoo+ESsEjtdGfVEVJ2RCwpQJVQq1 NwKPlMpxbYYVPJjir5RwdfEOfd2sKQckEBkPR+Stgh/QD3zHNsxWDTgBaOGCX58MaF 0ukAfUlmoFqgg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 724285C1A02; Wed, 30 Nov 2022 10:13:27 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , David Ahern , "David S. Miller" , Eric Dumazet , Hideaki YOSHIFUJI , Jakub Kicinski , Paolo Abeni , netdev@vger.kernel.org, "Paul E . McKenney" Subject: [PATCH rcu 15/16] net: Use call_rcu_hurry() for dst_release() Date: Wed, 30 Nov 2022 10:13:24 -0800 Message-Id: <20221130181325.1012760-15-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> References: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750946076842476089?= X-GMAIL-MSGID: =?utf-8?q?1750946076842476089?= From: "Joel Fernandes (Google)" In a networking test on ChromeOS, kernels built with the new CONFIG_RCU_LAZY=y Kconfig option fail a networking test in the teardown phase. This failure may be reproduced as follows: ip netns del The CONFIG_RCU_LAZY=y Kconfig option was introduced by earlier commits in this series for the benefit of certain battery-powered systems. This Kconfig option causes call_rcu() to delay its callbacks in order to batch them. This means that a given RCU grace period covers more callbacks, thus reducing the number of grace periods, in turn reducing the amount of energy consumed, which increases battery lifetime which can be a very good thing. This is not a subtle effect: In some important use cases, the battery lifetime is increased by more than 10%. This CONFIG_RCU_LAZY=y option is available only for CPUs that offload callbacks, for example, CPUs mentioned in the rcu_nocbs kernel boot parameter passed to kernels built with CONFIG_RCU_NOCB_CPU=y. Delaying callbacks is normally not a problem because most callbacks do nothing but free memory. If the system is short on memory, a shrinker will kick all currently queued lazy callbacks out of their laziness, thus freeing their memory in short order. Similarly, the rcu_barrier() function, which blocks until all currently queued callbacks are invoked, will also kick lazy callbacks, thus enabling rcu_barrier() to complete in a timely manner. However, there are some cases where laziness is not a good option. For example, synchronize_rcu() invokes call_rcu(), and blocks until the newly queued callback is invoked. It would not be a good for synchronize_rcu() to block for ten seconds, even on an idle system. Therefore, synchronize_rcu() invokes call_rcu_hurry() instead of call_rcu(). The arrival of a non-lazy call_rcu_hurry() callback on a given CPU kicks any lazy callbacks that might be already queued on that CPU. After all, if there is going to be a grace period, all callbacks might as well get full benefit from it. Yes, this could be done the other way around by creating a call_rcu_lazy(), but earlier experience with this approach and feedback at the 2022 Linux Plumbers Conference shifted the approach to call_rcu() being lazy with call_rcu_hurry() for the few places where laziness is inappropriate. Returning to the test failure, use of ftrace showed that this failure cause caused by the aadded delays due to this new lazy behavior of call_rcu() in kernels built with CONFIG_RCU_LAZY=y. Therefore, make dst_release() use call_rcu_hurry() in order to revert to the old test-failure-free behavior. [ paulmck: Apply s/call_rcu_flush/call_rcu_hurry/ feedback from Tejun Heo. ] Signed-off-by: Joel Fernandes (Google) Cc: David Ahern Cc: "David S. Miller" Cc: Eric Dumazet Cc: Hideaki YOSHIFUJI Cc: Jakub Kicinski Cc: Paolo Abeni Cc: Signed-off-by: Paul E. McKenney Reviewed-by: Eric Dumazet --- net/core/dst.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/core/dst.c b/net/core/dst.c index bc9c9be4e0801..a4e738d321ba2 100644 --- a/net/core/dst.c +++ b/net/core/dst.c @@ -174,7 +174,7 @@ void dst_release(struct dst_entry *dst) net_warn_ratelimited("%s: dst:%p refcnt:%d\n", __func__, dst, newrefcnt); if (!newrefcnt) - call_rcu(&dst->rcu_head, dst_destroy_rcu); + call_rcu_hurry(&dst->rcu_head, dst_destroy_rcu); } } EXPORT_SYMBOL(dst_release); From patchwork Wed Nov 30 18:13:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 27938 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1078100wrr; Wed, 30 Nov 2022 10:15:30 -0800 (PST) X-Google-Smtp-Source: AA0mqf6mhkwUdfAL7jOPuRXnO6E2FMw7TN9+NjxOUPksIZeUrd90EMWJWV7NFp0QeZvGnmKkdUDb X-Received: by 2002:a05:6402:41:b0:467:3ea2:3ec4 with SMTP id f1-20020a056402004100b004673ea23ec4mr57704025edu.351.1669832130048; Wed, 30 Nov 2022 10:15:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669832130; cv=none; d=google.com; s=arc-20160816; b=NgRyYzyZolx6B4808051chL8Mp1nwF8rJ97IsuOKW9HlKbdjPETFtgJJET3FSfWI/m Fg0yKl47BSCi+Bx5/dDHK9VnTDWgrIrSzbJ/3SFiqxQ3PubmC1NuWHqz0A8hV1m8fprY YByhZ6mKCsfnBkNTE4pLjLVMrwcXCYW27iCry1Vo3+/4rrqFlRw1hj2KS2aC9X9lxxRB X/7GW4V2dUXvavQ00w8N518DrLeyTi7tqj8C+nx+f/Ft70eAFlbbioo9aBQOFhcOeiWB GsbOZ7y29AA+CR+oAm8efppIR7rmI+TQ2dVTLQ+YydhGU71NsqkE9R/Tt3KwcCFB0LKl WMYg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Lv/uLvZfOF0iRiluE4xXFTPcAzql9p11wASsMjjHz2Y=; b=Bz7rximTC42i7OqkbgBLtsX6N6GVWPuAqNzxpGRBY2Ke3mxIC7GTeWRwNGNz0VeIVx FdD7ybjTYWFQB/6m23iKJb9cY53AQyG7yJpHEY42plxiKFiw3SrInP/HmfWs1hGb5GfL J6TKJg1Pkw8ySgs4rVJVnYBrHvOURvRYuWzUyhOE59cUDHL0mpKux7hjFHgob59T+Cm1 mwxOOs/rbQGay1g3zUAttSaDPV9CMnML0z0NRBbZAGoO2mTVyaymumxeHZl+yws9EMz9 2t6zIayvnIyVJdmLw7r0mffJstuXqOIaMHoZh0mz2CfDkbrINCgpKCtiTP159udy8wbf R3+A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="QmUth/WN"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y2-20020a50e602000000b0046a9b3521fasi1846213edm.134.2022.11.30.10.15.04; Wed, 30 Nov 2022 10:15:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="QmUth/WN"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229897AbiK3SOK (ORCPT + 99 others); Wed, 30 Nov 2022 13:14:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230114AbiK3SNb (ORCPT ); Wed, 30 Nov 2022 13:13:31 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EFFD3862D6; Wed, 30 Nov 2022 10:13:29 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7496461D69; Wed, 30 Nov 2022 18:13:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5EE5BC4347C; Wed, 30 Nov 2022 18:13:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669832008; bh=yoNO14jnT7/3wJpWHWo0CYgw8+T0b3nI0VWmVo/0VZk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QmUth/WNvneVJ7GY/NtQg2tQXkTTyS6dQZtGDqBepyOjRyaRmzdEVbxOwHWojNgNG fy274sxQ8p2fNmFr7zOX+v8PTdexYqqR+nEhTS6TAca3mgKJxgJmgNeZ7Qgfyb13vk KmRg7eHwgtZRQ3dkAQTxR3kVNdYZZUuawesr4MoOkISj5Ge9v87CEKqjYT49l8Q27B czHeZkDRTlPmeLD5CGnM9FyqIl1YmKi8jqqfbS6/XwRSsWdypQh1osFZVzDx+3doNT ZEc5DK7JuXrrQWn2jEe1fbX8zvfQHWLv13WWUGCyBNyPpiiXarzI79sX1I3H8M8AhX ZlhvX2BidzHTA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 744535C1A77; Wed, 30 Nov 2022 10:13:27 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Eric Dumazet , Joel Fernandes , David Ahern , "David S. Miller" , Hideaki YOSHIFUJI , Jakub Kicinski , Paolo Abeni , netdev@vger.kernel.org, "Paul E . McKenney" Subject: [PATCH rcu 16/16] net: devinet: Reduce refcount before grace period Date: Wed, 30 Nov 2022 10:13:25 -0800 Message-Id: <20221130181325.1012760-16-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> References: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750945895322861106?= X-GMAIL-MSGID: =?utf-8?q?1750945895322861106?= From: Eric Dumazet Currently, the inetdev_destroy() function waits for an RCU grace period before decrementing the refcount and freeing memory. This causes a delay with a new RCU configuration that tries to save power, which results in the network interface disappearing later than expected. The resulting delay causes test failures on ChromeOS. Refactor the code such that the refcount is freed before the grace period and memory is freed after. With this a ChromeOS network test passes that does 'ip netns del' and polls for an interface disappearing, now passes. Reported-by: Joel Fernandes (Google) Signed-off-by: Eric Dumazet Signed-off-by: Joel Fernandes (Google) Cc: David Ahern Cc: "David S. Miller" Cc: Hideaki YOSHIFUJI Cc: Jakub Kicinski Cc: Paolo Abeni Cc: Signed-off-by: Paul E. McKenney --- net/ipv4/devinet.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/net/ipv4/devinet.c b/net/ipv4/devinet.c index e8b9a9202fecd..b0acf6e19aed3 100644 --- a/net/ipv4/devinet.c +++ b/net/ipv4/devinet.c @@ -234,13 +234,20 @@ static void inet_free_ifa(struct in_ifaddr *ifa) call_rcu(&ifa->rcu_head, inet_rcu_free_ifa); } +static void in_dev_free_rcu(struct rcu_head *head) +{ + struct in_device *idev = container_of(head, struct in_device, rcu_head); + + kfree(rcu_dereference_protected(idev->mc_hash, 1)); + kfree(idev); +} + void in_dev_finish_destroy(struct in_device *idev) { struct net_device *dev = idev->dev; WARN_ON(idev->ifa_list); WARN_ON(idev->mc_list); - kfree(rcu_dereference_protected(idev->mc_hash, 1)); #ifdef NET_REFCNT_DEBUG pr_debug("%s: %p=%s\n", __func__, idev, dev ? dev->name : "NIL"); #endif @@ -248,7 +255,7 @@ void in_dev_finish_destroy(struct in_device *idev) if (!idev->dead) pr_err("Freeing alive in_device %p\n", idev); else - kfree(idev); + call_rcu(&idev->rcu_head, in_dev_free_rcu); } EXPORT_SYMBOL(in_dev_finish_destroy); @@ -298,12 +305,6 @@ static struct in_device *inetdev_init(struct net_device *dev) goto out; } -static void in_dev_rcu_put(struct rcu_head *head) -{ - struct in_device *idev = container_of(head, struct in_device, rcu_head); - in_dev_put(idev); -} - static void inetdev_destroy(struct in_device *in_dev) { struct net_device *dev; @@ -328,7 +329,7 @@ static void inetdev_destroy(struct in_device *in_dev) neigh_parms_release(&arp_tbl, in_dev->arp_parms); arp_ifdown(dev); - call_rcu(&in_dev->rcu_head, in_dev_rcu_put); + in_dev_put(in_dev); } int inet_addr_onlink(struct in_device *in_dev, __be32 a, __be32 b)