From patchwork Tue Nov 22 01:04:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24078 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1929781wrr; Mon, 21 Nov 2022 17:04:58 -0800 (PST) X-Google-Smtp-Source: AA0mqf4Wngtr2JP8jRWV7Ned4lNdWsI9knyggu3AEtk3VHph94gw76X2iPPevpV+CE95CttIWLhD X-Received: by 2002:a17:90a:7d0d:b0:1fa:b51d:8d83 with SMTP id g13-20020a17090a7d0d00b001fab51d8d83mr1539524pjl.169.1669079098161; Mon, 21 Nov 2022 17:04:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079098; cv=none; d=google.com; s=arc-20160816; b=AxgWQxGQYJHI4ivgL5Wb30gzprogir5+1MxfxiWX6osiJrgXcRTAJbgtI65DqjwhZJ 62nLrAstOPPJO5wu9OvwbN4Y5G+XIVgTYaXZY2OJX1lMqjWIzckzrqv7PPYjxR32bHzy TtrJ/4g8bPFpHJJfZYEnVRt6otpsh8n75YKMKqpj/7WoNBW20RZSH1jiOB+SJ9LEKwYl Ke70mXDqGb8+gEUsSadRPFS9LfUOAC7IF5palK06hU3KA8yw/VUX4xJgrhSJ+NrNj/y0 BLhe6Dtdd5BvmhbVMhwwkCM+PZn/sDImElVy0QW+7bLgYLABnBa5DvORGOG8NUTVkZko 6pmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=uuQBoT0/9VgjI8k7NsHO7zPPIVhrYWCEsChCDjUY0bA=; b=vFeHUaw5E6sC0Gp3o98W+heqfagzzlaV+WmU3hLkV9JWbFjzgFSdct1SnBHsVQrzqi nGRT/3UAWY9tUV+UjqXiYYf27nT3P8NngR0jVClhlcewVYb3/93/eeU4t6nJwVEyRvqd QMSvTv6ejto0bMo9BB2JrUhlu1pxA0FJkSh8lIhhKyh86wJP9UBDgR4qNSpzMeIU5bfx zXQYlsRX4vo1b0MHXuQ3hCigO1sSM93GHFT+B7ipqTlGk8KoIs7TV0HjhNW+s2siifAy 4tseNG3aRMK/awv13AdFsoXoGDxIOLak0gwRzGXv4aFo8Cpk8aJeWrhba6F6xlOZjzhX yflg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=JOQXCNm7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y1-20020a62ce01000000b00542c772d478si12442849pfg.247.2022.11.21.17.04.44; Mon, 21 Nov 2022 17:04:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=JOQXCNm7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232036AbiKVBEb (ORCPT + 99 others); Mon, 21 Nov 2022 20:04:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44780 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231928AbiKVBEY (ORCPT ); Mon, 21 Nov 2022 20:04:24 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4ECBDF40; Mon, 21 Nov 2022 17:04:23 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 65A6961519; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B7E53C433B5; Tue, 22 Nov 2022 01:04:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079062; bh=KTUh/BBOVf/jDi48jtrSyvr3uGxVW8ae61JQrk9NyJc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JOQXCNm70PQyyZeRLbwg93f07Dsei0nOfLuJ9J1re8t1PshudLXi0wOHO/zVlqGIJ fVVt/hwTYo9vGZqtrEO1CZRGiTXXVBXqaf+dkjkWlt60fJwBTjP/bp9KPDvz+MsFnM KDxvtvd9htvS0DItqONR8FyIvfCdbEc8wilPNw3hQOJfqZpXyDRbPaiNOBElGMjGAt vwn2Z/RP5gnqRTfvP6NmirIWhApQ/Z+zy6mgsJkEf4LTQXgUdo9Cb8FF3IN9mGnf6e +3vOtMoi/mz0ptRX5ZwaWDg8ZigkbAZDUqaO7IH64hw5l/IakTXYlKT/d1z3RzRbI1 5oNlhcSSFvNSg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 717D15C0641; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Zhen Lei , Joel Fernandes , Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH v2 rcu 01/16] rcu: Simplify rcu_init_nohz() cpumask handling Date: Mon, 21 Nov 2022 17:04:06 -0800 Message-Id: <20221122010421.3799681-1-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156284236219131?= X-GMAIL-MSGID: =?utf-8?q?1750156284236219131?= From: Zhen Lei In kernels built with either CONFIG_RCU_NOCB_CPU_DEFAULT_ALL=y or CONFIG_NO_HZ_FULL=y, additional CPUs must be added to rcu_nocb_mask. Except that kernels booted without the rcu_nocbs= will not have allocated rcu_nocb_mask. And the current rcu_init_nohz() function uses its need_rcu_nocb_mask and offload_all local variables to track the rcu_nocb and nohz_full state. But there is a much simpler approach, namely creating a cpumask pointer to track the default and then using cpumask_available() to check the rcu_nocb_mask state. This commit takes this approach, thereby simplifying and shortening the rcu_init_nohz() function. Signed-off-by: Zhen Lei Reviewed-by: Joel Fernandes (Google) Acked-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 34 +++++++++++----------------------- 1 file changed, 11 insertions(+), 23 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 0a5f0ef414845..ce526cc2791ca 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1210,45 +1210,33 @@ EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload); void __init rcu_init_nohz(void) { int cpu; - bool need_rcu_nocb_mask = false; - bool offload_all = false; struct rcu_data *rdp; - -#if defined(CONFIG_RCU_NOCB_CPU_DEFAULT_ALL) - if (!rcu_state.nocb_is_setup) { - need_rcu_nocb_mask = true; - offload_all = true; - } -#endif /* #if defined(CONFIG_RCU_NOCB_CPU_DEFAULT_ALL) */ + const struct cpumask *cpumask = NULL; #if defined(CONFIG_NO_HZ_FULL) - if (tick_nohz_full_running && !cpumask_empty(tick_nohz_full_mask)) { - need_rcu_nocb_mask = true; - offload_all = false; /* NO_HZ_FULL has its own mask. */ - } -#endif /* #if defined(CONFIG_NO_HZ_FULL) */ + if (tick_nohz_full_running && !cpumask_empty(tick_nohz_full_mask)) + cpumask = tick_nohz_full_mask; +#endif + + if (IS_ENABLED(CONFIG_RCU_NOCB_CPU_DEFAULT_ALL) && + !rcu_state.nocb_is_setup && !cpumask) + cpumask = cpu_possible_mask; - if (need_rcu_nocb_mask) { + if (cpumask) { if (!cpumask_available(rcu_nocb_mask)) { if (!zalloc_cpumask_var(&rcu_nocb_mask, GFP_KERNEL)) { pr_info("rcu_nocb_mask allocation failed, callback offloading disabled.\n"); return; } } + + cpumask_or(rcu_nocb_mask, rcu_nocb_mask, cpumask); rcu_state.nocb_is_setup = true; } if (!rcu_state.nocb_is_setup) return; -#if defined(CONFIG_NO_HZ_FULL) - if (tick_nohz_full_running) - cpumask_or(rcu_nocb_mask, rcu_nocb_mask, tick_nohz_full_mask); -#endif /* #if defined(CONFIG_NO_HZ_FULL) */ - - if (offload_all) - cpumask_setall(rcu_nocb_mask); - if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) { pr_info("\tNote: kernel parameter 'rcu_nocbs=', 'nohz_full', or 'isolcpus=' contains nonexistent CPUs.\n"); cpumask_and(rcu_nocb_mask, cpu_possible_mask, From patchwork Tue Nov 22 01:04:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24076 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1929662wrr; Mon, 21 Nov 2022 17:04:45 -0800 (PST) X-Google-Smtp-Source: AA0mqf5S29jQRz205vVfTUBy04SMKgTjob2SbmZz6tVzMeKHzUqB083oj8LJ1uShkT5IjcRpDne5 X-Received: by 2002:a17:902:e383:b0:186:9c70:9b7d with SMTP id g3-20020a170902e38300b001869c709b7dmr1963471ple.3.1669079085240; Mon, 21 Nov 2022 17:04:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079085; cv=none; d=google.com; s=arc-20160816; b=ttPaTqoNEUY/v4feJ1rZJG5fOt4XRyvNYc26vZL5A+dn86vmks85bWBI5gBpZr59Zr DxL1Bu1Ieb0K/ft31iY9rCzw3llUnNm1t1EjqG4NOxD27w92zLn8ktPYH0PYI2/iUGBC /rxvlpxAErwQacHjt43Lu+YBtCRK1Pb9dJ1DLF32OpBikUybKHqsqG6ckleSghH3X2qq zaA90ZPfPQCwGVBg2WtAiolUo2WDS7KxXxoDEWq0n5sHv2EoAJvywYaA8JfDr9guOWrO +fVzDDMiEFZkZi7DXK/rIR02CrI4JoqNnY0OPCDH8VdSWZ3nquu9aQtVGu6xYvCgOFW+ RIVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=U4oVFdZuYHclhhiYM3y5LbmYNw6QTk652e3IwS9BkbM=; b=WGWHbXoGrtijRENTj0abbIOYhU2oNhkJ7CG4n7pco89dKnXt6xpjlQI841BjgUfee/ YCUDT7Dj81K8kEv/iasU0zuFaJw7AtWqgI4X7TAc4eolPKgwD/SW3MOGhizyT9JP6NjQ GikNozmsfEXhQ/kLR4j0dcEOi718rBTt1KExKK+CYU0qpd5nHGSWFTTa1I6SqCUnNAqW oWAJwmzvJtIQeyAilDgpiuQCMPeORo8PiGYz7gTywDxJw8lTaIr4qdvnSbopkTjtowa9 vcXAmK7BHs+jsbdJ4G3TZs1tbRc2Qa1qAmJ0Jv72SnrUjcbEmf3l6dzry5c53TEfrs++ qVEQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=WD+d1eZ5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ju17-20020a170903429100b001890f5957c4si6979862plb.353.2022.11.21.17.04.32; Mon, 21 Nov 2022 17:04:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=WD+d1eZ5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231960AbiKVBE0 (ORCPT + 99 others); Mon, 21 Nov 2022 20:04:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44768 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231924AbiKVBEY (ORCPT ); Mon, 21 Nov 2022 20:04:24 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ACFA3DE9A; Mon, 21 Nov 2022 17:04:23 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4A8AA61511; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B06BBC433D6; Tue, 22 Nov 2022 01:04:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079062; bh=cd9rgMHV5ytGRmwwPo9gYzUKZlyUq56zzJaeb9ZJrZY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WD+d1eZ5hWZsn4OWdNBCqObluC84BcwFQc1I/202wtBNI1lrqmeLZYnqJmFhRutjM 88QawAXneNZ6ZoDbJB16S+LBDCrWPqIHNk1nJiHKQLhsZJmKh1X7+3RYba6JOKUuJ0 F8+GgwLr3axpteO3550Si3usU+zWLDUynWclDTMHREi1jg29LgheeO9g9V4McYjWDT cic5G6/57tHpDiFYUDWk7RbinwGOE4UaSvTC1myCs1rs/d6fNSuF2MTHAaCxN3jyMK ZyBVnu3wW3Jad8wia6bzdvawwqXqyxr9uwgcB97vtFf3bCEN/4KZ2EXBzF0i79/HvO dE/js4FqZfQ6g== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 738485C0909; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH v2 rcu 02/16] rcu: Fix late wakeup when flush of bypass cblist happens Date: Mon, 21 Nov 2022 17:04:07 -0800 Message-Id: <20221122010421.3799681-2-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156271106679106?= X-GMAIL-MSGID: =?utf-8?q?1750156271106679106?= From: "Joel Fernandes (Google)" When the bypass cblist gets too big or its timeout has occurred, it is flushed into the main cblist. However, the bypass timer is still running and the behavior is that it would eventually expire and wake the GP thread. Since we are going to use the bypass cblist for lazy CBs, do the wakeup soon as the flush for "too big or too long" bypass list happens. Otherwise, long delays can happen for callbacks which get promoted from lazy to non-lazy. This is a good thing to do anyway (regardless of future lazy patches), since it makes the behavior consistent with behavior of other code paths where flushing into the ->cblist makes the GP kthread into a non-sleeping state quickly. [ Frederic Weisbecker: Changes to avoid unnecessary GP-thread wakeups plus comment changes. ] Reviewed-by: Frederic Weisbecker Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index ce526cc2791ca..f77a6d7e13564 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -433,8 +433,9 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, if ((ncbs && j != READ_ONCE(rdp->nocb_bypass_first)) || ncbs >= qhimark) { rcu_nocb_lock(rdp); + *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); + if (!rcu_nocb_flush_bypass(rdp, rhp, j)) { - *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstQ")); @@ -447,7 +448,12 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, rcu_advance_cbs_nowake(rdp->mynode, rdp); rdp->nocb_gp_adv_time = j; } - rcu_nocb_unlock_irqrestore(rdp, flags); + + // The flush succeeded and we moved CBs into the regular list. + // Don't wait for the wake up timer as it may be too far ahead. + // Wake up the GP thread now instead, if the cblist was empty. + __call_rcu_nocb_wake(rdp, *was_alldone, flags); + return true; // Callback already enqueued. } From patchwork Tue Nov 22 01:04:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24077 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1929714wrr; Mon, 21 Nov 2022 17:04:51 -0800 (PST) X-Google-Smtp-Source: AA0mqf4nd4+q9GHaHsmpJam3r9aMSxASV3+OCDM2ZMI7cPg0aMEcwRcSC2Uqh0tNF2iJkLUFb48M X-Received: by 2002:a63:eb16:0:b0:477:6fe1:cd6d with SMTP id t22-20020a63eb16000000b004776fe1cd6dmr6862295pgh.334.1669079090929; Mon, 21 Nov 2022 17:04:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079090; cv=none; d=google.com; s=arc-20160816; b=cwxkvIU8zhZem/h3bO2+F1AhIxLJSF4K51LBHc1PvxFMXjhr06/SLPFgdZsTKVo8n3 KZYoiNLyTji8vGnPOU1gP3TRcuffmoRJo9WQO//Qn0ghQ45GTCzE81LX353siJj/elHq 3Z1TH3XUrGyYSGqqdI9m9Z9A349PNqn8VKBZJDkJch1jGRJNXj3R5IO80rdq+eeCcSrx ePNJqSnHELpqQhmCbAnCiH9ECy2eqir2AU0p1xPVRhvkozt6GN/XrJTrB5mkVS/SGphK 4ylcWR6pz7ZiPAv1aIJwHMv9ZmW/yTN61zm8WYCBBG7GP087KDim3ApxXbOTpJldgbC0 9qJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ozKhBfyN+FxyuLIwEkPQLFxihGTESoRyKpstG62wPHM=; b=WcdASKSS1JhBW2WOb+nC7y4ehoNX4N+nnJXTBHdqbP7t0v2741CGamtzl+mvJSx4sL unM4AFQTeEUcDuIuPYa68uFXNU/eUfUQEKqk3gi6jgNd3HkKFtLNsjTbhJ0BFkT1VFup hMzWHyAaWeV0yYYT/+K+wVAUm2TQ849Q5u6x6AS3alUiiD5hgL9QU02TlG0oF2FpXoNq /FyZzNW2RzAEruntBsMnumIxE9ly4O24s87Rhace+DNa5bxdXvEabXUvGH9jcpr2nxro fD1ACB+oMMQnqtnLPZv7iXqnFdDRpVK//gmA11AxbdtmU+BivsUuReYq36U3vTQF4YOS JApA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=FStEKe3U; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g3-20020a63e603000000b0046ef006f51dsi12914033pgh.425.2022.11.21.17.04.37; Mon, 21 Nov 2022 17:04:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=FStEKe3U; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231951AbiKVBE3 (ORCPT + 99 others); Mon, 21 Nov 2022 20:04:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231927AbiKVBEY (ORCPT ); Mon, 21 Nov 2022 20:04:24 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D505511A12; Mon, 21 Nov 2022 17:04:23 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5F9BC61505; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B38BBC433D7; Tue, 22 Nov 2022 01:04:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079062; bh=JgCYhsIq4xhCHA9e/N+ivpZlC63szLfECFNNQENWSoc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FStEKe3UKD8G3yecmoiH9U+hi00FGgXyD45l/lhOmtJ104VYltWb4uubccUFvAgr4 gyJ77WtydXtgorQjyx4cKIfVf0rQAm4qcRpfShZXhrdR3HFS8duP385sCvj47gn9GD UOQAV925FbSslF3kyd0GpnkH87vF16s5nN6QCDAwvC9ZgWkcWPvE/Y2gPQSe7NT+Qj iwlHLa6We21VIpsqRc+qDLbCvq3vPLRpjYLAfZR3SLEC8ekk94sMKjap2zwa2Sne/9 9DTkb67x+WBKkxFCa8NRDlzVzYKUij3AKfprKx02lfQuFOunOr2rCnThYSV3c+Pfrf RTeaRLM+MTfWA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 756A15C0D1D; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Frederic Weisbecker , Joel Fernandes , "Paul E . McKenney" Subject: [PATCH v2 rcu 03/16] rcu: Fix missing nocb gp wake on rcu_barrier() Date: Mon, 21 Nov 2022 17:04:08 -0800 Message-Id: <20221122010421.3799681-3-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156276722439850?= X-GMAIL-MSGID: =?utf-8?q?1750156276722439850?= From: Frederic Weisbecker In preparation for RCU lazy changes, wake up the RCU nocb gp thread if needed after an entrain. This change prevents the RCU barrier callback from waiting in the queue for several seconds before the lazy callbacks in front of it are serviced. Reported-by: Joel Fernandes (Google) Signed-off-by: Frederic Weisbecker Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 11 +++++++++++ kernel/rcu/tree.h | 1 + kernel/rcu/tree_nocb.h | 5 +++++ 3 files changed, 17 insertions(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 6bb8e72bc8151..fb7a1b95af71e 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3894,6 +3894,8 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) { unsigned long gseq = READ_ONCE(rcu_state.barrier_sequence); unsigned long lseq = READ_ONCE(rdp->barrier_seq_snap); + bool wake_nocb = false; + bool was_alldone = false; lockdep_assert_held(&rcu_state.barrier_lock); if (rcu_seq_state(lseq) || !rcu_seq_state(gseq) || rcu_seq_ctr(lseq) != rcu_seq_ctr(gseq)) @@ -3902,7 +3904,14 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) rdp->barrier_head.func = rcu_barrier_callback; debug_rcu_head_queue(&rdp->barrier_head); rcu_nocb_lock(rdp); + /* + * Flush bypass and wakeup rcuog if we add callbacks to an empty regular + * queue. This way we don't wait for bypass timer that can reach seconds + * if it's fully lazy. + */ + was_alldone = rcu_rdp_is_offloaded(rdp) && !rcu_segcblist_pend_cbs(&rdp->cblist); WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies)); + wake_nocb = was_alldone && rcu_segcblist_pend_cbs(&rdp->cblist); if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head)) { atomic_inc(&rcu_state.barrier_cpu_count); } else { @@ -3910,6 +3919,8 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) rcu_barrier_trace(TPS("IRQNQ"), -1, rcu_state.barrier_sequence); } rcu_nocb_unlock(rdp); + if (wake_nocb) + wake_nocb_gp(rdp, false); smp_store_release(&rdp->barrier_seq_snap, gseq); } diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index d4a97e40ea9c3..925dd98f8b23b 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -439,6 +439,7 @@ static void zero_cpu_stall_ticks(struct rcu_data *rdp); static struct swait_queue_head *rcu_nocb_gp_get(struct rcu_node *rnp); static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq); static void rcu_init_one_nocb(struct rcu_node *rnp); +static bool wake_nocb_gp(struct rcu_data *rdp, bool force); static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, unsigned long j); static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index f77a6d7e13564..094fd454b6c38 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1558,6 +1558,11 @@ static void rcu_init_one_nocb(struct rcu_node *rnp) { } +static bool wake_nocb_gp(struct rcu_data *rdp, bool force) +{ + return false; +} + static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, unsigned long j) { From patchwork Tue Nov 22 01:04:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24089 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1934041wrr; Mon, 21 Nov 2022 17:15:19 -0800 (PST) X-Google-Smtp-Source: AA0mqf6EMabyGpE9MipWUfn2hfiyzQx+3zXaAmIuaIkaHc4gyWNBV91JMv3DpZSd5WpfTMecnjRE X-Received: by 2002:a17:906:3607:b0:78d:585e:d183 with SMTP id q7-20020a170906360700b0078d585ed183mr17253245ejb.768.1669079719017; Mon, 21 Nov 2022 17:15:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079719; cv=none; d=google.com; s=arc-20160816; b=pSvTbBFcpNu0qujcpU3RGHCchsR2JHjlFKsLxAPUTSQ/LXVKZBB6m2PIn6+e2R1pGk ONUOrng5gFykvToH7QrWvaayP7Oo+kmfYWrlptUE30gIY6uYYQwCMH6KQYV+NfFy5ySH RxNBSqAb97TKupOlUmSzQI0HY8qUpM8Mn06eBYH4NhErzQKss/fAVTC+Ibqi1/rLce70 UXcf+V0XOnpYtlYowl/wmjGjUc2tZ7SwDZr+Mx7LuldNASDqb0gpZkhnYj9wRa7Y0XN7 VOAphTO9SJ1LA0YQGMdqx61cvp7GzEImMe0JJOV8WAlAcvLLG4P1GguYFL8CZ9v/4hRu LuFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eP7nmMNbze1GrMH+n1VNMRu75jKpSBcId/giopZn9Ic=; b=lvOGwO5feaYaldoVYcYTCG6Vr7pidqZ/9Xkgt0DGRMLfLOkTebUBJUawoTHdJTRQhc OhLccO+yXdDpDcjJ0SIlG0CzyUsVCv3WHYtlUbDkAZE1gOYkRtUCFDo8mqy31UdMoE0x qpT3dzYMqoFvf2wfaOkQWzCnKm7vf0nBo3srX6toKb1MkomuWXzk8xELE0uw92gAj5oM XR0Ur1icztPxOpZDodJofePnex5TbrlkMVGKu3F5tt8JQA1dt46sehJLbGOOw92G/6qr RbBzevPgHFw3xLHBrkO/qhIxAr/QknfMar9xItSjFNpSa5ASOuvYoep225pptMYq33gb 64yw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=WEhUFm5O; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id sg34-20020a170907a42200b00791a7e441a3si11483710ejc.459.2022.11.21.17.14.53; Mon, 21 Nov 2022 17:15:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=WEhUFm5O; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232202AbiKVBFT (ORCPT + 99 others); Mon, 21 Nov 2022 20:05:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232003AbiKVBE1 (ORCPT ); Mon, 21 Nov 2022 20:04:27 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8694A12D20; Mon, 21 Nov 2022 17:04:25 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 21A00B818E5; Tue, 22 Nov 2022 01:04:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BA2B0C43470; Tue, 22 Nov 2022 01:04:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079062; bh=Th6GP3/7ShYA47Vn7dip1vFOs5BaI6ZyRfur59xpoJU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WEhUFm5OVWcTDLrDOUhTmaXhvRqgaAvnl2uc8lMtrpuAGWXtgdsken8lZRqMlA4A0 fNtCkncPjhqwg4WGJFpqR1umetFJ+UyWXyagoA2mP0HnH5Syg1n+rJIIUm3aoRbfRD ylWqnfZRgUCDSiiEknMEdchFfozJ+F19t8kl0UMZbS80d+G29qWHN5lPl+3hBFCIBk 5/hlKqt4QyG1NSrnGs7zIF1EYUxsVpRtrStdD6YK1CDGQ/nNVoC4X0i6VfVsY2OV2b 5S4hdTAHx0eR1YNoieWL0OwdQPjBRTZvFb6wD4/EOVc7fBlvMydSS0CVa4NywloMLD 6OMWDUYfdcIGQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 786385C116C; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , Paul McKenney , Frederic Weisbecker Subject: [PATCH v2 rcu 04/16] rcu: Make call_rcu() lazy to save power Date: Mon, 21 Nov 2022 17:04:09 -0800 Message-Id: <20221122010421.3799681-4-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156935649522941?= X-GMAIL-MSGID: =?utf-8?q?1750156935649522941?= From: "Joel Fernandes (Google)" Implement timer-based RCU callback batching (also known as lazy callbacks). With this we save about 5-10% of power consumed due to RCU requests that happen when system is lightly loaded or idle. By default, all async callbacks (queued via call_rcu) are marked lazy. An alternate API call_rcu_flush() is provided for the few users, for example synchronize_rcu(), that need the old behavior. The batch is flushed whenever a certain amount of time has passed, or the batch on a particular CPU grows too big. Also memory pressure will flush it in a future patch. To handle several corner cases automagically (such as rcu_barrier() and hotplug), we re-use bypass lists which were originally introduced to address lock contention, to handle lazy CBs as well. The bypass list length has the lazy CB length included in it. A separate lazy CB length counter is also introduced to keep track of the number of lazy CBs. [ paulmck: Fix formatting of inline call_rcu_lazy() definition. ] [ paulmck: Apply Zqiang feedback. ] Suggested-by: Paul McKenney Acked-by: Frederic Weisbecker Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- include/linux/rcupdate.h | 9 +++ kernel/rcu/Kconfig | 8 ++ kernel/rcu/rcu.h | 8 ++ kernel/rcu/tiny.c | 2 +- kernel/rcu/tree.c | 129 ++++++++++++++++++++----------- kernel/rcu/tree.h | 11 ++- kernel/rcu/tree_exp.h | 2 +- kernel/rcu/tree_nocb.h | 159 +++++++++++++++++++++++++++++++-------- 8 files changed, 246 insertions(+), 82 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 08605ce7379d7..f6288c1124425 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -108,6 +108,15 @@ static inline int rcu_preempt_depth(void) #endif /* #else #ifdef CONFIG_PREEMPT_RCU */ +#ifdef CONFIG_RCU_LAZY +void call_rcu_flush(struct rcu_head *head, rcu_callback_t func); +#else +static inline void call_rcu_flush(struct rcu_head *head, rcu_callback_t func) +{ + call_rcu(head, func); +} +#endif + /* Internal to kernel */ void rcu_init(void); extern int rcu_scheduler_active; diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index d471d22a5e21b..d78f6181c8aad 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -311,4 +311,12 @@ config TASKS_TRACE_RCU_READ_MB Say N here if you hate read-side memory barriers. Take the default if you are unsure. +config RCU_LAZY + bool "RCU callback lazy invocation functionality" + depends on RCU_NOCB_CPU + default n + help + To save power, batch RCU callbacks and flush after delay, memory + pressure, or callback list growing too big. + endmenu # "RCU Subsystem" diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index be5979da07f59..65704cbc9df7b 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -474,6 +474,14 @@ enum rcutorture_type { INVALID_RCU_FLAVOR }; +#if defined(CONFIG_RCU_LAZY) +unsigned long rcu_lazy_get_jiffies_till_flush(void); +void rcu_lazy_set_jiffies_till_flush(unsigned long j); +#else +static inline unsigned long rcu_lazy_get_jiffies_till_flush(void) { return 0; } +static inline void rcu_lazy_set_jiffies_till_flush(unsigned long j) { } +#endif + #if defined(CONFIG_TREE_RCU) void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags, unsigned long *gp_seq); diff --git a/kernel/rcu/tiny.c b/kernel/rcu/tiny.c index a33a8d4942c37..810479cf17bae 100644 --- a/kernel/rcu/tiny.c +++ b/kernel/rcu/tiny.c @@ -44,7 +44,7 @@ static struct rcu_ctrlblk rcu_ctrlblk = { void rcu_barrier(void) { - wait_rcu_gp(call_rcu); + wait_rcu_gp(call_rcu_flush); } EXPORT_SYMBOL(rcu_barrier); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index fb7a1b95af71e..e3c939b507ac5 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2728,47 +2728,8 @@ static void check_cb_ovld(struct rcu_data *rdp) raw_spin_unlock_rcu_node(rnp); } -/** - * call_rcu() - Queue an RCU callback for invocation after a grace period. - * @head: structure to be used for queueing the RCU updates. - * @func: actual callback function to be invoked after the grace period - * - * The callback function will be invoked some time after a full grace - * period elapses, in other words after all pre-existing RCU read-side - * critical sections have completed. However, the callback function - * might well execute concurrently with RCU read-side critical sections - * that started after call_rcu() was invoked. - * - * RCU read-side critical sections are delimited by rcu_read_lock() - * and rcu_read_unlock(), and may be nested. In addition, but only in - * v5.0 and later, regions of code across which interrupts, preemption, - * or softirqs have been disabled also serve as RCU read-side critical - * sections. This includes hardware interrupt handlers, softirq handlers, - * and NMI handlers. - * - * Note that all CPUs must agree that the grace period extended beyond - * all pre-existing RCU read-side critical section. On systems with more - * than one CPU, this means that when "func()" is invoked, each CPU is - * guaranteed to have executed a full memory barrier since the end of its - * last RCU read-side critical section whose beginning preceded the call - * to call_rcu(). It also means that each CPU executing an RCU read-side - * critical section that continues beyond the start of "func()" must have - * executed a memory barrier after the call_rcu() but before the beginning - * of that RCU read-side critical section. Note that these guarantees - * include CPUs that are offline, idle, or executing in user mode, as - * well as CPUs that are executing in the kernel. - * - * Furthermore, if CPU A invoked call_rcu() and CPU B invoked the - * resulting RCU callback function "func()", then both CPU A and CPU B are - * guaranteed to execute a full memory barrier during the time interval - * between the call to call_rcu() and the invocation of "func()" -- even - * if CPU A and CPU B are the same CPU (but again only if the system has - * more than one CPU). - * - * Implementation of these memory-ordering guarantees is described here: - * Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst. - */ -void call_rcu(struct rcu_head *head, rcu_callback_t func) +static void +__call_rcu_common(struct rcu_head *head, rcu_callback_t func, bool lazy) { static atomic_t doublefrees; unsigned long flags; @@ -2809,7 +2770,7 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) } check_cb_ovld(rdp); - if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags)) + if (rcu_nocb_try_bypass(rdp, head, &was_alldone, flags, lazy)) return; // Enqueued onto ->nocb_bypass, so just leave. // If no-CBs CPU gets here, rcu_nocb_try_bypass() acquired ->nocb_lock. rcu_segcblist_enqueue(&rdp->cblist, head); @@ -2831,8 +2792,84 @@ void call_rcu(struct rcu_head *head, rcu_callback_t func) local_irq_restore(flags); } } -EXPORT_SYMBOL_GPL(call_rcu); +#ifdef CONFIG_RCU_LAZY +/** + * call_rcu_flush() - Queue RCU callback for invocation after grace period, and + * flush all lazy callbacks (including the new one) to the main ->cblist while + * doing so. + * + * @head: structure to be used for queueing the RCU updates. + * @func: actual callback function to be invoked after the grace period + * + * The callback function will be invoked some time after a full grace + * period elapses, in other words after all pre-existing RCU read-side + * critical sections have completed. + * + * Use this API instead of call_rcu() if you don't want the callback to be + * invoked after very long periods of time, which can happen on systems without + * memory pressure and on systems which are lightly loaded or mostly idle. + * This function will cause callbacks to be invoked sooner than later at the + * expense of extra power. Other than that, this function is identical to, and + * reuses call_rcu()'s logic. Refer to call_rcu() for more details about memory + * ordering and other functionality. + */ +void call_rcu_flush(struct rcu_head *head, rcu_callback_t func) +{ + return __call_rcu_common(head, func, false); +} +EXPORT_SYMBOL_GPL(call_rcu_flush); +#endif + +/** + * call_rcu() - Queue an RCU callback for invocation after a grace period. + * By default the callbacks are 'lazy' and are kept hidden from the main + * ->cblist to prevent starting of grace periods too soon. + * If you desire grace periods to start very soon, use call_rcu_flush(). + * + * @head: structure to be used for queueing the RCU updates. + * @func: actual callback function to be invoked after the grace period + * + * The callback function will be invoked some time after a full grace + * period elapses, in other words after all pre-existing RCU read-side + * critical sections have completed. However, the callback function + * might well execute concurrently with RCU read-side critical sections + * that started after call_rcu() was invoked. + * + * RCU read-side critical sections are delimited by rcu_read_lock() + * and rcu_read_unlock(), and may be nested. In addition, but only in + * v5.0 and later, regions of code across which interrupts, preemption, + * or softirqs have been disabled also serve as RCU read-side critical + * sections. This includes hardware interrupt handlers, softirq handlers, + * and NMI handlers. + * + * Note that all CPUs must agree that the grace period extended beyond + * all pre-existing RCU read-side critical section. On systems with more + * than one CPU, this means that when "func()" is invoked, each CPU is + * guaranteed to have executed a full memory barrier since the end of its + * last RCU read-side critical section whose beginning preceded the call + * to call_rcu(). It also means that each CPU executing an RCU read-side + * critical section that continues beyond the start of "func()" must have + * executed a memory barrier after the call_rcu() but before the beginning + * of that RCU read-side critical section. Note that these guarantees + * include CPUs that are offline, idle, or executing in user mode, as + * well as CPUs that are executing in the kernel. + * + * Furthermore, if CPU A invoked call_rcu() and CPU B invoked the + * resulting RCU callback function "func()", then both CPU A and CPU B are + * guaranteed to execute a full memory barrier during the time interval + * between the call to call_rcu() and the invocation of "func()" -- even + * if CPU A and CPU B are the same CPU (but again only if the system has + * more than one CPU). + * + * Implementation of these memory-ordering guarantees is described here: + * Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst. + */ +void call_rcu(struct rcu_head *head, rcu_callback_t func) +{ + return __call_rcu_common(head, func, IS_ENABLED(CONFIG_RCU_LAZY)); +} +EXPORT_SYMBOL_GPL(call_rcu); /* Maximum number of jiffies to wait before draining a batch. */ #define KFREE_DRAIN_JIFFIES (5 * HZ) @@ -3507,7 +3544,7 @@ void synchronize_rcu(void) if (rcu_gp_is_expedited()) synchronize_rcu_expedited(); else - wait_rcu_gp(call_rcu); + wait_rcu_gp(call_rcu_flush); return; } @@ -3910,7 +3947,7 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) * if it's fully lazy. */ was_alldone = rcu_rdp_is_offloaded(rdp) && !rcu_segcblist_pend_cbs(&rdp->cblist); - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); wake_nocb = was_alldone && rcu_segcblist_pend_cbs(&rdp->cblist); if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head)) { atomic_inc(&rcu_state.barrier_cpu_count); @@ -4336,7 +4373,7 @@ void rcutree_migrate_callbacks(int cpu) my_rdp = this_cpu_ptr(&rcu_data); my_rnp = my_rdp->mynode; rcu_nocb_lock(my_rdp); /* irqs already disabled. */ - WARN_ON_ONCE(!rcu_nocb_flush_bypass(my_rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(my_rdp, NULL, jiffies, false)); raw_spin_lock_rcu_node(my_rnp); /* irqs already disabled. */ /* Leverage recent GPs and set GP for new callbacks. */ needwake = rcu_advance_cbs(my_rnp, rdp) || diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index 925dd98f8b23b..fcb5d696eb170 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -263,14 +263,16 @@ struct rcu_data { unsigned long last_fqs_resched; /* Time of last rcu_resched(). */ unsigned long last_sched_clock; /* Jiffies of last rcu_sched_clock_irq(). */ + long lazy_len; /* Length of buffered lazy callbacks. */ int cpu; }; /* Values for nocb_defer_wakeup field in struct rcu_data. */ #define RCU_NOCB_WAKE_NOT 0 #define RCU_NOCB_WAKE_BYPASS 1 -#define RCU_NOCB_WAKE 2 -#define RCU_NOCB_WAKE_FORCE 3 +#define RCU_NOCB_WAKE_LAZY 2 +#define RCU_NOCB_WAKE 3 +#define RCU_NOCB_WAKE_FORCE 4 #define RCU_JIFFIES_TILL_FORCE_QS (1 + (HZ > 250) + (HZ > 500)) /* For jiffies_till_first_fqs and */ @@ -441,9 +443,10 @@ static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq); static void rcu_init_one_nocb(struct rcu_node *rnp); static bool wake_nocb_gp(struct rcu_data *rdp, bool force); static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - unsigned long j); + unsigned long j, bool lazy); static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags); + bool *was_alldone, unsigned long flags, + bool lazy); static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty, unsigned long flags); static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level); diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 18e9b4cd78ef8..5cac056007982 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -937,7 +937,7 @@ void synchronize_rcu_expedited(void) /* If expedited grace periods are prohibited, fall back to normal. */ if (rcu_gp_is_normal()) { - wait_rcu_gp(call_rcu); + wait_rcu_gp(call_rcu_flush); return; } diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 094fd454b6c38..d6e4c076b0515 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -256,6 +256,31 @@ static bool wake_nocb_gp(struct rcu_data *rdp, bool force) return __wake_nocb_gp(rdp_gp, rdp, force, flags); } +/* + * LAZY_FLUSH_JIFFIES decides the maximum amount of time that + * can elapse before lazy callbacks are flushed. Lazy callbacks + * could be flushed much earlier for a number of other reasons + * however, LAZY_FLUSH_JIFFIES will ensure no lazy callbacks are + * left unsubmitted to RCU after those many jiffies. + */ +#define LAZY_FLUSH_JIFFIES (10 * HZ) +static unsigned long jiffies_till_flush = LAZY_FLUSH_JIFFIES; + +#ifdef CONFIG_RCU_LAZY +// To be called only from test code. +void rcu_lazy_set_jiffies_till_flush(unsigned long jif) +{ + jiffies_till_flush = jif; +} +EXPORT_SYMBOL(rcu_lazy_set_jiffies_till_flush); + +unsigned long rcu_lazy_get_jiffies_till_flush(void) +{ + return jiffies_till_flush; +} +EXPORT_SYMBOL(rcu_lazy_get_jiffies_till_flush); +#endif + /* * Arrange to wake the GP kthread for this NOCB group at some future * time when it is safe to do so. @@ -269,10 +294,14 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, raw_spin_lock_irqsave(&rdp_gp->nocb_gp_lock, flags); /* - * Bypass wakeup overrides previous deferments. In case - * of callback storm, no need to wake up too early. + * Bypass wakeup overrides previous deferments. In case of + * callback storms, no need to wake up too early. */ - if (waketype == RCU_NOCB_WAKE_BYPASS) { + if (waketype == RCU_NOCB_WAKE_LAZY && + rdp->nocb_defer_wakeup == RCU_NOCB_WAKE_NOT) { + mod_timer(&rdp_gp->nocb_timer, jiffies + jiffies_till_flush); + WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); + } else if (waketype == RCU_NOCB_WAKE_BYPASS) { mod_timer(&rdp_gp->nocb_timer, jiffies + 2); WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); } else { @@ -293,10 +322,13 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, * proves to be initially empty, just return false because the no-CB GP * kthread may need to be awakened in this case. * + * Return true if there was something to be flushed and it succeeded, otherwise + * false. + * * Note that this function always returns true if rhp is NULL. */ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - unsigned long j) + unsigned long j, bool lazy) { struct rcu_cblist rcl; @@ -310,7 +342,20 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, /* Note: ->cblist.len already accounts for ->nocb_bypass contents. */ if (rhp) rcu_segcblist_inc_len(&rdp->cblist); /* Must precede enqueue. */ - rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); + + /* + * If the new CB requested was a lazy one, queue it onto the main + * ->cblist so we can take advantage of a sooner grade period. + */ + if (lazy && rhp) { + rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, NULL); + rcu_cblist_enqueue(&rcl, rhp); + WRITE_ONCE(rdp->lazy_len, 0); + } else { + rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); + WRITE_ONCE(rdp->lazy_len, 0); + } + rcu_segcblist_insert_pend_cbs(&rdp->cblist, &rcl); WRITE_ONCE(rdp->nocb_bypass_first, j); rcu_nocb_bypass_unlock(rdp); @@ -326,13 +371,13 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, * Note that this function always returns true if rhp is NULL. */ static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - unsigned long j) + unsigned long j, bool lazy) { if (!rcu_rdp_is_offloaded(rdp)) return true; rcu_lockdep_assert_cblist_protected(rdp); rcu_nocb_bypass_lock(rdp); - return rcu_nocb_do_flush_bypass(rdp, rhp, j); + return rcu_nocb_do_flush_bypass(rdp, rhp, j, lazy); } /* @@ -345,7 +390,7 @@ static void rcu_nocb_try_flush_bypass(struct rcu_data *rdp, unsigned long j) if (!rcu_rdp_is_offloaded(rdp) || !rcu_nocb_bypass_trylock(rdp)) return; - WARN_ON_ONCE(!rcu_nocb_do_flush_bypass(rdp, NULL, j)); + WARN_ON_ONCE(!rcu_nocb_do_flush_bypass(rdp, NULL, j, false)); } /* @@ -367,12 +412,14 @@ static void rcu_nocb_try_flush_bypass(struct rcu_data *rdp, unsigned long j) * there is only one CPU in operation. */ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags) + bool *was_alldone, unsigned long flags, + bool lazy) { unsigned long c; unsigned long cur_gp_seq; unsigned long j = jiffies; long ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + bool bypass_is_lazy = (ncbs == READ_ONCE(rdp->lazy_len)); lockdep_assert_irqs_disabled(); @@ -417,25 +464,29 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, // If there hasn't yet been all that many ->cblist enqueues // this jiffy, tell the caller to enqueue onto ->cblist. But flush // ->nocb_bypass first. - if (rdp->nocb_nobypass_count < nocb_nobypass_lim_per_jiffy) { + // Lazy CBs throttle this back and do immediate bypass queuing. + if (rdp->nocb_nobypass_count < nocb_nobypass_lim_per_jiffy && !lazy) { rcu_nocb_lock(rdp); *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstQ")); - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, j)); + + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, j, false)); WARN_ON_ONCE(rcu_cblist_n_cbs(&rdp->nocb_bypass)); return false; // Caller must enqueue the callback. } // If ->nocb_bypass has been used too long or is too full, // flush ->nocb_bypass to ->cblist. - if ((ncbs && j != READ_ONCE(rdp->nocb_bypass_first)) || + if ((ncbs && !bypass_is_lazy && j != READ_ONCE(rdp->nocb_bypass_first)) || + (ncbs && bypass_is_lazy && + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush))) || ncbs >= qhimark) { rcu_nocb_lock(rdp); *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); - if (!rcu_nocb_flush_bypass(rdp, rhp, j)) { + if (!rcu_nocb_flush_bypass(rdp, rhp, j, lazy)) { if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstQ")); @@ -463,13 +514,24 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); rcu_segcblist_inc_len(&rdp->cblist); /* Must precede enqueue. */ rcu_cblist_enqueue(&rdp->nocb_bypass, rhp); + + if (lazy) + WRITE_ONCE(rdp->lazy_len, rdp->lazy_len + 1); + if (!ncbs) { WRITE_ONCE(rdp->nocb_bypass_first, j); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstBQ")); } rcu_nocb_bypass_unlock(rdp); smp_mb(); /* Order enqueue before wake. */ - if (ncbs) { + // A wake up of the grace period kthread or timer adjustment + // needs to be done only if: + // 1. Bypass list was fully empty before (this is the first + // bypass list entry), or: + // 2. Both of these conditions are met: + // a. The bypass list previously had only lazy CBs, and: + // b. The new CB is non-lazy. + if (ncbs && (!bypass_is_lazy || lazy)) { local_irq_restore(flags); } else { // No-CBs GP kthread might be indefinitely asleep, if so, wake. @@ -497,8 +559,10 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, unsigned long flags) __releases(rdp->nocb_lock) { + long bypass_len; unsigned long cur_gp_seq; unsigned long j; + long lazy_len; long len; struct task_struct *t; @@ -512,9 +576,16 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone, } // Need to actually to a wakeup. len = rcu_segcblist_n_cbs(&rdp->cblist); + bypass_len = rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_len = READ_ONCE(rdp->lazy_len); if (was_alldone) { rdp->qlen_last_fqs_check = len; - if (!irqs_disabled_flags(flags)) { + // Only lazy CBs in bypass list + if (lazy_len && bypass_len == lazy_len) { + rcu_nocb_unlock_irqrestore(rdp, flags); + wake_nocb_gp_defer(rdp, RCU_NOCB_WAKE_LAZY, + TPS("WakeLazy")); + } else if (!irqs_disabled_flags(flags)) { /* ... if queue was empty ... */ rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); @@ -605,12 +676,12 @@ static void nocb_gp_sleep(struct rcu_data *my_rdp, int cpu) static void nocb_gp_wait(struct rcu_data *my_rdp) { bool bypass = false; - long bypass_ncbs; int __maybe_unused cpu = my_rdp->cpu; unsigned long cur_gp_seq; unsigned long flags; bool gotcbs = false; unsigned long j = jiffies; + bool lazy = false; bool needwait_gp = false; // This prevents actual uninitialized use. bool needwake; bool needwake_gp; @@ -640,24 +711,43 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) * won't be ignored for long. */ list_for_each_entry(rdp, &my_rdp->nocb_head_rdp, nocb_entry_rdp) { + long bypass_ncbs; + bool flush_bypass = false; + long lazy_ncbs; + trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("Check")); rcu_nocb_lock_irqsave(rdp, flags); lockdep_assert_held(&rdp->nocb_lock); bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); - if (bypass_ncbs && + lazy_ncbs = READ_ONCE(rdp->lazy_len); + + if (bypass_ncbs && (lazy_ncbs == bypass_ncbs) && + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush) || + bypass_ncbs > 2 * qhimark)) { + flush_bypass = true; + } else if (bypass_ncbs && (lazy_ncbs != bypass_ncbs) && (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) || bypass_ncbs > 2 * qhimark)) { - // Bypass full or old, so flush it. - (void)rcu_nocb_try_flush_bypass(rdp, j); - bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + flush_bypass = true; } else if (!bypass_ncbs && rcu_segcblist_empty(&rdp->cblist)) { rcu_nocb_unlock_irqrestore(rdp, flags); continue; /* No callbacks here, try next. */ } + + if (flush_bypass) { + // Bypass full or old, so flush it. + (void)rcu_nocb_try_flush_bypass(rdp, j); + bypass_ncbs = rcu_cblist_n_cbs(&rdp->nocb_bypass); + lazy_ncbs = READ_ONCE(rdp->lazy_len); + } + if (bypass_ncbs) { trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, - TPS("Bypass")); - bypass = true; + bypass_ncbs == lazy_ncbs ? TPS("Lazy") : TPS("Bypass")); + if (bypass_ncbs == lazy_ncbs) + lazy = true; + else + bypass = true; } rnp = rdp->mynode; @@ -705,12 +795,20 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) my_rdp->nocb_gp_gp = needwait_gp; my_rdp->nocb_gp_seq = needwait_gp ? wait_gp_seq : 0; - if (bypass && !rcu_nocb_poll) { - // At least one child with non-empty ->nocb_bypass, so set - // timer in order to avoid stranding its callbacks. - wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_BYPASS, - TPS("WakeBypassIsDeferred")); + // At least one child with non-empty ->nocb_bypass, so set + // timer in order to avoid stranding its callbacks. + if (!rcu_nocb_poll) { + // If bypass list only has lazy CBs. Add a deferred lazy wake up. + if (lazy && !bypass) { + wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_LAZY, + TPS("WakeLazyIsDeferred")); + // Otherwise add a deferred bypass wake up. + } else if (bypass) { + wake_nocb_gp_defer(my_rdp, RCU_NOCB_WAKE_BYPASS, + TPS("WakeBypassIsDeferred")); + } } + if (rcu_nocb_poll) { /* Polling, so trace if first poll in the series. */ if (gotcbs) @@ -1036,7 +1134,7 @@ static long rcu_nocb_rdp_deoffload(void *arg) * return false, which means that future calls to rcu_nocb_try_bypass() * will refuse to put anything into the bypass. */ - WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies)); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); /* * Start with invoking rcu_core() early. This way if the current thread * happens to preempt an ongoing call to rcu_core() in the middle, @@ -1278,6 +1376,7 @@ static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp) raw_spin_lock_init(&rdp->nocb_gp_lock); timer_setup(&rdp->nocb_timer, do_nocb_deferred_wakeup_timer, 0); rcu_cblist_init(&rdp->nocb_bypass); + WRITE_ONCE(rdp->lazy_len, 0); mutex_init(&rdp->nocb_gp_kthread_mutex); } @@ -1564,13 +1663,13 @@ static bool wake_nocb_gp(struct rcu_data *rdp, bool force) } static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - unsigned long j) + unsigned long j, bool lazy) { return true; } static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, - bool *was_alldone, unsigned long flags) + bool *was_alldone, unsigned long flags, bool lazy) { return false; } From patchwork Tue Nov 22 01:04:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24087 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1932079wrr; Mon, 21 Nov 2022 17:10:12 -0800 (PST) X-Google-Smtp-Source: AA0mqf4jBR+q0OGngq5gfYCCT6QNOhecK95FyHXtzHQsRgk37i50pmpJcx2S1qwnEXGI9Rwbe8Za X-Received: by 2002:aa7:cc14:0:b0:468:58d4:a10e with SMTP id q20-20020aa7cc14000000b0046858d4a10emr19208805edt.222.1669079412096; Mon, 21 Nov 2022 17:10:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079412; cv=none; d=google.com; s=arc-20160816; b=oKQ+NC1ofr0rVP/UCC82QPihp22NTrIp06EtYpa1OaLyeZTABPwl3P/T0eVg0LdvJz c0gq+h6ESuQr67jEME7r6tulp943UEh0jZ8xtTQgqfHAslu+kzaK/i/g8haDPEQM2+aN f456Pcukr4GkkiGnUWz/Wm87o8HkcL/zD5rwsrfpwf5jquk9nTqCF6o13V2f/T0NY6cy M+A6t8wXxw2MoCy7qRWKxKSZ2IACegScdFFHQhckARNoatXPOxPiO0xLQgRNqxx080TF RUtpu0wDYGbVhIEOIKFiQi84et/n1n6lJuhH30cFgYc3SMDJoN2A83JUYGZiZte6gdVL KDeA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ZtA1/GcSFcFiU1ppp5Q4H/NERsG7tztvT92JiROd1mE=; b=bUswySyl/rETOtc8qRPeg2eNGGPGqC0as0bMbp3i3WWR6lN07GdPxKamp/EbDt09by 4paOo7EmV6KGa/dpbGnPxQcCMOZs5CSxv3Dh+u4qQbMGlDeh4Zq9Eo0LyadB+6ec7LD5 0Wshk80AuPjGekoER8xqMCk2PqSR8BqpSc7V9/lGAJiT3BR+OOsZ/jbdI9VlSGG1F7Rt 3+GETW1KyIJQyIJJ0ce2y5G/hrwbmI58cySGdvkHO/oryHoZUyIZ9ABSaXjrDK6khf6s vof+Ty62oVHVs6FV2dQd4JZPMBqnd9zcn0p5jiFQUZ+bJ6gnIpn7hD19XnM4sqWhsXNY Rd/A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=msgDCNIf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ss28-20020a170907c01c00b007ad79c4f58fsi8761757ejc.120.2022.11.21.17.09.48; Mon, 21 Nov 2022 17:10:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=msgDCNIf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232127AbiKVBFJ (ORCPT + 99 others); Mon, 21 Nov 2022 20:05:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44780 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231970AbiKVBE0 (ORCPT ); Mon, 21 Nov 2022 20:04:26 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9D8F615804; Mon, 21 Nov 2022 17:04:25 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 36A14B818E6; Tue, 22 Nov 2022 01:04:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C09FDC433C1; Tue, 22 Nov 2022 01:04:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079062; bh=U3+4ieL/lmCXWeRZRiLB8u0S5rhXMcyArw0nT7Mm3tc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=msgDCNIf5wTHEMiqK1Oby1Kv9Mul5DyWDroLunYs3RPd6Q4u0lEwQZKrTci5eEb8g D3NdwkLIwRwI6jk7mIMqe8ZH/iMBjQIKoUAxKYnVJxFFjzd/gWwAgFKUW+BQiIcMaQ fMA6510Y+gWdcIfhQK17+Bh0yKljOgZ4RhZXP/BL/VD8U3Jr2whVpiM6gHRbs1wXoy z2JnJ8viIApXCPsicuQ8+9JM+QwsBoDnbkMs+M0qlwfhT8kx1/XcJFyAp8uKEcLGuL yWRCMfiBG4xzWEkPbjr8rXkELND3r3n7zETP0DjxmeBfT34P2t0T6klB+V7PgdumKq mNb4lVjLFgKCA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 799CB5C116D; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , "Paul E . McKenney" Subject: [PATCH v2 rcu 05/16] rcu: Refactor code a bit in rcu_nocb_do_flush_bypass() Date: Mon, 21 Nov 2022 17:04:10 -0800 Message-Id: <20221122010421.3799681-5-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156613250180528?= X-GMAIL-MSGID: =?utf-8?q?1750156613250180528?= From: "Joel Fernandes (Google)" This consolidates the code a bit and makes it cleaner. Functionally it is the same. Reported-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney Reviewed-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index d6e4c076b0515..213daf81c057f 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -327,10 +327,11 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, * * Note that this function always returns true if rhp is NULL. */ -static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, +static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp_in, unsigned long j, bool lazy) { struct rcu_cblist rcl; + struct rcu_head *rhp = rhp_in; WARN_ON_ONCE(!rcu_rdp_is_offloaded(rdp)); rcu_lockdep_assert_cblist_protected(rdp); @@ -345,16 +346,16 @@ static bool rcu_nocb_do_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, /* * If the new CB requested was a lazy one, queue it onto the main - * ->cblist so we can take advantage of a sooner grade period. + * ->cblist so that we can take advantage of the grace-period that will + * happen regardless. But queue it onto the bypass list first so that + * the lazy CB is ordered with the existing CBs in the bypass list. */ if (lazy && rhp) { - rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, NULL); - rcu_cblist_enqueue(&rcl, rhp); - WRITE_ONCE(rdp->lazy_len, 0); - } else { - rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); - WRITE_ONCE(rdp->lazy_len, 0); + rcu_cblist_enqueue(&rdp->nocb_bypass, rhp); + rhp = NULL; } + rcu_cblist_flush_enqueue(&rcl, &rdp->nocb_bypass, rhp); + WRITE_ONCE(rdp->lazy_len, 0); rcu_segcblist_insert_pend_cbs(&rdp->cblist, &rcl); WRITE_ONCE(rdp->nocb_bypass_first, j); From patchwork Tue Nov 22 01:04:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24081 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1929918wrr; Mon, 21 Nov 2022 17:05:14 -0800 (PST) X-Google-Smtp-Source: AA0mqf4DAn23Yg0m/zTqhZfo2q0UbDRsFfpfeht0UkJG2LbSbJLpk2F2dQZXMcr2lpuVzjFu0wOE X-Received: by 2002:a17:903:2601:b0:189:315f:d6fc with SMTP id jd1-20020a170903260100b00189315fd6fcmr1769329plb.35.1669079114556; Mon, 21 Nov 2022 17:05:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079114; cv=none; d=google.com; s=arc-20160816; b=meOiOX8aaADyfzew0e6SpYUmuNbZPS5e4P41DmsZBnzpvQPvcfCCsciIvZ30yiIFDt WR4nV+s5CR2LX8BrNfKu6/NHZXs5WGRywXVkkv9tqIEvjRiXTd9rK3YEjdR/1tSCbD4v LACMuXL8CXR32A7Rxer2GThbKa6Zexdmc6d95ybKH41VReGSLMYIUDDbh4mhYGsYUKtW hTlIiiZdfHCwjKKDobMGcb2BLVwJZkd1lf91GOsfpUcRKwWjXPrVA4/pm2j64jNWv7ON vxy9IPSWKW47ja6m9AUAv1LWURrK539q0gWOS6M7Vfju4WEykPFS3ZJtN2HwLnFTnXmM cbBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KpkkHRs6NSlaA30zXYSvuq9VaM3mDcAD5+jXqlQvvlo=; b=r8XEZ2LHGqeMdIKZE3MTo1qfukFJIdsEyGoR3raIz7W+T1IOuC86II1TOeMswf9Olt 1BxspDZ7Q+jUm3IDqRrbccuntIFCWBCZMgRrzYDsx03vkY2/Dlv8HGUe/BjgwegRLi+s 14JoSL/LKNsH6ABvhK1AjM/FGpYuPYXkvgEzxWGItmYPzUcy4JcUb9DVoXk0MTgBGaLX Vl9xh/yB1TlQ63fQ+RT+worCN7SBDDplD03AoB8fEzbK/jYGD7TRprbKGZHHa6TQVeQ4 GrIVTc2DzPhepvpxsOD8QJdR649o52WmRyMjbWyC9Fs3C9J9gEic/Np9BmEcRF7khKMq Mygw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=l80XZpuR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j190-20020a6380c7000000b00477312c910dsi10170588pgd.657.2022.11.21.17.05.01; Mon, 21 Nov 2022 17:05:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=l80XZpuR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232070AbiKVBEm (ORCPT + 99 others); Mon, 21 Nov 2022 20:04:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44808 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231935AbiKVBEZ (ORCPT ); Mon, 21 Nov 2022 20:04:25 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A08CADE9A; Mon, 21 Nov 2022 17:04:24 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id F0AE06151C; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1A663C43151; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079063; bh=hIFNQHcIbY8YtBkmBJx5icL1buQHNTtIxJeozRTrMW8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=l80XZpuRLUSSJsfBa+XlMP4gcQbNsw2NpaOiGOrWZDuoa2xpSHx4WmtNaVKRVe8O0 ZylXtgqWLL1Aof0vdX+L2M+9DM8OvANGbU/rLbX47UnsyD8H3pj1PARWK+P2skct8K sahbDCb47N+kfh7+vnqWmpMsjiEa/X9Fj66FDAXbXTv4xk3y0DVadRw18JldcpDO4d PKKT8fidKmlnBctadwf/v+xZWx1iyBlGYGs4DkXI5nuL7qy2+/vrFszkhlFCXlufO3 NljYnSE3YelGoIzCERzGJg014E5/c76UNpkHnNQwaLDuw+1uXCPn1p+Uk4F7p8GU/a AFvJewBwASYmA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 7BA6B5C1219; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Vineeth Pillai , Joel Fernandes , "Paul E . McKenney" Subject: [PATCH v2 rcu 06/16] rcu: Shrinker for lazy rcu Date: Mon, 21 Nov 2022 17:04:11 -0800 Message-Id: <20221122010421.3799681-6-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156301817061976?= X-GMAIL-MSGID: =?utf-8?q?1750156301817061976?= From: Vineeth Pillai The shrinker is used to speed up the free'ing of memory potentially held by RCU lazy callbacks. RCU kernel module test cases show this to be effective. Test is introduced in a later patch. Signed-off-by: Vineeth Pillai Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 52 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 213daf81c057f..9e1c8caec5ceb 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1312,6 +1312,55 @@ int rcu_nocb_cpu_offload(int cpu) } EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload); +static unsigned long +lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + + count += READ_ONCE(rdp->lazy_len); + } + + return count ? count : SHRINK_EMPTY; +} + +static unsigned long +lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu; + unsigned long flags; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + int _count = READ_ONCE(rdp->lazy_len); + + if (_count == 0) + continue; + rcu_nocb_lock_irqsave(rdp, flags); + WRITE_ONCE(rdp->lazy_len, 0); + rcu_nocb_unlock_irqrestore(rdp, flags); + wake_nocb_gp(rdp, false); + sc->nr_to_scan -= _count; + count += _count; + if (sc->nr_to_scan <= 0) + break; + } + return count ? count : SHRINK_STOP; +} + +static struct shrinker lazy_rcu_shrinker = { + .count_objects = lazy_rcu_shrink_count, + .scan_objects = lazy_rcu_shrink_scan, + .batch = 0, + .seeks = DEFAULT_SEEKS, +}; + void __init rcu_init_nohz(void) { int cpu; @@ -1342,6 +1391,9 @@ void __init rcu_init_nohz(void) if (!rcu_state.nocb_is_setup) return; + if (register_shrinker(&lazy_rcu_shrinker, "rcu-lazy")) + pr_err("Failed to register lazy_rcu shrinker!\n"); + if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) { pr_info("\tNote: kernel parameter 'rcu_nocbs=', 'nohz_full', or 'isolcpus=' contains nonexistent CPUs.\n"); cpumask_and(rcu_nocb_mask, cpu_possible_mask, From patchwork Tue Nov 22 01:04:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24080 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1929890wrr; Mon, 21 Nov 2022 17:05:11 -0800 (PST) X-Google-Smtp-Source: AA0mqf6Kgbx+/whyQwv5Zx+nI+kvm3xRwppXSlnO/S9eOFPIM3b14ZBehcyvoubTygpgPqX3K3hu X-Received: by 2002:a62:6445:0:b0:56c:3a0e:cf13 with SMTP id y66-20020a626445000000b0056c3a0ecf13mr1996596pfb.29.1669079111504; Mon, 21 Nov 2022 17:05:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079111; cv=none; d=google.com; s=arc-20160816; b=OXriw88f+jmdPpbxeo4dgoqlTN4Phtj/pyWdzcgw9ULrbCwrtCVWyiltObhVGqoz7Z IKyqoPUKR1YEmizbgMjCYY977Qxp/tgqn8wtFYRid6aerO6Bcs3q3IkrHAUC84Hil2XX hlVftEZb77x4Qa6PWChwyaIav6KF53xxmMT6SU51sLdHCQY7OuYgAh3nsfYVnzJH+upn vfZsZOpiuZv6j1poZsvGFqGgFQYxvg+Ylq9A2R/lAMxiF+YJi4BjWyvBU+baCmz/laZm 5rVf2FyKbqJIMS5rTlYvNzEjCyKkwkq9kMM7nx6SduJR+9Ni7VHf0y+8vDPEFkEAERy4 sQig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ZHuFqUUNuNBBbGtm2obtlgmElzjDDyVm88R/Db4kxYU=; b=dcBEMl0jtjAhB0VXsnIrtRfntMxwq9pE/zymn6nKxwuVnt2bkBkzqBQuY/zma1hRN2 O3a11JEuGmQhGP51h5mIEBi50cKACpAqgq25o6N3qU0Mpu7hw0zD/8ch2SC0E/dDtf6H T1lvISzAyfDlJBbi6z+gDlrkoxyhIdT6rP8ql/Jf3KcQAt8KFYuRRo+L8lNKp9LqP6hz ZTBA6kPJIwmaEMPSLzwHLUiy4QgA1dah8FBRYArYShNxXT/lBTav3PrmpynO/EWMsX57 vY1VN+fWS3HCkDZZ9kcxGYu7IOs332DE4JwhfSI1MY0fLZqiSgYUTbeHPcqQHgSmRU/j psTw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=NyFZp1u8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j191-20020a6380c8000000b0047731d8c0ddsi9996746pgd.511.2022.11.21.17.04.58; Mon, 21 Nov 2022 17:05:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=NyFZp1u8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232095AbiKVBEj (ORCPT + 99 others); Mon, 21 Nov 2022 20:04:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231934AbiKVBEZ (ORCPT ); Mon, 21 Nov 2022 20:04:25 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4807312D20; Mon, 21 Nov 2022 17:04:24 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id DACA76151B; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1A54FC43150; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079063; bh=B649m1ZX6zSaxE2rwxoHbFbQjFjTg7X6m3NxA9r7zOA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NyFZp1u84JOtAE8OwuebFFFgYT5PiGcvKQdE7X/tibjT5bljbTQ4ha1TUZcJmWmnM tNavfqq2R3fCSAuF1EbxMtejTujT+jmqbWEIrru9oIn0dTV4UuhqB2wdhxJYsKGZNj ah65W3f9GlKpVQX/PF1uQEvvBd7/NTAgTpEfouPIwMj/mYst07jKPAL2HkBZ+WK3O2 qCc1o/EhXzYA2XBOXp9fzyGdp1W9sNTye5prh+V4okAYHb89i4pIov7ThhO56HUDWK Zl1g2IacTjXp42bKIAMpl8kAvgNcPo+szNqHGERSk6nHe38x3gqYQyMMUOrBxWS0ji 6oIJl84aB2uUA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 7D6A55C12EB; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , "Paul E . McKenney" Subject: [PATCH v2 rcu 07/16] rcuscale: Add laziness and kfree tests Date: Mon, 21 Nov 2022 17:04:12 -0800 Message-Id: <20221122010421.3799681-7-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156298537291807?= X-GMAIL-MSGID: =?utf-8?q?1750156298537291807?= From: "Joel Fernandes (Google)" This commit adds 2 tests to rcuscale. The first one is a startup test to check whether we are not too lazy or too hard working. The second one causes kfree_rcu() itself to use call_rcu() and checks memory pressure. Testing indicates that the new call_rcu() keeps memory pressure under control roughly as well as does kfree_rcu(). [ paulmck: Apply checkpatch feedback. ] Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/rcuscale.c | 67 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 65 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 3ef02d4a81085..3baded807a616 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -95,6 +95,7 @@ torture_param(int, verbose, 1, "Enable verbose debugging printk()s"); torture_param(int, writer_holdoff, 0, "Holdoff (us) between GPs, zero to disable"); torture_param(int, kfree_rcu_test, 0, "Do we run a kfree_rcu() scale test?"); torture_param(int, kfree_mult, 1, "Multiple of kfree_obj size to allocate."); +torture_param(int, kfree_by_call_rcu, 0, "Use call_rcu() to emulate kfree_rcu()?"); static char *scale_type = "rcu"; module_param(scale_type, charp, 0444); @@ -659,6 +660,14 @@ struct kfree_obj { struct rcu_head rh; }; +/* Used if doing RCU-kfree'ing via call_rcu(). */ +static void kfree_call_rcu(struct rcu_head *rh) +{ + struct kfree_obj *obj = container_of(rh, struct kfree_obj, rh); + + kfree(obj); +} + static int kfree_scale_thread(void *arg) { @@ -696,6 +705,11 @@ kfree_scale_thread(void *arg) if (!alloc_ptr) return -ENOMEM; + if (kfree_by_call_rcu) { + call_rcu(&(alloc_ptr->rh), kfree_call_rcu); + continue; + } + // By default kfree_rcu_test_single and kfree_rcu_test_double are // initialized to false. If both have the same value (false or true) // both are randomly tested, otherwise only the one with value true @@ -767,11 +781,58 @@ kfree_scale_shutdown(void *arg) return -EINVAL; } +// Used if doing RCU-kfree'ing via call_rcu(). +static unsigned long jiffies_at_lazy_cb; +static struct rcu_head lazy_test1_rh; +static int rcu_lazy_test1_cb_called; +static void call_rcu_lazy_test1(struct rcu_head *rh) +{ + jiffies_at_lazy_cb = jiffies; + WRITE_ONCE(rcu_lazy_test1_cb_called, 1); +} + static int __init kfree_scale_init(void) { - long i; int firsterr = 0; + long i; + unsigned long jif_start; + unsigned long orig_jif; + + // Also, do a quick self-test to ensure laziness is as much as + // expected. + if (kfree_by_call_rcu && !IS_ENABLED(CONFIG_RCU_LAZY)) { + pr_alert("CONFIG_RCU_LAZY is disabled, falling back to kfree_rcu() for delayed RCU kfree'ing\n"); + kfree_by_call_rcu = 0; + } + + if (kfree_by_call_rcu) { + /* do a test to check the timeout. */ + orig_jif = rcu_lazy_get_jiffies_till_flush(); + + rcu_lazy_set_jiffies_till_flush(2 * HZ); + rcu_barrier(); + + jif_start = jiffies; + jiffies_at_lazy_cb = 0; + call_rcu(&lazy_test1_rh, call_rcu_lazy_test1); + + smp_cond_load_relaxed(&rcu_lazy_test1_cb_called, VAL == 1); + + rcu_lazy_set_jiffies_till_flush(orig_jif); + + if (WARN_ON_ONCE(jiffies_at_lazy_cb - jif_start < 2 * HZ)) { + pr_alert("ERROR: call_rcu() CBs are not being lazy as expected!\n"); + WARN_ON_ONCE(1); + return -1; + } + + if (WARN_ON_ONCE(jiffies_at_lazy_cb - jif_start > 3 * HZ)) { + pr_alert("ERROR: call_rcu() CBs are being too lazy!\n"); + WARN_ON_ONCE(1); + return -1; + } + } kfree_nrealthreads = compute_real(kfree_nthreads); /* Start up the kthreads. */ @@ -784,7 +845,9 @@ kfree_scale_init(void) schedule_timeout_uninterruptible(1); } - pr_alert("kfree object size=%zu\n", kfree_mult * sizeof(struct kfree_obj)); + pr_alert("kfree object size=%zu, kfree_by_call_rcu=%d\n", + kfree_mult * sizeof(struct kfree_obj), + kfree_by_call_rcu); kfree_reader_tasks = kcalloc(kfree_nrealthreads, sizeof(kfree_reader_tasks[0]), GFP_KERNEL); From patchwork Tue Nov 22 01:04:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24079 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1929867wrr; Mon, 21 Nov 2022 17:05:08 -0800 (PST) X-Google-Smtp-Source: AA0mqf7AxI1gAUkFMwFZsL2bnaPwZURW24ERsCFlFxdKwFa9x3wSCb5I21GiTQthtS6xdaAttPqD X-Received: by 2002:a63:4d01:0:b0:476:898c:deda with SMTP id a1-20020a634d01000000b00476898cdedamr1065739pgb.222.1669079108183; Mon, 21 Nov 2022 17:05:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079108; cv=none; d=google.com; s=arc-20160816; b=tSNzShbAW3gRuIwQmkSUuZbTOvkn865ZZ+x3+Gut0vKYyo1Xo6Y0UA3ZXZ1hINaXtZ 7XB3uMs6DmH8pPOgtvS8Gq2c1OTxdviFlxtl3LX2JrTuHZbK+Rmad0R5wkflu1nLeFBP R4eaVfXnskkluKjg2AEz41+c6JPbob53wGVPe+Gf1XdybrApISUtbudQuPJCSMnpNTzG WJUV7UrbnwTOiF1jEHuRm7oAIsO9FlKQavprDGpY9lxiPB5XKUOWcKy10GxQ+PkVCZKZ yC7+xpWbpS0GnXyLfu+5KC72KIVMPNEFWsGtzq6JNh7o4kWXZytsUD74ARnMfYJYst1c aHAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=T+B3Wd3V/zk3IHc0EsSAng9ddSG+4DT5R4CU+3RmirU=; b=Hufhz+BH0KrtMZCO+lrbdbID4cwc5++LLP9oqjXGz/IpNmlUId6JFpULgMGmTYV7Hy whQRNdwrARSq0MIfDO58nAn/9kXyzn1FJ1xJQ6zIAyCVeGqllyzgV8jhYpvziapMqq+3 2VZY4IDKHspGLWvxpcPIZlCgRtXuggYQDbmZuCpBfxcxt75lBj23D4pPYdPTrjv3V54+ WfWQFPZ2zHnGkpOujkOwTL9Q0GbZJg560vxy23HrEmKW6U+PbRNL09VDdLHclKkaaL5r r1o4YmjpkRjmy4M/BYAjfasLlZWNw6buKH/bOOt8znG3wuYHzRaMQFMKUi+wPlnhfycs BxeA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ouiCNi1M; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u14-20020a63470e000000b0044e12bf26ecsi13367416pga.465.2022.11.21.17.04.53; Mon, 21 Nov 2022 17:05:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ouiCNi1M; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232076AbiKVBEf (ORCPT + 99 others); Mon, 21 Nov 2022 20:04:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44780 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231933AbiKVBEZ (ORCPT ); Mon, 21 Nov 2022 20:04:25 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49D9D15804; Mon, 21 Nov 2022 17:04:24 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id DA1D0614C7; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1A511C4314F; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079063; bh=z9JjjzgTdogo9ezjaH1Kvhao6xVgeTuYWUcd1hbJd34=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ouiCNi1MquoLuOlDKxTEYf7uSk1aoeJVXXFQEgmoBoFB55aKExq8LU532Qtw1F2i9 icTvelfXLI8qLKPC1QyophlQshZT87N4wmcllBRKX60rg+hwLyCLcItLqgFBUugFU5 u5KJLARZ/rxaXegIpMETwQ+hG85w8RDUFy5rRn2rKJKFkzqP/ajNoG8kAnoVewObpK YJiwLIJMoPhBWBUAKJMkB4dmSM/m9gyGllbtvmXMHCNCnzn8tkQwWVW1fj9DSOH+II 27lgpW4F5ldo4aH+iVHDV3JVV+45e6OMjMpbNmrODFChzoHs+we9qOt6XKP6Gu52kS 2x+3g3Jk+ARvQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 7F8865C13DD; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , "Paul E . McKenney" , Dennis Zhou , Tejun Heo , Christoph Lameter , linux-mm@kvack.org Subject: [PATCH v2 rcu 08/16] percpu-refcount: Use call_rcu_flush() for atomic switch Date: Mon, 21 Nov 2022 17:04:13 -0800 Message-Id: <20221122010421.3799681-8-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156294993312766?= X-GMAIL-MSGID: =?utf-8?q?1750156294993312766?= From: "Joel Fernandes (Google)" Earlier commits in this series allow battery-powered systems to build their kernels with the default-disabled CONFIG_RCU_LAZY=y Kconfig option. This Kconfig option causes call_rcu() to delay its callbacks in order to batch callbacks. This means that a given RCU grace period covers more callbacks, thus reducing the number of grace periods, in turn reducing the amount of energy consumed, which increases battery lifetime which can be a very good thing. This is not a subtle effect: In some important use cases, the battery lifetime is increased by more than 10%. This CONFIG_RCU_LAZY=y option is available only for CPUs that offload callbacks, for example, CPUs mentioned in the rcu_nocbs kernel boot parameter passed to kernels built with CONFIG_RCU_NOCB_CPU=y. Delaying callbacks is normally not a problem because most callbacks do nothing but free memory. If the system is short on memory, a shrinker will kick all currently queued lazy callbacks out of their laziness, thus freeing their memory in short order. Similarly, the rcu_barrier() function, which blocks until all currently queued callbacks are invoked, will also kick lazy callbacks, thus enabling rcu_barrier() to complete in a timely manner. However, there are some cases where laziness is not a good option. For example, synchronize_rcu() invokes call_rcu(), and blocks until the newly queued callback is invoked. It would not be a good for synchronize_rcu() to block for ten seconds, even on an idle system. Therefore, synchronize_rcu() invokes call_rcu_flush() instead of call_rcu(). The arrival of a non-lazy call_rcu_flush() callback on a given CPU kicks any lazy callbacks that might be already queued on that CPU. After all, if there is going to be a grace period, all callbacks might as well get full benefit from it. Yes, this could be done the other way around by creating a call_rcu_lazy(), but earlier experience with this approach and feedback at the 2022 Linux Plumbers Conference shifted the approach to call_rcu() being lazy with call_rcu_flush() for the few places where laziness is inappropriate. And another call_rcu() instance that cannot be lazy is the one on the percpu refcounter's "per-CPU to atomic switch" code path, which uses RCU when switching to atomic mode. The enqueued callback wakes up waiters waiting in the percpu_ref_switch_waitq. Allowing this callback to be lazy would result in unacceptable slowdowns for users of per-CPU refcounts, such as blk_pre_runtime_suspend(). Therefore, make __percpu_ref_switch_to_atomic() use call_rcu_flush() in order to revert to the old behavior. Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: --- lib/percpu-refcount.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/percpu-refcount.c b/lib/percpu-refcount.c index e5c5315da2741..65c58a029297d 100644 --- a/lib/percpu-refcount.c +++ b/lib/percpu-refcount.c @@ -230,7 +230,8 @@ static void __percpu_ref_switch_to_atomic(struct percpu_ref *ref, percpu_ref_noop_confirm_switch; percpu_ref_get(ref); /* put after confirmation */ - call_rcu(&ref->data->rcu, percpu_ref_switch_to_atomic_rcu); + call_rcu_flush(&ref->data->rcu, + percpu_ref_switch_to_atomic_rcu); } static void __percpu_ref_switch_to_percpu(struct percpu_ref *ref) From patchwork Tue Nov 22 01:04:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24088 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1932415wrr; Mon, 21 Nov 2022 17:11:08 -0800 (PST) X-Google-Smtp-Source: AA0mqf45+56ZtXGEU99DKihtocr2fUVcriAAwskvOzYTjxDyzlPj74ONwrHcG1eowoaTSDag5NOm X-Received: by 2002:aa7:d94a:0:b0:463:6b8f:f203 with SMTP id l10-20020aa7d94a000000b004636b8ff203mr18621683eds.236.1669079468009; Mon, 21 Nov 2022 17:11:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079468; cv=none; d=google.com; s=arc-20160816; b=KVvBnb5NZ+zJQr5jvpOciawrXN4htEWHOwnEewdpKJi1TZ+AnmpFwuz/tVdqhHWqQv mww0QICJpPp/Uoo5UPzkFGPZ2YIfyw3Jh8NpoxtBRQd+6aCMZPuo+mqHOzMCBQJGz7M+ O3vqcXPA0+oPaP19pdZQ6NNpkzMfqMlmng+hbbrlv9HKwm9V0rRhJoKR4TfIybCuA+Qr p2bLxhtyf33gHcJF21NuPPWbdzFU7nYiJ4aVNq/qcYiNbCuXFpoDSkH7UGjpgwYmkCbv fbQZiqhxd8VI9wjwXP1RPk5wZrnPHUSQGDyOSs8rwBWGCGc5V5PEqyM2zHHljLnkjNsZ 8Mag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XOZV7LmQxRu1aXrcZKBm/XqijZLMhzG5l+1tT1eLahs=; b=UaYeTRxWtO3GLr8vI5+buNyZBgE+YtM33H4mTdsJyxj/UHXJMNLQKOvPDuEIF1aa8h H15iqF+lXxsEe2YAclwddZn16eJZs9bw7jW6iseEjeHQZ5L2FcsoFQj8sFDnF6tVz/+x WpAkG/lO8W+AeArtzCxjKuce8TEs/NPEjYztBJSCLdElYMgNR6pKOL5NSXAN3mMce2gV 7KVGuheu7cWyvAgV17EojKVOGEW+EQ+kIwciGM2ASAmCrirRw6RvcnVSbbbto03cx9na PkOpwFUoCgZ6HV+Lqq74ZMo5KJWgt0rGFqm0G+uSAP7sJsv+k8wVqeQqJrYA/jgFIZWZ fivg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=CFD1MnAS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id dm15-20020a170907948f00b0077441079bfbsi11481215ejc.434.2022.11.21.17.10.44; Mon, 21 Nov 2022 17:11:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=CFD1MnAS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231864AbiKVBFG (ORCPT + 99 others); Mon, 21 Nov 2022 20:05:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231981AbiKVBE0 (ORCPT ); Mon, 21 Nov 2022 20:04:26 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08CB81E711; Mon, 21 Nov 2022 17:04:26 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id B0FB8B818E7; Tue, 22 Nov 2022 01:04:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1E6D6C43152; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079063; bh=uyDGWyzfPaiQ/+Iwibul18/2tA5ta5E2a9H+CSNE/yA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CFD1MnASm3nHVWtRNYrL1YNSkJ5NsCu6EMrtc9wT9RYHFupnOEhBIAelJJBkbzfGL seRGP7i+PbVg1o0P3F1yYAPd3lyPeCP89i7G3HCyh1Yv2OdZmtWtWY2D7lQ6m2ik6S tTy8OF9izkP9LALFdQy2sxTkMETa6y9JhV0xijdnV7JSV0nk1zGEAM4OispMp4zXi2 20ag4F1cNLMmHLWWwfLlY+W9hIoK+z4kDZR0I9gS0SoA9wxDdH6r7XvH6mtchSbpcD DKKtg8DLBSgLnq7hdM6bn6h1dGUIgGSqhWoppMo0DBYToKWLv5JVS5Z2JO1tePUUKi cwSGc+lBsYzHg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 814A35C1409; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , "Paul E . McKenney" Subject: [PATCH v2 rcu 09/16] rcu/sync: Use call_rcu_flush() instead of call_rcu Date: Mon, 21 Nov 2022 17:04:14 -0800 Message-Id: <20221122010421.3799681-9-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156672153007511?= X-GMAIL-MSGID: =?utf-8?q?1750156672153007511?= From: "Joel Fernandes (Google)" call_rcu() changes to save power will slow down rcu sync. Use the call_rcu_flush() API instead which reverts to the old behavior. Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/sync.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/sync.c b/kernel/rcu/sync.c index 5cefc702158fe..bdce3b5d7f714 100644 --- a/kernel/rcu/sync.c +++ b/kernel/rcu/sync.c @@ -44,7 +44,7 @@ static void rcu_sync_func(struct rcu_head *rhp); static void rcu_sync_call(struct rcu_sync *rsp) { - call_rcu(&rsp->cb_head, rcu_sync_func); + call_rcu_flush(&rsp->cb_head, rcu_sync_func); } /** From patchwork Tue Nov 22 01:04:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24086 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1930206wrr; Mon, 21 Nov 2022 17:05:44 -0800 (PST) X-Google-Smtp-Source: AA0mqf7vy/HQR60Pbei3h0lvQsXOe5Asss8wFOFmwWqUgzm30Bef8KY1rOJbUCyMzFpEa22n5nBv X-Received: by 2002:a65:4c85:0:b0:46f:59bd:6125 with SMTP id m5-20020a654c85000000b0046f59bd6125mr19994186pgt.147.1669079144111; Mon, 21 Nov 2022 17:05:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079144; cv=none; d=google.com; s=arc-20160816; b=ulAcsA2th/5hHmIt0d02E2+3MeP1/cnW9GuFl5QT0vRBaBuCbmJQs+LNPT3iG3EewN vV0aI3PacmnVv9qvdSpnJHP9lqjit8aJVDO9WnZEXRd4azRRaz9cBadBl2hNdWuzshXk oqUlL1wldDOifZ1xSoQ5WW/dsN9ED2C7px6ig3YyO1oTuBBqeQ3y3z0pFyGrXudyMMAU j0+ucL5JD42frUoFusqe1+iJO/uoYu1oWjtB1KOXMYpflUzk2sTBR46Vek217rmjaAE3 ev7dISBznRkWaNqUrR1Tyd5x59XdAohKOSrjFvwGzHXpH8VHykbrd0/3lWETxca7RG95 8/Vg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=NKvxC3Rb6nkSb5iYzdJMXZ5bqB8v/yuo/n1QZuBus+8=; b=f2zhGrWYmGzmdzF9eEb8oGQTi4NUgQna8P7Ix6TvLRpXFBdlc2JlyuqLhTRPtqE8eK A5Bj5sdOosy3xfuGauMVpADHyTVXNmLqxo1hGhWJwpG6vfChGlV4Ys0S7Eq+ywKBeEGA 8pfsSQojsTZI6q3vSy3CG6ALQkQVryoqUHFaod/yTYeGXtnd5pm0bq8YV8YbS4wo3GwR OWzqaj7q8O6b6rx8cPN/uK9s/FFGyN4Q7WpQ0WD4IxuiOke1DGMqw2jRVSHC5T8IWpDQ yLBk5FjAX07gr5mHI8Q/wvK0516pbT2QZrzGARvgoUArc1sHH9QY2FEOKoamEjW+/ugJ 4T1A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=RZFElDl2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b12-20020a170902d50c00b00187204b35besi14341038plg.203.2022.11.21.17.05.30; Mon, 21 Nov 2022 17:05:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=RZFElDl2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232161AbiKVBFD (ORCPT + 99 others); Mon, 21 Nov 2022 20:05:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44822 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231978AbiKVBE0 (ORCPT ); Mon, 21 Nov 2022 20:04:26 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08BA51E3C8; Mon, 21 Nov 2022 17:04:25 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id A2C31B818BC; Tue, 22 Nov 2022 01:04:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2792FC43153; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079063; bh=tEjic2DFB31t8ylkVSWil0ue1UwVOCBgRrj8WJMVg8E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RZFElDl2xa4aU5vs66KvPsfhJM1wHZrsTZBTpjiD4mvRVzWPd/qFRpvxjTHLDtbpR 81R58GX8DRLeUP4eWFKGcCWLVnO3hj4SovbJDEh92KpZlSlmDtI9pvZzAxG20JMlG9 Y2iuiCPzH+qhERyfj4mAo6wFBtjvY9uYJAPqRXqpovdimnt9NhP4ByYdTZWy4USGMC BgdSNXKfQ64/8o92FmI0Ep069lODyXBJqCbHSBnGp5hY2GUWFneAMvJVet4vivyAsQ FBZBvm4LvG4XrYDxOEC5lZbtq8JWMq4MXt78BrPxWo+G1xzax7oRkLe9zOLFvusB8M HrsI8JtVdB9oQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 837DC5C146A; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , "Paul E . McKenney" Subject: [PATCH v2 rcu 10/16] rcu/rcuscale: Use call_rcu_flush() for async reader test Date: Mon, 21 Nov 2022 17:04:15 -0800 Message-Id: <20221122010421.3799681-10-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156332917602573?= X-GMAIL-MSGID: =?utf-8?q?1750156332917602573?= From: "Joel Fernandes (Google)" rcuscale uses call_rcu() to queue async readers. With recent changes to save power, the test will have fewer async readers in flight. Use the call_rcu_flush() API instead to revert to the old behavior. Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/rcuscale.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index 3baded807a616..3b149e745a0f6 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -176,7 +176,7 @@ static struct rcu_scale_ops rcu_ops = { .get_gp_seq = rcu_get_gp_seq, .gp_diff = rcu_seq_diff, .exp_completed = rcu_exp_batches_completed, - .async = call_rcu, + .async = call_rcu_flush, .gp_barrier = rcu_barrier, .sync = synchronize_rcu, .exp_sync = synchronize_rcu_expedited, From patchwork Tue Nov 22 01:04:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24082 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1929953wrr; Mon, 21 Nov 2022 17:05:17 -0800 (PST) X-Google-Smtp-Source: AA0mqf6nMj6zJC7ZaZIVNCnHqTV5DySKX2mgNY6bX0KZWC1o5a1STCOZhN1Ogk3uaEA6pnkE3Ppd X-Received: by 2002:a63:711e:0:b0:477:5654:c37e with SMTP id m30-20020a63711e000000b004775654c37emr10145097pgc.206.1669079117628; Mon, 21 Nov 2022 17:05:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079117; cv=none; d=google.com; s=arc-20160816; b=ewq0YVWt0CZfABso0IOyXUn/Vgg9xn+EyF7ZRGFgsNLkq/HvzwbzLc/rWRiTZn5odH 0qOFQ4lyAT4gVMm9FT0UiCMNQBU4t4Wm1xu8Ip1JCTwjamWayp+WRAa9PfLJnEmAUCQR NLErNE2oEXmSpXMdHKJJGCaYp8HkX2vO8q8fe4bm5bNX/hFg7qulo2JF9XYOmXppLAxj y6SaCSUknB7xlm+4rZIgOWhmjnK9k8hoRZX1tfmFhmDZwGlBt/IsptdfwUrwBsYJ52VC yQUIqEBuN8r6IWHVjPziISYljEgtFU0cqRfRl6c62Znc3hlE7NRMvqVQ/pE7IreyUkEg ySnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=rrIm2tPhLjnx4rUE0Af77vY86wceGm13xfYULUrPKV4=; b=sq0VinMZKKGVm9Ra8yHoFvNvJge0VyfEivvpKmAYVLoHqP71RyxBYypLzHJw+3MVnB MFVCGHn+L+RrztGTB/buCtWthdiBZqubCgAe7cn5BUbd3R98TVM5VGk00p4q15d9eVyZ nYVmpFpnuYErAcMp8q58Xu41W3f9vJ2vDfOMsQg/VaCRCWvosYFM9DFb1NleQ2bssrt0 GGLcz/LWySlBf3cer3AcB41lAT+fAOzz1ShojEI//EDcPduO2uBKY7MXZZpK5csh1n7R tgZb9JzqLUKOBhd7z+g4zZKDSsxVILWWCJTzE+eHf7FS/KvfdyqzSp6ptNZBF6SvdL+v qRNA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=SR10hgGo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o4-20020a656a44000000b00459ba7ace44si14549544pgu.514.2022.11.21.17.05.04; Mon, 21 Nov 2022 17:05:17 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=SR10hgGo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232119AbiKVBEp (ORCPT + 99 others); Mon, 21 Nov 2022 20:04:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231936AbiKVBEZ (ORCPT ); Mon, 21 Nov 2022 20:04:25 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A09EF14003; Mon, 21 Nov 2022 17:04:24 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id F422D6151D; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2375CC4314D; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079063; bh=wpKgIqAJ7kcPSavM/RzKNoKYI274ueWKeTYgl/A5jf8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SR10hgGofc1YR6A1qP88bTh66K8mo+POYKhycLn8nv1e9j2wUMJGi/j4h9olXJtSO +GSGs3Np2ZXT/chkhEZHj/A/0PcoJuqEINDnaroz0crkrT2WwfAidzt5fZvdLINC+6 3/RRZWLv0aqi7y93XNk+taL/6N3cXf1qcsr5ZGx5a6F13ntxyrtbIQ4bJVsU3ssml9 8YJKEpjMrAHhLsqfMzuWgyZIwBlpbBkt82i+yST/YX1ITaCpRFf3u+XjHOWFkrC2gP VpC7q8Ud4lFFjyx8TE62gAO5eHv//IKpitC8MTh5xAEIohwxQnn1vlEuiD1Xq9ZZZZ 6084RRt3sF4Rg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8554A5C146C; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , "Paul E . McKenney" Subject: [PATCH v2 rcu 11/16] rcu/rcutorture: Use call_rcu_flush() where needed Date: Mon, 21 Nov 2022 17:04:16 -0800 Message-Id: <20221122010421.3799681-11-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156304960638871?= X-GMAIL-MSGID: =?utf-8?q?1750156304960638871?= From: "Joel Fernandes (Google)" call_rcu() changes to save power will change the behavior of rcutorture tests. Use the call_rcu_flush() API instead which reverts to the old behavior. Reported-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/rcutorture.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 503c2aa845a4a..c8ddb4b635b77 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -510,7 +510,7 @@ static unsigned long rcu_no_completed(void) static void rcu_torture_deferred_free(struct rcu_torture *p) { - call_rcu(&p->rtort_rcu, rcu_torture_cb); + call_rcu_flush(&p->rtort_rcu, rcu_torture_cb); } static void rcu_sync_torture_init(void) @@ -551,7 +551,7 @@ static struct rcu_torture_ops rcu_ops = { .start_gp_poll_exp_full = start_poll_synchronize_rcu_expedited_full, .poll_gp_state_exp = poll_state_synchronize_rcu, .cond_sync_exp = cond_synchronize_rcu_expedited, - .call = call_rcu, + .call = call_rcu_flush, .cb_barrier = rcu_barrier, .fqs = rcu_force_quiescent_state, .stats = NULL, @@ -848,7 +848,7 @@ static void rcu_tasks_torture_deferred_free(struct rcu_torture *p) static void synchronize_rcu_mult_test(void) { - synchronize_rcu_mult(call_rcu_tasks, call_rcu); + synchronize_rcu_mult(call_rcu_tasks, call_rcu_flush); } static struct rcu_torture_ops tasks_ops = { @@ -3388,13 +3388,13 @@ static void rcu_test_debug_objects(void) /* Try to queue the rh2 pair of callbacks for the same grace period. */ preempt_disable(); /* Prevent preemption from interrupting test. */ rcu_read_lock(); /* Make it impossible to finish a grace period. */ - call_rcu(&rh1, rcu_torture_leak_cb); /* Start grace period. */ + call_rcu_flush(&rh1, rcu_torture_leak_cb); /* Start grace period. */ local_irq_disable(); /* Make it harder to start a new grace period. */ - call_rcu(&rh2, rcu_torture_leak_cb); - call_rcu(&rh2, rcu_torture_err_cb); /* Duplicate callback. */ + call_rcu_flush(&rh2, rcu_torture_leak_cb); + call_rcu_flush(&rh2, rcu_torture_err_cb); /* Duplicate callback. */ if (rhp) { - call_rcu(rhp, rcu_torture_leak_cb); - call_rcu(rhp, rcu_torture_err_cb); /* Another duplicate callback. */ + call_rcu_flush(rhp, rcu_torture_leak_cb); + call_rcu_flush(rhp, rcu_torture_err_cb); /* Another duplicate callback. */ } local_irq_enable(); rcu_read_unlock(); From patchwork Tue Nov 22 01:04:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24084 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1930114wrr; Mon, 21 Nov 2022 17:05:32 -0800 (PST) X-Google-Smtp-Source: AA0mqf4A4eBG9mjYRQ0yLcUEHMjRS3iDJ7GyumktP6AYgKq38/RHnQ1l8Wg1A7aerPExlGI7TgMk X-Received: by 2002:a65:55cf:0:b0:474:d6fa:f568 with SMTP id k15-20020a6555cf000000b00474d6faf568mr1291300pgs.54.1669079132495; Mon, 21 Nov 2022 17:05:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079132; cv=none; d=google.com; s=arc-20160816; b=QI3ZrUmkr2OpilCmQQaonbbp1gwd+25TI0fcGOowP/7eJZCbp1JBIxOaLC5RuMjCFG h/Uppax7e7hWZSpBtNN0fepXOz3gEHbAQfhWJgHMR70Bov586xNkV/05++Xh4pkFH0bf /IE5SIVF+SNgN2/8waGAgoAOzqITOGSFiVqYcTU4f4J5kvP491JHtT2aHzzoCeeSKJDj 2p6Y7/VMxnZqoJcTPF7xBQYgjORzGMIO0I5eA75RBV62Z72J5LJQS0oK2LKjKGpoVBzl jZ039PMf+7cclsKSPmvnGeLqMo8VdVBn9DxiKL2szhqmz3NpvFuHjO3YJK1+HjrfgGdJ Cfkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5LUNVZHwNVt96+8P/miAQJBJGuT5Kk3S2AJ+wtaDntE=; b=enYC6oxb0v4lMXY8FCei4Kswhvi+M27M6PZIcBwI9koavh2Cvsl3VYaXv/FCADtkEv P0DDnl5xgVNlVD+sxPqZByX25Bgskp9IVgXl0aY+YDxaUS6AMb+RKbROcBbUztE9jhPj GgWJL5cAfEudZbqzh8OrpcHLQzppe4kkeytQkgIOXCYzOTNRYp+RXjRd2m/b5ZGsd+dG +bALoDlcNgf/tyiOovcbslYnjEsPCJpuyXbADUzP+TxDvKpqWZDdZAJphg8gsP7rZn/p sYy8RQIoRJ1Nr1BmXzykP9OoBP8zrjU/iTLYjtbrCTBaljj2so6/XRuntU9vCRs/Ak9S Y1TA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=PK1gNZVJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g14-20020a65580e000000b004771126e2casi12564586pgr.265.2022.11.21.17.05.15; Mon, 21 Nov 2022 17:05:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=PK1gNZVJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232140AbiKVBE4 (ORCPT + 99 others); Mon, 21 Nov 2022 20:04:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231948AbiKVBEZ (ORCPT ); Mon, 21 Nov 2022 20:04:25 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4AD615FF5; Mon, 21 Nov 2022 17:04:24 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1301E6151A; Tue, 22 Nov 2022 01:04:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2F617C43155; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079063; bh=wpD5zzzo6w7p3WOWI5WdmBr7p4BboEaBNwhD+AKKK+A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PK1gNZVJ0kcEPvGy8l9MHsEaFYI4887MGYxjwddokCvV+5CR0pd79fl0p1glOCGRn 0UZ933OfAr+4Gq4dPyQQ4ggYYy7ccvWAq7u3GjOOQ7+zHHDtLvQye8pH8M+W0mlL9y sPKNHqGeUQX6wHnzugWH1EkLj7EBiasEKdjqPfB+3Mn6fRhazpJViEPEdVXvGMH8Mp 8V3kgK6niSFEXIPG0ggFZbhKEYuigziC8Lr+NN51KI8fNT4wMU7AyIBoB+ybCdNHHm eL1fFHymyVCC6ufBOz1XhfkVZUFfGDaG9XQKwLeKE7dV3bVxproQTa3yKrUxKkYe2e wEDKHMa+wadsw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8788A5C1523; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Uladzislau Rezki , Joel Fernandes , "James E.J. Bottomley" , "Martin K. Petersen" , linux-scsi@vger.kernel.org, "Paul E . McKenney" Subject: [PATCH v2 rcu 12/16] scsi/scsi_error: Use call_rcu_flush() instead of call_rcu() Date: Mon, 21 Nov 2022 17:04:17 -0800 Message-Id: <20221122010421.3799681-12-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156320553015342?= X-GMAIL-MSGID: =?utf-8?q?1750156320553015342?= From: Uladzislau Rezki Earlier commits in this series allow battery-powered systems to build their kernels with the default-disabled CONFIG_RCU_LAZY=y Kconfig option. This Kconfig option causes call_rcu() to delay its callbacks in order to batch them. This means that a given RCU grace period covers more callbacks, thus reducing the number of grace periods, in turn reducing the amount of energy consumed, which increases battery lifetime which can be a very good thing. This is not a subtle effect: In some important use cases, the battery lifetime is increased by more than 10%. This CONFIG_RCU_LAZY=y option is available only for CPUs that offload callbacks, for example, CPUs mentioned in the rcu_nocbs kernel boot parameter passed to kernels built with CONFIG_RCU_NOCB_CPU=y. Delaying callbacks is normally not a problem because most callbacks do nothing but free memory. If the system is short on memory, a shrinker will kick all currently queued lazy callbacks out of their laziness, thus freeing their memory in short order. Similarly, the rcu_barrier() function, which blocks until all currently queued callbacks are invoked, will also kick lazy callbacks, thus enabling rcu_barrier() to complete in a timely manner. However, there are some cases where laziness is not a good option. For example, synchronize_rcu() invokes call_rcu(), and blocks until the newly queued callback is invoked. It would not be a good for synchronize_rcu() to block for ten seconds, even on an idle system. Therefore, synchronize_rcu() invokes call_rcu_flush() instead of call_rcu(). The arrival of a non-lazy call_rcu_flush() callback on a given CPU kicks any lazy callbacks that might be already queued on that CPU. After all, if there is going to be a grace period, all callbacks might as well get full benefit from it. Yes, this could be done the other way around by creating a call_rcu_lazy(), but earlier experience with this approach and feedback at the 2022 Linux Plumbers Conference shifted the approach to call_rcu() being lazy with call_rcu_flush() for the few places where laziness is inappropriate. And another call_rcu() instance that cannot be lazy is the one in the scsi_eh_scmd_add() function. Leaving this instance lazy results in unacceptably slow boot times. Therefore, make scsi_eh_scmd_add() use call_rcu_flush() in order to revert to the old behavior. Tested-by: Joel Fernandes (Google) Signed-off-by: Uladzislau Rezki Signed-off-by: Joel Fernandes (Google) Cc: "James E.J. Bottomley" Cc: "Martin K. Petersen" Cc: Signed-off-by: Paul E. McKenney Reviewed-by: Bart Van Assche Acked-by: Martin K. Petersen --- drivers/scsi/scsi_error.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c index 6995c89792300..634672e67c81f 100644 --- a/drivers/scsi/scsi_error.c +++ b/drivers/scsi/scsi_error.c @@ -312,7 +312,7 @@ void scsi_eh_scmd_add(struct scsi_cmnd *scmd) * Ensure that all tasks observe the host state change before the * host_failed change. */ - call_rcu(&scmd->rcu, scsi_eh_inc_host_failed); + call_rcu_flush(&scmd->rcu, scsi_eh_inc_host_failed); } /** From patchwork Tue Nov 22 01:04:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24083 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1930009wrr; Mon, 21 Nov 2022 17:05:23 -0800 (PST) X-Google-Smtp-Source: AA0mqf7qOudaT/7KSB2oKNmqp5Jb7naCfBSEoPqJNAmeVSNSjxlvzjPElxGqAaZsvqXIWjHoWFF7 X-Received: by 2002:a63:4e0f:0:b0:476:91d6:e15e with SMTP id c15-20020a634e0f000000b0047691d6e15emr1231661pgb.455.1669079122991; Mon, 21 Nov 2022 17:05:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079122; cv=none; d=google.com; s=arc-20160816; b=wy3QngXwRq64ZK3RiPxg349oE/X9tkTO+gjz2pC9KrzaT8Jx91Lbrfo8F+GHgRm4I9 WRTuc4eeinhYlwM5AcmOZLUpVrOPXUjr8N1D8KKX29c1zNdsPFGJsddcYEgynGflMaf6 JAnWbI6HKhf2DTtgteGn9hNdSarBGnRSx1e8k+LPqFRg9/wS6Alv2vklM7ShKv5s72sV OUch/d4kjlMJm6RUUgnsGknyfDBeqaRw9vKrxU2Gj/W+VL6faEM+/Lk2X0qfoeBR6gK6 7UMm1AmWSkF/MeewceKcPtPCa96FnWmE93mZQu/YEYlUCqgwqbz8yScWOqabCAMLcY1Z rVJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5KvntsJrcIL0LlcK4loZGDfdWOzPQp8MEDfqtIbnV0s=; b=RrCFdeKHmlhhFkC6R3otVTSXtdQwwGZHU5Y+wfq5KxnuqoCaLNPA8qeU1LMdRXtuE8 QUInZsWJ1Q8hZ64J6AqoRxRcaJ2J3/hO/GtZa52P5Pe/QrPVdmB9+4bIrw7iQUIlXevQ 3rAKjmjzlP/Grkw6KumW++FkPoQkjdzC2ALAW5xhMQzGXFZRYv+n7TGXbursGjZTOLVA 5hUGRXn7PsaVgZ8MtsTpV59EHV1HZmZf71Z8w+j9zwpER4MbIQNxLlDbxL2mS/Vgt9Yp 9qdlN39RBIQusJLK/EtswL2Rndm/pxppidi8NqTh3zSKjUJA7RyxltJgdDJSYMyjE++g uMnw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ERo9qt7u; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k20-20020a056a00169400b005669b9f5e1bsi936281pfc.45.2022.11.21.17.05.09; Mon, 21 Nov 2022 17:05:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ERo9qt7u; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232134AbiKVBEv (ORCPT + 99 others); Mon, 21 Nov 2022 20:04:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44780 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231946AbiKVBEZ (ORCPT ); Mon, 21 Nov 2022 20:04:25 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D3F7C11A12; Mon, 21 Nov 2022 17:04:24 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 13F7F6151F; Tue, 22 Nov 2022 01:04:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3228FC43154; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079063; bh=JDkX7Fc0DOy+yBhbkuqH9S9Ejfgbk7JA8huLd1R6z8Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ERo9qt7uO7GVBBnrX8o2yXKM/JykcLBdKAbt/RQK4nUKXF1KXqh6rH97v9r5k9ZXA WumeDeAgQ6u2DwTKhLbgEUQi9W/b1FOGQkAEq75C51k0vVc6/BAUAIEWz/HBMK4vI5 l0puY9/6h3ocIVuzk2IqMT5KJCe+V61HRQok8UROZEnccnPyNxpaauTlmY/L4CEmXY 9hfAa7Osg5YjfWoq7Y0ORylTq6OuP+ZtK1SsgJyIGpu/HQWPMGq/ce3G253xKj4Ctz NVDwNSK7+OAzWB7T5QrsS8ecsRwC2aSLbLC1ByBPcM7Uiq7FBYS8zMeUuF27bmV1bV iYx8FNDVQU4bw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8954E5C155D; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Uladzislau Rezki , Joel Fernandes , Tejun Heo , Lai Jiangshan , "Paul E . McKenney" Subject: [PATCH v2 rcu 13/16] workqueue: Make queue_rcu_work() use call_rcu_flush() Date: Mon, 21 Nov 2022 17:04:18 -0800 Message-Id: <20221122010421.3799681-13-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156310750832834?= X-GMAIL-MSGID: =?utf-8?q?1750156310750832834?= From: Uladzislau Rezki Earlier commits in this series allow battery-powered systems to build their kernels with the default-disabled CONFIG_RCU_LAZY=y Kconfig option. This Kconfig option causes call_rcu() to delay its callbacks in order to batch them. This means that a given RCU grace period covers more callbacks, thus reducing the number of grace periods, in turn reducing the amount of energy consumed, which increases battery lifetime which can be a very good thing. This is not a subtle effect: In some important use cases, the battery lifetime is increased by more than 10%. This CONFIG_RCU_LAZY=y option is available only for CPUs that offload callbacks, for example, CPUs mentioned in the rcu_nocbs kernel boot parameter passed to kernels built with CONFIG_RCU_NOCB_CPU=y. Delaying callbacks is normally not a problem because most callbacks do nothing but free memory. If the system is short on memory, a shrinker will kick all currently queued lazy callbacks out of their laziness, thus freeing their memory in short order. Similarly, the rcu_barrier() function, which blocks until all currently queued callbacks are invoked, will also kick lazy callbacks, thus enabling rcu_barrier() to complete in a timely manner. However, there are some cases where laziness is not a good option. For example, synchronize_rcu() invokes call_rcu(), and blocks until the newly queued callback is invoked. It would not be a good for synchronize_rcu() to block for ten seconds, even on an idle system. Therefore, synchronize_rcu() invokes call_rcu_flush() instead of call_rcu(). The arrival of a non-lazy call_rcu_flush() callback on a given CPU kicks any lazy callbacks that might be already queued on that CPU. After all, if there is going to be a grace period, all callbacks might as well get full benefit from it. Yes, this could be done the other way around by creating a call_rcu_lazy(), but earlier experience with this approach and feedback at the 2022 Linux Plumbers Conference shifted the approach to call_rcu() being lazy with call_rcu_flush() for the few places where laziness is inappropriate. And another call_rcu() instance that cannot be lazy is the one in queue_rcu_work(), given that callers to queue_rcu_work() are not necessarily OK with long delays. Therefore, make queue_rcu_work() use call_rcu_flush() in order to revert to the old behavior. Signed-off-by: Uladzislau Rezki Signed-off-by: Joel Fernandes (Google) Cc: Tejun Heo Cc: Lai Jiangshan Signed-off-by: Paul E. McKenney --- kernel/workqueue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 7cd5f5e7e0a1b..b4b0e828b529e 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -1771,7 +1771,7 @@ bool queue_rcu_work(struct workqueue_struct *wq, struct rcu_work *rwork) if (!test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work))) { rwork->wq = wq; - call_rcu(&rwork->rcu, rcu_work_rcufn); + call_rcu_flush(&rwork->rcu, rcu_work_rcufn); return true; } From patchwork Tue Nov 22 01:04:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24085 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1930146wrr; Mon, 21 Nov 2022 17:05:36 -0800 (PST) X-Google-Smtp-Source: AA0mqf5PlU9HCC4ChTJE76Iq+Bvs4xP5YcACcnb8N29KTMWsnTP6+2IKbFzzakAB389zW/MpojCF X-Received: by 2002:a63:510:0:b0:470:537b:c39a with SMTP id 16-20020a630510000000b00470537bc39amr1881255pgf.614.1669079136329; Mon, 21 Nov 2022 17:05:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079136; cv=none; d=google.com; s=arc-20160816; b=KHak+zziHaB6sPTv9Y1WiKgbhuFdRtNzm4hM9JQJefNnsCNWPA2LSLtEORueaAvpRQ G9bTS2J2+R9ZyaGipOnwkLZ9QM3EFzyDdk5P6FODxGGRSwyfs1IqLz1qy8SN6I2kDsoP cgdsSYR5hfB1z3BLmJVKlIR/8w7C5hRoLS+4vd8FBYaJcJR9vuch9ty5ffPkl4JMJe2n XlCSiBaG0EOMCx15Ja3NaLF+cD/UoGUoV7v+GSK8kVgIZNIds3B0/AGCGROV6MlInsOm AzoLHCfOOG4qp62viObGDzVKqweJbeQXf3vYTiKXo/vNIz2o827LxPrkS5YgmRrf/E6V tlXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=6x7QQ4egTRAEeupnvkCpFQbLM5Qn1z5Y22rsc8OvGxY=; b=B1xla1tiusuLfV4BLuy2fB7m3RglvULVdXQJUes97mKHaNoWN9GsLcKD04+KguJ5RV AwNiHszQa4omFuPGNHRtmo9rlipdc3Tp/x8rMUishZbnTRpSPDB0AkjKA4Gs+7a7DAe/ FE1tlUJLA/VgZgp6TAIye4jjqc9ez2jUtyFSug4/ju+ODR0R0AYs7unipMTluRi/kvO6 j8wsTxcUrL9yjzzl5uxsvZHFMCXTUdRPL+bJb304JpOQfFP1/ZPCaaF3eeSlfgM6CeX1 reeusdQyfXg2lNZ0lBFfPJ6qKmXUCk452j32qlaB6bpn0KWjbnRY805VcuPqFAmTI/jB qoCQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=fjaKQJ3e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bt26-20020a056a00439a00b005370942e40dsi11906934pfb.324.2022.11.21.17.05.22; Mon, 21 Nov 2022 17:05:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=fjaKQJ3e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232152AbiKVBE7 (ORCPT + 99 others); Mon, 21 Nov 2022 20:04:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231956AbiKVBE0 (ORCPT ); Mon, 21 Nov 2022 20:04:26 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C42C1C901; Mon, 21 Nov 2022 17:04:25 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 75F0061511; Tue, 22 Nov 2022 01:04:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 33D5FC43156; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079063; bh=QNpGjRTwvCwUOFj/pR6pze23etHEHGqQMnnOZBmIHF0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fjaKQJ3eSDwRn22hicp3w41Zl2GWI3YjHGmDWDJVyAPiyxrZOPD78A4UgUF/RxlKB mXGO2rgS6XfFXotIqbsRqz4o+pd1q/Hs692e4No5+HTCd0nOByQWtv9EHTTE1hc24X CFJQp86yxs1lzaKlkx+DuntsELItJ0tfxyeN6coVE+eJQ+MD+thR7MIPPb4Ie090Xp UZm6mR87a4Top0HJCaCJNNz08fh8MG6AgBOxWL2doq4PriTEJZZbwiURgI4Xy6Flqh qn4dfosyXjLIxsH4W3GBjX+5zNJCur7P2JuBr9Pit8CDSkARL0mDq5+BiumQOmTzzD nUWm6MnXK4CZA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8B2805C1695; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , David Howells , Marc Dionne , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , linux-afs@lists.infradead.org, netdev@vger.kernel.org, "Paul E . McKenney" Subject: [PATCH v2 rcu 14/16] rxrpc: Use call_rcu_flush() instead of call_rcu() Date: Mon, 21 Nov 2022 17:04:19 -0800 Message-Id: <20221122010421.3799681-14-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156324710916857?= X-GMAIL-MSGID: =?utf-8?q?1750156324710916857?= From: "Joel Fernandes (Google)" Earlier commits in this series allow battery-powered systems to build their kernels with the default-disabled CONFIG_RCU_LAZY=y Kconfig option. This Kconfig option causes call_rcu() to delay its callbacks in order to batch them. This means that a given RCU grace period covers more callbacks, thus reducing the number of grace periods, in turn reducing the amount of energy consumed, which increases battery lifetime which can be a very good thing. This is not a subtle effect: In some important use cases, the battery lifetime is increased by more than 10%. This CONFIG_RCU_LAZY=y option is available only for CPUs that offload callbacks, for example, CPUs mentioned in the rcu_nocbs kernel boot parameter passed to kernels built with CONFIG_RCU_NOCB_CPU=y. Delaying callbacks is normally not a problem because most callbacks do nothing but free memory. If the system is short on memory, a shrinker will kick all currently queued lazy callbacks out of their laziness, thus freeing their memory in short order. Similarly, the rcu_barrier() function, which blocks until all currently queued callbacks are invoked, will also kick lazy callbacks, thus enabling rcu_barrier() to complete in a timely manner. However, there are some cases where laziness is not a good option. For example, synchronize_rcu() invokes call_rcu(), and blocks until the newly queued callback is invoked. It would not be a good for synchronize_rcu() to block for ten seconds, even on an idle system. Therefore, synchronize_rcu() invokes call_rcu_flush() instead of call_rcu(). The arrival of a non-lazy call_rcu_flush() callback on a given CPU kicks any lazy callbacks that might be already queued on that CPU. After all, if there is going to be a grace period, all callbacks might as well get full benefit from it. Yes, this could be done the other way around by creating a call_rcu_lazy(), but earlier experience with this approach and feedback at the 2022 Linux Plumbers Conference shifted the approach to call_rcu() being lazy with call_rcu_flush() for the few places where laziness is inappropriate. And another call_rcu() instance that cannot be lazy is the one in rxrpc_kill_connection(), which sometimes does a wakeup that should not be unduly delayed. Therefore, make rxrpc_kill_connection() use call_rcu_flush() in order to revert to the old behavior. Signed-off-by: Joel Fernandes (Google) Cc: David Howells Cc: Marc Dionne Cc: "David S. Miller" Cc: Eric Dumazet Cc: Jakub Kicinski Cc: Paolo Abeni Cc: Cc: Signed-off-by: Paul E. McKenney --- net/rxrpc/conn_object.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/rxrpc/conn_object.c b/net/rxrpc/conn_object.c index 22089e37e97f0..fdcfb509cc443 100644 --- a/net/rxrpc/conn_object.c +++ b/net/rxrpc/conn_object.c @@ -253,7 +253,7 @@ void rxrpc_kill_connection(struct rxrpc_connection *conn) * must carry a ref on the connection to prevent us getting here whilst * it is queued or running. */ - call_rcu(&conn->rcu, rxrpc_destroy_connection); + call_rcu_flush(&conn->rcu, rxrpc_destroy_connection); } /* From patchwork Tue Nov 22 01:04:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24091 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1934518wrr; Mon, 21 Nov 2022 17:16:29 -0800 (PST) X-Google-Smtp-Source: AA0mqf52SqVO9fPWt4GzeaWrq11OUjQiQ3pWtj+5shGdDXP3y3nZLXWk+5uUfwU452RmV8apQK8R X-Received: by 2002:aa7:9508:0:b0:56d:3180:e885 with SMTP id b8-20020aa79508000000b0056d3180e885mr6321104pfp.82.1669079788814; Mon, 21 Nov 2022 17:16:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079788; cv=none; d=google.com; s=arc-20160816; b=tzB0vZeQMVERARPILy8psxymSuXexN1lu1rQTwf7fJF0yRZo36X2214SvRe1SXmax6 9WUKhaV4geh7WnVKkOMTD7i23BYg9NLG4c1DKgkR4i8uWvr375tSejBFYOdm7OwKhGk8 T6oJnYnMHhZy0sP+x77VLzxadZ0GiC6YXQs8H2y9Kw0k1+UNDaJ1JZ7M4QALWW5Wry28 HoABj+R+n62IPcFARUrsHWMg027HFvx28M2cEkM3Up6XV0lo6Tvw6H83H7thWIzwYXVk K2qjf6SDjKt8MieMM45PXNuxWHXRFtRtzj0l7I1qRwXY5aQ/LPi8PkWFKj6xM2onXAlJ 0aNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XSj//TueDO+zTWpg5mKEZG+g/2SiLHDRLvn9bGE03Zo=; b=iYUd7DvdS3CXkKbnzrD3sMSjdJ8cm7UV0wjx9vfHxGRQpTocoVDIZ61JfZNoFMTHbV w4piH7IcLOSpZQWTuuqNEdJKo2P3cEM9XSdxtAEqN0O3PMLXzrBcRAsllLcgY0AguP2O 0MzCvjgvZgUIVG/tQ3TJeT4mCuxHW+IsOLMqUh0Oye2U9jkkC02dpuVLRkNX1+r8zGqQ OtHZQrEzDRqfl1rsOT9E2qDPsMeO7A+GRRhlIF1q+/rBYZwgAENQ/RiqEjoVYFkPNmSN ghCvO3Wo3ehgkYkjV/4m4G9WgvDSPeWKWBlgZRwDi+OI8mASENqRi3W+WU0kF2Le/Aq3 VpyA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=oj2o0MV7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m3-20020a170902db0300b001872cd80411si14130573plx.193.2022.11.21.17.16.15; Mon, 21 Nov 2022 17:16:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=oj2o0MV7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232106AbiKVBF3 (ORCPT + 99 others); Mon, 21 Nov 2022 20:05:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44860 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232004AbiKVBE1 (ORCPT ); Mon, 21 Nov 2022 20:04:27 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E04911A12; Mon, 21 Nov 2022 17:04:26 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 2B273B818DF; Tue, 22 Nov 2022 01:04:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3BA9BC4315E; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079063; bh=82QgUkdn96zAlaY+UFpNNtcTMhQgVxk6sdFaelyo3zA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oj2o0MV7bWppKYphpTZpnE345/Ekq1qeCL51DEXRnLTr9ODMUfEY6Po0mW1vDwE0N ZKt1Nr856cCvvre9rHNL7oio8FbnDnddN3xQDc/PpWXjmQUzB9Bbo4807N7jvKa+AX 87o4Tj/mVIAeC27B9POj/7hSsfsa5wE9QYk4sIycuuSeBgVhHY6Yc/Graw+BLBgdYe o8jkA8DFvGrnVI0c29rXdES4mjK6TiEOu0d3mqMORsQQgZ4cMEjqKNZT8H6/+6MXxU x4aRyTkMoQsvQgE7SOCx0yMQZ8zK5oriDMDd+lM5jCpeuQ8ZGIzm6AtDJsyBNP8HFk QSv+x/5jDzJLw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8CE7F5C18FF; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , David Ahern , "David S. Miller" , Eric Dumazet , Hideaki YOSHIFUJI , Jakub Kicinski , Paolo Abeni , netdev@vger.kernel.org, "Paul E . McKenney" Subject: [PATCH v2 rcu 15/16] net: Use call_rcu_flush() for dst_release() Date: Mon, 21 Nov 2022 17:04:20 -0800 Message-Id: <20221122010421.3799681-15-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750157008529652473?= X-GMAIL-MSGID: =?utf-8?q?1750157008529652473?= From: "Joel Fernandes (Google)" In a networking test on ChromeOS, kernels built with the new CONFIG_RCU_LAZY=y Kconfig option fail a networking test in the teardown phase. This failure may be reproduced as follows: ip netns del The CONFIG_RCU_LAZY=y Kconfig option was introduced by earlier commits in this series for the benefit of certain battery-powered systems. This Kconfig option causes call_rcu() to delay its callbacks in order to batch them. This means that a given RCU grace period covers more callbacks, thus reducing the number of grace periods, in turn reducing the amount of energy consumed, which increases battery lifetime which can be a very good thing. This is not a subtle effect: In some important use cases, the battery lifetime is increased by more than 10%. This CONFIG_RCU_LAZY=y option is available only for CPUs that offload callbacks, for example, CPUs mentioned in the rcu_nocbs kernel boot parameter passed to kernels built with CONFIG_RCU_NOCB_CPU=y. Delaying callbacks is normally not a problem because most callbacks do nothing but free memory. If the system is short on memory, a shrinker will kick all currently queued lazy callbacks out of their laziness, thus freeing their memory in short order. Similarly, the rcu_barrier() function, which blocks until all currently queued callbacks are invoked, will also kick lazy callbacks, thus enabling rcu_barrier() to complete in a timely manner. However, there are some cases where laziness is not a good option. For example, synchronize_rcu() invokes call_rcu(), and blocks until the newly queued callback is invoked. It would not be a good for synchronize_rcu() to block for ten seconds, even on an idle system. Therefore, synchronize_rcu() invokes call_rcu_flush() instead of call_rcu(). The arrival of a non-lazy call_rcu_flush() callback on a given CPU kicks any lazy callbacks that might be already queued on that CPU. After all, if there is going to be a grace period, all callbacks might as well get full benefit from it. Yes, this could be done the other way around by creating a call_rcu_lazy(), but earlier experience with this approach and feedback at the 2022 Linux Plumbers Conference shifted the approach to call_rcu() being lazy with call_rcu_flush() for the few places where laziness is inappropriate. Returning to the test failure, use of ftrace showed that this failure cause caused by the aadded delays due to this new lazy behavior of call_rcu() in kernels built with CONFIG_RCU_LAZY=y. Therefore, make dst_release() use call_rcu_flush() in order to revert to the old test-failure-free behavior. Signed-off-by: Joel Fernandes (Google) Cc: David Ahern Cc: "David S. Miller" Cc: Eric Dumazet Cc: Hideaki YOSHIFUJI Cc: Jakub Kicinski Cc: Paolo Abeni Cc: Signed-off-by: Paul E. McKenney --- net/core/dst.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/core/dst.c b/net/core/dst.c index bc9c9be4e0801..15b16322703f4 100644 --- a/net/core/dst.c +++ b/net/core/dst.c @@ -174,7 +174,7 @@ void dst_release(struct dst_entry *dst) net_warn_ratelimited("%s: dst:%p refcnt:%d\n", __func__, dst, newrefcnt); if (!newrefcnt) - call_rcu(&dst->rcu_head, dst_destroy_rcu); + call_rcu_flush(&dst->rcu_head, dst_destroy_rcu); } } EXPORT_SYMBOL(dst_release); From patchwork Tue Nov 22 01:04:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24090 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1934457wrr; Mon, 21 Nov 2022 17:16:18 -0800 (PST) X-Google-Smtp-Source: AA0mqf5bOaliXpMBahGzAfIOVSx4vtZCm7fpE2jm00VmNtWXvsmdhNj7W61PukkLMsi2CDvlYcI0 X-Received: by 2002:a62:7b57:0:b0:56b:dd28:4f6e with SMTP id w84-20020a627b57000000b0056bdd284f6emr5326466pfc.0.1669079778047; Mon, 21 Nov 2022 17:16:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079778; cv=none; d=google.com; s=arc-20160816; b=EJ5zZ2VTEqDWeCCBK1sxqLVd2V+eLqxdrT4zjDigtox4apNjVfp0x3YDRRa9e2ZOv3 GAfDNKqndlxnk0aCKsPCapxBxonFlkcFxXYaYA4N7V96UyH9dldp1Auv2F1EREA0W6gG 9tbxIz3jKRUiwGKjFjg4gCo/x43mCQDds1UODsMJW4G+kdDPalQneyGtybGCykSTMwHv iOCf5yOphgyEHl2WkfbGFhCc/8M/qcLA7E3GhDUTEfwvmqydh0dQbtBx3YTfN9blQ93e j70TpRf2CCSNunD21QE7PXkw06lMqrzouLk46BEx2YYFLah0JakPL2hsIt57WNYR9mSg GLUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Lv/uLvZfOF0iRiluE4xXFTPcAzql9p11wASsMjjHz2Y=; b=OQYBSRKorUxIo21b/oc3GsszAIEKOqSHdoGNibm+O3nMsByNgMgu9RrKIo9MmbSqgV R82fa9/JJxmpI2363U/hBFZh/0JlwRSq7y6d/Rw7SIV47Hsmd2O0uHqzQIQlviHMmH4i wouiaa6MjAJXpN5AgKyjN/vcuA6QEibtNHpPPB3gHvmO6dT+9v7p79+FKpA99sqDO3FS LTskupt1qQ1HOVQxfPhr3iXFjvaljndBfWUq9hcG9Alpli834MVha+RhzzRMLl7A+B6e OeCnsRNEwSdKBF6rCw+dRhq60KIqe3gMOJowiuWKd49ab6A/XoXXvA7fra7SR9WEfaGj VlcA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=TTez867U; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ob6-20020a17090b390600b0021111f5231bsi3026821pjb.22.2022.11.21.17.16.03; Mon, 21 Nov 2022 17:16:18 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=TTez867U; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232191AbiKVBFO (ORCPT + 99 others); Mon, 21 Nov 2022 20:05:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44808 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231954AbiKVBE0 (ORCPT ); Mon, 21 Nov 2022 20:04:26 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C2C1DF40; Mon, 21 Nov 2022 17:04:25 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2A4D06151E; Tue, 22 Nov 2022 01:04:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 75FB9C43166; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079063; bh=yoNO14jnT7/3wJpWHWo0CYgw8+T0b3nI0VWmVo/0VZk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TTez867UG51Rj10O9MBv8/3v+xkdrmwVzCDqX9ISWk5WFBZYKeDXeOODA9souyOXQ 53WH/uy/84JsK2tNKVG2Jvi0x8VhcDvWhBLP0mZQqp6x2bZJqPl3z4afQjFW7ETIX+ OpceptNxp1GZpjnzC6xWRWl5nqZDjSRBvAarT3P2OOH6PRYJOzrB6Cf9iXiQGD1ZKU zr7ICASGaotS6FZUO9zEcuJPFKFuknV5yCsy84GE3hS3YoH3Yjwn9N9Tqvdy3CCXYz O2lfCRltmaMbxQLYhotx/3UEQo1EP97pAcoNAmWX2AWBVjzrcYz67tww5M5f1CF82i m1p7zPsS+2dhA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8EC005C1C98; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Eric Dumazet , Joel Fernandes , David Ahern , "David S. Miller" , Hideaki YOSHIFUJI , Jakub Kicinski , Paolo Abeni , netdev@vger.kernel.org, "Paul E . McKenney" Subject: [PATCH v2 rcu 16/16] net: devinet: Reduce refcount before grace period Date: Mon, 21 Nov 2022 17:04:21 -0800 Message-Id: <20221122010421.3799681-16-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156997136342786?= X-GMAIL-MSGID: =?utf-8?q?1750156997136342786?= From: Eric Dumazet Currently, the inetdev_destroy() function waits for an RCU grace period before decrementing the refcount and freeing memory. This causes a delay with a new RCU configuration that tries to save power, which results in the network interface disappearing later than expected. The resulting delay causes test failures on ChromeOS. Refactor the code such that the refcount is freed before the grace period and memory is freed after. With this a ChromeOS network test passes that does 'ip netns del' and polls for an interface disappearing, now passes. Reported-by: Joel Fernandes (Google) Signed-off-by: Eric Dumazet Signed-off-by: Joel Fernandes (Google) Cc: David Ahern Cc: "David S. Miller" Cc: Hideaki YOSHIFUJI Cc: Jakub Kicinski Cc: Paolo Abeni Cc: Signed-off-by: Paul E. McKenney --- net/ipv4/devinet.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/net/ipv4/devinet.c b/net/ipv4/devinet.c index e8b9a9202fecd..b0acf6e19aed3 100644 --- a/net/ipv4/devinet.c +++ b/net/ipv4/devinet.c @@ -234,13 +234,20 @@ static void inet_free_ifa(struct in_ifaddr *ifa) call_rcu(&ifa->rcu_head, inet_rcu_free_ifa); } +static void in_dev_free_rcu(struct rcu_head *head) +{ + struct in_device *idev = container_of(head, struct in_device, rcu_head); + + kfree(rcu_dereference_protected(idev->mc_hash, 1)); + kfree(idev); +} + void in_dev_finish_destroy(struct in_device *idev) { struct net_device *dev = idev->dev; WARN_ON(idev->ifa_list); WARN_ON(idev->mc_list); - kfree(rcu_dereference_protected(idev->mc_hash, 1)); #ifdef NET_REFCNT_DEBUG pr_debug("%s: %p=%s\n", __func__, idev, dev ? dev->name : "NIL"); #endif @@ -248,7 +255,7 @@ void in_dev_finish_destroy(struct in_device *idev) if (!idev->dead) pr_err("Freeing alive in_device %p\n", idev); else - kfree(idev); + call_rcu(&idev->rcu_head, in_dev_free_rcu); } EXPORT_SYMBOL(in_dev_finish_destroy); @@ -298,12 +305,6 @@ static struct in_device *inetdev_init(struct net_device *dev) goto out; } -static void in_dev_rcu_put(struct rcu_head *head) -{ - struct in_device *idev = container_of(head, struct in_device, rcu_head); - in_dev_put(idev); -} - static void inetdev_destroy(struct in_device *in_dev) { struct net_device *dev; @@ -328,7 +329,7 @@ static void inetdev_destroy(struct in_device *in_dev) neigh_parms_release(&arp_tbl, in_dev->arp_parms); arp_ifdown(dev); - call_rcu(&in_dev->rcu_head, in_dev_rcu_put); + in_dev_put(in_dev); } int inet_addr_onlink(struct in_device *in_dev, __be32 a, __be32 b)