Message ID | 20230329160203.191380-5-frederic@kernel.org |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp537621vqo; Wed, 29 Mar 2023 09:28:04 -0700 (PDT) X-Google-Smtp-Source: AKy350YhsEzzHXsSNy38Yy+lxKSVXzSPrzMUqlCsacekpu4cj/HEidesvqYDJ0jVInJkWib9jSoL X-Received: by 2002:a17:90b:3a85:b0:23d:e0e8:f453 with SMTP id om5-20020a17090b3a8500b0023de0e8f453mr20434324pjb.38.1680107283940; Wed, 29 Mar 2023 09:28:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680107283; cv=none; d=google.com; s=arc-20160816; b=GdqTGDEIEQRMttN7eVfxdKsZ2U+NL8LJXDCGNLyFOFWzEOT4ey4gM7sXp45H0htZjx X0n710+uJ6ifuaHglHu95+/c341Geoe7mMejBFd7l8vs1YZCwX2w68mjgVsOZgIAd6BC Vv5UDMFdFGOUp8SMWMfkNsyUbs94prceJt6dFn3/tW8UC/fZHQpS/9Ap058onsTfMH4z 0ANqFkRuVbIfNufGK2GtUiyU+8cQHvJtK0GjrN4iCR71FhoTBo7MhN6dWY4BotabYdqI DLyE5guLOjXsrD1kfTO8TcKWH3e+ZwZGb3TWfAOnNGIi76Adkx4+63/fac9eOViznaFH yjxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=akVSJ2LyXWCgnCkenP6zVJjPdvdtBduMAT+INdXPpJE=; b=KlPuV0jvfsCrWNX7TXpuEE3lQzSQKEkyV3WPfQy76VufCIn40KCvA4M7C1JJwpfmjN pQkwTNqS+x3+fRUTTr6xuk+7mT7Y2uZcTvgR6Dcz+HNonkfw/FvTgdk0M/+EqoaRxcf/ hRhSMlC24gbowW3vGGWzqtHlvybKzSXwQuzkUhBrmpPG6CnjqCSEertAa9ud9dZ7elDq O4BN18y6UZRIuUTlVRNhqm2cWUJSKzNSIcdkTUtfvKCDzFnAuLtcyDEJpYATQQM8QiD7 HUklKBRlL1w5NDFqL5YgqbcAnraSoNN5Ux4GLWBg2RFDd6GPdz5hkDqBZsIqQSSr28Ws 5LBg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=T9MnKLbP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bg23-20020a17090b0d9700b0023cfd288a09si1623644pjb.157.2023.03.29.09.27.51; Wed, 29 Mar 2023 09:28:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=T9MnKLbP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230359AbjC2QE6 (ORCPT <rfc822;rua109.linux@gmail.com> + 99 others); Wed, 29 Mar 2023 12:04:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231585AbjC2QEV (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Wed, 29 Mar 2023 12:04:21 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B56C3619E; Wed, 29 Mar 2023 09:03:11 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7054461DA6; Wed, 29 Mar 2023 16:02:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2042CC433A8; Wed, 29 Mar 2023 16:02:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1680105737; bh=0K4ESNHhsDtk/yTZYL0aDVGEKmylHOGHPXnx46UNdgA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=T9MnKLbPrzarjfBI9Dw3rgLPHWQ8R6SP/XmHP0pf5K+NS6LV+GbdRHtt44caB17Aa Fovrx/MzSWysqABNpo4PTUYL30Yd0e/T14VLVTAG7i5Tij52icZET8bAihMSn2uncp 9dMU1P+Y0blFA4Z74TJYw8hmeSgYk6IBYNX6ovEdXytKo3dlUkiI98znrYWcl9kRBo 1slX4ifmJD5DWD+8I3Rbvnud97Fu8BfkTVZ+xehDhzeywWmnKwPeXuJ2bUC1uZ1sx1 rHHr6nT9u3Mw3v5d+dOFQGgwI7yM4iXHP8aK6LM2EiEjamYCebP3ZCgp8OBmd3WvAN ctWQZThLtRm/g== From: Frederic Weisbecker <frederic@kernel.org> To: "Paul E . McKenney" <paulmck@kernel.org> Cc: LKML <linux-kernel@vger.kernel.org>, Frederic Weisbecker <frederic@kernel.org>, rcu <rcu@vger.kernel.org>, Uladzislau Rezki <urezki@gmail.com>, Neeraj Upadhyay <quic_neeraju@quicinc.com>, Boqun Feng <boqun.feng@gmail.com>, Joel Fernandes <joel@joelfernandes.org> Subject: [PATCH 4/4] rcu/nocb: Make shrinker to iterate only NOCB CPUs Date: Wed, 29 Mar 2023 18:02:03 +0200 Message-Id: <20230329160203.191380-5-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230329160203.191380-1-frederic@kernel.org> References: <20230329160203.191380-1-frederic@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761720175050452593?= X-GMAIL-MSGID: =?utf-8?q?1761720175050452593?= |
Series |
rcu/nocb: Shrinker related boring fixes
|
|
Commit Message
Frederic Weisbecker
March 29, 2023, 4:02 p.m. UTC
Callbacks can only be queued as lazy on NOCB CPUs, therefore iterating
over the NOCB mask is enough for both counting and scanning. Just lock
the mostly uncontended barrier mutex on counting as well in order to
keep rcu_nocb_mask stable.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
kernel/rcu/tree_nocb.h | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
Comments
On Wed, Mar 29, 2023 at 06:02:03PM +0200, Frederic Weisbecker wrote: > Callbacks can only be queued as lazy on NOCB CPUs, therefore iterating > over the NOCB mask is enough for both counting and scanning. Just lock > the mostly uncontended barrier mutex on counting as well in order to > keep rcu_nocb_mask stable. > > Signed-off-by: Frederic Weisbecker <frederic@kernel.org> Looks plausible. ;-) What are you doing to test this? For that matter, what should rcutorture be doing to test this? My guess is that the current callback flooding in rcu_torture_fwd_prog_cr() should do the trick, but figured I should ask. Thanx, Paul > --- > kernel/rcu/tree_nocb.h | 17 ++++++++++++++--- > 1 file changed, 14 insertions(+), 3 deletions(-) > > diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h > index dfa9c10d6727..43229d2b0c44 100644 > --- a/kernel/rcu/tree_nocb.h > +++ b/kernel/rcu/tree_nocb.h > @@ -1319,13 +1319,22 @@ lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) > int cpu; > unsigned long count = 0; > > + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) > + return 0; > + > + /* Protect rcu_nocb_mask against concurrent (de-)offloading. */ > + if (!mutex_trylock(&rcu_state.barrier_mutex)) > + return 0; > + > /* Snapshot count of all CPUs */ > - for_each_possible_cpu(cpu) { > + for_each_cpu(cpu, rcu_nocb_mask) { > struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); > > count += READ_ONCE(rdp->lazy_len); > } > > + mutex_unlock(&rcu_state.barrier_mutex); > + > return count ? count : SHRINK_EMPTY; > } > > @@ -1336,6 +1345,8 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) > unsigned long flags; > unsigned long count = 0; > > + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) > + return 0; > /* > * Protect against concurrent (de-)offloading. Otherwise nocb locking > * may be ignored or imbalanced. > @@ -1351,11 +1362,11 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) > } > > /* Snapshot count of all CPUs */ > - for_each_possible_cpu(cpu) { > + for_each_cpu(cpu, rcu_nocb_mask) { > struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); > int _count; > > - if (!rcu_rdp_is_offloaded(rdp)) > + if (WARN_ON_ONCE(!rcu_rdp_is_offloaded(rdp))) > continue; > > if (!READ_ONCE(rdp->lazy_len)) > -- > 2.34.1 >
On Wed, Mar 29, 2023 at 01:58:06PM -0700, Paul E. McKenney wrote: > On Wed, Mar 29, 2023 at 06:02:03PM +0200, Frederic Weisbecker wrote: > > Callbacks can only be queued as lazy on NOCB CPUs, therefore iterating > > over the NOCB mask is enough for both counting and scanning. Just lock > > the mostly uncontended barrier mutex on counting as well in order to > > keep rcu_nocb_mask stable. > > > > Signed-off-by: Frederic Weisbecker <frederic@kernel.org> > > Looks plausible. ;-) > > What are you doing to test this? For that matter, what should rcutorture > be doing to test this? My guess is that the current callback flooding in > rcu_torture_fwd_prog_cr() should do the trick, but figured I should ask. All I did was to trigger these shrinker callbacks through debugfs (https://docs.kernel.org/admin-guide/mm/shrinker_debugfs.html) But rcutorture isn't testing it because: - No torture config has CONFIG_RCU_LAZY - rcutorture doesn't do any lazy call_rcu() (always calls hurry for the main RCU flavour). And I suspect rcutorture isn't ready for accepting the lazy delay, that would require some special treatment. Thanks.
On Wed, Mar 29, 2023 at 11:35:36PM +0200, Frederic Weisbecker wrote: > On Wed, Mar 29, 2023 at 01:58:06PM -0700, Paul E. McKenney wrote: > > On Wed, Mar 29, 2023 at 06:02:03PM +0200, Frederic Weisbecker wrote: > > > Callbacks can only be queued as lazy on NOCB CPUs, therefore iterating > > > over the NOCB mask is enough for both counting and scanning. Just lock > > > the mostly uncontended barrier mutex on counting as well in order to > > > keep rcu_nocb_mask stable. > > > > > > Signed-off-by: Frederic Weisbecker <frederic@kernel.org> > > > > Looks plausible. ;-) > > > > What are you doing to test this? For that matter, what should rcutorture > > be doing to test this? My guess is that the current callback flooding in > > rcu_torture_fwd_prog_cr() should do the trick, but figured I should ask. > > All I did was to trigger these shrinker callbacks through debugfs > (https://docs.kernel.org/admin-guide/mm/shrinker_debugfs.html) > > But rcutorture isn't testing it because: > > - No torture config has CONFIG_RCU_LAZY > - rcutorture doesn't do any lazy call_rcu() (always calls hurry for the > main RCU flavour). > > And I suspect rcutorture isn't ready for accepting the lazy delay, that would > require some special treatment. All fair points! And yes, any non-lazy callback would delazify everything, so as it is currently constituted, it would not be testing very much of the lazy-callback state space. Thanx, Paul
diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index dfa9c10d6727..43229d2b0c44 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1319,13 +1319,22 @@ lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) int cpu; unsigned long count = 0; + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) + return 0; + + /* Protect rcu_nocb_mask against concurrent (de-)offloading. */ + if (!mutex_trylock(&rcu_state.barrier_mutex)) + return 0; + /* Snapshot count of all CPUs */ - for_each_possible_cpu(cpu) { + for_each_cpu(cpu, rcu_nocb_mask) { struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); count += READ_ONCE(rdp->lazy_len); } + mutex_unlock(&rcu_state.barrier_mutex); + return count ? count : SHRINK_EMPTY; } @@ -1336,6 +1345,8 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) unsigned long flags; unsigned long count = 0; + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) + return 0; /* * Protect against concurrent (de-)offloading. Otherwise nocb locking * may be ignored or imbalanced. @@ -1351,11 +1362,11 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) } /* Snapshot count of all CPUs */ - for_each_possible_cpu(cpu) { + for_each_cpu(cpu, rcu_nocb_mask) { struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); int _count; - if (!rcu_rdp_is_offloaded(rdp)) + if (WARN_ON_ONCE(!rcu_rdp_is_offloaded(rdp))) continue; if (!READ_ONCE(rdp->lazy_len))