From patchwork Wed Mar 29 16:02:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 76703 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp537621vqo; Wed, 29 Mar 2023 09:28:04 -0700 (PDT) X-Google-Smtp-Source: AKy350YhsEzzHXsSNy38Yy+lxKSVXzSPrzMUqlCsacekpu4cj/HEidesvqYDJ0jVInJkWib9jSoL X-Received: by 2002:a17:90b:3a85:b0:23d:e0e8:f453 with SMTP id om5-20020a17090b3a8500b0023de0e8f453mr20434324pjb.38.1680107283940; Wed, 29 Mar 2023 09:28:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680107283; cv=none; d=google.com; s=arc-20160816; b=GdqTGDEIEQRMttN7eVfxdKsZ2U+NL8LJXDCGNLyFOFWzEOT4ey4gM7sXp45H0htZjx X0n710+uJ6ifuaHglHu95+/c341Geoe7mMejBFd7l8vs1YZCwX2w68mjgVsOZgIAd6BC Vv5UDMFdFGOUp8SMWMfkNsyUbs94prceJt6dFn3/tW8UC/fZHQpS/9Ap058onsTfMH4z 0ANqFkRuVbIfNufGK2GtUiyU+8cQHvJtK0GjrN4iCR71FhoTBo7MhN6dWY4BotabYdqI DLyE5guLOjXsrD1kfTO8TcKWH3e+ZwZGb3TWfAOnNGIi76Adkx4+63/fac9eOViznaFH yjxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=akVSJ2LyXWCgnCkenP6zVJjPdvdtBduMAT+INdXPpJE=; b=KlPuV0jvfsCrWNX7TXpuEE3lQzSQKEkyV3WPfQy76VufCIn40KCvA4M7C1JJwpfmjN pQkwTNqS+x3+fRUTTr6xuk+7mT7Y2uZcTvgR6Dcz+HNonkfw/FvTgdk0M/+EqoaRxcf/ hRhSMlC24gbowW3vGGWzqtHlvybKzSXwQuzkUhBrmpPG6CnjqCSEertAa9ud9dZ7elDq O4BN18y6UZRIuUTlVRNhqm2cWUJSKzNSIcdkTUtfvKCDzFnAuLtcyDEJpYATQQM8QiD7 HUklKBRlL1w5NDFqL5YgqbcAnraSoNN5Ux4GLWBg2RFDd6GPdz5hkDqBZsIqQSSr28Ws 5LBg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=T9MnKLbP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bg23-20020a17090b0d9700b0023cfd288a09si1623644pjb.157.2023.03.29.09.27.51; Wed, 29 Mar 2023 09:28:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=T9MnKLbP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230359AbjC2QE6 (ORCPT + 99 others); Wed, 29 Mar 2023 12:04:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231585AbjC2QEV (ORCPT ); Wed, 29 Mar 2023 12:04:21 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B56C3619E; Wed, 29 Mar 2023 09:03:11 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7054461DA6; Wed, 29 Mar 2023 16:02:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2042CC433A8; Wed, 29 Mar 2023 16:02:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1680105737; bh=0K4ESNHhsDtk/yTZYL0aDVGEKmylHOGHPXnx46UNdgA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=T9MnKLbPrzarjfBI9Dw3rgLPHWQ8R6SP/XmHP0pf5K+NS6LV+GbdRHtt44caB17Aa Fovrx/MzSWysqABNpo4PTUYL30Yd0e/T14VLVTAG7i5Tij52icZET8bAihMSn2uncp 9dMU1P+Y0blFA4Z74TJYw8hmeSgYk6IBYNX6ovEdXytKo3dlUkiI98znrYWcl9kRBo 1slX4ifmJD5DWD+8I3Rbvnud97Fu8BfkTVZ+xehDhzeywWmnKwPeXuJ2bUC1uZ1sx1 rHHr6nT9u3Mw3v5d+dOFQGgwI7yM4iXHP8aK6LM2EiEjamYCebP3ZCgp8OBmd3WvAN ctWQZThLtRm/g== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , rcu , Uladzislau Rezki , Neeraj Upadhyay , Boqun Feng , Joel Fernandes Subject: [PATCH 4/4] rcu/nocb: Make shrinker to iterate only NOCB CPUs Date: Wed, 29 Mar 2023 18:02:03 +0200 Message-Id: <20230329160203.191380-5-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230329160203.191380-1-frederic@kernel.org> References: <20230329160203.191380-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761720175050452593?= X-GMAIL-MSGID: =?utf-8?q?1761720175050452593?= Callbacks can only be queued as lazy on NOCB CPUs, therefore iterating over the NOCB mask is enough for both counting and scanning. Just lock the mostly uncontended barrier mutex on counting as well in order to keep rcu_nocb_mask stable. Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index dfa9c10d6727..43229d2b0c44 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1319,13 +1319,22 @@ lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) int cpu; unsigned long count = 0; + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) + return 0; + + /* Protect rcu_nocb_mask against concurrent (de-)offloading. */ + if (!mutex_trylock(&rcu_state.barrier_mutex)) + return 0; + /* Snapshot count of all CPUs */ - for_each_possible_cpu(cpu) { + for_each_cpu(cpu, rcu_nocb_mask) { struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); count += READ_ONCE(rdp->lazy_len); } + mutex_unlock(&rcu_state.barrier_mutex); + return count ? count : SHRINK_EMPTY; } @@ -1336,6 +1345,8 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) unsigned long flags; unsigned long count = 0; + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) + return 0; /* * Protect against concurrent (de-)offloading. Otherwise nocb locking * may be ignored or imbalanced. @@ -1351,11 +1362,11 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) } /* Snapshot count of all CPUs */ - for_each_possible_cpu(cpu) { + for_each_cpu(cpu, rcu_nocb_mask) { struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); int _count; - if (!rcu_rdp_is_offloaded(rdp)) + if (WARN_ON_ONCE(!rcu_rdp_is_offloaded(rdp))) continue; if (!READ_ONCE(rdp->lazy_len))