From patchwork Wed Mar 22 19:44:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 73651 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp2537826wrt; Wed, 22 Mar 2023 12:53:59 -0700 (PDT) X-Google-Smtp-Source: AK7set8niebKQ8CO5o39V9vy5sGL4f/gTdSGo8l9SV3FB+m86LMDPsXPqWtZgEkGYAqESLFd19pn X-Received: by 2002:a05:6a20:8f0f:b0:cc:70df:ae20 with SMTP id b15-20020a056a208f0f00b000cc70dfae20mr782983pzk.0.1679514839160; Wed, 22 Mar 2023 12:53:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679514839; cv=none; d=google.com; s=arc-20160816; b=KNBJxVtj3Uu8PsaGYeiwkKr35q1C5iVkeOepuXvrS35sL5BoTuBkeZ2xdviLmZ4vtV aFutduBbVCOzBht3QpKXJ9LorWWKSeIEjDrTi1VlllYXbVRFRY4rOAWBTN2f5OmGTQkj C5IYgYirBayY/RaqiCceFTgDgpmn0NQ4lMvMOiqEWyhgBr6UFuqeH162qu7xNGFxnKTq Fi0O/Ue7OaD/YdO6rmFZg0xT2Oe6TnF6/kEQBeF6CogRfVIRZGbYZVeRLvMc1FoIFKrD L7Sr5fPYzFrS/lgbHVyc/sUreMtngNphrKthSIVqRrebDpTOiq6xNcIyfmHVfQFr+GnL 1tyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=kpX6/yGKLnlgQaQIZH1x0b4a3n71Kp6mrU22uMIIRl0=; b=X2DHVZkoJXC0+NhmilGdILbCqmSn2G5SZ5W9ae6nLZ6MWtvPOQWKD4/onk0HijsHdb +WnMizzG+vxG4TXbiZVCPZFdIY7s9d5J3EinIgAlmgYYlLYnQrdsDa+OQiauANZsiG5h 8luR8zqKuQGdop8RrcffSv2/CkJ0TeCwfYwoloM3miJLYxEsDPjaXWEl7L5lMWrEjdb+ eUOHrVzPkNUYuyPy6vhd7ybA4UmxSUzXB38TCPqTa1aWnJI9wjZf0WbopNwOfCB/xT9S kV3dCGKYpKcu2b1xxwqRadOnEyFYYZue+BMglcA6MmIetJR/8gxqG4B1Zz0Lu9Ffoxu+ 3+UQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=sMzpGXOB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 18-20020a631552000000b004fbd23110d4si15737196pgv.773.2023.03.22.12.53.46; Wed, 22 Mar 2023 12:53:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=sMzpGXOB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229865AbjCVTpb (ORCPT + 99 others); Wed, 22 Mar 2023 15:45:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52626 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229928AbjCVTpT (ORCPT ); Wed, 22 Mar 2023 15:45:19 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C81F11EFE2; Wed, 22 Mar 2023 12:45:15 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 78F8CB81DEA; Wed, 22 Mar 2023 19:45:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5A46EC433AA; Wed, 22 Mar 2023 19:45:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679514313; bh=o/d+MExJk6dT2Q3smaaeBjF9BtLmW7hBw77NXSOsuZA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sMzpGXOBEC1RG/wSZamyrsu+iU5m/0UxfjbhgycVXYNRHFbH1PZzU36l2ZJ4mSV0j LkxFT6qscyXdSGYE7CmKa1TSgHrv8ZKVgvw5F65b2ML4Bc3nLmvr37dSZcgONG+5pJ mdw05b0W+QtvQbJBzNWZzZiXPt0XXlvpK3HV5gZA+FxyPMGB7CvKDCBH8iioZ4teNE d3i8DgCAfvDwfQLZEvkwaEIUyJ5yDb+kP0dxZBPQzjEi/l9KdYglF8ryipqJ9Imq7y zm+q6aVtWhZGneLo//5aGNwdBjFj+fGPKk8BA3LqKTtQbIbBhhQyejpRxL+Q7gZWZr Y82MOffgaHTBQ== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , rcu , Uladzislau Rezki , Neeraj Upadhyay , Boqun Feng , Joel Fernandes Subject: [PATCH 4/4] rcu/nocb: Make shrinker to iterate only NOCB CPUs Date: Wed, 22 Mar 2023 20:44:56 +0100 Message-Id: <20230322194456.2331527-5-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230322194456.2331527-1-frederic@kernel.org> References: <20230322194456.2331527-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-5.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761098951935719307?= X-GMAIL-MSGID: =?utf-8?q?1761098951935719307?= Callbacks can only be queued as lazy on NOCB CPUs, therefore iterating over the NOCB mask is enough for both counting and scanning. Just lock the mostly uncontended barrier mutex on counting as well in order to keep rcu_nocb_mask stable. Signed-off-by: Frederic Weisbecker Reviewed-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index a3dc7465b0b2..185c0c9a60d4 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1319,13 +1319,21 @@ lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) int cpu; unsigned long count = 0; + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) + return 0; + + /* Protect rcu_nocb_mask against concurrent (de-)offloading. */ + mutex_lock(&rcu_state.barrier_mutex); + /* Snapshot count of all CPUs */ - for_each_possible_cpu(cpu) { + for_each_cpu(cpu, rcu_nocb_mask) { struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); count += READ_ONCE(rdp->lazy_len); } + mutex_unlock(&rcu_state.barrier_mutex); + return count ? count : SHRINK_EMPTY; } @@ -1336,6 +1344,8 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) unsigned long flags; unsigned long count = 0; + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) + return 0; /* * Protect against concurrent (de-)offloading. Otherwise nocb locking * may be ignored or imbalanced. @@ -1343,7 +1353,7 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) mutex_lock(&rcu_state.barrier_mutex); /* Snapshot count of all CPUs */ - for_each_possible_cpu(cpu) { + for_each_cpu(cpu, rcu_nocb_mask) { struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); int _count;