From patchwork Tue Nov 22 01:04:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24081 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1929918wrr; Mon, 21 Nov 2022 17:05:14 -0800 (PST) X-Google-Smtp-Source: AA0mqf4DAn23Yg0m/zTqhZfo2q0UbDRsFfpfeht0UkJG2LbSbJLpk2F2dQZXMcr2lpuVzjFu0wOE X-Received: by 2002:a17:903:2601:b0:189:315f:d6fc with SMTP id jd1-20020a170903260100b00189315fd6fcmr1769329plb.35.1669079114556; Mon, 21 Nov 2022 17:05:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079114; cv=none; d=google.com; s=arc-20160816; b=meOiOX8aaADyfzew0e6SpYUmuNbZPS5e4P41DmsZBnzpvQPvcfCCsciIvZ30yiIFDt WR4nV+s5CR2LX8BrNfKu6/NHZXs5WGRywXVkkv9tqIEvjRiXTd9rK3YEjdR/1tSCbD4v LACMuXL8CXR32A7Rxer2GThbKa6Zexdmc6d95ybKH41VReGSLMYIUDDbh4mhYGsYUKtW hTlIiiZdfHCwjKKDobMGcb2BLVwJZkd1lf91GOsfpUcRKwWjXPrVA4/pm2j64jNWv7ON vxy9IPSWKW47ja6m9AUAv1LWURrK539q0gWOS6M7Vfju4WEykPFS3ZJtN2HwLnFTnXmM cbBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KpkkHRs6NSlaA30zXYSvuq9VaM3mDcAD5+jXqlQvvlo=; b=r8XEZ2LHGqeMdIKZE3MTo1qfukFJIdsEyGoR3raIz7W+T1IOuC86II1TOeMswf9Olt 1BxspDZ7Q+jUm3IDqRrbccuntIFCWBCZMgRrzYDsx03vkY2/Dlv8HGUe/BjgwegRLi+s 14JoSL/LKNsH6ABvhK1AjM/FGpYuPYXkvgEzxWGItmYPzUcy4JcUb9DVoXk0MTgBGaLX Vl9xh/yB1TlQ63fQ+RT+worCN7SBDDplD03AoB8fEzbK/jYGD7TRprbKGZHHa6TQVeQ4 GrIVTc2DzPhepvpxsOD8QJdR649o52WmRyMjbWyC9Fs3C9J9gEic/Np9BmEcRF7khKMq Mygw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=l80XZpuR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j190-20020a6380c7000000b00477312c910dsi10170588pgd.657.2022.11.21.17.05.01; Mon, 21 Nov 2022 17:05:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=l80XZpuR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232070AbiKVBEm (ORCPT + 99 others); Mon, 21 Nov 2022 20:04:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44808 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231935AbiKVBEZ (ORCPT ); Mon, 21 Nov 2022 20:04:25 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A08CADE9A; Mon, 21 Nov 2022 17:04:24 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id F0AE06151C; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1A663C43151; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079063; bh=hIFNQHcIbY8YtBkmBJx5icL1buQHNTtIxJeozRTrMW8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=l80XZpuRLUSSJsfBa+XlMP4gcQbNsw2NpaOiGOrWZDuoa2xpSHx4WmtNaVKRVe8O0 ZylXtgqWLL1Aof0vdX+L2M+9DM8OvANGbU/rLbX47UnsyD8H3pj1PARWK+P2skct8K sahbDCb47N+kfh7+vnqWmpMsjiEa/X9Fj66FDAXbXTv4xk3y0DVadRw18JldcpDO4d PKKT8fidKmlnBctadwf/v+xZWx1iyBlGYGs4DkXI5nuL7qy2+/vrFszkhlFCXlufO3 NljYnSE3YelGoIzCERzGJg014E5/c76UNpkHnNQwaLDuw+1uXCPn1p+Uk4F7p8GU/a AFvJewBwASYmA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 7BA6B5C1219; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Vineeth Pillai , Joel Fernandes , "Paul E . McKenney" Subject: [PATCH v2 rcu 06/16] rcu: Shrinker for lazy rcu Date: Mon, 21 Nov 2022 17:04:11 -0800 Message-Id: <20221122010421.3799681-6-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156301817061976?= X-GMAIL-MSGID: =?utf-8?q?1750156301817061976?= From: Vineeth Pillai The shrinker is used to speed up the free'ing of memory potentially held by RCU lazy callbacks. RCU kernel module test cases show this to be effective. Test is introduced in a later patch. Signed-off-by: Vineeth Pillai Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 52 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 213daf81c057f..9e1c8caec5ceb 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1312,6 +1312,55 @@ int rcu_nocb_cpu_offload(int cpu) } EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload); +static unsigned long +lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + + count += READ_ONCE(rdp->lazy_len); + } + + return count ? count : SHRINK_EMPTY; +} + +static unsigned long +lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu; + unsigned long flags; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + int _count = READ_ONCE(rdp->lazy_len); + + if (_count == 0) + continue; + rcu_nocb_lock_irqsave(rdp, flags); + WRITE_ONCE(rdp->lazy_len, 0); + rcu_nocb_unlock_irqrestore(rdp, flags); + wake_nocb_gp(rdp, false); + sc->nr_to_scan -= _count; + count += _count; + if (sc->nr_to_scan <= 0) + break; + } + return count ? count : SHRINK_STOP; +} + +static struct shrinker lazy_rcu_shrinker = { + .count_objects = lazy_rcu_shrink_count, + .scan_objects = lazy_rcu_shrink_scan, + .batch = 0, + .seeks = DEFAULT_SEEKS, +}; + void __init rcu_init_nohz(void) { int cpu; @@ -1342,6 +1391,9 @@ void __init rcu_init_nohz(void) if (!rcu_state.nocb_is_setup) return; + if (register_shrinker(&lazy_rcu_shrinker, "rcu-lazy")) + pr_err("Failed to register lazy_rcu shrinker!\n"); + if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) { pr_info("\tNote: kernel parameter 'rcu_nocbs=', 'nohz_full', or 'isolcpus=' contains nonexistent CPUs.\n"); cpumask_and(rcu_nocb_mask, cpu_possible_mask,