From patchwork Wed Oct 19 22:51:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 5903 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp573088wrs; Wed, 19 Oct 2022 15:57:30 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6BLES046/2imEXXANctubq13dTWd/4D0s38x9x73rJrYiG2C6gP0nJmAG0lElz75PXwagC X-Received: by 2002:a05:6402:35cc:b0:45d:4a39:2a43 with SMTP id z12-20020a05640235cc00b0045d4a392a43mr9651657edc.304.1666220249991; Wed, 19 Oct 2022 15:57:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666220249; cv=none; d=google.com; s=arc-20160816; b=yuKlRAuzWXTmMCrsJsbzlQlIJDloedTE5SmRlbo6dMqy9nAdGYN55oENCk40b5FQ6+ 1YDJe6BUQxIDbTzQrQ81AWvNb4A1pSFCXB/9OGcnLds2UuDrIZBZFqrjih3gmBIhQ7JK /B4qddF7/HjveY5aCURq4MnPgfoVvNQfvjXsfwqT882uITHryv80RLTYcfsst+g1fW2S FP4cK1afhKtbbwRre3vxvS3RHjQecKhxX6ILN7rUZN6wFF2WMl+PRQrq9QrCJte5tdse xb9XPSjIau52yWG+wuNud6Vw4Awn9sv4fb/jWw9DIksUxakt2zqGgwzpE3AU7wAc/ftw 3k/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KpkkHRs6NSlaA30zXYSvuq9VaM3mDcAD5+jXqlQvvlo=; b=QfdPedbxO5m2cqxQCVAHC6c74fH3wrpICJkCnow04dzks8paFfTfGHTV5uP0MFRe+e JRdygSMqxLI3Lai9SLsb02Z5bFeBU8sgX4BdZOtNMqqUwXv8R/jGuOMIj4h9hYvZUqKD yWAqaVVnIqr8gAQ5PLZs8xiwDqT/KGNSC0umHew0kbJn9HSOC3oCMLk6wATAEG/fVZDJ oP7tZWrLJrVEiJWGkjk17d+adfpi7lvNOaOp3w2DDg6382I7ggWzWgpk2YZsXTKRS+dy JQCAaPDTQfzwwcp0Qg6nTCpsBhRUGDlo+Em7jkEnU0McsH4xmxBc8oUWtBQ6WkmOI3Pt VQJA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=MAbTGcy7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cs17-20020a170906dc9100b007848e6eaef1si9613255ejc.733.2022.10.19.15.57.05; Wed, 19 Oct 2022 15:57:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=MAbTGcy7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230496AbiJSWwf (ORCPT + 99 others); Wed, 19 Oct 2022 18:52:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51160 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230313AbiJSWvy (ORCPT ); Wed, 19 Oct 2022 18:51:54 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B94F1905C2; Wed, 19 Oct 2022 15:51:50 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 37C07B82620; Wed, 19 Oct 2022 22:51:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 18ADEC4314C; Wed, 19 Oct 2022 22:51:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666219907; bh=hIFNQHcIbY8YtBkmBJx5icL1buQHNTtIxJeozRTrMW8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MAbTGcy7ARkabmWioJ3weutz0OZZjs+BAUteqFV69VdQSdNqMT9FxKkjNkPkaqTwh Oru8cXB60UytRoW9394LnIayvuYlEAOTHXodoTFMC30pFaxrw9HbkmQWe4RYYdT+em dsiNokspncAC8j/16EpELJfEJL7eSmo2Pe13KcNQehD8CFPLPNvICE/CPDSAf7HK3a YiUIxk9ov1aQ/alMvF/w66EKTpR1zjGcbCGll00dDyEdW5lh0UxAr+DHfmccAQsg7x QuXbu0Di5xVKAbWESeGLUHvR0h/HuVGX1R+Wz275LVWGZu8ZLAkCG/wyZ79RwutS+m dyMmPFn8ZZOgw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 80A4B5C0A40; Wed, 19 Oct 2022 15:51:46 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Vineeth Pillai , Joel Fernandes , "Paul E . McKenney" Subject: [PATCH rcu 06/14] rcu: Shrinker for lazy rcu Date: Wed, 19 Oct 2022 15:51:36 -0700 Message-Id: <20221019225144.2500095-6-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221019225138.GA2499943@paulmck-ThinkPad-P17-Gen-1> References: <20221019225138.GA2499943@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747158564456112187?= X-GMAIL-MSGID: =?utf-8?q?1747158564456112187?= From: Vineeth Pillai The shrinker is used to speed up the free'ing of memory potentially held by RCU lazy callbacks. RCU kernel module test cases show this to be effective. Test is introduced in a later patch. Signed-off-by: Vineeth Pillai Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 52 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 213daf81c057f..9e1c8caec5ceb 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1312,6 +1312,55 @@ int rcu_nocb_cpu_offload(int cpu) } EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload); +static unsigned long +lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + + count += READ_ONCE(rdp->lazy_len); + } + + return count ? count : SHRINK_EMPTY; +} + +static unsigned long +lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu; + unsigned long flags; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + int _count = READ_ONCE(rdp->lazy_len); + + if (_count == 0) + continue; + rcu_nocb_lock_irqsave(rdp, flags); + WRITE_ONCE(rdp->lazy_len, 0); + rcu_nocb_unlock_irqrestore(rdp, flags); + wake_nocb_gp(rdp, false); + sc->nr_to_scan -= _count; + count += _count; + if (sc->nr_to_scan <= 0) + break; + } + return count ? count : SHRINK_STOP; +} + +static struct shrinker lazy_rcu_shrinker = { + .count_objects = lazy_rcu_shrink_count, + .scan_objects = lazy_rcu_shrink_scan, + .batch = 0, + .seeks = DEFAULT_SEEKS, +}; + void __init rcu_init_nohz(void) { int cpu; @@ -1342,6 +1391,9 @@ void __init rcu_init_nohz(void) if (!rcu_state.nocb_is_setup) return; + if (register_shrinker(&lazy_rcu_shrinker, "rcu-lazy")) + pr_err("Failed to register lazy_rcu shrinker!\n"); + if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) { pr_info("\tNote: kernel parameter 'rcu_nocbs=', 'nohz_full', or 'isolcpus=' contains nonexistent CPUs.\n"); cpumask_and(rcu_nocb_mask, cpu_possible_mask,