From patchwork Wed Nov 30 18:13:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 27934 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1077858wrr; Wed, 30 Nov 2022 10:15:04 -0800 (PST) X-Google-Smtp-Source: AA0mqf7Z5TQ9z1hnIN5nv1iqwRpoHW1L05vF+20IrKUBhpIFDk1hCeqpBUIwPi4fykLFmzcBQM2f X-Received: by 2002:a17:907:d047:b0:7ba:530:223a with SMTP id vb7-20020a170907d04700b007ba0530223amr30642415ejc.215.1669832104280; Wed, 30 Nov 2022 10:15:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669832104; cv=none; d=google.com; s=arc-20160816; b=VIN1FQfIEqxzYAPNS09RJou1Mfb3gmJgLC78VYWoGorfibFgpIOKTIqnK4D3Y1pU/o 35c9zyrz/r2t33BBwAjkL1fz2+85ibkSkLMDPmEjbgTacFuiz0diVQMPcUuqvTxZ0smq FXYjYspTXwgoAtDwu7s/T/+W9fyZN/x9zt3kT9MEB+yG2n/2j9jH/IAESHEvMHVKg8Ye Qx0vOrYUxVAivelFuwrYUxGtJiKU5g1Pc4Eo9LdyFX91WL7pUiE6HtrTzGiD43DhD+Pg 6Yv+mqvRfiXovPyJMT3H43QJA7KPq7duvsEqSKauuWLmISbQkclcXiLwU+6/kT5k9Dyv JOuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KpkkHRs6NSlaA30zXYSvuq9VaM3mDcAD5+jXqlQvvlo=; b=bsMmOLLnCss2uHInWxPEYw7IEd3QrlThUl0onISHPLGalpUG+ZsL6//7wkoPWl08B1 IjPagJemUE96ul0XID0NmHn+WGxPs2e45MyMphuC/4qL/DO9wr6ekS8gBY/Z/EmdC+Jc 4LHZNn89oxc973Tk3cKVtXxlGIVSUcEHxdcyZcrFx8+uoDJGg6oEz/5HXKPFdt2N6rOM GNqsFm8/ln90ft+VvjGDitkSlrmlYXfKPPraEYPmb9IJv9UuPGpYq6+aS+E3aI5AlRU4 faVG20AAo3k1JEkKE2uU4xZOQsHe0nPZf2VQCJhWmADRB80yWswJyVJ1t9vGiL4gKMST 2NmA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ZcqbaoCw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gt11-20020a1709072d8b00b007a20c586f4esi2032590ejc.876.2022.11.30.10.14.39; Wed, 30 Nov 2022 10:15:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ZcqbaoCw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230103AbiK3SNv (ORCPT + 99 others); Wed, 30 Nov 2022 13:13:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229974AbiK3SNa (ORCPT ); Wed, 30 Nov 2022 13:13:30 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F20C880D5; Wed, 30 Nov 2022 10:13:29 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id DDDCC61D59; Wed, 30 Nov 2022 18:13:28 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EF12BC43150; Wed, 30 Nov 2022 18:13:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669832008; bh=hIFNQHcIbY8YtBkmBJx5icL1buQHNTtIxJeozRTrMW8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZcqbaoCwWgpfTBjbKPRFJJwRPJc5Ejc/7PIei5D9g6OZD+vcLZL44CqUzletkgDmn kmTWhtbegh8kbXGBylPtd1ALYn4P73Qd09zqgGld2TDoc3I4TMgmOxh0cyfMkOqOpu ZWtPl4JshImqKuU1LHowxQdfmR5kp6HqByOBJIq/kv/AgQydFU4yx4gFRqF2tzUT2X N+Ri6k718HDvY4R5aByGoXEDtUeJQU2zz9Dsf/P3gUJVhUqW20mGH2Oo6BBpPsftsa vNrtqqiRXNPbtUdm21PaHp2OTpsVqrYRya38dDVLf1ceFxbB7PXX1TQT6xSqUBU2gu 0mECVh3dOkCwQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 607D65C0F36; Wed, 30 Nov 2022 10:13:27 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Vineeth Pillai , Joel Fernandes , "Paul E . McKenney" Subject: [PATCH rcu 06/16] rcu: Shrinker for lazy rcu Date: Wed, 30 Nov 2022 10:13:15 -0800 Message-Id: <20221130181325.1012760-6-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> References: <20221130181316.GA1012431@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750945868541366097?= X-GMAIL-MSGID: =?utf-8?q?1750945868541366097?= From: Vineeth Pillai The shrinker is used to speed up the free'ing of memory potentially held by RCU lazy callbacks. RCU kernel module test cases show this to be effective. Test is introduced in a later patch. Signed-off-by: Vineeth Pillai Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 52 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 213daf81c057f..9e1c8caec5ceb 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1312,6 +1312,55 @@ int rcu_nocb_cpu_offload(int cpu) } EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload); +static unsigned long +lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + + count += READ_ONCE(rdp->lazy_len); + } + + return count ? count : SHRINK_EMPTY; +} + +static unsigned long +lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) +{ + int cpu; + unsigned long flags; + unsigned long count = 0; + + /* Snapshot count of all CPUs */ + for_each_possible_cpu(cpu) { + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + int _count = READ_ONCE(rdp->lazy_len); + + if (_count == 0) + continue; + rcu_nocb_lock_irqsave(rdp, flags); + WRITE_ONCE(rdp->lazy_len, 0); + rcu_nocb_unlock_irqrestore(rdp, flags); + wake_nocb_gp(rdp, false); + sc->nr_to_scan -= _count; + count += _count; + if (sc->nr_to_scan <= 0) + break; + } + return count ? count : SHRINK_STOP; +} + +static struct shrinker lazy_rcu_shrinker = { + .count_objects = lazy_rcu_shrink_count, + .scan_objects = lazy_rcu_shrink_scan, + .batch = 0, + .seeks = DEFAULT_SEEKS, +}; + void __init rcu_init_nohz(void) { int cpu; @@ -1342,6 +1391,9 @@ void __init rcu_init_nohz(void) if (!rcu_state.nocb_is_setup) return; + if (register_shrinker(&lazy_rcu_shrinker, "rcu-lazy")) + pr_err("Failed to register lazy_rcu shrinker!\n"); + if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) { pr_info("\tNote: kernel parameter 'rcu_nocbs=', 'nohz_full', or 'isolcpus=' contains nonexistent CPUs.\n"); cpumask_and(rcu_nocb_mask, cpu_possible_mask,