From patchwork Thu Jan 5 00:22:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 39181 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp40156wrt; Wed, 4 Jan 2023 16:24:59 -0800 (PST) X-Google-Smtp-Source: AMrXdXu8zQhDC1rj5Ks4+HlzkNCiIrXHh1gImwCytbbyzvxXlEK9NamrS6H0LBfq7UtHsGAamOP6 X-Received: by 2002:a05:6402:203:b0:48e:96d1:63c2 with SMTP id t3-20020a056402020300b0048e96d163c2mr9033222edv.14.1672878299804; Wed, 04 Jan 2023 16:24:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672878299; cv=none; d=google.com; s=arc-20160816; b=MjeWNWCw38VbQnpKun4AmVZJgcO2Iat4/2rwPzviMhR14O2dhSfsmIjufDYw1fAMOY K9Vxm5kj4D0oezHfifDBxEWqeEqt1FHYSW2mu13y/fwdkJh/fb4EJNpAC1qx7nYlpsSe GkkeLFRbJlKSuOXJASOiN3MkHfFhzqEbOeoDZTZyxh8GFWwWopIpI6j9xtmB53buY7ER 4puFvhG7Vpw4/jFIViJoWvgZcsyhDl983YLlws69fXaLFF8SDgbvUT3ql4nwHakU1NI3 ooXPwC9aiAvv/vihdMzRpWFvtD4ns9XUZydPmJZrdzWJbdBANmkhiSUcHjSyjFTHiVwj 71HA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=InhFNa3fSroqFXWKP16u4b6JhjX1O5MirgXuYC3quEg=; b=fxAbvc5T+29KOZ7chusUlfHJCpJwgq0mnapRy6d1e5KW3E9paGr2KGa5LpRHS7TYiu sUa2IQ1qXcwWMvAPgfXIsOYHd+x0/vD0HADlldlPxjRqAXkSLYJ+ENPYZU/F7rTvFABi 9FeIaZTx1ftoeKQS4NUDw1rR70/+FHqs35s2dRBa6HP+RAH0e/qVrZRTz0d6Kkp5zbyW U4hM0pkZr4i3I5lO2AJJQichzdf9YxjuGAQxzdc5W997d1+gmof1dRljFOC5E5WIxwna gjlT+ja6fU+Fyjqkbr6Ak1V9QhHT0mPuF3IHZEBqWDfzJGjreYW/PEbXjC0SWAZYmsbA wiuw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=NQ30BrwZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p29-20020a056402501d00b004614f376c2fsi31451360eda.88.2023.01.04.16.24.34; Wed, 04 Jan 2023 16:24:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=NQ30BrwZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235675AbjAEAXZ (ORCPT + 99 others); Wed, 4 Jan 2023 19:23:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32978 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235383AbjAEAXM (ORCPT ); Wed, 4 Jan 2023 19:23:12 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA7134436F; Wed, 4 Jan 2023 16:23:10 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 239C9B81985; Thu, 5 Jan 2023 00:23:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BE2CAC433EF; Thu, 5 Jan 2023 00:23:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672878187; bh=0ljvs2QOdGuLUAaxiiCnltc5mW3jT5j/qQAFulm3cTg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NQ30BrwZpdHVHwiV8AvKN4qD68IuNfd7pZ3HCV3tSA+xnYTPknssJ03j+KSIwDPRT d1LVu6c34jQAptXnXPtEJ93Qu4i+fagmjDXhWQJLcgvgWkxuuFDYZsVDKcWRDL8+YF LaOExSXR68T4b08uJAQhHjGKNo+nGSqNjkSZJuXZPq/Ksv17G9I0wuk/B9JRHneruR w1rspg+xPwl18VgiiV9bFfXBR1W1+o+BSJnAFDWgMK8jWZBz40eV3kNvnlFLHAZeuN mReSTicA3crOuq/1B2lZafVGei3PAuz9ACzVibvvsi3LYEBtWACDBpj3QbZBETq3+U 7thuX9zkbw0sg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 7327F5C05CA; Wed, 4 Jan 2023 16:23:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Zhao Mengmeng , "Paul E . McKenney" Subject: [PATCH rcu 01/10] rcu: Use hlist_nulls_next_rcu() in hlist_nulls_add_tail_rcu() Date: Wed, 4 Jan 2023 16:22:56 -0800 Message-Id: <20230105002305.1768591-1-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> References: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754140036062654678?= X-GMAIL-MSGID: =?utf-8?q?1754140036062654678?= From: Zhao Mengmeng In commit 8dbd76e79a16 ("tcp/dccp: fix possible race __inet_lookup_established()"), function hlist_nulls_add_tail_rcu() was added back, but the local variable *last* is of type hlist_nulls_node, so use hlist_nulls_next_rcu() instead of hlist_next_rcu(). Signed-off-by: Zhao Mengmeng Signed-off-by: Paul E. McKenney --- include/linux/rculist_nulls.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/rculist_nulls.h b/include/linux/rculist_nulls.h index d8afdb8784c1c..ba4c00dd8005a 100644 --- a/include/linux/rculist_nulls.h +++ b/include/linux/rculist_nulls.h @@ -139,7 +139,7 @@ static inline void hlist_nulls_add_tail_rcu(struct hlist_nulls_node *n, if (last) { n->next = last->next; n->pprev = &last->next; - rcu_assign_pointer(hlist_next_rcu(last), n); + rcu_assign_pointer(hlist_nulls_next_rcu(last), n); } else { hlist_nulls_add_head_rcu(n, h); } From patchwork Thu Jan 5 00:22:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 39186 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp40454wrt; Wed, 4 Jan 2023 16:25:57 -0800 (PST) X-Google-Smtp-Source: AMrXdXtVjWJzP/mjBAezlFHmHiLgDkcU7s1xFET5MV+YEn1yfFvhfEyzqoh+Lgu68pSApTET8Djl X-Received: by 2002:a05:6a20:289f:b0:af:d0d7:9f38 with SMTP id q31-20020a056a20289f00b000afd0d79f38mr53393195pzf.2.1672878357282; Wed, 04 Jan 2023 16:25:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672878357; cv=none; d=google.com; s=arc-20160816; b=wBRCKdICfEwK4Oymou7C3/65a2e5yZNi6GImWbKaoh5OByNsxnn7q++cuTC+Jt11JJ unhkWtGDpNzFpWTrIDcid3SL36/x1PAfnGEvzd7xYuvHjr+DU8JSmFghhqv5LzWahMam Q1v3qfjk+K56f+DiZOGBKUQ2iH6l/K6pgMqe4GU0aMTuCRNbhj9WeKgA7olwTESDQOnw E5BVRB3/518yiig6WraRn4HtYPR0kEuyUpkw/vDU8CfR6nkviHHRO9BfrUA4/u9nCfZf 2M9rg9/YXxIyRgheR7JPNoxKFk+RAtU3+LCp4EZPUUOdl6XXf1hAZNS/Vtpj1Dze51Pb LKyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=qwJKQwJfqz08k0wDSQICy5UGYCLJNhJt0Kpw2quq2PQ=; b=nz6nR1N/3Am3mmowRMeXqdXfGbKqRyL+x/rTIndq+1Kjd6jr+MUMx+qreXQooU1H7j ZE9zKmwyvEWHWverIcE6ABSbaSjHqwdUsUxaeMdDqS57k1xHtGwL+1wQ+/+PfEvbwNM5 eUA0fbx1g/dvcfccfUEAVgOcCtuV9NieVP2s8TYN1Vvn6nEfLFQ/T2qVnbiJWh7+mb3x FqkPjJ6ez5Q7ApETvc6HvVPiQ/73ssjiJ1LQoQj7acawQBhOR2mESN8htpZwt/0scmUz 7J9Uyvf9KRvwrAY2z7bZZiMZEtOqW5JN5ThSK8M9mgYFVE+/SczeoucTWH4eh0R6bT3r KbyA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=SCUL6KbQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v8-20020a63bf08000000b004a5b8385b68si4310455pgf.412.2023.01.04.16.25.44; Wed, 04 Jan 2023 16:25:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=SCUL6KbQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239823AbjAEAXb (ORCPT + 99 others); Wed, 4 Jan 2023 19:23:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235496AbjAEAXN (ORCPT ); Wed, 4 Jan 2023 19:23:13 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA84744372; Wed, 4 Jan 2023 16:23:10 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 1AE0EB8167B; Thu, 5 Jan 2023 00:23:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B908CC433D2; Thu, 5 Jan 2023 00:23:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672878187; bh=RVZbEyTmCniCww65JvMwyLYk21JOsmMrHWf5ALY+E3s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SCUL6KbQmIBZ8VvBA/pmHR6NC0WroSi1i+aHvUdwxTjFNUMIXXOLYQwmoLGKPgo5D mTx7e4rPGU1tHdND6mQdsBo+goFLWeKcSsVJNtZaYdOEnSxq2KMUBy7pdtVjCEvym7 jigU4sRoYWHPTQSjkBqIsP9U7a6vqYs5H2m9r9cToVoC6VlsxqzJwIlcxP0/Uk8H0y vza1fCeaOt+SjlAZCJ+o5Iv2LvGgW7ch7Yr3V/RgXyJIGlvjW/SVep92kDaapMO5M9 weVYXl8YDQcsLa/DSLK+/Dm1NdGYtYUSK5TWK9Y/v+tyGw66wDufDy93M9E3MtoCcX sSt0uFgT5+gGQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 755145C086D; Wed, 4 Jan 2023 16:23:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 02/10] rcu: Consolidate initialization and CPU-hotplug code Date: Wed, 4 Jan 2023 16:22:57 -0800 Message-Id: <20230105002305.1768591-2-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> References: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754140096275176277?= X-GMAIL-MSGID: =?utf-8?q?1754140096275176277?= This commit consolidates the initialization and CPU-hotplug code at the end of kernel/rcu/tree.c. This is strictly a code-motion commit. No functionality has changed. Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 314 +++++++++++++++++++++++----------------------- 1 file changed, 158 insertions(+), 156 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index cf34a961821ad..d3b082233b74b 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -144,14 +144,16 @@ static int rcu_scheduler_fully_active __read_mostly; static void rcu_report_qs_rnp(unsigned long mask, struct rcu_node *rnp, unsigned long gps, unsigned long flags); -static void rcu_init_new_rnp(struct rcu_node *rnp_leaf); -static void rcu_cleanup_dead_rnp(struct rcu_node *rnp_leaf); static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu); static void invoke_rcu_core(void); static void rcu_report_exp_rdp(struct rcu_data *rdp); static void sync_sched_exp_online_cleanup(int cpu); static void check_cb_ovld_locked(struct rcu_data *rdp, struct rcu_node *rnp); static bool rcu_rdp_is_offloaded(struct rcu_data *rdp); +static bool rcu_rdp_cpu_online(struct rcu_data *rdp); +static bool rcu_init_invoked(void); +static void rcu_cleanup_dead_rnp(struct rcu_node *rnp_leaf); +static void rcu_init_new_rnp(struct rcu_node *rnp_leaf); /* * rcuc/rcub/rcuop kthread realtime priority. The "rcuop" @@ -214,27 +216,6 @@ EXPORT_SYMBOL_GPL(rcu_get_gp_kthreads_prio); */ #define PER_RCU_NODE_PERIOD 3 /* Number of grace periods between delays for debugging. */ -/* - * Compute the mask of online CPUs for the specified rcu_node structure. - * This will not be stable unless the rcu_node structure's ->lock is - * held, but the bit corresponding to the current CPU will be stable - * in most contexts. - */ -static unsigned long rcu_rnp_online_cpus(struct rcu_node *rnp) -{ - return READ_ONCE(rnp->qsmaskinitnext); -} - -/* - * Is the CPU corresponding to the specified rcu_data structure online - * from RCU's perspective? This perspective is given by that structure's - * ->qsmaskinitnext field rather than by the global cpu_online_mask. - */ -static bool rcu_rdp_cpu_online(struct rcu_data *rdp) -{ - return !!(rdp->grpmask & rcu_rnp_online_cpus(rdp->mynode)); -} - /* * Return true if an RCU grace period is in progress. The READ_ONCE()s * permit this function to be invoked without holding the root rcu_node @@ -734,46 +715,6 @@ void rcu_request_urgent_qs_task(struct task_struct *t) smp_store_release(per_cpu_ptr(&rcu_data.rcu_urgent_qs, cpu), true); } -#if defined(CONFIG_PROVE_RCU) && defined(CONFIG_HOTPLUG_CPU) - -/* - * Is the current CPU online as far as RCU is concerned? - * - * Disable preemption to avoid false positives that could otherwise - * happen due to the current CPU number being sampled, this task being - * preempted, its old CPU being taken offline, resuming on some other CPU, - * then determining that its old CPU is now offline. - * - * Disable checking if in an NMI handler because we cannot safely - * report errors from NMI handlers anyway. In addition, it is OK to use - * RCU on an offline processor during initial boot, hence the check for - * rcu_scheduler_fully_active. - */ -bool rcu_lockdep_current_cpu_online(void) -{ - struct rcu_data *rdp; - bool ret = false; - - if (in_nmi() || !rcu_scheduler_fully_active) - return true; - preempt_disable_notrace(); - rdp = this_cpu_ptr(&rcu_data); - /* - * Strictly, we care here about the case where the current CPU is - * in rcu_cpu_starting() and thus has an excuse for rdp->grpmask - * not being up to date. So arch_spin_is_locked() might have a - * false positive if it's held by some *other* CPU, but that's - * OK because that just means a false *negative* on the warning. - */ - if (rcu_rdp_cpu_online(rdp) || arch_spin_is_locked(&rcu_state.ofl_lock)) - ret = true; - preempt_enable_notrace(); - return ret; -} -EXPORT_SYMBOL_GPL(rcu_lockdep_current_cpu_online); - -#endif /* #if defined(CONFIG_PROVE_RCU) && defined(CONFIG_HOTPLUG_CPU) */ - /* * When trying to report a quiescent state on behalf of some other CPU, * it is our responsibility to check for and handle potential overflow @@ -1350,13 +1291,6 @@ static void rcu_strict_gp_boundary(void *unused) invoke_rcu_core(); } -// Has rcu_init() been invoked? This is used (for example) to determine -// whether spinlocks may be acquired safely. -static bool rcu_init_invoked(void) -{ - return !!rcu_state.n_online_cpus; -} - // Make the polled API aware of the beginning of a grace period. static void rcu_poll_gp_seq_start(unsigned long *snap) { @@ -2091,92 +2025,6 @@ rcu_check_quiescent_state(struct rcu_data *rdp) rcu_report_qs_rdp(rdp); } -/* - * Near the end of the offline process. Trace the fact that this CPU - * is going offline. - */ -int rcutree_dying_cpu(unsigned int cpu) -{ - bool blkd; - struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); - struct rcu_node *rnp = rdp->mynode; - - if (!IS_ENABLED(CONFIG_HOTPLUG_CPU)) - return 0; - - blkd = !!(READ_ONCE(rnp->qsmask) & rdp->grpmask); - trace_rcu_grace_period(rcu_state.name, READ_ONCE(rnp->gp_seq), - blkd ? TPS("cpuofl-bgp") : TPS("cpuofl")); - return 0; -} - -/* - * All CPUs for the specified rcu_node structure have gone offline, - * and all tasks that were preempted within an RCU read-side critical - * section while running on one of those CPUs have since exited their RCU - * read-side critical section. Some other CPU is reporting this fact with - * the specified rcu_node structure's ->lock held and interrupts disabled. - * This function therefore goes up the tree of rcu_node structures, - * clearing the corresponding bits in the ->qsmaskinit fields. Note that - * the leaf rcu_node structure's ->qsmaskinit field has already been - * updated. - * - * This function does check that the specified rcu_node structure has - * all CPUs offline and no blocked tasks, so it is OK to invoke it - * prematurely. That said, invoking it after the fact will cost you - * a needless lock acquisition. So once it has done its work, don't - * invoke it again. - */ -static void rcu_cleanup_dead_rnp(struct rcu_node *rnp_leaf) -{ - long mask; - struct rcu_node *rnp = rnp_leaf; - - raw_lockdep_assert_held_rcu_node(rnp_leaf); - if (!IS_ENABLED(CONFIG_HOTPLUG_CPU) || - WARN_ON_ONCE(rnp_leaf->qsmaskinit) || - WARN_ON_ONCE(rcu_preempt_has_tasks(rnp_leaf))) - return; - for (;;) { - mask = rnp->grpmask; - rnp = rnp->parent; - if (!rnp) - break; - raw_spin_lock_rcu_node(rnp); /* irqs already disabled. */ - rnp->qsmaskinit &= ~mask; - /* Between grace periods, so better already be zero! */ - WARN_ON_ONCE(rnp->qsmask); - if (rnp->qsmaskinit) { - raw_spin_unlock_rcu_node(rnp); - /* irqs remain disabled. */ - return; - } - raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */ - } -} - -/* - * The CPU has been completely removed, and some other CPU is reporting - * this fact from process context. Do the remainder of the cleanup. - * There can only be one CPU hotplug operation at a time, so no need for - * explicit locking. - */ -int rcutree_dead_cpu(unsigned int cpu) -{ - struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); - struct rcu_node *rnp = rdp->mynode; /* Outgoing CPU's rdp & rnp. */ - - if (!IS_ENABLED(CONFIG_HOTPLUG_CPU)) - return 0; - - WRITE_ONCE(rcu_state.n_online_cpus, rcu_state.n_online_cpus - 1); - /* Adjust any no-longer-needed kthreads. */ - rcu_boost_kthread_setaffinity(rnp, -1); - // Stop-machine done, so allow nohz_full to disable tick. - tick_dep_clear(TICK_DEP_BIT_RCU); - return 0; -} - /* * Invoke any RCU callbacks that have made it to the end of their grace * period. Throttle as specified by rdp->blimit. @@ -4079,6 +3927,160 @@ void rcu_barrier(void) } EXPORT_SYMBOL_GPL(rcu_barrier); +/* + * Compute the mask of online CPUs for the specified rcu_node structure. + * This will not be stable unless the rcu_node structure's ->lock is + * held, but the bit corresponding to the current CPU will be stable + * in most contexts. + */ +static unsigned long rcu_rnp_online_cpus(struct rcu_node *rnp) +{ + return READ_ONCE(rnp->qsmaskinitnext); +} + +/* + * Is the CPU corresponding to the specified rcu_data structure online + * from RCU's perspective? This perspective is given by that structure's + * ->qsmaskinitnext field rather than by the global cpu_online_mask. + */ +static bool rcu_rdp_cpu_online(struct rcu_data *rdp) +{ + return !!(rdp->grpmask & rcu_rnp_online_cpus(rdp->mynode)); +} + +#if defined(CONFIG_PROVE_RCU) && defined(CONFIG_HOTPLUG_CPU) + +/* + * Is the current CPU online as far as RCU is concerned? + * + * Disable preemption to avoid false positives that could otherwise + * happen due to the current CPU number being sampled, this task being + * preempted, its old CPU being taken offline, resuming on some other CPU, + * then determining that its old CPU is now offline. + * + * Disable checking if in an NMI handler because we cannot safely + * report errors from NMI handlers anyway. In addition, it is OK to use + * RCU on an offline processor during initial boot, hence the check for + * rcu_scheduler_fully_active. + */ +bool rcu_lockdep_current_cpu_online(void) +{ + struct rcu_data *rdp; + bool ret = false; + + if (in_nmi() || !rcu_scheduler_fully_active) + return true; + preempt_disable_notrace(); + rdp = this_cpu_ptr(&rcu_data); + /* + * Strictly, we care here about the case where the current CPU is + * in rcu_cpu_starting() and thus has an excuse for rdp->grpmask + * not being up to date. So arch_spin_is_locked() might have a + * false positive if it's held by some *other* CPU, but that's + * OK because that just means a false *negative* on the warning. + */ + if (rcu_rdp_cpu_online(rdp) || arch_spin_is_locked(&rcu_state.ofl_lock)) + ret = true; + preempt_enable_notrace(); + return ret; +} +EXPORT_SYMBOL_GPL(rcu_lockdep_current_cpu_online); + +#endif /* #if defined(CONFIG_PROVE_RCU) && defined(CONFIG_HOTPLUG_CPU) */ + +// Has rcu_init() been invoked? This is used (for example) to determine +// whether spinlocks may be acquired safely. +static bool rcu_init_invoked(void) +{ + return !!rcu_state.n_online_cpus; +} + +/* + * Near the end of the offline process. Trace the fact that this CPU + * is going offline. + */ +int rcutree_dying_cpu(unsigned int cpu) +{ + bool blkd; + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + struct rcu_node *rnp = rdp->mynode; + + if (!IS_ENABLED(CONFIG_HOTPLUG_CPU)) + return 0; + + blkd = !!(READ_ONCE(rnp->qsmask) & rdp->grpmask); + trace_rcu_grace_period(rcu_state.name, READ_ONCE(rnp->gp_seq), + blkd ? TPS("cpuofl-bgp") : TPS("cpuofl")); + return 0; +} + +/* + * All CPUs for the specified rcu_node structure have gone offline, + * and all tasks that were preempted within an RCU read-side critical + * section while running on one of those CPUs have since exited their RCU + * read-side critical section. Some other CPU is reporting this fact with + * the specified rcu_node structure's ->lock held and interrupts disabled. + * This function therefore goes up the tree of rcu_node structures, + * clearing the corresponding bits in the ->qsmaskinit fields. Note that + * the leaf rcu_node structure's ->qsmaskinit field has already been + * updated. + * + * This function does check that the specified rcu_node structure has + * all CPUs offline and no blocked tasks, so it is OK to invoke it + * prematurely. That said, invoking it after the fact will cost you + * a needless lock acquisition. So once it has done its work, don't + * invoke it again. + */ +static void rcu_cleanup_dead_rnp(struct rcu_node *rnp_leaf) +{ + long mask; + struct rcu_node *rnp = rnp_leaf; + + raw_lockdep_assert_held_rcu_node(rnp_leaf); + if (!IS_ENABLED(CONFIG_HOTPLUG_CPU) || + WARN_ON_ONCE(rnp_leaf->qsmaskinit) || + WARN_ON_ONCE(rcu_preempt_has_tasks(rnp_leaf))) + return; + for (;;) { + mask = rnp->grpmask; + rnp = rnp->parent; + if (!rnp) + break; + raw_spin_lock_rcu_node(rnp); /* irqs already disabled. */ + rnp->qsmaskinit &= ~mask; + /* Between grace periods, so better already be zero! */ + WARN_ON_ONCE(rnp->qsmask); + if (rnp->qsmaskinit) { + raw_spin_unlock_rcu_node(rnp); + /* irqs remain disabled. */ + return; + } + raw_spin_unlock_rcu_node(rnp); /* irqs remain disabled. */ + } +} + +/* + * The CPU has been completely removed, and some other CPU is reporting + * this fact from process context. Do the remainder of the cleanup. + * There can only be one CPU hotplug operation at a time, so no need for + * explicit locking. + */ +int rcutree_dead_cpu(unsigned int cpu) +{ + struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); + struct rcu_node *rnp = rdp->mynode; /* Outgoing CPU's rdp & rnp. */ + + if (!IS_ENABLED(CONFIG_HOTPLUG_CPU)) + return 0; + + WRITE_ONCE(rcu_state.n_online_cpus, rcu_state.n_online_cpus - 1); + /* Adjust any no-longer-needed kthreads. */ + rcu_boost_kthread_setaffinity(rnp, -1); + // Stop-machine done, so allow nohz_full to disable tick. + tick_dep_clear(TICK_DEP_BIT_RCU); + return 0; +} + /* * Propagate ->qsinitmask bits up the rcu_node tree to account for the * first CPU in a given leaf rcu_node structure coming online. The caller From patchwork Thu Jan 5 00:22:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 39187 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp40856wrt; Wed, 4 Jan 2023 16:27:05 -0800 (PST) X-Google-Smtp-Source: AMrXdXsNkznLfkTNo+l1UfotDGRZLdEvRfDgY2/81eRobMxhpNLxnPPXp8+pRIpCDWLUMM2r7Hd7 X-Received: by 2002:a05:6a20:d695:b0:a2:c1f4:3c70 with SMTP id it21-20020a056a20d69500b000a2c1f43c70mr59574528pzb.8.1672878424997; Wed, 04 Jan 2023 16:27:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672878424; cv=none; d=google.com; s=arc-20160816; b=fo62tZ1nzQTlHjVLHlEvMqx9VPQCvwHVUzm6Kk7G9RaBzDjIiEmejiV5WAvCD/BLzh ipMNQHqj6Uop3a9OWd2w8g9c28TS0WLJ0XjDtXYZrR9bcpeos2/v3wTl8KRsmG0r1+gz sDELDW2AjlOJ0VRhSPuxHfbbyK8MqMM10UPUaZGMUfygDyTty5Wgn0Ry+shEwt1o9A45 AA21eiHTAp20Z14JO+9zvG81ZGNNrTawpFtn0ylZlXUN+EcCingo5VvcxedAfmuLuBLE TPevn1p9xtl4qFWtRpL+makyy7DLvH1vg5xBJzf97r1wgOd3EIocK9jAqT0Mw0QEmn+g Qmaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=IahGQLcF4unRBMGotMZ2SCMRxJ4Q7z3kHfv9HZcvFBo=; b=B6KQW6ypZamfeM960VJBWCS0JBZDsi2kX9NnJ/CcCVIaGpZ2cgCsYix8gFPMBn256L DksoTAugn8009ea0rI0SjOnk1VYpdGNDoprowzEFTYlI1hurRYVFSwkN93MNIeN1Gh+5 1gxV66PnSRLaPO1DYlCkHqaS7zxQw+mxfqEMhXBN9FGGKWYONRxFbboMAenqvD+Nk9jz JQKNURojIQmSqqOQLRrNklRv8t0a+no5y+iCi/tuTZ3w5w4otR3BEiSZGlRXBTQwZXji +9Or8Uk5YegVVX+zo2IKhwdjHhbsCQlmy+sDwJayJRzMXEA1i+RKlYDXWSuDtxTxcucg KqFA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=hpVqoG0S; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n6-20020a654506000000b00478446557b9si36126055pgq.677.2023.01.04.16.26.52; Wed, 04 Jan 2023 16:27:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=hpVqoG0S; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235499AbjAEAXm (ORCPT + 99 others); Wed, 4 Jan 2023 19:23:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32854 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235516AbjAEAXN (ORCPT ); Wed, 4 Jan 2023 19:23:13 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AAA4044375; Wed, 4 Jan 2023 16:23:10 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 281B6B81987; Thu, 5 Jan 2023 00:23:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C0088C433F1; Thu, 5 Jan 2023 00:23:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672878187; bh=/ECD69jxcNYYwEATRT9YBzkdJrzTS32gHvcq9AJnndw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hpVqoG0SK7CXriUdM8F38SKoj35jgBFggQkLZaV6ufIyNqjzg43BGlmAO1vjkW3Xu p6MDNItrdzIYkideWsLk8WNQnvME6pEvPFc48WeLg/a8BpwEj++b9zoYql7/8Y4p0P KeJIOK2D7qRqMmsncrUmMlBFDlsKE+gP/KnpEsxsQAAgvqzGajZtPl0YRblraWwx5x kfoduOq4aEAYOv3iNyWRySbznB4eGJTilik0beQPzXhXio78iuM7VxnSV2XN282fN0 li+3Ntqeo2uPkUJnLShuPPWyPfjfrE9hlsd2s30O6OFLurnP4ESyfD5js1LziGuD11 iKu/42ZLAC2EA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 779BB5C08E5; Wed, 4 Jan 2023 16:23:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 03/10] rcu: Throttle callback invocation based on number of ready callbacks Date: Wed, 4 Jan 2023 16:22:58 -0800 Message-Id: <20230105002305.1768591-3-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> References: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754140166939152365?= X-GMAIL-MSGID: =?utf-8?q?1754140166939152365?= Currently, rcu_do_batch() sizes its batches based on the total number of callbacks in the callback list. This can result in some strange choices, for example, if there was 12,800 callbacks in the list, but only 200 were ready to invoke, RCU would invoke 100 at a time (12,800 shifted down by seven bits). A more measured approach would use the number that were actually ready to invoke, an approach that has become feasible only recently given the per-segment ->seglen counts in ->cblist. This commit therefore bases the batch limit on the number of callbacks ready to invoke instead of on the total number of callbacks. Signed-off-by: Paul E. McKenney --- kernel/rcu/rcu_segcblist.c | 2 +- kernel/rcu/rcu_segcblist.h | 2 ++ kernel/rcu/tree.c | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/rcu_segcblist.c b/kernel/rcu/rcu_segcblist.c index c54ea2b6a36bc..f71fac422c8f6 100644 --- a/kernel/rcu/rcu_segcblist.c +++ b/kernel/rcu/rcu_segcblist.c @@ -89,7 +89,7 @@ static void rcu_segcblist_set_len(struct rcu_segcblist *rsclp, long v) } /* Get the length of a segment of the rcu_segcblist structure. */ -static long rcu_segcblist_get_seglen(struct rcu_segcblist *rsclp, int seg) +long rcu_segcblist_get_seglen(struct rcu_segcblist *rsclp, int seg) { return READ_ONCE(rsclp->seglen[seg]); } diff --git a/kernel/rcu/rcu_segcblist.h b/kernel/rcu/rcu_segcblist.h index 431cee212467d..4fe877f5f6540 100644 --- a/kernel/rcu/rcu_segcblist.h +++ b/kernel/rcu/rcu_segcblist.h @@ -15,6 +15,8 @@ static inline long rcu_cblist_n_cbs(struct rcu_cblist *rclp) return READ_ONCE(rclp->len); } +long rcu_segcblist_get_seglen(struct rcu_segcblist *rsclp, int seg); + /* Return number of callbacks in segmented callback list by summing seglen. */ long rcu_segcblist_n_segment_cbs(struct rcu_segcblist *rsclp); diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index d3b082233b74b..7d3a59d4f37ef 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2057,7 +2057,7 @@ static void rcu_do_batch(struct rcu_data *rdp) */ rcu_nocb_lock_irqsave(rdp, flags); WARN_ON_ONCE(cpu_is_offline(smp_processor_id())); - pending = rcu_segcblist_n_cbs(&rdp->cblist); + pending = rcu_segcblist_get_seglen(&rdp->cblist, RCU_DONE_TAIL); div = READ_ONCE(rcu_divisor); div = div < 0 ? 7 : div > sizeof(long) * 8 - 2 ? sizeof(long) * 8 - 2 : div; bl = max(rdp->blimit, pending >> div); From patchwork Thu Jan 5 00:22:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 39185 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp40401wrt; Wed, 4 Jan 2023 16:25:45 -0800 (PST) X-Google-Smtp-Source: AMrXdXu9iERm0VmX4t9b6fW7HcEUSd5SWGT1OidheL6SCL/YZu06CInwlihUThCTRQnRQIo+Jdbg X-Received: by 2002:a05:6a20:3b1d:b0:9d:efbe:a116 with SMTP id c29-20020a056a203b1d00b0009defbea116mr49334205pzh.38.1672878345666; Wed, 04 Jan 2023 16:25:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672878345; cv=none; d=google.com; s=arc-20160816; b=gjHB2ndpY2rqDE9mSdWPuM/VTzX25OtEseVPz8m49IXyoRThBzHZDjeQMVadqI/YsG 35vjwW903SAr9SvfhJWOz1Ql+2vWUG+w2Rsd9WkYwIgPYO8l+dxmI1u7VY6TxuQGZQ7+ sm+qvxqH+472yR/WyS6vrDH5Eg6gDSmLp/JAnEyfcTEFLxdCNI8oRZcAo+1Ef9rPJXKt /pomaLvSFPleq5VO6Xd1opLk2UQA+wh1XMu7OIKhIZh5bin5jfsxe15nE93VRESUm2Of RpmjRdNhtIrvLIQNirJPBv3Y8T6CLTH9ZM8aYZ8hBmxN2xoKhCnAOqLV9JxDyQ24BXBz r5WQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KbyAhZ/6PQSJFGzON275MfoYsdUyqX1iTjcRke7/b10=; b=XnYwYu6benHP54vliOY2mSyJmYtwaoyaMgXXoNBTNngtr0HTa17B9mZTD0XmTC/0G8 kqEUU2wRidwt/ZdCqxqpmdL6/LNkgcndmc//2+K9f7rWkwZ2Keh2mhVmeY+cDxpt7+Ag NRNIb2PsD6dGQ8QKeQyM1DAYxMQd06OJbfRLxd2kNhwsB00/97q78cQMqfTEEIW3IxX2 VP1BSxAryuDOmGDR+c9tozitPVLZDAnoHXFyWMlK1p8GJj+u8izkh+b6VynjEwF1xc6w 0pKgtPCKTMIqo7Z0CKxh4N+elsoyUgyVsYG015VIVGr0/PYfOLNk6tCTI9SmXXF6LfSu EHXQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=sGVTI7Qg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q202-20020a632ad3000000b00480a937f897si38992017pgq.650.2023.01.04.16.25.33; Wed, 04 Jan 2023 16:25:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=sGVTI7Qg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235716AbjAEAX1 (ORCPT + 99 others); Wed, 4 Jan 2023 19:23:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32976 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235470AbjAEAXM (ORCPT ); Wed, 4 Jan 2023 19:23:12 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA9A544374; Wed, 4 Jan 2023 16:23:10 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 1CD61B81984; Thu, 5 Jan 2023 00:23:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C1F92C433F0; Thu, 5 Jan 2023 00:23:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672878187; bh=ztl1EQiOdDcQZcotzokX82t3kWGoj7LXDHlcvmlwCU8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sGVTI7Qg58lsUbW1BNkoKK45kvR4sdZ/a992D/99AGedvCnCZSIqHj4dbrB7gnw2l wQSEtk02ZpdrfHG2cCe+xtrc2CxkeaWrNFn76ZXgMjlsjAMD7625j5J3Cs93HYMTf+ p1fXfTSkTccAHr/SeBda7YtysoiWMzbwV24TmBbB0rS/fuZSgelcByU7IhoZwjHy4L aPQlcqJ/0fplKWR7RZaitdz806uNp3VcWLLkmb9S4D4xiCDY2uB8jFJAT35DJxD8TK taQI2dt/DzcUdVu0P47i7F2WZKu0tWegdzU3+a16S55bvhK+SipdfWVQZu23w0x+AO lJplReIpqI9yA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 79AB35C1456; Wed, 4 Jan 2023 16:23:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 04/10] rcu: Upgrade header comment for poll_state_synchronize_rcu() Date: Wed, 4 Jan 2023 16:22:59 -0800 Message-Id: <20230105002305.1768591-4-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> References: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754140083750874498?= X-GMAIL-MSGID: =?utf-8?q?1754140083750874498?= This commit emphasizes the possibility of concurrent calls to synchronize_rcu() and synchronize_rcu_expedited() causing one or the other of the two grace periods being lost from the viewpoint of poll_state_synchronize_rcu(). If you cannot afford to lose grace periods this way, you should instead use the _full() variants of the polled RCU API, for example, poll_state_synchronize_rcu_full(). Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 7d3a59d4f37ef..0147e69ea85a9 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3559,7 +3559,9 @@ EXPORT_SYMBOL_GPL(start_poll_synchronize_rcu_full); * If @false is returned, it is the caller's responsibility to invoke this * function later on until it does return @true. Alternatively, the caller * can explicitly wait for a grace period, for example, by passing @oldstate - * to cond_synchronize_rcu() or by directly invoking synchronize_rcu(). + * to either cond_synchronize_rcu() or cond_synchronize_rcu_expedited() + * on the one hand or by directly invoking either synchronize_rcu() or + * synchronize_rcu_expedited() on the other. * * Yes, this function does not take counter wrap into account. * But counter wrap is harmless. If the counter wraps, we have waited for @@ -3570,6 +3572,12 @@ EXPORT_SYMBOL_GPL(start_poll_synchronize_rcu_full); * completed. Alternatively, they can use get_completed_synchronize_rcu() * to get a guaranteed-completed grace-period state. * + * In addition, because oldstate compresses the grace-period state for + * both normal and expedited grace periods into a single unsigned long, + * it can miss a grace period when synchronize_rcu() runs concurrently + * with synchronize_rcu_expedited(). If this is unacceptable, please + * instead use the _full() variant of these polling APIs. + * * This function provides the same memory-ordering guarantees that * would be provided by a synchronize_rcu() that was invoked at the call * to the function that provided @oldstate, and that returned at the end From patchwork Thu Jan 5 00:23:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 39182 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp40199wrt; Wed, 4 Jan 2023 16:25:05 -0800 (PST) X-Google-Smtp-Source: AMrXdXvMjKdnG4SNd7iQ5fcqXPT/RwINO8+Xq/RAkdE1QAN9DPqNcScQSwN3bV5zzAh+R3s9z8ZJ X-Received: by 2002:a17:907:9a85:b0:81b:fc79:be51 with SMTP id km5-20020a1709079a8500b0081bfc79be51mr44260203ejc.50.1672878305810; Wed, 04 Jan 2023 16:25:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672878305; cv=none; d=google.com; s=arc-20160816; b=JzC7ekhGoeGI0JFL0PlBeif6j7devrSExOv9Ztktq6jBMQk+lSjyzRY444SvC8lKOc Ejj8NEX4akHiSCdRY8To5bZr/gDmG5rDns5QgNVT6q2Qmxt+MtpzDBhtP5vzyxaP13ot E1Hx6JIEGudZrOc+1mxQsgCeR0Pdu/1yWaZuLFXPjc+bd2jFlYwUIRy7QFYYtv2QGkDj aHch9BPkPGdYtU2kLlry+jei29eeZ7w/o0q7hmseP9B7NX59rauYTzIB7jPEReNBd9p/ 1ELDtS98nH8gVO88MY9He838u5sgZz+APdF7f19Vo7cRMiZO9b8zZc9NBfTmH+nnioWd /Q7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=e9Gv/5dM21dmAIUxzuKNn71hP2ohWxpJ4xgg/0HGsHY=; b=U9SvOq2b5v7V92eYJ+qPAiKICxEW/Li+PLF0ZjQjo8xOoX7E6MKqd1NP3WUjpFv6PD yM+X/1eqCgFENE29YdgoWD3RtDkB70urPiz2P6IE3ajPj3j+PeOQfO9XKy+rR1W7vy2E Dcd8gd2qWL35DRDKsyJ00XYCQDV+Ztufekz++JljHr6Pp2C+r5OkcC4d9RxPb7Xjrd8r /Z88fyomNVTiAJY1P5PbauHgorlCYFYtj6bQHaN146ruxLlGaORG/eVta5yupRJn5exl mpi/6dYWBdZWPg4ujknp/nQ+t1/6CKwpYOghGTypA+bHnk6L+e4Bthr7GsVxIfEJxmph tAIQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=lO+DcO1D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qf11-20020a1709077f0b00b008401cc1a665si32992553ejc.259.2023.01.04.16.24.39; Wed, 04 Jan 2023 16:25:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=lO+DcO1D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240112AbjAEAXf (ORCPT + 99 others); Wed, 4 Jan 2023 19:23:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235393AbjAEAXN (ORCPT ); Wed, 4 Jan 2023 19:23:13 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA04044368; Wed, 4 Jan 2023 16:23:10 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 3C677B81983; Thu, 5 Jan 2023 00:23:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C3BD8C43392; Thu, 5 Jan 2023 00:23:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672878187; bh=hrYxt2M86I+YLxkqBWLEpmhGnP0U4qusfW3DVwJ922w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lO+DcO1DQJWobdC4TYtmZNiT4JyEMz7vzZ2AszirxPy8b3kCcyCawRA1PIcMj+mvo dsEttD17K3MEfR5RbFKzJAOXwCtFQeWRE9H9SQSiK+7E69bJolQxvUoX6La7NVx88E vgIJ53KJz/DFMFbSd+QbAUWujYZDqGr4oKmtBDyXoX0Ma1V7n/scnXiYsSu1ndyEBc uGqVzi9+Qwnx0cCJ2fXKlM41Hii2qNSKmCkaCE21nUsVUrIuWYwTxZMfuiJwrEssiv a3tCWnViVumHh4vfJpmZ+mAOQXumqEnmtK9MQc3Vw3f6BErUtyufnjg9Zm7m5+qQYc ACW0BZvV+6vAg== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 7BB955C149B; Wed, 4 Jan 2023 16:23:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , Masami Hiramatsu , Mathieu Desnoyers , Boqun Feng , Matthew Wilcox , Thomas Gleixner Subject: [PATCH rcu 05/10] rcu: Make RCU_LOCKDEP_WARN() avoid early lockdep checks Date: Wed, 4 Jan 2023 16:23:00 -0800 Message-Id: <20230105002305.1768591-5-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> References: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754140042591838643?= X-GMAIL-MSGID: =?utf-8?q?1754140042591838643?= Currently, RCU_LOCKDEP_WARN() checks the condition before checking to see if lockdep is still enabled. This is necessary to avoid the false-positive splats fixed by commit 3066820034b5dd ("rcu: Reject RCU_LOCKDEP_WARN() false positives"). However, the current state can result in false-positive splats during early boot before lockdep is fully initialized. This commit therefore checks debug_lockdep_rcu_enabled() both before and after checking the condition, thus avoiding both sets of false-positive error reports. Reported-by: Steven Rostedt Reported-by: Masami Hiramatsu (Google) Reported-by: Mathieu Desnoyers Signed-off-by: Paul E. McKenney Reviewed-by: Mathieu Desnoyers Cc: Boqun Feng Cc: Matthew Wilcox Cc: Thomas Gleixner --- include/linux/rcupdate.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 03abf883a281b..aa86de01aab6d 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -374,11 +374,18 @@ static inline int debug_lockdep_rcu_enabled(void) * RCU_LOCKDEP_WARN - emit lockdep splat if specified condition is met * @c: condition to check * @s: informative message + * + * This checks debug_lockdep_rcu_enabled() before checking (c) to + * prevent early boot splats due to lockdep not yet being initialized, + * and rechecks it after checking (c) to prevent false-positive splats + * due to races with lockdep being disabled. See commit 3066820034b5dd + * ("rcu: Reject RCU_LOCKDEP_WARN() false positives") for more detail. */ #define RCU_LOCKDEP_WARN(c, s) \ do { \ static bool __section(".data.unlikely") __warned; \ - if ((c) && debug_lockdep_rcu_enabled() && !__warned) { \ + if (debug_lockdep_rcu_enabled() && (c) && \ + debug_lockdep_rcu_enabled() && !__warned) { \ __warned = true; \ lockdep_rcu_suspicious(__FILE__, __LINE__, s); \ } \ From patchwork Thu Jan 5 00:23:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 39179 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp39882wrt; Wed, 4 Jan 2023 16:24:16 -0800 (PST) X-Google-Smtp-Source: AMrXdXutFpj6OHygpATYAKvB89EhWpDWaFY5XBDgxm2VxQE17VV1Tuq2ey4pi33+bNyXBHW1kUZ2 X-Received: by 2002:a17:907:c24e:b0:7c0:1db5:ea10 with SMTP id tj14-20020a170907c24e00b007c01db5ea10mr55640447ejc.53.1672878256010; Wed, 04 Jan 2023 16:24:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672878255; cv=none; d=google.com; s=arc-20160816; b=TI93D84eyNcWbRAYfyDy2sTPP5WJ2FCyxb3tOgD3c2PvK5VVEQJMGNjyFgcOVlHLd4 G2D0Uip5dizNntQ7MYxnc9hlQddFBZ/P/bwM2vTRfGXF3HQMngzHgGq4D3J0086X1Bxu ZO2xnTLuJwwqei/FJAI/b6q2ZNND3o/iBbQB71ydPgF+ST9vik+6qf7dr7ZyPJod9YOO u3RevI9fQEKS3XWWzBzDi7DtjEdwG9R6yYBGWGPL6Da9A4fHL/XlF1896a1EuyusiTuz 7GhfVNpr5jBHbeRQjB/d/oe0rbuO/abaXOiLUrtTSVTrTX0kQBY8LnRDeHnq2kUGvJW/ CTEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=239eL31BoO4eH9HOzJw04BakYiSF8/16uZSrwMMtazA=; b=BC0KP3yCpn4Mgd2dRnKnZW4yzhuk9qTSipz3+Mf5fszW3Ac46EmAJKBkNcoznQMkk0 0wqLvKymQf4yHarCuQtZCuAIxGrAjmwdWQQ19FIMMXUZYCvVTUJ9Z487UVyXhR/z+4/R +Va/67o98UerBHhB+ICShLkZVYdN5KYoJbQL/01E0UcKm3/5SwOo5SQLqXVP9x9JQYzk 416/d19H00ZAsP/dETsyxaB4LBSlAmDW8qj7Lj5L99JsVse25HhX3RUMk7WRM4zAd8Jm m6QCGQ1hBHVOxN79voCeSQKdvDeXzvCtu9msqL6vcDiwaYdj39sOx+rqxgIfRDoh2R69 bNmA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=BlfqUVD6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id dp22-20020a170906c15600b007c01839ded7si31769237ejc.993.2023.01.04.16.23.52; Wed, 04 Jan 2023 16:24:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=BlfqUVD6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235491AbjAEAXT (ORCPT + 99 others); Wed, 4 Jan 2023 19:23:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235468AbjAEAXL (ORCPT ); Wed, 4 Jan 2023 19:23:11 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3BA624434B; Wed, 4 Jan 2023 16:23:10 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C9F7561899; Thu, 5 Jan 2023 00:23:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1DFFDC433A0; Thu, 5 Jan 2023 00:23:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672878188; bh=fNTH5yIpqQhqXPGvfulmhhwbllvc7TzDs+475c3ZbT8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BlfqUVD6Nyn+zgTpC7gS9OqRe0xIO2vnskDN1tauZgcwdLjXC/aW2YOwqEsdDrRKc 3vszS3QAsEe72ALkbfzcb6FiYrV2MzeBiQ7kEi8lO/4a/fpsBOWIU4UlG9JyxzZrWZ Alz4M5qAmard7X4dyG/UvR9cv3QJuycbHL+gNBm6+ffs6FLID9zBA/XLDSuahmL8Lt dqN2FIl9gMmeq7aoYqf3Ot+p3oNeSisqRHj0ZYWdA4khhQy1+5C3mhnSCv6LhJURCS 6ZgxwXYNKHCnw31MqJfbi7+QLIS38CfsplWh9AiwpK//d4jRGjSCPoVDUAKSHnTbHx Y5fZV0hjER1+w== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 7D8F85C1ADF; Wed, 4 Jan 2023 16:23:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 06/10] rcu: Suppress smp_processor_id() complaint in synchronize_rcu_expedited_wait() Date: Wed, 4 Jan 2023 16:23:01 -0800 Message-Id: <20230105002305.1768591-6-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> References: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754139989815197476?= X-GMAIL-MSGID: =?utf-8?q?1754139989815197476?= The normal grace period's RCU CPU stall warnings are invoked from the scheduling-clock interrupt handler, and can thus invoke smp_processor_id() with impunity, which allows them to directly invoke dump_cpu_task(). In contrast, the expedited grace period's RCU CPU stall warnings are invoked from process context, which causes the dump_cpu_task() function's calls to smp_processor_id() to complain bitterly in debug kernels. This commit therefore causes synchronize_rcu_expedited_wait() to disable preemption around its call to dump_cpu_task(). Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_exp.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index ed6c3cce28f23..927abaf6c822e 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -667,7 +667,9 @@ static void synchronize_rcu_expedited_wait(void) mask = leaf_node_cpu_bit(rnp, cpu); if (!(READ_ONCE(rnp->expmask) & mask)) continue; + preempt_disable(); // For smp_processor_id() in dump_cpu_task(). dump_cpu_task(cpu); + preempt_enable(); } } jiffies_stall = 3 * rcu_exp_jiffies_till_stall_check() + 3; From patchwork Thu Jan 5 00:23:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 39184 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp40337wrt; Wed, 4 Jan 2023 16:25:34 -0800 (PST) X-Google-Smtp-Source: AMrXdXs/Sqij7gy4WowTyhckM3HeNFYU1n5j3pCnEeNmYtLGbGAldW/ztr68Yj62WkXQ0tlLeFFU X-Received: by 2002:a17:902:ead4:b0:189:c429:f615 with SMTP id p20-20020a170902ead400b00189c429f615mr49718522pld.1.1672878334163; Wed, 04 Jan 2023 16:25:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672878334; cv=none; d=google.com; s=arc-20160816; b=ptA8unbP34IEjsypL3Zd8Abn2a8Blx0wRvH9DhTJTAUu/CBE04t84r1i+WXING7Vr5 ETWU5afWTMNS4WTtnCPodgxkwUB1WtyHnYhE/s84RFzFxZ9EpwGzT6npy2VwblUxtkli FjV//s8SU6N+j+a/CDN1KBHe7Tsg85Ha6mkmEB0KFFQlgzuQT3Mj0Ii03ftG52ffvLDO Fw/itFKOMzOzi7THjQzccWQU7ysCtfSWpCuPHZqt/IZgmfJTrOevMHJ+qz+aDfBw10BV zXgSa19zy/Ivwj9C+sg5ezKYFS1wgtz6+Yf1eVSai+n1m7hAd2iIK0tmiR+Lr7KmAJJ/ qGZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Fxs+PTC10J4EG1Yb2+9bUE9JCYj2rzRD424tEqIfhB0=; b=x4tJC7Ks564MGOAdUhrCVcA6+cNEUzicB+56qhXvJSfuTcMjjV/Sr9eWKOvQIoUq1F LyF9AqNrKvl3Sd9gjsNNSHzLG/XhP5TxQVdy9ofGpoRN8nl0T+NzzDmJlsdMYz1w1qdZ IbPukHd5fiDe/p000M39cC7cMhi1sZ0MfAE7k8aAMbzLycQcxX49k7CWZmBI/EcwpVKi lfYpECdwWNDr4ecpiY56/oytLKd88UvAuceU4WuNg1oCyotyIJz5QT/0MZULJr2DJ7PI Vv1cKye9PhAUGfx7o/Vevfr7DAORvjyGcdbyUOqspRnsIr6bCFjr9fF4+3M+CRmPQpex G0gQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=GPKGOamf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e6-20020a170903240600b0018994e35cbfsi35764175plo.364.2023.01.04.16.25.21; Wed, 04 Jan 2023 16:25:34 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=GPKGOamf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240296AbjAEAXt (ORCPT + 99 others); Wed, 4 Jan 2023 19:23:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235437AbjAEAXP (ORCPT ); Wed, 4 Jan 2023 19:23:15 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE25D4435A; Wed, 4 Jan 2023 16:23:13 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 7CE8AB81982; Thu, 5 Jan 2023 00:23:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1E02FC433A4; Thu, 5 Jan 2023 00:23:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672878188; bh=2WSJ8n/mVkUaqRO+2OU4uVBj5A9i4K7XPBfFkEnpujQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GPKGOamfbyj/w6a6jbRFZwcg+TgZGWBmQ0+ZtF5mmm9iETuuYIIXFn89wPPPupKtV A/KwX+0ZZCcRtmqQi2QN/WjEkyAqCwkMxnd/jDwdo9mkAJlK/YBPH8t+xhaAiK2tIL HGLaWf+om9DQsYS1jREDI2jqwkZX8YcV9tucW61XOq2RvCZz7eAK+XDE+yfkBFG5rQ B3PzYb/nsPPDop+EZ+DR5+Y7T9MQmWAye2coVn+SrNELR7O5ebEOQhKS4f0vy5q70U 4fbvSSxTtmFu27iqzWxQrhG7Qg5bT41Kg3l+cYL6nYe028t/yjWIVKccq3zECCjubv ygU8w6/RzQdTw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 7F4CE5C1AE0; Wed, 4 Jan 2023 16:23:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Zqiang , "Paul E . McKenney" Subject: [PATCH rcu 07/10] rcu: Make rcu_blocking_is_gp() stop early-boot might_sleep() Date: Wed, 4 Jan 2023 16:23:02 -0800 Message-Id: <20230105002305.1768591-7-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> References: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754140072060788976?= X-GMAIL-MSGID: =?utf-8?q?1754140072060788976?= From: Zqiang Currently, rcu_blocking_is_gp() invokes might_sleep() even during early boot when interrupts are disabled and before the scheduler is scheduling. This is at best an accident waiting to happen. Therefore, this commit moves that might_sleep() under an rcu_scheduler_active check in order to ensure that might_sleep() is not invoked unless sleeping might actually happen. Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 0147e69ea85a9..15f9765064727 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3333,9 +3333,10 @@ void __init kfree_rcu_scheduler_running(void) */ static int rcu_blocking_is_gp(void) { - if (rcu_scheduler_active != RCU_SCHEDULER_INACTIVE) + if (rcu_scheduler_active != RCU_SCHEDULER_INACTIVE) { + might_sleep(); return false; - might_sleep(); /* Check for RCU read-side critical section. */ + } return true; } From patchwork Thu Jan 5 00:23:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 39180 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp39935wrt; Wed, 4 Jan 2023 16:24:26 -0800 (PST) X-Google-Smtp-Source: AMrXdXvYnSBIVH9/xVeHvTooD3XFRLJ2WNj2Ff8EehxapWJ0998n5VOmhDpSMvm6xYbyjkxY4/em X-Received: by 2002:a17:906:524b:b0:7c1:5b5e:4d78 with SMTP id y11-20020a170906524b00b007c15b5e4d78mr44290283ejm.51.1672878266227; Wed, 04 Jan 2023 16:24:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672878266; cv=none; d=google.com; s=arc-20160816; b=vxX5tOufWe3rSD7+J4pz3Jms/VuWZ1Ug/0CaHpSlGGxJxRIJuvERDepUg4jyoda0XK v/X9MSqKHs4XQ1PHJCPya5EzzrwE3dsD4Vq3TZ1iaB64Zslt3u0pI/xjtgAybliJ0G1/ LRTUVgTMPrv7iyofkF6hW3Q1A251JFXZ9EGwd8QDPHhP3EkRmQreRN12c3llctIMYnqJ tx3mYCZrvO0HQloJHdjMcrc4Z/pxW4Qb+TuR2A3y9Q835gSvOiQgn6MG8WnHF1j685jX yUpTht3irFTPVZVMi44SsSV4895DWUK5lagWcyi7TJA+6uvkbNutQiFmr3EdZDHPWvx0 F8QA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Dh1Taj1LoeyHXHTzT0TJPo66asWpO6nGlJnjo9qrG/g=; b=MDZvXubs7lYIrG5T8lHBUOgpUOJ54JysJX0C/lMIvJPyyH0lJBawktieA0V1I7bOoS l4DarBYZ41AEBFhZNeVBgiTBtSBc85CLyD+q+kXUAlrKmkoZ0fscCWznfvZVTglDLmYD YJ/7/Yj5Pp8V8kN0LqlbGJx+bLB2KGicONIeqsy3cZu0u7pJzwp8TMC44U9/Q9hW3Kk+ pxjiipb/cBc/gGPwQD7xnPa3D+f/ImcFERi+JsgNdKEm1sL9dO17lLyqoFTGEtl6OJ9D 6eLb3DFpx3jvp5tnHJwWgLWYguiP1e1foDxWTQYSjNSoBZwPzcOxc1bFi2W/qm3BXqcF Csrg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=FQS48QKD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id xh13-20020a170906da8d00b0084c4784684bsi24727295ejb.791.2023.01.04.16.24.02; Wed, 04 Jan 2023 16:24:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=FQS48QKD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235520AbjAEAXV (ORCPT + 99 others); Wed, 4 Jan 2023 19:23:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32968 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235064AbjAEAXM (ORCPT ); Wed, 4 Jan 2023 19:23:12 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E06A43A2E; Wed, 4 Jan 2023 16:23:10 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id DE8036189A; Thu, 5 Jan 2023 00:23:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 22C74C433AA; Thu, 5 Jan 2023 00:23:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672878188; bh=cQj+0MvK2dvCMDfEUq3ufEL/MhEK6ftbDWfUx6TMnEo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FQS48QKDf2Lpmmy3HwQPQyH0ez2nI90pzgXA2IgpC2we2flhJ64kQbkiKafaxswN6 PK5RUgoFwh26PWhWgqg2pFpPqDrMnOse8qPffSTUox8YCJ/pPLNzu2y4iRkO7eGwAc APXTqmI9J3IcjmoQkN2ofiUz3bgGmFwSSGbCi+Yaa1slUFQXEtDL+GRgSwlZYwiGCA qXKgxhtKd4fm+MAJbZbOsQVbJJC39h2IDrVww6Hkf0ubqsAxucBhh6jLyhrdc9O9JO XL8SNCExQiBgCbrnqZMx6u3ZQxZreuv4K7xY1RKEkkGy2njbVC9nj6CMzQo8FmUjWt pQL8dnm0J8Y3A== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 811CF5C1C5B; Wed, 4 Jan 2023 16:23:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" Subject: [PATCH rcu 08/10] rcu: Test synchronous RCU grace periods at the end of rcu_init() Date: Wed, 4 Jan 2023 16:23:03 -0800 Message-Id: <20230105002305.1768591-8-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> References: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754140000913551494?= X-GMAIL-MSGID: =?utf-8?q?1754140000913551494?= This commit tests synchronize_rcu() and synchronize_rcu_expedited() at the end of rcu_init(), in addition to the test already at the beginning of that function. These tests are run only in kernels built with CONFIG_PROVE_RCU=y. Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 2 ++ kernel/rcu/update.c | 1 + 2 files changed, 3 insertions(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 15f9765064727..80b84ae285b41 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4849,6 +4849,8 @@ void __init rcu_init(void) // Kick-start any polled grace periods that started early. if (!(per_cpu_ptr(&rcu_data, cpu)->mynode->exp_seq_poll_rq & 0x1)) (void)start_poll_synchronize_rcu_expedited(); + + rcu_test_sync_prims(); } #include "tree_stall.h" diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c index f5e6a2f95a2a0..587b97c401914 100644 --- a/kernel/rcu/update.c +++ b/kernel/rcu/update.c @@ -220,6 +220,7 @@ void rcu_test_sync_prims(void) { if (!IS_ENABLED(CONFIG_PROVE_RCU)) return; + pr_info("Running RCU synchronous self tests\n"); synchronize_rcu(); synchronize_rcu_expedited(); } From patchwork Thu Jan 5 00:23:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 39195 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp43071wrt; Wed, 4 Jan 2023 16:33:30 -0800 (PST) X-Google-Smtp-Source: AMrXdXvVGlC0t50E/P7jkuK5i4TI2gvrZLc1PjXpXeRIcddDL9xaUu47RAhYjLy8ZCLp1lpGtg8g X-Received: by 2002:a17:903:3043:b0:189:7548:20a7 with SMTP id u3-20020a170903304300b00189754820a7mr33638531pla.56.1672878809867; Wed, 04 Jan 2023 16:33:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672878809; cv=none; d=google.com; s=arc-20160816; b=VKFtS0AOINtmBbmath6P1PQ3YaxnrGFj46HezhGYRPPDRuNPuf8yiPV971kFzPHkNi mgvpRh8LZh4voShq3/Sn3NNHcJ7cZYZHY1F55SIwgE3idL6UZGBtXFtemJn2ffHQnTZT 9J1Fdvj0yPeyARgLXX59Kz9OFo7rmYxk6rPiYk6QYdVpzT4NILONoZDmuCRq/8r4AtnT 5H///zJoEKevl4yE4S4+T/6dGjMEOHpWom8Vdm1WobLt8dRBMM9ZxOnu2FJKbYiUeMHG wDA7XobHtrckgzJuDM6TFprrvso19DTuaSKkmDMSR6r8+aMkU/E7XF0lHZmlfn3pFyHB asLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vGOISKhZ4dJtfoFqp39gzFk6BTmj8X1+UJDeL27mgOM=; b=tK2lTPAFkGoDpfLabcmfAZw5DqJ8CzhYzJ+LLYQRsY8W7tKVeoWw1A9GkEiAQ0e7F+ ZuVZ3VuzmnMgQ3PxxyMDkKa5sG7VMwAl8CZJzym/SVbhIXAzPK7IWAn+bUHDeLoNpmlS 37JyiFFml86HY9d6jA1HLFE1CozfwgqvWi+m9Th62YjtP4hnIq5eyKi5OR1Aw4Fnz8+R sjZjGWvnsvK+nImxAXDr0W1REL7XhddpDKPr5Irtclorl+weBrHoxctioAEuyAzuFOH+ ALKAfsGKkqCeh5B2PJ8YrkCWO+mZrrIRUfSdSJU6X8phDVCN5bN27PmnpiScfYCiE9AY dytw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=FuMuwQZW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h18-20020a170902f71200b00192f4b0a9e1si2103496plo.20.2023.01.04.16.33.17; Wed, 04 Jan 2023 16:33:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=FuMuwQZW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235746AbjAEAX5 (ORCPT + 99 others); Wed, 4 Jan 2023 19:23:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235451AbjAEAXQ (ORCPT ); Wed, 4 Jan 2023 19:23:16 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EAC6F44360; Wed, 4 Jan 2023 16:23:13 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 9BEFFB81990; Thu, 5 Jan 2023 00:23:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 22C11C433A8; Thu, 5 Jan 2023 00:23:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672878188; bh=MI3c5QYGakL+wl+pPfIjyXc4qZP1Ce7fJ4Wh6kV0Nls=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FuMuwQZWImSI4uvXh25bZyrTECbTfsUQcHsxAcxzPyj5uDX3IUUIWSb0164looBl1 39qNBTnjnDgnu5ecFHmjdUsFNeG+ReoSu+1SsHtNCAODDaruP6e1BMYWbmdCQ505RP uymMeJPCQe3nA/NWZaDRcsXgBucBJRmHO9a/l1YiZ5J0Z/6gBJR82FEJppHO5uYFDX 7yGYGPcYL8t/QNCJwBos7xi8WF72mo5Gdlq3x00ZR/Y94iuogFSRV+UxHsQstT0lw0 AyQ3DlebyRg61Vm6tw/YDCboAR8O1fcK8hFXc/ofMPLUUoQhznNA+ArtGCuelH5as4 GAVr2FwxoJEkw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 82E155C1C5D; Wed, 4 Jan 2023 16:23:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Paul E. McKenney" , David Howells Subject: [PATCH rcu 09/10] rcu: Allow expedited RCU CPU stall warnings to dump task stacks Date: Wed, 4 Jan 2023 16:23:04 -0800 Message-Id: <20230105002305.1768591-9-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> References: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754140571065342928?= X-GMAIL-MSGID: =?utf-8?q?1754140571065342928?= This commit introduces the rcupdate.rcu_exp_stall_task_details kernel boot parameter, which cause expedited RCU CPU stall warnings to dump the stacks of any tasks blocking the current expedited grace period. Reported-by: David Howells Signed-off-by: Paul E. McKenney --- .../admin-guide/kernel-parameters.txt | 5 +++ kernel/rcu/rcu.h | 1 + kernel/rcu/tree_exp.h | 41 +++++++++++++++++++ kernel/rcu/update.c | 2 + 4 files changed, 49 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 6cfa6e3996cf7..aa453f9202d89 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -5113,6 +5113,11 @@ rcupdate.rcu_cpu_stall_timeout to be used (after conversion from seconds to milliseconds). + rcupdate.rcu_exp_stall_task_details= [KNL] + Print stack dumps of any tasks blocking the + current expedited RCU grace period during an + expedited RCU CPU stall warning. + rcupdate.rcu_expedited= [KNL] Use expedited grace-period primitives, for example, synchronize_rcu_expedited() instead diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index c5aa934de59b0..fa640c45172e9 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -224,6 +224,7 @@ extern int rcu_cpu_stall_ftrace_dump; extern int rcu_cpu_stall_suppress; extern int rcu_cpu_stall_timeout; extern int rcu_exp_cpu_stall_timeout; +extern bool rcu_exp_stall_task_details __read_mostly; int rcu_jiffies_till_stall_check(void); int rcu_exp_jiffies_till_stall_check(void); diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 927abaf6c822e..249c2967d9e6c 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -11,6 +11,7 @@ static void rcu_exp_handler(void *unused); static int rcu_print_task_exp_stall(struct rcu_node *rnp); +static void rcu_exp_print_detail_task_stall_rnp(struct rcu_node *rnp); /* * Record the start of an expedited grace period. @@ -671,6 +672,7 @@ static void synchronize_rcu_expedited_wait(void) dump_cpu_task(cpu); preempt_enable(); } + rcu_exp_print_detail_task_stall_rnp(rnp); } jiffies_stall = 3 * rcu_exp_jiffies_till_stall_check() + 3; panic_on_rcu_stall(); @@ -813,6 +815,36 @@ static int rcu_print_task_exp_stall(struct rcu_node *rnp) return ndetected; } +/* + * Scan the current list of tasks blocked within RCU read-side critical + * sections, dumping the stack of each that is blocking the current + * expedited grace period. + */ +static void rcu_exp_print_detail_task_stall_rnp(struct rcu_node *rnp) +{ + unsigned long flags; + struct task_struct *t; + + if (!rcu_exp_stall_task_details) + return; + raw_spin_lock_irqsave_rcu_node(rnp, flags); + if (!READ_ONCE(rnp->exp_tasks)) { + raw_spin_unlock_irqrestore_rcu_node(rnp, flags); + return; + } + t = list_entry(rnp->exp_tasks->prev, + struct task_struct, rcu_node_entry); + list_for_each_entry_continue(t, &rnp->blkd_tasks, rcu_node_entry) { + /* + * We could be printing a lot while holding a spinlock. + * Avoid triggering hard lockup. + */ + touch_nmi_watchdog(); + sched_show_task(t); + } + raw_spin_unlock_irqrestore_rcu_node(rnp, flags); +} + #else /* #ifdef CONFIG_PREEMPT_RCU */ /* Request an expedited quiescent state. */ @@ -885,6 +917,15 @@ static int rcu_print_task_exp_stall(struct rcu_node *rnp) return 0; } +/* + * Because preemptible RCU does not exist, we never have to print out + * tasks blocked within RCU read-side critical sections that are blocking + * the current expedited grace period. + */ +static void rcu_exp_print_detail_task_stall_rnp(struct rcu_node *rnp) +{ +} + #endif /* #else #ifdef CONFIG_PREEMPT_RCU */ /** diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c index 587b97c401914..6ed5020aee6d1 100644 --- a/kernel/rcu/update.c +++ b/kernel/rcu/update.c @@ -509,6 +509,8 @@ int rcu_cpu_stall_timeout __read_mostly = CONFIG_RCU_CPU_STALL_TIMEOUT; module_param(rcu_cpu_stall_timeout, int, 0644); int rcu_exp_cpu_stall_timeout __read_mostly = CONFIG_RCU_EXP_CPU_STALL_TIMEOUT; module_param(rcu_exp_cpu_stall_timeout, int, 0644); +bool rcu_exp_stall_task_details __read_mostly; +module_param(rcu_exp_stall_task_details, bool, 0644); #endif /* #ifdef CONFIG_RCU_STALL_COMMON */ // Suppress boot-time RCU CPU stall warnings and rcutorture writer stall From patchwork Thu Jan 5 00:23:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 39191 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp42804wrt; Wed, 4 Jan 2023 16:32:54 -0800 (PST) X-Google-Smtp-Source: AMrXdXvhzJN2xm1h95uN0hWZw9XGn2fDW14qNFEcZmiDQSgfbNc7EMW9i+aqEGnivVyFA1a8dSDz X-Received: by 2002:a17:903:2d0:b0:192:b5b3:7ff5 with SMTP id s16-20020a17090302d000b00192b5b37ff5mr21607812plk.50.1672878773971; Wed, 04 Jan 2023 16:32:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672878773; cv=none; d=google.com; s=arc-20160816; b=Sx+rmRR1MJ1xDhJtuejeUVsWYsgIPS68zYD5mT+7fNTCoVKmAkATbldahrUFflp65K Ddx1pgL9kMQFyNwlkvmDQRz3NqM9vfOhw3z3ZXts8K9ASBFekIN27lpYriPOmGPeYC1c l5fU6MWc+cXTuvCeA+8j0jAZ/UEEKUoKMqQ7IR/22DNkslKnJKax7t+2P+s5SgMFLDZ9 KOUJYiZOwFHAP7MxdDhXbs0vNNno/Cfr1e4FbGXHTY61HJL3aZSNRPhc5oRcJ6z/Eb3g SzOP6nQ/MA1apLrk/p1z9b4QqOMUNw4cdc+pCBDHPohnIWIdYCC/PsLDBjFTyFhIhaZW csqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Xj9cxwzeZNus67RUR3xjkZTN187+QgHa6HJRxuZEKt8=; b=usq+TLeQLYsGS+t3eAXWoZ4UMSjGSxKjS9ANRnUFQP/laPde5X7RGTVYmtimDNKNgN 75mzJXceX0IaYPIXqksLRfYVE8NMzLeghhCZSLbEgy6ct7In87ONCc+PxIq23v0oReG9 FrO0/wD7m7n6pN8jPcmOlfW9uA69t3U7TeoWXCOGJV8wZONtJ3Ow999/XSzZvwU15n7J 2uC16OqW3/7JOXpT7dEIVqP+Nd5FQNqSXLeM41Bh5oHb9MYZJs9009Mm5ynpM4IhnNwN njyR+iewDaWBTC7acCReY+N780jJfMHk+az1EmW5J+j0xSmrI4bGeKGiuNyzAeCuQfIn 5otw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ulKYLo2Z; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t6-20020a170902e84600b001887a1ffb17si39648983plg.223.2023.01.04.16.32.39; Wed, 04 Jan 2023 16:32:53 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ulKYLo2Z; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238837AbjAEAYE (ORCPT + 99 others); Wed, 4 Jan 2023 19:24:04 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33140 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235537AbjAEAXQ (ORCPT ); Wed, 4 Jan 2023 19:23:16 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE2E74435E; Wed, 4 Jan 2023 16:23:13 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 8A06EB8198A; Thu, 5 Jan 2023 00:23:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 22A1CC433A7; Thu, 5 Jan 2023 00:23:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1672878188; bh=Zj66HaKlbW7or4kBjbwXaM4/EWUfv7i59XPipGrmdK0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ulKYLo2ZelNq3JjSgOVb1UmueUei/z6vfdW5xa/K9HlH5Rsqxsctdb4/Tujqz4bh/ SLYLIFY/7LXH42rZs2DBm8dPtZ1qhcWCwxzBEygxCyfmvOdrUWWi0K5bldjYXGM4+E A9qGjQEKRNTZ8DR0SNwMy2p6ZCtRikhj613LcOEgJZs0oF7I4zuX7AzlLz0qOQKgTv HHus1peh8fjC+fXpJ/m3aPrcp1+8W+GvMdwq4JY1CdjHAejq388N7yzHmDMQp0FWMQ poja+Oartem/2us8mYOzK07V7SCE79MquyoyOJ7/B4JcXdza+m7UNi7/wbrns8bgin VY5gly0yNjKFw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 84A6B5C1C64; Wed, 4 Jan 2023 16:23:07 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Zqiang , "Paul E . McKenney" Subject: [PATCH rcu 10/10] rcu: Remove redundant call to rcu_boost_kthread_setaffinity() Date: Wed, 4 Jan 2023 16:23:05 -0800 Message-Id: <20230105002305.1768591-10-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> References: <20230105002257.GA1768487@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754140533102808696?= X-GMAIL-MSGID: =?utf-8?q?1754140533102808696?= From: Zqiang The rcu_boost_kthread_setaffinity() function is invoked at rcutree_online_cpu() and rcutree_offline_cpu() time, early in the online timeline and late in the offline timeline, respectively. It is also invoked from rcutree_dead_cpu(), however, in the absence of userspace manipulations (for which userspace must take responsibility), this call is redundant with that from rcutree_offline_cpu(). This redundancy can be demonstrated by printing out the relevant cpumasks This commit therefore removes the call to rcu_boost_kthread_setaffinity() from rcutree_dead_cpu(). Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 5 ----- 1 file changed, 5 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 80b84ae285b41..89313c7c17b62 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4076,15 +4076,10 @@ static void rcu_cleanup_dead_rnp(struct rcu_node *rnp_leaf) */ int rcutree_dead_cpu(unsigned int cpu) { - struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); - struct rcu_node *rnp = rdp->mynode; /* Outgoing CPU's rdp & rnp. */ - if (!IS_ENABLED(CONFIG_HOTPLUG_CPU)) return 0; WRITE_ONCE(rcu_state.n_online_cpus, rcu_state.n_online_cpus - 1); - /* Adjust any no-longer-needed kthreads. */ - rcu_boost_kthread_setaffinity(rnp, -1); // Stop-machine done, so allow nohz_full to disable tick. tick_dep_clear(TICK_DEP_BIT_RCU); return 0;