From patchwork Tue Nov 22 01:04:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24077 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1929714wrr; Mon, 21 Nov 2022 17:04:51 -0800 (PST) X-Google-Smtp-Source: AA0mqf4nd4+q9GHaHsmpJam3r9aMSxASV3+OCDM2ZMI7cPg0aMEcwRcSC2Uqh0tNF2iJkLUFb48M X-Received: by 2002:a63:eb16:0:b0:477:6fe1:cd6d with SMTP id t22-20020a63eb16000000b004776fe1cd6dmr6862295pgh.334.1669079090929; Mon, 21 Nov 2022 17:04:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079090; cv=none; d=google.com; s=arc-20160816; b=cwxkvIU8zhZem/h3bO2+F1AhIxLJSF4K51LBHc1PvxFMXjhr06/SLPFgdZsTKVo8n3 KZYoiNLyTji8vGnPOU1gP3TRcuffmoRJo9WQO//Qn0ghQ45GTCzE81LX353siJj/elHq 3Z1TH3XUrGyYSGqqdI9m9Z9A349PNqn8VKBZJDkJch1jGRJNXj3R5IO80rdq+eeCcSrx ePNJqSnHELpqQhmCbAnCiH9ECy2eqir2AU0p1xPVRhvkozt6GN/XrJTrB5mkVS/SGphK 4ylcWR6pz7ZiPAv1aIJwHMv9ZmW/yTN61zm8WYCBBG7GP087KDim3ApxXbOTpJldgbC0 9qJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ozKhBfyN+FxyuLIwEkPQLFxihGTESoRyKpstG62wPHM=; b=WcdASKSS1JhBW2WOb+nC7y4ehoNX4N+nnJXTBHdqbP7t0v2741CGamtzl+mvJSx4sL unM4AFQTeEUcDuIuPYa68uFXNU/eUfUQEKqk3gi6jgNd3HkKFtLNsjTbhJ0BFkT1VFup hMzWHyAaWeV0yYYT/+K+wVAUm2TQ849Q5u6x6AS3alUiiD5hgL9QU02TlG0oF2FpXoNq /FyZzNW2RzAEruntBsMnumIxE9ly4O24s87Rhace+DNa5bxdXvEabXUvGH9jcpr2nxro fD1ACB+oMMQnqtnLPZv7iXqnFdDRpVK//gmA11AxbdtmU+BivsUuReYq36U3vTQF4YOS JApA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=FStEKe3U; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g3-20020a63e603000000b0046ef006f51dsi12914033pgh.425.2022.11.21.17.04.37; Mon, 21 Nov 2022 17:04:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=FStEKe3U; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231951AbiKVBE3 (ORCPT + 99 others); Mon, 21 Nov 2022 20:04:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231927AbiKVBEY (ORCPT ); Mon, 21 Nov 2022 20:04:24 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D505511A12; Mon, 21 Nov 2022 17:04:23 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5F9BC61505; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B38BBC433D7; Tue, 22 Nov 2022 01:04:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079062; bh=JgCYhsIq4xhCHA9e/N+ivpZlC63szLfECFNNQENWSoc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FStEKe3UKD8G3yecmoiH9U+hi00FGgXyD45l/lhOmtJ104VYltWb4uubccUFvAgr4 gyJ77WtydXtgorQjyx4cKIfVf0rQAm4qcRpfShZXhrdR3HFS8duP385sCvj47gn9GD UOQAV925FbSslF3kyd0GpnkH87vF16s5nN6QCDAwvC9ZgWkcWPvE/Y2gPQSe7NT+Qj iwlHLa6We21VIpsqRc+qDLbCvq3vPLRpjYLAfZR3SLEC8ekk94sMKjap2zwa2Sne/9 9DTkb67x+WBKkxFCa8NRDlzVzYKUij3AKfprKx02lfQuFOunOr2rCnThYSV3c+Pfrf RTeaRLM+MTfWA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 756A15C0D1D; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, Frederic Weisbecker , Joel Fernandes , "Paul E . McKenney" Subject: [PATCH v2 rcu 03/16] rcu: Fix missing nocb gp wake on rcu_barrier() Date: Mon, 21 Nov 2022 17:04:08 -0800 Message-Id: <20221122010421.3799681-3-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156276722439850?= X-GMAIL-MSGID: =?utf-8?q?1750156276722439850?= From: Frederic Weisbecker In preparation for RCU lazy changes, wake up the RCU nocb gp thread if needed after an entrain. This change prevents the RCU barrier callback from waiting in the queue for several seconds before the lazy callbacks in front of it are serviced. Reported-by: Joel Fernandes (Google) Signed-off-by: Frederic Weisbecker Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/tree.c | 11 +++++++++++ kernel/rcu/tree.h | 1 + kernel/rcu/tree_nocb.h | 5 +++++ 3 files changed, 17 insertions(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 6bb8e72bc8151..fb7a1b95af71e 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3894,6 +3894,8 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) { unsigned long gseq = READ_ONCE(rcu_state.barrier_sequence); unsigned long lseq = READ_ONCE(rdp->barrier_seq_snap); + bool wake_nocb = false; + bool was_alldone = false; lockdep_assert_held(&rcu_state.barrier_lock); if (rcu_seq_state(lseq) || !rcu_seq_state(gseq) || rcu_seq_ctr(lseq) != rcu_seq_ctr(gseq)) @@ -3902,7 +3904,14 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) rdp->barrier_head.func = rcu_barrier_callback; debug_rcu_head_queue(&rdp->barrier_head); rcu_nocb_lock(rdp); + /* + * Flush bypass and wakeup rcuog if we add callbacks to an empty regular + * queue. This way we don't wait for bypass timer that can reach seconds + * if it's fully lazy. + */ + was_alldone = rcu_rdp_is_offloaded(rdp) && !rcu_segcblist_pend_cbs(&rdp->cblist); WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies)); + wake_nocb = was_alldone && rcu_segcblist_pend_cbs(&rdp->cblist); if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head)) { atomic_inc(&rcu_state.barrier_cpu_count); } else { @@ -3910,6 +3919,8 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) rcu_barrier_trace(TPS("IRQNQ"), -1, rcu_state.barrier_sequence); } rcu_nocb_unlock(rdp); + if (wake_nocb) + wake_nocb_gp(rdp, false); smp_store_release(&rdp->barrier_seq_snap, gseq); } diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index d4a97e40ea9c3..925dd98f8b23b 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -439,6 +439,7 @@ static void zero_cpu_stall_ticks(struct rcu_data *rdp); static struct swait_queue_head *rcu_nocb_gp_get(struct rcu_node *rnp); static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq); static void rcu_init_one_nocb(struct rcu_node *rnp); +static bool wake_nocb_gp(struct rcu_data *rdp, bool force); static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, unsigned long j); static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index f77a6d7e13564..094fd454b6c38 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1558,6 +1558,11 @@ static void rcu_init_one_nocb(struct rcu_node *rnp) { } +static bool wake_nocb_gp(struct rcu_data *rdp, bool force) +{ + return false; +} + static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, unsigned long j) {