From patchwork Sun Oct 16 16:22:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 3057 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp1077873wrs; Sun, 16 Oct 2022 09:24:48 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5mm9KhUyRxHt9TX86lsbWNy3UiG1dBKTeVbPkud07ga8/42bc5sgVh+BOhW14Ue47bwzs5 X-Received: by 2002:a63:8648:0:b0:461:722b:ffc8 with SMTP id x69-20020a638648000000b00461722bffc8mr7276862pgd.118.1665937487608; Sun, 16 Oct 2022 09:24:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1665937487; cv=none; d=google.com; s=arc-20160816; b=tsgh/A4jItJx2r7gsoERfpe0MhsvYRcCSouC5xddepGPPpHoMtJG1SurEPVT+NJWdj 9woL6/Fc5Afj+Yu4rjWZ00CNX94UbkzPJZmgyHMoOeEBlcUZptoXJGSnlmDZAtKoIM/g wj1YfdRg5x7xsOTfXLo9elO9O5zL+CzuBu8cdHQ3Oo1u+uEYqTQhYwX3eTmi2xh+GytJ EBPwQKg8DY9vLe9nUUB28/62aXgTimtC3jGA+nmSt2iTCVZn6b7/gkBx6J3nI7068e1a sRgTllWauGx5ssIwCS9NH5mjM1+4WNKIoNvINaGVp9zYbTekyV/glxsn9ROe7GUjl9wX vYqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2MoWqjHuV8nLLRCYHvb4Cnre7Xc0pX7FZvINFrXyaV4=; b=HRvoXIVAlOUU5XZzdOJLEIP/jUAcr1FMIUdQl216/cOOQGeRDRD/SZmXWho6qmqnEj NhWoLj64CYsndMtSkmoPt02mZs5IPaSaJByGBuSMS4f4A7DkPaYlKjhqWiYJzG3g4T3L Vcrralzv26vSO7KSKIWoHvdYwuY3V53bM81TnDcqYtaeP7OGRuynfcAp5oGusNhWQude uELf+EggZEh+HwK4/pwKg/K5mzfMBMDjoy4dERahFx5LW0VPOdtsoUuYmugxJP49d4Ap NEbwuB1RgTLlKvp17mKqefrPX5OpDRomZFlbzF9BrVYSa6ocWTQLbF8U+TLfb63nTJNf tk/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b="yj/YGf+8"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p4-20020a170902a40400b0017f9636902fsi8852505plq.391.2022.10.16.09.24.35; Sun, 16 Oct 2022 09:24:47 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b="yj/YGf+8"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229904AbiJPQXr (ORCPT + 99 others); Sun, 16 Oct 2022 12:23:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57856 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229934AbiJPQXj (ORCPT ); Sun, 16 Oct 2022 12:23:39 -0400 Received: from mail-qk1-x72a.google.com (mail-qk1-x72a.google.com [IPv6:2607:f8b0:4864:20::72a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D026D3CBCB for ; Sun, 16 Oct 2022 09:23:36 -0700 (PDT) Received: by mail-qk1-x72a.google.com with SMTP id j21so5395221qkk.9 for ; Sun, 16 Oct 2022 09:23:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2MoWqjHuV8nLLRCYHvb4Cnre7Xc0pX7FZvINFrXyaV4=; b=yj/YGf+8EA2ZpPX+5IFaseWn+7sFZFipCIU5JHL04HFdaMTYUJQQ4hCZrnCkhzCiXI WKJ5akQdzdCq4T8fLwF/E0+LjQ0eHnJ/viRtqu1ppvmLMWELcLSrbwMal8aPxNIfjDwO zk+vuG4c0Kv+ddJgdg6JRrPhXXitTZs9uaV9A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2MoWqjHuV8nLLRCYHvb4Cnre7Xc0pX7FZvINFrXyaV4=; b=4/0cYqhEzWTqgrQ47yDZ5DB4e97JCQ/lL6QzVZeaYqqvkJNfR+B4hTF9Eibg2j4Lq1 xW2jKLrRYXRnZeLkpaIpqsD4ZIPMQzXGrpu+qTcG/T7ODJgfLIjR57timrRgRsvYlyoa qNP4YAn7DNVljmE82hzmQ2iVmcG9hIJXpgrBUSKxt7UzDzh7LcTJtK8duLQlpJWUoeKl MsTDXZAnz0JrvQlrP5DCaF3mk9i6hv1Fr8Ot4Gdm86n3Txx+tFJuFknoPfV7WzqB5663 v+crWrDuM+t+Q2IHKr3L/QY89dq0G/v7wFkSnOb/cOi4kd+cgabUTVczOPA0lpSpNjzk vDyQ== X-Gm-Message-State: ACrzQf0N4xscoteFX4v9TmuG2d7jk1U82w5WmdotlkVmxOktmns7/T3S 3tsvw3tOjuM8hzxAGScl0NckaA== X-Received: by 2002:a05:620a:1729:b0:6ee:cf01:6810 with SMTP id az41-20020a05620a172900b006eecf016810mr4905001qkb.555.1665937415383; Sun, 16 Oct 2022 09:23:35 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (228.221.150.34.bc.googleusercontent.com. [34.150.221.228]) by smtp.gmail.com with ESMTPSA id x19-20020ac87ed3000000b003436103df40sm6001207qtj.8.2022.10.16.09.23.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 16 Oct 2022 09:23:35 -0700 (PDT) From: "Joel Fernandes (Google)" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, frederic@kernel.org, paulmck@kernel.org, Joel Fernandes Subject: [PATCH v9 01/13] rcu: Fix missing nocb gp wake on rcu_barrier() Date: Sun, 16 Oct 2022 16:22:53 +0000 Message-Id: <20221016162305.2489629-2-joel@joelfernandes.org> X-Mailer: git-send-email 2.38.0.413.g74048e4d9e-goog In-Reply-To: <20221016162305.2489629-1-joel@joelfernandes.org> References: <20221016162305.2489629-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1746862066821145414?= X-GMAIL-MSGID: =?utf-8?q?1746862066821145414?= From: Frederic Weisbecker In preparation of RCU lazy changes, wake up the RCU nocb gp thread if needed after an entrain. Otherwise, the RCU barrier callback can wait in the queue for several seconds before the lazy callbacks in front of it are serviced. Reported-by: Joel Fernandes (Google) Signed-off-by: Frederic Weisbecker Signed-off-by: Joel Fernandes (Google) Change-Id: I830269cd41b18862a1a58b26ce3292c6c4457bc7 --- kernel/rcu/tree.c | 11 +++++++++++ kernel/rcu/tree.h | 1 + kernel/rcu/tree_nocb.h | 5 +++++ 3 files changed, 17 insertions(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 5ec97e3f7468..67a1ae5151f5 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3894,6 +3894,8 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) { unsigned long gseq = READ_ONCE(rcu_state.barrier_sequence); unsigned long lseq = READ_ONCE(rdp->barrier_seq_snap); + bool wake_nocb = false; + bool was_alldone = false; lockdep_assert_held(&rcu_state.barrier_lock); if (rcu_seq_state(lseq) || !rcu_seq_state(gseq) || rcu_seq_ctr(lseq) != rcu_seq_ctr(gseq)) @@ -3902,7 +3904,14 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) rdp->barrier_head.func = rcu_barrier_callback; debug_rcu_head_queue(&rdp->barrier_head); rcu_nocb_lock(rdp); + /* + * Flush bypass and wakeup rcuog if we add callbacks to an empty regular + * queue. This way we don't wait for bypass timer that can reach seconds + * if it's fully lazy. + */ + was_alldone = rcu_rdp_is_offloaded(rdp) && !rcu_segcblist_pend_cbs(&rdp->cblist); WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies)); + wake_nocb = was_alldone && rcu_segcblist_pend_cbs(&rdp->cblist); if (rcu_segcblist_entrain(&rdp->cblist, &rdp->barrier_head)) { atomic_inc(&rcu_state.barrier_cpu_count); } else { @@ -3910,6 +3919,8 @@ static void rcu_barrier_entrain(struct rcu_data *rdp) rcu_barrier_trace(TPS("IRQNQ"), -1, rcu_state.barrier_sequence); } rcu_nocb_unlock(rdp); + if (wake_nocb) + wake_nocb_gp(rdp, false); smp_store_release(&rdp->barrier_seq_snap, gseq); } diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index d4a97e40ea9c..925dd98f8b23 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -439,6 +439,7 @@ static void zero_cpu_stall_ticks(struct rcu_data *rdp); static struct swait_queue_head *rcu_nocb_gp_get(struct rcu_node *rnp); static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq); static void rcu_init_one_nocb(struct rcu_node *rnp); +static bool wake_nocb_gp(struct rcu_data *rdp, bool force); static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, unsigned long j); static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index f77a6d7e1356..094fd454b6c3 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1558,6 +1558,11 @@ static void rcu_init_one_nocb(struct rcu_node *rnp) { } +static bool wake_nocb_gp(struct rcu_data *rdp, bool force) +{ + return false; +} + static bool rcu_nocb_flush_bypass(struct rcu_data *rdp, struct rcu_head *rhp, unsigned long j) {