From patchwork Wed Nov 15 19:11:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 165555 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2754781vqg; Wed, 15 Nov 2023 11:12:03 -0800 (PST) X-Google-Smtp-Source: AGHT+IFWIh5Z4fU7dnx8ci0f/0jjqBF+NV0scVdlSqlz31IPcRiaUrLGfiOyz7RiuelVtIcuqsGK X-Received: by 2002:a05:6a00:1ca9:b0:6c3:4bf2:7486 with SMTP id y41-20020a056a001ca900b006c34bf27486mr9597540pfw.7.1700075522864; Wed, 15 Nov 2023 11:12:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700075522; cv=none; d=google.com; s=arc-20160816; b=UWNzvqesqYWBrvVrcA0m26inxZ6w2CggzQwct8wwTVADo/71nnfg9MBL42L5x9JK0d H/ZRicRHps0XGIp+ymQ3FOc8apytEHMJFQyBc8btai3pkuUIpie5yn4pWzR4uTX+l9rE ULoPbWbpi34v1NgpOvtc0KqPlSSPAwWl0Kmg+E1T6QV6K7NCCz1XbG0Ztf3ugeQ8SDvI 5HXXJwkdwOdJmzDkfjWs0fYIePo4W77yxZfkbjU/mFDiea4HlLMKz5uTR7+9KsXVGH08 POhrkVl7iYL699tT8k838sVxciKL4sq4VW0hEtoMaVQCCH16bIkc3nKJXko5VkcHpB4M nv+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=uqCNq68y1zZJq4Kv6zn+GugeX/BUSKyWHsU5LJTN4eE=; fh=Hg33MEMv6OAYg0vV8rrONkXSgLqcq8ZX1if6rnFlFSg=; b=UFGUqSf8xAhO4zri0jnclixeVMtra2EYs2e0XfuB3D8iZk/s4GUY44TNxoyRag33gK bmbRYBOfyjVZLsHJdhIGudeJuhFxg1bkPezlovbO51fpkXzEwLi9FyKxP0I97WzFlRFA cXGvuZnqWl6XVQ1DrJdGqQO0NFMFLPKqlLbVLegqD2ZhI1M22PW+7AgrKuTHDMgN28Ca dJVx12Vfb253hTQUZ9aktYtKJXvmzoDZqgldc6k16HNwBkbOfbjvCvzKXk9ic6ES2J0h 2aNWvo1ezCI/sk+VPemXzdhjcgIb9TSzAgddOg1wQ+kR2tVi0rgs55guXlOOhkpxxm+9 BTYQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=DQn+FuTY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id q8-20020a63e208000000b005bdcad0ac5esi10250773pgh.833.2023.11.15.11.12.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 11:12:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=DQn+FuTY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 1033E8043CA7; Wed, 15 Nov 2023 11:11:55 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232589AbjKOTLi (ORCPT + 29 others); Wed, 15 Nov 2023 14:11:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36240 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230006AbjKOTLg (ORCPT ); Wed, 15 Nov 2023 14:11:36 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4C4912C; Wed, 15 Nov 2023 11:11:32 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A301CC433C9; Wed, 15 Nov 2023 19:11:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1700075492; bh=+Lz3UclCdi8Chr5LH8Nj8EvSuzzxtIfVUkXPRBwTd6w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DQn+FuTYTgFeL/Jp2HT/7etsJiu/0YXKTPBfYyT+ArYrZBFdVGim0VLPpDkf1ko1N mzl28sS02q6k/mtc2tPIg5aX1tgtN/R57kWY5lQMNSQCK62ScVpE6U7WzzvhQ/uU1B Lg+EkucCzz2n+K1xmmX0tQVp9myVLE77/picPzOTfM+hTrUdgzAoK3X095ttdUi4O7 iWYrJZC7wGDSxk0N6ardAWFMRq4bopnZAGrxGcZ6VfqNtzFjMJ9KnsnE9WWtPcUJh3 ENZ62N3OORYIDxARdyAqFOYIeb3cLU6RspyuMopq5l8ZCk+9cqvSheS8NR06VcE+F5 QReAOZ2SlRG1A== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Boqun Feng , Joel Fernandes , Neeraj Upadhyay , Uladzislau Rezki , Zqiang , rcu , "Paul E. McKenney" Subject: [PATCH 1/3] rcu: Rename jiffies_till_flush to jiffies_lazy_flush Date: Wed, 15 Nov 2023 14:11:26 -0500 Message-ID: <20231115191128.15615-2-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231115191128.15615-1-frederic@kernel.org> References: <20231115191128.15615-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.3 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 15 Nov 2023 11:11:55 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782658391220361318 X-GMAIL-MSGID: 1782658391220361318 The variable name jiffies_till_flush is too generic and therefore: * It may shadow a global variable * It doesn't tell on what it operates Make the name more precise, along with the related APIs. Reviewed-by: Joel Fernandes (Google) Signed-off-by: Frederic Weisbecker --- kernel/rcu/rcu.h | 8 ++++---- kernel/rcu/rcuscale.c | 6 +++--- kernel/rcu/tree_nocb.h | 22 +++++++++++----------- 3 files changed, 18 insertions(+), 18 deletions(-) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index b531c33e9545..ff41423cd61c 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -541,11 +541,11 @@ enum rcutorture_type { }; #if defined(CONFIG_RCU_LAZY) -unsigned long rcu_lazy_get_jiffies_till_flush(void); -void rcu_lazy_set_jiffies_till_flush(unsigned long j); +unsigned long rcu_get_jiffies_lazy_flush(void); +void rcu_set_jiffies_lazy_flush(unsigned long j); #else -static inline unsigned long rcu_lazy_get_jiffies_till_flush(void) { return 0; } -static inline void rcu_lazy_set_jiffies_till_flush(unsigned long j) { } +static inline unsigned long rcu_get_jiffies_lazy_flush(void) { return 0; } +static inline void rcu_set_jiffies_lazy_flush(unsigned long j) { } #endif #if defined(CONFIG_TREE_RCU) diff --git a/kernel/rcu/rcuscale.c b/kernel/rcu/rcuscale.c index ffdb30495e3c..8db4fedaaa1e 100644 --- a/kernel/rcu/rcuscale.c +++ b/kernel/rcu/rcuscale.c @@ -764,9 +764,9 @@ kfree_scale_init(void) if (kfree_by_call_rcu) { /* do a test to check the timeout. */ - orig_jif = rcu_lazy_get_jiffies_till_flush(); + orig_jif = rcu_get_jiffies_lazy_flush(); - rcu_lazy_set_jiffies_till_flush(2 * HZ); + rcu_set_jiffies_lazy_flush(2 * HZ); rcu_barrier(); jif_start = jiffies; @@ -775,7 +775,7 @@ kfree_scale_init(void) smp_cond_load_relaxed(&rcu_lazy_test1_cb_called, VAL == 1); - rcu_lazy_set_jiffies_till_flush(orig_jif); + rcu_set_jiffies_lazy_flush(orig_jif); if (WARN_ON_ONCE(jiffies_at_lazy_cb - jif_start < 2 * HZ)) { pr_alert("ERROR: call_rcu() CBs are not being lazy as expected!\n"); diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 4efbf7333d4e..aecef51166c7 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -256,6 +256,7 @@ static bool wake_nocb_gp(struct rcu_data *rdp, bool force) return __wake_nocb_gp(rdp_gp, rdp, force, flags); } +#ifdef CONFIG_RCU_LAZY /* * LAZY_FLUSH_JIFFIES decides the maximum amount of time that * can elapse before lazy callbacks are flushed. Lazy callbacks @@ -264,21 +265,20 @@ static bool wake_nocb_gp(struct rcu_data *rdp, bool force) * left unsubmitted to RCU after those many jiffies. */ #define LAZY_FLUSH_JIFFIES (10 * HZ) -static unsigned long jiffies_till_flush = LAZY_FLUSH_JIFFIES; +static unsigned long jiffies_lazy_flush = LAZY_FLUSH_JIFFIES; -#ifdef CONFIG_RCU_LAZY // To be called only from test code. -void rcu_lazy_set_jiffies_till_flush(unsigned long jif) +void rcu_set_jiffies_lazy_flush(unsigned long jif) { - jiffies_till_flush = jif; + jiffies_lazy_flush = jif; } -EXPORT_SYMBOL(rcu_lazy_set_jiffies_till_flush); +EXPORT_SYMBOL(rcu_set_jiffies_lazy_flush); -unsigned long rcu_lazy_get_jiffies_till_flush(void) +unsigned long rcu_get_jiffies_lazy_flush(void) { - return jiffies_till_flush; + return jiffies_lazy_flush; } -EXPORT_SYMBOL(rcu_lazy_get_jiffies_till_flush); +EXPORT_SYMBOL(rcu_get_jiffies_lazy_flush); #endif /* @@ -299,7 +299,7 @@ static void wake_nocb_gp_defer(struct rcu_data *rdp, int waketype, */ if (waketype == RCU_NOCB_WAKE_LAZY && rdp->nocb_defer_wakeup == RCU_NOCB_WAKE_NOT) { - mod_timer(&rdp_gp->nocb_timer, jiffies + jiffies_till_flush); + mod_timer(&rdp_gp->nocb_timer, jiffies + rcu_get_jiffies_lazy_flush()); WRITE_ONCE(rdp_gp->nocb_defer_wakeup, waketype); } else if (waketype == RCU_NOCB_WAKE_BYPASS) { mod_timer(&rdp_gp->nocb_timer, jiffies + 2); @@ -482,7 +482,7 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, // flush ->nocb_bypass to ->cblist. if ((ncbs && !bypass_is_lazy && j != READ_ONCE(rdp->nocb_bypass_first)) || (ncbs && bypass_is_lazy && - (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush))) || + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + rcu_get_jiffies_lazy_flush()))) || ncbs >= qhimark) { rcu_nocb_lock(rdp); *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); @@ -723,7 +723,7 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) lazy_ncbs = READ_ONCE(rdp->lazy_len); if (bypass_ncbs && (lazy_ncbs == bypass_ncbs) && - (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + jiffies_till_flush) || + (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + rcu_get_jiffies_lazy_flush()) || bypass_ncbs > 2 * qhimark)) { flush_bypass = true; } else if (bypass_ncbs && (lazy_ncbs != bypass_ncbs) && From patchwork Wed Nov 15 19:11:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 165554 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2754778vqg; Wed, 15 Nov 2023 11:12:02 -0800 (PST) X-Google-Smtp-Source: AGHT+IFJQQd/GMHmvcs8FDwQ6h/J28FflaoFwisvWBsPOyHG3IWKLvQzFn+eFpr2C+ku2zweq9Qq X-Received: by 2002:a17:903:c3:b0:1cc:237c:75d6 with SMTP id x3-20020a17090300c300b001cc237c75d6mr52427plc.67.1700075522470; Wed, 15 Nov 2023 11:12:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700075522; cv=none; d=google.com; s=arc-20160816; b=yj+iA3RfVYVBRCUxjrZBiEcmP0fXOZl23NSKn/OEVYSbQ5RM4EAofbEJKxyTcdyKae PRR2I9EK4zdGKaxPQmMfd2u4rJU67j8eV7M6F9U7gpVi1OArnAs5/MxB62aFaXQe6dy8 UERLluH6Wnxgmp8hVWrqCYZxWMB+Fvl7SKtrQGwSWAUnmOOp1zpIqt9lAKHImEsjlQ3g 00B4zo1KvRD+4Rkonx3RDD1wH1O/q/MAVolZ1SnRzhDHjHzi1hlaNFPGJFoW2ESNglCy OLutyszq92lVzNbiupVKrX1anMwImHeh3ntrqtIOoqnhurytxlRhqOLDN3AmIQU0JlAa 4Cvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3zzM0xsVrtY4/PlTPNZ1+yGgz9ciOG4K1aya7rU4eMI=; fh=Hg33MEMv6OAYg0vV8rrONkXSgLqcq8ZX1if6rnFlFSg=; b=IMx2KmiMtxQdBgRuhqaajjHdRFk1C4fJoc17ls06xIh3NAV5Qbiv/5tkIXagL3JXFo JaWtzzkZ1WNCe7X6FG7ssNUnPxvdrHaxjAditik4JYkXtiX1MQQXEWO9V7/3KIcbVglo yff+fj30S42Wln8LWg0H0UlfAop+6MYJuBzDUPAEqujUFX75z3+jPh5595B1i+Q3fv3F 1vIy43xxDUiigWxGYawA7meYVtL5VAvSHW1wTgp4n0YPmZS92uFfNKnK1p/AendsgmOs TnzkBlbMwuqPWub9VAALvYk8AkyK5KCOUdYTKNVcAzU0l8+7Y5Hi+TLzP8e0GSGkpaFu cBGw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=QR4M2pHC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id p10-20020a170902e74a00b001ca336f48bdsi10942556plf.556.2023.11.15.11.12.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 11:12:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=QR4M2pHC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 6023A8028A68; Wed, 15 Nov 2023 11:11:57 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232781AbjKOTLj (ORCPT + 29 others); Wed, 15 Nov 2023 14:11:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36252 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230262AbjKOTLh (ORCPT ); Wed, 15 Nov 2023 14:11:37 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 02F2B199; Wed, 15 Nov 2023 11:11:33 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BECE3C433C8; Wed, 15 Nov 2023 19:11:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1700075493; bh=b4GSpROCgIcQBEluuLqw37z2afZmrOSo6EsA/1KM+bk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QR4M2pHCubk+BGD+JllWKxf+LOjqI6w2qgt7rivD1NfHkmRwiLuzYbfI0TdABfioF tUXQgiioySp63uLWj50R0lLVZogLGyGIbaAjsIp6S+OtfjYVLglJY3Z8hjC5NNqrZm GgZviXe0mb0f7tugAXGxug1v+FwbmTLEmzp4QhfVfbB4wvH7K3EXrbEino/aVRnvly 0adAe5dIKoteYZG02+E8957bOc1VN30Daml7eripT7HxyZMu+RuWFM3yr2GAg5R9i3 uumlxSMw9OGa+Qh8wnUpdYMXaEc76zt+yMQOruz9CW6/8TWc+B8A3JrTj4qoOu9LJ9 mWGDFUgA80aAQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Boqun Feng , Joel Fernandes , Neeraj Upadhyay , Uladzislau Rezki , Zqiang , rcu , "Paul E. McKenney" Subject: [PATCH 2/3] rcu/nocb: Remove needless LOAD-ACQUIRE Date: Wed, 15 Nov 2023 14:11:27 -0500 Message-ID: <20231115191128.15615-3-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231115191128.15615-1-frederic@kernel.org> References: <20231115191128.15615-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.3 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Wed, 15 Nov 2023 11:11:57 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782658391046563042 X-GMAIL-MSGID: 1782658391046563042 The LOAD-ACQUIRE access performed on rdp->nocb_cb_sleep advertizes ordering callback execution against grace period completion. However this is contradicted by the following: * This LOAD-ACQUIRE doesn't pair with anything. The only counterpart barrier that can be found is the smp_mb() placed after callbacks advancing in nocb_gp_wait(). However the barrier is placed _after_ ->nocb_cb_sleep write. * Callbacks can be concurrently advanced between the LOAD-ACQUIRE on ->nocb_cb_sleep and the call to rcu_segcblist_extract_done_cbs() in rcu_do_batch(), making any ordering based on ->nocb_cb_sleep broken. * Both rcu_segcblist_extract_done_cbs() and rcu_advance_cbs() are called under the nocb_lock, the latter hereby providing already the desired ACQUIRE semantics. Therefore it is safe to access ->nocb_cb_sleep with a simple compiler barrier. Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index aecef51166c7..eb27878d46f1 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -933,8 +933,7 @@ static void nocb_cb_wait(struct rcu_data *rdp) swait_event_interruptible_exclusive(rdp->nocb_cb_wq, nocb_cb_wait_cond(rdp)); - // VVV Ensure CB invocation follows _sleep test. - if (smp_load_acquire(&rdp->nocb_cb_sleep)) { // ^^^ + if (READ_ONCE(rdp->nocb_cb_sleep)) { WARN_ON(signal_pending(current)); trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("WokeEmpty")); } From patchwork Wed Nov 15 19:11:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 165556 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2754884vqg; Wed, 15 Nov 2023 11:12:13 -0800 (PST) X-Google-Smtp-Source: AGHT+IHb9AbVw7NaJLUvPnpY5zDDBnLkPlRRj5DJmFb3cksqY2nTlfmohg8X3NX4Dmu+4D5SoG5t X-Received: by 2002:a05:6a21:7889:b0:187:15e2:fe02 with SMTP id bf9-20020a056a21788900b0018715e2fe02mr6546908pzc.13.1700075533613; Wed, 15 Nov 2023 11:12:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700075533; cv=none; d=google.com; s=arc-20160816; b=gB7kFwi57O+ih3RGQKhArjk8ekTpn7eYHenwSZmNlSoDdjreEfWNmyqLLchvtIbvNz unM0YDmUAowbDKrDga0GU2/K/ZSlP5ENTrnUgAkE0Qz6SZxEpMk5cfGPelqFSB8IF29j /QKmWj54WT1xVt570Z3/q2eXfMCrGjv8qNcuKafkEhXfs41duxeSvOXjXiVCLzXNKKrt msBtJ3E6IW8yr/7uXXgKzZKxO8AMPOuPumjGau2wYRZndLyPZIVLpYw0tpNRj51Hezxr +QulxBvFgHUS2zzVIf08yzVQ659o2FmvvuGMJ1+8a08VKC39VOBeSUlsSiW4YSrsqjLQ c7JA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=OiPI8PbzeQzW6a6UQl68ApbLi7mbAqrXUL2mGgUo1VM=; fh=Hg33MEMv6OAYg0vV8rrONkXSgLqcq8ZX1if6rnFlFSg=; b=aZiIRIrI/fBxKrBbVrE2JFh4nvkjanJuZkWKOvZSZw2zcGkg/M36VNBP9Qr6cPzlVm xTKf0CoZSLC4vNbTjpnoNFS28W4QC4sScMAGk/6my8wZLNTTIHmSfWlFyQmBnDIGv3Iy u8d7zN37KwAqA/lX/ogrspEeCoZ6eDj8lNFAMo8Ny6JIUESI7LhSGkdCnoycC9iLjJs8 3wZn2YY6Be2C69PVOwLO7YSTiSdkqm2om5Qkyv9ymR5Zv4HWLCIs9/MskRtnKsb+5UWm Ka0aSPQlPVpj4lXWyJy88XTZc40p+3JjXxpGp5abl2exycIK7FC4HVTiOwAlGM1Hd6ZO jkYg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=pRqQ355M; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id f6-20020a63e306000000b00566022bbcafsi10370698pgh.566.2023.11.15.11.12.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 11:12:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=pRqQ355M; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 53CD48080D45; Wed, 15 Nov 2023 11:12:11 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232814AbjKOTLn (ORCPT + 29 others); Wed, 15 Nov 2023 14:11:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36272 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232757AbjKOTLi (ORCPT ); Wed, 15 Nov 2023 14:11:38 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 18CE2FA; Wed, 15 Nov 2023 11:11:35 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D9CFCC433CA; Wed, 15 Nov 2023 19:11:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1700075494; bh=+CwFhpZFJagUYR6sRV+zAT+zjPxnRMlVxvZF4rtuuss=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pRqQ355MpVkvPG4Din2KnuRYnR/dMc2x1hGi2P6G7B+yVZ2hOFIBMQ4lM3iTqMSXE RJW6mwFeShgda59qG3xg/J3BomgtT4vSBO/ZEL6/FvYY9DK8HUDM/cfcJSEyWOh6CG Hoputu2RO/KdslIEI5HYGubKKLr3tPFbzt4Z+qYBkob18Fma8mEn+Rnwt3viklXZio zA1ZVP9AwOV/D2JIpJ4fMqNutCtaEcFYOSFVe5NubGQfEJGZpCqMWTZOxMSFmK09Kk svMEGvysR9VqV2n68d/M+7+Jdwhh4ahR3lfOpoAfGM2IMbsEBbJsyNoYMoSHaJKnYz YhIHx16iU/LBQ== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , Boqun Feng , Joel Fernandes , Neeraj Upadhyay , Uladzislau Rezki , Zqiang , rcu , "Paul E. McKenney" Subject: [PATCH 3/3] rcu/nocb: Remove needless full barrier after callback advancing Date: Wed, 15 Nov 2023 14:11:28 -0500 Message-ID: <20231115191128.15615-4-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231115191128.15615-1-frederic@kernel.org> References: <20231115191128.15615-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.3 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Wed, 15 Nov 2023 11:12:11 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782658402560599097 X-GMAIL-MSGID: 1782658402560599097 A full barrier is issued from nocb_gp_wait() upon callbacks advancing to order grace period completion with callbacks execution. However these two events are already ordered by the smp_mb__after_unlock_lock() barrier within the call to raw_spin_lock_rcu_node() that is necessary for callbacks advancing to happen. The following litmus test shows the kind of guarantee that this barrier provides: C smp_mb__after_unlock_lock {} // rcu_gp_cleanup() P0(spinlock_t *rnp_lock, int *gpnum) { // Grace period cleanup increase gp sequence number spin_lock(rnp_lock); WRITE_ONCE(*gpnum, 1); spin_unlock(rnp_lock); } // nocb_gp_wait() P1(spinlock_t *rnp_lock, spinlock_t *nocb_lock, int *gpnum, int *cb_ready) { int r1; // Call rcu_advance_cbs() from nocb_gp_wait() spin_lock(nocb_lock); spin_lock(rnp_lock); smp_mb__after_unlock_lock(); r1 = READ_ONCE(*gpnum); WRITE_ONCE(*cb_ready, 1); spin_unlock(rnp_lock); spin_unlock(nocb_lock); } // nocb_cb_wait() P2(spinlock_t *nocb_lock, int *cb_ready, int *cb_executed) { int r2; // rcu_do_batch() -> rcu_segcblist_extract_done_cbs() spin_lock(nocb_lock); r2 = READ_ONCE(*cb_ready); spin_unlock(nocb_lock); // Actual callback execution WRITE_ONCE(*cb_executed, 1); } P3(int *cb_executed, int *gpnum) { int r3; WRITE_ONCE(*cb_executed, 2); smp_mb(); r3 = READ_ONCE(*gpnum); } exists (1:r1=1 /\ 2:r2=1 /\ cb_executed=2 /\ 3:r3=0) (* Bad outcome. *) Here the bad outcome only occurs if the smp_mb__after_unlock_lock() is removed. This barrier orders the grace period completion against callbacks advancing and even later callbacks invocation, thanks to the opportunistic propagation via the ->nocb_lock to nocb_cb_wait(). Therefore the smp_mb() placed after callbacks advancing can be safely removed. Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree.c | 6 ++++++ kernel/rcu/tree_nocb.h | 1 - 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 3ac3c846105f..c557302fc4f5 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -2113,6 +2113,12 @@ static void rcu_do_batch(struct rcu_data *rdp) * Extract the list of ready callbacks, disabling IRQs to prevent * races with call_rcu() from interrupt handlers. Leave the * callback counts, as rcu_barrier() needs to be conservative. + * + * Callbacks execution is fully ordered against preceding grace period + * completion (materialized by rnp->gp_seq update) thanks to the + * smp_mb__after_unlock_lock() upon node locking required for callbacks + * advancing. In NOCB mode this ordering is then further relayed through + * the nocb locking that protects both callbacks advancing and extraction. */ rcu_nocb_lock_irqsave(rdp, flags); WARN_ON_ONCE(cpu_is_offline(smp_processor_id())); diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index eb27878d46f1..d82f96a66600 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -779,7 +779,6 @@ static void nocb_gp_wait(struct rcu_data *my_rdp) if (rcu_segcblist_ready_cbs(&rdp->cblist)) { needwake = rdp->nocb_cb_sleep; WRITE_ONCE(rdp->nocb_cb_sleep, false); - smp_mb(); /* CB invocation -after- GP end. */ } else { needwake = false; }