From patchwork Tue Oct 17 12:19:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 154234 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4092082vqb; Tue, 17 Oct 2023 05:19:51 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFNkCIlx5oJQe1Bf7Z5MI/rL4+MuuQ1DEvoh7ASh7Uhzgg4D2xq6Sokll7JVyrZIgaq8UBv X-Received: by 2002:a17:903:340c:b0:1ca:1ce1:bfac with SMTP id ke12-20020a170903340c00b001ca1ce1bfacmr2062748plb.1.1697545191182; Tue, 17 Oct 2023 05:19:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697545191; cv=none; d=google.com; s=arc-20160816; b=QwIYUJ5goCQnByu53TFPcwbPgN+h3CETL5jBGEO7Rex3oGCFulYs2PQNAIMICEUEmM PHgqW87gRmBJ0otUOtus0OyJn54xv6o+W+65m22kBINMdZsTirYEwCb1hg9PZlsMNdwp 6STeFX9CBuyelFsxbhGU3awrNXRF9rY44X2cJBMi8YB0NP4wTurGvSiK0XprGl4hREcR 7NqS/lVGSpTK9o3V/CV9UBlti7vxwFdz7ydXLREzNrynt1rVAKNdcuHvJf9bEf+5FREM 8h9kOOJbXs65qGfYRdPyyr80FeBj88a+0KR28YEBCRBDEEZNdW7gyEMdRx5OM5YEJ+dW 1AlQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5220byjYiS9sIkEdrurHlSWwJR0Y4aZbZTrxatThIv4=; fh=lu4ND71kuCJoIq12QTO73nSs+Y7WmfMYUMyi8q/LiXw=; b=NGcv/nkQNN94rJNtk5akpfGbjCnm7nud0TUNPAzZnF7+DfXamtxMTcJI49dZtDYKbn hF2Ut5GJkYtCVE3Hzo36sFuz39Yda7xScdQ5/TwToGHWYmqnNCPSdheaIQzQ5F8CJoEn 3jPWUFLbAAa6rJR2kmI/+KCtqv7fu3wylJzwBCGDRQWh3QPsY3YKzwVzdM+WtYPyV0Qj jWczUIUUgMXnT2WvtXZkw/NwbEYbD2Ng3xX3mK5QSlYVZ8yN1OzSpglWGbjFQHO2c0l7 e8v60Dh9wJxsHwLfcBAAYcYcy6EOXJPr+9btX3Ft9VkUXGOMjh8E9xxYLOu+isP06tho yK3w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=TTiYxOGU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id c12-20020a170903234c00b001bbad1883d5si1824628plh.293.2023.10.17.05.19.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 05:19:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=TTiYxOGU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 3395A801C09F; Tue, 17 Oct 2023 05:19:50 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343584AbjJQMTr (ORCPT + 19 others); Tue, 17 Oct 2023 08:19:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49622 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234808AbjJQMTb (ORCPT ); Tue, 17 Oct 2023 08:19:31 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7262C3C0B; Tue, 17 Oct 2023 05:19:18 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6FC6BC433CA; Tue, 17 Oct 2023 12:19:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1697545157; bh=h9+pFFk6S6/pR3kzOkCCUyr4lY/4hLmqaxjAdWypwXw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TTiYxOGUpdxl6LnEq0oYQHVVWNtIIFS0BNm10v346xnZCDzUSzpWjskcFRF5HpDUq g4xKUIiR3l0nyHMboEFtRZ+B20T+xF9aWoculjjsXgoRAqq1eC8gYcWqRsFcNqhUxt 50INoQ5wOe/MgwIFM7Eo7xDC4/fvVuLlth7fpn5ba9cyxyj81Jn0VP5HyNGAriH09O 3tc5q+YuSqUcduYHj2CkebBIkkXy5VKVyKP78mwK0XG4oWtN3GBLiye8mtWt5Hfrj8 ENxImfz+yapBpyjn9qAyUyAwCcYER8UnzkcGd4ay8H9fa35EGO+SoABpltdsL15QSQ onIIgMpDKFTpg== From: Frederic Weisbecker To: LKML Cc: "Paul E. McKenney" , Boqun Feng , Joel Fernandes , Josh Triplett , Mathieu Desnoyers , Neeraj Upadhyay , Steven Rostedt , Uladzislau Rezki , rcu , Linus Torvalds , Frederic Weisbecker Subject: [PATCH 2/3] rcu-tasks: Pull sampling of ->percpu_dequeue_lim out of loop Date: Tue, 17 Oct 2023 14:19:04 +0200 Message-Id: <20231017121905.1156166-3-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231017121905.1156166-1-frederic@kernel.org> References: <20231017121905.1156166-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Tue, 17 Oct 2023 05:19:50 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780005146640498828 X-GMAIL-MSGID: 1780005146640498828 From: "Paul E. McKenney" The rcu_tasks_need_gpcb() samples ->percpu_dequeue_lim as part of the condition clause of a "for" loop, which is a bit confusing. This commit therefore hoists this sampling out of the loop, using the result loaded in the condition clause. So why does this work in the face of a concurrent switch from single-CPU queueing to per-CPU queueing? o The call_rcu_tasks_generic() that makes the change has already enqueued its callback, which means that all of the other CPU's callback queues are empty. o For the call_rcu_tasks_generic() that first notices the switch to per-CPU queues, the smp_store_release() used to update ->percpu_enqueue_lim pairs with the raw_spin_trylock_rcu_node()'s full barrier that is between the READ_ONCE(rtp->percpu_enqueue_shift) and the rcu_segcblist_enqueue() that enqueues the callback. o Because this CPU's queue is empty (unless it happens to be the original single queue, in which case there is no need for synchronization), this call_rcu_tasks_generic() will do an irq_work_queue() to schedule a handler for the needed rcuwait_wake_up() call. This call will be ordered after the first call_rcu_tasks_generic() function's change to ->percpu_dequeue_lim. o This rcuwait_wake_up() will either happen before or after the set_current_state() in rcuwait_wait_event(). If it happens before, the "condition" argument's call to rcu_tasks_need_gpcb() will be ordered after the original change, and all callbacks on all CPUs will be visible. Otherwise, if it happens after, then the grace-period kthread's state will be set back to running, which will result in a later call to rcuwait_wait_event() and thus to rcu_tasks_need_gpcb(), which will again see the change. So it all works out. Suggested-by: Linus Torvalds Signed-off-by: Paul E. McKenney Signed-off-by: Frederic Weisbecker --- kernel/rcu/tasks.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 83049a893de5..94bb5abdbb37 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -432,6 +432,7 @@ static void rcu_barrier_tasks_generic(struct rcu_tasks *rtp) static int rcu_tasks_need_gpcb(struct rcu_tasks *rtp) { int cpu; + int dequeue_limit; unsigned long flags; bool gpdone = poll_state_synchronize_rcu(rtp->percpu_dequeue_gpseq); long n; @@ -439,7 +440,8 @@ static int rcu_tasks_need_gpcb(struct rcu_tasks *rtp) long ncbsnz = 0; int needgpcb = 0; - for (cpu = 0; cpu < smp_load_acquire(&rtp->percpu_dequeue_lim); cpu++) { + dequeue_limit = smp_load_acquire(&rtp->percpu_dequeue_lim); + for (cpu = 0; cpu < dequeue_limit; cpu++) { struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rtp->rtpcpu, cpu); /* Advance and accelerate any new callbacks. */