From patchwork Wed Oct 19 22:51:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 5901 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp572968wrs; Wed, 19 Oct 2022 15:57:06 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7HQwOyciiEklBbo3cvzK013IYSbp0ds4DEcdBZ2zfdDkvM/kZUe3MoKYVQM66yLgiQo4U0 X-Received: by 2002:a05:6402:3988:b0:44e:6f08:ddfb with SMTP id fk8-20020a056402398800b0044e6f08ddfbmr9479722edb.89.1666220226258; Wed, 19 Oct 2022 15:57:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666220226; cv=none; d=google.com; s=arc-20160816; b=IBFkq1KEl0nFtH39/jkAZDBLMf2gHyUv0opF3m+Em+M23bTf4jcgAzJUMDw9ZTRCaQ sGv3QxpWHfql5Nmzjf5B/ZF83Ve89LEJDFMkrpMf7Lt3mvahFpZR7Q3hQ3WI8e8iiF6z u1LkfvetSeTcT94vvSWtfrInuqjQ8tt77KuJe4IKpsepV14yagit3waOGKlfAstF9yhu d6jvS1IfxswthOUcWbF/nqQrVh+NBEVdHhHB4Hyc/+G2aD17CUq3mo0DAfSOis/T+I1h T7QkSSRmeghGdcBXgdHDupqkC4bH360+Z2H47NQDdCb/ptHAwEjriR0Dim2GCv6c8CXz G/yw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=U4oVFdZuYHclhhiYM3y5LbmYNw6QTk652e3IwS9BkbM=; b=T5Tgn3v0eXkqwPnilSdxWpv4vYkmuBcUpKkW8e63n2mtZK2adFeyrZK61BImHAnrLS DKMgQZcTQXynRZ4j+QVo02OCGq8xL/iwOzCEUU0FKtYXlX9rKWwCg4nA+3qZn/tpWx8R 9VEWRc+F7Wg2AJssgNELXoKGrK80VYAfGmM++z0g5wBI68o9Qnrtv2lz/WCAvAUJQ1UP AZd7BIys6ccLfjDq7W+g56Ln6E7zY+lAzwmU8hWIysNzgCL6AxY1DYzRO52ZGP1C4NpT 5LP57hBnveofjxvasnaIVz3oY7v+lYyVDrgJeaWyFGYDvXCTxpH0ttH4N26BsuHq2yKb GWpg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=kFduOefc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id eb11-20020a0564020d0b00b0045d189ac60esi17393384edb.401.2022.10.19.15.56.41; Wed, 19 Oct 2022 15:57:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=kFduOefc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230498AbiJSWw1 (ORCPT + 99 others); Wed, 19 Oct 2022 18:52:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51268 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230452AbiJSWvx (ORCPT ); Wed, 19 Oct 2022 18:51:53 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B528C190472; Wed, 19 Oct 2022 15:51:50 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 618D9CE242E; Wed, 19 Oct 2022 22:51:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B7EA2C433D6; Wed, 19 Oct 2022 22:51:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666219906; bh=cd9rgMHV5ytGRmwwPo9gYzUKZlyUq56zzJaeb9ZJrZY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kFduOefcF3DL5P78TykY4+I41g8zwf9ipFCX1ghnGq+rZ2F7oeLDQ3t6/PGJHe5FT lhhVZ1vE522OBbGx19z+6enNorzxpeNLqUQqIB49qVbTuc4u0zKMUKcQV3o1/ebZQP IUKC2sdvWq9i9w/EgpY1/G59nv+d3Hd/RoeLr85Yreq5Dv0ELqosSmsSPMf5iakGgI a10LJ/nGnHeUBJf7EYdPQRXemru4WmQJPebsrjdXUEB+1Ua8tZibX/zkrBIBcIYA9o 5vBM4oRDM4qnD6o5tpJIRuI/AeTFdp+F0capyD+rTMI/DjP2mczY42srq6bGppUraC 30lefLU9VMigQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 787575C0879; Wed, 19 Oct 2022 15:51:46 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH rcu 02/14] rcu: Fix late wakeup when flush of bypass cblist happens Date: Wed, 19 Oct 2022 15:51:32 -0700 Message-Id: <20221019225144.2500095-2-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221019225138.GA2499943@paulmck-ThinkPad-P17-Gen-1> References: <20221019225138.GA2499943@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747158539776672368?= X-GMAIL-MSGID: =?utf-8?q?1747158539776672368?= From: "Joel Fernandes (Google)" When the bypass cblist gets too big or its timeout has occurred, it is flushed into the main cblist. However, the bypass timer is still running and the behavior is that it would eventually expire and wake the GP thread. Since we are going to use the bypass cblist for lazy CBs, do the wakeup soon as the flush for "too big or too long" bypass list happens. Otherwise, long delays can happen for callbacks which get promoted from lazy to non-lazy. This is a good thing to do anyway (regardless of future lazy patches), since it makes the behavior consistent with behavior of other code paths where flushing into the ->cblist makes the GP kthread into a non-sleeping state quickly. [ Frederic Weisbecker: Changes to avoid unnecessary GP-thread wakeups plus comment changes. ] Reviewed-by: Frederic Weisbecker Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index ce526cc2791ca..f77a6d7e13564 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -433,8 +433,9 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, if ((ncbs && j != READ_ONCE(rdp->nocb_bypass_first)) || ncbs >= qhimark) { rcu_nocb_lock(rdp); + *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); + if (!rcu_nocb_flush_bypass(rdp, rhp, j)) { - *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstQ")); @@ -447,7 +448,12 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, rcu_advance_cbs_nowake(rdp->mynode, rdp); rdp->nocb_gp_adv_time = j; } - rcu_nocb_unlock_irqrestore(rdp, flags); + + // The flush succeeded and we moved CBs into the regular list. + // Don't wait for the wake up timer as it may be too far ahead. + // Wake up the GP thread now instead, if the cblist was empty. + __call_rcu_nocb_wake(rdp, *was_alldone, flags); + return true; // Callback already enqueued. }