From patchwork Tue Nov 22 01:04:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 24076 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1929662wrr; Mon, 21 Nov 2022 17:04:45 -0800 (PST) X-Google-Smtp-Source: AA0mqf5S29jQRz205vVfTUBy04SMKgTjob2SbmZz6tVzMeKHzUqB083oj8LJ1uShkT5IjcRpDne5 X-Received: by 2002:a17:902:e383:b0:186:9c70:9b7d with SMTP id g3-20020a170902e38300b001869c709b7dmr1963471ple.3.1669079085240; Mon, 21 Nov 2022 17:04:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669079085; cv=none; d=google.com; s=arc-20160816; b=ttPaTqoNEUY/v4feJ1rZJG5fOt4XRyvNYc26vZL5A+dn86vmks85bWBI5gBpZr59Zr DxL1Bu1Ieb0K/ft31iY9rCzw3llUnNm1t1EjqG4NOxD27w92zLn8ktPYH0PYI2/iUGBC /rxvlpxAErwQacHjt43Lu+YBtCRK1Pb9dJ1DLF32OpBikUybKHqsqG6ckleSghH3X2qq zaA90ZPfPQCwGVBg2WtAiolUo2WDS7KxXxoDEWq0n5sHv2EoAJvywYaA8JfDr9guOWrO +fVzDDMiEFZkZi7DXK/rIR02CrI4JoqNnY0OPCDH8VdSWZ3nquu9aQtVGu6xYvCgOFW+ RIVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=U4oVFdZuYHclhhiYM3y5LbmYNw6QTk652e3IwS9BkbM=; b=WGWHbXoGrtijRENTj0abbIOYhU2oNhkJ7CG4n7pco89dKnXt6xpjlQI841BjgUfee/ YCUDT7Dj81K8kEv/iasU0zuFaJw7AtWqgI4X7TAc4eolPKgwD/SW3MOGhizyT9JP6NjQ GikNozmsfEXhQ/kLR4j0dcEOi718rBTt1KExKK+CYU0qpd5nHGSWFTTa1I6SqCUnNAqW oWAJwmzvJtIQeyAilDgpiuQCMPeORo8PiGYz7gTywDxJw8lTaIr4qdvnSbopkTjtowa9 vcXAmK7BHs+jsbdJ4G3TZs1tbRc2Qa1qAmJ0Jv72SnrUjcbEmf3l6dzry5c53TEfrs++ qVEQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=WD+d1eZ5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ju17-20020a170903429100b001890f5957c4si6979862plb.353.2022.11.21.17.04.32; Mon, 21 Nov 2022 17:04:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=WD+d1eZ5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231960AbiKVBE0 (ORCPT + 99 others); Mon, 21 Nov 2022 20:04:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44768 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231924AbiKVBEY (ORCPT ); Mon, 21 Nov 2022 20:04:24 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ACFA3DE9A; Mon, 21 Nov 2022 17:04:23 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4A8AA61511; Tue, 22 Nov 2022 01:04:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B06BBC433D6; Tue, 22 Nov 2022 01:04:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1669079062; bh=cd9rgMHV5ytGRmwwPo9gYzUKZlyUq56zzJaeb9ZJrZY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WD+d1eZ5hWZsn4OWdNBCqObluC84BcwFQc1I/202wtBNI1lrqmeLZYnqJmFhRutjM 88QawAXneNZ6ZoDbJB16S+LBDCrWPqIHNk1nJiHKQLhsZJmKh1X7+3RYba6JOKUuJ0 F8+GgwLr3axpteO3550Si3usU+zWLDUynWclDTMHREi1jg29LgheeO9g9V4McYjWDT cic5G6/57tHpDiFYUDWk7RbinwGOE4UaSvTC1myCs1rs/d6fNSuF2MTHAaCxN3jyMK ZyBVnu3wW3Jad8wia6bzdvawwqXqyxr9uwgcB97vtFf3bCEN/4KZ2EXBzF0i79/HvO dE/js4FqZfQ6g== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 738485C0909; Mon, 21 Nov 2022 17:04:22 -0800 (PST) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, "Joel Fernandes (Google)" , Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH v2 rcu 02/16] rcu: Fix late wakeup when flush of bypass cblist happens Date: Mon, 21 Nov 2022 17:04:07 -0800 Message-Id: <20221122010421.3799681-2-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> References: <20221122010408.GA3799268@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750156271106679106?= X-GMAIL-MSGID: =?utf-8?q?1750156271106679106?= From: "Joel Fernandes (Google)" When the bypass cblist gets too big or its timeout has occurred, it is flushed into the main cblist. However, the bypass timer is still running and the behavior is that it would eventually expire and wake the GP thread. Since we are going to use the bypass cblist for lazy CBs, do the wakeup soon as the flush for "too big or too long" bypass list happens. Otherwise, long delays can happen for callbacks which get promoted from lazy to non-lazy. This is a good thing to do anyway (regardless of future lazy patches), since it makes the behavior consistent with behavior of other code paths where flushing into the ->cblist makes the GP kthread into a non-sleeping state quickly. [ Frederic Weisbecker: Changes to avoid unnecessary GP-thread wakeups plus comment changes. ] Reviewed-by: Frederic Weisbecker Signed-off-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney --- kernel/rcu/tree_nocb.h | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index ce526cc2791ca..f77a6d7e13564 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -433,8 +433,9 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, if ((ncbs && j != READ_ONCE(rdp->nocb_bypass_first)) || ncbs >= qhimark) { rcu_nocb_lock(rdp); + *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); + if (!rcu_nocb_flush_bypass(rdp, rhp, j)) { - *was_alldone = !rcu_segcblist_pend_cbs(&rdp->cblist); if (*was_alldone) trace_rcu_nocb_wake(rcu_state.name, rdp->cpu, TPS("FirstQ")); @@ -447,7 +448,12 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, rcu_advance_cbs_nowake(rdp->mynode, rdp); rdp->nocb_gp_adv_time = j; } - rcu_nocb_unlock_irqrestore(rdp, flags); + + // The flush succeeded and we moved CBs into the regular list. + // Don't wait for the wake up timer as it may be too far ahead. + // Wake up the GP thread now instead, if the cblist was empty. + __call_rcu_nocb_wake(rdp, *was_alldone, flags); + return true; // Callback already enqueued. }