From patchwork Wed Mar 15 19:43:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 70422 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp99121wrt; Wed, 15 Mar 2023 13:12:58 -0700 (PDT) X-Google-Smtp-Source: AK7set9JvX7yYLCVJeKUKPSPBMxB+zRccEThKcuXHPIiPXdKS1EDZSwG0eGT51FlfbDbvG02718P X-Received: by 2002:a17:903:430b:b0:1a0:600b:fbc6 with SMTP id jz11-20020a170903430b00b001a0600bfbc6mr655886plb.28.1678911178464; Wed, 15 Mar 2023 13:12:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678911178; cv=none; d=google.com; s=arc-20160816; b=ANtRdHXzhvEgRoOfzmzfodjLl2HfYMyzW1yicsJX0a4O0pVYOOckjLTPLXZ1toiaTd 5OcJIo9xgEBzLRudl+hkW0ChMJP+/PKqONcFNklrRaPLCWQ+UUDOnccCMC3bMdiy7aNb MtDFTNQU80Q/ltxJOApqzhyEFsviOTjiOQ2afPL1TBJ37/l7hEu8JZJi3OL0PoVyfjKB 5FJgm2OGjkLYNac0sN2r0AZMUxx1Rp6yj58Jib2v+0b6H5eF76BvKeSatyX2UnISTjPr qDjUZ1LxzVqKiWUx+B3xRsEYYH2z9SDZSTkd3A6dx1M5uNhOgkhfC4KBu+EGi/v9cvhO MUMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=jqFrNqITGGDLlhPYnkJjrrLRlwOYf2ibTxsThXGpk8A=; b=FqMpxNOdeqTecEsDuOtYutyxJBMAkjiGuqohaATHKfXWghUl6IthcJhKli6XtEKnB2 eAivJ5gJXq/y0CRRhT7z2kffGjsCLBNT7NBiyfpiTO9AbET/ocBUKds1CAI9sz1KVblB WjQnPmXYE6NwxgMN0+R+QLR2y9L+ERP6g/CyrFe1OU63H8MeQAdQMvGCweJjjwLLF48g i9JColKq3qXKwEOHZ+C1T0LGf6x3wgsQSDONX3FFgjuqUXxh7fyq23OldybgiZfpHSJ4 COLO1yn/FPqj/nVr9hqfdDuOxRCRogLYgXHuyk2roSUb6MGCTWoxcrQ75X8gQFHIxKoO mA+A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=VEyLp9Oj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t11-20020a6549cb000000b00502fd2d2901si3608263pgs.343.2023.03.15.13.12.42; Wed, 15 Mar 2023 13:12:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=VEyLp9Oj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232223AbjCOTof (ORCPT + 99 others); Wed, 15 Mar 2023 15:44:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232082AbjCOTod (ORCPT ); Wed, 15 Mar 2023 15:44:33 -0400 Received: from mail-qt1-x832.google.com (mail-qt1-x832.google.com [IPv6:2607:f8b0:4864:20::832]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 69C53F756 for ; Wed, 15 Mar 2023 12:44:30 -0700 (PDT) Received: by mail-qt1-x832.google.com with SMTP id s12so17450351qtq.11 for ; Wed, 15 Mar 2023 12:44:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1678909469; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=jqFrNqITGGDLlhPYnkJjrrLRlwOYf2ibTxsThXGpk8A=; b=VEyLp9OjLmJF5YzG1BBVxZJwxAFxMj+a4hkb4juSICDtdMWn8iT2nSGV1mAMg+JGAf usO9i856r9iQFDut5RGxvV/cwJqOKC6yQiQqrFSVmBrJqPxApDUBvZ0+dfZ3iMEtsCLO XfvJmsZnSgkJMKtqKzf1Wmk2eEKQgYX45tu9c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678909469; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=jqFrNqITGGDLlhPYnkJjrrLRlwOYf2ibTxsThXGpk8A=; b=0NLZ9wH4oeZD6AelT24To12kN7qsTbLECnj8SngmxoP6N/wqfwJ6xupgTbWMH7h/yN SaVPJ5Q1KVv/gdpFAo2QIAp7tJQ5Rkule4T6c+UPoZc9sGimLWRa9MPyCX028nog+Hlk Rd0m53XFYUcXg5SD0Ui1PUyp83vZ+CYaY6A4u2r2siLo/T8YXBSC9rqe8+LWnfFwRrIV mOUTZ0k23Q2RzsE3pkwcic4SIVwg/XX6FXFl5nRB/6unRO608UlzWURMIzw0rkCPetAi hWR6kpkCm6/36NfK0P6xGZusG6zgWdS6Hi4Z8nr/XNMtKSrtcOHwPwBD8W0P9faI0iX8 T3xw== X-Gm-Message-State: AO0yUKX8hwA3LGul97sCkB+r5/UeqcO7pMtDPO7Z4DInYrfd2+95RdDD H+ol/yL+q9oQ+ksggsxDcwW/UeTQnRmWINIfL2U= X-Received: by 2002:a05:622a:60f:b0:3b7:ec70:30af with SMTP id z15-20020a05622a060f00b003b7ec7030afmr1753785qta.46.1678909468933; Wed, 15 Mar 2023 12:44:28 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id c15-20020ac8660f000000b003b86b088755sm4346666qtp.15.2023.03.15.12.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 12:44:28 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes Cc: Zqiang , rcu@vger.kernel.org Subject: [PATCH 1/9] rcu: Fix set/clear TICK_DEP_BIT_RCU_EXP bitmask race Date: Wed, 15 Mar 2023 19:43:41 +0000 Message-Id: <20230315194349.10798-1-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760465968045217106?= X-GMAIL-MSGID: =?utf-8?q?1760465968045217106?= From: Zqiang For kernels built with CONFIG_NO_HZ_FULL=y, the following scenario can result in the scheduling-clock interrupt remaining enabled on a holdout CPU after its quiescent state has been reported: CPU1 CPU2 rcu_report_exp_cpu_mult synchronize_rcu_expedited_wait acquires rnp->lock mask = rnp->expmask; for_each_leaf_node_cpu_mask(rnp, cpu, mask) rnp->expmask = rnp->expmask & ~mask; rdp = per_cpu_ptr(&rcu_data, cpu1); for_each_leaf_node_cpu_mask(rnp, cpu, mask) rdp = per_cpu_ptr(&rcu_data, cpu1); if (!rdp->rcu_forced_tick_exp) continue; rdp->rcu_forced_tick_exp = true; tick_dep_set_cpu(cpu1, TICK_DEP_BIT_RCU_EXP); The problem is that CPU2's sampling of rnp->expmask is obsolete by the time it invokes tick_dep_set_cpu(), and CPU1 is not guaranteed to see CPU2's store to ->rcu_forced_tick_exp in time to clear it. And even if CPU1 does see that store, it might invoke tick_dep_clear_cpu() before CPU2 got around to executing its tick_dep_set_cpu(), which would still leave the victim CPU with its scheduler-clock tick running. Either way, an nohz_full real-time application running on the victim CPU would have its latency needlessly degraded. Note that expedited RCU grace periods look at context-tracking information, and so if the CPU is executing in nohz_full usermode throughout, that CPU cannot be victimized in this manner. This commit therefore causes synchronize_rcu_expedited_wait to hold the rcu_node structure's ->lock when checking for holdout CPUs, setting TICK_DEP_BIT_RCU_EXP, and invoking tick_dep_set_cpu(), thus preventing this race. Signed-off-by: Zqiang Reviewed-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_exp.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 249c2967d9e6..7cc4856da081 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -594,6 +594,7 @@ static void synchronize_rcu_expedited_wait(void) struct rcu_data *rdp; struct rcu_node *rnp; struct rcu_node *rnp_root = rcu_get_root(); + unsigned long flags; trace_rcu_exp_grace_period(rcu_state.name, rcu_exp_gp_seq_endval(), TPS("startwait")); jiffies_stall = rcu_exp_jiffies_till_stall_check(); @@ -602,17 +603,17 @@ static void synchronize_rcu_expedited_wait(void) if (synchronize_rcu_expedited_wait_once(1)) return; rcu_for_each_leaf_node(rnp) { + raw_spin_lock_irqsave_rcu_node(rnp, flags); mask = READ_ONCE(rnp->expmask); for_each_leaf_node_cpu_mask(rnp, cpu, mask) { rdp = per_cpu_ptr(&rcu_data, cpu); if (rdp->rcu_forced_tick_exp) continue; rdp->rcu_forced_tick_exp = true; - preempt_disable(); if (cpu_online(cpu)) tick_dep_set_cpu(cpu, TICK_DEP_BIT_RCU_EXP); - preempt_enable(); } + raw_spin_unlock_irqrestore_rcu_node(rnp, flags); } j = READ_ONCE(jiffies_till_first_fqs); if (synchronize_rcu_expedited_wait_once(j + HZ))