From patchwork Wed Mar 15 19:43:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 70422 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp99121wrt; Wed, 15 Mar 2023 13:12:58 -0700 (PDT) X-Google-Smtp-Source: AK7set9JvX7yYLCVJeKUKPSPBMxB+zRccEThKcuXHPIiPXdKS1EDZSwG0eGT51FlfbDbvG02718P X-Received: by 2002:a17:903:430b:b0:1a0:600b:fbc6 with SMTP id jz11-20020a170903430b00b001a0600bfbc6mr655886plb.28.1678911178464; Wed, 15 Mar 2023 13:12:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678911178; cv=none; d=google.com; s=arc-20160816; b=ANtRdHXzhvEgRoOfzmzfodjLl2HfYMyzW1yicsJX0a4O0pVYOOckjLTPLXZ1toiaTd 5OcJIo9xgEBzLRudl+hkW0ChMJP+/PKqONcFNklrRaPLCWQ+UUDOnccCMC3bMdiy7aNb MtDFTNQU80Q/ltxJOApqzhyEFsviOTjiOQ2afPL1TBJ37/l7hEu8JZJi3OL0PoVyfjKB 5FJgm2OGjkLYNac0sN2r0AZMUxx1Rp6yj58Jib2v+0b6H5eF76BvKeSatyX2UnISTjPr qDjUZ1LxzVqKiWUx+B3xRsEYYH2z9SDZSTkd3A6dx1M5uNhOgkhfC4KBu+EGi/v9cvhO MUMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=jqFrNqITGGDLlhPYnkJjrrLRlwOYf2ibTxsThXGpk8A=; b=FqMpxNOdeqTecEsDuOtYutyxJBMAkjiGuqohaATHKfXWghUl6IthcJhKli6XtEKnB2 eAivJ5gJXq/y0CRRhT7z2kffGjsCLBNT7NBiyfpiTO9AbET/ocBUKds1CAI9sz1KVblB WjQnPmXYE6NwxgMN0+R+QLR2y9L+ERP6g/CyrFe1OU63H8MeQAdQMvGCweJjjwLLF48g i9JColKq3qXKwEOHZ+C1T0LGf6x3wgsQSDONX3FFgjuqUXxh7fyq23OldybgiZfpHSJ4 COLO1yn/FPqj/nVr9hqfdDuOxRCRogLYgXHuyk2roSUb6MGCTWoxcrQ75X8gQFHIxKoO mA+A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=VEyLp9Oj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t11-20020a6549cb000000b00502fd2d2901si3608263pgs.343.2023.03.15.13.12.42; Wed, 15 Mar 2023 13:12:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=VEyLp9Oj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232223AbjCOTof (ORCPT + 99 others); Wed, 15 Mar 2023 15:44:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232082AbjCOTod (ORCPT ); Wed, 15 Mar 2023 15:44:33 -0400 Received: from mail-qt1-x832.google.com (mail-qt1-x832.google.com [IPv6:2607:f8b0:4864:20::832]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 69C53F756 for ; Wed, 15 Mar 2023 12:44:30 -0700 (PDT) Received: by mail-qt1-x832.google.com with SMTP id s12so17450351qtq.11 for ; Wed, 15 Mar 2023 12:44:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1678909469; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=jqFrNqITGGDLlhPYnkJjrrLRlwOYf2ibTxsThXGpk8A=; b=VEyLp9OjLmJF5YzG1BBVxZJwxAFxMj+a4hkb4juSICDtdMWn8iT2nSGV1mAMg+JGAf usO9i856r9iQFDut5RGxvV/cwJqOKC6yQiQqrFSVmBrJqPxApDUBvZ0+dfZ3iMEtsCLO XfvJmsZnSgkJMKtqKzf1Wmk2eEKQgYX45tu9c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678909469; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=jqFrNqITGGDLlhPYnkJjrrLRlwOYf2ibTxsThXGpk8A=; b=0NLZ9wH4oeZD6AelT24To12kN7qsTbLECnj8SngmxoP6N/wqfwJ6xupgTbWMH7h/yN SaVPJ5Q1KVv/gdpFAo2QIAp7tJQ5Rkule4T6c+UPoZc9sGimLWRa9MPyCX028nog+Hlk Rd0m53XFYUcXg5SD0Ui1PUyp83vZ+CYaY6A4u2r2siLo/T8YXBSC9rqe8+LWnfFwRrIV mOUTZ0k23Q2RzsE3pkwcic4SIVwg/XX6FXFl5nRB/6unRO608UlzWURMIzw0rkCPetAi hWR6kpkCm6/36NfK0P6xGZusG6zgWdS6Hi4Z8nr/XNMtKSrtcOHwPwBD8W0P9faI0iX8 T3xw== X-Gm-Message-State: AO0yUKX8hwA3LGul97sCkB+r5/UeqcO7pMtDPO7Z4DInYrfd2+95RdDD H+ol/yL+q9oQ+ksggsxDcwW/UeTQnRmWINIfL2U= X-Received: by 2002:a05:622a:60f:b0:3b7:ec70:30af with SMTP id z15-20020a05622a060f00b003b7ec7030afmr1753785qta.46.1678909468933; Wed, 15 Mar 2023 12:44:28 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id c15-20020ac8660f000000b003b86b088755sm4346666qtp.15.2023.03.15.12.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 12:44:28 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes Cc: Zqiang , rcu@vger.kernel.org Subject: [PATCH 1/9] rcu: Fix set/clear TICK_DEP_BIT_RCU_EXP bitmask race Date: Wed, 15 Mar 2023 19:43:41 +0000 Message-Id: <20230315194349.10798-1-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760465968045217106?= X-GMAIL-MSGID: =?utf-8?q?1760465968045217106?= From: Zqiang For kernels built with CONFIG_NO_HZ_FULL=y, the following scenario can result in the scheduling-clock interrupt remaining enabled on a holdout CPU after its quiescent state has been reported: CPU1 CPU2 rcu_report_exp_cpu_mult synchronize_rcu_expedited_wait acquires rnp->lock mask = rnp->expmask; for_each_leaf_node_cpu_mask(rnp, cpu, mask) rnp->expmask = rnp->expmask & ~mask; rdp = per_cpu_ptr(&rcu_data, cpu1); for_each_leaf_node_cpu_mask(rnp, cpu, mask) rdp = per_cpu_ptr(&rcu_data, cpu1); if (!rdp->rcu_forced_tick_exp) continue; rdp->rcu_forced_tick_exp = true; tick_dep_set_cpu(cpu1, TICK_DEP_BIT_RCU_EXP); The problem is that CPU2's sampling of rnp->expmask is obsolete by the time it invokes tick_dep_set_cpu(), and CPU1 is not guaranteed to see CPU2's store to ->rcu_forced_tick_exp in time to clear it. And even if CPU1 does see that store, it might invoke tick_dep_clear_cpu() before CPU2 got around to executing its tick_dep_set_cpu(), which would still leave the victim CPU with its scheduler-clock tick running. Either way, an nohz_full real-time application running on the victim CPU would have its latency needlessly degraded. Note that expedited RCU grace periods look at context-tracking information, and so if the CPU is executing in nohz_full usermode throughout, that CPU cannot be victimized in this manner. This commit therefore causes synchronize_rcu_expedited_wait to hold the rcu_node structure's ->lock when checking for holdout CPUs, setting TICK_DEP_BIT_RCU_EXP, and invoking tick_dep_set_cpu(), thus preventing this race. Signed-off-by: Zqiang Reviewed-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_exp.h | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 249c2967d9e6..7cc4856da081 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -594,6 +594,7 @@ static void synchronize_rcu_expedited_wait(void) struct rcu_data *rdp; struct rcu_node *rnp; struct rcu_node *rnp_root = rcu_get_root(); + unsigned long flags; trace_rcu_exp_grace_period(rcu_state.name, rcu_exp_gp_seq_endval(), TPS("startwait")); jiffies_stall = rcu_exp_jiffies_till_stall_check(); @@ -602,17 +603,17 @@ static void synchronize_rcu_expedited_wait(void) if (synchronize_rcu_expedited_wait_once(1)) return; rcu_for_each_leaf_node(rnp) { + raw_spin_lock_irqsave_rcu_node(rnp, flags); mask = READ_ONCE(rnp->expmask); for_each_leaf_node_cpu_mask(rnp, cpu, mask) { rdp = per_cpu_ptr(&rcu_data, cpu); if (rdp->rcu_forced_tick_exp) continue; rdp->rcu_forced_tick_exp = true; - preempt_disable(); if (cpu_online(cpu)) tick_dep_set_cpu(cpu, TICK_DEP_BIT_RCU_EXP); - preempt_enable(); } + raw_spin_unlock_irqrestore_rcu_node(rnp, flags); } j = READ_ONCE(jiffies_till_first_fqs); if (synchronize_rcu_expedited_wait_once(j + HZ)) From patchwork Wed Mar 15 19:43:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 70419 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp92108wrt; Wed, 15 Mar 2023 12:58:07 -0700 (PDT) X-Google-Smtp-Source: AK7set+bHYYx1AyMz8/gl+hgAgsinJKX0+SrVXIgjA0iH5wZI67EyAemZlbTzzhrH0XtV8QktjZd X-Received: by 2002:a05:6a20:4408:b0:cd:3069:28cd with SMTP id ce8-20020a056a20440800b000cd306928cdmr969394pzb.39.1678910287200; Wed, 15 Mar 2023 12:58:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678910287; cv=none; d=google.com; s=arc-20160816; b=qt48OhakGuAXI+Us1XQN9DLyHBg2dySHlb4nUlyg/j+NzSaJ/tiJpOMzvr5QaAwMmO Mh63dBayqUMlg9qAtMd1sBw4fwihKiCe8WmcgGVcbK3Vh7QWi6LSxFrIcRvnP2tsJk2c UAriXnNHX0U9ADRK6EKwYF3doeG6nx212yZFR/S68b5kECqXyVgaM6X3famGTacrobAg F8wrDfbjZjeVdLST3dxIirQgjJEKxqY6PsSc3KvCvQfwC6vTbvwHYpheW/o5nupDxfBa xMWIadBxe5V4q/+Ye1+plywbfw0rfS4BitsLBc0Y3QohA/sXtY735px2h61PvkQzMCXJ u0hA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=v18L/HyzbX8RIGPkv74gEAaNuKF2umfPmevTB4XROmQ=; b=DBVPQimvI2CgLNgkurEaZZsIL8ncfD7fsPqOMZeb5Q2Yy9UcQ0+UNWWHBfl83OH6x6 8tANnawc+qPRr9PkQ/UcSUathLdq0QpqoSwDLckjeFy3V2PaaWGaJjI6ggVDETqysch1 sB7m6hsflZ2geEjlS/9JRwaQ+YcTi4vbMIfthYW8/A5iSV6Awo3ZtpWv+NwRKQ8kxjJK 7qu64XPweBTu5gkFliiKYxM7MxE2da4aYMVp7aqMRzGHy2/6viJDYnWy+lrjPzwQZFpY CPbTZnm2tk7zBTnKBUpXfeerdSH7EVHPqjtpOYjo8SP3cwjMIaULtfA3/6eVkVZ0RVrI 8IhQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=LAK5vaKQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p2-20020a634202000000b005030c349b9fsi5543434pga.577.2023.03.15.12.57.54; Wed, 15 Mar 2023 12:58:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=LAK5vaKQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232545AbjCOTom (ORCPT + 99 others); Wed, 15 Mar 2023 15:44:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47552 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232375AbjCOTog (ORCPT ); Wed, 15 Mar 2023 15:44:36 -0400 Received: from mail-qt1-x82d.google.com (mail-qt1-x82d.google.com [IPv6:2607:f8b0:4864:20::82d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B0A8ACA0B for ; Wed, 15 Mar 2023 12:44:31 -0700 (PDT) Received: by mail-qt1-x82d.google.com with SMTP id r5so17505135qtp.4 for ; Wed, 15 Mar 2023 12:44:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1678909470; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=v18L/HyzbX8RIGPkv74gEAaNuKF2umfPmevTB4XROmQ=; b=LAK5vaKQ7yNuOXwJ/bmj+BxFvK6WV9egm/yBtxMZRQ86LkU+ecP3uQvSIt9nrBkN+u Zx3IzVf7Qvxjmyq2vNKFl9qY9SmA2yHS3dUcZJYtGNJdYLUNEp14r4a9nDEFBOf2I51P K36kb3x302AK7FfweoSCaIdBujCIksdSNSirg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678909470; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v18L/HyzbX8RIGPkv74gEAaNuKF2umfPmevTB4XROmQ=; b=M8oV4xLsK7P6zFpwteiJoXb6gSY9PKxwDGJxixRbz+dYxrX94JpipncPHEEv00Xgid RZLwUZMDG2qP5hGPeA1HWQpAk6qPLAHHgbdvWHS6LtyFhSqc4+c0HEReVNytBkOGjkDI EfDQXGhVWfjmwpRSghnXZDxntS/1+cNnMYeGJ9AW7RPpfXarUWCGgVFXHhsOF5mjyb4f ezVSNJaCrt3crrC8jRDCNRT1+NchUbu+iPsBfSXhtm9SNeZCNJd0z1Z+BdkwDoXVTG3i gDz9vueQi9NwVD9m8m0hLfxpkNzPYSNp5QjAXdkMNcfNZJ3aId7BSrMhFhUZ4xQzr8cP /Flw== X-Gm-Message-State: AO0yUKWHVBcBO6Ps8k88WbOHdl4P0Q/6sa3BTYW622BbRw/4eJnINCn6 tmEOdh3+RJnmVCdiUzPUCjo4y0znVL/xhF20lS0= X-Received: by 2002:a05:622a:174c:b0:3bf:d1b3:2bbb with SMTP id l12-20020a05622a174c00b003bfd1b32bbbmr2125188qtk.13.1678909470197; Wed, 15 Mar 2023 12:44:30 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id c15-20020ac8660f000000b003b86b088755sm4346666qtp.15.2023.03.15.12.44.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 12:44:29 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, Steven Rostedt , Masami Hiramatsu , Frederic Weisbecker , Thomas Gleixner , Ingo Molnar , "Paul E. McKenney" Cc: Zqiang , Anna-Maria Behnsen , Frederic Weisbecker , Joel Fernandes , linux-trace-kernel@vger.kernel.org Subject: [PATCH 2/9] rcu: Fix missing TICK_DEP_MASK_RCU_EXP dependency check Date: Wed, 15 Mar 2023 19:43:42 +0000 Message-Id: <20230315194349.10798-2-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog In-Reply-To: <20230315194349.10798-1-joel@joelfernandes.org> References: <20230315194349.10798-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760465033146911145?= X-GMAIL-MSGID: =?utf-8?q?1760465033146911145?= From: Zqiang This commit adds checks for the TICK_DEP_MASK_RCU_EXP bit, thus enabling RCU expedited grace periods to actually force-enable scheduling-clock interrupts on holdout CPUs. Fixes: df1e849ae455 ("rcu: Enable tick for nohz_full CPUs slow to provide expedited QS") Signed-off-by: Zqiang Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Frederic Weisbecker Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Anna-Maria Behnsen Acked-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- include/trace/events/timer.h | 3 ++- kernel/time/tick-sched.c | 5 +++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/include/trace/events/timer.h b/include/trace/events/timer.h index 2e713a7d9aa3..3e8619c72f77 100644 --- a/include/trace/events/timer.h +++ b/include/trace/events/timer.h @@ -371,7 +371,8 @@ TRACE_EVENT(itimer_expire, tick_dep_name(PERF_EVENTS) \ tick_dep_name(SCHED) \ tick_dep_name(CLOCK_UNSTABLE) \ - tick_dep_name_end(RCU) + tick_dep_name(RCU) \ + tick_dep_name_end(RCU_EXP) #undef tick_dep_name #undef tick_dep_mask_name diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index b0e3c9205946..ba2ac1469d47 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -281,6 +281,11 @@ static bool check_tick_dependency(atomic_t *dep) return true; } + if (val & TICK_DEP_MASK_RCU_EXP) { + trace_tick_stop(0, TICK_DEP_MASK_RCU_EXP); + return true; + } + return false; } From patchwork Wed Mar 15 19:43:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 70417 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp91857wrt; Wed, 15 Mar 2023 12:57:30 -0700 (PDT) X-Google-Smtp-Source: AK7set9MB50aKT+D4gzytQ+r76JAjkfMqu1jlIeS4ytYYN/4uMM72aidAB8lSGIYRlG+ZgilIdSs X-Received: by 2002:a17:902:fa4f:b0:19c:be03:d1a3 with SMTP id lb15-20020a170902fa4f00b0019cbe03d1a3mr569962plb.40.1678910250718; Wed, 15 Mar 2023 12:57:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678910250; cv=none; d=google.com; s=arc-20160816; b=VSZX8ysrW++K+EHLWfwyoFsaUWoaq/SsR6wHym/AxQsgt3S1bkSZln5pN5FhYHXlDw 6+LQHXYahwfjxc/D7D1uG2kp3G3tocabQh75xMNJBYK0/Q8VfD12ZaDBfBJeNGUk2dEL 9/ScsL8vb+kyFMKGKDQecf0VT1QoIkxJe0mu+EqO9Y38wewpxj9NVRcowHDT5tNAAhM4 WHZJnnpzlTiMA+ZdvaKl5Y5J0IwM73p9h+4au03sQpCkLwz33jxgx1+/HGt9iDOfHSMK kd+J1P6H8iDLy3KkRMVJuQQT/JR/4G9litN0gjOo9rtMfYJy5InqkuGnIa3lPvQqIlFC rARw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=HkEQfoeE/r4WLsh8kNGxAmPOHd334+QHCg51Sh3MFAg=; b=tVHMeN2KH0BBkKGSe46Le7L5n9vpfd58rzQLG/ryQSWAK7iXRLJVIbjn1o5w1bKLsq 2od+M7YKjMG+kR3un4kJHKHBRbOy64x2rXHuxEMR1bzQBqDG+SaxmAwGu11XIeHKvBeP iaaLFdifrMJfLe9lPniqZboyYYBO4f9j95o9fSP3Z/QAILPS3FsPzxLVwvuY+HDet9p6 W6i7EC3RzCg8hkjsjNwL59Am6x0q/CyD9STYx/uIwCVqTcAkMUSHQT/RCftNhco+1NO6 tqY5utak8KGZ8q8pXC4Taa84ZVCHrzDALyzJk9YXd4XnQH11CB9aWX6O1nRUq5s5NQd7 O/qw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=NsbVHBk5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p2-20020a170902780200b0019e8928b3besi5920408pll.248.2023.03.15.12.57.17; Wed, 15 Mar 2023 12:57:30 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=NsbVHBk5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232345AbjCOTpK (ORCPT + 99 others); Wed, 15 Mar 2023 15:45:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47774 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232186AbjCOToj (ORCPT ); Wed, 15 Mar 2023 15:44:39 -0400 Received: from mail-qt1-x82d.google.com (mail-qt1-x82d.google.com [IPv6:2607:f8b0:4864:20::82d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A90611656 for ; Wed, 15 Mar 2023 12:44:32 -0700 (PDT) Received: by mail-qt1-x82d.google.com with SMTP id h19so1348214qtn.1 for ; Wed, 15 Mar 2023 12:44:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1678909471; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HkEQfoeE/r4WLsh8kNGxAmPOHd334+QHCg51Sh3MFAg=; b=NsbVHBk5ucAJrVO0iXKlJhOtGw18iXzyZV1oDL1C6C9ke9qWst/j13WDaObekR3JVD Gy9u/S/7VpT3P5tCkuY6pVAwD/NlOuk1Bc2MzuzCcDgDBgAzUFWtF30KYky7wh+DeyGP 13pEof4SKJJFM4rs5V9fWt8k7WF0aESemcz5Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678909471; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HkEQfoeE/r4WLsh8kNGxAmPOHd334+QHCg51Sh3MFAg=; b=KGg40vZH6icYwlvgRJonZHR4m1mQulrONYRgRGofTCHuJBwB2kdcO0QOzMqSRTUodi SSJVicD5RibAeUZpH0/5ELqoY+DY0fkKgjRZp2ZBydE1qpqXAsqJ3lt1IBV186rdqsKM otbGxqznqzBgFWuyEIAPGrB5wFuBBMNkQOueDMCDGpcTGSuz5ho55pD/GRWSxMv+5bhy p73AqmImZSWFfDAHQ3EIYQNbOu1s6NBmtarBens31mNYJKlsYsTsf/rMFuUz/6We/fe1 WnGa74W+8nXknHTKQztdbc6MHK2rXfX1UWgtJyW/3Y8WmPzcLqx83+vEH/CDTnstV/kr +QQg== X-Gm-Message-State: AO0yUKWdDJ6hepmUu0O63i+ue3tEHkS0YchDHUCAndy8tBBpKnD1WL4+ BQ6TDKgVXmazRWx9rtDrg49TVsfGJ9NOaIwbZrY= X-Received: by 2002:a05:622a:180d:b0:3bf:e2f8:c6a4 with SMTP id t13-20020a05622a180d00b003bfe2f8c6a4mr1984193qtc.61.1678909471190; Wed, 15 Mar 2023 12:44:31 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id c15-20020ac8660f000000b003b86b088755sm4346666qtp.15.2023.03.15.12.44.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 12:44:30 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, Thomas Gleixner , Peter Zijlstra , Andy Lutomirski , Frederic Weisbecker , Ingo Molnar Cc: "Paul E . McKenney" , Joel Fernandes Subject: [PATCH 3/9] entry/rcu: Check TIF_RESCHED _after_ delayed RCU wake-up Date: Wed, 15 Mar 2023 19:43:43 +0000 Message-Id: <20230315194349.10798-3-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog In-Reply-To: <20230315194349.10798-1-joel@joelfernandes.org> References: <20230315194349.10798-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760464995167647711?= X-GMAIL-MSGID: =?utf-8?q?1760464995167647711?= From: Frederic Weisbecker RCU sometimes needs to perform a delayed wake up for specific kthreads handling offloaded callbacks (RCU_NOCB). These wakeups are performed by timers and upon entry to idle (also to guest and to user on nohz_full). However the delayed wake-up on kernel exit is actually performed after the thread flags are fetched towards the fast path check for work to do on exit to user. As a result, and if there is no other pending work to do upon that kernel exit, the current task will resume to userspace with TIF_RESCHED set and the pending wake up ignored. Fix this with fetching the thread flags _after_ the delayed RCU-nocb kthread wake-up. Fixes: 47b8ff194c1f ("entry: Explicitly flush pending rcuog wakeup before last rescheduling point") Signed-off-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/entry/common.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/entry/common.c b/kernel/entry/common.c index 846add8394c4..a134e26b58c6 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -192,13 +192,14 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs *regs, static void exit_to_user_mode_prepare(struct pt_regs *regs) { - unsigned long ti_work = read_thread_flags(); + unsigned long ti_work; lockdep_assert_irqs_disabled(); /* Flush pending rcuog wakeup before the last need_resched() check */ tick_nohz_user_enter_prepare(); + ti_work = read_thread_flags(); if (unlikely(ti_work & EXIT_TO_USER_MODE_WORK)) ti_work = exit_to_user_mode_loop(regs, ti_work); From patchwork Wed Mar 15 19:43:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 70426 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp99491wrt; Wed, 15 Mar 2023 13:13:50 -0700 (PDT) X-Google-Smtp-Source: AK7set/e9VBhWAm4L9ClUfqmec823IVXJ+c+/NMbc5E4Jhddyt+RNrzvGPgIwj8zZYVs/oceVoom X-Received: by 2002:a17:90b:3851:b0:237:91df:9fcd with SMTP id nl17-20020a17090b385100b0023791df9fcdmr794646pjb.48.1678911230114; Wed, 15 Mar 2023 13:13:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678911230; cv=none; d=google.com; s=arc-20160816; b=t+AVFGB3BwneqfOVEe1Z+7Z4WSp8Wsx/Bhxe1GfZK+i27oeovVWToFQwdBaN4gotn1 Qx5Lb+NUBnh8NRpC6tQh+trtl0poBnEqUNihBfm1qFB4kXpYaH5weGy8l2VLBlSeRIDB 1WEG4dLXoI1ocQcZ94zOzw3ARGeYsNt5ix1Jp5NzMoToEplRJQa1EbIyfXNGOa1p8FQ2 8mCQ0dbA4f0//W76uuoIHeBkDLioZyj2Rm/fJRzJmczMxS41tfdgkCBfrnGVSaMxucfm 4ojkHQq7Vr/CXlHBAQ0hhhVl/qaLGyj7oQFWVjGOJnzS5l+DNtGj7auGH0qG8bAOcWw0 DQPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=VzLowNCggdMuHVa7t9ihdZAmx6P+qhTQOwYOWCMVniY=; b=JIstvMuUDVWaIFN4Gy18eGPoqC3FdsOU1lmZglFd3V6I9QQnENC6QbbfH3VirFzBJK qX7CmurQ4khepO7gy1CdK1ExEjaxOkLMAHmNMroPMWSFZP0c+XDDfSjAXrdh8nlDrdJs NpzmfXRu3dhHDcfPdTLJZ5ztuZqv3NVQNv7EdkzRl7IejLBOO4O5oUvBcSThAMhz79KH X70PS1uEziCYz5NlBnksf+dD4lVOv1aX7SSh2PH3HUtL6lZ24B9bXNOZDZSViFHrh3wX JcuSXcuBQsxmwuHfuqcB/yNV6GHqub2k+sEzJr9Ta7ZOkWX1XMEtBbpI7BnZJuOpjCQL 83Mw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=X5E0FVsO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v13-20020a17090a458d00b0023f09ab33a8si1231073pjg.188.2023.03.15.13.13.34; Wed, 15 Mar 2023 13:13:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=X5E0FVsO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232011AbjCOTpP (ORCPT + 99 others); Wed, 15 Mar 2023 15:45:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47954 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232543AbjCOTom (ORCPT ); Wed, 15 Mar 2023 15:44:42 -0400 Received: from mail-qt1-x833.google.com (mail-qt1-x833.google.com [IPv6:2607:f8b0:4864:20::833]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D513F12CC9 for ; Wed, 15 Mar 2023 12:44:33 -0700 (PDT) Received: by mail-qt1-x833.google.com with SMTP id x1so1751749qtr.7 for ; Wed, 15 Mar 2023 12:44:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1678909473; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VzLowNCggdMuHVa7t9ihdZAmx6P+qhTQOwYOWCMVniY=; b=X5E0FVsOr52URNj8IKnNJAMyRNN6iAb0W+Te/IP/Rkl5Hpf1U/QqPijvWm3XlbaicC 3mlCYV4ql1Go4r8wPQCmoR9J9NpghgNsyUoeZ8oRouPfvr2gPKGhgdMmlX/lLdQcMHMI R6nYUPAL6WViNKApXzNGke+SYrLWQYJ/Zfe8Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678909473; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VzLowNCggdMuHVa7t9ihdZAmx6P+qhTQOwYOWCMVniY=; b=MoF+9rFbgSIukMwMLn7DKceq3rtqfdMOTB+7Z/uEVPTeFo7wmYLr1JKP6jHKOTSCfq ONOnHQhs3gIRPrZfPu+pdESz+1G5DejmbuBw50T7M81HNslDALtpF9dSELYXN9EM8eKs iauxFf1qtiYtNXpQg2v9Xh0DpGN6s6JOU8I90yI3QXI8UVY7Yi4nxclWiSRSajXM0yq7 iWWCX4ikcIfztVB+ttTwTOz3mQ7DVIKMtVwWapJ0s7ueIrQv+3NzTowxNtWIXsKeYaka 6KlJ1qiiPD9MyDz7PfXLsnq9enD0xch+iikpyZoOly0yQOkJCRgQYN9cfISQKemmOLbx XN1A== X-Gm-Message-State: AO0yUKVvKLzPMVkBAZhTwGkI8iseB+mux3nfz20ibhbIEVWzI9yrZbzp e8E6QkTJhQk0BajnxGmeFBC38pogqqCZZk+n9pQ= X-Received: by 2002:ac8:5886:0:b0:3bd:16cf:2f33 with SMTP id t6-20020ac85886000000b003bd16cf2f33mr1586127qta.63.1678909472162; Wed, 15 Mar 2023 12:44:32 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id c15-20020ac8660f000000b003b86b088755sm4346666qtp.15.2023.03.15.12.44.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 12:44:31 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes Cc: Zqiang , rcu@vger.kernel.org Subject: [PATCH 4/9] rcu: Register rcu-lazy shrinker only for CONFIG_RCU_LAZY=y kernels Date: Wed, 15 Mar 2023 19:43:44 +0000 Message-Id: <20230315194349.10798-4-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog In-Reply-To: <20230315194349.10798-1-joel@joelfernandes.org> References: <20230315194349.10798-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760466022114436426?= X-GMAIL-MSGID: =?utf-8?q?1760466022114436426?= From: Zqiang The lazy_rcu_shrink_count() shrinker function is registered even in kernels built with CONFIG_RCU_LAZY=n, in which case this function uselessly consumes cycles learning that no CPU has any lazy callbacks queued. This commit therefore registers this shrinker function only in the kernels built with CONFIG_RCU_LAZY=y, where it might actually do something useful. Signed-off-by: Zqiang Reviewed-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) Reviewed-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 9e1c8caec5ce..f2280616f9d5 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1312,6 +1312,7 @@ int rcu_nocb_cpu_offload(int cpu) } EXPORT_SYMBOL_GPL(rcu_nocb_cpu_offload); +#ifdef CONFIG_RCU_LAZY static unsigned long lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) { @@ -1360,6 +1361,7 @@ static struct shrinker lazy_rcu_shrinker = { .batch = 0, .seeks = DEFAULT_SEEKS, }; +#endif // #ifdef CONFIG_RCU_LAZY void __init rcu_init_nohz(void) { @@ -1391,8 +1393,10 @@ void __init rcu_init_nohz(void) if (!rcu_state.nocb_is_setup) return; +#ifdef CONFIG_RCU_LAZY if (register_shrinker(&lazy_rcu_shrinker, "rcu-lazy")) pr_err("Failed to register lazy_rcu shrinker!\n"); +#endif // #ifdef CONFIG_RCU_LAZY if (!cpumask_subset(rcu_nocb_mask, cpu_possible_mask)) { pr_info("\tNote: kernel parameter 'rcu_nocbs=', 'nohz_full', or 'isolcpus=' contains nonexistent CPUs.\n"); From patchwork Wed Mar 15 19:43:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 70437 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp118201wrt; Wed, 15 Mar 2023 13:56:30 -0700 (PDT) X-Google-Smtp-Source: AK7set9Dtg00o7+xbtATPXEsPH8/A6rLV7HuogpR3+lMGpRcMKhI+RFjgvZndiQdBz3EYVEpfqpo X-Received: by 2002:a62:1c46:0:b0:625:d697:c386 with SMTP id c67-20020a621c46000000b00625d697c386mr691581pfc.22.1678913789874; Wed, 15 Mar 2023 13:56:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678913789; cv=none; d=google.com; s=arc-20160816; b=wj6PL/HWYHgEzeH4z43ve3bRJOAJJ871rn4bsJUvZ/VTMhJ4uwgOb0cueNdJ7nI6pN z3Zpd75EdX1ZOfsTKFTYAp7++zjXC+OXbreespCG3g82Dj4vU6h6lWqaqPm5GFHiDEgt yT5h+d3clhLVu5yvuTiuWXPuejFPKRsySRjwMulE1VfBJRoLAgiCC5uMqMJBlMCDdIYr RNeEUFYAIj0lA6DSwGoPsrDGJrn987sCaTE3UjzjjCYi3nJPPS8CK2HfC0YHDLfsMmn5 kgDyiJz5NUAWF/rvx7b9ec4Dkz1nt3EOSf15ghecj+0T0N3502585E7WdGqKRc09l+dL da9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ThU1XJtO1aLExHCNuOZtJgSY8HbvsH7BE8/boByrYno=; b=jV1s6FVhDEsbtv5GRROJC5rnodp09vjAEWJetvXfvmGl+p+eN5hVYpkHRRYY8Lmgik QIUgaxlhu9H6ahrBA0ZnyJDfGoltS9uGlBiNaasikaO9BEkfCB2x7onipw7r/BYunjDz CO2h7toGr/+Y8wdLIEP0gLeoMj/Rxp2MJaHVQUdY2oNJ0IOC7+nxmPc+roZHcmxVsw61 hsGcj7BgFPspR4wtjUK5OCn0DiOZnL3iXqPgNPrUq7vv/oNVN3xu1AkXjh96+m8HOaDO LzegF0dFVSLVmqZCX1jwwSiSodstRvemVzMTgy+LuSQijLegTN8E/Ltr51MyzbstgSAC IFPQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=ag5X7WW6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h126-20020a636c84000000b0050bcc13077asi5386798pgc.29.2023.03.15.13.56.14; Wed, 15 Mar 2023 13:56:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=ag5X7WW6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231701AbjCOTpT (ORCPT + 99 others); Wed, 15 Mar 2023 15:45:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47712 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232654AbjCOToq (ORCPT ); Wed, 15 Mar 2023 15:44:46 -0400 Received: from mail-qt1-x82b.google.com (mail-qt1-x82b.google.com [IPv6:2607:f8b0:4864:20::82b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B389D199FE for ; Wed, 15 Mar 2023 12:44:34 -0700 (PDT) Received: by mail-qt1-x82b.google.com with SMTP id r16so17474515qtx.9 for ; Wed, 15 Mar 2023 12:44:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1678909473; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ThU1XJtO1aLExHCNuOZtJgSY8HbvsH7BE8/boByrYno=; b=ag5X7WW6rfx5+7t1/eBsvLbRkB0y7Vb/fHHyhSxrRthTHHyDI7UjKzwlHHipX9cFtF Kka1hk8oKQDU69luA4VDWIBakHNnDiGrOW8TUl6Tbt1TJYOVUHXpB2E2TdW5uov5uc98 nrZnUhAVT3BLC9hxVe2eeMxuyFGzvMETCnPyU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678909473; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ThU1XJtO1aLExHCNuOZtJgSY8HbvsH7BE8/boByrYno=; b=Fhj64vxTzuqChlK/ZrFgvhjtaqqTVSs8WPcvFkwztxAQ/0BW/QPYOJ6mpabmJuExkB jbQNvuOiM6LxGvhj+8NiXC1+7dRfVcNn4w8bvlXSwo1Q9CbVq8EBqFKQA4pD1VbrldJF gPNnrMDEP8YCkNW6EyjsImHDMgrAgDpo6KCoHmZhBno8h1TKS7Esy+O6xBUH2hjYT5Vs aLUbgUe9EWwEKYSxLLpCCIC8kenus+5fNckJYf1Qvue8GwRrRoAOAjDIPo5zYP2Il5EV QjbpXqSuOHEGuOTAK8Q439XA37tnAVxk1GTc3IwsMY2bu9JAGVtrtPyAy3tZU1/VeOov CdNQ== X-Gm-Message-State: AO0yUKWZdzk2HagI5bEy8a+2AF2mwGdrN1/5hBkvXhHcobLxSUaWWhvP oZcN2OJBK9ygB3t8AUKq7KQE/1AwVqXdgoPsjzQ= X-Received: by 2002:a05:622a:1902:b0:3b6:93fe:443c with SMTP id w2-20020a05622a190200b003b693fe443cmr1567792qtc.32.1678909473597; Wed, 15 Mar 2023 12:44:33 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id c15-20020ac8660f000000b003b86b088755sm4346666qtp.15.2023.03.15.12.44.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 12:44:33 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes Cc: Zqiang , rcu@vger.kernel.org Subject: [PATCH 5/9] rcu: Remove never-set needwake assignment from rcu_report_qs_rdp() Date: Wed, 15 Mar 2023 19:43:45 +0000 Message-Id: <20230315194349.10798-5-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog In-Reply-To: <20230315194349.10798-1-joel@joelfernandes.org> References: <20230315194349.10798-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760468706206244555?= X-GMAIL-MSGID: =?utf-8?q?1760468706206244555?= From: Zqiang The rcu_accelerate_cbs() function is invoked by rcu_report_qs_rdp() only if there is a grace period in progress that is still blocked by at least one CPU on this rcu_node structure. This means that rcu_accelerate_cbs() should never return the value true, and thus that this function should never set the needwake variable and in turn never invoke rcu_gp_kthread_wake(). This commit therefore removes the needwake variable and the invocation of rcu_gp_kthread_wake() in favor of a WARN_ON_ONCE() on the call to rcu_accelerate_cbs(). The purpose of this new WARN_ON_ONCE() is to detect situations where the system's opinion differs from ours. Signed-off-by: Zqiang Reviewed-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 8e880c09ab59..e80e8f128c57 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -1955,7 +1955,6 @@ rcu_report_qs_rdp(struct rcu_data *rdp) { unsigned long flags; unsigned long mask; - bool needwake = false; bool needacc = false; struct rcu_node *rnp; @@ -1987,7 +1986,12 @@ rcu_report_qs_rdp(struct rcu_data *rdp) * NOCB kthreads have their own way to deal with that... */ if (!rcu_rdp_is_offloaded(rdp)) { - needwake = rcu_accelerate_cbs(rnp, rdp); + /* + * The current GP has not yet ended, so it + * should not be possible for rcu_accelerate_cbs() + * to return true. So complain, but don't awaken. + */ + WARN_ON_ONCE(rcu_accelerate_cbs(rnp, rdp)); } else if (!rcu_segcblist_completely_offloaded(&rdp->cblist)) { /* * ...but NOCB kthreads may miss or delay callbacks acceleration @@ -1999,8 +2003,6 @@ rcu_report_qs_rdp(struct rcu_data *rdp) rcu_disable_urgency_upon_qs(rdp); rcu_report_qs_rnp(mask, rnp, rnp->gp_seq, flags); /* ^^^ Released rnp->lock */ - if (needwake) - rcu_gp_kthread_wake(); if (needacc) { rcu_nocb_lock_irqsave(rdp, flags); From patchwork Wed Mar 15 19:43:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 70416 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp91543wrt; Wed, 15 Mar 2023 12:56:36 -0700 (PDT) X-Google-Smtp-Source: AK7set89yzjrQws+Tx61f+ynQfnYJOpIwr3k5nM/cz4ovgkXoLBXPkPnizlqj+S1v5IEKuPCFzVo X-Received: by 2002:a17:902:d505:b0:19f:31cc:47fc with SMTP id b5-20020a170902d50500b0019f31cc47fcmr745840plg.39.1678910196654; Wed, 15 Mar 2023 12:56:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678910196; cv=none; d=google.com; s=arc-20160816; b=WUbdrdgvHmIcitUI1t2Hu5mauTI8y/egLyKCMKPKSY2+qgDbmKZvNlPcGjIwjALjMo GCtEYxu2n9D0Y0iaAxK9W/o0jLt2OHMEO4fJF7P05XCaEKoJOiDmFKuBrIbLuB9p+dfl 9Og2/e5ifBnIJElAyYi8y6kZxkYW9249ZWm7pG22BoXR4vyiMTSjvmzsnBRFpIEWR2IA v3/ufTh36K+fVAEx2+M3Qu6UGZivkSduNkJzLuu4AnghA5ogfuTNYT731+w/zHrJboMT wZ3+6AwTGDDUtOYvvBx7qKhxdzJkmzURzWu4CvLifavFilVAybNgTc4O+Tp+S9RYlQF9 8zTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3K8lekfNTHeJ4C0iv99fjGghn93IIX905cIt06EpTfQ=; b=XqkoEW73oL8GSJ/eJXzKiiu8rErG/E4VaTfRpRoEiCqLmgGeF9f2me5Vwm/tPA4GVE gZqbx6Kf2J52UHh81T6S086Fge+a1IpVLZsGrhzt+BxZ4tV2Gox7InKrUnleYomcJUci 2N23Mnvlc83QvIMdWDIKHOzd056hjHMMHoSCjfRFNZRc2O5XUwRvGcZV/WQo0ZkoDnDk sPArcRXy8fdue62VFBFMK9CffDAtZSjK1ANq5i2/7FxvCdwW0zPTDvfz+woApgNo20YJ 9ZdBUJqCBEPovz12hurSjxqEyWqrkSoEpZRFDQqFtt+ba5FCDJPRjMSpOxPzbmy4Zxab nnZQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=cKuWC+aH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d4-20020a17090aa68400b0023d0131c21asi2342590pjq.66.2023.03.15.12.56.24; Wed, 15 Mar 2023 12:56:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=cKuWC+aH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231631AbjCOTpX (ORCPT + 99 others); Wed, 15 Mar 2023 15:45:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232670AbjCOTor (ORCPT ); Wed, 15 Mar 2023 15:44:47 -0400 Received: from mail-qt1-x829.google.com (mail-qt1-x829.google.com [IPv6:2607:f8b0:4864:20::829]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC750311D4 for ; Wed, 15 Mar 2023 12:44:35 -0700 (PDT) Received: by mail-qt1-x829.google.com with SMTP id c18so17507935qte.5 for ; Wed, 15 Mar 2023 12:44:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1678909475; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3K8lekfNTHeJ4C0iv99fjGghn93IIX905cIt06EpTfQ=; b=cKuWC+aHDyxATa6HN+YqE2E5Okxl3eR7i8zhoQiYjbzAOf2vN+cKz3FWTds5n8vFyZ 4FHsCNQavFITJZv8en/rMnQhzx8Bu1Ufn4X/caDxoFZldJHM/1+37+83SF2L6Q6HM4me bh0PyPNf3ZJ5AH49Aq7N5gzNJEuxy5pZTENp0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678909475; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3K8lekfNTHeJ4C0iv99fjGghn93IIX905cIt06EpTfQ=; b=PuoMuObrcNYjjQGdIN/5wTDIDzp22wL6bGUK4g4Zb3/GW9Q/KcyaNOe6XmxpO7cyEN iZ+Ttoxjr6x/aqrvnz0zVm5Sm04sXWnl3RD82oof/lcDQKNJ6FoQ50JGG18vRHYf+SZp IxerqY6Gp9zhgJpG8sH7neSEzomakket3xDx5MQbeWU43ukGLBY9bnUMSuTCJGuPGnpj X9GvMsRe4fVXGrpY/vhwx0Rw6lHKtbVJipGiadaOgO4YJBi5xhgzpi6oy9Yd7obat166 9NQcyK4AuMcjxghlenpWBAgdcutDQqicHqaVV6rSXtDLEjaH2mgQeS7GT6qCYCVjlGWp z7KQ== X-Gm-Message-State: AO0yUKU+v0jk9NKS/s/hwp4PVhK2OVSUXLYa0nJtl7iJyJ6UfUM5CIVf txtq5z0ckSKiB6m/xHrXZBax1zeHmB/e2iFqXlM= X-Received: by 2002:a05:622a:30d:b0:3bf:c407:10ca with SMTP id q13-20020a05622a030d00b003bfc40710camr1981909qtw.10.1678909474756; Wed, 15 Mar 2023 12:44:34 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id c15-20020ac8660f000000b003b86b088755sm4346666qtp.15.2023.03.15.12.44.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 12:44:34 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes Cc: Zqiang , rcu@vger.kernel.org Subject: [PATCH 6/9] rcu: Permit start_poll_synchronize_rcu_expedited() to be invoked early Date: Wed, 15 Mar 2023 19:43:46 +0000 Message-Id: <20230315194349.10798-6-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog In-Reply-To: <20230315194349.10798-1-joel@joelfernandes.org> References: <20230315194349.10798-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760464938431383700?= X-GMAIL-MSGID: =?utf-8?q?1760464938431383700?= From: Zqiang According to the commit log of the patch that added it to the kernel, start_poll_synchronize_rcu_expedited() can be invoked very early, as in long before rcu_init() has been invoked. But before rcu_init(), the rcu_data structure's ->mynode field has not yet been initialized. This means that the start_poll_synchronize_rcu_expedited() function's attempt to set the CPU's leaf rcu_node structure's ->exp_seq_poll_rq field will result in a segmentation fault. This commit therefore causes start_poll_synchronize_rcu_expedited() to set ->exp_seq_poll_rq only after rcu_init() has initialized all CPUs' rcu_data structures' ->mynode fields. It also removes the check from the rcu_init() function so that start_poll_synchronize_rcu_expedited( is unconditionally invoked. Yes, this might result in an unnecessary boot-time grace period, but this is down in the noise. Signed-off-by: Zqiang Reviewed-by: Frederic Weisbecker Reviewed-by: Joel Fernandes (Google) Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 5 ++--- kernel/rcu/tree_exp.h | 5 +++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index e80e8f128c57..90d54571126a 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -4942,9 +4942,8 @@ void __init rcu_init(void) else qovld_calc = qovld; - // Kick-start any polled grace periods that started early. - if (!(per_cpu_ptr(&rcu_data, cpu)->mynode->exp_seq_poll_rq & 0x1)) - (void)start_poll_synchronize_rcu_expedited(); + // Kick-start in case any polled grace periods started early. + (void)start_poll_synchronize_rcu_expedited(); rcu_test_sync_prims(); } diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 7cc4856da081..5343f32e7d67 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -1066,9 +1066,10 @@ unsigned long start_poll_synchronize_rcu_expedited(void) if (rcu_init_invoked()) raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags); if (!poll_state_synchronize_rcu(s)) { - rnp->exp_seq_poll_rq = s; - if (rcu_init_invoked()) + if (rcu_init_invoked()) { + rnp->exp_seq_poll_rq = s; queue_work(rcu_gp_wq, &rnp->exp_poll_wq); + } } if (rcu_init_invoked()) raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags); From patchwork Wed Mar 15 19:43:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 70425 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp99394wrt; Wed, 15 Mar 2023 13:13:38 -0700 (PDT) X-Google-Smtp-Source: AK7set/6mHbVahvDWV6Bm6GZZ2e659ubUhssFeNXb/ew5+dyJBg/rdj/8m2V+1ExfSD6h1stHwPr X-Received: by 2002:a17:902:e311:b0:1a0:4b23:84a with SMTP id q17-20020a170902e31100b001a04b23084amr607901plc.46.1678911218169; Wed, 15 Mar 2023 13:13:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678911218; cv=none; d=google.com; s=arc-20160816; b=jny+OoJb93D5Jo7akwYR38Le7zRTAIlTJmKN8XJUEMcxjVnnl56TAINGU1CJICb58E mchaFkyikrRUt1/PkMzB3en2b191UcGPoRcjeY32fQucHHORw5DglvFfMUoqEktB5RCQ Mj/wGBRrqtvDiQ5Y1CyQN/DaY4P8zVY5tmGPHegRewOBuym1BEqGoi6b38I8ga0Ag1V9 CZMnf6dVxUEbfIUxp7+QiYd7t92aO9cHIdpDlWfIOgFgnabF9wEfWE4tKyjUZqB78CJy 1vkT2XkNfcS+QR31qKkzAoQfAP41NgKuMGixdh1qgW5qpiC/DjDEIVllKsqOb0uxIiN/ TGkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=208GxfDZm7oSyOpPWnvVrrbR0LSr9U2B1RRikPcTIHU=; b=x6abIDg00o6WV3aO61uso0A2wJlmEZMjTCPFlffQDcDbMONRY634GRlD6DU2FVeIVs PyN9PYn6xOpPGe/p6mVl57WjGWdHxA30J5LikZvmNnIQA9HgBBdhniPzthdD3Mo/+3q/ cbwVcNNx4hxxhO3ZWgI0zoHN/APCbHdmXdOo4VOYx02rMU5/+mks71hwdRSVv9ryQMh7 K4asXlv0qze1fR2mufKy8MTyx1iGOvHVppjpbnFda8NgE3hir//H6jRR8lJn+TYrUgNc UxKykXm5dvLR0emHMXLOAk3hRcwpUL9177AhsSV4UNj66unegyQXuT/I28SAFvVIE5ox pctw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=SwfZcZz2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ji7-20020a170903324700b0019d211fced4si6078923plb.187.2023.03.15.13.13.23; Wed, 15 Mar 2023 13:13:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=SwfZcZz2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232532AbjCOTpd (ORCPT + 99 others); Wed, 15 Mar 2023 15:45:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48128 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232713AbjCOTor (ORCPT ); Wed, 15 Mar 2023 15:44:47 -0400 Received: from mail-qt1-x833.google.com (mail-qt1-x833.google.com [IPv6:2607:f8b0:4864:20::833]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2DBF23B0F5 for ; Wed, 15 Mar 2023 12:44:37 -0700 (PDT) Received: by mail-qt1-x833.google.com with SMTP id t9so1349914qtx.8 for ; Wed, 15 Mar 2023 12:44:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1678909476; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=208GxfDZm7oSyOpPWnvVrrbR0LSr9U2B1RRikPcTIHU=; b=SwfZcZz22GQ86AfZbWhrkMmC0APeen3E/TYOlJxwfPQj5qa3k998D+cUVTsr+OrgyK WvbfHiF17cuaKhsWGLmCE3S/m6YnKZ9M/k74AU81Siisp2uBs9tI/TxBZD5TyjWVi8wS x5b4+KHAhLFQQO1DjIaf7rO6WNKuOgp35UykA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678909476; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=208GxfDZm7oSyOpPWnvVrrbR0LSr9U2B1RRikPcTIHU=; b=zRGPSlsXlFXHe8yGkQHRG1VmF5qy1VIMX/TpTLUTfnt/AERVYcDk8QBJxMBb/mndKL oKNvAcDozWma5k3VGFsR7L24FqWUBp0LmneiLlusb1guIUcxzIXa/ejBLbcQ1RP65848 kxtuXmi30lGCGLCShZXJpg7GroDyD2NTcpTzEov9tLSf9icXuDxnVrFTyjifER9jt3j+ uWx2EoXV4ozbnHdUmwYw5qAxmrajplu/j1uEzz9SrJsZpCp9MS86V+PZ0dZ4rYEiTRVd myRCBEUT3I726SJEUBe71a2aKt1fTHiiv+QN0yihF1sAD47xnS6Qmj141UEXKq76AhzV uHHQ== X-Gm-Message-State: AO0yUKUV7tuemThZ3O973Tah0Mu4vGLk6Pk+EsS/PqAhfLtEDbalZkPj idHMZ7eOjAabVV5aXMhQSyqVO3qK+D1g5KvBRVA= X-Received: by 2002:ac8:5c13:0:b0:3d4:8ce9:cef7 with SMTP id i19-20020ac85c13000000b003d48ce9cef7mr2038670qti.8.1678909475786; Wed, 15 Mar 2023 12:44:35 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id c15-20020ac8660f000000b003b86b088755sm4346666qtp.15.2023.03.15.12.44.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 12:44:35 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes Cc: Mark Brown , rcu@vger.kernel.org Subject: [PATCH 7/9] rcu-tasks: Report stalls during synchronize_srcu() in rcu_tasks_postscan() Date: Wed, 15 Mar 2023 19:43:47 +0000 Message-Id: <20230315194349.10798-7-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog In-Reply-To: <20230315194349.10798-1-joel@joelfernandes.org> References: <20230315194349.10798-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760466009337414461?= X-GMAIL-MSGID: =?utf-8?q?1760466009337414461?= From: Neeraj Upadhyay The call to synchronize_srcu() from rcu_tasks_postscan() can be stalled by a task getting stuck in do_exit() between that function's calls to exit_tasks_rcu_start() and exit_tasks_rcu_finish(). To ease diagnosis of this situation, print a stall warning message every rcu_task_stall_info period when rcu_tasks_postscan() is stalled. [ paulmck: Adjust to handle CONFIG_SMP=n. ] Reported-by: Mark Brown Link: https://lore.kernel.org/rcu/20230111212736.GA1062057@paulmck-ThinkPad-P17-Gen-1/ Signed-off-by: Neeraj Upadhyay Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tasks.h | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index bfb5e1549f2b..baf7ec178155 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -139,6 +139,12 @@ static struct rcu_tasks rt_name = \ /* Track exiting tasks in order to allow them to be waited for. */ DEFINE_STATIC_SRCU(tasks_rcu_exit_srcu); +#ifdef CONFIG_TASKS_RCU +/* Report delay in synchronize_srcu() completion in rcu_tasks_postscan(). */ +static void tasks_rcu_exit_srcu_stall(struct timer_list *unused); +static DEFINE_TIMER(tasks_rcu_exit_srcu_stall_timer, tasks_rcu_exit_srcu_stall); +#endif + /* Avoid IPIing CPUs early in the grace period. */ #define RCU_TASK_IPI_DELAY (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB) ? HZ / 2 : 0) static int rcu_task_ipi_delay __read_mostly = RCU_TASK_IPI_DELAY; @@ -830,6 +836,13 @@ static void rcu_tasks_pertask(struct task_struct *t, struct list_head *hop) /* Processing between scanning taskslist and draining the holdout list. */ static void rcu_tasks_postscan(struct list_head *hop) { + int rtsi = READ_ONCE(rcu_task_stall_info); + + if (!IS_ENABLED(CONFIG_TINY_RCU)) { + tasks_rcu_exit_srcu_stall_timer.expires = jiffies + rtsi; + add_timer(&tasks_rcu_exit_srcu_stall_timer); + } + /* * Exiting tasks may escape the tasklist scan. Those are vulnerable * until their final schedule() with TASK_DEAD state. To cope with @@ -848,6 +861,9 @@ static void rcu_tasks_postscan(struct list_head *hop) * call to synchronize_rcu(). */ synchronize_srcu(&tasks_rcu_exit_srcu); + + if (!IS_ENABLED(CONFIG_TINY_RCU)) + del_timer_sync(&tasks_rcu_exit_srcu_stall_timer); } /* See if tasks are still holding out, complain if so. */ @@ -923,6 +939,21 @@ static void rcu_tasks_postgp(struct rcu_tasks *rtp) void call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func); DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_wait_gp, call_rcu_tasks, "RCU Tasks"); +static void tasks_rcu_exit_srcu_stall(struct timer_list *unused) +{ +#ifndef CONFIG_TINY_RCU + int rtsi; + + rtsi = READ_ONCE(rcu_task_stall_info); + pr_info("%s: %s grace period number %lu (since boot) gp_state: %s is %lu jiffies old.\n", + __func__, rcu_tasks.kname, rcu_tasks.tasks_gp_seq, + tasks_gp_state_getname(&rcu_tasks), jiffies - rcu_tasks.gp_jiffies); + pr_info("Please check any exiting tasks stuck between calls to exit_tasks_rcu_start() and exit_tasks_rcu_finish()\n"); + tasks_rcu_exit_srcu_stall_timer.expires = jiffies + rtsi; + add_timer(&tasks_rcu_exit_srcu_stall_timer); +#endif // #ifndef CONFIG_TINY_RCU +} + /** * call_rcu_tasks() - Queue an RCU for invocation task-based grace period * @rhp: structure to be used for queueing the RCU updates. From patchwork Wed Mar 15 19:43:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 70420 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp93026wrt; Wed, 15 Mar 2023 13:00:26 -0700 (PDT) X-Google-Smtp-Source: AK7set8/bJPlRG/l9fsb1ktkKHXG0n3zBllG9dhisw8EyUyknwq37nud5B3Y01GOoCQVCBrtaK33 X-Received: by 2002:a17:90b:3010:b0:233:feb4:895f with SMTP id hg16-20020a17090b301000b00233feb4895fmr938073pjb.44.1678910426274; Wed, 15 Mar 2023 13:00:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678910426; cv=none; d=google.com; s=arc-20160816; b=NH3A3KOSHDGPtRmtZuZc/xDeCRoXq06LC23c/SgDFIVx7xrKytkjs2/sz10ceKnIgz WWxfvUcGED68snb5dyyCWnfx0Kz4So0qnaImc26p5iZUrNMfKkzxnE3sqsiO+7130Emk P/Aqls1fSE/IouapzvWHrsRRd44jnplgHmUd8YdAp32h3n3OI5XoKva+T4OIWm2fPsGl G7ulHjw0QBO2rJfbnaC/aMj8XfDpiw8AFMxboQyZpJUjZKbpJpqJYc1jyy1bKyo47BJO BdeNdhdaDfUfGVMrWFFS3G1D29lunWNMG1x34Q6I49iEnaS/fJGAghprNuj08asIWWTn Xbdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=kmL3NaGdkJr0Bc+bh875hISga550hRk4RtUTBtwwn0w=; b=vwu/uzcDd242w/p2cY6fUCeHcjQMK5iaJGErQpN959HzFFtwB2HfxAFjCJDc1K4yw3 CQZp6+4+Z6SUgeSdta++0dvPjs5GDfASHKkcXXs6jPMwz/ZZxqFaXOKv00ECxnSy9Kqa xwCksfeZbwtDy6jCdmWF4cG3MpzWqUNqAUbEOEQcR89OenKOwJKy7NjftgxH1XmuEJ7u S7zw1tEx7dLvJB2qPsGACH9GOXPRISVpBJltsU0gJN2+H+MC9DgaxkUrLTkV96xuJCic +PlUbUAGYcvy9Apt6bEoaCO0Cv3nhinjpsSRnzzCingzejaM0y04dA8fZJbn9VjHqWgN mRzw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=bnBx2KLF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id nv1-20020a17090b1b4100b002301345a517si2587336pjb.17.2023.03.15.13.00.13; Wed, 15 Mar 2023 13:00:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=bnBx2KLF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232603AbjCOTpa (ORCPT + 99 others); Wed, 15 Mar 2023 15:45:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47952 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232736AbjCOTor (ORCPT ); Wed, 15 Mar 2023 15:44:47 -0400 Received: from mail-qt1-x830.google.com (mail-qt1-x830.google.com [IPv6:2607:f8b0:4864:20::830]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2267F298EA for ; Wed, 15 Mar 2023 12:44:38 -0700 (PDT) Received: by mail-qt1-x830.google.com with SMTP id l13so17525326qtv.3 for ; Wed, 15 Mar 2023 12:44:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1678909477; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kmL3NaGdkJr0Bc+bh875hISga550hRk4RtUTBtwwn0w=; b=bnBx2KLF4/8ayT1hsKRH/YELY31dEXlopSuwsSIBqqg+vaXM/EzMzHk3VEelU5122J jwQb+81WA0fyC2BsjfFRu6dch564OgwoXcO8sX+yjQIZbugkKoJ2HG34dTE3ocYhrCar 8S/A3URcPENhyHM0kLeY152ccpK6HGiSkJAnw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678909477; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kmL3NaGdkJr0Bc+bh875hISga550hRk4RtUTBtwwn0w=; b=zu8fqmkYNf5WtbJAVAbP03AbeP7wU7lZiux8VU39ybdHF/3M83OrPzaWtvTldMQZoZ hEMPzJJ9D5oRLo70lKeDuHeNb+du9wslgHCDyi9gYxjLubKb1R8OlbAvPTYfW8W2PnAt KYgY7+Tz3Ex6RoyX1P8zQo3p5DUmekAxrpDoLc0gIN/3L+QgAWG7THCQy5pJWMaGLwgy p0WRAEjFRdpMj0kQTha5ZnPTWETxSwBct094099/LvLTsozIqu9QFCORgSOqD+/bEdTJ biPer5tHYzldTyFHWVFI07TeOdmAY28r0A/LmbAyi+cUCKODpt1QbbUGp5Fhq+8QtHY4 vTpg== X-Gm-Message-State: AO0yUKVN/VDzwNjiSijyFJ0o7Aa2dB3Wvp14YlutEsf7IZBzmSb+tv6t zTRDvwKW5pC+LuMfn6bs27dIGs89e/JqToZJaj0= X-Received: by 2002:ac8:5989:0:b0:3bf:c5a7:595f with SMTP id e9-20020ac85989000000b003bfc5a7595fmr1903191qte.21.1678909476912; Wed, 15 Mar 2023 12:44:36 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id c15-20020ac8660f000000b003b86b088755sm4346666qtp.15.2023.03.15.12.44.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 12:44:36 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes , Ingo Molnar , Thomas Gleixner Cc: Zheng Yejian , stable@vger.kernel.org, rcu@vger.kernel.org Subject: [PATCH 8/9] rcu: Avoid stack overflow due to __rcu_irq_enter_check_tick() being kprobe-ed Date: Wed, 15 Mar 2023 19:43:48 +0000 Message-Id: <20230315194349.10798-8-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog In-Reply-To: <20230315194349.10798-1-joel@joelfernandes.org> References: <20230315194349.10798-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760465179330193231?= X-GMAIL-MSGID: =?utf-8?q?1760465179330193231?= From: Zheng Yejian Registering a kprobe on __rcu_irq_enter_check_tick() can cause kernel stack overflow as shown below. This issue can be reproduced by enabling CONFIG_NO_HZ_FULL and booting the kernel with argument "nohz_full=", and then giving the following commands at the shell prompt: # cd /sys/kernel/tracing/ # echo 'p:mp1 __rcu_irq_enter_check_tick' >> kprobe_events # echo 1 > events/kprobes/enable This commit therefore adds __rcu_irq_enter_check_tick() to the kprobes blacklist using NOKPROBE_SYMBOL(). Insufficient stack space to handle exception! ESR: 0x00000000f2000004 -- BRK (AArch64) FAR: 0x0000ffffccf3e510 Task stack: [0xffff80000ad30000..0xffff80000ad38000] IRQ stack: [0xffff800008050000..0xffff800008058000] Overflow stack: [0xffff089c36f9f310..0xffff089c36fa0310] CPU: 5 PID: 190 Comm: bash Not tainted 6.2.0-rc2-00320-g1f5abbd77e2c #19 Hardware name: linux,dummy-virt (DT) pstate: 400003c5 (nZcv DAIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : __rcu_irq_enter_check_tick+0x0/0x1b8 lr : ct_nmi_enter+0x11c/0x138 sp : ffff80000ad30080 x29: ffff80000ad30080 x28: ffff089c82e20000 x27: 0000000000000000 x26: 0000000000000000 x25: ffff089c02a8d100 x24: 0000000000000000 x23: 00000000400003c5 x22: 0000ffffccf3e510 x21: ffff089c36fae148 x20: ffff80000ad30120 x19: ffffa8da8fcce148 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: ffffa8da8e44ea6c x14: ffffa8da8e44e968 x13: ffffa8da8e03136c x12: 1fffe113804d6809 x11: ffff6113804d6809 x10: 0000000000000a60 x9 : dfff800000000000 x8 : ffff089c026b404f x7 : 00009eec7fb297f7 x6 : 0000000000000001 x5 : ffff80000ad30120 x4 : dfff800000000000 x3 : ffffa8da8e3016f4 x2 : 0000000000000003 x1 : 0000000000000000 x0 : 0000000000000000 Kernel panic - not syncing: kernel stack overflow CPU: 5 PID: 190 Comm: bash Not tainted 6.2.0-rc2-00320-g1f5abbd77e2c #19 Hardware name: linux,dummy-virt (DT) Call trace: dump_backtrace+0xf8/0x108 show_stack+0x20/0x30 dump_stack_lvl+0x68/0x84 dump_stack+0x1c/0x38 panic+0x214/0x404 add_taint+0x0/0xf8 panic_bad_stack+0x144/0x160 handle_bad_stack+0x38/0x58 __bad_stack+0x78/0x7c __rcu_irq_enter_check_tick+0x0/0x1b8 arm64_enter_el1_dbg.isra.0+0x14/0x20 el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 arm64_enter_el1_dbg.isra.0+0x14/0x20 el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 arm64_enter_el1_dbg.isra.0+0x14/0x20 el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 [...] el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 arm64_enter_el1_dbg.isra.0+0x14/0x20 el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 arm64_enter_el1_dbg.isra.0+0x14/0x20 el1_dbg+0x2c/0x90 el1h_64_sync_handler+0xcc/0xe8 el1h_64_sync+0x64/0x68 __rcu_irq_enter_check_tick+0x0/0x1b8 el1_interrupt+0x28/0x60 el1h_64_irq_handler+0x18/0x28 el1h_64_irq+0x64/0x68 __ftrace_set_clr_event_nolock+0x98/0x198 __ftrace_set_clr_event+0x58/0x80 system_enable_write+0x144/0x178 vfs_write+0x174/0x738 ksys_write+0xd0/0x188 __arm64_sys_write+0x4c/0x60 invoke_syscall+0x64/0x180 el0_svc_common.constprop.0+0x84/0x160 do_el0_svc+0x48/0xe8 el0_svc+0x34/0xd0 el0t_64_sync_handler+0xb8/0xc0 el0t_64_sync+0x190/0x194 SMP: stopping secondary CPUs Kernel Offset: 0x28da86000000 from 0xffff800008000000 PHYS_OFFSET: 0xfffff76600000000 CPU features: 0x00000,01a00100,0000421b Memory Limit: none Link: https://lore.kernel.org/all/20221119040049.795065-1-zhengyejian1@huawei.com/ Fixes: aaf2bc50df1f ("rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()") Signed-off-by: Zheng Yejian Cc: stable@vger.kernel.org Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 90d54571126a..ee27a03d7576 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -640,6 +640,7 @@ void __rcu_irq_enter_check_tick(void) } raw_spin_unlock_rcu_node(rdp->mynode); } +NOKPROBE_SYMBOL(__rcu_irq_enter_check_tick); #endif /* CONFIG_NO_HZ_FULL */ /* From patchwork Wed Mar 15 19:43:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joel Fernandes X-Patchwork-Id: 70428 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp101705wrt; Wed, 15 Mar 2023 13:18:48 -0700 (PDT) X-Google-Smtp-Source: AK7set++U7uGoQXb2oQ/rJ3PBaUVciQWDGqjxy1KG5ovPZmgTxFY14vBB/QS3piLz+4zWMc1c7gI X-Received: by 2002:a17:903:41c5:b0:19d:297:f30b with SMTP id u5-20020a17090341c500b0019d0297f30bmr904496ple.19.1678911528335; Wed, 15 Mar 2023 13:18:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678911528; cv=none; d=google.com; s=arc-20160816; b=gEbg5V+tSDvrUABRTI1eAVOdM28CGzlKUMx2jzxOSKrptzD5qZ+8TgypLvHARzPZIB +L0PWmp0pKbQFuchYIilpmYrhpJftgq2UaCsbxpPXfxgnBd+/b4q8eZlFX4aVa+Vazh+ hrdR6L53OHzCGSJmWSGOAKnLYEHrurfT/mrfwc0Vnpn1XkE1VHRMeByislLGsPQorbvB WScMTd6aTJkMec19uMJgE9YXMQejXvt4zp/32+LfidVnj2SLS8ZjKgEKQLs/QdmuZ7Ai 6k1yTNs5vpmnTZTsAa7GAs+gGBWKFK69q0mjSdhtEFa96uFxABLnHgS4z7/RI4K1kMQq dQiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4B9TRjdncmXbZW5HDxJoPgY1Lgs+0sxOKkTAY/VF78g=; b=OXAYlUKa8S3pzZK6Z1QdNtcQBHHVjaAWSprme8WRoQRklOHtgJzBRAz+gpmu7zGuIS hXvdv3peOiKrI0BKkFOs1eYOlSeIaG8LK8n4+VbOh/JovWH5NQS/T0dFxgXHkay6sw3M udeDuJX6RzP8LZnWyV+9Vrxt78L0D68jm72rIO3B0Cv/9r70vLy6o/TT8Yn0AO3wj6FX 58nzkS5/vf6mgq3WRqBuqXeBSkUOiiHk7A6+5SSS5pIM1I0gtUdExe9JnQUbkDNvTevI 2Y8L5S18u1wXrBoKqWMY9p+gMEIgZJkk7P2GkXTuLmPo0tC/D8bRirjsofWI5D5P6PQR AMZA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=dqOfXnxH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ld12-20020a170902facc00b0019abd3a98efsi5764246plb.91.2023.03.15.13.18.33; Wed, 15 Mar 2023 13:18:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@joelfernandes.org header.s=google header.b=dqOfXnxH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232503AbjCOTp0 (ORCPT + 99 others); Wed, 15 Mar 2023 15:45:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48178 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232746AbjCOTor (ORCPT ); Wed, 15 Mar 2023 15:44:47 -0400 Received: from mail-qt1-x82c.google.com (mail-qt1-x82c.google.com [IPv6:2607:f8b0:4864:20::82c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9D6646C6A5 for ; Wed, 15 Mar 2023 12:44:38 -0700 (PDT) Received: by mail-qt1-x82c.google.com with SMTP id r5so17505561qtp.4 for ; Wed, 15 Mar 2023 12:44:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelfernandes.org; s=google; t=1678909478; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4B9TRjdncmXbZW5HDxJoPgY1Lgs+0sxOKkTAY/VF78g=; b=dqOfXnxHTLjoCGihltc9NkJBBAVuV0+9aZqiLbjL4hII42e0QglBr/nfEAbRs2bOkJ GJMPOjeYCEpzI+Scjv/06JFOBqgaiyJfIHVZH/2XR0pVlXRzCzTggamJVGUci2HlXgXm tuGi6ULhvXLuP1gdGRXg0ylLcoiFMYJdJDCmk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678909478; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4B9TRjdncmXbZW5HDxJoPgY1Lgs+0sxOKkTAY/VF78g=; b=5xsZRpKGOV9W6f1ARhZ+CzXdSpdAGjaScHNPfvRsVbucddFJDXgSV71mtZ34QLYoOw b7MfmC1nZy2az/luMoGDMvVV6kVcXn79VL5YEjXYD+rFafusX+CuIS5X09iWvRSKYTyG Zzi7UtJq/VVqNQfEdS2Q3xKELthcjSwsEVGwuo3o2T4M98xK27M3MbEO+6LcCF8y5lzT aBZ/syv3CbSMhEHhex9zQ33XDvSQqqK8yK6lh4D5ziq5OIiL80HQaKgSIbQnifGPCFoQ GSraSEvIWMqedx/3457f6gnFbENzKBQzhQOWY/0MJrcVYqIvYSPzosw9JuKMd24QFjDO 93jw== X-Gm-Message-State: AO0yUKWvgt/lJXeb3Sd9oUNJkdhvcIv8o5Llsw3hRdmeIL11bbfr8ZJS P5/JRtS5cEfcU6a5A9VW7rlw5axbVlzkSwr9FKU= X-Received: by 2002:a05:622a:44f:b0:3b8:340b:1aab with SMTP id o15-20020a05622a044f00b003b8340b1aabmr1820256qtx.25.1678909477913; Wed, 15 Mar 2023 12:44:37 -0700 (PDT) Received: from joelboxx.c.googlers.com.com (129.239.188.35.bc.googleusercontent.com. [35.188.239.129]) by smtp.gmail.com with ESMTPSA id c15-20020ac8660f000000b003b86b088755sm4346666qtp.15.2023.03.15.12.44.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Mar 2023 12:44:37 -0700 (PDT) From: "Joel Fernandes (Google)" To: linux-kernel@vger.kernel.org, "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Joel Fernandes Cc: Zqiang , rcu@vger.kernel.org Subject: [PATCH 9/9] rcu: Protect rcu_print_task_exp_stall() ->exp_tasks access Date: Wed, 15 Mar 2023 19:43:49 +0000 Message-Id: <20230315194349.10798-9-joel@joelfernandes.org> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog In-Reply-To: <20230315194349.10798-1-joel@joelfernandes.org> References: <20230315194349.10798-1-joel@joelfernandes.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760466334536058817?= X-GMAIL-MSGID: =?utf-8?q?1760466334536058817?= From: Zqiang For kernels built with CONFIG_PREEMPT_RCU=y, the following scenario can result in a NULL-pointer dereference: CPU1 CPU2 rcu_preempt_deferred_qs_irqrestore rcu_print_task_exp_stall if (special.b.blocked) READ_ONCE(rnp->exp_tasks) != NULL raw_spin_lock_rcu_node np = rcu_next_node_entry(t, rnp) if (&t->rcu_node_entry == rnp->exp_tasks) WRITE_ONCE(rnp->exp_tasks, np) .... raw_spin_unlock_irqrestore_rcu_node raw_spin_lock_irqsave_rcu_node t = list_entry(rnp->exp_tasks->prev, struct task_struct, rcu_node_entry) (if rnp->exp_tasks is NULL, this will dereference a NULL pointer) The problem is that CPU2 accesses the rcu_node structure's->exp_tasks field without holding the rcu_node structure's ->lock and CPU2 did not observe CPU1's change to rcu_node structure's ->exp_tasks in time. Therefore, if CPU1 sets rcu_node structure's->exp_tasks pointer to NULL, then CPU2 might dereference that NULL pointer. This commit therefore holds the rcu_node structure's ->lock while accessing that structure's->exp_tasks field. [ paulmck: Apply Frederic Weisbecker feedback. ] Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney Signed-off-by: Joel Fernandes (Google) --- kernel/rcu/tree_exp.h | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h index 5343f32e7d67..3b7abb58157d 100644 --- a/kernel/rcu/tree_exp.h +++ b/kernel/rcu/tree_exp.h @@ -803,9 +803,11 @@ static int rcu_print_task_exp_stall(struct rcu_node *rnp) int ndetected = 0; struct task_struct *t; - if (!READ_ONCE(rnp->exp_tasks)) - return 0; raw_spin_lock_irqsave_rcu_node(rnp, flags); + if (!rnp->exp_tasks) { + raw_spin_unlock_irqrestore_rcu_node(rnp, flags); + return 0; + } t = list_entry(rnp->exp_tasks->prev, struct task_struct, rcu_node_entry); list_for_each_entry_continue(t, &rnp->blkd_tasks, rcu_node_entry) {