From patchwork Mon Oct 24 11:32:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Greg KH X-Patchwork-Id: 10038 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp622554wru; Mon, 24 Oct 2022 12:45:45 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4Qr5z694md8yBCXoOyPvI+XP7g7jYH5/iKz70wxd+5NUrpeKN6CYDrE6HhqfVlLflVLSaZ X-Received: by 2002:a17:907:3f23:b0:78e:260a:fc33 with SMTP id hq35-20020a1709073f2300b0078e260afc33mr29569603ejc.152.1666640745180; Mon, 24 Oct 2022 12:45:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666640745; cv=none; d=google.com; s=arc-20160816; b=DGbJeb6n8hJ0aqeAtElqvsHVPZEVa+/9ykCDLsp17ejJNmfy+SDJ8RydotD5jbCJcS Im9Tw6wDGS6JNvkCdpD5bDMXlPkgxoaTWbs/rB5RSq8SurhqBRosSF6fCDUHr1kPeP8p brEZEdVAHum0fzKdUAlzjMFHkNZQj09UYz7UKH0BZKE6nSmpE+7o3oM6qDsRBsZzvtIs tROqjT5Sdw3zc4UTc2T0D2xBWJ21zrEBUigQjO6fYUjvdSLrhC3feVZO3VnCMb2n4kZz mnnVd1pEQp+28FfCrFRG/x/wE2r8DFgzbolZIWZwMzF/C4nCn6yMooLv8wvnGpsi7pTs R9Gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=OCmaoVMNGbdsxz6Wx+89uDdo+ENG5djb8QVKYMqU8sk=; b=EWGUz4hIe3B58f4wvY3P7Cd8xTuRV3nPSuv7pJAwtt6MQQv0pMC/qoggZFcKeiwLrk 0MwBYjBufLFaQWPReV3fh0A9SOkZMqfq5hzrsreGWxlgf5qFmQTZH8hMZ8kdXn6hEpSL GdPkYtftDbCCDxFP7KmT427IkSKOFEEsJ+Mu4pXy+zBALW0Ner0DfAyLDRIuu8Lriecm 3y4E2JIbhDHrhTg3jA5jmblr5S2j3Zn72iFXxWVyylASMa8v4qhpt+ytmP+hy3Mq8ff/ T2Pzcr9A5o60CJqE6I8LOEAL8tKl/IYVqfooMF8wu6cVR5Qh0oya1diYMZfkClqNCSDS xpzg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linuxfoundation.org header.s=korg header.b=p3s6Auyy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m10-20020a056402510a00b0045cba2d74c3si677293edd.541.2022.10.24.12.45.20; Mon, 24 Oct 2022 12:45:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linuxfoundation.org header.s=korg header.b=p3s6Auyy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231771AbiJXTm4 (ORCPT + 99 others); Mon, 24 Oct 2022 15:42:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37558 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233521AbiJXTlg (ORCPT ); Mon, 24 Oct 2022 15:41:36 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B2CE9BE3B; Mon, 24 Oct 2022 11:11:27 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id ECBBEB818AA; Mon, 24 Oct 2022 12:52:01 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4CF94C433C1; Mon, 24 Oct 2022 12:52:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1666615920; bh=x3EheLOKxEvpnbaFHL+iXxWZuiofgRfuUGkpAb7ktM8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=p3s6AuyyKttJxUjdmc6ZZclteR05xCL3OkzRR25a0hDu901rhn9K+XxBBz/9nOqiG B98TogeoB/aOR9a2xaIMo15tpRwxT29Pu45672Iv0+CAXnIV9uOQG17lI/PhGIQSDx EjeyEPlxI0o61GmB4ogP+Dda5s7ueCkd7uHHBTnE= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Zqiang , "Paul E. McKenney" , Sasha Levin Subject: [PATCH 5.15 401/530] rcu: Avoid triggering strict-GP irq-work when RCU is idle Date: Mon, 24 Oct 2022 13:32:25 +0200 Message-Id: <20221024113103.228321513@linuxfoundation.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221024113044.976326639@linuxfoundation.org> References: <20221024113044.976326639@linuxfoundation.org> User-Agent: quilt/0.67 MIME-Version: 1.0 X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747599486176370988?= X-GMAIL-MSGID: =?utf-8?q?1747599486176370988?= From: Zqiang [ Upstream commit 621189a1fe93cb2b34d62c5cdb9e258bca044813 ] Kernels built with PREEMPT_RCU=y and RCU_STRICT_GRACE_PERIOD=y trigger irq-work from rcu_read_unlock(), and the resulting irq-work handler invokes rcu_preempt_deferred_qs_handle(). The point of this triggering is to force grace periods to end quickly in order to give tools like KASAN a better chance of detecting RCU usage bugs such as leaking RCU-protected pointers out of an RCU read-side critical section. However, this irq-work triggering is unconditional. This works, but there is no point in doing this irq-work unless the current grace period is waiting on the running CPU or task, which is not the common case. After all, in the common case there are many rcu_read_unlock() calls per CPU per grace period. This commit therefore triggers the irq-work only when the current grace period is waiting on the running CPU or task. This change was tested as follows on a four-CPU system: echo rcu_preempt_deferred_qs_handler > /sys/kernel/debug/tracing/set_ftrace_filter echo 1 > /sys/kernel/debug/tracing/function_profile_enabled insmod rcutorture.ko sleep 20 rmmod rcutorture.ko echo 0 > /sys/kernel/debug/tracing/function_profile_enabled echo > /sys/kernel/debug/tracing/set_ftrace_filter This procedure produces results in this per-CPU set of files: /sys/kernel/debug/tracing/trace_stat/function* Sample output from one of these files is as follows: Function Hit Time Avg s^2 -------- --- ---- --- --- rcu_preempt_deferred_qs_handle 838746 182650.3 us 0.217 us 0.004 us The baseline sum of the "Hit" values (the number of calls to this function) was 3,319,015. With this commit, that sum was 1,140,359, for a 2.9x reduction. The worst-case variance across the CPUs was less than 25%, so this large effect size is statistically significant. The raw data is available in the Link: URL. Link: https://lore.kernel.org/all/20220808022626.12825-1-qiang1.zhang@intel.com/ Signed-off-by: Zqiang Signed-off-by: Paul E. McKenney Signed-off-by: Sasha Levin --- kernel/rcu/tree_plugin.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index ef2dd131e955..f1a73a1f8472 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -638,7 +638,8 @@ static void rcu_read_unlock_special(struct task_struct *t) expboost = (t->rcu_blocked_node && READ_ONCE(t->rcu_blocked_node->exp_tasks)) || (rdp->grpmask & READ_ONCE(rnp->expmask)) || - IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) || + (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) && + ((rdp->grpmask & READ_ONCE(rnp->qsmask)) || t->rcu_blocked_node)) || (IS_ENABLED(CONFIG_RCU_BOOST) && irqs_were_disabled && t->rcu_blocked_node); // Need to defer quiescent state until everything is enabled.