From patchwork Wed Oct 19 22:58:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 5906 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp573976wrs; Wed, 19 Oct 2022 15:59:51 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7rug3KAktDyzEIVcuPUBf08xhWHVkK4IITbvzISqQSILbcbbgLmwJlLq7FEumGZYfqFbTY X-Received: by 2002:a17:906:9bc3:b0:78d:816f:3743 with SMTP id de3-20020a1709069bc300b0078d816f3743mr8718276ejc.380.1666220390906; Wed, 19 Oct 2022 15:59:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666220390; cv=none; d=google.com; s=arc-20160816; b=HTvNM0V4Ym+2NEBsf3BM/Yq7Z7iD4wVd4jXh8OeiVDuIK8JUmD82zMD3rTAYtSmYA1 KpOcqZHe2loND7aOnTs//PrE66Fqwf3fsW58fpyL7kX8UbtUdbOiQ07YsIZKUzhLF0af 7qqdZKaxOjmxSbDVxYBIQ3loXZdeh+wXQWlkVM4qed8iDcvEhsU5OepHzsWcc4ybo8ZE fv7Y1M3FHWUqIpFxmnEq0skLTupZ/JRyDPCnZO7Chg4gJh+FSyJ2C7GdUhed0x3SyVTA QoAdLHsqO/zKlTO1thMluVwTstadhSkwCzCEBRSPR0igP/LrYbeU3oFX/fSWHcw4UpWD DHjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XIdHpfb8QQNygxA9tgb5GJjZxzX6zY4hk75Cthr8VJg=; b=RXL1lbhcsCApWotN+j7yhby9+vfsUZj4jPnHa/AjwHHzJhEV23q3QTHY6nBW2saqPC nUuZhCO8YEMcxElCr+ZHMsQ0qgvxsRAxXdMZib2cgBi7h/eNvapbQWUZmidYweKIZg5d uSDmvoPoqssfk9oxCZ69V1a5uu6kMH+CrOHRk5DnABPFq00IkSrhXFSXjGZsNHWc3Gnd fFi9yaquCIp6iKcqb7ZYFWlX6wulRBnhaJe7S8oz3g3/UPNN4h3dq18Kr37OYG1p+WQJ cxZ3LOYOjDh7s/Vt0ZmmiSXmqd/YagRAYrj9tTcM0wxFgxDU1MMcg3270B8LdtCzXmew Lxxg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=cnfRlwEu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h2-20020a50c382000000b0045729070194si15784633edf.517.2022.10.19.15.59.25; Wed, 19 Oct 2022 15:59:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=cnfRlwEu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230032AbiJSW6w (ORCPT + 99 others); Wed, 19 Oct 2022 18:58:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230440AbiJSW6u (ORCPT ); Wed, 19 Oct 2022 18:58:50 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 02D8A1CBA94; Wed, 19 Oct 2022 15:58:49 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9195D61920; Wed, 19 Oct 2022 22:58:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EA56CC433C1; Wed, 19 Oct 2022 22:58:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666220328; bh=lC2ZhkuunFA2ksKMDckPtMceJ6zVcRmtLkRgyIgtcls=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cnfRlwEud8zypsIPnkwkCkn4HzzhbvLYXCgPFisO8zPb3C9evnYf+kZ3SQCxwnJHu 5R9xpAjW3LITHmqT4vuhUGiA6f3pnLgLI1uyh2NP3h/sgiOfkqgUUb+PO5BAhzptBn dgGpyTdlk4eBWNcDLem7An8S6TbmugUekGrA3uLk1rv/K6gNaoJ416WnYw19pRhroW 7rXKmCMtL/u2/czbqQraIF1CRXk/WWpE2VQqm3SVOyUOt393LxO9arF800iV3GXnmN jEEvmzO5mbnEppwdS85MqBgABaBfTg783962fITifkN43P+5H2drvNoQk4319L54O+ oGF8SQRQ88njA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id AA4CD5C06B4; Wed, 19 Oct 2022 15:58:47 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Frederic Weisbecker , Thomas Gleixner , John Ogness , Petr Mladek Subject: [PATCH v3 rcu 01/11] srcu: Convert ->srcu_lock_count and ->srcu_unlock_count to atomic Date: Wed, 19 Oct 2022 15:58:36 -0700 Message-Id: <20221019225846.2501109-1-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> References: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747158712391829995?= X-GMAIL-MSGID: =?utf-8?q?1747158712391829995?= NMI-safe variants of srcu_read_lock() and srcu_read_unlock() are needed by printk(), which on many architectures entails read-modify-write atomic operations. This commit prepares Tree SRCU for this change by making both ->srcu_lock_count and ->srcu_unlock_count by atomic_long_t. [ paulmck: Apply feedback from John Ogness. ] Link: https://lore.kernel.org/all/20220910221947.171557773@linutronix.de/ Signed-off-by: Paul E. McKenney Reviewed-by: Frederic Weisbecker Cc: Thomas Gleixner Cc: John Ogness Cc: Petr Mladek --- include/linux/srcutree.h | 4 ++-- kernel/rcu/srcutree.c | 24 ++++++++++++------------ 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index e3014319d1ade..0c4eca07d78d5 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -23,8 +23,8 @@ struct srcu_struct; */ struct srcu_data { /* Read-side state. */ - unsigned long srcu_lock_count[2]; /* Locks per CPU. */ - unsigned long srcu_unlock_count[2]; /* Unlocks per CPU. */ + atomic_long_t srcu_lock_count[2]; /* Locks per CPU. */ + atomic_long_t srcu_unlock_count[2]; /* Unlocks per CPU. */ /* Update-side state. */ spinlock_t __private lock ____cacheline_internodealigned_in_smp; diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 1c304fec89c02..25e9458da6a26 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -417,7 +417,7 @@ static unsigned long srcu_readers_lock_idx(struct srcu_struct *ssp, int idx) for_each_possible_cpu(cpu) { struct srcu_data *cpuc = per_cpu_ptr(ssp->sda, cpu); - sum += READ_ONCE(cpuc->srcu_lock_count[idx]); + sum += atomic_long_read(&cpuc->srcu_lock_count[idx]); } return sum; } @@ -434,7 +434,7 @@ static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx) for_each_possible_cpu(cpu) { struct srcu_data *cpuc = per_cpu_ptr(ssp->sda, cpu); - sum += READ_ONCE(cpuc->srcu_unlock_count[idx]); + sum += atomic_long_read(&cpuc->srcu_unlock_count[idx]); } return sum; } @@ -503,10 +503,10 @@ static bool srcu_readers_active(struct srcu_struct *ssp) for_each_possible_cpu(cpu) { struct srcu_data *cpuc = per_cpu_ptr(ssp->sda, cpu); - sum += READ_ONCE(cpuc->srcu_lock_count[0]); - sum += READ_ONCE(cpuc->srcu_lock_count[1]); - sum -= READ_ONCE(cpuc->srcu_unlock_count[0]); - sum -= READ_ONCE(cpuc->srcu_unlock_count[1]); + sum += atomic_long_read(&cpuc->srcu_lock_count[0]); + sum += atomic_long_read(&cpuc->srcu_lock_count[1]); + sum -= atomic_long_read(&cpuc->srcu_unlock_count[0]); + sum -= atomic_long_read(&cpuc->srcu_unlock_count[1]); } return sum; } @@ -636,7 +636,7 @@ int __srcu_read_lock(struct srcu_struct *ssp) int idx; idx = READ_ONCE(ssp->srcu_idx) & 0x1; - this_cpu_inc(ssp->sda->srcu_lock_count[idx]); + this_cpu_inc(ssp->sda->srcu_lock_count[idx].counter); smp_mb(); /* B */ /* Avoid leaking the critical section. */ return idx; } @@ -650,7 +650,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock); void __srcu_read_unlock(struct srcu_struct *ssp, int idx) { smp_mb(); /* C */ /* Avoid leaking the critical section. */ - this_cpu_inc(ssp->sda->srcu_unlock_count[idx]); + this_cpu_inc(ssp->sda->srcu_unlock_count[idx].counter); } EXPORT_SYMBOL_GPL(__srcu_read_unlock); @@ -1687,8 +1687,8 @@ void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf) struct srcu_data *sdp; sdp = per_cpu_ptr(ssp->sda, cpu); - u0 = data_race(sdp->srcu_unlock_count[!idx]); - u1 = data_race(sdp->srcu_unlock_count[idx]); + u0 = data_race(atomic_long_read(&sdp->srcu_unlock_count[!idx])); + u1 = data_race(atomic_long_read(&sdp->srcu_unlock_count[idx])); /* * Make sure that a lock is always counted if the corresponding @@ -1696,8 +1696,8 @@ void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf) */ smp_rmb(); - l0 = data_race(sdp->srcu_lock_count[!idx]); - l1 = data_race(sdp->srcu_lock_count[idx]); + l0 = data_race(atomic_long_read(&sdp->srcu_lock_count[!idx])); + l1 = data_race(atomic_long_read(&sdp->srcu_lock_count[idx])); c0 = l0 - u0; c1 = l1 - u1; From patchwork Wed Oct 19 22:58:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 5912 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp574565wrs; Wed, 19 Oct 2022 16:00:55 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4px+7yLAIj9ZJlGURbUdrwV5AewWIzpgMWhMfm/RrKn+N98TQJ1V1rq7tXjv3Fc8XioTuh X-Received: by 2002:a17:906:cc52:b0:78d:d477:5b7f with SMTP id mm18-20020a170906cc5200b0078dd4775b7fmr8613233ejb.558.1666220455623; Wed, 19 Oct 2022 16:00:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666220455; cv=none; d=google.com; s=arc-20160816; b=KkL+wWp7FSQR7D3J8hrizK9zQatTceZ8fg8+vfKHnSpNR6dJFcl2r3t+MKLTW7o06j yGi8XFXFlUwj2uuVLlBGA3wtWiF8/5gHXauqiqya7mHnce4iU/7LQepU3+szvS9EAddc GRln2BrC3a3py7G+APg0sKmP5hTMdt/vivpwFqX+5PBhLO2lDd4wVtHnvHFnuxRE0d9o dmjHu+36iiXyclqQE4K15JuOj/iFAKW/mjPgFP/VhOH+PdvuEcvDxdtL695OurnOH0wv 6rReiAAPoQ1jv9o3qZgpCmm4q0wlV+oleKPqZR1+204qgMk4Pdt3VVrt7iHwCnCXNDkz HKWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=zK+9MmvvhTfPLDlvN5RU41V+SOQQaX5MUjyEAuLLr+Q=; b=JjPcPTE2tDq9Cxpihg9LQmuy6Sin+mmi7mIHMJ5KD9OLXOweYHdiPDz022w/jBmtuf pDYPtr5d5tmeFs6zq2kQ6eIgN7xO7Po1qw9ZAFGashkF+m0D3GRIuOUwuxIb3xzvIioQ H3OhnhAJlPjWyD9fVV/wdJM7s5N9vQYw5DdC2cyoBoyfQ5cYmsJwMbCDFQbqxIQOd+Np sAHZaO4A2yZbD+GVHTHRtEGjCmvJr/OKtPS46ZKOMD5vNCoB1fcqyLEtTALwRbEK68p/ xN3/bgxnlALN75Fxv26AMrAC5WwH3bOnWB1qNwgo90WChkRrfpMm/3ByNjc33YJuV/or 0aRg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="aSL//Xz1"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s17-20020aa7c551000000b0045c9dbe290csi13849661edr.406.2022.10.19.16.00.30; Wed, 19 Oct 2022 16:00:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="aSL//Xz1"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231728AbiJSW7P (ORCPT + 99 others); Wed, 19 Oct 2022 18:59:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44508 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231317AbiJSW6y (ORCPT ); Wed, 19 Oct 2022 18:58:54 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 214531CBAAF; Wed, 19 Oct 2022 15:58:51 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 5ADD5B82622; Wed, 19 Oct 2022 22:58:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EEA8AC4314C; Wed, 19 Oct 2022 22:58:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666220328; bh=XNWl0HIokgwRVMPagCZ5aA6xkbCFeaPBVMDULtjlh9U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aSL//Xz1geDrL70We06qUFfxPQB1fefbnlBS69m60Xc9ptHZUuYCYSx0bFKWDEmZB kAhMsNar9Pa3GHSQ0Jgzi8eDZg6HJNUfQtJCzkF006C6zT4vwIwZQOazU7xWTEqEfW WFdRN9WRt1VZdpLXfCFwWhxa/9o0+ZkA/iu56sd5mG24kAcLRhnepJrKLKduPKw/xP 4HH8j7XDITWJg1Z1T1o5oXHMwi2tAwCKFhXAMV581PgKBWHFNsxW8ncPAGCvtY5vrt 8SzJmbDfQc3nahuTzMqNaOibBp9D1pd+w4akppt8objl6lcsFDLxDWD1y2NqYL/qPB XBQUs9KgsyosA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id ACFC35C0879; Wed, 19 Oct 2022 15:58:47 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Randy Dunlap , Frederic Weisbecker , Thomas Gleixner , John Ogness , Petr Mladek Subject: [PATCH v3 rcu 02/11] srcu: Create an srcu_read_lock_nmisafe() and srcu_read_unlock_nmisafe() Date: Wed, 19 Oct 2022 15:58:37 -0700 Message-Id: <20221019225846.2501109-2-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> References: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747158780615660984?= X-GMAIL-MSGID: =?utf-8?q?1747158780615660984?= On strict load-store architectures, the use of this_cpu_inc() by srcu_read_lock() and srcu_read_unlock() is not NMI-safe in TREE SRCU. To see this suppose that an NMI arrives in the middle of srcu_read_lock(), just after it has read ->srcu_lock_count, but before it has written the incremented value back to memory. If that NMI handler also does srcu_read_lock() and srcu_read_lock() on that same srcu_struct structure, then upon return from that NMI handler, the interrupted srcu_read_lock() will overwrite the NMI handler's update to ->srcu_lock_count, but leave unchanged the NMI handler's update by srcu_read_unlock() to ->srcu_unlock_count. This can result in a too-short SRCU grace period, which can in turn result in arbitrary memory corruption. If the NMI handler instead interrupts the srcu_read_unlock(), this can result in eternal SRCU grace periods, which is not much better. This commit therefore creates a pair of new srcu_read_lock_nmisafe() and srcu_read_unlock_nmisafe() functions, which allow SRCU readers in both NMI handlers and in process and IRQ context. It is bad practice to mix the existing and the new _nmisafe() primitives on the same srcu_struct structure. Use one set or the other, not both. Just to underline that "bad practice" point, using srcu_read_lock() at process level and srcu_read_lock_nmisafe() in your NMI handler will not, repeat NOT, work. If you do not immediately understand why this is the case, please review the earlier paragraphs in this commit log. [ paulmck: Apply kernel test robot feedback. ] [ paulmck: Apply feedback from Randy Dunlap. ] [ paulmck: Apply feedback from John Ogness. ] Link: https://lore.kernel.org/all/20220910221947.171557773@linutronix.de/ Signed-off-by: Paul E. McKenney Acked-by: Randy Dunlap # build-tested Reviewed-by: Frederic Weisbecker Cc: Thomas Gleixner Cc: John Ogness Cc: Petr Mladek --- arch/Kconfig | 3 +++ include/linux/srcu.h | 39 ++++++++++++++++++++++++++++++++++++ include/linux/srcutiny.h | 11 ++++++++++ include/linux/srcutree.h | 3 +++ kernel/rcu/Kconfig | 3 +++ kernel/rcu/rcutorture.c | 11 ++++++++-- kernel/rcu/srcutree.c | 43 ++++++++++++++++++++++++++++++++++++---- 7 files changed, 107 insertions(+), 6 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 8f138e580d1ae..6b95244c3057d 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -468,6 +468,9 @@ config ARCH_WANT_IRQS_OFF_ACTIVATE_MM config ARCH_HAVE_NMI_SAFE_CMPXCHG bool +config ARCH_HAS_NMI_SAFE_THIS_CPU_OPS + bool + config HAVE_ALIGNED_STRUCT_PAGE bool help diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 01226e4d960a0..2cc8321c0c86a 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -52,6 +52,8 @@ int init_srcu_struct(struct srcu_struct *ssp); #else /* Dummy definition for things like notifiers. Actual use gets link error. */ struct srcu_struct { }; +int __srcu_read_lock_nmisafe(struct srcu_struct *ssp, bool chknmisafe) __acquires(ssp); +void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx, bool chknmisafe) __releases(ssp); #endif void call_srcu(struct srcu_struct *ssp, struct rcu_head *head, @@ -166,6 +168,25 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp) return retval; } +/** + * srcu_read_lock_nmisafe - register a new reader for an SRCU-protected structure. + * @ssp: srcu_struct in which to register the new reader. + * + * Enter an SRCU read-side critical section, but in an NMI-safe manner. + * See srcu_read_lock() for more information. + */ +static inline int srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp) +{ + int retval; + + if (IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) + retval = __srcu_read_lock_nmisafe(ssp); + else + retval = __srcu_read_lock(ssp); + rcu_lock_acquire(&(ssp)->dep_map); + return retval; +} + /* Used by tracing, cannot be traced and cannot invoke lockdep. */ static inline notrace int srcu_read_lock_notrace(struct srcu_struct *ssp) __acquires(ssp) @@ -191,6 +212,24 @@ static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx) __srcu_read_unlock(ssp, idx); } +/** + * srcu_read_unlock_nmisafe - unregister a old reader from an SRCU-protected structure. + * @ssp: srcu_struct in which to unregister the old reader. + * @idx: return value from corresponding srcu_read_lock(). + * + * Exit an SRCU read-side critical section, but in an NMI-safe manner. + */ +static inline void srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) + __releases(ssp) +{ + WARN_ON_ONCE(idx & ~0x1); + rcu_lock_release(&(ssp)->dep_map); + if (IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) + __srcu_read_unlock_nmisafe(ssp, idx); + else + __srcu_read_unlock(ssp, idx); +} + /* Used by tracing, cannot be traced and cannot call lockdep. */ static inline notrace void srcu_read_unlock_notrace(struct srcu_struct *ssp, int idx) __releases(ssp) diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 5aa5e0faf6a12..278331bd77660 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -90,4 +90,15 @@ static inline void srcu_torture_stats_print(struct srcu_struct *ssp, data_race(READ_ONCE(ssp->srcu_idx_max))); } +static inline int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) +{ + BUG(); + return 0; +} + +static inline void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) +{ + BUG(); +} + #endif diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 0c4eca07d78d5..d45dd507f4a56 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -154,4 +154,7 @@ void synchronize_srcu_expedited(struct srcu_struct *ssp); void srcu_barrier(struct srcu_struct *ssp); void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf); +int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp); +void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) __releases(ssp); + #endif diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index d471d22a5e21b..f53ad63b2bc63 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -72,6 +72,9 @@ config TREE_SRCU help This option selects the full-fledged version of SRCU. +config NEED_SRCU_NMI_SAFE + def_bool HAVE_NMI && !ARCH_HAS_NMI_SAFE_THIS_CPU_OPS && !TINY_SRCU + config TASKS_RCU_GENERIC def_bool TASKS_RCU || TASKS_RUDE_RCU || TASKS_TRACE_RCU select SRCU diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 503c2aa845a4a..b4c74ce102256 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -615,10 +615,14 @@ static struct rcu_torture_ops rcu_busted_ops = { DEFINE_STATIC_SRCU(srcu_ctl); static struct srcu_struct srcu_ctld; static struct srcu_struct *srcu_ctlp = &srcu_ctl; +static struct rcu_torture_ops srcud_ops; static int srcu_torture_read_lock(void) __acquires(srcu_ctlp) { - return srcu_read_lock(srcu_ctlp); + if (cur_ops == &srcud_ops) + return srcu_read_lock_nmisafe(srcu_ctlp); + else + return srcu_read_lock(srcu_ctlp); } static void @@ -642,7 +646,10 @@ srcu_read_delay(struct torture_random_state *rrsp, struct rt_read_seg *rtrsp) static void srcu_torture_read_unlock(int idx) __releases(srcu_ctlp) { - srcu_read_unlock(srcu_ctlp, idx); + if (cur_ops == &srcud_ops) + srcu_read_unlock_nmisafe(srcu_ctlp, idx); + else + srcu_read_unlock(srcu_ctlp, idx); } static int torture_srcu_read_lock_held(void) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 25e9458da6a26..32a94b254d29f 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -654,6 +654,41 @@ void __srcu_read_unlock(struct srcu_struct *ssp, int idx) } EXPORT_SYMBOL_GPL(__srcu_read_unlock); +#ifdef CONFIG_NEED_SRCU_NMI_SAFE + +/* + * Counts the new reader in the appropriate per-CPU element of the + * srcu_struct, but in an NMI-safe manner using RMW atomics. + * Returns an index that must be passed to the matching srcu_read_unlock(). + */ +int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) +{ + int idx; + struct srcu_data *sdp = raw_cpu_ptr(ssp->sda); + + idx = READ_ONCE(ssp->srcu_idx) & 0x1; + atomic_long_inc(&sdp->srcu_lock_count[idx]); + smp_mb__after_atomic(); /* B */ /* Avoid leaking the critical section. */ + return idx; +} +EXPORT_SYMBOL_GPL(__srcu_read_lock_nmisafe); + +/* + * Removes the count for the old reader from the appropriate per-CPU + * element of the srcu_struct. Note that this may well be a different + * CPU than that which was incremented by the corresponding srcu_read_lock(). + */ +void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) +{ + struct srcu_data *sdp = raw_cpu_ptr(ssp->sda); + + smp_mb__before_atomic(); /* C */ /* Avoid leaking the critical section. */ + atomic_long_inc(&sdp->srcu_unlock_count[idx]); +} +EXPORT_SYMBOL_GPL(__srcu_read_unlock_nmisafe); + +#endif // CONFIG_NEED_SRCU_NMI_SAFE + /* * Start an SRCU grace period. */ @@ -1090,7 +1125,7 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp, int ss_state; check_init_srcu_struct(ssp); - idx = srcu_read_lock(ssp); + idx = __srcu_read_lock_nmisafe(ssp); ss_state = smp_load_acquire(&ssp->srcu_size_state); if (ss_state < SRCU_SIZE_WAIT_CALL) sdp = per_cpu_ptr(ssp->sda, 0); @@ -1123,7 +1158,7 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp, srcu_funnel_gp_start(ssp, sdp, s, do_norm); else if (needexp) srcu_funnel_exp_start(ssp, sdp_mynode, s); - srcu_read_unlock(ssp, idx); + __srcu_read_unlock_nmisafe(ssp, idx); return s; } @@ -1427,13 +1462,13 @@ void srcu_barrier(struct srcu_struct *ssp) /* Initial count prevents reaching zero until all CBs are posted. */ atomic_set(&ssp->srcu_barrier_cpu_cnt, 1); - idx = srcu_read_lock(ssp); + idx = __srcu_read_lock_nmisafe(ssp); if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, 0)); else for_each_possible_cpu(cpu) srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, cpu)); - srcu_read_unlock(ssp, idx); + __srcu_read_unlock_nmisafe(ssp, idx); /* Remove the initial count, at which point reaching zero can happen. */ if (atomic_dec_and_test(&ssp->srcu_barrier_cpu_cnt)) From patchwork Wed Oct 19 22:58:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 5913 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp574635wrs; Wed, 19 Oct 2022 16:01:05 -0700 (PDT) X-Google-Smtp-Source: AMsMyM71sc9fSZQnWucaDiQde91DD75jZv8FDc3+TFdwZsMov0tOn6AuDW+ielx8Y4ML1tvY0OLj X-Received: by 2002:a17:907:6ea9:b0:794:8b93:2e44 with SMTP id sh41-20020a1709076ea900b007948b932e44mr3329522ejc.407.1666220465451; Wed, 19 Oct 2022 16:01:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666220465; cv=none; d=google.com; s=arc-20160816; b=XGPcpjPvH2JUBRnpiaZG+evJFkUZylUWzVQrFB0ck7fvDsJTVhvwhIS4TRfIoWMcN3 0X44+7E9zbMcs4blOyiGfOM4dhA205DmO+rdBO3Y3P3P7kiE2PGBUhd/H4/OAPH9x4O8 wstfMRQySje+um5JAXXxQXUkVJCbkWoXITwrWVLxDgZsBoOBvvkPEgiaNQRpn+WalmWu APgwUoc0Vg/yr1/ijbD9HAsKLQDIq/k+Gg80sgtbjt7Tf14e24yqfFy8xBggt5401KZb hxvT2ZnIXyoN90DhbPojznBogfhSg5v+IUAq7/9dAxWVC6wfoqusiHta/knh4Zdt17Rh zYzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ppbOHQWLHBoHG+0bS4TaiZ2HszHSqHaEdDd2Me2Wg6c=; b=A6HEFA50b0rKcpVXelxgamFjO5EanxlSqPADjUEw72x8+zFanZO1EoZEWM19dgMN8s GUGcymO9/uHqI9oNIbeHxSRaInQHf2YFOwJYmStBrnbZQdOig4Tbd+aG8IYCxn6L9enW l7hE3MRHBZJuKCclN9CFdKlLnft7uTRtFGF53GT3/4CTJzxJX9aVUHRrO9LlgbTYApzR 2UfWyuH4ysQJL5EGoZtmbNAnJoB3vW8vH4L3EF7XQ3nZfb0wF7jqFsm9ogr/IZhRCJhF Mh+jkfg1zDKVmyEPJTlusJcTOg9e8NeotwcXMMAbwZggx0rlOs61ofA+0V/MpsvGk21H qA3g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=cU8gnXdG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ds14-20020a170907724e00b0078c3197bf86si14361319ejc.533.2022.10.19.16.00.39; Wed, 19 Oct 2022 16:01:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=cU8gnXdG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231779AbiJSW7W (ORCPT + 99 others); Wed, 19 Oct 2022 18:59:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44516 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231297AbiJSW6x (ORCPT ); Wed, 19 Oct 2022 18:58:53 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 212511CBAAD; Wed, 19 Oct 2022 15:58:51 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 58FD9B8261D; Wed, 19 Oct 2022 22:58:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F1D9EC43145; Wed, 19 Oct 2022 22:58:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666220328; bh=BdQBnT/5Lg+ztsBrYfPMi3iUxQTo6LmC+1YzXMhFGpc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cU8gnXdGPAvsqb8jSRTMD5Tspc+o/pThgguWUy9C0+HKiSB02l5GfNIGxS0JkL7uX pq+RNB4RMzE074QpEWxilIEtr7DP28edGfkQiaGDDJT2n8K1gKZlATm9DXGgaKo8wG hDWPKAtu3fuXKGyMMvmjFBcs3gV9VQjqWDVSyyr2rbyzRt0hU2S/kJ0OCsX9eRyAy5 NEQ/jAccQDyxbgxzafRZOvp74EWbisM43SneTv+DSiK+TjDYrUYdhvKA7Y95NiHV4v Y+rymIeTADKMHoCQUGs00uf3WFhiACFYWcP3ZkoSodalaYjLJfkN8YHvwGgjBfbTku PN9wNMrWoqdSw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id AF0AF5C0890; Wed, 19 Oct 2022 15:58:47 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Frederic Weisbecker , Thomas Gleixner , John Ogness , Petr Mladek Subject: [PATCH v3 rcu 03/11] srcu: Check for consistent per-CPU per-srcu_struct NMI safety Date: Wed, 19 Oct 2022 15:58:38 -0700 Message-Id: <20221019225846.2501109-3-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> References: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747158790513648936?= X-GMAIL-MSGID: =?utf-8?q?1747158790513648936?= This commit adds runtime checks to verify that a given srcu_struct uses consistent NMI-safe (or not) read-side primitives on a per-CPU basis. Link: https://lore.kernel.org/all/20220910221947.171557773@linutronix.de/ Signed-off-by: Paul E. McKenney Reviewed-by: Frederic Weisbecker Cc: Thomas Gleixner Cc: John Ogness Cc: Petr Mladek --- include/linux/srcu.h | 4 ++-- include/linux/srcutiny.h | 4 ++-- include/linux/srcutree.h | 9 +++++++-- kernel/rcu/srcutree.c | 38 ++++++++++++++++++++++++++++++++------ 4 files changed, 43 insertions(+), 12 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 2cc8321c0c86a..565f60d574847 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -180,7 +180,7 @@ static inline int srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp int retval; if (IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) - retval = __srcu_read_lock_nmisafe(ssp); + retval = __srcu_read_lock_nmisafe(ssp, true); else retval = __srcu_read_lock(ssp); rcu_lock_acquire(&(ssp)->dep_map); @@ -225,7 +225,7 @@ static inline void srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) WARN_ON_ONCE(idx & ~0x1); rcu_lock_release(&(ssp)->dep_map); if (IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) - __srcu_read_unlock_nmisafe(ssp, idx); + __srcu_read_unlock_nmisafe(ssp, idx, true); else __srcu_read_unlock(ssp, idx); } diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 278331bd77660..f890301f123df 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -90,13 +90,13 @@ static inline void srcu_torture_stats_print(struct srcu_struct *ssp, data_race(READ_ONCE(ssp->srcu_idx_max))); } -static inline int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) +static inline int __srcu_read_lock_nmisafe(struct srcu_struct *ssp, bool chknmisafe) { BUG(); return 0; } -static inline void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) +static inline void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx, bool chknmisafe) { BUG(); } diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index d45dd507f4a56..35ffdedf86ccb 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -25,6 +25,7 @@ struct srcu_data { /* Read-side state. */ atomic_long_t srcu_lock_count[2]; /* Locks per CPU. */ atomic_long_t srcu_unlock_count[2]; /* Unlocks per CPU. */ + int srcu_nmi_safety; /* NMI-safe srcu_struct structure? */ /* Update-side state. */ spinlock_t __private lock ____cacheline_internodealigned_in_smp; @@ -42,6 +43,10 @@ struct srcu_data { struct srcu_struct *ssp; }; +#define SRCU_NMI_UNKNOWN 0x0 +#define SRCU_NMI_NMI_UNSAFE 0x1 +#define SRCU_NMI_NMI_SAFE 0x2 + /* * Node in SRCU combining tree, similar in function to rcu_data. */ @@ -154,7 +159,7 @@ void synchronize_srcu_expedited(struct srcu_struct *ssp); void srcu_barrier(struct srcu_struct *ssp); void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf); -int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp); -void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) __releases(ssp); +int __srcu_read_lock_nmisafe(struct srcu_struct *ssp, bool chknmisafe) __acquires(ssp); +void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx, bool chknmisafe) __releases(ssp); #endif diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 32a94b254d29f..30575864fcfa3 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -626,6 +626,26 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) } EXPORT_SYMBOL_GPL(cleanup_srcu_struct); +/* + * Check for consistent NMI safety. + */ +static void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe) +{ + int nmi_safe_mask = 1 << nmi_safe; + int old_nmi_safe_mask; + struct srcu_data *sdp; + + if (!IS_ENABLED(CONFIG_PROVE_RCU)) + return; + sdp = raw_cpu_ptr(ssp->sda); + old_nmi_safe_mask = READ_ONCE(sdp->srcu_nmi_safety); + if (!old_nmi_safe_mask) { + WRITE_ONCE(sdp->srcu_nmi_safety, nmi_safe_mask); + return; + } + WARN_ONCE(old_nmi_safe_mask != nmi_safe_mask, "CPU %d old state %d new state %d\n", sdp->cpu, old_nmi_safe_mask, nmi_safe_mask); +} + /* * Counts the new reader in the appropriate per-CPU element of the * srcu_struct. @@ -638,6 +658,7 @@ int __srcu_read_lock(struct srcu_struct *ssp) idx = READ_ONCE(ssp->srcu_idx) & 0x1; this_cpu_inc(ssp->sda->srcu_lock_count[idx].counter); smp_mb(); /* B */ /* Avoid leaking the critical section. */ + srcu_check_nmi_safety(ssp, false); return idx; } EXPORT_SYMBOL_GPL(__srcu_read_lock); @@ -651,6 +672,7 @@ void __srcu_read_unlock(struct srcu_struct *ssp, int idx) { smp_mb(); /* C */ /* Avoid leaking the critical section. */ this_cpu_inc(ssp->sda->srcu_unlock_count[idx].counter); + srcu_check_nmi_safety(ssp, false); } EXPORT_SYMBOL_GPL(__srcu_read_unlock); @@ -661,7 +683,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock); * srcu_struct, but in an NMI-safe manner using RMW atomics. * Returns an index that must be passed to the matching srcu_read_unlock(). */ -int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) +int __srcu_read_lock_nmisafe(struct srcu_struct *ssp, bool chknmisafe) { int idx; struct srcu_data *sdp = raw_cpu_ptr(ssp->sda); @@ -669,6 +691,8 @@ int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) idx = READ_ONCE(ssp->srcu_idx) & 0x1; atomic_long_inc(&sdp->srcu_lock_count[idx]); smp_mb__after_atomic(); /* B */ /* Avoid leaking the critical section. */ + if (chknmisafe) + srcu_check_nmi_safety(ssp, true); return idx; } EXPORT_SYMBOL_GPL(__srcu_read_lock_nmisafe); @@ -678,12 +702,14 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock_nmisafe); * element of the srcu_struct. Note that this may well be a different * CPU than that which was incremented by the corresponding srcu_read_lock(). */ -void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) +void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx, bool chknmisafe) { struct srcu_data *sdp = raw_cpu_ptr(ssp->sda); smp_mb__before_atomic(); /* C */ /* Avoid leaking the critical section. */ atomic_long_inc(&sdp->srcu_unlock_count[idx]); + if (chknmisafe) + srcu_check_nmi_safety(ssp, true); } EXPORT_SYMBOL_GPL(__srcu_read_unlock_nmisafe); @@ -1125,7 +1151,7 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp, int ss_state; check_init_srcu_struct(ssp); - idx = __srcu_read_lock_nmisafe(ssp); + idx = __srcu_read_lock_nmisafe(ssp, false); ss_state = smp_load_acquire(&ssp->srcu_size_state); if (ss_state < SRCU_SIZE_WAIT_CALL) sdp = per_cpu_ptr(ssp->sda, 0); @@ -1158,7 +1184,7 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp, srcu_funnel_gp_start(ssp, sdp, s, do_norm); else if (needexp) srcu_funnel_exp_start(ssp, sdp_mynode, s); - __srcu_read_unlock_nmisafe(ssp, idx); + __srcu_read_unlock_nmisafe(ssp, idx, false); return s; } @@ -1462,13 +1488,13 @@ void srcu_barrier(struct srcu_struct *ssp) /* Initial count prevents reaching zero until all CBs are posted. */ atomic_set(&ssp->srcu_barrier_cpu_cnt, 1); - idx = __srcu_read_lock_nmisafe(ssp); + idx = __srcu_read_lock_nmisafe(ssp, false); if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, 0)); else for_each_possible_cpu(cpu) srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, cpu)); - __srcu_read_unlock_nmisafe(ssp, idx); + __srcu_read_unlock_nmisafe(ssp, idx, false); /* Remove the initial count, at which point reaching zero can happen. */ if (atomic_dec_and_test(&ssp->srcu_barrier_cpu_cnt)) From patchwork Wed Oct 19 22:58:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 5909 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp574032wrs; Wed, 19 Oct 2022 16:00:00 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6I0hcBzSj1m3RVOE0x6bqoitUYrOf3QELKgCUQtAITVEeXEpiXzmYWv2p8dRoqPpsDKKp/ X-Received: by 2002:a05:6402:1604:b0:458:b2b7:bd5a with SMTP id f4-20020a056402160400b00458b2b7bd5amr9195354edv.73.1666220399899; Wed, 19 Oct 2022 15:59:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666220399; cv=none; d=google.com; s=arc-20160816; b=RdVGnG4/vaESgQAX4dG6Uwbq74r4rlYrnAU9qK/ESCRF1jbOrx9iMlqDFFYmuvHOm4 lwUM2WwAgut99NSqiOZG45ERG0FKLJCZBRLirmbY6ee7R2Ev1yOJYzP1t4hQ/MgC/kpp 4QI3DHHqKjz/6r0NKCFq+QtWkQ6t+z1Q/T1rm55KJQCRQ8fMv3G55IqDIRoCwdoKFQqM XZTBiUoEFADvqQrbmqiqI+Pb5VO+Fcz5NLmIAFyasBf57+2pYLxeEC0GITBOZmDIC+fY X+8hfQe8QTkZIKchNBCq+u5g0rz75WODwVqc6zHPgJYXaINjT0Ld20WuUJ9BP82wq08/ DQ/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=o9C7HFdJe+TpxLr3OltZKAXHOpT8mFPedow8UuaZ3Uk=; b=zenDVnNmLtOGUUiLHJSX/vXqSPrrH9FBZZdc/rctGZtU5+PdCHR7nR5Bi9HMncfdan CKUx+qikxbs/quFXIS4oxkAAFm1d2iaIwVdmZIXQ1BJlX65VzGhYMltAvO6G5IsWLX0w nUic/MYlqV5v7UiWO/7xuz48AjXeKEy4iQvN/5aP7ZNQMsa1B5WxAcqq+194ClFb2xm0 46mfy0w+k4ojjvwOi5grX+/cAYJ5CUDscPTHge1koknKrwD4EVqUnO6QgB5i5sIUyjmV whx+TqPirI22YZ4NDMSpSz/wczFus9gJcVuA+kPIaA/0G133EQgYRVbj2PCKTpcYYvAr XPPw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=q7M1Vv9l; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id rs8-20020a170907036800b0078306c5c48asi12428840ejb.250.2022.10.19.15.59.35; Wed, 19 Oct 2022 15:59:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=q7M1Vv9l; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231687AbiJSW7N (ORCPT + 99 others); Wed, 19 Oct 2022 18:59:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44514 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231294AbiJSW6x (ORCPT ); Wed, 19 Oct 2022 18:58:53 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2021B1C97D5; Wed, 19 Oct 2022 15:58:51 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 4BDE4B82620; Wed, 19 Oct 2022 22:58:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 01CEAC4314D; Wed, 19 Oct 2022 22:58:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666220328; bh=uYczQLLZZuNFqlq5NKe5CWa3nO/1WLyhpggz2xg9uhA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=q7M1Vv9lN62MBvoQb5g5kOtBWau7pjX4utTN721A0qI1T3bW4dJJ465cA0rrNCWZU ANWg7UiQUTzktsQaVP+ihkXfhqwFjV0sZqx/nuwYgBeX0T9J0a7LTYsC4GRYtc9ef1 F2gCDcG418R5GTqMIZWJI+77HtI4oACAfA2X7DlJ8N57t/knxdBQkCRINgRoGGDS2D DkK3qIWyhQ2V+8cZiLGrXhdHH4iiFDX3ifV0Gtb+GL7Twu+Kpw7wNrmA+1T3GyJjOz t3G9FWGOW+shnkH3J2Num1HsvLc3XiZyQvthJ3KATqOHFim1VOZEPHcO3S30c3Gnjx Dy/UY6bj23tNA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id B14B45C0920; Wed, 19 Oct 2022 15:58:47 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Frederic Weisbecker , Thomas Gleixner , John Ogness , Petr Mladek Subject: [PATCH v3 rcu 04/11] srcu: Check for consistent global per-srcu_struct NMI safety Date: Wed, 19 Oct 2022 15:58:39 -0700 Message-Id: <20221019225846.2501109-4-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> References: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747158722109231310?= X-GMAIL-MSGID: =?utf-8?q?1747158722109231310?= This commit adds runtime checks to verify that a given srcu_struct uses consistent NMI-safe (or not) read-side primitives globally, but based on the per-CPU data. These global checks are made by the grace-period code that must scan the srcu_data structures anyway, and are done only in kernels built with CONFIG_PROVE_RCU=y. Link: https://lore.kernel.org/all/20220910221947.171557773@linutronix.de/ Signed-off-by: Paul E. McKenney Reviewed-by: Frederic Weisbecker Cc: Thomas Gleixner Cc: John Ogness Cc: Petr Mladek --- kernel/rcu/srcutree.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 30575864fcfa3..87ae6f5c1edae 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -429,13 +429,18 @@ static unsigned long srcu_readers_lock_idx(struct srcu_struct *ssp, int idx) static unsigned long srcu_readers_unlock_idx(struct srcu_struct *ssp, int idx) { int cpu; + unsigned long mask = 0; unsigned long sum = 0; for_each_possible_cpu(cpu) { struct srcu_data *cpuc = per_cpu_ptr(ssp->sda, cpu); sum += atomic_long_read(&cpuc->srcu_unlock_count[idx]); + if (IS_ENABLED(CONFIG_PROVE_RCU)) + mask = mask | READ_ONCE(cpuc->srcu_nmi_safety); } + WARN_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && (mask & (mask >> 1)), + "Mixed NMI-safe readers for srcu_struct at %ps.\n", ssp); return sum; } From patchwork Wed Oct 19 22:58:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 5916 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp575311wrs; Wed, 19 Oct 2022 16:02:26 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4okxqlqR+nF39EVrHOpx2bWnPt+qI3skj7wnxD4XG8kR6aHZYeAQFL4TpJvo1oCKLbPszp X-Received: by 2002:a05:6402:1e8e:b0:45c:af84:63dd with SMTP id f14-20020a0564021e8e00b0045caf8463ddmr9670912edf.190.1666220546420; Wed, 19 Oct 2022 16:02:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666220546; cv=none; d=google.com; s=arc-20160816; b=HNYYlgV6+7sHuRJdys1TRPtLHNGhZ59aAtBfMjdgOB0Xt983tzcZarpH35zxHIA+Xf NLbnE3RyZnCQdPOba4l6p5U67JqPcLfTv5BU1BRL1kYNZU2uOSM5UM8MlW2ldEOOJbfz krPI+KseZx9UUex/knpiaFVOdj+09JrVALvaP/zLQvhU6DlIogN5B4kMsDy8b1hFru61 H0C6KOGj6WL7Dw5sCl2UmM9GW2DBd3h0dh5LmBmaFcsYkJs7y4tRnsHIATywAOOdBW3V IlEaDb9ZuIpSrifSW0RQQo+7lZ7LoiTA9HfgVbwXQBAZX5ash0PILy7qnRcKCf5UUMPN QJHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=M+08Rpyk7mX+8l1elKiciqp6OQscgdiUIswQxTNpI94=; b=0TmX7ifiYycTIu4mXr/KDIuwyQ3UcfYU7QugzDBehvRxG+J0WEicQOgy+BoytaPanT D88cWGc2RMnVAg6aAD0Vgg2fitw0XJ6Hq7uxpb3oWOL1NTfsV/EX5NADfCqFtjYYcSmx N/yLqV5p7pDNLk2gDpsfLpcXaNag+PpnolXTxtIfIhXElBgXBH9yYiq4mkaSnt2S5bLy 2gzrfSjHhJ9o8bNQdwd1QEEsfaHnpLScEOETIjCOx6o/pxHbuwY9PCNAJ+tG+xqGI4Ca BGRoNP+G75GSDXAwOlL34T6H516mFo+nH3rQf3QX+Gv3W/wCfJ8dMdoU/l1Oz3TLxvSH NgBw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=VHSeKWao; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d9-20020a50fb09000000b0045fccff0195si617412edq.313.2022.10.19.16.02.00; Wed, 19 Oct 2022 16:02:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=VHSeKWao; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231640AbiJSW7J (ORCPT + 99 others); Wed, 19 Oct 2022 18:59:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44508 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230440AbiJSW6x (ORCPT ); Wed, 19 Oct 2022 18:58:53 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 21ACD1CC3EE; Wed, 19 Oct 2022 15:58:51 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 67BB4B8261C; Wed, 19 Oct 2022 22:58:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0B161C4314F; Wed, 19 Oct 2022 22:58:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666220328; bh=U7EwiwQT1WIRqzav2oy6CKAJs3S0FYOZZwSmr789m/Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VHSeKWaok+7TErScKygedoM5pxkR28Xt8UhuuCNhxDJHK7bcHWtlUo29Lkc/PrbWb 1b9KpVyaPa/k44ez7NM+GkChSe45i0RD+vtTlIqQ6KxgGj/b9CDGM448AOGXxUYPP2 IplplV+7Up+araeM5uzRfUHNI+bW0O5cMtB1jJs61eMWuWvebcb3m9qoppgC7/igUE SFyASmOxJsO2CWeaTUAm1K8FeIMB3Q3H4IxZD5O/bgLm8gCRPAx8m6ZJH0ml46kKm9 YCAHYFxGInG9rn/bBct8txWus9bYK/kj5LZGBfamSJch5BepTlpx2X4qvil5B1lHMZ hZSCNYcoONwtA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id B34FC5C0A04; Wed, 19 Oct 2022 15:58:47 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Frederic Weisbecker , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , John Ogness , Petr Mladek , x86@kernel.org Subject: [PATCH v3 rcu 05/11] arch/x86: Add ARCH_HAS_NMI_SAFE_THIS_CPU_OPS Kconfig option Date: Wed, 19 Oct 2022 15:58:40 -0700 Message-Id: <20221019225846.2501109-5-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> References: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747158875723990743?= X-GMAIL-MSGID: =?utf-8?q?1747158875723990743?= The x86 architecture uses an add-to-memory instruction to implement this_cpu_add(), which is NMI safe. This means that the old and more-efficient srcu_read_lock() may be used in NMI context, without the need for srcu_read_lock_nmisafe(). Therefore, add the new Kconfig option ARCH_HAS_NMI_SAFE_THIS_CPU_OPS to arch/x86/Kconfig, which will cause NEED_SRCU_NMI_SAFE to be deselected, thus preserving the current srcu_read_lock() behavior. Link: https://lore.kernel.org/all/20220910221947.171557773@linutronix.de/ Signed-off-by: Paul E. McKenney Reviewed-by: Frederic Weisbecker Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: John Ogness Cc: Petr Mladek Cc: --- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 6d1879ef933a2..bcb3190eaa266 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -81,6 +81,7 @@ config X86 select ARCH_HAS_KCOV if X86_64 select ARCH_HAS_MEM_ENCRYPT select ARCH_HAS_MEMBARRIER_SYNC_CORE + select ARCH_HAS_NMI_SAFE_THIS_CPU_OPS select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE select ARCH_HAS_PMEM_API if X86_64 select ARCH_HAS_PTE_DEVMAP if X86_64 From patchwork Wed Oct 19 22:58:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 5908 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp573978wrs; Wed, 19 Oct 2022 15:59:51 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7Mn2DhO1D7YPrwhx7nwNNyf7iBZY6wZLklilQsuid6BA+5oYMuM2B7kTYVyrMVmeBa7fCt X-Received: by 2002:a05:6402:5cd:b0:446:5965:f4af with SMTP id n13-20020a05640205cd00b004465965f4afmr9512315edx.12.1666220391252; Wed, 19 Oct 2022 15:59:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666220391; cv=none; d=google.com; s=arc-20160816; b=ccj+fDgWXvwXFPSVzB4/omz6G6Zo/QA2p88NYk+mO4L6MjCbtNZ2JxzzsvklyCM4wv PMPUkRWlxwg4ERjcDiBzuxziUql7k8OUtMAjmTu5SoELOunDivtFK/xkYo2cY9F6aeNt c9ZFFQ0+DeY8PhjLwjqaYkmCfc1KSl7IpeADowTYNORTbTkPKFly9va4B1PF+TXfxdbk LLle2OFVkHqUPOsbpI1oJhLpBR3zlU1wU9vacNNO6Y0QaaD4rI65NW/7ItQb1aaxVGM2 oxkVb6SA6kTh1i4IQ2JhliA5oDnvqxg6IYUdKsgSpHDWb9WM6fYK0Kjoa7Kg2vRxm/+t wR6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Z5fvIEf2ZB/UvPQnfu2JGU0LjkOn0x+0dXuoiPOSSDk=; b=Q4n2zyBaiVOoDrIeqvva08vBp1cCWZSoyHJyLt/zbXgJmxwtzHec832V67RNVEzmiL P4Uyfp4QzSQpoqPQTOIVJ3ueiIJE6oF9ec+SvAaC9qrIAs7YiCK0D8QdpXY9m8TLcxx4 D9BrL7ZR+ivNpKp8UJH9I6wR7yMeDTr2+zMwBmmsGlmC5OgcxVMtXywYlt3xGXeNJFF+ M8oTO8F/pzSMCxINwwKfORoWmY/LbIk6urgfB+/WlwWcj2aXVg0KneB4hHGL2dLC6coY 1yhZfVPt4NjwUPsC/7RltGVathNt+tf09tekkcLmegYBMFt4KXr4BwjdWjUKthw8NNsa yTlA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=VfqMvXca; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q20-20020a1709064c9400b00781d82a6fc9si12492228eju.264.2022.10.19.15.59.26; Wed, 19 Oct 2022 15:59:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=VfqMvXca; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231324AbiJSW6y (ORCPT + 99 others); Wed, 19 Oct 2022 18:58:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44494 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231209AbiJSW6w (ORCPT ); Wed, 19 Oct 2022 18:58:52 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 134321CC768; Wed, 19 Oct 2022 15:58:50 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6074F619E8; Wed, 19 Oct 2022 22:58:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4F393C4315C; Wed, 19 Oct 2022 22:58:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666220328; bh=Nb+xsXAeDl9rjVwYT8+uivwa9boPW8pnKYqH3COPbBY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VfqMvXcayMmhZ5/iuy3j3lXwyVT0aBX9tfr5xmh7SpBuHwKwazuBBRp/wJlePRVBr mh7NW+8FEl+rP7Bu4leZvBTLjzJzpKMMWOXoDn+LTxRpuVzWLVJqAJOl8nWaopjnzv kARyhKaFZJSQ5r3CgujnfJnwIQ0uUgPV5WkMHfLOU2CBMkjxREKJe4nEZFBwp7ODwa AbCK5NvLIINjI72QR/DrBWPIO2DVw4C2QCMshI6JjQoFMDTAsyFvx4/5SoEMrgeGUa J2t32wNwZJr07kzRfhQM3X4LuQ61Z0SrNeNFHE0hp2ba/rUMHOtnlm1OXlh51UFI0q EQtpf4HBUuGIQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id B59075C0A40; Wed, 19 Oct 2022 15:58:47 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay , Frederic Weisbecker , Boqun Feng , Catalin Marinas , Will Deacon , Thomas Gleixner , John Ogness , Petr Mladek , linux-arm-kernel@lists.infradead.org Subject: [PATCH v3 rcu 06/11] arch/arm64: Add ARCH_HAS_NMI_SAFE_THIS_CPU_OPS Kconfig option Date: Wed, 19 Oct 2022 15:58:41 -0700 Message-Id: <20221019225846.2501109-6-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> References: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747158712678332606?= X-GMAIL-MSGID: =?utf-8?q?1747158712678332606?= The arm64 architecture uses either an LL/SC loop (old systems) or an LSE stadd instruction (new systems) to implement this_cpu_add(), both of which are NMI safe. This means that the old and more-efficient srcu_read_lock() may be used in NMI context, without the need for srcu_read_lock_nmisafe(). Therefore, add the new Kconfig option ARCH_HAS_NMI_SAFE_THIS_CPU_OPS to arch/arm64/Kconfig, which will cause NEED_SRCU_NMI_SAFE to be deselected, thus preserving the current srcu_read_lock() behavior. Link: https://lore.kernel.org/all/20220910221947.171557773@linutronix.de/ Suggested-by: Neeraj Upadhyay Suggested-by: Frederic Weisbecker Suggested-by: Boqun Feng Signed-off-by: Paul E. McKenney Cc: Catalin Marinas Cc: Will Deacon Cc: Thomas Gleixner Cc: John Ogness Cc: Petr Mladek Cc: --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 505c8a1ccbe0c..099ee812f3f18 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -31,6 +31,7 @@ config ARM64 select ARCH_HAS_KCOV select ARCH_HAS_KEEPINITRD select ARCH_HAS_MEMBARRIER_SYNC_CORE + select ARCH_HAS_NMI_SAFE_THIS_CPU_OPS select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE select ARCH_HAS_PTE_DEVMAP select ARCH_HAS_PTE_SPECIAL From patchwork Wed Oct 19 22:58:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 5915 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp574924wrs; Wed, 19 Oct 2022 16:01:41 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6xdjO8h68a8799Xud45FShUXv9wvb2y/vOJz3Plln6PV73OFcoObs++T9pdNjIsNRttNw3 X-Received: by 2002:a17:907:724c:b0:791:a7ff:aa7a with SMTP id ds12-20020a170907724c00b00791a7ffaa7amr4457805ejc.105.1666220501011; Wed, 19 Oct 2022 16:01:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666220501; cv=none; d=google.com; s=arc-20160816; b=QgaRYYnNiKRO2iUct2mxrXtNz5ey13HkT/t3Mld1nNUeUaYd+W4fI8OwpxkF7N1O2/ d26NpW9CxQyDgt81zK8/blkFWkv13lrcf6upKImV5a2M1a8/Ve3efRUHzdIVH4pH8ApQ RxTkPiwitbnyroqJLGUyZ3ET2SO6OgBAwWuUXChudCMnKaIivX+8sRztSC7LBKqgswA/ XUvEkdiMgOovx4SUyyqHAgKBmUTlkv41tcUcORdqbxop0riw8MppIG7K+QGoepjI2kdz 3vXYwNK5IkSegpmhxU3unszdAQYQLXb8fRcIE69ksjA61wl9LpYgiCXLFQBHdnz+6o1L TkKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=osKOGsjv7drK1beFepCiYzd4ihVSqTp/Fips9Ur2opE=; b=jIzjzzXFaDtvXwvKSncpUymH+gd6/qldtpdI2P0F/GsXbYZGRTnFl7fM6tG2SI/qpf waKCoLnJdUVVMB0fo4lZV2j895LfhFjq/KcuuInG4BpEDA72xuQ6IEinJS3mPG1USnoc W+8i7jPc9XZmaBJN1fTG4x6VnxSQSGUYuZo9O6Po4M3e1wxnv/gbc7M4Py9zPZ95PxY3 oeSjFkFlr9iL0/DjchI616+ID2w4OhoBlYaLhT54N3ch5vWYiXN6aUEWmNbo6Gy57UFb JM1RuucG2pqMyoshwVXY4Myo+urccOStDSeCPj16SHCztqOz3FhFS1c8+0llWGjWjqmW GDAg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=uN2Y657P; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id xf13-20020a17090731cd00b00788361f96a2si17097613ejb.776.2022.10.19.16.01.17; Wed, 19 Oct 2022 16:01:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=uN2Y657P; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230195AbiJSW7d (ORCPT + 99 others); Wed, 19 Oct 2022 18:59:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44510 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231392AbiJSW65 (ORCPT ); Wed, 19 Oct 2022 18:58:57 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A9821CC762; Wed, 19 Oct 2022 15:58:54 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 7E0CAB82625; Wed, 19 Oct 2022 22:58:50 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4F2CFC43158; Wed, 19 Oct 2022 22:58:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666220328; bh=q+Nh2vqtEVu5d5wW0RvHG/vkrFq6MOVxBevrDf1eSrY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uN2Y657PIuHGDtEfThRsABBfDWmMY2nPkfEzviDg/z53XUHh45lsuuSSilpTZJSpz 3NHYNYSreQ/zQyMhpaYFRd684T2E8r+C4cuCUGN8buVHW+bgLaKsVO5niw4BheQgfv oI5CKDeCHTWd2QrYmZc51KrYDFFP33b8kAwHeXne1pIAaKFfwlxGCJq7Uzo7gTSBRC lXUSAqY6VVWYmet5AZvswNsazH0XCr6W2/y4lIkcQ/XP8gY6a5SvaeLcOPBmnkRYcq /qOtDVcglyW/+UqM11sLKetAjdUp4xjzvBGSdoE0dPfB32hce9PNR70MKY34hCpHgA t5I0+l6rN+eLw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id B781A5C0AC5; Wed, 19 Oct 2022 15:58:47 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay , Frederic Weisbecker , Boqun Feng , Huacai Chen , WANG Xuerui , Thomas Gleixner , John Ogness , Petr Mladek , loongarch@lists.linux.dev Subject: [PATCH v3 rcu 07/11] arch/loongarch: Add ARCH_HAS_NMI_SAFE_THIS_CPU_OPS Kconfig option Date: Wed, 19 Oct 2022 15:58:42 -0700 Message-Id: <20221019225846.2501109-7-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> References: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747158828162108167?= X-GMAIL-MSGID: =?utf-8?q?1747158828162108167?= The loongarch architecture uses the atomic read-modify-write amadd instruction to implement this_cpu_add(), which is NMI safe. This means that the old and more-efficient srcu_read_lock() may be used in NMI context, without the need for srcu_read_lock_nmisafe(). Therefore, add the new Kconfig option ARCH_HAS_NMI_SAFE_THIS_CPU_OPS to arch/x86/Kconfig, which will cause NEED_SRCU_NMI_SAFE to be deselected, thus preserving the current srcu_read_lock() behavior. Link: https://lore.kernel.org/all/20220910221947.171557773@linutronix.de/ Suggested-by: Neeraj Upadhyay Suggested-by: Frederic Weisbecker Suggested-by: Boqun Feng Signed-off-by: Paul E. McKenney Cc: Huacai Chen Cc: WANG Xuerui Cc: Thomas Gleixner Cc: John Ogness Cc: Petr Mladek Cc: --- arch/loongarch/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index 903096bd87f88..386adde2feffb 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -10,6 +10,7 @@ config LOONGARCH select ARCH_ENABLE_MEMORY_HOTPLUG select ARCH_ENABLE_MEMORY_HOTREMOVE select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI + select ARCH_HAS_NMI_SAFE_THIS_CPU_OPS select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST select ARCH_INLINE_READ_LOCK if !PREEMPTION From patchwork Wed Oct 19 22:58:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 5910 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp574033wrs; Wed, 19 Oct 2022 16:00:00 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5GpPnT9fHV+qc48kByMQ8Ppnf5ePdcBLdp2ixbfeZBIQiWhnzFtnRirDtepsBRob4ohplX X-Received: by 2002:a17:907:7f0b:b0:78d:c16e:e5fc with SMTP id qf11-20020a1709077f0b00b0078dc16ee5fcmr8795342ejc.713.1666220400226; Wed, 19 Oct 2022 16:00:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666220400; cv=none; d=google.com; s=arc-20160816; b=iM0s0Jiku5MVXO5gWKzaopteuIR7KadVuycdb6+SQo6hIMoQgWi8JaZxJt4HxSs6Jz eLDasc/9i4QAdFOmkPS+X+pIqXpmS2HQ9VIKxHOV3sPcFrPliR0SGPjJIWqEMsGqepTH RmMGCVNLBQGb8DyUFMUJMeS242yZNNA+5499awXN9pwi52Q1cdIDQCNaQVBdoKPulbLJ FODXUucV6iPE6vqV4KqAxiWT5JUd6IRFEs4c4S2WF2VstjMq2TwqCW2G46HExzgwCG1A CMALVWZikASX5loMzoKO44r79AABi4veLAVeCv+vgL521kffMs89efZbS6PoJZUeZ/gS 8hZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ocPTlUmllcXNl5vpULlQnXC9rXwFmIY7TCJ7zxkqIX8=; b=0w59FmxklHdq/1a0P3BPB8eQXiOpM8tPr0BTurwGcKWk+mwSGZzbByyy1xX7G/Pa9p NCuZ3rtdpmmA1ALYermLTQQeXEUsczmYgmQRk30btKEWO5AGRyn4FPfKmmEK/pxVDfHg 0KKcwAAiqL1TLBW1MKsUVUtWEUWyj9FkA7mcInBcTuGJnkaCXADrfbNM7W62xcnM13ka 4UnN9gSnG76CAdpH1LSY2aTNaLv3D/k77wx3EtTaoPStfOuOm3tJzI9jcR6gJaHiCr7F 6vGGmbwFY/BAZ0D8Q09qtZO3gX3Bpql6K77GA0NO77LUJM2YMrSsj7I/mhWupisGCZbN 4+TQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=BNcVR0ah; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t4-20020a508d44000000b004574154f09asi14721552edt.529.2022.10.19.15.59.35; Wed, 19 Oct 2022 16:00:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=BNcVR0ah; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231559AbiJSW7G (ORCPT + 99 others); Wed, 19 Oct 2022 18:59:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44490 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231190AbiJSW6w (ORCPT ); Wed, 19 Oct 2022 18:58:52 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E8DA21CBA82; Wed, 19 Oct 2022 15:58:49 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2E89A619D2; Wed, 19 Oct 2022 22:58:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4F438C4315D; Wed, 19 Oct 2022 22:58:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666220328; bh=FsGsTqRYffVxY8Tir1OmIHFlULaz8yaYl41bZtGWV/s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BNcVR0ahlFzw6b+PSF6Ag1ZOkRdVOoCNxNeLVlSqdB/xb/QBx8Oy05CFGtm+ymepY F9Cb8e1kDXGYBFOxmUEN6DEgTvqjf+3IfS4EILZ0xl0y3r3Gj/x9haeO6Q6nJDbBeB bFBfxwtOkN2KJdKDNTG0L0tJnXBHYu6mDamjbZUF4+tIi6+IoiJQm2f/XwMrR1G8aa EQN/ByNpoPWbaepW39zGL/TdsoURAJTRs5UBfWyA9S9ldgHCwLPRlTV/iyVPZgbVfQ TYj4iIDGYDRyu0ynwgbxmaHVm+eLqJF5y3Usbs4c9sPmncdGU+nyqWtVjl69OeNcFH C51yosxbFgh9w== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id B99C05C0B8F; Wed, 19 Oct 2022 15:58:47 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, "Paul E. McKenney" , Neeraj Upadhyay , Frederic Weisbecker , Boqun Feng , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Thomas Gleixner , John Ogness , Petr Mladek , linux-s390@vger.kernel.org Subject: [PATCH v3 rcu 08/11] arch/s390: Add ARCH_HAS_NMI_SAFE_THIS_CPU_OPS Kconfig option Date: Wed, 19 Oct 2022 15:58:43 -0700 Message-Id: <20221019225846.2501109-8-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> References: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747158722795554636?= X-GMAIL-MSGID: =?utf-8?q?1747158722795554636?= The s390 architecture uses either a cmpxchg loop (old systems) or the laa add-to-memory instruction (new systems) to implement this_cpu_add(), both of which are NMI safe. This means that the old and more-efficient srcu_read_lock() may be used in NMI context, without the need for srcu_read_lock_nmisafe(). Therefore, add the new Kconfig option ARCH_HAS_NMI_SAFE_THIS_CPU_OPS to arch/arm64/Kconfig, which will cause NEED_SRCU_NMI_SAFE to be deselected, thus preserving the current srcu_read_lock() behavior. Link: https://lore.kernel.org/all/20220910221947.171557773@linutronix.de/ Suggested-by: Neeraj Upadhyay Suggested-by: Frederic Weisbecker Suggested-by: Boqun Feng Signed-off-by: Paul E. McKenney Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Alexander Gordeev Cc: Christian Borntraeger Cc: Sven Schnelle Cc: Thomas Gleixner Cc: John Ogness Cc: Petr Mladek Cc: Acked-by: Heiko Carstens --- arch/s390/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index 318fce77601d3..0acdfda332908 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -73,6 +73,7 @@ config S390 select ARCH_HAS_GIGANTIC_PAGE select ARCH_HAS_KCOV select ARCH_HAS_MEM_ENCRYPT + select ARCH_HAS_NMI_SAFE_THIS_CPU_OPS select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_SCALED_CPUTIME select ARCH_HAS_SET_MEMORY From patchwork Wed Oct 19 22:58:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 5907 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp573977wrs; Wed, 19 Oct 2022 15:59:51 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5vzThptQM/zrxJwDgHst2OYdnoXN3z9j2ub54RG8GPutwN5qAOthgaubwomqcQnn0XjdEs X-Received: by 2002:a17:907:1dcf:b0:78d:8afc:bc9f with SMTP id og15-20020a1709071dcf00b0078d8afcbc9fmr8479219ejc.518.1666220391093; Wed, 19 Oct 2022 15:59:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666220391; cv=none; d=google.com; s=arc-20160816; b=NgUIAP+9Z8VjhMQvfxy/czjO6+rNY2jw+DkX+8RiyEVtP4RTziJMeYJn6xnZBrRpj5 7tkYTG1YOYLr7dU2nysz5K7+UkGclAJ/qDmxliWyABSIYoC7EapDn2OsU915eXJTMgLN IAJNBGcuUPYqQguYGAvO1uEkdwubqlfPwA0eZTfXZ4tEVvNE9WP7MhTXZ0rnpFN9hgd/ lfTp8lgcKU7GHxYvgNKt0GaahWkbvxG/CiBS/r2pFZKSB+uFFNO5ifOpm4KimuBqHPxv 7d0RDFP6k2vzBXbQK7syaWH9xHN2Nbpi5hvheQYl0wLQ0ZJVo8fYMYVEFNp90gATcEr9 qnGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=msKO2Qnqow59JhV6NbNITt2ZMC1PJB69eLmjXFHEv8Y=; b=jXpyUE8+y35t6hnU5knJKqZhpeacI+9GVTy+pBuis3Hkv0zmild7YZ+2H/x3VhgW3S TwYpMUGRYlSPORjru+ana33TGqChZGPsWkteldXF25UIqXFjDNSrg+Q0ni2ecxrDvyyq XuUDaTBraJZpsm3PthiYXF4YjBb2Hf05KvxysPtvKowvg/LyVql3CamZVcIpkYEEGBwd mHWAjXDq0IwXgES/EYBKZS4BNXGE4AMxJ6vD1KCGu8qraev75aY4QjRUeUYtYj1AcgZz wMjfdXC974Pmj7gazjic5F8e4jQ80wqKFRfTOKjxaswe0OCK8s+gqE0ffVAZjiNqzvX/ M+9A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=tBBT+O+t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q13-20020a056402518d00b0045e00384e8asi5442791edd.380.2022.10.19.15.59.26; Wed, 19 Oct 2022 15:59:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=tBBT+O+t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231424AbiJSW66 (ORCPT + 99 others); Wed, 19 Oct 2022 18:58:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44492 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231206AbiJSW6w (ORCPT ); Wed, 19 Oct 2022 18:58:52 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 11DC31CC741; Wed, 19 Oct 2022 15:58:49 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 32619619D6; Wed, 19 Oct 2022 22:58:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 53B38C4315E; Wed, 19 Oct 2022 22:58:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666220328; bh=vcOf9CMrU+6NN6qjpATXbQFTM1BXdbCbv7vFEbo3qjM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tBBT+O+tdFhyraHK1mTg82F6rvkz9Afq3DB3uSS+If39hRs3+6Rwd6Ixl1N/VNOol DEBxL+oTy7c6xrpF/vTdn1VBUBX2MqOCLp91dt3myP+U6qlEBcwaW4QnhOMiBMum/c sWHM+yE/NMFPt6J6boNnvxQxbBD1bwuyubIAwqTl6dpze6T3hOxAyOHXVHHGerKGa0 HRgPT4SGikXeEoUrsl8yuwJO8hzvrTNbn7xP9ItcNCgWeEGIWzV3wjEJa4iuV1s34n x7pAKLdDdeWMywbF12fYUNc9iGL7wV0fd47O7rSQsvkPOOzwAfOPFNYJXxMwor35oh VEIihypUyELbA== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id BB57A5C0BE8; Wed, 19 Oct 2022 15:58:47 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH v3 rcu 09/11] srcu: Warn when NMI-unsafe API is used in NMI Date: Wed, 19 Oct 2022 15:58:44 -0700 Message-Id: <20221019225846.2501109-9-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> References: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747158712967584966?= X-GMAIL-MSGID: =?utf-8?q?1747158712967584966?= From: Frederic Weisbecker Using the NMI-unsafe reader API from within an NMI handler is very likely to be buggy for three reasons: 1) NMIs aren't strictly re-entrant (a pending nested NMI will execute at the end of the current one) so it should be fine to use a non-atomic increment here. However, breakpoints can still interrupt NMIs and if a breakpoint callback has a reader on that same ssp, a racy increment can happen. 2) If the only reader site for a given srcu_struct structure is in an NMI handler, then RCU should be used instead of SRCU. 3) Because of the previous reason (2), an srcu_struct structure having an SRCU read side critical section in an NMI handler is likely to have another one from a task context. For all these reasons, warn if an NMI-unsafe reader API is used from an NMI handler. Signed-off-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 87ae6f5c1edae..18bb696cff8ca 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -642,6 +642,8 @@ static void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe) if (!IS_ENABLED(CONFIG_PROVE_RCU)) return; + /* NMI-unsafe use in NMI is a bad sign */ + WARN_ON_ONCE(!nmi_safe && in_nmi()); sdp = raw_cpu_ptr(ssp->sda); old_nmi_safe_mask = READ_ONCE(sdp->srcu_nmi_safety); if (!old_nmi_safe_mask) { From patchwork Wed Oct 19 22:58:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 5914 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp574810wrs; Wed, 19 Oct 2022 16:01:28 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4w4M7jgZKd6x5T+VztufdPyMjea6K+aCaUhpsVBck2AvVc1HnXCJPFqnSGcaJOhcuHSt/n X-Received: by 2002:a05:6402:190f:b0:45d:2c25:3a1d with SMTP id e15-20020a056402190f00b0045d2c253a1dmr9841834edz.175.1666220488203; Wed, 19 Oct 2022 16:01:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666220488; cv=none; d=google.com; s=arc-20160816; b=cCtZKsantiK9CwfdMoN0mg8T3DmnzjzFXFkhdvKsRkQi0OMPI3UPv+FmFMaUZVuKD8 tKU4MTIKpSLQCT5MO5YRJcrhxtUz6U+zaoi5vhvv/GdvdmXiU6gI744xVd4kT7waf4eI NoUCbbtcpG7InmYhV1mao2dbMGzQyRtGDsYxVGSvGUWQxg641wNZQANIypIax09PBXrm YGce6Wlg5SzyYWI0vlZ/+Mb02X29lf6i/XWqoDxPHAxrEn1sW6ywSQB4dfXvIJkwbp/x lG4oOyJp9jAw1C1NIkBv6GkzT1+Sv/F2Cyqt7STz8K1hw48FHE4r3yNs4UN4huaGnH0F TEiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5ABMxQ8PFy2khu6uvyHYp+aMLoQBLJ9EGauB7WZlpCc=; b=waYMswfqCisfFmpvJcz6lJHI6d2EgQK1s+50A35A1lCVO86AahOrqwG4uQbUcLUI8l F0YSds+Mfo98g468Xjwe8cyBejuRE8ppUJB56Y8UWMRtmgVWfF2eRNbvKYw/EqjniDf8 g+uuCWmW09Ah9MfKCRVKcrzdDI4zztx3NDZROUAG44qI7pApMDJRsN6Qd8DDN4kj1zzX JC7T/hFWlwMtum6vfZj82YpMt50RxbiEjzwH75UUcuYNGI70pgSIGurXv0H0q5pT1fMS KaWgCWszEwF+G6lCEsLiAxXjoTnF08AMs0MEiskAf+fVSncfoqUSFoE/wqx1kyYxvNwz rcsw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=rrgNEcCv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m2-20020a1709062ac200b0078da856a4ecsi12238263eje.391.2022.10.19.16.01.01; Wed, 19 Oct 2022 16:01:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=rrgNEcCv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231294AbiJSW71 (ORCPT + 99 others); Wed, 19 Oct 2022 18:59:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44510 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231375AbiJSW6z (ORCPT ); Wed, 19 Oct 2022 18:58:55 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9D83A1CD688; Wed, 19 Oct 2022 15:58:52 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 7DB39B82624; Wed, 19 Oct 2022 22:58:50 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 59E48C4315B; Wed, 19 Oct 2022 22:58:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666220328; bh=Lw7Q0GwSzhjX27kJ3r/Pu5VkOmW/ZLbkQEvvvISFP7I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rrgNEcCvnt9q2Ei1GCPn0fArdrA82frgyah4oWktNFVTH01HA4b2GnkskY46J8peD 3AzWfEUiIuOHcksy1hvxYjs5auNbV0LjN2jRF7FgONsTlUYJ0yF5FPygzUThbcn/ks 5ngY96d2lwGndXGz5zEeYU5a1RnZk3ZkR0+MiAwZpgw/duszA+iyiaiyoXcGEPyBak tIAES6BW4ZKPKhINAaCaSllhJBB+B2+eYyjjFda7Ubsa/haPD+unNMQQKX+QtBP7v8 2ReYkEMsdLQBjt7RIVd9D2jqojVRVRuXEhk7qga4dvKR1vKaVdRbCs+uGCDGb9Td6p YUwBPwuuRZqoQ== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id BD0FC5C0D2B; Wed, 19 Oct 2022 15:58:47 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH v3 rcu 10/11] srcu: Explain the reason behind the read side critical section on GP start Date: Wed, 19 Oct 2022 15:58:45 -0700 Message-Id: <20221019225846.2501109-10-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> References: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747158814934961199?= X-GMAIL-MSGID: =?utf-8?q?1747158814934961199?= From: Frederic Weisbecker Tell about the need to protect against concurrent updaters who may overflow the GP counter behind the current update. Reported-by: Paul E. McKenney Signed-off-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- kernel/rcu/srcutree.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 18bb696cff8ca..272830a87e566 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1158,6 +1158,11 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp, int ss_state; check_init_srcu_struct(ssp); + /* + * While starting a new grace period, make sure we are in an + * SRCU read-side critical section so that the grace-period + * sequence number cannot wrap around in the meantime. + */ idx = __srcu_read_lock_nmisafe(ssp, false); ss_state = smp_load_acquire(&ssp->srcu_size_state); if (ss_state < SRCU_SIZE_WAIT_CALL) From patchwork Wed Oct 19 22:58:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 5911 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp574180wrs; Wed, 19 Oct 2022 16:00:17 -0700 (PDT) X-Google-Smtp-Source: AMsMyM59mqvNiMNOS07Ra2PQhWk32XIU7ZsyKIisYVt+hepKOa6x1JAMCaMdOYOKf7K/fFxI2mmv X-Received: by 2002:a17:906:3852:b0:78d:b3d2:97a9 with SMTP id w18-20020a170906385200b0078db3d297a9mr8307225ejc.565.1666220416919; Wed, 19 Oct 2022 16:00:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666220416; cv=none; d=google.com; s=arc-20160816; b=Aptqf0CDJR2rWBW+j7xcE/UGIkhZ3l2werRT2APkjERDvLVBb03iHH7zvXzGJEbCH1 Mi6dRabNylGD1gtCmG01LT+Qi1cYQNQH+ZiHbj0AXGNaxu3p4szsEcNysmwN/UfouX5X 1u7+GC81iYx6ClrCsXeRQVmu3cveGWN2aCZzZnTSKXSKHZtmYaeSuhjX8F3SQb/uxHE+ odEMpoupG8M3jEOqbCpF4R5KP65SYYNEC5ecYkBnTR399VO4SKp/XfYy2EAJFobeCsFK v5rpab4lwQA4amhBPj06PKfvwSx0ic0G+kdl7LgFOdFvHGOs/Y40zMpobn+1DkSpDpK2 BT2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=6IrD6+esoaXHdR+2Vcns6GlKr8lG7SnOc0L5ImkKbU0=; b=AgDzFgZWa1lwg3AhdK/h0DK35uM4sxaowsh4jm+xukUHJW8Du6mDm2+y/bkFLuOScs h50Ryz1jur9eR3hIvcOLQZV/3AtI5suNBNXFOVHUb/NLmWhxsUR0SjqMBpecoaIgrT3V o4LLCf3Eo1GQfOSnc2bGpnix0Wp3SThp4MvcePEtLUh43W8A96fmb6tnijHN6aQXx1sd f18Pcwu4HS5HSixEMXKYQ5elNiQvGsabZSTZAcMsSC2Vpopw+32fbudqkz86oMbvB1C5 u3Jjo1Dp5d7AxlvY0w0c3pCOLNvC2MUcCQ+hZdAAwiVpxN+IVzQqW1IN5yivADZdPgt3 uKaw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=XdE4x2ZJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hr2-20020a1709073f8200b007919b3ad78esi6208980ejc.495.2022.10.19.15.59.26; Wed, 19 Oct 2022 16:00:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=XdE4x2ZJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231497AbiJSW7C (ORCPT + 99 others); Wed, 19 Oct 2022 18:59:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44496 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231244AbiJSW6w (ORCPT ); Wed, 19 Oct 2022 18:58:52 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 140221CCCEE; Wed, 19 Oct 2022 15:58:50 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5CFF661962; Wed, 19 Oct 2022 22:58:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 53C47C4315F; Wed, 19 Oct 2022 22:58:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666220328; bh=UBuj76kHE7Oii/4AoWLoto3e/2iEtgpEdWdwXnDVmFQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XdE4x2ZJHM7aJD5PUYc0rXTv6uWHLp+BmrOXwgZ/n795GNKG82V1omLedMJT1KxNB 5hoLRNWWWs3f1jubTVC13kelpwCZWqdWhy6qJn/xf28/L79Awu7xQbUndIWsHEN7sW joJn7gFP+lVxKMeRx+SOLREoRFegwJ/JsWjjl1D1CdPgqHpQZJOoII6TtyuE5ANasw TD2HurdcYQDgGua0p1LaKqJuJo1G35agZD3Dpp2DrMKPuJKhZofYkvgdFbCkD2VUwp EOJKtHDCqgCn0m+/oLE7KGU7E4fNyqmRqSZu0GATBLwFfCiaOgXOEXFloHYoRzo8bH dQd8KEm9c5XTw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id BEF155C0DA6; Wed, 19 Oct 2022 15:58:47 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@fb.com, rostedt@goodmis.org, Frederic Weisbecker , "Paul E . McKenney" Subject: [PATCH v3 rcu 11/11] srcu: Debug NMI safety even on archs that don't require it Date: Wed, 19 Oct 2022 15:58:46 -0700 Message-Id: <20221019225846.2501109-11-paulmck@kernel.org> X-Mailer: git-send-email 2.31.1.189.g2e36527f23 In-Reply-To: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> References: <20221019225838.GA2500612@paulmck-ThinkPad-P17-Gen-1> MIME-Version: 1.0 X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747158739929845780?= X-GMAIL-MSGID: =?utf-8?q?1747158739929845780?= From: Frederic Weisbecker Currently the NMI safety debugging is only performed on architectures that don't support NMI-safe this_cpu_inc(). Reorder the code so that other architectures like x86 also detect bad uses. [ paulmck: Apply kernel test robot, Stephen Rothwell, and Zqiang feedback. ] Signed-off-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney --- include/linux/srcu.h | 44 +++++++++++++++++++++++++++++++--------- include/linux/srcutiny.h | 12 ----------- include/linux/srcutree.h | 7 ------- kernel/rcu/srcutree.c | 25 +++++++++-------------- 4 files changed, 44 insertions(+), 44 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 565f60d574847..f0814ffca34bb 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -52,8 +52,6 @@ int init_srcu_struct(struct srcu_struct *ssp); #else /* Dummy definition for things like notifiers. Actual use gets link error. */ struct srcu_struct { }; -int __srcu_read_lock_nmisafe(struct srcu_struct *ssp, bool chknmisafe) __acquires(ssp); -void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx, bool chknmisafe) __releases(ssp); #endif void call_srcu(struct srcu_struct *ssp, struct rcu_head *head, @@ -66,6 +64,20 @@ unsigned long get_state_synchronize_srcu(struct srcu_struct *ssp); unsigned long start_poll_synchronize_srcu(struct srcu_struct *ssp); bool poll_state_synchronize_srcu(struct srcu_struct *ssp, unsigned long cookie); +#ifdef CONFIG_NEED_SRCU_NMI_SAFE +int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp); +void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) __releases(ssp); +#else +static inline int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) +{ + return __srcu_read_lock(ssp); +} +static inline void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) +{ + __srcu_read_unlock(ssp, idx); +} +#endif /* CONFIG_NEED_SRCU_NMI_SAFE */ + #ifdef CONFIG_SRCU void srcu_init(void); #else /* #ifdef CONFIG_SRCU */ @@ -106,6 +118,18 @@ static inline int srcu_read_lock_held(const struct srcu_struct *ssp) #endif /* #else #ifdef CONFIG_DEBUG_LOCK_ALLOC */ +#define SRCU_NMI_UNKNOWN 0x0 +#define SRCU_NMI_UNSAFE 0x1 +#define SRCU_NMI_SAFE 0x2 + +#if defined(CONFIG_PROVE_RCU) && defined(CONFIG_TREE_SRCU) +void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe); +#else +static inline void srcu_check_nmi_safety(struct srcu_struct *ssp, + bool nmi_safe) { } +#endif + + /** * srcu_dereference_check - fetch SRCU-protected pointer for later dereferencing * @p: the pointer to fetch and protect for later dereferencing @@ -163,6 +187,7 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp) { int retval; + srcu_check_nmi_safety(ssp, false); retval = __srcu_read_lock(ssp); rcu_lock_acquire(&(ssp)->dep_map); return retval; @@ -179,10 +204,8 @@ static inline int srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp { int retval; - if (IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) - retval = __srcu_read_lock_nmisafe(ssp, true); - else - retval = __srcu_read_lock(ssp); + srcu_check_nmi_safety(ssp, true); + retval = __srcu_read_lock_nmisafe(ssp); rcu_lock_acquire(&(ssp)->dep_map); return retval; } @@ -193,6 +216,7 @@ srcu_read_lock_notrace(struct srcu_struct *ssp) __acquires(ssp) { int retval; + srcu_check_nmi_safety(ssp, false); retval = __srcu_read_lock(ssp); return retval; } @@ -208,6 +232,7 @@ static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx) __releases(ssp) { WARN_ON_ONCE(idx & ~0x1); + srcu_check_nmi_safety(ssp, false); rcu_lock_release(&(ssp)->dep_map); __srcu_read_unlock(ssp, idx); } @@ -223,17 +248,16 @@ static inline void srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) __releases(ssp) { WARN_ON_ONCE(idx & ~0x1); + srcu_check_nmi_safety(ssp, true); rcu_lock_release(&(ssp)->dep_map); - if (IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)) - __srcu_read_unlock_nmisafe(ssp, idx, true); - else - __srcu_read_unlock(ssp, idx); + __srcu_read_unlock_nmisafe(ssp, idx); } /* Used by tracing, cannot be traced and cannot call lockdep. */ static inline notrace void srcu_read_unlock_notrace(struct srcu_struct *ssp, int idx) __releases(ssp) { + srcu_check_nmi_safety(ssp, false); __srcu_read_unlock(ssp, idx); } diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index f890301f123df..f3a4d65b91efd 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -89,16 +89,4 @@ static inline void srcu_torture_stats_print(struct srcu_struct *ssp, data_race(READ_ONCE(ssp->srcu_idx)), data_race(READ_ONCE(ssp->srcu_idx_max))); } - -static inline int __srcu_read_lock_nmisafe(struct srcu_struct *ssp, bool chknmisafe) -{ - BUG(); - return 0; -} - -static inline void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx, bool chknmisafe) -{ - BUG(); -} - #endif diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 35ffdedf86ccb..c689a81752c9a 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -43,10 +43,6 @@ struct srcu_data { struct srcu_struct *ssp; }; -#define SRCU_NMI_UNKNOWN 0x0 -#define SRCU_NMI_NMI_UNSAFE 0x1 -#define SRCU_NMI_NMI_SAFE 0x2 - /* * Node in SRCU combining tree, similar in function to rcu_data. */ @@ -159,7 +155,4 @@ void synchronize_srcu_expedited(struct srcu_struct *ssp); void srcu_barrier(struct srcu_struct *ssp); void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf); -int __srcu_read_lock_nmisafe(struct srcu_struct *ssp, bool chknmisafe) __acquires(ssp); -void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx, bool chknmisafe) __releases(ssp); - #endif diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 272830a87e566..ca4b5dcec675b 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -631,17 +631,16 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) } EXPORT_SYMBOL_GPL(cleanup_srcu_struct); +#ifdef CONFIG_PROVE_RCU /* * Check for consistent NMI safety. */ -static void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe) +void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe) { int nmi_safe_mask = 1 << nmi_safe; int old_nmi_safe_mask; struct srcu_data *sdp; - if (!IS_ENABLED(CONFIG_PROVE_RCU)) - return; /* NMI-unsafe use in NMI is a bad sign */ WARN_ON_ONCE(!nmi_safe && in_nmi()); sdp = raw_cpu_ptr(ssp->sda); @@ -652,6 +651,8 @@ static void srcu_check_nmi_safety(struct srcu_struct *ssp, bool nmi_safe) } WARN_ONCE(old_nmi_safe_mask != nmi_safe_mask, "CPU %d old state %d new state %d\n", sdp->cpu, old_nmi_safe_mask, nmi_safe_mask); } +EXPORT_SYMBOL_GPL(srcu_check_nmi_safety); +#endif /* CONFIG_PROVE_RCU */ /* * Counts the new reader in the appropriate per-CPU element of the @@ -665,7 +666,6 @@ int __srcu_read_lock(struct srcu_struct *ssp) idx = READ_ONCE(ssp->srcu_idx) & 0x1; this_cpu_inc(ssp->sda->srcu_lock_count[idx].counter); smp_mb(); /* B */ /* Avoid leaking the critical section. */ - srcu_check_nmi_safety(ssp, false); return idx; } EXPORT_SYMBOL_GPL(__srcu_read_lock); @@ -679,7 +679,6 @@ void __srcu_read_unlock(struct srcu_struct *ssp, int idx) { smp_mb(); /* C */ /* Avoid leaking the critical section. */ this_cpu_inc(ssp->sda->srcu_unlock_count[idx].counter); - srcu_check_nmi_safety(ssp, false); } EXPORT_SYMBOL_GPL(__srcu_read_unlock); @@ -690,7 +689,7 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock); * srcu_struct, but in an NMI-safe manner using RMW atomics. * Returns an index that must be passed to the matching srcu_read_unlock(). */ -int __srcu_read_lock_nmisafe(struct srcu_struct *ssp, bool chknmisafe) +int __srcu_read_lock_nmisafe(struct srcu_struct *ssp) { int idx; struct srcu_data *sdp = raw_cpu_ptr(ssp->sda); @@ -698,8 +697,6 @@ int __srcu_read_lock_nmisafe(struct srcu_struct *ssp, bool chknmisafe) idx = READ_ONCE(ssp->srcu_idx) & 0x1; atomic_long_inc(&sdp->srcu_lock_count[idx]); smp_mb__after_atomic(); /* B */ /* Avoid leaking the critical section. */ - if (chknmisafe) - srcu_check_nmi_safety(ssp, true); return idx; } EXPORT_SYMBOL_GPL(__srcu_read_lock_nmisafe); @@ -709,14 +706,12 @@ EXPORT_SYMBOL_GPL(__srcu_read_lock_nmisafe); * element of the srcu_struct. Note that this may well be a different * CPU than that which was incremented by the corresponding srcu_read_lock(). */ -void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx, bool chknmisafe) +void __srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) { struct srcu_data *sdp = raw_cpu_ptr(ssp->sda); smp_mb__before_atomic(); /* C */ /* Avoid leaking the critical section. */ atomic_long_inc(&sdp->srcu_unlock_count[idx]); - if (chknmisafe) - srcu_check_nmi_safety(ssp, true); } EXPORT_SYMBOL_GPL(__srcu_read_unlock_nmisafe); @@ -1163,7 +1158,7 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp, * SRCU read-side critical section so that the grace-period * sequence number cannot wrap around in the meantime. */ - idx = __srcu_read_lock_nmisafe(ssp, false); + idx = __srcu_read_lock_nmisafe(ssp); ss_state = smp_load_acquire(&ssp->srcu_size_state); if (ss_state < SRCU_SIZE_WAIT_CALL) sdp = per_cpu_ptr(ssp->sda, 0); @@ -1196,7 +1191,7 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp, srcu_funnel_gp_start(ssp, sdp, s, do_norm); else if (needexp) srcu_funnel_exp_start(ssp, sdp_mynode, s); - __srcu_read_unlock_nmisafe(ssp, idx, false); + __srcu_read_unlock_nmisafe(ssp, idx); return s; } @@ -1500,13 +1495,13 @@ void srcu_barrier(struct srcu_struct *ssp) /* Initial count prevents reaching zero until all CBs are posted. */ atomic_set(&ssp->srcu_barrier_cpu_cnt, 1); - idx = __srcu_read_lock_nmisafe(ssp, false); + idx = __srcu_read_lock_nmisafe(ssp); if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, 0)); else for_each_possible_cpu(cpu) srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, cpu)); - __srcu_read_unlock_nmisafe(ssp, idx, false); + __srcu_read_unlock_nmisafe(ssp, idx); /* Remove the initial count, at which point reaching zero can happen. */ if (atomic_dec_and_test(&ssp->srcu_barrier_cpu_cnt))