From patchwork Fri Mar 24 00:19:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74271 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp54548wrt; Thu, 23 Mar 2023 17:20:53 -0700 (PDT) X-Google-Smtp-Source: AKy350ZL0P2XoDiJgE61NE/jvjQPg4VpTPVvyHsGYO554kJ5zH0vq6DEk91qgTV2FwpprDQNKwoE X-Received: by 2002:aa7:c786:0:b0:4fb:eda4:c093 with SMTP id n6-20020aa7c786000000b004fbeda4c093mr1263267eds.13.1679617253687; Thu, 23 Mar 2023 17:20:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679617253; cv=none; d=google.com; s=arc-20160816; b=GPKo9hoBuNJ+hX40TMAdvBkfZLjkMkcFbuciRKom0k3YBjoaMG0q13ob9aLE7lcFd4 Uo3OtVyK8i+2PJ/dLz+c2aaqaM3fJEDk9VauCbaMclSfBoh/veN42UXYsgLrSG3kWF7B p/xvcvcHq8gl8Zv4vBjrEPlVi7zvQcmyUxaGdJhBMIg40zeY4ZxPm8PQaYKuvo9Xy8s6 YruLGwIOBMC4ql9N3faNwLfhKI+nOv4By7/YimZI3b4c/wVhfhZNs/yEDewDYY8mGwIL WP4za25NXy0m0M6B09xJFhexfY+UCio4h3uxycjgS+kYX0Mdf73QtSf1zuFxNQvHzkI2 ZC9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vVEP0vQ/DN83w+OnorEHs0iEkvEk7Oh1zfOMFu6sTuo=; b=TioQahXIqXfEHVro91QJ35GOgCI/Pm5/y9aGQ2ivHaYDVppEN2oGMEy7/ryq3He87H 9sGwKcx4CCf7ESWkiwgSFOGqEi3IKgGNVPso5tKFAPCq+0kiTUe5qbwFuotQPQl8rnRx hBDv7Nj88n1EaIaKUsGkJALCqT2KU30lqpQcfRhyEzDAFgzQ34t7CjTfib7JFF57buV/ hUFCpp0siVa650So6RyuUctJC7aasdZ8Ydpd0ylS7EbFJff35LpI4mSH0ZM9NG7IAh7R LFeKY8faWzeux3UKEPgzcIhJwkti98R/cmVeiUk6Lg7GRr2+8om8GkIOoLT7YttrtDe2 VW/A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=WeG19BfY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w13-20020aa7dccd000000b004c5a8aa02fcsi21068911edu.435.2023.03.23.17.20.30; Thu, 23 Mar 2023 17:20:53 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=WeG19BfY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231173AbjCXATr (ORCPT + 99 others); Thu, 23 Mar 2023 20:19:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57724 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230506AbjCXATq (ORCPT ); Thu, 23 Mar 2023 20:19:46 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 001DC2D177; Thu, 23 Mar 2023 17:19:41 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7679D628E3; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DAE52C433EF; Fri, 24 Mar 2023 00:19:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617180; bh=Dy5gUgxa2Ux36xNFueJhEIxHE/YNXOYB6Npc4f9H5Y8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WeG19BfYL7hBWzqROTU8ovzKbWMk1QCSR90fZ6CwvpocH07kIFC/amEUmPqIs8WkP JojEDVWNOLdSFnDOmq24IrBeFqr8UXorLQHSkCzLwR6lO18gdZI9E4VJSHjRwJwuoL +hNxwZVtbV+IujcutqCpgXwHBEDt2/uHpSaNCXnIEHDsLY6yTYIJgSxK5ZgnqcLM9Y /P+2dQXTb2dzgSzk4AOBjGAcTbC+9x+lk6qWmGpu2Mrgj5m2aCmaVxZfra/oHBAJdS LKqF5yztJokJdJSKVfctop1Xk3Zvlu6JjGJCQltdD7chkmCiInzVc0Qz2FRqdAVBQs Cqmf5lt3s7yRg== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id 891E21540379; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 01/19] srcu: Add whitespace to __SRCU_STRUCT_INIT() & __DEFINE_SRCU() Date: Thu, 23 Mar 2023 17:19:20 -0700 Message-Id: <20230324001938.3443499-1-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-5.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761206341372917462?= X-GMAIL-MSGID: =?utf-8?q?1761206341372917462?= This is a whitespace-only commit with no change in functionality. Its purpose is to prepare for later commits that: (1) Cause statically allocated srcu_struct structures to rely on compile-time initialization and (2) Move fields from the srcu_struct structure to a new srcu_usage structure. Signed-off-by: Paul E. McKenney Cc: Christoph Hellwig --- include/linux/srcutree.h | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index a6910805f9c5..ac8af12f93b3 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -121,13 +121,13 @@ struct srcu_struct { #define SRCU_STATE_SCAN1 1 #define SRCU_STATE_SCAN2 2 -#define __SRCU_STRUCT_INIT(name, pcpu_name) \ -{ \ - .sda = &pcpu_name, \ - .lock = __SPIN_LOCK_UNLOCKED(name.lock), \ - .srcu_gp_seq_needed = -1UL, \ - .work = __DELAYED_WORK_INITIALIZER(name.work, NULL, 0), \ - __SRCU_DEP_MAP_INIT(name) \ +#define __SRCU_STRUCT_INIT(name, pcpu_name) \ +{ \ + .sda = &pcpu_name, \ + .lock = __SPIN_LOCK_UNLOCKED(name.lock), \ + .srcu_gp_seq_needed = -1UL, \ + .work = __DELAYED_WORK_INITIALIZER(name.work, NULL, 0), \ + __SRCU_DEP_MAP_INIT(name) \ } /* @@ -150,15 +150,15 @@ struct srcu_struct { * See include/linux/percpu-defs.h for the rules on per-CPU variables. */ #ifdef MODULE -# define __DEFINE_SRCU(name, is_static) \ - is_static struct srcu_struct name; \ - extern struct srcu_struct * const __srcu_struct_##name; \ - struct srcu_struct * const __srcu_struct_##name \ +# define __DEFINE_SRCU(name, is_static) \ + is_static struct srcu_struct name; \ + extern struct srcu_struct * const __srcu_struct_##name; \ + struct srcu_struct * const __srcu_struct_##name \ __section("___srcu_struct_ptrs") = &name #else -# define __DEFINE_SRCU(name, is_static) \ - static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data); \ - is_static struct srcu_struct name = \ +# define __DEFINE_SRCU(name, is_static) \ + static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data); \ + is_static struct srcu_struct name = \ __SRCU_STRUCT_INIT(name, name##_srcu_data) #endif #define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */) From patchwork Fri Mar 24 00:19:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74272 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp54608wrt; Thu, 23 Mar 2023 17:21:01 -0700 (PDT) X-Google-Smtp-Source: AKy350Ydz365pox8u+gYPQXtvwxEmCxKQX7404zi9i4zMWS7zq9WVXIkZ74zJIdcHKCaDi3wkQZn X-Received: by 2002:a05:6402:5178:b0:4fa:57bf:141a with SMTP id d24-20020a056402517800b004fa57bf141amr1355177ede.32.1679617261186; Thu, 23 Mar 2023 17:21:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679617261; cv=none; d=google.com; s=arc-20160816; b=iuv3s0VO2ZMdpFGjsnc1t2UjUYVtqeo3PbK/CrIVXtla69rp37SIapgZX02e6cxYW5 Eihd9FQspxiONSn+9udJPcTh6zr1tABY7w1OeFyjodygu7PrrtB/icsYw9+zeiA8V8a0 /muU/vvmgDMIAq2jJtBtgCsdp7u4oTqUEcTodqZVOqF5HaYGh2DuTlwfBIGCgmr5Yl8w 7/EsceVRIPcj2vJaLEqj2P5/jtNqKhTmfS38qE0t/0nRuELensjFdV08TCtgTQQKqdqf WVXeepYW5BLGwEFXP9jwtNd1zV0GvzPqExbVCebL8Qf3KZteFkyh+YzJuHLMuPwpKtL9 0MUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=y6amKrIc+PHTg3Vz9MbcQEaZiYCrGoBLPGOIHbvDCrw=; b=YoCuOgjz1IjNi9pinkECYXQT2oYF//6hNvmEoj90LQEO5iqeDG7ZtTaQ7bxtEBOq0z gITMg/knr91gyOCHprSYIOzvUKuUJS5DW1D6UxFQ5VK0+xG3zfJX3g4A+XZhWhzhflt+ Z6udtL8szYm6X6rkLmCiW6syxWGHXuxWW+hmRXvaemIPymP9KEIrUhF1iFIKGUEsCpan VoSAVOJnm4Nj4dbQCvR70ziX86ILpqkXrvLo0VxrDQsOz7cVz57mYYGoxFbqWPMO5/3K r9VGiuPB8pP47ays6GbawQ5sqm80RNNc6cuDmzI4NA4vAjroFD1y8NFFUgDfU9AEQ5Qy A9hg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="nP3/V17k"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m1-20020aa7c481000000b004fedca1007dsi21870140edq.161.2023.03.23.17.20.37; Thu, 23 Mar 2023 17:21:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="nP3/V17k"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231248AbjCXAT4 (ORCPT + 99 others); Thu, 23 Mar 2023 20:19:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57752 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229917AbjCXATq (ORCPT ); Thu, 23 Mar 2023 20:19:46 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1FA8E2DE4A; Thu, 23 Mar 2023 17:19:42 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9660D62920; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ED9A0C433D2; Fri, 24 Mar 2023 00:19:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=NfvU5usTkW6B5h8S2WMilLDlOwDmJEkpL6G4bgxR4N4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nP3/V17kUQWQO1UOGwccGf/h74BiIzZMw0wXVLx8lRLykGQ2lnft1/BUCKGbJ/HlR mBKW5xOJ3+Cca3vQMBS3TtZQ7ePFFNIFk+HKoH/DYvX2QdI1hSUqraHM7DRfET66r3 c+AdrN8d/V8b3kqwfyYeOJL93sqrVOFrVLWsag/KHddUMXDTVCYmAhxlYnmKcuAHuy m7FzAYmSgBc/TMpq4/2lmMKNI1y4W3Knf6utQvm6RBvZHW073radP024xKkPzHTZKL CBReo9bfi1XsuD5DFtXWJ97WAZsYDrlcb/GQ5N80NzoiZtxzPqgu/rVBuCDPZENwNp EqCwPeMoDziGQ== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id 8B7841540338; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 02/19] srcu: Use static init for statically allocated in-module srcu_struct Date: Thu, 23 Mar 2023 17:19:21 -0700 Message-Id: <20230324001938.3443499-2-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-5.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761206349562141117?= X-GMAIL-MSGID: =?utf-8?q?1761206349562141117?= Further shrinking the srcu_struct structure is eased by requiring that in-module srcu_struct structures rely more heavily on static initialization. In particular, this preserves the property that a module-load-time srcu_struct initialization can fail only due to memory-allocation failure of the per-CPU srcu_data structures. It might also slightly improve robustness by keeping the number of memory allocations that must succeed down percpu_alloc() call. This is in preparation for splitting an srcu_usage structure out of the srcu_struct structure. [ paulmck: Fold in qiang1.zhang@intel.com feedback. ] Signed-off-by: Paul E. McKenney Cc: Christoph Hellwig --- include/linux/srcutree.h | 19 ++++++++++++++----- kernel/rcu/srcutree.c | 19 +++++++++++++------ 2 files changed, 27 insertions(+), 11 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index ac8af12f93b3..428480152375 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -121,15 +121,24 @@ struct srcu_struct { #define SRCU_STATE_SCAN1 1 #define SRCU_STATE_SCAN2 2 -#define __SRCU_STRUCT_INIT(name, pcpu_name) \ -{ \ - .sda = &pcpu_name, \ +#define __SRCU_STRUCT_INIT_COMMON(name) \ .lock = __SPIN_LOCK_UNLOCKED(name.lock), \ .srcu_gp_seq_needed = -1UL, \ .work = __DELAYED_WORK_INITIALIZER(name.work, NULL, 0), \ - __SRCU_DEP_MAP_INIT(name) \ + __SRCU_DEP_MAP_INIT(name) + +#define __SRCU_STRUCT_INIT_MODULE(name) \ +{ \ + __SRCU_STRUCT_INIT_COMMON(name) \ } +#define __SRCU_STRUCT_INIT(name, pcpu_name) \ +{ \ + .sda = &pcpu_name, \ + __SRCU_STRUCT_INIT_COMMON(name) \ +} + + /* * Define and initialize a srcu struct at build time. * Do -not- call init_srcu_struct() nor cleanup_srcu_struct() on it. @@ -151,7 +160,7 @@ struct srcu_struct { */ #ifdef MODULE # define __DEFINE_SRCU(name, is_static) \ - is_static struct srcu_struct name; \ + is_static struct srcu_struct name = __SRCU_STRUCT_INIT_MODULE(name); \ extern struct srcu_struct * const __srcu_struct_##name; \ struct srcu_struct * const __srcu_struct_##name \ __section("___srcu_struct_ptrs") = &name diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index cd46fe063e50..7a6d9452a5d0 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1895,13 +1895,14 @@ void __init srcu_init(void) static int srcu_module_coming(struct module *mod) { int i; + struct srcu_struct *ssp; struct srcu_struct **sspp = mod->srcu_struct_ptrs; - int ret; for (i = 0; i < mod->num_srcu_structs; i++) { - ret = init_srcu_struct(*(sspp++)); - if (WARN_ON_ONCE(ret)) - return ret; + ssp = *(sspp++); + ssp->sda = alloc_percpu(struct srcu_data); + if (WARN_ON_ONCE(!ssp->sda)) + return -ENOMEM; } return 0; } @@ -1910,10 +1911,16 @@ static int srcu_module_coming(struct module *mod) static void srcu_module_going(struct module *mod) { int i; + struct srcu_struct *ssp; struct srcu_struct **sspp = mod->srcu_struct_ptrs; - for (i = 0; i < mod->num_srcu_structs; i++) - cleanup_srcu_struct(*(sspp++)); + for (i = 0; i < mod->num_srcu_structs; i++) { + ssp = *(sspp++); + if (!rcu_seq_state(smp_load_acquire(&ssp->srcu_sup->srcu_gp_seq_needed)) && + !WARN_ON_ONCE(!ssp->srcu_sup->sda_is_static)) + cleanup_srcu_struct(ssp); + free_percpu(ssp->sda); + } } /* Handle one module, either coming or going. */ From patchwork Fri Mar 24 00:19:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74275 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp56775wrt; Thu, 23 Mar 2023 17:26:37 -0700 (PDT) X-Google-Smtp-Source: AKy350ZXya/e6HgFtF0T0R3/DhDGp1ScmLv9MwJrs7lGnhaKD/A0ZC6+Z+uw9KT/DjVhOisGGKMh X-Received: by 2002:a17:906:f193:b0:92b:eca6:43fc with SMTP id gs19-20020a170906f19300b0092beca643fcmr859606ejb.64.1679617596785; Thu, 23 Mar 2023 17:26:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679617596; cv=none; d=google.com; s=arc-20160816; b=evOs+oZjIYMRjxiOjsEWRzE1XQaPaROmBw4cebzrBOBwtH5Vllmzgs6KcaXTnsHq+s bXWTtfVWr1iQF97wgrNxHwjTFjBW/eloNG3iA/JLi29s1oyCfqS9ThJ90UC0EFZOHhWC K5whcGmWwEtiaVl+1O8yPWKVS/1KkSilm2w2Q/Gu7Gl/r6mcNz/Y7+VohJCeeD3eEmXF LfM0HU6xm5GalxfuEERICDzj2cbthOghqo0yvPRIZtuw25b3I9J8+QBI1u667IoTynjV 4ID9lC2jG9AXU+Epn+OdND+RZ93+vif88CpEAROeBnfrnD3n7lmvG32mvZAo2MOVot87 XSng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=egL5YRcbqwVZOmHdsTv5SVJdZauQNaLjuhZ5GcoaKTw=; b=dISSMtfPrj2ywNiWHvGZHjJ+ppEVbSrcwb89sXB6r380pW/9mH548ls4MDByBms/Pv ljP2qWojxfONC65yXpxnyDEvyA394mx3PZhT8s/AAXAfNRLxMwiuikQcZcNtaNriZSaq 29I5JbgB9Vk+DSnkIIqjqfmwFlaQXjWfhKH2QCAmdz/VY0WBF9Bbq5duB8SqmgK6li+s xw4gWImhQ8vu6JCQ1HItgZ6a2kz0WA73H2ODxJj0BckM7aUJTz/oyxJE2/bSIXRlW8Pv 60cW4iNUT9tOf8DMzYTS92jBYKCga+H+6TYASXkmBIOxA2pmK+Usg5YYcZvqam71R2Qo OHIA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=pNvyVgFb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ty12-20020a170907c70c00b00939577d1dbbsi10326030ejc.73.2023.03.23.17.26.13; Thu, 23 Mar 2023 17:26:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=pNvyVgFb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231419AbjCXAUD (ORCPT + 99 others); Thu, 23 Mar 2023 20:20:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57776 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230486AbjCXATq (ORCPT ); Thu, 23 Mar 2023 20:19:46 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D5B12DE63; Thu, 23 Mar 2023 17:19:42 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9E01562922; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F3D03C4339B; Fri, 24 Mar 2023 00:19:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=0OKDq/T3sDmYtszQPVaOvqY8WMmnghojks+2Z1U5srw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pNvyVgFbgEukhtTuu4UgE2tFC/2yD4ByiHzvgTYPWU+nb6T6T6EduEkHHNKOTLDgk c3MUo2YA+f6/0i+Zs2+x2AfzvzP0Sh6FiAjOoGGoYE2/aidH6DdRrh5qUL3hWk/jL8 5UbyOY9HwB7Vs6JdwzPOhZ2mRXeuVJGI4MhjN0b7tEn4TF6gLRhfO/h2p7OwLoWpgt DFXuQCHg0NnOucrpCmRwh9r0mrS5ygFjaTjp6RJaQ073sapuzE6nxa+2u7ghSC8ED7 65Dzb0JVsj9Kh1QNFB5kqRJt+ON9jtl13NksyuJIsU7Sq83LrDXibyJstRrUmvk5uO cA0T73/oj1Q/Q== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id 910D81540398; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" , "Rafael J. Wysocki" , =?utf-8?b?TWljaGHFgiBN?= =?utf-8?b?aXJvc8WCYXc=?= , Dmitry Osipenko Subject: [PATCH RFC rcu 03/19] srcu: Begin offloading srcu_struct fields to srcu_update Date: Thu, 23 Mar 2023 17:19:22 -0700 Message-Id: <20230324001938.3443499-3-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_FILL_THIS_FORM_SHORT autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761206700860617259?= X-GMAIL-MSGID: =?utf-8?q?1761206700860617259?= The current srcu_struct structure is on the order of 200 bytes in size (depending on architecture and .config), which is much better than the old-style 26K bytes, but still all too inconvenient when one is trying to achieve good cache locality on a fastpath involving SRCU readers. However, only a few fields in srcu_struct are used by SRCU readers. The remaining fields could be offloaded to a new srcu_update structure, thus shrinking the srcu_struct structure down to a few tens of bytes. This commit begins this noble quest, a quest that is complicated by open-coded initialization of the srcu_struct within the srcu_notifier_head structure. This complication is addressed by updating the srcu_notifier_head structure's open coding, given that there does not appear to be a straightforward way of abstracting that initialization. This commit moves only the ->node pointer to srcu_update. Later commits will move additional fields. [ paulmck: Fold in qiang1.zhang@intel.com's memory-leak fix. ] Link: https://lore.kernel.org/all/20230320055751.4120251-1-qiang1.zhang@intel.com/ Suggested-by: Christoph Hellwig Signed-off-by: Paul E. McKenney Cc: "Rafael J. Wysocki" Cc: "Michał Mirosław" Cc: Dmitry Osipenko Acked-by: Rafael J. Wysocki --- include/linux/notifier.h | 5 ++++- include/linux/srcutiny.h | 6 +++--- include/linux/srcutree.h | 27 ++++++++++++++++++--------- kernel/rcu/rcu.h | 6 ++++-- kernel/rcu/srcutree.c | 28 +++++++++++++++++++--------- 5 files changed, 48 insertions(+), 24 deletions(-) diff --git a/include/linux/notifier.h b/include/linux/notifier.h index aef88c2d1173..2aba75145144 100644 --- a/include/linux/notifier.h +++ b/include/linux/notifier.h @@ -73,6 +73,9 @@ struct raw_notifier_head { struct srcu_notifier_head { struct mutex mutex; +#ifdef CONFIG_TREE_SRCU + struct srcu_usage srcuu; +#endif struct srcu_struct srcu; struct notifier_block __rcu *head; }; @@ -107,7 +110,7 @@ extern void srcu_init_notifier_head(struct srcu_notifier_head *nh); { \ .mutex = __MUTEX_INITIALIZER(name.mutex), \ .head = NULL, \ - .srcu = __SRCU_STRUCT_INIT(name.srcu, pcpu), \ + .srcu = __SRCU_STRUCT_INIT(name.srcu, name.srcuu, pcpu), \ } #define ATOMIC_NOTIFIER_HEAD(name) \ diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 5aa5e0faf6a1..ebd72491af99 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -31,7 +31,7 @@ struct srcu_struct { void srcu_drive_gp(struct work_struct *wp); -#define __SRCU_STRUCT_INIT(name, __ignored) \ +#define __SRCU_STRUCT_INIT(name, __ignored, ___ignored) \ { \ .srcu_wq = __SWAIT_QUEUE_HEAD_INITIALIZER(name.srcu_wq), \ .srcu_cb_tail = &name.srcu_cb_head, \ @@ -44,9 +44,9 @@ void srcu_drive_gp(struct work_struct *wp); * Tree SRCU, which needs some per-CPU data. */ #define DEFINE_SRCU(name) \ - struct srcu_struct name = __SRCU_STRUCT_INIT(name, name) + struct srcu_struct name = __SRCU_STRUCT_INIT(name, name, name) #define DEFINE_STATIC_SRCU(name) \ - static struct srcu_struct name = __SRCU_STRUCT_INIT(name, name) + static struct srcu_struct name = __SRCU_STRUCT_INIT(name, name, name) void synchronize_srcu(struct srcu_struct *ssp); diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 428480152375..2689e64024bb 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -57,11 +57,17 @@ struct srcu_node { int grphi; /* Biggest CPU for node. */ }; +/* + * Per-SRCU-domain structure, update-side data linked from srcu_struct. + */ +struct srcu_usage { + struct srcu_node *node; /* Combining tree. */ +}; + /* * Per-SRCU-domain structure, similar in function to rcu_state. */ struct srcu_struct { - struct srcu_node *node; /* Combining tree. */ struct srcu_node *level[RCU_NUM_LVLS + 1]; /* First node at each level. */ int srcu_size_state; /* Small-to-big transition state. */ @@ -90,6 +96,7 @@ struct srcu_struct { unsigned long reschedule_count; struct delayed_work work; struct lockdep_map dep_map; + struct srcu_usage *srcu_sup; /* Update-side data. */ }; // Values for size state variable (->srcu_size_state). Once the state @@ -121,24 +128,24 @@ struct srcu_struct { #define SRCU_STATE_SCAN1 1 #define SRCU_STATE_SCAN2 2 -#define __SRCU_STRUCT_INIT_COMMON(name) \ +#define __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ .lock = __SPIN_LOCK_UNLOCKED(name.lock), \ .srcu_gp_seq_needed = -1UL, \ .work = __DELAYED_WORK_INITIALIZER(name.work, NULL, 0), \ + .srcu_sup = &usage_name, \ __SRCU_DEP_MAP_INIT(name) -#define __SRCU_STRUCT_INIT_MODULE(name) \ +#define __SRCU_STRUCT_INIT_MODULE(name, usage_name) \ { \ - __SRCU_STRUCT_INIT_COMMON(name) \ + __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ } -#define __SRCU_STRUCT_INIT(name, pcpu_name) \ +#define __SRCU_STRUCT_INIT(name, usage_name, pcpu_name) \ { \ .sda = &pcpu_name, \ - __SRCU_STRUCT_INIT_COMMON(name) \ + __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ } - /* * Define and initialize a srcu struct at build time. * Do -not- call init_srcu_struct() nor cleanup_srcu_struct() on it. @@ -160,15 +167,17 @@ struct srcu_struct { */ #ifdef MODULE # define __DEFINE_SRCU(name, is_static) \ - is_static struct srcu_struct name = __SRCU_STRUCT_INIT_MODULE(name); \ + static struct srcu_usage name##_srcu_usage; \ + is_static struct srcu_struct name = __SRCU_STRUCT_INIT_MODULE(name, name##_srcu_usage); \ extern struct srcu_struct * const __srcu_struct_##name; \ struct srcu_struct * const __srcu_struct_##name \ __section("___srcu_struct_ptrs") = &name #else # define __DEFINE_SRCU(name, is_static) \ static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data); \ + static struct srcu_usage name##_srcu_usage; \ is_static struct srcu_struct name = \ - __SRCU_STRUCT_INIT(name, name##_srcu_data) + __SRCU_STRUCT_INIT(name, name##_srcu_usage, name##_srcu_data) #endif #define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */) #define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static) diff --git a/kernel/rcu/rcu.h b/kernel/rcu/rcu.h index a3adcf9a9919..4a1b9622598b 100644 --- a/kernel/rcu/rcu.h +++ b/kernel/rcu/rcu.h @@ -378,11 +378,13 @@ extern void rcu_init_geometry(void); * specified state structure (for SRCU) or the only rcu_state structure * (for RCU). */ -#define srcu_for_each_node_breadth_first(sp, rnp) \ +#define _rcu_for_each_node_breadth_first(sp, rnp) \ for ((rnp) = &(sp)->node[0]; \ (rnp) < &(sp)->node[rcu_num_nodes]; (rnp)++) #define rcu_for_each_node_breadth_first(rnp) \ - srcu_for_each_node_breadth_first(&rcu_state, rnp) + _rcu_for_each_node_breadth_first(&rcu_state, rnp) +#define srcu_for_each_node_breadth_first(ssp, rnp) \ + _rcu_for_each_node_breadth_first(ssp->srcu_sup, rnp) /* * Scan the leaves of the rcu_node hierarchy for the rcu_state structure. diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 7a6d9452a5d0..ad1d5ca42a99 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -173,12 +173,12 @@ static bool init_srcu_struct_nodes(struct srcu_struct *ssp, gfp_t gfp_flags) /* Initialize geometry if it has not already been initialized. */ rcu_init_geometry(); - ssp->node = kcalloc(rcu_num_nodes, sizeof(*ssp->node), gfp_flags); - if (!ssp->node) + ssp->srcu_sup->node = kcalloc(rcu_num_nodes, sizeof(*ssp->srcu_sup->node), gfp_flags); + if (!ssp->srcu_sup->node) return false; /* Work out the overall tree geometry. */ - ssp->level[0] = &ssp->node[0]; + ssp->level[0] = &ssp->srcu_sup->node[0]; for (i = 1; i < rcu_num_lvls; i++) ssp->level[i] = ssp->level[i - 1] + num_rcu_lvl[i - 1]; rcu_init_levelspread(levelspread, num_rcu_lvl); @@ -195,7 +195,7 @@ static bool init_srcu_struct_nodes(struct srcu_struct *ssp, gfp_t gfp_flags) snp->srcu_gp_seq_needed_exp = SRCU_SNP_INIT_SEQ; snp->grplo = -1; snp->grphi = -1; - if (snp == &ssp->node[0]) { + if (snp == &ssp->srcu_sup->node[0]) { /* Root node, special case. */ snp->srcu_parent = NULL; continue; @@ -236,8 +236,12 @@ static bool init_srcu_struct_nodes(struct srcu_struct *ssp, gfp_t gfp_flags) */ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) { + if (!is_static) + ssp->srcu_sup = kzalloc(sizeof(*ssp->srcu_sup), GFP_KERNEL); + if (!ssp->srcu_sup) + return -ENOMEM; ssp->srcu_size_state = SRCU_SIZE_SMALL; - ssp->node = NULL; + ssp->srcu_sup->node = NULL; mutex_init(&ssp->srcu_cb_mutex); mutex_init(&ssp->srcu_gp_mutex); ssp->srcu_idx = 0; @@ -249,8 +253,11 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) ssp->sda_is_static = is_static; if (!is_static) ssp->sda = alloc_percpu(struct srcu_data); - if (!ssp->sda) + if (!ssp->sda) { + if (!is_static) + kfree(ssp->srcu_sup); return -ENOMEM; + } init_srcu_struct_data(ssp); ssp->srcu_gp_seq_needed_exp = 0; ssp->srcu_last_gp_end = ktime_get_mono_fast_ns(); @@ -259,6 +266,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) if (!ssp->sda_is_static) { free_percpu(ssp->sda); ssp->sda = NULL; + kfree(ssp->srcu_sup); return -ENOMEM; } } else { @@ -656,13 +664,15 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) rcu_seq_current(&ssp->srcu_gp_seq), ssp->srcu_gp_seq_needed); return; /* Caller forgot to stop doing call_srcu()? */ } + kfree(ssp->srcu_sup->node); + ssp->srcu_sup->node = NULL; + ssp->srcu_size_state = SRCU_SIZE_SMALL; if (!ssp->sda_is_static) { free_percpu(ssp->sda); ssp->sda = NULL; + kfree(ssp->srcu_sup); + ssp->srcu_sup = NULL; } - kfree(ssp->node); - ssp->node = NULL; - ssp->srcu_size_state = SRCU_SIZE_SMALL; } EXPORT_SYMBOL_GPL(cleanup_srcu_struct); From patchwork Fri Mar 24 00:19:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74290 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp65803wrt; Thu, 23 Mar 2023 17:52:43 -0700 (PDT) X-Google-Smtp-Source: AKy350YJlXGvTgbeNK6TXUBYDhKSpFC7MKcky9nMaI2lr2OR1ueoMQK75iqdQvdSHLaiYdKdrkdJ X-Received: by 2002:a17:906:d7a2:b0:92a:f756:28fa with SMTP id pk2-20020a170906d7a200b0092af75628famr945496ejb.31.1679619162778; Thu, 23 Mar 2023 17:52:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679619162; cv=none; d=google.com; s=arc-20160816; b=RO5mhHu1NrzVItvDCP3SJ48el0W84f8m8uKJ6/er04tLQ1y9slqDu8OOsWrCMO2mLi n9AavFb29diO3pP5U88GHVbE3hlzS6SeUMjH6h1gyCP7chmqUwGLUUVafXlQYFrGqXkV 00CgtqdzQ3DQW32+AEVhj6ekzv32gSSBEVrSCTYDix3KwTGikstyRCb2oU5/Scs1WTJ4 grBjCmIXWbuD8v3BCaYpA/Su+HeTk49/ZGhyAzGQp9lPyBF1XJFphXDJ5GKWrPX4iS3W fYjWFjgZktOY0Mwmv3GaYWbsyJcu4bx4EIqeNeF3zdtZbqiYwy3GkcnBXFCp+13wsViH 8H+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Be339qI3Bq3vB8OsLaeGPTLF5B/nrT83vuWY6kGPi+g=; b=s/Obx/I2bBDXlCaWJJ/Um1GSyqvIheBu5q47QueDNxpJiG/9lJhsmpeZpFjWQ9HtLt qRxBGzTxn2rjgD4PyFcAPiZS2RMjb82F9F5+7YEUggM1R3bSbn50UZkAHRY/CohLW6/F sa01x/mWDK1j1XbCwhxAMFzAmcHrfqmrL9Y7mJufoHEakMeBRjKSgxM0ajg8oGyAScEH 2Q5QNi9s/veyBCHB8b6ruE2g6+UGKWVwujBQxgXoNp5O3xE1ECF6f3lJaaEYilOPRAcR VtVNUgSmqG+2qsX9jcGAYWdXi0LyjJ75wGd6Z9TUdrt2yz2II5lWhzR9eFEWbqpgWYh3 ajgQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=r0K3ytQW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gb4-20020a170907960400b00939adedc8e7si8819365ejc.226.2023.03.23.17.52.19; Thu, 23 Mar 2023 17:52:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=r0K3ytQW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231442AbjCXAUL (ORCPT + 99 others); Thu, 23 Mar 2023 20:20:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231241AbjCXATu (ORCPT ); Thu, 23 Mar 2023 20:19:50 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDECA2C66C; Thu, 23 Mar 2023 17:19:43 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 66EAAB82245; Fri, 24 Mar 2023 00:19:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0FBF4C4339C; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=l31ZVxzvsf5Mo/krVC1TzBNoZigl26axBa/w/xOsuuY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=r0K3ytQWk89vnjC3zZ2d7/NNM+0eQFQpeAuvXn1ohJlVwm75gD9staupAQosnD0cD BtCRxT1fsTXKJYttuZUS9by6VyUSXd2lDUDZkiG8eeXgCPDftTzjxPDNMtt/+O2bVr 14iki5/0ubn/M67CJIFC8VMafg5XIqX6MRBJcSMRYE6Epjlg25w0ZHhPljSVpNA+te bL2coV5i0P2WO3jJcb9/3lJyO5EIJXT74zeXKOBbB8VtlwT/BOZo4x9bGN/RnoQyub KtwyxYywUx7LRaRCebxaGP1iGYW+52qmpN9QrSgBB4txO+UfAGVZ5Ojet139G+fUqJ 9QFHLrS1ET53g== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id 95153154039D; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 04/19] srcu: Move ->level from srcu_struct to srcu_usage Date: Thu, 23 Mar 2023 17:19:23 -0700 Message-Id: <20230324001938.3443499-4-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-5.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761208343255680123?= X-GMAIL-MSGID: =?utf-8?q?1761208343255680123?= This commit moves the ->level[] array from the srcu_struct structure to the srcu_usage structure to reduce the size of the former in order to improve cache locality. Suggested-by: Christoph Hellwig Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 4 ++-- kernel/rcu/srcutree.c | 14 +++++++------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 2689e64024bb..362c8f39c53d 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -62,14 +62,14 @@ struct srcu_node { */ struct srcu_usage { struct srcu_node *node; /* Combining tree. */ + struct srcu_node *level[RCU_NUM_LVLS + 1]; + /* First node at each level. */ }; /* * Per-SRCU-domain structure, similar in function to rcu_state. */ struct srcu_struct { - struct srcu_node *level[RCU_NUM_LVLS + 1]; - /* First node at each level. */ int srcu_size_state; /* Small-to-big transition state. */ struct mutex srcu_cb_mutex; /* Serialize CB preparation. */ spinlock_t __private lock; /* Protect counters and size state. */ diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index ad1d5ca42a99..90d753e10e33 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -178,9 +178,9 @@ static bool init_srcu_struct_nodes(struct srcu_struct *ssp, gfp_t gfp_flags) return false; /* Work out the overall tree geometry. */ - ssp->level[0] = &ssp->srcu_sup->node[0]; + ssp->srcu_sup->level[0] = &ssp->srcu_sup->node[0]; for (i = 1; i < rcu_num_lvls; i++) - ssp->level[i] = ssp->level[i - 1] + num_rcu_lvl[i - 1]; + ssp->srcu_sup->level[i] = ssp->srcu_sup->level[i - 1] + num_rcu_lvl[i - 1]; rcu_init_levelspread(levelspread, num_rcu_lvl); /* Each pass through this loop initializes one srcu_node structure. */ @@ -202,10 +202,10 @@ static bool init_srcu_struct_nodes(struct srcu_struct *ssp, gfp_t gfp_flags) } /* Non-root node. */ - if (snp == ssp->level[level + 1]) + if (snp == ssp->srcu_sup->level[level + 1]) level++; - snp->srcu_parent = ssp->level[level - 1] + - (snp - ssp->level[level]) / + snp->srcu_parent = ssp->srcu_sup->level[level - 1] + + (snp - ssp->srcu_sup->level[level]) / levelspread[level - 1]; } @@ -214,7 +214,7 @@ static bool init_srcu_struct_nodes(struct srcu_struct *ssp, gfp_t gfp_flags) * leaves of the srcu_node tree. */ level = rcu_num_lvls - 1; - snp_first = ssp->level[level]; + snp_first = ssp->srcu_sup->level[level]; for_each_possible_cpu(cpu) { sdp = per_cpu_ptr(ssp->sda, cpu); sdp->mynode = &snp_first[cpu / levelspread[level]]; @@ -889,7 +889,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) srcu_for_each_node_breadth_first(ssp, snp) { spin_lock_irq_rcu_node(snp); cbs = false; - last_lvl = snp >= ssp->level[rcu_num_lvls - 1]; + last_lvl = snp >= ssp->srcu_sup->level[rcu_num_lvls - 1]; if (last_lvl) cbs = ss_state < SRCU_SIZE_BIG || snp->srcu_have_cbs[idx] == gpseq; snp->srcu_have_cbs[idx] = gpseq; From patchwork Fri Mar 24 00:19:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74274 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp56686wrt; Thu, 23 Mar 2023 17:26:23 -0700 (PDT) X-Google-Smtp-Source: AKy350Z8X9WywlQi42EjSvYsUsIBMZ2cVgiw/CYRVSjgK+EN6PQFaOFxpJw0aIJbPHivMO/gPuge X-Received: by 2002:aa7:cd8b:0:b0:501:d3a1:9a52 with SMTP id x11-20020aa7cd8b000000b00501d3a19a52mr1156927edv.33.1679617583707; Thu, 23 Mar 2023 17:26:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679617583; cv=none; d=google.com; s=arc-20160816; b=kWCz078m22lyYL8/eXLz/VTKhjWKcx8FAM2j3QW8hLeaHpuFp+eKzqdewm9/gdW8o0 KL3Gzd4asNIOKgFv5525zLi9WI5L7cYQdtcxtodbpV/YrP+q7Q7l0OvgfyYD/8E0fRx4 /zBwJE3eXD/ghLkVa3IH6Hczlxmzslcd2MR73s1iTI+sK/aDq+2uYoyXhIw0Hb+zAxjK 6gIWAOYnOV2CEOFD3E/MHtWzQohouQGyb0hV6z5S9sWoucrvp1W4pE+Q4laxzS6ocfmJ uZm0bQ632BaEAzbCGWRJqcQYzZEudunV3NB8yLiu0LEMgiqYt59oA3lBCJjo4U6yfcnx jpdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+oiPJzl6eQi4eOJrTI3YiT7mso2/HXyMcXOvw0mKKK4=; b=UkOmKGKH6F7/YTSrh2sRxkyCjTLQ9h5Md1ef4NZaJn6tCY6+YJ3ko0R+ZfYm/Q12g9 dg+mwnnoI4Q/qTyC1UiCBO/SFMKICWcJCXheI8tNmBq/wcIoZNiRvuynelGjKCW4OOXj oTvVzJl3G+0ZoO34smvvneDkBZUcLVc8MICQIxddGz4S8HCCeJjc8y4E4SI8jx+eKwYh 5bX8QdoTo9WvMhoUExG2yxaVaZCXJvhAD92tiKJOKOuF//fXUxMgsIv0OUqINFYOCSnL WmNAQkxDxDJBTHzVVGE2zoU2bMGbN4wei1laKbhSFCCTNHoPpxpxwqqtWT61kw020QLi ISHA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Witaechd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e11-20020a056402088b00b0050203525481si2745205edy.188.2023.03.23.17.26.00; Thu, 23 Mar 2023 17:26:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Witaechd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231556AbjCXAUO (ORCPT + 99 others); Thu, 23 Mar 2023 20:20:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231240AbjCXATu (ORCPT ); Thu, 23 Mar 2023 20:19:50 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0BB6E2DE7B; Thu, 23 Mar 2023 17:19:44 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 767AAB822A6; Fri, 24 Mar 2023 00:19:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 20ED7C4339E; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=02q/OWqQRR5LuTJFEoF1ibkG9lx5Gkwwg1gtcCvUyAI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WitaechdF+UVTeKMRghz048IL1tjLUpNxWNIBHDdg3lFobWazzXUOomhkTztr7NPu HBce1UJvGbiwcptwp3imR1KssRsB4U+w3vnt+Y/+TDkSUbBOJVc0Kt90Y0o/aTmtoY MokOY3ks5N6PWdLuuvG401Q2fApH55ocgWl/hNI/DecejGatRhrf0uMhyVymFBZXYv dXGnnl9VnXko503f6bBJtxrLhlEq/hLPYbAPv3x2WCjjYHgwNWVkJMcKcXkp3F8ROF n0vZVVxVnfdbDnZqfQ2JusCRvB8DgE5IiD8VXh7QZvVy06uF/lN4gQD5lj2HYcvTW6 7ken20WLJCDGg== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id 98870154039E; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 05/19] srcu: Move ->srcu_size_state from srcu_struct to srcu_usage Date: Thu, 23 Mar 2023 17:19:24 -0700 Message-Id: <20230324001938.3443499-5-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-5.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761206687665707995?= X-GMAIL-MSGID: =?utf-8?q?1761206687665707995?= This commit moves the ->srcu_size_state field from the srcu_struct structure to the srcu_usage structure to reduce the size of the former in order to improve cache locality. Suggested-by: Christoph Hellwig Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 2 +- kernel/rcu/srcutree.c | 37 +++++++++++++++++++------------------ 2 files changed, 20 insertions(+), 19 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 362c8f39c53d..72fb01fb2eb5 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -64,13 +64,13 @@ struct srcu_usage { struct srcu_node *node; /* Combining tree. */ struct srcu_node *level[RCU_NUM_LVLS + 1]; /* First node at each level. */ + int srcu_size_state; /* Small-to-big transition state. */ }; /* * Per-SRCU-domain structure, similar in function to rcu_state. */ struct srcu_struct { - int srcu_size_state; /* Small-to-big transition state. */ struct mutex srcu_cb_mutex; /* Serialize CB preparation. */ spinlock_t __private lock; /* Protect counters and size state. */ struct mutex srcu_gp_mutex; /* Serialize GP work. */ diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 90d753e10e33..2717217de136 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -225,7 +225,7 @@ static bool init_srcu_struct_nodes(struct srcu_struct *ssp, gfp_t gfp_flags) } sdp->grpmask = 1 << (cpu - sdp->mynode->grplo); } - smp_store_release(&ssp->srcu_size_state, SRCU_SIZE_WAIT_BARRIER); + smp_store_release(&ssp->srcu_sup->srcu_size_state, SRCU_SIZE_WAIT_BARRIER); return true; } @@ -240,7 +240,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) ssp->srcu_sup = kzalloc(sizeof(*ssp->srcu_sup), GFP_KERNEL); if (!ssp->srcu_sup) return -ENOMEM; - ssp->srcu_size_state = SRCU_SIZE_SMALL; + ssp->srcu_sup->srcu_size_state = SRCU_SIZE_SMALL; ssp->srcu_sup->node = NULL; mutex_init(&ssp->srcu_cb_mutex); mutex_init(&ssp->srcu_gp_mutex); @@ -261,7 +261,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) init_srcu_struct_data(ssp); ssp->srcu_gp_seq_needed_exp = 0; ssp->srcu_last_gp_end = ktime_get_mono_fast_ns(); - if (READ_ONCE(ssp->srcu_size_state) == SRCU_SIZE_SMALL && SRCU_SIZING_IS_INIT()) { + if (READ_ONCE(ssp->srcu_sup->srcu_size_state) == SRCU_SIZE_SMALL && SRCU_SIZING_IS_INIT()) { if (!init_srcu_struct_nodes(ssp, GFP_ATOMIC)) { if (!ssp->sda_is_static) { free_percpu(ssp->sda); @@ -270,7 +270,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) return -ENOMEM; } } else { - WRITE_ONCE(ssp->srcu_size_state, SRCU_SIZE_BIG); + WRITE_ONCE(ssp->srcu_sup->srcu_size_state, SRCU_SIZE_BIG); } } smp_store_release(&ssp->srcu_gp_seq_needed, 0); /* Init done. */ @@ -315,7 +315,7 @@ EXPORT_SYMBOL_GPL(init_srcu_struct); static void __srcu_transition_to_big(struct srcu_struct *ssp) { lockdep_assert_held(&ACCESS_PRIVATE(ssp, lock)); - smp_store_release(&ssp->srcu_size_state, SRCU_SIZE_ALLOC); + smp_store_release(&ssp->srcu_sup->srcu_size_state, SRCU_SIZE_ALLOC); } /* @@ -326,10 +326,10 @@ static void srcu_transition_to_big(struct srcu_struct *ssp) unsigned long flags; /* Double-checked locking on ->srcu_size-state. */ - if (smp_load_acquire(&ssp->srcu_size_state) != SRCU_SIZE_SMALL) + if (smp_load_acquire(&ssp->srcu_sup->srcu_size_state) != SRCU_SIZE_SMALL) return; spin_lock_irqsave_rcu_node(ssp, flags); - if (smp_load_acquire(&ssp->srcu_size_state) != SRCU_SIZE_SMALL) { + if (smp_load_acquire(&ssp->srcu_sup->srcu_size_state) != SRCU_SIZE_SMALL) { spin_unlock_irqrestore_rcu_node(ssp, flags); return; } @@ -345,7 +345,7 @@ static void spin_lock_irqsave_check_contention(struct srcu_struct *ssp) { unsigned long j; - if (!SRCU_SIZING_IS_CONTEND() || ssp->srcu_size_state) + if (!SRCU_SIZING_IS_CONTEND() || ssp->srcu_sup->srcu_size_state) return; j = jiffies; if (ssp->srcu_size_jiffies != j) { @@ -666,7 +666,7 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) } kfree(ssp->srcu_sup->node); ssp->srcu_sup->node = NULL; - ssp->srcu_size_state = SRCU_SIZE_SMALL; + ssp->srcu_sup->srcu_size_state = SRCU_SIZE_SMALL; if (!ssp->sda_is_static) { free_percpu(ssp->sda); ssp->sda = NULL; @@ -770,7 +770,7 @@ static void srcu_gp_start(struct srcu_struct *ssp) struct srcu_data *sdp; int state; - if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) + if (smp_load_acquire(&ssp->srcu_sup->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) sdp = per_cpu_ptr(ssp->sda, get_boot_cpu_id()); else sdp = this_cpu_ptr(ssp->sda); @@ -880,7 +880,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) /* A new grace period can start at this point. But only one. */ /* Initiate callback invocation as needed. */ - ss_state = smp_load_acquire(&ssp->srcu_size_state); + ss_state = smp_load_acquire(&ssp->srcu_sup->srcu_size_state); if (ss_state < SRCU_SIZE_WAIT_BARRIER) { srcu_schedule_cbs_sdp(per_cpu_ptr(ssp->sda, get_boot_cpu_id()), cbdelay); @@ -940,7 +940,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) if (ss_state == SRCU_SIZE_ALLOC) init_srcu_struct_nodes(ssp, GFP_KERNEL); else - smp_store_release(&ssp->srcu_size_state, ss_state + 1); + smp_store_release(&ssp->srcu_sup->srcu_size_state, ss_state + 1); } } @@ -1002,7 +1002,7 @@ static void srcu_funnel_gp_start(struct srcu_struct *ssp, struct srcu_data *sdp, unsigned long snp_seq; /* Ensure that snp node tree is fully initialized before traversing it */ - if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) + if (smp_load_acquire(&ssp->srcu_sup->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) snp_leaf = NULL; else snp_leaf = sdp->mynode; @@ -1229,7 +1229,7 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp, * sequence number cannot wrap around in the meantime. */ idx = __srcu_read_lock_nmisafe(ssp); - ss_state = smp_load_acquire(&ssp->srcu_size_state); + ss_state = smp_load_acquire(&ssp->srcu_sup->srcu_size_state); if (ss_state < SRCU_SIZE_WAIT_CALL) sdp = per_cpu_ptr(ssp->sda, get_boot_cpu_id()); else @@ -1568,7 +1568,7 @@ void srcu_barrier(struct srcu_struct *ssp) atomic_set(&ssp->srcu_barrier_cpu_cnt, 1); idx = __srcu_read_lock_nmisafe(ssp); - if (smp_load_acquire(&ssp->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) + if (smp_load_acquire(&ssp->srcu_sup->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) srcu_barrier_one_cpu(ssp, per_cpu_ptr(ssp->sda, get_boot_cpu_id())); else for_each_possible_cpu(cpu) @@ -1806,7 +1806,7 @@ void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf) int cpu; int idx; unsigned long s0 = 0, s1 = 0; - int ss_state = READ_ONCE(ssp->srcu_size_state); + int ss_state = READ_ONCE(ssp->srcu_sup->srcu_size_state); int ss_state_idx = ss_state; idx = ssp->srcu_idx & 0x1; @@ -1893,8 +1893,9 @@ void __init srcu_init(void) ssp = list_first_entry(&srcu_boot_list, struct srcu_struct, work.work.entry); list_del_init(&ssp->work.work.entry); - if (SRCU_SIZING_IS(SRCU_SIZING_INIT) && ssp->srcu_size_state == SRCU_SIZE_SMALL) - ssp->srcu_size_state = SRCU_SIZE_ALLOC; + if (SRCU_SIZING_IS(SRCU_SIZING_INIT) && + ssp->srcu_sup->srcu_size_state == SRCU_SIZE_SMALL) + ssp->srcu_sup->srcu_size_state = SRCU_SIZE_ALLOC; queue_work(rcu_gp_wq, &ssp->work.work); } } From patchwork Fri Mar 24 00:19:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74283 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp64174wrt; Thu, 23 Mar 2023 17:47:04 -0700 (PDT) X-Google-Smtp-Source: AKy350Z/FqdZRuhy17WAzJHuxLuxQ8j5CdYWwbZe5cvir8ykVu1cLgJ4TMJ0+5oi6aLxwBwpSZGQ X-Received: by 2002:a17:906:5d14:b0:92f:e7e2:b7b3 with SMTP id g20-20020a1709065d1400b0092fe7e2b7b3mr1091752ejt.5.1679618824274; Thu, 23 Mar 2023 17:47:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679618824; cv=none; d=google.com; s=arc-20160816; b=O+HkCmnQoJPU+iLtpz0kLew/mMXvKpprGEXp8FBKUY2sKORkKe7aW7CTVJLuqQdZ+2 1KNcdk8OZ57CmhneBCsnX9yzP3rtWNnLXFbhmwj/LNQacOg6yAGP009mY90lCb/qPewG 4n8RWKeddDz05zXx+rA6JnNRccJ3cUJnrRSpehuU7HMOOULABQmNSFZpPTW5ANy1hBDP 7tudXoJLKSKUJ42oJYtEcD45WE9Zn2QNTom5E/q8AaWm3TrfwHMw7zDpbDsEfCXFXoFE 8UcmvWGq6XTZzx+EbZf9mYagO4i/vL/hfBQ65laXUyfiJyrUpGuqNh8BZg0/NP4XEvVX zw9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=tcQ1U0+rAiTGV+XDKJv9lJ4z7dPGWgxCUYD1wQVldHc=; b=GASHfRiB6pwXRplFYjHvzfD7rA60670YvHHxOKpaO8eFqCiKrrLPaQCs8+eDnm8zs7 VARC2HG3hqbcowQYusVbTPouDJ2AE2YYPzFd6RA7TqCdvmwPDLuVQnaMnQ0vgM42CEwk rLEvhf8nJkE8W9BHdng0fNJZtuxgYxG/hf+YaJxDUhQHL1iAPNvO0MCn2qbWc30pGogk YWW53Vq3wMSJdRT+ANNpcLajRzF5RmkgOg2J7Fk6cVDFVKdMAr69NjxYIoJRLcgpo2+I 7V90L27ElSLQ0DEMAEh3zC+Opw1rcGXUliQ5rJ4WiozNsm7QqUtSThSHF59wuyWAySq9 llow== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=o7EV7iCL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id mp8-20020a1709071b0800b0093defbd6271si658502ejc.1016.2023.03.23.17.46.39; Thu, 23 Mar 2023 17:47:04 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=o7EV7iCL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231614AbjCXAUY (ORCPT + 99 others); Thu, 23 Mar 2023 20:20:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231278AbjCXATw (ORCPT ); Thu, 23 Mar 2023 20:19:52 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 822D52E0C5; Thu, 23 Mar 2023 17:19:45 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 43C2FCE23F2; Fri, 24 Mar 2023 00:19:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 587A2C433A0; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=1zZMbnj6wCHzqV26Ik2U3pZ5xQ0e/amfS4bQebuVTDo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=o7EV7iCLpr0TJDTZJ4avWQJqOKyCa7BhzDmRu1B4UgeeQUswVdFBXUU0tYiizhpGH pViUmMTWcGtxz6RvMn/WIq4bQYfvwJOwOxxYl5htEUlPzWznd//kbNyjSYkEaWy0jQ 73HeQloSXoSOcpgViMPrQRS7+QvJh6X5HtL7z69Iq6K7Xi3jyncu+KR5DiJKtSW8YB xVW3mQ39iPBbaeqdnF0H7TYpuUZylGnXHxwuJj0S3wYSTbthNuFhWvIQibdFcZoe1i Brum04832rhsxlvlwWT2YlxcadWbSL0Nqmw4HZ7R6lX0Fqj0OgPuPrtU0N/5hFWk9d gYMF/FQxTzUlA== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id 9BE43154039F; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 06/19] srcu: Move ->srcu_cb_mutex from srcu_struct to srcu_usage Date: Thu, 23 Mar 2023 17:19:25 -0700 Message-Id: <20230324001938.3443499-6-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761207988522779234?= X-GMAIL-MSGID: =?utf-8?q?1761207988522779234?= This commit moves the ->srcu_cb_mutex field from the srcu_struct structure to the srcu_usage structure to reduce the size of the former in order to improve cache locality. Suggested-by: Christoph Hellwig Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 2 +- kernel/rcu/srcutree.c | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 72fb01fb2eb5..69d49de87b67 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -65,13 +65,13 @@ struct srcu_usage { struct srcu_node *level[RCU_NUM_LVLS + 1]; /* First node at each level. */ int srcu_size_state; /* Small-to-big transition state. */ + struct mutex srcu_cb_mutex; /* Serialize CB preparation. */ }; /* * Per-SRCU-domain structure, similar in function to rcu_state. */ struct srcu_struct { - struct mutex srcu_cb_mutex; /* Serialize CB preparation. */ spinlock_t __private lock; /* Protect counters and size state. */ struct mutex srcu_gp_mutex; /* Serialize GP work. */ unsigned int srcu_idx; /* Current rdr array element. */ diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 2717217de136..6b6d92fe41cf 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -242,7 +242,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) return -ENOMEM; ssp->srcu_sup->srcu_size_state = SRCU_SIZE_SMALL; ssp->srcu_sup->node = NULL; - mutex_init(&ssp->srcu_cb_mutex); + mutex_init(&ssp->srcu_sup->srcu_cb_mutex); mutex_init(&ssp->srcu_gp_mutex); ssp->srcu_idx = 0; ssp->srcu_gp_seq = 0; @@ -861,7 +861,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) int ss_state; /* Prevent more than one additional grace period. */ - mutex_lock(&ssp->srcu_cb_mutex); + mutex_lock(&ssp->srcu_sup->srcu_cb_mutex); /* End the current grace period. */ spin_lock_irq_rcu_node(ssp); @@ -921,7 +921,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) } /* Callback initiation done, allow grace periods after next. */ - mutex_unlock(&ssp->srcu_cb_mutex); + mutex_unlock(&ssp->srcu_sup->srcu_cb_mutex); /* Start a new grace period if needed. */ spin_lock_irq_rcu_node(ssp); From patchwork Fri Mar 24 00:19:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74286 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp64891wrt; Thu, 23 Mar 2023 17:49:31 -0700 (PDT) X-Google-Smtp-Source: AKy350YeRMxsCAu0/3nVoJ1Gzj7+vMQCyscf1aojKasDAolBLFtgccI9So+2H3moyN8Xqie4tDqG X-Received: by 2002:a17:907:86ac:b0:92b:eefb:b966 with SMTP id qa44-20020a17090786ac00b0092beefbb966mr1218074ejc.0.1679618971752; Thu, 23 Mar 2023 17:49:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679618971; cv=none; d=google.com; s=arc-20160816; b=QWwoaG5JrC9mRlz7E0TH0RAX39IOdSUvJGHOSk5eoaNnpPSeKmfV/1G5zGkaPe/u43 B8kYrtUP2ggUBcpr/1eEvnOmXBmYxS5wtY2nF9e37vcR9/Wg7jGuWG6UZrsBIPD6kdyQ EgoQU+7RXiJS23AJeCRrPndrOc1zZmF95z6pDvhgfgonDSu2u8Pw2qkgqJG+FKgP0DTY /zl9y1Lgn3oYx/Ukq+ge2Cndp31o7PL2mB8X1h8aQOVMROtqVJLCWhuW2e0h6PjuADaa ejS0Tv58VDLfAHH0AG32DJKQHm97EYucoRB8yqhWpw0Bbn6DnhlEra48uRrKElg2PxpM LUpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=zUj/nNlUH5zrhof4Tp3Y9Qd0pRIq5dwaahQ5+e8VQw8=; b=TfJSK7kJg7ArfG5RVeWfqrYPx+2iwdxfEdg2sCOBRdyrygI0+SkauGqalVLgG6nBmS X6Z42ICe8ijzNRpx6DIh0hQ8PbC2n2yKBQghxCw6q2C+wZEX1QMQhOXDwxM461SGtGHn fxJ4H/sFHhIiINTAv24avZ7hGxOi8z/64VatveqD4W5dg0F/N0xaaT5IQS81aP+qiJ/g uWz2e+DTciFTIhB5+IGOxuIZZuZP9pFaypl9aRJhfuEvNrlN40+R0z3GcPCXJ39hF9Ht nhfGMhnXAaUYAaTPH+uQjw+KzG29dWoxGtFfW69y0o9XPZDeAeNbbRBZSJQSh55h+F7G iiMw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=UkDVJd8N; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i20-20020a170906115400b008b17cc2e38asi18535685eja.582.2023.03.23.17.49.07; Thu, 23 Mar 2023 17:49:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=UkDVJd8N; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231572AbjCXAUS (ORCPT + 99 others); Thu, 23 Mar 2023 20:20:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57944 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231252AbjCXATu (ORCPT ); Thu, 23 Mar 2023 20:19:50 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3BE612D148; Thu, 23 Mar 2023 17:19:44 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id E0BA7B821CB; Fri, 24 Mar 2023 00:19:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5F655C433A4; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=CyI2x+E6wMeERtsPHUF/+GOvXVfY6z2JZMMsSarhSLI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UkDVJd8No6s7ukbT7sfNiUaDEYE3eq3U0X7dM3eEWUKY3SmlKyafcpOQBRf665Q78 BYL+i9XhxHCsMfQ9wtGa5mqWFz0vc57gjbHg6PimsNdHIq/DCc1KXmyZ0nD/2fOfD3 P4q+2TFGrkDCVN2hu4kWz2K2S1icmXON9VSrVCv/W3rTrCSeVPtf0G1cfOXVV2Foia DP9rB+sYeqQi3+ocq8JY4EHNhXiSq43ACZRKBml1Y3GM5mcrKx5qAv2unJcuxwH/E/ v562+4FFHgJDaIhBJpCBJdyKBfbp4GNDcBZhM+Ch/8z4/iGyDl1MQUhqb6JbovPkw1 GjAGxcWT84DTw== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id A0E1615403A0; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 07/19] srcu: Move ->lock initialization after srcu_usage allocation Date: Thu, 23 Mar 2023 17:19:26 -0700 Message-Id: <20230324001938.3443499-7-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761208143271945318?= X-GMAIL-MSGID: =?utf-8?q?1761208143271945318?= Currently, both __init_srcu_struct() in CONFIG_DEBUG_LOCK_ALLOC=y kernels and init_srcu_struct() in CONFIG_DEBUG_LOCK_ALLOC=n kernel initialize the srcu_struct structure's ->lock before the srcu_usage structure has been allocated. This of course prevents the ->lock from being moved to the srcu_usage structure, so this commit moves the initialization into the init_srcu_struct_fields() after the srcu_usage structure has been allocated. Signed-off-by: Paul E. McKenney Cc: Christoph Hellwig --- kernel/rcu/srcutree.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 6b6d92fe41cf..f6f37e34a1e0 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -240,6 +240,8 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) ssp->srcu_sup = kzalloc(sizeof(*ssp->srcu_sup), GFP_KERNEL); if (!ssp->srcu_sup) return -ENOMEM; + if (!is_static) + spin_lock_init(&ACCESS_PRIVATE(ssp, lock)); ssp->srcu_sup->srcu_size_state = SRCU_SIZE_SMALL; ssp->srcu_sup->node = NULL; mutex_init(&ssp->srcu_sup->srcu_cb_mutex); @@ -285,7 +287,6 @@ int __init_srcu_struct(struct srcu_struct *ssp, const char *name, /* Don't re-initialize a lock while it is held. */ debug_check_no_locks_freed((void *)ssp, sizeof(*ssp)); lockdep_init_map(&ssp->dep_map, name, key, 0); - spin_lock_init(&ACCESS_PRIVATE(ssp, lock)); return init_srcu_struct_fields(ssp, false); } EXPORT_SYMBOL_GPL(__init_srcu_struct); @@ -302,7 +303,6 @@ EXPORT_SYMBOL_GPL(__init_srcu_struct); */ int init_srcu_struct(struct srcu_struct *ssp) { - spin_lock_init(&ACCESS_PRIVATE(ssp, lock)); return init_srcu_struct_fields(ssp, false); } EXPORT_SYMBOL_GPL(init_srcu_struct); From patchwork Fri Mar 24 00:19:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74285 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp64562wrt; Thu, 23 Mar 2023 17:48:27 -0700 (PDT) X-Google-Smtp-Source: AKy350Z63GDb1/PzgYvxwZbC6/YfUAilNLvRbZe7KALmofXYAVPTk0ydCo8khSL852kLwVRHCvxo X-Received: by 2002:a17:906:e0c6:b0:93b:a133:f7e6 with SMTP id gl6-20020a170906e0c600b0093ba133f7e6mr790748ejb.46.1679618906906; Thu, 23 Mar 2023 17:48:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679618906; cv=none; d=google.com; s=arc-20160816; b=pj/UNPSaRBsWmbnMNo2EuNgT4+Wgs2E7ntt8zefXfaBF89lRIQ9dw2+0hi96qst9i6 bUx6YkF0OdcDgjSbK4/Dkx2GzFpSGLqVx2/+dA5PdGSA3Dx3fq4ySrKZvkWdAAr6qBXE D1P50UOebwXeza1RM6Sb82hWhjDPN8bS5VAC2oQNaGFfNdknS/lQQWzEScPbbEjdua1F /ZKvbvCStCHlINO6hUQuwlsV2QZDrIVlIT3+ro+isgF//2F3R8N+l9iJGEoKZIeO1Cxd vtDPEZBLTwX7gpghHKw+MknAY6FwMQuptrSlHRHjAsFAFMOjfUxkET8bta1s6bgUNR6B WNyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3sMNRdS+w8qrLvl0booAi/vvUpZbONFvwYe4azVaktE=; b=ryf1xbpA2P985SN/+FejlCr9RMGkvYrtSvQRF0foE6iqn4P7WDZCDuD/Jm02hytYik IeaC42g8xCHceeJ4m9mnzmd+hT9fCoAekiAJHO8L0vLCepJMtliWh4cicfPKAiXbGPy4 3bbQkuHsDt9/oLP/DRthdSrqd4hR7/xtmUCndKsnVlRQlq320+H7YfId9NsmMqnyvJEE U8JiN+icYFcyZZS12eraDY6DdOT6DxRlTjIWcqA7UuAXvLKioW1cZ5LlbT29bfr6MXU0 SzLab+A/2GHWH+Gimp5gBKxIxP+IdhmlkpqL0zK6zFc13iszGQeluU/qTQKj1lMnbmt+ cwzw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=pgOlU74j; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id sh8-20020a1709076e8800b0093d0d964aebsi3012170ejc.124.2023.03.23.17.48.02; Thu, 23 Mar 2023 17:48:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=pgOlU74j; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231649AbjCXAU3 (ORCPT + 99 others); Thu, 23 Mar 2023 20:20:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57888 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231183AbjCXATz (ORCPT ); Thu, 23 Mar 2023 20:19:55 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 628242DE43; Thu, 23 Mar 2023 17:19:45 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 50419CE23E7; Fri, 24 Mar 2023 00:19:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5FB24C433A7; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=Kuzm8Bj8iVO+o7GdhoPvUAfNWxPS8pezXjecL0pzOgg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pgOlU74jyJQT+/HmpEyFq20STsjbFdQzqCPGC+G0Jnl6thtH7jZqaHglhAy2wMbUX 0moIAiZvsoam0bE/n93phwLkCwtKWLfFsC0ohfGeFHyw1Ox84Kouu5yWSxZQKr/K8s 47EVYY/oLK7TEHrtCyr5rNCfNF05qDr4mMFBdKFdl0H9sqxCwzdEmMkmWQYPNguRBy hEedKx4E44MSLSc+l3XykylX3llMevP+VLxVGo+8znBAj49DShOhS03GC7uoMgi5ua S30IP49k438xZpKdlDQjyRv8FmvjlkyV2DZb9t4gN1AP49zPHNY2aq7xeygqgObQdF uUh6tTbQZz22g== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id A533D15403A1; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 08/19] srcu: Move ->lock from srcu_struct to srcu_usage Date: Thu, 23 Mar 2023 17:19:27 -0700 Message-Id: <20230324001938.3443499-8-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761208074674445482?= X-GMAIL-MSGID: =?utf-8?q?1761208074674445482?= This commit moves the ->lock field from the srcu_struct structure to the srcu_usage structure to reduce the size of the former in order to improve cache locality. Suggested-by: Christoph Hellwig Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 11 +++++--- kernel/rcu/srcutree.c | 56 ++++++++++++++++++++-------------------- 2 files changed, 35 insertions(+), 32 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 69d49de87b67..6fe5743c1179 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -66,13 +66,13 @@ struct srcu_usage { /* First node at each level. */ int srcu_size_state; /* Small-to-big transition state. */ struct mutex srcu_cb_mutex; /* Serialize CB preparation. */ + spinlock_t __private lock; /* Protect counters and size state. */ }; /* * Per-SRCU-domain structure, similar in function to rcu_state. */ struct srcu_struct { - spinlock_t __private lock; /* Protect counters and size state. */ struct mutex srcu_gp_mutex; /* Serialize GP work. */ unsigned int srcu_idx; /* Current rdr array element. */ unsigned long srcu_gp_seq; /* Grace-period seq #. */ @@ -129,7 +129,6 @@ struct srcu_struct { #define SRCU_STATE_SCAN2 2 #define __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ - .lock = __SPIN_LOCK_UNLOCKED(name.lock), \ .srcu_gp_seq_needed = -1UL, \ .work = __DELAYED_WORK_INITIALIZER(name.work, NULL, 0), \ .srcu_sup = &usage_name, \ @@ -167,7 +166,9 @@ struct srcu_struct { */ #ifdef MODULE # define __DEFINE_SRCU(name, is_static) \ - static struct srcu_usage name##_srcu_usage; \ + static struct srcu_usage name##_srcu_usage = { \ + .lock = __SPIN_LOCK_UNLOCKED(name.lock), \ + }; \ is_static struct srcu_struct name = __SRCU_STRUCT_INIT_MODULE(name, name##_srcu_usage); \ extern struct srcu_struct * const __srcu_struct_##name; \ struct srcu_struct * const __srcu_struct_##name \ @@ -175,7 +176,9 @@ struct srcu_struct { #else # define __DEFINE_SRCU(name, is_static) \ static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data); \ - static struct srcu_usage name##_srcu_usage; \ + static struct srcu_usage name##_srcu_usage = { \ + .lock = __SPIN_LOCK_UNLOCKED(name.lock), \ + }; \ is_static struct srcu_struct name = \ __SRCU_STRUCT_INIT(name, name##_srcu_usage, name##_srcu_data) #endif diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index f6f37e34a1e0..579c43ff0f4b 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -103,7 +103,7 @@ do { \ #define spin_trylock_irqsave_rcu_node(p, flags) \ ({ \ - bool ___locked = spin_trylock_irqsave(&ACCESS_PRIVATE(p, lock), flags); \ + bool ___locked = spin_trylock_irqsave(&ACCESS_PRIVATE(p, lock), flags); \ \ if (___locked) \ smp_mb__after_unlock_lock(); \ @@ -241,7 +241,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) if (!ssp->srcu_sup) return -ENOMEM; if (!is_static) - spin_lock_init(&ACCESS_PRIVATE(ssp, lock)); + spin_lock_init(&ACCESS_PRIVATE(ssp->srcu_sup, lock)); ssp->srcu_sup->srcu_size_state = SRCU_SIZE_SMALL; ssp->srcu_sup->node = NULL; mutex_init(&ssp->srcu_sup->srcu_cb_mutex); @@ -314,7 +314,7 @@ EXPORT_SYMBOL_GPL(init_srcu_struct); */ static void __srcu_transition_to_big(struct srcu_struct *ssp) { - lockdep_assert_held(&ACCESS_PRIVATE(ssp, lock)); + lockdep_assert_held(&ACCESS_PRIVATE(ssp->srcu_sup, lock)); smp_store_release(&ssp->srcu_sup->srcu_size_state, SRCU_SIZE_ALLOC); } @@ -328,13 +328,13 @@ static void srcu_transition_to_big(struct srcu_struct *ssp) /* Double-checked locking on ->srcu_size-state. */ if (smp_load_acquire(&ssp->srcu_sup->srcu_size_state) != SRCU_SIZE_SMALL) return; - spin_lock_irqsave_rcu_node(ssp, flags); + spin_lock_irqsave_rcu_node(ssp->srcu_sup, flags); if (smp_load_acquire(&ssp->srcu_sup->srcu_size_state) != SRCU_SIZE_SMALL) { - spin_unlock_irqrestore_rcu_node(ssp, flags); + spin_unlock_irqrestore_rcu_node(ssp->srcu_sup, flags); return; } __srcu_transition_to_big(ssp); - spin_unlock_irqrestore_rcu_node(ssp, flags); + spin_unlock_irqrestore_rcu_node(ssp->srcu_sup, flags); } /* @@ -369,9 +369,9 @@ static void spin_lock_irqsave_sdp_contention(struct srcu_data *sdp, unsigned lon if (spin_trylock_irqsave_rcu_node(sdp, *flags)) return; - spin_lock_irqsave_rcu_node(ssp, *flags); + spin_lock_irqsave_rcu_node(ssp->srcu_sup, *flags); spin_lock_irqsave_check_contention(ssp); - spin_unlock_irqrestore_rcu_node(ssp, *flags); + spin_unlock_irqrestore_rcu_node(ssp->srcu_sup, *flags); spin_lock_irqsave_rcu_node(sdp, *flags); } @@ -383,9 +383,9 @@ static void spin_lock_irqsave_sdp_contention(struct srcu_data *sdp, unsigned lon */ static void spin_lock_irqsave_ssp_contention(struct srcu_struct *ssp, unsigned long *flags) { - if (spin_trylock_irqsave_rcu_node(ssp, *flags)) + if (spin_trylock_irqsave_rcu_node(ssp->srcu_sup, *flags)) return; - spin_lock_irqsave_rcu_node(ssp, *flags); + spin_lock_irqsave_rcu_node(ssp->srcu_sup, *flags); spin_lock_irqsave_check_contention(ssp); } @@ -404,13 +404,13 @@ static void check_init_srcu_struct(struct srcu_struct *ssp) /* The smp_load_acquire() pairs with the smp_store_release(). */ if (!rcu_seq_state(smp_load_acquire(&ssp->srcu_gp_seq_needed))) /*^^^*/ return; /* Already initialized. */ - spin_lock_irqsave_rcu_node(ssp, flags); + spin_lock_irqsave_rcu_node(ssp->srcu_sup, flags); if (!rcu_seq_state(ssp->srcu_gp_seq_needed)) { - spin_unlock_irqrestore_rcu_node(ssp, flags); + spin_unlock_irqrestore_rcu_node(ssp->srcu_sup, flags); return; } init_srcu_struct_fields(ssp, true); - spin_unlock_irqrestore_rcu_node(ssp, flags); + spin_unlock_irqrestore_rcu_node(ssp->srcu_sup, flags); } /* @@ -774,7 +774,7 @@ static void srcu_gp_start(struct srcu_struct *ssp) sdp = per_cpu_ptr(ssp->sda, get_boot_cpu_id()); else sdp = this_cpu_ptr(ssp->sda); - lockdep_assert_held(&ACCESS_PRIVATE(ssp, lock)); + lockdep_assert_held(&ACCESS_PRIVATE(ssp->srcu_sup, lock)); WARN_ON_ONCE(ULONG_CMP_GE(ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)); spin_lock_rcu_node(sdp); /* Interrupts already disabled. */ rcu_segcblist_advance(&sdp->srcu_cblist, @@ -864,7 +864,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) mutex_lock(&ssp->srcu_sup->srcu_cb_mutex); /* End the current grace period. */ - spin_lock_irq_rcu_node(ssp); + spin_lock_irq_rcu_node(ssp->srcu_sup); idx = rcu_seq_state(ssp->srcu_gp_seq); WARN_ON_ONCE(idx != SRCU_STATE_SCAN2); if (ULONG_CMP_LT(READ_ONCE(ssp->srcu_gp_seq), READ_ONCE(ssp->srcu_gp_seq_needed_exp))) @@ -875,7 +875,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) gpseq = rcu_seq_current(&ssp->srcu_gp_seq); if (ULONG_CMP_LT(ssp->srcu_gp_seq_needed_exp, gpseq)) WRITE_ONCE(ssp->srcu_gp_seq_needed_exp, gpseq); - spin_unlock_irq_rcu_node(ssp); + spin_unlock_irq_rcu_node(ssp->srcu_sup); mutex_unlock(&ssp->srcu_gp_mutex); /* A new grace period can start at this point. But only one. */ @@ -924,15 +924,15 @@ static void srcu_gp_end(struct srcu_struct *ssp) mutex_unlock(&ssp->srcu_sup->srcu_cb_mutex); /* Start a new grace period if needed. */ - spin_lock_irq_rcu_node(ssp); + spin_lock_irq_rcu_node(ssp->srcu_sup); gpseq = rcu_seq_current(&ssp->srcu_gp_seq); if (!rcu_seq_state(gpseq) && ULONG_CMP_LT(gpseq, ssp->srcu_gp_seq_needed)) { srcu_gp_start(ssp); - spin_unlock_irq_rcu_node(ssp); + spin_unlock_irq_rcu_node(ssp->srcu_sup); srcu_reschedule(ssp, 0); } else { - spin_unlock_irq_rcu_node(ssp); + spin_unlock_irq_rcu_node(ssp->srcu_sup); } /* Transition to big if needed. */ @@ -975,7 +975,7 @@ static void srcu_funnel_exp_start(struct srcu_struct *ssp, struct srcu_node *snp spin_lock_irqsave_ssp_contention(ssp, &flags); if (ULONG_CMP_LT(ssp->srcu_gp_seq_needed_exp, s)) WRITE_ONCE(ssp->srcu_gp_seq_needed_exp, s); - spin_unlock_irqrestore_rcu_node(ssp, flags); + spin_unlock_irqrestore_rcu_node(ssp->srcu_sup, flags); } /* @@ -1064,7 +1064,7 @@ static void srcu_funnel_gp_start(struct srcu_struct *ssp, struct srcu_data *sdp, else if (list_empty(&ssp->work.work.entry)) list_add(&ssp->work.work.entry, &srcu_boot_list); } - spin_unlock_irqrestore_rcu_node(ssp, flags); + spin_unlock_irqrestore_rcu_node(ssp->srcu_sup, flags); } /* @@ -1621,17 +1621,17 @@ static void srcu_advance_state(struct srcu_struct *ssp) */ idx = rcu_seq_state(smp_load_acquire(&ssp->srcu_gp_seq)); /* ^^^ */ if (idx == SRCU_STATE_IDLE) { - spin_lock_irq_rcu_node(ssp); + spin_lock_irq_rcu_node(ssp->srcu_sup); if (ULONG_CMP_GE(ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)) { WARN_ON_ONCE(rcu_seq_state(ssp->srcu_gp_seq)); - spin_unlock_irq_rcu_node(ssp); + spin_unlock_irq_rcu_node(ssp->srcu_sup); mutex_unlock(&ssp->srcu_gp_mutex); return; } idx = rcu_seq_state(READ_ONCE(ssp->srcu_gp_seq)); if (idx == SRCU_STATE_IDLE) srcu_gp_start(ssp); - spin_unlock_irq_rcu_node(ssp); + spin_unlock_irq_rcu_node(ssp->srcu_sup); if (idx != SRCU_STATE_IDLE) { mutex_unlock(&ssp->srcu_gp_mutex); return; /* Someone else started the grace period. */ @@ -1645,10 +1645,10 @@ static void srcu_advance_state(struct srcu_struct *ssp) return; /* readers present, retry later. */ } srcu_flip(ssp); - spin_lock_irq_rcu_node(ssp); + spin_lock_irq_rcu_node(ssp->srcu_sup); rcu_seq_set_state(&ssp->srcu_gp_seq, SRCU_STATE_SCAN2); ssp->srcu_n_exp_nodelay = 0; - spin_unlock_irq_rcu_node(ssp); + spin_unlock_irq_rcu_node(ssp->srcu_sup); } if (rcu_seq_state(READ_ONCE(ssp->srcu_gp_seq)) == SRCU_STATE_SCAN2) { @@ -1732,7 +1732,7 @@ static void srcu_reschedule(struct srcu_struct *ssp, unsigned long delay) { bool pushgp = true; - spin_lock_irq_rcu_node(ssp); + spin_lock_irq_rcu_node(ssp->srcu_sup); if (ULONG_CMP_GE(ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)) { if (!WARN_ON_ONCE(rcu_seq_state(ssp->srcu_gp_seq))) { /* All requests fulfilled, time to go idle. */ @@ -1742,7 +1742,7 @@ static void srcu_reschedule(struct srcu_struct *ssp, unsigned long delay) /* Outstanding request and no GP. Start one. */ srcu_gp_start(ssp); } - spin_unlock_irq_rcu_node(ssp); + spin_unlock_irq_rcu_node(ssp->srcu_sup); if (pushgp) queue_delayed_work(rcu_gp_wq, &ssp->work, delay); From patchwork Fri Mar 24 00:19:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74282 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp63284wrt; Thu, 23 Mar 2023 17:44:28 -0700 (PDT) X-Google-Smtp-Source: AKy350YRttZwljmDM4ckCRr7hFqp6SBaNoz9fHwJf0Yd+fwGurOWFwko7Wx0UrAHy0K9UxYx2ZzD X-Received: by 2002:a17:906:5e45:b0:931:3a85:d0bf with SMTP id b5-20020a1709065e4500b009313a85d0bfmr744969eju.17.1679618668170; Thu, 23 Mar 2023 17:44:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679618668; cv=none; d=google.com; s=arc-20160816; b=DvXajcf/0JuujQozG323dINxFktvGdTTzx61Wzph0X7Na/inXyZltDt8eo14yBbThO XTdJX5MxIGRA2Mrqye9zjKYMbOFMzPwyZqXzfBSH4AEn0UmkK+kmpN/i0gR0TB02DOa/ IIrQx7Wo68gYlqavS0Haa4RzKHtUf/ZOMMHcHtMUuXVSQB0AczQN1pdqRINODzUm6Bq+ do4BdLOg6/3U+YEXD9oQX+xqZ3gP7Tz5dO8aFPBID1C1UbOINuRZA489vbBTTyJClZTv HjQZ5K2TvKoIXnv0HCTBq7GVsNBEUlOtwQtNwA2RXxzJT9+tDKYBqJ+NlGcuMn4EkYgf XMLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Sa8HWPQBGzvsmbKIdL/CTMVI0xo8NwymVmEhecm0uIs=; b=jYMTI6FjpcmrfvelPY+XuKEw3liISXcNPlFFLUn6ElU6vEe30f1Z7rBNbKosdRxrra 3rd5MbIUMGa59oo8Dp3Y+KgInDP8Ihao8ro3zG2WGl3ZXc23fwRK9sDxtayiKmvCLOQa f4aTZNni0K1bgYZ+nndB/R8Mk4b9XblS3caXBK5o6CRi55POb+HRxiPUIsuxc+FVJCp9 TunY3F/8kcJkgTheXxEI9UGYGxc4GdLT39wB0Hi7gymPzpHxdVA1UpRLgudfLI0cJmDA O90jmrkew4w8Ew7hAQIuYR8dGqt5HANW9FaUaStB3rgCoCHAUVoU4DQmKtM4mvA788hv BQkg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=VgYiTMlq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s27-20020a170906355b00b0093083e8ec20si12049565eja.56.2023.03.23.17.44.04; Thu, 23 Mar 2023 17:44:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=VgYiTMlq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229917AbjCXAUV (ORCPT + 99 others); Thu, 23 Mar 2023 20:20:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57980 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231267AbjCXATw (ORCPT ); Thu, 23 Mar 2023 20:19:52 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6FB332D15C; Thu, 23 Mar 2023 17:19:44 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 08C6EB822B2; Fri, 24 Mar 2023 00:19:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6F6BAC433A8; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=1z9mhKOSF9Ot+/9TtqVkMS1pJCE+8rIytPo17nS5a68=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VgYiTMlqqifxl++S+gtgsG3HPNfaeKWsMx3kYYo8nWyB9C23J9eMsk7qGs5Nm8xNo flSNHsMMcBqxTF3Mp5taBQKYwHuLG0Xrgmp/nC6x1/Rfd3VdNhHtnB2QN7piCBo8ZR i4xYJUaqL3uYrVbCp5tUhwuSnIawcaJoHsf5dOKHOVqu68qg1BF5SgC/n1lrYKP0Hz 6SpdT/rtdEXmHO7SGg76CWA7O42OBNTYW1d+Mhk73kuGIOfg6Astq48LIqworxFy1+ 9Nqk7niQxo0IvtXPdnGZw2UiMibBvY5qU1+M+kz8KGNwsnRHWM0U7NbDfgBBZJVmkd 4+TBHtvH8c63g== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id A88F315403A2; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 09/19] srcu: Move ->srcu_gp_mutex from srcu_struct to srcu_usage Date: Thu, 23 Mar 2023 17:19:28 -0700 Message-Id: <20230324001938.3443499-9-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-5.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761207824778299777?= X-GMAIL-MSGID: =?utf-8?q?1761207824778299777?= This commit moves the ->srcu_gp_mutex field from the srcu_struct structure to the srcu_usage structure to reduce the size of the former in order to improve cache locality. Suggested-by: Christoph Hellwig Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 2 +- kernel/rcu/srcutree.c | 14 +++++++------- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 6fe5743c1179..6402cd5a93ef 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -67,13 +67,13 @@ struct srcu_usage { int srcu_size_state; /* Small-to-big transition state. */ struct mutex srcu_cb_mutex; /* Serialize CB preparation. */ spinlock_t __private lock; /* Protect counters and size state. */ + struct mutex srcu_gp_mutex; /* Serialize GP work. */ }; /* * Per-SRCU-domain structure, similar in function to rcu_state. */ struct srcu_struct { - struct mutex srcu_gp_mutex; /* Serialize GP work. */ unsigned int srcu_idx; /* Current rdr array element. */ unsigned long srcu_gp_seq; /* Grace-period seq #. */ unsigned long srcu_gp_seq_needed; /* Latest gp_seq needed. */ diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 579c43ff0f4b..3bf3408c7716 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -245,7 +245,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) ssp->srcu_sup->srcu_size_state = SRCU_SIZE_SMALL; ssp->srcu_sup->node = NULL; mutex_init(&ssp->srcu_sup->srcu_cb_mutex); - mutex_init(&ssp->srcu_gp_mutex); + mutex_init(&ssp->srcu_sup->srcu_gp_mutex); ssp->srcu_idx = 0; ssp->srcu_gp_seq = 0; ssp->srcu_barrier_seq = 0; @@ -876,7 +876,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) if (ULONG_CMP_LT(ssp->srcu_gp_seq_needed_exp, gpseq)) WRITE_ONCE(ssp->srcu_gp_seq_needed_exp, gpseq); spin_unlock_irq_rcu_node(ssp->srcu_sup); - mutex_unlock(&ssp->srcu_gp_mutex); + mutex_unlock(&ssp->srcu_sup->srcu_gp_mutex); /* A new grace period can start at this point. But only one. */ /* Initiate callback invocation as needed. */ @@ -1607,7 +1607,7 @@ static void srcu_advance_state(struct srcu_struct *ssp) { int idx; - mutex_lock(&ssp->srcu_gp_mutex); + mutex_lock(&ssp->srcu_sup->srcu_gp_mutex); /* * Because readers might be delayed for an extended period after @@ -1625,7 +1625,7 @@ static void srcu_advance_state(struct srcu_struct *ssp) if (ULONG_CMP_GE(ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)) { WARN_ON_ONCE(rcu_seq_state(ssp->srcu_gp_seq)); spin_unlock_irq_rcu_node(ssp->srcu_sup); - mutex_unlock(&ssp->srcu_gp_mutex); + mutex_unlock(&ssp->srcu_sup->srcu_gp_mutex); return; } idx = rcu_seq_state(READ_ONCE(ssp->srcu_gp_seq)); @@ -1633,7 +1633,7 @@ static void srcu_advance_state(struct srcu_struct *ssp) srcu_gp_start(ssp); spin_unlock_irq_rcu_node(ssp->srcu_sup); if (idx != SRCU_STATE_IDLE) { - mutex_unlock(&ssp->srcu_gp_mutex); + mutex_unlock(&ssp->srcu_sup->srcu_gp_mutex); return; /* Someone else started the grace period. */ } } @@ -1641,7 +1641,7 @@ static void srcu_advance_state(struct srcu_struct *ssp) if (rcu_seq_state(READ_ONCE(ssp->srcu_gp_seq)) == SRCU_STATE_SCAN1) { idx = 1 ^ (ssp->srcu_idx & 1); if (!try_check_zero(ssp, idx, 1)) { - mutex_unlock(&ssp->srcu_gp_mutex); + mutex_unlock(&ssp->srcu_sup->srcu_gp_mutex); return; /* readers present, retry later. */ } srcu_flip(ssp); @@ -1659,7 +1659,7 @@ static void srcu_advance_state(struct srcu_struct *ssp) */ idx = 1 ^ (ssp->srcu_idx & 1); if (!try_check_zero(ssp, idx, 2)) { - mutex_unlock(&ssp->srcu_gp_mutex); + mutex_unlock(&ssp->srcu_sup->srcu_gp_mutex); return; /* readers present, retry later. */ } ssp->srcu_n_exp_nodelay = 0; From patchwork Fri Mar 24 00:19:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74281 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp62899wrt; Thu, 23 Mar 2023 17:43:23 -0700 (PDT) X-Google-Smtp-Source: AKy350ZO7L1kmtpg8HdLX0Yq6xQC8zxlNFYhDa5oyVnr5hB9X3DMv1YM8bjoA8jX4p9YmjFuVfQT X-Received: by 2002:a17:906:da82:b0:933:4d47:55b7 with SMTP id xh2-20020a170906da8200b009334d4755b7mr1036969ejb.2.1679618603793; Thu, 23 Mar 2023 17:43:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679618603; cv=none; d=google.com; s=arc-20160816; b=sSD3Rj/lOv1+Q1UNYRU+pAelFkXcdVMd0hN1iTc25yvx4x468CFyJyV4viZEbUyJAQ J8xfsdCz8I0oNFGG+tpHNPbsLiNPGzWfON/QZigp7VK36sfl+3uM2AuoWCSGCmRW3qME OGwcVK8NTwS/wRxK85D+WWjo/q9h6e3TC+DPOYIAn+oZtfftuKU/t9v924xtErcbZYrE NN0K5WFeggEU6GalmalfK6ehzIBuXa0xGoFYaEBg6GTtbugadp13HjBo+o5xXxY5IrZO O6i1Bgtv0ZMPWiqb6uXgKmT5rCcvDa5fy4IWopJT7qVAh1pmJ/fCVVLJlTb91T5Ao0HX p0Tg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=a76M8pul09DvX6/ZGvCtOR9pRHqPDwTa85g47mKcPiA=; b=fOVQosxBNyrEbjx5esvPxnNyiaMUKEDcFPo3svM6jnX7UmUTTUc8UBROxvnS2JFyOu RUuYyRUQE0yEuBt609pclyeJ7MaUqFQtJHTBMMOMVuykWIcE61IlM+XboWT3miRRh/lx VVVARd6JTeojB6MwgPiCWc0NVaFJH40DjSgWaO+MZKMXIGfn4/WSA9n2e8HA2JWeyW7X OsplEX5lrIp8Os8IvDB6RC0yWP/VL7wVrKAuonrXMWCZJUCO6bCNyRukgqiwk5E2s3Vx 8c8Dw9BTKqPIgS/8a0c02mD5wck/TFwyEw5fsjth6/PX/ZQsqpoh1S/hwV6+Iooq6U50 +ZJg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=mH55LsPq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gb4-20020a170907960400b00939adedc8e7si8819365ejc.226.2023.03.23.17.42.58; Thu, 23 Mar 2023 17:43:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=mH55LsPq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231624AbjCXAU0 (ORCPT + 99 others); Thu, 23 Mar 2023 20:20:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57886 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231288AbjCXATz (ORCPT ); Thu, 23 Mar 2023 20:19:55 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B9D62DE7D; Thu, 23 Mar 2023 17:19:44 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 0CDB2B822B3; Fri, 24 Mar 2023 00:19:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 71034C433A1; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=pnrOHquvhl3zA9Hvs7AJrVyQy+64qTthIMKn436C2t0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mH55LsPqqBI49scsA5yLcfhU7/TSuLA9wtC/B7jB34nUPm+/ifnBfhBV9u28BjJF9 8rCIUtdkDmNH3P2kje7152V9bngGZyYc2n1XtmNscoywVvXcAyxDYzw5Q8P7r52pbG gsEKhC59fyFj/in+ryR9Bf8AneFOZu5T94H2poMZRGUfNfwSoxURfvq7+sWcFOEc4J DnWX9Jq0MNNL1a2oy4chY5pfvvQf865EH4Zz58K+k3iQ+XLCOcURbV7M3q9gfLnEvw PRRYGhZAd0ZlK25otSRmclkfTw2858UeFPrFm1g2dcXFRwHx9OxlwC+Pt/lNxw1kZf wxuMGAYKrRUEg== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id AC23215403A3; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 10/19] srcu: Move grace-period fields from srcu_struct to srcu_usage Date: Thu, 23 Mar 2023 17:19:29 -0700 Message-Id: <20230324001938.3443499-10-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761207757085465401?= X-GMAIL-MSGID: =?utf-8?q?1761207757085465401?= This commit moves the ->srcu_gp_seq, ->srcu_gp_seq_needed, ->srcu_gp_seq_needed_exp, ->srcu_gp_start, and ->srcu_last_gp_end fields from the srcu_struct structure to the srcu_usage structure to reduce the size of the former in order to improve cache locality. Suggested-by: Christoph Hellwig Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 25 ++++---- kernel/rcu/srcutree.c | 126 +++++++++++++++++++-------------------- 2 files changed, 76 insertions(+), 75 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 6402cd5a93ef..7458df6bbe00 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -68,6 +68,11 @@ struct srcu_usage { struct mutex srcu_cb_mutex; /* Serialize CB preparation. */ spinlock_t __private lock; /* Protect counters and size state. */ struct mutex srcu_gp_mutex; /* Serialize GP work. */ + unsigned long srcu_gp_seq; /* Grace-period seq #. */ + unsigned long srcu_gp_seq_needed; /* Latest gp_seq needed. */ + unsigned long srcu_gp_seq_needed_exp; /* Furthest future exp GP. */ + unsigned long srcu_gp_start; /* Last GP start timestamp (jiffies) */ + unsigned long srcu_last_gp_end; /* Last GP end timestamp (ns) */ }; /* @@ -75,11 +80,6 @@ struct srcu_usage { */ struct srcu_struct { unsigned int srcu_idx; /* Current rdr array element. */ - unsigned long srcu_gp_seq; /* Grace-period seq #. */ - unsigned long srcu_gp_seq_needed; /* Latest gp_seq needed. */ - unsigned long srcu_gp_seq_needed_exp; /* Furthest future exp GP. */ - unsigned long srcu_gp_start; /* Last GP start timestamp (jiffies) */ - unsigned long srcu_last_gp_end; /* Last GP end timestamp (ns) */ unsigned long srcu_size_jiffies; /* Current contention-measurement interval. */ unsigned long srcu_n_lock_retries; /* Contention events in current interval. */ unsigned long srcu_n_exp_nodelay; /* # expedited no-delays in current GP phase. */ @@ -128,8 +128,13 @@ struct srcu_struct { #define SRCU_STATE_SCAN1 1 #define SRCU_STATE_SCAN2 2 -#define __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ +#define __SRCU_USAGE_INIT(name) \ +{ \ + .lock = __SPIN_LOCK_UNLOCKED(name.lock), \ .srcu_gp_seq_needed = -1UL, \ +} + +#define __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ .work = __DELAYED_WORK_INITIALIZER(name.work, NULL, 0), \ .srcu_sup = &usage_name, \ __SRCU_DEP_MAP_INIT(name) @@ -166,9 +171,7 @@ struct srcu_struct { */ #ifdef MODULE # define __DEFINE_SRCU(name, is_static) \ - static struct srcu_usage name##_srcu_usage = { \ - .lock = __SPIN_LOCK_UNLOCKED(name.lock), \ - }; \ + static struct srcu_usage name##_srcu_usage = __SRCU_USAGE_INIT(name##_srcu_usage); \ is_static struct srcu_struct name = __SRCU_STRUCT_INIT_MODULE(name, name##_srcu_usage); \ extern struct srcu_struct * const __srcu_struct_##name; \ struct srcu_struct * const __srcu_struct_##name \ @@ -176,9 +179,7 @@ struct srcu_struct { #else # define __DEFINE_SRCU(name, is_static) \ static DEFINE_PER_CPU(struct srcu_data, name##_srcu_data); \ - static struct srcu_usage name##_srcu_usage = { \ - .lock = __SPIN_LOCK_UNLOCKED(name.lock), \ - }; \ + static struct srcu_usage name##_srcu_usage = __SRCU_USAGE_INIT(name##_srcu_usage); \ is_static struct srcu_struct name = \ __SRCU_STRUCT_INIT(name, name##_srcu_usage, name##_srcu_data) #endif diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 3bf3408c7716..6c1ac78b1cfc 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -135,8 +135,8 @@ static void init_srcu_struct_data(struct srcu_struct *ssp) spin_lock_init(&ACCESS_PRIVATE(sdp, lock)); rcu_segcblist_init(&sdp->srcu_cblist); sdp->srcu_cblist_invoking = false; - sdp->srcu_gp_seq_needed = ssp->srcu_gp_seq; - sdp->srcu_gp_seq_needed_exp = ssp->srcu_gp_seq; + sdp->srcu_gp_seq_needed = ssp->srcu_sup->srcu_gp_seq; + sdp->srcu_gp_seq_needed_exp = ssp->srcu_sup->srcu_gp_seq; sdp->mynode = NULL; sdp->cpu = cpu; INIT_WORK(&sdp->work, srcu_invoke_callbacks); @@ -247,7 +247,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) mutex_init(&ssp->srcu_sup->srcu_cb_mutex); mutex_init(&ssp->srcu_sup->srcu_gp_mutex); ssp->srcu_idx = 0; - ssp->srcu_gp_seq = 0; + ssp->srcu_sup->srcu_gp_seq = 0; ssp->srcu_barrier_seq = 0; mutex_init(&ssp->srcu_barrier_mutex); atomic_set(&ssp->srcu_barrier_cpu_cnt, 0); @@ -261,8 +261,8 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) return -ENOMEM; } init_srcu_struct_data(ssp); - ssp->srcu_gp_seq_needed_exp = 0; - ssp->srcu_last_gp_end = ktime_get_mono_fast_ns(); + ssp->srcu_sup->srcu_gp_seq_needed_exp = 0; + ssp->srcu_sup->srcu_last_gp_end = ktime_get_mono_fast_ns(); if (READ_ONCE(ssp->srcu_sup->srcu_size_state) == SRCU_SIZE_SMALL && SRCU_SIZING_IS_INIT()) { if (!init_srcu_struct_nodes(ssp, GFP_ATOMIC)) { if (!ssp->sda_is_static) { @@ -275,7 +275,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) WRITE_ONCE(ssp->srcu_sup->srcu_size_state, SRCU_SIZE_BIG); } } - smp_store_release(&ssp->srcu_gp_seq_needed, 0); /* Init done. */ + smp_store_release(&ssp->srcu_sup->srcu_gp_seq_needed, 0); /* Init done. */ return 0; } @@ -402,10 +402,10 @@ static void check_init_srcu_struct(struct srcu_struct *ssp) unsigned long flags; /* The smp_load_acquire() pairs with the smp_store_release(). */ - if (!rcu_seq_state(smp_load_acquire(&ssp->srcu_gp_seq_needed))) /*^^^*/ + if (!rcu_seq_state(smp_load_acquire(&ssp->srcu_sup->srcu_gp_seq_needed))) /*^^^*/ return; /* Already initialized. */ spin_lock_irqsave_rcu_node(ssp->srcu_sup, flags); - if (!rcu_seq_state(ssp->srcu_gp_seq_needed)) { + if (!rcu_seq_state(ssp->srcu_sup->srcu_gp_seq_needed)) { spin_unlock_irqrestore_rcu_node(ssp->srcu_sup, flags); return; } @@ -616,11 +616,11 @@ static unsigned long srcu_get_delay(struct srcu_struct *ssp) unsigned long j; unsigned long jbase = SRCU_INTERVAL; - if (ULONG_CMP_LT(READ_ONCE(ssp->srcu_gp_seq), READ_ONCE(ssp->srcu_gp_seq_needed_exp))) + if (ULONG_CMP_LT(READ_ONCE(ssp->srcu_sup->srcu_gp_seq), READ_ONCE(ssp->srcu_sup->srcu_gp_seq_needed_exp))) jbase = 0; - if (rcu_seq_state(READ_ONCE(ssp->srcu_gp_seq))) { + if (rcu_seq_state(READ_ONCE(ssp->srcu_sup->srcu_gp_seq))) { j = jiffies - 1; - gpstart = READ_ONCE(ssp->srcu_gp_start); + gpstart = READ_ONCE(ssp->srcu_sup->srcu_gp_start); if (time_after(j, gpstart)) jbase += j - gpstart; if (!jbase) { @@ -656,12 +656,12 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) if (WARN_ON(rcu_segcblist_n_cbs(&sdp->srcu_cblist))) return; /* Forgot srcu_barrier(), so just leak it! */ } - if (WARN_ON(rcu_seq_state(READ_ONCE(ssp->srcu_gp_seq)) != SRCU_STATE_IDLE) || - WARN_ON(rcu_seq_current(&ssp->srcu_gp_seq) != ssp->srcu_gp_seq_needed) || + if (WARN_ON(rcu_seq_state(READ_ONCE(ssp->srcu_sup->srcu_gp_seq)) != SRCU_STATE_IDLE) || + WARN_ON(rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq) != ssp->srcu_sup->srcu_gp_seq_needed) || WARN_ON(srcu_readers_active(ssp))) { pr_info("%s: Active srcu_struct %p read state: %d gp state: %lu/%lu\n", - __func__, ssp, rcu_seq_state(READ_ONCE(ssp->srcu_gp_seq)), - rcu_seq_current(&ssp->srcu_gp_seq), ssp->srcu_gp_seq_needed); + __func__, ssp, rcu_seq_state(READ_ONCE(ssp->srcu_sup->srcu_gp_seq)), + rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq), ssp->srcu_sup->srcu_gp_seq_needed); return; /* Caller forgot to stop doing call_srcu()? */ } kfree(ssp->srcu_sup->node); @@ -775,18 +775,18 @@ static void srcu_gp_start(struct srcu_struct *ssp) else sdp = this_cpu_ptr(ssp->sda); lockdep_assert_held(&ACCESS_PRIVATE(ssp->srcu_sup, lock)); - WARN_ON_ONCE(ULONG_CMP_GE(ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)); + WARN_ON_ONCE(ULONG_CMP_GE(ssp->srcu_sup->srcu_gp_seq, ssp->srcu_sup->srcu_gp_seq_needed)); spin_lock_rcu_node(sdp); /* Interrupts already disabled. */ rcu_segcblist_advance(&sdp->srcu_cblist, - rcu_seq_current(&ssp->srcu_gp_seq)); + rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); (void)rcu_segcblist_accelerate(&sdp->srcu_cblist, - rcu_seq_snap(&ssp->srcu_gp_seq)); + rcu_seq_snap(&ssp->srcu_sup->srcu_gp_seq)); spin_unlock_rcu_node(sdp); /* Interrupts remain disabled. */ - WRITE_ONCE(ssp->srcu_gp_start, jiffies); + WRITE_ONCE(ssp->srcu_sup->srcu_gp_start, jiffies); WRITE_ONCE(ssp->srcu_n_exp_nodelay, 0); smp_mb(); /* Order prior store to ->srcu_gp_seq_needed vs. GP start. */ - rcu_seq_start(&ssp->srcu_gp_seq); - state = rcu_seq_state(ssp->srcu_gp_seq); + rcu_seq_start(&ssp->srcu_sup->srcu_gp_seq); + state = rcu_seq_state(ssp->srcu_sup->srcu_gp_seq); WARN_ON_ONCE(state != SRCU_STATE_SCAN1); } @@ -865,16 +865,16 @@ static void srcu_gp_end(struct srcu_struct *ssp) /* End the current grace period. */ spin_lock_irq_rcu_node(ssp->srcu_sup); - idx = rcu_seq_state(ssp->srcu_gp_seq); + idx = rcu_seq_state(ssp->srcu_sup->srcu_gp_seq); WARN_ON_ONCE(idx != SRCU_STATE_SCAN2); - if (ULONG_CMP_LT(READ_ONCE(ssp->srcu_gp_seq), READ_ONCE(ssp->srcu_gp_seq_needed_exp))) + if (ULONG_CMP_LT(READ_ONCE(ssp->srcu_sup->srcu_gp_seq), READ_ONCE(ssp->srcu_sup->srcu_gp_seq_needed_exp))) cbdelay = 0; - WRITE_ONCE(ssp->srcu_last_gp_end, ktime_get_mono_fast_ns()); - rcu_seq_end(&ssp->srcu_gp_seq); - gpseq = rcu_seq_current(&ssp->srcu_gp_seq); - if (ULONG_CMP_LT(ssp->srcu_gp_seq_needed_exp, gpseq)) - WRITE_ONCE(ssp->srcu_gp_seq_needed_exp, gpseq); + WRITE_ONCE(ssp->srcu_sup->srcu_last_gp_end, ktime_get_mono_fast_ns()); + rcu_seq_end(&ssp->srcu_sup->srcu_gp_seq); + gpseq = rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq); + if (ULONG_CMP_LT(ssp->srcu_sup->srcu_gp_seq_needed_exp, gpseq)) + WRITE_ONCE(ssp->srcu_sup->srcu_gp_seq_needed_exp, gpseq); spin_unlock_irq_rcu_node(ssp->srcu_sup); mutex_unlock(&ssp->srcu_sup->srcu_gp_mutex); /* A new grace period can start at this point. But only one. */ @@ -925,9 +925,9 @@ static void srcu_gp_end(struct srcu_struct *ssp) /* Start a new grace period if needed. */ spin_lock_irq_rcu_node(ssp->srcu_sup); - gpseq = rcu_seq_current(&ssp->srcu_gp_seq); + gpseq = rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq); if (!rcu_seq_state(gpseq) && - ULONG_CMP_LT(gpseq, ssp->srcu_gp_seq_needed)) { + ULONG_CMP_LT(gpseq, ssp->srcu_sup->srcu_gp_seq_needed)) { srcu_gp_start(ssp); spin_unlock_irq_rcu_node(ssp->srcu_sup); srcu_reschedule(ssp, 0); @@ -960,7 +960,7 @@ static void srcu_funnel_exp_start(struct srcu_struct *ssp, struct srcu_node *snp if (snp) for (; snp != NULL; snp = snp->srcu_parent) { sgsne = READ_ONCE(snp->srcu_gp_seq_needed_exp); - if (WARN_ON_ONCE(rcu_seq_done(&ssp->srcu_gp_seq, s)) || + if (WARN_ON_ONCE(rcu_seq_done(&ssp->srcu_sup->srcu_gp_seq, s)) || (!srcu_invl_snp_seq(sgsne) && ULONG_CMP_GE(sgsne, s))) return; spin_lock_irqsave_rcu_node(snp, flags); @@ -973,8 +973,8 @@ static void srcu_funnel_exp_start(struct srcu_struct *ssp, struct srcu_node *snp spin_unlock_irqrestore_rcu_node(snp, flags); } spin_lock_irqsave_ssp_contention(ssp, &flags); - if (ULONG_CMP_LT(ssp->srcu_gp_seq_needed_exp, s)) - WRITE_ONCE(ssp->srcu_gp_seq_needed_exp, s); + if (ULONG_CMP_LT(ssp->srcu_sup->srcu_gp_seq_needed_exp, s)) + WRITE_ONCE(ssp->srcu_sup->srcu_gp_seq_needed_exp, s); spin_unlock_irqrestore_rcu_node(ssp->srcu_sup, flags); } @@ -1010,7 +1010,7 @@ static void srcu_funnel_gp_start(struct srcu_struct *ssp, struct srcu_data *sdp, if (snp_leaf) /* Each pass through the loop does one level of the srcu_node tree. */ for (snp = snp_leaf; snp != NULL; snp = snp->srcu_parent) { - if (WARN_ON_ONCE(rcu_seq_done(&ssp->srcu_gp_seq, s)) && snp != snp_leaf) + if (WARN_ON_ONCE(rcu_seq_done(&ssp->srcu_sup->srcu_gp_seq, s)) && snp != snp_leaf) return; /* GP already done and CBs recorded. */ spin_lock_irqsave_rcu_node(snp, flags); snp_seq = snp->srcu_have_cbs[idx]; @@ -1037,20 +1037,20 @@ static void srcu_funnel_gp_start(struct srcu_struct *ssp, struct srcu_data *sdp, /* Top of tree, must ensure the grace period will be started. */ spin_lock_irqsave_ssp_contention(ssp, &flags); - if (ULONG_CMP_LT(ssp->srcu_gp_seq_needed, s)) { + if (ULONG_CMP_LT(ssp->srcu_sup->srcu_gp_seq_needed, s)) { /* * Record need for grace period s. Pair with load * acquire setting up for initialization. */ - smp_store_release(&ssp->srcu_gp_seq_needed, s); /*^^^*/ + smp_store_release(&ssp->srcu_sup->srcu_gp_seq_needed, s); /*^^^*/ } - if (!do_norm && ULONG_CMP_LT(ssp->srcu_gp_seq_needed_exp, s)) - WRITE_ONCE(ssp->srcu_gp_seq_needed_exp, s); + if (!do_norm && ULONG_CMP_LT(ssp->srcu_sup->srcu_gp_seq_needed_exp, s)) + WRITE_ONCE(ssp->srcu_sup->srcu_gp_seq_needed_exp, s); /* If grace period not already in progress, start it. */ - if (!WARN_ON_ONCE(rcu_seq_done(&ssp->srcu_gp_seq, s)) && - rcu_seq_state(ssp->srcu_gp_seq) == SRCU_STATE_IDLE) { - WARN_ON_ONCE(ULONG_CMP_GE(ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)); + if (!WARN_ON_ONCE(rcu_seq_done(&ssp->srcu_sup->srcu_gp_seq, s)) && + rcu_seq_state(ssp->srcu_sup->srcu_gp_seq) == SRCU_STATE_IDLE) { + WARN_ON_ONCE(ULONG_CMP_GE(ssp->srcu_sup->srcu_gp_seq, ssp->srcu_sup->srcu_gp_seq_needed)); srcu_gp_start(ssp); // And how can that list_add() in the "else" clause @@ -1184,18 +1184,18 @@ static bool srcu_might_be_idle(struct srcu_struct *ssp) /* First, see if enough time has passed since the last GP. */ t = ktime_get_mono_fast_ns(); - tlast = READ_ONCE(ssp->srcu_last_gp_end); + tlast = READ_ONCE(ssp->srcu_sup->srcu_last_gp_end); if (exp_holdoff == 0 || time_in_range_open(t, tlast, tlast + exp_holdoff)) return false; /* Too soon after last GP. */ /* Next, check for probable idleness. */ - curseq = rcu_seq_current(&ssp->srcu_gp_seq); + curseq = rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq); smp_mb(); /* Order ->srcu_gp_seq with ->srcu_gp_seq_needed. */ - if (ULONG_CMP_LT(curseq, READ_ONCE(ssp->srcu_gp_seq_needed))) + if (ULONG_CMP_LT(curseq, READ_ONCE(ssp->srcu_sup->srcu_gp_seq_needed))) return false; /* Grace period in progress, so not idle. */ smp_mb(); /* Order ->srcu_gp_seq with prior access. */ - if (curseq != rcu_seq_current(&ssp->srcu_gp_seq)) + if (curseq != rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)) return false; /* GP # changed, so not idle. */ return true; /* With reasonable probability, idle! */ } @@ -1238,8 +1238,8 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp, if (rhp) rcu_segcblist_enqueue(&sdp->srcu_cblist, rhp); rcu_segcblist_advance(&sdp->srcu_cblist, - rcu_seq_current(&ssp->srcu_gp_seq)); - s = rcu_seq_snap(&ssp->srcu_gp_seq); + rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); + s = rcu_seq_snap(&ssp->srcu_sup->srcu_gp_seq); (void)rcu_segcblist_accelerate(&sdp->srcu_cblist, s); if (ULONG_CMP_LT(sdp->srcu_gp_seq_needed, s)) { sdp->srcu_gp_seq_needed = s; @@ -1452,7 +1452,7 @@ unsigned long get_state_synchronize_srcu(struct srcu_struct *ssp) // Any prior manipulation of SRCU-protected data must happen // before the load from ->srcu_gp_seq. smp_mb(); - return rcu_seq_snap(&ssp->srcu_gp_seq); + return rcu_seq_snap(&ssp->srcu_sup->srcu_gp_seq); } EXPORT_SYMBOL_GPL(get_state_synchronize_srcu); @@ -1499,7 +1499,7 @@ EXPORT_SYMBOL_GPL(start_poll_synchronize_srcu); */ bool poll_state_synchronize_srcu(struct srcu_struct *ssp, unsigned long cookie) { - if (!rcu_seq_done(&ssp->srcu_gp_seq, cookie)) + if (!rcu_seq_done(&ssp->srcu_sup->srcu_gp_seq, cookie)) return false; // Ensure that the end of the SRCU grace period happens before // any subsequent code that the caller might execute. @@ -1619,16 +1619,16 @@ static void srcu_advance_state(struct srcu_struct *ssp) * The load-acquire ensures that we see the accesses performed * by the prior grace period. */ - idx = rcu_seq_state(smp_load_acquire(&ssp->srcu_gp_seq)); /* ^^^ */ + idx = rcu_seq_state(smp_load_acquire(&ssp->srcu_sup->srcu_gp_seq)); /* ^^^ */ if (idx == SRCU_STATE_IDLE) { spin_lock_irq_rcu_node(ssp->srcu_sup); - if (ULONG_CMP_GE(ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)) { - WARN_ON_ONCE(rcu_seq_state(ssp->srcu_gp_seq)); + if (ULONG_CMP_GE(ssp->srcu_sup->srcu_gp_seq, ssp->srcu_sup->srcu_gp_seq_needed)) { + WARN_ON_ONCE(rcu_seq_state(ssp->srcu_sup->srcu_gp_seq)); spin_unlock_irq_rcu_node(ssp->srcu_sup); mutex_unlock(&ssp->srcu_sup->srcu_gp_mutex); return; } - idx = rcu_seq_state(READ_ONCE(ssp->srcu_gp_seq)); + idx = rcu_seq_state(READ_ONCE(ssp->srcu_sup->srcu_gp_seq)); if (idx == SRCU_STATE_IDLE) srcu_gp_start(ssp); spin_unlock_irq_rcu_node(ssp->srcu_sup); @@ -1638,7 +1638,7 @@ static void srcu_advance_state(struct srcu_struct *ssp) } } - if (rcu_seq_state(READ_ONCE(ssp->srcu_gp_seq)) == SRCU_STATE_SCAN1) { + if (rcu_seq_state(READ_ONCE(ssp->srcu_sup->srcu_gp_seq)) == SRCU_STATE_SCAN1) { idx = 1 ^ (ssp->srcu_idx & 1); if (!try_check_zero(ssp, idx, 1)) { mutex_unlock(&ssp->srcu_sup->srcu_gp_mutex); @@ -1646,12 +1646,12 @@ static void srcu_advance_state(struct srcu_struct *ssp) } srcu_flip(ssp); spin_lock_irq_rcu_node(ssp->srcu_sup); - rcu_seq_set_state(&ssp->srcu_gp_seq, SRCU_STATE_SCAN2); + rcu_seq_set_state(&ssp->srcu_sup->srcu_gp_seq, SRCU_STATE_SCAN2); ssp->srcu_n_exp_nodelay = 0; spin_unlock_irq_rcu_node(ssp->srcu_sup); } - if (rcu_seq_state(READ_ONCE(ssp->srcu_gp_seq)) == SRCU_STATE_SCAN2) { + if (rcu_seq_state(READ_ONCE(ssp->srcu_sup->srcu_gp_seq)) == SRCU_STATE_SCAN2) { /* * SRCU read-side critical sections are normally short, @@ -1688,7 +1688,7 @@ static void srcu_invoke_callbacks(struct work_struct *work) rcu_cblist_init(&ready_cbs); spin_lock_irq_rcu_node(sdp); rcu_segcblist_advance(&sdp->srcu_cblist, - rcu_seq_current(&ssp->srcu_gp_seq)); + rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); if (sdp->srcu_cblist_invoking || !rcu_segcblist_ready_cbs(&sdp->srcu_cblist)) { spin_unlock_irq_rcu_node(sdp); @@ -1716,7 +1716,7 @@ static void srcu_invoke_callbacks(struct work_struct *work) spin_lock_irq_rcu_node(sdp); rcu_segcblist_add_len(&sdp->srcu_cblist, -len); (void)rcu_segcblist_accelerate(&sdp->srcu_cblist, - rcu_seq_snap(&ssp->srcu_gp_seq)); + rcu_seq_snap(&ssp->srcu_sup->srcu_gp_seq)); sdp->srcu_cblist_invoking = false; more = rcu_segcblist_ready_cbs(&sdp->srcu_cblist); spin_unlock_irq_rcu_node(sdp); @@ -1733,12 +1733,12 @@ static void srcu_reschedule(struct srcu_struct *ssp, unsigned long delay) bool pushgp = true; spin_lock_irq_rcu_node(ssp->srcu_sup); - if (ULONG_CMP_GE(ssp->srcu_gp_seq, ssp->srcu_gp_seq_needed)) { - if (!WARN_ON_ONCE(rcu_seq_state(ssp->srcu_gp_seq))) { + if (ULONG_CMP_GE(ssp->srcu_sup->srcu_gp_seq, ssp->srcu_sup->srcu_gp_seq_needed)) { + if (!WARN_ON_ONCE(rcu_seq_state(ssp->srcu_sup->srcu_gp_seq))) { /* All requests fulfilled, time to go idle. */ pushgp = false; } - } else if (!rcu_seq_state(ssp->srcu_gp_seq)) { + } else if (!rcu_seq_state(ssp->srcu_sup->srcu_gp_seq)) { /* Outstanding request and no GP. Start one. */ srcu_gp_start(ssp); } @@ -1784,7 +1784,7 @@ void srcutorture_get_gp_data(enum rcutorture_type test_type, if (test_type != SRCU_FLAVOR) return; *flags = 0; - *gp_seq = rcu_seq_current(&ssp->srcu_gp_seq); + *gp_seq = rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq); } EXPORT_SYMBOL_GPL(srcutorture_get_gp_data); @@ -1813,7 +1813,7 @@ void srcu_torture_stats_print(struct srcu_struct *ssp, char *tt, char *tf) if (ss_state < 0 || ss_state >= ARRAY_SIZE(srcu_size_state_name)) ss_state_idx = ARRAY_SIZE(srcu_size_state_name) - 1; pr_alert("%s%s Tree SRCU g%ld state %d (%s)", - tt, tf, rcu_seq_current(&ssp->srcu_gp_seq), ss_state, + tt, tf, rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq), ss_state, srcu_size_state_name[ss_state_idx]); if (!ssp->sda) { // Called after cleanup_srcu_struct(), perhaps. From patchwork Fri Mar 24 00:19:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74278 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp57968wrt; Thu, 23 Mar 2023 17:30:03 -0700 (PDT) X-Google-Smtp-Source: AKy350bIECQiQB/dFfK+qiJ2lqDYmKRpIX3C48iGF8cHnDgOmgUD3MsQcNZlZjUllTNWVuMGm3Wg X-Received: by 2002:a17:907:c008:b0:8ad:51e9:cd57 with SMTP id ss8-20020a170907c00800b008ad51e9cd57mr994266ejc.49.1679617803245; Thu, 23 Mar 2023 17:30:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679617803; cv=none; d=google.com; s=arc-20160816; b=RtWlxcBW7o0Uz8S1SvCgyWZl6HgdlEzZAbaRE0fADu5vD4kApLQnN7Tp0WW0KswhHj 1YPanWrK1IONFSIaORIxoscsW9mwmzYvjmnkpycnDGTpm/HXnAOsXeRBqv3rAhec7Wl0 yTvxvRLK8nrCMLe8z8eLwh43JdLT/T/akgzsy6Wk3Pfd+A8pnLkpywE8UE39LatOaZH9 KO/xv3TEjoWSJgJiBADwssU1H9tjiHHkLNBvBCZKjzUrds1z9UpW8V8qVfQ6piBaU5Si JUHqpVQrDazZvnd/QfbnRchB3ZbGkt3Cj76RJXDf5+CBJBQ/jYYK4safVR89HpGu7w5R fHlQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=z2lhkFvMr7OwZdhJTxF7/jhl2APOeYm4lK0mvBBQq7U=; b=booU4lQGY1knxwe3DWAdk81glYV1GwEOqSC5DtjbiPeNUQDedAN2uFZB/zug8RlM+w DbLlgnnxREskjgdqUjUcpP3gXP5BorMze2xYBShX/5/HoLShNCGF1/X0faUkEKJWtF/i lrTJqjIQTW/ASZMYn1mUlJ0kmIZl2q2qlYLdHm+FUSanJDozfQZhld8JCsVnGzCR/m0g 5wRCwHAGTuM1uqOfmob2jaoiYqAK9SZEmU9S4BydJeJQ0+eICZWfYitm/h1cjkwKxs08 qPxvAowV1G0sZYVJ0SjFJQkysOsaYYEvrmq30WbAR6ZEOCbrPRSbnNji+YHWiT4UZ5Kw lORw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Yfz1FPE9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gl14-20020a170906e0ce00b009324539297csi13271466ejb.208.2023.03.23.17.29.39; Thu, 23 Mar 2023 17:30:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Yfz1FPE9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231364AbjCXAUA (ORCPT + 99 others); Thu, 23 Mar 2023 20:20:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57754 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230375AbjCXATq (ORCPT ); Thu, 23 Mar 2023 20:19:46 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C51992D174; Thu, 23 Mar 2023 17:19:42 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4DA0F62926; Fri, 24 Mar 2023 00:19:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 78130C433AA; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=RoSQbzhbQELx/c2FQsLQzy2pL9pqIn71fkr9W5LQAqE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Yfz1FPE9CogRTIXNDhDP2T6oWMeLLachsiSzYRCLrvfpOqPNTnPhR7YttabfOXnO3 mHUqTIdoQ7weGQPfq/NTya4OscFlzHiK7ylZ+y03g6lLtcephz2iLLyDwvmns50TmF dWJGe2q4RwWV8j69WtR+0h6/2V5f7ZiLlwftUNppfkz375nCs/r7Xbmd6Adfhpyti7 0FAuLuXTuH5/SSYlh9nFWBguvxfANqQ11fm2axRd6V+6gBw6VXFKfqIgmgO6SAm+bR RT649DPgOLeDyJYSoySdklFcWAzHppWbq8n8svZkTtHcyasYJ4V/asyj+M9XE0LViF 5JIoMYbAa5jtQ== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id AF94B15403A4; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 11/19] srcu: Move heuristics fields from srcu_struct to srcu_usage Date: Thu, 23 Mar 2023 17:19:30 -0700 Message-Id: <20230324001938.3443499-11-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-5.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761206917593477105?= X-GMAIL-MSGID: =?utf-8?q?1761206917593477105?= This commit moves the ->srcu_size_jiffies, ->srcu_n_lock_retries, and ->srcu_n_exp_nodelay fields from the srcu_struct structure to the srcu_usage structure to reduce the size of the former in order to improve cache locality. Suggested-by: Christoph Hellwig Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 6 +++--- kernel/rcu/srcutree.c | 18 +++++++++--------- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 7458df6bbe00..82d07466da93 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -73,6 +73,9 @@ struct srcu_usage { unsigned long srcu_gp_seq_needed_exp; /* Furthest future exp GP. */ unsigned long srcu_gp_start; /* Last GP start timestamp (jiffies) */ unsigned long srcu_last_gp_end; /* Last GP end timestamp (ns) */ + unsigned long srcu_size_jiffies; /* Current contention-measurement interval. */ + unsigned long srcu_n_lock_retries; /* Contention events in current interval. */ + unsigned long srcu_n_exp_nodelay; /* # expedited no-delays in current GP phase. */ }; /* @@ -80,9 +83,6 @@ struct srcu_usage { */ struct srcu_struct { unsigned int srcu_idx; /* Current rdr array element. */ - unsigned long srcu_size_jiffies; /* Current contention-measurement interval. */ - unsigned long srcu_n_lock_retries; /* Contention events in current interval. */ - unsigned long srcu_n_exp_nodelay; /* # expedited no-delays in current GP phase. */ struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ bool sda_is_static; /* May ->sda be passed to free_percpu()? */ unsigned long srcu_barrier_seq; /* srcu_barrier seq #. */ diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 6c1ac78b1cfc..3f389b3a25aa 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -348,11 +348,11 @@ static void spin_lock_irqsave_check_contention(struct srcu_struct *ssp) if (!SRCU_SIZING_IS_CONTEND() || ssp->srcu_sup->srcu_size_state) return; j = jiffies; - if (ssp->srcu_size_jiffies != j) { - ssp->srcu_size_jiffies = j; - ssp->srcu_n_lock_retries = 0; + if (ssp->srcu_sup->srcu_size_jiffies != j) { + ssp->srcu_sup->srcu_size_jiffies = j; + ssp->srcu_sup->srcu_n_lock_retries = 0; } - if (++ssp->srcu_n_lock_retries <= small_contention_lim) + if (++ssp->srcu_sup->srcu_n_lock_retries <= small_contention_lim) return; __srcu_transition_to_big(ssp); } @@ -624,8 +624,8 @@ static unsigned long srcu_get_delay(struct srcu_struct *ssp) if (time_after(j, gpstart)) jbase += j - gpstart; if (!jbase) { - WRITE_ONCE(ssp->srcu_n_exp_nodelay, READ_ONCE(ssp->srcu_n_exp_nodelay) + 1); - if (READ_ONCE(ssp->srcu_n_exp_nodelay) > srcu_max_nodelay_phase) + WRITE_ONCE(ssp->srcu_sup->srcu_n_exp_nodelay, READ_ONCE(ssp->srcu_sup->srcu_n_exp_nodelay) + 1); + if (READ_ONCE(ssp->srcu_sup->srcu_n_exp_nodelay) > srcu_max_nodelay_phase) jbase = 1; } } @@ -783,7 +783,7 @@ static void srcu_gp_start(struct srcu_struct *ssp) rcu_seq_snap(&ssp->srcu_sup->srcu_gp_seq)); spin_unlock_rcu_node(sdp); /* Interrupts remain disabled. */ WRITE_ONCE(ssp->srcu_sup->srcu_gp_start, jiffies); - WRITE_ONCE(ssp->srcu_n_exp_nodelay, 0); + WRITE_ONCE(ssp->srcu_sup->srcu_n_exp_nodelay, 0); smp_mb(); /* Order prior store to ->srcu_gp_seq_needed vs. GP start. */ rcu_seq_start(&ssp->srcu_sup->srcu_gp_seq); state = rcu_seq_state(ssp->srcu_sup->srcu_gp_seq); @@ -1647,7 +1647,7 @@ static void srcu_advance_state(struct srcu_struct *ssp) srcu_flip(ssp); spin_lock_irq_rcu_node(ssp->srcu_sup); rcu_seq_set_state(&ssp->srcu_sup->srcu_gp_seq, SRCU_STATE_SCAN2); - ssp->srcu_n_exp_nodelay = 0; + ssp->srcu_sup->srcu_n_exp_nodelay = 0; spin_unlock_irq_rcu_node(ssp->srcu_sup); } @@ -1662,7 +1662,7 @@ static void srcu_advance_state(struct srcu_struct *ssp) mutex_unlock(&ssp->srcu_sup->srcu_gp_mutex); return; /* readers present, retry later. */ } - ssp->srcu_n_exp_nodelay = 0; + ssp->srcu_sup->srcu_n_exp_nodelay = 0; srcu_gp_end(ssp); /* Releases ->srcu_gp_mutex. */ } } From patchwork Fri Mar 24 00:19:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74280 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp60326wrt; Thu, 23 Mar 2023 17:35:55 -0700 (PDT) X-Google-Smtp-Source: AKy350ac5zkHWprsYSsaay/mRP7gogJuG9Ny9bjHIiJD24/Pd9LG/3QcVWEUqvxki56NK/Az22wK X-Received: by 2002:aa7:c1d7:0:b0:4fa:ba31:6c66 with SMTP id d23-20020aa7c1d7000000b004faba316c66mr1164427edp.42.1679618155289; Thu, 23 Mar 2023 17:35:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679618155; cv=none; d=google.com; s=arc-20160816; b=X+Vn7WuNHhmnXaxXF+U5TfJ5KOHEi/dcjG5zsaZ14vHhOvcpxqPA3wtscFP7f6FrrK hJsMOomqxiylgTUolw0zKUBlATYSC6yVcibrolOKJSZXqvV9Fh5cE3xoSB/00XtIAXKC uCJALomaaL9b1WFKdKWyXKro/5rcHNBcJp1Z+m3l2A2w6Cs2XRXBESmh1SVrTyYps+wt RHkKhzVH+AySKsW0TaX46jbV6WzJ1PCo5g5+8a//EXQvCTMFqQUMH/rVHGUqv72yDI07 gz51kIxvN1IBdQf+xo5c9pT519Zpj7KV/Et01T0z6vvnh/RTP/o43A05PsVepUv5em4k +Bcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=aF85cbbUyN0T72a2IKMtmWipCcHZd8Ak9Drq2lz8wd8=; b=WbnPi1xAhaN7AsoWG2jA3WOf/46ZM6hILu9gohVCc7b2bdkd08UXjQ0gdTNwQUsDm5 10XBQlqtuSn8+Q80JRHtdqgl9UcxzabLjwLOGSFUFwLEMNotHuxl2WbF4TzYDSHHiWHD DvcyXyfTyh//fkdoaG46YZ7KiFQxAeoh7H0KC5RUk7VLP/GXqSJtcEnPznvplUX9EiXq eB+2z0CO705kjvT2B+A8qOBADuumj1h0hEDI+iZsd50FmBPn9EL/xlC61XwrXTP6Kkom 7zEi1BEzRIKXM7V0il3/E1eZ3b4U4Xfb9Sbqnm8ycqU+jdiu9GnJ1ttWv5tV6DojAp8W MHkg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=g2MtUvvi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b15-20020aa7c90f000000b00501d50c1416si9694009edt.411.2023.03.23.17.35.28; Thu, 23 Mar 2023 17:35:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=g2MtUvvi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231461AbjCXAUG (ORCPT + 99 others); Thu, 23 Mar 2023 20:20:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231206AbjCXATs (ORCPT ); Thu, 23 Mar 2023 20:19:48 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1AC922DE65; Thu, 23 Mar 2023 17:19:43 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7AE6D62929; Fri, 24 Mar 2023 00:19:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7F2EDC433AC; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=l0KonFEulcToOd8dBD5wwjfw4yz2wDL/Nm7i9FpxVDI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=g2MtUvviljmMLyxWmLKa7HWhACub3Fq+C+FCDyUL3exH403jVAko+RiNnT+MSoeSA hdgLtw8nTREUcp/jiVtKghd59cNElVIEuheqai2P/89fE+MKi/jkuRvRNTtvdc5uVG EDU1VrQ0WwAP47aALarNeAxnFSP3hb1IF5nlIC9kCcMyK62oYpo4Q5prVnJj/PXwqK jRxnd1oe2HX1jBe0UpNZlPWMDKnpvXum11E/j0yWOQTTCSSVeuko8ZzHpfuVuZItWM yhdOAzVsBDz34dPt18MRUFcppImxkTd2xcliH7x8Aum+dx4tAMSJiVmCybbkW1FtV0 A5jZBKkwGbV/w== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id B311615403A5; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 12/19] srcu: Move ->sda_is_static from srcu_struct to srcu_usage Date: Thu, 23 Mar 2023 17:19:31 -0700 Message-Id: <20230324001938.3443499-12-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-5.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761207286984320848?= X-GMAIL-MSGID: =?utf-8?q?1761207286984320848?= This commit moves the ->sda_is_static field from the srcu_struct structure to the srcu_usage structure to reduce the size of the former in order to improve cache locality. Suggested-by: Christoph Hellwig Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 2 +- kernel/rcu/srcutree.c | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 82d07466da93..ca48b97d9f3b 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -76,6 +76,7 @@ struct srcu_usage { unsigned long srcu_size_jiffies; /* Current contention-measurement interval. */ unsigned long srcu_n_lock_retries; /* Contention events in current interval. */ unsigned long srcu_n_exp_nodelay; /* # expedited no-delays in current GP phase. */ + bool sda_is_static; /* May ->sda be passed to free_percpu()? */ }; /* @@ -84,7 +85,6 @@ struct srcu_usage { struct srcu_struct { unsigned int srcu_idx; /* Current rdr array element. */ struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ - bool sda_is_static; /* May ->sda be passed to free_percpu()? */ unsigned long srcu_barrier_seq; /* srcu_barrier seq #. */ struct mutex srcu_barrier_mutex; /* Serialize barrier ops. */ struct completion srcu_barrier_completion; diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 3f389b3a25aa..7c192f7fb559 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -252,7 +252,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) mutex_init(&ssp->srcu_barrier_mutex); atomic_set(&ssp->srcu_barrier_cpu_cnt, 0); INIT_DELAYED_WORK(&ssp->work, process_srcu); - ssp->sda_is_static = is_static; + ssp->srcu_sup->sda_is_static = is_static; if (!is_static) ssp->sda = alloc_percpu(struct srcu_data); if (!ssp->sda) { @@ -265,7 +265,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) ssp->srcu_sup->srcu_last_gp_end = ktime_get_mono_fast_ns(); if (READ_ONCE(ssp->srcu_sup->srcu_size_state) == SRCU_SIZE_SMALL && SRCU_SIZING_IS_INIT()) { if (!init_srcu_struct_nodes(ssp, GFP_ATOMIC)) { - if (!ssp->sda_is_static) { + if (!ssp->srcu_sup->sda_is_static) { free_percpu(ssp->sda); ssp->sda = NULL; kfree(ssp->srcu_sup); @@ -667,7 +667,7 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) kfree(ssp->srcu_sup->node); ssp->srcu_sup->node = NULL; ssp->srcu_sup->srcu_size_state = SRCU_SIZE_SMALL; - if (!ssp->sda_is_static) { + if (!ssp->srcu_sup->sda_is_static) { free_percpu(ssp->sda); ssp->sda = NULL; kfree(ssp->srcu_sup); From patchwork Fri Mar 24 00:19:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74279 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp58894wrt; Thu, 23 Mar 2023 17:32:00 -0700 (PDT) X-Google-Smtp-Source: AKy350bhetNyQAF+QCki0+QrnVzhx3RxGNm5nGxbQCpzlSnKHyQK1yUYnHSqI1P8/FC5QwMX7HPJ X-Received: by 2002:a17:906:2484:b0:930:8a26:eed8 with SMTP id e4-20020a170906248400b009308a26eed8mr851051ejb.71.1679617920362; Thu, 23 Mar 2023 17:32:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679617920; cv=none; d=google.com; s=arc-20160816; b=Ty3ItFWCTLIPJkMPrJwUnHqRLpCxLEow9o0/M/OGB2u5UxkMv/NQLzVLSYlW9E9RVY 5/ClODdsHsN9ZaH43lTR0ArZYg5Eb9f1yhDnRFx+fnA33haBatYEMFAnLawPpscKimaJ Mi4LOTG+q9bMeude5Yx0EJQ0Wn4+jKYcLnQnGws9icz3D2D9mdlDWMT9MXhRLA9RR562 r2GREYJGxBuEQhmndys+BqFcLrf7Qsb/l5HGCIkRyZ8MBdd9rUS4nhPn2Pkk9TbkvKcV JAfWZhsDKYy6pCvRz7SewqaEeQG8ZeOoj4hApmoUsD//uwdysFO/df1jPatxm10FSxj+ yrWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5eL0zhRwmik17ORdqu3Eo0lKkq9JuEDQfRia1Sy0mPc=; b=hB1+n8lrHqmLqyRjXmbxSpAxRbe7G2dv2IUKLpzX4cy9pCyyP45m+7D3Mt5LAz+M05 7WQLPpdxn6QTB6tSgUBzLzR/qB0lSyyqgi1YnpgHaiBkUV8VNeUoIr2d8DNfbVjfDcGe /rfbE2xepXzy6drtTLsN4O/Dq+Eq0Q4yWc9R2gwdSmojSPUK9HEnt9Gc/G6onjT2y5NQ BjzJlFFXOAN1VaHfxHag5x6gSXKhxpsZeiXjw5PceYVDV4qZd528+kzd1oOqip9Oaugd uPLw1ZIa66/Z+BDBAyrk3s8jAiTp5Hs9mbv+rXwoGtxD81f49vYY0gYbHE7m1+f/c5LQ TrIQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=DZvZzEbC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b14-20020a170906660e00b008e259a32abcsi5182910ejp.761.2023.03.23.17.31.36; Thu, 23 Mar 2023 17:32:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=DZvZzEbC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231691AbjCXAUp (ORCPT + 99 others); Thu, 23 Mar 2023 20:20:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58420 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231350AbjCXAT7 (ORCPT ); Thu, 23 Mar 2023 20:19:59 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B871D2E0DE; Thu, 23 Mar 2023 17:19:50 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id A0634B822B4; Fri, 24 Mar 2023 00:19:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8B765C433AF; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=MAdfEhnT/3cOhBFkqmfwCHXtUyR9uHGuueAhpuWvzO8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DZvZzEbCURXVTVMgdsW0b4RUQqRZQC8Gxdn3yZSP2HpWQH0YR3aadA+gWCTkLpCAb yN5LowvL9vTfqLTL2IqosQf7S6HpCFJAoHG0CJk4FlsWQzOMn2fOXq77PWtnGw45IK dnEXnSkv1G/k876/cMUVWwG52OWGNhl40NczbU0EZD7Oa2olWCOq5Fcdm/wbIaqqw6 zYjLPh24yS1gR5qTU7yEHHhOnvFZokJIWJH5yyT1gdbsuV8h0JQf/41rQ7J1ZYL5kR +CfDjQBV0nWLc0DG8ZWSIhLed2ux12RnLILG+glMaEyC98sRcM+OiTJ0+CqipveKxx BsJp3ai+R4Vyg== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id B693A15403A6; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 13/19] srcu: Move srcu_barrier() fields from srcu_struct to srcu_usage Date: Thu, 23 Mar 2023 17:19:32 -0700 Message-Id: <20230324001938.3443499-13-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-5.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761207040306398020?= X-GMAIL-MSGID: =?utf-8?q?1761207040306398020?= This commit moves the ->srcu_barrier_seq, ->srcu_barrier_mutex, ->srcu_barrier_completion, and ->srcu_barrier_cpu_cnt fields from the srcu_struct structure to the srcu_usage structure to reduce the size of the former in order to improve cache locality. Suggested-by: Christoph Hellwig Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 14 +++++++------- kernel/rcu/srcutree.c | 38 +++++++++++++++++++------------------- 2 files changed, 26 insertions(+), 26 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index ca48b97d9f3b..70f5a3abe80a 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -77,6 +77,13 @@ struct srcu_usage { unsigned long srcu_n_lock_retries; /* Contention events in current interval. */ unsigned long srcu_n_exp_nodelay; /* # expedited no-delays in current GP phase. */ bool sda_is_static; /* May ->sda be passed to free_percpu()? */ + unsigned long srcu_barrier_seq; /* srcu_barrier seq #. */ + struct mutex srcu_barrier_mutex; /* Serialize barrier ops. */ + struct completion srcu_barrier_completion; + /* Awaken barrier rq at end. */ + atomic_t srcu_barrier_cpu_cnt; /* # CPUs not yet posting a */ + /* callback for the barrier */ + /* operation. */ }; /* @@ -85,13 +92,6 @@ struct srcu_usage { struct srcu_struct { unsigned int srcu_idx; /* Current rdr array element. */ struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ - unsigned long srcu_barrier_seq; /* srcu_barrier seq #. */ - struct mutex srcu_barrier_mutex; /* Serialize barrier ops. */ - struct completion srcu_barrier_completion; - /* Awaken barrier rq at end. */ - atomic_t srcu_barrier_cpu_cnt; /* # CPUs not yet posting a */ - /* callback for the barrier */ - /* operation. */ unsigned long reschedule_jiffies; unsigned long reschedule_count; struct delayed_work work; diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 7c192f7fb559..1df4a1467765 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -248,9 +248,9 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) mutex_init(&ssp->srcu_sup->srcu_gp_mutex); ssp->srcu_idx = 0; ssp->srcu_sup->srcu_gp_seq = 0; - ssp->srcu_barrier_seq = 0; - mutex_init(&ssp->srcu_barrier_mutex); - atomic_set(&ssp->srcu_barrier_cpu_cnt, 0); + ssp->srcu_sup->srcu_barrier_seq = 0; + mutex_init(&ssp->srcu_sup->srcu_barrier_mutex); + atomic_set(&ssp->srcu_sup->srcu_barrier_cpu_cnt, 0); INIT_DELAYED_WORK(&ssp->work, process_srcu); ssp->srcu_sup->sda_is_static = is_static; if (!is_static) @@ -1518,8 +1518,8 @@ static void srcu_barrier_cb(struct rcu_head *rhp) sdp = container_of(rhp, struct srcu_data, srcu_barrier_head); ssp = sdp->ssp; - if (atomic_dec_and_test(&ssp->srcu_barrier_cpu_cnt)) - complete(&ssp->srcu_barrier_completion); + if (atomic_dec_and_test(&ssp->srcu_sup->srcu_barrier_cpu_cnt)) + complete(&ssp->srcu_sup->srcu_barrier_completion); } /* @@ -1533,13 +1533,13 @@ static void srcu_barrier_cb(struct rcu_head *rhp) static void srcu_barrier_one_cpu(struct srcu_struct *ssp, struct srcu_data *sdp) { spin_lock_irq_rcu_node(sdp); - atomic_inc(&ssp->srcu_barrier_cpu_cnt); + atomic_inc(&ssp->srcu_sup->srcu_barrier_cpu_cnt); sdp->srcu_barrier_head.func = srcu_barrier_cb; debug_rcu_head_queue(&sdp->srcu_barrier_head); if (!rcu_segcblist_entrain(&sdp->srcu_cblist, &sdp->srcu_barrier_head)) { debug_rcu_head_unqueue(&sdp->srcu_barrier_head); - atomic_dec(&ssp->srcu_barrier_cpu_cnt); + atomic_dec(&ssp->srcu_sup->srcu_barrier_cpu_cnt); } spin_unlock_irq_rcu_node(sdp); } @@ -1552,20 +1552,20 @@ void srcu_barrier(struct srcu_struct *ssp) { int cpu; int idx; - unsigned long s = rcu_seq_snap(&ssp->srcu_barrier_seq); + unsigned long s = rcu_seq_snap(&ssp->srcu_sup->srcu_barrier_seq); check_init_srcu_struct(ssp); - mutex_lock(&ssp->srcu_barrier_mutex); - if (rcu_seq_done(&ssp->srcu_barrier_seq, s)) { + mutex_lock(&ssp->srcu_sup->srcu_barrier_mutex); + if (rcu_seq_done(&ssp->srcu_sup->srcu_barrier_seq, s)) { smp_mb(); /* Force ordering following return. */ - mutex_unlock(&ssp->srcu_barrier_mutex); + mutex_unlock(&ssp->srcu_sup->srcu_barrier_mutex); return; /* Someone else did our work for us. */ } - rcu_seq_start(&ssp->srcu_barrier_seq); - init_completion(&ssp->srcu_barrier_completion); + rcu_seq_start(&ssp->srcu_sup->srcu_barrier_seq); + init_completion(&ssp->srcu_sup->srcu_barrier_completion); /* Initial count prevents reaching zero until all CBs are posted. */ - atomic_set(&ssp->srcu_barrier_cpu_cnt, 1); + atomic_set(&ssp->srcu_sup->srcu_barrier_cpu_cnt, 1); idx = __srcu_read_lock_nmisafe(ssp); if (smp_load_acquire(&ssp->srcu_sup->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) @@ -1576,12 +1576,12 @@ void srcu_barrier(struct srcu_struct *ssp) __srcu_read_unlock_nmisafe(ssp, idx); /* Remove the initial count, at which point reaching zero can happen. */ - if (atomic_dec_and_test(&ssp->srcu_barrier_cpu_cnt)) - complete(&ssp->srcu_barrier_completion); - wait_for_completion(&ssp->srcu_barrier_completion); + if (atomic_dec_and_test(&ssp->srcu_sup->srcu_barrier_cpu_cnt)) + complete(&ssp->srcu_sup->srcu_barrier_completion); + wait_for_completion(&ssp->srcu_sup->srcu_barrier_completion); - rcu_seq_end(&ssp->srcu_barrier_seq); - mutex_unlock(&ssp->srcu_barrier_mutex); + rcu_seq_end(&ssp->srcu_sup->srcu_barrier_seq); + mutex_unlock(&ssp->srcu_sup->srcu_barrier_mutex); } EXPORT_SYMBOL_GPL(srcu_barrier); From patchwork Fri Mar 24 00:19:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74287 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp64989wrt; Thu, 23 Mar 2023 17:49:54 -0700 (PDT) X-Google-Smtp-Source: AKy350YLOPLuuz8hRqKoClxD246tpZkxZwxU5R/nJ25iJfGw5xHoSJ0AycM0ZdCvGfgo/rg6OKm+ X-Received: by 2002:aa7:c54f:0:b0:4fb:7ad6:907a with SMTP id s15-20020aa7c54f000000b004fb7ad6907amr1394058edr.13.1679618994711; Thu, 23 Mar 2023 17:49:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679618994; cv=none; d=google.com; s=arc-20160816; b=lzV98sczO0vrwRSCJS+qKPMrypd5RQ3Qbnf7vlqRP+tlVpNYKptQZSLl1PzFthDd7A qTKvx5UZ4oHwfRHPN5gTKv/teA8Vx7W3m8joqfOwb5fyWqP5jbAaqy9WhsIDVhKJexEW zHajZUK/AlTBOIO5Dv3lL/lkmmwIVA0TThUa1Tce5ny6L8UK9hVyG4hS/zxiBM+7Tb99 8xhBQy9Qpe6M0vDggrOw7RntainlKsURUZWu19r7khhZMsF9iFBGiMKqx6YEJ8xnuVy+ l6V8IxsTwFSbSDIyKWrrVsXPpXvM5Pd6cIeSWCf8aoue+oMZheSw+h8Tj+bzvnlbwBnQ tLRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=T22cdluc1R+bZbCRq7kPOGDr/3sX6UIw5TWMWpcoSBw=; b=NVXfRSNkI0GHAMLuHrcsIqgi8R2kprAIqQqT7xS3NRjeoaBMaxHClACfzuC7CTJckk 6XBmhAQoVCd39YuzYKh4moozES/gfy419CFegbcsnDyMGxEjMyJ84QfH1UmP4yv6ACq9 fMJiPQ6FmbA9s5itl4Z7vju5picc+di7ZtYEl/YX6I51kbMUESSsMuj2hkuBYegWRx4B 1isZCAbOEyk1H7U4eL9yYPnVMrL1savPEsrBwK7MVyxvzHNDfNMAaFj6L8Gq6o/8sZBf dpyimUsXPZrYc3sY/6oo1g59AW34yxj5Jvp6oQACgk4amKARQm3jWnWucVI1fpcFUykh H9Sw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="Zi9iN/Bn"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f23-20020a056402005700b00501e2c6548csi7996001edu.489.2023.03.23.17.49.31; Thu, 23 Mar 2023 17:49:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="Zi9iN/Bn"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231665AbjCXAUg (ORCPT + 99 others); Thu, 23 Mar 2023 20:20:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231301AbjCXAT4 (ORCPT ); Thu, 23 Mar 2023 20:19:56 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5830915575; Thu, 23 Mar 2023 17:19:46 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A7AB06292C; Fri, 24 Mar 2023 00:19:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9B5D0C433EF; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=z5Ko9dauL9ZQTfvwUYHm0X/HwTNs9oz5LfSEuyWupsQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Zi9iN/BnqN8vWk5eaBPUG5Bi6wFtRUnEW66phG77T8rwiJiPq8MBNVgQjDNBSSL8O h58ljkpwD+SxGP247TYY7T9WFuA/njTe2KW3mTWqobbIISOOakFa4BRt8ANvQuitwo X/wUdkDSmQyfklz4mIXaSP/MuDHknjw9OLdC04556fFItqJVBB7TtJIW8u5MSm0/fK 7BgOGsnDj6a7dQ55VoMytqyaLjElw9/ooxN1BxudtLXPxDYdX+eeJPc4OrT5ti1bFf 0Ur5MLUQLw8UdAoQPKkrvsqDDoSLjPMx1SMkWptDNjh9cGzSQ5aEP5F4XC4eeLMSaZ gLimYs1iv5QSg== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id BA0D315403A7; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 14/19] srcu: Move work-scheduling fields from srcu_struct to srcu_usage Date: Thu, 23 Mar 2023 17:19:33 -0700 Message-Id: <20230324001938.3443499-14-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-5.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761208166918315716?= X-GMAIL-MSGID: =?utf-8?q?1761208166918315716?= This commit moves the ->reschedule_jiffies, ->reschedule_count, and ->work fields from the srcu_struct structure to the srcu_usage structure to reduce the size of the former in order to improve cache locality. However, this means that the container_of() calls cannot get a pointer to the srcu_struct because they are no longer in the srcu_struct. This issue is addressed by adding a ->srcu_ssp field in the srcu_usage structure that references the corresponding srcu_struct structure. And given the presence of the sup pointer to the srcu_usage structure, replace some ssp->srcu_usage-> instances with sup->. [ paulmck Apply feedback from kernel test robot. ] Link: https://lore.kernel.org/oe-kbuild-all/202303191400.iO5BOqka-lkp@intel.com/ Suggested-by: Christoph Hellwig Signed-off-by: Paul E. McKenney --- include/linux/srcutree.h | 9 +++++---- kernel/rcu/srcutree.c | 41 +++++++++++++++++++++------------------- 2 files changed, 27 insertions(+), 23 deletions(-) diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h index 70f5a3abe80a..8f3f72480e78 100644 --- a/include/linux/srcutree.h +++ b/include/linux/srcutree.h @@ -84,6 +84,10 @@ struct srcu_usage { atomic_t srcu_barrier_cpu_cnt; /* # CPUs not yet posting a */ /* callback for the barrier */ /* operation. */ + unsigned long reschedule_jiffies; + unsigned long reschedule_count; + struct delayed_work work; + struct srcu_struct *srcu_ssp; }; /* @@ -92,9 +96,6 @@ struct srcu_usage { struct srcu_struct { unsigned int srcu_idx; /* Current rdr array element. */ struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ - unsigned long reschedule_jiffies; - unsigned long reschedule_count; - struct delayed_work work; struct lockdep_map dep_map; struct srcu_usage *srcu_sup; /* Update-side data. */ }; @@ -132,10 +133,10 @@ struct srcu_struct { { \ .lock = __SPIN_LOCK_UNLOCKED(name.lock), \ .srcu_gp_seq_needed = -1UL, \ + .work = __DELAYED_WORK_INITIALIZER(name.work, NULL, 0), \ } #define __SRCU_STRUCT_INIT_COMMON(name, usage_name) \ - .work = __DELAYED_WORK_INITIALIZER(name.work, NULL, 0), \ .srcu_sup = &usage_name, \ __SRCU_DEP_MAP_INIT(name) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 1df4a1467765..22dc266d2090 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -251,7 +251,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) ssp->srcu_sup->srcu_barrier_seq = 0; mutex_init(&ssp->srcu_sup->srcu_barrier_mutex); atomic_set(&ssp->srcu_sup->srcu_barrier_cpu_cnt, 0); - INIT_DELAYED_WORK(&ssp->work, process_srcu); + INIT_DELAYED_WORK(&ssp->srcu_sup->work, process_srcu); ssp->srcu_sup->sda_is_static = is_static; if (!is_static) ssp->sda = alloc_percpu(struct srcu_data); @@ -275,6 +275,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp, bool is_static) WRITE_ONCE(ssp->srcu_sup->srcu_size_state, SRCU_SIZE_BIG); } } + ssp->srcu_sup->srcu_ssp = ssp; smp_store_release(&ssp->srcu_sup->srcu_gp_seq_needed, 0); /* Init done. */ return 0; } @@ -647,7 +648,7 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) return; /* Just leak it! */ if (WARN_ON(srcu_readers_active(ssp))) return; /* Just leak it! */ - flush_delayed_work(&ssp->work); + flush_delayed_work(&ssp->srcu_sup->work); for_each_possible_cpu(cpu) { struct srcu_data *sdp = per_cpu_ptr(ssp->sda, cpu); @@ -1059,10 +1060,10 @@ static void srcu_funnel_gp_start(struct srcu_struct *ssp, struct srcu_data *sdp, // can only be executed during early boot when there is only // the one boot CPU running with interrupts still disabled. if (likely(srcu_init_done)) - queue_delayed_work(rcu_gp_wq, &ssp->work, + queue_delayed_work(rcu_gp_wq, &ssp->srcu_sup->work, !!srcu_get_delay(ssp)); - else if (list_empty(&ssp->work.work.entry)) - list_add(&ssp->work.work.entry, &srcu_boot_list); + else if (list_empty(&ssp->srcu_sup->work.work.entry)) + list_add(&ssp->srcu_sup->work.work.entry, &srcu_boot_list); } spin_unlock_irqrestore_rcu_node(ssp->srcu_sup, flags); } @@ -1745,7 +1746,7 @@ static void srcu_reschedule(struct srcu_struct *ssp, unsigned long delay) spin_unlock_irq_rcu_node(ssp->srcu_sup); if (pushgp) - queue_delayed_work(rcu_gp_wq, &ssp->work, delay); + queue_delayed_work(rcu_gp_wq, &ssp->srcu_sup->work, delay); } /* @@ -1756,22 +1757,24 @@ static void process_srcu(struct work_struct *work) unsigned long curdelay; unsigned long j; struct srcu_struct *ssp; + struct srcu_usage *sup; - ssp = container_of(work, struct srcu_struct, work.work); + sup = container_of(work, struct srcu_usage, work.work); + ssp = sup->srcu_ssp; srcu_advance_state(ssp); curdelay = srcu_get_delay(ssp); if (curdelay) { - WRITE_ONCE(ssp->reschedule_count, 0); + WRITE_ONCE(sup->reschedule_count, 0); } else { j = jiffies; - if (READ_ONCE(ssp->reschedule_jiffies) == j) { - WRITE_ONCE(ssp->reschedule_count, READ_ONCE(ssp->reschedule_count) + 1); - if (READ_ONCE(ssp->reschedule_count) > srcu_max_nodelay) + if (READ_ONCE(sup->reschedule_jiffies) == j) { + WRITE_ONCE(sup->reschedule_count, READ_ONCE(sup->reschedule_count) + 1); + if (READ_ONCE(sup->reschedule_count) > srcu_max_nodelay) curdelay = 1; } else { - WRITE_ONCE(ssp->reschedule_count, 1); - WRITE_ONCE(ssp->reschedule_jiffies, j); + WRITE_ONCE(sup->reschedule_count, 1); + WRITE_ONCE(sup->reschedule_jiffies, j); } } srcu_reschedule(ssp, curdelay); @@ -1870,7 +1873,7 @@ early_initcall(srcu_bootup_announce); void __init srcu_init(void) { - struct srcu_struct *ssp; + struct srcu_usage *sup; /* Decide on srcu_struct-size strategy. */ if (SRCU_SIZING_IS(SRCU_SIZING_AUTO)) { @@ -1890,13 +1893,13 @@ void __init srcu_init(void) */ srcu_init_done = true; while (!list_empty(&srcu_boot_list)) { - ssp = list_first_entry(&srcu_boot_list, struct srcu_struct, + sup = list_first_entry(&srcu_boot_list, struct srcu_usage, work.work.entry); - list_del_init(&ssp->work.work.entry); + list_del_init(&sup->work.work.entry); if (SRCU_SIZING_IS(SRCU_SIZING_INIT) && - ssp->srcu_sup->srcu_size_state == SRCU_SIZE_SMALL) - ssp->srcu_sup->srcu_size_state = SRCU_SIZE_ALLOC; - queue_work(rcu_gp_wq, &ssp->work.work); + sup->srcu_size_state == SRCU_SIZE_SMALL) + sup->srcu_size_state = SRCU_SIZE_ALLOC; + queue_work(rcu_gp_wq, &sup->work.work); } } From patchwork Fri Mar 24 00:19:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74292 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp67951wrt; Thu, 23 Mar 2023 17:59:55 -0700 (PDT) X-Google-Smtp-Source: AKy350blT/FO28ZQQigSKgWod7fl0MEgBf6EjwHliei3DS4/F02IzOhHimo5qjdqpKbjIpBHmcXI X-Received: by 2002:a62:7904:0:b0:5df:9809:2e95 with SMTP id u4-20020a627904000000b005df98092e95mr1290538pfc.11.1679619595147; Thu, 23 Mar 2023 17:59:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679619595; cv=none; d=google.com; s=arc-20160816; b=wLxr6yh2AsDmVgPnrC1xkhE+AgeowY0YHpEHVmXU+Yz/yeay+9vLrhS5xP7LnM5dP8 kSN0TsAWhU70PrxOV1yBelHJpTlqcQPpfS9zV383yAwFAzGY5hhnncwjalYjbU7cePaU rRFN1atgMOQlW/2Kd7n371vk9UWPsY4GOof9hIZCqF4QwYQDDRPiPqufIlKEQtDBiilF SPHeAYWiq5Qn69yQuKg2ewaYIjgcZxeC1f8wZIl3pDESKh7CjepPQo2J2VZR8qfhn/q+ bh+yDGDwl7hApZUMtRtpLuX2TUCAoYEG954WLjgLt1mjwbWvII/0NMo3ZFCUCaQC/39/ zJOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ZNGeluT5aedVXH1cEwsBQPOQhag2nD9X+FhRnCz/DIU=; b=ZaGfiQiUYp8yuhMvZyGSKd/aUU+kSNG3CYXim3igAmvs/G/oHl/ql7XmXRsVRoU46N 0EA8KTg10LNw2PHuzraDnVc2g1SnrdtH5rp60/5T7WzaZonVd/qNPuzmsh0w0Nlx2OAv 8ClfEeiFLy7/oBIBJcpz0vZo6KItZ/T3xZVRLzJcBgp5BASaNCtRwLhDT/Hct0UeBNQN kwWHmY/+uWVB/Y+NV3ARzHO508whYfTKiZGZhgF5dvbX//mBEz9NzyepnyKRx7c8Ewhi IGJkt0/Mb6gd/NyY+vEE5s/0TDZ8gzeQJDEqRdpYOjXIXbq8MMg6Hu8EwYBxPMmC0Tgg vwbw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=AQhNQzGk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a14-20020a634d0e000000b00503829012b7si19758024pgb.355.2023.03.23.17.59.34; Thu, 23 Mar 2023 17:59:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=AQhNQzGk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231698AbjCXAUr (ORCPT + 99 others); Thu, 23 Mar 2023 20:20:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58416 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231346AbjCXAT7 (ORCPT ); Thu, 23 Mar 2023 20:19:59 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B843C23C6A; Thu, 23 Mar 2023 17:19:50 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id B50C2B822B7; Fri, 24 Mar 2023 00:19:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A558EC433B0; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=NdzYQ3Je9Z5Q5tvkS8TNBgXRfqR10wf6pC1f2vVyHzA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AQhNQzGkkOYUSB/savXCPXJ8MgV7N9T2fyxbdc9gzd9ikLSLEs3D1Ly2vM+MM3+Cb 9tnt+pjS5xOBBR7aQZVhhhZk7zEM07G7yj8EjuLe0fVSXgE5X8RBwnMp+63VtGJ7L+ qRNfjhgeQzwG8Aoq/8xeEzPczsdjs+fckzmnxrewG5hv9XifBT0L20qG3FCSqwn9Kw ILowEqFrJyc8seOceGXFEZZM9uwq7s0NfSYiIFNzWI0wCdcnnm1eRIWiTkrubKjhzX 3OJtTeKMOWkJZbayyPKsuiiXHjyVmIVTV/YC7zYqGLBW7sWsN1ikMnizPP6r7flX5/ XU8DlBMFWsvpw== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id BD6EE15403A8; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 15/19] srcu: Fix long lines in srcu_get_delay() Date: Thu, 23 Mar 2023 17:19:34 -0700 Message-Id: <20230324001938.3443499-15-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-5.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761208796200133690?= X-GMAIL-MSGID: =?utf-8?q?1761208796200133690?= This commit creates an srcu_usage pointer named "sup" as a shorter synonym for the "ssp->srcu_sup" that was bloating several lines of code. Signed-off-by: Paul E. McKenney Cc: Christoph Hellwig --- kernel/rcu/srcutree.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 22dc266d2090..f6bb9fbe1b9c 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -616,17 +616,18 @@ static unsigned long srcu_get_delay(struct srcu_struct *ssp) unsigned long gpstart; unsigned long j; unsigned long jbase = SRCU_INTERVAL; + struct srcu_usage *sup = ssp->srcu_sup; - if (ULONG_CMP_LT(READ_ONCE(ssp->srcu_sup->srcu_gp_seq), READ_ONCE(ssp->srcu_sup->srcu_gp_seq_needed_exp))) + if (ULONG_CMP_LT(READ_ONCE(sup->srcu_gp_seq), READ_ONCE(sup->srcu_gp_seq_needed_exp))) jbase = 0; - if (rcu_seq_state(READ_ONCE(ssp->srcu_sup->srcu_gp_seq))) { + if (rcu_seq_state(READ_ONCE(sup->srcu_gp_seq))) { j = jiffies - 1; - gpstart = READ_ONCE(ssp->srcu_sup->srcu_gp_start); + gpstart = READ_ONCE(sup->srcu_gp_start); if (time_after(j, gpstart)) jbase += j - gpstart; if (!jbase) { - WRITE_ONCE(ssp->srcu_sup->srcu_n_exp_nodelay, READ_ONCE(ssp->srcu_sup->srcu_n_exp_nodelay) + 1); - if (READ_ONCE(ssp->srcu_sup->srcu_n_exp_nodelay) > srcu_max_nodelay_phase) + WRITE_ONCE(sup->srcu_n_exp_nodelay, READ_ONCE(sup->srcu_n_exp_nodelay) + 1); + if (READ_ONCE(sup->srcu_n_exp_nodelay) > srcu_max_nodelay_phase) jbase = 1; } } From patchwork Fri Mar 24 00:19:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74284 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp64434wrt; Thu, 23 Mar 2023 17:48:02 -0700 (PDT) X-Google-Smtp-Source: AKy350aIQay+rs8eQkNXrsu7N4dmQL6kYZZweLlI5JJV1mbMQ6MNys0s7PjoOyfy9xfy8UrSI/yP X-Received: by 2002:a17:906:3c55:b0:925:a734:a012 with SMTP id i21-20020a1709063c5500b00925a734a012mr870963ejg.12.1679618882185; Thu, 23 Mar 2023 17:48:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679618882; cv=none; d=google.com; s=arc-20160816; b=V/QsmFahIxEPct04m0hszlhCmr6ed6tNch4W+um4cSXToP//kRqdjtyObhLsqzVwlA TEkA17hbqOEYd4l7yPPn6pOyCnbncyfq6wBCpRjJj/mNFOiifBifZiBy9URWLWx7jcnJ qTFCVVS9VnOOdtxlgyQNoTnR+tZGGKTcGJp3zOQidzHT4Xpw/XLx56IvBy5QkpfKWLYl audA5FRCu6q2n3vcGp+FRokDpZ9SHBX1QXX8JK4Mdiotva76oE7orXTLIJKIILiIOBRY Er/Xf68+gvHWMOKCgVKpgEFzGMSZMPFLkz9TKXFGLyB9hagsPl7pqrxtKWgNSQnqMh2m fteQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=z49kUsG66xw2BRZ1PpIfR5fEJs+lugznHS8Y6Sedr2I=; b=Oi7MTQ1eyiPe8RVm5ISNqyOoHH9GrvjOabHtU/hxzhdpyTsi1AVJbKsYcXi1fD/EPY eK7CkaLTsIVUxO4VBFIZoysbngib1qqZAUL/Y5e9NxmYrjn6CZvhsSfHDmqbpqUTwpR5 jqWQ8QneCarwXkiLiHJ/JmtTDQpL0UKFr7QDEQZ070+gWkFffPLsnTdsVN1KXiSKqQ3h k15hn4ytxYD8QqfbbPlzxJ6+LHlMDaLM9Yt9x+GFg6AaX3uQbQ7AGjI9ddEsfqCkgada 4i6eP28dKfMPbIWp3x0iE/rUs9MYWLrgfdO+GWWaCH8y+8cE1P2Hlyoa39i+t9EnPCts dwhA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=NlAoMMGX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u5-20020a17090626c500b0093a142182f3si7780623ejc.400.2023.03.23.17.47.37; Thu, 23 Mar 2023 17:48:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=NlAoMMGX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231656AbjCXAUd (ORCPT + 99 others); Thu, 23 Mar 2023 20:20:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231231AbjCXAT4 (ORCPT ); Thu, 23 Mar 2023 20:19:56 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6328E193F7; Thu, 23 Mar 2023 17:19:46 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2365262928; Fri, 24 Mar 2023 00:19:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B2DBDC433AE; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=T/GGGc2qd6p6ruITBApJ92taK2jjXJ9En+4S9TDTB3o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NlAoMMGXEqbSWbkEiGsC26LyU9nrPu82Fmqoo7F5tlti3T3uoHlLMBZ/avS4LFQ3j r9GfwppRqIpDJOaNLgXxO4npmKCtXq3yFxZ3Tx/2Qdl5ivBTSgFnAwqfeyPsj3dYT2 M4fwLLmnb7eeKxue6VQ5z9wL1SLr+yd6fDdyCET/4FY5cMpTyW2SeLqrO7yaOtjYFL W6iKfw5DiJKacHjrGUc+yM3buLaSvkGMowADxuqnm9KqiR+MeaQc2PA+HUXDF7iI2e cu+2jMfbaGHOaElqsz4iqkSamKDSGN7kMzEohQww8kRmdD87pKgsqB9Ao8+tITunna 0JhIp8Uk659Qw== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id C0EA915403A9; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 16/19] srcu: Fix long lines in cleanup_srcu_struct() Date: Thu, 23 Mar 2023 17:19:35 -0700 Message-Id: <20230324001938.3443499-16-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-5.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761208049154870438?= X-GMAIL-MSGID: =?utf-8?q?1761208049154870438?= This commit creates an srcu_usage pointer named "sup" as a shorter synonym for the "ssp->srcu_sup" that was bloating several lines of code. Signed-off-by: Paul E. McKenney Cc: Christoph Hellwig --- kernel/rcu/srcutree.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index f6bb9fbe1b9c..fd88a98b7254 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -644,12 +644,13 @@ static unsigned long srcu_get_delay(struct srcu_struct *ssp) void cleanup_srcu_struct(struct srcu_struct *ssp) { int cpu; + struct srcu_usage *sup = ssp->srcu_sup; if (WARN_ON(!srcu_get_delay(ssp))) return; /* Just leak it! */ if (WARN_ON(srcu_readers_active(ssp))) return; /* Just leak it! */ - flush_delayed_work(&ssp->srcu_sup->work); + flush_delayed_work(&sup->work); for_each_possible_cpu(cpu) { struct srcu_data *sdp = per_cpu_ptr(ssp->sda, cpu); @@ -658,21 +659,21 @@ void cleanup_srcu_struct(struct srcu_struct *ssp) if (WARN_ON(rcu_segcblist_n_cbs(&sdp->srcu_cblist))) return; /* Forgot srcu_barrier(), so just leak it! */ } - if (WARN_ON(rcu_seq_state(READ_ONCE(ssp->srcu_sup->srcu_gp_seq)) != SRCU_STATE_IDLE) || - WARN_ON(rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq) != ssp->srcu_sup->srcu_gp_seq_needed) || + if (WARN_ON(rcu_seq_state(READ_ONCE(sup->srcu_gp_seq)) != SRCU_STATE_IDLE) || + WARN_ON(rcu_seq_current(&sup->srcu_gp_seq) != sup->srcu_gp_seq_needed) || WARN_ON(srcu_readers_active(ssp))) { pr_info("%s: Active srcu_struct %p read state: %d gp state: %lu/%lu\n", - __func__, ssp, rcu_seq_state(READ_ONCE(ssp->srcu_sup->srcu_gp_seq)), - rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq), ssp->srcu_sup->srcu_gp_seq_needed); + __func__, ssp, rcu_seq_state(READ_ONCE(sup->srcu_gp_seq)), + rcu_seq_current(&sup->srcu_gp_seq), sup->srcu_gp_seq_needed); return; /* Caller forgot to stop doing call_srcu()? */ } - kfree(ssp->srcu_sup->node); - ssp->srcu_sup->node = NULL; - ssp->srcu_sup->srcu_size_state = SRCU_SIZE_SMALL; - if (!ssp->srcu_sup->sda_is_static) { + kfree(sup->node); + sup->node = NULL; + sup->srcu_size_state = SRCU_SIZE_SMALL; + if (!sup->sda_is_static) { free_percpu(ssp->sda); ssp->sda = NULL; - kfree(ssp->srcu_sup); + kfree(sup); ssp->srcu_sup = NULL; } } From patchwork Fri Mar 24 00:19:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74273 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp56338wrt; Thu, 23 Mar 2023 17:25:35 -0700 (PDT) X-Google-Smtp-Source: AKy350YxrAa+L5Q/JRXWTPgrgVYU6dQ3vowaBU76djCSCgf1y3DW3sHBOICFPmmHAxb0pWeryMlz X-Received: by 2002:a50:fe8f:0:b0:502:1d20:b760 with SMTP id d15-20020a50fe8f000000b005021d20b760mr775193edt.12.1679617535626; Thu, 23 Mar 2023 17:25:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679617535; cv=none; d=google.com; s=arc-20160816; b=Iice1p5EsYfJLNnadBn9/fo40lcCQI9KK35clajnB53CqAdhuMsKsq0rESSS7t2TDJ bDb18amEhYdRI7GpTp12kWsjvR9nf/5V9cy8GULXsY78uEOtfXhteCOdOwS4LIs9jR/d x86p0WkQvf2D4bNe4nGATjagbx483oVr4tWXqmzlcRtnG6rSCK50v+nW4WqEDV+aysxo X88/1yDvGm4cbYk1H2OYkSpcQn2+GrS9XBVpLWT7ReS6/cjM1pueQe/C7d0uSudxhJmr T0kiNymF2fBfp5QabOdfkPZ9BwrWq6dLFX7YF9RCsHw1ioYZ9xXuaRQEu8rjd5CZgyYw XB+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=BeVGKuuSeEFw9zadumMaMfazgLiXufxYNrYt0k7UJro=; b=UEMTlE1sghJkfcCs1Z7xOFWyd6WNZKWUAUcMAp/V6hCie/JiVsnGVzPfo6WIEb58co CE3P3tW3Wpq3ohaIC1UGfkAApfnQP//yLTHU0FnPhupebu/RtJCVhpCvSAIA0+Vb/EGg ySmZSXaeKE+Y21z91a0jlCjnhliBH1VmDDlPu9Lj+fd+4+0Hk73Njm1yfj2DFdC6+ms9 vuHojUVi2bInWjyHYDJ/JsZoBHwFSluHnVqLbI52mw+wvSXO1gQ0+mNj3Iw3U4PUrsQ0 IcAj0zIq+sUYIfWA712LtSy0Br7R0m4r8PaXjJ6nR/VFNghejgzbXEWmPMSgEbNRywb2 ApXQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=JOWtHCxn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e11-20020a056402088b00b0050203525481si2745205edy.188.2023.03.23.17.25.10; Thu, 23 Mar 2023 17:25:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=JOWtHCxn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231706AbjCXAUu (ORCPT + 99 others); Thu, 23 Mar 2023 20:20:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58430 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231357AbjCXAT7 (ORCPT ); Thu, 23 Mar 2023 20:19:59 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 179812E81D; Thu, 23 Mar 2023 17:19:51 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id D5554B822B5; Fri, 24 Mar 2023 00:19:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BD355C433B4; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=BGUB2fistUJ4gqE7rwALc7ULAezsO9b+PlI8e9ct9as=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JOWtHCxnhgFIhmU3FDT2IfbmcHRtpzmIK7aszFfqC+k5EDcplZ5lMpbmLjERbZ83W dRWcyYeUuTXQVF7j7eEaO4BGkjiVvlvVkEiWEkZXGVUOYeUSgW1Yc2SWPOUrcbThMF /0cLNxoB3dcJ0y8PXNKzMsLNi5fikH3H6tf7L1cR4tW0lsVU6BOmQZE65dbKjrYQCw pX/fpUr9jb9MD9p/qwwZmYet/3D6TfCjzMU3p13XsMLfqq0ei+kYCiaTDkCaz51a0o vUHNqLMlIbsbyHpcUyqxcaU+ooVWJo67ISAdaOqLK2ahtmCX/7DvIE0PStmXcrwgaX OIdZGCRF0oiuQ== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id C44BC15403AA; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 17/19] srcu: Fix long lines in srcu_gp_end() Date: Thu, 23 Mar 2023 17:19:36 -0700 Message-Id: <20230324001938.3443499-17-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-5.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761206636993680634?= X-GMAIL-MSGID: =?utf-8?q?1761206636993680634?= This commit creates an srcu_usage pointer named "sup" as a shorter synonym for the "ssp->srcu_sup" that was bloating several lines of code. Signed-off-by: Paul E. McKenney Cc: Christoph Hellwig --- kernel/rcu/srcutree.c | 41 +++++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 20 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index fd88a98b7254..fcb2bac7bb4b 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -862,28 +862,29 @@ static void srcu_gp_end(struct srcu_struct *ssp) unsigned long sgsne; struct srcu_node *snp; int ss_state; + struct srcu_usage *sup = ssp->srcu_sup; /* Prevent more than one additional grace period. */ - mutex_lock(&ssp->srcu_sup->srcu_cb_mutex); + mutex_lock(&sup->srcu_cb_mutex); /* End the current grace period. */ - spin_lock_irq_rcu_node(ssp->srcu_sup); - idx = rcu_seq_state(ssp->srcu_sup->srcu_gp_seq); + spin_lock_irq_rcu_node(sup); + idx = rcu_seq_state(sup->srcu_gp_seq); WARN_ON_ONCE(idx != SRCU_STATE_SCAN2); - if (ULONG_CMP_LT(READ_ONCE(ssp->srcu_sup->srcu_gp_seq), READ_ONCE(ssp->srcu_sup->srcu_gp_seq_needed_exp))) + if (ULONG_CMP_LT(READ_ONCE(sup->srcu_gp_seq), READ_ONCE(sup->srcu_gp_seq_needed_exp))) cbdelay = 0; - WRITE_ONCE(ssp->srcu_sup->srcu_last_gp_end, ktime_get_mono_fast_ns()); - rcu_seq_end(&ssp->srcu_sup->srcu_gp_seq); - gpseq = rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq); - if (ULONG_CMP_LT(ssp->srcu_sup->srcu_gp_seq_needed_exp, gpseq)) - WRITE_ONCE(ssp->srcu_sup->srcu_gp_seq_needed_exp, gpseq); - spin_unlock_irq_rcu_node(ssp->srcu_sup); - mutex_unlock(&ssp->srcu_sup->srcu_gp_mutex); + WRITE_ONCE(sup->srcu_last_gp_end, ktime_get_mono_fast_ns()); + rcu_seq_end(&sup->srcu_gp_seq); + gpseq = rcu_seq_current(&sup->srcu_gp_seq); + if (ULONG_CMP_LT(sup->srcu_gp_seq_needed_exp, gpseq)) + WRITE_ONCE(sup->srcu_gp_seq_needed_exp, gpseq); + spin_unlock_irq_rcu_node(sup); + mutex_unlock(&sup->srcu_gp_mutex); /* A new grace period can start at this point. But only one. */ /* Initiate callback invocation as needed. */ - ss_state = smp_load_acquire(&ssp->srcu_sup->srcu_size_state); + ss_state = smp_load_acquire(&sup->srcu_size_state); if (ss_state < SRCU_SIZE_WAIT_BARRIER) { srcu_schedule_cbs_sdp(per_cpu_ptr(ssp->sda, get_boot_cpu_id()), cbdelay); @@ -892,7 +893,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) srcu_for_each_node_breadth_first(ssp, snp) { spin_lock_irq_rcu_node(snp); cbs = false; - last_lvl = snp >= ssp->srcu_sup->level[rcu_num_lvls - 1]; + last_lvl = snp >= sup->level[rcu_num_lvls - 1]; if (last_lvl) cbs = ss_state < SRCU_SIZE_BIG || snp->srcu_have_cbs[idx] == gpseq; snp->srcu_have_cbs[idx] = gpseq; @@ -924,18 +925,18 @@ static void srcu_gp_end(struct srcu_struct *ssp) } /* Callback initiation done, allow grace periods after next. */ - mutex_unlock(&ssp->srcu_sup->srcu_cb_mutex); + mutex_unlock(&sup->srcu_cb_mutex); /* Start a new grace period if needed. */ - spin_lock_irq_rcu_node(ssp->srcu_sup); - gpseq = rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq); + spin_lock_irq_rcu_node(sup); + gpseq = rcu_seq_current(&sup->srcu_gp_seq); if (!rcu_seq_state(gpseq) && - ULONG_CMP_LT(gpseq, ssp->srcu_sup->srcu_gp_seq_needed)) { + ULONG_CMP_LT(gpseq, sup->srcu_gp_seq_needed)) { srcu_gp_start(ssp); - spin_unlock_irq_rcu_node(ssp->srcu_sup); + spin_unlock_irq_rcu_node(sup); srcu_reschedule(ssp, 0); } else { - spin_unlock_irq_rcu_node(ssp->srcu_sup); + spin_unlock_irq_rcu_node(sup); } /* Transition to big if needed. */ @@ -943,7 +944,7 @@ static void srcu_gp_end(struct srcu_struct *ssp) if (ss_state == SRCU_SIZE_ALLOC) init_srcu_struct_nodes(ssp, GFP_KERNEL); else - smp_store_release(&ssp->srcu_sup->srcu_size_state, ss_state + 1); + smp_store_release(&sup->srcu_size_state, ss_state + 1); } } From patchwork Fri Mar 24 00:19:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74288 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp65082wrt; Thu, 23 Mar 2023 17:50:19 -0700 (PDT) X-Google-Smtp-Source: AKy350aRFDNOQ2YRsQ3nU/bI9nwielQ4C2PNUte/wWt5H9T9VoLvg36pHrVB8aHay8zM4XNXUxFp X-Received: by 2002:a17:906:578c:b0:88f:a236:69e6 with SMTP id k12-20020a170906578c00b0088fa23669e6mr900652ejq.7.1679619018987; Thu, 23 Mar 2023 17:50:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679619018; cv=none; d=google.com; s=arc-20160816; b=PweDHkth1u4FA/mH+ICtbe3aiEnD1wjZlG95uXPWOFsAfJ6c4psnL9Z23asBAuks6/ lgQnyHx/A/4GDsuawtHBhLMCxIeigbfURLfhS1iYkLk7PwIFWFZ6E53NWDE4L1klmexd l8iKjJ9uitoU4+gPLgUtNtS3unhwNwioJmw58KqKDC4a24Czg4v0MbM3nPFYwth6zCcc eVAV2CH9WBRCXKr4lzW7Ypxzt558b4LtX/G3f1uWAZn1AVzd8+AZOk1q8cpWr1uvpooK psttC2rQRlMs8vOO7nUU44RWJQzCpngyWhA9QBaZNvC4M+NRZn9dbbh0ZPmltMbW4GRZ OSqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=r2C5UffqU3WUbDmM2TqbNoRVL4JcT10ICkXo2xM2DaU=; b=GGlITYCDC1on/+E4hqK/TwEsMfrtTpIWwFeFJApiSs/DGDE7oxT4zgzAZwjDRf8ho2 hzVcXKJxJI4T4DFmrhTwbd5l2HkuIcJfw5eNkhrSv4dy6112DWsBO2BgEZieehl5NAN2 2Jp12ANwRQ3g9wtDm3iZK3Ohe+MOjfPjguYkPCS9V+I+2AjRzu5C7FYrbsstd5t5c78o MrUaKTV0Z1icjKrDugEZ+tiTZxOlH5+NCD35Y97AW+OJTAlCCFyyYB3jAt0hRfLcuMnS 5Em7yIlfPOAggkrLlRts0Jtvw2jYOdlmFTE6NlhKd1yteZvgK50wBNRPi8ZM/ZdXLSLF ramA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=VLuPzNlC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id vo10-20020a170907a80a00b00931f078ce63si12600779ejc.278.2023.03.23.17.49.54; Thu, 23 Mar 2023 17:50:18 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=VLuPzNlC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231683AbjCXAUm (ORCPT + 99 others); Thu, 23 Mar 2023 20:20:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58134 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231355AbjCXAT7 (ORCPT ); Thu, 23 Mar 2023 20:19:59 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0994A2E0E3; Thu, 23 Mar 2023 17:19:51 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id E0F84B822B8; Fri, 24 Mar 2023 00:19:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BC604C433B3; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=n01ySLfbpYhus3Mob79To2m1LPQToRsdp4kgb/5Hf30=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VLuPzNlCYuKg3TaBy/0difG1UhzPlTqt2rEuEtkPtyAaxZU3H0ZuprO6BsoMhLNyO JgmqxDdZPk1h4spbzVRTjjafiVD1bpQgETQMit9Yq/zxc38SrF0/L00TowiAQynwZG pZBIR5nIrrraXyhU6ix/xtC7+oNqcPvAOzaIr93nZoPw8xy7d3BpvRcqo7oZtUkU8k fG14zhuZxOjNsVd+iJ79V6TqA9+K2z4M+pAEjR777LowwZfgJs66XvQGKX0po4GDmz vvGqZfamClj6Hm6Yndi+TOlCftlntgYK8FFRMYfOppX2GpaDvFmjRchmttFNv30oU4 KSVTmaCeasItA== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id C78F315403AB; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 18/19] srcu: Fix long lines in srcu_funnel_gp_start() Date: Thu, 23 Mar 2023 17:19:37 -0700 Message-Id: <20230324001938.3443499-18-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761208192620745175?= X-GMAIL-MSGID: =?utf-8?q?1761208192620745175?= This commit creates an srcu_usage pointer named "sup" as a shorter synonym for the "ssp->srcu_sup" that was bloating several lines of code. Signed-off-by: Paul E. McKenney Cc: Christoph Hellwig --- kernel/rcu/srcutree.c | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index fcb2bac7bb4b..61dd47981b40 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1004,9 +1004,10 @@ static void srcu_funnel_gp_start(struct srcu_struct *ssp, struct srcu_data *sdp, struct srcu_node *snp; struct srcu_node *snp_leaf; unsigned long snp_seq; + struct srcu_usage *sup = ssp->srcu_sup; /* Ensure that snp node tree is fully initialized before traversing it */ - if (smp_load_acquire(&ssp->srcu_sup->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) + if (smp_load_acquire(&sup->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) snp_leaf = NULL; else snp_leaf = sdp->mynode; @@ -1014,7 +1015,7 @@ static void srcu_funnel_gp_start(struct srcu_struct *ssp, struct srcu_data *sdp, if (snp_leaf) /* Each pass through the loop does one level of the srcu_node tree. */ for (snp = snp_leaf; snp != NULL; snp = snp->srcu_parent) { - if (WARN_ON_ONCE(rcu_seq_done(&ssp->srcu_sup->srcu_gp_seq, s)) && snp != snp_leaf) + if (WARN_ON_ONCE(rcu_seq_done(&sup->srcu_gp_seq, s)) && snp != snp_leaf) return; /* GP already done and CBs recorded. */ spin_lock_irqsave_rcu_node(snp, flags); snp_seq = snp->srcu_have_cbs[idx]; @@ -1041,20 +1042,20 @@ static void srcu_funnel_gp_start(struct srcu_struct *ssp, struct srcu_data *sdp, /* Top of tree, must ensure the grace period will be started. */ spin_lock_irqsave_ssp_contention(ssp, &flags); - if (ULONG_CMP_LT(ssp->srcu_sup->srcu_gp_seq_needed, s)) { + if (ULONG_CMP_LT(sup->srcu_gp_seq_needed, s)) { /* * Record need for grace period s. Pair with load * acquire setting up for initialization. */ - smp_store_release(&ssp->srcu_sup->srcu_gp_seq_needed, s); /*^^^*/ + smp_store_release(&sup->srcu_gp_seq_needed, s); /*^^^*/ } - if (!do_norm && ULONG_CMP_LT(ssp->srcu_sup->srcu_gp_seq_needed_exp, s)) - WRITE_ONCE(ssp->srcu_sup->srcu_gp_seq_needed_exp, s); + if (!do_norm && ULONG_CMP_LT(sup->srcu_gp_seq_needed_exp, s)) + WRITE_ONCE(sup->srcu_gp_seq_needed_exp, s); /* If grace period not already in progress, start it. */ - if (!WARN_ON_ONCE(rcu_seq_done(&ssp->srcu_sup->srcu_gp_seq, s)) && - rcu_seq_state(ssp->srcu_sup->srcu_gp_seq) == SRCU_STATE_IDLE) { - WARN_ON_ONCE(ULONG_CMP_GE(ssp->srcu_sup->srcu_gp_seq, ssp->srcu_sup->srcu_gp_seq_needed)); + if (!WARN_ON_ONCE(rcu_seq_done(&sup->srcu_gp_seq, s)) && + rcu_seq_state(sup->srcu_gp_seq) == SRCU_STATE_IDLE) { + WARN_ON_ONCE(ULONG_CMP_GE(sup->srcu_gp_seq, sup->srcu_gp_seq_needed)); srcu_gp_start(ssp); // And how can that list_add() in the "else" clause @@ -1063,12 +1064,12 @@ static void srcu_funnel_gp_start(struct srcu_struct *ssp, struct srcu_data *sdp, // can only be executed during early boot when there is only // the one boot CPU running with interrupts still disabled. if (likely(srcu_init_done)) - queue_delayed_work(rcu_gp_wq, &ssp->srcu_sup->work, + queue_delayed_work(rcu_gp_wq, &sup->work, !!srcu_get_delay(ssp)); - else if (list_empty(&ssp->srcu_sup->work.work.entry)) - list_add(&ssp->srcu_sup->work.work.entry, &srcu_boot_list); + else if (list_empty(&sup->work.work.entry)) + list_add(&sup->work.work.entry, &srcu_boot_list); } - spin_unlock_irqrestore_rcu_node(ssp->srcu_sup, flags); + spin_unlock_irqrestore_rcu_node(sup, flags); } /* From patchwork Fri Mar 24 00:19:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 74277 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp57607wrt; Thu, 23 Mar 2023 17:29:06 -0700 (PDT) X-Google-Smtp-Source: AKy350Y6XrA68wXJlq9uEma6OXERxROKzWvRWLF+GIIGjAabFdifmiaC89G46fEHKpjNz4L3gg6H X-Received: by 2002:aa7:cb94:0:b0:4fa:bb8d:1d8e with SMTP id r20-20020aa7cb94000000b004fabb8d1d8emr1187653edt.19.1679617746110; Thu, 23 Mar 2023 17:29:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679617746; cv=none; d=google.com; s=arc-20160816; b=pQ0sPg9xSa8EhKzdHCWxumSFza1mjllF7coEoo9CPRYX4rvfor97Cfy8t9V5Ko7z8B gn/TlIaO4E5u0cD4rpr8aJ+/ItuwFA1Rqj9HcpZRpLKrXXZ44sL7xFlzw+lKJyhnJa9U HtaFl3yK1RF3dViUtVEZiMeSmN3FvltJFCyazO+goFE+i5kmCEbeNrzasct84VocQmYU UVh9WM1/BXgwwZjOkwRuGs8KzZanPSTV2F3wEk4Bxwhe/pXkfMk8NBcF4GIImC165zrd jOE2uhgwiB3PWN1D7iYU+QHk88TNGw45qwQRW+OL25dMnSZ1uFVBAp8tKysyxJLsWZmO KkPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5vwP7lNawK861+LKdrOa5LLWTcc5zxxbAZSEju/1cEI=; b=abLUTRV/ln4bAd5EFRsHRQLbKEfo17g0R7cxG3n7ToqM+Drv8HBY31qyRIS3taH7OX SYtnKmfSbUwmEoGD7gRAQAbM1Tty+AslfcCjI0AHw+xktTe7rYDs8Hv5rcpNbIOvHwc0 lQEQ2KfjAPC+NllH5e1BSqj8zgcx5dcozSLOq2IeOE5AkgSBsqJkjAc6NE6I2ns/Ekph DNAZwy8c1SS+ir2asYmoVBxKw57og2WLEm9bRmbsEukdLlu/gLn4dN6pA5JyBrrOQg2i 7rEH7u6xP0uPBu2RMgXcg7OtPTX2k6UkwjJqaYSC8fT5OWFLLLyN0wLlQwIs8U3dKD0A /1Tw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=EmlRlA7W; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d21-20020a50fb15000000b004c074367506si19179043edq.307.2023.03.23.17.28.42; Thu, 23 Mar 2023 17:29:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=EmlRlA7W; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231673AbjCXAUj (ORCPT + 99 others); Thu, 23 Mar 2023 20:20:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57944 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231320AbjCXAT6 (ORCPT ); Thu, 23 Mar 2023 20:19:58 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F2F972D174; Thu, 23 Mar 2023 17:19:46 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3B1116292F; Fri, 24 Mar 2023 00:19:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CCCF4C4339B; Fri, 24 Mar 2023 00:19:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679617181; bh=4rd83tdZ5NNo2vvnVsKePCSQIIPMOL6CP78Y+oTke8o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EmlRlA7W5p/wwSxQ6XMac/me0inaXakFPYzNL6YG+CoWtWMc/U7os7JWF9cRUozHJ TYvx4qJ/O6FAAIYhZzMBkmaiT3mFBxE7IpYHnYLsMHIc0OAnS2+uaU8CnbS0FbQjfq NZB5TM+4SIQG5kkalKO2YoCuFXt7ToBzraKaV43Ain2ECE1bAQKlc60xWEKUGHuTx8 lxyfqgfk53YjI61sla9+9IulzO7h0DzRuNofruhQLifUBvYUsYRHoq35FFk3a4zRJb mZUjUnalQDWOGakEFde3WIwcpodbScLvpIY9TBAprppmKOek0AZqQZsyTqz/3tCtgK RHaugZziaQs/g== Received: by paulmck-ThinkPad-P72.home (Postfix, from userid 1000) id CADDC15403AC; Thu, 23 Mar 2023 17:19:40 -0700 (PDT) From: "Paul E. McKenney" To: rcu@vger.kernel.org Cc: linux-kernel@vger.kernel.org, kernel-team@meta.com, rostedt@goodmis.org, hch@lst.de, "Paul E. McKenney" Subject: [PATCH RFC rcu 19/19] srcu: Remove extraneous parentheses from srcu_read_lock() etc. Date: Thu, 23 Mar 2023 17:19:38 -0700 Message-Id: <20230324001938.3443499-19-paulmck@kernel.org> X-Mailer: git-send-email 2.40.0.rc2 In-Reply-To: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> References: <3db82572-f156-4a5d-b711-841aa28bd996@paulmck-laptop> MIME-Version: 1.0 X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761206857702851541?= X-GMAIL-MSGID: =?utf-8?q?1761206857702851541?= This commit removes extraneous parentheses from srcu_read_lock(), srcu_read_lock_nmisafe(), srcu_read_unlock(), and srcu_read_unlock_nmisafe(). Looks like someone was once a macro. Signed-off-by: Paul E. McKenney Cc: Christoph Hellwig --- include/linux/srcu.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 41c4b26fb1c1..eb92a50a4599 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -212,7 +212,7 @@ static inline int srcu_read_lock(struct srcu_struct *ssp) __acquires(ssp) srcu_check_nmi_safety(ssp, false); retval = __srcu_read_lock(ssp); - srcu_lock_acquire(&(ssp)->dep_map); + srcu_lock_acquire(&ssp->dep_map); return retval; } @@ -229,7 +229,7 @@ static inline int srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp srcu_check_nmi_safety(ssp, true); retval = __srcu_read_lock_nmisafe(ssp); - rcu_lock_acquire(&(ssp)->dep_map); + rcu_lock_acquire(&ssp->dep_map); return retval; } @@ -284,7 +284,7 @@ static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx) { WARN_ON_ONCE(idx & ~0x1); srcu_check_nmi_safety(ssp, false); - srcu_lock_release(&(ssp)->dep_map); + srcu_lock_release(&ssp->dep_map); __srcu_read_unlock(ssp, idx); } @@ -300,7 +300,7 @@ static inline void srcu_read_unlock_nmisafe(struct srcu_struct *ssp, int idx) { WARN_ON_ONCE(idx & ~0x1); srcu_check_nmi_safety(ssp, true); - rcu_lock_release(&(ssp)->dep_map); + rcu_lock_release(&ssp->dep_map); __srcu_read_unlock_nmisafe(ssp, idx); }