From patchwork Tue Oct 3 23:28:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 148100 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2a8e:b0:403:3b70:6f57 with SMTP id in14csp2413245vqb; Tue, 3 Oct 2023 16:29:22 -0700 (PDT) X-Google-Smtp-Source: AGHT+IETktlNNiFl5ha7rntS9luQgqUW0YCwKb4eNcN2Ek0Moef6k08rFcA2WkH+DxpuNDtefYpR X-Received: by 2002:a17:90a:2fc8:b0:277:4be4:7a84 with SMTP id n8-20020a17090a2fc800b002774be47a84mr6257772pjm.19.1696375761823; Tue, 03 Oct 2023 16:29:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696375761; cv=none; d=google.com; s=arc-20160816; b=jSLmUOsRQOdYn6sbRhzKJeXir3qmniWkRNS6kniz4mbgoM/72Ky9B64NUaJ6vC8HXk CeBhhT3V5q7DNVk+MvOMs6CoIp7Mw8vsP9NTN2XP4yRcHOlXzXbrlPoug09TMemPXg0O UlYp8eLz1UjecduP6BdXfUHcH38jzgPCRLzwc8MSd8WdQxbF5NjKu6lyD+XyJz1C11wS nHKxi7L477X97IZpjWecRSIi4uKTvR6bVA0zvWZqJbqODsEej2mbLcXHx9Hzl3QattIt KKxgfNAtJzbjzO6AKIbopC6fkum7zdb1HB4/wGwPwk9t6Ec4smVMeO4NTriC9D5SIs5D XwKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ZMuof7S0ptwTdKK259pTP8dtsk+Rgu2JXsfSapu/puU=; fh=T6CBLmOzI04hocTAvxGzPC/84cjG79C3tLw5KlpjaeE=; b=G9/l2ueL3OetBZnN1TiK8q4Luxe3rA/FdmnVVJQaxafzZF2WO+7b1HqvKFyaRLjNV0 89EfhF51/IDUJnCNQiNKiLJCj/nHilUd6c8U5XqelX+srANuy5tSH7QMfOXNUel170zT EfYB9raps2LCLzEPjdoY+kda8Uq3hYvfEd9NtadyseFSDNzalnkvWJypWQspSks4ZFyE QFSXv4ZEW+O+xx8g6f4LUj9szMgnY37uzYzXw2rCBAQDVQ061ex2zsjIjJV1CIhtVOin UC8fRpGB4EBEtpFWeybvr5aL9FCuMD3PYElnaDAteQBG7SUixbCvDUS+vXm47wfRPQPi KUWw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=tAVXdvQD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id p11-20020a17090a348b00b002719ad11d97si270857pjb.137.2023.10.03.16.29.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 16:29:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=tAVXdvQD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 069378313024; Tue, 3 Oct 2023 16:29:21 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235505AbjJCX3T (ORCPT + 17 others); Tue, 3 Oct 2023 19:29:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52980 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236052AbjJCX3R (ORCPT ); Tue, 3 Oct 2023 19:29:17 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E77E1BD; Tue, 3 Oct 2023 16:29:13 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3D2B6C433C7; Tue, 3 Oct 2023 23:29:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1696375753; bh=SeJHCTpKmvGZu4Eri7N06Cc3UjYAkpJhCi+GE0MfaBE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tAVXdvQDOshnwP7RiEpZBQNaMyg87EVmQYfybY4RJsaRDcsnWJ0doRAHE02jHwjzl x3Gak5bzB5M4vfjOcV9bhQ5DZw4tXvNyH9fEY+4XL/HEL8l3H4K2g5WZ8ZIKnoDbs9 Vj+Xl7yNFdnDplAhdnKNV/tjbz1cnh3wIzpbihaK1eqDlOtjlXNVR9bIXNyvGfypEm ygXG2wWRF18xWrJPKAh97N1+jik/gdlmbs3Kt1t8ExrO7xDeeoJKQIwoFL44MXou5C zpDszM2snNCzdj2YqmbVEF14BN1oTysf2DnF4wNhCN7pGd5AfktnK6wgEjnJaObOJ/ xsxTD0qxTBfAw== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Yong He , Neeraj upadhyay , Joel Fernandes , Zhouyi Zhou , Boqun Feng , Uladzislau Rezki , RCU Subject: [PATCH 1/5] srcu: Fix callbacks acceleration mishandling Date: Wed, 4 Oct 2023 01:28:59 +0200 Message-ID: <20231003232903.7109-2-frederic@kernel.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231003232903.7109-1-frederic@kernel.org> References: <20231003232903.7109-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Tue, 03 Oct 2023 16:29:21 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778778910929459353 X-GMAIL-MSGID: 1778778910929459353 SRCU callbacks acceleration might fail if the preceding callbacks advance also fails. This can happen when the following steps are met: 1) The RCU_WAIT_TAIL segment has callbacks (say for gp_num 8) and the RCU_NEXT_READY_TAIL also has callbacks (say for gp_num 12). 2) The grace period for RCU_WAIT_TAIL is observed as started but not yet completed so rcu_seq_current() returns 4 + SRCU_STATE_SCAN1 = 5. 3) This value is passed to rcu_segcblist_advance() which can't move any segment forward and fails. 4) srcu_gp_start_if_needed() still proceeds with callback acceleration. But then the call to rcu_seq_snap() observes the grace period for the RCU_WAIT_TAIL segment (gp_num 8) as completed and the subsequent one for the RCU_NEXT_READY_TAIL segment as started (ie: 8 + SRCU_STATE_SCAN1 = 9) so it returns a snapshot of the next grace period, which is 16. 5) The value of 16 is passed to rcu_segcblist_accelerate() but the freshly enqueued callback in RCU_NEXT_TAIL can't move to RCU_NEXT_READY_TAIL which already has callbacks for a previous grace period (gp_num = 12). So acceleration fails. 6) Note in all these steps, srcu_invoke_callbacks() hadn't had a chance to run srcu_invoke_callbacks(). Then some very bad outcome may happen if the following happens: 7) Some other CPU races and starts the grace period number 16 before the CPU handling previous steps had a chance. Therefore srcu_gp_start() isn't called on the latter sdp to fix the acceleration leak from previous steps with a new pair of call to advance/accelerate. 8) The grace period 16 completes and srcu_invoke_callbacks() is finally called. All the callbacks from previous grace periods (8 and 12) are correctly advanced and executed but callbacks in RCU_NEXT_READY_TAIL still remain. Then rcu_segcblist_accelerate() is called with a snaphot of 20. 9) Since nothing started the grace period number 20, callbacks stay unhandled. This has been reported in real load: [3144162.608392] INFO: task kworker/136:12:252684 blocked for more than 122 seconds. [3144162.615986] Tainted: G O K 5.4.203-1-tlinux4-0011.1 #1 [3144162.623053] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [3144162.631162] kworker/136:12 D 0 252684 2 0x90004000 [3144162.631189] Workqueue: kvm-irqfd-cleanup irqfd_shutdown [kvm] [3144162.631192] Call Trace: [3144162.631202] __schedule+0x2ee/0x660 [3144162.631206] schedule+0x33/0xa0 [3144162.631209] schedule_timeout+0x1c4/0x340 [3144162.631214] ? update_load_avg+0x82/0x660 [3144162.631217] ? raw_spin_rq_lock_nested+0x1f/0x30 [3144162.631218] wait_for_completion+0x119/0x180 [3144162.631220] ? wake_up_q+0x80/0x80 [3144162.631224] __synchronize_srcu.part.19+0x81/0xb0 [3144162.631226] ? __bpf_trace_rcu_utilization+0x10/0x10 [3144162.631227] synchronize_srcu+0x5f/0xc0 [3144162.631236] irqfd_shutdown+0x3c/0xb0 [kvm] [3144162.631239] ? __schedule+0x2f6/0x660 [3144162.631243] process_one_work+0x19a/0x3a0 [3144162.631244] worker_thread+0x37/0x3a0 [3144162.631247] kthread+0x117/0x140 [3144162.631247] ? process_one_work+0x3a0/0x3a0 [3144162.631248] ? __kthread_cancel_work+0x40/0x40 [3144162.631250] ret_from_fork+0x1f/0x30 Fix this with taking the snapshot for acceleration _before_ the read of the current grace period number. The only side effect of this solution is that callbacks advancing happen then _after_ the full barrier in rcu_seq_snap(). This is not a problem because that barrier only cares about: 1) Ordering accesses of the update side before call_srcu() so they don't bleed. 2) See all the accesses prior to the grace period of the current gp_num The only things callbacks advancing need to be ordered against are carried by snp locking. Reported-by: Yong He Co-developed-by: Yong He Co-developed-by: Joel Fernandes Co-developed-by: Neeraj upadhyay Link: http://lore.kernel.org/CANZk6aR+CqZaqmMWrC2eRRPY12qAZnDZLwLnHZbNi=xXMB401g@mail.gmail.com Fixes: da915ad5cf25 ("srcu: Parallelize callback handling") Signed-off-by: Frederic Weisbecker --- kernel/rcu/srcutree.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 5602042856b1..9fab9ac36996 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1244,10 +1244,37 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp, spin_lock_irqsave_sdp_contention(sdp, &flags); if (rhp) rcu_segcblist_enqueue(&sdp->srcu_cblist, rhp); - rcu_segcblist_advance(&sdp->srcu_cblist, - rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); + /* + * The snapshot for acceleration must be taken _before_ the read of the + * current gp sequence used for advancing, otherwise advancing may fail + * and acceleration may then fail too. + * + * This could happen if: + * + * 1) The RCU_WAIT_TAIL segment has callbacks (gp_num = X + 4) and the + * RCU_NEXT_READY_TAIL also has callbacks (gp_num = X + 8). + * + * 2) The grace period for RCU_WAIT_TAIL is seen as started but not + * completed so rcu_seq_current() returns X + SRCU_STATE_SCAN1. + * + * 3) This value is passed to rcu_segcblist_advance() which can't move + * any segment forward and fails. + * + * 4) srcu_gp_start_if_needed() still proceeds with callback acceleration. + * But then the call to rcu_seq_snap() observes the grace period for the + * RCU_WAIT_TAIL segment as completed and the subsequent one for the + * RCU_NEXT_READY_TAIL segment as started (ie: X + 4 + SRCU_STATE_SCAN1) + * so it returns a snapshot of the next grace period, which is X + 12. + * + * 5) The value of X + 12 is passed to rcu_segcblist_accelerate() but the + * freshly enqueued callback in RCU_NEXT_TAIL can't move to + * RCU_NEXT_READY_TAIL which already has callbacks for a previous grace + * period (gp_num = X + 8). So acceleration fails. + */ s = rcu_seq_snap(&ssp->srcu_sup->srcu_gp_seq); - (void)rcu_segcblist_accelerate(&sdp->srcu_cblist, s); + rcu_segcblist_advance(&sdp->srcu_cblist, + rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); + WARN_ON_ONCE(!rcu_segcblist_accelerate(&sdp->srcu_cblist, s) && rhp); if (ULONG_CMP_LT(sdp->srcu_gp_seq_needed, s)) { sdp->srcu_gp_seq_needed = s; needgp = true; From patchwork Tue Oct 3 23:29:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 148101 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2a8e:b0:403:3b70:6f57 with SMTP id in14csp2413287vqb; Tue, 3 Oct 2023 16:29:26 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFCO4p12jOWRCm8u0qZQxUzFLuqT7qFv9GGOhHj6Lc0f699c7rsp23aBVW4JNTIYhLKvScq X-Received: by 2002:a05:6808:16a4:b0:3a4:8140:97e8 with SMTP id bb36-20020a05680816a400b003a4814097e8mr1145322oib.14.1696375766365; Tue, 03 Oct 2023 16:29:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696375766; cv=none; d=google.com; s=arc-20160816; b=LF/oxAaFKiJbDFdbVlhHGT39rHeJSFBXnPdbGClIUJhtKyGsNUO3WfkUrgIyDn411Q KDkwosB+xeg8GZTFDEbpnu6ZHy1pQKlzDsduKYyp4JLUCnpGckyN9rdvcjxVIUj1AGaT i60mMnT2ZOe3mxSDGycYYKIet6RfIZY5EOoSmFGHh0AJEdtUtmc45rw8Ch861I/jFzwL ZKyil56+Yvkc7v+sSnjU/3KFImtolcBm28zgCNTjtyN5NALoiEief0tdK/e06g4qgwHF oWPet+OqziV7DrnEOqfysniKsvR+PjNTlck1SOnw1GOQaLseAGJ3loF1C6QhcpMUYBia sx8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=OomvQ/ntS+F+KEZmgft8HnZu68PFoO+LAgoZegTZ1i4=; fh=F5rAnPhwm0E3NuqHck/WDxlMw3n2ra+y1fhewNj930Q=; b=mi4dpv5vmk+MqE1oJlVacIrDa2xSM9m17/L5s0+o+g7f1PfEFpBpXIkqIAWsDwoEqi id87rgMwpbX3Ds9Hv9It4bL10Yv3DfKSb6obBJZA+Y+ThUviXX4xSF1WOhEkKXS3X+HN NAgtFr6s5qB4rh5OJ02+K0Ne8FDyujO+LD0lmTnCDzzU99jyZRWZmsMc8o36wJJ+Lesl Byf8ysUQmmzKTHcdTjQmc2ZW1Z7+AaYHQRcII1HxT5eCNZLH4B4b98vs8mwO2SMKl9Sz fxHcW7LzOGv6NysuFeJUSk8dpZtM+uH2cFUq0Rgevz4gBns+OnVElsazBYyzcoJuAKRO g91Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="GdXvGbF/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id x65-20020a636344000000b0057751b7788esi2444510pgb.488.2023.10.03.16.29.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 16:29:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="GdXvGbF/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 9B8BE831301F; Tue, 3 Oct 2023 16:29:25 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235508AbjJCX3W (ORCPT + 17 others); Tue, 3 Oct 2023 19:29:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42918 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236052AbjJCX3U (ORCPT ); Tue, 3 Oct 2023 19:29:20 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98382AF; Tue, 3 Oct 2023 16:29:16 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0AB0DC433CB; Tue, 3 Oct 2023 23:29:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1696375756; bh=CgxY0LqJVEeiQkh649qa4IrU9jtDS9lKPNw7zJo199M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GdXvGbF/AWxAYuxckUhWjwbmC2g8wVhiZA+GsiwvL2w7wW+3ZfahRlEMz5eqlQiuZ A8IJPuFHWxv1q/wlKOLNIJY1WtvDfMnk8i3N7Q48BOaAgWNkNDhGw1mujdUdI2cw3y 81C8Ty5CpnNlhTT9W1J4Ucp/RoAQYbKGdM0M9gRpiDwWR9pFGqCdZPBn7WzFKFbdBX Y8SWXIfR5NbltrTaLhYEjbCq/H50C4OQPp+WNM5isy2ztzmkDsomYm/ok6XTUImG1a milVgIIcWpGmrvc69skdFZ9uWD0ycflSvBh02qpWAjAkibTQBcGs+PfDUT//aZRWL3 E5H2XdgVc5U5g== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Yong He , Neeraj upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , RCU Subject: [PATCH 2/5] srcu: Only accelerate on enqueue time Date: Wed, 4 Oct 2023 01:29:00 +0200 Message-ID: <20231003232903.7109-3-frederic@kernel.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231003232903.7109-1-frederic@kernel.org> References: <20231003232903.7109-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Tue, 03 Oct 2023 16:29:25 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778778915747167357 X-GMAIL-MSGID: 1778778915747167357 Acceleration in SRCU happens on enqueue time for each new callback. This operation is expected not to fail and therefore any similar attempt from other places shouldn't find any remaining callbacks to accelerate. Moreover accelerations performed beyond enqueue time are error prone because rcu_seq_snap() then may return the snapshot for a new grace period that is not going to be started. Remove these dangerous and needless accelerations and introduce instead assertions reporting leaking unaccelerated callbacks beyond enqueue time. Co-developed-by: Yong He Co-developed-by: Joel Fernandes Co-developed-by: Neeraj upadhyay Signed-off-by: Frederic Weisbecker Reviewed-by: Like Xu --- kernel/rcu/srcutree.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 9fab9ac36996..560e99ec5333 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -784,8 +784,7 @@ static void srcu_gp_start(struct srcu_struct *ssp) spin_lock_rcu_node(sdp); /* Interrupts already disabled. */ rcu_segcblist_advance(&sdp->srcu_cblist, rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); - (void)rcu_segcblist_accelerate(&sdp->srcu_cblist, - rcu_seq_snap(&ssp->srcu_sup->srcu_gp_seq)); + WARN_ON_ONCE(!rcu_segcblist_segempty(&sdp->srcu_cblist, RCU_NEXT_TAIL)); spin_unlock_rcu_node(sdp); /* Interrupts remain disabled. */ WRITE_ONCE(ssp->srcu_sup->srcu_gp_start, jiffies); WRITE_ONCE(ssp->srcu_sup->srcu_n_exp_nodelay, 0); @@ -1721,6 +1720,7 @@ static void srcu_invoke_callbacks(struct work_struct *work) ssp = sdp->ssp; rcu_cblist_init(&ready_cbs); spin_lock_irq_rcu_node(sdp); + WARN_ON_ONCE(!rcu_segcblist_segempty(&sdp->srcu_cblist, RCU_NEXT_TAIL)); rcu_segcblist_advance(&sdp->srcu_cblist, rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); if (sdp->srcu_cblist_invoking || @@ -1750,8 +1750,6 @@ static void srcu_invoke_callbacks(struct work_struct *work) */ spin_lock_irq_rcu_node(sdp); rcu_segcblist_add_len(&sdp->srcu_cblist, -len); - (void)rcu_segcblist_accelerate(&sdp->srcu_cblist, - rcu_seq_snap(&ssp->srcu_sup->srcu_gp_seq)); sdp->srcu_cblist_invoking = false; more = rcu_segcblist_ready_cbs(&sdp->srcu_cblist); spin_unlock_irq_rcu_node(sdp); From patchwork Tue Oct 3 23:29:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 148102 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2a8e:b0:403:3b70:6f57 with SMTP id in14csp2413322vqb; Tue, 3 Oct 2023 16:29:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHtqjM2Os7lPpdYIRsf6fTx6mLehOJzIIrK7cXelM9IOucsnN78QIrCvNoSImMaWTE+aHy/ X-Received: by 2002:a17:902:db06:b0:1c7:73aa:636 with SMTP id m6-20020a170902db0600b001c773aa0636mr998604plx.43.1696375772100; Tue, 03 Oct 2023 16:29:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696375772; cv=none; d=google.com; s=arc-20160816; b=u5NVEPfo88lFgziMn+IWnZqvyCkwYvsUDFKp8iBpzfsBcWZW/jNe8b0KpZ13ylkWwD AcLqBI0X0XoMvtPQnePV9h4lIHhMrlm6Go2dFwhJpcty+y0gG5NS/s1Qn8VHxbNIdW0F 4VLj21lRHI5/tqE4qEz22ipkXQRRoJ6x/2k1fYnY4Qey8pEwjf3gxPL5fxFldatTBw79 Y9gtiRPsyXV3Ib9aRSOWT0T6Nce9hOxi3Fexy4yUFbGDgsRFutEyYjL4Hx4r6hTg8ZRY vIbCGXLxmoJmwPfP/JfBSPBhFI5M6cTC8fTjnTbHlkqXMIN26kqdZWer7qh/GXsfcwC0 u5fw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=DZWGUkO4n24VfH8ClmxNLaBb5TyVpHF0VreKRIvjYBA=; fh=F5rAnPhwm0E3NuqHck/WDxlMw3n2ra+y1fhewNj930Q=; b=U4twLFdhjj+qeew3wWLEpaxyAR9z3yU5yR7zaQYFXdzZnCEhlmkJ/lLJHcaeEvvjWC IEr8pPY1pSMY5tiVYxTgS70YnBYw2c/1SCuvhfTubFT1Fohh+/3pB3zWufzIhTO2yOqK qtH3die5vxg0pJZyMXtNbvpwAEXXYlnvfVZAVS/Zwx9d6yZup4eGf6cQF4zKa4AgZn+0 QI/Wb2VR1NX20p7lNQ7/7XsMnD9g4dyMPpB1PnfKGJ5W+7kvUaI3mZvIIZaK91lvOyQP hf2XvlFViSz3xfZwCESdjt37351vYYV/JChRo8bT+pH++QxTK/fZKUtBTNijCbOGMQjt OK5g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=rxBFBO9f; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id e4-20020a170902cf4400b001c470c5906bsi2325215plg.221.2023.10.03.16.29.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 16:29:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=rxBFBO9f; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 5AE44831303B; Tue, 3 Oct 2023 16:29:31 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236394AbjJCX30 (ORCPT + 17 others); Tue, 3 Oct 2023 19:29:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42984 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236651AbjJCX3X (ORCPT ); Tue, 3 Oct 2023 19:29:23 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3E766DC; Tue, 3 Oct 2023 16:29:19 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A0123C433C7; Tue, 3 Oct 2023 23:29:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1696375758; bh=v3XWzp5y3b223EZI0Adeq7leNILWY4Ny2s3VXNJdVDs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rxBFBO9fexrgAIW8L6p/0lDFR+U7WAcxIocLnY5i/UnkELy4bW/aP796oq73qPyDe GgqjDQK5xFLEp4d9OxZ7X1QP5pGMLkxZDlpb4v7PNi+0s+guASOFbwUYqeCBCMaUFC lk7UIp12bc6DjYNl2ccy5N5xrHAPtnGqFQDiXFEjxGN8IEKvA9qPfwRgQzBL7HGvIx AgMOTbk04C7loWaFhVWUwVft01hR9G4WyrkiM3bnFKNyFDSkH+4bdVgsMUedtGVf7o 1PLlBmv41w4svke1Z8RSZd1rj1N0UBikaJTJq2i8hidOcZuLTXbNwmJ2gKPM2Va0Bk 4Ja4UJsw9BAKQ== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Yong He , Neeraj upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , RCU Subject: [PATCH 3/5] srcu: Remove superfluous callbacks advancing from srcu_start_gp() Date: Wed, 4 Oct 2023 01:29:01 +0200 Message-ID: <20231003232903.7109-4-frederic@kernel.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231003232903.7109-1-frederic@kernel.org> References: <20231003232903.7109-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Tue, 03 Oct 2023 16:29:31 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778778921233549329 X-GMAIL-MSGID: 1778778921233549329 Callbacks advancing on SRCU must be performed on two specific places: 1) On enqueue time in order to make room for the acceleration of the new callback. 2) On invocation time in order to move the callbacks ready to invoke. Any other callback advancing callsite is needless. Remove the remaining one in srcu_gp_start(). Co-developed-by: Yong He Co-developed-by: Joel Fernandes Co-developed-by: Neeraj upadhyay Signed-off-by: Frederic Weisbecker --- kernel/rcu/srcutree.c | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 560e99ec5333..e9356a103626 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -772,20 +772,10 @@ EXPORT_SYMBOL_GPL(__srcu_read_unlock_nmisafe); */ static void srcu_gp_start(struct srcu_struct *ssp) { - struct srcu_data *sdp; int state; - if (smp_load_acquire(&ssp->srcu_sup->srcu_size_state) < SRCU_SIZE_WAIT_BARRIER) - sdp = per_cpu_ptr(ssp->sda, get_boot_cpu_id()); - else - sdp = this_cpu_ptr(ssp->sda); lockdep_assert_held(&ACCESS_PRIVATE(ssp->srcu_sup, lock)); WARN_ON_ONCE(ULONG_CMP_GE(ssp->srcu_sup->srcu_gp_seq, ssp->srcu_sup->srcu_gp_seq_needed)); - spin_lock_rcu_node(sdp); /* Interrupts already disabled. */ - rcu_segcblist_advance(&sdp->srcu_cblist, - rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); - WARN_ON_ONCE(!rcu_segcblist_segempty(&sdp->srcu_cblist, RCU_NEXT_TAIL)); - spin_unlock_rcu_node(sdp); /* Interrupts remain disabled. */ WRITE_ONCE(ssp->srcu_sup->srcu_gp_start, jiffies); WRITE_ONCE(ssp->srcu_sup->srcu_n_exp_nodelay, 0); smp_mb(); /* Order prior store to ->srcu_gp_seq_needed vs. GP start. */ From patchwork Tue Oct 3 23:29:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 148103 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2a8e:b0:403:3b70:6f57 with SMTP id in14csp2413330vqb; Tue, 3 Oct 2023 16:29:33 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGnbyyYzRQEMa72TZMUJFfSnvyWqkJDBT1eyC+3AcgI4QvhLD1bQqHCuXkomFgsnJ8/Ctyu X-Received: by 2002:a05:6a21:9989:b0:14c:daa9:5e22 with SMTP id ve9-20020a056a21998900b0014cdaa95e22mr858910pzb.45.1696375773444; Tue, 03 Oct 2023 16:29:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696375773; cv=none; d=google.com; s=arc-20160816; b=RtvSstxJ6CN/k8VPVIxEEAbqHgRnLeqSXumBD1VJjs1VDzXcUeiLpOFt3GDLiFCLim +c2tq85TiHxGVT8PPpPC6TtwEXKiLz5+ppDeM2rm5dm6/cW2jFnSAJaZd1g0rIefd9Xy tPTPPGKnKciBh/NiSpd4LMcNX+pLqsvVRg4dESKDF1eRdolKSZ8Es+Nxxt2Bcq5iGmhm 3T2HbKumu8gKHgEPuRIFTnCY6fHM3b2vJcw0Bc0WUTZDpDBIanHFGbU/LH69b0ntj4kH n/4rwArGS1X43kO4DZiaq/NeddDb/Z54SalcGXjBtYo8dEXWdhdhg9frhIVxXEVGkw85 hAfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CkbI8qBGRRYpFx8wwdG54GwR9iHuJjuN3v1zcB47Xgg=; fh=F5rAnPhwm0E3NuqHck/WDxlMw3n2ra+y1fhewNj930Q=; b=ZfCa56J3wAojMx874A07mNrWVmcB3C22gmyWj3PD3Z9dXAhunqy+VMouUqlikn0OUk ZQaQSn92hBC5ytw3Q/tgPSnmNrXE9z0hiMXzNvRbWYtY76umIlrz2vYMNfb+/R6YDRDu Zsq40lq+5BzxVuBXtdBG1shAXWII2HaX+Lm7RMerKE7BV/JcVcoX2M8IKVjvJlZNc+l2 CohD8ZlF8WthMHa9e+tqwsvMAEuM/ijdxPdE/+cfwblhzEKGudQABJXZR0lxKbaNplfW ewf1EeVsi7oZ/046sWptfk7f3LXtBOR6balXmxSghDAXz9UDKhkRkgjCABk+HhUaEx+x hh9A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Vb8yt7rj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id m18-20020a170902db1200b001bf741231f7si2706101plx.570.2023.10.03.16.29.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 16:29:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Vb8yt7rj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id D8FC08315C8D; Tue, 3 Oct 2023 16:29:32 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236799AbjJCX33 (ORCPT + 17 others); Tue, 3 Oct 2023 19:29:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43032 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236317AbjJCX3Z (ORCPT ); Tue, 3 Oct 2023 19:29:25 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0116890; Tue, 3 Oct 2023 16:29:21 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 827CEC433CC; Tue, 3 Oct 2023 23:29:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1696375761; bh=AYpsqjGYQerYpEpgb3/3JuONsnqpe9k5KpicXU2iC5A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Vb8yt7rj1+eASMqdgd8uOd7YztQbD2LY9C8D6UKrRfCDQEWlcvcvXP+pYsO1KYl3F wKo7Ipua6EfHoRWZg6Gh9t3205hor55zobDIFfZQK+/0sBOlmNyXAzW5kyHOveRedJ tlpo2xc6XMDQkLBC/d/cAioxpFZ/Tb40pP0QU+M4nhFeRad5DIaZ7oQZLxwrmVzEvr BH2bP6bdKh3ey9tW6/I+GMMIihWm9SNc5TC0i9t83zpy4b+Ge+x4qt/Sohw3GLHgN+ LUFyxegCWNp/NlJAT09G8e1UXQu8Xg/FZnZ79+rrMLxk6hfFfzi3QJZ0bKBz97ZPSI a7srEntCpnslg== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Yong He , Neeraj upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , RCU Subject: [PATCH 4/5] srcu: No need to advance/accelerate if no callback enqueued Date: Wed, 4 Oct 2023 01:29:02 +0200 Message-ID: <20231003232903.7109-5-frederic@kernel.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231003232903.7109-1-frederic@kernel.org> References: <20231003232903.7109-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Tue, 03 Oct 2023 16:29:32 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778778922754959644 X-GMAIL-MSGID: 1778778922754959644 While in grace period start, there is nothing to accelerate and therefore no need to advance the callbacks either if no callback is to be enqueued. Spare these needless operations in this case. Signed-off-by: Frederic Weisbecker --- kernel/rcu/srcutree.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index e9356a103626..2bfc8ed1eed2 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1261,9 +1261,11 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp, * period (gp_num = X + 8). So acceleration fails. */ s = rcu_seq_snap(&ssp->srcu_sup->srcu_gp_seq); - rcu_segcblist_advance(&sdp->srcu_cblist, - rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); - WARN_ON_ONCE(!rcu_segcblist_accelerate(&sdp->srcu_cblist, s) && rhp); + if (rhp) { + rcu_segcblist_advance(&sdp->srcu_cblist, + rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); + WARN_ON_ONCE(!rcu_segcblist_accelerate(&sdp->srcu_cblist, s)); + } if (ULONG_CMP_LT(sdp->srcu_gp_seq_needed, s)) { sdp->srcu_gp_seq_needed = s; needgp = true; From patchwork Tue Oct 3 23:29:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 148104 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2a8e:b0:403:3b70:6f57 with SMTP id in14csp2413366vqb; Tue, 3 Oct 2023 16:29:39 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGk9qZJQA7AbSbfpR1B4YTOomhGVoHMghgYf4DRnO9AibgyQa3sS3g9EuwXysYD3gtQLiCJ X-Received: by 2002:a05:6358:591c:b0:143:7d73:6e66 with SMTP id g28-20020a056358591c00b001437d736e66mr1060600rwf.1.1696375778913; Tue, 03 Oct 2023 16:29:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696375778; cv=none; d=google.com; s=arc-20160816; b=mWn7APr1WaLkbZIvJXWshjLPkntXp/nmMn3+2RQHWgU8XzhWtfMp7G05Y5/rwjyNyC NYwyhHdD3WlGjBaoTRqrqg8JAo/e9A9jRRsew3zci4CvbfUDUf/mZfvhBJHWE4x94sO3 bHPFvLTL0wh08bKaI16SwpKhwHNtW1IvXN6m/LQqWrxLrtXLhcPKFBaeKMYNzh/P1WtF WtjosSZHCFnwianbCmkVgcy5HUTSJQlSIEUWmdCf1HRxBFr+pRvf3lVlWNa1mcIqVzqX WtmuGgJAQThLd9bhgOKNTIIun95At97Ihl4EWdvVjET6Qbn5FFgJtxdnTJAexGKtIVb7 ly5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=mVKbX4aKsx1H5J6w9BmWyps9Z918RvdMzTu3RurVgzU=; fh=F5rAnPhwm0E3NuqHck/WDxlMw3n2ra+y1fhewNj930Q=; b=WLGbj1Hhi2SUzcGVdo8iFrHkNOXUEXnTn8+Qhympk5Ocgm0Q73zWXdD3n80JgqHcvG my+ecKZtBcLC0jMeIjTwkko2hFYUk5BhlNQjhN9yVfB/KGo11OEu8lbtyBqyyWVpg/Gh quetN2EL4stx4hLXOQTmiW1gAU3yLoTX/bt24iiCaabxR7HPGZFEeas9QxDSxEneo5wN uz0yEnZaAEu4cVb620746pGsouWWvZvDTGMP2fZkCkM3Gephc8DA4EawtuzszkMjYxM/ 5o9o0523uu3Gm9Fql8LiSlR8zLBRFvBKYzNWb9h3Rb6SDh8Hghhde6q2eZ08LiS2JGd9 OhJQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ckZ84jL+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id a21-20020a17090aa51500b00263a2156cd5si295922pjq.30.2023.10.03.16.29.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Oct 2023 16:29:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=ckZ84jL+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 0B5F081B94D4; Tue, 3 Oct 2023 16:29:38 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237072AbjJCX3h (ORCPT + 17 others); Tue, 3 Oct 2023 19:29:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53746 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236767AbjJCX33 (ORCPT ); Tue, 3 Oct 2023 19:29:29 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D0F92F0; Tue, 3 Oct 2023 16:29:24 -0700 (PDT) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 33129C433CA; Tue, 3 Oct 2023 23:29:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1696375764; bh=0rRo8FAmb5oxl6EZiJPhw9pHIJx5I2TobFmAd6Jx3js=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ckZ84jL+5znrqWKHYEA+9b1Rw3RT0Gi+UXmpmDTGlZhoPinert8WUNxoDLU4o56nk dmedaAxQJKrj7ntNQVKbPtnO2xlcoeh9TptXuh3uHCcLUzchAgQQSAxWftjDL7dHI4 F0GlfzowXeBcgPjhVYmPfr5Vtqv1pKYzqUVAEbkdOOLw4rADsiMRppbrszlTsiYIXp najipfPoK5n//X9Kg/cvv3L1npieqZqr+QdozUZ9ALZVuzsIdeTaN32AS/NkaSNHQV vrd4UXNW8B3aGZGyz0IN4a6Eb7zGdywILKMBiwNxLEtYz+WsYVe5USMhK4ClYS+sxK PV7LRnOBhW8iw== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , Yong He , Neeraj upadhyay , Joel Fernandes , Boqun Feng , Uladzislau Rezki , RCU Subject: [PATCH 5/5] srcu: Explain why callbacks invocations can't run concurrently Date: Wed, 4 Oct 2023 01:29:03 +0200 Message-ID: <20231003232903.7109-6-frederic@kernel.org> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231003232903.7109-1-frederic@kernel.org> References: <20231003232903.7109-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Tue, 03 Oct 2023 16:29:38 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778778928978909766 X-GMAIL-MSGID: 1778778928978909766 If an SRCU barrier is queued while callbacks are running and a new callbacks invocator for the same sdp were to run concurrently, the RCU barrier might execute too early. As this requirement is non-obvious, make sure to keep a record. Signed-off-by: Frederic Weisbecker --- kernel/rcu/srcutree.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c index 2bfc8ed1eed2..0351a4e83529 100644 --- a/kernel/rcu/srcutree.c +++ b/kernel/rcu/srcutree.c @@ -1715,6 +1715,11 @@ static void srcu_invoke_callbacks(struct work_struct *work) WARN_ON_ONCE(!rcu_segcblist_segempty(&sdp->srcu_cblist, RCU_NEXT_TAIL)); rcu_segcblist_advance(&sdp->srcu_cblist, rcu_seq_current(&ssp->srcu_sup->srcu_gp_seq)); + /* + * Although this function is theoretically re-entrant, concurrent + * callbacks invocation is disallowed to avoid executing an SRCU barrier + * too early. + */ if (sdp->srcu_cblist_invoking || !rcu_segcblist_ready_cbs(&sdp->srcu_cblist)) { spin_unlock_irq_rcu_node(sdp); @@ -1745,6 +1750,7 @@ static void srcu_invoke_callbacks(struct work_struct *work) sdp->srcu_cblist_invoking = false; more = rcu_segcblist_ready_cbs(&sdp->srcu_cblist); spin_unlock_irq_rcu_node(sdp); + /* An SRCU barrier or callbacks from previous nesting work pending */ if (more) srcu_schedule_cbs_sdp(sdp, 0); }