From patchwork Wed Mar 29 16:02:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 76700 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp534696vqo; Wed, 29 Mar 2023 09:22:52 -0700 (PDT) X-Google-Smtp-Source: AKy350Z+mxAiWECuKhjAaxfiuASyIVmUSMXhYVY7oiQtaUhyOX+gcGqOw/PAhMeM6mmYQP0DvOra X-Received: by 2002:a17:907:2063:b0:92e:d6e6:d8a4 with SMTP id qp3-20020a170907206300b0092ed6e6d8a4mr19697916ejb.44.1680106972537; Wed, 29 Mar 2023 09:22:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680106972; cv=none; d=google.com; s=arc-20160816; b=cqCWF5btYlt9859kMXav29enerh1lyZy+9T7vBEcUVtTrG01TG/bk4m/Q912hTA/3O MkLATkJhkZX/ufo6/paqyDBd8CSFbvdQsMGrNFmUY88mKVcGFkdkwdj4RdBnx9Kcz2qy EZOM9fMy+WriH/zm8dIRR4XnNHw2y3IgxAo241D6V7y09Jc2jWXNWNXVtaJ4yWoEYRF7 LH8JY87cXplTskonv2dQlnsMCCfvWJ83h44JX7K88M3+VLaqNeFjVDXmWJjj+moGRRVg 7Q87L4M37o3rcElJ1pi9OgTS3hyMxhCjN3TC6EcVlCb+nYZsAZu4D5JFqC9fbVqIyJl9 QRvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1FdTF/dkJtJfmYs6OZ2NwpuUcaQOn4VutCH6oNYKohw=; b=N4MMG6v4Jclk5dIcBTHiAh5irLrhhqWUmBi5yfdTvYnqQRQ0CcGLEf8GG3fI03j2ri ARLoAKMztfwSsh42+Z8CiCV/ps/R0icKwvG/MgoR47fUEf0yWYkOtorPr4HoFY8jD6O9 bn/zprGtQY/FHjWvn4wBH5wATjgeN7YopomZZWRs/89wXiDIK7zGotyLCvdIGJi+UTXh e9jU1v/ajHO5C5DEJ6LfZFmc5q9KZEZ6cEuTwwnBzb0DT4EjiFYTf7n1g7fa9nBQyJva 8Xlzki2aNk16vJuEQhC9mcgauvPA3CIOgUZcCN7ye36+6RlBN5sjsxrcws57i9IdsBzx taTg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=GXJMVZfZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j24-20020a170906255800b008d7f2747fd8si29568426ejb.161.2023.03.29.09.22.28; Wed, 29 Mar 2023 09:22:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=GXJMVZfZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231150AbjC2QFI (ORCPT + 99 others); Wed, 29 Mar 2023 12:05:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231178AbjC2QEf (ORCPT ); Wed, 29 Mar 2023 12:04:35 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 157F47DBA; Wed, 29 Mar 2023 09:03:31 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CA5CA61DA0; Wed, 29 Mar 2023 16:02:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 76468C433EF; Wed, 29 Mar 2023 16:02:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1680105731; bh=Fp/ndTMmWDQvJYRUYSiB/n1pWGiQmprjbLIvXN1t7Yo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GXJMVZfZjTUnt6ouzt7l7m2vLdEpVbUYF3zXJObcVdJJ25rx/QAGDa+UpVUX5FE0p yr9Nrt6tM4pt81wfx84r5lPmsnUfyLbv3QrLoXbP+LziiVJ06zK3GQsjq+RFuVFwiq v/5qNkDMeWEGa054oW0J6Lk4jzXytIB0ApwtctH1DbKmdcqmNH0qiX5j553slYPOp5 aaTi7iyeF5s8UlNX+OkcbFgJMznNcCAKMSsCCYFsqremZD8TJRiiv6HxfgLeXT3yDy 7Z//tvu6PB1lD4HVnV8kem77vQWp8PlIRW8pyBKiBT1n6SzZs5RnfTlJ2C+xWJMZaU 7Qo2OdpvBGN6g== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , rcu , Uladzislau Rezki , Neeraj Upadhyay , Boqun Feng , Joel Fernandes Subject: [PATCH 1/4] rcu/nocb: Protect lazy shrinker against concurrent (de-)offloading Date: Wed, 29 Mar 2023 18:02:00 +0200 Message-Id: <20230329160203.191380-2-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230329160203.191380-1-frederic@kernel.org> References: <20230329160203.191380-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761719848720012779?= X-GMAIL-MSGID: =?utf-8?q?1761719848720012779?= The shrinker may run concurrently with callbacks (de-)offloading. As such, calling rcu_nocb_lock() is very dangerous because it does a conditional locking. The worst outcome is that rcu_nocb_lock() doesn't lock but rcu_nocb_unlock() eventually unlocks, or the reverse, creating an imbalance. Fix this with protecting against (de-)offloading using the barrier mutex. Although if the barrier mutex is contended, which should be rare, then step aside so as not to trigger a mutex VS allocation dependency chain. Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index f2280616f9d5..1a86883902ce 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1336,13 +1336,33 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) unsigned long flags; unsigned long count = 0; + /* + * Protect against concurrent (de-)offloading. Otherwise nocb locking + * may be ignored or imbalanced. + */ + if (!mutex_trylock(&rcu_state.barrier_mutex)) { + /* + * But really don't insist if barrier_mutex is contended since we + * can't guarantee that it will never engage in a dependency + * chain involving memory allocation. The lock is seldom contended + * anyway. + */ + return 0; + } + /* Snapshot count of all CPUs */ for_each_possible_cpu(cpu) { struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); - int _count = READ_ONCE(rdp->lazy_len); + int _count; + + if (!rcu_rdp_is_offloaded(rdp)) + continue; + + _count = READ_ONCE(rdp->lazy_len); if (_count == 0) continue; + rcu_nocb_lock_irqsave(rdp, flags); WRITE_ONCE(rdp->lazy_len, 0); rcu_nocb_unlock_irqrestore(rdp, flags); @@ -1352,6 +1372,9 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) if (sc->nr_to_scan <= 0) break; } + + mutex_unlock(&rcu_state.barrier_mutex); + return count ? count : SHRINK_STOP; } From patchwork Wed Mar 29 16:02:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 76704 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp537705vqo; Wed, 29 Mar 2023 09:28:13 -0700 (PDT) X-Google-Smtp-Source: AKy350arWPE8t7xe9Y2MMPsn3aUlZ5S2sEeBUq4Af0MaTQ9bZlfAFdqCVOK1wlNswnJtlipzBfx/ X-Received: by 2002:aa7:d54f:0:b0:502:3376:7872 with SMTP id u15-20020aa7d54f000000b0050233767872mr16315522edr.35.1680107293477; Wed, 29 Mar 2023 09:28:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680107293; cv=none; d=google.com; s=arc-20160816; b=prUeWon14cJUYzzU+BUk8Z/AH/Qk7wQhP8Hilzspm7oBBzHwF7SJTehaJJUBEcxPAL R4evt2f0/bOTdQc94kTchrvDeOsfx120Ci1x2T6dttx1v9c+oku5GJ7+7kcDqmio/le7 QuPc07GcTPSmujS2Pjca3AJfiLv8/LMSoK8UgQie05A7PXOKGiDImcxPuw4K6Ceb8yKy MDjbW21/77rVCvUCU7puyPcF8xmwZ7bPwfGzbvd1Ig2l1OuBm9vxh3Pj3umr1Qlt0Y8U /lvNaVb5BDI7IJ93Uhkr+gZznV2j5l9e/MSDe9d+ojoZOIFsXMufZJ1OL2ybY7K/U3nm Wq3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=v01Hw6x0CUiP4NL/z6mcjh/halOX2MBGCfK09KIAllQ=; b=XdYjNxAD/kToCq2MZfzUl73PSEqLhmMx+GxMg4AB3CDekQYTg2FpNXG8rQOMug02/U hryqA/Bki8WKlWT9lK2E5LOweLqIqKySnuR96Bxo4jClG358zrxvMef3190uxaT52o9J FdbLmWyJhvK34MS3Q7R9aJ3+8vVRlvYNhojzHRgqJQsWZvtjR/qrQa5lvXOwXyjLtpFh 4uTeJNcGHHXe38ILPnuSGbaRyDtFW1pXXtKh9msjZMxJb1Ia9JGvoVxOte2SiuV698mZ /UyDdWk8hV0LJsTO0VkF62+D47y9gMLU4foPRYu2WXTPY7C2QIC5OrOcHwNlVUfHuAhk x8FQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=mM1UAcje; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d22-20020aa7c1d6000000b004ab4f62171asi34776268edp.279.2023.03.29.09.27.50; Wed, 29 Mar 2023 09:28:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=mM1UAcje; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231223AbjC2QFL (ORCPT + 99 others); Wed, 29 Mar 2023 12:05:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231285AbjC2QEk (ORCPT ); Wed, 29 Mar 2023 12:04:40 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D2DE4C16; Wed, 29 Mar 2023 09:03:46 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id BDF39B82380; Wed, 29 Mar 2023 16:02:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ABE73C433A1; Wed, 29 Mar 2023 16:02:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1680105733; bh=qp5qTHymeNKXf/igbwuWFINn/OHqxS5wqWdXe8S1EME=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mM1UAcjeDDMImv5k9KeshViE+mnte2MLkwfhDMuMk3mam4OB+U+s9icDSTJiY6tj2 mpV0By8nybN+Dgd8/UsXyNaX0vKIF6Wsj1oYa1nO7Ro+o3qlvIQ6RmJP0DQBtpP2Vm lINjK5j4VPanlu31F6MfIammRNoXjwvehMtGoBzrQPEQMKVN2HJjL/Go+QZeE+JOA3 BCfSnQzEC4MnFn0lZJ3KjD6gofBMB+dceLNEydiTGDxuzL4Bk0Tus2PSqy3dsyVrDv GcLelC4UmxcmsywSnqEBIBbrM36mqdisHzvvceK86rBfsbDqZxqwPAgzhp18r48E2m VsKkc1l1zbseA== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , rcu , Uladzislau Rezki , Neeraj Upadhyay , Boqun Feng , Joel Fernandes Subject: [PATCH 2/4] rcu/nocb: Fix shrinker race against callback enqueuer Date: Wed, 29 Mar 2023 18:02:01 +0200 Message-Id: <20230329160203.191380-3-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230329160203.191380-1-frederic@kernel.org> References: <20230329160203.191380-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761720185243612576?= X-GMAIL-MSGID: =?utf-8?q?1761720185243612576?= The shrinker resets the lazy callbacks counter in order to trigger the pending lazy queue flush though the rcuog kthread. The counter reset is protected by the ->nocb_lock against concurrent accesses...except for one of them. Here is a list of existing synchronized readers/writer: 1) The first lazy enqueuer (incrementing ->lazy_len to 1) does so under ->nocb_lock and ->nocb_bypass_lock. 2) The further lazy enqueuers (incrementing ->lazy_len above 1) do so under ->nocb_bypass_lock _only_. 3) The lazy flush checks and resets to 0 under ->nocb_lock and ->nocb_bypass_lock. The shrinker protects its ->lazy_len reset against cases 1) and 3) but not against 2). As such, setting ->lazy_len to 0 under the ->nocb_lock may be cancelled right away by an overwrite from an enqueuer, leading rcuog to ignore the flush. To avoid that, use the proper bypass flush API which takes care of all those details. Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 1a86883902ce..c321fce2af8e 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1364,7 +1364,7 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) continue; rcu_nocb_lock_irqsave(rdp, flags); - WRITE_ONCE(rdp->lazy_len, 0); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); sc->nr_to_scan -= _count; From patchwork Wed Mar 29 16:02:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 76708 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp541318vqo; Wed, 29 Mar 2023 09:33:45 -0700 (PDT) X-Google-Smtp-Source: AK7set/I45VtYgjiwFO3n9AD61IsirWjHVGbci1TbWefqWRHC4Q/1aIE5iDW3E6I3YwNAGthHSXy X-Received: by 2002:a05:6a20:b55c:b0:da:a7db:48bb with SMTP id ev28-20020a056a20b55c00b000daa7db48bbmr16010896pzb.42.1680107625560; Wed, 29 Mar 2023 09:33:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680107625; cv=none; d=google.com; s=arc-20160816; b=hlN86BAUaBKpdUv1iOH9CAvAmaCGqZ1ZvaV75vaEd5UYXjDnIn8v7LHEuN5LakVh1a ITbEy2ciePIKiGG36bXi1xfM2phOJvjMyx0S2mWTmN1ctP6/+8VEy0t4XFOV+3VJc8O5 qgIojZpkTNxOUHRABC3CkwIyEI6swgj3L9tP5Wm4rfrN+Tg+pKZUCUwkABCCZn4pIgzP 6Tf7SNoqJf8QrvTpE+MWKNyFXygsUanHWisZml780+gvKqhJzSnuzi2VsLlGho+aezRN Z3JqAzcfVKnQ9BIXFZMauP1eYF8NX5rLCYJQ7lE1UN2EEvZtlsLu8ZoXaGwq0/h7dUrD 5gXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=pxRhWFyb27BvogwhMK7gi/x5UwosXX92hs5VIGhZfD0=; b=FK+7Ddt/8+lPBCxqF/JcD8YOz/tpf2Ft8cLiDsHfJUcHipMHK3by/QLcW65yg14wlb iXaHM26n9SIHxmw+LaMtX1b6Vu8Z/n7X1xn0KC2AP8kheJTt4xCJEvBdbS/i8q5VESZP FCoFAftEZtM/euoL9kzhX7zsikphNmrFQ3WZZ0CeqaRI9FlMgs6LQCDS1j/g8pMARxmg C0KqP8dt2ygvjAsljdkCXTjAv0kwAIPjlGKYO6SEKXAlYgYifgDmDm8vubv1/BSBhXVM RjcrW5+1NnO8TyP5L4gGWiAZOH8xPusn3yvo4CiCSiBOEsFY+qmXYlSNUVQV+wPsg4x9 J7zA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=We6Sppzs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d17-20020a056a00245100b00628030da823si23505332pfj.318.2023.03.29.09.33.32; Wed, 29 Mar 2023 09:33:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=We6Sppzs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231215AbjC2QEz (ORCPT + 99 others); Wed, 29 Mar 2023 12:04:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60208 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231582AbjC2QEU (ORCPT ); Wed, 29 Mar 2023 12:04:20 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF252558E; Wed, 29 Mar 2023 09:03:10 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3F88461DAA; Wed, 29 Mar 2023 16:02:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DFC75C4339C; Wed, 29 Mar 2023 16:02:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1680105735; bh=+h4XK6+UI6FKKHszmm1wj9jmGuHr+bG0mxcM6PK0eo8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=We6SppzsOzlK2//1daxu59Df7ANxDJyMznwP+OkNrFmQmFcLdYSH8atr1ezJd2MQj WGhWJt5bkH3VrhROVfbJh02EKGkS4HvFe7yGH7qz1L3O1hPSNGdLY0xegZuyYPbhO4 HFJ39S8in9KjgKsomrNfCm6kkGsr3ndqR6XGOtnZzBDu5NF8SysXHICJOoLTeTUZlf t8uQZIL7RyPCnaGpbfjU1TLNYO2Hj0+jgz4c93W2sJu3rtVZTMLioZzU+iPZjGmRfC D6tzE56MJESRUNbCdPEERhTIsa0krSKTMJRV4SDPnbDJ+cY+nwa+73GhrVeGcZadG2 6mVRYA++CYdug== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , rcu , Uladzislau Rezki , Neeraj Upadhyay , Boqun Feng , Joel Fernandes Subject: [PATCH 3/4] rcu/nocb: Recheck lazy callbacks under the ->nocb_lock from shrinker Date: Wed, 29 Mar 2023 18:02:02 +0200 Message-Id: <20230329160203.191380-4-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230329160203.191380-1-frederic@kernel.org> References: <20230329160203.191380-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761720533679437799?= X-GMAIL-MSGID: =?utf-8?q?1761720533679437799?= The ->lazy_len is only checked locklessly. Recheck again under the ->nocb_lock to avoid spending more time on flushing/waking if not necessary. The ->lazy_len can still increment concurrently (from 1 to infinity) but under the ->nocb_lock we at least know for sure if there are lazy callbacks at all (->lazy_len > 0). Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index c321fce2af8e..dfa9c10d6727 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1358,12 +1358,20 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) if (!rcu_rdp_is_offloaded(rdp)) continue; + if (!READ_ONCE(rdp->lazy_len)) + continue; + + rcu_nocb_lock_irqsave(rdp, flags); + /* + * Recheck under the nocb lock. Since we are not holding the bypass + * lock we may still race with increments from the enqueuer but still + * we know for sure if there is at least one lazy callback. + */ _count = READ_ONCE(rdp->lazy_len); - - if (_count == 0) + if (!_count) { + rcu_nocb_unlock_irqrestore(rdp, flags); continue; - - rcu_nocb_lock_irqsave(rdp, flags); + } WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); From patchwork Wed Mar 29 16:02:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 76703 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp537621vqo; Wed, 29 Mar 2023 09:28:04 -0700 (PDT) X-Google-Smtp-Source: AKy350YhsEzzHXsSNy38Yy+lxKSVXzSPrzMUqlCsacekpu4cj/HEidesvqYDJ0jVInJkWib9jSoL X-Received: by 2002:a17:90b:3a85:b0:23d:e0e8:f453 with SMTP id om5-20020a17090b3a8500b0023de0e8f453mr20434324pjb.38.1680107283940; Wed, 29 Mar 2023 09:28:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680107283; cv=none; d=google.com; s=arc-20160816; b=GdqTGDEIEQRMttN7eVfxdKsZ2U+NL8LJXDCGNLyFOFWzEOT4ey4gM7sXp45H0htZjx X0n710+uJ6ifuaHglHu95+/c341Geoe7mMejBFd7l8vs1YZCwX2w68mjgVsOZgIAd6BC Vv5UDMFdFGOUp8SMWMfkNsyUbs94prceJt6dFn3/tW8UC/fZHQpS/9Ap058onsTfMH4z 0ANqFkRuVbIfNufGK2GtUiyU+8cQHvJtK0GjrN4iCR71FhoTBo7MhN6dWY4BotabYdqI DLyE5guLOjXsrD1kfTO8TcKWH3e+ZwZGb3TWfAOnNGIi76Adkx4+63/fac9eOViznaFH yjxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=akVSJ2LyXWCgnCkenP6zVJjPdvdtBduMAT+INdXPpJE=; b=KlPuV0jvfsCrWNX7TXpuEE3lQzSQKEkyV3WPfQy76VufCIn40KCvA4M7C1JJwpfmjN pQkwTNqS+x3+fRUTTr6xuk+7mT7Y2uZcTvgR6Dcz+HNonkfw/FvTgdk0M/+EqoaRxcf/ hRhSMlC24gbowW3vGGWzqtHlvybKzSXwQuzkUhBrmpPG6CnjqCSEertAa9ud9dZ7elDq O4BN18y6UZRIuUTlVRNhqm2cWUJSKzNSIcdkTUtfvKCDzFnAuLtcyDEJpYATQQM8QiD7 HUklKBRlL1w5NDFqL5YgqbcAnraSoNN5Ux4GLWBg2RFDd6GPdz5hkDqBZsIqQSSr28Ws 5LBg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=T9MnKLbP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bg23-20020a17090b0d9700b0023cfd288a09si1623644pjb.157.2023.03.29.09.27.51; Wed, 29 Mar 2023 09:28:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=T9MnKLbP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230359AbjC2QE6 (ORCPT + 99 others); Wed, 29 Mar 2023 12:04:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231585AbjC2QEV (ORCPT ); Wed, 29 Mar 2023 12:04:21 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B56C3619E; Wed, 29 Mar 2023 09:03:11 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7054461DA6; Wed, 29 Mar 2023 16:02:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2042CC433A8; Wed, 29 Mar 2023 16:02:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1680105737; bh=0K4ESNHhsDtk/yTZYL0aDVGEKmylHOGHPXnx46UNdgA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=T9MnKLbPrzarjfBI9Dw3rgLPHWQ8R6SP/XmHP0pf5K+NS6LV+GbdRHtt44caB17Aa Fovrx/MzSWysqABNpo4PTUYL30Yd0e/T14VLVTAG7i5Tij52icZET8bAihMSn2uncp 9dMU1P+Y0blFA4Z74TJYw8hmeSgYk6IBYNX6ovEdXytKo3dlUkiI98znrYWcl9kRBo 1slX4ifmJD5DWD+8I3Rbvnud97Fu8BfkTVZ+xehDhzeywWmnKwPeXuJ2bUC1uZ1sx1 rHHr6nT9u3Mw3v5d+dOFQGgwI7yM4iXHP8aK6LM2EiEjamYCebP3ZCgp8OBmd3WvAN ctWQZThLtRm/g== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , rcu , Uladzislau Rezki , Neeraj Upadhyay , Boqun Feng , Joel Fernandes Subject: [PATCH 4/4] rcu/nocb: Make shrinker to iterate only NOCB CPUs Date: Wed, 29 Mar 2023 18:02:03 +0200 Message-Id: <20230329160203.191380-5-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230329160203.191380-1-frederic@kernel.org> References: <20230329160203.191380-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761720175050452593?= X-GMAIL-MSGID: =?utf-8?q?1761720175050452593?= Callbacks can only be queued as lazy on NOCB CPUs, therefore iterating over the NOCB mask is enough for both counting and scanning. Just lock the mostly uncontended barrier mutex on counting as well in order to keep rcu_nocb_mask stable. Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index dfa9c10d6727..43229d2b0c44 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1319,13 +1319,22 @@ lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) int cpu; unsigned long count = 0; + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) + return 0; + + /* Protect rcu_nocb_mask against concurrent (de-)offloading. */ + if (!mutex_trylock(&rcu_state.barrier_mutex)) + return 0; + /* Snapshot count of all CPUs */ - for_each_possible_cpu(cpu) { + for_each_cpu(cpu, rcu_nocb_mask) { struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); count += READ_ONCE(rdp->lazy_len); } + mutex_unlock(&rcu_state.barrier_mutex); + return count ? count : SHRINK_EMPTY; } @@ -1336,6 +1345,8 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) unsigned long flags; unsigned long count = 0; + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) + return 0; /* * Protect against concurrent (de-)offloading. Otherwise nocb locking * may be ignored or imbalanced. @@ -1351,11 +1362,11 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) } /* Snapshot count of all CPUs */ - for_each_possible_cpu(cpu) { + for_each_cpu(cpu, rcu_nocb_mask) { struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); int _count; - if (!rcu_rdp_is_offloaded(rdp)) + if (WARN_ON_ONCE(!rcu_rdp_is_offloaded(rdp))) continue; if (!READ_ONCE(rdp->lazy_len))