From patchwork Wed Mar 29 16:02:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 76704 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp537705vqo; Wed, 29 Mar 2023 09:28:13 -0700 (PDT) X-Google-Smtp-Source: AKy350arWPE8t7xe9Y2MMPsn3aUlZ5S2sEeBUq4Af0MaTQ9bZlfAFdqCVOK1wlNswnJtlipzBfx/ X-Received: by 2002:aa7:d54f:0:b0:502:3376:7872 with SMTP id u15-20020aa7d54f000000b0050233767872mr16315522edr.35.1680107293477; Wed, 29 Mar 2023 09:28:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680107293; cv=none; d=google.com; s=arc-20160816; b=prUeWon14cJUYzzU+BUk8Z/AH/Qk7wQhP8Hilzspm7oBBzHwF7SJTehaJJUBEcxPAL R4evt2f0/bOTdQc94kTchrvDeOsfx120Ci1x2T6dttx1v9c+oku5GJ7+7kcDqmio/le7 QuPc07GcTPSmujS2Pjca3AJfiLv8/LMSoK8UgQie05A7PXOKGiDImcxPuw4K6Ceb8yKy MDjbW21/77rVCvUCU7puyPcF8xmwZ7bPwfGzbvd1Ig2l1OuBm9vxh3Pj3umr1Qlt0Y8U /lvNaVb5BDI7IJ93Uhkr+gZznV2j5l9e/MSDe9d+ojoZOIFsXMufZJ1OL2ybY7K/U3nm Wq3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=v01Hw6x0CUiP4NL/z6mcjh/halOX2MBGCfK09KIAllQ=; b=XdYjNxAD/kToCq2MZfzUl73PSEqLhmMx+GxMg4AB3CDekQYTg2FpNXG8rQOMug02/U hryqA/Bki8WKlWT9lK2E5LOweLqIqKySnuR96Bxo4jClG358zrxvMef3190uxaT52o9J FdbLmWyJhvK34MS3Q7R9aJ3+8vVRlvYNhojzHRgqJQsWZvtjR/qrQa5lvXOwXyjLtpFh 4uTeJNcGHHXe38ILPnuSGbaRyDtFW1pXXtKh9msjZMxJb1Ia9JGvoVxOte2SiuV698mZ /UyDdWk8hV0LJsTO0VkF62+D47y9gMLU4foPRYu2WXTPY7C2QIC5OrOcHwNlVUfHuAhk x8FQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=mM1UAcje; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d22-20020aa7c1d6000000b004ab4f62171asi34776268edp.279.2023.03.29.09.27.50; Wed, 29 Mar 2023 09:28:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=mM1UAcje; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231223AbjC2QFL (ORCPT + 99 others); Wed, 29 Mar 2023 12:05:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231285AbjC2QEk (ORCPT ); Wed, 29 Mar 2023 12:04:40 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D2DE4C16; Wed, 29 Mar 2023 09:03:46 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id BDF39B82380; Wed, 29 Mar 2023 16:02:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id ABE73C433A1; Wed, 29 Mar 2023 16:02:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1680105733; bh=qp5qTHymeNKXf/igbwuWFINn/OHqxS5wqWdXe8S1EME=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mM1UAcjeDDMImv5k9KeshViE+mnte2MLkwfhDMuMk3mam4OB+U+s9icDSTJiY6tj2 mpV0By8nybN+Dgd8/UsXyNaX0vKIF6Wsj1oYa1nO7Ro+o3qlvIQ6RmJP0DQBtpP2Vm lINjK5j4VPanlu31F6MfIammRNoXjwvehMtGoBzrQPEQMKVN2HJjL/Go+QZeE+JOA3 BCfSnQzEC4MnFn0lZJ3KjD6gofBMB+dceLNEydiTGDxuzL4Bk0Tus2PSqy3dsyVrDv GcLelC4UmxcmsywSnqEBIBbrM36mqdisHzvvceK86rBfsbDqZxqwPAgzhp18r48E2m VsKkc1l1zbseA== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , rcu , Uladzislau Rezki , Neeraj Upadhyay , Boqun Feng , Joel Fernandes Subject: [PATCH 2/4] rcu/nocb: Fix shrinker race against callback enqueuer Date: Wed, 29 Mar 2023 18:02:01 +0200 Message-Id: <20230329160203.191380-3-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230329160203.191380-1-frederic@kernel.org> References: <20230329160203.191380-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761720185243612576?= X-GMAIL-MSGID: =?utf-8?q?1761720185243612576?= The shrinker resets the lazy callbacks counter in order to trigger the pending lazy queue flush though the rcuog kthread. The counter reset is protected by the ->nocb_lock against concurrent accesses...except for one of them. Here is a list of existing synchronized readers/writer: 1) The first lazy enqueuer (incrementing ->lazy_len to 1) does so under ->nocb_lock and ->nocb_bypass_lock. 2) The further lazy enqueuers (incrementing ->lazy_len above 1) do so under ->nocb_bypass_lock _only_. 3) The lazy flush checks and resets to 0 under ->nocb_lock and ->nocb_bypass_lock. The shrinker protects its ->lazy_len reset against cases 1) and 3) but not against 2). As such, setting ->lazy_len to 0 under the ->nocb_lock may be cancelled right away by an overwrite from an enqueuer, leading rcuog to ignore the flush. To avoid that, use the proper bypass flush API which takes care of all those details. Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index 1a86883902ce..c321fce2af8e 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1364,7 +1364,7 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) continue; rcu_nocb_lock_irqsave(rdp, flags); - WRITE_ONCE(rdp->lazy_len, 0); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); sc->nr_to_scan -= _count;