From patchwork Wed Mar 22 19:44:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 73648 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp2534592wrt; Wed, 22 Mar 2023 12:46:14 -0700 (PDT) X-Google-Smtp-Source: AK7set89mpsYcg7tZ2mLQe4NLB+zQ5QInI1oZufkG79kZ9Aiux4CTsAW1t++sPBizLYidb/bDePv X-Received: by 2002:a05:6a20:1221:b0:d9:b59c:d09e with SMTP id v33-20020a056a20122100b000d9b59cd09emr655303pzf.11.1679514374162; Wed, 22 Mar 2023 12:46:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679514374; cv=none; d=google.com; s=arc-20160816; b=ZAxdAm+tsuf4UJsU+z1vMKQ3HAfUgLwtobzmoWZgH4JlIHPudAvvNXzC8bWuIh0ny+ xG3jji1uVdX7ljYY3ZCmMkEhhQtnM1V980QlaTwkCraMB4xmi6gwi5CriQARbhvV+mNp rOUgZ27AabsBWSzUejHhCtPEa7FtU2AU0Mh5Pm8/7ogM1BR0dUp8hoOAii1ROcAr1krV TURO+RSvhMVoZgC2jgPyGJ/iy26pZZvLbyqS9KoWsTqWruIEl8yfJrMEbTNWvWY4dU0J SVh2Z7jy7ll0SuKIIL5VJ7FBBS+UBtB9U3FtvLmGkUqQtvrUAkcomw+Qs/5xJ3SaDLlG suSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=mmglLE5oH+V8Tbw5lSX8yjxfe/MV+7OxcRsttZXtJoU=; b=Bw0WsaC/bynEKeCo0kMuZOZ+ajHCIfHJ+/ZqRjd6vOt4eaPSZbyUX0GOQ4XNu6wLEF 6LvkkpULN5YKY9C/KyK8WcXG8rw64ZT4UZMT58XLJgZ2cBGwyJWtMj5TVkO48SGwB2rk qGa/aivX1eURTKsvwgIF3uO+ItH6YmofMvO1jzIVBDMSEDlBHnbaLUOStatXfvm9saeC Fbvo14XF9z3epum5PtcM9VIdecMuNkprsCr1Nf+X4nvD/7fAjUCClRJo8KLsgK6uu0Uw lWHDFyq43Tl38SowB99Z+Jmn9DLPvL3HO+94doQDspL4MHkfjREI2FkBM+BGjOK+YbuH 05+w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=KsIagmLy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 18-20020a630412000000b004fc29da6e55si16269867pge.674.2023.03.22.12.46.01; Wed, 22 Mar 2023 12:46:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=KsIagmLy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230010AbjCVTpW (ORCPT + 99 others); Wed, 22 Mar 2023 15:45:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229838AbjCVTpS (ORCPT ); Wed, 22 Mar 2023 15:45:18 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6135E3E0BD; Wed, 22 Mar 2023 12:45:07 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id EF741622B2; Wed, 22 Mar 2023 19:45:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8E6C7C4339E; Wed, 22 Mar 2023 19:45:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679514306; bh=a59gUDfTCHexacSrsbU3Hhq+A3tfJIUWFd5dyE5PoH0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KsIagmLyRTqKsfR7uIUg9MjUB6redqBnrKvV76N6o1Vx+jY7uIq9Y/alf5qVC/2ey ohoXznoXXIOPSCCOrKmSUnBTPG7XgSSOYnAU0cwsvTb1jWWtbM7mUf8bn8zfxUqq7d sUKfStasnjoGheU7ibY1W95SuRdThbFK9Ec/iAOHru2Z9iREVbwiQjm2+mgwBrHgpT Rv8lUWHtGFZv1KFuOaULMtS5KzIMTqZzlDzOsR6T4Fl82KuJ/xakJuFUd6056Ubqmc H7gSghF0E0NFS/fX2z1J5oKfP42YjQUcRkcAmriowRcVz/Mqs+RrElKEEgy3Ag1wLo 1ff65pFu9K4+A== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , rcu , Uladzislau Rezki , Neeraj Upadhyay , Boqun Feng , Joel Fernandes Subject: [PATCH 1/4] rcu/nocb: Protect lazy shrinker against concurrent (de-)offloading Date: Wed, 22 Mar 2023 20:44:53 +0100 Message-Id: <20230322194456.2331527-2-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230322194456.2331527-1-frederic@kernel.org> References: <20230322194456.2331527-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-5.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761098464543116465?= X-GMAIL-MSGID: =?utf-8?q?1761098464543116465?= The shrinker may run concurrently with callbacks (de-)offloading. As such, calling rcu_nocb_lock() is very dangerous because it does a conditional locking. The worst outcome is that rcu_nocb_lock() doesn't lock but rcu_nocb_unlock() eventually unlocks, or the reverse, creating an imbalance. Fix this with protecting against (de-)offloading using the barrier mutex. Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index f2280616f9d5..dd9b655ae533 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1336,13 +1336,25 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) unsigned long flags; unsigned long count = 0; + /* + * Protect against concurrent (de-)offloading. Otherwise nocb locking + * may be ignored or imbalanced. + */ + mutex_lock(&rcu_state.barrier_mutex); + /* Snapshot count of all CPUs */ for_each_possible_cpu(cpu) { struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); - int _count = READ_ONCE(rdp->lazy_len); + int _count; + + if (!rcu_rdp_is_offloaded(rdp)) + continue; + + _count = READ_ONCE(rdp->lazy_len); if (_count == 0) continue; + rcu_nocb_lock_irqsave(rdp, flags); WRITE_ONCE(rdp->lazy_len, 0); rcu_nocb_unlock_irqrestore(rdp, flags); @@ -1352,6 +1364,9 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) if (sc->nr_to_scan <= 0) break; } + + mutex_unlock(&rcu_state.barrier_mutex); + return count ? count : SHRINK_STOP; } From patchwork Wed Mar 22 19:44:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 73649 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp2536688wrt; Wed, 22 Mar 2023 12:51:12 -0700 (PDT) X-Google-Smtp-Source: AK7set++DxDpXAfNCEGSpRqpotmnyfn4ARTHd5S3MutD3ZkPZ/8gOSN5sYxsP6zyy5KE0gIFcouF X-Received: by 2002:a17:902:da8b:b0:1a1:ca37:525a with SMTP id j11-20020a170902da8b00b001a1ca37525amr4457685plx.36.1679514672524; Wed, 22 Mar 2023 12:51:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679514672; cv=none; d=google.com; s=arc-20160816; b=Pb7g1AldpIq68Twj7mck5CgAln4nWtsQPzX3m2TPvhaeLZS9/3rWq6jWMd5jEk6NWl N77qyCH8G7KY31bHOuIdAoSmbmahjEdpEIXCndbtqRbs1ZjMxf2DTzz+aISQCFAdXULa yUaBxjsoMVld8Cj1ZMO7ahTaN9S2y4Z4bTyhLbgxOGVaxVHbA/YOCOSN0JpBmlGwmpQ5 rE2suUGsQewIKadW0cmoxn8S8Q4N6e2UjqXDd43pOEZNULF2YHzc5j1FGrnKlY3qq8Bo skOqOHTnRGo/yNtOP9GS3Ye6CZYA2/7XZLLhqcCQ9/IiN+DJ6Qo8XcOaDKqGsVn/Wo6/ oWZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=29VBGH4PlF4WusIiZ9owbNMBnJCZ2q9O7kpR2ktbG+E=; b=0becKQ0yoR0SGvTgCryh2xiHEd5mIExKBRk8eiO1r+mjdWkBpwL0k+kqdPebrbgEhQ Ty2482ljE6/8Pk6F2x1/B+0nHPvM4MRw71+MaLhOnx22xJssSIhwZmMfL0zOxsrShFM3 0SEv9PKBLPhcQVVW0hSIrExcb+17NDIvfF9Eeb7AiAGF795Hpa44GzfZwQgH/HjHcR85 bp88GRShJmKMKrZR8eOjADCaLYavuqGGflQ3XiviE6iiDi2fJZFBrAef2j7Tj7Ppp1rP DzGuyta28NLRAxEeRqmMw8sqhsZuZPxqZdXOv6WNWmkfqvg0kj/C9JSGloXi9StV6K3N hUpw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=q4ElfwI+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p12-20020a170902e74c00b001a045fcd743si17427411plf.142.2023.03.22.12.50.59; Wed, 22 Mar 2023 12:51:12 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=q4ElfwI+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230182AbjCVTp2 (ORCPT + 99 others); Wed, 22 Mar 2023 15:45:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52614 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229871AbjCVTpS (ORCPT ); Wed, 22 Mar 2023 15:45:18 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AEC273BC54; Wed, 22 Mar 2023 12:45:09 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 39A03622B3; Wed, 22 Mar 2023 19:45:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CF921C4339C; Wed, 22 Mar 2023 19:45:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679514308; bh=0Nr44nYCi7dzslDo1zOnGsqBigDrConQ2cePzusvGBA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=q4ElfwI+077Zl/ofdd0x4Ay6VqneIR52LHDh8XjmdxboYiaUFydahOC+vJdDP3gVM qKcsiuWKsMku4MXcouUMFY1ScE5FU63BdMz/qvvDXmr/AZV7oPpzqxEvDSG7ZWC5VG V8oTfqnZjRS0wCjLYzNexf3mUk13See+COaCA70KUfzrfUHDjbZB0nODKhY9g9GEf8 VlO40NfZqTBD6abnXkfgTUW2DfAPV2va79D0yiYOm/MWe4O3X9NKiweMNPuOVmIVlW w9kbIexABoAg5VoldEaKqo16pOjfJbMT35K/KjLJDwpMnw+JO9srJ+pZ11tOc5oRAy k2dQYxj84yMGg== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , rcu , Uladzislau Rezki , Neeraj Upadhyay , Boqun Feng , Joel Fernandes Subject: [PATCH 2/4] rcu/nocb: Fix shrinker race against callback enqueuer Date: Wed, 22 Mar 2023 20:44:54 +0100 Message-Id: <20230322194456.2331527-3-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230322194456.2331527-1-frederic@kernel.org> References: <20230322194456.2331527-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761098777173070069?= X-GMAIL-MSGID: =?utf-8?q?1761098777173070069?= The shrinker resets the lazy callbacks counter in order to trigger the pending lazy queue flush though the rcuog kthread. The counter reset is protected by the ->nocb_lock against concurrent accesses...except for one of them. Here is a list of existing synchronized readers/writer: 1) The first lazy enqueuer (incrementing ->lazy_len to 1) does so under ->nocb_lock and ->nocb_bypass_lock. 2) The further lazy enqueuers (incrementing ->lazy_len above 1) do so under ->nocb_bypass_lock _only_. 3) The lazy flush checks and resets to 0 under ->nocb_lock and ->nocb_bypass_lock. The shrinker protects its ->lazy_len reset against cases 1) and 3) but not against 2). As such, setting ->lazy_len to 0 under the ->nocb_lock may be cancelled right away by an overwrite from an enqueuer, leading rcuog to ignore the flush. To avoid that, use the proper bypass flush API which takes care of all those details. Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index dd9b655ae533..cb57e8312231 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1356,7 +1356,7 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) continue; rcu_nocb_lock_irqsave(rdp, flags); - WRITE_ONCE(rdp->lazy_len, 0); + WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); sc->nr_to_scan -= _count; From patchwork Wed Mar 22 19:44:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 73650 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp2537437wrt; Wed, 22 Mar 2023 12:53:05 -0700 (PDT) X-Google-Smtp-Source: AK7set/G00roMR5bOMT7r/ig8CkPVOygRXIRKiPHZYT+pDLYd2vhF1vV9xgmTjYWTbPZG2FtboTE X-Received: by 2002:a17:90b:350e:b0:23f:2c65:fab7 with SMTP id ls14-20020a17090b350e00b0023f2c65fab7mr4707228pjb.42.1679514785673; Wed, 22 Mar 2023 12:53:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679514785; cv=none; d=google.com; s=arc-20160816; b=G3LqzyPn+Wbstj5bL+V75ufIZwb/KgITiFnPqCp3HHkvXkG1yshcjbiHAWLR3R/Yxm KdypzJ2Vv8a6jbkMSHIU/ULoJcmeMS+SaDIL2Q1bGm4CoZlQdw2fenbnbadnWjpOfz2u mZfENBLtdmxRQL9bVc1kUU9QkO+O0sesF9kAel+H42HNc8urUaI+Zqox3tqaG+kLZbyO xpF91DMZ/c0G26Ry0jy5vdkao19Sld7fCoYP1cCluFxlzSSBAwlK7IZEtwtrTPjmuI26 g3ffdt0zBLGew7dWXE5416ktWNPGXugtPClydN1ezdtdnOwY3Uz/gAkr2H/4KqPLRRAS fp6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QWh0w4a34SfD/fykTzaCD4NrRTzf1m09qyJhbCAzRJc=; b=pfIlfD8wOZhdXUr5z3+AS7kQWRTKVH9J71rf3xdMgL+aBzQE9JTfh3dhh1hfC+Q/Pv lJVI1YXL8kG+vermIGNpYt7hTcBFN+4nbTTdKFs3MqTEwy3G0YV3phapFULunASU14ot H6TraPyXz9pAkhpdj62qYiUqU3PyktQdqY+sCYhzPWeaXiHq6J+nO6j4xfrbYygmXc/+ dwQ9g8sibEYDOhCC2LXmB8PdTUxPyIsQ1hqKYO2Vqt+856vdJ0WEPf/eolI5Q1UZ620R oNAHjELfgOvvZG/L8vNOZJk31wfU+PsGR36tOFjHPU/l6A8Jju5iQKPcMgA0DRi9YeEk uY8A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Z4SeTifB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z7-20020a17090abd8700b0023f37b648d9si16274830pjr.157.2023.03.22.12.52.52; Wed, 22 Mar 2023 12:53:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Z4SeTifB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230076AbjCVTp0 (ORCPT + 99 others); Wed, 22 Mar 2023 15:45:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229846AbjCVTpS (ORCPT ); Wed, 22 Mar 2023 15:45:18 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE7ED32CEB; Wed, 22 Mar 2023 12:45:11 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7BD9A622B1; Wed, 22 Mar 2023 19:45:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1B7CDC4339B; Wed, 22 Mar 2023 19:45:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679514310; bh=mP8fCieTmDyHtxHWglNueIzXLMyc5wglCPH3QTQMCKk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Z4SeTifBUVRdEuhbIjeDAFkhZN28kaeoNGLv044rupmYaop8EMvGXNg2ZnWWILzay 0/ndpYG1LRNCIps3ZYm+OjblNQ2Jx0Z3xyIAkpKvkR1m1tRxv8P7j1HqZY7LEGGRy1 X0ePCHGOVygrJy6zx3cr2agJePWejOCbV1uAqk+a8eH1knyQ1aUreHgnlv+jkpKuci KXqPE/9qDyKZR1ZiG2tgroIW0BCGzx+YdpgkmPLqfpoduP4LSHPPzwj0YANEhzCJeA z1vOPQ5UwH7MIwmZQr92YILDBs0gB2vgKDgl5LGO7A2ob8JooFjAcnOmAIp+xfTlVy KBFcXUO9lpmjA== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , rcu , Uladzislau Rezki , Neeraj Upadhyay , Boqun Feng , Joel Fernandes Subject: [PATCH 3/4] rcu/nocb: Recheck lazy callbacks under the ->nocb_lock from shrinker Date: Wed, 22 Mar 2023 20:44:55 +0100 Message-Id: <20230322194456.2331527-4-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230322194456.2331527-1-frederic@kernel.org> References: <20230322194456.2331527-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761098895817003241?= X-GMAIL-MSGID: =?utf-8?q?1761098895817003241?= The ->lazy_len is only checked locklessly. Recheck again under the ->nocb_lock to avoid spending more time on flushing/waking if not necessary. The ->lazy_len can still increment concurrently (from 1 to infinity) but under the ->nocb_lock we at least know for sure if there are lazy callbacks at all (->lazy_len > 0). Signed-off-by: Frederic Weisbecker --- kernel/rcu/tree_nocb.h | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index cb57e8312231..a3dc7465b0b2 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1350,12 +1350,20 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) if (!rcu_rdp_is_offloaded(rdp)) continue; + if (!READ_ONCE(rdp->lazy_len)) + continue; + + rcu_nocb_lock_irqsave(rdp, flags); + /* + * Recheck under the nocb lock. Since we are not holding the bypass + * lock we may still race with increments from the enqueuer but still + * we know for sure if there is at least one lazy callback. + */ _count = READ_ONCE(rdp->lazy_len); - - if (_count == 0) + if (!_count) { + rcu_nocb_unlock_irqrestore(rdp, flags); continue; - - rcu_nocb_lock_irqsave(rdp, flags); + } WARN_ON_ONCE(!rcu_nocb_flush_bypass(rdp, NULL, jiffies, false)); rcu_nocb_unlock_irqrestore(rdp, flags); wake_nocb_gp(rdp, false); From patchwork Wed Mar 22 19:44:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 73651 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp2537826wrt; Wed, 22 Mar 2023 12:53:59 -0700 (PDT) X-Google-Smtp-Source: AK7set8niebKQ8CO5o39V9vy5sGL4f/gTdSGo8l9SV3FB+m86LMDPsXPqWtZgEkGYAqESLFd19pn X-Received: by 2002:a05:6a20:8f0f:b0:cc:70df:ae20 with SMTP id b15-20020a056a208f0f00b000cc70dfae20mr782983pzk.0.1679514839160; Wed, 22 Mar 2023 12:53:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679514839; cv=none; d=google.com; s=arc-20160816; b=KNBJxVtj3Uu8PsaGYeiwkKr35q1C5iVkeOepuXvrS35sL5BoTuBkeZ2xdviLmZ4vtV aFutduBbVCOzBht3QpKXJ9LorWWKSeIEjDrTi1VlllYXbVRFRY4rOAWBTN2f5OmGTQkj C5IYgYirBayY/RaqiCceFTgDgpmn0NQ4lMvMOiqEWyhgBr6UFuqeH162qu7xNGFxnKTq Fi0O/Ue7OaD/YdO6rmFZg0xT2Oe6TnF6/kEQBeF6CogRfVIRZGbYZVeRLvMc1FoIFKrD L7Sr5fPYzFrS/lgbHVyc/sUreMtngNphrKthSIVqRrebDpTOiq6xNcIyfmHVfQFr+GnL 1tyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=kpX6/yGKLnlgQaQIZH1x0b4a3n71Kp6mrU22uMIIRl0=; b=X2DHVZkoJXC0+NhmilGdILbCqmSn2G5SZ5W9ae6nLZ6MWtvPOQWKD4/onk0HijsHdb +WnMizzG+vxG4TXbiZVCPZFdIY7s9d5J3EinIgAlmgYYlLYnQrdsDa+OQiauANZsiG5h 8luR8zqKuQGdop8RrcffSv2/CkJ0TeCwfYwoloM3miJLYxEsDPjaXWEl7L5lMWrEjdb+ eUOHrVzPkNUYuyPy6vhd7ybA4UmxSUzXB38TCPqTa1aWnJI9wjZf0WbopNwOfCB/xT9S kV3dCGKYpKcu2b1xxwqRadOnEyFYYZue+BMglcA6MmIetJR/8gxqG4B1Zz0Lu9Ffoxu+ 3+UQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=sMzpGXOB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 18-20020a631552000000b004fbd23110d4si15737196pgv.773.2023.03.22.12.53.46; Wed, 22 Mar 2023 12:53:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=sMzpGXOB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229865AbjCVTpb (ORCPT + 99 others); Wed, 22 Mar 2023 15:45:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52626 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229928AbjCVTpT (ORCPT ); Wed, 22 Mar 2023 15:45:19 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C81F11EFE2; Wed, 22 Mar 2023 12:45:15 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 78F8CB81DEA; Wed, 22 Mar 2023 19:45:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5A46EC433AA; Wed, 22 Mar 2023 19:45:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679514313; bh=o/d+MExJk6dT2Q3smaaeBjF9BtLmW7hBw77NXSOsuZA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sMzpGXOBEC1RG/wSZamyrsu+iU5m/0UxfjbhgycVXYNRHFbH1PZzU36l2ZJ4mSV0j LkxFT6qscyXdSGYE7CmKa1TSgHrv8ZKVgvw5F65b2ML4Bc3nLmvr37dSZcgONG+5pJ mdw05b0W+QtvQbJBzNWZzZiXPt0XXlvpK3HV5gZA+FxyPMGB7CvKDCBH8iioZ4teNE d3i8DgCAfvDwfQLZEvkwaEIUyJ5yDb+kP0dxZBPQzjEi/l9KdYglF8ryipqJ9Imq7y zm+q6aVtWhZGneLo//5aGNwdBjFj+fGPKk8BA3LqKTtQbIbBhhQyejpRxL+Q7gZWZr Y82MOffgaHTBQ== From: Frederic Weisbecker To: "Paul E . McKenney" Cc: LKML , Frederic Weisbecker , rcu , Uladzislau Rezki , Neeraj Upadhyay , Boqun Feng , Joel Fernandes Subject: [PATCH 4/4] rcu/nocb: Make shrinker to iterate only NOCB CPUs Date: Wed, 22 Mar 2023 20:44:56 +0100 Message-Id: <20230322194456.2331527-5-frederic@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230322194456.2331527-1-frederic@kernel.org> References: <20230322194456.2331527-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-5.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761098951935719307?= X-GMAIL-MSGID: =?utf-8?q?1761098951935719307?= Callbacks can only be queued as lazy on NOCB CPUs, therefore iterating over the NOCB mask is enough for both counting and scanning. Just lock the mostly uncontended barrier mutex on counting as well in order to keep rcu_nocb_mask stable. Signed-off-by: Frederic Weisbecker Reviewed-by: Joel Fernandes (Google) --- kernel/rcu/tree_nocb.h | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h index a3dc7465b0b2..185c0c9a60d4 100644 --- a/kernel/rcu/tree_nocb.h +++ b/kernel/rcu/tree_nocb.h @@ -1319,13 +1319,21 @@ lazy_rcu_shrink_count(struct shrinker *shrink, struct shrink_control *sc) int cpu; unsigned long count = 0; + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) + return 0; + + /* Protect rcu_nocb_mask against concurrent (de-)offloading. */ + mutex_lock(&rcu_state.barrier_mutex); + /* Snapshot count of all CPUs */ - for_each_possible_cpu(cpu) { + for_each_cpu(cpu, rcu_nocb_mask) { struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); count += READ_ONCE(rdp->lazy_len); } + mutex_unlock(&rcu_state.barrier_mutex); + return count ? count : SHRINK_EMPTY; } @@ -1336,6 +1344,8 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) unsigned long flags; unsigned long count = 0; + if (WARN_ON_ONCE(!cpumask_available(rcu_nocb_mask))) + return 0; /* * Protect against concurrent (de-)offloading. Otherwise nocb locking * may be ignored or imbalanced. @@ -1343,7 +1353,7 @@ lazy_rcu_shrink_scan(struct shrinker *shrink, struct shrink_control *sc) mutex_lock(&rcu_state.barrier_mutex); /* Snapshot count of all CPUs */ - for_each_possible_cpu(cpu) { + for_each_cpu(cpu, rcu_nocb_mask) { struct rcu_data *rdp = per_cpu_ptr(&rcu_data, cpu); int _count;