From patchwork Mon Mar 13 11:28:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 68758 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1129639wrd; Mon, 13 Mar 2023 04:37:25 -0700 (PDT) X-Google-Smtp-Source: AK7set/GwZg4oYZCB3eSXvNQqw8/yRaWbS1g3jBnXoo1/DZmfYzU7im/efTaCAkRB8/SpInG9WK1 X-Received: by 2002:a17:902:aa4a:b0:19e:3922:b7d8 with SMTP id c10-20020a170902aa4a00b0019e3922b7d8mr27883712plr.12.1678707444729; Mon, 13 Mar 2023 04:37:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678707444; cv=none; d=google.com; s=arc-20160816; b=mnkOFTCljWX+Pa6H7XQ986z3OibYqBIHvAUIuQ/qO9l1TMIqcE/ZQ+PtwUE0/5IEhl EWFlGHi7GgM4bq27hNPv+VcvwWOyxyykxfQMmVBGeY4Ri0mZ835yStm8+ILNbfeQ2HAa FweZUDS2rfsZfGREZz3230j8eVECYLZdl+WtApKY3yDmC7MpjNwz7EjOej7oaFQsgmQF 1cjW1eAErBcRsNURXBINi2u/2XVDEA6kpwYQ5riIzhr9n7dCe7n/ZsfOFmNUQor2FvNV X445xYvKiQdpdI3M/zTNn+VVIjNwZnQXH8F/V/+mqz2SZZbT3SApNBtA8RAgKZ/4d7au Fltg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=cHjpkGxbHVFFSyOk4f8KxselJo2cIaqjL6okRMIx/Nw=; b=ihY4qUU6P08Ut9NXBVTXju+/jt91DDsI4fMFUkc3mbZcZOsy+TxDiiLEeZlLkt+OL/ /PA/xyOnrWsexqGML8U84MYzX3PcEYCgGo/G0Eb4NFyv5mutd466np4o+3cgZ7s2S0bt 8mOC4CcgjcOsSEwh+oXTvdpC/rRE4NS7dP0tRvctPx/iRKMRL8HhMhPsIz/SMFps1/Gx r8V2pQYkVtp1AcJmwP6yiRSsCTNCb/rL96FPQSeTfq2MLVM7o4yaC/JAEQGFo8t907/X Wed6P3BrHXtWfHpU+F8mAo+pRt/AzH2aAjbOpXP1nDL0PSTGDHk76XkRRHsZ4bqv2z+e 5AhQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=cMAG05TH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k5-20020a63d845000000b0050726756edesi6350059pgj.76.2023.03.13.04.37.09; Mon, 13 Mar 2023 04:37:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=cMAG05TH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230193AbjCMLaO (ORCPT + 99 others); Mon, 13 Mar 2023 07:30:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55032 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230167AbjCMLaH (ORCPT ); Mon, 13 Mar 2023 07:30:07 -0400 Received: from mail-pf1-x42b.google.com (mail-pf1-x42b.google.com [IPv6:2607:f8b0:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 55B2F4C6FC for ; Mon, 13 Mar 2023 04:29:43 -0700 (PDT) Received: by mail-pf1-x42b.google.com with SMTP id n16so1816290pfu.11 for ; Mon, 13 Mar 2023 04:29:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678706983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cHjpkGxbHVFFSyOk4f8KxselJo2cIaqjL6okRMIx/Nw=; b=cMAG05THHEL45EY/ZCrlGRIbq1kt4I6LPUMAH/dQjouL+NMY60brEhRYxofz4Sc3yO jVllerX4euE/VSk8gxVy2qydl3B1I0ZvizXWnB/k4AlXoCTSsvfCFO9lUUclwXD/c7zx +Lstf6Kcx2x9pLpDuB/8MwyscxtFe5rDeDGAP/r3rkvK8Nru8nZQojqHvbsNe7a2Ab7/ gUPJ2C1u8cgdUyoJ+NnEXAlZ1izrfSpHumWDpFe9DMlhF0rV7W8iS0Gr8SVDtkxfdNr/ N3obCyhq6rnOWhxjxKbMgLykNy2bkGDKgby8e4YMGNLWFieQ1ksal3K4NhZ3AUVdAOK+ UsUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678706983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cHjpkGxbHVFFSyOk4f8KxselJo2cIaqjL6okRMIx/Nw=; b=W8QPQQxqVIyFmyp8lbMo8UIKoTm3iLRfsQNS0g/u0doZJmwY60m09905ShojbH1f4r 5uvj5mxC5bH/iXyJQQKb8wjYX0pGdVU7zLxWickgnbf5P1n+fcgDAI/RsGjAZiRUYOjq hdYoFnXcr5J6ijjoQlgQBW4kjzcio+4FBwnUU159O57otp9ko+xpC0u/dY7boCosYvLc Z5HVDVSF5f17PDh6AD4VTeQIO2pxf1E7+IfQWIQ/ngEE+Hf7z9tAFtf9+Owb/5zasrbo 9UGty363kvrvdtDzSHrDQDszVHvyWInhMED8HlyHjiFZnkStHtRPBNE1T0PL8Fph2z00 6F4A== X-Gm-Message-State: AO0yUKXhf4rZ4I0WIWLpHFs5Q8XivotVrMfWX900IEbaCfG8dnvUR+GG NmE8oHGpB4zCj1Z2isTjim83ew== X-Received: by 2002:a05:6a00:2148:b0:606:d3d1:4cc4 with SMTP id o8-20020a056a00214800b00606d3d14cc4mr10842399pfk.3.1678706982825; Mon, 13 Mar 2023 04:29:42 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id n2-20020a654882000000b0050300a7c8c2sm4390827pgs.89.2023.03.13.04.29.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 04:29:42 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, christian.koenig@amd.com, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v5 1/8] mm: vmscan: add a map_nr_max field to shrinker_info Date: Mon, 13 Mar 2023 19:28:12 +0800 Message-Id: <20230313112819.38938-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230313112819.38938-1-zhengqi.arch@bytedance.com> References: <20230313112819.38938-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760252337199085446?= X-GMAIL-MSGID: =?utf-8?q?1760252337199085446?= To prepare for the subsequent lockless memcg slab shrink, add a map_nr_max field to struct shrinker_info to records its own real shrinker_nr_max. Signed-off-by: Qi Zheng Suggested-by: Kirill Tkhai Acked-by: Vlastimil Babka Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 1 + mm/vmscan.c | 35 ++++++++++++++++++----------------- 2 files changed, 19 insertions(+), 17 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..aa69ea98e2d8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -97,6 +97,7 @@ struct shrinker_info { struct rcu_head rcu; atomic_long_t *nr_deferred; unsigned long *map; + int map_nr_max; }; struct lruvec_stats_percpu { diff --git a/mm/vmscan.c b/mm/vmscan.c index 9414226218f0..9a2a6301052c 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -226,7 +226,8 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, static int expand_one_shrinker_info(struct mem_cgroup *memcg, int map_size, int defer_size, - int old_map_size, int old_defer_size) + int old_map_size, int old_defer_size, + int new_nr_max) { struct shrinker_info *new, *old; struct mem_cgroup_per_node *pn; @@ -240,12 +241,17 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, if (!old) return 0; + /* Already expanded this shrinker_info */ + if (new_nr_max <= old->map_nr_max) + continue; + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); if (!new) return -ENOMEM; new->nr_deferred = (atomic_long_t *)(new + 1); new->map = (void *)new->nr_deferred + defer_size; + new->map_nr_max = new_nr_max; /* map: set all old bits, clear all new bits */ memset(new->map, (int)0xff, old_map_size); @@ -295,6 +301,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) } info->nr_deferred = (atomic_long_t *)(info + 1); info->map = (void *)info->nr_deferred + defer_size; + info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -302,23 +309,14 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) return ret; } -static inline bool need_expand(int nr_max) -{ - return round_up(nr_max, BITS_PER_LONG) > - round_up(shrinker_nr_max, BITS_PER_LONG); -} - static int expand_shrinker_info(int new_id) { int ret = 0; - int new_nr_max = new_id + 1; + int new_nr_max = round_up(new_id + 1, BITS_PER_LONG); int map_size, defer_size = 0; int old_map_size, old_defer_size = 0; struct mem_cgroup *memcg; - if (!need_expand(new_nr_max)) - goto out; - if (!root_mem_cgroup) goto out; @@ -332,7 +330,8 @@ static int expand_shrinker_info(int new_id) memcg = mem_cgroup_iter(NULL, NULL, NULL); do { ret = expand_one_shrinker_info(memcg, map_size, defer_size, - old_map_size, old_defer_size); + old_map_size, old_defer_size, + new_nr_max); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -352,9 +351,11 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) rcu_read_lock(); info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); - /* Pairs with smp mb in shrink_slab() */ - smp_mb__before_atomic(); - set_bit(shrinker_id, info->map); + if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { + /* Pairs with smp mb in shrink_slab() */ + smp_mb__before_atomic(); + set_bit(shrinker_id, info->map); + } rcu_read_unlock(); } } @@ -432,7 +433,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); - for (i = 0; i < shrinker_nr_max; i++) { + for (i = 0; i < child_info->map_nr_max; i++) { nr = atomic_long_read(&child_info->nr_deferred[i]); atomic_long_add(nr, &parent_info->nr_deferred[i]); } @@ -899,7 +900,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, From patchwork Mon Mar 13 11:28:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 68770 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1133888wrd; Mon, 13 Mar 2023 04:47:55 -0700 (PDT) X-Google-Smtp-Source: AK7set8TQ9yTrBfEXjw1lo9T77m4gkOz6alXWa51UjwDNn9199qz78RyL2b67X1WbdNr53mezDBt X-Received: by 2002:a05:6a20:8f17:b0:d5:8a5:e61e with SMTP id b23-20020a056a208f1700b000d508a5e61emr346574pzk.11.1678708075147; Mon, 13 Mar 2023 04:47:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678708075; cv=none; d=google.com; s=arc-20160816; b=JhjmeLPSMCtPw+u7EBW+Id0phvncGb1bY8nVFLFH9ohBeTRJLsZV2L55wBUWK47DAQ v+m5VlbKsKOTmrMhQHhVvehNBsdMFLofw1RVNGI7CGJqbNjGg9tJZxNvofz3MBu0Eais fwER/IzdOKtpLSkyhZ0FccRUgBSpmuLBCh8oALTjFPhD2mEsmfJ1OM/rinuDC0HhLGyO XC3a/jJX4AI21CvzzCyXP4L4eWfwG5iMTEuRk8hVqXobPfdjJ1EyywoRPbJRvkNEYqhi m+iBhOR6u9AKSWwQA0VK8/7g/Qy+5arcJ9509kupz7B6vXqMq0+3deuULusq7cqSc+C5 yoAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=o+PIjzMrqqus+vCKR4kW46vOjsCmmZLpdvQvuCO0Xe0=; b=XkplpzIZBsc+9ZSqEbYtF/7EyLPINHYmN/bm2YZCeVA0DarpcGkLeKJhmi3MdMUAWL mf/m6i0EH+ujH3n6qvzct4kbfmZv4wFkmw2hHGCUfT7mGQO+kZ5YrvY2lHudcQ8nm96N 5LJgg5i1t6DontsN+ra8502HvyIu5E3OLD22wr8BlScsufbsbD8Srel9vYtQ4lR8E05D aII4STtr+J9k1CkAlV36G7F6fci2e4Ggq3jHD5CEz/vAW11Cp1BZcCGP1mQI7adisZ1F uykXpFvzRQrADp9AzD9kxbrPHy74MbnUhA0tpCx5uN8RaMKte6aGL12MENazJLZQAWR9 dn9g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=LKdBuqoP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 142-20020a630094000000b00508aefbec7asi5679609pga.300.2023.03.13.04.47.40; Mon, 13 Mar 2023 04:47:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=LKdBuqoP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230110AbjCMLaB (ORCPT + 99 others); Mon, 13 Mar 2023 07:30:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55146 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230135AbjCML3x (ORCPT ); Mon, 13 Mar 2023 07:29:53 -0400 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C818559EE for ; Mon, 13 Mar 2023 04:29:51 -0700 (PDT) Received: by mail-pj1-x1032.google.com with SMTP id u3-20020a17090a450300b00239db6d7d47so11320778pjg.4 for ; Mon, 13 Mar 2023 04:29:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678706990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=o+PIjzMrqqus+vCKR4kW46vOjsCmmZLpdvQvuCO0Xe0=; b=LKdBuqoPZ3Eoe3IWRve5czAmJNAuSXHJEZyt6qMgQ2QWE0FfdegRHxqdr14XQLU0rx PsAdOS3EITkjvH5yWgAVHjIGJIgtokVqDvCFC26F4koRE2Dg75XvyXjyLjnvaxEhaWs0 dhQ9CFD461CllT/ivbb0KEKQLcI9xDxnAgIIgqPeywEBe7ZlcK3Kpq2x4YNZJlXiWmJW wFSsnsQNtCw4pUPGf8nkmUQNQWO9giNg3PiXQrgZK9dUZR0DabpxIf6V5RpwknTkHH+s 11as/+u7W1eLQqbscYrAfCE12YP+G4kZxz1LGruS5fzUaeaeMeP5o6+X9fPn1O+TsKdH kbvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678706990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o+PIjzMrqqus+vCKR4kW46vOjsCmmZLpdvQvuCO0Xe0=; b=FvriqHfG4WwNH44Z8cRjr2FyKqxVr0XUARuitkSFjSL8da5/5BFYocl9uHOqscBwd2 liXp7UPGsu1JEKSaeLMDJdit18DdariIiJPjK5HsGsb4l86mAQRp+DqGITEX4nNqUz8T 7E2hYS2hxwhdeUoelDncTp1trpRT6usWLC1Dd2cqkRnx+a3QuK5Hvzz/C6btmDxZL0RQ OMNXhcqcK9cJ/c8Il5mQC5pxKzkdp5zuO94We5xbWYHe4GrSLAZq0ryguq5t8XoFuWl2 a2N37ovdHLILO2qdLTbujR49Pxd8V7cwemx/YimaflkCG2NCKOqd9gCiXLtFjB8kBIom nx2g== X-Gm-Message-State: AO0yUKVPwqUmwo+93scnoD65Wf6F4k01o7p5zF/lzF5mpOlFpiOO2wDX MnfrcM5v4rLAY/9zTivibxAHOQ== X-Received: by 2002:a05:6a20:841c:b0:c7:af88:4199 with SMTP id c28-20020a056a20841c00b000c7af884199mr17859458pzd.6.1678706990517; Mon, 13 Mar 2023 04:29:50 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id n2-20020a654882000000b0050300a7c8c2sm4390827pgs.89.2023.03.13.04.29.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 04:29:50 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, christian.koenig@amd.com, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v5 2/8] mm: vmscan: make global slab shrink lockless Date: Mon, 13 Mar 2023 19:28:13 +0800 Message-Id: <20230313112819.38938-3-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230313112819.38938-1-zhengqi.arch@bytedance.com> References: <20230313112819.38938-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760252998947547636?= X-GMAIL-MSGID: =?utf-8?q?1760252998947547636?= The shrinker_rwsem is a global read-write lock in shrinkers subsystem, which protects most operations such as slab shrink, registration and unregistration of shrinkers, etc. This can easily cause problems in the following cases. 1) When the memory pressure is high and there are many filesystems mounted or unmounted at the same time, slab shrink will be affected (down_read_trylock() failed). Such as the real workload mentioned by Kirill Tkhai: ``` One of the real workloads from my experience is start of an overcommitted node containing many starting containers after node crash (or many resuming containers after reboot for kernel update). In these cases memory pressure is huge, and the node goes round in long reclaim. ``` 2) If a shrinker is blocked (such as the case mentioned in [1]) and a writer comes in (such as mount a fs), then this writer will be blocked and cause all subsequent shrinker-related operations to be blocked. Even if there is no competitor when shrinking slab, there may still be a problem. If we have a long shrinker list and we do not reclaim enough memory with each shrinker, then the down_read_trylock() may be called with high frequency. Because of the poor multicore scalability of atomic operations, this can lead to a significant drop in IPC (instructions per cycle). So many times in history ([2],[3],[4],[5]), some people wanted to replace shrinker_rwsem trylock with SRCU in the slab shrink, but all these patches were abandoned because SRCU was not unconditionally enabled. But now, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), the SRCU is unconditionally enabled. So it's time to use SRCU to protect readers who previously held shrinker_rwsem. This commit uses SRCU to make global slab shrink lockless, the memcg slab shrink is handled in the subsequent patch. [1]. https://lore.kernel.org/lkml/20191129214541.3110-1-ptikhomirov@virtuozzo.com/ [2]. https://lore.kernel.org/all/1437080113.3596.2.camel@stgolabs.net/ [3]. https://lore.kernel.org/lkml/1510609063-3327-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp/ [4]. https://lore.kernel.org/lkml/153365347929.19074.12509495712735843805.stgit@localhost.localdomain/ [5]. https://lore.kernel.org/lkml/20210927074823.5825-1-sultan@kerneltoast.com/ Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- mm/vmscan.c | 28 ++++++++++++---------------- 1 file changed, 12 insertions(+), 16 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 9a2a6301052c..db2ed6e08f67 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -202,6 +203,7 @@ static void set_task_reclaim_state(struct task_struct *task, LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); +DEFINE_SRCU(shrinker_srcu); #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -700,7 +702,7 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); - list_add_tail(&shrinker->list, &shrinker_list); + list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); up_write(&shrinker_rwsem); @@ -754,13 +756,15 @@ void unregister_shrinker(struct shrinker *shrinker) return; down_write(&shrinker_rwsem); - list_del(&shrinker->list); + list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_remove(shrinker); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); + debugfs_remove_recursive(debugfs_entry); kfree(shrinker->nr_deferred); @@ -780,6 +784,7 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -990,6 +995,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, { unsigned long ret, freed = 0; struct shrinker *shrinker; + int srcu_idx; /* * The root memcg might be allocated even though memcg is disabled @@ -1001,10 +1007,10 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) return shrink_slab_memcg(gfp_mask, nid, memcg, priority); - if (!down_read_trylock(&shrinker_rwsem)) - goto out; + srcu_idx = srcu_read_lock(&shrinker_srcu); - list_for_each_entry(shrinker, &shrinker_list, list) { + list_for_each_entry_srcu(shrinker, &shrinker_list, list, + srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -1015,19 +1021,9 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; freed += ret; - /* - * Bail out if someone want to register a new shrinker to - * prevent the registration from being stalled for long periods - * by parallel ongoing shrinking. - */ - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } } - up_read(&shrinker_rwsem); -out: + srcu_read_unlock(&shrinker_srcu, srcu_idx); cond_resched(); return freed; } From patchwork Mon Mar 13 11:28:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 68769 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1133826wrd; Mon, 13 Mar 2023 04:47:45 -0700 (PDT) X-Google-Smtp-Source: AK7set/i1mNaDBUhfuLv9auXcLr5LBH0oOwDQT5dut+pmBWVbn5fyDeuKpJVE5cOdxPAiTU/Szul X-Received: by 2002:a17:90b:1e41:b0:237:39b1:7b1 with SMTP id pi1-20020a17090b1e4100b0023739b107b1mr35417239pjb.39.1678708065195; Mon, 13 Mar 2023 04:47:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678708065; cv=none; d=google.com; s=arc-20160816; b=NZAn2U2s7rm66VkdU4Su/Q9VqpoWCLSMLkMKOTI/7ctqcOQDaeRju0PKmMKYNeFcVm xBM3hCP52r5rWkODtLOqoIDmo97AJJKAdFgKymkYseCd6uPO8rpelZSzCwRIVJD5XIg8 uduzNBa1VU3JOn5Ih4Ss83Hn9FxNEUygQ7hG0G156eplW5vHbCkcw//mReozuD/q1BQ6 EjeLfDOl8NZkrX2gwiXgNKOAq6yL0WrjNfevJChgTanoeMZALxIMVi1PI/sjp39QH4n/ gdAK6LiTE6Km8lXPmtYMLrPCQ7lVyTtcz1VljWFjgequ40oE+19oa4f7t6+HKwpCs2Ot pufw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CwHKJa6QSzn+fFUbr3GdLYPgv95cnerwTX8zoVbLwNQ=; b=d3oFV1tuS1eEsKV21wORfl2p9LzM0u2/cTOk0rKazn/BVCmDOMswAMDEbaSQ84vzBy pIYGt+lkNs1iXJ+aLFjpv+S9AUL1HMFZudRRXduThyZ68wElYSOlKM3vz5DV5SjGf0Zj TrBcWNe5ZYgr4hwrk4qZ7MybbgXKwlea/CqtD79cHckMVv2rTWokPW6W0uIkl0AN5efd 9u3x9XenNcuxv0BtlAIr+2tcUe2memzaB/qhp7/eXPx5PX8VaigkIzlltxSuANhKULWq z5r1TofqO8/guOWC+IIZ6XQkU6of/gOzHXBNgqwNNSwIzYSuefT44KTeaDGV78nvBIaa 77Vg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=VL291Bsm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 25-20020a17090a001900b00237413f8c6esi6442712pja.154.2023.03.13.04.47.32; Mon, 13 Mar 2023 04:47:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=VL291Bsm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230287AbjCMLa1 (ORCPT + 99 others); Mon, 13 Mar 2023 07:30:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55212 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230215AbjCMLaP (ORCPT ); Mon, 13 Mar 2023 07:30:15 -0400 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39CBF4DE14 for ; Mon, 13 Mar 2023 04:29:57 -0700 (PDT) Received: by mail-pl1-x62f.google.com with SMTP id u5so12549026plq.7 for ; Mon, 13 Mar 2023 04:29:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678706997; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CwHKJa6QSzn+fFUbr3GdLYPgv95cnerwTX8zoVbLwNQ=; b=VL291BsmdjOKqQ02b55ZkiNe1Gxiw0B9HIG72ICowrQQIz60/2k76pkbV6ohUH0jkm Jox9Y09fVi77XloTu403ELBMBr76X8dzxeQ3haW2GgqK0Esu9ZkoXpWzHaSH6tAaMoDV tmDAGCPgOSqsKKkamJFDdXSO6nIaXZBF40p+RTjEGc53dohjfIGG8/C+G7GvyIESgath WeikExSTqhrXJdcJWm/xMTEGyKADTPQDEs+Ees4T5pfkypk5jFJZpd/7M1ecIk1aLgTn YzvxNO/avEaPXMEeDvs/G5Ff4veN37dFXuRX65vfKGciMuehn4VVeyzAB73LmFAqM9LG sxIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678706997; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CwHKJa6QSzn+fFUbr3GdLYPgv95cnerwTX8zoVbLwNQ=; b=xxYdK0ETSpv07mu7DxsKn8vdTgXep7G0mZN2EZhS+4vcoj6xXRJWV/ZnW8UZBTqKlU W7pkoKKRse/pGvOxwZ6mD8K4rcEGYx2E9kJ3lj7bx72uSeHWSEcvERUXQ6AjlD39SU/9 rioD8KXEcAcOG+REy1ggA3oUF0Arx0VpxD+TChNRGb9j7ZOcI8CtbGo91u7aISy55nOy L1IpqEuGBtq0s5iZUaiy/Q5TrC9pdUdQO7CuHJQ0sIFYx4W9vj2Q584St7JuTvA9j4LI tEuY5EcU4lkg8shtHtGfFei+Ks0/FGWy65DEx0UAvZxeTeGtILhYC9E1AclajQFN551p 0TXA== X-Gm-Message-State: AO0yUKU9GdbvodDFvVK+hBLRiR+8b5/1Fb+ZVnwn33ysX2f79e0441H8 K0HIWOOQR90wQ9oNt//TLxbLkQ== X-Received: by 2002:a17:90a:990e:b0:233:a836:15f4 with SMTP id b14-20020a17090a990e00b00233a83615f4mr12917519pjp.1.1678706996712; Mon, 13 Mar 2023 04:29:56 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id n2-20020a654882000000b0050300a7c8c2sm4390827pgs.89.2023.03.13.04.29.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 04:29:56 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, christian.koenig@amd.com, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng , Vlastimil Babka Subject: [PATCH v5 3/8] mm: vmscan: make memcg slab shrink lockless Date: Mon, 13 Mar 2023 19:28:14 +0800 Message-Id: <20230313112819.38938-4-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230313112819.38938-1-zhengqi.arch@bytedance.com> References: <20230313112819.38938-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760252988141643396?= X-GMAIL-MSGID: =?utf-8?q?1760252988141643396?= Like global slab shrink, this commit also uses SRCU to make memcg slab shrink lockless. We can reproduce the down_read_trylock() hotspot through the following script: ``` DIR="/root/shrinker/memcg/mnt" do_create() { mkdir -p /sys/fs/cgroup/memory/test mkdir -p /sys/fs/cgroup/perf_event/test echo 4G > /sys/fs/cgroup/memory/test/memory.limit_in_bytes for i in `seq 0 $1`; do mkdir -p /sys/fs/cgroup/memory/test/$i; echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; echo $$ > /sys/fs/cgroup/perf_event/test/cgroup.procs; mkdir -p $DIR/$i; done } do_mount() { for i in `seq $1 $2`; do mount -t tmpfs $i $DIR/$i; done } do_touch() { for i in `seq $1 $2`; do echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; echo $$ > /sys/fs/cgroup/perf_event/test/cgroup.procs; dd if=/dev/zero of=$DIR/$i/file$i bs=1M count=1 & done } case "$1" in touch) do_touch $2 $3 ;; test) do_create 4000 do_mount 0 4000 do_touch 0 3000 ;; *) exit 1 ;; esac ``` Save the above script, then run test and touch commands. Then we can use the following perf command to view hotspots: perf top -U -F 999 1) Before applying this patchset: 32.31% [kernel] [k] down_read_trylock 19.40% [kernel] [k] pv_native_safe_halt 16.24% [kernel] [k] up_read 15.70% [kernel] [k] shrink_slab 4.69% [kernel] [k] _find_next_bit 2.62% [kernel] [k] shrink_node 1.78% [kernel] [k] shrink_lruvec 0.76% [kernel] [k] do_shrink_slab 2) After applying this patchset: 27.83% [kernel] [k] _find_next_bit 16.97% [kernel] [k] shrink_slab 15.82% [kernel] [k] pv_native_safe_halt 9.58% [kernel] [k] shrink_node 8.31% [kernel] [k] shrink_lruvec 5.64% [kernel] [k] do_shrink_slab 3.88% [kernel] [k] mem_cgroup_iter At the same time, we use the following perf command to capture IPC information: perf stat -e cycles,instructions -G test -a --repeat 5 -- sleep 10 1) Before applying this patchset: Performance counter stats for 'system wide' (5 runs): 454187219766 cycles test ( +- 1.84% ) 78896433101 instructions test # 0.17 insn per cycle ( +- 0.44% ) 10.0020430 +- 0.0000366 seconds time elapsed ( +- 0.00% ) 2) After applying this patchset: Performance counter stats for 'system wide' (5 runs): 841954709443 cycles test ( +- 15.80% ) (98.69%) 527258677936 instructions test # 0.63 insn per cycle ( +- 15.11% ) (98.68%) 10.01064 +- 0.00831 seconds time elapsed ( +- 0.08% ) We can see that IPC drops very seriously when calling down_read_trylock() at high frequency. After using SRCU, the IPC is at a normal level. Signed-off-by: Qi Zheng Acked-by: Kirill Tkhai Acked-by: Vlastimil Babka --- mm/vmscan.c | 45 ++++++++++++++++++++++++++------------------- 1 file changed, 26 insertions(+), 19 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index db2ed6e08f67..ce7834030f75 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -222,8 +222,21 @@ static inline int shrinker_defer_size(int nr_items) static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, int nid) { - return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - lockdep_is_held(&shrinker_rwsem)); + return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu, + lockdep_is_held(&shrinker_rwsem)); +} + +static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, + int nid) +{ + return srcu_dereference(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu); +} + +static void free_shrinker_info_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct shrinker_info, rcu)); } static int expand_one_shrinker_info(struct mem_cgroup *memcg, @@ -264,7 +277,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, defer_size - old_defer_size); rcu_assign_pointer(pn->shrinker_info, new); - kvfree_rcu(old, rcu); + call_srcu(&shrinker_srcu, &old->rcu, free_shrinker_info_rcu); } return 0; @@ -350,15 +363,16 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { struct shrinker_info *info; + int srcu_idx; - rcu_read_lock(); - info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); set_bit(shrinker_id, info->map); } - rcu_read_unlock(); + srcu_read_unlock(&shrinker_srcu, srcu_idx); } } @@ -372,7 +386,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) return -ENOSYS; down_write(&shrinker_rwsem); - /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -406,7 +419,7 @@ static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); } @@ -415,7 +428,7 @@ static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } @@ -893,15 +906,14 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; + int srcu_idx; int i; if (!mem_cgroup_online(memcg)) return 0; - if (!down_read_trylock(&shrinker_rwsem)) - return 0; - - info = shrinker_info_protected(memcg, nid); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; @@ -951,14 +963,9 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, i); } freed += ret; - - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } } unlock: - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); return freed; } #else /* CONFIG_MEMCG */ From patchwork Mon Mar 13 11:28:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 68768 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1133751wrd; Mon, 13 Mar 2023 04:47:37 -0700 (PDT) X-Google-Smtp-Source: AK7set+DpvjuCkd9vtipcOOgXsHS/N5t0Kn8qFPF3K9lKB/Hme/ecRYnjKOH2bImtoY7fHYWnFvj X-Received: by 2002:a17:902:f7c3:b0:1a0:4ebd:15d5 with SMTP id h3-20020a170902f7c300b001a04ebd15d5mr2169302plw.67.1678708056769; Mon, 13 Mar 2023 04:47:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678708056; cv=none; d=google.com; s=arc-20160816; b=gWxL1zCF/a6VUXoNE2Do2GtRHSqr5Leg/e4p4bWxIf1j5jDPnbY+/PfEgV4rMrQ2DP Siv8upHb4R1AJGJqFQ8QMROE3hyVyo2P4bUADRZquoMz1LP+Zy85w4VA+OTkDU9zvwrV 9dwztbZY3v+oa0+/2dWolT9U1Tvva+pHm9Omr419gibVlLI/FiAH/T/9TOM56rguimuN gX+RrFqn2HWQjB6S+06oOzUuCz7ij5ris4IK63kh2IANOMVtoK9VtedMKe+2TvmAwceF gadRJ/UuC5Cw4FM+4JCf92yU24FGUthi0cwnzdEiP9miV8K0Jvhwlcroo6DKE1UK5zVb w5tQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=joJBORpGAhUrTa0QaRmWtbHEppwsJHnBnzSS2GFBqpE=; b=fYvJhnZR69QjH8OLzMHA1t4c17+y625f8tjYAuwN9EJ3tY7FPSx/rxA25kTBi0aqR+ l6O9kKN/BYecSux9PakDzBRYcYDOJokYm68mAVoEHjO8eyWmSkBJHBTNSYNq1VmzAxm5 tUrV6O72lz7txUNxN5b6nc/spYOw6Gb8m6EUOzXBHFk8dnUTYSH0AO/6x1aDfpOboqiV M+LOUe3fdnrGldGFp73jdYUamExRMuJ6t6VbsJWQv44d8ddhqFnyZs9/Z7kiobwvztjo G8Ff5B+IkpktleBr2Uf6wVqb9yRAatiTwruDtBEjRgNguST5fp4nv/CiRTNlCzzoATzT Gebw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=gxoAK0RJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a21-20020a63e855000000b00502d6f5fb2dsi6592831pgk.805.2023.03.13.04.47.23; Mon, 13 Mar 2023 04:47:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=gxoAK0RJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230309AbjCMLam (ORCPT + 99 others); Mon, 13 Mar 2023 07:30:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55368 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230338AbjCMLa3 (ORCPT ); Mon, 13 Mar 2023 07:30:29 -0400 Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80E5D5B5D1 for ; Mon, 13 Mar 2023 04:30:03 -0700 (PDT) Received: by mail-pg1-x52b.google.com with SMTP id x37so2227816pga.1 for ; Mon, 13 Mar 2023 04:30:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678707003; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=joJBORpGAhUrTa0QaRmWtbHEppwsJHnBnzSS2GFBqpE=; b=gxoAK0RJLeAUG9DMQX3AJ19hEgYUC73gn5qDU+xe20VchIqQl4+4Bs8TMoR+xUoHKj ND9SsfWlrkOJpCkSW9SdC0rMfbp2pkoC+L+Zsl8WbNt1tGBTAG2LXwZTOyMQ1iLcCwQd GxK8VSLfc4QaraebduBNkUss03AkdhgD0s4yXld1S43G6GFxDxgVa30i3fDKmLe2+vsn L0khGXfZPxYQ4maITGQLyGizgnKgpvbt2obFC6FA/0459J8cOE4CnMELpfc1XfocweG3 UCt1lbaasgjVAeNzd0lchgP8FL0nLDv/KuKuOjPDpVlt6/LjMfKic6NSPuZGnknpbpu6 jdFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678707003; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=joJBORpGAhUrTa0QaRmWtbHEppwsJHnBnzSS2GFBqpE=; b=DoSFHOjibqoqPPp/r0SL8YX+I0VcsqopQhiotEnSi9PpWyYkY52mxKdQ9aflGPH/1o OZasjuqFarBa1P4o59cIvzAxV66ldxhZy/IVjXXPcf3raf0vhHdX5DRTp3ZaUpySAqw2 bG5HZfLjBEDPfh9wS3M923NxECHxQ++c6IdgYblangee23hQAA7XnczwQL8vVRa0qJqs Fm1P7yBvvHztE06DvSn8eQWWp555qlT9Cuyos5NE2YVHboRxk0dZL0UVVr58PS9nsVkw csOIGD12MMqk6yFDSdgm9Ux+V8jcgKcoWje8qzoFvifVCW0dk+J4zhMXwu7Pus1uI+Qn lXNg== X-Gm-Message-State: AO0yUKUCUgmbsQ11bG8kWcUFojFjcpukxQgB6gFCWZ2VXJa15jZHx6/V 15yT/nv07ctoH/HCc3d3DAbnOA== X-Received: by 2002:a05:6a00:2148:b0:606:d3d1:4cc4 with SMTP id o8-20020a056a00214800b00606d3d14cc4mr10843183pfk.3.1678707003039; Mon, 13 Mar 2023 04:30:03 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id n2-20020a654882000000b0050300a7c8c2sm4390827pgs.89.2023.03.13.04.29.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 04:30:02 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, christian.koenig@amd.com, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v5 4/8] mm: vmscan: add shrinker_srcu_generation Date: Mon, 13 Mar 2023 19:28:15 +0800 Message-Id: <20230313112819.38938-5-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230313112819.38938-1-zhengqi.arch@bytedance.com> References: <20230313112819.38938-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760252979176911550?= X-GMAIL-MSGID: =?utf-8?q?1760252979176911550?= From: Kirill Tkhai After we make slab shrink lockless with SRCU, the longest sleep unregister_shrinker() will be a sleep waiting for all do_shrink_slab() calls. To avoid long unbreakable action in the unregister_shrinker(), add shrinker_srcu_generation to restore a check similar to the rwsem_is_contendent() check that we had before. And for memcg slab shrink, we unlock SRCU and continue iterations from the next shrinker id. Signed-off-by: Kirill Tkhai Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka --- mm/vmscan.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index ce7834030f75..5c2a22454320 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -204,6 +204,7 @@ static void set_task_reclaim_state(struct task_struct *task, LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); DEFINE_SRCU(shrinker_srcu); +static atomic_t shrinker_srcu_generation = ATOMIC_INIT(0); #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -776,6 +777,7 @@ void unregister_shrinker(struct shrinker *shrinker) debugfs_entry = shrinker_debugfs_remove(shrinker); up_write(&shrinker_rwsem); + atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); debugfs_remove_recursive(debugfs_entry); @@ -797,6 +799,7 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); + atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -906,18 +909,20 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; - int srcu_idx; - int i; + int srcu_idx, generation; + int i = 0; if (!mem_cgroup_online(memcg)) return 0; +again: srcu_idx = srcu_read_lock(&shrinker_srcu); info = shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, info->map_nr_max) { + generation = atomic_read(&shrinker_srcu_generation); + for_each_set_bit_from(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -963,6 +968,11 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, i); } freed += ret; + if (atomic_read(&shrinker_srcu_generation) != generation) { + srcu_read_unlock(&shrinker_srcu, srcu_idx); + i++; + goto again; + } } unlock: srcu_read_unlock(&shrinker_srcu, srcu_idx); @@ -1002,7 +1012,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, { unsigned long ret, freed = 0; struct shrinker *shrinker; - int srcu_idx; + int srcu_idx, generation; /* * The root memcg might be allocated even though memcg is disabled @@ -1016,6 +1026,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, srcu_idx = srcu_read_lock(&shrinker_srcu); + generation = atomic_read(&shrinker_srcu_generation); list_for_each_entry_srcu(shrinker, &shrinker_list, list, srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc = { @@ -1028,6 +1039,11 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; freed += ret; + + if (atomic_read(&shrinker_srcu_generation) != generation) { + freed = freed ? : 1; + break; + } } srcu_read_unlock(&shrinker_srcu, srcu_idx); From patchwork Mon Mar 13 11:28:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 68786 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1136442wrd; Mon, 13 Mar 2023 04:54:34 -0700 (PDT) X-Google-Smtp-Source: AK7set/UobJtZa7NEa56FNguS8FPEb2NOyZnQ46YOzF/CMn49eX1Knbb8DLe43j4+WemAhj3vUvU X-Received: by 2002:a05:6a20:a89d:b0:d4:489f:57d3 with SMTP id ca29-20020a056a20a89d00b000d4489f57d3mr3294541pzb.45.1678708473866; Mon, 13 Mar 2023 04:54:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678708473; cv=none; d=google.com; s=arc-20160816; b=PNo/IiLOfNWDXAluE6yzgFlEJBiu0f31fQaW22UpQaZ9h/lKMYtaHSa+/MdpFA+34I ve+slB/wKotVUcBvQhOOKOf+nvm8jCSyAuyMs5fviYguN6t5tqw/1KK1xQdjwtC4ysht wsYJ+iHojN5tnHcZvoDtuRepdeVVy6pzelxUxVGWdqNSZZEVrbV8f24ASB7KIVipzwjb SnTxk//G+FZBhLRfjJX8X/PWrmNifaO4NzRQqddBuiBIVSAKzMR+zUkOVmZq0EI9bxkd mlsfRHgPOoB7VgtNpzcou85NwY9QwzFIjyszdC2hT9PTeF2KHge8th3swKylQvO7RHbT GLxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Oxd2ThxTJ6UijCJ4SDDL7FUC7F7cO/btWP9US9pLSzo=; b=lCRsFwbcIcDv6C079PRVn7jjkgXPn/jZodoTaL1c6bqljK7B7jNIJFb2hxQ+buiR7V lpTbhpAfG3Y6u+uTl+y2Taxy7oloInbPiPYJT4oQTXNhMQKIhGwJ6av1wtCenyUPFIW0 cMZo+/j/TuJjqol76Z9WJe0rrj2nyWi5Dz17R/iTjg/7aetnvsh/mWVJkdH6lYrLTe6y J76EOANvssZFo9mfqy2DOkknsFSkRxdbpNYHKAetwunrH1grsTEV0Ir4Q/0do76++bWB idCfTMG/fzicqpcCm38YJRWijNncQS7VYdVATgXxxRZvVBOyTxKRLMFRiWH+ZNbs5XSx 3ldQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=MvlGryeR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h6-20020aa79f46000000b005d8fd3b36ecsi184082pfr.228.2023.03.13.04.54.19; Mon, 13 Mar 2023 04:54:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=MvlGryeR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230194AbjCMLaZ (ORCPT + 99 others); Mon, 13 Mar 2023 07:30:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230162AbjCMLaN (ORCPT ); Mon, 13 Mar 2023 07:30:13 -0400 Received: from mail-pl1-x631.google.com (mail-pl1-x631.google.com [IPv6:2607:f8b0:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CFBEA570AC for ; Mon, 13 Mar 2023 04:30:09 -0700 (PDT) Received: by mail-pl1-x631.google.com with SMTP id u5so12549651plq.7 for ; Mon, 13 Mar 2023 04:30:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678707009; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Oxd2ThxTJ6UijCJ4SDDL7FUC7F7cO/btWP9US9pLSzo=; b=MvlGryeRy82G7VkgqEKuMpQfzAezLp2xXrHbHKrQM/eEt81Oq/AfYVdlB/GIobl8It 7j2/egtK7yEFhJCwQO1OEf29hlspWlTFr6UYJKW30dRAjzdPoQtwo+yeUuFCO5XjM1fR QTpRAX/3KhabhS2FMYkh8JFMbk32znCcnVB0I8hQW5KZXN6S0Oxh6LXvEwjcTlYALXpV G6Kh9bDRNBabSRjM+xuJvnmQWOWsrkeBNmVU+B+cq0i32zf+AmsgqKMTYjYBMPOnHkL9 Scfe6Zs4rWOLcpW74stS1wV1wpoCCzVwzmigR2g2N6q5dgesopZRoP+7gE0A+6gqZQ3p 5mjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678707009; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Oxd2ThxTJ6UijCJ4SDDL7FUC7F7cO/btWP9US9pLSzo=; b=3JxYf2QSH3XS4UaJemiAHwqhbY8tJDc3FGUBxeTwnJYBTVU/Phz0A+tptEBxuya4N4 ho7lgbRpwKeEZfF1IKt36HH5Ta0D5yMiBJ2PYf3zfF9N60b47pDS3TDMGDVMZQlL+q8V HLLpZpUlHyxhvEC9GEeEhGGvR/99tGrfCUpHcPw1+zc0GM+zcI6BHV1EQCsQ4uL2t0/A VCmG689yP5v9HjRjj0BIoTjM5GDRPO6JKJGnBXeIE4mNesFswxV5XgEsbeEop7nnWvd0 H0rzSVMKiSaGejWQX7cV5CWGREH4DiQ3O/u+EYo/DD855Tf48LJtCth+HONG0Nu3eAob gjHQ== X-Gm-Message-State: AO0yUKWNLuxUNd0L66Wqcr4+E85sDoXtBwKExFQdRY+3czVVhoN8g99c Bsx4NigkK1NBD83wOtfBITxqLg== X-Received: by 2002:a05:6a20:6914:b0:cc:4118:65c4 with SMTP id q20-20020a056a20691400b000cc411865c4mr12735774pzj.5.1678707009311; Mon, 13 Mar 2023 04:30:09 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id n2-20020a654882000000b0050300a7c8c2sm4390827pgs.89.2023.03.13.04.30.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 04:30:08 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, christian.koenig@amd.com, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v5 5/8] mm: shrinkers: make count and scan in shrinker debugfs lockless Date: Mon, 13 Mar 2023 19:28:16 +0800 Message-Id: <20230313112819.38938-6-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230313112819.38938-1-zhengqi.arch@bytedance.com> References: <20230313112819.38938-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760253416638878839?= X-GMAIL-MSGID: =?utf-8?q?1760253416638878839?= Like global and memcg slab shrink, also use SRCU to make count and scan operations in memory shrinker debugfs lockless. Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- mm/shrinker_debug.c | 25 ++++++++----------------- 1 file changed, 8 insertions(+), 17 deletions(-) diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 39c3491e28a3..37d54d037495 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -5,10 +5,12 @@ #include #include #include +#include /* defined in vmscan.c */ extern struct rw_semaphore shrinker_rwsem; extern struct list_head shrinker_list; +extern struct srcu_struct shrinker_srcu; static DEFINE_IDA(shrinker_debugfs_ida); static struct dentry *shrinker_debugfs_root; @@ -49,18 +51,13 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) struct mem_cgroup *memcg; unsigned long total; bool memcg_aware; - int ret, nid; + int ret = 0, nid, srcu_idx; count_per_node = kcalloc(nr_node_ids, sizeof(unsigned long), GFP_KERNEL); if (!count_per_node) return -ENOMEM; - ret = down_read_killable(&shrinker_rwsem); - if (ret) { - kfree(count_per_node); - return ret; - } - rcu_read_lock(); + srcu_idx = srcu_read_lock(&shrinker_srcu); memcg_aware = shrinker->flags & SHRINKER_MEMCG_AWARE; @@ -91,8 +88,7 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) } } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); - rcu_read_unlock(); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); kfree(count_per_node); return ret; @@ -115,9 +111,8 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, .gfp_mask = GFP_KERNEL, }; struct mem_cgroup *memcg = NULL; - int nid; + int nid, srcu_idx; char kbuf[72]; - ssize_t ret; read_len = size < (sizeof(kbuf) - 1) ? size : (sizeof(kbuf) - 1); if (copy_from_user(kbuf, buf, read_len)) @@ -146,11 +141,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, return -EINVAL; } - ret = down_read_killable(&shrinker_rwsem); - if (ret) { - mem_cgroup_put(memcg); - return ret; - } + srcu_idx = srcu_read_lock(&shrinker_srcu); sc.nid = nid; sc.memcg = memcg; @@ -159,7 +150,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, shrinker->scan_objects(shrinker, &sc); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); mem_cgroup_put(memcg); return size; From patchwork Mon Mar 13 11:28:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 68787 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1136471wrd; Mon, 13 Mar 2023 04:54:39 -0700 (PDT) X-Google-Smtp-Source: AK7set/xIcVuThaF3Q+jYdmuo5tUIbV91WlGnxvJJL2mrtGrVns7AHNR4TkmReS2DinRzh2iE9mD X-Received: by 2002:a17:90a:1913:b0:233:76bd:9faa with SMTP id 19-20020a17090a191300b0023376bd9faamr35808727pjg.47.1678708479054; Mon, 13 Mar 2023 04:54:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678708479; cv=none; d=google.com; s=arc-20160816; b=Jmzk8Jbpou+Uq81ub1cCDuzVALqSw0pSTxmIpNsMGxaw+x6+8UmIpNaYAS09NGNxnx Va079ppo4WUQCsjwFGPjLbi3DWcgzwJjsZ4WINh4BSnhXIVpwRAdtqpeAetpQiHfjEND 4SE5ajl94OfsBHmWXECqBqKsAotVKAyk/UnZuV+YMM7PRFZjLNFACPAgyeOdr2za8Z1e aRMhEusFISNcYfH8terr9TznYpX7NRuJFxOOAsJjzbKC5qZBKdHCSZhH2kvSeMpP3sQ6 UGHklYZiBLgAQsZeVQQdJQzgwJtOoCm533fTxidTJ86zMoFcZfapoUxOg6Sw2zebyARv 5C4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=H5UXC4Y8L+sYixm/ZiEHHormamWIB6IrtYM6DyQ3d6c=; b=f/rqgHATuIia2vFp4fz56dew90ezzeDbckDpZS8tcHVEmcG9yFI5uwqvRZyxcznpDp uot9LriWGOhKxx88DCeRhpU4b11TwijbwCLOrD50+B6tzIORR8JHg91yXO4I0oSuR4Hm txAru8neku3rcbw8AeVhBLkiMSodUw9QnC/oxm061+6Pdjl9saYFoAeunFzDlO2EYBLj RGVUdbbhIWAREb1hgXpO34KnXSRbSRKfRbVC0l/tyjdt7yLG/m/Zn2uaMz/CRgI5T3eF s2syuywpFZsB0nzH06AQ9EKcS5xlujH67pPClFz7uFi9W+HQYPklLnsH5Dlb5+NMQM7G C+qg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=ItJnDFmM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t1-20020a17090abc4100b0023417fa8d32si6554428pjv.20.2023.03.13.04.54.26; Mon, 13 Mar 2023 04:54:39 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=ItJnDFmM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230411AbjCMLak (ORCPT + 99 others); Mon, 13 Mar 2023 07:30:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55940 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230164AbjCMLa1 (ORCPT ); Mon, 13 Mar 2023 07:30:27 -0400 Received: from mail-pl1-x62c.google.com (mail-pl1-x62c.google.com [IPv6:2607:f8b0:4864:20::62c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1129D5849B for ; Mon, 13 Mar 2023 04:30:16 -0700 (PDT) Received: by mail-pl1-x62c.google.com with SMTP id k2so4674684pll.8 for ; Mon, 13 Mar 2023 04:30:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678707015; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H5UXC4Y8L+sYixm/ZiEHHormamWIB6IrtYM6DyQ3d6c=; b=ItJnDFmMsgazPNOaXu8i9CtcCnSzNN57aJj3idjpUOYKHDhuP27hoj7ePdkQNEOby+ riTdnuWQLKyP3nQPylZlMs6mnXVeCSnfHP6Q7AeVCEJSYIy+l6HW9Tbx7RDnuCzBTX04 lg/9ublTRynkmyfmxHDT8KqSD1ZrXcBSzOxYBDdTI0PM8nEOM7P/oyB09c6XGMeI9FY0 ZynsjAfzuUGtByb52mixStZA+f5WzLTbVCttXhkplGLfpxGeI3fKJtOb5yyJciSuqDaa kNFAiFqnIfkQtLxoqnTuMDedbMN6EiT6dScRhSmKgSpp0QGmx1ct8f81dbBiredODvuA ZyxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678707015; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H5UXC4Y8L+sYixm/ZiEHHormamWIB6IrtYM6DyQ3d6c=; b=pHAIhElgJXev6bGADw6nqK58a6+1bA8STqTctBzMzs+cnn9VQnoSNDEGbcQQauAXPl PungvdgrKJO4TAHT94ftp78DvHD8BEVqCSjDi5EGsEjY6tFapKyrEMTuL3VFFj9FCMYt igVMLTE/uNNm6d1uLK+CYbzX4zFR6Vf0Iims/6J3HdocX/NPJhJRObFBZj4S5y1fllY0 C8Tt9UgAZ5rHrob4YML/8PATDWlOx3yP9jiHF9jUM1vj6CBJ84lW+daAiTDNDrTHi6KJ DcqcW2ppUtfGYfXoA3zTb9kXecYQO7rfZChNNO+gV6soZRqFqGhVquNJLOwkCRdROpYD A5Dw== X-Gm-Message-State: AO0yUKVTmbq3+h14hpNj6GiJdbt2mTDp0b9J8KaUwHcfVaFFUGAxdDIA BJTqMkfIi7HYAKFnCk8fHqXNJg== X-Received: by 2002:a05:6a20:440d:b0:cd:18a2:f6cc with SMTP id ce13-20020a056a20440d00b000cd18a2f6ccmr13960885pzb.3.1678707015460; Mon, 13 Mar 2023 04:30:15 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id n2-20020a654882000000b0050300a7c8c2sm4390827pgs.89.2023.03.13.04.30.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 04:30:15 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, christian.koenig@amd.com, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v5 6/8] mm: vmscan: hold write lock to reparent shrinker nr_deferred Date: Mon, 13 Mar 2023 19:28:17 +0800 Message-Id: <20230313112819.38938-7-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230313112819.38938-1-zhengqi.arch@bytedance.com> References: <20230313112819.38938-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760253422091712306?= X-GMAIL-MSGID: =?utf-8?q?1760253422091712306?= For now, reparent_shrinker_deferred() is the only holder of read lock of shrinker_rwsem. And it already holds the global cgroup_mutex, so it will not be called in parallel. Therefore, in order to convert shrinker_rwsem to shrinker_mutex later, here we change to hold the write lock of shrinker_rwsem to reparent. Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 5c2a22454320..8c1ae7ea8dea 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -445,7 +445,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_read(&shrinker_rwsem); + down_write(&shrinker_rwsem); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -454,7 +454,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_read(&shrinker_rwsem); + up_write(&shrinker_rwsem); } static bool cgroup_reclaim(struct scan_control *sc) From patchwork Mon Mar 13 11:28:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 68793 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1137412wrd; Mon, 13 Mar 2023 04:57:20 -0700 (PDT) X-Google-Smtp-Source: AK7set9ppZ1k4MYCOw0PVINjimgDVZnGpbkPtk+LxUifJW4gJOu8b2fiqPDo4kQHw7UqsnmE8dBt X-Received: by 2002:a05:6a20:1a12:b0:cc:8e18:420f with SMTP id cj18-20020a056a201a1200b000cc8e18420fmr30082783pzb.35.1678708640629; Mon, 13 Mar 2023 04:57:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678708640; cv=none; d=google.com; s=arc-20160816; b=dPmJwf1FotoFdd4cFVkN0qAr+i4OvMqAvqhn7BsFmRjYN6cquZ8WIRCWUw3sAsgvZz Vu/MdtmIp4ojVm/H7AE2E7NexrvvQ7rXDHDbaZbwBGqfS8mE1Race3N7d0ZoDuPbCOLg 7+bj1jBcY+Id3jBDQJNP/zc0Q2Cet8aZ/j1u35IpWlTpgF+UCiTjI99uDfoexipqnEs7 6ZIFAM7FOARLHy/EuSgW+mmmcvPZMuqPWqsHKxOysC2SJBdF3LjNB/55CWb+8uIeUhMf dUZvBa0xN1fg1JNNfDz9Bp66ZXI5s8fEPNNz3gQz3IeYQNetmx0E9rg7+IdmJEeNkR01 fxCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0Z4Tof/PnwUFj5i0NNzN56+j5kNnB7gWDICsVdyoruc=; b=WE1XiHowkXnHDpfsOlxfA9+lniDYNFlZdY8fVb2nWe/5vW9SofjQA6SIEamdgXxBDW RQuCsyO8XYhw53IliGxiZqljL64umRZt6DQ12vFp5o6H8tQTEMZ88zVbBAVLcjZmgoWO AqitBx2eLCNTVud8MbfGEnkw6rDGmP46a4nw+fTgfUOUpNBo8xzPALGvDd6jqcNUjP9t gxBq8TgGJaBsvXof6aePEmG+pDYlrOh1vW8O3GpypByGqxd5RmXfTwIfHPYHZMk4pvrg vyReVmw6TP8OBzueTRjkPoirKe8rt3/cqA9iJw9Ig6PuPc6IouBj8spC+Aibo3Q5HAyw 0aqA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=LueY7xOn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u4-20020a655c04000000b004faeaac4dacsi6533859pgr.753.2023.03.13.04.57.07; Mon, 13 Mar 2023 04:57:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=LueY7xOn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230055AbjCMLba (ORCPT + 99 others); Mon, 13 Mar 2023 07:31:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56384 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230336AbjCMLbC (ORCPT ); Mon, 13 Mar 2023 07:31:02 -0400 Received: from mail-pg1-x52e.google.com (mail-pg1-x52e.google.com [IPv6:2607:f8b0:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BCD1962B53 for ; Mon, 13 Mar 2023 04:30:22 -0700 (PDT) Received: by mail-pg1-x52e.google.com with SMTP id 132so6666485pgh.13 for ; Mon, 13 Mar 2023 04:30:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678707022; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0Z4Tof/PnwUFj5i0NNzN56+j5kNnB7gWDICsVdyoruc=; b=LueY7xOnIXHIbNor8k5oEpAk+tX93l8rCkl4eUPXSjj/FR0coKA+VxIcbga8n7x+3/ UVYQ+xwM5v7aAKibviyLXFIf9yfA5DqdUrbSgL59K0+yWXOscDZ9kE0PPEbSKSAKGOeW fYdcX0uAb27i11e16ubA0DvK2mQmR+nk8oZAtv85shH1l8grZ50ghPaV1UKNldaTWetj vt+eouSQMr0HwGA/8pBbk59dg1Pgdhd0cKvtaYLaKLCdPxJ8k+oWK21Mgq9NtpfoT2OG W0HAgtKswNDuPDIH/Ey2qVGkFfAIGkpwaECFhXVuXnSl/4V4k4XFXHrDTRlXi/isujEx xkxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678707022; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0Z4Tof/PnwUFj5i0NNzN56+j5kNnB7gWDICsVdyoruc=; b=nNUI68runvX8mQmO8km0VUGlP4ncRHDzN/N/eiwRc0Cz6Eqh4e/13dL18Ktgg3Rzmr bmykdNOihfNhI6t753yqO8j+YCTj1+TRBir7oYUHeRaqJ9Q/4tAdES0Q1K3e4cGNJH1b mjIl/hFq2qsUuh0UzEQvH+sJ4HpSfhCTTXZsISCQjaqcAlEZyJ5J0v23NyzbKt/aEJd7 DSnlv7+6fnwxMN4vD/6fMWVou4YnTsvBM0skdvi2H0ZB59D8Ed7jNrxDYsoY4W1oAiYV sEaJAm9ena/kaRh5aMcJgQs5s+vkivTYbf7sFOQZ1wJ51gdy+nGhEYO4EbbpMcHmh8nT b2Ew== X-Gm-Message-State: AO0yUKUEf+Mb938cW1BWhe9iUETJdGl8emeYAhGLMzhPHxFgWw1KZ437 Dq3YSxDOVDAefnAQEb+Jmtxlhg== X-Received: by 2002:a05:6a00:2148:b0:606:d3d1:4cc4 with SMTP id o8-20020a056a00214800b00606d3d14cc4mr10844065pfk.3.1678707021778; Mon, 13 Mar 2023 04:30:21 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id n2-20020a654882000000b0050300a7c8c2sm4390827pgs.89.2023.03.13.04.30.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 04:30:21 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, christian.koenig@amd.com, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v5 7/8] mm: vmscan: remove shrinker_rwsem from synchronize_shrinkers() Date: Mon, 13 Mar 2023 19:28:18 +0800 Message-Id: <20230313112819.38938-8-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230313112819.38938-1-zhengqi.arch@bytedance.com> References: <20230313112819.38938-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760253591608484412?= X-GMAIL-MSGID: =?utf-8?q?1760253591608484412?= Currently, the synchronize_shrinkers() is only used by TTM pool. It only requires that no shrinkers run in parallel, and doesn't care about registering and unregistering of shrinkers. Since slab shrink is protected by SRCU, synchronize_srcu() is sufficient to ensure that no shrinker is running in parallel. So the shrinker_rwsem in synchronize_shrinkers() is no longer needed, just remove it. Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- mm/vmscan.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 8c1ae7ea8dea..2b22a42d83c4 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -790,15 +790,11 @@ EXPORT_SYMBOL(unregister_shrinker); /** * synchronize_shrinkers - Wait for all running shrinkers to complete. * - * This is equivalent to calling unregister_shrink() and register_shrinker(), - * but atomically and with less overhead. This is useful to guarantee that all - * shrinker invocations have seen an update, before freeing memory, similar to - * rcu. + * This is useful to guarantee that all shrinker invocations have seen an + * update, before freeing memory. */ void synchronize_shrinkers(void) { - down_write(&shrinker_rwsem); - up_write(&shrinker_rwsem); atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } From patchwork Mon Mar 13 11:28:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 68773 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1135568wrd; Mon, 13 Mar 2023 04:52:19 -0700 (PDT) X-Google-Smtp-Source: AK7set/JCjL9Z45aE/D4EfExy+UsJcUgjk46+kN9W6DdeyRB9Y8yCE5DZv9xKh5pb/8zHffsqJLO X-Received: by 2002:a17:903:2282:b0:19a:a9d8:e47f with SMTP id b2-20020a170903228200b0019aa9d8e47fmr42997558plh.36.1678708339590; Mon, 13 Mar 2023 04:52:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678708339; cv=none; d=google.com; s=arc-20160816; b=LOfb9LI0b3WHFTmny3TsSOoK0zRdUKyryWGfz49w9zbtQHSsSwMC3QelCSpNeCgn8x FyML42MkTLz/2P/f+YfNJCNFYj0u1gmE8Gg52bIGAPEChDoefCze25vzW2WPYYUNP57Z mtEwaXxeLNfpgmmkEJZP2vk91NZ17gtVDH6BowCugdR0cxP0MSomwwQvuYmtf+lGwFmd 4HH5w1yrKmnd0TaG6SmJk/dcpeeQ5SbqVbz5UC4NY3qSRu8YIo9e0Lr3HSkx4hZ2W6VD ozp5MF2e35/ffvQWQePp+ENw67nO+HHkdMI9dxc9nhUQ0kThjkacNN8SVcJX8D2yA1JX zHEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=veHvRh5W1aQDA0P3DcAtGH917iTj6D7LjZJelB9m0SE=; b=yKJ96YhHVf8gs7u8FGyy1cAgEEHeoYAgXdnOF9pzcXiLe9f2cO4VvJl4h++5ElylXy tnFySqTIMHIZ7thFZlftCqbuav+NI5QxiBWE1qQSeLJ86WTMbTbDxAZDjKJvf6mag/7k OErWFCArQdC+a3KgQNRstKL7ZYI8DfXONuUuuWiCFsIF6UTAlQf9fkI2gJEV127BumU3 kQexbR3CsN6c7JQWc1H/BYeFXir1v5UBvWl8usXvA454wyStJ+uA7wIrZ9j9DmKVg4/b DDJbmEwlNtEpUBhELGJ3h6xipVRE4UxgCPGbZge2jWckAdKevOcWe6OR5LkyhIN6M9bV 9GJw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="SVZ/kd4y"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q22-20020a170902789600b0019cac5266b1si6606158pll.604.2023.03.13.04.52.04; Mon, 13 Mar 2023 04:52:19 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="SVZ/kd4y"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229616AbjCMLbq (ORCPT + 99 others); Mon, 13 Mar 2023 07:31:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57196 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229849AbjCMLbT (ORCPT ); Mon, 13 Mar 2023 07:31:19 -0400 Received: from mail-pg1-x52a.google.com (mail-pg1-x52a.google.com [IPv6:2607:f8b0:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 922EF4C6FC for ; Mon, 13 Mar 2023 04:30:28 -0700 (PDT) Received: by mail-pg1-x52a.google.com with SMTP id q23so6686432pgt.7 for ; Mon, 13 Mar 2023 04:30:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678707028; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=veHvRh5W1aQDA0P3DcAtGH917iTj6D7LjZJelB9m0SE=; b=SVZ/kd4yVajGRKXhMfnj08cBdN48skJmJEGIspTu2+CDNIdJP4iVJq7QNuWbryFTFQ 5WArVBLbgGw6BT+81/ZTiH5DRhGi3JDvekIZehLu2+/q199ZQJ1MzMGsH3B6ep70Nh2j O9CDfWe6oBvsg9PWBDzLHKCNF/NITz+vSqLJI6Zvlx3SIOWybhEFZsRLhi+R8hbquZII EF5ApH6pEdxse23guW/P/4ktZI7CDraqOYOVYGnfra1lYuYfsl2CmxifRbGwgg74BxXU wCwmiL2GpNFoGQ1kXBpQ/7VvFCZVNNvKkhzQxx4qZOb24QFhzFh+GYU9RthjQ0RmrpAP KQEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678707028; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=veHvRh5W1aQDA0P3DcAtGH917iTj6D7LjZJelB9m0SE=; b=JtqEQ93g3hTofb2OW6+1uSCip/vUN0B+V/PvLuVA6edCG5+uv2aRauQws3VhUJtDA/ XJLrdkd7+tXV8Vk5AyeuRDJ4sdHZXj87/q4FMdaYdaD/IvvfxqGOjDu4Bku7GLg+bwwc k4Twv0dG0leXKSxJWr3aZrH14r/kg41tjumQfeQzNZd4DQrNjqdfkCY2dN72RPWOJrik oldyThsUXpXNS80Z/wee4Wix/DhrRGibJpZhvPNZfw+TK8HWP2zPjgKQxWNg8QAuaVuV w+3jQ/jmH4uYbTniqGYIgcWZgFLPuRc6A1iLjCeYmTQHzJSj3E+0LBL5gtZ9h49ewCbp wADg== X-Gm-Message-State: AO0yUKV6on+PfXkCFrPL7hKsZCIkYZmjP7O7/8+waHYPLv7v7qfm95Z8 aryhdoYwBSSMplnK7w+CMuZhkA== X-Received: by 2002:a62:8784:0:b0:623:77a8:8f65 with SMTP id i126-20020a628784000000b0062377a88f65mr4740537pfe.3.1678707027920; Mon, 13 Mar 2023 04:30:27 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id n2-20020a654882000000b0050300a7c8c2sm4390827pgs.89.2023.03.13.04.30.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Mar 2023 04:30:27 -0700 (PDT) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, christian.koenig@amd.com, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v5 8/8] mm: shrinkers: convert shrinker_rwsem to mutex Date: Mon, 13 Mar 2023 19:28:19 +0800 Message-Id: <20230313112819.38938-9-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230313112819.38938-1-zhengqi.arch@bytedance.com> References: <20230313112819.38938-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760253275751654402?= X-GMAIL-MSGID: =?utf-8?q?1760253275751654402?= Now there are no readers of shrinker_rwsem, so we can simply replace it with mutex lock. Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- drivers/md/dm-cache-metadata.c | 2 +- drivers/md/dm-thin-metadata.c | 2 +- fs/super.c | 2 +- mm/shrinker_debug.c | 14 +++++++------- mm/vmscan.c | 34 +++++++++++++++++----------------- 5 files changed, 27 insertions(+), 27 deletions(-) diff --git a/drivers/md/dm-cache-metadata.c b/drivers/md/dm-cache-metadata.c index acffed750e3e..9e0c69958587 100644 --- a/drivers/md/dm-cache-metadata.c +++ b/drivers/md/dm-cache-metadata.c @@ -1828,7 +1828,7 @@ int dm_cache_metadata_abort(struct dm_cache_metadata *cmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * cmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs cmd root_lock). - * - must take shrinker_rwsem without holding cmd->root_lock + * - must take shrinker_mutex without holding cmd->root_lock */ new_bm = dm_block_manager_create(cmd->bdev, DM_CACHE_METADATA_BLOCK_SIZE << SECTOR_SHIFT, CACHE_MAX_CONCURRENT_LOCKS); diff --git a/drivers/md/dm-thin-metadata.c b/drivers/md/dm-thin-metadata.c index fd464fb024c3..9f5cb52c5763 100644 --- a/drivers/md/dm-thin-metadata.c +++ b/drivers/md/dm-thin-metadata.c @@ -1887,7 +1887,7 @@ int dm_pool_abort_metadata(struct dm_pool_metadata *pmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * pmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs pmd root_lock). - * - must take shrinker_rwsem without holding pmd->root_lock + * - must take shrinker_mutex without holding pmd->root_lock */ new_bm = dm_block_manager_create(pmd->bdev, THIN_METADATA_BLOCK_SIZE << SECTOR_SHIFT, THIN_MAX_CONCURRENT_LOCKS); diff --git a/fs/super.c b/fs/super.c index 84332d5cb817..91a4037b1d95 100644 --- a/fs/super.c +++ b/fs/super.c @@ -54,7 +54,7 @@ static char *sb_writers_name[SB_FREEZE_LEVELS] = { * One thing we have to be careful of with a per-sb shrinker is that we don't * drop the last active reference to the superblock from within the shrinker. * If that happens we could trigger unregistering the shrinker from within the - * shrinker path and that leads to deadlock on the shrinker_rwsem. Hence we + * shrinker path and that leads to deadlock on the shrinker_mutex. Hence we * take a passive reference to the superblock to avoid this from occurring. */ static unsigned long super_cache_scan(struct shrinker *shrink, diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 37d54d037495..fdd155fd35ed 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -8,7 +8,7 @@ #include /* defined in vmscan.c */ -extern struct rw_semaphore shrinker_rwsem; +extern struct mutex shrinker_mutex; extern struct list_head shrinker_list; extern struct srcu_struct shrinker_srcu; @@ -168,7 +168,7 @@ int shrinker_debugfs_add(struct shrinker *shrinker) char buf[128]; int id; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); /* debugfs isn't initialized yet, add debugfs entries later. */ if (!shrinker_debugfs_root) @@ -211,7 +211,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) if (!new) return -ENOMEM; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); old = shrinker->name; shrinker->name = new; @@ -229,7 +229,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) shrinker->debugfs_entry = entry; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); kfree_const(old); @@ -241,7 +241,7 @@ struct dentry *shrinker_debugfs_remove(struct shrinker *shrinker) { struct dentry *entry = shrinker->debugfs_entry; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); kfree_const(shrinker->name); shrinker->name = NULL; @@ -266,14 +266,14 @@ static int __init shrinker_debugfs_init(void) shrinker_debugfs_root = dentry; /* Create debugfs entries for shrinkers registered at boot */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_for_each_entry(shrinker, &shrinker_list, list) if (!shrinker->debugfs_entry) { ret = shrinker_debugfs_add(shrinker); if (ret) break; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } diff --git a/mm/vmscan.c b/mm/vmscan.c index 2b22a42d83c4..8faac4310cb5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -35,7 +35,7 @@ #include #include #include -#include +#include #include #include #include @@ -202,7 +202,7 @@ static void set_task_reclaim_state(struct task_struct *task, } LIST_HEAD(shrinker_list); -DECLARE_RWSEM(shrinker_rwsem); +DEFINE_MUTEX(shrinker_mutex); DEFINE_SRCU(shrinker_srcu); static atomic_t shrinker_srcu_generation = ATOMIC_INIT(0); @@ -225,7 +225,7 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, { return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, &shrinker_srcu, - lockdep_is_held(&shrinker_rwsem)); + lockdep_is_held(&shrinker_mutex)); } static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, @@ -304,7 +304,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) int nid, size, ret = 0; int map_size, defer_size = 0; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); map_size = shrinker_map_size(shrinker_nr_max); defer_size = shrinker_defer_size(shrinker_nr_max); size = map_size + defer_size; @@ -320,7 +320,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -336,7 +336,7 @@ static int expand_shrinker_info(int new_id) if (!root_mem_cgroup) goto out; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); map_size = shrinker_map_size(new_nr_max); defer_size = shrinker_defer_size(new_nr_max); @@ -386,7 +386,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) if (mem_cgroup_disabled()) return -ENOSYS; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -400,7 +400,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) shrinker->id = id; ret = 0; unlock: - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -410,7 +410,7 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) BUG_ON(id < 0); - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); idr_remove(&shrinker_idr, id); } @@ -445,7 +445,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -454,7 +454,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } static bool cgroup_reclaim(struct scan_control *sc) @@ -703,9 +703,9 @@ void free_prealloced_shrinker(struct shrinker *shrinker) shrinker->name = NULL; #endif if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); unregister_memcg_shrinker(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return; } @@ -715,11 +715,11 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } static int __register_shrinker(struct shrinker *shrinker) @@ -769,13 +769,13 @@ void unregister_shrinker(struct shrinker *shrinker) if (!(shrinker->flags & SHRINKER_REGISTERED)) return; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_remove(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu);