From patchwork Tue Mar 7 06:55:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 65294 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2284018wrd; Mon, 6 Mar 2023 23:03:01 -0800 (PST) X-Google-Smtp-Source: AK7set+mcECTbihB5A/U1RCMX2PeBXIrpknd2GBZErmcbqbNpwu6zs/B5bSPcicYjmVRB4scf7fn X-Received: by 2002:a17:907:8e91:b0:8e3:da5c:56f7 with SMTP id tx17-20020a1709078e9100b008e3da5c56f7mr13772684ejc.76.1678172580957; Mon, 06 Mar 2023 23:03:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678172580; cv=none; d=google.com; s=arc-20160816; b=PPzmKELg/Bk+sYJ6gB2TLBM1Tle76gwLU70oLwmWSJ/OlrsWzR8JS06HINcbL4pafB QoAuYDSNmw2JI2+PUVeQ7XJQsqN1L7JGCVOMGVCSVCHS6QqeMjby4Nx5s8wFviVIC4fR 0QPNVAAs2XtfPGYiFbd7AhIt+592dhKMIk0VPtHRpF4BJJ39NZ+GF4jx57kOYUBvvkQZ TOKlCNk/4b08ZuDUmfs6z1USs9UqxBYjA6ii31BJttqNVG6YOxU8aBQmAEfqxSOlUDCn 6S7REaF/rJS9cJwUYMck8OGN2pC59NupRFy8Yxw2NSODXzVpbqUTUyPGDjfbwlZtYn9w SOvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ER53Y4QROl37259eb6APCiUTawgi9lN6yGxvXXCwSQY=; b=uXd4po/Wit26Gmhl3G2hXLbFq8dTmg6Xw0/PA/q9q/WiGBFIIgb7V12nD9VWGceplA g7GA1OIpHAelHMrpCXE2I91jii15YR62eg0qBIg/Tm0ZvsW9R/p1f9eDc5mPGMTSO5iH tRE1Qr78OGNiz/ltKzAlvqvqkHVMSfFCkzS6FdGsxrkqaSblkQn+zmr7YjpOJO/PfrDa zA+X9USC8d+wlh/Pe49blTrrsGRwBt9xlzWCUUg41W+tzHSvJVSMHkj8oucX0ir+vM1R eReKc5zd0U8uWHNif8Ywium9lsl15wy6neuZ/tJClk2NpAvKWjh+LWvYGSdX9E/H73NS YHjA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="e8fFg/1Z"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m8-20020aa7c488000000b004c698b5018esi402573edq.347.2023.03.06.23.02.36; Mon, 06 Mar 2023 23:03:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="e8fFg/1Z"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231191AbjCGG6H (ORCPT + 99 others); Tue, 7 Mar 2023 01:58:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231179AbjCGG5e (ORCPT ); Tue, 7 Mar 2023 01:57:34 -0500 Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E8087848DB for ; Mon, 6 Mar 2023 22:56:34 -0800 (PST) Received: by mail-pj1-x102f.google.com with SMTP id p3-20020a17090ad30300b0023a1cd5065fso11048881pju.0 for ; Mon, 06 Mar 2023 22:56:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678172189; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ER53Y4QROl37259eb6APCiUTawgi9lN6yGxvXXCwSQY=; b=e8fFg/1ZMWrDBYFiJGMfpaUyPuJWpCLpWWZq+9/2MnqY7SWb0mfveVeBZnGqQEScRg SRpcjengd7bibS0+sCjtTNAacG5ZTh6UzMHGYHGhLesxbisy2K8z9u5WBafxqENJ3cle +aareyjuZRIYh2a2Es/Zb9QV30TYYKVFPL1aymJscHSeoeSE0VJP3FXWjmFFNrBEf2bt qFIAa0MZmcpqbFGrop17AAgZicTZfc+oz5iTO0ILfWsXzt07POPSqui0chCoPVnuTfpa jEpmE0LmM4BFlo/QSmO5S+SEmZWa4v5rkEHKIznIKzxalv0FQT/NqtWCuLU5TuTPr1k7 kJ+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678172189; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ER53Y4QROl37259eb6APCiUTawgi9lN6yGxvXXCwSQY=; b=OoWQe99pMt8R+oCbBgCHxTNBphX9YG6m6QhfmrkoqZNMjihvHwLSIYpSeeiMUmYEFV OUFYU87oHPipXPz9Hkk2wd9VpG2Aq/HlzVakdBwXLaY1d+2aiERye3AiXhENzZIEjHwj +73IGTR6QtXrU/iIo/5rOrFfPLlEt+ZZ/HehrUMwZ/Zk8oHeRx464Gk1fqsF7EEaXHK+ /+Gov2WBwOsRt69QMdn/UkUL6NRnwIz6zKJOYj82uVWn0cjcJCFM3hNaXsgqcaAgX+XI TATPDo24Nx2eChjRjqf5e/6fmaB2tPKrvoQbJDGSzuOEXk2zW5xdZ6YzqiG3XIV9cyTR mrRQ== X-Gm-Message-State: AO0yUKVqJsZ/LpJYHUZoKKqavxpKs3QazA3QOpfOGbKX5fO7C1z7qRrS ZyEdBLj6PxohgFQ5g76Ub+LsBg== X-Received: by 2002:a05:6a20:8404:b0:c0:2875:9e8c with SMTP id c4-20020a056a20840400b000c028759e8cmr16929564pzd.1.1678172189271; Mon, 06 Mar 2023 22:56:29 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id e5-20020a62ee05000000b00608dae58695sm7230854pfi.209.2023.03.06.22.56.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 22:56:28 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 1/8] mm: vmscan: add a map_nr_max field to shrinker_info Date: Tue, 7 Mar 2023 14:55:58 +0800 Message-Id: <20230307065605.58209-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230307065605.58209-1-zhengqi.arch@bytedance.com> References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759691492663675125?= X-GMAIL-MSGID: =?utf-8?q?1759691492663675125?= To prepare for the subsequent lockless memcg slab shrink, add a map_nr_max field to struct shrinker_info to records its own real shrinker_nr_max. Suggested-by: Kirill Tkhai Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka --- include/linux/memcontrol.h | 1 + mm/vmscan.c | 41 ++++++++++++++++++++++---------------- 2 files changed, 25 insertions(+), 17 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..aa69ea98e2d8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -97,6 +97,7 @@ struct shrinker_info { struct rcu_head rcu; atomic_long_t *nr_deferred; unsigned long *map; + int map_nr_max; }; struct lruvec_stats_percpu { diff --git a/mm/vmscan.c b/mm/vmscan.c index 9414226218f0..2dcc01682026 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -224,9 +224,16 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, lockdep_is_held(&shrinker_rwsem)); } +static inline bool need_expand(int new_nr_max, int old_nr_max) +{ + return round_up(new_nr_max, BITS_PER_LONG) > + round_up(old_nr_max, BITS_PER_LONG); +} + static int expand_one_shrinker_info(struct mem_cgroup *memcg, int map_size, int defer_size, - int old_map_size, int old_defer_size) + int old_map_size, int old_defer_size, + int new_nr_max) { struct shrinker_info *new, *old; struct mem_cgroup_per_node *pn; @@ -240,12 +247,17 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, if (!old) return 0; + /* Already expanded this shrinker_info */ + if (!need_expand(new_nr_max, old->map_nr_max)) + continue; + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); if (!new) return -ENOMEM; new->nr_deferred = (atomic_long_t *)(new + 1); new->map = (void *)new->nr_deferred + defer_size; + new->map_nr_max = new_nr_max; /* map: set all old bits, clear all new bits */ memset(new->map, (int)0xff, old_map_size); @@ -295,6 +307,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) } info->nr_deferred = (atomic_long_t *)(info + 1); info->map = (void *)info->nr_deferred + defer_size; + info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -302,23 +315,14 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) return ret; } -static inline bool need_expand(int nr_max) -{ - return round_up(nr_max, BITS_PER_LONG) > - round_up(shrinker_nr_max, BITS_PER_LONG); -} - static int expand_shrinker_info(int new_id) { int ret = 0; - int new_nr_max = new_id + 1; + int new_nr_max = round_up(new_id + 1, BITS_PER_LONG); int map_size, defer_size = 0; int old_map_size, old_defer_size = 0; struct mem_cgroup *memcg; - if (!need_expand(new_nr_max)) - goto out; - if (!root_mem_cgroup) goto out; @@ -332,7 +336,8 @@ static int expand_shrinker_info(int new_id) memcg = mem_cgroup_iter(NULL, NULL, NULL); do { ret = expand_one_shrinker_info(memcg, map_size, defer_size, - old_map_size, old_defer_size); + old_map_size, old_defer_size, + new_nr_max); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -352,9 +357,11 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) rcu_read_lock(); info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); - /* Pairs with smp mb in shrink_slab() */ - smp_mb__before_atomic(); - set_bit(shrinker_id, info->map); + if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { + /* Pairs with smp mb in shrink_slab() */ + smp_mb__before_atomic(); + set_bit(shrinker_id, info->map); + } rcu_read_unlock(); } } @@ -432,7 +439,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); - for (i = 0; i < shrinker_nr_max; i++) { + for (i = 0; i < child_info->map_nr_max; i++) { nr = atomic_long_read(&child_info->nr_deferred[i]); atomic_long_add(nr, &parent_info->nr_deferred[i]); } @@ -899,7 +906,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, From patchwork Tue Mar 7 06:55:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 65297 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2289947wrd; Mon, 6 Mar 2023 23:20:26 -0800 (PST) X-Google-Smtp-Source: AK7set/graIptWJghpatO0YSobPjCD7rM40DjAnS9hSXJ71OGmiWWtjjRC0PkwCyqXgO65N5sdVQ X-Received: by 2002:a17:902:d38c:b0:19c:d6d0:7887 with SMTP id e12-20020a170902d38c00b0019cd6d07887mr12751431pld.30.1678173626254; Mon, 06 Mar 2023 23:20:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678173626; cv=none; d=google.com; s=arc-20160816; b=ABUhSxUzVj7bJnlUrxVoyyzKcuIotpp9ylbPnkhf2LPG2SbMKZzG8UrtEbmk9+VoEr CifiUpzjpdW+jdM1fYFeCw7PVuL5aRlYv7U1Xivat9wh5zPC+ZOdTBJjBlIrKDJKRkD+ coWX3JJ2cWnLDqgZUT17NKDuDZnWADP4HxDSJH7rpb3EaKejEN+nkJHybvtO3JG9W4z0 NhSjz5j80s9VKCNkQDhxbr/oMIM7ATqx+5Wcq1y1S6Fa9jDk2dCKER6US52/nQJIxoYI N4K6/H7QDkq8SI6xDxZjxQDZeUO5S/01jjkQ5PJ+uoMOWQ1kPx2pj+z+M9wwdFIPsiqu sM0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ASSD7BghNw/J/hUaiXgWQCknh8Fl8R4TjAxXoZe8rSo=; b=GlPgL9S6RpAUr9receuSYsgGWY0mdrkck/aAO5OizZj+SiTSwFxZ6p4yfVVgTAzAIF qLFYKnE11eLFdwxMK10nxG+WfsBEw8vK3fmkQXzvS14IqdwvR2JwfBaNYaMadHF0H6+e TBC5X8am7l8ESPipIDst/FOhD0G7YezxFxbMqQk/Vqr4CK+pQRiJv0CbkCPfXmzpfeL2 d7to/+UcVtxd4rAvE2MiRl8loaoe+vVi9Bd9ikYj6LyclHYZ/yvyNC4u0IcQMsRiKbcd f+h41L6yMPYz4l/2gGQpN+ztoFirrJGoBf23Qx83VIXMqEKPXybV7HGYfmDJoQYp8mq8 hd7Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=gwtF2j6j; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ks5-20020a170903084500b0019a78d4c761si11152234plb.93.2023.03.06.23.20.13; Mon, 06 Mar 2023 23:20:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=gwtF2j6j; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231234AbjCGG6P (ORCPT + 99 others); Tue, 7 Mar 2023 01:58:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59144 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229627AbjCGG5q (ORCPT ); Tue, 7 Mar 2023 01:57:46 -0500 Received: from mail-pg1-x529.google.com (mail-pg1-x529.google.com [IPv6:2607:f8b0:4864:20::529]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 04BED86176 for ; Mon, 6 Mar 2023 22:56:44 -0800 (PST) Received: by mail-pg1-x529.google.com with SMTP id 130so7025201pgg.3 for ; Mon, 06 Mar 2023 22:56:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678172196; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ASSD7BghNw/J/hUaiXgWQCknh8Fl8R4TjAxXoZe8rSo=; b=gwtF2j6jEh4RIfI6JeZzkRF/pMyUNSc09ILbLx3nVVaFUO9uEdhj33JRDIszQXQGC7 tFr1nfLXPXzxu61Qs4pZfvZ+lgsXotnOK4kxSA6ErTDJ3W9Yv9udOYJFj30nHSvBFyCK 6zXZzdciINciHjYiRTh9V7nvLCLygsVJ/+eAkyvaE7v2gDCPhNda8nR9vHnymWdta9XX nom7SbNGo2amnoXsU0yzWSGLVguPni1/Zb7foDqZHxvqZtJ3jEqF2qkhQsumFjGyA+Oj mW773gxFLzs4+Pd46Q3RYSLslTYbgRVmbhEIspK60CCkH26JKY6i5r6Y+hVwWdew41+A rdQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678172196; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ASSD7BghNw/J/hUaiXgWQCknh8Fl8R4TjAxXoZe8rSo=; b=p4YMOrYtQpyNoQJQHU3Qyk2/DIs4eVyLwuZ+c9ucVNHkwvecUHjyfBsibSldVxFK7V T4129v1Sx0fHV7NfD6wKlkF2YNM9WTllwyYJa9QnaNeOW/u9CWPiE4R3Dd+NFmJbF7TT yPxXDV7GRXRehlQjC0Ooc1LuGfDsw9/CgFpKgtZNqvr/KDu4STTIQtrnmAltwDVyB1UP LO/s488Bur/bBZ8eaYwhx0vggiF9GtE+36aeCC2PyXIeqF8b18HuIUoFvU16hXvjDtMt EH+ebkcTdDl/88jO3hm5D4eH3BVSbQ5l5Hti62QfUVv4BtYdVvnW0jnx4jUlxq1MRcOW DIHg== X-Gm-Message-State: AO0yUKVhbrf2t+Ob9/laA7EYOCjXC6p01p+qsGXuM+EJhiRtgO8o4sK/ WU0wN1CBB2lDYD3kxoHa8jxp5w== X-Received: by 2002:a05:6a00:2d89:b0:5e2:3086:f977 with SMTP id fb9-20020a056a002d8900b005e23086f977mr13567615pfb.2.1678172196363; Mon, 06 Mar 2023 22:56:36 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id e5-20020a62ee05000000b00608dae58695sm7230854pfi.209.2023.03.06.22.56.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 22:56:35 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 2/8] mm: vmscan: make global slab shrink lockless Date: Tue, 7 Mar 2023 14:55:59 +0800 Message-Id: <20230307065605.58209-3-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230307065605.58209-1-zhengqi.arch@bytedance.com> References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759692587936802656?= X-GMAIL-MSGID: =?utf-8?q?1759692587936802656?= The shrinker_rwsem is a global read-write lock in shrinkers subsystem, which protects most operations such as slab shrink, registration and unregistration of shrinkers, etc. This can easily cause problems in the following cases. 1) When the memory pressure is high and there are many filesystems mounted or unmounted at the same time, slab shrink will be affected (down_read_trylock() failed). Such as the real workload mentioned by Kirill Tkhai: ``` One of the real workloads from my experience is start of an overcommitted node containing many starting containers after node crash (or many resuming containers after reboot for kernel update). In these cases memory pressure is huge, and the node goes round in long reclaim. ``` 2) If a shrinker is blocked (such as the case mentioned in [1]) and a writer comes in (such as mount a fs), then this writer will be blocked and cause all subsequent shrinker-related operations to be blocked. Even if there is no competitor when shrinking slab, there may still be a problem. If we have a long shrinker list and we do not reclaim enough memory with each shrinker, then the down_read_trylock() may be called with high frequency. Because of the poor multicore scalability of atomic operations, this can lead to a significant drop in IPC (instructions per cycle). So many times in history ([2],[3],[4],[5]), some people wanted to replace shrinker_rwsem trylock with SRCU in the slab shrink, but all these patches were abandoned because SRCU was not unconditionally enabled. But now, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), the SRCU is unconditionally enabled. So it's time to use SRCU to protect readers who previously held shrinker_rwsem. This commit uses SRCU to make global slab shrink lockless, the memcg slab shrink is handled in the subsequent patch. [1]. https://lore.kernel.org/lkml/20191129214541.3110-1-ptikhomirov@virtuozzo.com/ [2]. https://lore.kernel.org/all/1437080113.3596.2.camel@stgolabs.net/ [3]. https://lore.kernel.org/lkml/1510609063-3327-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp/ [4]. https://lore.kernel.org/lkml/153365347929.19074.12509495712735843805.stgit@localhost.localdomain/ [5]. https://lore.kernel.org/lkml/20210927074823.5825-1-sultan@kerneltoast.com/ Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- mm/vmscan.c | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 2dcc01682026..8515ac40bcaf 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -202,6 +202,7 @@ static void set_task_reclaim_state(struct task_struct *task, LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); +DEFINE_SRCU(shrinker_srcu); #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -706,7 +707,7 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); - list_add_tail(&shrinker->list, &shrinker_list); + list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); up_write(&shrinker_rwsem); @@ -760,13 +761,15 @@ void unregister_shrinker(struct shrinker *shrinker) return; down_write(&shrinker_rwsem); - list_del(&shrinker->list); + list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_remove(shrinker); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); + debugfs_remove_recursive(debugfs_entry); kfree(shrinker->nr_deferred); @@ -786,6 +789,7 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -996,6 +1000,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, { unsigned long ret, freed = 0; struct shrinker *shrinker; + int srcu_idx; /* * The root memcg might be allocated even though memcg is disabled @@ -1007,10 +1012,10 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) return shrink_slab_memcg(gfp_mask, nid, memcg, priority); - if (!down_read_trylock(&shrinker_rwsem)) - goto out; + srcu_idx = srcu_read_lock(&shrinker_srcu); - list_for_each_entry(shrinker, &shrinker_list, list) { + list_for_each_entry_srcu(shrinker, &shrinker_list, list, + srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -1021,19 +1026,9 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; freed += ret; - /* - * Bail out if someone want to register a new shrinker to - * prevent the registration from being stalled for long periods - * by parallel ongoing shrinking. - */ - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } } - up_read(&shrinker_rwsem); -out: + srcu_read_unlock(&shrinker_srcu, srcu_idx); cond_resched(); return freed; } From patchwork Tue Mar 7 06:56:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 65298 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2290897wrd; Mon, 6 Mar 2023 23:23:51 -0800 (PST) X-Google-Smtp-Source: AK7set+hlNEvcp093N981UuDLSLw76q4R8GMtgCje2tCzEl//6Cob66l9IYXCk58De1AqfnGpTDT X-Received: by 2002:a17:902:e84f:b0:19a:8ce1:2c55 with SMTP id t15-20020a170902e84f00b0019a8ce12c55mr16629918plg.8.1678173831362; Mon, 06 Mar 2023 23:23:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678173831; cv=none; d=google.com; s=arc-20160816; b=MqoGrsFVMvbBdHT9gdA/RCrKstwe48+MnSo2MUWTiamnvXvexqdbrg2/vvlpb4WOgz 5YNgKFIVJKcX1R+0N74XlDEC/q5LWCObskp39klnWjFrRiaS8ITBTnMziY+Qw4E9l87g 9cJGaDYJH30BaAt1O9Je4aoQTTVztuelemBxmkbrmspUhHUWr6sz7ZJMccTUu73e2cTq g+WEPNGrXyAK3cdnfWvS0NyJyGE13Ct5ofWaZOUeWzXhxKvoEjv+5f3U+HDQwNjwiTWg arpp5RPNFiDU4QfYvHc5ZLFqJ8dDDXlcq0Vy+WWlN16nVKvLrF8lJb3RcqEE16RfJfm/ wOGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9rhvfVxo/VIqlZiNKGwl0zk05JZT0WS+du3pcLhwX24=; b=lAl//XYGtHPuy0YCFhTnxFVrFYTB+fC7hXDzDJynQj0pB61QAwbUSz/W7dSMZu9oK2 bXukGU3VqbJpVKywyCEfmp5k8BFWM8aWmGLlGF6OQjMwXhom0UXT1usRbkvbDiFn3zU8 9IGgPCHg8UMch5YDXwrmiAiOJs7sh0RCH+AQkOWdrWQgPmV0u6asyBRYs5me72nSagWr WXWDHWjTvZHZINmzMuyEC+zcjWbQl+3j+/NZuYan8UC6BozsAg/0P41vvRxxWNjdIMyn 3gC5byWf00pO4+6Kl2n2hAjgOGKdFEGFoy/VCoY1cGuluBN/ZC2MFrCWCIB8RjYq4OGE m2ag== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=iOzGoBBF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r4-20020a632b04000000b0050726756ee5si8353880pgr.98.2023.03.06.23.23.38; Mon, 06 Mar 2023 23:23:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=iOzGoBBF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231145AbjCGG6X (ORCPT + 99 others); Tue, 7 Mar 2023 01:58:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58942 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229871AbjCGG5r (ORCPT ); Tue, 7 Mar 2023 01:57:47 -0500 Received: from mail-pl1-x62a.google.com (mail-pl1-x62a.google.com [IPv6:2607:f8b0:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5575D88D9D for ; Mon, 6 Mar 2023 22:56:47 -0800 (PST) Received: by mail-pl1-x62a.google.com with SMTP id a9so13057180plh.11 for ; Mon, 06 Mar 2023 22:56:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678172203; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9rhvfVxo/VIqlZiNKGwl0zk05JZT0WS+du3pcLhwX24=; b=iOzGoBBFwi7lf9XHmfcZfShTltMjKW7fB7JTbgosxQmnrGMtoxbAgVo1GO+pB9geGD ux0EoYY0pV+ERjF8F18TEfHq9gvIyT4dhtCOozSdJ5U3Uc7KDQZjV/RZ0bw9vjZZGL8h 6AvKxfXxIsouFF4LnTb66UYjTB0h3F16LVfPmemtGR2hTzHJvTn0cjtGnyb6+IXsRvFv yWUb+9Dses9zqWd2+QitUQbqI7oCCIQFPmCp3hOV3Kywr901CALSY+b2ZE0aQz5c+ubl QFR4X7Ka+3ROzvlIIttem0P+mhUUNQnXkpmli1fLe9fgOOGOH4R00OxZ5OvRzTgdpGxM QZow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678172203; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9rhvfVxo/VIqlZiNKGwl0zk05JZT0WS+du3pcLhwX24=; b=ViHArO+ruGALDjkiGiUi1C/fnSOKjAIxOgRK5E7HFkW6a7LOkGmHWnnSnFJ9wXhFgQ 9U/ku2w+UbBsuYJNeHjYxeQUiVOyrF5Mlg3JDeuVoL5MQgSCr4gyGHqm14Nj3PTUGgAM 2e83EJBAApLn5TR4zvLGHZNCpYhWH6ApWEF20qlTI8srnjXBS3aJgvZ24Po5NrnnngsF YyUQfm/yPaUZv7wbp09RcImUlEwHturySmgCS2zCvzaMqHbxqE5jxOe/m/eMtUbLqs2a eadnkCZwQrbG5HKnY2/zl0jLmJgG+BgfSftWKuvSvmrzJKPwt4Ci8JRY/MfUlAMJd0zL 8SjQ== X-Gm-Message-State: AO0yUKVT7RDsQI9oMyvPwt4cRTgKSIm4uct6sQ4R1x24DvE2VG5fnQNq 6B8dxhNlUuseZ/ynHx/l7Ltx7A== X-Received: by 2002:a05:6a20:1585:b0:cb:c276:58bb with SMTP id h5-20020a056a20158500b000cbc27658bbmr14626109pzj.4.1678172203481; Mon, 06 Mar 2023 22:56:43 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id e5-20020a62ee05000000b00608dae58695sm7230854pfi.209.2023.03.06.22.56.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 22:56:43 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 3/8] mm: vmscan: make memcg slab shrink lockless Date: Tue, 7 Mar 2023 14:56:00 +0800 Message-Id: <20230307065605.58209-4-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230307065605.58209-1-zhengqi.arch@bytedance.com> References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759692803450155525?= X-GMAIL-MSGID: =?utf-8?q?1759692803450155525?= Like global slab shrink, this commit also uses SRCU to make memcg slab shrink lockless. We can reproduce the down_read_trylock() hotspot through the following script: ``` DIR="/root/shrinker/memcg/mnt" do_create() { mkdir -p /sys/fs/cgroup/memory/test mkdir -p /sys/fs/cgroup/perf_event/test echo 4G > /sys/fs/cgroup/memory/test/memory.limit_in_bytes for i in `seq 0 $1`; do mkdir -p /sys/fs/cgroup/memory/test/$i; echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; echo $$ > /sys/fs/cgroup/perf_event/test/cgroup.procs; mkdir -p $DIR/$i; done } do_mount() { for i in `seq $1 $2`; do mount -t tmpfs $i $DIR/$i; done } do_touch() { for i in `seq $1 $2`; do echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; echo $$ > /sys/fs/cgroup/perf_event/test/cgroup.procs; dd if=/dev/zero of=$DIR/$i/file$i bs=1M count=1 & done } case "$1" in touch) do_touch $2 $3 ;; test) do_create 4000 do_mount 0 4000 do_touch 0 3000 ;; *) exit 1 ;; esac ``` Save the above script, then run test and touch commands. Then we can use the following perf command to view hotspots: perf top -U -F 999 1) Before applying this patchset: 32.31% [kernel] [k] down_read_trylock 19.40% [kernel] [k] pv_native_safe_halt 16.24% [kernel] [k] up_read 15.70% [kernel] [k] shrink_slab 4.69% [kernel] [k] _find_next_bit 2.62% [kernel] [k] shrink_node 1.78% [kernel] [k] shrink_lruvec 0.76% [kernel] [k] do_shrink_slab 2) After applying this patchset: 27.83% [kernel] [k] _find_next_bit 16.97% [kernel] [k] shrink_slab 15.82% [kernel] [k] pv_native_safe_halt 9.58% [kernel] [k] shrink_node 8.31% [kernel] [k] shrink_lruvec 5.64% [kernel] [k] do_shrink_slab 3.88% [kernel] [k] mem_cgroup_iter At the same time, we use the following perf command to capture IPC information: perf stat -e cycles,instructions -G test -a --repeat 5 -- sleep 10 1) Before applying this patchset: Performance counter stats for 'system wide' (5 runs): 454187219766 cycles test ( +- 1.84% ) 78896433101 instructions test # 0.17 insn per cycle ( +- 0.44% ) 10.0020430 +- 0.0000366 seconds time elapsed ( +- 0.00% ) 2) After applying this patchset: Performance counter stats for 'system wide' (5 runs): 841954709443 cycles test ( +- 15.80% ) (98.69%) 527258677936 instructions test # 0.63 insn per cycle ( +- 15.11% ) (98.68%) 10.01064 +- 0.00831 seconds time elapsed ( +- 0.08% ) We can see that IPC drops very seriously when calling down_read_trylock() at high frequency. After using SRCU, the IPC is at a normal level. Signed-off-by: Qi Zheng Acked-by: Kirill Tkhai Acked-by: Vlastimil Babka --- mm/vmscan.c | 46 +++++++++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 19 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 8515ac40bcaf..1de9bc3e5aa2 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -221,8 +222,21 @@ static inline int shrinker_defer_size(int nr_items) static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, int nid) { - return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - lockdep_is_held(&shrinker_rwsem)); + return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu, + lockdep_is_held(&shrinker_rwsem)); +} + +static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, + int nid) +{ + return srcu_dereference(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu); +} + +static void free_shrinker_info_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct shrinker_info, rcu)); } static inline bool need_expand(int new_nr_max, int old_nr_max) @@ -269,7 +283,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, defer_size - old_defer_size); rcu_assign_pointer(pn->shrinker_info, new); - kvfree_rcu(old, rcu); + call_srcu(&shrinker_srcu, &old->rcu, free_shrinker_info_rcu); } return 0; @@ -355,15 +369,16 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { struct shrinker_info *info; + int srcu_idx; - rcu_read_lock(); - info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); set_bit(shrinker_id, info->map); } - rcu_read_unlock(); + srcu_read_unlock(&shrinker_srcu, srcu_idx); } } @@ -377,7 +392,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) return -ENOSYS; down_write(&shrinker_rwsem); - /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -411,7 +425,7 @@ static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); } @@ -420,7 +434,7 @@ static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } @@ -898,15 +912,14 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; + int srcu_idx; int i; if (!mem_cgroup_online(memcg)) return 0; - if (!down_read_trylock(&shrinker_rwsem)) - return 0; - - info = shrinker_info_protected(memcg, nid); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; @@ -956,14 +969,9 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, i); } freed += ret; - - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } } unlock: - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); return freed; } #else /* CONFIG_MEMCG */ From patchwork Tue Mar 7 06:56:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 65306 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2292645wrd; Mon, 6 Mar 2023 23:30:00 -0800 (PST) X-Google-Smtp-Source: AK7set+mvthzgVIKFNMTkCz8xwNwT0Hxpntu1T4trOakeaGxLyruMFOx1iCu21/FuRPLmqoeiYdP X-Received: by 2002:a17:906:348b:b0:8ef:3e41:e719 with SMTP id g11-20020a170906348b00b008ef3e41e719mr13796938ejb.63.1678174200453; Mon, 06 Mar 2023 23:30:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678174200; cv=none; d=google.com; s=arc-20160816; b=Qcb0rPRm0hIyIkzL8YBEy9PaQQiJI75YH0Mp8evwRhFVCpIQIcmGBiehya6NkfCm+W /cqPKEo4WtJM8WzO+wVh7zQq+SwWNkpzaywmBAILNdxsEQQpK0nrDFwUvCcYagB136sH oQMjn10DqrkzTHJBS9aDSWZXG/RTWBgEIa/R2MO3Wf6jw7OQzNiE33K2R0eyzILhc9gy hF3wl3WV09+uICmj+gg1T0/uNNduV04iWAwNUEPsYvDSNBpvi2mpgyR/d36dpNa8v2sI A5lf5UniOcB9tZ6YnAxT0o/kYyBOjHrGXBKFw4RPW85p9PdhnxzmdijwgJJpN8wRkQeP bu9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=hlfbC5hLUOyZ5eiKWKe3FNliR+/5Zth66owFIX3duxQ=; b=rsk0ZIezb014ZMXjaibXp7g9WVGBSjs9u1phtGWPAwQpas9J3VyizXUQbnNq1JmcMc fj8etC/i+j6hS20kqrqwsaE1daAChQwakfQLgVwcd8Qp1fMbejJns9+yR+0ojcGaF6H/ W/uG/zMW2hIOACXSEvOMoXU6/pn1i7JGavWjzD17Q2ywWMOTE8sGNIi3RWj4C7hsSsOv 4cPCpA177CtLSzz89qPTJgWy0XvMdAVkQNqFgnZzEsGuFrZPHsD4MKaBT95ntVYTFvBZ GhnRSGrFajWiDv9svKCMQcckNO8a1MdnTqxw9OCG+wXopx8gtQI4FTo+pww6LT2CgtkB Ow9w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=j8xEkJaY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qa35-20020a17090786a300b008d28622c8d9si1836340ejc.728.2023.03.06.23.29.36; Mon, 06 Mar 2023 23:30:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=j8xEkJaY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230522AbjCGG55 (ORCPT + 99 others); Tue, 7 Mar 2023 01:57:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59190 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230520AbjCGG52 (ORCPT ); Tue, 7 Mar 2023 01:57:28 -0500 Received: from mail-pf1-x42e.google.com (mail-pf1-x42e.google.com [IPv6:2607:f8b0:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC6E288D8D for ; Mon, 6 Mar 2023 22:56:51 -0800 (PST) Received: by mail-pf1-x42e.google.com with SMTP id c4so7447508pfl.0 for ; Mon, 06 Mar 2023 22:56:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678172211; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hlfbC5hLUOyZ5eiKWKe3FNliR+/5Zth66owFIX3duxQ=; b=j8xEkJaYof6JnjV2P0RsveJBbsnpcn9RY4V+ifKMT79Be93zvpgebNY5nSfWdFD0NU 1JW1AYTHaO+N+RFx2VLXc9tAFRn/DsFZYM5L65TiSMr1hMRgmru3xvORasp/75YD6J/y rGn2vVrH8HJoacjH3PbF9GnOpdyo3gX0mQEUZgRXnTBovRgY8MVxTw8BI9uz52Q/LQHG E2dcdZEf3BV3Hh0v3X7btzqm0okyXgWthyJJzVeextpZ9F1g0oAYjtHaaqZ/iEWplil2 10Sb5xodkY2XripisufbYdb8ro/zRPnyPWGLDA9/6bcUB7x768fm0jxge+41ve9sBsI8 3UMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678172211; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hlfbC5hLUOyZ5eiKWKe3FNliR+/5Zth66owFIX3duxQ=; b=fjP0dCIu/84us1lEdw/SflpGfLuJnx93KGDJG4qUK2B1FjlmT5LjXAphpK0Gajd+jH 7BedkZtDo0VxDWTk0gAlRqfpbnXOMllXFHjuBTHUwofJEBqhwQszlppp4IkKJ1O06Dmt YcocLFosrxX5d0WpdOTJrt2EUiMR8sdG1AqHVzyTxMWPWg/r9/bgekEg15z3duY99G6p G0kpie5vGBLea7HFIo7oCBT5MSvQHKS+cIX24G/scT1JoQ9P5AiaTX6tPn/8JNyDaJLU GjMYRrJngZ/Sfwu2+19bSHYZ6MfkWKRv7F+xv4eX262zIY84mPYfPGmCgkbra4Hs/2gE iCiw== X-Gm-Message-State: AO0yUKUQBODxZg1lfwOldqPIjN4TlkZ++jyj9/uRzC1mDK2vudsb+Ugz g1+/7LXw9iz2uJoimMwU9Um/yg== X-Received: by 2002:a05:6a00:4087:b0:5d1:f76:d1d7 with SMTP id bw7-20020a056a00408700b005d10f76d1d7mr14022848pfb.1.1678172211105; Mon, 06 Mar 2023 22:56:51 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id e5-20020a62ee05000000b00608dae58695sm7230854pfi.209.2023.03.06.22.56.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 22:56:50 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 4/8] mm: vmscan: add shrinker_srcu_generation Date: Tue, 7 Mar 2023 14:56:01 +0800 Message-Id: <20230307065605.58209-5-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230307065605.58209-1-zhengqi.arch@bytedance.com> References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759693190582466140?= X-GMAIL-MSGID: =?utf-8?q?1759693190582466140?= From: Kirill Tkhai After we make slab shrink lockless with SRCU, the longest sleep unregister_shrinker() will be a sleep waiting for all do_shrink_slab() calls. To aviod long unbreakable action in the unregister_shrinker(), add shrinker_srcu_generation to restore a check similar to the rwsem_is_contendent() check that we had before. And for memcg slab shrink, we unlock SRCU and continue iterations from the next shrinker id. Signed-off-by: Kirill Tkhai Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka --- mm/vmscan.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 1de9bc3e5aa2..9a5a3da5c8b5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -204,6 +204,7 @@ static void set_task_reclaim_state(struct task_struct *task, LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); DEFINE_SRCU(shrinker_srcu); +static atomic_t shrinker_srcu_generation = ATOMIC_INIT(0); #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -782,6 +783,7 @@ void unregister_shrinker(struct shrinker *shrinker) debugfs_entry = shrinker_debugfs_remove(shrinker); up_write(&shrinker_rwsem); + atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); debugfs_remove_recursive(debugfs_entry); @@ -803,6 +805,7 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); + atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -912,18 +915,20 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; - int srcu_idx; - int i; + int srcu_idx, generation; + int i = 0; if (!mem_cgroup_online(memcg)) return 0; +again: srcu_idx = srcu_read_lock(&shrinker_srcu); info = shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, info->map_nr_max) { + generation = atomic_read(&shrinker_srcu_generation); + for_each_set_bit_from(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -969,6 +974,11 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, i); } freed += ret; + if (atomic_read(&shrinker_srcu_generation) != generation) { + srcu_read_unlock(&shrinker_srcu, srcu_idx); + i++; + goto again; + } } unlock: srcu_read_unlock(&shrinker_srcu, srcu_idx); @@ -1008,7 +1018,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, { unsigned long ret, freed = 0; struct shrinker *shrinker; - int srcu_idx; + int srcu_idx, generation; /* * The root memcg might be allocated even though memcg is disabled @@ -1022,6 +1032,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, srcu_idx = srcu_read_lock(&shrinker_srcu); + generation = atomic_read(&shrinker_srcu_generation); list_for_each_entry_srcu(shrinker, &shrinker_list, list, srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc = { @@ -1034,6 +1045,11 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; freed += ret; + + if (atomic_read(&shrinker_srcu_generation) != generation) { + freed = freed ? : 1; + break; + } } srcu_read_unlock(&shrinker_srcu, srcu_idx); From patchwork Tue Mar 7 06:56:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 65295 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2286769wrd; Mon, 6 Mar 2023 23:10:26 -0800 (PST) X-Google-Smtp-Source: AK7set8sbIS+SaDBWXrH2mWmbW5vmqVC8cg11tSNot3pjql3/C8XleQTHGc1F5ZgArremCXwVIoK X-Received: by 2002:a17:902:ecc1:b0:19e:23c1:4c2e with SMTP id a1-20020a170902ecc100b0019e23c14c2emr16265000plh.59.1678173026028; Mon, 06 Mar 2023 23:10:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678173026; cv=none; d=google.com; s=arc-20160816; b=BVU1Zv3yzPiwwwKzwYBgYSu4H1X0Fh24/6V7kWTODNj8kQKwO+JjR2SAnxcj3yqhsV uhvP3KN1gtjnuLDPg77HpVrXUtwMIZrC1VigQRd8Oiem+zJX28dYJsFi2IigeQ6/HFd2 Ko+pyMW8DhOx7ueXnKBJePb4yBI8ojlYiOQKb2v6qjgiLdIwe+ruWxzdctiuvqVhJaGW uvUOhX439AIqDsvEx3DRFqzc5G1MQrIPCocOPOWOMqU/c1o+IhIpGWEt3Cl0jDVJ36xs sgKRpY4MYCnUsF1v9jGFjLxCjWgLRZb2fq1+2hlDrpUzeeRJPLD99iJvKSZNFO7QLNqN pRiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=si4i3xYcl0oslNsMEwQ4bD/IMZqZAsePzzOsdLXazR8hdGujuluzRJuQbx5iiveKpH Jo6bWXhuPFztmop9EjZXannKy6dp6BMrgPRA/piPrfpgzOoxNc3OCbow9HYbLGbwOA9y G3Zfj4SAcsLUqWfXFWoMsLsyciOHB3c1VdMhlTu+YnwKcJhX36eWpqjQBVNuA16Fgdlo bPdV1ZG4USd4o+wSjl2cr0gywZaHofuuFdE4cda56lH/irmMpi9IpvS4J0TJL0NpcBbJ zPhTJjHZmyqA+z0+6QrIAbUIQls4Ydqv49u2PFd2wfR7pJuHFNE7UYHcFSRAbse6NMXj V98g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=PS4W+FgT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k24-20020a170902761800b0019a7375dfcasi10734323pll.247.2023.03.06.23.10.10; Mon, 06 Mar 2023 23:10:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=PS4W+FgT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230478AbjCGG6l (ORCPT + 99 others); Tue, 7 Mar 2023 01:58:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231174AbjCGG6F (ORCPT ); Tue, 7 Mar 2023 01:58:05 -0500 Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CF50159405 for ; Mon, 6 Mar 2023 22:57:03 -0800 (PST) Received: by mail-pl1-x62d.google.com with SMTP id p20so13041455plw.13 for ; Mon, 06 Mar 2023 22:57:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678172219; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=PS4W+FgTgUCjDbGtg+QwCoSMgK+PZFy9BY2F1ZThxrjZVSL14yAr0wpqGWIeHjNUI0 MjcnS5KpgGlvLOSQCUAdkAjQZHluL/aF5sSdv42dRLNv4KIIzh+aWWZDjamM2HQABfg/ ke15HkP04BOxxAYCBWXNW3bEgB1hjXjR2Uv8/YCxNcS8GL93fqqgDCLTaxFq1I/kUgHl LQ/t9QNRUeTUcPEAjMgUEh4v5iqcEyjJI/gJPK3velHfAXIwIh9N1aKawGtD5vTgD5rp 0q/DlCg3cZIA9QSknjY+zItjUDbIe0idSmifezJtWhrwh7Hi0S+Ju/qHW2bubpQjjP41 H5IQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678172219; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=P301nGzFxa2873ciT6LZL7aoZjIXfzYghyRui4ZQr7XfFgopxoO/EwyPDM1ME4jbzZ gP8Hrs5LCwpHb7+/BOxt+tyAvCMil2kEQmjrtsVF5Y8THUm0sSdm0Lk0yTMriejL2hff Bc8PH16fnaIvNPDfPEAcJGKIPjKiTCLndzjjUVPVH8n1SxewDUgeuXelME0NA2mh2Ag9 GiSeP6LaUamaKC9q1h1mlIO8m1/2DE2ZF/XFdkcoQ4QaQViDEDaWAF+iHzpWyOwbXQSp /BSne+R6Dli1nXnYBy0iPUbxF9E0ySOpZdHXjwfYMX28oRjLwo6lwJqkKkdCOQB6a+Zb yA9w== X-Gm-Message-State: AO0yUKWJ/zTvj7Js1E6UVDpGWeu37OJWfif6OPtEcJC9xEqNLaMwMD2z VkqJO6HZogQSF/otzXNDs29Hng== X-Received: by 2002:a05:6a20:6914:b0:cc:4118:65c4 with SMTP id q20-20020a056a20691400b000cc411865c4mr14603250pzj.5.1678172219371; Mon, 06 Mar 2023 22:56:59 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id e5-20020a62ee05000000b00608dae58695sm7230854pfi.209.2023.03.06.22.56.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 22:56:58 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 5/8] mm: shrinkers: make count and scan in shrinker debugfs lockless Date: Tue, 7 Mar 2023 14:56:02 +0800 Message-Id: <20230307065605.58209-6-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230307065605.58209-1-zhengqi.arch@bytedance.com> References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759691959057423529?= X-GMAIL-MSGID: =?utf-8?q?1759691959057423529?= Like global and memcg slab shrink, also use SRCU to make count and scan operations in memory shrinker debugfs lockless. Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- mm/shrinker_debug.c | 24 +++++++----------------- 1 file changed, 7 insertions(+), 17 deletions(-) diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 39c3491e28a3..6aa7a7ec69da 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -9,6 +9,7 @@ /* defined in vmscan.c */ extern struct rw_semaphore shrinker_rwsem; extern struct list_head shrinker_list; +extern struct srcu_struct shrinker_srcu; static DEFINE_IDA(shrinker_debugfs_ida); static struct dentry *shrinker_debugfs_root; @@ -49,18 +50,13 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) struct mem_cgroup *memcg; unsigned long total; bool memcg_aware; - int ret, nid; + int ret = 0, nid, srcu_idx; count_per_node = kcalloc(nr_node_ids, sizeof(unsigned long), GFP_KERNEL); if (!count_per_node) return -ENOMEM; - ret = down_read_killable(&shrinker_rwsem); - if (ret) { - kfree(count_per_node); - return ret; - } - rcu_read_lock(); + srcu_idx = srcu_read_lock(&shrinker_srcu); memcg_aware = shrinker->flags & SHRINKER_MEMCG_AWARE; @@ -91,8 +87,7 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) } } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); - rcu_read_unlock(); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); kfree(count_per_node); return ret; @@ -115,9 +110,8 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, .gfp_mask = GFP_KERNEL, }; struct mem_cgroup *memcg = NULL; - int nid; + int nid, srcu_idx; char kbuf[72]; - ssize_t ret; read_len = size < (sizeof(kbuf) - 1) ? size : (sizeof(kbuf) - 1); if (copy_from_user(kbuf, buf, read_len)) @@ -146,11 +140,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, return -EINVAL; } - ret = down_read_killable(&shrinker_rwsem); - if (ret) { - mem_cgroup_put(memcg); - return ret; - } + srcu_idx = srcu_read_lock(&shrinker_srcu); sc.nid = nid; sc.memcg = memcg; @@ -159,7 +149,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, shrinker->scan_objects(shrinker, &sc); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); mem_cgroup_put(memcg); return size; From patchwork Tue Mar 7 06:56:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 65299 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2291093wrd; Mon, 6 Mar 2023 23:24:26 -0800 (PST) X-Google-Smtp-Source: AK7set9CQueKDko4xgG2MQwsJj/4Z9kkMaFkOS8grRKDqsjcOUvesuwa11UX6XpSZL3h1tnC97t9 X-Received: by 2002:a17:902:8d8a:b0:19e:e172:2a40 with SMTP id v10-20020a1709028d8a00b0019ee1722a40mr1180523plo.65.1678173866569; Mon, 06 Mar 2023 23:24:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678173866; cv=none; d=google.com; s=arc-20160816; b=IU+gSvGYiOOu+JPWaET4jh+00tfkmRozH/Rj1mEOcoY6OcGKNJq3vEw+E9tKBnh9vc jAuY0jC2NhhRktR+h25/XI9oxeRIa55bbpF2cD0EVb0rAYEGS5fJ86No96KWPmB8mOc3 2CTWQV3KLhunqSOzVbUOpk3cgAvkUOVZgSTAA3mete/g2Eu40QziT1auhk546Wt8A+tV qgGwmxE5lPOOxFGRfT/MyBSJrxFkDd7sXT/FVTKyYiIK4KqZaJzwXwURAPd2rdBhTqq4 w/BDOaQ8mE0jS7hNyM1IEZRBUts+vIkomtAmOOkBeW+14a2a25zmRuWAjEnQnNmC96EY p1AA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=6RpTkxvvrfdHRVC7O4Q9cB0zRrVMaNNv+4vslhRNXIw=; b=NBrLKv0MMH8/1DuDi5q4msUx9HgB6ekJXHEsFd4io1ZFfDQGKCfZq5VRjhdumE8D/J 0Zl7QK7M5iSrZe3cEY3vB/R2dMyX28taMJwvBBtLv/yTSbIx763wFFTivVUhwcmmQ9UR E/SXx2QTLucCVdz5Hi7A3RlUAFXqlBKOOzzUaZKEFE1r6vq7heCAxmA2V9y/M/M92/aK vZwdsuzZq5sOoxT3v233SrzGeA0XxxEd2Edx1gFx0V1jOnPu/5BwEbfBc8IWhsdgZeUz GVQVIj0d5l0ZY2twrBBQkGMS2+8xLFJ9075l1YJbXHg3WX+ttQ1sOwjjgEZQ7Hcwu2eX KRvg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=W4VHauOi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x4-20020a170902ec8400b0019cb63d38e3si12316344plg.589.2023.03.06.23.24.13; Mon, 06 Mar 2023 23:24:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=W4VHauOi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231228AbjCGG6n (ORCPT + 99 others); Tue, 7 Mar 2023 01:58:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59102 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231205AbjCGG6I (ORCPT ); Tue, 7 Mar 2023 01:58:08 -0500 Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BDCEC88EF7 for ; Mon, 6 Mar 2023 22:57:08 -0800 (PST) Received: by mail-pf1-x42d.google.com with SMTP id b20so7405966pfo.6 for ; Mon, 06 Mar 2023 22:57:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678172226; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6RpTkxvvrfdHRVC7O4Q9cB0zRrVMaNNv+4vslhRNXIw=; b=W4VHauOiBpt34sXN0NGeRnCvFFOrK6bGxqZvOkEMVUkrE3vG6k1a/43ATXBWF9oL4D HYBUo7vHZe/W31enhNJ7CVLqdmu5Y02HPFFflxIozl0b/QDd6RN4k6EwvF+awGPNHWe4 Rx/0HE2KR+Qq7m9hXa35Ea3oyuplOTrjAbd44mvSt3lLuSOT9OuJKMHLt7IsuNAw2GaM bzcYuW9uMvkR+Z6LM4ysopbsVwXnebzGgDNnTsrwb/NRdLlO7ExqCFm1QRYlGeJTOVKA QV50Vul9k2ren/2UFSJ3/jtJDhKNqxK+2HgjtMnrdUW17iEio32nEMoMmRRjN2BUDePl lK1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678172226; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6RpTkxvvrfdHRVC7O4Q9cB0zRrVMaNNv+4vslhRNXIw=; b=GnLWipTVEHy3FRJf67IT4TYp9UdVo0ZWfhZLTXmC123DAC/d8xXZ3EROT9rl9nD1HP c9N50gSC1RW8Nug15UNtiJdTSNDpb/QX7PAp54zX1fZJQux1HWr6EXyyy03nuTSzjgbL WNb31kztTSxb7xb2Hdur92yGb/oMDI5asX0OpPbEnEMJzooUp5I9owT9xmOKSaMz0Lmu boGEGjw2Yx9yjjrg8N14ZwQcFWX9fynrhssuTSCl65Y314JvSOeNTwbwT6xODz52gy0G kD5HVzfUGFOJ/vdGiAUxHX1Q0bGlhjrMNj7rys/p8nMg0LDDgAQIBwoHpS17duNYUevM Unyw== X-Gm-Message-State: AO0yUKXdvkGiEhaYXwnsvD+PFNomGoUxCUIqeO0HwRelejZGypK4TN+E MB4B/WahfaHvwmchtK8DVpBTIw== X-Received: by 2002:aa7:8892:0:b0:5d1:bbd2:7604 with SMTP id z18-20020aa78892000000b005d1bbd27604mr12474224pfe.2.1678172226558; Mon, 06 Mar 2023 22:57:06 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id e5-20020a62ee05000000b00608dae58695sm7230854pfi.209.2023.03.06.22.56.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 22:57:06 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 6/8] mm: vmscan: hold write lock to reparent shrinker nr_deferred Date: Tue, 7 Mar 2023 14:56:03 +0800 Message-Id: <20230307065605.58209-7-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230307065605.58209-1-zhengqi.arch@bytedance.com> References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759692840657547486?= X-GMAIL-MSGID: =?utf-8?q?1759692840657547486?= For now, reparent_shrinker_deferred() is the only holder of read lock of shrinker_rwsem. And it already holds the global cgroup_mutex, so it will not be called in parallel. Therefore, in order to convert shrinker_rwsem to shrinker_mutex later, here we change to hold the write lock of shrinker_rwsem to reparent. Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 9a5a3da5c8b5..7aaf6f94ac1b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -451,7 +451,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_read(&shrinker_rwsem); + down_write(&shrinker_rwsem); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -460,7 +460,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_read(&shrinker_rwsem); + up_write(&shrinker_rwsem); } static bool cgroup_reclaim(struct scan_control *sc) From patchwork Tue Mar 7 06:56:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 65301 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2292122wrd; Mon, 6 Mar 2023 23:28:15 -0800 (PST) X-Google-Smtp-Source: AK7set+ZbXED234yWfz1kUEWUNqPiZbb1Lbm36r0eCYpW+RopES0zFl5WAb7WSXweb/Wcff10SUd X-Received: by 2002:a17:903:40c4:b0:19d:74c:78e5 with SMTP id t4-20020a17090340c400b0019d074c78e5mr13259587pld.50.1678174095215; Mon, 06 Mar 2023 23:28:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678174095; cv=none; d=google.com; s=arc-20160816; b=ao0ODBUgV3ITB1LaLae63/QzLj05T6kDUKXNLeRbNFW1tMBySYP3QGRzATxgepYQAk ZWY9eB+kXueRD/Jm9aC6Ff9f2Q4utGdW0OPBcUOzuu/ocw03FL4TGIivMG7kfdmykNkJ lFOXZjUjJHrxcH/YLpZHpqEGPjbMZh7oKjpRvKqiGgnmZjFbeJc8swDDsOcedWOavObs cObIkPuc12hwsw0mv6cfFnBrfAsNhvsXHHZkdZjPSd0tL6MAKzJ8Hlu6dsX4fjeT9mOp gYxwqzE7CwYF50QdCgLvFXktdoLnjTx7Qd40nhK//isLEJU/RO8CdTnR3DmfYO6l13mD Mdcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=pKI1kuAHKypUXeareR/09n2feCgpJViSdOybhZI+EsM=; b=rvjurZ7V+0O/TBsqAiCeGWAl+CZnm/lc5XjI7NdHKpxA3vRzmAwIBMtJzysj3ATPci Bb+/JsX+hdS6cVaybdM02AsoSmpcXUA6ZKP6416J0yqsYGBOjG7/3QkwlHZbaXerIB+I PD/nQYsMpmCgHHSj41dm4/ji4tOgQKwabASXGHYVFfN1K7fFCewqOaaEOHDOWhwldNiU he0uOnk60pxlFlYshimh2wfPsZ3p0/W8YiCjsoLr7BMW08UFqtSuRISn15V3A1vcbj5J giTPccm4ijBINwkrTxBNhnNkagyP+BieaLr5f6WavgODOk0EllGao7Lebx3anafjm0Us d+RQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=MyoQeF+Z; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x4-20020a170902ec8400b0019cb63d38e3si12316344plg.589.2023.03.06.23.28.03; Mon, 06 Mar 2023 23:28:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=MyoQeF+Z; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231262AbjCGG6s (ORCPT + 99 others); Tue, 7 Mar 2023 01:58:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59152 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231208AbjCGG6L (ORCPT ); Tue, 7 Mar 2023 01:58:11 -0500 Received: from mail-pg1-x534.google.com (mail-pg1-x534.google.com [IPv6:2607:f8b0:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4DC5688EE1 for ; Mon, 6 Mar 2023 22:57:15 -0800 (PST) Received: by mail-pg1-x534.google.com with SMTP id 16so6992496pge.11 for ; Mon, 06 Mar 2023 22:57:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678172234; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pKI1kuAHKypUXeareR/09n2feCgpJViSdOybhZI+EsM=; b=MyoQeF+Z1Ozq9bE9/PbK9injww7cXqExwhY7rvSLxR4g/4O+Nfc9fGmLGzFYJMpAHD Zob+jZZ+kbE9jLOG+OEUynRAhk/1wDg68aB229TqJDeviGbMSvnk+L9gn1ZEZC2xzZew OV+vxUHqrr+Tv0KeVOcVBlszBPmxncShiPqOZhplHv4c4JKf7I0IqXVPOMWMvDJaZt4R LtsYlahwTiomz3DM5CScou3sBZL/2DaB/HRpvPb940m2H7IkHsGjeco1KuyTRwCNdWya 0gnRlmOrxjy+r8Q8lkkhv4vWG6f46ZgtO0AxZHLWg6LCI26AiYHKnM1ulIrPiJzRhtno V4Qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678172234; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pKI1kuAHKypUXeareR/09n2feCgpJViSdOybhZI+EsM=; b=HzWzQa3kD3hTa5qVzS3lohAw4fLh4w5nrgE7KegPjcLJTKx67U5gKcBk0FxBQyP07C 2RLk6/8+FY0cGh7H6joFAS1X+2j1lsBAxdv65Hxfue+bjJHNu9zu545r8VgdMOGWR0Mu bmW4rGB0mAHgqSlsgkyPn09o2CTrDHYmx8x7mnInLUOyUfSaV2NWhfXOLs42iw35DnW6 zbI/c43+vSe4heQCaFlGczMa/o1/m6RROdXYd+r806YWWSLXphWyaxulWala39Gr5h5h zhp9Qmcs58L4stAnaR8Wdyk3cDXoZaPXzt3Vf9dA3i671i/s3Kv2UZLLfHyLNTukjW68 w91w== X-Gm-Message-State: AO0yUKUtv9QRPkLKXgPCB97o+0p9kT6BLc8HzNHwPFQQ9AHMpiMQ/gzx dOLIyXFlDWWIiJ82jDjSjpH2OA== X-Received: by 2002:aa7:8892:0:b0:5d1:bbd2:7604 with SMTP id z18-20020aa78892000000b005d1bbd27604mr12474542pfe.2.1678172233918; Mon, 06 Mar 2023 22:57:13 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id e5-20020a62ee05000000b00608dae58695sm7230854pfi.209.2023.03.06.22.57.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 22:57:13 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 7/8] mm: vmscan: remove shrinker_rwsem from synchronize_shrinkers() Date: Tue, 7 Mar 2023 14:56:04 +0800 Message-Id: <20230307065605.58209-8-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230307065605.58209-1-zhengqi.arch@bytedance.com> References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759693079798972959?= X-GMAIL-MSGID: =?utf-8?q?1759693079798972959?= Now there are no readers of shrinker_rwsem, so synchronize_shrinkers() does not need to hold the writer of shrinker_rwsem to wait for all running shinkers to complete, synchronize_srcu() is enough. Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- mm/vmscan.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 7aaf6f94ac1b..ac7ab4aa344f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -796,15 +796,11 @@ EXPORT_SYMBOL(unregister_shrinker); /** * synchronize_shrinkers - Wait for all running shrinkers to complete. * - * This is equivalent to calling unregister_shrink() and register_shrinker(), - * but atomically and with less overhead. This is useful to guarantee that all - * shrinker invocations have seen an update, before freeing memory, similar to - * rcu. + * This is useful to guarantee that all shrinker invocations have seen an + * update, before freeing memory. */ void synchronize_shrinkers(void) { - down_write(&shrinker_rwsem); - up_write(&shrinker_rwsem); atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } From patchwork Tue Mar 7 06:56:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 65296 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2289307wrd; Mon, 6 Mar 2023 23:18:26 -0800 (PST) X-Google-Smtp-Source: AK7set+eOV8w+urwetkcGQBpv0hKh7gvPD7A+oM3fNN0Fz1ghWjMuACibKZpO9MCTuo/TfZpJ0NT X-Received: by 2002:a17:902:aa02:b0:19e:2eb5:712d with SMTP id be2-20020a170902aa0200b0019e2eb5712dmr12524703plb.25.1678173506663; Mon, 06 Mar 2023 23:18:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678173506; cv=none; d=google.com; s=arc-20160816; b=MuEId+Jvj2+hjhyM3hBY26akuSL5BEzoGLsyldmc1CUoMrvjmuMnazgUidhuKPeaxR 7wl1TQA+PB1NvWsUxxlcOIpwUiyScEXomqkR0ZVxOEjrYo67nK3ny5z0VgfR+JSb1PI4 AZizN+xGYQtofqWfz0bjbeNWPGvCVU6YBduNqgb6vlFoaY/Bd5BtGxChnpKDsguuXD1z rLhX40AhHPqzmXSZlDaLdHzvnicdlfy+kJCAKo2zzWKM9pHsyiVA5fW2M23A5JCqqtwi UxQFqHSGMwhcB/45J+vENBvmJaurWOnaWb31HQHAjlg3d+3HaFDbmJJJPsgYvrByrIYY OjGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JEwsIZrv65iqCvc8ubXlIP3Z3XNAqh1J9iA+XAih3cE=; b=loKHssINbbTdwpP+inQc3lZ+Zv7Sa0PGgYr/Qs9oKkya3sPNM5Iu4+mymMFFKlrVKi 3q75+WyQnjpvnXDDRkk4wfRP0ADpY9sl9W+nxGoChxdRzcyFexG2BVSOztpR0sYd6YN0 ABFSKS7nELEEj1rdE1L/dCZtRRBZEGgtgUbgRE1+xpUEL5QM1d8hOxkfGE2/ldeuOJ9P G14HIaxBspT20b0NVPPixHIx/j7ZzcmiEOY5QoFmotzJLdnBXZCZtA6GnPN1IUr8q6eW ePYfbL3lzJKbVJ0ASVXnFfQ1ArfupsSXt/qwO+115+jg0/3KeXPm2rdk1v50gb+AWLAQ r4FQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=Rz+OlQxX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u19-20020a170902a61300b0019ce4348dcasi11206472plq.113.2023.03.06.23.18.13; Mon, 06 Mar 2023 23:18:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=Rz+OlQxX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231268AbjCGG6x (ORCPT + 99 others); Tue, 7 Mar 2023 01:58:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231140AbjCGG6W (ORCPT ); Tue, 7 Mar 2023 01:58:22 -0500 Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D29C688EFE for ; Mon, 6 Mar 2023 22:57:23 -0800 (PST) Received: by mail-pf1-x436.google.com with SMTP id fa28so7389169pfb.12 for ; Mon, 06 Mar 2023 22:57:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1678172241; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JEwsIZrv65iqCvc8ubXlIP3Z3XNAqh1J9iA+XAih3cE=; b=Rz+OlQxXJ+H/lFVEkuWxvVRfLx2yy/2tPDqB+C/WLB56VLSKXLTwwrE98jNSXgcCeH Wp6ocsD+lSOvkeaVrQZW5FS2yg4o9s1dz1qiHMRxFASBIpc2JVftANvxB++8sPANbxME twinknB2RKFoFIRTXehrwocBnnddLU0nmhVWDA0ZICfv6A2YIPyUIpqw70fmmMDpC9LO go9gjPkLef7kHYIg+lwI+6Lb//VWhFFJMJNWrvhgTYNSkuf3AGI3BRtPFYG9oGd6YtoV hEYb4BtRHKy7EJ9lOblKA1RqHDTg9Y++t4SZN/OXDN6aiH/uovJaGTEGR8hEEHJgMKni koTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678172241; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JEwsIZrv65iqCvc8ubXlIP3Z3XNAqh1J9iA+XAih3cE=; b=ehN3InZsIlu/xjmg99Jm4SZByxvUeqIudFvSuGUk8w1EkikX1yqEfT3B3wW1UuonoY yqWW5VgF7VUsyvAtixXhtyhQorLv4r4MJoRTO2xrnOwiA5xA+cylcj6KtuLHLIihXx2X PuWu/OXMtCI6mYCTUSI2YgfWr/w6iARL22CnZvHKJtVPjcyOb05vLlYjLCF7GwH1tPTQ uAOmzrsFGrGGixAq7XwHiZw6nGlvrP51Q8ANLkASm+2kbT2g9Few9LboTfQ90ZiEkbvl ESRJYdPKTFU7z8lYoNLvV7J4zZm7ysVUjWtE5S0sr3QmtumjbANxQoxSt+n0pp7/a3EF YZkQ== X-Gm-Message-State: AO0yUKU4D04pQeArkZ/8kIhZ7M8ulTDtMxIYvSgmLO0HOBB8DlAFbSXz bh0TBIxnb4csY25Avene9J005A== X-Received: by 2002:a05:6a00:1d8c:b0:61a:50f:cad1 with SMTP id z12-20020a056a001d8c00b0061a050fcad1mr9376180pfw.3.1678172241246; Mon, 06 Mar 2023 22:57:21 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.245]) by smtp.gmail.com with ESMTPSA id e5-20020a62ee05000000b00608dae58695sm7230854pfi.209.2023.03.06.22.57.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Mar 2023 22:57:20 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com, rppt@kernel.org Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 8/8] mm: shrinkers: convert shrinker_rwsem to mutex Date: Tue, 7 Mar 2023 14:56:05 +0800 Message-Id: <20230307065605.58209-9-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230307065605.58209-1-zhengqi.arch@bytedance.com> References: <20230307065605.58209-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759692462852411970?= X-GMAIL-MSGID: =?utf-8?q?1759692462852411970?= Now there are no readers of shrinker_rwsem, so we can simply replace it with mutex lock. Signed-off-by: Qi Zheng Acked-by: Vlastimil Babka Acked-by: Kirill Tkhai --- drivers/md/dm-cache-metadata.c | 2 +- drivers/md/dm-thin-metadata.c | 2 +- fs/super.c | 2 +- mm/shrinker_debug.c | 14 +++++++------- mm/vmscan.c | 34 +++++++++++++++++----------------- 5 files changed, 27 insertions(+), 27 deletions(-) diff --git a/drivers/md/dm-cache-metadata.c b/drivers/md/dm-cache-metadata.c index acffed750e3e..9e0c69958587 100644 --- a/drivers/md/dm-cache-metadata.c +++ b/drivers/md/dm-cache-metadata.c @@ -1828,7 +1828,7 @@ int dm_cache_metadata_abort(struct dm_cache_metadata *cmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * cmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs cmd root_lock). - * - must take shrinker_rwsem without holding cmd->root_lock + * - must take shrinker_mutex without holding cmd->root_lock */ new_bm = dm_block_manager_create(cmd->bdev, DM_CACHE_METADATA_BLOCK_SIZE << SECTOR_SHIFT, CACHE_MAX_CONCURRENT_LOCKS); diff --git a/drivers/md/dm-thin-metadata.c b/drivers/md/dm-thin-metadata.c index fd464fb024c3..9f5cb52c5763 100644 --- a/drivers/md/dm-thin-metadata.c +++ b/drivers/md/dm-thin-metadata.c @@ -1887,7 +1887,7 @@ int dm_pool_abort_metadata(struct dm_pool_metadata *pmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * pmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs pmd root_lock). - * - must take shrinker_rwsem without holding pmd->root_lock + * - must take shrinker_mutex without holding pmd->root_lock */ new_bm = dm_block_manager_create(pmd->bdev, THIN_METADATA_BLOCK_SIZE << SECTOR_SHIFT, THIN_MAX_CONCURRENT_LOCKS); diff --git a/fs/super.c b/fs/super.c index 84332d5cb817..91a4037b1d95 100644 --- a/fs/super.c +++ b/fs/super.c @@ -54,7 +54,7 @@ static char *sb_writers_name[SB_FREEZE_LEVELS] = { * One thing we have to be careful of with a per-sb shrinker is that we don't * drop the last active reference to the superblock from within the shrinker. * If that happens we could trigger unregistering the shrinker from within the - * shrinker path and that leads to deadlock on the shrinker_rwsem. Hence we + * shrinker path and that leads to deadlock on the shrinker_mutex. Hence we * take a passive reference to the superblock to avoid this from occurring. */ static unsigned long super_cache_scan(struct shrinker *shrink, diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 6aa7a7ec69da..b0f6aff372df 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -7,7 +7,7 @@ #include /* defined in vmscan.c */ -extern struct rw_semaphore shrinker_rwsem; +extern struct mutex shrinker_mutex; extern struct list_head shrinker_list; extern struct srcu_struct shrinker_srcu; @@ -167,7 +167,7 @@ int shrinker_debugfs_add(struct shrinker *shrinker) char buf[128]; int id; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); /* debugfs isn't initialized yet, add debugfs entries later. */ if (!shrinker_debugfs_root) @@ -210,7 +210,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) if (!new) return -ENOMEM; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); old = shrinker->name; shrinker->name = new; @@ -228,7 +228,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) shrinker->debugfs_entry = entry; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); kfree_const(old); @@ -240,7 +240,7 @@ struct dentry *shrinker_debugfs_remove(struct shrinker *shrinker) { struct dentry *entry = shrinker->debugfs_entry; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); kfree_const(shrinker->name); shrinker->name = NULL; @@ -265,14 +265,14 @@ static int __init shrinker_debugfs_init(void) shrinker_debugfs_root = dentry; /* Create debugfs entries for shrinkers registered at boot */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_for_each_entry(shrinker, &shrinker_list, list) if (!shrinker->debugfs_entry) { ret = shrinker_debugfs_add(shrinker); if (ret) break; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } diff --git a/mm/vmscan.c b/mm/vmscan.c index ac7ab4aa344f..c00302fabc3d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -35,7 +35,7 @@ #include #include #include -#include +#include #include #include #include @@ -202,7 +202,7 @@ static void set_task_reclaim_state(struct task_struct *task, } LIST_HEAD(shrinker_list); -DECLARE_RWSEM(shrinker_rwsem); +DEFINE_MUTEX(shrinker_mutex); DEFINE_SRCU(shrinker_srcu); static atomic_t shrinker_srcu_generation = ATOMIC_INIT(0); @@ -225,7 +225,7 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, { return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, &shrinker_srcu, - lockdep_is_held(&shrinker_rwsem)); + lockdep_is_held(&shrinker_mutex)); } static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, @@ -310,7 +310,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) int nid, size, ret = 0; int map_size, defer_size = 0; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); map_size = shrinker_map_size(shrinker_nr_max); defer_size = shrinker_defer_size(shrinker_nr_max); size = map_size + defer_size; @@ -326,7 +326,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -342,7 +342,7 @@ static int expand_shrinker_info(int new_id) if (!root_mem_cgroup) goto out; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); map_size = shrinker_map_size(new_nr_max); defer_size = shrinker_defer_size(new_nr_max); @@ -392,7 +392,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) if (mem_cgroup_disabled()) return -ENOSYS; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -406,7 +406,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) shrinker->id = id; ret = 0; unlock: - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -416,7 +416,7 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) BUG_ON(id < 0); - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); idr_remove(&shrinker_idr, id); } @@ -451,7 +451,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -460,7 +460,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } static bool cgroup_reclaim(struct scan_control *sc) @@ -709,9 +709,9 @@ void free_prealloced_shrinker(struct shrinker *shrinker) shrinker->name = NULL; #endif if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); unregister_memcg_shrinker(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return; } @@ -721,11 +721,11 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } static int __register_shrinker(struct shrinker *shrinker) @@ -775,13 +775,13 @@ void unregister_shrinker(struct shrinker *shrinker) if (!(shrinker->flags & SHRINKER_REGISTERED)) return; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_remove(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu);