From patchwork Sun Feb 26 14:46:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 61623 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1973267wrd; Sun, 26 Feb 2023 06:54:20 -0800 (PST) X-Google-Smtp-Source: AK7set/kjZZgGKgi5sslv53IcDjLxFYDPBmkWO1Q8NUOaL6xX7nkINZKv9QbO5TEHNRxQLLe6H0A X-Received: by 2002:a05:6a00:18aa:b0:5e6:b80d:c76f with SMTP id x42-20020a056a0018aa00b005e6b80dc76fmr5857797pfh.0.1677423260220; Sun, 26 Feb 2023 06:54:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677423260; cv=none; d=google.com; s=arc-20160816; b=FcxshooIfP9P8nrhTWQMNIfKGQ6Ca3w9rHCtbUVOpV0rFGRCWb6Ari4SWx72uayMbS XkyWA9CyLptIwUwdnwETebCgeEu9WOznSsZM6zayXO41JmEjEJOzOqPlITUu+eH/86rc 1xuVwoK4cAXSXh8ukqYoPqTGHvxOtaGfTgA9rEnZ0l8dD5z58tliQE7O0xe6JfQWR72M LIqaN4/+riJyakE7Cqu0AGySNv7ZybjS13CszOhu0iqLVEuk41Iy/8XPgp8cGEoe9N/C dZtFh8uy2vPCiYc7yg52cYLJ/J1nUsHPvlIvpO5OhljJmKxLjPIJRDOyrhEkW1zaPmvx xh+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jZz7Wrj6RpcNJkd0HlCRkRoarWFliGItBtXoMhl1VyQ=; b=xIcz7Zn+gwJZBZqK6Qdl2SV0ibzoYtFSRxjsG01Wu7PjxI/0c0tZBN2GZtXHgGXOba p8On9wIz9QRwUIzwkmsY+0Nq/LdMBQkjPty4ddb5DnYBRgBXwOXwmzZngRTVEf7mfIg+ litCqAC2zuSGwR8VI8qfU9LLMVDjpJF3Y/2zQs8W0xonI4U/7uiwcEy7tw2mdJhKLh9j FIRQnzi2Z0/gASi03G17e8/bFctzj4nUtExyy4ou5fJOxRI/NXlzP4YAdrKcQZI7QbQW 2u12MpXJ3RL6P4bSYOV55GgzlGlTuK3vnm/GPg/QjGgTT0V0SAIf9UY92ZE4orSyoNPP zlzQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=OEQgF8XF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x69-20020a638648000000b004fbc4d51d21si4424864pgd.342.2023.02.26.06.54.07; Sun, 26 Feb 2023 06:54:20 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=OEQgF8XF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230401AbjBZOxy (ORCPT + 99 others); Sun, 26 Feb 2023 09:53:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55102 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230330AbjBZOxQ (ORCPT ); Sun, 26 Feb 2023 09:53:16 -0500 Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4650A46BF for ; Sun, 26 Feb 2023 06:49:48 -0800 (PST) Received: by mail-pj1-x1035.google.com with SMTP id qa18-20020a17090b4fd200b0023750b675f5so7526296pjb.3 for ; Sun, 26 Feb 2023 06:49:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jZz7Wrj6RpcNJkd0HlCRkRoarWFliGItBtXoMhl1VyQ=; b=OEQgF8XFhS1YnbfOQOJh1IFogO5l8aVUQrCwR9nyy7oN1K9Ji93rp0LItnd8h4XE6q oXvlgHNvYZY7Sru38EHw5XpbSsjVDCpErYLQOtN774iKL22Mm3meSyNn/DoDpseqe7VK M148czmxLrasp9EktCEGPZBaRaJCIL/h+oxW/Len8VuNVSZLJgVQPTgQZl2H0oXCxaKT th31uEB6vb+9QTQp2lgTemj0Wo5Eo3476YMBJNBoEXl1eXjK1lRElSTkvEo/pHe8O8zz kKJciJ2YT4YMKnGfyaACr4ey76nUPX2Xt475p0Ss+/Ot3sZ5uU3wsAqqCutxvYDzlxUi SM6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jZz7Wrj6RpcNJkd0HlCRkRoarWFliGItBtXoMhl1VyQ=; b=LAVMf7rtYuChFwk9LjoYprScrpXUZ2a8xb9PjRk7p9mL4nWq3TlQv0olhss1Km1CJE wSQT3A25hcZGg9YBRvc4PK8g4qZgb6VZph1euCVuQLVslQtuNFgm0sy2fkHniY4TNDpj 2pDMfIhI4Cjpl/L+AiqKWT1J/n00s7a008R4+5bydOF4TTN8U2ree6Y0OfQnCZErtY5v Ey6jqf6kctQMU3F4J+Dss4ueq/ewQAgTAAX8dYae6fYyEjB6++HlrCZMl+tsZUfu7kXZ gdJxtgeMMJXAXr6EnD2/eH2J/bDUINcZEgdb+Ulo7qdmWxsnwHTTZY/6zGm4PG6UTjSa xVDg== X-Gm-Message-State: AO0yUKVmocvjQkl7hhdIB6ay0VrLEH0BM6xwpgaJyKE+shBu6sKIGUjn KFVJ5JJSEItLqMk1jDwZleeVDA== X-Received: by 2002:a17:902:d4c8:b0:19a:7217:32af with SMTP id o8-20020a170902d4c800b0019a721732afmr23248669plg.5.1677422896084; Sun, 26 Feb 2023 06:48:16 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:15 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 1/8] mm: vmscan: add a map_nr_max field to shrinker_info Date: Sun, 26 Feb 2023 22:46:48 +0800 Message-Id: <20230226144655.79778-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758905772769136753?= X-GMAIL-MSGID: =?utf-8?q?1758905772769136753?= To prepare for the subsequent lockless memcg slab shrink, add a map_nr_max field to struct shrinker_info to records its own real shrinker_nr_max. Suggested-by: Kirill Tkhai Signed-off-by: Qi Zheng --- include/linux/memcontrol.h | 1 + mm/vmscan.c | 41 ++++++++++++++++++++++---------------- 2 files changed, 25 insertions(+), 17 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..aa69ea98e2d8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -97,6 +97,7 @@ struct shrinker_info { struct rcu_head rcu; atomic_long_t *nr_deferred; unsigned long *map; + int map_nr_max; }; struct lruvec_stats_percpu { diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8..546c07ccb3bc 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -224,9 +224,16 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, lockdep_is_held(&shrinker_rwsem)); } +static inline bool need_expand(int new_nr_max, int old_nr_max) +{ + return round_up(new_nr_max, BITS_PER_LONG) > + round_up(old_nr_max, BITS_PER_LONG); +} + static int expand_one_shrinker_info(struct mem_cgroup *memcg, int map_size, int defer_size, - int old_map_size, int old_defer_size) + int old_map_size, int old_defer_size, + int new_nr_max) { struct shrinker_info *new, *old; struct mem_cgroup_per_node *pn; @@ -240,12 +247,17 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, if (!old) return 0; + /* Already expanded this shrinker_info */ + if (!need_expand(new_nr_max, old->map_nr_max)) + return 0; + new = kvmalloc_node(sizeof(*new) + size, GFP_KERNEL, nid); if (!new) return -ENOMEM; new->nr_deferred = (atomic_long_t *)(new + 1); new->map = (void *)new->nr_deferred + defer_size; + new->map_nr_max = new_nr_max; /* map: set all old bits, clear all new bits */ memset(new->map, (int)0xff, old_map_size); @@ -295,6 +307,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) } info->nr_deferred = (atomic_long_t *)(info + 1); info->map = (void *)info->nr_deferred + defer_size; + info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } up_write(&shrinker_rwsem); @@ -302,23 +315,14 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) return ret; } -static inline bool need_expand(int nr_max) -{ - return round_up(nr_max, BITS_PER_LONG) > - round_up(shrinker_nr_max, BITS_PER_LONG); -} - static int expand_shrinker_info(int new_id) { int ret = 0; - int new_nr_max = new_id + 1; + int new_nr_max = round_up(new_id + 1, BITS_PER_LONG); int map_size, defer_size = 0; int old_map_size, old_defer_size = 0; struct mem_cgroup *memcg; - if (!need_expand(new_nr_max)) - goto out; - if (!root_mem_cgroup) goto out; @@ -332,7 +336,8 @@ static int expand_shrinker_info(int new_id) memcg = mem_cgroup_iter(NULL, NULL, NULL); do { ret = expand_one_shrinker_info(memcg, map_size, defer_size, - old_map_size, old_defer_size); + old_map_size, old_defer_size, + new_nr_max); if (ret) { mem_cgroup_iter_break(NULL, memcg); goto out; @@ -352,9 +357,11 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) rcu_read_lock(); info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); - /* Pairs with smp mb in shrink_slab() */ - smp_mb__before_atomic(); - set_bit(shrinker_id, info->map); + if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { + /* Pairs with smp mb in shrink_slab() */ + smp_mb__before_atomic(); + set_bit(shrinker_id, info->map); + } rcu_read_unlock(); } } @@ -432,7 +439,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); - for (i = 0; i < shrinker_nr_max; i++) { + for (i = 0; i < child_info->map_nr_max; i++) { nr = atomic_long_read(&child_info->nr_deferred[i]); atomic_long_add(nr, &parent_info->nr_deferred[i]); } @@ -899,7 +906,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, shrinker_nr_max) { + for_each_set_bit(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, From patchwork Sun Feb 26 14:46:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 61622 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1973252wrd; Sun, 26 Feb 2023 06:54:18 -0800 (PST) X-Google-Smtp-Source: AK7set+UW/U+5mrvxuuYUOUjOS34ut/GQzU4SVu4uNUrIiNqMT5IzUIOnmrZhfxgU6GOwTZJ6rD2 X-Received: by 2002:a17:90a:bf0d:b0:233:a836:15f4 with SMTP id c13-20020a17090abf0d00b00233a83615f4mr22118442pjs.1.1677423257996; Sun, 26 Feb 2023 06:54:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677423257; cv=none; d=google.com; s=arc-20160816; b=uOIXf4BOESM+1UL/TpjOuAq+pejemo+XHugbdoRLGrBu0cnCmr9S7EzeYJX5dhhSJN lAcXgz2tT2LmO4U0YjW+01MY/Pzf046r/P70poc26W3Ry2jXdBKzeXMF2PADgL4Y/ekx Q3DxC85q5eZsfRb2uZsK3Jxixk63MLJU0PTJNs/b4iun1qhb3EAtNoJEYQC1Jl8Gdbk7 xhQGKqpkaHR42hY+QoCtdwUHsB5UHUvkj/Pkhm+RqR8gOwBWOmcMocOrwxYOdIZWdks1 Beu7Qm/OpJ8VNyqhGWLJQc1ArEJ/Oni+c2O/MHTR5RRcU2cSNP23jIKaYy1jbSn+jrtX zI3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=xJVL9LTM3YR7R2xJV6pOC5vrFajMNWUdaUGoIRklN4g=; b=CqT66g4yBFEiVqwPoUD1WVpxrj3TVtLb/YAdPUbxTart0VbB9hZIY5j3SBvzYc4Czv MjyJgst9jyjjjRyVF0lu1125MAkiU1clFGNNbDgpXq+w13L/bhnTypfeXgz1B+z0wmzH GN/lZEoSzI29YDLjya7lyDmLtP4Zo2en55ncpdGQ0jGXwnKFBdKPdtEET/WbQ7XjZEM6 3KA6lMdSKHmI3r5sh5++scgpsC+a4SPZ7svQWXFqh91zLx9g7oOKGFsnZgwCABqA6ebf DF0zSGGGT3rznP6561/3GL11i62FqsynyFsy6tvPKOnltzMtFtRgSHmJIrt1bzFOkjQ/ m+IQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=ALVQXXOf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id iq17-20020a17090afb5100b00233efa8d3e5si7763541pjb.20.2023.02.26.06.54.05; Sun, 26 Feb 2023 06:54:17 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=ALVQXXOf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230386AbjBZOxv (ORCPT + 99 others); Sun, 26 Feb 2023 09:53:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44776 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230208AbjBZOxL (ORCPT ); Sun, 26 Feb 2023 09:53:11 -0500 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE4E91A4A4 for ; Sun, 26 Feb 2023 06:49:30 -0800 (PST) Received: by mail-pl1-x632.google.com with SMTP id i3so4209103plg.6 for ; Sun, 26 Feb 2023 06:49:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1677422902; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xJVL9LTM3YR7R2xJV6pOC5vrFajMNWUdaUGoIRklN4g=; b=ALVQXXOfu8Q3gVhZC/7gcT/As/x0PQNyX2nqdY6UDPAFzmCXukW/onK5TzeE9pcwvk vXJzQkiBP+sXbTYXsw+HpB7JRRbrwmsjkOEq5DQ1S1Eac1N0S/0/DsgoqOjYjHQYnPSR P4R9Pfo0jggo+dcJDr8XJLDD9doWZVqghSIkea6uqcKjvoOktQF/G7VDIKPmpqIX0uaq CUp5L8HF8li82mResfb29HBRedCwzzg7BH3UMRf7q7wMBQ/7+ybQtKL36gQDJZu0gwHP FCqtYPLddEJNMwfVTwEbd49hBEqsrqinMleX5XcFcj1s7BFSCNs7Wfig3bKPX0H4mz4V BKmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677422902; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xJVL9LTM3YR7R2xJV6pOC5vrFajMNWUdaUGoIRklN4g=; b=kx69l754ul2Sqs4Xs1g0SDtaTzNs2jNSxZqK9n+D2Zq0rrLCiVbdckVUIiuaRFhYTN c8QFmWshy3iFhhmzlXyLSlk/1fDoNOiHxW1+vHbc0iCpx2Dw+x4OXTXVp0ukGQ7Ne5k+ UvS+k/AgAxDmhemEXQe8TrIZPkq5GXPe+2yQjS8UGeTOdJpV6Of0Tkv7eEIa/JVJeowc D/IDUbzatyVvj9Fe4I2YZ61IAo+6U8bnjnhFRtb7DREZowZN6A90cQ+HM+c9ovAbGL3G Cqxihy/TiyA0BK74Oj8KRH+jLbd2U0qomOvEVmdo+4hoFb32WNKseBr2FRxZqMd/znAt Vy5A== X-Gm-Message-State: AO0yUKW0avX26tNNgSBfu+/wImHGDSfTj6u7rPky0oodb/epom0Yc/kc gxCwgvkCoE4L9SQdenslcYDjew== X-Received: by 2002:a17:902:e84b:b0:19a:7439:3e98 with SMTP id t11-20020a170902e84b00b0019a74393e98mr23301977plg.4.1677422901796; Sun, 26 Feb 2023 06:48:21 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:21 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 2/8] mm: vmscan: make global slab shrink lockless Date: Sun, 26 Feb 2023 22:46:49 +0800 Message-Id: <20230226144655.79778-3-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758905769918062338?= X-GMAIL-MSGID: =?utf-8?q?1758905769918062338?= The shrinker_rwsem is a global lock in shrinkers subsystem, it is easy to cause blocking in the following cases: a. the write lock of shrinker_rwsem was held for too long. For example, there are many memcgs in the system, which causes some paths to hold locks and traverse it for too long. (e.g. expand_shrinker_info()) b. the read lock of shrinker_rwsem was held for too long, and a writer came at this time. Then this writer will be forced to wait and block all subsequent readers. For example: - be scheduled when the read lock of shrinker_rwsem is held in do_shrink_slab() - some shrinker are blocked for too long. Like the case mentioned in the patchset[1]. Therefore, many times in history ([2],[3],[4],[5]), some people wanted to replace shrinker_rwsem reader with SRCU, but they all gave up because SRCU was not unconditionally enabled. But now, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), the SRCU is unconditionally enabled. So it's time to use SRCU to protect readers who previously held shrinker_rwsem. [1]. https://lore.kernel.org/lkml/20191129214541.3110-1-ptikhomirov@virtuozzo.com/ [2]. https://lore.kernel.org/all/1437080113.3596.2.camel@stgolabs.net/ [3]. https://lore.kernel.org/lkml/1510609063-3327-1-git-send-email-penguin-kernel@I-love.SAKURA.ne.jp/ [4]. https://lore.kernel.org/lkml/153365347929.19074.12509495712735843805.stgit@localhost.localdomain/ [5]. https://lore.kernel.org/lkml/20210927074823.5825-1-sultan@kerneltoast.com/ Signed-off-by: Qi Zheng --- mm/vmscan.c | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 546c07ccb3bc..2a21a84d3db1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -202,6 +202,7 @@ static void set_task_reclaim_state(struct task_struct *task, LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); +DEFINE_SRCU(shrinker_srcu); #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -706,7 +707,7 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { down_write(&shrinker_rwsem); - list_add_tail(&shrinker->list, &shrinker_list); + list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); up_write(&shrinker_rwsem); @@ -760,13 +761,15 @@ void unregister_shrinker(struct shrinker *shrinker) return; down_write(&shrinker_rwsem); - list_del(&shrinker->list); + list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_remove(shrinker); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); + debugfs_remove_recursive(debugfs_entry); kfree(shrinker->nr_deferred); @@ -786,6 +789,7 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); + synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -996,6 +1000,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, { unsigned long ret, freed = 0; struct shrinker *shrinker; + int srcu_idx; /* * The root memcg might be allocated even though memcg is disabled @@ -1007,10 +1012,10 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg)) return shrink_slab_memcg(gfp_mask, nid, memcg, priority); - if (!down_read_trylock(&shrinker_rwsem)) - goto out; + srcu_idx = srcu_read_lock(&shrinker_srcu); - list_for_each_entry(shrinker, &shrinker_list, list) { + list_for_each_entry_srcu(shrinker, &shrinker_list, list, + srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -1021,19 +1026,9 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; freed += ret; - /* - * Bail out if someone want to register a new shrinker to - * prevent the registration from being stalled for long periods - * by parallel ongoing shrinking. - */ - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } } - up_read(&shrinker_rwsem); -out: + srcu_read_unlock(&shrinker_srcu, srcu_idx); cond_resched(); return freed; } From patchwork Sun Feb 26 14:46:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 61624 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1973378wrd; Sun, 26 Feb 2023 06:54:40 -0800 (PST) X-Google-Smtp-Source: AK7set+WaBG6utrTtwRVbfORsniSyQ1c5wZzO0/5WAG61FDVqOmY2/MMjQ7nraElqwIkx/RYSaMS X-Received: by 2002:a05:6a00:1d20:b0:5e4:f141:568b with SMTP id a32-20020a056a001d2000b005e4f141568bmr7400570pfx.3.1677423280113; Sun, 26 Feb 2023 06:54:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677423280; cv=none; d=google.com; s=arc-20160816; b=LJZVFylxTWS5SnOZtheKP8qWyl3IsymQqW5UJTOcW7ogT+dBBSGs4G2W0jSG9YlDQc GhTEE2UKpfpHNnyzr8UFv1GIvj4dOPdpbq0UDZBsoB3XBM7Bu22b3bVmNk+eTYIYmPUf F9eQMtpZcsLKxYSK1o8apcEqvd3kk+qk3VaYkWsP8qHb6JBB5PB+75hX9obFJI24D9ej p9WdsJ0GgMLk091ift46Kw10ySgd5cj1h+4semHffUROQG3Yp9AWZ1P6Iizk/KnHwQuY xIOQkdkYRWLXC1e5WJYvzysspiyqDeiInwFonAkuWAXIsGQKq8/OSckWaF0xoU2B1X6l EJSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=gvRso6AuWsz0X9JCAZwWH9GIT0TN5B8gocPlJ++Lup8=; b=LBwlSfSI5WsknSHGwo0t0vUR/spmLMgJHRJm3qsDqUk4GYbXFQa5LnFt7mhg5+b2lJ 0xUHtxM8sUoropGeHMhWgugSLAPsPafp/OIK++RBUiHhlL/zXTj/0JG8CrJ61FFuOWng smWRfeTR/25OzO/J29Dk8w3dfehYHhjYgJUcWcrreFTQGcr5hS51VCyVAvXTIdZ01wsj gyGKGK8eS/stQY4DvPR+2SnNlLZe0CNEW+J7lK0/HV/zA58mlYBAjPBVJAB4of7Vm81k hOqOqLd4CunjRZp4LUJpy7TWOsmrMPzGxm1ERWKBHoOsA/UfkrgFb8Shum4YShOUOgvD NNlA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=LzQ+kcdg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c20-20020a056a00249400b005a8d987d003si5151216pfv.55.2023.02.26.06.54.27; Sun, 26 Feb 2023 06:54:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=LzQ+kcdg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230363AbjBZOyL (ORCPT + 99 others); Sun, 26 Feb 2023 09:54:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54748 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230317AbjBZOxb (ORCPT ); Sun, 26 Feb 2023 09:53:31 -0500 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A688814213 for ; Sun, 26 Feb 2023 06:49:43 -0800 (PST) Received: by mail-pl1-x62f.google.com with SMTP id v11so680713plz.8 for ; Sun, 26 Feb 2023 06:49:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gvRso6AuWsz0X9JCAZwWH9GIT0TN5B8gocPlJ++Lup8=; b=LzQ+kcdgxQjvYV6/aBTUiIdXty1SduBzej96ViyOvdlzB3Fsg4Lcdgbz3j4hmKO+8+ TIcLX9NYqDrIQ5/ezPI3RaFfYwQ1ZEDeoEpqbn5P8z2NLz+savfhhRBcSPVq7BwrsDXr R1O+OCmiRS291b9X4XNwI5b5W0qVttXgY3Oqixpog+eVUFyly+Rm9YGT9YU8Ro/3F5Pw d/nQ47A2O7bUl+GBn4gAU0PulKhyQ7Bu/CfMAW5zYIkO2kY76GczKehRR3kCEzOW262q ug1kCR6iRSBx0q+cNBVn+mB+nddqScXB+yCOQ55hN5exNnBS/N7LGAnn4tefuskRqhyI gJZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gvRso6AuWsz0X9JCAZwWH9GIT0TN5B8gocPlJ++Lup8=; b=HhtZSwgclEP/XebHoIteUQPhgLBJgRKHRySrYebUH6CQB1wHmMpU7u46yPLU2dcGuf KLwv/hd7sYlm+YVzzJS54bQX//MyxAXOakbSmiTJ7rEKiY/tBZOJXkKl2VTGGTpvdNXi jjq17L/aboFZnEXqxAJRyLymi5FruidXvdO8+9BvwVeOVdWHVpTZ4Hb3OgJUgZY5syT+ tnSeT9zR1+7/XSmTW9n04lyN9VPiK4lYs/ty38O9yodaWm2XiVBlzxxGnAvfJgOwKFh6 ksC3IDkjYKwkyUoQwudnfnXZDZ5/W8htryt5zCHtK/9HMEWs0cLO3FVYMPWnXo5t3QgS fryg== X-Gm-Message-State: AO0yUKUTQHkrXvEwvkAVUeG9zm0gphN9148oDbrsynVxVZ8QolxlU+oH cJC5q9xyg6M2o11Z0vGk91ZiiA== X-Received: by 2002:a17:903:230f:b0:19a:7060:948 with SMTP id d15-20020a170903230f00b0019a70600948mr21481855plh.1.1677422907679; Sun, 26 Feb 2023 06:48:27 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:27 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 3/8] mm: vmscan: make memcg slab shrink lockless Date: Sun, 26 Feb 2023 22:46:50 +0800 Message-Id: <20230226144655.79778-4-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758905793225542558?= X-GMAIL-MSGID: =?utf-8?q?1758905793225542558?= Like global slab shrink, since commit 1cd0bd06093c ("rcu: Remove CONFIG_SRCU"), it's time to use SRCU to protect readers who previously held shrinker_rwsem. We can test with the following script: ``` DIR="/root/shrinker/memcg/mnt" do_create() { mkdir /sys/fs/cgroup/memory/test echo 200M > /sys/fs/cgroup/memory/test/memory.limit_in_bytes for i in `seq 0 $1`; do mkdir /sys/fs/cgroup/memory/test/$i; echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; mkdir -p $DIR/$i; done } do_mount() { for i in `seq $1 $2`; do mount -t tmpfs $i $DIR/$i; done } do_touch() { for i in `seq $1 $2`; do echo $$ > /sys/fs/cgroup/memory/test/$i/cgroup.procs; dd if=/dev/zero of=$DIR/$i/file$i bs=1M count=1 & done } do_create 2000 do_mount 0 2000 do_touch 0 1000 ``` Before applying: 46.60% [kernel] [k] down_read_trylock 18.70% [kernel] [k] up_read 15.44% [kernel] [k] shrink_slab 4.37% [kernel] [k] _find_next_bit 2.75% [kernel] [k] xa_load 2.07% [kernel] [k] idr_find 1.73% [kernel] [k] do_shrink_slab 1.42% [kernel] [k] shrink_lruvec 0.74% [kernel] [k] shrink_node 0.60% [kernel] [k] list_lru_count_one After applying: 19.53% [kernel] [k] _find_next_bit 14.63% [kernel] [k] do_shrink_slab 14.58% [kernel] [k] shrink_slab 11.83% [kernel] [k] shrink_lruvec 9.33% [kernel] [k] __blk_flush_plug 6.67% [kernel] [k] mem_cgroup_iter 3.73% [kernel] [k] list_lru_count_one 2.43% [kernel] [k] shrink_node 1.96% [kernel] [k] super_cache_count 1.78% [kernel] [k] __rcu_read_unlock 1.38% [kernel] [k] __srcu_read_lock 1.30% [kernel] [k] xas_descend We can see that the readers is no longer blocked. Signed-off-by: Qi Zheng --- mm/vmscan.c | 46 +++++++++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 19 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 2a21a84d3db1..490764f8e085 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include @@ -221,8 +222,21 @@ static inline int shrinker_defer_size(int nr_items) static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, int nid) { - return rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_info, - lockdep_is_held(&shrinker_rwsem)); + return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu, + lockdep_is_held(&shrinker_rwsem)); +} + +static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, + int nid) +{ + return srcu_dereference(memcg->nodeinfo[nid]->shrinker_info, + &shrinker_srcu); +} + +static void free_shrinker_info_rcu(struct rcu_head *head) +{ + kvfree(container_of(head, struct shrinker_info, rcu)); } static inline bool need_expand(int new_nr_max, int old_nr_max) @@ -269,7 +283,7 @@ static int expand_one_shrinker_info(struct mem_cgroup *memcg, defer_size - old_defer_size); rcu_assign_pointer(pn->shrinker_info, new); - kvfree_rcu(old, rcu); + call_srcu(&shrinker_srcu, &old->rcu, free_shrinker_info_rcu); } return 0; @@ -355,15 +369,16 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) { if (shrinker_id >= 0 && memcg && !mem_cgroup_is_root(memcg)) { struct shrinker_info *info; + int srcu_idx; - rcu_read_lock(); - info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { /* Pairs with smp mb in shrink_slab() */ smp_mb__before_atomic(); set_bit(shrinker_id, info->map); } - rcu_read_unlock(); + srcu_read_unlock(&shrinker_srcu, srcu_idx); } } @@ -377,7 +392,6 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) return -ENOSYS; down_write(&shrinker_rwsem); - /* This may call shrinker, so it must use down_read_trylock() */ id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -411,7 +425,7 @@ static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_xchg(&info->nr_deferred[shrinker->id], 0); } @@ -420,7 +434,7 @@ static long add_nr_deferred_memcg(long nr, int nid, struct shrinker *shrinker, { struct shrinker_info *info; - info = shrinker_info_protected(memcg, nid); + info = shrinker_info_srcu(memcg, nid); return atomic_long_add_return(nr, &info->nr_deferred[shrinker->id]); } @@ -898,15 +912,14 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; + int srcu_idx; int i; if (!mem_cgroup_online(memcg)) return 0; - if (!down_read_trylock(&shrinker_rwsem)) - return 0; - - info = shrinker_info_protected(memcg, nid); + srcu_idx = srcu_read_lock(&shrinker_srcu); + info = shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; @@ -956,14 +969,9 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, i); } freed += ret; - - if (rwsem_is_contended(&shrinker_rwsem)) { - freed = freed ? : 1; - break; - } } unlock: - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); return freed; } #else /* CONFIG_MEMCG */ From patchwork Sun Feb 26 14:46:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 61625 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1973441wrd; Sun, 26 Feb 2023 06:54:58 -0800 (PST) X-Google-Smtp-Source: AK7set/E1EmSZpegv19GSbJueaqDaI3Id9sAygoS3C9xVy4yRJoPL4GPjy5VCXtvKKShWg5djYVp X-Received: by 2002:a05:6a20:7f9c:b0:cc:4118:65c4 with SMTP id d28-20020a056a207f9c00b000cc411865c4mr15061044pzj.5.1677423297799; Sun, 26 Feb 2023 06:54:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677423297; cv=none; d=google.com; s=arc-20160816; b=kzrLIzz5oHai90TuiSPRWh59BiGSrYt/zwv7zrxYAMYGQvPaC2t9EOJ07fOwmzHmf9 BK1+LaDrGv9rqGLobF0J/HizbiqA54EHO+8vGXFLa4yv/eKEJVjKmfzfAZH6hbYJr6K0 0f5upYbt6Oo3MRszpp4y2WFCtnW5JccmuC/JuHrTXAHlQZU9d+//H9UZ7NfPhHWqmtja kpk3tvRVYLbXyJeOEFrccuaq8OF8ZdX604TAeMjpxdJe4wUtVsIphpITlt/m6qgQyE8b iSwC4d0v5S8xX9kWbZPtahbAggJIJP4I4EtOcQAH3+F/OZBknCfQb4mwfBzSPmJxNk/u 3+QA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=zCpviF+GWpW2q6LnTDjAWSvloYoZWztn7l/3EVg4xkQ=; b=Y6t+VE+jzRs5QHLEZJgf2Wl3Gut9JQSxjDVAA7fEnA4vf0me4JEA4dFCLq049kantY Pi5NfbgSp7v3/lljKtzxH6akCQBXTKKe32nfCwQhulMxvI31k6aPXRdquA0+XYPwZ4dU t2UPDUSAdaTF0RRSM6JoOrhMFnMj/usNKeKrW/19+4FgoSLZ24PYZSmEbwEBSeQnnrgt 2pLNuYzBL2JD8YFePyoXLiraspczf3p+2tCTsNEizYOnLBuVcI7wxRQuZkQBD+mhN5zt +++fNzzqllYLhH5Tejq0ANhea1e8D6UNamsAgxTpQ8VOhyrd3/3w1Y66U3EHqRdC60FT ACRQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=CsO5aMnV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e1-20020a63e001000000b00502f344050csi4790543pgh.313.2023.02.26.06.54.43; Sun, 26 Feb 2023 06:54:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=CsO5aMnV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230463AbjBZOy0 (ORCPT + 99 others); Sun, 26 Feb 2023 09:54:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55210 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230222AbjBZOxo (ORCPT ); Sun, 26 Feb 2023 09:53:44 -0500 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2C235278 for ; Sun, 26 Feb 2023 06:49:51 -0800 (PST) Received: by mail-pj1-x1036.google.com with SMTP id y2so3594626pjg.3 for ; Sun, 26 Feb 2023 06:49:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zCpviF+GWpW2q6LnTDjAWSvloYoZWztn7l/3EVg4xkQ=; b=CsO5aMnVYOQmnyjcQDWffm4ibQqMe8oamD5qA31APfwGf7gzHtJWrrI9olR/qCLH3m +Dg9XEOXL03hNqlNAICq5ya5pc+4BAUaxlKS89rulXGKJbAvILD5lY4IpjVkqVKRPLvh 40gqi3qq8AOZ1mGSmaZCdTm8hikTcJ9NRlhGcNGYHgwL8+XrC0lLnzZo9UmwdvnTP54e eUYL2ktptCc5CxNqWcNGg2KrPSG53OlDew6ZxZjtTpBl8B2IbscuG/h4GalyWHIQrMzw 3OBbkZBqaEHnKb1ujMBfyfX5Hxby7rTMu2I2Ehi1U62/G1skp5O5yL+5jSowBJMa0FAX IEvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zCpviF+GWpW2q6LnTDjAWSvloYoZWztn7l/3EVg4xkQ=; b=mrCCfqqu93S3EXxAvrWxSDkhl6NF85GhSsMMj9vhz+0FhMc37WrvMNUVfYN2NZ8+vw o1/K6GG+x+4199CnRuukfCOwvwyMMBkbgGUMfEQzfkUEJCwysq2MyyBufBJxmt509uOM BLQhY/IXOEjqktk10+YAdelrH+xj6ddtCpBqj7sbvMDfQgg3XmeBE5qSgFEmYm+sX1pi fILHmI3gArL9FIGTTDAW7VdnCdIxX6Oj2et0bNaJD9Q0ZKryFfn2tiQgPIeK2GUbjqy/ 5XwK+pr4aIDzFBpzrQE4gFCG/fV6o2y6PAUN0D+XzBOBLqkvCuWEjIJP9A48p2Q0Roag lotA== X-Gm-Message-State: AO0yUKXOPTaAiWl6OIJ5uZW2pqpvG4T2KIsjNgWS/KWiU54DvVvBIeOf vFBsGAdqbc7u62Ny89pfXSlSsg== X-Received: by 2002:a17:902:ab0c:b0:19a:b1ac:45d4 with SMTP id ik12-20020a170902ab0c00b0019ab1ac45d4mr23569774plb.3.1677422913584; Sun, 26 Feb 2023 06:48:33 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:33 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 4/8] mm: vmscan: add shrinker_srcu_generation Date: Sun, 26 Feb 2023 22:46:51 +0800 Message-Id: <20230226144655.79778-5-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758905812005881159?= X-GMAIL-MSGID: =?utf-8?q?1758905812005881159?= From: Kirill Tkhai After we make slab shrink lockless with SRCU, the longest sleep unregister_shrinker() will be a sleep waiting for all do_shrink_slab() calls. To aviod long unbreakable action in the unregister_shrinker(), add shrinker_srcu_generation to restore a check similar to the rwsem_is_contendent() check that we had before. And for memcg slab shrink, we unlock SRCU and continue iterations from the next shrinker id. Signed-off-by: Kirill Tkhai Signed-off-by: Qi Zheng --- mm/vmscan.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 490764f8e085..99e852c0ab9e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -204,6 +204,7 @@ static void set_task_reclaim_state(struct task_struct *task, LIST_HEAD(shrinker_list); DECLARE_RWSEM(shrinker_rwsem); DEFINE_SRCU(shrinker_srcu); +static atomic_t shrinker_srcu_generation = ATOMIC_INIT(0); #ifdef CONFIG_MEMCG static int shrinker_nr_max; @@ -782,6 +783,7 @@ void unregister_shrinker(struct shrinker *shrinker) debugfs_entry = shrinker_debugfs_remove(shrinker); up_write(&shrinker_rwsem); + atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); debugfs_remove_recursive(debugfs_entry); @@ -803,6 +805,7 @@ void synchronize_shrinkers(void) { down_write(&shrinker_rwsem); up_write(&shrinker_rwsem); + atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } EXPORT_SYMBOL(synchronize_shrinkers); @@ -912,18 +915,20 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, { struct shrinker_info *info; unsigned long ret, freed = 0; - int srcu_idx; - int i; + int srcu_idx, generation; + int i = 0; if (!mem_cgroup_online(memcg)) return 0; +again: srcu_idx = srcu_read_lock(&shrinker_srcu); info = shrinker_info_srcu(memcg, nid); if (unlikely(!info)) goto unlock; - for_each_set_bit(i, info->map, info->map_nr_max) { + generation = atomic_read(&shrinker_srcu_generation); + for_each_set_bit_from(i, info->map, info->map_nr_max) { struct shrink_control sc = { .gfp_mask = gfp_mask, .nid = nid, @@ -969,6 +974,11 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, set_shrinker_bit(memcg, nid, i); } freed += ret; + if (atomic_read(&shrinker_srcu_generation) != generation) { + srcu_read_unlock(&shrinker_srcu, srcu_idx); + i++; + goto again; + } } unlock: srcu_read_unlock(&shrinker_srcu, srcu_idx); @@ -1008,7 +1018,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, { unsigned long ret, freed = 0; struct shrinker *shrinker; - int srcu_idx; + int srcu_idx, generation; /* * The root memcg might be allocated even though memcg is disabled @@ -1022,6 +1032,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, srcu_idx = srcu_read_lock(&shrinker_srcu); + generation = atomic_read(&shrinker_srcu_generation); list_for_each_entry_srcu(shrinker, &shrinker_list, list, srcu_read_lock_held(&shrinker_srcu)) { struct shrink_control sc = { @@ -1034,6 +1045,11 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid, if (ret == SHRINK_EMPTY) ret = 0; freed += ret; + + if (atomic_read(&shrinker_srcu_generation) != generation) { + freed = freed ? : 1; + break; + } } srcu_read_unlock(&shrinker_srcu, srcu_idx); From patchwork Sun Feb 26 14:46:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 61629 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1975530wrd; Sun, 26 Feb 2023 07:01:14 -0800 (PST) X-Google-Smtp-Source: AK7set8qV5JxnIkD84bqMUctZYOi18grGj2gFVocHtpAu1xd5bYDW1OlYOZ5xBxPHTlSRasnDNme X-Received: by 2002:a17:906:2cf:b0:878:5fa4:7443 with SMTP id 15-20020a17090602cf00b008785fa47443mr28526080ejk.3.1677423673780; Sun, 26 Feb 2023 07:01:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677423673; cv=none; d=google.com; s=arc-20160816; b=Zjbbj7xWR7n+2O0hdG3I7sO2O+QxR8F7rNEdSIRy0uGcqZmwyEKCD1a7EBzc5ZiILc Wr9ngD8G6gFuVyUGtrjbuRv4hVvX1NX0l1LjhgswoDR54YEZnBs9E+vT4fIl4Die9fuU mlJt90vY6iY34fuEc61O9dhe+2g1SQAhjdgZn+WmdV4avWA7qMnm6TVYNNUvqpCoP+8W yJHthgQOPzNAkFBj1mzicRfTkL+ZqTVKm+Q7tVFH34HOpa3x3wNzufGz13y7mYsTRW7P j7lMmXhOuvnJhHs+Z/+z6pDgGMrSVpVAXZPg7wktVjFQrhi9kaluffyu4Mp4qWpEPN/H x2Ow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=ezHTKFyHKlt4tIfSbiwLjpEeB+RP9TUhDRuQZdjQ+5vB0a0UkT7YpC5A3CEfgty1it zB2cljlre4ydnKElKtQRJ6So6trWAEvWaXuSq5we3kNv8IVIhLMa89ice32cNCHeg/Fb nQSQmsCCbxv15c6Ra1qWd5CLMLSyXX9xtfuUTl2IIcwkg+GsnMTPFhxPOgxGWf/ZQ0jW urIOHxJnVZ4XL53CbzzCynM+dq+XHxtlMA53HDI5qfMuB67XcLqEoPDcRmtqTOlrsJtn 2W46P3X/sug/XtCqgPmBD1F22PK+aoCEMRA+3SUrrzs8QC4XlbfpPuIqcTgKKJ7VWHC0 yP2A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=PLf5B69J; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e17-20020a170906045100b008b79a5e743esi5411970eja.253.2023.02.26.07.00.48; Sun, 26 Feb 2023 07:01:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=PLf5B69J; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230514AbjBZO62 (ORCPT + 99 others); Sun, 26 Feb 2023 09:58:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231555AbjBZOzp (ORCPT ); Sun, 26 Feb 2023 09:55:45 -0500 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 38AE114223 for ; Sun, 26 Feb 2023 06:51:05 -0800 (PST) Received: by mail-pl1-f173.google.com with SMTP id p20so2987759plw.13 for ; Sun, 26 Feb 2023 06:51:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=PLf5B69Jh3pz5GXkfJ0kzyAzNE9QmfDh24Eym4+RIsIY+kh7/HFsEQ2eahiZrlAtR+ 4MxgpRyO+JR9bXbQmyfDQ7Xzvmlu8kCxHtggdFCMFTGHU/zrqq+iyrlQZ1x2SIvb3hxE GprjsegftQOiKVGJiPih2MUwmk/ZVPeSzgftTgUUilU4HagO6HlGsAjToabMn30Ye2jJ D1sW5UHlLVaMC0PQVaPxtdHsFbPDqUB7VsgFSalXcyA5wqm6EJXO+viNO6MMi5fGoOK4 u4SJ8zlHcmMYd0X0NrZ/p1mkOI0zz70UPWZaATCxnS4NbaHp2WXdx+JDlArqTdLRnVOn lPsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4ra+jS9qPdJDcLL3tCNVjDgmNt8q5h7NH0ipc1XhyKg=; b=FJApLAFXdKlFvXPpOomb1WAu7un9LMHeCfjsahFcsjzLGYM2Ou/Hca9QDkKQPkPD+7 SaqANBo09j1HGyU1BQeWlrvMuE65kFcklZLJD6KF0uw6t6Vy7iEy77uooypxyIZIb9OT vAkXoCd+h41iwolBToRK8kAuWLCF5HeBg2i/YKNNjGpThyRUT3Sq8kKrnzF8JP0lcvDC yMQ0nho4hCn7oe7tT2W38RFycCJRYKy7lRHzNxZheev9Ep+eZj+LxBipDUTqzmWXeHjp ZRAOGg22LI4lq4axzzK79bNA31Ei9/EcWPGb+eJYAK4tfz8KZH1HUspK9p8BwTkXYKeH fs7A== X-Gm-Message-State: AO0yUKWMb6pWVbtvSHKMaHWNSNLPKq7SUBw0CJ/ctNk9TRsBXJO+REFo zfi1wG7tkA3JPZoBALHPBUfN/g== X-Received: by 2002:a17:903:2291:b0:19a:8202:2dad with SMTP id b17-20020a170903229100b0019a82022dadmr23250501plh.2.1677422919354; Sun, 26 Feb 2023 06:48:39 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:39 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 5/8] mm: shrinkers: make count and scan in shrinker debugfs lockless Date: Sun, 26 Feb 2023 22:46:52 +0800 Message-Id: <20230226144655.79778-6-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758906206107588414?= X-GMAIL-MSGID: =?utf-8?q?1758906206107588414?= Like global and memcg slab shrink, also use SRCU to make count and scan operations in memory shrinker debugfs lockless. Signed-off-by: Qi Zheng --- mm/shrinker_debug.c | 24 +++++++----------------- 1 file changed, 7 insertions(+), 17 deletions(-) diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 39c3491e28a3..6aa7a7ec69da 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -9,6 +9,7 @@ /* defined in vmscan.c */ extern struct rw_semaphore shrinker_rwsem; extern struct list_head shrinker_list; +extern struct srcu_struct shrinker_srcu; static DEFINE_IDA(shrinker_debugfs_ida); static struct dentry *shrinker_debugfs_root; @@ -49,18 +50,13 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) struct mem_cgroup *memcg; unsigned long total; bool memcg_aware; - int ret, nid; + int ret = 0, nid, srcu_idx; count_per_node = kcalloc(nr_node_ids, sizeof(unsigned long), GFP_KERNEL); if (!count_per_node) return -ENOMEM; - ret = down_read_killable(&shrinker_rwsem); - if (ret) { - kfree(count_per_node); - return ret; - } - rcu_read_lock(); + srcu_idx = srcu_read_lock(&shrinker_srcu); memcg_aware = shrinker->flags & SHRINKER_MEMCG_AWARE; @@ -91,8 +87,7 @@ static int shrinker_debugfs_count_show(struct seq_file *m, void *v) } } while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)) != NULL); - rcu_read_unlock(); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); kfree(count_per_node); return ret; @@ -115,9 +110,8 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, .gfp_mask = GFP_KERNEL, }; struct mem_cgroup *memcg = NULL; - int nid; + int nid, srcu_idx; char kbuf[72]; - ssize_t ret; read_len = size < (sizeof(kbuf) - 1) ? size : (sizeof(kbuf) - 1); if (copy_from_user(kbuf, buf, read_len)) @@ -146,11 +140,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, return -EINVAL; } - ret = down_read_killable(&shrinker_rwsem); - if (ret) { - mem_cgroup_put(memcg); - return ret; - } + srcu_idx = srcu_read_lock(&shrinker_srcu); sc.nid = nid; sc.memcg = memcg; @@ -159,7 +149,7 @@ static ssize_t shrinker_debugfs_scan_write(struct file *file, shrinker->scan_objects(shrinker, &sc); - up_read(&shrinker_rwsem); + srcu_read_unlock(&shrinker_srcu, srcu_idx); mem_cgroup_put(memcg); return size; From patchwork Sun Feb 26 14:46:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 61627 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1974163wrd; Sun, 26 Feb 2023 06:57:48 -0800 (PST) X-Google-Smtp-Source: AK7set9nb3snXzY1KMymk9vhdp/mdNDn8WCS7vKyRa+RaojlelMW8li5ZIa+I9R4h/a1A+x6YmAl X-Received: by 2002:a05:6402:274c:b0:4af:81f1:5b02 with SMTP id z12-20020a056402274c00b004af81f15b02mr9290485edd.2.1677423468599; Sun, 26 Feb 2023 06:57:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677423468; cv=none; d=google.com; s=arc-20160816; b=h/BlU6U5fMzSkG40JMvnJWRXJJdv4q6ZEgghDDzqmyEQyH/qfe2l+XeMh7l9NVV/m0 EyPmKbJUhVk1Y7kA0ksnNT5Ijtv1FV5hglwrZmrW1ZV0gmMts+ZgKRsze8XnNrIT9i2S 0tdSoGm+KWMDTP9HUOo+48fQAdU+6AM8hkYyGezc/N5gtmtEqrMlV/ZZgAfR7fbS9fUU bwgUi+YOSuJNJttvxekHX/4Q5NbpKidoKRSca93FhrR8R+ivYdtrc6nrOFIEjKazb9h5 1eTDN8B54d2EDNRtlHVUq/0fkY43jDRVJmF+kwKkPUOzgTsflreHXj3zRYXcHLe7tAOY 8lbg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=c6cuoJEo92PScDqgnKjc0MgPFLiMUUPUvBDJj7SgxHo=; b=owbiAwS6TA2k0cf63vAzxy5iq9CZM+i9n5AlOOpxKiMk+VXDho7B0mnVhjZ6UGjlDN nDAsJGO1me4cK9ZegXDgIgwhewDrxfXufQL5rodXTbUcgub1SDRv/avjYP34M9wWR+EL Sepw6fWilU88nRX9s6l6642axD/cw2CbXPonbljZUfQtL9q+nfu5bEIGScD2lTR32WPo uVPJalHBu/+BI8U7v5cDWtJkRNsCCRn/kPcJgiVvd1D0SaqVYQKxODLP0j8W995g2d86 cksXJbvB7fAjiYepf8jTKYZ79iUyiKTGviGoz4RHdcK4w7FBnCmqkmK1AMC4r1shhLxg 5GFg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=lIKiq8Er; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s16-20020a056402165000b004aad23a85a6si5454359edx.81.2023.02.26.06.57.26; Sun, 26 Feb 2023 06:57:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=lIKiq8Er; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230481AbjBZO4q (ORCPT + 99 others); Sun, 26 Feb 2023 09:56:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54712 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230151AbjBZOyE (ORCPT ); Sun, 26 Feb 2023 09:54:04 -0500 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 292599763 for ; Sun, 26 Feb 2023 06:50:04 -0800 (PST) Received: by mail-pl1-x62f.google.com with SMTP id n6so2822311plf.5 for ; Sun, 26 Feb 2023 06:50:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1677422925; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=c6cuoJEo92PScDqgnKjc0MgPFLiMUUPUvBDJj7SgxHo=; b=lIKiq8ErtDRmfz+i/ano83moR805w7pQEVFBs5DWz8iogOWtMgCKlWMtTjjG4CUYGf qZUEa/ayLAYcSmEePRMBNnD4UMS+HxFoUzmlroe0ZVHuqZ3z2bv2HGbh7Yq0wKCvHk4k QE8yyj88IWGiNyUTVZhSPdvEoYldgr4C6pMLd3y42cOtg65TmytTMhFeDnjLzWUSuqD4 9aNiX3gjRzbrv4MYxQIZGaPfntfaShxVFiNdV3/73U+gNlaqjtHmoW98ZiAC2kGLB21U 7G4hnPw+a24oAWIJ7kfT7KLIQYmuet0PscCXlnmSZhQ2rIyHh9ik/cyvUbqhLd+kEnv8 Uwjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677422925; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=c6cuoJEo92PScDqgnKjc0MgPFLiMUUPUvBDJj7SgxHo=; b=NnSkB4euJawKue+gYwn7w1aVgBcrOOIx/1LFZJAunt6QHlVTAT0NtzZk0MFa7TeSS1 3rpDbrWfSSpMLjz9LrxAoST4Yi7jUqBdYKJFuRIc0evIMxX1yfcxSun60ChsPNlH+2bH HYWyfCgbHTPZDaoO4HEK3n2sh8vX4usCIUyoB2YeJG/anMyAHo85h1ITLR+Q9tLrGyBv Ta79JOyTic+v61QNC3lpILuELVYQYQmEJ2xuSpJvJHSTD96O2BTs8KI11sTGxT9kJbxz mvUFAl+GefHyWwDWNUu1k+5yzrYqbnNkApy4c8+IRMWZhtz+Civ+d1QH6GiKG8qUufpt psNw== X-Gm-Message-State: AO0yUKV3yP3Sxn1qxSxtx/cm6t09wfVUG3NfhZTUypGer4ylaePsl1KX Q95oTmmGkKbVeLt9FQhH+m7XwQ== X-Received: by 2002:a17:902:bb0f:b0:197:8e8e:f15 with SMTP id im15-20020a170902bb0f00b001978e8e0f15mr23385229plb.6.1677422925086; Sun, 26 Feb 2023 06:48:45 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:44 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 6/8] mm: vmscan: hold write lock to reparent shrinker nr_deferred Date: Sun, 26 Feb 2023 22:46:53 +0800 Message-Id: <20230226144655.79778-7-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758905991331839865?= X-GMAIL-MSGID: =?utf-8?q?1758905991331839865?= For now, reparent_shrinker_deferred() is the only holder of read lock of shrinker_rwsem. And it already holds the global cgroup_mutex, so it will not be called in parallel. Therefore, in order to convert shrinker_rwsem to shrinker_mutex later, here we change to hold the write lock of shrinker_rwsem to reparent. Signed-off-by: Qi Zheng --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 99e852c0ab9e..16ff64813175 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -451,7 +451,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_read(&shrinker_rwsem); + down_write(&shrinker_rwsem); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -460,7 +460,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_read(&shrinker_rwsem); + up_write(&shrinker_rwsem); } static bool cgroup_reclaim(struct scan_control *sc) From patchwork Sun Feb 26 14:46:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 61626 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1973687wrd; Sun, 26 Feb 2023 06:56:09 -0800 (PST) X-Google-Smtp-Source: AK7set/uxBYPETPK9iUx5DMd2K6KZxa0UC7DDhWJ1gEeCCj2BNVZxB0Nu1PstvIVxD2uJ7MQ2sBq X-Received: by 2002:a17:903:2291:b0:19a:8202:2dad with SMTP id b17-20020a170903229100b0019a82022dadmr23266378plh.2.1677423369296; Sun, 26 Feb 2023 06:56:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677423369; cv=none; d=google.com; s=arc-20160816; b=gu9VZ5bKC4lOGH/nxv/UMiE0zrjg+AZQ9aFughBkox0h8eQqD8kILXGnrO3U4gm+HT UmDtfRHjndwggFgQzveYJng9AO0P8KQZtRrFJJttu4b94L9aayFH6aDvfi5lv2i/D6Ms Eav61Xb5HAdEVaNJerL0/zZeNdxTIqaYDHOZPFMdcBhhRT0tIAuQNZV61S7Po4gr9USO VzN0pV34AFH4bD1bcXdwcbx1kpYA3hRQHZ36yene1oy+J1oWh3BjubAt3xbvrzowzOPo zi+zE77emDrM348MsoTArvxihkHuJOn3oCa/cdJQOqpwTXczzTuItbHRjS010OoV9ac/ Oo0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=rPbK2mA8SftO30bweBpSZUhXCxYptD2QUCquMlUkzZQ=; b=AO81B5SaGLa5e+Fft82QJp3/zmNifOcXIl58dqHj4LiGy3dlonGeZjYrGz+7b4gc/Z NkSUGwTZgzE0U45+uMqhzvSsTwlgtQrdoUPecpZKEDzKA1QW8Wa8ofsK4sK3C9ZgBUN4 JVeQPW6XmVrvfjgNka0iMSL1utU36jU4dJbDTYFqYoZbd9oPcbQl21bIri2VX/jZMLVQ P2aJmxDVOgTmxk+qXJpPOBVR9pPBL7NydU7oC5Nbq60sDb/3lAex75pnGifNawvWf4h+ TJA3G5zl2hH0M+CvUmOKJf2NAFHMDrn2Pl7merbKFya43rwe+fgAGQWyIFvfpb5Ft2bp CX7Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=RLNAX82e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ko15-20020a17090307cf00b0019cfc412deesi2889966plb.465.2023.02.26.06.55.57; Sun, 26 Feb 2023 06:56:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=RLNAX82e; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230477AbjBZOye (ORCPT + 99 others); Sun, 26 Feb 2023 09:54:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56440 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230323AbjBZOxt (ORCPT ); Sun, 26 Feb 2023 09:53:49 -0500 Received: from mail-pl1-x62a.google.com (mail-pl1-x62a.google.com [IPv6:2607:f8b0:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E58AA19BA for ; Sun, 26 Feb 2023 06:50:05 -0800 (PST) Received: by mail-pl1-x62a.google.com with SMTP id i10so4190450plr.9 for ; Sun, 26 Feb 2023 06:50:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rPbK2mA8SftO30bweBpSZUhXCxYptD2QUCquMlUkzZQ=; b=RLNAX82ej6WiOp+gJxxrYUevqQBzdDGYpvRp2WltPYHzsFagaq47ozL06y+Hy6OgmK JK9iGaHN7dJvbFey7hQXB7aig8jRb8Fzh2ihtgS4KPi3JJgVeer/QP403MbnYAJfXhak bYN0OJJ/krh591ZGhC6ZmW4Bo9d/0ELhp3VjgtJ7eyFa0hScQMuwE81nRfrktkjFE6Os OjbYs65uBBhgwi4YlDp3YuXDzBBPVqJDlkDh/2welq1ly/HLAfshGinj1f1s9zkkezz/ 2xhknARP+taXDeqDm4xAjtZ8aw0C7dGDxIicSpRhb2gQC9so4gEZ84pp+H7E0riZtKLc Tpsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rPbK2mA8SftO30bweBpSZUhXCxYptD2QUCquMlUkzZQ=; b=lTqDzm1efkpTNt4k/tKU05+Of29BHCQCgBFmQ5Wdvz7aV0BfbDeD/EPr2i5xRmx9V/ Siqc42IS5V1pdzJuae3TQ0MaDeJDZatYBCRrWIo0aEtBZcuL7XZtgwyvgMKhiiPiWVxc IGljWhfcBHQNsJ638ojfT8d0qEbdx+134jJD0fFPsXXlX2gTLA/iOWnVVNrJi2jhJEfg xMKB4OIrWNfx70QcCbj2CuX3DbE7tgyzeE9A/DFBTcLX8yuEM0XIny0EX6v41jL2LT4a 9ZKzD2SvIRdO+2KfxsT4Xe04L8fywOhggSkzM/aUYxxShLEhUzaQqWWx3qQZzJGtoSU2 LeRg== X-Gm-Message-State: AO0yUKUtq9TLwP/GkJDiNfkS87JRAzn9T49lmsXQLdDGRDzLDq0K/5Zf CUvbSW7UAJB7mBU64B9UXMHY+Q== X-Received: by 2002:a17:902:d4c8:b0:19a:7217:32af with SMTP id o8-20020a170902d4c800b0019a721732afmr23249925plg.5.1677422930708; Sun, 26 Feb 2023 06:48:50 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:50 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 7/8] mm: vmscan: remove shrinker_rwsem from synchronize_shrinkers() Date: Sun, 26 Feb 2023 22:46:54 +0800 Message-Id: <20230226144655.79778-8-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758905886915283305?= X-GMAIL-MSGID: =?utf-8?q?1758905886915283305?= Now there are no readers of shrinker_rwsem, so synchronize_shrinkers() does not need to hold the writer of shrinker_rwsem to wait for all running shinkers to complete, synchronize_srcu() is enough. Signed-off-by: Qi Zheng --- mm/vmscan.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 16ff64813175..2d71fd565c78 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -796,15 +796,11 @@ EXPORT_SYMBOL(unregister_shrinker); /** * synchronize_shrinkers - Wait for all running shrinkers to complete. * - * This is equivalent to calling unregister_shrink() and register_shrinker(), - * but atomically and with less overhead. This is useful to guarantee that all - * shrinker invocations have seen an update, before freeing memory, similar to - * rcu. + * This is useful to guarantee that all shrinker invocations have seen an + * update, before freeing memory. */ void synchronize_shrinkers(void) { - down_write(&shrinker_rwsem); - up_write(&shrinker_rwsem); atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu); } From patchwork Sun Feb 26 14:46:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 61628 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1974311wrd; Sun, 26 Feb 2023 06:58:18 -0800 (PST) X-Google-Smtp-Source: AK7set+cDytB7Apo8hJ0fAmk+tdbG7eZ0j8TN6OWKWXhCBp5QOEXWF1zCsaFDJT036kC+3T1cHh3 X-Received: by 2002:a17:906:d4:b0:8f5:14ab:94bc with SMTP id 20-20020a17090600d400b008f514ab94bcmr8493815eji.6.1677423497966; Sun, 26 Feb 2023 06:58:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677423497; cv=none; d=google.com; s=arc-20160816; b=lqeWRMnK71euWV61n1LDmKTFZYRT0lvAmWMMi/Gz9sVrXT+OJNzC7zht5SA4UR7VkR mZFD6m99wzFmsQCiPvfBORBjc0YHLZJ+FN4zxpYfNKF09RwXt5Y/U7Dgf2VHojMgqT7V 7Pv91PS0cbp/26S6yRCDAeb8CJ3WUn/1tSITkFLC8C2E09Ohi4DWR1vJE8wAlsrfCHxc 3GcqZr5WyTodTlzmjRi3dgW7rd9rQZHom7EBRccWnY0FJM0SzKx3NmOKbCDg7lSWTTte ZptuX0tNGun1W8shon0/IMDhsBc4PXutTI2jlEFKS1CPYf4tz9Jdi5HtYqrPkF8kQOVh AT8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=f5xm65H9BLTSDcpU/nYsAyAok8lJrBUsJgeNL2FDrMI=; b=egyj8Og4V+TJPivs694sBDpm5DpL22/I5+c1Z9iaZeZ4g5ahTKB2PS3WX4dkJE5OR7 R2xfQP1BEmgyBJ1MSTXbLjGwN5BTJ9f2c20GXIb+8VPo6KLbvNL2EJCnW/P5ReP/Tux/ KX3ehC2o6JRUFxacJzACfeYlWsIDowWndhHtE83bcDM0G8QYnxmt+B3XiBlQNQZlY0RM R1xIpJILGvjTMalH5UhFrEatwNTpMWbOe2ctZxGMzY2OspLfiS7qrsx53S5Fn6JqBKog ClDaNA5fPIbW7y9X3WPD9kcC5b+cq0kK3aDChXPp8YlKOneYd4hF1SdxL25ZXBV6bhch PgaQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=DU9ImCfz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v17-20020a170906339100b008c5b7ddf5fcsi4508399eja.346.2023.02.26.06.57.53; Sun, 26 Feb 2023 06:58:17 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=DU9ImCfz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230523AbjBZO5K (ORCPT + 99 others); Sun, 26 Feb 2023 09:57:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55222 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230321AbjBZOyT (ORCPT ); Sun, 26 Feb 2023 09:54:19 -0500 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D61CF1ADFD for ; Sun, 26 Feb 2023 06:50:14 -0800 (PST) Received: by mail-pj1-x1036.google.com with SMTP id k21-20020a17090aaa1500b002376652e160so3811503pjq.0 for ; Sun, 26 Feb 2023 06:50:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=f5xm65H9BLTSDcpU/nYsAyAok8lJrBUsJgeNL2FDrMI=; b=DU9ImCfzPDt6r5qsdIJgFlMqPyv1XNzrFdxX1MYevySp7rV/vBy4oHtHlXM0ehu+/q e7Sjxf8qsqyQGdUSKu80ZuyH8gv9rpDyqCA+6sYlagLU/RVK4FI8pl7xZiXV+Ojb41zx ezPpwq3uJvNV5TVTXIF7WKeiEmB772qpgZU/rJEebve9b47zOunZTUMe/f+gk7PISSjc Ehcf7cS8nj1sKVkD/JveUqUJzsUCxQfCPh/rzpzR2amvQmZsujUa/Epz4Eonu2OG9utb VbC/USw9zVaaeY/TFpc3MuKTAHI4ssQxadvh9v+878R2uidsPviikdk/mxseIFk0IHSC +N3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=f5xm65H9BLTSDcpU/nYsAyAok8lJrBUsJgeNL2FDrMI=; b=ymEbfvJJflIhHcxxQE6BoqxhGeoqCLhJXYgVuxlV97/IVd1UihoKpx/CWKHzOIzRCr JvpzA2OjjJVrVtxK9YU+bpd/nic4OYuP/0GzSUKDxfaIJWnkecyzsf39L+TjifeNfatH LjPAJhpvO1eQYKTyZ+tAc0O0gshce6cSOcwdNO2/dQlqhsL8qRCuSUJiJ7RVq+X2zxHG Mfue5gjC4q7eS6EdhUHZIpRw4dyJdc3Ipy8w92qqVqYBKC6f/sqAnFbZONUmg8ukf5XO uOUbBLruTtMl01XueY4e+dYrxWwoRhf/OX6zBX4cr/X54DXXnNNo+whM3kz0pCCcinMo XoBg== X-Gm-Message-State: AO0yUKVy+ht/6CI68ZBL7QtyC1hYcKXWhRbxLDk3PHmHEMi2jRQtyjXh EETxISgFJfYxYZw1sMYhRZYPdg== X-Received: by 2002:a17:902:d4c8:b0:19a:7217:32af with SMTP id o8-20020a170902d4c800b0019a721732afmr23250125plg.5.1677422936561; Sun, 26 Feb 2023 06:48:56 -0800 (PST) Received: from localhost.localdomain ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id y20-20020a170902ed5400b0019c2cf12d15sm2755589plb.116.2023.02.26.06.48.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 26 Feb 2023 06:48:56 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, tkhai@ya.ru, hannes@cmpxchg.org, shakeelb@google.com, mhocko@kernel.org, roman.gushchin@linux.dev, muchun.song@linux.dev, david@redhat.com, shy828301@gmail.com Cc: sultan@kerneltoast.com, dave@stgolabs.net, penguin-kernel@I-love.SAKURA.ne.jp, paulmck@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v3 8/8] mm: shrinkers: convert shrinker_rwsem to mutex Date: Sun, 26 Feb 2023 22:46:55 +0800 Message-Id: <20230226144655.79778-9-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20230226144655.79778-1-zhengqi.arch@bytedance.com> References: <20230226144655.79778-1-zhengqi.arch@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758906021998857886?= X-GMAIL-MSGID: =?utf-8?q?1758906021998857886?= Now there are no readers of shrinker_rwsem, so we can simply replace it with mutex lock. Signed-off-by: Qi Zheng --- drivers/md/dm-cache-metadata.c | 2 +- drivers/md/dm-thin-metadata.c | 2 +- fs/super.c | 2 +- mm/shrinker_debug.c | 14 +++++++------- mm/vmscan.c | 34 +++++++++++++++++----------------- 5 files changed, 27 insertions(+), 27 deletions(-) diff --git a/drivers/md/dm-cache-metadata.c b/drivers/md/dm-cache-metadata.c index acffed750e3e..9e0c69958587 100644 --- a/drivers/md/dm-cache-metadata.c +++ b/drivers/md/dm-cache-metadata.c @@ -1828,7 +1828,7 @@ int dm_cache_metadata_abort(struct dm_cache_metadata *cmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * cmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs cmd root_lock). - * - must take shrinker_rwsem without holding cmd->root_lock + * - must take shrinker_mutex without holding cmd->root_lock */ new_bm = dm_block_manager_create(cmd->bdev, DM_CACHE_METADATA_BLOCK_SIZE << SECTOR_SHIFT, CACHE_MAX_CONCURRENT_LOCKS); diff --git a/drivers/md/dm-thin-metadata.c b/drivers/md/dm-thin-metadata.c index fd464fb024c3..9f5cb52c5763 100644 --- a/drivers/md/dm-thin-metadata.c +++ b/drivers/md/dm-thin-metadata.c @@ -1887,7 +1887,7 @@ int dm_pool_abort_metadata(struct dm_pool_metadata *pmd) * Replacement block manager (new_bm) is created and old_bm destroyed outside of * pmd root_lock to avoid ABBA deadlock that would result (due to life-cycle of * shrinker associated with the block manager's bufio client vs pmd root_lock). - * - must take shrinker_rwsem without holding pmd->root_lock + * - must take shrinker_mutex without holding pmd->root_lock */ new_bm = dm_block_manager_create(pmd->bdev, THIN_METADATA_BLOCK_SIZE << SECTOR_SHIFT, THIN_MAX_CONCURRENT_LOCKS); diff --git a/fs/super.c b/fs/super.c index 84332d5cb817..91a4037b1d95 100644 --- a/fs/super.c +++ b/fs/super.c @@ -54,7 +54,7 @@ static char *sb_writers_name[SB_FREEZE_LEVELS] = { * One thing we have to be careful of with a per-sb shrinker is that we don't * drop the last active reference to the superblock from within the shrinker. * If that happens we could trigger unregistering the shrinker from within the - * shrinker path and that leads to deadlock on the shrinker_rwsem. Hence we + * shrinker path and that leads to deadlock on the shrinker_mutex. Hence we * take a passive reference to the superblock to avoid this from occurring. */ static unsigned long super_cache_scan(struct shrinker *shrink, diff --git a/mm/shrinker_debug.c b/mm/shrinker_debug.c index 6aa7a7ec69da..b0f6aff372df 100644 --- a/mm/shrinker_debug.c +++ b/mm/shrinker_debug.c @@ -7,7 +7,7 @@ #include /* defined in vmscan.c */ -extern struct rw_semaphore shrinker_rwsem; +extern struct mutex shrinker_mutex; extern struct list_head shrinker_list; extern struct srcu_struct shrinker_srcu; @@ -167,7 +167,7 @@ int shrinker_debugfs_add(struct shrinker *shrinker) char buf[128]; int id; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); /* debugfs isn't initialized yet, add debugfs entries later. */ if (!shrinker_debugfs_root) @@ -210,7 +210,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) if (!new) return -ENOMEM; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); old = shrinker->name; shrinker->name = new; @@ -228,7 +228,7 @@ int shrinker_debugfs_rename(struct shrinker *shrinker, const char *fmt, ...) shrinker->debugfs_entry = entry; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); kfree_const(old); @@ -240,7 +240,7 @@ struct dentry *shrinker_debugfs_remove(struct shrinker *shrinker) { struct dentry *entry = shrinker->debugfs_entry; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); kfree_const(shrinker->name); shrinker->name = NULL; @@ -265,14 +265,14 @@ static int __init shrinker_debugfs_init(void) shrinker_debugfs_root = dentry; /* Create debugfs entries for shrinkers registered at boot */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_for_each_entry(shrinker, &shrinker_list, list) if (!shrinker->debugfs_entry) { ret = shrinker_debugfs_add(shrinker); if (ret) break; } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } diff --git a/mm/vmscan.c b/mm/vmscan.c index 2d71fd565c78..6c5d21ba0c9a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -35,7 +35,7 @@ #include #include #include -#include +#include #include #include #include @@ -202,7 +202,7 @@ static void set_task_reclaim_state(struct task_struct *task, } LIST_HEAD(shrinker_list); -DECLARE_RWSEM(shrinker_rwsem); +DEFINE_MUTEX(shrinker_mutex); DEFINE_SRCU(shrinker_srcu); static atomic_t shrinker_srcu_generation = ATOMIC_INIT(0); @@ -225,7 +225,7 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, { return srcu_dereference_check(memcg->nodeinfo[nid]->shrinker_info, &shrinker_srcu, - lockdep_is_held(&shrinker_rwsem)); + lockdep_is_held(&shrinker_mutex)); } static struct shrinker_info *shrinker_info_srcu(struct mem_cgroup *memcg, @@ -310,7 +310,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) int nid, size, ret = 0; int map_size, defer_size = 0; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); map_size = shrinker_map_size(shrinker_nr_max); defer_size = shrinker_defer_size(shrinker_nr_max); size = map_size + defer_size; @@ -326,7 +326,7 @@ int alloc_shrinker_info(struct mem_cgroup *memcg) info->map_nr_max = shrinker_nr_max; rcu_assign_pointer(memcg->nodeinfo[nid]->shrinker_info, info); } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -342,7 +342,7 @@ static int expand_shrinker_info(int new_id) if (!root_mem_cgroup) goto out; - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); map_size = shrinker_map_size(new_nr_max); defer_size = shrinker_defer_size(new_nr_max); @@ -392,7 +392,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) if (mem_cgroup_disabled()) return -ENOSYS; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); id = idr_alloc(&shrinker_idr, shrinker, 0, 0, GFP_KERNEL); if (id < 0) goto unlock; @@ -406,7 +406,7 @@ static int prealloc_memcg_shrinker(struct shrinker *shrinker) shrinker->id = id; ret = 0; unlock: - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return ret; } @@ -416,7 +416,7 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) BUG_ON(id < 0); - lockdep_assert_held(&shrinker_rwsem); + lockdep_assert_held(&shrinker_mutex); idr_remove(&shrinker_idr, id); } @@ -451,7 +451,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) parent = root_mem_cgroup; /* Prevent from concurrent shrinker_info expand */ - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); for_each_node(nid) { child_info = shrinker_info_protected(memcg, nid); parent_info = shrinker_info_protected(parent, nid); @@ -460,7 +460,7 @@ void reparent_shrinker_deferred(struct mem_cgroup *memcg) atomic_long_add(nr, &parent_info->nr_deferred[i]); } } - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } static bool cgroup_reclaim(struct scan_control *sc) @@ -709,9 +709,9 @@ void free_prealloced_shrinker(struct shrinker *shrinker) shrinker->name = NULL; #endif if (shrinker->flags & SHRINKER_MEMCG_AWARE) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); unregister_memcg_shrinker(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); return; } @@ -721,11 +721,11 @@ void free_prealloced_shrinker(struct shrinker *shrinker) void register_shrinker_prepared(struct shrinker *shrinker) { - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_add_tail_rcu(&shrinker->list, &shrinker_list); shrinker->flags |= SHRINKER_REGISTERED; shrinker_debugfs_add(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); } static int __register_shrinker(struct shrinker *shrinker) @@ -775,13 +775,13 @@ void unregister_shrinker(struct shrinker *shrinker) if (!(shrinker->flags & SHRINKER_REGISTERED)) return; - down_write(&shrinker_rwsem); + mutex_lock(&shrinker_mutex); list_del_rcu(&shrinker->list); shrinker->flags &= ~SHRINKER_REGISTERED; if (shrinker->flags & SHRINKER_MEMCG_AWARE) unregister_memcg_shrinker(shrinker); debugfs_entry = shrinker_debugfs_remove(shrinker); - up_write(&shrinker_rwsem); + mutex_unlock(&shrinker_mutex); atomic_inc(&shrinker_srcu_generation); synchronize_srcu(&shrinker_srcu);