From patchwork Thu Nov 2 03:23:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160770 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp101759vqu; Wed, 1 Nov 2023 20:26:14 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF9crgtKRjbfsE8OyFjrgF5jWpDk9mqXeARA/64z8pI63+oa/3kkqPXZlNnWzX4XVmxNaK4 X-Received: by 2002:a17:902:fa85:b0:1cc:4467:a563 with SMTP id lc5-20020a170902fa8500b001cc4467a563mr9946521plb.3.1698895574443; Wed, 01 Nov 2023 20:26:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698895574; cv=none; d=google.com; s=arc-20160816; b=fCsUq5XQFptesBOR7234SqJ6dJXsVTWKmYMRQrNpnLSxjp/BvUsSP2UhWJRD/Wrq6r ZfWtgr7xsdYu6NUUMhLBIDDjRS8gEAA9ob65YfAXsYHqqYi6SQepl/bs+Frqm9VxAdUH 7wgukgTU32PXUEKuCzVcc7iXtRP9Fljyao93O4hRllj803S8IXPbT+3ucfJxcFjQ8sou wuDJPVBn49zPhochYr9T5YhnGYR1SZ7RLAGnvJvVqHQh+025iQNnrohV7x7WFbQBekAn GafIGJOF9GCqYI0OuxDDt76LC4gztjrlm5nuot0pV0KW/hcFhPjF2c/b389yFRnY1lmp Bmjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=SxN7Gc7oPUWEeX+rwHiBiXl514BfxOyW9F4Ey0p4ljE=; fh=Dd5ryTbz4gXtDU9jzXqg2hD87XusQgPqunF08GDmhnA=; b=y6ZtRZ8vtWz174RdTjqoaaZFz5NdCfbb+6BWApgHgdLlWAPPm0qk7SS6lqbfaC8igI oEXzlcjiC8CKh03inaznysialmFkbBUuANBZY7bK7PBkOjNgHQNxD0UtzxrsWaqTpH5i bqV9gZrT8uF/3CEqfYx0MNPrlohjT62DwbZB12GlVu2dHNU9Oe1Puyfj31lbKhDxEW9d w4RtUk2yMtzVpOcKUdqUYTkVBNB64/den2A6Ck2/5zs2YgNPgD9NJBj03i8ojWw8uNRu IEHnb9zsKqS8GlOML6LiUxquospUizeacZY/0GyxD4XzOSYEN+t0w7qSheVFHRCM8MFi B9ew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=KQjfhe1X; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id w18-20020a170902e89200b001c72699dcd3si4608668plg.367.2023.11.01.20.26.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 20:26:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=KQjfhe1X; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 0BFD3807DED8; Wed, 1 Nov 2023 20:25:12 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348344AbjKBDYs (ORCPT + 35 others); Wed, 1 Nov 2023 23:24:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47620 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348329AbjKBDYr (ORCPT ); Wed, 1 Nov 2023 23:24:47 -0400 Received: from out-170.mta1.migadu.com (out-170.mta1.migadu.com [IPv6:2001:41d0:203:375::aa]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96595110 for ; Wed, 1 Nov 2023 20:24:41 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698895480; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SxN7Gc7oPUWEeX+rwHiBiXl514BfxOyW9F4Ey0p4ljE=; b=KQjfhe1X0vzh80JaLfhIpxkVvacWOXmJrdGiki+nTBnVHkJnDiiAVRsK2kELO7fDNxvQI6 G6ybzeZ5YjP1xah95OqvCvgzwmI2JTnc6KOhO8JBYI5WOuwCfe9ULPpNMpW0LdOZ/2n96O //Wvc+JEmR8lCOYJBPeBl/5I81NJdYw= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [PATCH v5 1/9] slub: Reflow ___slab_alloc() Date: Thu, 2 Nov 2023 03:23:22 +0000 Message-Id: <20231102032330.1036151-2-chengming.zhou@linux.dev> In-Reply-To: <20231102032330.1036151-1-chengming.zhou@linux.dev> References: <20231102032330.1036151-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Wed, 01 Nov 2023 20:25:12 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781421125892042065 X-GMAIL-MSGID: 1781421125892042065 From: Chengming Zhou The get_partial() interface used in ___slab_alloc() may return a single object in the "kmem_cache_debug(s)" case, in which we will just return the "freelist" object. Move this handling up to prepare for later changes. And the "pfmemalloc_match()" part is not needed for node partial slab, since we already check this in the get_partial_node(). Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 63d281dfacdb..0b0fdc8c189f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3216,8 +3216,21 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, pc.slab = &slab; pc.orig_size = orig_size; freelist = get_partial(s, node, &pc); - if (freelist) - goto check_new_slab; + if (freelist) { + if (kmem_cache_debug(s)) { + /* + * For debug caches here we had to go through + * alloc_single_from_partial() so just store the + * tracking info and return the object. + */ + if (s->flags & SLAB_STORE_USER) + set_track(s, freelist, TRACK_ALLOC, addr); + + return freelist; + } + + goto retry_load_slab; + } slub_put_cpu_ptr(s->cpu_slab); slab = new_slab(s, gfpflags, node); @@ -3253,20 +3266,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, inc_slabs_node(s, slab_nid(slab), slab->objects); -check_new_slab: - - if (kmem_cache_debug(s)) { - /* - * For debug caches here we had to go through - * alloc_single_from_partial() so just store the tracking info - * and return the object - */ - if (s->flags & SLAB_STORE_USER) - set_track(s, freelist, TRACK_ALLOC, addr); - - return freelist; - } - if (unlikely(!pfmemalloc_match(slab, gfpflags))) { /* * For !pfmemalloc_match() case we don't load freelist so that From patchwork Thu Nov 2 03:23:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160768 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp101382vqu; Wed, 1 Nov 2023 20:24:59 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHd/dERNdNF/5O893O2E9FwUeTaoafWA9WSOi9/VdMP65Knn9sV3QnIxSQWIc5V9BFIG+41 X-Received: by 2002:a81:9ac9:0:b0:5a7:c973:c82 with SMTP id r192-20020a819ac9000000b005a7c9730c82mr15597000ywg.13.1698895498913; Wed, 01 Nov 2023 20:24:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698895498; cv=none; d=google.com; s=arc-20160816; b=ffppzP4DVlJnr6Fzq1Lrh+lDuasx4evlonFOewWo7l/YvTS5UOk/sgaWZI8m7/G2i5 YdviELRgzw8Lhcgz0raaxEjK9brcLarf6+dffIq5F3M41CVT6cuSDSdJvrqlJ0rbTMme 1lk9ARuoeGiNX/0F9dwxoaCd3rg71goUX2FXcT5a6NesTAqDDAXDZGGVBhbMg7prj/3x LaQ7SxsqDZ2YvrjhGRZlk54Ebn3xgAxN0SE9mMWaEnq85WCB45MLACSwJPn27Z4JgSOz rM1pr3z0k4Tw9s6UoffQxoE+wL29+6NVwPCRXIDdC9GkVrXjYDerAvJ2HgbXFPJhpDoL 1Jsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=rXE5qC66xflKju/b5K0r0xT61UeednKHJ/BZyj3zT9w=; fh=Dd5ryTbz4gXtDU9jzXqg2hD87XusQgPqunF08GDmhnA=; b=QHM8zkLMbFgE8Kz/e5AiCuSx5usepf9yl6y2JThcJG6LxzQN4pRhrICeupoZU3vr/7 JVIkokYQC2ty3mbEji7xfL1wmTaZxPkXOSI0tt0e1hWMwCEbpFXMlHXXMFR8pN9QwkW+ 8C2xkl5YnVxlSnHIM2zqaBzcTUMBC0wnT7gTTxQkWe+COUUKIF95hjnM0D8eY0kE+OVm K6i+N+uaW13EyvfnUtnCgJDkS6YJ9/oJmw7Y0GcYOUvhO6yW4thYii2WU6ZRL19HPqZF zdnnZXKqSx0nux13F2Du1tcQGWSgyx2SZCDgsJNw7RApQCLESm43WPzD3MOr0vyFAUB8 Rlfg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="Hb4c5/5c"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id y8-20020a655b48000000b005b891b74b31si932911pgr.625.2023.11.01.20.24.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 20:24:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="Hb4c5/5c"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id A525781B93DE; Wed, 1 Nov 2023 20:24:57 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348374AbjKBDYx (ORCPT + 35 others); Wed, 1 Nov 2023 23:24:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47632 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348330AbjKBDYv (ORCPT ); Wed, 1 Nov 2023 23:24:51 -0400 Received: from out-188.mta1.migadu.com (out-188.mta1.migadu.com [95.215.58.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0DD6A113 for ; Wed, 1 Nov 2023 20:24:44 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698895483; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rXE5qC66xflKju/b5K0r0xT61UeednKHJ/BZyj3zT9w=; b=Hb4c5/5cMu557sfyb4Qm+N+E2afW9N5tC2dmGC7W2tqshdmHyIQi0ignzc96Sfv3lH/6v4 /pMsyHBXFmwbe/kt7UshRgo+l+u5TurmWX3zHZI3IJ9XGdtBS8AY4ZNbtEcl52hm/S2deO rdYAz7rU1LmdQ25nUsrxBo6s5PDHjlI= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [PATCH v5 2/9] slub: Change get_partial() interfaces to return slab Date: Thu, 2 Nov 2023 03:23:23 +0000 Message-Id: <20231102032330.1036151-3-chengming.zhou@linux.dev> In-Reply-To: <20231102032330.1036151-1-chengming.zhou@linux.dev> References: <20231102032330.1036151-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 01 Nov 2023 20:24:57 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781421046972545837 X-GMAIL-MSGID: 1781421046972545837 From: Chengming Zhou We need all get_partial() related interfaces to return a slab, instead of returning the freelist (or object). Use the partial_context.object to return back freelist or object for now. This patch shouldn't have any functional changes. Suggested-by: Vlastimil Babka Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 63 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 33 insertions(+), 30 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 0b0fdc8c189f..03384cd965c5 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -204,9 +204,9 @@ DEFINE_STATIC_KEY_FALSE(slub_debug_enabled); /* Structure holding parameters for get_partial() call chain */ struct partial_context { - struct slab **slab; gfp_t flags; unsigned int orig_size; + void *object; }; static inline bool kmem_cache_debug(struct kmem_cache *s) @@ -2269,10 +2269,11 @@ static inline bool pfmemalloc_match(struct slab *slab, gfp_t gfpflags); /* * Try to allocate a partial slab from a specific node. */ -static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, - struct partial_context *pc) +static struct slab *get_partial_node(struct kmem_cache *s, + struct kmem_cache_node *n, + struct partial_context *pc) { - struct slab *slab, *slab2; + struct slab *slab, *slab2, *partial = NULL; void *object = NULL; unsigned long flags; unsigned int partial_slabs = 0; @@ -2288,27 +2289,28 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, spin_lock_irqsave(&n->list_lock, flags); list_for_each_entry_safe(slab, slab2, &n->partial, slab_list) { - void *t; - if (!pfmemalloc_match(slab, pc->flags)) continue; if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) { object = alloc_single_from_partial(s, n, slab, pc->orig_size); - if (object) + if (object) { + partial = slab; + pc->object = object; break; + } continue; } - t = acquire_slab(s, n, slab, object == NULL); - if (!t) + object = acquire_slab(s, n, slab, object == NULL); + if (!object) break; - if (!object) { - *pc->slab = slab; + if (!partial) { + partial = slab; + pc->object = object; stat(s, ALLOC_FROM_PARTIAL); - object = t; } else { put_cpu_partial(s, slab, 0); stat(s, CPU_PARTIAL_NODE); @@ -2324,20 +2326,21 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, } spin_unlock_irqrestore(&n->list_lock, flags); - return object; + return partial; } /* * Get a slab from somewhere. Search in increasing NUMA distances. */ -static void *get_any_partial(struct kmem_cache *s, struct partial_context *pc) +static struct slab *get_any_partial(struct kmem_cache *s, + struct partial_context *pc) { #ifdef CONFIG_NUMA struct zonelist *zonelist; struct zoneref *z; struct zone *zone; enum zone_type highest_zoneidx = gfp_zone(pc->flags); - void *object; + struct slab *slab; unsigned int cpuset_mems_cookie; /* @@ -2372,8 +2375,8 @@ static void *get_any_partial(struct kmem_cache *s, struct partial_context *pc) if (n && cpuset_zone_allowed(zone, pc->flags) && n->nr_partial > s->min_partial) { - object = get_partial_node(s, n, pc); - if (object) { + slab = get_partial_node(s, n, pc); + if (slab) { /* * Don't check read_mems_allowed_retry() * here - if mems_allowed was updated in @@ -2381,7 +2384,7 @@ static void *get_any_partial(struct kmem_cache *s, struct partial_context *pc) * between allocation and the cpuset * update */ - return object; + return slab; } } } @@ -2393,17 +2396,18 @@ static void *get_any_partial(struct kmem_cache *s, struct partial_context *pc) /* * Get a partial slab, lock it and return it. */ -static void *get_partial(struct kmem_cache *s, int node, struct partial_context *pc) +static struct slab *get_partial(struct kmem_cache *s, int node, + struct partial_context *pc) { - void *object; + struct slab *slab; int searchnode = node; if (node == NUMA_NO_NODE) searchnode = numa_mem_id(); - object = get_partial_node(s, get_node(s, searchnode), pc); - if (object || node != NUMA_NO_NODE) - return object; + slab = get_partial_node(s, get_node(s, searchnode), pc); + if (slab || node != NUMA_NO_NODE) + return slab; return get_any_partial(s, pc); } @@ -3213,10 +3217,10 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, new_objects: pc.flags = gfpflags; - pc.slab = &slab; pc.orig_size = orig_size; - freelist = get_partial(s, node, &pc); - if (freelist) { + slab = get_partial(s, node, &pc); + if (slab) { + freelist = pc.object; if (kmem_cache_debug(s)) { /* * For debug caches here we had to go through @@ -3408,12 +3412,11 @@ static void *__slab_alloc_node(struct kmem_cache *s, void *object; pc.flags = gfpflags; - pc.slab = &slab; pc.orig_size = orig_size; - object = get_partial(s, node, &pc); + slab = get_partial(s, node, &pc); - if (object) - return object; + if (slab) + return pc.object; slab = new_slab(s, gfpflags, node); if (unlikely(!slab)) { From patchwork Thu Nov 2 03:23:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160769 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp101391vqu; Wed, 1 Nov 2023 20:25:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFFfohttLuSDRqETqC+K8o0yIuhUd/2oeWFpK2NRyFmTzNO4g2nNpM6xrDFEEaTuY7l8NiU X-Received: by 2002:a05:6a21:99a4:b0:181:9b30:89d6 with SMTP id ve36-20020a056a2199a400b001819b3089d6mr889175pzb.27.1698895502478; Wed, 01 Nov 2023 20:25:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698895502; cv=none; d=google.com; s=arc-20160816; b=rs4/uIJVWBlb0YeeNx8INDfiRW+47QlUEHUP9gAcn23WUfZz903J06H+1FZWt8p+nq 9MGMV/ar7H8/T3aL0kTkA/QkNkPacr1s9fM/zQT5aaDZe5q8LsPZOXFgiktPOiEjOl1h GaBxBsnt85iy4CZd+MJCGHIJG+LI/qEJ6SQJ3auwRPS63aC5GhckUh+vMcj3Wp3oEFmF /f0rbyAV7cotAWqG8GoN6vfNIEhKYupWCXDaSnsqcEY0zRTzNTJGdDL+alTmxeltcFrT 48fExifNigNn/atcS84H46tgrKXx2S1zaNajkWotk2eVM3qsah7NbS3T5fIU36PD7buo VCOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=IPS3iWC0ordY9Z/gFrasLVfoOU0ee6NMzPFkJtmjEdA=; fh=mNx4emnblabKsSvYnWt3rJfr5klL7EHzY3/GxImb0OY=; b=L0Ub74VHRuNi5XhcHBSuA0Qnts5OKvGRpS+RPzyIw59Q/acRjhizSHFG31/0SxObGf nbdMXJ4a/dLCzIMQ/NkUvMIkOyY9IS31+N8nblwJjM6HF5Hp08uOA7v9SwTA+99wUTqS ZbfVmGarCysGxbr1sdGaRED6HOPnDmdW5OrcliXCyJim7NbgecHZIzpWyvvo2zarsCQ3 7SOXgNH96pCCSyGMAhRII0ALOMweS52BPSkmiecqYuOYYYk/jBjeGITqnbX+IAVP3dEi 1vzlDaSbtyuZEv2/B1sHPs8Nsw2M/VPQq+sYTLla4WUFEIRpkzbQyXGb8Fy7OnljyB7t yTgQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="v1GKoBc/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id l16-20020a170903245000b001c88fc3c593si4471986pls.560.2023.11.01.20.25.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 20:25:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="v1GKoBc/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id BD5EB81B93DC; Wed, 1 Nov 2023 20:25:00 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348375AbjKBDY6 (ORCPT + 35 others); Wed, 1 Nov 2023 23:24:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47648 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348366AbjKBDYw (ORCPT ); Wed, 1 Nov 2023 23:24:52 -0400 Received: from out-179.mta1.migadu.com (out-179.mta1.migadu.com [95.215.58.179]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71B6D120 for ; Wed, 1 Nov 2023 20:24:49 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698895487; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IPS3iWC0ordY9Z/gFrasLVfoOU0ee6NMzPFkJtmjEdA=; b=v1GKoBc/GWi/6wA+3MMyQ6L15tftj7e2VTfNMKRmvlXo85TA0YUcUSnVVJ8Kga11asCboO btJ3b5nOE+cya1I08GWukxT+lICauUTNrau2aealf4F9CdmRScWkmlPsoHeV6UcabxAXZx LYjHaUg928T27sxO5LCHMigYm2gu+Xw= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou , Matthew Wilcox Subject: [PATCH v5 3/9] slub: Keep track of whether slub is on the per-node partial list Date: Thu, 2 Nov 2023 03:23:24 +0000 Message-Id: <20231102032330.1036151-4-chengming.zhou@linux.dev> In-Reply-To: <20231102032330.1036151-1-chengming.zhou@linux.dev> References: <20231102032330.1036151-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 01 Nov 2023 20:25:00 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781421050466029106 X-GMAIL-MSGID: 1781421050466029106 From: Chengming Zhou Now we rely on the "frozen" bit to see if we should manipulate the slab->slab_list, which will be changed in the following patch. Instead we introduce another way to keep track of whether slub is on the per-node partial list, here we reuse the PG_workingset bit. We use __set_bit and __clear_bit directly instead of the atomic version for better performance and it's safe since it's protected by the slub node list_lock. Suggested-by: Matthew Wilcox Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/mm/slub.c b/mm/slub.c index 03384cd965c5..eed8ae0dbaf9 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2116,6 +2116,25 @@ static void discard_slab(struct kmem_cache *s, struct slab *slab) free_slab(s, slab); } +/* + * SLUB reuses PG_workingset bit to keep track of whether it's on + * the per-node partial list. + */ +static inline bool slab_test_node_partial(const struct slab *slab) +{ + return folio_test_workingset((struct folio *)slab_folio(slab)); +} + +static inline void slab_set_node_partial(struct slab *slab) +{ + __set_bit(PG_workingset, folio_flags(slab_folio(slab), 0)); +} + +static inline void slab_clear_node_partial(struct slab *slab) +{ + __clear_bit(PG_workingset, folio_flags(slab_folio(slab), 0)); +} + /* * Management of partially allocated slabs. */ @@ -2127,6 +2146,7 @@ __add_partial(struct kmem_cache_node *n, struct slab *slab, int tail) list_add_tail(&slab->slab_list, &n->partial); else list_add(&slab->slab_list, &n->partial); + slab_set_node_partial(slab); } static inline void add_partial(struct kmem_cache_node *n, @@ -2141,6 +2161,7 @@ static inline void remove_partial(struct kmem_cache_node *n, { lockdep_assert_held(&n->list_lock); list_del(&slab->slab_list); + slab_clear_node_partial(slab); n->nr_partial--; } @@ -4833,6 +4854,7 @@ static int __kmem_cache_do_shrink(struct kmem_cache *s) if (free == slab->objects) { list_move(&slab->slab_list, &discard); + slab_clear_node_partial(slab); n->nr_partial--; dec_slabs_node(s, node, slab->objects); } else if (free <= SHRINK_PROMOTE_MAX) From patchwork Thu Nov 2 03:23:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160771 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp101764vqu; Wed, 1 Nov 2023 20:26:15 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH3lio75Z8ToVOqOMDT9Oc/WhXT0K0F+eQ9rmIWiR8PgfxjzZm8kywOWMX04agdL+I+aWNy X-Received: by 2002:a05:690c:dc6:b0:5ad:4975:c860 with SMTP id db6-20020a05690c0dc600b005ad4975c860mr22328656ywb.39.1698895575542; Wed, 01 Nov 2023 20:26:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698895575; cv=none; d=google.com; s=arc-20160816; b=IhNee8tBlfKRz28SS8Z4juvlY3X/14RyWxUM8LLd4j2wVqH/XeLTa5+oeEFxjGEKGr pdQD0qgyx464l5m9EY1lZmdyi2XdCML164vK+E7JbQjx6wLjrt8xxvSmG3rvDEoZihaC d5pZnj7VX8gXxkWU5lLbD4d5eNy3oFhUAPgRH+AphaePjTTa4ZFLgW1cKkGTkJUNQnzA ZwfTrZW8gA2TfZAgcAuwG3o/WItYRyygZIA2D2LYU8tDVL4eechWcKZi95hBtaPDnerw RzN7Gy8cfM6ZgU0BIhawQLyDRJ/OyKtFfaGspsws0h8l27hrYCuAFeeUkdqE9uF1mMIe jWoQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=f6Me2hDPNAg0QwflvoQbBjNc8QPP2obW8dtDV2OvsdM=; fh=Dd5ryTbz4gXtDU9jzXqg2hD87XusQgPqunF08GDmhnA=; b=zSxdyBQvKXNXMpy3JnSI/qMfWhqTqt/3FDEhUFuKOuAOQRAGBrv6PucFd9yO8bk5pv qp8iYkv3WOP/UPUYNlZI3xijuVd/B5YAF55JcY4+AAKM8WkZCQFURwiGe7mcm1fZL886 WCfe29PbsCRUjUkBnMkCpkhD3Mb6zUQmnweGcNS6AzXeb1Ppb2exv58unPBCYYDJEgWj 5Sg3Y0CYBat2HaMSyQS0u1gIRA1YVxjYoA/A+5+70jU6pLyWOppIMQTWh6PDI0NWebvu UgLxNTXfNQMK9bvWUUYuFqPGvQOIK8zvvS0Pr0LuUSia90WV18+LtjqsP31D3ihWEW/T Nfxg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=BPwts8cC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id h1-20020a636c01000000b005b9a1494637si1096662pgc.317.2023.11.01.20.26.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 20:26:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=BPwts8cC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 5E34B8062DDB; Wed, 1 Nov 2023 20:25:40 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348396AbjKBDZI (ORCPT + 35 others); Wed, 1 Nov 2023 23:25:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35146 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348366AbjKBDZA (ORCPT ); Wed, 1 Nov 2023 23:25:00 -0400 Received: from out-173.mta1.migadu.com (out-173.mta1.migadu.com [IPv6:2001:41d0:203:375::ad]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9EF8A128 for ; Wed, 1 Nov 2023 20:24:54 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698895493; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f6Me2hDPNAg0QwflvoQbBjNc8QPP2obW8dtDV2OvsdM=; b=BPwts8cCYtipUTdyFWricBp8OGyxnWzOEVZlgGagJ9J46S70yP6mdX0Zikc3F0F2DBJvXT /yMeu2ntPNaXcbiqN3TobVLGvOh9MDaYMJyNpEP8dSexDMJlp27RoUaeceT+iumgN10Z/7 Sn3Ry4EaMJKSfkAaTC3ksH7jqeLcYEc= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [PATCH v5 4/9] slub: Prepare __slab_free() for unfrozen partial slab out of node partial list Date: Thu, 2 Nov 2023 03:23:25 +0000 Message-Id: <20231102032330.1036151-5-chengming.zhou@linux.dev> In-Reply-To: <20231102032330.1036151-1-chengming.zhou@linux.dev> References: <20231102032330.1036151-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Wed, 01 Nov 2023 20:25:40 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781421126918864295 X-GMAIL-MSGID: 1781421126918864295 From: Chengming Zhou Now the partially empty slub will be frozen when taken out of node partial list, so the __slab_free() will know from "was_frozen" that the partially empty slab is not on node partial list and is a cpu or cpu partial slab of some cpu. But we will change this, make partial slabs leave the node partial list with unfrozen state, so we need to change __slab_free() to use the new slab_test_node_partial() we just introduced. Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/mm/slub.c b/mm/slub.c index eed8ae0dbaf9..1880b483350e 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3631,6 +3631,7 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, unsigned long counters; struct kmem_cache_node *n = NULL; unsigned long flags; + bool on_node_partial; stat(s, FREE_SLOWPATH); @@ -3678,6 +3679,7 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, */ spin_lock_irqsave(&n->list_lock, flags); + on_node_partial = slab_test_node_partial(slab); } } @@ -3706,6 +3708,15 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, return; } + /* + * This slab was partially empty but not on the per-node partial list, + * in which case we shouldn't manipulate its list, just return. + */ + if (prior && !on_node_partial) { + spin_unlock_irqrestore(&n->list_lock, flags); + return; + } + if (unlikely(!new.inuse && n->nr_partial >= s->min_partial)) goto slab_empty; From patchwork Thu Nov 2 03:23:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160772 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp101877vqu; Wed, 1 Nov 2023 20:26:38 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFIR+pmCrKuotCEPyOJ8DIz4hpkWA+3xxh+WUmboiwRdZITj3K2XQ32DWVWPwigrg+rLwPM X-Received: by 2002:a05:6830:7181:b0:6b9:a926:4a12 with SMTP id el1-20020a056830718100b006b9a9264a12mr24515819otb.28.1698895598600; Wed, 01 Nov 2023 20:26:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698895598; cv=none; d=google.com; s=arc-20160816; b=UC+NMUR+2K+YikAqZCrx2h/jfmhLfW9I0isCkt6ZKNOvnl0mn8vh/EK1AkqPteqyns SB+ZmGog+yCEQemg5hZYv3smuH3DFaLjmcu30mAHQpcnrtZzP9ycI9fkjNvoTaDJhyAf AzaI+fYt6j+36Y4TKGOY85DHfOmncFt1n48RYI8dx50EsNkZcZk5gdvcKoVHuWDwgl3d h/n0trWjVqYJ2Y6ekDGz0CJlLbV/a/XcD6KPazNLmllXImf+o6mSP1g5jdlOtcgAWxaR ofEsw6yYYCu04Q8Cn8eLR3Ll71tOKFaaBi60bPW6SvtRyIHbH4z1PqpCLg1tZRtVGj7P vpbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Y0gxyVelJQEnoALRuyqBzkeEChILTc95BveKj67/aCg=; fh=Dd5ryTbz4gXtDU9jzXqg2hD87XusQgPqunF08GDmhnA=; b=PtYm1VUHl8vi2LdCN2H998MfdeUAGUmIA0m5RiQkFpk1e7xo2aljhtBRyRqRYbchJk u2oRuAaezudqsC4Vtdx1DAd84rXU6gQZPBGWhUMfwYH2BWxLt+/qa/dmNfI5qgbUl4Gh fS/BHcVIkIYBdRJmsXL7iG0zxi2OzuFsj4LSrEg/WlBnW8GXmIyrbGJyiTPSDHJ23jth 6DlZoGODfPca0/rLxopCj3OOB7CBML9WFeAWEpsfFzvD9wlHwpsapnPF9bA8ffYLbh56 sFRbMx9k3v9SD1gOZiZsjBtqz7paHJrv5gKAjTHfEVN1DZgm+7MXoTpDF0s8TXdwssl9 6YKA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=lFfmSKyh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id k7-20020a056a00168700b0069025078128si2881681pfc.240.2023.11.01.20.26.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 20:26:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=lFfmSKyh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 8A30F80792C3; Wed, 1 Nov 2023 20:25:34 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348401AbjKBDZO (ORCPT + 35 others); Wed, 1 Nov 2023 23:25:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35330 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348415AbjKBDZG (ORCPT ); Wed, 1 Nov 2023 23:25:06 -0400 Received: from out-185.mta1.migadu.com (out-185.mta1.migadu.com [IPv6:2001:41d0:203:375::b9]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 973D913E for ; Wed, 1 Nov 2023 20:25:00 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698895498; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y0gxyVelJQEnoALRuyqBzkeEChILTc95BveKj67/aCg=; b=lFfmSKyhGA3gVCpgOxB6Id7LBL0rMpEGl5gAgGZuLLsAnnTy6no1BEn+SeD5u7+7EuXT02 tiwcrE0IU15hQr17N0/O6wDqFW8ovwBIDEs0JU6KOd0J6sBKfSBYsgMLgyQ2qyobNj/QlF NFtNE/LIi0zdnMrItnwzeHEGbAdss+8= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [PATCH v5 5/9] slub: Introduce freeze_slab() Date: Thu, 2 Nov 2023 03:23:26 +0000 Message-Id: <20231102032330.1036151-6-chengming.zhou@linux.dev> In-Reply-To: <20231102032330.1036151-1-chengming.zhou@linux.dev> References: <20231102032330.1036151-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 01 Nov 2023 20:25:34 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781421151173536240 X-GMAIL-MSGID: 1781421151173536240 From: Chengming Zhou We will have unfrozen slabs out of the node partial list later, so we need a freeze_slab() function to freeze the partial slab and get its freelist. Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/mm/slub.c b/mm/slub.c index 1880b483350e..edf567971679 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3098,6 +3098,33 @@ static inline void *get_freelist(struct kmem_cache *s, struct slab *slab) return freelist; } +/* + * Freeze the partial slab and return the pointer to the freelist. + */ +static inline void *freeze_slab(struct kmem_cache *s, struct slab *slab) +{ + struct slab new; + unsigned long counters; + void *freelist; + + do { + freelist = slab->freelist; + counters = slab->counters; + + new.counters = counters; + VM_BUG_ON(new.frozen); + + new.inuse = slab->objects; + new.frozen = 1; + + } while (!slab_update_freelist(s, slab, + freelist, counters, + NULL, new.counters, + "freeze_slab")); + + return freelist; +} + /* * Slow path. The lockless freelist is empty or we need to perform * debugging duties. From patchwork Thu Nov 2 03:23:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160776 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp102192vqu; Wed, 1 Nov 2023 20:27:37 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEr4jx5x3iIGWQfz+Mw3s6SPbpFutXUTQOLY5RusTSmwhjDf33QImmGRgsfPkLd1ja71KYx X-Received: by 2002:a05:6830:2644:b0:6d3:1369:aed with SMTP id f4-20020a056830264400b006d313690aedmr6859979otu.13.1698895657643; Wed, 01 Nov 2023 20:27:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698895657; cv=none; d=google.com; s=arc-20160816; b=qeTDnBlB7jhN5rT40jX7xyE/pg55wqlaWzwIMmS3QxQgVgJCQqtCLELOIHzmN+aEgF AzrkIIU2N4wEAydEWaOkFPIq7QqQK0i4rZRfXsU2aWrXN3ft7Kv0GcFmOWcPlzxgM6zD eQEBDIFQz7l9rG3yBSaBCE7JfipGhTaSh7P/vawfwR/3KFijOZeUJYumGro788j6Y+6x Gj1gqvRAWscXBngbghGInc9zURhH633ePBtuRAxlf4xQOzJAav8dIxD9yr3KJ3X11hu/ DKsnh3Owt7UaNvjtPEBVrcHFiwVQT+T7T23sCpJh2xzS/qT6SNqH/iKm6CMgzrW37wB0 wDZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eS7MrX3LWikN1DLYb4lOax9/7h+5CRyI/1/wKTB6RQU=; fh=Dd5ryTbz4gXtDU9jzXqg2hD87XusQgPqunF08GDmhnA=; b=kiFZVp7sl8tuOmZtZZIcmq8sz1Gl1gc0Q2Uw2EEgYaVYdw4DF4W8z+FEsAoeC9MDB8 BPOKYISSZsjT3r94OH2UtToUwY5U6sfn5LZ/WeRhbgyrisHIEuNzax65YrzmUbuy/CAQ jYLXuqXRVcjCY7FoWEYWCgbIllg4SRfHapPELyzDGadwezxrlGhcYOofAVkJ795a5PJz lfqqVO1wfzCHioasgaiZpDontYrRTOn9dGrStxY/V9g2iCHldIwnpf5zZOV0Yu2UYUYT ORKOe61v3Ra9WTf0wXytT/UuEmgcFCJp1m04iH7H1k+jtaqmOT1bZv3bG5/3is5EZ92x 4aGw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="d9B/WWGs"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id x27-20020aa7941b000000b00690fe3ec830si2782539pfo.55.2023.11.01.20.27.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 20:27:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="d9B/WWGs"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 4C88880C8DE3; Wed, 1 Nov 2023 20:27:27 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348398AbjKBDZU (ORCPT + 35 others); Wed, 1 Nov 2023 23:25:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348393AbjKBDZN (ORCPT ); Wed, 1 Nov 2023 23:25:13 -0400 Received: from out-172.mta1.migadu.com (out-172.mta1.migadu.com [IPv6:2001:41d0:203:375::ac]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3EF83113 for ; Wed, 1 Nov 2023 20:25:05 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698895503; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eS7MrX3LWikN1DLYb4lOax9/7h+5CRyI/1/wKTB6RQU=; b=d9B/WWGsa8AoP9ussGNC0kPusAYCFPBeAQmfPhKZ7WEK7TjlN5ddZZc39LBarNoAdB8jIQ DDraEREuZQYSs4zN95VOQr3psrEiMT5K6FXATomckUDVHfNmgi+DDJbjmbTyYpRL/M4j0p 9h17G+GlKq2BMQwQZj0ZOUhtnjzLGpI= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [PATCH v5 6/9] slub: Delay freezing of partial slabs Date: Thu, 2 Nov 2023 03:23:27 +0000 Message-Id: <20231102032330.1036151-7-chengming.zhou@linux.dev> In-Reply-To: <20231102032330.1036151-1-chengming.zhou@linux.dev> References: <20231102032330.1036151-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 01 Nov 2023 20:27:27 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781421213387803714 X-GMAIL-MSGID: 1781421213387803714 From: Chengming Zhou Now we will freeze slabs when moving them out of node partial list to cpu partial list, this method needs two cmpxchg_double operations: 1. freeze slab (acquire_slab()) under the node list_lock 2. get_freelist() when pick used in ___slab_alloc() Actually we don't need to freeze when moving slabs out of node partial list, we can delay freezing to when use slab freelist in ___slab_alloc(), so we can save one cmpxchg_double(). And there are other good points: - The moving of slabs between node partial list and cpu partial list becomes simpler, since we don't need to freeze or unfreeze at all. - The node list_lock contention would be less, since we don't need to freeze any slab under the node list_lock. We can achieve this because there is no concurrent path would manipulate the partial slab list except the __slab_free() path, which is now serialized by slab_test_node_partial() under the list_lock. Since the slab returned by get_partial() interfaces is not frozen anymore and no freelist is returned in the partial_context, so we need to use the introduced freeze_slab() to freeze it and get its freelist. Similarly, the slabs on the CPU partial list are not frozen anymore, we need to freeze_slab() on it before use. We can now delete acquire_slab() as it became unused. Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Vlastimil Babka Tested-by: Mark Brown Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 113 +++++++++++------------------------------------------- 1 file changed, 23 insertions(+), 90 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index edf567971679..bcb5b2c4e213 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2234,51 +2234,6 @@ static void *alloc_single_from_new_slab(struct kmem_cache *s, return object; } -/* - * Remove slab from the partial list, freeze it and - * return the pointer to the freelist. - * - * Returns a list of objects or NULL if it fails. - */ -static inline void *acquire_slab(struct kmem_cache *s, - struct kmem_cache_node *n, struct slab *slab, - int mode) -{ - void *freelist; - unsigned long counters; - struct slab new; - - lockdep_assert_held(&n->list_lock); - - /* - * Zap the freelist and set the frozen bit. - * The old freelist is the list of objects for the - * per cpu allocation list. - */ - freelist = slab->freelist; - counters = slab->counters; - new.counters = counters; - if (mode) { - new.inuse = slab->objects; - new.freelist = NULL; - } else { - new.freelist = freelist; - } - - VM_BUG_ON(new.frozen); - new.frozen = 1; - - if (!__slab_update_freelist(s, slab, - freelist, counters, - new.freelist, new.counters, - "acquire_slab")) - return NULL; - - remove_partial(n, slab); - WARN_ON(!freelist); - return freelist; -} - #ifdef CONFIG_SLUB_CPU_PARTIAL static void put_cpu_partial(struct kmem_cache *s, struct slab *slab, int drain); #else @@ -2295,7 +2250,6 @@ static struct slab *get_partial_node(struct kmem_cache *s, struct partial_context *pc) { struct slab *slab, *slab2, *partial = NULL; - void *object = NULL; unsigned long flags; unsigned int partial_slabs = 0; @@ -2314,7 +2268,7 @@ static struct slab *get_partial_node(struct kmem_cache *s, continue; if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) { - object = alloc_single_from_partial(s, n, slab, + void *object = alloc_single_from_partial(s, n, slab, pc->orig_size); if (object) { partial = slab; @@ -2324,13 +2278,10 @@ static struct slab *get_partial_node(struct kmem_cache *s, continue; } - object = acquire_slab(s, n, slab, object == NULL); - if (!object) - break; + remove_partial(n, slab); if (!partial) { partial = slab; - pc->object = object; stat(s, ALLOC_FROM_PARTIAL); } else { put_cpu_partial(s, slab, 0); @@ -2629,9 +2580,6 @@ static void __unfreeze_partials(struct kmem_cache *s, struct slab *partial_slab) unsigned long flags = 0; while (partial_slab) { - struct slab new; - struct slab old; - slab = partial_slab; partial_slab = slab->next; @@ -2644,23 +2592,7 @@ static void __unfreeze_partials(struct kmem_cache *s, struct slab *partial_slab) spin_lock_irqsave(&n->list_lock, flags); } - do { - - old.freelist = slab->freelist; - old.counters = slab->counters; - VM_BUG_ON(!old.frozen); - - new.counters = old.counters; - new.freelist = old.freelist; - - new.frozen = 0; - - } while (!__slab_update_freelist(s, slab, - old.freelist, old.counters, - new.freelist, new.counters, - "unfreezing slab")); - - if (unlikely(!new.inuse && n->nr_partial >= s->min_partial)) { + if (unlikely(!slab->inuse && n->nr_partial >= s->min_partial)) { slab->next = slab_to_discard; slab_to_discard = slab; } else { @@ -3167,7 +3099,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, node = NUMA_NO_NODE; goto new_slab; } -redo: if (unlikely(!node_match(slab, node))) { /* @@ -3243,7 +3174,8 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, new_slab: - if (slub_percpu_partial(c)) { +#ifdef CONFIG_SLUB_CPU_PARTIAL + while (slub_percpu_partial(c)) { local_lock_irqsave(&s->cpu_slab->lock, flags); if (unlikely(c->slab)) { local_unlock_irqrestore(&s->cpu_slab->lock, flags); @@ -3255,12 +3187,22 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, goto new_objects; } - slab = c->slab = slub_percpu_partial(c); + slab = slub_percpu_partial(c); slub_set_percpu_partial(c, slab); local_unlock_irqrestore(&s->cpu_slab->lock, flags); stat(s, CPU_PARTIAL_ALLOC); - goto redo; + + if (unlikely(!node_match(slab, node) || + !pfmemalloc_match(slab, gfpflags))) { + slab->next = NULL; + __unfreeze_partials(s, slab); + continue; + } + + freelist = freeze_slab(s, slab); + goto retry_load_slab; } +#endif new_objects: @@ -3268,8 +3210,8 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, pc.orig_size = orig_size; slab = get_partial(s, node, &pc); if (slab) { - freelist = pc.object; if (kmem_cache_debug(s)) { + freelist = pc.object; /* * For debug caches here we had to go through * alloc_single_from_partial() so just store the @@ -3281,6 +3223,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, return freelist; } + freelist = freeze_slab(s, slab); goto retry_load_slab; } @@ -3682,18 +3625,8 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, was_frozen = new.frozen; new.inuse -= cnt; if ((!new.inuse || !prior) && !was_frozen) { - - if (kmem_cache_has_cpu_partial(s) && !prior) { - - /* - * Slab was on no list before and will be - * partially empty - * We can defer the list move and instead - * freeze it. - */ - new.frozen = 1; - - } else { /* Needs to be taken off a list */ + /* Needs to be taken off a list */ + if (!kmem_cache_has_cpu_partial(s) || prior) { n = get_node(s, slab_nid(slab)); /* @@ -3723,9 +3656,9 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, * activity can be necessary. */ stat(s, FREE_FROZEN); - } else if (new.frozen) { + } else if (kmem_cache_has_cpu_partial(s) && !prior) { /* - * If we just froze the slab then put it onto the + * If we started with a full slab then put it onto the * per cpu partial list. */ put_cpu_partial(s, slab, 1); From patchwork Thu Nov 2 03:23:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160775 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp102184vqu; Wed, 1 Nov 2023 20:27:36 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF0ouByK3uaYvZ9ixNQqDwLgAaqfdZ9SFie1k46YTiMN10kuQDFYf3aN4vK+ZivdjTAw6Ji X-Received: by 2002:a05:6830:3d0e:b0:6c0:7bab:3457 with SMTP id eu14-20020a0568303d0e00b006c07bab3457mr16561083otb.20.1698895656542; Wed, 01 Nov 2023 20:27:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698895656; cv=none; d=google.com; s=arc-20160816; b=vSRawYDpO1WbSsbioOzhW0bmne/Q2m5b/NDRWOK5AC7B4sFIndA1asyEDnQ1DsLxPZ RXs0r8jWEn1+O2qlL3wEO7PH/H7kZ8LNJ+SEp2T3/vTovHGti3sQHSvYMnlYFYwsOUPe 4DZ7WqP9U5LqdJbKUgv1G746kyWjBcJ5HhvV8exi3rWgHLR6if671bbribPImiDSCvWW KSRsjRJRlVARAIXRtoF8E6yZMuVHXl5PmILm80Lwu7VqSvEzXvTwwhQlyvVh3pDBSEvW sDChXCbZpBmfpONlbJFD33uT53WRDYTcamch48jA6KszT5SJA3u/IptI2/trCUbsm3xo UTFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=bbgfkr34Ut5bOat3PkSebFKZ4A5Cv7UHaj4yNDSPhuU=; fh=Dd5ryTbz4gXtDU9jzXqg2hD87XusQgPqunF08GDmhnA=; b=nQfjh2wnUIeQTVzFBvSSfCCMxzyNUr+xaYeDPTc/wmn2TsfXpi/0xXEQSZPijBO70A v+/PRxzIddoOItLZjmmvT0TvI1Er/MFJG6+xEBFZ4VNVYOS8PmShdyqY13lW7r2Svr8V FDTgPbyzo5Qt4CwTN53Gabe5Gf8+e0SBpPOADcO6JGHUqoTnKfZ0fP22AdGvTD4k443L ENN96+i1ypRntG5cx8h8FZ4khb3pwGDbE8NkPMN6b4YtUMHl1Y2ffPkMGrGep+laaaEt QBXbnZ/Qb1nqh24k9twTm9KDGm065/BimpkwnstfgBRgw0OvS76jHJ0tymF0nrPtlYhB xI0Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=gHbx+Mfk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id v129-20020a626187000000b006901387b09bsi2676216pfb.344.2023.11.01.20.27.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 20:27:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=gHbx+Mfk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 5FBF580314EF; Wed, 1 Nov 2023 20:27:25 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348390AbjKBDZY (ORCPT + 35 others); Wed, 1 Nov 2023 23:25:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55802 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348397AbjKBDZR (ORCPT ); Wed, 1 Nov 2023 23:25:17 -0400 Received: from out-186.mta1.migadu.com (out-186.mta1.migadu.com [IPv6:2001:41d0:203:375::ba]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 89BB0127 for ; Wed, 1 Nov 2023 20:25:11 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698895509; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bbgfkr34Ut5bOat3PkSebFKZ4A5Cv7UHaj4yNDSPhuU=; b=gHbx+MfkKMaXjWLoeHR6R2/oAAyFc9gfvo3ohtd4K4S4h0A6KE0hMrTwtFkBS0N7jtWhrk xt6ezAgfua8IzYH47q+ur2/jbneBGFE275j69jA9HgOkeQ4CEUTLkH6FIThRIPBcrygeuj fPAujlzG9ChYjmdwj+8gHQRxUvZnQVo= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [PATCH v5 7/9] slub: Optimize deactivate_slab() Date: Thu, 2 Nov 2023 03:23:28 +0000 Message-Id: <20231102032330.1036151-8-chengming.zhou@linux.dev> In-Reply-To: <20231102032330.1036151-1-chengming.zhou@linux.dev> References: <20231102032330.1036151-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 01 Nov 2023 20:27:25 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781421211959672411 X-GMAIL-MSGID: 1781421211959672411 From: Chengming Zhou Since the introduce of unfrozen slabs on cpu partial list, we don't need to synchronize the slab frozen state under the node list_lock. The caller of deactivate_slab() and the caller of __slab_free() won't manipulate the slab list concurrently. So we can get node list_lock in the last stage if we really need to manipulate the slab list in this path. Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 79 ++++++++++++++++++------------------------------------- 1 file changed, 26 insertions(+), 53 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index bcb5b2c4e213..d137468fe4b9 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2468,10 +2468,8 @@ static void init_kmem_cache_cpus(struct kmem_cache *s) static void deactivate_slab(struct kmem_cache *s, struct slab *slab, void *freelist) { - enum slab_modes { M_NONE, M_PARTIAL, M_FREE, M_FULL_NOLIST }; struct kmem_cache_node *n = get_node(s, slab_nid(slab)); int free_delta = 0; - enum slab_modes mode = M_NONE; void *nextfree, *freelist_iter, *freelist_tail; int tail = DEACTIVATE_TO_HEAD; unsigned long flags = 0; @@ -2509,65 +2507,40 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, /* * Stage two: Unfreeze the slab while splicing the per-cpu * freelist to the head of slab's freelist. - * - * Ensure that the slab is unfrozen while the list presence - * reflects the actual number of objects during unfreeze. - * - * We first perform cmpxchg holding lock and insert to list - * when it succeed. If there is mismatch then the slab is not - * unfrozen and number of objects in the slab may have changed. - * Then release lock and retry cmpxchg again. */ -redo: - - old.freelist = READ_ONCE(slab->freelist); - old.counters = READ_ONCE(slab->counters); - VM_BUG_ON(!old.frozen); - - /* Determine target state of the slab */ - new.counters = old.counters; - if (freelist_tail) { - new.inuse -= free_delta; - set_freepointer(s, freelist_tail, old.freelist); - new.freelist = freelist; - } else - new.freelist = old.freelist; - - new.frozen = 0; + do { + old.freelist = READ_ONCE(slab->freelist); + old.counters = READ_ONCE(slab->counters); + VM_BUG_ON(!old.frozen); + + /* Determine target state of the slab */ + new.counters = old.counters; + new.frozen = 0; + if (freelist_tail) { + new.inuse -= free_delta; + set_freepointer(s, freelist_tail, old.freelist); + new.freelist = freelist; + } else { + new.freelist = old.freelist; + } + } while (!slab_update_freelist(s, slab, + old.freelist, old.counters, + new.freelist, new.counters, + "unfreezing slab")); + /* + * Stage three: Manipulate the slab list based on the updated state. + */ if (!new.inuse && n->nr_partial >= s->min_partial) { - mode = M_FREE; + stat(s, DEACTIVATE_EMPTY); + discard_slab(s, slab); + stat(s, FREE_SLAB); } else if (new.freelist) { - mode = M_PARTIAL; - /* - * Taking the spinlock removes the possibility that - * acquire_slab() will see a slab that is frozen - */ spin_lock_irqsave(&n->list_lock, flags); - } else { - mode = M_FULL_NOLIST; - } - - - if (!slab_update_freelist(s, slab, - old.freelist, old.counters, - new.freelist, new.counters, - "unfreezing slab")) { - if (mode == M_PARTIAL) - spin_unlock_irqrestore(&n->list_lock, flags); - goto redo; - } - - - if (mode == M_PARTIAL) { add_partial(n, slab, tail); spin_unlock_irqrestore(&n->list_lock, flags); stat(s, tail); - } else if (mode == M_FREE) { - stat(s, DEACTIVATE_EMPTY); - discard_slab(s, slab); - stat(s, FREE_SLAB); - } else if (mode == M_FULL_NOLIST) { + } else { stat(s, DEACTIVATE_FULL); } } From patchwork Thu Nov 2 03:23:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160773 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp101886vqu; Wed, 1 Nov 2023 20:26:41 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEACZ/nHtBXwdeaGynJAe2A9P0bEyAn+IbQ+dSsgV+7AagSFcTRns7XYh12+/6w6NU/hvUb X-Received: by 2002:a05:6a21:35c7:b0:181:3af0:97e8 with SMTP id ba7-20020a056a2135c700b001813af097e8mr4522307pzc.55.1698895600872; Wed, 01 Nov 2023 20:26:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698895600; cv=none; d=google.com; s=arc-20160816; b=ojU8Bt2YaRjeCLIBYE+nFozIB+rpKbEiVq6Vhwt7mrU/QF4Hu9j5Huy1zLPoJ0aU69 pz8+Bm6lSTxOQhZcrKVDKn7FWaoFAT/gZcKqTCJVJXsZA88IwE7wjqrogldymdVbYO7O 1q3me6faoU136CzUjRcQ1uzvm4lRvAUvo3Evr+EEqxg+nayAJO+vyiMe+Hu0oYWoGI2w sjyVgZRaDLrIOtfIt+bkXcA8TJ0wD6kJe7GYuXpTO0HU2FhVztGOb8npK1QW7s+n7cgd GZ7/17i4fQTn7UT4fdXJtWVqagj9tRA4UeAo/UWy12S2Wc/ZmzUOH8srk44hYgwPANej ZyVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=N3Xfo0y6jZ0YfVFtmWrT6BPsa9Nsm+KQGum2QWoNV0I=; fh=Dd5ryTbz4gXtDU9jzXqg2hD87XusQgPqunF08GDmhnA=; b=DJgM7i3u257WZ+6mr6cTDLJekbx3b5lc6MKNESIy3ZlsbNnJ1v0rPVPjSf5Q15Asnj wN8ybWD71OPvmMcnBoWJEs7EJWEI5mAhzf9P3uRREpTG7d66KyLdzYPhlScz2CeD1Zd2 Iq9/DSsrBWLzl/2uTq2ntSzrSwQaruuGfSb+lZc9e70ctZfvo8TZcw5u1dqrA5QPysWB Z/bJ/6ZOXbYno9mpkZTvwAxulcPE3EtPB0XJqIrx0oZ3WvqGH+aO5I/ze+SXlsQiTabU qZ/jNxCQCjk7Y6U1igLTD4MhYd6CcTMAmaYEuI/iCxLktXbm4evLctQ7aR7gqy0Bz3o8 YucQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=k4QQibDC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id 68-20020a17090a09ca00b0026b7f68f386si2002953pjo.73.2023.11.01.20.26.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 20:26:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=k4QQibDC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 7CB8880A2649; Wed, 1 Nov 2023 20:25:56 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348426AbjKBDZm (ORCPT + 35 others); Wed, 1 Nov 2023 23:25:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58276 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348420AbjKBDZf (ORCPT ); Wed, 1 Nov 2023 23:25:35 -0400 Received: from out-179.mta1.migadu.com (out-179.mta1.migadu.com [IPv6:2001:41d0:203:375::b3]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 251D2186 for ; Wed, 1 Nov 2023 20:25:24 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698895522; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=N3Xfo0y6jZ0YfVFtmWrT6BPsa9Nsm+KQGum2QWoNV0I=; b=k4QQibDC1ld0UrW3iBDGGZ48phYAHJ1+HACNNhCO/C89ykyWOPzBIhsNfTvaTJprJCvNXn xOQ7UO16oxS2c3AfRJJiFnt6rOq3FciJQ8ilKoxXDPfHcQOyfEz6OK1DOpVwj61rke/jDb 3Vlw5deJI3Z6PFrtYghaLkTN4KlSeUA= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [PATCH v5 8/9] slub: Rename all *unfreeze_partials* functions to *put_partials* Date: Thu, 2 Nov 2023 03:23:29 +0000 Message-Id: <20231102032330.1036151-9-chengming.zhou@linux.dev> In-Reply-To: <20231102032330.1036151-1-chengming.zhou@linux.dev> References: <20231102032330.1036151-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 01 Nov 2023 20:25:56 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781421153660998078 X-GMAIL-MSGID: 1781421153660998078 From: Chengming Zhou Since all partial slabs on the CPU partial list are not frozen anymore, we don't unfreeze when moving cpu partial slabs to node partial list, it's better to rename these functions. Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index d137468fe4b9..c20bdf5dab0f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2546,7 +2546,7 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, } #ifdef CONFIG_SLUB_CPU_PARTIAL -static void __unfreeze_partials(struct kmem_cache *s, struct slab *partial_slab) +static void __put_partials(struct kmem_cache *s, struct slab *partial_slab) { struct kmem_cache_node *n = NULL, *n2 = NULL; struct slab *slab, *slab_to_discard = NULL; @@ -2588,9 +2588,9 @@ static void __unfreeze_partials(struct kmem_cache *s, struct slab *partial_slab) } /* - * Unfreeze all the cpu partial slabs. + * Put all the cpu partial slabs to the node partial list. */ -static void unfreeze_partials(struct kmem_cache *s) +static void put_partials(struct kmem_cache *s) { struct slab *partial_slab; unsigned long flags; @@ -2601,11 +2601,11 @@ static void unfreeze_partials(struct kmem_cache *s) local_unlock_irqrestore(&s->cpu_slab->lock, flags); if (partial_slab) - __unfreeze_partials(s, partial_slab); + __put_partials(s, partial_slab); } -static void unfreeze_partials_cpu(struct kmem_cache *s, - struct kmem_cache_cpu *c) +static void put_partials_cpu(struct kmem_cache *s, + struct kmem_cache_cpu *c) { struct slab *partial_slab; @@ -2613,7 +2613,7 @@ static void unfreeze_partials_cpu(struct kmem_cache *s, c->partial = NULL; if (partial_slab) - __unfreeze_partials(s, partial_slab); + __put_partials(s, partial_slab); } /* @@ -2626,7 +2626,7 @@ static void unfreeze_partials_cpu(struct kmem_cache *s, static void put_cpu_partial(struct kmem_cache *s, struct slab *slab, int drain) { struct slab *oldslab; - struct slab *slab_to_unfreeze = NULL; + struct slab *slab_to_put = NULL; unsigned long flags; int slabs = 0; @@ -2641,7 +2641,7 @@ static void put_cpu_partial(struct kmem_cache *s, struct slab *slab, int drain) * per node partial list. Postpone the actual unfreezing * outside of the critical section. */ - slab_to_unfreeze = oldslab; + slab_to_put = oldslab; oldslab = NULL; } else { slabs = oldslab->slabs; @@ -2657,17 +2657,17 @@ static void put_cpu_partial(struct kmem_cache *s, struct slab *slab, int drain) local_unlock_irqrestore(&s->cpu_slab->lock, flags); - if (slab_to_unfreeze) { - __unfreeze_partials(s, slab_to_unfreeze); + if (slab_to_put) { + __put_partials(s, slab_to_put); stat(s, CPU_PARTIAL_DRAIN); } } #else /* CONFIG_SLUB_CPU_PARTIAL */ -static inline void unfreeze_partials(struct kmem_cache *s) { } -static inline void unfreeze_partials_cpu(struct kmem_cache *s, - struct kmem_cache_cpu *c) { } +static inline void put_partials(struct kmem_cache *s) { } +static inline void put_partials_cpu(struct kmem_cache *s, + struct kmem_cache_cpu *c) { } #endif /* CONFIG_SLUB_CPU_PARTIAL */ @@ -2709,7 +2709,7 @@ static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) stat(s, CPUSLAB_FLUSH); } - unfreeze_partials_cpu(s, c); + put_partials_cpu(s, c); } struct slub_flush_work { @@ -2737,7 +2737,7 @@ static void flush_cpu_slab(struct work_struct *w) if (c->slab) flush_slab(s, c); - unfreeze_partials(s); + put_partials(s); } static bool has_cpu_slab(int cpu, struct kmem_cache *s) @@ -3168,7 +3168,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, if (unlikely(!node_match(slab, node) || !pfmemalloc_match(slab, gfpflags))) { slab->next = NULL; - __unfreeze_partials(s, slab); + __put_partials(s, slab); continue; } From patchwork Thu Nov 2 03:23:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160774 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp101896vqu; Wed, 1 Nov 2023 20:26:43 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHxtoumDXPX3nq/Cz5YZ/uuzhCpJVINSzOFie1bLLSQkwXy5qQeGlBM0wp11rOEq3qNfK2q X-Received: by 2002:a17:90a:e516:b0:280:1dca:f699 with SMTP id t22-20020a17090ae51600b002801dcaf699mr10787245pjy.42.1698895603436; Wed, 01 Nov 2023 20:26:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698895603; cv=none; d=google.com; s=arc-20160816; b=k5YjzoazaRYKBjxnEZK4ne7BWawFNG56hC77EkvfSlkCkJN7p2YUUA8eh0xka21cJm gwYKjWUrnx6Tj9mYPJuR5b6PAVEyZJeT+FV/GGxLXTvVfpfdJqf+392aXHeTTGTLGNox pwMp2nUPLO9cJQXbtv7W2Obyn3qZ4aQ3gEcPQpvb4lemNrCwredisfEH5pXM0tvlV7FB JDWgZkIZnAWnRTPgv8od+1lSCImlQf8R6XmojJz9R7oYFOQ8Y8bhp1OVNOfk4usHm7Zj mOrpRwSemNtSKx6xe2y2XQawwHEHhncNPgLdl3u537PtWqxl7fVeiWEBbvEfv57WVH2Z mCfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=gJd616cEH+YooKkPi26MjXWgnIGEXuyIPk1hxd0FhTk=; fh=Dd5ryTbz4gXtDU9jzXqg2hD87XusQgPqunF08GDmhnA=; b=qTqaGI/cEjsDiaRdzhV6/BqXbwl7vjDO2f1ap1RsZHo++HDaAQBQu/mQi5mmM9h9+L P5sV3/wdO0H50nY8MybCIhT5STYbMKsy6XDubtESl0Zi+pGOIeP54GM/EN770Reeo0rK 7vjVoFQFhgreeXHvV5E1Yvt3/EOiThZqBA6JTRbLADnwGpT4pSpwGrLuroRlsUodmK8H oIE4e7Ci6TcYY40bdLJW4gIJOUr4N3NjVPbFezoJ11i37atKytv3oJqx5Twh6D/AHQwD cwEm+77KUQLEtsaQ6m4DL39goOiGrBJcOTXIVlH49KWNq+eyfaOkSuvS/R/zm57RX7a9 e9TQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=nDjeGraa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id w89-20020a17090a6be200b00280571bb749si2010926pjj.150.2023.11.01.20.26.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Nov 2023 20:26:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=nDjeGraa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 001CD80A9930; Wed, 1 Nov 2023 20:26:00 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348415AbjKBDZt (ORCPT + 35 others); Wed, 1 Nov 2023 23:25:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348419AbjKBDZk (ORCPT ); Wed, 1 Nov 2023 23:25:40 -0400 Received: from out-179.mta1.migadu.com (out-179.mta1.migadu.com [95.215.58.179]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CDB87119 for ; Wed, 1 Nov 2023 20:25:33 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698895532; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gJd616cEH+YooKkPi26MjXWgnIGEXuyIPk1hxd0FhTk=; b=nDjeGraaC0yMn15FP6J1O98CFrfWFi/efFlDQyWlgTKRBWcIkb3Bueh3V63chVaCnBjh/z aT+DJ4liAnoOXN6AjPz5ujQz9YSWQl2pq1MhhIZ8NmN0T3o3AMQcWG/ZORLSwwjhMGbr7d UKvLJzRg/tomBOgosqWMv4T+MnQQ2Cg= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [PATCH v5 9/9] slub: Update frozen slabs documentations in the source Date: Thu, 2 Nov 2023 03:23:30 +0000 Message-Id: <20231102032330.1036151-10-chengming.zhou@linux.dev> In-Reply-To: <20231102032330.1036151-1-chengming.zhou@linux.dev> References: <20231102032330.1036151-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 01 Nov 2023 20:26:01 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781421156484605234 X-GMAIL-MSGID: 1781421156484605234 From: Chengming Zhou The current updated scheme (which this series implemented) is: - node partial slabs: PG_Workingset && !frozen - cpu partial slabs: !PG_Workingset && !frozen - cpu slabs: !PG_Workingset && frozen - full slabs: !PG_Workingset && !frozen The most important change is that "frozen" bit is not set for the cpu partial slabs anymore, __slab_free() will grab node list_lock then check by !PG_Workingset that it's not on a node partial list. And the "frozen" bit is still kept for the cpu slabs for performance, since we don't need to grab node list_lock to check whether the PG_Workingset is set or not if the "frozen" bit is set in __slab_free(). Update related documentations and comments in the source. Signed-off-by: Chengming Zhou Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Acked-by: Christoph Lameter (Ampere) --- mm/slub.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index c20bdf5dab0f..a307d319e82c 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -76,13 +76,22 @@ * * Frozen slabs * - * If a slab is frozen then it is exempt from list management. It is not - * on any list except per cpu partial list. The processor that froze the + * If a slab is frozen then it is exempt from list management. It is + * the cpu slab which is actively allocated from by the processor that + * froze it and it is not on any list. The processor that froze the * slab is the one who can perform list operations on the slab. Other * processors may put objects onto the freelist but the processor that * froze the slab is the only one that can retrieve the objects from the * slab's freelist. * + * CPU partial slabs + * + * The partially empty slabs cached on the CPU partial list are used + * for performance reasons, which speeds up the allocation process. + * These slabs are not frozen, but are also exempt from list management, + * by clearing the PG_workingset flag when moving out of the node + * partial list. Please see __slab_free() for more details. + * * list_lock * * The list_lock protects the partial and full list on each node and @@ -2617,8 +2626,7 @@ static void put_partials_cpu(struct kmem_cache *s, } /* - * Put a slab that was just frozen (in __slab_free|get_partial_node) into a - * partial slab slot if available. + * Put a slab into a partial slab slot if available. * * If we did not find a slot then simply move all the partials to the * per node partial list.