From patchwork Tue Oct 31 14:07:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160126 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b90f:0:b0:403:3b70:6f57 with SMTP id t15csp267289vqg; Tue, 31 Oct 2023 07:09:21 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHw8zhQkGwtUJpr0p4QfZAIw5eF7FXincZKI5/yC1HHOLt7LOBg4ERvDCDTfBakalCJ3VMU X-Received: by 2002:a17:903:68d:b0:1ca:7439:f74f with SMTP id ki13-20020a170903068d00b001ca7439f74fmr11808065plb.60.1698761360987; Tue, 31 Oct 2023 07:09:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698761360; cv=none; d=google.com; s=arc-20160816; b=XfDBW/WmXDA3YIXlIJJe2CvNi2L8QG9EWDXi5hELfMeGQlU44WVtYK6pAHwskZAAEo g0Dagu6TTjL4PQA115izHMpXmZw9CY2TYobnYUQy60p23cRVzEwXC7BNefvDZl1UAd4S 4oFRbQIvhc+7urA3CeZhWPdssxkpd5JT8YOd8l4JEF7BtCGdtyU7q6+/7uHvo7p8IWLV LV353tnpIw7dvTLjUN3MoRsxZQnLACY7aRfhJYmTvJCb0GRYvlPGtjk7c28LDB/pOuaF 7tDN4UZD+KP8Kg1o7TUGr9AUF3LXv5kL5s1Wg4WcfZPFWjxD0upUxy6oMhFAZ41WYg1o JLtQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=V/1IQ8DC4Ww2WANT0Yk1WGGMP6T1XolwvFVe/nWWtdo=; fh=1GJaSiew/4nhYvae+yJYXPcARWi0PNZhXd1mB5ASXRg=; b=mZ/itlPyaBxnlSAvGnW59yY4bOXdj3GoTWCKtU7SkfPIk21m5dQSy8/phH2TO1kbD3 QeV6rZWxnqWL2BxDuXFV2kbB5atfGqE5xUYs3N8nTMvtV8eB4dFqZ6VYv2Ibqkec/RQo qLwXjdlWZlbt/ugCva0TysUZUFyMX3xSmBJUGR4x6lXQF6HD05PY1h4KBa1u+B9lWs2W 96Qbj1xLCEyuzHRKWt1gijD8cOI6hs99zddWtC4l7ZzzWYXpFFJWiMzaHePgdBSP4hVV MJfuySFqQRB+vJ+kOZMjIfztBwK1oVTEvMejdKZaIQ3JmaMrr+iQj0e3u+N2M0CT277O rZTw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=fWm3Ia9W; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id c10-20020a170903234a00b001cc56120079si1083409plh.84.2023.10.31.07.09.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Oct 2023 07:09:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=fWm3Ia9W; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 1AB36805D563; Tue, 31 Oct 2023 07:09:18 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230504AbjJaOJI (ORCPT + 33 others); Tue, 31 Oct 2023 10:09:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229518AbjJaOJF (ORCPT ); Tue, 31 Oct 2023 10:09:05 -0400 Received: from out-179.mta1.migadu.com (out-179.mta1.migadu.com [IPv6:2001:41d0:203:375::b3]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA376C1 for ; Tue, 31 Oct 2023 07:09:02 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698761341; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V/1IQ8DC4Ww2WANT0Yk1WGGMP6T1XolwvFVe/nWWtdo=; b=fWm3Ia9WLFuFmGWFVj3nkTO3KKIcArYI3sKwoNpxt4Vp+VSnp85jAdULf4fwS1xOUGBFRy qaL4xq+9jHvu/YO3QBypb2sb88cVvTthrSNRtZkaROqU8A7pGtYB9RLWsXKVwXBPAiXyCD wIoAlyBL54p89iiOAU9sfE8NjzIGnzM= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, willy@infradead.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v4 1/9] slub: Reflow ___slab_alloc() Date: Tue, 31 Oct 2023 14:07:33 +0000 Message-Id: <20231031140741.79387-2-chengming.zhou@linux.dev> In-Reply-To: <20231031140741.79387-1-chengming.zhou@linux.dev> References: <20231031140741.79387-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Tue, 31 Oct 2023 07:09:18 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781280393154897776 X-GMAIL-MSGID: 1781280393154897776 From: Chengming Zhou The get_partial() interface used in ___slab_alloc() may return a single object in the "kmem_cache_debug(s)" case, in which we will just return the "freelist" object. Move this handling up to prepare for later changes. And the "pfmemalloc_match()" part is not needed for node partial slab, since we already check this in the get_partial_node(). Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka --- mm/slub.c | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 63d281dfacdb..0b0fdc8c189f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3216,8 +3216,21 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, pc.slab = &slab; pc.orig_size = orig_size; freelist = get_partial(s, node, &pc); - if (freelist) - goto check_new_slab; + if (freelist) { + if (kmem_cache_debug(s)) { + /* + * For debug caches here we had to go through + * alloc_single_from_partial() so just store the + * tracking info and return the object. + */ + if (s->flags & SLAB_STORE_USER) + set_track(s, freelist, TRACK_ALLOC, addr); + + return freelist; + } + + goto retry_load_slab; + } slub_put_cpu_ptr(s->cpu_slab); slab = new_slab(s, gfpflags, node); @@ -3253,20 +3266,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, inc_slabs_node(s, slab_nid(slab), slab->objects); -check_new_slab: - - if (kmem_cache_debug(s)) { - /* - * For debug caches here we had to go through - * alloc_single_from_partial() so just store the tracking info - * and return the object - */ - if (s->flags & SLAB_STORE_USER) - set_track(s, freelist, TRACK_ALLOC, addr); - - return freelist; - } - if (unlikely(!pfmemalloc_match(slab, gfpflags))) { /* * For !pfmemalloc_match() case we don't load freelist so that From patchwork Tue Oct 31 14:07:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160127 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b90f:0:b0:403:3b70:6f57 with SMTP id t15csp267315vqg; Tue, 31 Oct 2023 07:09:23 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHIlJeaqs3LOgP58NGr0LxYG4MimPVcn8YnlUk4efnAo6RQBwJWP9HoWzZXEf7ia5ylSEFm X-Received: by 2002:a17:90a:d497:b0:27f:ecd9:6d0e with SMTP id s23-20020a17090ad49700b0027fecd96d0emr11398436pju.34.1698761363344; Tue, 31 Oct 2023 07:09:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698761363; cv=none; d=google.com; s=arc-20160816; b=oYecRHMQX2Crqsxk1EIFpnCpbsImX+e+3KmkWrZu/PlaN/FbWy2wLh3wmm6ClktZFL c9VorsTfYg9FgGtidkrgKext08XoNVUA3exYr5FpKYFgBPLpgifGIb1ZAJh1zvNFMqUa 1jFhuVPB/L38onW7p0mwnc5pP6CHsmmGrMnPgsuOck1pnMxaYdwayImckcJsVJ7imE3Y kR19kDDh6g6JSm0oHdFty8Xb+pG0jVtgOLa5Nc10de8mMHT5heYAlVHsmr975rysIeGX vZ4PVy/JO8QYmVq4E6JhP1ujnJDXAvLst3x9mHNYf1uxontUViT7av8+W9WoLjdUj2gc B4wA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=GvFpTjudBUawLgUZhfA4XumfhFt8DHF3SUizR1gsSN0=; fh=1GJaSiew/4nhYvae+yJYXPcARWi0PNZhXd1mB5ASXRg=; b=bsq/+hoXHUiV3gtGjgYj1KoBMIZOmFUnZZBE+DiTpMCHXo/3oYuitaNTbWRgsCubvO t7nZEA7Tj+4/kKW433qPpzrIL399fQPxcmfllqfmeS6QPRa4dF8xZtR0EVUIq1xrq5su X5WtH2Jg3nQ4CN9llGxRVvU8zJAFRVYzMvQrPqe16GMgSFM4Cg4y/VrWdxxrvlvpKQ8q tUnNiIlwust9nJSTmXwDXcDPobMyyufMV6MN/i3f3fhbHDeOa4Xq/dte0COjlREWmsaD jT/6n6hQXFWis/m60tRiDFgOsHX7Gl8FK6FytX1caxsbD8AVeLaRfLu0fcw3k9lChhDu pfXA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=TXFp5ZrS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id t17-20020a17090b019100b002791d79bbf9si992179pjs.187.2023.10.31.07.09.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Oct 2023 07:09:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=TXFp5ZrS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 0DF6280A4F68; Tue, 31 Oct 2023 07:09:21 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231779AbjJaOJK (ORCPT + 33 others); Tue, 31 Oct 2023 10:09:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34430 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231286AbjJaOJI (ORCPT ); Tue, 31 Oct 2023 10:09:08 -0400 Received: from out-188.mta1.migadu.com (out-188.mta1.migadu.com [95.215.58.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A73AE6 for ; Tue, 31 Oct 2023 07:09:05 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698761343; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GvFpTjudBUawLgUZhfA4XumfhFt8DHF3SUizR1gsSN0=; b=TXFp5ZrS2RLrdK40RbVab8YMC1O20QrLDJHbtWDv0h7N6MLbta00vjmZGm7DnKTG5RBgxq h6YdjNWfhUe2h4dCRzYErdTwo+9MGcYO5LIMetNw+He4mdD9iExl/e/+7gz+qY3n+fIq95 TDslQnTghxmWva6tPEM6zC6/ZgI5/0A= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, willy@infradead.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v4 2/9] slub: Change get_partial() interfaces to return slab Date: Tue, 31 Oct 2023 14:07:34 +0000 Message-Id: <20231031140741.79387-3-chengming.zhou@linux.dev> In-Reply-To: <20231031140741.79387-1-chengming.zhou@linux.dev> References: <20231031140741.79387-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Tue, 31 Oct 2023 07:09:21 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781280395258076535 X-GMAIL-MSGID: 1781280395258076535 From: Chengming Zhou We need all get_partial() related interfaces to return a slab, instead of returning the freelist (or object). Use the partial_context.object to return back freelist or object for now. This patch shouldn't have any functional changes. Suggested-by: Vlastimil Babka Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka --- mm/slub.c | 63 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 33 insertions(+), 30 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 0b0fdc8c189f..03384cd965c5 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -204,9 +204,9 @@ DEFINE_STATIC_KEY_FALSE(slub_debug_enabled); /* Structure holding parameters for get_partial() call chain */ struct partial_context { - struct slab **slab; gfp_t flags; unsigned int orig_size; + void *object; }; static inline bool kmem_cache_debug(struct kmem_cache *s) @@ -2269,10 +2269,11 @@ static inline bool pfmemalloc_match(struct slab *slab, gfp_t gfpflags); /* * Try to allocate a partial slab from a specific node. */ -static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, - struct partial_context *pc) +static struct slab *get_partial_node(struct kmem_cache *s, + struct kmem_cache_node *n, + struct partial_context *pc) { - struct slab *slab, *slab2; + struct slab *slab, *slab2, *partial = NULL; void *object = NULL; unsigned long flags; unsigned int partial_slabs = 0; @@ -2288,27 +2289,28 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, spin_lock_irqsave(&n->list_lock, flags); list_for_each_entry_safe(slab, slab2, &n->partial, slab_list) { - void *t; - if (!pfmemalloc_match(slab, pc->flags)) continue; if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) { object = alloc_single_from_partial(s, n, slab, pc->orig_size); - if (object) + if (object) { + partial = slab; + pc->object = object; break; + } continue; } - t = acquire_slab(s, n, slab, object == NULL); - if (!t) + object = acquire_slab(s, n, slab, object == NULL); + if (!object) break; - if (!object) { - *pc->slab = slab; + if (!partial) { + partial = slab; + pc->object = object; stat(s, ALLOC_FROM_PARTIAL); - object = t; } else { put_cpu_partial(s, slab, 0); stat(s, CPU_PARTIAL_NODE); @@ -2324,20 +2326,21 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, } spin_unlock_irqrestore(&n->list_lock, flags); - return object; + return partial; } /* * Get a slab from somewhere. Search in increasing NUMA distances. */ -static void *get_any_partial(struct kmem_cache *s, struct partial_context *pc) +static struct slab *get_any_partial(struct kmem_cache *s, + struct partial_context *pc) { #ifdef CONFIG_NUMA struct zonelist *zonelist; struct zoneref *z; struct zone *zone; enum zone_type highest_zoneidx = gfp_zone(pc->flags); - void *object; + struct slab *slab; unsigned int cpuset_mems_cookie; /* @@ -2372,8 +2375,8 @@ static void *get_any_partial(struct kmem_cache *s, struct partial_context *pc) if (n && cpuset_zone_allowed(zone, pc->flags) && n->nr_partial > s->min_partial) { - object = get_partial_node(s, n, pc); - if (object) { + slab = get_partial_node(s, n, pc); + if (slab) { /* * Don't check read_mems_allowed_retry() * here - if mems_allowed was updated in @@ -2381,7 +2384,7 @@ static void *get_any_partial(struct kmem_cache *s, struct partial_context *pc) * between allocation and the cpuset * update */ - return object; + return slab; } } } @@ -2393,17 +2396,18 @@ static void *get_any_partial(struct kmem_cache *s, struct partial_context *pc) /* * Get a partial slab, lock it and return it. */ -static void *get_partial(struct kmem_cache *s, int node, struct partial_context *pc) +static struct slab *get_partial(struct kmem_cache *s, int node, + struct partial_context *pc) { - void *object; + struct slab *slab; int searchnode = node; if (node == NUMA_NO_NODE) searchnode = numa_mem_id(); - object = get_partial_node(s, get_node(s, searchnode), pc); - if (object || node != NUMA_NO_NODE) - return object; + slab = get_partial_node(s, get_node(s, searchnode), pc); + if (slab || node != NUMA_NO_NODE) + return slab; return get_any_partial(s, pc); } @@ -3213,10 +3217,10 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, new_objects: pc.flags = gfpflags; - pc.slab = &slab; pc.orig_size = orig_size; - freelist = get_partial(s, node, &pc); - if (freelist) { + slab = get_partial(s, node, &pc); + if (slab) { + freelist = pc.object; if (kmem_cache_debug(s)) { /* * For debug caches here we had to go through @@ -3408,12 +3412,11 @@ static void *__slab_alloc_node(struct kmem_cache *s, void *object; pc.flags = gfpflags; - pc.slab = &slab; pc.orig_size = orig_size; - object = get_partial(s, node, &pc); + slab = get_partial(s, node, &pc); - if (object) - return object; + if (slab) + return pc.object; slab = new_slab(s, gfpflags, node); if (unlikely(!slab)) { From patchwork Tue Oct 31 14:07:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160130 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b90f:0:b0:403:3b70:6f57 with SMTP id t15csp267515vqg; Tue, 31 Oct 2023 07:09:37 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFxNs4n0DHgGLqa7BnIo8KrPhEoccmjATRe/vavPqg7IZCBdiAD8pu8Ao4a0XsoGBqwVoaO X-Received: by 2002:a17:902:e5cf:b0:1cc:3c6c:ce23 with SMTP id u15-20020a170902e5cf00b001cc3c6cce23mr6955155plf.42.1698761377214; Tue, 31 Oct 2023 07:09:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698761377; cv=none; d=google.com; s=arc-20160816; b=dFI/JcUIceKlHG8ujQq3Plqjq4X9sF0/sRQBkcYWkF319dqtyv8qF5k9A18dEQ2KNt BI0peNsC/m8Evy7OPGunlFLx637H/LwOpa21l3yOwpc2TrkIfY/HbFkHPrqSKld4vfkU DV9I9Jxgh8JYwAMeDTMg/T8ebW2QpFo5zRGrz5jMHluzDcuHn8ZbsMG0NQgeEfDVQiti 7imJl7gUKxWb591UoRDG4BPyQt/g01JZ0yIB6ily9kqMLuky4KhfGuay+QIFgskMkP37 ymKqrut4Uv0HNZ9OEMaBBYIwM29Wvcz7nwJarrj+Gnf68/FsDgWxYloXCUQ0KtujookU K1bA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=A++jOVGS7LIP72VwRB52HksoP76pYwnYqPVPh8XlVaY=; fh=1GJaSiew/4nhYvae+yJYXPcARWi0PNZhXd1mB5ASXRg=; b=XTMGLjCACtmNGM+WIuXuc1M6inRt+j4vx2UHQZTpjqp7EV92sQKgIB0Nz4rPGq3LQ6 tECTpL1++xgDNjDdHyekPLo05Ig1flshBH4qB2I2GXERldreYMcCOkX4ZdWhZhKU1kvo mObLcVzNT0kbYYU164FlODj/tCLxAGcdEdJ197qgwjiJZfSYZn6cVmvzOUsq6X1P2lam j555bvs6E8a+yjyg3REpxz96s1rOiDe718PGr4cDKWCazM06JA56ZxcmcheG8j8VJLiF R9I53Ccy599Rv/oPd4/eT/tgpXXc5WEqVJ52L/A1R7JfZwfQ4P8Y2/Lu35slqEhXw359 XhqA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=e0oJj347; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id u7-20020a170903124700b001c72476c96fsi1061933plh.143.2023.10.31.07.09.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Oct 2023 07:09:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=e0oJj347; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 044AB802B137; Tue, 31 Oct 2023 07:09:32 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233530AbjJaOJN (ORCPT + 33 others); Tue, 31 Oct 2023 10:09:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34474 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231286AbjJaOJL (ORCPT ); Tue, 31 Oct 2023 10:09:11 -0400 Received: from out-181.mta1.migadu.com (out-181.mta1.migadu.com [IPv6:2001:41d0:203:375::b5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 492E9F1 for ; Tue, 31 Oct 2023 07:09:08 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698761346; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A++jOVGS7LIP72VwRB52HksoP76pYwnYqPVPh8XlVaY=; b=e0oJj347aPZSwbHKIh3vXLsSkKARGwrOKNnsPyGmHd3T7kZSTXQX3ogXWfbcaQarI3N9T+ K6a8+/g2w/Lvo1/fNOwaYaNltKZM0McSUApZ+72CI1R6rJaPoGntDGGb/xG8G5fgvNBo8e 6XB8zAf1SLuL1e087s8VurnX6PJPY5s= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, willy@infradead.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v4 3/9] slub: Keep track of whether slub is on the per-node partial list Date: Tue, 31 Oct 2023 14:07:35 +0000 Message-Id: <20231031140741.79387-4-chengming.zhou@linux.dev> In-Reply-To: <20231031140741.79387-1-chengming.zhou@linux.dev> References: <20231031140741.79387-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Tue, 31 Oct 2023 07:09:32 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781280409618663887 X-GMAIL-MSGID: 1781280409618663887 From: Chengming Zhou Now we rely on the "frozen" bit to see if we should manipulate the slab->slab_list, which will be changed in the following patch. Instead we introduce another way to keep track of whether slub is on the per-node partial list, here we reuse the PG_workingset bit. We use __set_bit and __clear_bit directly instead of the atomic version for better performance and it's safe since it's protected by the slub node list_lock. Suggested-by: Matthew Wilcox Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka --- mm/slub.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/mm/slub.c b/mm/slub.c index 03384cd965c5..eed8ae0dbaf9 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2116,6 +2116,25 @@ static void discard_slab(struct kmem_cache *s, struct slab *slab) free_slab(s, slab); } +/* + * SLUB reuses PG_workingset bit to keep track of whether it's on + * the per-node partial list. + */ +static inline bool slab_test_node_partial(const struct slab *slab) +{ + return folio_test_workingset((struct folio *)slab_folio(slab)); +} + +static inline void slab_set_node_partial(struct slab *slab) +{ + __set_bit(PG_workingset, folio_flags(slab_folio(slab), 0)); +} + +static inline void slab_clear_node_partial(struct slab *slab) +{ + __clear_bit(PG_workingset, folio_flags(slab_folio(slab), 0)); +} + /* * Management of partially allocated slabs. */ @@ -2127,6 +2146,7 @@ __add_partial(struct kmem_cache_node *n, struct slab *slab, int tail) list_add_tail(&slab->slab_list, &n->partial); else list_add(&slab->slab_list, &n->partial); + slab_set_node_partial(slab); } static inline void add_partial(struct kmem_cache_node *n, @@ -2141,6 +2161,7 @@ static inline void remove_partial(struct kmem_cache_node *n, { lockdep_assert_held(&n->list_lock); list_del(&slab->slab_list); + slab_clear_node_partial(slab); n->nr_partial--; } @@ -4833,6 +4854,7 @@ static int __kmem_cache_do_shrink(struct kmem_cache *s) if (free == slab->objects) { list_move(&slab->slab_list, &discard); + slab_clear_node_partial(slab); n->nr_partial--; dec_slabs_node(s, node, slab->objects); } else if (free <= SHRINK_PROMOTE_MAX) From patchwork Tue Oct 31 14:07:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160128 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b90f:0:b0:403:3b70:6f57 with SMTP id t15csp267393vqg; Tue, 31 Oct 2023 07:09:28 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFr+N97I+0ErKqsC5JwTsekJDA7LX8qaheO84yF5ftvxpsDl1aeqREB0oh/JfEqXqoFRE8U X-Received: by 2002:a17:902:fac4:b0:1cc:40eb:79ae with SMTP id ld4-20020a170902fac400b001cc40eb79aemr6165598plb.63.1698761368560; Tue, 31 Oct 2023 07:09:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698761368; cv=none; d=google.com; s=arc-20160816; b=1CUKxrR8LLqxZubGj/04mVm9+AlQAu0/fON26dx7OcB00WK5AO65dD9Vy3ugT2ZMHH DX9dxuiZy6ih6ieg+J732YcNO9ArBV6NCJN53c0CkOU1EVUTjHPMC+hSUh/G9bDzhkeY 02k3kQB+OIrJZVXRgVcRKFRywlmBzWow6mLXaNUQcpYkubGI7BKq8+5E1hm6yBo5UWsW unHDP3FWzJbfVusSy9UIkFqKjUE6v2xUVyn4emmo6gANN6MXvaDeyGHKMJonZVpM+9If vrYYQTu3JrfXtQj23m3udh88Jr/MRG471MC7R/ZSCKdF64ypwKO7HMMmW9soQgoIiiGR /z/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=v/W13f1yvwrCHZgH+TSl+cdWReIuGoRVTB5psVe8An4=; fh=1GJaSiew/4nhYvae+yJYXPcARWi0PNZhXd1mB5ASXRg=; b=kitofonmTNE88/SS/eQ8g6bXCiL7b3xUmw/Fz6p2qF9fs4YJoMQYwJ7MbeTo3bVLdb e2AaSfNsLoH0o/1BGTMeFZSqcuJqQ9ViNIxGLrGgRSDH+7SIUKzFeje1+CrdxwjwxY+z SWa/O9nYO3XINFzTj81NU6rdyfu3g9Dw/Raj5uSq/AQTk9IxdfjSS67RSOTSxnzVOfVP q+QhmVBxYAk72Nz7KyuyX3rbfFBa7lSPnERf6J77/58XOKRFgmwzhTaKr6qzd6EfHP3G m7aNZxG7muoVN8Qt/WRvdgkuksBPWW6qRGrDddIeoq2bcyM7iIq+kXa+MyGBXNfU4+rm Hd7g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=wNIpzakv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id w15-20020a170902e88f00b001cc2a6624e5si1068536plg.307.2023.10.31.07.09.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Oct 2023 07:09:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=wNIpzakv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 608968039444; Tue, 31 Oct 2023 07:09:26 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231286AbjJaOJQ (ORCPT + 33 others); Tue, 31 Oct 2023 10:09:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34586 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233719AbjJaOJO (ORCPT ); Tue, 31 Oct 2023 10:09:14 -0400 Received: from out-185.mta1.migadu.com (out-185.mta1.migadu.com [95.215.58.185]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39D9D115 for ; Tue, 31 Oct 2023 07:09:11 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698761349; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v/W13f1yvwrCHZgH+TSl+cdWReIuGoRVTB5psVe8An4=; b=wNIpzakvsd93W80vtkHwQf555uZ1X7UCHSu5UZdhAwR+WInzzIHY0wg3MqdVqIXyuFrveg zfUzHrnr02/YVCm2lpOuVKjDOu31IpZFnrJaagZexEAqG5577/evBuPT6k6264Mc9xVBB2 K4yu/KTcypElGi304xzyFujmguhhPdg= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, willy@infradead.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v4 4/9] slub: Prepare __slab_free() for unfrozen partial slab out of node partial list Date: Tue, 31 Oct 2023 14:07:36 +0000 Message-Id: <20231031140741.79387-5-chengming.zhou@linux.dev> In-Reply-To: <20231031140741.79387-1-chengming.zhou@linux.dev> References: <20231031140741.79387-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Tue, 31 Oct 2023 07:09:26 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781280400639465341 X-GMAIL-MSGID: 1781280400639465341 From: Chengming Zhou Now the partially empty slub will be frozen when taken out of node partial list, so the __slab_free() will know from "was_frozen" that the partially empty slab is not on node partial list and is a cpu or cpu partial slab of some cpu. But we will change this, make partial slabs leave the node partial list with unfrozen state, so we need to change __slab_free() to use the new slab_test_node_partial() we just introduced. Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka --- mm/slub.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/mm/slub.c b/mm/slub.c index eed8ae0dbaf9..1880b483350e 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3631,6 +3631,7 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, unsigned long counters; struct kmem_cache_node *n = NULL; unsigned long flags; + bool on_node_partial; stat(s, FREE_SLOWPATH); @@ -3678,6 +3679,7 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, */ spin_lock_irqsave(&n->list_lock, flags); + on_node_partial = slab_test_node_partial(slab); } } @@ -3706,6 +3708,15 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, return; } + /* + * This slab was partially empty but not on the per-node partial list, + * in which case we shouldn't manipulate its list, just return. + */ + if (prior && !on_node_partial) { + spin_unlock_irqrestore(&n->list_lock, flags); + return; + } + if (unlikely(!new.inuse && n->nr_partial >= s->min_partial)) goto slab_empty; From patchwork Tue Oct 31 14:07:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160129 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b90f:0:b0:403:3b70:6f57 with SMTP id t15csp267479vqg; Tue, 31 Oct 2023 07:09:35 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEaTqKqZlc9obI0TZBIYdM4QekcmGPkYQ641WjrZKhSKLkWmQ3q7Z+nVqlz2XLf5D5ESBwr X-Received: by 2002:a05:6a20:1447:b0:17b:2b7e:923c with SMTP id a7-20020a056a20144700b0017b2b7e923cmr12350124pzi.16.1698761374803; Tue, 31 Oct 2023 07:09:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698761374; cv=none; d=google.com; s=arc-20160816; b=Cvyzx6RPpDiwQv/zFQG/WmhmZC+26gzq6Gce/WJxVSFVnNO/rM9DjMSJU77WixkKxw Gnku86LIUj8t7yFEzUAetJP9f0CamqydmRFyqaq2MS+wiSJyEzn7L3TgUsZWubyNIgKQ NbYK9xLduhALng0+7uVFEUyTMoJkUWGUnnDXh0vg25uLCYVAu3xyBGikL/q/RSMiYGHr MEPcC/ndv8rkiS+knP950bVccBSl/5oePur00mNqkUphD87O5boymfhf+BzxEa6MszDo V/oZ2brnEr1JP93rofLSLB8ZGb2oEIr9ID74wz+p847Q8LOv+UmkHOF4oAp2nxPf9+gl GDyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=DSV2hJzhN1EMBaKNKe5GTqHBigf8Oik9SH9kDO4tfhs=; fh=1GJaSiew/4nhYvae+yJYXPcARWi0PNZhXd1mB5ASXRg=; b=FZeY0HTE90m2lkqMam/r/+8byAUYwjziHvmmo4nwsM82QsjsbZOCLFWKdJtQdj6PPC IB2SgpGu+/7377iNzsBtKrhzfBImoofWftvqHDyrNT0PgQijYvG0t/fgDFdvGBU1cGrB YQTRT4HiRbStJ7NZSsMWvlcaHQliBAjVsJaVa5pY4EkWxJdZD0FidPrKdL9BaSqTs77B fTz9k8wqhp2IXt7qVDcfQDeOc+rsU7o3ShyQjRgsmUW6Ay5yme+NA0DFOXhlz/UYtblh h2uyiDoZxycvcFaB5sorIVeTUClTUwUEA5tKTLItkeAZCJPDjgkQzI4FFcamp66S99CO JQrg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=WHMjfXAw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id f16-20020aa79d90000000b00690258a9777si1038473pfq.20.2023.10.31.07.09.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Oct 2023 07:09:34 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=WHMjfXAw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 6E35D80BEF0A; Tue, 31 Oct 2023 07:09:31 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235216AbjJaOJY (ORCPT + 33 others); Tue, 31 Oct 2023 10:09:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34678 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233536AbjJaOJQ (ORCPT ); Tue, 31 Oct 2023 10:09:16 -0400 Received: from out-182.mta1.migadu.com (out-182.mta1.migadu.com [95.215.58.182]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 14BA0125 for ; Tue, 31 Oct 2023 07:09:13 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698761352; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DSV2hJzhN1EMBaKNKe5GTqHBigf8Oik9SH9kDO4tfhs=; b=WHMjfXAw75r95lcc5nFUiMN89Tof/YlZMyVPqZKNOdvO1yteE6uHCrbGBMOqY7xCBTII+D bv4O0JIZHkzqYTy9jSpAEEAubWuuo1+LX71TrqtSTlbNRuljW0aerBJZZx3KeiAGfHntFE 9WQV2XL6SGskdWaTT/AwR5nu7UHQO8I= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, willy@infradead.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v4 5/9] slub: Introduce freeze_slab() Date: Tue, 31 Oct 2023 14:07:37 +0000 Message-Id: <20231031140741.79387-6-chengming.zhou@linux.dev> In-Reply-To: <20231031140741.79387-1-chengming.zhou@linux.dev> References: <20231031140741.79387-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Tue, 31 Oct 2023 07:09:31 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781280407307690967 X-GMAIL-MSGID: 1781280407307690967 From: Chengming Zhou We will have unfrozen slabs out of the node partial list later, so we need a freeze_slab() function to freeze the partial slab and get its freelist. Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/mm/slub.c b/mm/slub.c index 1880b483350e..edf567971679 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3098,6 +3098,33 @@ static inline void *get_freelist(struct kmem_cache *s, struct slab *slab) return freelist; } +/* + * Freeze the partial slab and return the pointer to the freelist. + */ +static inline void *freeze_slab(struct kmem_cache *s, struct slab *slab) +{ + struct slab new; + unsigned long counters; + void *freelist; + + do { + freelist = slab->freelist; + counters = slab->counters; + + new.counters = counters; + VM_BUG_ON(new.frozen); + + new.inuse = slab->objects; + new.frozen = 1; + + } while (!slab_update_freelist(s, slab, + freelist, counters, + NULL, new.counters, + "freeze_slab")); + + return freelist; +} + /* * Slow path. The lockless freelist is empty or we need to perform * debugging duties. From patchwork Tue Oct 31 14:07:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160131 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b90f:0:b0:403:3b70:6f57 with SMTP id t15csp267786vqg; Tue, 31 Oct 2023 07:09:55 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGaFRl0kThHlsPOFhfFDTQYdcLJGanBMhlzKBWm3p/50xC88gsqLqsvP0JJ3vbnKpyCdJQK X-Received: by 2002:a05:6a21:7897:b0:138:68b9:138e with SMTP id bf23-20020a056a21789700b0013868b9138emr3619440pzc.8.1698761394948; Tue, 31 Oct 2023 07:09:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698761394; cv=none; d=google.com; s=arc-20160816; b=qCkwGj6NT8nU+GKq6f4XiALORi9tjfZl5ah6F7OE9IYFB+x+BxnPRSXba3GaVkfigU +fQjCgBNbcqgI7EpjGdsOrfWm5bd3fNHSme6gSAzl99rXbBIhjNCaVVaDMbZW0nafyEm eDactkEwWJGe9w2msuMCMKYeVvGaaJ+GK3Rx8xmTsLXCqIIhYYeY0Ousoh8WXBG46aCX uWYARXevuCe5x6CP/+2/PzPbyjSZ9P1PLendWB0ul5LtmG+L3xiBQJAcrvsPLa8y6Avy FMz2ZRQMPJrp9nbrQvebU/Jlv18iopkzh0/n/d1rF93RTNo5CsB2YJOE8pOsYoVXqj46 fwwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=WFFMQyhLZJnEzwgDfbrlXS+sPYcJwR5pdI64P4BtRGU=; fh=1GJaSiew/4nhYvae+yJYXPcARWi0PNZhXd1mB5ASXRg=; b=Qur1xi/AxVK+9fmxTvjANfDRzYybxY8EaMJ2r0uuYWuB0HSFw386+K+Fex44Z7hBqK r0rp2J7hofehXMO40CL+CdOxOeEOMV2SlPd00taZEC/Ysz7AdjuB37FxVYCEDxU4ZUlZ 0brOe8jx2+dgwaVPQCqSy+8UmCCpBBAJHb73ulfGk8q7uwSixvWFCe4pCimuTLRbIcLy 7KPACZ6L/51fl6vLa3DeCDrUZxcoOt+C8WYf54XQ3g4Pjjjq3gVytOgsjGXd5iYweutH Red+o/g3A1nHi3aEFfZ8+vYmJhuX6ztVtMlh9hmlGqzNKnEESjELkDaLDTRGbgd3/5ie 268g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=T7PF2RDx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id a25-20020a656419000000b005777bea0b6asi1078732pgv.859.2023.10.31.07.09.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Oct 2023 07:09:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=T7PF2RDx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 1314B8037AA8; Tue, 31 Oct 2023 07:09:52 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235293AbjJaOJb (ORCPT + 33 others); Tue, 31 Oct 2023 10:09:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34960 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233464AbjJaOJY (ORCPT ); Tue, 31 Oct 2023 10:09:24 -0400 Received: from out-180.mta1.migadu.com (out-180.mta1.migadu.com [IPv6:2001:41d0:203:375::b4]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F7F4133 for ; Tue, 31 Oct 2023 07:09:16 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698761355; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WFFMQyhLZJnEzwgDfbrlXS+sPYcJwR5pdI64P4BtRGU=; b=T7PF2RDxrnhml9AGrzBwH7Qb6gebmTAsuAiIOValuuZMya2Gd9T1jQvXTSlgNdQynWleC7 BIbYj4Hhvn5EHHFj1uNZNgRFHhPDpDZmezPhOhX5zL2VsVdotFZfFjnqhG8XarSBkxuXgN ddq0HXNsmojE5zrxEzjWz16CJWMGtZU= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, willy@infradead.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v4 6/9] slub: Delay freezing of partial slabs Date: Tue, 31 Oct 2023 14:07:38 +0000 Message-Id: <20231031140741.79387-7-chengming.zhou@linux.dev> In-Reply-To: <20231031140741.79387-1-chengming.zhou@linux.dev> References: <20231031140741.79387-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Tue, 31 Oct 2023 07:09:52 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781280428236525853 X-GMAIL-MSGID: 1781280428236525853 From: Chengming Zhou Now we will freeze slabs when moving them out of node partial list to cpu partial list, this method needs two cmpxchg_double operations: 1. freeze slab (acquire_slab()) under the node list_lock 2. get_freelist() when pick used in ___slab_alloc() Actually we don't need to freeze when moving slabs out of node partial list, we can delay freezing to when use slab freelist in ___slab_alloc(), so we can save one cmpxchg_double(). And there are other good points: - The moving of slabs between node partial list and cpu partial list becomes simpler, since we don't need to freeze or unfreeze at all. - The node list_lock contention would be less, since we don't need to freeze any slab under the node list_lock. We can achieve this because there is no concurrent path would manipulate the partial slab list except the __slab_free() path, which is now serialized by slab_test_node_partial() under the list_lock. Since the slab returned by get_partial() interfaces is not frozen anymore and no freelist is returned in the partial_context, so we need to use the introduced freeze_slab() to freeze it and get its freelist. Similarly, the slabs on the CPU partial list are not frozen anymore, we need to freeze_slab() on it before use. We can now delete acquire_slab() as it became unused. Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka --- mm/slub.c | 113 +++++++++++------------------------------------------- 1 file changed, 23 insertions(+), 90 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index edf567971679..bcb5b2c4e213 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2234,51 +2234,6 @@ static void *alloc_single_from_new_slab(struct kmem_cache *s, return object; } -/* - * Remove slab from the partial list, freeze it and - * return the pointer to the freelist. - * - * Returns a list of objects or NULL if it fails. - */ -static inline void *acquire_slab(struct kmem_cache *s, - struct kmem_cache_node *n, struct slab *slab, - int mode) -{ - void *freelist; - unsigned long counters; - struct slab new; - - lockdep_assert_held(&n->list_lock); - - /* - * Zap the freelist and set the frozen bit. - * The old freelist is the list of objects for the - * per cpu allocation list. - */ - freelist = slab->freelist; - counters = slab->counters; - new.counters = counters; - if (mode) { - new.inuse = slab->objects; - new.freelist = NULL; - } else { - new.freelist = freelist; - } - - VM_BUG_ON(new.frozen); - new.frozen = 1; - - if (!__slab_update_freelist(s, slab, - freelist, counters, - new.freelist, new.counters, - "acquire_slab")) - return NULL; - - remove_partial(n, slab); - WARN_ON(!freelist); - return freelist; -} - #ifdef CONFIG_SLUB_CPU_PARTIAL static void put_cpu_partial(struct kmem_cache *s, struct slab *slab, int drain); #else @@ -2295,7 +2250,6 @@ static struct slab *get_partial_node(struct kmem_cache *s, struct partial_context *pc) { struct slab *slab, *slab2, *partial = NULL; - void *object = NULL; unsigned long flags; unsigned int partial_slabs = 0; @@ -2314,7 +2268,7 @@ static struct slab *get_partial_node(struct kmem_cache *s, continue; if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) { - object = alloc_single_from_partial(s, n, slab, + void *object = alloc_single_from_partial(s, n, slab, pc->orig_size); if (object) { partial = slab; @@ -2324,13 +2278,10 @@ static struct slab *get_partial_node(struct kmem_cache *s, continue; } - object = acquire_slab(s, n, slab, object == NULL); - if (!object) - break; + remove_partial(n, slab); if (!partial) { partial = slab; - pc->object = object; stat(s, ALLOC_FROM_PARTIAL); } else { put_cpu_partial(s, slab, 0); @@ -2629,9 +2580,6 @@ static void __unfreeze_partials(struct kmem_cache *s, struct slab *partial_slab) unsigned long flags = 0; while (partial_slab) { - struct slab new; - struct slab old; - slab = partial_slab; partial_slab = slab->next; @@ -2644,23 +2592,7 @@ static void __unfreeze_partials(struct kmem_cache *s, struct slab *partial_slab) spin_lock_irqsave(&n->list_lock, flags); } - do { - - old.freelist = slab->freelist; - old.counters = slab->counters; - VM_BUG_ON(!old.frozen); - - new.counters = old.counters; - new.freelist = old.freelist; - - new.frozen = 0; - - } while (!__slab_update_freelist(s, slab, - old.freelist, old.counters, - new.freelist, new.counters, - "unfreezing slab")); - - if (unlikely(!new.inuse && n->nr_partial >= s->min_partial)) { + if (unlikely(!slab->inuse && n->nr_partial >= s->min_partial)) { slab->next = slab_to_discard; slab_to_discard = slab; } else { @@ -3167,7 +3099,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, node = NUMA_NO_NODE; goto new_slab; } -redo: if (unlikely(!node_match(slab, node))) { /* @@ -3243,7 +3174,8 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, new_slab: - if (slub_percpu_partial(c)) { +#ifdef CONFIG_SLUB_CPU_PARTIAL + while (slub_percpu_partial(c)) { local_lock_irqsave(&s->cpu_slab->lock, flags); if (unlikely(c->slab)) { local_unlock_irqrestore(&s->cpu_slab->lock, flags); @@ -3255,12 +3187,22 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, goto new_objects; } - slab = c->slab = slub_percpu_partial(c); + slab = slub_percpu_partial(c); slub_set_percpu_partial(c, slab); local_unlock_irqrestore(&s->cpu_slab->lock, flags); stat(s, CPU_PARTIAL_ALLOC); - goto redo; + + if (unlikely(!node_match(slab, node) || + !pfmemalloc_match(slab, gfpflags))) { + slab->next = NULL; + __unfreeze_partials(s, slab); + continue; + } + + freelist = freeze_slab(s, slab); + goto retry_load_slab; } +#endif new_objects: @@ -3268,8 +3210,8 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, pc.orig_size = orig_size; slab = get_partial(s, node, &pc); if (slab) { - freelist = pc.object; if (kmem_cache_debug(s)) { + freelist = pc.object; /* * For debug caches here we had to go through * alloc_single_from_partial() so just store the @@ -3281,6 +3223,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, return freelist; } + freelist = freeze_slab(s, slab); goto retry_load_slab; } @@ -3682,18 +3625,8 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, was_frozen = new.frozen; new.inuse -= cnt; if ((!new.inuse || !prior) && !was_frozen) { - - if (kmem_cache_has_cpu_partial(s) && !prior) { - - /* - * Slab was on no list before and will be - * partially empty - * We can defer the list move and instead - * freeze it. - */ - new.frozen = 1; - - } else { /* Needs to be taken off a list */ + /* Needs to be taken off a list */ + if (!kmem_cache_has_cpu_partial(s) || prior) { n = get_node(s, slab_nid(slab)); /* @@ -3723,9 +3656,9 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, * activity can be necessary. */ stat(s, FREE_FROZEN); - } else if (new.frozen) { + } else if (kmem_cache_has_cpu_partial(s) && !prior) { /* - * If we just froze the slab then put it onto the + * If we started with a full slab then put it onto the * per cpu partial list. */ put_cpu_partial(s, slab, 1); From patchwork Tue Oct 31 14:07:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160134 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b90f:0:b0:403:3b70:6f57 with SMTP id t15csp268461vqg; Tue, 31 Oct 2023 07:10:44 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGay1oBZQhwtQC5x9jB2hmH2ljgt5EPBPp9LhwWqZVWNVjp9jf0IWuwdCGU9GSslUrQdd3W X-Received: by 2002:a05:6a00:23c6:b0:68f:c7c5:a73a with SMTP id g6-20020a056a0023c600b0068fc7c5a73amr16491643pfc.16.1698761443918; Tue, 31 Oct 2023 07:10:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698761443; cv=none; d=google.com; s=arc-20160816; b=pruo2us50RBCXl9SGWjuyeln5FgNU8HiqnA/8ty6WNgFDfwxi3ZYlEzumE0QbYt0Wb wJMLfAFmkyTtP7VdiXiNvHgsko40geoZOvVfKPIa7fRotQI5m8NVluFVOvMZlQbBDL3Z wIwcHgi/DsaiIcC96riddGiXwdTz0ey8t5HC9DxxlIsI4yBE2ZQehE3B2TezbKKAH1cf myaPWJ63GT0Mwlh70X0Nl1COgukMz0fU5DhuPyADc1ezavujM3N8MYZrrnro4SZpV1I7 UEedVd+SKRR6nJCVmPswyIYxw1IMsaFRBbWGa8wcpEoyRFPi5T+7gXe6didl4ilXj7mL 4RDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=71jccDuvmHUN0B2hAN0chVUZ3a/I3ZWuScusSDMRbZM=; fh=1GJaSiew/4nhYvae+yJYXPcARWi0PNZhXd1mB5ASXRg=; b=NZMqFelvxX6BerNouI230PFOmYIxyguCHia9zagchtZGulHygVnSSt+OusgMHCTvK9 6ktgbzS1QxqSjyPVGynIfx0E2g5jcXDCPbIgzXtNQmn2ccBUUFosbR2JllaNlab3jE4V YzHyhQrNP/mJvh217BS75OhekiUGWef7/vzoE9R6ssXw0lQPiW5GQX5rtWHrwAamWvhW bqmDgz9WcHKFdn/rhjEcjwhvJCpYxhQJIJklYWoeu7mfuCN78o9eh3/aAR9o2VeBsaEj uDqGdQi6tlj5ZOfJPoIX/XO8UUoUGlXfvwN+vujIUl9eCh6TUhn8WGkbTnA/PGU1Rcq9 LxaA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=EgWg7Jyu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id bv70-20020a632e49000000b005b88dfa676esi1051719pgb.158.2023.10.31.07.10.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Oct 2023 07:10:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=EgWg7Jyu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id EB7D08088A62; Tue, 31 Oct 2023 07:10:20 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235780AbjJaOJd (ORCPT + 33 others); Tue, 31 Oct 2023 10:09:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34978 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234896AbjJaOJY (ORCPT ); Tue, 31 Oct 2023 10:09:24 -0400 Received: from out-175.mta1.migadu.com (out-175.mta1.migadu.com [95.215.58.175]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2B0A183 for ; Tue, 31 Oct 2023 07:09:19 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698761358; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=71jccDuvmHUN0B2hAN0chVUZ3a/I3ZWuScusSDMRbZM=; b=EgWg7JyuljiAgmtHeNxZPVDCEzu19y0onJXhXs8aae8cO3ZMQHroKY26QyUFiFaebUhzCc RGV6bWjUO05MthcYp2hTuKmvk++h80PzPUQqKT6vJe0w3lTBc8M2DdpDA2JE+ug45MtF26 X/0C77Wy9EZYl/zFaNFqEn7aLM2cGT8= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, willy@infradead.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v4 7/9] slub: Optimize deactivate_slab() Date: Tue, 31 Oct 2023 14:07:39 +0000 Message-Id: <20231031140741.79387-8-chengming.zhou@linux.dev> In-Reply-To: <20231031140741.79387-1-chengming.zhou@linux.dev> References: <20231031140741.79387-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Tue, 31 Oct 2023 07:10:21 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781280479739208327 X-GMAIL-MSGID: 1781280479739208327 From: Chengming Zhou Since the introduce of unfrozen slabs on cpu partial list, we don't need to synchronize the slab frozen state under the node list_lock. The caller of deactivate_slab() and the caller of __slab_free() won't manipulate the slab list concurrently. So we can get node list_lock in the last stage if we really need to manipulate the slab list in this path. Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka --- mm/slub.c | 76 +++++++++++++++++++------------------------------------ 1 file changed, 26 insertions(+), 50 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index bcb5b2c4e213..c429f8baba5f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2468,10 +2468,8 @@ static void init_kmem_cache_cpus(struct kmem_cache *s) static void deactivate_slab(struct kmem_cache *s, struct slab *slab, void *freelist) { - enum slab_modes { M_NONE, M_PARTIAL, M_FREE, M_FULL_NOLIST }; struct kmem_cache_node *n = get_node(s, slab_nid(slab)); int free_delta = 0; - enum slab_modes mode = M_NONE; void *nextfree, *freelist_iter, *freelist_tail; int tail = DEACTIVATE_TO_HEAD; unsigned long flags = 0; @@ -2512,62 +2510,40 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, * * Ensure that the slab is unfrozen while the list presence * reflects the actual number of objects during unfreeze. - * - * We first perform cmpxchg holding lock and insert to list - * when it succeed. If there is mismatch then the slab is not - * unfrozen and number of objects in the slab may have changed. - * Then release lock and retry cmpxchg again. */ -redo: - - old.freelist = READ_ONCE(slab->freelist); - old.counters = READ_ONCE(slab->counters); - VM_BUG_ON(!old.frozen); - - /* Determine target state of the slab */ - new.counters = old.counters; - if (freelist_tail) { - new.inuse -= free_delta; - set_freepointer(s, freelist_tail, old.freelist); - new.freelist = freelist; - } else - new.freelist = old.freelist; - - new.frozen = 0; + do { + old.freelist = READ_ONCE(slab->freelist); + old.counters = READ_ONCE(slab->counters); + VM_BUG_ON(!old.frozen); + + /* Determine target state of the slab */ + new.counters = old.counters; + new.frozen = 0; + if (freelist_tail) { + new.inuse -= free_delta; + set_freepointer(s, freelist_tail, old.freelist); + new.freelist = freelist; + } else { + new.freelist = old.freelist; + } + } while (!slab_update_freelist(s, slab, + old.freelist, old.counters, + new.freelist, new.counters, + "unfreezing slab")); + /* + * Stage three: Manipulate the slab list based on the updated state. + */ if (!new.inuse && n->nr_partial >= s->min_partial) { - mode = M_FREE; + stat(s, DEACTIVATE_EMPTY); + discard_slab(s, slab); + stat(s, FREE_SLAB); } else if (new.freelist) { - mode = M_PARTIAL; - /* - * Taking the spinlock removes the possibility that - * acquire_slab() will see a slab that is frozen - */ spin_lock_irqsave(&n->list_lock, flags); - } else { - mode = M_FULL_NOLIST; - } - - - if (!slab_update_freelist(s, slab, - old.freelist, old.counters, - new.freelist, new.counters, - "unfreezing slab")) { - if (mode == M_PARTIAL) - spin_unlock_irqrestore(&n->list_lock, flags); - goto redo; - } - - - if (mode == M_PARTIAL) { add_partial(n, slab, tail); spin_unlock_irqrestore(&n->list_lock, flags); stat(s, tail); - } else if (mode == M_FREE) { - stat(s, DEACTIVATE_EMPTY); - discard_slab(s, slab); - stat(s, FREE_SLAB); - } else if (mode == M_FULL_NOLIST) { + } else { stat(s, DEACTIVATE_FULL); } } From patchwork Tue Oct 31 14:07:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160133 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b90f:0:b0:403:3b70:6f57 with SMTP id t15csp267981vqg; Tue, 31 Oct 2023 07:10:08 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFwlhY3L1i2vfHNC30P0k43767zOkU9Hb+ya37I/yVdQcHClhz07/c72vcxhYNN7L4qvRUm X-Received: by 2002:a17:902:e0c5:b0:1ca:8b90:1cbd with SMTP id e5-20020a170902e0c500b001ca8b901cbdmr9516065pla.0.1698761408126; Tue, 31 Oct 2023 07:10:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698761408; cv=none; d=google.com; s=arc-20160816; b=mTMIFsKJqME4brurb1ddxSKNn2pHurN5Og8A0pVcKOWBtjeHwl12CxPjoGfpL94RQ8 afdY3TnaH6wOaB/RgMG+Fu1Vj2ZVdXLGic1PJ8hzWEaVAb7wDCEuM2PHpeVlUexCi4DP xYlnbM87qSCjp6QV4tS1ikVtLPqNmaEEOH04I83G8wWP2KbcFPdf77ulgkMwl5Cf9Kzu oPRycueHxiFVBG3xrMyrnDK1ZbCm9bv0Fw2VKe6HOhLwDN810zPe1y+0a6V6cbKqUySh E8gon87fIgnk1lEPZ10nJdn8Gwev15afz5PDmTvxcgWH+FGHTy4gOvHATwzQXnF/jhm+ X7aw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=WNirFUhouaQ2FdRmy2RjQgkCONe7ugTNtD0fPI9SkvQ=; fh=1GJaSiew/4nhYvae+yJYXPcARWi0PNZhXd1mB5ASXRg=; b=Ecpd8Txim0Q7WUnJN5Lm8GL8oBUpHnrM0+N2g20CTSLmmMLaqhDfxUXMK+APCq9DOG A86gByfmlzlx78wu3wgjO49mhw7Ts+IZme92HqgTA4eMgbIVsN0+3yYtO69D2VcSij4f k/zSFZxbdkrirhFvcpKFvKhfLqJAJcQ3XP0Spa8nDl7Nf5KUppMXk+9FiwDq9ospMNpi LYqC3iPycn+ARm++ly2An48h2BfLD7D/i3NkrUkPLdlf2gccXebjKR3p7WfWJjDd5i9/ Rt1XwuVkxwjCHUO2zOGXE1QSXjl8YcIZNWO4/ErDr4X/NWfKd7/7jxd9qrncY70urEH/ hflQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=r1MiH4vM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id lb7-20020a170902fa4700b001ca4e2a35efsi1039029plb.45.2023.10.31.07.10.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Oct 2023 07:10:08 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=r1MiH4vM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 7EEA6802B00D; Tue, 31 Oct 2023 07:10:05 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236046AbjJaOJm (ORCPT + 33 others); Tue, 31 Oct 2023 10:09:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35090 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235514AbjJaOJZ (ORCPT ); Tue, 31 Oct 2023 10:09:25 -0400 Received: from out-179.mta1.migadu.com (out-179.mta1.migadu.com [95.215.58.179]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B79B9121 for ; Tue, 31 Oct 2023 07:09:22 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698761361; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WNirFUhouaQ2FdRmy2RjQgkCONe7ugTNtD0fPI9SkvQ=; b=r1MiH4vM84Jj3nwHog6emae9i1zx3YqzKMYp0IBEloU7g/8tB45Y1H9zMFiMKQbkD/H0Qe VuvBkAF8ea4irChfcbAhAq0tXL092g71ipfL7eKZwrga14ekhl47LZcRSrMGn4jQOfzSeP bhCSvGRNAWF820iIR0MuX+4oZgKIP/E= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, willy@infradead.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v4 8/9] slub: Rename all *unfreeze_partials* functions to *put_partials* Date: Tue, 31 Oct 2023 14:07:40 +0000 Message-Id: <20231031140741.79387-9-chengming.zhou@linux.dev> In-Reply-To: <20231031140741.79387-1-chengming.zhou@linux.dev> References: <20231031140741.79387-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Tue, 31 Oct 2023 07:10:05 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781280442619271850 X-GMAIL-MSGID: 1781280442619271850 From: Chengming Zhou Since all partial slabs on the CPU partial list are not frozen anymore, we don't unfreeze when moving cpu partial slabs to node partial list, it's better to rename these functions. Signed-off-by: Chengming Zhou Reviewed-by: Vlastimil Babka --- mm/slub.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index c429f8baba5f..bb7368047103 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2549,7 +2549,7 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, } #ifdef CONFIG_SLUB_CPU_PARTIAL -static void __unfreeze_partials(struct kmem_cache *s, struct slab *partial_slab) +static void put_partials_node(struct kmem_cache *s, struct slab *partial_slab) { struct kmem_cache_node *n = NULL, *n2 = NULL; struct slab *slab, *slab_to_discard = NULL; @@ -2591,9 +2591,9 @@ static void __unfreeze_partials(struct kmem_cache *s, struct slab *partial_slab) } /* - * Unfreeze all the cpu partial slabs. + * Put all the cpu partial slabs to the node partial list. */ -static void unfreeze_partials(struct kmem_cache *s) +static void put_partials(struct kmem_cache *s) { struct slab *partial_slab; unsigned long flags; @@ -2604,11 +2604,11 @@ static void unfreeze_partials(struct kmem_cache *s) local_unlock_irqrestore(&s->cpu_slab->lock, flags); if (partial_slab) - __unfreeze_partials(s, partial_slab); + put_partials_node(s, partial_slab); } -static void unfreeze_partials_cpu(struct kmem_cache *s, - struct kmem_cache_cpu *c) +static void put_partials_cpu(struct kmem_cache *s, + struct kmem_cache_cpu *c) { struct slab *partial_slab; @@ -2616,7 +2616,7 @@ static void unfreeze_partials_cpu(struct kmem_cache *s, c->partial = NULL; if (partial_slab) - __unfreeze_partials(s, partial_slab); + put_partials_node(s, partial_slab); } /* @@ -2629,7 +2629,7 @@ static void unfreeze_partials_cpu(struct kmem_cache *s, static void put_cpu_partial(struct kmem_cache *s, struct slab *slab, int drain) { struct slab *oldslab; - struct slab *slab_to_unfreeze = NULL; + struct slab *slab_to_put = NULL; unsigned long flags; int slabs = 0; @@ -2644,7 +2644,7 @@ static void put_cpu_partial(struct kmem_cache *s, struct slab *slab, int drain) * per node partial list. Postpone the actual unfreezing * outside of the critical section. */ - slab_to_unfreeze = oldslab; + slab_to_put = oldslab; oldslab = NULL; } else { slabs = oldslab->slabs; @@ -2660,17 +2660,17 @@ static void put_cpu_partial(struct kmem_cache *s, struct slab *slab, int drain) local_unlock_irqrestore(&s->cpu_slab->lock, flags); - if (slab_to_unfreeze) { - __unfreeze_partials(s, slab_to_unfreeze); + if (slab_to_put) { + put_partials_node(s, slab_to_put); stat(s, CPU_PARTIAL_DRAIN); } } #else /* CONFIG_SLUB_CPU_PARTIAL */ -static inline void unfreeze_partials(struct kmem_cache *s) { } -static inline void unfreeze_partials_cpu(struct kmem_cache *s, - struct kmem_cache_cpu *c) { } +static inline void put_partials(struct kmem_cache *s) { } +static inline void put_partials_cpu(struct kmem_cache *s, + struct kmem_cache_cpu *c) { } #endif /* CONFIG_SLUB_CPU_PARTIAL */ @@ -2712,7 +2712,7 @@ static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) stat(s, CPUSLAB_FLUSH); } - unfreeze_partials_cpu(s, c); + put_partials_cpu(s, c); } struct slub_flush_work { @@ -2740,7 +2740,7 @@ static void flush_cpu_slab(struct work_struct *w) if (c->slab) flush_slab(s, c); - unfreeze_partials(s); + put_partials(s); } static bool has_cpu_slab(int cpu, struct kmem_cache *s) @@ -3171,7 +3171,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, if (unlikely(!node_match(slab, node) || !pfmemalloc_match(slab, gfpflags))) { slab->next = NULL; - __unfreeze_partials(s, slab); + put_partials_node(s, slab); continue; } From patchwork Tue Oct 31 14:07:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 160132 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b90f:0:b0:403:3b70:6f57 with SMTP id t15csp267864vqg; Tue, 31 Oct 2023 07:10:00 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFCkMSrYm4EKd8AttynTNvPiLeelm9EvvAPNm2S3UZ0usPSk1pM1k1DBwt83Rt2xeIfWKpm X-Received: by 2002:a17:902:f9ce:b0:1ca:a07b:36d with SMTP id kz14-20020a170902f9ce00b001caa07b036dmr10185317plb.48.1698761400541; Tue, 31 Oct 2023 07:10:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698761400; cv=none; d=google.com; s=arc-20160816; b=uxY6BJq4KkkoWbJQwB1New62W+G1J7EHIdDyuA5PJZMSsQtWiWHqTr4RbfCEDvgfFO M4uvs+hjL5H4G5kmLn3M1oLFxTryXNIH7WXDC5jiLhyanRzV878ethAVShY5dYQUA2U3 0FrzXcMaZN85sGqLsynWJlEKdN6vmUCHDXeVxJ0d6lKYV8m4K2UjNgcsJdZhyDqFtOdm nagcAg4MWkyAlOb2poPWJlzMyBVaPBs4RKX1b5YTet7XNrTVd2rqNfd2h3/Kf5iyEI0x Gdm6M7YMHB9HFObUEWxOWrA+AkveZHhwXMLd47uZ/rKGHzXL6BKqD8z4IcmV4KguXh2T GGAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1rWlR4h5g4eDWf2XTzUlOrJLNxfEOq6mT+00ilTz0G0=; fh=1GJaSiew/4nhYvae+yJYXPcARWi0PNZhXd1mB5ASXRg=; b=qwVGt+T70LcpVz5TnfVK+XUcgK/QZz+Is8PuaosTIqltEXvg2E52KtXg4+zQI2sffX KFwyU3LWd96Q0WYNbyyWDkHZFRHjr9aAPSQHLbC08IzKOpT7CLtS0DxCHdIsO7eKNxvm Xu11HNC2Gu3yppz8uV0ATDXeZImfopCWZ0qSdS251HMAwt+jKe4Yspwb6+OOTKxWhR1+ 9GnWso2G3f0i+6Z4Ao7NPINSQH4sFDrL3Lo6WxxAfj89jGAZH7gNMDfNkxjSdXXjhbri f2iwr3S0lsIaLDYaBVH8x13l7gfJpOXOdA7NuOwH6OE0DGmhcavylMfiQkUIltzqv54N 4SSg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Y0fRMVJb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id j1-20020a170902da8100b001c60636e426si1063701plx.432.2023.10.31.07.09.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 31 Oct 2023 07:10:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Y0fRMVJb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 003D68037AA3; Tue, 31 Oct 2023 07:09:57 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236130AbjJaOJp (ORCPT + 33 others); Tue, 31 Oct 2023 10:09:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34810 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235747AbjJaOJb (ORCPT ); Tue, 31 Oct 2023 10:09:31 -0400 Received: from out-181.mta1.migadu.com (out-181.mta1.migadu.com [IPv6:2001:41d0:203:375::b5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 26E4019E for ; Tue, 31 Oct 2023 07:09:25 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698761363; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1rWlR4h5g4eDWf2XTzUlOrJLNxfEOq6mT+00ilTz0G0=; b=Y0fRMVJblxItnO7fa29gryhY0OuDSuYpX3u9Ss3dhqwRf1OZ/GJBzbFxGqASfXa3ZkcDEd llhl/Boestl6/Pzbcckw30/f5cm8ZIVtqAPveVEahzHWy0Mm0D45/0Br2r/TLM+w+opTst LbLFwmWZyebAWLmuYs8XzdN4+enFRJY= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, willy@infradead.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v4 9/9] slub: Update frozen slabs documentations in the source Date: Tue, 31 Oct 2023 14:07:41 +0000 Message-Id: <20231031140741.79387-10-chengming.zhou@linux.dev> In-Reply-To: <20231031140741.79387-1-chengming.zhou@linux.dev> References: <20231031140741.79387-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Tue, 31 Oct 2023 07:09:58 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781280434110941845 X-GMAIL-MSGID: 1781280434110941845 From: Chengming Zhou The current updated scheme (which this series implemented) is: - node partial slabs: PG_Workingset && !frozen - cpu partial slabs: !PG_Workingset && !frozen - cpu slabs: !PG_Workingset && frozen - full slabs: !PG_Workingset && !frozen The most important change is that "frozen" bit is not set for the cpu partial slabs anymore, __slab_free() will grab node list_lock then check by !PG_Workingset that it's not on a node partial list. And the "frozen" bit is still kept for the cpu slabs for performance, since we don't need to grab node list_lock to check whether the PG_Workingset is set or not if the "frozen" bit is set in __slab_free(). Update related documentations and comments in the source. Signed-off-by: Chengming Zhou --- mm/slub.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index bb7368047103..89d3f7a18a73 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -76,13 +76,22 @@ * * Frozen slabs * - * If a slab is frozen then it is exempt from list management. It is not - * on any list except per cpu partial list. The processor that froze the + * If a slab is frozen then it is exempt from list management. It is + * the cpu slab which is actively allocated from by the processor that + * froze it and it is not on any list. The processor that froze the * slab is the one who can perform list operations on the slab. Other * processors may put objects onto the freelist but the processor that * froze the slab is the only one that can retrieve the objects from the * slab's freelist. * + * CPU partial slabs + * + * The partially empty slabs cached on the CPU partial list are used + * for performance reasons, which speeds up the allocation process. + * These slabs are not frozen, but also exempt from list management, + * by clearing the PG_workingset flag when moving out of the node + * partial list. Please see __slab_free() for more details. + * * list_lock * * The list_lock protects the partial and full list on each node and @@ -2620,8 +2629,7 @@ static void put_partials_cpu(struct kmem_cache *s, } /* - * Put a slab that was just frozen (in __slab_free|get_partial_node) into a - * partial slab slot if available. + * Put a slab into a partial slab slot if available. * * If we did not find a slot then simply move all the partials to the * per node partial list.