From patchwork Sat Oct 21 14:43:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 156454 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce89:0:b0:403:3b70:6f57 with SMTP id p9csp328322vqx; Sat, 21 Oct 2023 07:44:39 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHV74vheAFPJeR6M1y/DbfJusb2GDT823RpcueBfxFQja/q0PSGpnBrTUDYgjr+FNlyDUhv X-Received: by 2002:a62:7911:0:b0:68b:e29c:b69 with SMTP id u17-20020a627911000000b0068be29c0b69mr5935125pfc.9.1697899479527; Sat, 21 Oct 2023 07:44:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697899479; cv=none; d=google.com; s=arc-20160816; b=kg0HhjHpBfzfRLY3Jf8nJdzyvyZg1InUvylvc5kK0TmWplQeP5OgCNGRGN4gGe0XXv 3W1HR6/u5Qhtu+ssDnu+0I4bDHPEBfzJ2k743JvPBj/21GD6yFKPVrzgeC0RjXmHmB6L MmWwbi9hT0d4C/SQItnjhNqoOt1NJHqJYvYDmr4YIOwTTIffpC9NpwDh8ett93flMDVC QnsBIT/gWy6I6g6+zi0JZNSZetY3uA0fcV+pOzioMAFXwlctMQCAf6+q+7TMgh8cPYeT DAB8Uls8SKzWaeVaFOxR1sfMG/qvcwwkiZ5keqtcjJsgnp8ArFkV4MmaFq3Kje6zCudj 7HGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=McUlAOC6AXMF3DQxwjxFsXChYXBELSqYw7TfeM0+yv0=; fh=SudcMr3Ofqd+t62q757mMk2HbWTWI1wXv43XX7ahhjA=; b=tLwTYgLPJbclsYIRZQ711H20Y8k7k9DEZ5xenIO62DlkpR/MF0K4r6LrlUv22HRPHh 1iW3AzPG5BXBr/ujqX8VA+ZDE75qI3IbANucuukr8lCfVatdUPfms6DQGFG4l5jxoL9v ZJNzZinr9VKNDFiW4TDxUaEV2VoM7zhtZI6cK1GT8Ywt+ruY7YbVfJd7wFuqeRFLh011 2sqpxOnZiN1tmfQ9TeR+mPAKx8VZHD7cTWWo6rura/99G/WSaRhd0wS7TM+YAa58EuOL OaT4u9ff2XjcEVS8rgjlPQMA6DtReX8K3khB4LK/DeFh+vRWTD0AK/QUBG/55YQU2huJ A2rw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=U7c03tZf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id w15-20020a637b0f000000b0056546b5fef4si3676746pgc.232.2023.10.21.07.44.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Oct 2023 07:44:39 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=U7c03tZf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 8B6848047D76; Sat, 21 Oct 2023 07:44:29 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231409AbjJUOoB (ORCPT + 26 others); Sat, 21 Oct 2023 10:44:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59066 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229588AbjJUOoA (ORCPT ); Sat, 21 Oct 2023 10:44:00 -0400 Received: from out-196.mta1.migadu.com (out-196.mta1.migadu.com [IPv6:2001:41d0:203:375::c4]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ACA95C0 for ; Sat, 21 Oct 2023 07:43:57 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1697899435; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=McUlAOC6AXMF3DQxwjxFsXChYXBELSqYw7TfeM0+yv0=; b=U7c03tZf2fCoW5T3wE2WN887d5Ut/z0dqbsdzuHPFwfuTnJjKkXIqgu1qkz/ZM1D7dvN/0 +t0ZPZQ04YUwljOgsVlPl0zjwouHH5C+41xLj3a5pQygz/jzSeLcr3C5ImZ/LyIGOBMkuM Zx55Qopachdrboij0Lf1+nGLH57UfVI= From: chengming.zhou@linux.dev To: cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, willy@infradead.org, pcc@google.com, tytso@mit.edu, maz@kernel.org, ruansy.fnst@fujitsu.com, vishal.moola@gmail.com, lrh2000@pku.edu.cn, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v2 1/6] slub: Keep track of whether slub is on the per-node partial list Date: Sat, 21 Oct 2023 14:43:12 +0000 Message-Id: <20231021144317.3400916-2-chengming.zhou@linux.dev> In-Reply-To: <20231021144317.3400916-1-chengming.zhou@linux.dev> References: <20231021144317.3400916-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Sat, 21 Oct 2023 07:44:29 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780376644579783622 X-GMAIL-MSGID: 1780376644579783622 From: Chengming Zhou Now we rely on the "frozen" bit to see if we should manipulate the slab->slab_list, which will be changed in the following patch. Instead we introduce another way to keep track of whether slub is on the per-node partial list, here we reuse the PG_workingset bit. Signed-off-by: Chengming Zhou --- include/linux/page-flags.h | 2 ++ mm/slab.h | 19 +++++++++++++++++++ mm/slub.c | 3 +++ 3 files changed, 24 insertions(+) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index a88e64acebfe..e8b1be71d722 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -478,6 +478,8 @@ PAGEFLAG(Active, active, PF_HEAD) __CLEARPAGEFLAG(Active, active, PF_HEAD) TESTCLEARFLAG(Active, active, PF_HEAD) PAGEFLAG(Workingset, workingset, PF_HEAD) TESTCLEARFLAG(Workingset, workingset, PF_HEAD) + __SETPAGEFLAG(Workingset, workingset, PF_HEAD) + __CLEARPAGEFLAG(Workingset, workingset, PF_HEAD) __PAGEFLAG(Slab, slab, PF_NO_TAIL) PAGEFLAG(Checked, checked, PF_NO_COMPOUND) /* Used by some filesystems */ diff --git a/mm/slab.h b/mm/slab.h index 8cd3294fedf5..9cff64cae8de 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -193,6 +193,25 @@ static inline void __slab_clear_pfmemalloc(struct slab *slab) __folio_clear_active(slab_folio(slab)); } +/* + * Slub reuse PG_workingset bit to keep track of whether it's on + * the per-node partial list. + */ +static inline bool slab_test_node_partial(const struct slab *slab) +{ + return folio_test_workingset((struct folio *)slab_folio(slab)); +} + +static inline void slab_set_node_partial(struct slab *slab) +{ + __folio_set_workingset(slab_folio(slab)); +} + +static inline void slab_clear_node_partial(struct slab *slab) +{ + __folio_clear_workingset(slab_folio(slab)); +} + static inline void *slab_address(const struct slab *slab) { return folio_address(slab_folio(slab)); diff --git a/mm/slub.c b/mm/slub.c index 63d281dfacdb..3fad4edca34b 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2127,6 +2127,7 @@ __add_partial(struct kmem_cache_node *n, struct slab *slab, int tail) list_add_tail(&slab->slab_list, &n->partial); else list_add(&slab->slab_list, &n->partial); + slab_set_node_partial(slab); } static inline void add_partial(struct kmem_cache_node *n, @@ -2141,6 +2142,7 @@ static inline void remove_partial(struct kmem_cache_node *n, { lockdep_assert_held(&n->list_lock); list_del(&slab->slab_list); + slab_clear_node_partial(slab); n->nr_partial--; } @@ -4831,6 +4833,7 @@ static int __kmem_cache_do_shrink(struct kmem_cache *s) if (free == slab->objects) { list_move(&slab->slab_list, &discard); + slab_clear_node_partial(slab); n->nr_partial--; dec_slabs_node(s, node, slab->objects); } else if (free <= SHRINK_PROMOTE_MAX) From patchwork Sat Oct 21 14:43:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 156457 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce89:0:b0:403:3b70:6f57 with SMTP id p9csp329041vqx; Sat, 21 Oct 2023 07:46:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEabA79+sbBh1KK8whjsyOmaLiXxHmiEv3ddzx/a7Ft+S6TIW7tKwDnjokCaUMmRo4C6Lmp X-Received: by 2002:a05:6358:6383:b0:13f:411:c1a9 with SMTP id k3-20020a056358638300b0013f0411c1a9mr5839259rwh.17.1697899562498; Sat, 21 Oct 2023 07:46:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697899562; cv=none; d=google.com; s=arc-20160816; b=QLniYTi9o4iqvYOPqYJc+dVW+kVW2QYbqeNebMGQkYr+oZNchlHp1qxbAojwD0Xiab QOzrLZqm3nMxOr/lqfPt3bRUcbYZhLj8e9Jx4FFAnHKNze60Wx6+Go0fAAS0PednyWyW vl/tW3/clwnUq96GSSG78Y7uZhEKD1vlLlNCZPBD2MPFXnv4XBWmo8Q7Jf7FG3kkMKfD oZUhoQvlAEu+UtDvpWltBEj5gnXqfCrDpglF2eJgKv97XeJEPjL7GdxoCgMMLU+Ykk4p qUACNcuwQJEVb4ii9V0EnX+CBQTJYgTvmJvv5P925ObwyUwJLQdeLf/v9nzQBNmXt35a NHbg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5cQE9ZFnILnFxpwPZY6R/Y63Il/vf0eHaCXoNa8ST4o=; fh=SudcMr3Ofqd+t62q757mMk2HbWTWI1wXv43XX7ahhjA=; b=d2nkbFCaZ2Maryi7QTbSWkqBleZhKsVKdhKneGVhqTbTKLkh+qf+8dJZXlxFKa1ex0 aC/B1EnZgTWTvGXzdM2PklQ14rh8riV6lAp8IkWHVHxpsCzHbg+MYHD4JTqqR73Xi/ce IqleG1NNA4vXbZkZELBGrd43dhtfWgk+SrlifyNyxWdRwcsXGcmv1YlR2EG1/7VrsGe3 sULi38CCXM0IHH1AiLV4tcqBpdvRhjcGokirnKk34hx4mV3j7HbLCHNqBpkcrlsLORPy yFQdsjzMnk5SyVdKXl/IgEh5AQRWV5JC7UVKs0uJNk7KDvQlSdMWfJ9CdExQvUQ4IrmA 4okQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=xX8mHLDH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id b26-20020a63931a000000b0056fa1ac3722si3641270pge.698.2023.10.21.07.46.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Oct 2023 07:46:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=xX8mHLDH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 4FC3E807C745; Sat, 21 Oct 2023 07:45:13 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231436AbjJUOoM (ORCPT + 26 others); Sat, 21 Oct 2023 10:44:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46166 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229588AbjJUOoK (ORCPT ); Sat, 21 Oct 2023 10:44:10 -0400 Received: from out-209.mta1.migadu.com (out-209.mta1.migadu.com [IPv6:2001:41d0:203:375::d1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F13E7D52 for ; Sat, 21 Oct 2023 07:44:04 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1697899443; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5cQE9ZFnILnFxpwPZY6R/Y63Il/vf0eHaCXoNa8ST4o=; b=xX8mHLDHA6jFLEhdUxuEf1tXL2PZcg10QlIE6EE+u+VIO1Ec/ICxJ/WjowvxWiwRTrQeqm rLm5/Uzh0gR0aIfDdIxK6teeg/FWiuTnzKUIMz43WCQ8Awhnrqp5HujqEJ+KKY2sWYUBDV RSgUvb/mgs0boYQF+AOgd4q17KMw5RI= From: chengming.zhou@linux.dev To: cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, willy@infradead.org, pcc@google.com, tytso@mit.edu, maz@kernel.org, ruansy.fnst@fujitsu.com, vishal.moola@gmail.com, lrh2000@pku.edu.cn, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v2 2/6] slub: Prepare __slab_free() for unfrozen partial slab out of node partial list Date: Sat, 21 Oct 2023 14:43:13 +0000 Message-Id: <20231021144317.3400916-3-chengming.zhou@linux.dev> In-Reply-To: <20231021144317.3400916-1-chengming.zhou@linux.dev> References: <20231021144317.3400916-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Sat, 21 Oct 2023 07:45:13 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780376731122996351 X-GMAIL-MSGID: 1780376731122996351 From: Chengming Zhou Now the partial slub will be frozen when taken out of node partial list, so the __slab_free() will know from "was_frozen" that the partial slab is not on node partial list and is used by one kmem_cache_cpu. But we will change this, make partial slabs leave the node partial list with unfrozen state, so we need to change __slab_free() to use the new slab_test_node_partial() we just introduced. Signed-off-by: Chengming Zhou --- mm/slub.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/mm/slub.c b/mm/slub.c index 3fad4edca34b..adeff8df85ec 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3610,6 +3610,7 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, unsigned long counters; struct kmem_cache_node *n = NULL; unsigned long flags; + bool on_node_partial; stat(s, FREE_SLOWPATH); @@ -3657,6 +3658,7 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, */ spin_lock_irqsave(&n->list_lock, flags); + on_node_partial = slab_test_node_partial(slab); } } @@ -3685,6 +3687,15 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, return; } + /* + * This slab was not full and not on the per-node partial list either, + * in which case we shouldn't manipulate its list, just early return. + */ + if (prior && !on_node_partial) { + spin_unlock_irqrestore(&n->list_lock, flags); + return; + } + if (unlikely(!new.inuse && n->nr_partial >= s->min_partial)) goto slab_empty; From patchwork Sat Oct 21 14:43:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 156458 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce89:0:b0:403:3b70:6f57 with SMTP id p9csp329042vqx; Sat, 21 Oct 2023 07:46:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFOUiH9KPmrAGt/Oh8g/22OEFjXxHbJi1Xnb17hoE9RucLw5qyeQ9pnxtBFjs3x1+BV8/Ql X-Received: by 2002:a05:6a21:9981:b0:17a:f4b6:bf89 with SMTP id ve1-20020a056a21998100b0017af4b6bf89mr5558974pzb.31.1697899562502; Sat, 21 Oct 2023 07:46:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697899562; cv=none; d=google.com; s=arc-20160816; b=dUkj6agq3mGwVoW1O8MYtcLtfjsR2HZuF4Nzvr3NOP13Fm081PhXlL9/eTKjGxI0eE J8MRpDxFgLiCEucQdJennqgHrcklPkB3V/RZqfu37nbeMrwaBeYk/dxM4qOomdM6WH92 dDevYX8HM2sAncfgIod6aGM8kXIVn//ANGuL6JBlW1RX9PgkDZ5jHlaxeszoGMlJH0yg stpLIYe2jhDjWAd7OCc6RzaCuagacht+punzSMx9/4TXsxP0+UfAP/O4+fYkXeNU3TrO b67qvNAX9/Q0HWEJMXynS60DM2gAEb3GoMEaTtsgFKp3H4fgDNb8sTokl3FQPzdQLLen 8uzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Fc6duZ/q+apDqgoP4cLzy3p5t8cD8/eKarQEAhD6cwA=; fh=SudcMr3Ofqd+t62q757mMk2HbWTWI1wXv43XX7ahhjA=; b=yV9j29FKzh5K1QzZtoSDvJydV2xcIRswDT3znxxBD/9trYBfXuTwQ7IQIcfI7/QnXa Wup5pot1c9eTIb+3y+CGEJsCm30Vy39jGdfvR8/pMbRh6pNDxLUkxDSJ66dfQYr0bN09 Hvovup4+Ph8jchcH7lDinQlgyyKKRfA5vzHTt5iD131GmI3l8/cg6KWATTAGUcEnrsd3 WW76EMsuUUENp/FYeD4uk4GzPFuLCCNE7M/2hbKUjt25G8urTj7XBom8XqpoEtGItv/A l4rdImbuDSrcg2aeI328D5nSRV8fRD+inzKXpn8O2AgKhRHruIvFE8d/RxP3UdPGd6Tr yUBQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=VwdHbGN6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id by38-20020a056a0205a600b0057795cb4f16si3935724pgb.684.2023.10.21.07.46.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Oct 2023 07:46:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=VwdHbGN6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id C30B6807C649; Sat, 21 Oct 2023 07:45:30 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231422AbjJUOoT (ORCPT + 26 others); Sat, 21 Oct 2023 10:44:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40102 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231429AbjJUOoR (ORCPT ); Sat, 21 Oct 2023 10:44:17 -0400 Received: from out-210.mta1.migadu.com (out-210.mta1.migadu.com [IPv6:2001:41d0:203:375::d2]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8CBC0D7C for ; Sat, 21 Oct 2023 07:44:13 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1697899451; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fc6duZ/q+apDqgoP4cLzy3p5t8cD8/eKarQEAhD6cwA=; b=VwdHbGN6iqPqOPPmjkoCZCqeya5Wrnfy29DfCh17oso4mTQaIrjGQ/bBj7cuImP6Ak0ZDt 74Ji4pLY0KWggdSHOhw8II53OTlr2K3tbgg2NViY04lO63dYbYrl8s/FsQ+pKx4DbOjdEG yYcnIac3YU/gH6s55G+958BKYZfu70Y= From: chengming.zhou@linux.dev To: cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, willy@infradead.org, pcc@google.com, tytso@mit.edu, maz@kernel.org, ruansy.fnst@fujitsu.com, vishal.moola@gmail.com, lrh2000@pku.edu.cn, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v2 3/6] slub: Don't freeze slabs for cpu partial Date: Sat, 21 Oct 2023 14:43:14 +0000 Message-Id: <20231021144317.3400916-4-chengming.zhou@linux.dev> In-Reply-To: <20231021144317.3400916-1-chengming.zhou@linux.dev> References: <20231021144317.3400916-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Sat, 21 Oct 2023 07:45:30 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780376731252078086 X-GMAIL-MSGID: 1780376731252078086 From: Chengming Zhou Now we will freeze slabs when moving them out of node partial list to cpu partial list, this method needs two cmpxchg_double operations: 1. freeze slab (acquire_slab()) under the node list_lock 2. get_freelist() when pick used in ___slab_alloc() Actually we don't need to freeze when moving slabs out of node partial list, we can delay freeze to use slab freelist in ___slab_alloc(), so we can save one cmpxchg_double(). And there are other good points: 1. The moving of slabs between node partial list and cpu partial list becomes simpler, since we don't need to freeze or unfreeze at all. 2. The node list_lock contention would be less, since we only need to freeze one slab under the node list_lock. (In fact, we can first move slabs out of node partial list, don't need to freeze any slab at all, so the contention on slab won't transfer to the node list_lock contention.) We can achieve this because there is no concurrent path would manipulate the partial slab list except the __slab_free() path, which is serialized now. Note this patch just change the parts of moving the partial slabs for easy code review, we will fix other parts in the following patches. Specifically this patch change three paths: 1. get partial slab from node: get_partial_node() 2. put partial slab to node: __unfreeze_partials() 3. cache partail slab on cpu when __slab_free() Signed-off-by: Chengming Zhou --- mm/slub.c | 63 +++++++++++++++++-------------------------------------- 1 file changed, 19 insertions(+), 44 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index adeff8df85ec..61ee82ea21b6 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2277,7 +2277,9 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, struct slab *slab, *slab2; void *object = NULL; unsigned long flags; +#ifdef CONFIG_SLUB_CPU_PARTIAL unsigned int partial_slabs = 0; +#endif /* * Racy check. If we mistakenly see no partial slabs then we @@ -2303,20 +2305,22 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, continue; } - t = acquire_slab(s, n, slab, object == NULL); - if (!t) - break; - if (!object) { - *pc->slab = slab; - stat(s, ALLOC_FROM_PARTIAL); - object = t; - } else { - put_cpu_partial(s, slab, 0); - stat(s, CPU_PARTIAL_NODE); - partial_slabs++; + t = acquire_slab(s, n, slab, object == NULL); + if (t) { + *pc->slab = slab; + stat(s, ALLOC_FROM_PARTIAL); + object = t; + continue; + } } + #ifdef CONFIG_SLUB_CPU_PARTIAL + remove_partial(n, slab); + put_cpu_partial(s, slab, 0); + stat(s, CPU_PARTIAL_NODE); + partial_slabs++; + if (!kmem_cache_has_cpu_partial(s) || partial_slabs > s->cpu_partial_slabs / 2) break; @@ -2606,9 +2610,6 @@ static void __unfreeze_partials(struct kmem_cache *s, struct slab *partial_slab) unsigned long flags = 0; while (partial_slab) { - struct slab new; - struct slab old; - slab = partial_slab; partial_slab = slab->next; @@ -2621,23 +2622,7 @@ static void __unfreeze_partials(struct kmem_cache *s, struct slab *partial_slab) spin_lock_irqsave(&n->list_lock, flags); } - do { - - old.freelist = slab->freelist; - old.counters = slab->counters; - VM_BUG_ON(!old.frozen); - - new.counters = old.counters; - new.freelist = old.freelist; - - new.frozen = 0; - - } while (!__slab_update_freelist(s, slab, - old.freelist, old.counters, - new.freelist, new.counters, - "unfreezing slab")); - - if (unlikely(!new.inuse && n->nr_partial >= s->min_partial)) { + if (unlikely(!slab->inuse && n->nr_partial >= s->min_partial)) { slab->next = slab_to_discard; slab_to_discard = slab; } else { @@ -3634,18 +3619,8 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, was_frozen = new.frozen; new.inuse -= cnt; if ((!new.inuse || !prior) && !was_frozen) { - - if (kmem_cache_has_cpu_partial(s) && !prior) { - - /* - * Slab was on no list before and will be - * partially empty - * We can defer the list move and instead - * freeze it. - */ - new.frozen = 1; - - } else { /* Needs to be taken off a list */ + /* Needs to be taken off a list */ + if (!kmem_cache_has_cpu_partial(s) || prior) { n = get_node(s, slab_nid(slab)); /* @@ -3675,7 +3650,7 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, * activity can be necessary. */ stat(s, FREE_FROZEN); - } else if (new.frozen) { + } else if (kmem_cache_has_cpu_partial(s) && !prior) { /* * If we just froze the slab then put it onto the * per cpu partial list. From patchwork Sat Oct 21 14:43:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 156459 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce89:0:b0:403:3b70:6f57 with SMTP id p9csp329263vqx; Sat, 21 Oct 2023 07:46:25 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGg7Xa3wXO5N5pJ8qebmhN8NjWEv1bNWsm4bHDNGJ1XXnmtvQopOBVxJ9J34kP6hXhapThe X-Received: by 2002:a17:903:11cd:b0:1c7:29fd:33b6 with SMTP id q13-20020a17090311cd00b001c729fd33b6mr4340785plh.40.1697899585271; Sat, 21 Oct 2023 07:46:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697899585; cv=none; d=google.com; s=arc-20160816; b=lBlvs1G1Z1KhIQiJYiG191MAVDlW/0i89NdkQfOl81cX4Um3IzEzHXtxniut42vdR5 HabnJzI1qPP4hnCrU3T3cE1qThmVLQc8QXHWvphPI9QbI/y/70Pi0/xuKE1byLlH5wnT OZ9TW/BpXw5EtnsjbYqXdXmjrzrKQL/mUumkYpQqquJ2P4Px4GRvMlZrv24q3hGS/i3L 8jOenlu1hqbtSUMMTimivYIbx3q6GayZSeW0I8eotKoNy3DSfmH0f/s+IEqiPQ9ltsPZ saNAyRyEg98pKj62nJsSSxwQSE0arbskCsXdCa8EU87ERuGRko+4UxCaeVvPFQFNHlG4 KlVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=nWmtXPvIlob45jfvRK/RuModS0a7G4PIYKuavHC2hOU=; fh=SudcMr3Ofqd+t62q757mMk2HbWTWI1wXv43XX7ahhjA=; b=BPp4xxjJvndAO95dvFUOw1XwsAJL3HNhGzg/XcpkWWyfvlJ++NqlR1efJ088sqWj7T lhy5orWafCAZRGywGTfHJmjX7sE2QWmTRNxVc6nf85bFW4CgmbCG5lwBL8qEtXr3RcwB TZikofo3v9hmtYbgakn0Ux8vrOmUCMW7TnF/gbPo0LWBkZ/nzzRvo20SMiekPwboPLWr 3v1Ea5tGBttbJ3zcrgJRT28L1Ia7sAeqCyOgVCj9blabe9DiqBTg00d/u+quSTbEWs2H mNLmhUx/mh6pi666yLM8X0uhfpdw4XVk3HPhEKoxd8n8KxThg6twKisxbEDgdt8Wt6iq LYNg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=PRz3bzrm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id b19-20020a170902ed1300b001c9d9050b37si3434631pld.260.2023.10.21.07.46.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Oct 2023 07:46:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=PRz3bzrm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 5A33E8083AB0; Sat, 21 Oct 2023 07:45:26 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231487AbjJUOo3 (ORCPT + 26 others); Sat, 21 Oct 2023 10:44:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50018 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231476AbjJUOo2 (ORCPT ); Sat, 21 Oct 2023 10:44:28 -0400 Received: from out-191.mta1.migadu.com (out-191.mta1.migadu.com [95.215.58.191]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6EA1610D1 for ; Sat, 21 Oct 2023 07:44:20 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1697899458; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nWmtXPvIlob45jfvRK/RuModS0a7G4PIYKuavHC2hOU=; b=PRz3bzrmS8EEMlbH+t4XO9cRe385YlpMQbBC+PJkV+mYGBOQDvOobTnnsiBIQwzbWk3dGH 53MwwtqQ/0FHj6idagxgeusX/wx+o/QH+LERI272V7ONmJtNIhQsZj5RYWyxmNgUWef9WU zq8+O024g84F8rDMjvJ1FPttWXr+WJw= From: chengming.zhou@linux.dev To: cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, willy@infradead.org, pcc@google.com, tytso@mit.edu, maz@kernel.org, ruansy.fnst@fujitsu.com, vishal.moola@gmail.com, lrh2000@pku.edu.cn, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v2 4/6] slub: Simplify acquire_slab() Date: Sat, 21 Oct 2023 14:43:15 +0000 Message-Id: <20231021144317.3400916-5-chengming.zhou@linux.dev> In-Reply-To: <20231021144317.3400916-1-chengming.zhou@linux.dev> References: <20231021144317.3400916-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Sat, 21 Oct 2023 07:45:26 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780376755538752635 X-GMAIL-MSGID: 1780376755538752635 From: Chengming Zhou Now the object == NULL is always true, simplify acquire_slab(). Signed-off-by: Chengming Zhou --- mm/slub.c | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 61ee82ea21b6..9f0b80fefc70 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2222,8 +2222,7 @@ static void *alloc_single_from_new_slab(struct kmem_cache *s, * Returns a list of objects or NULL if it fails. */ static inline void *acquire_slab(struct kmem_cache *s, - struct kmem_cache_node *n, struct slab *slab, - int mode) + struct kmem_cache_node *n, struct slab *slab) { void *freelist; unsigned long counters; @@ -2239,12 +2238,8 @@ static inline void *acquire_slab(struct kmem_cache *s, freelist = slab->freelist; counters = slab->counters; new.counters = counters; - if (mode) { - new.inuse = slab->objects; - new.freelist = NULL; - } else { - new.freelist = freelist; - } + new.inuse = slab->objects; + new.freelist = NULL; VM_BUG_ON(new.frozen); new.frozen = 1; @@ -2306,7 +2301,7 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, } if (!object) { - t = acquire_slab(s, n, slab, object == NULL); + t = acquire_slab(s, n, slab); if (t) { *pc->slab = slab; stat(s, ALLOC_FROM_PARTIAL); From patchwork Sat Oct 21 14:43:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 156455 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce89:0:b0:403:3b70:6f57 with SMTP id p9csp328701vqx; Sat, 21 Oct 2023 07:45:22 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFKyx+MhAZy+cpt6s+hfeTPiNQG4oEGsTzy8gqAKysJGOIsWPD3McWkDYxKQtTzasJlWFDg X-Received: by 2002:a05:6a20:8e14:b0:157:54fd:5c26 with SMTP id y20-20020a056a208e1400b0015754fd5c26mr4514325pzj.38.1697899522297; Sat, 21 Oct 2023 07:45:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697899522; cv=none; d=google.com; s=arc-20160816; b=s1aD1wY8HQ6CUzYk8iXNkL1kQUheAlqwIS0JSw4NlltQo8Vf7PvPZcIfMtvvn5s6Ee p0gf0xThr5IhdbT88i5jAQL58M61OWCRJ3B3/jt6wNV790ixYpNjrlKWgWLId3fNePxS Lg4Xfce86eq3bdTqRgETt+V2qTnEg202hfYJQD3f2Rzx+NTS3530PAL5VsnKVU9n1OyH Gnmr7emaOVwO3PVXHs0DSdMObPzB12vkyfGyolvScyA5GQAezINJ+zfVPFmIPGo3QH0a 0OKOqGMQB3IvjcvQRC1Qy1C0heyE3GPwbaQdEqeoKVYatf3Odbp/Y7eMoBpa8HntvIqM 4hQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=tHWic5xUklM9ZxSjedPZ6GHTnBcatj0QDeFJJJdaZnI=; fh=SudcMr3Ofqd+t62q757mMk2HbWTWI1wXv43XX7ahhjA=; b=Q82cyTA+nQs9pfpurFIJjN4FDEMCUQnAJxhOO5VV3jf7gmSwMJOFy55wC19+Akjkxr rMtT+nJcQ7bCRlsY895lHhbv25bXvolfxZkTkqPfxfi70oZy7QjXHJOR90CKZzqfzEB8 smRyh6cXGqK2j4aabinoqlWokxLAwEqHWW9MggTSPlPTMVSIZffYI7xKPsHRbl1eD9XV Aj51aXnDujI+zmcOGTb0/B+MXm/XpKOLyg4inbMOOX5oc0PYvc2KrvlD2Wp57PlJS5u/ vAXUoxrBEd+CSY8mu2wlY+4Bc+Wxe45S/Ktle2QdYBlg/gx7UO6PX1BXc7/lp856lOGm bRQg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=xZpLnPA5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id e24-20020a17090a9a9800b00278ff770796si6288899pjp.88.2023.10.21.07.45.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Oct 2023 07:45:22 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=xZpLnPA5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id A8AFD8060050; Sat, 21 Oct 2023 07:45:15 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231481AbjJUOoi (ORCPT + 26 others); Sat, 21 Oct 2023 10:44:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50154 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231496AbjJUOog (ORCPT ); Sat, 21 Oct 2023 10:44:36 -0400 Received: from out-198.mta1.migadu.com (out-198.mta1.migadu.com [IPv6:2001:41d0:203:375::c6]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C9CE310CF for ; Sat, 21 Oct 2023 07:44:27 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1697899466; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tHWic5xUklM9ZxSjedPZ6GHTnBcatj0QDeFJJJdaZnI=; b=xZpLnPA5vle6JTPcFbJ0n67rIsmg5XrvSZmL7TRfIktWc0DANpJ7X1ziqse/jIv0Pz1dUf o7Qc3QwNW64n42WyqJ9gTrBPvWfFdigRhBwW2USYW3Yx3/zSf5l0CB9wXdlFqaxE+Px/su W+CQPFSDyncBRZEtuylakjuQDjHwMzc= From: chengming.zhou@linux.dev To: cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, willy@infradead.org, pcc@google.com, tytso@mit.edu, maz@kernel.org, ruansy.fnst@fujitsu.com, vishal.moola@gmail.com, lrh2000@pku.edu.cn, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v2 5/6] slub: Introduce get_cpu_partial() Date: Sat, 21 Oct 2023 14:43:16 +0000 Message-Id: <20231021144317.3400916-6-chengming.zhou@linux.dev> In-Reply-To: <20231021144317.3400916-1-chengming.zhou@linux.dev> References: <20231021144317.3400916-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Sat, 21 Oct 2023 07:45:15 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780376689395477596 X-GMAIL-MSGID: 1780376689395477596 From: Chengming Zhou Since the slabs on cpu partial list are not frozen anymore, we introduce get_cpu_partial() to get a frozen slab with its freelist from cpu partial list. It's now much like getting a frozen slab with its freelist from node partial list. Another change is about get_partial(), which can return no frozen slab when all slabs are failed when acquire_slab(), but get some unfreeze slabs in its cpu partial list, so we need to check this rare case to avoid allocating a new slab. Signed-off-by: Chengming Zhou --- mm/slub.c | 87 +++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 68 insertions(+), 19 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 9f0b80fefc70..7fae959c56eb 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3055,6 +3055,68 @@ static inline void *get_freelist(struct kmem_cache *s, struct slab *slab) return freelist; } +#ifdef CONFIG_SLUB_CPU_PARTIAL + +static void *get_cpu_partial(struct kmem_cache *s, struct kmem_cache_cpu *c, + struct slab **slabptr, int node, gfp_t gfpflags) +{ + unsigned long flags; + struct slab *slab; + struct slab new; + unsigned long counters; + void *freelist; + + while (slub_percpu_partial(c)) { + local_lock_irqsave(&s->cpu_slab->lock, flags); + if (unlikely(!slub_percpu_partial(c))) { + local_unlock_irqrestore(&s->cpu_slab->lock, flags); + /* we were preempted and partial list got empty */ + return NULL; + } + + slab = slub_percpu_partial(c); + slub_set_percpu_partial(c, slab); + local_unlock_irqrestore(&s->cpu_slab->lock, flags); + stat(s, CPU_PARTIAL_ALLOC); + + if (unlikely(!node_match(slab, node) || + !pfmemalloc_match(slab, gfpflags))) { + slab->next = NULL; + __unfreeze_partials(s, slab); + continue; + } + + do { + freelist = slab->freelist; + counters = slab->counters; + + new.counters = counters; + VM_BUG_ON(new.frozen); + + new.inuse = slab->objects; + new.frozen = 1; + } while (!__slab_update_freelist(s, slab, + freelist, counters, + NULL, new.counters, + "get_cpu_partial")); + + *slabptr = slab; + return freelist; + } + + return NULL; +} + +#else /* CONFIG_SLUB_CPU_PARTIAL */ + +static void *get_cpu_partial(struct kmem_cache *s, struct kmem_cache_cpu *c, + struct slab **slabptr, int node, gfp_t gfpflags) +{ + return NULL; +} + +#endif + /* * Slow path. The lockless freelist is empty or we need to perform * debugging duties. @@ -3097,7 +3159,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, node = NUMA_NO_NODE; goto new_slab; } -redo: if (unlikely(!node_match(slab, node))) { /* @@ -3173,24 +3234,9 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, new_slab: - if (slub_percpu_partial(c)) { - local_lock_irqsave(&s->cpu_slab->lock, flags); - if (unlikely(c->slab)) { - local_unlock_irqrestore(&s->cpu_slab->lock, flags); - goto reread_slab; - } - if (unlikely(!slub_percpu_partial(c))) { - local_unlock_irqrestore(&s->cpu_slab->lock, flags); - /* we were preempted and partial list got empty */ - goto new_objects; - } - - slab = c->slab = slub_percpu_partial(c); - slub_set_percpu_partial(c, slab); - local_unlock_irqrestore(&s->cpu_slab->lock, flags); - stat(s, CPU_PARTIAL_ALLOC); - goto redo; - } + freelist = get_cpu_partial(s, c, &slab, node, gfpflags); + if (freelist) + goto retry_load_slab; new_objects: @@ -3201,6 +3247,9 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, if (freelist) goto check_new_slab; + if (slub_percpu_partial(c)) + goto new_slab; + slub_put_cpu_ptr(s->cpu_slab); slab = new_slab(s, gfpflags, node); c = slub_get_cpu_ptr(s->cpu_slab); From patchwork Sat Oct 21 14:43:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 156456 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce89:0:b0:403:3b70:6f57 with SMTP id p9csp328926vqx; Sat, 21 Oct 2023 07:45:48 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHotQcX7zA0GXRT8fP/4mq52yKjVN+rAQEZQbapLMqveuJ74BwEdxlq0FKVGYdeLN+FjwKc X-Received: by 2002:a05:6a20:7d81:b0:162:4f45:b415 with SMTP id v1-20020a056a207d8100b001624f45b415mr5653883pzj.51.1697899548410; Sat, 21 Oct 2023 07:45:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697899548; cv=none; d=google.com; s=arc-20160816; b=YWzdyMS8ZwmJ6JruiktaqHBPqmE+4dn4HRfMC+wlKXvqf+3yT42D06onN6EAwFufLy HAamZtwRMJ4r73scPmRsEv3ZObuHnaUAU/SgWDEw06fTd+sTcltns47kcTn8M2zCefQ1 mgRsZdDiEAjHgnnhCCaRRl0ovt1tG6OYhLGMmh+RHVZ35D2sjiuMgQUGCOD9oPmyjGPI Gi0Bjx3NmuD9uwoQF8XYyAiod/ZJZJpA8bdX5kVRZ3Q8NA1ADTVC+1bBjcC2ZGzko9iH uMwjv7BEeeu0rc8JYwptzeBGXX9p+X49VjoUtiBli5Ya82kz7IDmQgtEkT48YdFixwVd DCew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=R6H+QGgDZUUSLw4c5cGh7W9M5d6koieepx0jv1lm/PY=; fh=SudcMr3Ofqd+t62q757mMk2HbWTWI1wXv43XX7ahhjA=; b=KMtl4y/e2+j7hd75/GtHCvMyittu1cjI2oG5KYoywMttI252IxKC81+SAOHdehO1vC LulbjkPC/sOe1PaIv/NWvX1IXp9Ct7Yn1rWuk7Yyz4VhMQdXdJoP6BuSGG3fh3FN+c/R OdLMS+LxnTDw6ihwgSMg4bzBKmyI8ZS41DNCUtddfV9YullZoS4ywBNRzDck3owLxEW1 lxoKTJa03ccpzHeYw8MzCtA3BfOcEhNoV5/2bc4reL+BrPlKzLMUg7VNbYrCxNrHHwj6 Ak6XVQg1hhkIb0iGucbLhXPJgcUd8861s55YLkr2sxdxWtOGD3kE87co1meyA3w3aaeQ QOGg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=nBlo2iOc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id j187-20020a638bc4000000b005780c629ec8si3731053pge.617.2023.10.21.07.45.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Oct 2023 07:45:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=nBlo2iOc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id AA93880B19DE; Sat, 21 Oct 2023 07:45:43 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231501AbjJUOor (ORCPT + 26 others); Sat, 21 Oct 2023 10:44:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47384 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231409AbjJUOoq (ORCPT ); Sat, 21 Oct 2023 10:44:46 -0400 Received: from out-200.mta1.migadu.com (out-200.mta1.migadu.com [95.215.58.200]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 33EA410EA for ; Sat, 21 Oct 2023 07:44:35 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1697899473; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=R6H+QGgDZUUSLw4c5cGh7W9M5d6koieepx0jv1lm/PY=; b=nBlo2iOcMyh6STRK6arKHuu2OtwYxeBOEZUYaYEoR/aFOVymXtmXiP9LRc647ZUIG7Opee H74yoyYmsNA9MpuqduzBOiT57hUIKMOEgLDg2dwwGPS9Y+i0M3NpXEK/xqZUJNEg8VERd+ ScKNE5ibttZh3eJk/XwCQSEMgTaBodM= From: chengming.zhou@linux.dev To: cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, willy@infradead.org, pcc@google.com, tytso@mit.edu, maz@kernel.org, ruansy.fnst@fujitsu.com, vishal.moola@gmail.com, lrh2000@pku.edu.cn, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v2 6/6] slub: Optimize deactivate_slab() Date: Sat, 21 Oct 2023 14:43:17 +0000 Message-Id: <20231021144317.3400916-7-chengming.zhou@linux.dev> In-Reply-To: <20231021144317.3400916-1-chengming.zhou@linux.dev> References: <20231021144317.3400916-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Sat, 21 Oct 2023 07:45:43 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780376716648653634 X-GMAIL-MSGID: 1780376716648653634 From: Chengming Zhou Since the introduce of unfrozen slabs on cpu partial list, we don't need to synchronize the slab frozen state under the node list_lock. The caller of deactivate_slab() and the caller of __slab_free() won't manipulate the slab list concurrently. So we can get node list_lock in the stage three if we need to manipulate the slab list in this path. Signed-off-by: Chengming Zhou --- mm/slub.c | 70 ++++++++++++++++++++----------------------------------- 1 file changed, 25 insertions(+), 45 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 7fae959c56eb..29a60bfbf9c5 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2493,10 +2493,8 @@ static void init_kmem_cache_cpus(struct kmem_cache *s) static void deactivate_slab(struct kmem_cache *s, struct slab *slab, void *freelist) { - enum slab_modes { M_NONE, M_PARTIAL, M_FREE, M_FULL_NOLIST }; struct kmem_cache_node *n = get_node(s, slab_nid(slab)); int free_delta = 0; - enum slab_modes mode = M_NONE; void *nextfree, *freelist_iter, *freelist_tail; int tail = DEACTIVATE_TO_HEAD; unsigned long flags = 0; @@ -2543,58 +2541,40 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, * unfrozen and number of objects in the slab may have changed. * Then release lock and retry cmpxchg again. */ -redo: - - old.freelist = READ_ONCE(slab->freelist); - old.counters = READ_ONCE(slab->counters); - VM_BUG_ON(!old.frozen); - - /* Determine target state of the slab */ - new.counters = old.counters; - if (freelist_tail) { - new.inuse -= free_delta; - set_freepointer(s, freelist_tail, old.freelist); - new.freelist = freelist; - } else - new.freelist = old.freelist; + do { + old.freelist = READ_ONCE(slab->freelist); + old.counters = READ_ONCE(slab->counters); + VM_BUG_ON(!old.frozen); + + /* Determine target state of the slab */ + new.counters = old.counters; + new.frozen = 0; + if (freelist_tail) { + new.inuse -= free_delta; + set_freepointer(s, freelist_tail, old.freelist); + new.freelist = freelist; + } else + new.freelist = old.freelist; - new.frozen = 0; + } while (!slab_update_freelist(s, slab, + old.freelist, old.counters, + new.freelist, new.counters, + "unfreezing slab")); + /* + * Stage three: Manipulate the slab list based on the updated state. + */ if (!new.inuse && n->nr_partial >= s->min_partial) { - mode = M_FREE; + stat(s, DEACTIVATE_EMPTY); + discard_slab(s, slab); + stat(s, FREE_SLAB); } else if (new.freelist) { - mode = M_PARTIAL; - /* - * Taking the spinlock removes the possibility that - * acquire_slab() will see a slab that is frozen - */ spin_lock_irqsave(&n->list_lock, flags); - } else { - mode = M_FULL_NOLIST; - } - - - if (!slab_update_freelist(s, slab, - old.freelist, old.counters, - new.freelist, new.counters, - "unfreezing slab")) { - if (mode == M_PARTIAL) - spin_unlock_irqrestore(&n->list_lock, flags); - goto redo; - } - - - if (mode == M_PARTIAL) { add_partial(n, slab, tail); spin_unlock_irqrestore(&n->list_lock, flags); stat(s, tail); - } else if (mode == M_FREE) { - stat(s, DEACTIVATE_EMPTY); - discard_slab(s, slab); - stat(s, FREE_SLAB); - } else if (mode == M_FULL_NOLIST) { + } else stat(s, DEACTIVATE_FULL); - } } #ifdef CONFIG_SLUB_CPU_PARTIAL