From patchwork Wed Nov 29 09:53:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 171221 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a5a7:0:b0:403:3b70:6f57 with SMTP id d7csp229385vqn; Wed, 29 Nov 2023 01:54:50 -0800 (PST) X-Google-Smtp-Source: AGHT+IG+G+GOF2SNW4Zp/S8zQOmEEdGge3Gc7JQGEcwF7uaB/R8QQ/6WPty0IYxIl9DCJpM9tNaK X-Received: by 2002:a05:6a20:1594:b0:18d:b43:78f9 with SMTP id h20-20020a056a20159400b0018d0b4378f9mr1333473pzj.39.1701251690338; Wed, 29 Nov 2023 01:54:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701251690; cv=none; d=google.com; s=arc-20160816; b=sSGJntl0W6w/ehUQiyhLJs1Bw7kU/ptMGL0O4EETnKxzwjfFgTM4MxSm8ZKRP73LUn 3QPsC3eyVRlKnMCWHL8TznZhgqcRdMUuKK8AF5HLxfV+Jm4KR7MJ1P/Geq5SZo7XfyH1 hMOSOp7GTqwYidERdZ74q1RfGo1Z+a2tn0PrLj0Z6IEwYdYyr75SzBkthdkffXi60LiB 9XuEAiBLeBOQ9Qdwygliret0shDJ4XZTVFO5zWXuXcz3xBc16K7mo5VB7kqjHblhderf 0i2GFgsuFAeCb6xn72CmYDtviQI5HtWP2BBum7cmrmFGZyR4VQRjvdeY8CqVvhKJglTn 21Kg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:in-reply-to:references:message-id :content-transfer-encoding:mime-version:subject:date:from; bh=r0ZIwChSWj55C5ZofQebA+Ysb/A45+GXttFMd4vVFZY=; fh=uYnIsWZ9n80gkcnhZMgoujzKtxA7UEl4GQvTBFLbimw=; b=EJk00yymxVDIzvu5i9LAbisy7T+cCs1uhrX6NHLjauSczfy5TTCv1FKIxZ7AyqHf1r 3YL9znsSnMgawys8I+QCKXKyeLwCs58eemPub/OzR2zv4hKi2bfCqn8e4OvlPJr35MQO +fWDRtu3cd+XYfGd3v4axH9PaBczx1XYk6SHPfYtvsqJVmc3oyf5uU0d0+hp3uPpVrGP e87CbbR8m1Ke68z0ZE5NNOlYUmTd4Pcd1AmZomlpLdBCLx+LjB4kYHIxY2AEHgUwmPdT uUFSJcYDaJthfcTmXw9a5ZltyPuYaWM3z4ryE1nkRen23Z9IPrqf/dBmgW7yt6PcUT6m fTJw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id p1-20020a17090ab90100b00285ada58d5bsi955612pjr.49.2023.11.29.01.54.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Nov 2023 01:54:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id BCB8F805F49B; Wed, 29 Nov 2023 01:54:44 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231351AbjK2Jyc (ORCPT + 99 others); Wed, 29 Nov 2023 04:54:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35046 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231274AbjK2Jxy (ORCPT ); Wed, 29 Nov 2023 04:53:54 -0500 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2a07:de40:b251:101:10:150:64:1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E584E2681 for ; Wed, 29 Nov 2023 01:53:39 -0800 (PST) Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 0AF1221991; Wed, 29 Nov 2023 09:53:37 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id E23D613A97; Wed, 29 Nov 2023 09:53:36 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id AGjyNiAKZ2UrfQAAD6G6ig (envelope-from ); Wed, 29 Nov 2023 09:53:36 +0000 From: Vlastimil Babka Date: Wed, 29 Nov 2023 10:53:27 +0100 Subject: [PATCH RFC v3 2/9] mm/slub: introduce __kmem_cache_free_bulk() without free hooks MIME-Version: 1.0 Message-Id: <20231129-slub-percpu-caches-v3-2-6bcf536772bc@suse.cz> References: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> In-Reply-To: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Matthew Wilcox , "Liam R. Howlett" Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Alexander Potapenko , Marco Elver , Dmitry Vyukov , linux-mm@kvack.org, linux-kernel@vger.kernel.org, maple-tree@lists.infradead.org, kasan-dev@googlegroups.com, Vlastimil Babka X-Mailer: b4 0.12.4 X-Spamd-Bar: +++++++++++++++++ X-Spam-Score: 17.13 X-Rspamd-Server: rspamd1 Authentication-Results: smtp-out1.suse.de; dkim=none; spf=softfail (smtp-out1.suse.de: 2a07:de40:b281:104:10:150:64:97 is neither permitted nor denied by domain of vbabka@suse.cz) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspamd-Queue-Id: 0AF1221991 X-Spamd-Result: default: False [17.13 / 50.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; BAYES_SPAM(5.09)[99.98%]; SPAMHAUS_XBL(0.00)[2a07:de40:b281:104:10:150:64:97:from]; TO_DN_SOME(0.00)[]; R_SPF_SOFTFAIL(4.60)[~all:c]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[]; NEURAL_HAM_SHORT(-0.20)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(2.20)[]; MIME_TRACE(0.00)[0:+]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(1.20)[suse.cz]; NEURAL_SPAM_LONG(2.85)[0.813]; RCPT_COUNT_TWELVE(0.00)[17]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.cz:email]; FUZZY_BLOCKED(0.00)[rspamd.com]; FREEMAIL_CC(0.00)[linux-foundation.org,linux.dev,gmail.com,google.com,kvack.org,vger.kernel.org,lists.infradead.org,googlegroups.com,suse.cz]; RCVD_TLS_ALL(0.00)[]; SUSPICIOUS_RECIPS(1.50)[] X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Wed, 29 Nov 2023 01:54:45 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783891692547235098 X-GMAIL-MSGID: 1783891692547235098 Currently, when __kmem_cache_alloc_bulk() fails, it frees back the objects that were allocated before the failure, using kmem_cache_free_bulk(). Because kmem_cache_free_bulk() calls the free hooks (kasan etc.) and those expect objects processed by the post alloc hooks, slab_post_alloc_hook() is called before kmem_cache_free_bulk(). This is wasteful, although not a big concern in practice for the very rare error path. But in order to efficiently handle percpu array batch refill and free in the following patch, we will also need a variant of kmem_cache_free_bulk() that avoids the free hooks. So introduce it first and use it in the error path too. As a consequence, __kmem_cache_alloc_bulk() no longer needs the objcg parameter, remove it. Signed-off-by: Vlastimil Babka --- mm/slub.c | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index f0cd55bb4e11..16748aeada8f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3919,6 +3919,27 @@ int build_detached_freelist(struct kmem_cache *s, size_t size, return same; } +/* + * Internal bulk free of objects that were not initialised by the post alloc + * hooks and thus should not be processed by the free hooks + */ +static void __kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p) +{ + if (!size) + return; + + do { + struct detached_freelist df; + + size = build_detached_freelist(s, size, p, &df); + if (!df.slab) + continue; + + do_slab_free(df.s, df.slab, df.freelist, df.tail, df.cnt, + _RET_IP_); + } while (likely(size)); +} + /* Note that interrupts must be enabled when calling this function. */ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p) { @@ -3940,7 +3961,7 @@ EXPORT_SYMBOL(kmem_cache_free_bulk); #ifndef CONFIG_SLUB_TINY static inline int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, - size_t size, void **p, struct obj_cgroup *objcg) + size_t size, void **p) { struct kmem_cache_cpu *c; unsigned long irqflags; @@ -4004,14 +4025,13 @@ static inline int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, error: slub_put_cpu_ptr(s->cpu_slab); - slab_post_alloc_hook(s, objcg, flags, i, p, false, s->object_size); - kmem_cache_free_bulk(s, i, p); + __kmem_cache_free_bulk(s, i, p); return 0; } #else /* CONFIG_SLUB_TINY */ static int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, - size_t size, void **p, struct obj_cgroup *objcg) + size_t size, void **p) { int i; @@ -4034,8 +4054,7 @@ static int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, return i; error: - slab_post_alloc_hook(s, objcg, flags, i, p, false, s->object_size); - kmem_cache_free_bulk(s, i, p); + __kmem_cache_free_bulk(s, i, p); return 0; } #endif /* CONFIG_SLUB_TINY */ @@ -4055,7 +4074,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, if (unlikely(!s)) return 0; - i = __kmem_cache_alloc_bulk(s, flags, size, p, objcg); + i = __kmem_cache_alloc_bulk(s, flags, size, p); /* * memcg and kmem_cache debug support and memory initialization. From patchwork Wed Nov 29 09:53:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 171224 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a5a7:0:b0:403:3b70:6f57 with SMTP id d7csp229475vqn; Wed, 29 Nov 2023 01:55:08 -0800 (PST) X-Google-Smtp-Source: AGHT+IE99lUjMyi7TS7iFt+9J09TrmBRrb00tBHUcHLYod4ktFdSVy7wjG85m9VjTZ5/wtRShPGE X-Received: by 2002:a9d:75cd:0:b0:6d4:80ee:f5b with SMTP id c13-20020a9d75cd000000b006d480ee0f5bmr19546355otl.35.1701251708120; Wed, 29 Nov 2023 01:55:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701251708; cv=none; d=google.com; s=arc-20160816; b=oJW4B+Q8RNsKuROD571P7a5UPgLB3AJT7q1H9qM46dzR9h297UadE6AvabSKvCtgW8 Y8bcq76aQfecYel8bIFSlFHAFm7u3nByX+uq11RURs/TxLfWfaC3nSEYUQqQnF4Z62AT cepIkBey0rwzHm520iN+ELP5ftl+NhOW8sgzR1JruUaZkMGTWAXOWt2pUe+abqrrdNg5 uGhWPpV3bRcvSMSdlzeRqlQOZgrPXG6K9vY32JhSeKopoWZj4cqHP46BCtFWYFiK4owS h0NwvpL+/LjRES3BiHBB5OnUxeb4UjPey/93Rn5T/2Rc8J3LloNFZWH+iDz7rIoJogdx +S9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:in-reply-to:references:message-id :content-transfer-encoding:mime-version:subject:date:from; bh=P2ukcSj2GFDFWZJr5dQPgX+igAnhpGokILF+Ev+OXuM=; fh=uYnIsWZ9n80gkcnhZMgoujzKtxA7UEl4GQvTBFLbimw=; b=vvG7GuU1ySa2PLOAgSqfIWJTVi91u667E+8qLbaGefVad3Y7YVeYDw+kXCW+QxL4VM UlWQ+F1gGesYxeAmttXylGYzG5nBy5X/y/jRYE9Wrkrj2Xkq2xZSL5Esww0r3t4NBfLP D4VZTQBKBXw2sH6pkrAX9QYnXYw0HdVpfBFoqzTGveCRdaIAuVPWE64c7wVqvTYQLcVk FYq52EF7+O0uXUfbUUkFske3H4FZ4ytaZgFK7yAQwJDCyK78f919t2eljTEFNOvE3Lp0 agmtHzA2CPFnlm2ndwnC2Gna3wQqlObKUOmVMHeFe+wxHmtgugpxZk2u5zZ0rJfTpgDc sgbg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id a2-20020a63d402000000b005b86142ee56si13285936pgh.277.2023.11.29.01.55.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Nov 2023 01:55:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 43CA9822037D; Wed, 29 Nov 2023 01:55:05 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232627AbjK2Jyh (ORCPT + 99 others); Wed, 29 Nov 2023 04:54:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57744 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231776AbjK2Jxz (ORCPT ); Wed, 29 Nov 2023 04:53:55 -0500 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EEC042682 for ; Wed, 29 Nov 2023 01:53:39 -0800 (PST) Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 2B8351F8B3; Wed, 29 Nov 2023 09:53:37 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 08AC613A98; Wed, 29 Nov 2023 09:53:37 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id qLDSASEKZ2UrfQAAD6G6ig (envelope-from ); Wed, 29 Nov 2023 09:53:37 +0000 From: Vlastimil Babka Date: Wed, 29 Nov 2023 10:53:28 +0100 Subject: [PATCH RFC v3 3/9] mm/slub: handle bulk and single object freeing separately MIME-Version: 1.0 Message-Id: <20231129-slub-percpu-caches-v3-3-6bcf536772bc@suse.cz> References: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> In-Reply-To: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Matthew Wilcox , "Liam R. Howlett" Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Alexander Potapenko , Marco Elver , Dmitry Vyukov , linux-mm@kvack.org, linux-kernel@vger.kernel.org, maple-tree@lists.infradead.org, kasan-dev@googlegroups.com, Vlastimil Babka X-Mailer: b4 0.12.4 X-Spamd-Bar: +++++++++ X-Spam-Score: 9.03 X-Rspamd-Server: rspamd1 Authentication-Results: smtp-out2.suse.de; dkim=none; spf=softfail (smtp-out2.suse.de: 2a07:de40:b281:104:10:150:64:97 is neither permitted nor denied by domain of vbabka@suse.cz) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspamd-Queue-Id: 2B8351F8B3 X-Spamd-Result: default: False [9.03 / 50.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; SPAMHAUS_XBL(0.00)[2a07:de40:b281:104:10:150:64:97:from]; TO_DN_SOME(0.00)[]; R_SPF_SOFTFAIL(4.60)[~all]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[]; NEURAL_HAM_SHORT(-0.20)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(2.20)[]; MIME_TRACE(0.00)[0:+]; MID_RHS_MATCH_FROM(0.00)[]; BAYES_HAM(-3.00)[100.00%]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(1.20)[suse.cz]; NEURAL_SPAM_LONG(2.84)[0.810]; RCPT_COUNT_TWELVE(0.00)[17]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.cz:email]; FUZZY_BLOCKED(0.00)[rspamd.com]; FREEMAIL_CC(0.00)[linux-foundation.org,linux.dev,gmail.com,google.com,kvack.org,vger.kernel.org,lists.infradead.org,googlegroups.com,suse.cz]; RCVD_TLS_ALL(0.00)[]; SUSPICIOUS_RECIPS(1.50)[] X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 29 Nov 2023 01:55:05 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783891711244063510 X-GMAIL-MSGID: 1783891711244063510 Until now we have a single function slab_free() handling both single object freeing and bulk freeing with neccessary hooks, the latter case requiring slab_free_freelist_hook(). It should be however better to distinguish the two scenarios for the following reasons: - code simpler to follow for the single object case - better code generation - although inlining should eliminate the slab_free_freelist_hook() in case no debugging options are enabled, it seems it's not perfect. When e.g. KASAN is enabled, we're imposing additional unnecessary overhead for single object freeing. - preparation to add percpu array caches in later patches Therefore, simplify slab_free() for the single object case by dropping unnecessary parameters and calling only slab_free_hook() instead of slab_free_freelist_hook(). Rename the bulk variant to slab_free_bulk() and adjust callers accordingly. While at it, flip (and document) slab_free_hook() return value so that it returns true when the freeing can proceed, which matches the logic of slab_free_freelist_hook() and is not confusingly the opposite. Additionally we can simplify a bit by changing the tail parameter of do_slab_free() when freeing a single object - instead of NULL we can set equal to head. bloat-o-meter shows small code reduction with a .config that has KASAN etc disabled: add/remove: 0/0 grow/shrink: 0/4 up/down: 0/-118 (-118) Function old new delta kmem_cache_alloc_bulk 1203 1196 -7 kmem_cache_free 861 835 -26 __kmem_cache_free 741 704 -37 kmem_cache_free_bulk 911 863 -48 Signed-off-by: Vlastimil Babka --- mm/slub.c | 57 ++++++++++++++++++++++++++++++++++----------------------- 1 file changed, 34 insertions(+), 23 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 16748aeada8f..7d23f10d42e6 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1770,9 +1770,12 @@ static bool freelist_corrupted(struct kmem_cache *s, struct slab *slab, /* * Hooks for other subsystems that check memory allocations. In a typical * production configuration these hooks all should produce no code at all. + * + * Returns true if freeing of the object can proceed, false if its reuse + * was delayed by KASAN quarantine. */ -static __always_inline bool slab_free_hook(struct kmem_cache *s, - void *x, bool init) +static __always_inline +bool slab_free_hook(struct kmem_cache *s, void *x, bool init) { kmemleak_free_recursive(x, s->flags); kmsan_slab_free(s, x); @@ -1805,7 +1808,7 @@ static __always_inline bool slab_free_hook(struct kmem_cache *s, s->size - s->inuse - rsize); } /* KASAN might put x into memory quarantine, delaying its reuse. */ - return kasan_slab_free(s, x, init); + return !kasan_slab_free(s, x, init); } static inline bool slab_free_freelist_hook(struct kmem_cache *s, @@ -1815,7 +1818,7 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s, void *object; void *next = *head; - void *old_tail = *tail ? *tail : *head; + void *old_tail = *tail; if (is_kfence_address(next)) { slab_free_hook(s, next, false); @@ -1831,7 +1834,7 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s, next = get_freepointer(s, object); /* If object's reuse doesn't have to be delayed */ - if (!slab_free_hook(s, object, slab_want_init_on_free(s))) { + if (slab_free_hook(s, object, slab_want_init_on_free(s))) { /* Move object to the new freelist */ set_freepointer(s, object, *head); *head = object; @@ -1846,9 +1849,6 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s, } } while (object != old_tail); - if (*head == *tail) - *tail = NULL; - return *head != NULL; } @@ -3743,7 +3743,6 @@ static __always_inline void do_slab_free(struct kmem_cache *s, struct slab *slab, void *head, void *tail, int cnt, unsigned long addr) { - void *tail_obj = tail ? : head; struct kmem_cache_cpu *c; unsigned long tid; void **freelist; @@ -3762,14 +3761,14 @@ static __always_inline void do_slab_free(struct kmem_cache *s, barrier(); if (unlikely(slab != c->slab)) { - __slab_free(s, slab, head, tail_obj, cnt, addr); + __slab_free(s, slab, head, tail, cnt, addr); return; } if (USE_LOCKLESS_FAST_PATH()) { freelist = READ_ONCE(c->freelist); - set_freepointer(s, tail_obj, freelist); + set_freepointer(s, tail, freelist); if (unlikely(!__update_cpu_freelist_fast(s, freelist, head, tid))) { note_cmpxchg_failure("slab_free", s, tid); @@ -3786,7 +3785,7 @@ static __always_inline void do_slab_free(struct kmem_cache *s, tid = c->tid; freelist = c->freelist; - set_freepointer(s, tail_obj, freelist); + set_freepointer(s, tail, freelist); c->freelist = head; c->tid = next_tid(tid); @@ -3799,15 +3798,27 @@ static void do_slab_free(struct kmem_cache *s, struct slab *slab, void *head, void *tail, int cnt, unsigned long addr) { - void *tail_obj = tail ? : head; - - __slab_free(s, slab, head, tail_obj, cnt, addr); + __slab_free(s, slab, head, tail, cnt, addr); } #endif /* CONFIG_SLUB_TINY */ -static __fastpath_inline void slab_free(struct kmem_cache *s, struct slab *slab, - void *head, void *tail, void **p, int cnt, - unsigned long addr) +static __fastpath_inline +void slab_free(struct kmem_cache *s, struct slab *slab, void *object, + unsigned long addr) +{ + bool init; + + memcg_slab_free_hook(s, slab, &object, 1); + + init = !is_kfence_address(object) && slab_want_init_on_free(s); + + if (likely(slab_free_hook(s, object, init))) + do_slab_free(s, slab, object, object, 1, addr); +} + +static __fastpath_inline +void slab_free_bulk(struct kmem_cache *s, struct slab *slab, void *head, + void *tail, void **p, int cnt, unsigned long addr) { memcg_slab_free_hook(s, slab, p, cnt); /* @@ -3821,13 +3832,13 @@ static __fastpath_inline void slab_free(struct kmem_cache *s, struct slab *slab, #ifdef CONFIG_KASAN_GENERIC void ___cache_free(struct kmem_cache *cache, void *x, unsigned long addr) { - do_slab_free(cache, virt_to_slab(x), x, NULL, 1, addr); + do_slab_free(cache, virt_to_slab(x), x, x, 1, addr); } #endif void __kmem_cache_free(struct kmem_cache *s, void *x, unsigned long caller) { - slab_free(s, virt_to_slab(x), x, NULL, &x, 1, caller); + slab_free(s, virt_to_slab(x), x, caller); } void kmem_cache_free(struct kmem_cache *s, void *x) @@ -3836,7 +3847,7 @@ void kmem_cache_free(struct kmem_cache *s, void *x) if (!s) return; trace_kmem_cache_free(_RET_IP_, x, s); - slab_free(s, virt_to_slab(x), x, NULL, &x, 1, _RET_IP_); + slab_free(s, virt_to_slab(x), x, _RET_IP_); } EXPORT_SYMBOL(kmem_cache_free); @@ -3953,8 +3964,8 @@ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p) if (!df.slab) continue; - slab_free(df.s, df.slab, df.freelist, df.tail, &p[size], df.cnt, - _RET_IP_); + slab_free_bulk(df.s, df.slab, df.freelist, df.tail, &p[size], + df.cnt, _RET_IP_); } while (likely(size)); } EXPORT_SYMBOL(kmem_cache_free_bulk); From patchwork Wed Nov 29 09:53:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 171223 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a5a7:0:b0:403:3b70:6f57 with SMTP id d7csp229461vqn; Wed, 29 Nov 2023 01:55:06 -0800 (PST) X-Google-Smtp-Source: AGHT+IHb2AdcgqT9sYryHEi2KwA637vWdYGSWUK83XfoHjAjiHLYGlc6e1/n/LxP3d599BZ5fO/8 X-Received: by 2002:a05:6830:3b0e:b0:6c6:4843:2abb with SMTP id dk14-20020a0568303b0e00b006c648432abbmr17382837otb.12.1701251706008; Wed, 29 Nov 2023 01:55:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701251705; cv=none; d=google.com; s=arc-20160816; b=TVtdXlur7hLHN5gs1KfzcxmPaBG+siAtC1w/VwH8Nj8TQ+yKIuqk8FT2UfJU660idT hEqKuPmJfuoCk3Q6ILQfVCaaM63tC1+PM9VPp2v7wx4ERju5eFiqtDQqM8Fgi9dqo0dd VRdj90Rm35EuElCLceEGC1HJD4HZZ8qEtT7Dd+ombVHFTjOQ4vXYN+CHuYyH3zr0D33O W9DX9qGRC8GK7chgIVdfbUsQ+2Msb/PwqETPn/ontL5yXxeSSUJGCwm6FbQFq87/Ya+Q nJ9zTG7iTw7Vwx1M+vAHI/rR+MOHizMAXprG2nj8O9PewUtz45li7qnpmMj1YlOZRAkD /O+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:in-reply-to:references:message-id :content-transfer-encoding:mime-version:subject:date:from :dkim-signature:dkim-signature; bh=Z/MdY/rw/HJ2oq6eayRQ1jISyVijjFBbUkDkU5iKbc8=; fh=uYnIsWZ9n80gkcnhZMgoujzKtxA7UEl4GQvTBFLbimw=; b=EAQhCvdg3mANy1ctWqMKL6iabfHaZVVgj3jkiZA2YgFqJZEuWs76kQfaZGnKSfYwOR SbRwZfuqTtiN1suRP5PRdbhK4/IDqIf+fT1EP83C2cqHd4tLdSQFW8jym706pQoTHKB0 sCY0seQCENGtb2xGVagAHOc1TjWAMnw0Jy97u4BhG8TGuIq9X8wkC3d9CEI0LwVBNcgg 2sPEA9h2NQAiOALLlHDsTQPlpo1y9gA/34h8er/sHUchYy6rRF9IyM+N0GLvMnGGFR0X lp4xZqH+XJdGMosmNB6xEHoocmdArFtm+16mi1rzK3NV9Jirq3Fgy/16w9KqchdwwlEH oMEQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=JSaWHQ5n; dkim=neutral (no key) header.i=@suse.cz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id b19-20020a63d313000000b00584ca25959csi14608294pgg.540.2023.11.29.01.55.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Nov 2023 01:55:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=JSaWHQ5n; dkim=neutral (no key) header.i=@suse.cz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id E39E980AD0B5; Wed, 29 Nov 2023 01:54:50 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231650AbjK2Jyf (ORCPT + 99 others); Wed, 29 Nov 2023 04:54:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231414AbjK2Jxz (ORCPT ); Wed, 29 Nov 2023 04:53:55 -0500 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2a07:de40:b251:101:10:150:64:2]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0CF3D2683 for ; Wed, 29 Nov 2023 01:53:40 -0800 (PST) Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 44CA81F8B9; Wed, 29 Nov 2023 09:53:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1701251617; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z/MdY/rw/HJ2oq6eayRQ1jISyVijjFBbUkDkU5iKbc8=; b=JSaWHQ5ngHNR9h0BjZPRO9pLV15+bY/sEiVQLXEtaalvdmi6pmxe9eubOmuCdSSXeNh/Sb Nh4DIf+6GCzwc4Z9cmo4/0h6AQF8+HTMP/IskJTEjbgBOYzfWDDmc9KEj599VKQ98q9w7M A6yjwcRMpx9Ot6/k9lkxkNg8+uMSuPs= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1701251617; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z/MdY/rw/HJ2oq6eayRQ1jISyVijjFBbUkDkU5iKbc8=; b=F3IPQ3NkAI4OgYj5R6JdEz852oBdNpOhcIfg0KleOLz7BA8tL0iO7eyPul69JqVGKjb/Zt uYSShywwUe5LzyAQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 240B213A9A; Wed, 29 Nov 2023 09:53:37 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id +GmBCCEKZ2UrfQAAD6G6ig (envelope-from ); Wed, 29 Nov 2023 09:53:37 +0000 From: Vlastimil Babka Date: Wed, 29 Nov 2023 10:53:29 +0100 Subject: [PATCH RFC v3 4/9] mm/slub: free KFENCE objects in slab_free_hook() MIME-Version: 1.0 Message-Id: <20231129-slub-percpu-caches-v3-4-6bcf536772bc@suse.cz> References: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> In-Reply-To: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Matthew Wilcox , "Liam R. Howlett" Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Alexander Potapenko , Marco Elver , Dmitry Vyukov , linux-mm@kvack.org, linux-kernel@vger.kernel.org, maple-tree@lists.infradead.org, kasan-dev@googlegroups.com, Vlastimil Babka X-Mailer: b4 0.12.4 Authentication-Results: smtp-out2.suse.de; none X-Spam-Level: X-Spamd-Result: default: False [-2.80 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; BAYES_HAM(-3.00)[100.00%]; RCVD_COUNT_THREE(0.00)[3]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; RCPT_COUNT_TWELVE(0.00)[17]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.cz:email]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_CC(0.00)[linux-foundation.org,linux.dev,gmail.com,google.com,kvack.org,vger.kernel.org,lists.infradead.org,googlegroups.com,suse.cz]; RCVD_TLS_ALL(0.00)[]; SUSPICIOUS_RECIPS(1.50)[] X-Spam-Score: -2.80 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 29 Nov 2023 01:54:51 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783891708991348152 X-GMAIL-MSGID: 1783891708991348152 When freeing an object that was allocated from KFENCE, we do that in the slowpath __slab_free(), relying on the fact that KFENCE "slab" cannot be the cpu slab, so the fastpath has to fallback to the slowpath. This optimization doesn't help much though, because is_kfence_address() is checked earlier anyway during the free hook processing or detached freelist building. Thus we can simplify the code by making the slab_free_hook() free the KFENCE object immediately, similarly to KASAN quarantine. In slab_free_hook() we can place kfence_free() above init processing, as callers have been making sure to set init to false for KFENCE objects. This simplifies slab_free(). This places it also above kasan_slab_free() which is ok as that skips KFENCE objects anyway. While at it also determine the init value in slab_free_freelist_hook() outside of the loop. This change will also make introducing per cpu array caches easier. Signed-off-by: Vlastimil Babka Tested-by: Marco Elver --- mm/slub.c | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 7d23f10d42e6..59912a376c6d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1772,7 +1772,7 @@ static bool freelist_corrupted(struct kmem_cache *s, struct slab *slab, * production configuration these hooks all should produce no code at all. * * Returns true if freeing of the object can proceed, false if its reuse - * was delayed by KASAN quarantine. + * was delayed by KASAN quarantine, or it was returned to KFENCE. */ static __always_inline bool slab_free_hook(struct kmem_cache *s, void *x, bool init) @@ -1790,6 +1790,9 @@ bool slab_free_hook(struct kmem_cache *s, void *x, bool init) __kcsan_check_access(x, s->object_size, KCSAN_ACCESS_WRITE | KCSAN_ACCESS_ASSERT); + if (kfence_free(kasan_reset_tag(x))) + return false; + /* * As memory initialization might be integrated into KASAN, * kasan_slab_free and initialization memset's must be @@ -1819,22 +1822,25 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s, void *object; void *next = *head; void *old_tail = *tail; + bool init; if (is_kfence_address(next)) { slab_free_hook(s, next, false); - return true; + return false; } /* Head and tail of the reconstructed freelist */ *head = NULL; *tail = NULL; + init = slab_want_init_on_free(s); + do { object = next; next = get_freepointer(s, object); /* If object's reuse doesn't have to be delayed */ - if (slab_free_hook(s, object, slab_want_init_on_free(s))) { + if (slab_free_hook(s, object, init)) { /* Move object to the new freelist */ set_freepointer(s, object, *head); *head = object; @@ -3619,9 +3625,6 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, stat(s, FREE_SLOWPATH); - if (kfence_free(head)) - return; - if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) { free_to_partial_list(s, slab, head, tail, cnt, addr); return; @@ -3806,13 +3809,9 @@ static __fastpath_inline void slab_free(struct kmem_cache *s, struct slab *slab, void *object, unsigned long addr) { - bool init; - memcg_slab_free_hook(s, slab, &object, 1); - init = !is_kfence_address(object) && slab_want_init_on_free(s); - - if (likely(slab_free_hook(s, object, init))) + if (likely(slab_free_hook(s, object, slab_want_init_on_free(s)))) do_slab_free(s, slab, object, object, 1, addr); } From patchwork Wed Nov 29 09:53:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 171225 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a5a7:0:b0:403:3b70:6f57 with SMTP id d7csp229670vqn; Wed, 29 Nov 2023 01:55:44 -0800 (PST) X-Google-Smtp-Source: AGHT+IHMHhOQevsuC7pxwNmBonJBWNnzmz6U4UclS1loC1Bc+XIgcJdXUYcFP3wzJIIZOhjNc5Rs X-Received: by 2002:a17:90b:1a89:b0:285:9912:a4c4 with SMTP id ng9-20020a17090b1a8900b002859912a4c4mr16790080pjb.41.1701251744612; Wed, 29 Nov 2023 01:55:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701251744; cv=none; d=google.com; s=arc-20160816; b=c9+HdjwLv0b6aswY0TQEsikxnZelXa0zZYAFl6MELImDEyvDkkdjI9qidbI8dwYQuJ EERNYyycs+I5jt6+cwpracpN18TFWcL/HlH9gbGYaal0xzsLFsIV5DWMJzZvtiST3KxV vgnlkvstiTMsV2kgayaZkfzIdp7SlyUBIqW7LTbQyPhryA8ADawKfN73LOBYAGDQjvQh lPontpJn8zdYkeahbCAchz08UV+AwRPJjO+3wJXBKdDdlxjjP5ZIFkwTvCHIZAoo6RrS S7TcLq4eMQxqiXJtE0PVUQCZFXGrNgSPXwZouEdIM0dCWzqnX0icMXyE4RiCkC+qU/D6 Mx/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:in-reply-to:references:message-id :content-transfer-encoding:mime-version:subject:date:from; bh=mCZNnnAgtZ9BCKVsGZOHUXKdRtd+yAwyaxi7IiapAks=; fh=uYnIsWZ9n80gkcnhZMgoujzKtxA7UEl4GQvTBFLbimw=; b=C/XE0YdCSxIQ5MQFn8bee+30sVhopeP1+2TaIYxfOjDCyyFivPv/vM2EygciIOcPKp +mSn8+8GZcsYYAnBVuqKb9QrCrt8w8/DgQPxCQi/zYgSyJ0RBcZBxBLZKZuVfsqipoNn IvIY6d9/QeEOcdadOt1G3zKwhCltGcpbAQ1ueadbybFN1+nttQQ7DfR4pv1yQoejTxAV oRCzu1ELFAexXRy9Lc0Low43itoRQqQLk6TVoQl33u3i1fyTcN5FHofk2psx3QmfxjF7 KFTNahCh65doW6yXF6SWYqAk9PkBE18l99BR86lbBj5upNbm4ntSx/SgVCJcHpHEx9vh 6vuw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id f20-20020a17090ac29400b00262ca5c4c12si935419pjt.178.2023.11.29.01.55.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Nov 2023 01:55:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 56545809BE4D; Wed, 29 Nov 2023 01:55:35 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232700AbjK2Jyk (ORCPT + 99 others); Wed, 29 Nov 2023 04:54:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230349AbjK2Jx5 (ORCPT ); Wed, 29 Nov 2023 04:53:57 -0500 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2a07:de40:b251:101:10:150:64:1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7D6B319AD for ; Wed, 29 Nov 2023 01:53:39 -0800 (PST) Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id E262721992; Wed, 29 Nov 2023 09:53:37 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 3E5FA13A9B; Wed, 29 Nov 2023 09:53:37 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id yPPuDiEKZ2UrfQAAD6G6ig (envelope-from ); Wed, 29 Nov 2023 09:53:37 +0000 From: Vlastimil Babka Date: Wed, 29 Nov 2023 10:53:30 +0100 Subject: [PATCH RFC v3 5/9] mm/slub: add opt-in percpu array cache of objects MIME-Version: 1.0 Message-Id: <20231129-slub-percpu-caches-v3-5-6bcf536772bc@suse.cz> References: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> In-Reply-To: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Matthew Wilcox , "Liam R. Howlett" Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Alexander Potapenko , Marco Elver , Dmitry Vyukov , linux-mm@kvack.org, linux-kernel@vger.kernel.org, maple-tree@lists.infradead.org, kasan-dev@googlegroups.com, Vlastimil Babka X-Mailer: b4 0.12.4 X-Spam-Level: X-Rspamd-Server: rspamd1 Authentication-Results: smtp-out1.suse.de; none X-Rspamd-Queue-Id: E262721992 X-Spam-Score: -4.00 X-Spamd-Result: default: False [-4.00 / 50.00]; TAGGED_RCPT(0.00)[]; REPLY(-4.00)[] X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Wed, 29 Nov 2023 01:55:35 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783891749069529778 X-GMAIL-MSGID: 1783891749069529778 kmem_cache_setup_percpu_array() will allocate a per-cpu array for caching alloc/free objects of given size for the cache. The cache has to be created with SLAB_NO_MERGE flag. When empty, half of the array is filled by an internal bulk alloc operation. When full, half of the array is flushed by an internal bulk free operation. The array does not distinguish NUMA locality of the cached objects. If an allocation is requested with kmem_cache_alloc_node() with numa node not equal to NUMA_NO_NODE, the array is bypassed. The bulk operations exposed to slab users also try to utilize the array when possible, but leave the array empty or full and use the bulk alloc/free only to finish the operation itself. If kmemcg is enabled and active, bulk freeing skips the array completely as it would be less efficient to use it. The locking scheme is copied from the page allocator's pcplists, based on embedded spin locks. Interrupts are not disabled, only preemption (cpu migration on RT). Trylock is attempted to avoid deadlock due to an interrupt; trylock failure means the array is bypassed. Sysfs stat counters alloc_cpu_cache and free_cpu_cache count objects allocated or freed using the percpu array; counters cpu_cache_refill and cpu_cache_flush count objects refilled or flushed form the array. kmem_cache_prefill_percpu_array() can be called to ensure the array on the current cpu to at least the given number of objects. However this is only opportunistic as there's no cpu pinning between the prefill and usage, and trylocks may fail when the usage is in an irq handler. Therefore allocations cannot rely on the array for success even after the prefill. But misses should be rare enough that e.g. GFP_ATOMIC allocations should be acceptable after the refill. When slub_debug is enabled for a cache with percpu array, the objects in the array are considered as allocated from the slub_debug perspective, and the alloc/free debugging hooks occur when moving the objects between the array and slab pages. This means that e.g. an use-after-free that occurs for an object cached in the array is undetected. Collected alloc/free stacktraces might also be less useful. This limitation could be changed in the future. On the other hand, KASAN, kmemcg and other hooks are executed on actual allocations and frees by kmem_cache users even if those use the array, so their debugging or accounting accuracy should be unaffected. Signed-off-by: Vlastimil Babka --- include/linux/slab.h | 4 + include/linux/slub_def.h | 12 ++ mm/Kconfig | 1 + mm/slub.c | 457 ++++++++++++++++++++++++++++++++++++++++++++++- 4 files changed, 468 insertions(+), 6 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index d6d6ffeeb9a2..fe0c0981be59 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -197,6 +197,8 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name, void kmem_cache_destroy(struct kmem_cache *s); int kmem_cache_shrink(struct kmem_cache *s); +int kmem_cache_setup_percpu_array(struct kmem_cache *s, unsigned int count); + /* * Please use this macro to create slab caches. Simply specify the * name of the structure and maybe some flags that are listed above. @@ -512,6 +514,8 @@ void kmem_cache_free(struct kmem_cache *s, void *objp); void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p); int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p); +int kmem_cache_prefill_percpu_array(struct kmem_cache *s, unsigned int count, gfp_t gfp); + static __always_inline void kfree_bulk(size_t size, void **p) { kmem_cache_free_bulk(NULL, size, p); diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index deb90cf4bffb..2083aa849766 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -13,8 +13,10 @@ #include enum stat_item { + ALLOC_PCA, /* Allocation from percpu array cache */ ALLOC_FASTPATH, /* Allocation from cpu slab */ ALLOC_SLOWPATH, /* Allocation by getting a new cpu slab */ + FREE_PCA, /* Free to percpu array cache */ FREE_FASTPATH, /* Free to cpu slab */ FREE_SLOWPATH, /* Freeing not to cpu slab */ FREE_FROZEN, /* Freeing to frozen slab */ @@ -39,6 +41,8 @@ enum stat_item { CPU_PARTIAL_FREE, /* Refill cpu partial on free */ CPU_PARTIAL_NODE, /* Refill cpu partial from node partial */ CPU_PARTIAL_DRAIN, /* Drain cpu partial to node partial */ + PCA_REFILL, /* Refilling empty percpu array cache */ + PCA_FLUSH, /* Flushing full percpu array cache */ NR_SLUB_STAT_ITEMS }; @@ -66,6 +70,13 @@ struct kmem_cache_cpu { }; #endif /* CONFIG_SLUB_TINY */ +struct slub_percpu_array { + spinlock_t lock; + unsigned int count; + unsigned int used; + void * objects[]; +}; + #ifdef CONFIG_SLUB_CPU_PARTIAL #define slub_percpu_partial(c) ((c)->partial) @@ -99,6 +110,7 @@ struct kmem_cache { #ifndef CONFIG_SLUB_TINY struct kmem_cache_cpu __percpu *cpu_slab; #endif + struct slub_percpu_array __percpu *cpu_array; /* Used for retrieving partial slabs, etc. */ slab_flags_t flags; unsigned long min_partial; diff --git a/mm/Kconfig b/mm/Kconfig index 89971a894b60..aa53c51bb4a6 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -237,6 +237,7 @@ choice config SLAB_DEPRECATED bool "SLAB (DEPRECATED)" depends on !PREEMPT_RT + depends on BROKEN help Deprecated and scheduled for removal in a few cycles. Replaced by SLUB. diff --git a/mm/slub.c b/mm/slub.c index 59912a376c6d..f08bd71c244f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -188,6 +188,79 @@ do { \ #define USE_LOCKLESS_FAST_PATH() (false) #endif +/* copy/pasted from mm/page_alloc.c */ + +#if defined(CONFIG_SMP) || defined(CONFIG_PREEMPT_RT) +/* + * On SMP, spin_trylock is sufficient protection. + * On PREEMPT_RT, spin_trylock is equivalent on both SMP and UP. + */ +#define pcp_trylock_prepare(flags) do { } while (0) +#define pcp_trylock_finish(flag) do { } while (0) +#else + +/* UP spin_trylock always succeeds so disable IRQs to prevent re-entrancy. */ +#define pcp_trylock_prepare(flags) local_irq_save(flags) +#define pcp_trylock_finish(flags) local_irq_restore(flags) +#endif + +/* + * Locking a pcp requires a PCP lookup followed by a spinlock. To avoid + * a migration causing the wrong PCP to be locked and remote memory being + * potentially allocated, pin the task to the CPU for the lookup+lock. + * preempt_disable is used on !RT because it is faster than migrate_disable. + * migrate_disable is used on RT because otherwise RT spinlock usage is + * interfered with and a high priority task cannot preempt the allocator. + */ +#ifndef CONFIG_PREEMPT_RT +#define pcpu_task_pin() preempt_disable() +#define pcpu_task_unpin() preempt_enable() +#else +#define pcpu_task_pin() migrate_disable() +#define pcpu_task_unpin() migrate_enable() +#endif + +/* + * Generic helper to lookup and a per-cpu variable with an embedded spinlock. + * Return value should be used with equivalent unlock helper. + */ +#define pcpu_spin_lock(type, member, ptr) \ +({ \ + type *_ret; \ + pcpu_task_pin(); \ + _ret = this_cpu_ptr(ptr); \ + spin_lock(&_ret->member); \ + _ret; \ +}) + +#define pcpu_spin_trylock(type, member, ptr) \ +({ \ + type *_ret; \ + pcpu_task_pin(); \ + _ret = this_cpu_ptr(ptr); \ + if (!spin_trylock(&_ret->member)) { \ + pcpu_task_unpin(); \ + _ret = NULL; \ + } \ + _ret; \ +}) + +#define pcpu_spin_unlock(member, ptr) \ +({ \ + spin_unlock(&ptr->member); \ + pcpu_task_unpin(); \ +}) + +/* struct slub_percpu_array specific helpers. */ +#define pca_spin_lock(ptr) \ + pcpu_spin_lock(struct slub_percpu_array, lock, ptr) + +#define pca_spin_trylock(ptr) \ + pcpu_spin_trylock(struct slub_percpu_array, lock, ptr) + +#define pca_spin_unlock(ptr) \ + pcpu_spin_unlock(lock, ptr) + #ifndef CONFIG_SLUB_TINY #define __fastpath_inline __always_inline #else @@ -3454,6 +3527,78 @@ static __always_inline void maybe_wipe_obj_freeptr(struct kmem_cache *s, 0, sizeof(void *)); } +static bool refill_pca(struct kmem_cache *s, unsigned int count, gfp_t gfp); + +static __fastpath_inline +void *alloc_from_pca(struct kmem_cache *s, gfp_t gfp) +{ + unsigned long __maybe_unused UP_flags; + struct slub_percpu_array *pca; + void *object; + +retry: + pcp_trylock_prepare(UP_flags); + pca = pca_spin_trylock(s->cpu_array); + + if (unlikely(!pca)) { + pcp_trylock_finish(UP_flags); + return NULL; + } + + if (unlikely(pca->used == 0)) { + unsigned int batch = pca->count / 2; + + pca_spin_unlock(pca); + pcp_trylock_finish(UP_flags); + + if (!gfpflags_allow_blocking(gfp) || in_irq()) + return NULL; + + if (refill_pca(s, batch, gfp)) + goto retry; + + return NULL; + } + + object = pca->objects[--pca->used]; + + pca_spin_unlock(pca); + pcp_trylock_finish(UP_flags); + + stat(s, ALLOC_PCA); + + return object; +} + +static __fastpath_inline +int alloc_from_pca_bulk(struct kmem_cache *s, size_t size, void **p) +{ + unsigned long __maybe_unused UP_flags; + struct slub_percpu_array *pca; + + pcp_trylock_prepare(UP_flags); + pca = pca_spin_trylock(s->cpu_array); + + if (unlikely(!pca)) { + size = 0; + goto failed; + } + + if (pca->used < size) + size = pca->used; + + for (int i = size; i > 0;) { + p[--i] = pca->objects[--pca->used]; + } + + pca_spin_unlock(pca); + stat_add(s, ALLOC_PCA, size); + +failed: + pcp_trylock_finish(UP_flags); + return size; +} + /* * Inlined fastpath so that allocation functions (kmalloc, kmem_cache_alloc) * have the fastpath folded into their functions. So no function call @@ -3479,7 +3624,11 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list if (unlikely(object)) goto out; - object = __slab_alloc_node(s, gfpflags, node, addr, orig_size); + if (s->cpu_array && (node == NUMA_NO_NODE)) + object = alloc_from_pca(s, gfpflags); + + if (!object) + object = __slab_alloc_node(s, gfpflags, node, addr, orig_size); maybe_wipe_obj_freeptr(s, object); init = slab_want_init_on_alloc(gfpflags, s); @@ -3726,6 +3875,81 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, discard_slab(s, slab); } +static bool flush_pca(struct kmem_cache *s, unsigned int count); + +static __fastpath_inline +bool free_to_pca(struct kmem_cache *s, void *object) +{ + unsigned long __maybe_unused UP_flags; + struct slub_percpu_array *pca; + +retry: + pcp_trylock_prepare(UP_flags); + pca = pca_spin_trylock(s->cpu_array); + + if (!pca) { + pcp_trylock_finish(UP_flags); + return false; + } + + if (pca->used == pca->count) { + unsigned int batch = pca->count / 2; + + pca_spin_unlock(pca); + pcp_trylock_finish(UP_flags); + + if (in_irq()) + return false; + + if (!flush_pca(s, batch)) + return false; + + goto retry; + } + + pca->objects[pca->used++] = object; + + pca_spin_unlock(pca); + pcp_trylock_finish(UP_flags); + + stat(s, FREE_PCA); + + return true; +} + +static __fastpath_inline +size_t free_to_pca_bulk(struct kmem_cache *s, size_t size, void **p) +{ + unsigned long __maybe_unused UP_flags; + struct slub_percpu_array *pca; + bool init; + + pcp_trylock_prepare(UP_flags); + pca = pca_spin_trylock(s->cpu_array); + + if (unlikely(!pca)) { + size = 0; + goto failed; + } + + if (pca->count - pca->used < size) + size = pca->count - pca->used; + + init = slab_want_init_on_free(s); + + for (size_t i = 0; i < size; i++) { + if (likely(slab_free_hook(s, p[i], init))) + pca->objects[pca->used++] = p[i]; + } + + pca_spin_unlock(pca); + stat_add(s, FREE_PCA, size); + +failed: + pcp_trylock_finish(UP_flags); + return size; +} + #ifndef CONFIG_SLUB_TINY /* * Fastpath with forced inlining to produce a kfree and kmem_cache_free that @@ -3811,7 +4035,12 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object, { memcg_slab_free_hook(s, slab, &object, 1); - if (likely(slab_free_hook(s, object, slab_want_init_on_free(s)))) + if (unlikely(!slab_free_hook(s, object, slab_want_init_on_free(s)))) + return; + + if (s->cpu_array) + free_to_pca(s, object); + else do_slab_free(s, slab, object, object, 1, addr); } @@ -3956,6 +4185,26 @@ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p) if (!size) return; + /* + * In case the objects might need memcg_slab_free_hook(), skip the array + * because the hook is not effective with single objects and benefits + * from groups of objects from a single slab that the detached freelist + * builds. But once we build the detached freelist, it's wasteful to + * throw it away and put the objects into the array. + * + * XXX: This test could be cache-specific if it was not possible to use + * __GFP_ACCOUNT with caches that are not SLAB_ACCOUNT + */ + if (s && s->cpu_array && !memcg_kmem_online()) { + size_t pca_freed = free_to_pca_bulk(s, size, p); + + if (pca_freed == size) + return; + + p += pca_freed; + size -= pca_freed; + } + do { struct detached_freelist df; @@ -4073,7 +4322,8 @@ static int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p) { - int i; + int from_pca = 0; + int allocated = 0; struct obj_cgroup *objcg = NULL; if (!size) @@ -4084,19 +4334,147 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, if (unlikely(!s)) return 0; - i = __kmem_cache_alloc_bulk(s, flags, size, p); + if (s->cpu_array) + from_pca = alloc_from_pca_bulk(s, size, p); + + if (from_pca < size) { + allocated = __kmem_cache_alloc_bulk(s, flags, size-from_pca, + p+from_pca); + if (allocated == 0 && from_pca > 0) { + __kmem_cache_free_bulk(s, from_pca, p); + } + } + + allocated += from_pca; /* * memcg and kmem_cache debug support and memory initialization. * Done outside of the IRQ disabled fastpath loop. */ - if (i != 0) + if (allocated != 0) slab_post_alloc_hook(s, objcg, flags, size, p, slab_want_init_on_alloc(flags, s), s->object_size); - return i; + return allocated; } EXPORT_SYMBOL(kmem_cache_alloc_bulk); +static bool refill_pca(struct kmem_cache *s, unsigned int count, gfp_t gfp) +{ + void *objects[32]; + unsigned int batch, allocated; + unsigned long __maybe_unused UP_flags; + struct slub_percpu_array *pca; + +bulk_alloc: + batch = min(count, 32U); + + allocated = __kmem_cache_alloc_bulk(s, gfp, batch, &objects[0]); + if (!allocated) + return false; + + pcp_trylock_prepare(UP_flags); + pca = pca_spin_trylock(s->cpu_array); + if (!pca) { + pcp_trylock_finish(UP_flags); + return false; + } + + batch = min(allocated, pca->count - pca->used); + + for (unsigned int i = 0; i < batch; i++) { + pca->objects[pca->used++] = objects[i]; + } + + pca_spin_unlock(pca); + pcp_trylock_finish(UP_flags); + + stat_add(s, PCA_REFILL, batch); + + /* + * We could have migrated to a different cpu or somebody else freed to the + * pca while we were bulk allocating, and now we have too many objects + */ + if (batch < allocated) { + __kmem_cache_free_bulk(s, allocated - batch, &objects[batch]); + } else { + count -= batch; + if (count > 0) + goto bulk_alloc; + } + + return true; +} + +static bool flush_pca(struct kmem_cache *s, unsigned int count) +{ + void *objects[32]; + unsigned int batch, remaining; + unsigned long __maybe_unused UP_flags; + struct slub_percpu_array *pca; + +next_batch: + batch = min(count, 32); + + pcp_trylock_prepare(UP_flags); + pca = pca_spin_trylock(s->cpu_array); + if (!pca) { + pcp_trylock_finish(UP_flags); + return false; + } + + batch = min(batch, pca->used); + + for (unsigned int i = 0; i < batch; i++) { + objects[i] = pca->objects[--pca->used]; + } + + remaining = pca->used; + + pca_spin_unlock(pca); + pcp_trylock_finish(UP_flags); + + __kmem_cache_free_bulk(s, batch, &objects[0]); + + stat_add(s, PCA_FLUSH, batch); + + if (batch < count && remaining > 0) { + count -= batch; + goto next_batch; + } + + return true; +} + +/* Do not call from irq handler nor with irqs disabled */ +int kmem_cache_prefill_percpu_array(struct kmem_cache *s, unsigned int count, + gfp_t gfp) +{ + struct slub_percpu_array *pca; + unsigned int used; + + lockdep_assert_no_hardirq(); + + if (!s->cpu_array) + return -EINVAL; + + /* racy but we don't care */ + pca = raw_cpu_ptr(s->cpu_array); + + used = READ_ONCE(pca->used); + + if (used >= count) + return 0; + + if (pca->count < count) + return -EINVAL; + + count -= used; + + if (!refill_pca(s, count, gfp)) + return -ENOMEM; + + return 0; +} /* * Object placement in a slab is made very easy because we always start at @@ -5167,6 +5545,65 @@ int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) return 0; } +/** + * kmem_cache_setup_percpu_array - Create a per-cpu array cache for the cache + * @s: The cache to add per-cpu array. Must be created with SLAB_NO_MERGE flag. + * @count: Size of the per-cpu array. + * + * After this call, allocations from the cache go through a percpu array. When + * it becomes empty, half is refilled with a bulk allocation. When it becomes + * full, half is flushed with a bulk free operation. + * + * Using the array cache is not guaranteed, i.e. it can be bypassed if its lock + * cannot be obtained. The array cache also does not distinguish NUMA nodes, so + * allocations via kmem_cache_alloc_node() with a node specified other than + * NUMA_NO_NODE will bypass the cache. + * + * Bulk allocation and free operations also try to use the array. + * + * kmem_cache_prefill_percpu_array() can be used to pre-fill the array cache + * before e.g. entering a restricted context. It is however not guaranteed that + * the caller will be able to subsequently consume the prefilled cache. Such + * failures should be however sufficiently rare so after the prefill, + * allocations using GFP_ATOMIC | __GFP_NOFAIL are acceptable for objects up to + * the prefilled amount. + * + * Limitations: when slub_debug is enabled for the cache, all relevant actions + * (i.e. poisoning, obtaining stacktraces) and checks happen when objects move + * between the array cache and slab pages, which may result in e.g. not + * detecting a use-after-free while the object is in the array cache, and the + * stacktraces may be less useful. + * + * Return: 0 if OK, -EINVAL on caches without SLAB_NO_MERGE or with the array + * already created, -ENOMEM when the per-cpu array creation fails. + */ +int kmem_cache_setup_percpu_array(struct kmem_cache *s, unsigned int count) +{ + int cpu; + + if (WARN_ON_ONCE(!(s->flags & SLAB_NO_MERGE))) + return -EINVAL; + + if (s->cpu_array) + return -EINVAL; + + s->cpu_array = __alloc_percpu(struct_size(s->cpu_array, objects, count), + sizeof(void *)); + + if (!s->cpu_array) + return -ENOMEM; + + for_each_possible_cpu(cpu) { + struct slub_percpu_array *pca = per_cpu_ptr(s->cpu_array, cpu); + + spin_lock_init(&pca->lock); + pca->count = count; + pca->used = 0; + } + + return 0; +} + #ifdef SLAB_SUPPORTS_SYSFS static int count_inuse(struct slab *slab) { @@ -5944,8 +6381,10 @@ static ssize_t text##_store(struct kmem_cache *s, \ } \ SLAB_ATTR(text); \ +STAT_ATTR(ALLOC_PCA, alloc_cpu_cache); STAT_ATTR(ALLOC_FASTPATH, alloc_fastpath); STAT_ATTR(ALLOC_SLOWPATH, alloc_slowpath); +STAT_ATTR(FREE_PCA, free_cpu_cache); STAT_ATTR(FREE_FASTPATH, free_fastpath); STAT_ATTR(FREE_SLOWPATH, free_slowpath); STAT_ATTR(FREE_FROZEN, free_frozen); @@ -5970,6 +6409,8 @@ STAT_ATTR(CPU_PARTIAL_ALLOC, cpu_partial_alloc); STAT_ATTR(CPU_PARTIAL_FREE, cpu_partial_free); STAT_ATTR(CPU_PARTIAL_NODE, cpu_partial_node); STAT_ATTR(CPU_PARTIAL_DRAIN, cpu_partial_drain); +STAT_ATTR(PCA_REFILL, cpu_cache_refill); +STAT_ATTR(PCA_FLUSH, cpu_cache_flush); #endif /* CONFIG_SLUB_STATS */ #ifdef CONFIG_KFENCE @@ -6031,8 +6472,10 @@ static struct attribute *slab_attrs[] = { &remote_node_defrag_ratio_attr.attr, #endif #ifdef CONFIG_SLUB_STATS + &alloc_cpu_cache_attr.attr, &alloc_fastpath_attr.attr, &alloc_slowpath_attr.attr, + &free_cpu_cache_attr.attr, &free_fastpath_attr.attr, &free_slowpath_attr.attr, &free_frozen_attr.attr, @@ -6057,6 +6500,8 @@ static struct attribute *slab_attrs[] = { &cpu_partial_free_attr.attr, &cpu_partial_node_attr.attr, &cpu_partial_drain_attr.attr, + &cpu_cache_refill_attr.attr, + &cpu_cache_flush_attr.attr, #endif #ifdef CONFIG_FAILSLAB &failslab_attr.attr, From patchwork Wed Nov 29 09:53:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 171220 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a5a7:0:b0:403:3b70:6f57 with SMTP id d7csp229327vqn; Wed, 29 Nov 2023 01:54:42 -0800 (PST) X-Google-Smtp-Source: AGHT+IHeZyiIPSr7n6H9Y/snX84uzX4RY25dVFHYT4U7a0QR84D0UdqLu+leSPMNk+r8MVDmKby1 X-Received: by 2002:a17:902:dad0:b0:1cf:da41:7293 with SMTP id q16-20020a170902dad000b001cfda417293mr10021104plx.15.1701251682372; Wed, 29 Nov 2023 01:54:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701251682; cv=none; d=google.com; s=arc-20160816; b=tt0O+/wTc5+GAZ3MRHSJky0O80vqO0ijNig0sXk1kofOkxk+5RxDgtaxrbjDj2Stxq 4fWi/2x6o/RMo0I4Ta1VHjrpEgATVI6zLNjgXa67GCGSEsTTSthK0oWD6s3/TKu9/tXS huXPA4V3Q6kKPsvXbNtgX1X45L+UIpLMTXysIlKAWbzRNNwB9h9C2yxrdK2RPjfXA/Bj RnXFMzKYZD8pOgU2xIuJnslc9c7W87bz6k5wLyTHS+itSzfrtaCxCrE8aB3QkyOC44WS Y6d6GUpv0OQmZFTTIts8DuwCWzJ7ZQ5oxT75C4o33rpr3WyceDENbg5xVoYsKzfEsZy0 nBbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:in-reply-to:references:message-id :content-transfer-encoding:mime-version:subject:date:from; bh=PX3MvAB9bT05yeaZSR4WJfiyLfcVNVB/Ayxh6IAIMM0=; fh=uYnIsWZ9n80gkcnhZMgoujzKtxA7UEl4GQvTBFLbimw=; b=ksUgkNtd6M4yqYH+Beisr0sw0//eaXzRdPGDSvnR7IhXHS5AHeUUdg6FADwfXlZQ4/ OW38KvQNx55FayZu8YupYBfco2DacB1zOsln+azhhC2YCMT/uJb+7xJ3ShXmJBgfsmdX jXZwYiSInmD44NS8nNPEHbFUJ93TaeYK27F/ct/xBsWHfZjImkFQdbR+xhS6eZQWYXEt jSrX8hUEPMJyso6Rg2HopGJQr+KqcmjII95bFaRkLn0UYDMmUeoyw9l+82g/1twLvhRV xOX/PAA3QN8w5CYsKlmN4wGP4KTDcmbFdkJ5DRmhwPRGgO7F7vhwAwQassUyyqInX7Rk FhCw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id u6-20020a170902e80600b001cfdaca2b92si5379963plg.386.2023.11.29.01.54.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Nov 2023 01:54:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 4F8AD804A9E9; Wed, 29 Nov 2023 01:54:35 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231255AbjK2JyL (ORCPT + 99 others); Wed, 29 Nov 2023 04:54:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55430 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231247AbjK2Jxw (ORCPT ); Wed, 29 Nov 2023 04:53:52 -0500 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2a07:de40:b251:101:10:150:64:1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96CE319BC for ; Wed, 29 Nov 2023 01:53:39 -0800 (PST) Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id E628721995; Wed, 29 Nov 2023 09:53:37 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 59C0B13A9D; Wed, 29 Nov 2023 09:53:37 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id yN2eFSEKZ2UrfQAAD6G6ig (envelope-from ); Wed, 29 Nov 2023 09:53:37 +0000 From: Vlastimil Babka Date: Wed, 29 Nov 2023 10:53:31 +0100 Subject: [PATCH RFC v3 6/9] tools: Add SLUB percpu array functions for testing MIME-Version: 1.0 Message-Id: <20231129-slub-percpu-caches-v3-6-6bcf536772bc@suse.cz> References: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> In-Reply-To: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Matthew Wilcox , "Liam R. Howlett" Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Alexander Potapenko , Marco Elver , Dmitry Vyukov , linux-mm@kvack.org, linux-kernel@vger.kernel.org, maple-tree@lists.infradead.org, kasan-dev@googlegroups.com, Vlastimil Babka X-Mailer: b4 0.12.4 X-Spam-Level: X-Rspamd-Server: rspamd1 Authentication-Results: smtp-out1.suse.de; none X-Rspamd-Queue-Id: E628721995 X-Spam-Score: -4.00 X-Spamd-Result: default: False [-4.00 / 50.00]; TAGGED_RCPT(0.00)[]; REPLY(-4.00)[] X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 29 Nov 2023 01:54:35 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783891684377595482 X-GMAIL-MSGID: 1783891684377595482 From: "Liam R. Howlett" Support new percpu array functions to the test code so they can be used in the maple tree testing. Signed-off-by: Liam R. Howlett Signed-off-by: Vlastimil Babka --- tools/include/linux/slab.h | 4 ++++ tools/testing/radix-tree/linux.c | 14 ++++++++++++++ tools/testing/radix-tree/linux/kernel.h | 1 + 3 files changed, 19 insertions(+) diff --git a/tools/include/linux/slab.h b/tools/include/linux/slab.h index 311759ea25e9..1043f9c5ef4e 100644 --- a/tools/include/linux/slab.h +++ b/tools/include/linux/slab.h @@ -7,6 +7,7 @@ #define SLAB_PANIC 2 #define SLAB_RECLAIM_ACCOUNT 0x00020000UL /* Objects are reclaimable */ +#define SLAB_NO_MERGE 0x01000000UL /* Prevent merging with compatible kmem caches */ #define kzalloc_node(size, flags, node) kmalloc(size, flags) @@ -45,4 +46,7 @@ void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list); int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size, void **list); +int kmem_cache_setup_percpu_array(struct kmem_cache *s, unsigned int count); +int kmem_cache_prefill_percpu_array(struct kmem_cache *s, unsigned int count, + gfp_t gfp); #endif /* _TOOLS_SLAB_H */ diff --git a/tools/testing/radix-tree/linux.c b/tools/testing/radix-tree/linux.c index 61fe2601cb3a..3c9372afe9bc 100644 --- a/tools/testing/radix-tree/linux.c +++ b/tools/testing/radix-tree/linux.c @@ -187,6 +187,20 @@ int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size, return size; } +int kmem_cache_setup_percpu_array(struct kmem_cache *s, unsigned int count) +{ + return 0; +} + +int kmem_cache_prefill_percpu_array(struct kmem_cache *s, unsigned int count, + gfp_t gfp) +{ + if (count > s->non_kernel) + return s->non_kernel; + + return count; +} + struct kmem_cache * kmem_cache_create(const char *name, unsigned int size, unsigned int align, unsigned int flags, void (*ctor)(void *)) diff --git a/tools/testing/radix-tree/linux/kernel.h b/tools/testing/radix-tree/linux/kernel.h index c5c9d05f29da..fc75018974de 100644 --- a/tools/testing/radix-tree/linux/kernel.h +++ b/tools/testing/radix-tree/linux/kernel.h @@ -15,6 +15,7 @@ #define printk printf #define pr_err printk +#define pr_warn printk #define pr_info printk #define pr_debug printk #define pr_cont printk From patchwork Wed Nov 29 09:53:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 171219 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a5a7:0:b0:403:3b70:6f57 with SMTP id d7csp229233vqn; Wed, 29 Nov 2023 01:54:22 -0800 (PST) X-Google-Smtp-Source: AGHT+IFlHjg67MtJzhOQMcsAR8G8yYBjAnNY9dJ/AaMHGkp1LPI//G3+X/IlKLs7wbxkPgWE/vcL X-Received: by 2002:a05:6830:b87:b0:6d8:1992:5465 with SMTP id a7-20020a0568300b8700b006d819925465mr11048145otv.17.1701251661801; Wed, 29 Nov 2023 01:54:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701251661; cv=none; d=google.com; s=arc-20160816; b=VRIDeVmfhWJz1STb1/2VMUsPuGV2YTCIyNNbxS3YsPErAGZneV4sDaqQd+bSfl8dUa QVLyua1AoHY2eTXiv0b49zINeyrO6Nl8c6Z37R2/SFOb7S6DxhX45A+ogH+qquE1SGbv xcFdV3rKe+GX0dDjwmMXyR3IJWunKF+tNTXKQkeZ0LlPuy8/EUj8P85LFfazEmWUjm4O DPdVgmTT6N/TsFPcGmQ4pg4+VuDiQc3is6CKzTduzY6hOoHF/eARShPyNgPT0mKoRW5s 6VketWfi6ZVvvXijYQSd0TOMx2YM0fTHDZRhR8mK+J+qB6liwPe9o1yq0o/fKOQpDp2D 4Reg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:in-reply-to:references:message-id :content-transfer-encoding:mime-version:subject:date:from; bh=sIzawMHw0qEvdtsrb8CaYai+9KHMb5Dh5VZ/ALQ2CnQ=; fh=BM+YXXPDItJMOh98QtZoCEH1RXzAZsCss4yDzENT5Z4=; b=KLqvtgEFbJZlxip5dLAT3tW+CsSQEVRO44b0p1hXF4SH4BpE7ICUzjsTVWCTVaLybR f/R2lUnTsY4Uem0IM/B2EuNxXyCOAiPPzwdK5XQOZzq4DX92Prms28vS5olDyHMM8LNS ww+D+h3HWkV/nXqS0HOHjkItZsxbqEmg2B119YM6l75Jw4sb6lianEbS9yK8sXaZ9/iD r68Srqze74YZ5XoP8BaIeQvYghvbLBxt2F9SOs1XICmlOXshje31nxoCMjT6mMkspX3B 9qlbComudQd7jIeHPxYc8gJ+ZrhkJf7EfXKRl9TWzzsQU0qnaLgUfYmydEy7zkBbf460 zB5g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id 13-20020a63114d000000b005b96af23fecsi14609056pgr.496.2023.11.29.01.54.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Nov 2023 01:54:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 6C01580AD0A6; Wed, 29 Nov 2023 01:54:18 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231709AbjK2JyJ (ORCPT + 99 others); Wed, 29 Nov 2023 04:54:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231190AbjK2Jxv (ORCPT ); Wed, 29 Nov 2023 04:53:51 -0500 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9C4991FF9 for ; Wed, 29 Nov 2023 01:53:39 -0800 (PST) Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id EE57F1F8BB; Wed, 29 Nov 2023 09:53:37 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 73E2B13A9E; Wed, 29 Nov 2023 09:53:37 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id eE8DHCEKZ2UrfQAAD6G6ig (envelope-from ); Wed, 29 Nov 2023 09:53:37 +0000 From: Vlastimil Babka Date: Wed, 29 Nov 2023 10:53:32 +0100 Subject: [PATCH RFC v3 7/9] maple_tree: use slub percpu array MIME-Version: 1.0 Message-Id: <20231129-slub-percpu-caches-v3-7-6bcf536772bc@suse.cz> References: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> In-Reply-To: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Matthew Wilcox , "Liam R. Howlett" Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Alexander Potapenko , Marco Elver , Dmitry Vyukov , linux-mm@kvack.org, linux-kernel@vger.kernel.org, maple-tree@lists.infradead.org, kasan-dev@googlegroups.com X-Mailer: b4 0.12.4 X-Spamd-Bar: ++++++++++++ X-Spam-Score: 12.58 X-Rspamd-Server: rspamd1 Authentication-Results: smtp-out2.suse.de; dkim=none; spf=softfail (smtp-out2.suse.de: 2a07:de40:b281:104:10:150:64:97 is neither permitted nor denied by domain of vbabka@suse.cz) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspamd-Queue-Id: EE57F1F8BB X-Spamd-Result: default: False [12.58 / 50.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; SPAMHAUS_XBL(0.00)[2a07:de40:b281:104:10:150:64:97:from]; TO_DN_SOME(0.00)[]; R_SPF_SOFTFAIL(4.60)[~all:c]; R_RATELIMIT(0.00)[to_ip_from(RLhc4kaujr6ihojcnjq7c1jwbi)]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[]; NEURAL_HAM_SHORT(-0.20)[-0.999]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(2.20)[]; MIME_TRACE(0.00)[0:+]; MID_RHS_MATCH_FROM(0.00)[]; BAYES_HAM(-0.00)[28.50%]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(1.20)[suse.cz]; NEURAL_SPAM_LONG(3.39)[0.969]; RCPT_COUNT_TWELVE(0.00)[16]; FUZZY_BLOCKED(0.00)[rspamd.com]; FREEMAIL_CC(0.00)[linux-foundation.org,linux.dev,gmail.com,google.com,kvack.org,vger.kernel.org,lists.infradead.org,googlegroups.com]; RCVD_TLS_ALL(0.00)[]; SUSPICIOUS_RECIPS(1.50)[] X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 29 Nov 2023 01:54:18 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783891662163296673 X-GMAIL-MSGID: 1783891662163296673 Just make sure the maple_node_cache has a percpu array of size 32. Will break with CONFIG_SLAB. --- lib/maple_tree.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index bb24d84a4922..d9e7088fd9a7 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -6213,9 +6213,16 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp) void __init maple_tree_init(void) { + int ret; + maple_node_cache = kmem_cache_create("maple_node", sizeof(struct maple_node), sizeof(struct maple_node), - SLAB_PANIC, NULL); + SLAB_PANIC | SLAB_NO_MERGE, NULL); + + ret = kmem_cache_setup_percpu_array(maple_node_cache, 32); + + if (ret) + pr_warn("error %d creating percpu_array for maple_node_cache\n", ret); } /** From patchwork Wed Nov 29 09:53:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 171222 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a5a7:0:b0:403:3b70:6f57 with SMTP id d7csp229387vqn; Wed, 29 Nov 2023 01:54:50 -0800 (PST) X-Google-Smtp-Source: AGHT+IHTaah9l5C9Q5xYzOJriWu1WCnUs/OXamGNtFhzMxWhm3wonhatlBKt1HpnodjDG1gCBhbr X-Received: by 2002:a92:d586:0:b0:35c:92a4:7111 with SMTP id a6-20020a92d586000000b0035c92a47111mr14793063iln.16.1701251690622; Wed, 29 Nov 2023 01:54:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701251690; cv=none; d=google.com; s=arc-20160816; b=WgTl8W9n++BWdTYwIMD6dPZyGvQQv4aUw4sq/gqBopOy2Fbsnhl8Oo5K6q/DpcKd54 Eu8eaCmZ7CsGnoDDlfjScYupK6I28izIXSrDPfyGzF6YkKMIW2EF6AmGJFBykK+WisIo 2788bMnf0t06Lzj7cLv/Qu24Dp9h0yW9X7rhpBLSxcfZV1VjQiGBFsC6v4bw8OFDl0kk 2UCKbSqTYnhKjNePgXgqALgALdOCIEOyg0zq2eNE5IkXjomRYev/XYby/5iGdYxg9t2J kY3z8w7oFgqFWKgozERZlrl8ZoIci9SySDCyyH186PxiCEVKE/COjYJpAWSADyZN2VHh /8QA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:in-reply-to:references:message-id :content-transfer-encoding:mime-version:subject:date:from; bh=ItnxEFdMdReUcgI2eC5m3fUPnWv97oIpUOC4WdcPO3w=; fh=uYnIsWZ9n80gkcnhZMgoujzKtxA7UEl4GQvTBFLbimw=; b=ycFugbwXKSh/3uTKTv63IT/BoUpZ3j2k/M+7mqawhBoKKzhJQFdS5BcKKYaJ6H5IhO svMS2YUyWegnfGgdwS/WcZyt6cVqLyDK4fV+RgwVS4NTZSJfoIt6oGaJia7ApsSl+tbH xD9vGk+p6JZFyI0YSlRMqwd3vGx4EEBnDgOh2tjC2Vf3ob820Y2cQLFKyU1aR7bUzoQm t167eiXS9jQxS7nE7T6UHgOJjTrjQToVCs87ONjD+C4dak72jeg0YE4ilflwU+fI1QGf m6iPaUJf18UuUdfubIGENkoPnCQz1utnpZiPB/eLXSGhUl6xazMcCVHgS7QqOSW1F+po 2pmQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id r3-20020a62e403000000b006cc0519f59fsi9202008pfh.131.2023.11.29.01.54.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Nov 2023 01:54:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 40C20804BC1A; Wed, 29 Nov 2023 01:54:41 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231749AbjK2JyN (ORCPT + 99 others); Wed, 29 Nov 2023 04:54:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35010 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231400AbjK2Jxx (ORCPT ); Wed, 29 Nov 2023 04:53:53 -0500 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9700A19BF for ; Wed, 29 Nov 2023 01:53:39 -0800 (PST) Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id EEF061F8BD; Wed, 29 Nov 2023 09:53:37 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 911A113AA0; Wed, 29 Nov 2023 09:53:37 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id YPESIyEKZ2UrfQAAD6G6ig (envelope-from ); Wed, 29 Nov 2023 09:53:37 +0000 From: Vlastimil Babka Date: Wed, 29 Nov 2023 10:53:33 +0100 Subject: [PATCH RFC v3 8/9] maple_tree: Remove MA_STATE_PREALLOC MIME-Version: 1.0 Message-Id: <20231129-slub-percpu-caches-v3-8-6bcf536772bc@suse.cz> References: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> In-Reply-To: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Matthew Wilcox , "Liam R. Howlett" Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Alexander Potapenko , Marco Elver , Dmitry Vyukov , linux-mm@kvack.org, linux-kernel@vger.kernel.org, maple-tree@lists.infradead.org, kasan-dev@googlegroups.com, Vlastimil Babka X-Mailer: b4 0.12.4 X-Spam-Level: X-Rspamd-Server: rspamd1 Authentication-Results: smtp-out2.suse.de; none X-Rspamd-Queue-Id: EEF061F8BD X-Spam-Score: -4.00 X-Spamd-Result: default: False [-4.00 / 50.00]; TAGGED_RCPT(0.00)[]; REPLY(-4.00)[] X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 29 Nov 2023 01:54:41 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783891693085409584 X-GMAIL-MSGID: 1783891693085409584 From: "Liam R. Howlett" MA_SATE_PREALLOC was added to catch any writes that try to allocate when the maple state is being used in preallocation mode. This can safely be removed in favour of the percpu array of nodes. Note that mas_expected_entries() still expects no allocations during operation and so MA_STATE_BULK can be used in place of preallocations for this case, which is primarily used for forking. Signed-off-by: Liam R. Howlett Signed-off-by: Vlastimil Babka --- lib/maple_tree.c | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index d9e7088fd9a7..f5c0bca2c5d7 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -68,11 +68,9 @@ * Maple state flags * * MA_STATE_BULK - Bulk insert mode * * MA_STATE_REBALANCE - Indicate a rebalance during bulk insert - * * MA_STATE_PREALLOC - Preallocated nodes, WARN_ON allocation */ #define MA_STATE_BULK 1 #define MA_STATE_REBALANCE 2 -#define MA_STATE_PREALLOC 4 #define ma_parent_ptr(x) ((struct maple_pnode *)(x)) #define mas_tree_parent(x) ((unsigned long)(x->tree) | MA_ROOT_PARENT) @@ -1255,11 +1253,8 @@ static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp) return; mas_set_alloc_req(mas, 0); - if (mas->mas_flags & MA_STATE_PREALLOC) { - if (allocated) - return; - WARN_ON(!allocated); - } + if (mas->mas_flags & MA_STATE_BULK) + return; if (!allocated || mas->alloc->node_count == MAPLE_ALLOC_SLOTS) { node = (struct maple_alloc *)mt_alloc_one(gfp); @@ -5518,7 +5513,6 @@ int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp) /* node store, slot store needs one node */ ask_now: mas_node_count_gfp(mas, request, gfp); - mas->mas_flags |= MA_STATE_PREALLOC; if (likely(!mas_is_err(mas))) return 0; @@ -5561,7 +5555,7 @@ void mas_destroy(struct ma_state *mas) mas->mas_flags &= ~MA_STATE_REBALANCE; } - mas->mas_flags &= ~(MA_STATE_BULK|MA_STATE_PREALLOC); + mas->mas_flags &= ~MA_STATE_BULK; total = mas_allocated(mas); while (total) { @@ -5610,9 +5604,6 @@ int mas_expected_entries(struct ma_state *mas, unsigned long nr_entries) * of nodes during the operation. */ - /* Optimize splitting for bulk insert in-order */ - mas->mas_flags |= MA_STATE_BULK; - /* * Avoid overflow, assume a gap between each entry and a trailing null. * If this is wrong, it just means allocation can happen during @@ -5629,8 +5620,9 @@ int mas_expected_entries(struct ma_state *mas, unsigned long nr_entries) /* Add working room for split (2 nodes) + new parents */ mas_node_count_gfp(mas, nr_nodes + 3, GFP_KERNEL); - /* Detect if allocations run out */ - mas->mas_flags |= MA_STATE_PREALLOC; + /* Optimize splitting for bulk insert in-order */ + mas->mas_flags |= MA_STATE_BULK; + if (!mas_is_err(mas)) return 0; From patchwork Wed Nov 29 09:53:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 171232 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a5a7:0:b0:403:3b70:6f57 with SMTP id d7csp246117vqn; Wed, 29 Nov 2023 02:34:52 -0800 (PST) X-Google-Smtp-Source: AGHT+IHH73GCmCX2EYnkcMF6PNfMWAVvQHq6a7nXU5iDSF4tsjIGzXdtdj5ymYT0f/fbL1MwGVYZ X-Received: by 2002:a92:d5cf:0:b0:35c:d27c:431b with SMTP id d15-20020a92d5cf000000b0035cd27c431bmr10069595ilq.11.1701254091963; Wed, 29 Nov 2023 02:34:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701254091; cv=none; d=google.com; s=arc-20160816; b=wf1WD6/z1yUvQtgMnuVS5N1VHpDRQTZlk6idzCWCkqjVzop37XLQ208yTqKANT6Di8 hdrCoMG0xZ6XCAS+bbRXo+ux/tbvMzR0CwfmHm/2UMxuCKgLg0zYg/xicaahS4DAFsb5 DIlERgGyQOr/ZSV2W5YHrRdxolbtOo6w6aBx5QN+fFZMRT48dXJ8Ss9OhpPp+kUba3X+ eJb3sqptzQR3Ein4H0dVLWU4I7KtKQ1HbxZK75RtPo72B9RrMiHaS31D94t738zpylOs m/ybccK60dBHlv8qMR3RfnEKvu6D+9GgaVh/jauXpOdToSZvo9r0vhnuXXYoMVqK+hN+ SXlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:in-reply-to:references:message-id :content-transfer-encoding:mime-version:subject:date:from :dkim-signature:dkim-signature; bh=OzJx1XLcKpfOVPPxGeC45HXa1Ww2Cw/dmnDirEUeMrA=; fh=BM+YXXPDItJMOh98QtZoCEH1RXzAZsCss4yDzENT5Z4=; b=mg1BilieOGYSRrZtpoIWUJLUfiCV+/VAjw7liwU4oqT6nF53e0rkobT6I+agpqh+0i jiWQTPsz1dk4ubwNFqGdL4rikdI8lnndg90/tGd0AEc3oZVMNyww4S+FKMZ7tyz6Q3+w QkeJ9BfeuwRS2BosL5Q1PtsJBCK0oeCA6jDM//Li9nsNy+RC/KaGCfoyUJC4FHXrDhqV +6Ngqr1u1hK05TVfK5XESneuDxCNHbHM4cvxmtIqNQ7t1aaBSNWoskGMsr1oU9k3F0yo JtYnkB/SQLnTVYh665eaEsGtRKH2hqSRF4Nl2kCQSsMcW9f2KZ4LxiR9pHsA2S7itH5s zrUw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=inSOz4yT; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b=Fx7hfg5V; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id bw28-20020a056a02049c00b005b88dfa676esi14709160pgb.158.2023.11.29.02.34.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Nov 2023 02:34:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=inSOz4yT; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b=Fx7hfg5V; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id CDD8B80B31C1; Wed, 29 Nov 2023 02:34:46 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232004AbjK2JyR (ORCPT + 99 others); Wed, 29 Nov 2023 04:54:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231246AbjK2Jxw (ORCPT ); Wed, 29 Nov 2023 04:53:52 -0500 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC0831BC1 for ; Wed, 29 Nov 2023 01:53:39 -0800 (PST) Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id F1F931F8BE; Wed, 29 Nov 2023 09:53:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1701251618; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OzJx1XLcKpfOVPPxGeC45HXa1Ww2Cw/dmnDirEUeMrA=; b=inSOz4yT8TFgPaEVNqR0q6mY6UH6gJHUbkvbIB60MQaG+WfZTTe8fyxGmqgFcEPzn+zbkH X4wFuffCvNZPe31CV8gw6BNVz2L7dpmQFwcoVuq+TvUiUMdeAYgBVI4N/mYaqnMfUmr0nc /CknMU6FEMRIquT7ifcgQ/wnwv5HELU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1701251618; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OzJx1XLcKpfOVPPxGeC45HXa1Ww2Cw/dmnDirEUeMrA=; b=Fx7hfg5Vv0T5+dLFQWNf1/uBwvIhSApKaJpzlzazw7YdiXTEwqcjlUJKSONFGFXMIuuIIj UDa9Bi0THVElqdBA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id AC77013AA1; Wed, 29 Nov 2023 09:53:37 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id aFDSKSEKZ2UrfQAAD6G6ig (envelope-from ); Wed, 29 Nov 2023 09:53:37 +0000 From: Vlastimil Babka Date: Wed, 29 Nov 2023 10:53:34 +0100 Subject: [PATCH RFC v3 9/9] maple_tree: replace preallocation with slub percpu array prefill MIME-Version: 1.0 Message-Id: <20231129-slub-percpu-caches-v3-9-6bcf536772bc@suse.cz> References: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> In-Reply-To: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Matthew Wilcox , "Liam R. Howlett" Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Alexander Potapenko , Marco Elver , Dmitry Vyukov , linux-mm@kvack.org, linux-kernel@vger.kernel.org, maple-tree@lists.infradead.org, kasan-dev@googlegroups.com X-Mailer: b4 0.12.4 Authentication-Results: smtp-out2.suse.de; none X-Spam-Level: X-Spamd-Result: default: False [0.20 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; BAYES_HAM(-0.00)[29.13%]; RCVD_COUNT_THREE(0.00)[3]; R_RATELIMIT(0.00)[to_ip_from(RLtz7ce9b89hw8xzamye9qeynd)]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; RCPT_COUNT_TWELVE(0.00)[16]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_CC(0.00)[linux-foundation.org,linux.dev,gmail.com,google.com,kvack.org,vger.kernel.org,lists.infradead.org,googlegroups.com]; RCVD_TLS_ALL(0.00)[]; SUSPICIOUS_RECIPS(1.50)[] X-Spam-Score: 0.20 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 29 Nov 2023 02:34:47 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783894210890529132 X-GMAIL-MSGID: 1783894210890529132 With the percpu array we can try not doing the preallocations in maple tree, and instead make sure the percpu array is prefilled, and using GFP_ATOMIC in places that relied on the preallocation (in case we miss or fail trylock on the array), i.e. mas_store_prealloc(). For now simply add __GFP_NOFAIL there as well. --- lib/maple_tree.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index f5c0bca2c5d7..d84a0c0fe83b 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5452,7 +5452,12 @@ void mas_store_prealloc(struct ma_state *mas, void *entry) mas_wr_store_setup(&wr_mas); trace_ma_write(__func__, mas, 0, entry); + +retry: mas_wr_store_entry(&wr_mas); + if (unlikely(mas_nomem(mas, GFP_ATOMIC | __GFP_NOFAIL))) + goto retry; + MAS_WR_BUG_ON(&wr_mas, mas_is_err(mas)); mas_destroy(mas); } @@ -5471,8 +5476,6 @@ int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp) MA_WR_STATE(wr_mas, mas, entry); unsigned char node_size; int request = 1; - int ret; - if (unlikely(!mas->index && mas->last == ULONG_MAX)) goto ask_now; @@ -5512,16 +5515,8 @@ int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp) /* node store, slot store needs one node */ ask_now: - mas_node_count_gfp(mas, request, gfp); - if (likely(!mas_is_err(mas))) - return 0; + return kmem_cache_prefill_percpu_array(maple_node_cache, request, gfp); - mas_set_alloc_req(mas, 0); - ret = xa_err(mas->node); - mas_reset(mas); - mas_destroy(mas); - mas_reset(mas); - return ret; } EXPORT_SYMBOL_GPL(mas_preallocate);