From patchwork Mon Nov 13 19:13:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 164619 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp1421214vqg; Mon, 13 Nov 2023 11:16:58 -0800 (PST) X-Google-Smtp-Source: AGHT+IGZMu/CpySMY0dNsCjkt2jpjAzTrjHBfD9SpgPl+d1mKp5pJ5pQMvib9uIcN3BvO7FR0E3s X-Received: by 2002:a17:903:4d2:b0:1cc:1ee2:d41d with SMTP id jm18-20020a17090304d200b001cc1ee2d41dmr145388plb.39.1699903018484; Mon, 13 Nov 2023 11:16:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1699903018; cv=none; d=google.com; s=arc-20160816; b=NNkAq+Bh6EKjsFv6/s62MDmaEeEAyosGYZnergKiXf0bRtRE1o1qmeGSQ2a0w+gVN5 /Ma/NApfA6V1QRsFX92KUKXTfStcrzozL2nq2Z91HlQg0M9GoXNX0yuq+kvs1wORJiEi neDTZXKqFOeJEuJiFV6A3PtlhqOH5wEb7obC6fe/k5I2XtVp+nBjGXf9NLXrbLJMxuhR w3GmMMXdmhUoGGUW9H9GD4J7OZ7shfp5cii9SqyeiV5bAkQ/nuBw4T9hbFg6wY8f0BnV tTdddDztAYj3tzuY2d0rl5n4MPFpcUlRhfjGkEpcVK9y9bH5YDE68VL9m5kuyxJDPINJ sf+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=AvYULxpvTpi0KZyzNL3YX1bvF9SuuOz0+MetYRBUoJQ=; fh=RPAmrUlnQQdc1FhCirEqyhGh/OnPyRxUfAdj7ygPMx4=; b=uYl7qTkx3BM6ZMds35wYcJiNSOKoX9UdOcW2AuztTf5RLCBlHi9vURssJWQwR5W++s QZYWyE3yhtc8L34ijXlSzqGipNd/d+B/IS/FTlk6efYgMCWYKUbR63Y2fIGy1V9O9p0t Q7wyz659Yp1DeSJH+euxjiq9soiVeGSC1JuSf/cAuGVQzNYWhdz7fjLu4wX5Oz8OUu76 UiHr0EJuw03BeVO7PgRzay1S5TJ8FggoH8fqWXRxiotPJCGoRzIiuwd1BHch9cxyMowq ReCwerzBhPKM0OkJjAvIqHGHrab1tfIXsDW2Gh1gzuA+3R91w6lRJOszVPZ4EWxeeaiJ nFVg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=oIHlLgRj; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b=yC6gI2Oa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id l14-20020a170903120e00b001ca119ef46csi6511357plh.478.2023.11.13.11.16.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Nov 2023 11:16:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=oIHlLgRj; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b=yC6gI2Oa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id BC52880A220E; Mon, 13 Nov 2023 11:15:41 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232176AbjKMTOy (ORCPT + 30 others); Mon, 13 Nov 2023 14:14:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42652 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231844AbjKMTOT (ORCPT ); Mon, 13 Nov 2023 14:14:19 -0500 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35C3C1731; Mon, 13 Nov 2023 11:14:16 -0800 (PST) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id E153B1F88E; Mon, 13 Nov 2023 19:14:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1699902854; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AvYULxpvTpi0KZyzNL3YX1bvF9SuuOz0+MetYRBUoJQ=; b=oIHlLgRjvw7JPEbJGFmSOpsKrINW0f9XdqQWVeRsISFOABjZ4DK8pmw1yqPCX9sGOBppD7 /2PaS3UjA8toXgHPERa2WAt2ew9Q/P4mq1rIHw9O8CtTaonrLfZW/F/b1AC8WVMaSEimF7 oFMsRG1N0wxBUjIAKDrXoROaTC6D8HQ= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1699902854; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AvYULxpvTpi0KZyzNL3YX1bvF9SuuOz0+MetYRBUoJQ=; b=yC6gI2OaZlpEgEV5qLkZwNfRnz195znLfF6b/XZpYlw9m0kJIbvR1gOmpqfTy7c7Hdbuie qkHrE5YZ8HMzm4Dw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 9CF6613398; Mon, 13 Nov 2023 19:14:14 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id sLfCJYZ1UmVFOgAAMHmgww (envelope-from ); Mon, 13 Nov 2023 19:14:14 +0000 From: Vlastimil Babka To: David Rientjes , Christoph Lameter , Pekka Enberg , Joonsoo Kim Cc: Andrew Morton , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, patches@lists.linux.dev, Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Marco Elver , Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Kees Cook , kasan-dev@googlegroups.com, cgroups@vger.kernel.org, Vlastimil Babka Subject: [PATCH 16/20] mm/slab: move kmalloc_slab() to mm/slab.h Date: Mon, 13 Nov 2023 20:13:57 +0100 Message-ID: <20231113191340.17482-38-vbabka@suse.cz> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231113191340.17482-22-vbabka@suse.cz> References: <20231113191340.17482-22-vbabka@suse.cz> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Mon, 13 Nov 2023 11:15:42 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782477507124678448 X-GMAIL-MSGID: 1782477507124678448 In preparation for the next patch, move the kmalloc_slab() function to the header, as it will have callers from two files, and make it inline. To avoid unnecessary bloat, remove all size checks/warnings from kmalloc_slab() as they just duplicate those in callers, especially after recent changes to kmalloc_size_roundup(). We just need to adjust handling of zero size in __do_kmalloc_node(). Also we can stop handling NULL result from kmalloc_slab() there as that now cannot happen (unless called too early during boot). The size_index array becomes visible so rename it to a more specific kmalloc_size_index. Signed-off-by: Vlastimil Babka Reviewed-by: Kees Cook --- mm/slab.h | 28 ++++++++++++++++++++++++++-- mm/slab_common.c | 43 ++++++++----------------------------------- 2 files changed, 34 insertions(+), 37 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 179467e8aacc..744384efa7be 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -388,8 +388,32 @@ extern const struct kmalloc_info_struct { void setup_kmalloc_cache_index_table(void); void create_kmalloc_caches(slab_flags_t); -/* Find the kmalloc slab corresponding for a certain size */ -struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags, unsigned long caller); +extern u8 kmalloc_size_index[24]; + +static inline unsigned int size_index_elem(unsigned int bytes) +{ + return (bytes - 1) / 8; +} + +/* + * Find the kmem_cache structure that serves a given size of + * allocation + * + * This assumes size is larger than zero and not larger than + * KMALLOC_MAX_CACHE_SIZE and the caller must check that. + */ +static inline struct kmem_cache * +kmalloc_slab(size_t size, gfp_t flags, unsigned long caller) +{ + unsigned int index; + + if (size <= 192) + index = kmalloc_size_index[size_index_elem(size)]; + else + index = fls(size - 1); + + return kmalloc_caches[kmalloc_type(flags, caller)][index]; +} void *__kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node, size_t orig_size, diff --git a/mm/slab_common.c b/mm/slab_common.c index f4f275613d2a..31ade17a7ad9 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -665,7 +665,7 @@ EXPORT_SYMBOL(random_kmalloc_seed); * of two cache sizes there. The size of larger slabs can be determined using * fls. */ -static u8 size_index[24] __ro_after_init = { +u8 kmalloc_size_index[24] __ro_after_init = { 3, /* 8 */ 4, /* 16 */ 5, /* 24 */ @@ -692,33 +692,6 @@ static u8 size_index[24] __ro_after_init = { 2 /* 192 */ }; -static inline unsigned int size_index_elem(unsigned int bytes) -{ - return (bytes - 1) / 8; -} - -/* - * Find the kmem_cache structure that serves a given size of - * allocation - */ -struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags, unsigned long caller) -{ - unsigned int index; - - if (size <= 192) { - if (!size) - return ZERO_SIZE_PTR; - - index = size_index[size_index_elem(size)]; - } else { - if (WARN_ON_ONCE(size > KMALLOC_MAX_CACHE_SIZE)) - return NULL; - index = fls(size - 1); - } - - return kmalloc_caches[kmalloc_type(flags, caller)][index]; -} - size_t kmalloc_size_roundup(size_t size) { if (size && size <= KMALLOC_MAX_CACHE_SIZE) { @@ -843,9 +816,9 @@ void __init setup_kmalloc_cache_index_table(void) for (i = 8; i < KMALLOC_MIN_SIZE; i += 8) { unsigned int elem = size_index_elem(i); - if (elem >= ARRAY_SIZE(size_index)) + if (elem >= ARRAY_SIZE(kmalloc_size_index)) break; - size_index[elem] = KMALLOC_SHIFT_LOW; + kmalloc_size_index[elem] = KMALLOC_SHIFT_LOW; } if (KMALLOC_MIN_SIZE >= 64) { @@ -854,7 +827,7 @@ void __init setup_kmalloc_cache_index_table(void) * is 64 byte. */ for (i = 64 + 8; i <= 96; i += 8) - size_index[size_index_elem(i)] = 7; + kmalloc_size_index[size_index_elem(i)] = 7; } @@ -865,7 +838,7 @@ void __init setup_kmalloc_cache_index_table(void) * instead. */ for (i = 128 + 8; i <= 192; i += 8) - size_index[size_index_elem(i)] = 8; + kmalloc_size_index[size_index_elem(i)] = 8; } } @@ -977,10 +950,10 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller return ret; } - s = kmalloc_slab(size, flags, caller); + if (unlikely(!size)) + return ZERO_SIZE_PTR; - if (unlikely(ZERO_OR_NULL_PTR(s))) - return s; + s = kmalloc_slab(size, flags, caller); ret = __kmem_cache_alloc_node(s, flags, node, size, caller); ret = kasan_kmalloc(s, ret, size, flags);