From patchwork Fri Sep 15 10:59:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matteo Rizzo X-Patchwork-Id: 140427 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1035026vqi; Fri, 15 Sep 2023 06:12:56 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH2g3yvHsSV8JvOV2kmuJQdA+D9+77QS8IIa6Y4wLZblZOgDnChwdntb9IIhJ297U4nr2Mx X-Received: by 2002:a17:90b:194e:b0:269:4fe8:687 with SMTP id nk14-20020a17090b194e00b002694fe80687mr2183327pjb.19.1694783576055; Fri, 15 Sep 2023 06:12:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694783576; cv=none; d=google.com; s=arc-20160816; b=RHO65tDCthfbkiNXiQGvL/jL/BV+NakZfUkTRA5XbXiJd6+C8lGLtnx1dpAqrCXVfL gaYtYuFb/xR+ietHzO7PcutGFe/qpjAzuylzSakuhLSIa8Pwma0byzmJsO1+nhRWDFZr SFDPN5tz8gmf086S2cjBcsiR/SQ5uuy+wgMBEL3tGT7yc+M3QePkfO66c+yiaePnD1NI KGwpIy+HngVRRbwb0lR3sI35NTsHL/tEx+WQBFKuWO2iaGWSK4COnylp+hgNKWWqCL3n r7dFc7qXqVg5HJ+uYhHnU4JflYVsLmgMF7leClr3Qdi6JYofAOek+a91FYvnHBurCQjf A8QA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=Rgcw5kSlwTUZsnv8liQ/ZCTx1m6umum1AvnN+yk4ci0=; fh=hxrcP/evkFXgdiqiA6zEdvE31LUTUVye/z/fSCqBw68=; b=De1J3VtAeDLQlTnO6kPTuSMr8f4ttBNV4UGH1mn3y8WyecG+3ItFiel52GurGOguWx 9gtJWtALsm63JCe7yVfysLT2AHVN6hEmz95qgfj5HffGWV36Z7gGSRO+LPWSRrrxnWDx A3/6VI7aTtYmatbHEAw+JCt/4SOwLpof8zzlftOf1APGjnZ4rcbWAZP6UAhqk7Zs7QbY 8FK9/CwSERxoq/33wb3oK+6lhRlIbVbb/BVogcCd7ckALHW+XDxR2jgcFdLF0YG8tXYA /wFtQ6cLKrHWi13Ffk8ROY96am5EGjef6F3s+gVXY5evgUWLM8ryTyU9DzUwX7QQ29Yi 0vYg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=YNj4Reoe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id a19-20020a17090acb9300b00252d84b7af0si3362132pju.181.2023.09.15.06.12.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 06:12:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=YNj4Reoe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id A43BC84049EE; Fri, 15 Sep 2023 04:01:48 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234359AbjIOLAo (ORCPT + 32 others); Fri, 15 Sep 2023 07:00:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48184 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234311AbjIOLAd (ORCPT ); Fri, 15 Sep 2023 07:00:33 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 90EC61BC for ; Fri, 15 Sep 2023 04:00:02 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-59bdb9fe821so26710147b3.0 for ; Fri, 15 Sep 2023 04:00:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694775601; x=1695380401; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Rgcw5kSlwTUZsnv8liQ/ZCTx1m6umum1AvnN+yk4ci0=; b=YNj4ReoeIxRcOkpVq1wXRHDJ4kBOiomlcsoOyIJsPeKpmSe6sUS5axgSozT2cUiDNF UqI+EpaBrjvu03Qa7dvD7SvLO6kwCl712RRDvHxermpsNFzBOfEx0m2UTPF3z4jl/yLX jAzEnQWYLo1Gr8mGDGsrEewV5ZCk+Qp1X4OdwvOo2McV2JMSEODYU6cn8qcsuVFHSs9D pe7+CgbeAs/3c5IuHTQe+Yisg3r0qXFFz2uPZuY9j0Zk/bEVTlEwga3EDXK+r/YFMjye M2PHAbgQnmBkUROVb6AQ29yPfOW61rTSNVZXaPYhPAELyisAzA3LWZgqHhCCRktRPcAY HeEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694775601; x=1695380401; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Rgcw5kSlwTUZsnv8liQ/ZCTx1m6umum1AvnN+yk4ci0=; b=tGgXusZqk5IQ8QBTh8B6jr8PVEE6dXbjeOrTfMzYfFGyNcqoSt8UaSZudU6kR1f44P TlA8kU30/uBvam2zfv2V8XHd1f/f64a1V1YnroFGqKvvFoUWUZEHLKs8B6MWoIprLkrm vx1TcapFqolhY1+cV+ctXunlqep9kN828dDYSSH4jLH/HqdVzs7ArlEJpeWgGHolojPV 29JdN9xSjC8V99H3SxW3Q0uhLu+snTApEwdW34qEDkJcedb/27+frUrih3RfjxgJOJAV Y1kN5FP7i4Gt0Vu5PyJ1xUmP8qPTJ9I1dVVfYvnPjlN9PtZYfiMrsFh2XL0OhTYDciMI K6Hg== X-Gm-Message-State: AOJu0YzMt8OhCKL557OFF5gaTTnddrJBFCXJnhy3TY7GzNIjiDqVO/J+ H6xr1Mk+VyDDZoXQRKdibKgnCwgyeRvcRD5v6A== X-Received: from mr-cloudtop2.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:2a6]) (user=matteorizzo job=sendgmr) by 2002:a81:ae43:0:b0:59b:f493:813d with SMTP id g3-20020a81ae43000000b0059bf493813dmr28901ywk.1.1694775601139; Fri, 15 Sep 2023 04:00:01 -0700 (PDT) Date: Fri, 15 Sep 2023 10:59:28 +0000 In-Reply-To: <20230915105933.495735-1-matteorizzo@google.com> Mime-Version: 1.0 References: <20230915105933.495735-1-matteorizzo@google.com> X-Mailer: git-send-email 2.42.0.459.ge4e396fd5e-goog Message-ID: <20230915105933.495735-10-matteorizzo@google.com> Subject: [RFC PATCH 09/14] mm/slub: add the slab freelists to kmem_cache From: Matteo Rizzo To: cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, keescook@chromium.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, corbet@lwn.net, luto@kernel.org, peterz@infradead.org Cc: jannh@google.com, matteorizzo@google.com, evn@google.com, poprdi@google.com, jordyzomer@google.com X-Spam-Status: No, score=-8.4 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 15 Sep 2023 04:01:48 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777109382997031748 X-GMAIL-MSGID: 1777109382997031748 From: Jann Horn With SLAB_VIRTUAL enabled, unused slabs which still have virtual memory allocated to them but no physical memory are kept in a per-cache list so that they can be reused later if the cache needs to grow again. Signed-off-by: Jann Horn Co-developed-by: Matteo Rizzo Signed-off-by: Matteo Rizzo Reviewed-by: Kees Cook --- include/linux/slub_def.h | 16 ++++++++++++++++ mm/slub.c | 23 +++++++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index 0adf5ba8241b..693e9bb34edc 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -86,6 +86,20 @@ struct kmem_cache_cpu { /* * Slab cache management. */ +struct kmem_cache_virtual { +#ifdef CONFIG_SLAB_VIRTUAL + /* Protects freed_slabs and freed_slabs_min */ + spinlock_t freed_slabs_lock; + /* + * Slabs on this list have virtual memory of size oo allocated to them + * but no physical memory + */ + struct list_head freed_slabs; + /* Same as freed_slabs but with memory of size min */ + struct list_head freed_slabs_min; +#endif +}; + struct kmem_cache { #ifndef CONFIG_SLUB_TINY struct kmem_cache_cpu __percpu *cpu_slab; @@ -107,6 +121,8 @@ struct kmem_cache { /* Allocation and freeing of slabs */ struct kmem_cache_order_objects min; + struct kmem_cache_virtual virtual; + gfp_t allocflags; /* gfp flags to use on each alloc */ int refcount; /* Refcount for slab cache destroy */ void (*ctor)(void *); diff --git a/mm/slub.c b/mm/slub.c index 42e7cc0b4452..4f77e5d4fe6c 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4510,8 +4510,20 @@ static int calculate_sizes(struct kmem_cache *s) return !!oo_objects(s->oo); } +static inline void slab_virtual_open(struct kmem_cache *s) +{ +#ifdef CONFIG_SLAB_VIRTUAL + /* WARNING: this stuff will be relocated in bootstrap()! */ + spin_lock_init(&s->virtual.freed_slabs_lock); + INIT_LIST_HEAD(&s->virtual.freed_slabs); + INIT_LIST_HEAD(&s->virtual.freed_slabs_min); +#endif +} + static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) { + slab_virtual_open(s); + s->flags = kmem_cache_flags(s->size, flags, s->name); #ifdef CONFIG_SLAB_FREELIST_HARDENED s->random = get_random_long(); @@ -4994,6 +5006,16 @@ static int slab_memory_callback(struct notifier_block *self, * that may be pointing to the wrong kmem_cache structure. */ +static inline void slab_virtual_bootstrap(struct kmem_cache *s, struct kmem_cache *static_cache) +{ + slab_virtual_open(s); + +#ifdef CONFIG_SLAB_VIRTUAL + list_splice(&static_cache->virtual.freed_slabs, &s->virtual.freed_slabs); + list_splice(&static_cache->virtual.freed_slabs_min, &s->virtual.freed_slabs_min); +#endif +} + static struct kmem_cache * __init bootstrap(struct kmem_cache *static_cache) { int node; @@ -5001,6 +5023,7 @@ static struct kmem_cache * __init bootstrap(struct kmem_cache *static_cache) struct kmem_cache_node *n; memcpy(s, static_cache, kmem_cache->object_size); + slab_virtual_bootstrap(s, static_cache); /* * This runs very early, and only the boot processor is supposed to be