From patchwork Fri Mar 1 17:07:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 208958 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:fa17:b0:10a:f01:a869 with SMTP id ju23csp19100dyc; Fri, 1 Mar 2024 09:09:43 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWlnyQ/rfwy6Kk0KdbYUt3dMGZlXLUaLdNm78+xMGCq7XAYzdLkdir2mIywsEsjoizyVzQzlXcufYEbE/sV16ZBOSeLIA== X-Google-Smtp-Source: AGHT+IG9WiPcZmAKVDqzM/TZ+YKEOdWBzPJ2IyHnzeDCOhUPPFIxZHZbxyQU+6BFabghu9dlZume X-Received: by 2002:a05:620a:1265:b0:787:d4d7:1a72 with SMTP id b5-20020a05620a126500b00787d4d71a72mr2603000qkl.46.1709312983379; Fri, 01 Mar 2024 09:09:43 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709312983; cv=pass; d=google.com; s=arc-20160816; b=ce2ZV0Z4c8rwPaVOb4DfASer7vM1iyBcfzKKEKkxIvWB05XvbMjKJYpo+FVWOC7ekt 0CoruDDLYEsdCuUMMHF41g4Mt8+pVVb/hH5haFZ3iPCfUL+xo049nfg0glev0P5a0IlO X4uBw4VG1FgbcKb5n+AiotrwOZpAyoTvRnSB3L0PccnURAVfhciyePDP3KZ2Xl6JXC+j qqy2VFs66H6S5/pfGlsm5OdrCT+dzRDXh7xjSbVxpMtMlzinZyhnWP7FU+VOo+PGpc2D BPnFGQSqZFzX4A1dQCRKPb60trtPoJ3OAdw4cFLsOAnMhtqDjFV9oi2zyl/zdI7tA6Oz kXUA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:list-unsubscribe:list-subscribe:list-id:precedence :subject:date:from; bh=XQGU1Qk0c+C8g3iQCGZYbUkVzv136rhWCBBorzn4hKY=; fh=WGGwdVVTGLpVYCxjY83qTUlyyBzuw+WV0hgXC1Po200=; b=UiPTVv6geLex28dX0pXAHYNKRfiAg85WgDBWvTqfaDICOP4Gj8jxWAxsxwssDo8/81 Q6p8TmhhF5WONG1qtAckfqAljaHH7g9Q7r+2RmAEy0jfApUU9/0npp6ZwbuQoOCU6v+n H+YFly93r64hXeZY7r/ExzuIO1HiTzPZ6Fy/9J8qpaLBHVTk4S7vdBCMNMGY5f6MEp2A ucxHe6ce3CvkjU+M2nnN7R6+tXV6JfKUME7etwrnZTI7tbjXgVAoGM/3EQFjnuGEKKTY xoWXniffclTSk6zgurVg1v1PWjd/glww2WZu8p7quBJhwKqRqVmniC5AxPWP/813SeBa Z5/g==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=suse.cz); spf=pass (google.com: domain of linux-kernel+bounces-88802-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-88802-ouuuleilei=gmail.com@vger.kernel.org" Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id wl11-20020a05620a57cb00b00785ab75cfd1si3896191qkn.449.2024.03.01.09.09.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Mar 2024 09:09:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-88802-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=suse.cz); spf=pass (google.com: domain of linux-kernel+bounces-88802-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-88802-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 5D30B1C252DE for ; Fri, 1 Mar 2024 17:08:55 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 059153FE0; Fri, 1 Mar 2024 17:07:19 +0000 (UTC) Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 825972571; Fri, 1 Mar 2024 17:07:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709312835; cv=none; b=YgHiBCCspc1Or7Cpfog9PuvfRaY3z/t/o063qu0I8Mox7P35ZS7M0ig9j4O8upDmAXsVVSQgOtUSy42G184Fwd+fr3ZQVOi3eGP4RBE7bG0c5jEE4dpGzuL8uVI0pqsZFSBfZc4Lzda42bHP8dRHAx0wObPSsgF/0yLwdO163pQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709312835; c=relaxed/simple; bh=Ss/e4gGxC24daD3nyMEINkiPC9L7D1njVPpAZVM+9sg=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Q5muM0cTLIdxbHj2QWTOyU3PkSGhlaTw34/m7y/Ok/UQ4SrJrIZ6NTrNsuiCJwPbsF8kKZ8gFFhjR1/za8AdreoGilGsDWXQ3qgpgg/71/vv5xwcdKKupDnB5qsvGiXM2xj4QuEIvmBoSbevzqoTYvBYh8fDaiVF7u1zxCn0Zds= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id B170733D2C; Fri, 1 Mar 2024 17:07:10 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 7CD6813AA3; Fri, 1 Mar 2024 17:07:10 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 6PIzHj4L4mUcGQAAD6G6ig (envelope-from ); Fri, 01 Mar 2024 17:07:10 +0000 From: Vlastimil Babka Date: Fri, 01 Mar 2024 18:07:08 +0100 Subject: [PATCH RFC 1/4] mm, slab: move memcg charging to post-alloc hook Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240301-slab-memcg-v1-1-359328a46596@suse.cz> References: <20240301-slab-memcg-v1-0-359328a46596@suse.cz> In-Reply-To: <20240301-slab-memcg-v1-0-359328a46596@suse.cz> To: Linus Torvalds , Josh Poimboeuf , Jeff Layton , Chuck Lever , Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Alexander Viro , Christian Brauner , Jan Kara Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.13.0 Authentication-Results: smtp-out1.suse.de; none X-Spamd-Result: default: False [-4.00 / 50.00]; TAGGED_RCPT(0.00)[]; REPLY(-4.00)[] X-Rspamd-Server: rspamd1.dmz-prg2.suse.org X-Rspamd-Queue-Id: B170733D2C X-Spam-Level: X-Spam-Score: -4.00 X-Spam-Flag: NO X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792344570771074668 X-GMAIL-MSGID: 1792344570771074668 The MEMCG_KMEM integration with slab currently relies on two hooks during allocation. memcg_slab_pre_alloc_hook() determines the objcg and charges it, and memcg_slab_post_alloc_hook() assigns the objcg pointer to the allocated object(s). As Linus pointed out, this is unnecessarily complex. Failing to charge due to memcg limits should be rare, so we can optimistically allocate the object(s) and do the charging together with assigning the objcg pointer in a single post_alloc hook. In the rare case the charging fails, we can free the object(s) back. This simplifies the code (no need to pass around the objcg pointer) and potentially allows to separate charging from allocation in cases where it's common that the allocation would be immediately freed, and the memcg handling overhead could be saved. Suggested-by: Linus Torvalds Link: https://lore.kernel.org/all/CAHk-=whYOOdM7jWy5jdrAm8LxcgCMFyk2bt8fYYvZzM4U-zAQA@mail.gmail.com/ Signed-off-by: Vlastimil Babka --- mm/slub.c | 180 +++++++++++++++++++++++++++----------------------------------- 1 file changed, 77 insertions(+), 103 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 2ef88bbf56a3..7022a1246bab 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1897,23 +1897,36 @@ static inline size_t obj_full_size(struct kmem_cache *s) return s->size + sizeof(struct obj_cgroup *); } -/* - * Returns false if the allocation should fail. - */ -static bool __memcg_slab_pre_alloc_hook(struct kmem_cache *s, - struct list_lru *lru, - struct obj_cgroup **objcgp, - size_t objects, gfp_t flags) +static bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, + struct list_lru *lru, + gfp_t flags, size_t size, + void **p) { + struct obj_cgroup *objcg; + struct slab *slab; + unsigned long off; + size_t i; + /* * The obtained objcg pointer is safe to use within the current scope, * defined by current task or set_active_memcg() pair. * obj_cgroup_get() is used to get a permanent reference. */ - struct obj_cgroup *objcg = current_obj_cgroup(); + objcg = current_obj_cgroup(); if (!objcg) return true; + /* + * slab_alloc_node() avoids the NULL check, so we might be called with a + * single NULL object. kmem_cache_alloc_bulk() aborts if it can't fill + * the whole requested size. + * return success as there's nothing to free back + */ + if (unlikely(*p == NULL)) + return true; + + flags &= gfp_allowed_mask; + if (lru) { int ret; struct mem_cgroup *memcg; @@ -1926,71 +1939,51 @@ static bool __memcg_slab_pre_alloc_hook(struct kmem_cache *s, return false; } - if (obj_cgroup_charge(objcg, flags, objects * obj_full_size(s))) + if (obj_cgroup_charge(objcg, flags, size * obj_full_size(s))) return false; - *objcgp = objcg; + for (i = 0; i < size; i++) { + slab = virt_to_slab(p[i]); + + if (!slab_objcgs(slab) && + memcg_alloc_slab_cgroups(slab, s, flags, false)) { + obj_cgroup_uncharge(objcg, obj_full_size(s)); + continue; + } + + off = obj_to_index(s, slab, p[i]); + obj_cgroup_get(objcg); + slab_objcgs(slab)[off] = objcg; + mod_objcg_state(objcg, slab_pgdat(slab), + cache_vmstat_idx(s), obj_full_size(s)); + } + return true; } -/* - * Returns false if the allocation should fail. - */ +static void memcg_alloc_abort_single(struct kmem_cache *s, void *object); + static __fastpath_inline -bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, struct list_lru *lru, - struct obj_cgroup **objcgp, size_t objects, - gfp_t flags) +bool memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, + gfp_t flags, size_t size, void **p) { - if (!memcg_kmem_online()) + if (likely(!memcg_kmem_online())) return true; if (likely(!(flags & __GFP_ACCOUNT) && !(s->flags & SLAB_ACCOUNT))) return true; - return likely(__memcg_slab_pre_alloc_hook(s, lru, objcgp, objects, - flags)); -} - -static void __memcg_slab_post_alloc_hook(struct kmem_cache *s, - struct obj_cgroup *objcg, - gfp_t flags, size_t size, - void **p) -{ - struct slab *slab; - unsigned long off; - size_t i; - - flags &= gfp_allowed_mask; - - for (i = 0; i < size; i++) { - if (likely(p[i])) { - slab = virt_to_slab(p[i]); - - if (!slab_objcgs(slab) && - memcg_alloc_slab_cgroups(slab, s, flags, false)) { - obj_cgroup_uncharge(objcg, obj_full_size(s)); - continue; - } + if (likely(__memcg_slab_post_alloc_hook(s, lru, flags, size, p))) + return true; - off = obj_to_index(s, slab, p[i]); - obj_cgroup_get(objcg); - slab_objcgs(slab)[off] = objcg; - mod_objcg_state(objcg, slab_pgdat(slab), - cache_vmstat_idx(s), obj_full_size(s)); - } else { - obj_cgroup_uncharge(objcg, obj_full_size(s)); - } + if (likely(size == 1)) { + memcg_alloc_abort_single(s, p); + *p = NULL; + } else { + kmem_cache_free_bulk(s, size, p); } -} - -static __fastpath_inline -void memcg_slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, - gfp_t flags, size_t size, void **p) -{ - if (likely(!memcg_kmem_online() || !objcg)) - return; - return __memcg_slab_post_alloc_hook(s, objcg, flags, size, p); + return false; } static void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, @@ -2029,14 +2022,6 @@ void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, __memcg_slab_free_hook(s, slab, p, objects, objcgs); } - -static inline -void memcg_slab_alloc_error_hook(struct kmem_cache *s, int objects, - struct obj_cgroup *objcg) -{ - if (objcg) - obj_cgroup_uncharge(objcg, objects * obj_full_size(s)); -} #else /* CONFIG_MEMCG_KMEM */ static inline struct mem_cgroup *memcg_from_slab_obj(void *ptr) { @@ -2047,31 +2032,18 @@ static inline void memcg_free_slab_cgroups(struct slab *slab) { } -static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, - struct list_lru *lru, - struct obj_cgroup **objcgp, - size_t objects, gfp_t flags) -{ - return true; -} - -static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, - struct obj_cgroup *objcg, +static inline bool memcg_slab_post_alloc_hook(struct kmem_cache *s, + struct list_lru *lru, gfp_t flags, size_t size, void **p) { + return true; } static inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects) { } - -static inline -void memcg_slab_alloc_error_hook(struct kmem_cache *s, int objects, - struct obj_cgroup *objcg) -{ -} #endif /* CONFIG_MEMCG_KMEM */ /* @@ -3751,10 +3723,7 @@ noinline int should_failslab(struct kmem_cache *s, gfp_t gfpflags) ALLOW_ERROR_INJECTION(should_failslab, ERRNO); static __fastpath_inline -struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, - struct list_lru *lru, - struct obj_cgroup **objcgp, - size_t size, gfp_t flags) +struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, gfp_t flags) { flags &= gfp_allowed_mask; @@ -3763,14 +3732,11 @@ struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, if (unlikely(should_failslab(s, flags))) return NULL; - if (unlikely(!memcg_slab_pre_alloc_hook(s, lru, objcgp, size, flags))) - return NULL; - return s; } static __fastpath_inline -void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, +bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, gfp_t flags, size_t size, void **p, bool init, unsigned int orig_size) { @@ -3819,7 +3785,7 @@ void slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, kmsan_slab_alloc(s, p[i], init_flags); } - memcg_slab_post_alloc_hook(s, objcg, flags, size, p); + return memcg_slab_post_alloc_hook(s, lru, flags, size, p); } /* @@ -3836,10 +3802,9 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list gfp_t gfpflags, int node, unsigned long addr, size_t orig_size) { void *object; - struct obj_cgroup *objcg = NULL; bool init = false; - s = slab_pre_alloc_hook(s, lru, &objcg, 1, gfpflags); + s = slab_pre_alloc_hook(s, gfpflags); if (unlikely(!s)) return NULL; @@ -3856,8 +3821,10 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list /* * When init equals 'true', like for kzalloc() family, only * @orig_size bytes might be zeroed instead of s->object_size + * In case this fails due to memcg_slab_post_alloc_hook(), + * object is set to NULL */ - slab_post_alloc_hook(s, objcg, gfpflags, 1, &object, init, orig_size); + slab_post_alloc_hook(s, lru, gfpflags, 1, &object, init, orig_size); return object; } @@ -4300,6 +4267,16 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object, do_slab_free(s, slab, object, object, 1, addr); } +#ifdef CONFIG_MEMCG_KMEM +/* Do not inline the rare memcg charging failed path into the allocation path */ +static noinline +void memcg_alloc_abort_single(struct kmem_cache *s, void *object) +{ + if (likely(slab_free_hook(s, object, slab_want_init_on_free(s)))) + do_slab_free(s, virt_to_slab(object), object, object, 1, _RET_IP_); +} +#endif + static __fastpath_inline void slab_free_bulk(struct kmem_cache *s, struct slab *slab, void *head, void *tail, void **p, int cnt, unsigned long addr) @@ -4635,29 +4612,26 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, void **p) { int i; - struct obj_cgroup *objcg = NULL; if (!size) return 0; - /* memcg and kmem_cache debug support */ - s = slab_pre_alloc_hook(s, NULL, &objcg, size, flags); + s = slab_pre_alloc_hook(s, flags); if (unlikely(!s)) return 0; i = __kmem_cache_alloc_bulk(s, flags, size, p); + if (unlikely(i == 0)) + return 0; /* * memcg and kmem_cache debug support and memory initialization. * Done outside of the IRQ disabled fastpath loop. */ - if (likely(i != 0)) { - slab_post_alloc_hook(s, objcg, flags, size, p, - slab_want_init_on_alloc(flags, s), s->object_size); - } else { - memcg_slab_alloc_error_hook(s, size, objcg); + if (unlikely(!slab_post_alloc_hook(s, NULL, flags, size, p, + slab_want_init_on_alloc(flags, s), s->object_size))) { + return 0; } - return i; } EXPORT_SYMBOL(kmem_cache_alloc_bulk); From patchwork Fri Mar 1 17:07:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 208957 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:fa17:b0:10a:f01:a869 with SMTP id ju23csp18986dyc; Fri, 1 Mar 2024 09:09:34 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUCDW/yMWaxpNQ8oXMIdZgLC03JbpCElsraBw1aSvAtJ8ghxTBeUxyIqh62qJD8+uj1pUFd1qUrasZTIPOCgKvjhJbM8Q== X-Google-Smtp-Source: AGHT+IEOHYELvVxM4IdafcE1fyQt0FLGv1Zqe0BlnnZDyb3UjGMHQ6HFNypRe+xbSxbRGS+vGkg5 X-Received: by 2002:ac8:5a83:0:b0:42e:cbce:c285 with SMTP id c3-20020ac85a83000000b0042ecbcec285mr2281133qtc.0.1709312974035; Fri, 01 Mar 2024 09:09:34 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709312974; cv=pass; d=google.com; s=arc-20160816; b=vIj9Qmf7dCFmBAR+krYuJBwzvctGJcS8mgAy4BswpcSZo2EzDvzM0jRwQWBO2ChM/1 iSrfMEivvhUnTBbOb+J+mThgOTBkr+dM2TUuDmfUCdVH4TpXqW9mSKt9cv6Eo+G/SJNQ ER5ixPegOCZwYt4tOihn/CJpJFCONtM3sGH8Ytnf54QX+5jX+WutDJ2mleYaG66gIb0z ZwoZK1YJMczZtY1hmoE+Q+szXXh54m7OwQZHR8D+asrqaes262RGgnnkVyeaEva0UeqQ mFpb1iDD6MaPZBgggz4LeJjn6TnRcAwRwqc8R7zG1r/M7299Ntfw7YFITA98paAzfXe9 CexA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:list-unsubscribe:list-subscribe:list-id:precedence :subject:date:from; bh=zaY7Cl6H+W5VUviiIrAaZrT1balgaCewR6uDC3lJgT0=; fh=WGGwdVVTGLpVYCxjY83qTUlyyBzuw+WV0hgXC1Po200=; b=PMWM2/ONQQJABgHl6v72g7IZy+R6mFlj89yb3EXrCCxOxcZcxjOCxdLEWNG5wvyVHZ iZJzcIcP64Yj8U8yZKq7BY2sF7UED4iLR7iDylh0SR1oNu9FmbXImWu1xCy0WM38brxP xutiAtxyMNw0TcIS/eu6qg+IvYhU2bVBCVkNcQZLVJnw6goy86fNfDhtMk7g+YO1l8zG X6mFowoY3eXuGBsCHgCftQxTI7gX7udzRDm+HZqokEBjqM7TH7lli8FNQ0GoeAHwmJOP dURQEfk5hRXmhBwuOo8YslYn37ZVhWTSeaf5zf9ICI8IuknXqB4AzCCFv3BnOwlM+oRe bDqg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=suse.cz); spf=pass (google.com: domain of linux-kernel+bounces-88800-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-88800-ouuuleilei=gmail.com@vger.kernel.org" Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id dt45-20020a05620a47ad00b007879cd000eesi4071965qkb.454.2024.03.01.09.09.33 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Mar 2024 09:09:34 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-88800-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=suse.cz); spf=pass (google.com: domain of linux-kernel+bounces-88800-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-88800-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 7CD931C24616 for ; Fri, 1 Mar 2024 17:08:41 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B287736118; Fri, 1 Mar 2024 17:07:17 +0000 (UTC) Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9785646A4; Fri, 1 Mar 2024 17:07:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709312834; cv=none; b=IPOQbl3bvRqdO2UP81MCm8VsIC8egP5z4KR/0Quos6B8PsptzowHYTpzq0mEoBfOjXoAVnPToEyIAVAELegANUSoXo9Bj/WcSlcFwYBchvPiF9VZfKXfAzanAy0joLuonoq5FG4iohviWxAmdkyuCqtH6fH9mlZbfFLF1kI/QfQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709312834; c=relaxed/simple; bh=tRcNrdDwOlxAEXA83t7g4wJfIiyligzzo/g9KIbU0M8=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=VjnATXkxCDgv2HT4hj1cCoFwKpxOCgZrA7x8zk+KiVMZ7Jpz+bisIFx9A+5Kgp8zlA0iZl+ZFoTyvTFiiFgILfvgHrVR3Jx4B+NTwYnUrG4K2kiVjniLi0K32abiIqD33uKJsgVIp9j0mOgIB2bN0YgywC5jEp70ua/L6IRIJaQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id C4782207B6; Fri, 1 Mar 2024 17:07:10 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id A05F613AB0; Fri, 1 Mar 2024 17:07:10 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id +FbfJj4L4mUcGQAAD6G6ig (envelope-from ); Fri, 01 Mar 2024 17:07:10 +0000 From: Vlastimil Babka Date: Fri, 01 Mar 2024 18:07:09 +0100 Subject: [PATCH RFC 2/4] mm, slab: move slab_memcg hooks to mm/memcontrol.c Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240301-slab-memcg-v1-2-359328a46596@suse.cz> References: <20240301-slab-memcg-v1-0-359328a46596@suse.cz> In-Reply-To: <20240301-slab-memcg-v1-0-359328a46596@suse.cz> To: Linus Torvalds , Josh Poimboeuf , Jeff Layton , Chuck Lever , Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Alexander Viro , Christian Brauner , Jan Kara Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.13.0 X-Spam-Level: Authentication-Results: smtp-out2.suse.de; none X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Spamd-Result: default: False [-4.00 / 50.00]; TAGGED_RCPT(0.00)[]; REPLY(-4.00)[] X-Spam-Score: -4.00 X-Rspamd-Queue-Id: C4782207B6 X-Spam-Flag: NO X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792344560955460598 X-GMAIL-MSGID: 1792344560955460598 The hooks make multiple calls to functions in mm/memcontrol.c, including to th current_obj_cgroup() marked __always_inline. It might be faster to make a single call to the hook in mm/memcontrol.c instead. The hooks also don't use almost anything from mm/slub.c. obj_full_size() can move with the hooks and cache_vmstat_idx() to the internal mm/slab.h Signed-off-by: Vlastimil Babka --- mm/memcontrol.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++ mm/slab.h | 10 ++++++ mm/slub.c | 100 -------------------------------------------------------- 3 files changed, 100 insertions(+), 100 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e4c8735e7c85..37ee9356a26c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3575,6 +3575,96 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) refill_obj_stock(objcg, size, true); } +static inline size_t obj_full_size(struct kmem_cache *s) +{ + /* + * For each accounted object there is an extra space which is used + * to store obj_cgroup membership. Charge it too. + */ + return s->size + sizeof(struct obj_cgroup *); +} + +bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, + gfp_t flags, size_t size, void **p) +{ + struct obj_cgroup *objcg; + struct slab *slab; + unsigned long off; + size_t i; + + /* + * The obtained objcg pointer is safe to use within the current scope, + * defined by current task or set_active_memcg() pair. + * obj_cgroup_get() is used to get a permanent reference. + */ + objcg = current_obj_cgroup(); + if (!objcg) + return true; + + /* + * slab_alloc_node() avoids the NULL check, so we might be called with a + * single NULL object. kmem_cache_alloc_bulk() aborts if it can't fill + * the whole requested size. + * return success as there's nothing to free back + */ + if (unlikely(*p == NULL)) + return true; + + flags &= gfp_allowed_mask; + + if (lru) { + int ret; + struct mem_cgroup *memcg; + + memcg = get_mem_cgroup_from_objcg(objcg); + ret = memcg_list_lru_alloc(memcg, lru, flags); + css_put(&memcg->css); + + if (ret) + return false; + } + + if (obj_cgroup_charge(objcg, flags, size * obj_full_size(s))) + return false; + + for (i = 0; i < size; i++) { + slab = virt_to_slab(p[i]); + + if (!slab_objcgs(slab) && + memcg_alloc_slab_cgroups(slab, s, flags, false)) { + obj_cgroup_uncharge(objcg, obj_full_size(s)); + continue; + } + + off = obj_to_index(s, slab, p[i]); + obj_cgroup_get(objcg); + slab_objcgs(slab)[off] = objcg; + mod_objcg_state(objcg, slab_pgdat(slab), + cache_vmstat_idx(s), obj_full_size(s)); + } + + return true; +} + +void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, + void **p, int objects, struct obj_cgroup **objcgs) +{ + for (int i = 0; i < objects; i++) { + struct obj_cgroup *objcg; + unsigned int off; + + off = obj_to_index(s, slab, p[i]); + objcg = objcgs[off]; + if (!objcg) + continue; + + objcgs[off] = NULL; + obj_cgroup_uncharge(objcg, obj_full_size(s)); + mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), + -obj_full_size(s)); + obj_cgroup_put(objcg); + } +} #endif /* CONFIG_MEMCG_KMEM */ /* diff --git a/mm/slab.h b/mm/slab.h index 54deeb0428c6..3f170673fa55 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -541,6 +541,12 @@ static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t fla return false; } +static inline enum node_stat_item cache_vmstat_idx(struct kmem_cache *s) +{ + return (s->flags & SLAB_RECLAIM_ACCOUNT) ? + NR_SLAB_RECLAIMABLE_B : NR_SLAB_UNRECLAIMABLE_B; +} + #ifdef CONFIG_MEMCG_KMEM /* * slab_objcgs - get the object cgroups vector associated with a slab @@ -564,6 +570,10 @@ int memcg_alloc_slab_cgroups(struct slab *slab, struct kmem_cache *s, gfp_t gfp, bool new_slab); void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, enum node_stat_item idx, int nr); +bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, + gfp_t flags, size_t size, void **p); +void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, + void **p, int objects, struct obj_cgroup **objcgs); #else /* CONFIG_MEMCG_KMEM */ static inline struct obj_cgroup **slab_objcgs(struct slab *slab) { diff --git a/mm/slub.c b/mm/slub.c index 7022a1246bab..64da169d672a 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1875,12 +1875,6 @@ static bool freelist_corrupted(struct kmem_cache *s, struct slab *slab, #endif #endif /* CONFIG_SLUB_DEBUG */ -static inline enum node_stat_item cache_vmstat_idx(struct kmem_cache *s) -{ - return (s->flags & SLAB_RECLAIM_ACCOUNT) ? - NR_SLAB_RECLAIMABLE_B : NR_SLAB_UNRECLAIMABLE_B; -} - #ifdef CONFIG_MEMCG_KMEM static inline void memcg_free_slab_cgroups(struct slab *slab) { @@ -1888,79 +1882,6 @@ static inline void memcg_free_slab_cgroups(struct slab *slab) slab->memcg_data = 0; } -static inline size_t obj_full_size(struct kmem_cache *s) -{ - /* - * For each accounted object there is an extra space which is used - * to store obj_cgroup membership. Charge it too. - */ - return s->size + sizeof(struct obj_cgroup *); -} - -static bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, - struct list_lru *lru, - gfp_t flags, size_t size, - void **p) -{ - struct obj_cgroup *objcg; - struct slab *slab; - unsigned long off; - size_t i; - - /* - * The obtained objcg pointer is safe to use within the current scope, - * defined by current task or set_active_memcg() pair. - * obj_cgroup_get() is used to get a permanent reference. - */ - objcg = current_obj_cgroup(); - if (!objcg) - return true; - - /* - * slab_alloc_node() avoids the NULL check, so we might be called with a - * single NULL object. kmem_cache_alloc_bulk() aborts if it can't fill - * the whole requested size. - * return success as there's nothing to free back - */ - if (unlikely(*p == NULL)) - return true; - - flags &= gfp_allowed_mask; - - if (lru) { - int ret; - struct mem_cgroup *memcg; - - memcg = get_mem_cgroup_from_objcg(objcg); - ret = memcg_list_lru_alloc(memcg, lru, flags); - css_put(&memcg->css); - - if (ret) - return false; - } - - if (obj_cgroup_charge(objcg, flags, size * obj_full_size(s))) - return false; - - for (i = 0; i < size; i++) { - slab = virt_to_slab(p[i]); - - if (!slab_objcgs(slab) && - memcg_alloc_slab_cgroups(slab, s, flags, false)) { - obj_cgroup_uncharge(objcg, obj_full_size(s)); - continue; - } - - off = obj_to_index(s, slab, p[i]); - obj_cgroup_get(objcg); - slab_objcgs(slab)[off] = objcg; - mod_objcg_state(objcg, slab_pgdat(slab), - cache_vmstat_idx(s), obj_full_size(s)); - } - - return true; -} - static void memcg_alloc_abort_single(struct kmem_cache *s, void *object); static __fastpath_inline @@ -1986,27 +1907,6 @@ bool memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru, return false; } -static void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, - void **p, int objects, - struct obj_cgroup **objcgs) -{ - for (int i = 0; i < objects; i++) { - struct obj_cgroup *objcg; - unsigned int off; - - off = obj_to_index(s, slab, p[i]); - objcg = objcgs[off]; - if (!objcg) - continue; - - objcgs[off] = NULL; - obj_cgroup_uncharge(objcg, obj_full_size(s)); - mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), - -obj_full_size(s)); - obj_cgroup_put(objcg); - } -} - static __fastpath_inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, void **p, int objects) From patchwork Fri Mar 1 17:07:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 208956 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:fa17:b0:10a:f01:a869 with SMTP id ju23csp18890dyc; Fri, 1 Mar 2024 09:09:25 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWFLw1fA8FVVaXp45zHkrZY0A2ocD9Qe5+bRb4N36a0AROU6uVuWH5wRSaZ53eqNG4gJedPD9HAb+ereYmKafnW5Ua0HQ== X-Google-Smtp-Source: AGHT+IFqxui2zFrLF8FBRhvfJVGZ9vzUvexRRsvjNPAnIsvs1VcLTVGa4XxbwmhaVmHYF33XYYHd X-Received: by 2002:a05:622a:1004:b0:42e:cf93:5fff with SMTP id d4-20020a05622a100400b0042ecf935fffmr1561254qte.30.1709312965451; Fri, 01 Mar 2024 09:09:25 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709312965; cv=pass; d=google.com; s=arc-20160816; b=CGo2rpu9cF7pL7vJ7UMh/971HMhx/+3BL/VpiWJa0TOtB9mDWUqSsAodjpyQsQ/4++ PVXYE2zQNJb1immutPmxpHtI+4s7c1zBvoOYFhtfywIifynBtwHhVZPTKiN0KyZg/gtu Y1zE8Vefbhwz/LqEgkZ53gQNgAU3ka9WgKpGcdT/JCX8hTla448tG83tDbFwyki80ITk ju11fuRHgoFevajbB35xFUvvoyrset12XZUwwqYB73V7+ssOVZGwocgc7ySd0GP5qYgx GZxMZc8rde6Pv90LdZZxy/HK2KTp7dxvTh3+IqoCBr9c+IE5HSihThwrnAVd/04DxdP1 lT5w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:list-unsubscribe:list-subscribe:list-id:precedence :subject:date:from; bh=ejITEmwOvuELk0FkAjL3jJy28ouM8QRgbljo3Xli1AE=; fh=WGGwdVVTGLpVYCxjY83qTUlyyBzuw+WV0hgXC1Po200=; b=o2jlKtNqHwP7sxlvxTrjiZzVClGl5l3obcFbzor72vcpz8BhPWnwX1y5QukTbuSKFa iEW7lM/lSc/1QW6ZT92Q8XnQNdKCqxkcnPHp6OYM5rWW6F1ZREND3lTCHDAPyqNnR6TS TcPmXqQBjXfSureKWPI1RAb9Rm5Hvfe6Lr8anxjbmUKxwtJol+DIfzGxBI9CX7bIGxl5 P0R8usrPnznsgttRun15q8iXXpotJj/UqGPNURAwGCvScnz5WQbeU8KDBlmkughcwmG/ QjzQgleDuZjERq9wVcOPKWG0qZ8XPV3Ul6GIOfq5InqntX9AbEawpCHSY52VgNrgjPYB JNzQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=suse.cz); spf=pass (google.com: domain of linux-kernel+bounces-88801-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-88801-ouuuleilei=gmail.com@vger.kernel.org" Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id f3-20020ac85d03000000b0042ec0a915a7si3594229qtx.293.2024.03.01.09.09.25 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Mar 2024 09:09:25 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-88801-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=suse.cz); spf=pass (google.com: domain of linux-kernel+bounces-88801-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-88801-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 3863E1C2359E for ; Fri, 1 Mar 2024 17:08:29 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 19DBF282F9; Fri, 1 Mar 2024 17:07:17 +0000 (UTC) Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 825612568; Fri, 1 Mar 2024 17:07:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709312835; cv=none; b=b4/kvy6Xlh1kitVddBiw76qSuVMadggiFdeC7djlHEAaMbXur2mSv05UnUaDyHmvgUqt660tm38UFD6n5n/5inHJ6H3IJND69h7qxXCjXv4WTcnWyBYGwRktXcct402m1LB5zxEKOJif8gdykmD9QDsKIIVGBsVODyy++GR6rd0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709312835; c=relaxed/simple; bh=zHTjE2s9eU4e8ejE9RWwR3BzqKotd3aHXC+td0OJ2X8=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=c2v8nSbcaEzcMqlR/wb3cfwLq4oBIrwDGtUmEnSqHY/dZ7gH3MJkFxn8I1iWOQ9N0lMFFxPhje4Pcg6FXvWId/uqnSuBIl47YBmtGIwlNBa5Uhf3+uDAvP2O8TrtMWlCCoqE1Bmi4RDzj4EYDFpznfxk29NVJbr77XDrij2l/bI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id E827E33D2B; Fri, 1 Mar 2024 17:07:10 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id C31ED13A59; Fri, 1 Mar 2024 17:07:10 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id AAZcLz4L4mUcGQAAD6G6ig (envelope-from ); Fri, 01 Mar 2024 17:07:10 +0000 From: Vlastimil Babka Date: Fri, 01 Mar 2024 18:07:10 +0100 Subject: [PATCH RFC 3/4] mm, slab: introduce kmem_cache_charge() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240301-slab-memcg-v1-3-359328a46596@suse.cz> References: <20240301-slab-memcg-v1-0-359328a46596@suse.cz> In-Reply-To: <20240301-slab-memcg-v1-0-359328a46596@suse.cz> To: Linus Torvalds , Josh Poimboeuf , Jeff Layton , Chuck Lever , Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Alexander Viro , Christian Brauner , Jan Kara Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.13.0 Authentication-Results: smtp-out1.suse.de; none X-Spamd-Result: default: False [-4.00 / 50.00]; TAGGED_RCPT(0.00)[]; REPLY(-4.00)[] X-Rspamd-Server: rspamd1.dmz-prg2.suse.org X-Rspamd-Queue-Id: E827E33D2B X-Spam-Level: X-Spam-Score: -4.00 X-Spam-Flag: NO X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792344552008150206 X-GMAIL-MSGID: 1792344552008150206 As suggested by Linus, introduce a slab API function to memcg-charge a an object that was previously allocated without __GFP_ACCOUNT and from a cache that's not SLAB_ACCOUNT. This may be useful when it's likely the object is to be freed soon, and thus the charging/uncharging overhead can be avoided. In case kmem_cache_charge() is called on an already-charged object, it's a no-op. Suggested-by: Linus Torvalds Link: https://lore.kernel.org/all/CAHk-=whYOOdM7jWy5jdrAm8LxcgCMFyk2bt8fYYvZzM4U-zAQA@mail.gmail.com/ Signed-off-by: Vlastimil Babka --- include/linux/slab.h | 10 ++++++++++ mm/slub.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index b5f5ee8308d0..0c3acb2fa3e6 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -491,6 +491,16 @@ void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags) __assume_slab_ali void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags) __assume_slab_alignment __malloc; void kmem_cache_free(struct kmem_cache *s, void *objp); +#ifdef CONFIG_MEMCG_KMEM +int kmem_cache_charge(struct kmem_cache *s, gfp_t flags, void *objp); +#else +static inline int +kmem_cache_charge(struct kmem_cache *s, gfp_t flags, void *objp) +{ + return 0; +} +#endif + /* * Bulk allocation and freeing operations. These are accelerated in an diff --git a/mm/slub.c b/mm/slub.c index 64da169d672a..72b61b379ba1 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4241,6 +4241,35 @@ void kmem_cache_free(struct kmem_cache *s, void *x) } EXPORT_SYMBOL(kmem_cache_free); +#ifdef CONFIG_MEMCG_KMEM +int kmem_cache_charge(struct kmem_cache *s, gfp_t flags, void *x) +{ + struct obj_cgroup ** objcg; + struct slab *slab; + + s = cache_from_obj(s, x); + if (!s) + return -EINVAL; + + if (likely(!memcg_kmem_online())) + return 0; + + /* was it already accounted? */ + slab = virt_to_slab(x); + if ((objcg = slab_objcgs(slab))) { + unsigned int off = obj_to_index(s, slab, x); + + if (objcg[off]) + return 0; + } + + if (!memcg_slab_post_alloc_hook(s, NULL, flags, 1, &x)) + return -ENOMEM; + + return 0; +} +#endif + static void free_large_kmalloc(struct folio *folio, void *object) { unsigned int order = folio_order(folio); From patchwork Fri Mar 1 17:07:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 208979 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:fa17:b0:10a:f01:a869 with SMTP id ju23csp39205dyc; Fri, 1 Mar 2024 09:43:55 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVibza3eP+CNYOuWVLPvQhLEUFsgKmvqJwrXwadTXs9Q8SExLN9FNJ2REtdJOf07V42Ata5a3ZWZL/wt6lIoGzGF4i1oA== X-Google-Smtp-Source: AGHT+IEQdW/H/yt/p2nSdiRyoda0GREV2jfoGySGh76Cs0aS2ebDDhMRPAWvX9mgFu7i+oGCkDvg X-Received: by 2002:a17:902:ea0a:b0:1dc:30ab:406b with SMTP id s10-20020a170902ea0a00b001dc30ab406bmr2652806plg.13.1709315035379; Fri, 01 Mar 2024 09:43:55 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709315035; cv=pass; d=google.com; s=arc-20160816; b=rmSWL3jMX3U3PSTlqp8bw3CWwYOvVotB1OtvqOjtvrFsm2UoZdd6MIjKUKKjMu11oK aGRClZZA8tbgfswflVOEBbReXcmROM4pPWRBUwxU5FNHRqlk93c5Omr7jqR7FAc+4ciB SZmML8lucpStw5THF1/+Z2YD8dn3rQRwY5N/7XL4spGaF/V9XUlpmdSHxoSQnU2Hq53E yddXu9RxeoMZDU9KnVEoplLe3vWU/Q56cKES6oHt7H7awdhJlHCFZyCtZ+X+devgTcBX JCfZmx2Q3AmtyWSUHCGY41IXnEpOIvwYkH8r2b7tjOYHXvTU8yS/t19Uc6sDD26ey/Id yD6A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:list-unsubscribe:list-subscribe:list-id:precedence :subject:date:from:dkim-signature:dkim-signature:dkim-signature :dkim-signature; bh=3OiKnePSvssqSsATKZD6FgDSrsGbw6Gz9LH7uaXGsMo=; fh=WGGwdVVTGLpVYCxjY83qTUlyyBzuw+WV0hgXC1Po200=; b=gs5Qyq02HS1isMrtd0J7kKUeR5/jH6Tr6b3lbGzHDY6NDcM20uLUcuK7H+MfYERGQZ SfcW+uRliF9ocwJdVEWcPs5CFvWEnMSnnDk+SBfBw6U+BhAxqRT2IZpBhayEnQJBDYcC NBPcvOszP8W/sP2F81XrOCW0IvhhEQxL+Z7oSfUX2wCgsf2D1nU4woaB7ypYL3zNYzk8 b+r0CU75Wz2hFZpTMhaMOjv0+L17MbXnL4zOGEyI4TutW4PIuF8e1pZA+pAEEP7LZzQD cY7jYxubloBxSuqPnd59sLzYFefuWCAcdmAZvvKxJsfXt3KsZFeWOunCa7TJApuCftNg Wj+w==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=hR4RI3M7; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=hR4RI3M7; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519; arc=pass (i=1 spf=pass spfdomain=suse.cz dkim=pass dkdomain=suse.cz dkim=pass dkdomain=suse.cz); spf=pass (google.com: domain of linux-kernel+bounces-88799-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-88799-ouuuleilei=gmail.com@vger.kernel.org" Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id w13-20020a170902e88d00b001d9e1ab2acbsi4057308plg.356.2024.03.01.09.43.54 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Mar 2024 09:43:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-88799-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=hR4RI3M7; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=hR4RI3M7; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519; arc=pass (i=1 spf=pass spfdomain=suse.cz dkim=pass dkdomain=suse.cz dkim=pass dkdomain=suse.cz); spf=pass (google.com: domain of linux-kernel+bounces-88799-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-88799-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id ED4F9B2A16D for ; Fri, 1 Mar 2024 17:08:53 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9730338398; Fri, 1 Mar 2024 17:07:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="hR4RI3M7"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="o05E46K6"; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="hR4RI3M7"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="o05E46K6" Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C4FBA46B7; Fri, 1 Mar 2024 17:07:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709312834; cv=none; b=aSP5QSHl+bulrDBSro5CHLmmUSNwv0xOh2W7PCHX9bzVMw+iLY9BY8gk7cqQ7COLt84qFZlEVgcqME6gXzVsVb8xGCtTodcpd49dfWYZxSSlZ0QBB+354r8UWV09wXnoh3iLrxkdWqHlb10l7N1m0Vb4D6ijFhoqBMDkJvwgPB4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709312834; c=relaxed/simple; bh=QnfkJAoInt5fUE+416z0sQ20bCXrkSWJKNSLICwxmGM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=p95dDAU7RfLK74kD+9IggMtkH5zglerivEdHaMv236iJh6+dGhFTqrcAIe6/PqFS0LKbAwSI4cdmhZo0bk2FzmAdWOSnyBoHTPLywNe/xHhx8npCU3++sYE3ZVCDxjp+7rOOwj1Dk06nZWlYPqojnfhENh7p1D4iaNMHfeZ2xbI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=hR4RI3M7; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=o05E46K6; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=hR4RI3M7; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=o05E46K6; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 13FDA207B4; Fri, 1 Mar 2024 17:07:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1709312831; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3OiKnePSvssqSsATKZD6FgDSrsGbw6Gz9LH7uaXGsMo=; b=hR4RI3M7QdmI77qwLuZOBimpV0p2ze0qGZw1B4hIMLU2F/D7t3Gu5VbTAR9gTJgDi6vCeY CHs1xCbyeczWk9Hb3kbBfU7sfuhKG4jF0YdapV00Mtliv7GsO2UuM0v+bywHfHlN1udd4J we+zKQVacDSXAT9F9S8KKuGQXrqXA18= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1709312831; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3OiKnePSvssqSsATKZD6FgDSrsGbw6Gz9LH7uaXGsMo=; b=o05E46K6EN+fy853AkUBuf6HrQD5ZThoOdIKf4tjmcYR9J8tcaU11fvMi6vu+GSL77uZkk Y/9da7wAoHpaIGBA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1709312831; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3OiKnePSvssqSsATKZD6FgDSrsGbw6Gz9LH7uaXGsMo=; b=hR4RI3M7QdmI77qwLuZOBimpV0p2ze0qGZw1B4hIMLU2F/D7t3Gu5VbTAR9gTJgDi6vCeY CHs1xCbyeczWk9Hb3kbBfU7sfuhKG4jF0YdapV00Mtliv7GsO2UuM0v+bywHfHlN1udd4J we+zKQVacDSXAT9F9S8KKuGQXrqXA18= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1709312831; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3OiKnePSvssqSsATKZD6FgDSrsGbw6Gz9LH7uaXGsMo=; b=o05E46K6EN+fy853AkUBuf6HrQD5ZThoOdIKf4tjmcYR9J8tcaU11fvMi6vu+GSL77uZkk Y/9da7wAoHpaIGBA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id E618113AA3; Fri, 1 Mar 2024 17:07:10 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id MKjgNz4L4mUcGQAAD6G6ig (envelope-from ); Fri, 01 Mar 2024 17:07:10 +0000 From: Vlastimil Babka Date: Fri, 01 Mar 2024 18:07:11 +0100 Subject: [PATCH RFC 4/4] UNFINISHED mm, fs: use kmem_cache_charge() in path_openat() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240301-slab-memcg-v1-4-359328a46596@suse.cz> References: <20240301-slab-memcg-v1-0-359328a46596@suse.cz> In-Reply-To: <20240301-slab-memcg-v1-0-359328a46596@suse.cz> To: Linus Torvalds , Josh Poimboeuf , Jeff Layton , Chuck Lever , Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Alexander Viro , Christian Brauner , Jan Kara Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-fsdevel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.13.0 Authentication-Results: smtp-out2.suse.de; none X-Spam-Level: X-Spam-Score: -6.80 X-Spamd-Result: default: False [-6.80 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; REPLY(-4.00)[]; BAYES_HAM(-3.00)[100.00%]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_RATELIMIT(0.00)[to_ip_from(RL8ogcagzi1y561i1mcnzpnkwh)]; RCVD_COUNT_THREE(0.00)[3]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; RCPT_COUNT_TWELVE(0.00)[24]; FREEMAIL_TO(0.00)[linux-foundation.org,kernel.org,oracle.com,linux.com,google.com,lge.com,linux.dev,gmail.com,cmpxchg.org,zeniv.linux.org.uk,suse.cz]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; SUSPICIOUS_RECIPS(1.50)[] X-Spam-Flag: NO X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792346722299573439 X-GMAIL-MSGID: 1792346722299573439 This is just an example of using the kmem_cache_charge() API. I think it's placed in a place that's applicable for Linus's example [1] although he mentions do_dentry_open() - I have followed from strace() showing openat(2) to path_openat() doing the alloc_empty_file(). The idea is that filp_cachep stops being SLAB_ACCOUNT. Allocations that want to be accounted immediately can use GFP_KERNEL_ACCOUNT. I did that in alloc_empty_file_noaccount() (despite the contradictory name but the noaccount refers to something else, right?) as IIUC it's about kernel-internal opens. alloc_empty_file() is now not doing the accounting, so I added kmem_account_file() that calls the new kmem_cache_charge() API. Why is this unfinished: - there are other callers of alloc_empty_file() which I didn't adjust so they simply became memcg-unaccounted. I haven't investigated for which ones it would make also sense to separate the allocation and accounting. Maybe alloc_empty_file() would need to get a parameter to control this. - I don't know how to properly unwind the accounting failure case. It seems like a new case because when we succeed the open, there's no further error path at least in path_openat(). Basically it boils down I'm unfamiliar with VFS so this depends if this approach is deemed useful enough to finish it. [1] https://lore.kernel.org/all/CAHk-=whYOOdM7jWy5jdrAm8LxcgCMFyk2bt8fYYvZzM4U-zAQA@mail.gmail.com/ Not-signed-off-by: Vlastimil Babka --- fs/file_table.c | 9 +++++++-- fs/internal.h | 1 + fs/namei.c | 4 +++- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/fs/file_table.c b/fs/file_table.c index b991f90571b4..6401b6f175ae 100644 --- a/fs/file_table.c +++ b/fs/file_table.c @@ -223,6 +223,11 @@ struct file *alloc_empty_file(int flags, const struct cred *cred) return ERR_PTR(-ENFILE); } +int kmem_account_file(struct file *f) +{ + return kmem_cache_charge(filp_cachep, GFP_KERNEL_ACCOUNT, f); +} + /* * Variant of alloc_empty_file() that doesn't check and modify nr_files. * @@ -234,7 +239,7 @@ struct file *alloc_empty_file_noaccount(int flags, const struct cred *cred) struct file *f; int error; - f = kmem_cache_zalloc(filp_cachep, GFP_KERNEL); + f = kmem_cache_zalloc(filp_cachep, GFP_KERNEL_ACCOUNT); if (unlikely(!f)) return ERR_PTR(-ENOMEM); @@ -468,7 +473,7 @@ void __init files_init(void) { filp_cachep = kmem_cache_create("filp", sizeof(struct file), 0, SLAB_TYPESAFE_BY_RCU | SLAB_HWCACHE_ALIGN | - SLAB_PANIC | SLAB_ACCOUNT, NULL); + SLAB_PANIC, NULL); percpu_counter_init(&nr_files, 0, GFP_KERNEL); } diff --git a/fs/internal.h b/fs/internal.h index b67406435fc0..06ada11b71d0 100644 --- a/fs/internal.h +++ b/fs/internal.h @@ -96,6 +96,7 @@ extern void chroot_fs_refs(const struct path *, const struct path *); struct file *alloc_empty_file(int flags, const struct cred *cred); struct file *alloc_empty_file_noaccount(int flags, const struct cred *cred); struct file *alloc_empty_backing_file(int flags, const struct cred *cred); +int kmem_account_file(struct file *file); static inline void file_put_write_access(struct file *file) { diff --git a/fs/namei.c b/fs/namei.c index 4e0de939fea1..fcf3f3fcd059 100644 --- a/fs/namei.c +++ b/fs/namei.c @@ -3799,8 +3799,10 @@ static struct file *path_openat(struct nameidata *nd, terminate_walk(nd); } if (likely(!error)) { - if (likely(file->f_mode & FMODE_OPENED)) + if (likely(file->f_mode & FMODE_OPENED)) { + kmem_account_file(file); return file; + } WARN_ON(1); error = -EINVAL; }