From patchwork Wed Sep 27 15:08:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 145581 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp2882329vqu; Wed, 27 Sep 2023 13:25:42 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGwQ/mUW7Sxygia/3wuCosWvQZA9eSdmIWxkAmhsTc61IG4DS6ZWo4EQJnaZn8okJIpD/v4 X-Received: by 2002:a05:6a20:12d6:b0:157:d7cd:ebd with SMTP id v22-20020a056a2012d600b00157d7cd0ebdmr3579598pzg.1.1695846342066; Wed, 27 Sep 2023 13:25:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695846342; cv=none; d=google.com; s=arc-20160816; b=kDfJxQMe/luYYHOkp3iX8dqryvJErcBxb7e46KPbBLkvDN8Tq3yM1+b9rIg/c7EkIt vLenkbBWYGv24jtFyQaVJos4M74jDNyb5UCBxiDt8/uWFhF9tfe6B2GNUj+vDmAB8PcP sRUVA9bLw6T3R+zm7wLiVz39MsCvhgLjlJORiQOeEjuK1mLPKAzT8J+kq+R2fxYBNbEC ZAe9DsZ74UjPZ3vV2QAcBEXICdl53AQjHsTtfcBuRB7kRhefDUiMsTK+DN2A7XAB+oZ9 JUXY+d/9Sx4fipIA7pFwTpPwr//B4mFiveJvmCVsTa9NzwGZfh6xNTLg3XVukn3wPcaC fRhw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=i5Ua/wdmvBo/wu1nXQ0Cl5cuox9DFv/B5F3hb3WMJFo=; fh=J+ThODwbvpIaijJ/a/gIBgC20RizoNlQ1sjxEf1nY4o=; b=WkzsaVsVqW5wnySyFufmBy3tFGGcnz06ktuZhAuD4iU6SYSiD3ujZpCRtwbR+xrEkW tA7PpmIZDFUcj7UTyaZs4W37fjLfJG0b7pBgZJo5NO9FhLfeaX9G8YfMgvCd4/6t9XZ0 8a8i6PgNVpgjn6zRgBXUPFavtK5JTV07FYpcYsOB2q87IHebHwUtT01toD5WErj+UCq2 k9GNOrZkMAnoEr6Xa6zAYkGiT+5hC529N5p9lxz+KyP54wwudf/jKleOO+0obRJwjXUL Mxekx+cca4gyQUOvYnJCwA0pLxeRedebpe0sNIb5h50fU7e5twz+e7nKJFlldqVmOgv/ YGKg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Q8hMpLXl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id pf2-20020a17090b1d8200b0024e47fae466si7418254pjb.180.2023.09.27.13.25.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Sep 2023 13:25:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Q8hMpLXl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id C431D80BEF32; Wed, 27 Sep 2023 08:09:36 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232348AbjI0PJC (ORCPT + 19 others); Wed, 27 Sep 2023 11:09:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232336AbjI0PI7 (ORCPT ); Wed, 27 Sep 2023 11:08:59 -0400 Received: from out-203.mta1.migadu.com (out-203.mta1.migadu.com [IPv6:2001:41d0:203:375::cb]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B02F1139 for ; Wed, 27 Sep 2023 08:08:57 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1695827335; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i5Ua/wdmvBo/wu1nXQ0Cl5cuox9DFv/B5F3hb3WMJFo=; b=Q8hMpLXlwyHYuhYTY7QmN3ucJ4BymSYDpw0OZAW82OI1HjO9QhUqV7hIgxRE3/+F1Zdf1X 9pwvp7A1SXynG+K4IEOnW40uYr4B8pTu6BDJ26Y+WSYnLIfaJXQFLXbhSCwQWYyEHZxai0 mR7jQfl6X8WWs0zHRyt6d4SZPcg5cVY= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , Roman Gushchin Subject: [PATCH rfc 1/5] mm: kmem: optimize get_obj_cgroup_from_current() Date: Wed, 27 Sep 2023 08:08:28 -0700 Message-ID: <20230927150832.335132-2-roman.gushchin@linux.dev> In-Reply-To: <20230927150832.335132-1-roman.gushchin@linux.dev> References: <20230927150832.335132-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 27 Sep 2023 08:09:36 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778223773707846902 X-GMAIL-MSGID: 1778223773707846902 Manually inline memcg_kmem_bypass() and active_memcg() to speed up get_obj_cgroup_from_current() by avoiding duplicate in_task() checks and active_memcg() readings. Also add a likely() macro to __get_obj_cgroup_from_memcg(): obj_cgroup_tryget() should succeed at almost all times except a very unlikely race with the memcg deletion path. Signed-off-by: Roman Gushchin (Cruise) Acked-by: Shakeel Butt --- mm/memcontrol.c | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 9741d62d0424..16ac2a5838fb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1068,19 +1068,6 @@ struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) } EXPORT_SYMBOL(get_mem_cgroup_from_mm); -static __always_inline bool memcg_kmem_bypass(void) -{ - /* Allow remote memcg charging from any context. */ - if (unlikely(active_memcg())) - return false; - - /* Memcg to charge can't be determined. */ - if (!in_task() || !current->mm || (current->flags & PF_KTHREAD)) - return true; - - return false; -} - /** * mem_cgroup_iter - iterate over memory cgroup hierarchy * @root: hierarchy root @@ -3007,7 +2994,7 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { objcg = rcu_dereference(memcg->objcg); - if (objcg && obj_cgroup_tryget(objcg)) + if (likely(objcg && obj_cgroup_tryget(objcg))) break; objcg = NULL; } @@ -3016,16 +3003,23 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { - struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg; + struct obj_cgroup *objcg; - if (memcg_kmem_bypass()) - return NULL; + if (in_task()) { + memcg = current->active_memcg; + + /* Memcg to charge can't be determined. */ + if (likely(!memcg) && (!current->mm || (current->flags & PF_KTHREAD))) + return NULL; + } else { + memcg = this_cpu_read(int_active_memcg); + if (likely(!memcg)) + return NULL; + } rcu_read_lock(); - if (unlikely(active_memcg())) - memcg = active_memcg(); - else + if (!memcg) memcg = mem_cgroup_from_task(current); objcg = __get_obj_cgroup_from_memcg(memcg); rcu_read_unlock(); From patchwork Wed Sep 27 15:08:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 145748 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp3093440vqu; Wed, 27 Sep 2023 22:48:00 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEcP/wHe/XQMoJrUjkcUEDeIuEgou6mveKO013aA1sPfZSV8WLh19NIRQHA8ch9jG25s6dK X-Received: by 2002:a05:6358:591f:b0:143:9075:6d9c with SMTP id g31-20020a056358591f00b0014390756d9cmr308760rwf.5.1695880080571; Wed, 27 Sep 2023 22:48:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695880080; cv=none; d=google.com; s=arc-20160816; b=BmQ6kPT3CuKdDPx+QrTVi4n266F4Nm1k91Et+R9UD9621EaRRaTP7Mhc5ZEKec8UEp DEvYn42/MMyaa+quY2RfcT30/hDkt6t3tdNft1FrFGJDADINHbXg4He1QqCJ1P4GkyHC PPtHIH6cr6gtT43MA9HmL+j8ZCB7dWbqG5NeIm8cRaFbaJqtjr47kOMBEGTRYTKdZRAs m8Fn85BZjM16tS2on6TK/487qTyFf4weUbcNuR/GB2BqpZFcR/k4LDRiGJroPvdGjqLR oWqjxsCxeod1YZKOqq6RYlBRkjlhGjjy9OOmYGlcTryeLkFW0QJxrHEA4xJXPqse3eHI XIKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Rcjc8iZxDm0DIYnvNFEg9AlvdD1o0gQiWksXrzSEaP4=; fh=J+ThODwbvpIaijJ/a/gIBgC20RizoNlQ1sjxEf1nY4o=; b=WpRNEqe3JNa/GXrG+JqLX7f1lDZx3ecIEFMOb8zoELAyD9hbNQoVTmNi1bE08YPSVA wm2cP9F0KiGtWDCzZ2/5eDj3lPfkgysCvzDgZMqzGSOCC2/ffezMg/mobVsZt1I8hr3e GIL9PIgb7UHt6TCEAcJRa7z+1Jq295DBh2u+jEZYRr71DhE46+ZlPc/dtAU/jlVafmtt EM2CyHg3egDk2BvH8zbF3Iu+ra8BBhe7GqZoFp67YRA+S6bBdzF4WEqLoG6ue+OSiAa0 b+EQhb4bPOooyxB1Tjh3sRUzb78buZ0HbQI7Rl40gUHUCgOICTbR2C5OCAwSJEy5Fb0V uu+Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=UsUhkDML; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id pf11-20020a17090b1d8b00b002776ebaec7fsi9491320pjb.122.2023.09.27.22.48.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Sep 2023 22:48:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=UsUhkDML; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 37A0F8029280; Wed, 27 Sep 2023 08:09:29 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232345AbjI0PJF (ORCPT + 19 others); Wed, 27 Sep 2023 11:09:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232346AbjI0PJB (ORCPT ); Wed, 27 Sep 2023 11:09:01 -0400 Received: from out-192.mta1.migadu.com (out-192.mta1.migadu.com [IPv6:2001:41d0:203:375::c0]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D8C19191 for ; Wed, 27 Sep 2023 08:08:58 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1695827337; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Rcjc8iZxDm0DIYnvNFEg9AlvdD1o0gQiWksXrzSEaP4=; b=UsUhkDMLWoqWyU+5nbHGAFCVQZ5r2vg+XsHsShhnKPKLyiziXVyVxR9lcV7QyDWCZsJ8KL 7K/IZURrLDyBSSujwFVKE2aqWxPUYANqDoYmQ7nFA92KgzQTpifzj6n13nvCNmo4/FEFGw NyMkSUQBXz32JTg1sqlghlRIHfkR4f8= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , Roman Gushchin Subject: [PATCH rfc 2/5] mm: kmem: add direct objcg pointer to task_struct Date: Wed, 27 Sep 2023 08:08:29 -0700 Message-ID: <20230927150832.335132-3-roman.gushchin@linux.dev> In-Reply-To: <20230927150832.335132-1-roman.gushchin@linux.dev> References: <20230927150832.335132-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 27 Sep 2023 08:09:30 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778259151174882823 X-GMAIL-MSGID: 1778259151174882823 To charge a freshly allocated kernel object to a memory cgroup, the kernel needs to obtain an objcg pointer. Currently it does it indirectly by obtaining the memcg pointer first and then calling to __get_obj_cgroup_from_memcg(). Usually tasks spend their entire life belonging to the same object cgroup. So it makes sense to save the objcg pointer on task_struct directly, so it can be obtained faster. It requires some work on fork, exit and cgroup migrate paths, but these paths are way colder. To avoid any costly synchronization the following rules are applied: 1) A task sets it's objcg pointer itself. 2) If a task is being migrated to another cgroup, the least significant bit of the objcg pointer is set. 3) On the allocation path the objcg pointer is obtained locklessly using the READ_ONCE() macro and the least significant bit is checked. If it set, the task updates it's objcg before proceeding with an allocation. 4) Operations 1) and 4) are synchronized via a new spinlock, so that if a task is moved twice, the update bit can't be lost. This allows to keep the hot path fully lockless. Because the task is keeping a reference to the objcg, it can't go away while the task is alive. This commit doesn't change the way the remote memcg charging works. Signed-off-by: Roman Gushchin (Cruise) --- include/linux/memcontrol.h | 10 ++++ include/linux/sched.h | 4 ++ mm/memcontrol.c | 107 +++++++++++++++++++++++++++++++++---- 3 files changed, 112 insertions(+), 9 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ab94ad4597d0..84425bfe4124 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -553,6 +553,16 @@ static inline bool folio_memcg_kmem(struct folio *folio) return folio->memcg_data & MEMCG_DATA_KMEM; } +static inline bool current_objcg_needs_update(struct obj_cgroup *objcg) +{ + return (struct obj_cgroup *)((unsigned long)objcg & 0x1); +} + +static inline struct obj_cgroup * +current_objcg_clear_update_flag(struct obj_cgroup *objcg) +{ + return (struct obj_cgroup *)((unsigned long)objcg & ~0x1); +} #else static inline bool folio_memcg_kmem(struct folio *folio) diff --git a/include/linux/sched.h b/include/linux/sched.h index 77f01ac385f7..60de42715b56 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1443,6 +1443,10 @@ struct task_struct { struct mem_cgroup *active_memcg; #endif +#ifdef CONFIG_MEMCG_KMEM + struct obj_cgroup *objcg; +#endif + #ifdef CONFIG_BLK_CGROUP struct gendisk *throttle_disk; #endif diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 16ac2a5838fb..7f33a503d600 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3001,6 +3001,47 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) return objcg; } +static DEFINE_SPINLOCK(current_objcg_lock); + +static struct obj_cgroup *current_objcg_update(struct obj_cgroup *old) +{ + struct mem_cgroup *memcg; + struct obj_cgroup *objcg; + unsigned long flags; + + old = current_objcg_clear_update_flag(old); + if (old) + obj_cgroup_put(old); + + spin_lock_irqsave(¤t_objcg_lock, flags); + rcu_read_lock(); + memcg = mem_cgroup_from_task(current); + for (; memcg != root_mem_cgroup; memcg = parent_mem_cgroup(memcg)) { + objcg = rcu_dereference(memcg->objcg); + if (objcg && obj_cgroup_tryget(objcg)) + break; + objcg = NULL; + } + rcu_read_unlock(); + + WRITE_ONCE(current->objcg, objcg); + spin_unlock_irqrestore(¤t_objcg_lock, flags); + + return objcg; +} + +static inline void current_objcg_set_needs_update(struct task_struct *task) +{ + struct obj_cgroup *objcg; + unsigned long flags; + + spin_lock_irqsave(¤t_objcg_lock, flags); + objcg = READ_ONCE(task->objcg); + objcg = (struct obj_cgroup *)((unsigned long)objcg | 0x1); + WRITE_ONCE(task->objcg, objcg); + spin_unlock_irqrestore(¤t_objcg_lock, flags); +} + __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { struct mem_cgroup *memcg; @@ -3008,19 +3049,26 @@ __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) if (in_task()) { memcg = current->active_memcg; + if (unlikely(memcg)) + goto from_memcg; - /* Memcg to charge can't be determined. */ - if (likely(!memcg) && (!current->mm || (current->flags & PF_KTHREAD))) - return NULL; + objcg = READ_ONCE(current->objcg); + if (unlikely(current_objcg_needs_update(objcg))) + objcg = current_objcg_update(objcg); + + if (objcg) { + obj_cgroup_get(objcg); + return objcg; + } } else { memcg = this_cpu_read(int_active_memcg); - if (likely(!memcg)) - return NULL; + if (unlikely(memcg)) + goto from_memcg; } + return NULL; +from_memcg: rcu_read_lock(); - if (!memcg) - memcg = mem_cgroup_from_task(current); objcg = __get_obj_cgroup_from_memcg(memcg); rcu_read_unlock(); return objcg; @@ -6345,6 +6393,22 @@ static void mem_cgroup_move_task(void) mem_cgroup_clear_mc(); } } + +#ifdef CONFIG_MEMCG_KMEM +static void mem_cgroup_fork(struct task_struct *task) +{ + task->objcg = (struct obj_cgroup *)0x1; +} + +static void mem_cgroup_exit(struct task_struct *task) +{ + struct obj_cgroup *objcg = current_objcg_clear_update_flag(task->objcg); + + if (objcg) + obj_cgroup_put(objcg); +} +#endif + #else /* !CONFIG_MMU */ static int mem_cgroup_can_attach(struct cgroup_taskset *tset) { @@ -6359,7 +6423,7 @@ static void mem_cgroup_move_task(void) #endif #ifdef CONFIG_LRU_GEN -static void mem_cgroup_attach(struct cgroup_taskset *tset) +static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset) { struct task_struct *task; struct cgroup_subsys_state *css; @@ -6377,10 +6441,29 @@ static void mem_cgroup_attach(struct cgroup_taskset *tset) task_unlock(task); } #else +static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset) {} +#endif /* CONFIG_LRU_GEN */ + +#ifdef CONFIG_MEMCG_KMEM +static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset) +{ + struct task_struct *task; + struct cgroup_subsys_state *css; + + cgroup_taskset_for_each(task, css, tset) + current_objcg_set_needs_update(task); +} +#else +static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset) {} +#endif /* CONFIG_MEMCG_KMEM */ + +#if defined(CONFIG_LRU_GEN) || defined(CONFIG_MEMCG_KMEM) static void mem_cgroup_attach(struct cgroup_taskset *tset) { + mem_cgroup_lru_gen_attach(tset); + mem_cgroup_kmem_attach(tset); } -#endif /* CONFIG_LRU_GEN */ +#endif static int seq_puts_memcg_tunable(struct seq_file *m, unsigned long value) { @@ -6824,9 +6907,15 @@ struct cgroup_subsys memory_cgrp_subsys = { .css_reset = mem_cgroup_css_reset, .css_rstat_flush = mem_cgroup_css_rstat_flush, .can_attach = mem_cgroup_can_attach, +#if defined(CONFIG_LRU_GEN) || defined(CONFIG_MEMCG_KMEM) .attach = mem_cgroup_attach, +#endif .cancel_attach = mem_cgroup_cancel_attach, .post_attach = mem_cgroup_move_task, +#ifdef CONFIG_MEMCG_KMEM + .fork = mem_cgroup_fork, + .exit = mem_cgroup_exit, +#endif .dfl_cftypes = memory_files, .legacy_cftypes = mem_cgroup_legacy_files, .early_init = 0, From patchwork Wed Sep 27 15:08:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 146050 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp3249689vqu; Thu, 28 Sep 2023 04:51:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF37QV5Y2UA6KY8mvcqjuYb1+uMbNWz3sNIgpspXX/iqCEEeCKzY6Y0+Boo1KUmc8iNg9XA X-Received: by 2002:a05:6a20:948f:b0:161:2cf2:75ec with SMTP id hs15-20020a056a20948f00b001612cf275ecmr864972pzb.49.1695901876567; Thu, 28 Sep 2023 04:51:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695901876; cv=none; d=google.com; s=arc-20160816; b=B5I35YBTiHN0pcyNL6AlRzWnM0pIJwnI42z0EVWfK1rhgClOAfJp7Y+pb2ZaAQ4Ryw rups58OAs2dpc7NDcDkKciR+alQiJFrkoIuB7YhHXwpQVd5WKWqsho/yq9TuXyQyFZOG 6TxgFB22p1PkHp3aWOK2MeNY8imPkTjZOSQiyMlpm1zZiftN+ikfJwAa/XDR4ORccNNo fzwC7KlYaOVZE2EJVCYc1QCdGEjvRie+qxjoRSCz2gVp/ZDf3HC8FTA5CCKAvb9yJG/w u0mCnFUltxEUJqsn4JFLezybBX5F7eVSJSZGXiKfZgIEvRxavgx8SWO4eW/lGnCIe/Pg a7Sw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=NJPTphOT9Ah3iL7MZ9KbERD0ROIs5HQQzvQjuLNJnns=; fh=J+ThODwbvpIaijJ/a/gIBgC20RizoNlQ1sjxEf1nY4o=; b=khPkqVCWsaKc3p2p+jnD7Wh/rWh55iO/syhwyvj97Q5G8uSg5toQHimplWVYnEK3YS hG2Mup3SsJJyXgUNLyDrhxQsWxCZz94eJK389BkHPHGG1XLfWlEriSC4clra0r+RtZ5y 3/ZIsCp8raaWf0GxrkwtUobgOf3PUse5rOh9CokTal0/DAucNYwZ69rrXxzD3Wc0GKTN SBv0lKsBwUK/XFTwKNcQ1ltNQ0RY0xeXHxD+gk9j4m2B3qk1F1tgaWIYLKvJ4W2tAmlp PP3j0jXd/kEdjSkl1wnG4qTemIkWYwKdf9JsM5WHZHer2wmDLKrNSxvM4xqWHtYom/75 fKFw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=pNocy6o9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id 19-20020a631753000000b00577ae8b603fsi17359408pgx.314.2023.09.28.04.51.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Sep 2023 04:51:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=pNocy6o9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id DFF128058A04; Wed, 27 Sep 2023 08:09:39 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232366AbjI0PJG (ORCPT + 19 others); Wed, 27 Sep 2023 11:09:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232355AbjI0PJD (ORCPT ); Wed, 27 Sep 2023 11:09:03 -0400 Received: from out-201.mta1.migadu.com (out-201.mta1.migadu.com [IPv6:2001:41d0:203:375::c9]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45E3B191 for ; Wed, 27 Sep 2023 08:09:01 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1695827339; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NJPTphOT9Ah3iL7MZ9KbERD0ROIs5HQQzvQjuLNJnns=; b=pNocy6o9sigu+gpBp2nZU0nrp9UmQjhqPdMh8hND8MX7ixqhRzLWJmdV9/s/k8Rs1A5Qa5 6IMESlh4qTyehhfmr0rDc9in0xsrFhOJIDRwOsOUFW2sCFbjw/rB+qADTL7a3cw1BDr2Zq iq5OmF6L+8t6+ySa0+DaftZUm1opJI4= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , Roman Gushchin Subject: [PATCH rfc 3/5] mm: kmem: make memcg keep a reference to the original objcg Date: Wed, 27 Sep 2023 08:08:30 -0700 Message-ID: <20230927150832.335132-4-roman.gushchin@linux.dev> In-Reply-To: <20230927150832.335132-1-roman.gushchin@linux.dev> References: <20230927150832.335132-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 27 Sep 2023 08:09:40 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778282005980813398 X-GMAIL-MSGID: 1778282005980813398 Keep a reference to the original objcg object for the entire life of a memcg structure. This allows to simplify the synchronization on the kernel memory allocation paths: pinning a (live) memcg will also pin the corresponding objcg. The memory overhead of this change is minimal because object cgroups usually outlive their corresponding memory cgroups even without this change, so it's only an additional pointer per memcg. Signed-off-by: Roman Gushchin (Cruise) --- include/linux/memcontrol.h | 8 +++++++- mm/memcontrol.c | 5 +++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 84425bfe4124..412ff0e8694d 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -299,7 +299,13 @@ struct mem_cgroup { #ifdef CONFIG_MEMCG_KMEM int kmemcg_id; - struct obj_cgroup __rcu *objcg; + /* + * memcg->objcg is wiped out as a part of the objcg repaprenting + * process. memcg->orig_objcg preserves a pointer (and a reference) + * to the original objcg until the end of live of memcg. + */ + struct obj_cgroup __rcu *objcg; + struct obj_cgroup *orig_objcg; /* list of inherited objcgs, protected by objcg_lock */ struct list_head objcg_list; #endif diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7f33a503d600..4815f897758c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3803,6 +3803,8 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) objcg->memcg = memcg; rcu_assign_pointer(memcg->objcg, objcg); + obj_cgroup_get(objcg); + memcg->orig_objcg = objcg; static_branch_enable(&memcg_kmem_online_key); @@ -5297,6 +5299,9 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg) { int node; + if (memcg->orig_objcg) + obj_cgroup_put(memcg->orig_objcg); + for_each_node(node) free_mem_cgroup_per_node_info(memcg, node); kfree(memcg->vmstats); From patchwork Wed Sep 27 15:08:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 145447 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp2736038vqu; Wed, 27 Sep 2023 09:09:40 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEUilLwfvKZTEiw7pzWhRVl2yBT1i9aO8j8XO6FuF28G8JGh7KmdUEq1CBdn384A3l0UEAM X-Received: by 2002:a05:6a00:b4f:b0:690:f877:aa1e with SMTP id p15-20020a056a000b4f00b00690f877aa1emr4221100pfo.12.1695830980210; Wed, 27 Sep 2023 09:09:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695830980; cv=none; d=google.com; s=arc-20160816; b=pTY+ztc+/cTbgXGS+Mi8zc/I4/c21WhV/juhjRP3D5IynhqJd7SAXwbPRH7a/E6lYK Y7VgZ81w6NWp5guhQz4YjnSp1ku4rus8Jrx2UBGfoxAVaH6Wemm9Nvr/VPo757c+TDzv b1ySdbY/SNMiLRPmZuzywvcD09EzBwXYEH62o26q+cqnd2kWM7bFOuhTSYI1TZtsh7ea 67lOjwQSDPCV5x6Ot8VKuWske/ttUrQzH1iapfKcCVr4EZgOTivR1o5c8LuWxlOiSUHB Ibq2aqgAyM63b2fJvui/F0MaASYwWbcbQtveJFoyMbezlUO59NUa7ZeNrot1l2zmAma6 /yNQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jeQJ/eIJXU0AYA9yk+s4aO7OKAgQk2qdJmGkoMOLMDk=; fh=J+ThODwbvpIaijJ/a/gIBgC20RizoNlQ1sjxEf1nY4o=; b=KloBBxiO7mtfiZbrsL4KyqXRw4gktv39030MYNkVDOed50JK6OfotrNUl6sjEsnJph frpp6RqqUbmVsVnKN9Q1OlvrZo9p/QtYX54hJdXFnw7GHmk+ExK9X7gcSJXjonoEfReG hZwoBqKlhZxi5rKCQhtW14ec8B5bcn3S1JR0G8Min+6H3U8VdmulDjNYfVlNHlp2xncZ pB/sWDyiX2UhZJ1Q6w0PoEXWeTxANQWOaim9SC4YFznxNboleVNwjPogcloswU+Q4l5M FreSQzOwrNNJh+GyCyzWl3R8ZYw7/Giya6aUAlwz94XuMS1Jl+ZrPQbH0I8jSbAN25BJ YDDw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=c6ZuccPL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id g19-20020a631113000000b0057851fc2b4asi16039650pgl.55.2023.09.27.09.09.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Sep 2023 09:09:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=c6ZuccPL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 847BF8026969; Wed, 27 Sep 2023 08:09:50 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232417AbjI0PJQ (ORCPT + 19 others); Wed, 27 Sep 2023 11:09:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58990 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232336AbjI0PJE (ORCPT ); Wed, 27 Sep 2023 11:09:04 -0400 Received: from out-201.mta1.migadu.com (out-201.mta1.migadu.com [IPv6:2001:41d0:203:375::c9]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4779A1B1 for ; Wed, 27 Sep 2023 08:09:03 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1695827341; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jeQJ/eIJXU0AYA9yk+s4aO7OKAgQk2qdJmGkoMOLMDk=; b=c6ZuccPLxLC+Ce4wWZ9aCvoGVbip0UFScbqDK/VCUHTzImAhP4V7Ea1dW8CIa6yy8rjLZJ 0Sp2j/SlVWnSus6gz0MlC0eiKozxqUtiaYSQ8bVhbIgA/srrcTqQNsoM1ecUX491WcPVhw WEyQ3zRkmTvotbuHgAt2bGqQFn9TnMI= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , Roman Gushchin Subject: [PATCH rfc 4/5] mm: kmem: scoped objcg protection Date: Wed, 27 Sep 2023 08:08:31 -0700 Message-ID: <20230927150832.335132-5-roman.gushchin@linux.dev> In-Reply-To: <20230927150832.335132-1-roman.gushchin@linux.dev> References: <20230927150832.335132-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 27 Sep 2023 08:09:50 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778207666167153601 X-GMAIL-MSGID: 1778207666167153601 Switch to a scope-based protection of the objcg pointer on slab/kmem allocation paths. Instead of using the get_() semantics in the pre-allocation hook and put the reference afterwards, let's rely on the fact that objcg is pinned by the scope. It's possible because: 1) if the objcg is received from the current task struct, the task is keeping a reference to the objcg. 2) if the objcg is received from an active memcg (remote charging), the memcg is pinned by the scope and has a reference to the corresponding objcg. Signed-off-by: Roman Gushchin (Cruise) --- include/linux/memcontrol.h | 6 +++++ mm/memcontrol.c | 46 ++++++++++++++++++++++++++++++++++++-- mm/slab.h | 10 +++------ 3 files changed, 53 insertions(+), 9 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 412ff0e8694d..9a5212d3b9d8 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1779,6 +1779,12 @@ bool mem_cgroup_kmem_disabled(void); int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order); void __memcg_kmem_uncharge_page(struct page *page, int order); +/* + * The returned objcg pointer is safe to use without additional + * protection within a scope, refer to the implementation for the + * additional details. + */ +struct obj_cgroup *current_obj_cgroup(void); struct obj_cgroup *get_obj_cgroup_from_current(void); struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4815f897758c..76557370f212 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3074,6 +3074,48 @@ __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) return objcg; } +__always_inline struct obj_cgroup *current_obj_cgroup(void) +{ + struct mem_cgroup *memcg; + struct obj_cgroup *objcg; + + if (in_task()) { + memcg = current->active_memcg; + if (unlikely(memcg)) + goto from_memcg; + + objcg = READ_ONCE(current->objcg); + if (unlikely(current_objcg_needs_update(objcg))) + objcg = current_objcg_update(objcg); + /* + * Objcg reference is kept by the task, so it's safe + * to use the objcg by the current task. + */ + return objcg; + } else { + memcg = this_cpu_read(int_active_memcg); + if (unlikely(memcg)) + goto from_memcg; + } + return NULL; + +from_memcg: + for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { + /* + * Memcg pointer is protected by scope (see set_active_memcg()) + * and is pinning the corresponding objcg, so objcg can't go + * away and can be used within the scope without any additional + * protection. + */ + objcg = rcu_dereference_check(memcg->objcg, 1); + if (likely(objcg)) + break; + objcg = NULL; + } + + return objcg; +} + struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio) { struct obj_cgroup *objcg; @@ -3168,15 +3210,15 @@ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) struct obj_cgroup *objcg; int ret = 0; - objcg = get_obj_cgroup_from_current(); + objcg = current_obj_cgroup(); if (objcg) { ret = obj_cgroup_charge_pages(objcg, gfp, 1 << order); if (!ret) { + obj_cgroup_get(objcg); page->memcg_data = (unsigned long)objcg | MEMCG_DATA_KMEM; return 0; } - obj_cgroup_put(objcg); } return ret; } diff --git a/mm/slab.h b/mm/slab.h index 799a315695c6..8cd3294fedf5 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -484,7 +484,7 @@ static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, if (!(flags & __GFP_ACCOUNT) && !(s->flags & SLAB_ACCOUNT)) return true; - objcg = get_obj_cgroup_from_current(); + objcg = current_obj_cgroup(); if (!objcg) return true; @@ -497,17 +497,14 @@ static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, css_put(&memcg->css); if (ret) - goto out; + return false; } if (obj_cgroup_charge(objcg, flags, objects * obj_full_size(s))) - goto out; + return false; *objcgp = objcg; return true; -out: - obj_cgroup_put(objcg); - return false; } static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, @@ -542,7 +539,6 @@ static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, obj_cgroup_uncharge(objcg, obj_full_size(s)); } } - obj_cgroup_put(objcg); } static inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, From patchwork Wed Sep 27 15:08:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 145452 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp2741503vqu; Wed, 27 Sep 2023 09:17:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHMYngjSYw8kQlA1Bh/0l81Ijz9V2F+RrOYVgDIWlsF7iP+3SlgapygRIzVwlcwB5TSuPrO X-Received: by 2002:a05:6808:2cf:b0:3a7:6ff5:c628 with SMTP id a15-20020a05680802cf00b003a76ff5c628mr2548020oid.11.1695831436008; Wed, 27 Sep 2023 09:17:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695831435; cv=none; d=google.com; s=arc-20160816; b=hHCuMo3yEgfVu/cEqvO50CSVmIuGCag8UMbTuA62KPYZD9NmlqyN587wKU2u7S0Btr 0hHCgooMvqiCOOu9LAmCd9ajJLWyOYHSdKaIg5Pk1TNWu27jSWT+RX+C0Y3AMNd+t/3K H9SP/VbreQ3yUPOrNfE075IhXIruC9BFK5xk8YGwZ/YMrOESLPPbDnIP9VHjWW+L6bxc IPw8M/fc3fDhZZTA7g5hV67cP2lW7IHaRq1CxoAAwLN/tsbe00/48HUW2Opx+EymH0BO wH5EFxgwqpGxEj4FJVAHfBnCe2CZqvWvKhmm9kkzpGRapaQnXeijgnSf+PuQBnEyDAda CVPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=j6TjL0tOzMm9uw8Y9niPXJ1ezOtPxfS6WilpL+cESRE=; fh=J+ThODwbvpIaijJ/a/gIBgC20RizoNlQ1sjxEf1nY4o=; b=ju0f98RYvLdEmAXytVjIsrKyATVwFQSMsRHIH4azppEBNBSJyFbfgM6fHLeS6X6I50 LPwXN0nj21u9j+DYayLjjmwOgMvslVoKytu1stR652NT8s8SAkuWPtj9aamfbVZs1H4k BzXSqwPLAnwWVvJraGwXw/3N2p4T/4EYwVUIKgU+UVxKK5RkRMuHX4YQ29xBlR0/aA17 y12KYqZkCgxcTBG2/SOHYh3MglF6LHj9dPBquDLXgtyWSpTMTxOCLjCn2T3qj3gNv074 MWEDDYB3uWC/ekOR2SKXac7K84NvahkVkdt7nBChwovDt+kqm7w6tdcO5Rxf4wSFEQjI FmcQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="V//ihQR3"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id f20-20020a637554000000b0057790a5a8e7si16600389pgn.190.2023.09.27.09.17.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Sep 2023 09:17:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="V//ihQR3"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 02D7A80D04FE; Wed, 27 Sep 2023 08:09:58 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232380AbjI0PJW (ORCPT + 19 others); Wed, 27 Sep 2023 11:09:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56808 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232381AbjI0PJO (ORCPT ); Wed, 27 Sep 2023 11:09:14 -0400 Received: from out-193.mta1.migadu.com (out-193.mta1.migadu.com [IPv6:2001:41d0:203:375::c1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9D6811AC for ; Wed, 27 Sep 2023 08:09:07 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1695827344; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j6TjL0tOzMm9uw8Y9niPXJ1ezOtPxfS6WilpL+cESRE=; b=V//ihQR3BNX8mCaIN2RHbsj24ybf5tsZR80ZurjIq/P4zdNE6/CNiqXGaUHELfnQzFCvFu ald/Rulhzhr/xKdfWTbsCyE4U6wbBWcnVys7yuOgrgVNmXQqA6dKbwvtqUzJLSfKLVSbkz VEv/GP4pB24vCqh16SVt3PIzQgO5+VI= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , Roman Gushchin Subject: [PATCH rfc 5/5] percpu: scoped objcg protection Date: Wed, 27 Sep 2023 08:08:32 -0700 Message-ID: <20230927150832.335132-6-roman.gushchin@linux.dev> In-Reply-To: <20230927150832.335132-1-roman.gushchin@linux.dev> References: <20230927150832.335132-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 27 Sep 2023 08:09:58 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778208143748472695 X-GMAIL-MSGID: 1778208143748472695 Similar to slab and kmem, switch to a scope-based protection of the objcg pointer to avoid. Signed-off-by: Roman Gushchin (Cruise) --- mm/percpu.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/mm/percpu.c b/mm/percpu.c index a7665de8485f..f53ba692d67a 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1628,14 +1628,12 @@ static bool pcpu_memcg_pre_alloc_hook(size_t size, gfp_t gfp, if (!memcg_kmem_online() || !(gfp & __GFP_ACCOUNT)) return true; - objcg = get_obj_cgroup_from_current(); + objcg = current_obj_cgroup(); if (!objcg) return true; - if (obj_cgroup_charge(objcg, gfp, pcpu_obj_full_size(size))) { - obj_cgroup_put(objcg); + if (obj_cgroup_charge(objcg, gfp, pcpu_obj_full_size(size))) return false; - } *objcgp = objcg; return true; @@ -1649,6 +1647,7 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg, return; if (likely(chunk && chunk->obj_cgroups)) { + obj_cgroup_get(objcg); chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = objcg; rcu_read_lock(); @@ -1657,7 +1656,6 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg, rcu_read_unlock(); } else { obj_cgroup_uncharge(objcg, pcpu_obj_full_size(size)); - obj_cgroup_put(objcg); } }