From patchwork Tue Oct 10 00:09:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 150422 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp2190681vqo; Mon, 9 Oct 2023 17:10:14 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFEV1WtGUoc3EXZJ3AaXWmrDc/xEAwSXnuI2jtLjh7Jd1ODEpef5Fp1Agqu0pTMmX8VJfJ0 X-Received: by 2002:a17:902:d488:b0:1c8:9a60:387f with SMTP id c8-20020a170902d48800b001c89a60387fmr4645755plg.56.1696896613664; Mon, 09 Oct 2023 17:10:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696896613; cv=none; d=google.com; s=arc-20160816; b=QIxI1aEAaA89CTuuAlhdjTiXg9q3UNT9u8xGJQio9dJUP8AA6fc4XBe+hzXcpLkl7u Zroq/rcytp8xg7jAYOZJD9WB8ovD6PBcSBtNqmChuPCfmM3QLU/mHC+oyl0A1S6CRTO+ Tk4bna79+vDPAALLeajGaUYEvxzwKaZOxuT9KaW/oYELDfaWUTQf+EfR6YgEbNBGV9C1 cP3yM1vIQmVSzxazxfpMssJ4fe+O1zMo7LACtP6LhQkW9CwbeTtcBF9xL6uQ3uvxYRRv XdvLxdEIlOT6wByiTHZzrR10LIPnTPiJLBvPKhBRVv/dyvCaEh+vBbBWBrU15ZjXhR7x viHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=6BDA26poRN0lVkuNk023R6ElLY4Pdz30rttSzJzWmU0=; fh=eIV/wktpjV+EeUDLeXhmhAmcuYEtVvgOWNik/7Ld4iQ=; b=opBd/QWrx8RaO3Lk01jO4qbM6P8kcn7yKUDn+gTBQ1sqfbWK+yhjmIxe+xEIgJUny9 r+xEPMavA34Dl2M9zjXEys+E6cOaJEhHJzyMDHLnuM1fkZQXr3R6XCGXD7zVVU6woCl+ crLoAaSUda0PhozjEDAvUFZAcurnsiSp1SHuO1dvJspd7hFAR7If2mE2WOSC3ERRuoDT JatEKU/5/Q71J5dr2SKJRVsEmChEQugIUbbll2c+QRfWtH0FfEpDcbFG6PDQ9p9u/U4h 6slHcpynvlRb3dABBi07XD0u5QMj+6MC6GHoTGCU9KYy2gSwtHkVzh4yBI9Pzhl5t9hQ AUWA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=ft97nbcy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id p20-20020a170902e35400b001c0ab540e62si9943342plc.277.2023.10.09.17.10.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 17:10:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=ft97nbcy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 9F53080FDDC2; Mon, 9 Oct 2023 17:10:12 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1379156AbjJJAKF (ORCPT + 18 others); Mon, 9 Oct 2023 20:10:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42702 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1379138AbjJJAKD (ORCPT ); Mon, 9 Oct 2023 20:10:03 -0400 Received: from out-199.mta0.migadu.com (out-199.mta0.migadu.com [91.218.175.199]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 72B9CA4 for ; Mon, 9 Oct 2023 17:10:02 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1696896599; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6BDA26poRN0lVkuNk023R6ElLY4Pdz30rttSzJzWmU0=; b=ft97nbcyFPqeUqZ4lisCE8IdqHS0q8h0Wnsl4HIdSRB7hY45IRuQS+OuTRiaAGnlAil0tV 6n6m78HGgBIkfz0ByXmzPARF8sfyEGsDiqZLrbTdayMXeSR6UAx8qwH0gjpYcPSmPHKjtQ qzYOWVCL8FrVR9Z2F277gR71ZjBKiJU= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , David Rientjes , Vlastimil Babka , Naresh Kamboju , Roman Gushchin Subject: [PATCH v2 1/5] mm: kmem: optimize get_obj_cgroup_from_current() Date: Mon, 9 Oct 2023 17:09:25 -0700 Message-ID: <20231010000929.450702-2-roman.gushchin@linux.dev> In-Reply-To: <20231010000929.450702-1-roman.gushchin@linux.dev> References: <20231010000929.450702-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 09 Oct 2023 17:10:12 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779325063358327741 X-GMAIL-MSGID: 1779325063358327741 Manually inline memcg_kmem_bypass() and active_memcg() to speed up get_obj_cgroup_from_current() by avoiding duplicate in_task() checks and active_memcg() readings. Also add a likely() macro to __get_obj_cgroup_from_memcg(): obj_cgroup_tryget() should succeed at almost all times except a very unlikely race with the memcg deletion path. Signed-off-by: Roman Gushchin (Cruise) Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- mm/memcontrol.c | 34 ++++++++++++++-------------------- 1 file changed, 14 insertions(+), 20 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 9741d62d0424..16ac2a5838fb 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1068,19 +1068,6 @@ struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) } EXPORT_SYMBOL(get_mem_cgroup_from_mm); -static __always_inline bool memcg_kmem_bypass(void) -{ - /* Allow remote memcg charging from any context. */ - if (unlikely(active_memcg())) - return false; - - /* Memcg to charge can't be determined. */ - if (!in_task() || !current->mm || (current->flags & PF_KTHREAD)) - return true; - - return false; -} - /** * mem_cgroup_iter - iterate over memory cgroup hierarchy * @root: hierarchy root @@ -3007,7 +2994,7 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { objcg = rcu_dereference(memcg->objcg); - if (objcg && obj_cgroup_tryget(objcg)) + if (likely(objcg && obj_cgroup_tryget(objcg))) break; objcg = NULL; } @@ -3016,16 +3003,23 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { - struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg; + struct obj_cgroup *objcg; - if (memcg_kmem_bypass()) - return NULL; + if (in_task()) { + memcg = current->active_memcg; + + /* Memcg to charge can't be determined. */ + if (likely(!memcg) && (!current->mm || (current->flags & PF_KTHREAD))) + return NULL; + } else { + memcg = this_cpu_read(int_active_memcg); + if (likely(!memcg)) + return NULL; + } rcu_read_lock(); - if (unlikely(active_memcg())) - memcg = active_memcg(); - else + if (!memcg) memcg = mem_cgroup_from_task(current); objcg = __get_obj_cgroup_from_memcg(memcg); rcu_read_unlock(); From patchwork Tue Oct 10 00:09:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 150423 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp2190701vqo; Mon, 9 Oct 2023 17:10:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE4BqX7Pzn/1cjI5ZnhdFl6+/t9i1/22ZtiTNXjlN7ZXthBIG8f833UaC/a83/3nLAFNJK0 X-Received: by 2002:a17:90b:3ec6:b0:27c:bcb3:55b1 with SMTP id rm6-20020a17090b3ec600b0027cbcb355b1mr3366445pjb.15.1696896616265; Mon, 09 Oct 2023 17:10:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696896616; cv=none; d=google.com; s=arc-20160816; b=0aeY3fJ1Kf9uRlEPWibNgKjdnuBdxB86zrqXTtJIA7jPbGOE2wpKxExuKJQ79EZDNf YnVAm5rikNTpQHzJ+pS1vWHj9i/u3zf2h/g3rmFTjiWoQ2JzSgwcmX3hAIvzn+HQnY+h XYyMRUbAlUOQcnB5WOLQjwCp1bSaeJtL91EqPfWJIvZn0rgEGQg4YD+xTYjkwT5w8tdU WlO023xM7ouf0OwVKjyH1A2cUBC6FspJmeHMe4bk4aNozLYzxnTRMZ+m/NC+zf7nwV+3 /W8+l6yhLtGf16grSgEbyIbtsfQP58Yp0FeR4UEQDQa8qS4uzzuWkvNUUSajVCJk8Ixf eLnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=UB8UCsgBBNwm/lYUhug39JXjgrJB25jOFlVEA0SwZ1w=; fh=eIV/wktpjV+EeUDLeXhmhAmcuYEtVvgOWNik/7Ld4iQ=; b=qkHa1A1so4xUXYmiezqD/eZZxbh29RPFVunaGX9hwcwVqazkmeUOysuQwXV2ReDwyd tFcQ1Q3xB1uLJ9VOJy3Y6vm5UwvtxGvcBFCQce6IUYJqN+cdSa0irKeAnvWEWGbmhoPt FVNq8isHoOVkkN5mhXOmzfJ8xvtFnfbcgdyzfGeFoQhgE3E0j8XWedXPdeIE+bC+2vXx cAhkgyx1WATT94Qx1StUsC/pU4EJrb61Gf01tp1xtpOOTUgTtNCGk14suS31CuVoCI0T zCa6TAxT8pkTlsCoeT7Hm2Jh5JLb3qucvr/nKwxkVI4uchPbo/R8vV+ATOWtshqptQxN Qt8g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=vYGiTYSo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id qe17-20020a17090b4f9100b00258996c09e4si14023154pjb.22.2023.10.09.17.10.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 17:10:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=vYGiTYSo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id ADE0F80FDDCF; Mon, 9 Oct 2023 17:10:15 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1379176AbjJJAKL (ORCPT + 18 others); Mon, 9 Oct 2023 20:10:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42708 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1379150AbjJJAKF (ORCPT ); Mon, 9 Oct 2023 20:10:05 -0400 Received: from out-199.mta0.migadu.com (out-199.mta0.migadu.com [91.218.175.199]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 68DA4A9 for ; Mon, 9 Oct 2023 17:10:03 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1696896601; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UB8UCsgBBNwm/lYUhug39JXjgrJB25jOFlVEA0SwZ1w=; b=vYGiTYSof5OwIfp3SCsmVVAqIIOHV/2QRu9KSEpIse6pPitUQLPoDY3+768geHIe1OotqT gQVtl8qNLJ/m8/vVb6QjGtwGNF5Ng1ivDTPPJOXsC152HcMV8t2pO85z3+N7HypmzHOCBd PAkUbeSbCfQEs0F1f4lfujM5IDfCuz4= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , David Rientjes , Vlastimil Babka , Naresh Kamboju , Roman Gushchin Subject: [PATCH v2 2/5] mm: kmem: add direct objcg pointer to task_struct Date: Mon, 9 Oct 2023 17:09:26 -0700 Message-ID: <20231010000929.450702-3-roman.gushchin@linux.dev> In-Reply-To: <20231010000929.450702-1-roman.gushchin@linux.dev> References: <20231010000929.450702-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 09 Oct 2023 17:10:15 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779325066093227037 X-GMAIL-MSGID: 1779325066093227037 To charge a freshly allocated kernel object to a memory cgroup, the kernel needs to obtain an objcg pointer. Currently it does it indirectly by obtaining the memcg pointer first and then calling to __get_obj_cgroup_from_memcg(). Usually tasks spend their entire life belonging to the same object cgroup. So it makes sense to save the objcg pointer on task_struct directly, so it can be obtained faster. It requires some work on fork, exit and cgroup migrate paths, but these paths are way colder. To avoid any costly synchronization the following rules are applied: 1) A task sets it's objcg pointer itself. 2) If a task is being migrated to another cgroup, the least significant bit of the objcg pointer is set atomically. 3) On the allocation path the objcg pointer is obtained locklessly using the READ_ONCE() macro and the least significant bit is checked. If it's set, the following procedure is used to update it locklessly: - task->objcg is zeroed using cmpxcg - new objcg pointer is obtained - task->objcg is updated using try_cmpxchg - operation is repeated if try_cmpxcg fails It guarantees that no updates will be lost if task migration is racing against objcg pointer update. It also allows to keep both read and write paths fully lockless. Because the task is keeping a reference to the objcg, it can't go away while the task is alive. This commit doesn't change the way the remote memcg charging works. Signed-off-by: Roman Gushchin (Cruise) Acked-by: Johannes Weiner --- include/linux/sched.h | 4 ++ mm/memcontrol.c | 130 +++++++++++++++++++++++++++++++++++++++--- 2 files changed, 125 insertions(+), 9 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 77f01ac385f7..60de42715b56 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1443,6 +1443,10 @@ struct task_struct { struct mem_cgroup *active_memcg; #endif +#ifdef CONFIG_MEMCG_KMEM + struct obj_cgroup *objcg; +#endif + #ifdef CONFIG_BLK_CGROUP struct gendisk *throttle_disk; #endif diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 16ac2a5838fb..a2efeea43058 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -249,6 +249,8 @@ struct mem_cgroup *vmpressure_to_memcg(struct vmpressure *vmpr) return container_of(vmpr, struct mem_cgroup, vmpressure); } +#define CURRENT_OBJCG_UPDATE_FLAG 0x1UL + #ifdef CONFIG_MEMCG_KMEM static DEFINE_SPINLOCK(objcg_lock); @@ -3001,6 +3003,50 @@ static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *memcg) return objcg; } +static struct obj_cgroup *current_objcg_update(void) +{ + struct mem_cgroup *memcg; + struct obj_cgroup *old, *objcg = NULL; + + do { + /* Atomically drop the update bit. */ + old = xchg(¤t->objcg, NULL); + if (old) { + old = (struct obj_cgroup *) + ((unsigned long)old & ~CURRENT_OBJCG_UPDATE_FLAG); + if (old) + obj_cgroup_put(old); + + old = NULL; + } + + /* Obtain the new objcg pointer. */ + rcu_read_lock(); + memcg = mem_cgroup_from_task(current); + /* + * The current task can be asynchronously moved to another + * memcg and the previous memcg can be offlined. So let's + * get the memcg pointer and try get a reference to objcg + * under a rcu read lock. + */ + for (; memcg != root_mem_cgroup; memcg = parent_mem_cgroup(memcg)) { + objcg = rcu_dereference(memcg->objcg); + if (likely(objcg && obj_cgroup_tryget(objcg))) + break; + objcg = NULL; + } + rcu_read_unlock(); + + /* + * Try set up a new objcg pointer atomically. If it + * fails, it means the update flag was set concurrently, so + * the whole procedure should be repeated. + */ + } while (!try_cmpxchg(¤t->objcg, &old, objcg)); + + return objcg; +} + __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { struct mem_cgroup *memcg; @@ -3008,19 +3054,26 @@ __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) if (in_task()) { memcg = current->active_memcg; + if (unlikely(memcg)) + goto from_memcg; - /* Memcg to charge can't be determined. */ - if (likely(!memcg) && (!current->mm || (current->flags & PF_KTHREAD))) - return NULL; + objcg = READ_ONCE(current->objcg); + if (unlikely((unsigned long)objcg & CURRENT_OBJCG_UPDATE_FLAG)) + objcg = current_objcg_update(); + + if (objcg) { + obj_cgroup_get(objcg); + return objcg; + } } else { memcg = this_cpu_read(int_active_memcg); - if (likely(!memcg)) - return NULL; + if (unlikely(memcg)) + goto from_memcg; } + return NULL; +from_memcg: rcu_read_lock(); - if (!memcg) - memcg = mem_cgroup_from_task(current); objcg = __get_obj_cgroup_from_memcg(memcg); rcu_read_unlock(); return objcg; @@ -6345,6 +6398,7 @@ static void mem_cgroup_move_task(void) mem_cgroup_clear_mc(); } } + #else /* !CONFIG_MMU */ static int mem_cgroup_can_attach(struct cgroup_taskset *tset) { @@ -6358,8 +6412,39 @@ static void mem_cgroup_move_task(void) } #endif +#ifdef CONFIG_MEMCG_KMEM +static void mem_cgroup_fork(struct task_struct *task) +{ + /* + * Set the update flag to cause task->objcg to be initialized lazily + * on the first allocation. It can be done without any synchronization + * because it's always performed on the current task, so does + * current_objcg_update(). + */ + task->objcg = (struct obj_cgroup *)CURRENT_OBJCG_UPDATE_FLAG; +} + +static void mem_cgroup_exit(struct task_struct *task) +{ + struct obj_cgroup *objcg = task->objcg; + + objcg = (struct obj_cgroup *) + ((unsigned long)objcg & ~CURRENT_OBJCG_UPDATE_FLAG); + if (objcg) + obj_cgroup_put(objcg); + + /* + * Some kernel allocations can happen after this point, + * but let's ignore them. It can be done without any synchronization + * because it's always performed on the current task, so does + * current_objcg_update(). + */ + task->objcg = NULL; +} +#endif + #ifdef CONFIG_LRU_GEN -static void mem_cgroup_attach(struct cgroup_taskset *tset) +static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset) { struct task_struct *task; struct cgroup_subsys_state *css; @@ -6377,10 +6462,31 @@ static void mem_cgroup_attach(struct cgroup_taskset *tset) task_unlock(task); } #else +static void mem_cgroup_lru_gen_attach(struct cgroup_taskset *tset) {} +#endif /* CONFIG_LRU_GEN */ + +#ifdef CONFIG_MEMCG_KMEM +static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset) +{ + struct task_struct *task; + struct cgroup_subsys_state *css; + + cgroup_taskset_for_each(task, css, tset) { + /* atomically set the update bit */ + set_bit(0, (unsigned long *)¤t->objcg); + } +} +#else +static void mem_cgroup_kmem_attach(struct cgroup_taskset *tset) {} +#endif /* CONFIG_MEMCG_KMEM */ + +#if defined(CONFIG_LRU_GEN) || defined(CONFIG_MEMCG_KMEM) static void mem_cgroup_attach(struct cgroup_taskset *tset) { + mem_cgroup_lru_gen_attach(tset); + mem_cgroup_kmem_attach(tset); } -#endif /* CONFIG_LRU_GEN */ +#endif static int seq_puts_memcg_tunable(struct seq_file *m, unsigned long value) { @@ -6824,9 +6930,15 @@ struct cgroup_subsys memory_cgrp_subsys = { .css_reset = mem_cgroup_css_reset, .css_rstat_flush = mem_cgroup_css_rstat_flush, .can_attach = mem_cgroup_can_attach, +#if defined(CONFIG_LRU_GEN) || defined(CONFIG_MEMCG_KMEM) .attach = mem_cgroup_attach, +#endif .cancel_attach = mem_cgroup_cancel_attach, .post_attach = mem_cgroup_move_task, +#ifdef CONFIG_MEMCG_KMEM + .fork = mem_cgroup_fork, + .exit = mem_cgroup_exit, +#endif .dfl_cftypes = memory_files, .legacy_cftypes = mem_cgroup_legacy_files, .early_init = 0, From patchwork Tue Oct 10 00:09:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 150424 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp2190724vqo; Mon, 9 Oct 2023 17:10:20 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEWm+NJJtMWeVCqUwWqBOeh2Lk6axUY+zF82qoTowWdMbZceIFztvq4cQJw8FIWZPgeIDov X-Received: by 2002:a17:90b:33c5:b0:27b:4909:aad1 with SMTP id lk5-20020a17090b33c500b0027b4909aad1mr5401746pjb.29.1696896620230; Mon, 09 Oct 2023 17:10:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696896620; cv=none; d=google.com; s=arc-20160816; b=ZfbKAS9YCYM19PT+KmLNBEDIkYmQDPImu00fWl5F4QDv+LYkA4PUIt5LwND6mv0c42 9DgBJwHju4DUfWJ1bSd9BB9KLZF9gtDKUbefpPJLYNgCIZBT+EdEBjV2YH7GBMm5Fk5T Asp84Q8rpUYzmizLFsO1dvEA1Pz9mBzVOhEjGm56ojcD/+wDwOK1r5iJyfV4+6VYH5ZM YCoDicvRJ4DLux2FE4zhZSZgaRGVtzVM2j32/3tWBLJmg8BJIavSksNTAVBFowS/qcfT qZ9ZBUhQfSSpiCYFGIRZEdjM3Xyi1KBUiHLCky31CCpKGkTKKDyaoP2fB3znMZGrgvgP XYYg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jfnE2/YTK2KTtvwJ+nJA5/HU7PGBa8pUnySPUFyhBIQ=; fh=eIV/wktpjV+EeUDLeXhmhAmcuYEtVvgOWNik/7Ld4iQ=; b=G4eTs+Exme2QPP15OIMOvYlp3DjzeqXQNZi0iVoCzboQwA1Hsm2aOkpiFZ6CC90c8w UW3do+mmu9JbYy5dTGh5sm3KCqIDH0Kqb5TyR1gnBQL0FpBjcBc6VpNLHQ5ibQLdNSP4 04Y9DJc0jUlsyEJ7wIogf0iQaXkmaXJ/D608Cc4pPWdF+Xo4hsLdVtxGleljhsXQhrII lot5unl04qqCeLwPZlQlxBf0aXT3cDNEqvqjxdOLyIwTPBmW/nd/qwBxBD9XYODWXfMo zGsfqSIh4VbI4VmeRdi4c6wLxf+cNPTu9zsC3+FPkP8Ge/kUfTbp4jXqORPUduboC78S zA7Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=qLsl13mT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id m3-20020a17090aab0300b00276bdabe471si12225765pjq.163.2023.10.09.17.10.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 17:10:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=qLsl13mT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id E8E8A80FDDDB; Mon, 9 Oct 2023 17:10:18 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1379138AbjJJAKN (ORCPT + 18 others); Mon, 9 Oct 2023 20:10:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51046 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1379166AbjJJAKK (ORCPT ); Mon, 9 Oct 2023 20:10:10 -0400 Received: from out-195.mta0.migadu.com (out-195.mta0.migadu.com [91.218.175.195]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 67C6DCF for ; Mon, 9 Oct 2023 17:10:08 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1696896604; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jfnE2/YTK2KTtvwJ+nJA5/HU7PGBa8pUnySPUFyhBIQ=; b=qLsl13mTbXz/4i7wT4UNbU/Yuhrs6p0pMqUkgW5cQutUavEUX53hOM0UdEJeTs+zo2p5sJ GQojwhgKw9URgh6Cnj8JskC/4by4Jp7qKD398cvjsd/sOFJjD43axUombC6Fgh4TUL5CuQ PNw7ZFBmb9QNHRWuBYOV43b0EnZyUDM= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , David Rientjes , Vlastimil Babka , Naresh Kamboju , Roman Gushchin Subject: [PATCH v2 3/5] mm: kmem: make memcg keep a reference to the original objcg Date: Mon, 9 Oct 2023 17:09:27 -0700 Message-ID: <20231010000929.450702-4-roman.gushchin@linux.dev> In-Reply-To: <20231010000929.450702-1-roman.gushchin@linux.dev> References: <20231010000929.450702-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 09 Oct 2023 17:10:19 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779325070278075630 X-GMAIL-MSGID: 1779325070278075630 Keep a reference to the original objcg object for the entire life of a memcg structure. This allows to simplify the synchronization on the kernel memory allocation paths: pinning a (live) memcg will also pin the corresponding objcg. The memory overhead of this change is minimal because object cgroups usually outlive their corresponding memory cgroups even without this change, so it's only an additional pointer per memcg. Signed-off-by: Roman Gushchin (Cruise) Acked-by: Shakeel Butt --- include/linux/memcontrol.h | 8 +++++++- mm/memcontrol.c | 5 +++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ab94ad4597d0..277690af383d 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -299,7 +299,13 @@ struct mem_cgroup { #ifdef CONFIG_MEMCG_KMEM int kmemcg_id; - struct obj_cgroup __rcu *objcg; + /* + * memcg->objcg is wiped out as a part of the objcg repaprenting + * process. memcg->orig_objcg preserves a pointer (and a reference) + * to the original objcg until the end of live of memcg. + */ + struct obj_cgroup __rcu *objcg; + struct obj_cgroup *orig_objcg; /* list of inherited objcgs, protected by objcg_lock */ struct list_head objcg_list; #endif diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a2efeea43058..490c5f67be79 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3808,6 +3808,8 @@ static int memcg_online_kmem(struct mem_cgroup *memcg) objcg->memcg = memcg; rcu_assign_pointer(memcg->objcg, objcg); + obj_cgroup_get(objcg); + memcg->orig_objcg = objcg; static_branch_enable(&memcg_kmem_online_key); @@ -5302,6 +5304,9 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg) { int node; + if (memcg->orig_objcg) + obj_cgroup_put(memcg->orig_objcg); + for_each_node(node) free_mem_cgroup_per_node_info(memcg, node); kfree(memcg->vmstats); From patchwork Tue Oct 10 00:09:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 150425 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp2191160vqo; Mon, 9 Oct 2023 17:11:31 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEH3mi3q6T1Id74837wFO50S7kPc78AjEVtacVQDpGQJ/vxeDa0vSH7CzM7I7xyQjuYvORG X-Received: by 2002:a17:90a:d707:b0:27c:e74a:a4b1 with SMTP id y7-20020a17090ad70700b0027ce74aa4b1mr426891pju.36.1696896691522; Mon, 09 Oct 2023 17:11:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696896691; cv=none; d=google.com; s=arc-20160816; b=IzmUtcfi/sCvrky2ztjCMa+nT6y6aea7RDINe0PSCqk0Fgzl2DPwNEylNWuvv5YVvr NqbYhVvI6Rpa+dNXxfr0ev2mRFqeSTOQyVcEjx+d7a8QqLXgMerYy6LQzBKD4zNYqGms O3IFBnUuPe2mZbs8DrbKfv1kmlF8kQK7CaZUBfILVXaxxkQFmxcX8NLvlD39DZPLGgmz Ix0RqHGbciE7KsvlZ75KxOcF+umY3MgdLXHYLcm4FI4mCVyI4/eo9092Zq30VanqE0U4 AkSJawTwT6vXZ6QDGRx9vdILX7HJbDVMGvog3BFPKIIT4W8erc3xz2bC4tbq3tzgBKq1 VqKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vaNtRxcBkqs+qNsGw0HnKynEFDwRPPwwrEaTCT/4xFw=; fh=eIV/wktpjV+EeUDLeXhmhAmcuYEtVvgOWNik/7Ld4iQ=; b=GwvLogQ6H836wLBmvjrULWPu5ErtmAiRkW53lYgXbFCagEpER9ot7EHNvLcf3v041i WqPYb7oyBsegiaedJ9+P/dXxq1R59spNgmVVen1xBoeECVyJpOFa3pDYzS9rRQAbrxcv otOMZ5KAi1eueJLsIUVWVUE92lokj1RK9mlVDR/alvQoRP5yAFDWwL5Ma2Ld8yGo2cnI eUukGtZZhTgTc18QBD7ge9KpbqLfK4vqPEN7kKrnGvYOOvXJ8U7DlYCHJA6qfJ+v2IAz pXXm2A2p4Wg+emsbP5jt/bUxI1syYSxKwYAZgZs4Oa2c+GGC1R8bKpd7xpk28zl00Z9Z x0/A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="M/HNb3YK"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id j2-20020a170902da8200b001c449fb7bfcsi11364837plx.353.2023.10.09.17.11.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 17:11:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="M/HNb3YK"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 885F980811F0; Mon, 9 Oct 2023 17:10:28 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1379186AbjJJAKP (ORCPT + 18 others); Mon, 9 Oct 2023 20:10:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51046 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1379169AbjJJAKK (ORCPT ); Mon, 9 Oct 2023 20:10:10 -0400 Received: from out-210.mta0.migadu.com (out-210.mta0.migadu.com [91.218.175.210]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A83FDA for ; Mon, 9 Oct 2023 17:10:09 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1696896607; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vaNtRxcBkqs+qNsGw0HnKynEFDwRPPwwrEaTCT/4xFw=; b=M/HNb3YKc2bGW5FqYd/GesbtjL4WMaYzcJf13vLo0uLcpBE3PKKz+0p+JT6Cdmcu7vY5Yn RNsW9yYFROw7J8lJJuMjKiT3ExifQsuvx6IMfw/EZFZy1RZTpeDFncxOFPz8hXFU61Wa2S ta2O/F0zqFj/J+i3jJVN9WgsTLIfc8Q= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , David Rientjes , Vlastimil Babka , Naresh Kamboju , Roman Gushchin Subject: [PATCH v2 4/5] mm: kmem: scoped objcg protection Date: Mon, 9 Oct 2023 17:09:28 -0700 Message-ID: <20231010000929.450702-5-roman.gushchin@linux.dev> In-Reply-To: <20231010000929.450702-1-roman.gushchin@linux.dev> References: <20231010000929.450702-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=2.7 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Mon, 09 Oct 2023 17:10:28 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779325144967583712 X-GMAIL-MSGID: 1779325144967583712 Switch to a scope-based protection of the objcg pointer on slab/kmem allocation paths. Instead of using the get_() semantics in the pre-allocation hook and put the reference afterwards, let's rely on the fact that objcg is pinned by the scope. It's possible because: 1) if the objcg is received from the current task struct, the task is keeping a reference to the objcg. 2) if the objcg is received from an active memcg (remote charging), the memcg is pinned by the scope and has a reference to the corresponding objcg. Signed-off-by: Roman Gushchin (Cruise) Acked-by: Shakeel Butt --- include/linux/memcontrol.h | 6 +++++ mm/memcontrol.c | 47 ++++++++++++++++++++++++++++++++++++-- mm/slab.h | 10 +++----- 3 files changed, 54 insertions(+), 9 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 277690af383d..0e53b890f063 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1769,6 +1769,12 @@ bool mem_cgroup_kmem_disabled(void); int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order); void __memcg_kmem_uncharge_page(struct page *page, int order); +/* + * The returned objcg pointer is safe to use without additional + * protection within a scope, refer to the implementation for the + * additional details. + */ +struct obj_cgroup *current_obj_cgroup(void); struct obj_cgroup *get_obj_cgroup_from_current(void); struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 490c5f67be79..84724fb875ce 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3079,6 +3079,49 @@ __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) return objcg; } +__always_inline struct obj_cgroup *current_obj_cgroup(void) +{ + struct mem_cgroup *memcg; + struct obj_cgroup *objcg; + + if (in_task()) { + memcg = current->active_memcg; + if (unlikely(memcg)) + goto from_memcg; + + objcg = READ_ONCE(current->objcg); + if (unlikely((unsigned long)objcg & CURRENT_OBJCG_UPDATE_FLAG)) + objcg = current_objcg_update(); + /* + * Objcg reference is kept by the task, so it's safe + * to use the objcg by the current task. + */ + return objcg; + } + + memcg = this_cpu_read(int_active_memcg); + if (unlikely(memcg)) + goto from_memcg; + + return NULL; + +from_memcg: + for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) { + /* + * Memcg pointer is protected by scope (see set_active_memcg()) + * and is pinning the corresponding objcg, so objcg can't go + * away and can be used within the scope without any additional + * protection. + */ + objcg = rcu_dereference_check(memcg->objcg, 1); + if (likely(objcg)) + break; + objcg = NULL; + } + + return objcg; +} + struct obj_cgroup *get_obj_cgroup_from_folio(struct folio *folio) { struct obj_cgroup *objcg; @@ -3173,15 +3216,15 @@ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) struct obj_cgroup *objcg; int ret = 0; - objcg = get_obj_cgroup_from_current(); + objcg = current_obj_cgroup(); if (objcg) { ret = obj_cgroup_charge_pages(objcg, gfp, 1 << order); if (!ret) { + obj_cgroup_get(objcg); page->memcg_data = (unsigned long)objcg | MEMCG_DATA_KMEM; return 0; } - obj_cgroup_put(objcg); } return ret; } diff --git a/mm/slab.h b/mm/slab.h index 799a315695c6..8cd3294fedf5 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -484,7 +484,7 @@ static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, if (!(flags & __GFP_ACCOUNT) && !(s->flags & SLAB_ACCOUNT)) return true; - objcg = get_obj_cgroup_from_current(); + objcg = current_obj_cgroup(); if (!objcg) return true; @@ -497,17 +497,14 @@ static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, css_put(&memcg->css); if (ret) - goto out; + return false; } if (obj_cgroup_charge(objcg, flags, objects * obj_full_size(s))) - goto out; + return false; *objcgp = objcg; return true; -out: - obj_cgroup_put(objcg); - return false; } static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, @@ -542,7 +539,6 @@ static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, obj_cgroup_uncharge(objcg, obj_full_size(s)); } } - obj_cgroup_put(objcg); } static inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, From patchwork Tue Oct 10 00:09:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 150426 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp2193456vqo; Mon, 9 Oct 2023 17:17:06 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEkwulDDtFUK0x1p4lgOhG8ADdLt6+ceXv6icsk8jVJFDuP9xYMUFoH6+u8KAIrCy4EPGMX X-Received: by 2002:a05:6870:7196:b0:1d6:7f77:c922 with SMTP id d22-20020a056870719600b001d67f77c922mr19402130oah.28.1696897025987; Mon, 09 Oct 2023 17:17:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696897025; cv=none; d=google.com; s=arc-20160816; b=NeODA0EBt9+iTP4LiPLJTW3dEmMQzSgsEA6/h1oyWu2FMwAOK0SSnVbwv8v+s6yvlT lEryBjTFRBjnnQ/e+GivFzLvDbQsEULWHVGijC+1X+7Q6YBHl8uhe3dfacy7hVpNXaSu +vP67V8HWmD4iovylvYlRCeFSmu0VzFHf2ph99rwj4HGctMie4zjiVsoRB169a/2DFzf e7nObyiuGqcaotQf7vk6tKJU2cajPtqHleY1AAX10qEZ+CRUco8zDIGVM0odFaQ2j/61 xYTH9KvHfhhGAA6iND1ZZiX8bsEz80g4G/XHV8UP/vLmILrzKMAM1bKI2tJzlq4ZleMv FKSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=j6TjL0tOzMm9uw8Y9niPXJ1ezOtPxfS6WilpL+cESRE=; fh=eIV/wktpjV+EeUDLeXhmhAmcuYEtVvgOWNik/7Ld4iQ=; b=DYTPL5mAzwnKTDcJADZOhxoeC/vv0L1F4fHE4IO7yY8J6ygx72mn2azmf6Yz5NMtsM TuNyj+mVP7Orcf9POi8lxZ0z39IHZ7ImtkwAAXPBQxAmz3w/xATMRLsQ/rivD0AHHqcy Cb0NXU43RaYJCnqZGoxpA7F8dqrehEVUuld8cJAbw+yIbKuEwhFWcf/AvKSvqvLgXxq1 hi/LPFDhOU/WmqYFuzsoxjXBpXNbGjX87okDHgPk7yM0VbZAEYCazamN7ycILrYB/G5C gNt20ejWOoNDKQVaZs7mvASl7yUIL5yR/Uvc5sZXUUxq6ghh3mUFN6GmOpMfqQgdpYsW Jm1w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=ZGPVwiM2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id dc4-20020a056a0035c400b00690cff4a2b6si8116713pfb.112.2023.10.09.17.17.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 17:17:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=ZGPVwiM2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id EA7F680617C3; Mon, 9 Oct 2023 17:16:20 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1379226AbjJJAKZ (ORCPT + 18 others); Mon, 9 Oct 2023 20:10:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1379191AbjJJAKV (ORCPT ); Mon, 9 Oct 2023 20:10:21 -0400 Received: from out-210.mta0.migadu.com (out-210.mta0.migadu.com [91.218.175.210]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 994C1BA for ; Mon, 9 Oct 2023 17:10:13 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1696896610; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j6TjL0tOzMm9uw8Y9niPXJ1ezOtPxfS6WilpL+cESRE=; b=ZGPVwiM2h+XunCiZLqM4FyYGsiSKjC0phq0S+wyUEtVkyO+vvaGmE0lOyAzfhgRkZqIQVI z+aLmjDpmuOiL6ag53Op+5oV0AIkM4FSmAvmbCqi/H5wKmFVEnAUYzw7h9+t++JC6EK5WG wCu/Uex1P/e7AHO9YRFh11WGzRmaLv8= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , David Rientjes , Vlastimil Babka , Naresh Kamboju , Roman Gushchin Subject: [PATCH v2 5/5] percpu: scoped objcg protection Date: Mon, 9 Oct 2023 17:09:29 -0700 Message-ID: <20231010000929.450702-6-roman.gushchin@linux.dev> In-Reply-To: <20231010000929.450702-1-roman.gushchin@linux.dev> References: <20231010000929.450702-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 09 Oct 2023 17:16:20 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779325495714662913 X-GMAIL-MSGID: 1779325495714662913 Similar to slab and kmem, switch to a scope-based protection of the objcg pointer to avoid. Signed-off-by: Roman Gushchin (Cruise) Acked-by: Shakeel Butt --- mm/percpu.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/mm/percpu.c b/mm/percpu.c index a7665de8485f..f53ba692d67a 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1628,14 +1628,12 @@ static bool pcpu_memcg_pre_alloc_hook(size_t size, gfp_t gfp, if (!memcg_kmem_online() || !(gfp & __GFP_ACCOUNT)) return true; - objcg = get_obj_cgroup_from_current(); + objcg = current_obj_cgroup(); if (!objcg) return true; - if (obj_cgroup_charge(objcg, gfp, pcpu_obj_full_size(size))) { - obj_cgroup_put(objcg); + if (obj_cgroup_charge(objcg, gfp, pcpu_obj_full_size(size))) return false; - } *objcgp = objcg; return true; @@ -1649,6 +1647,7 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg, return; if (likely(chunk && chunk->obj_cgroups)) { + obj_cgroup_get(objcg); chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = objcg; rcu_read_lock(); @@ -1657,7 +1656,6 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg, rcu_read_unlock(); } else { obj_cgroup_uncharge(objcg, pcpu_obj_full_size(size)); - obj_cgroup_put(objcg); } }