Message ID | 20231016221900.4031141-1-roman.gushchin@linux.dev |
---|---|
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp3755877vqb; Mon, 16 Oct 2023 15:19:30 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEfBTq2Jszdu8FKttz9nlcvZMqKIh8yeJVgfs2fizKW3bC509i5AGzFdJq/t7LvNgBayQtk X-Received: by 2002:a05:6a00:1d09:b0:6bc:ff89:a2fc with SMTP id a9-20020a056a001d0900b006bcff89a2fcmr502594pfx.2.1697494770648; Mon, 16 Oct 2023 15:19:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697494770; cv=none; d=google.com; s=arc-20160816; b=EXIfdRyezxY/DVIdwAmKhK8lNgEgFQzgSBEitBziipXnz34A8P7An1cZH7XP7H5eO4 NhG2e4VvwdqYnk97HhABBL7TyYwiZcu1dka8hIbhbN4utTx5fewFbK7hnFqOXNotjFp3 2nK5z1IpBKP958EagC++cQbQBwRaGyFVVQ96XHyS3cF8eYwHxfTzlHSWakigVHoXGO8D fAiauFDpFXFsUGGmVjhIsIeEzDEf3IaibQgiBcVTWEYIbMlr56VXdKz6rPdYd6AlqgBe NRFys0V0+b1xJpmyXlk94O29AqewLClgni//qCVCu1+hVvk8DMIbqqROKJYxUrV+v/rQ Hziw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=me+EtXsO4OQNKVxpXtZoBKeIB+zeonfL/v0A4vdt7vw=; fh=TyPg14X/GXePfX2HFCpC8BoYIRT/2UqYFN7bT1+n50k=; b=sP48MVhgNtvUeZmnj1N5aWeOL6aLXWAKXF3zteJR8SGWSltwptZ1E5i23mybp/A7eb O+H6Xuh0wE1mFwgpUAuk4OOcCJH1hXWeKPE6QLXOM5iLJChM60/gPqFBhIOV+S/j5O3/ ng7FQsc+C/5rI80Kl8cxDvfr3MruCMLcJQb6MoF8eGmrSZmEe24YXtCgVqytw/u1Ng5a j/QkzQFv60Ru+TSB96dT93DOcbQBorJcPsSATKJENwQlWloi+tJr7DlEZ+RB+3ubEaYU sotYsx1MngYUuIrswtMZY31WOeiDakg368YrB6QlKEhppmv8+cBL3yq9foG0ngAYVVkl X1NA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linux.dev header.s=key1 header.b=UerryqrH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id q5-20020a656a85000000b005a9fb6bf73esi281867pgu.293.2023.10.16.15.19.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 15:19:30 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=fail header.i=@linux.dev header.s=key1 header.b=UerryqrH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id D39FD8028872; Mon, 16 Oct 2023 15:19:29 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234134AbjJPWT1 (ORCPT <rfc822;hjfbswb@gmail.com> + 18 others); Mon, 16 Oct 2023 18:19:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56110 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229943AbjJPWTZ (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Mon, 16 Oct 2023 18:19:25 -0400 Received: from out-204.mta1.migadu.com (out-204.mta1.migadu.com [IPv6:2001:41d0:203:375::cc]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6754795 for <linux-kernel@vger.kernel.org>; Mon, 16 Oct 2023 15:19:23 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1697494761; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=me+EtXsO4OQNKVxpXtZoBKeIB+zeonfL/v0A4vdt7vw=; b=UerryqrHGKvuW2aa+eacApzC4j9BLgQ297VVhGO3cJayVal+++VpmtupMGR/y5uY5xhzLn 4pT1IG4+1slJUAqTXFz6w8YCrOwL/ErDBejwExf7A+gSs8JY6Bs25PkBJLSkskM0n53L7j Ju6egteINuuAxB10wAujofYeB9m+rck= From: Roman Gushchin <roman.gushchin@linux.dev> To: Andrew Morton <akpm@linux-foundation.org> Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner <hannes@cmpxchg.org>, Michal Hocko <mhocko@kernel.org>, Shakeel@vger.kernel.org, Butt@vger.kernel.org, shakeelb@google.com, Muchun Song <muchun.song@linux.dev>, Dennis Zhou <dennis@kernel.org>, David Rientjes <rientjes@google.com>, Vlastimil Babka <vbabka@suse.cz>, Naresh Kamboju <naresh.kamboju@linaro.org>, Roman Gushchin <roman.gushchin@linux.dev> Subject: [PATCH v3 0/5] mm: improve performance of accounted kernel memory allocations Date: Mon, 16 Oct 2023 15:18:55 -0700 Message-ID: <20231016221900.4031141-1-roman.gushchin@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 16 Oct 2023 15:19:29 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779952276238911951 X-GMAIL-MSGID: 1779952276238911951 |
Series |
mm: improve performance of accounted kernel memory allocations
|
|
Message
Roman Gushchin
Oct. 16, 2023, 10:18 p.m. UTC
This patchset improves the performance of accounted kernel memory allocations
by ~30% as measured by a micro-benchmark [1]. The benchmark is very
straightforward: 1M of 64 bytes-large kmalloc() allocations.
Below are results with the disabled kernel memory accounting, the original state
and with this patchset applied.
| | Kmem disabled | Original | Patched | Delta |
|-------------+---------------+----------+---------+--------|
| User cgroup | 29764 | 84548 | 59078 | -30.0% |
| Root cgroup | 29742 | 48342 | 31501 | -34.8% |
As we can see, the patchset removes the majority of the overhead when there is
no actual accounting (a task belongs to the root memory cgroup) and almost
halves the accounting overhead otherwise.
The main idea is to get rid of unnecessary memcg to objcg conversions and switch
to a scope-based protection of objcgs, which eliminates extra operations with
objcg reference counters under a rcu read lock. More details are provided in
individual commit descriptions.
v3:
- fixed a bug spotted by Shakeel
- added some comments, per Shakeel
v2:
- fixed a bug discovered by Naresh Kamboju
- code changes asked by Johannes (added comments, open-coded bit ops)
- merged in a couple of small fixes
v1:
- made the objcg update fully lockless
- fixed !CONFIG_MMU build issues
rfc:
https://lwn.net/Articles/945722/
--
[1]:
static int memory_alloc_test(struct seq_file *m, void *v)
{
unsigned long i, j;
void **ptrs;
ktime_t start, end;
s64 delta, min_delta = LLONG_MAX;
ptrs = kvmalloc(sizeof(void *) * 1000000, GFP_KERNEL);
if (!ptrs)
return -ENOMEM;
for (j = 0; j < 100; j++) {
start = ktime_get();
for (i = 0; i < 1000000; i++)
ptrs[i] = kmalloc(64, GFP_KERNEL_ACCOUNT);
end = ktime_get();
delta = ktime_us_delta(end, start);
if (delta < min_delta)
min_delta = delta;
for (i = 0; i < 1000000; i++)
kfree(ptrs[i]);
}
kvfree(ptrs);
seq_printf(m, "%lld us\n", min_delta);
return 0;
}
--
Signed-off-by: Roman Gushchin (Cruise) <roman.gushchin@linux.dev>
Roman Gushchin (5):
mm: kmem: optimize get_obj_cgroup_from_current()
mm: kmem: add direct objcg pointer to task_struct
mm: kmem: make memcg keep a reference to the original objcg
mm: kmem: scoped objcg protection
percpu: scoped objcg protection
include/linux/memcontrol.h | 17 +++-
include/linux/sched.h | 4 +
include/linux/sched/mm.h | 4 +
mm/memcontrol.c | 204 ++++++++++++++++++++++++++++++++-----
mm/percpu.c | 8 +-
mm/slab.h | 15 +--
6 files changed, 214 insertions(+), 38 deletions(-)