Message ID | 20231010000929.450702-1-roman.gushchin@linux.dev |
---|---|
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp2190633vqo; Mon, 9 Oct 2023 17:10:07 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEKsjyPJuB2lND+kFSy8rIWCBaRXFQzt2QdC2arqWYPV4ObmPtZtc6Jy7NAoLLFFt4l7mJ6 X-Received: by 2002:a05:6808:3bc:b0:3a4:225d:82c0 with SMTP id n28-20020a05680803bc00b003a4225d82c0mr18353677oie.31.1696896607486; Mon, 09 Oct 2023 17:10:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696896607; cv=none; d=google.com; s=arc-20160816; b=FphTPH7lv2NvuTHxDoszWe9o61vP53bGaQrh+pNHIpN5fj9XptOFCpGLHCxjZWTnF/ nf34MckEoufRrgT5n8VuTUS+bSkW5ezLaOY9B63DsfMh5Q3xrisnMqzwEWyKZ1ITGxsi ZO5okrVK49D+bHc95I+evgIX0fTbzh8Qs1MSCXzMnpp4AO0X7l2XITnWUbjIUrvF7FdA bN/6qzfNSQPP5ELxqc6s2HIsHWiBI0+SMFFj8Jurfn4yzzixuVGKjfDNVDbuzazDNq9p D0dsHI14P0zr4mGvai65INWXKK3Qlin52OQOg08o3ON4yw9U/X51xTPxJNvWjM4X5S1j SafA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=Zu2srZGRza0i12D/f9sOSWHgmsyy71BOzkS7lMMPjis=; fh=eIV/wktpjV+EeUDLeXhmhAmcuYEtVvgOWNik/7Ld4iQ=; b=g1mll8EtyFTHOfev8eN5ILNkR/t+KG4E3mGi42EuzBKheGZbDZpVsCZlNdpzh1tIx1 mbSZ63VQudyF4ZSWXFhOWlPC66buBoCaJ3BEqYT7aNBMoDy53kQ2IfQ0ijmYoka1WLlc wQXTQfBXB7gyImkWPWV7iTnctsDMkifT+x6w9SIthRKvSLgAEjIRQVW+gQ1Dv6VuvySt yEnRZxNFopUVVU8ha86ZBwK0sHf60Kj/l5QlYx5civHvwRg6TX2pmGSQ/s4jhvVu1jh2 PrpbhiOdBXa8MtbuVJpv2JL8j6dXtgFxrLs25Qd91UFipFFD5ScjfDGI2xJmL9I4dYwO 3iWA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=DnHfbVma; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id z17-20020a631911000000b00577f4d736bbsi10168763pgl.373.2023.10.09.17.10.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 17:10:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=DnHfbVma; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id BAAE380FC73A; Mon, 9 Oct 2023 17:10:06 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1379142AbjJJAKC (ORCPT <rfc822;makky5685@gmail.com> + 18 others); Mon, 9 Oct 2023 20:10:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42680 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1379138AbjJJAKB (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Mon, 9 Oct 2023 20:10:01 -0400 Received: from out-210.mta0.migadu.com (out-210.mta0.migadu.com [91.218.175.210]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12F4EAC for <linux-kernel@vger.kernel.org>; Mon, 9 Oct 2023 17:09:59 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1696896596; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=Zu2srZGRza0i12D/f9sOSWHgmsyy71BOzkS7lMMPjis=; b=DnHfbVmaqpEspZYM0pD00NCFUZGLSVOOtcdBPfHVbSOARxf2o1fJ/qt7mqIbtSIPUqM1Xs AlBwkqXu83Dk3Ms55eJaUyr5vrS7cGAm3R0CItbKKNrFTRdhXFx9Lg2iXrue1HccXLZJDr mkrBXCUahp/34s8KWx+jyweMYmAnjR8= From: Roman Gushchin <roman.gushchin@linux.dev> To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner <hannes@cmpxchg.org>, Michal Hocko <mhocko@kernel.org>, Shakeel Butt <shakeelb@google.com>, Muchun Song <muchun.song@linux.dev>, Dennis Zhou <dennis@kernel.org>, Andrew Morton <akpm@linux-foundation.org>, David Rientjes <rientjes@google.com>, Vlastimil Babka <vbabka@suse.cz>, Naresh Kamboju <naresh.kamboju@linaro.org>, Roman Gushchin <roman.gushchin@linux.dev> Subject: [PATCH v2 0/5] mm: improve performance of accounted kernel memory allocations Date: Mon, 9 Oct 2023 17:09:24 -0700 Message-ID: <20231010000929.450702-1-roman.gushchin@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 09 Oct 2023 17:10:06 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779325057252499008 X-GMAIL-MSGID: 1779325057252499008 |
Series |
mm: improve performance of accounted kernel memory allocations
|
|
Message
Roman Gushchin
Oct. 10, 2023, 12:09 a.m. UTC
This patchset improves the performance of accounted kernel memory allocations
by ~30% as measured by a micro-benchmark [1]. The benchmark is very
straightforward: 1M of 64 bytes-large kmalloc() allocations.
Below are results with the disabled kernel memory accounting, the original state
and with this patchset applied.
| | Kmem disabled | Original | Patched | Delta |
|-------------+---------------+----------+---------+--------|
| User cgroup | 29764 | 84548 | 59078 | -30.0% |
| Root cgroup | 29742 | 48342 | 31501 | -34.8% |
As we can see, the patchset removes the majority of the overhead when there is
no actual accounting (a task belongs to the root memory cgroup) and almost
halves the accounting overhead otherwise.
The main idea is to get rid of unnecessary memcg to objcg conversions and switch
to a scope-based protection of objcgs, which eliminates extra operations with
objcg reference counters under a rcu read lock. More details are provided in
individual commit descriptions.
v2:
- fixed a bug discovered by Naresh Kamboju
- code changes asked by Johannes (added comments, open-coded bit ops)
- merged in a couple of small fixes
v1:
- made the objcg update fully lockless
- fixed !CONFIG_MMU build issues
rfc:
https://lwn.net/Articles/945722/
--
[1]:
static int memory_alloc_test(struct seq_file *m, void *v)
{
unsigned long i, j;
void **ptrs;
ktime_t start, end;
s64 delta, min_delta = LLONG_MAX;
ptrs = kvmalloc(sizeof(void *) * 1000000, GFP_KERNEL);
if (!ptrs)
return -ENOMEM;
for (j = 0; j < 100; j++) {
start = ktime_get();
for (i = 0; i < 1000000; i++)
ptrs[i] = kmalloc(64, GFP_KERNEL_ACCOUNT);
end = ktime_get();
delta = ktime_us_delta(end, start);
if (delta < min_delta)
min_delta = delta;
for (i = 0; i < 1000000; i++)
kfree(ptrs[i]);
}
kvfree(ptrs);
seq_printf(m, "%lld us\n", min_delta);
return 0;
}
--
Signed-off-by: Roman Gushchin (Cruise) <roman.gushchin@linux.dev>
Roman Gushchin (5):
mm: kmem: optimize get_obj_cgroup_from_current()
mm: kmem: add direct objcg pointer to task_struct
mm: kmem: make memcg keep a reference to the original objcg
mm: kmem: scoped objcg protection
percpu: scoped objcg protection
include/linux/memcontrol.h | 14 ++-
include/linux/sched.h | 4 +
mm/memcontrol.c | 204 ++++++++++++++++++++++++++++++++-----
mm/percpu.c | 8 +-
mm/slab.h | 10 +-
5 files changed, 202 insertions(+), 38 deletions(-)
Comments
Andrew, can you, please, drop v1 entirely (with all follow-up fixes) and replace it with v2? Thanks!