Message ID | 20230929180056.1122002-1-roman.gushchin@linux.dev |
---|---|
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2a8e:b0:403:3b70:6f57 with SMTP id in14csp137786vqb; Fri, 29 Sep 2023 18:18:21 -0700 (PDT) X-Google-Smtp-Source: AGHT+IESk48MMnGxm8ywVEufDtGmJiVsPLXVAhzpFAuESHDILtE2r0LQzn8GziWYV+Lk4Hnrxmmk X-Received: by 2002:a17:90a:e48:b0:274:a241:a7a8 with SMTP id p8-20020a17090a0e4800b00274a241a7a8mr8900225pja.0.1696036701168; Fri, 29 Sep 2023 18:18:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696036701; cv=none; d=google.com; s=arc-20160816; b=quhyx8E2VhXhntV9tOrg3imaWvOQ+std1GmXZt47Ms+YMvqT1/WSibZ/J3NSiFQo9Q Wg2mRlEoBrZJFjFCmcstLWEAcXbKXl53Rb/6JNFHAxcMqkXCLn3EhTPMIp2eSkDI/OvS xYBdGUNHabyQu3TOjEhTxedYutVZT4Uqss0R2E/ju6yk9I72qAfTFoKSeBZRxuWkejSz EuLCvtslI2t9YVdFxBpdOLTqqOfYQ1ksGDI1/M0zK6+h2f09H2sU3/XwqM06Je1kRtF5 AWWNX7PndxM4PUu2x6ytPDDorP1bJMr2dr1OqQ9uh7HTYkVffNyq2NwRPLLl6+uRX4Ra FWKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=6tcTuWYRALIgJf3jd0QQAQe8tlt9e/aWb2Z4sY08mGQ=; fh=GrOh8lyhpWnv6fQ7PEuyh7hTNeBQucAMexY2PVem5IM=; b=TF9HspGdNgEt8sNWQque/lJ3fdlwuUsrMFh3GW7qnfOd/I3pclIEUkKtZKQGLyy5+F rMFrhweZB0HTOC0b52x6BJGit7pCdkLnbweyBQI4AuMxUBDb0jWSnCELNTDDIH/uHp9x fFgFIBaO6fS3kiazLCRHNJeYP8vx/4QKFziAD5k+jXFqCZ8v/qMrpiAlzB+C1Ww7/r5n gfGcsBV/1Q7N6KqtuDMXBS3lRJlZELr5BKIhS9Cff6g5R3+C2NYjf+sch9U13szsh0ye mPqQafYX7N1dz3goH2LSekekHmW6G0fhzlwF/F63DsaXF0UvfCbaPJ/cLrIHT9jjJrCy R+Ag== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=SkOuMpvS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id pc5-20020a17090b3b8500b0027740943ec2si2755326pjb.158.2023.09.29.18.18.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Sep 2023 18:18:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=SkOuMpvS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id D691080ED9A7; Fri, 29 Sep 2023 11:01:58 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233353AbjI2SBV (ORCPT <rfc822;pwkd43@gmail.com> + 19 others); Fri, 29 Sep 2023 14:01:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232748AbjI2SBT (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Fri, 29 Sep 2023 14:01:19 -0400 Received: from out-195.mta0.migadu.com (out-195.mta0.migadu.com [91.218.175.195]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B6E119F for <linux-kernel@vger.kernel.org>; Fri, 29 Sep 2023 11:01:17 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1696010475; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=6tcTuWYRALIgJf3jd0QQAQe8tlt9e/aWb2Z4sY08mGQ=; b=SkOuMpvSIJQsX1dDkYHCau3cvj0O3IYvhitbxnrKAG2QyDLxJyA2P4OufwQXTxcFl3Nqf1 hBwl2Nq2rZ+L+tq8JEqSRm6O9MoulQjBqZEZYio1AlRAkoXBirXAhyCeetKPYbHVW7NG0V EW1XF9qCTWMH8bocXcrlnOOLzPABKJM= From: Roman Gushchin <roman.gushchin@linux.dev> To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner <hannes@cmpxchg.org>, Michal Hocko <mhocko@kernel.org>, Shakeel Butt <shakeelb@google.com>, Muchun Song <muchun.song@linux.dev>, Dennis Zhou <dennis@kernel.org>, Andrew Morton <akpm@linux-foundation.org>, David Rientjes <rientjes@google.com>, Vlastimil Babka <vbabka@suse.cz>, Roman Gushchin <roman.gushchin@linux.dev> Subject: [PATCH v1 0/5] mm: improve performance of accounted kernel memory allocations Date: Fri, 29 Sep 2023 11:00:50 -0700 Message-ID: <20230929180056.1122002-1-roman.gushchin@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 29 Sep 2023 11:01:58 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778423379719696569 X-GMAIL-MSGID: 1778423379719696569 |
Series |
mm: improve performance of accounted kernel memory allocations
|
|
Message
Roman Gushchin
Sept. 29, 2023, 6 p.m. UTC
This patchset improves the performance of accounted kernel memory allocations
by ~30% as measured by a micro-benchmark [1]. The benchmark is very
straightforward: 1M of 64 bytes-large kmalloc() allocations.
Below are results with the disabled kernel memory accounting, the original state
and with this patchset applied.
| | Kmem disabled | Original | Patched | Delta |
|-------------+---------------+----------+---------+--------|
| User cgroup | 29764 | 84548 | 59078 | -30.0% |
| Root cgroup | 29742 | 48342 | 31501 | -34.8% |
As we can see, the patchset removes the majority of the overhead when there is
no actual accounting (a task belongs to the root memory cgroup) and almost
halves the accounting overhead otherwise.
The main idea is to get rid of unnecessary memcg to objcg conversions and switch
to a scope-based protection of objcgs, which eliminates extra operations with
objcg reference counters under a rcu read lock. More details are provided in
individual commit descriptions.
v1:
- made the objcg update fully lockless
- fixed !CONFIG_MMU build issues
rfc:
https://lwn.net/Articles/945722/
--
[1]:
static int memory_alloc_test(struct seq_file *m, void *v)
{
unsigned long i, j;
void **ptrs;
ktime_t start, end;
s64 delta, min_delta = LLONG_MAX;
ptrs = kvmalloc(sizeof(void *) * 1000000, GFP_KERNEL);
if (!ptrs)
return -ENOMEM;
for (j = 0; j < 100; j++) {
start = ktime_get();
for (i = 0; i < 1000000; i++)
ptrs[i] = kmalloc(64, GFP_KERNEL_ACCOUNT);
end = ktime_get();
delta = ktime_us_delta(end, start);
if (delta < min_delta)
min_delta = delta;
for (i = 0; i < 1000000; i++)
kfree(ptrs[i]);
}
kvfree(ptrs);
seq_printf(m, "%lld us\n", min_delta);
return 0;
}
--
Signed-off-by: Roman Gushchin (Cruise) <roman.gushchin@linux.dev>
Roman Gushchin (5):
mm: kmem: optimize get_obj_cgroup_from_current()
mm: kmem: add direct objcg pointer to task_struct
mm: kmem: make memcg keep a reference to the original objcg
mm: kmem: scoped objcg protection
percpu: scoped objcg protection
include/linux/memcontrol.h | 24 ++++-
include/linux/sched.h | 4 +
mm/memcontrol.c | 184 ++++++++++++++++++++++++++++++++-----
mm/percpu.c | 8 +-
mm/slab.h | 10 +-
5 files changed, 192 insertions(+), 38 deletions(-)
Comments
On Fri, Sep 29, 2023 at 11:00:50AM -0700, Roman Gushchin <roman.gushchin@linux.dev> wrote: > This patchset improves the performance of accounted kernel memory allocations > by ~30% as measured by a micro-benchmark [1]. The benchmark is very > straightforward: 1M of 64 bytes-large kmalloc() allocations. Nice. Have you tried how these +34% compose with -34% reported way back [1] when file lock accounting was added (because your benchmark and lock1 sound quite similar)? (BTW Is that your motivation (too)?) Thanks, Michal [1] https://lore.kernel.org/r/20210907150757.GE17617@xsang-OptiPlex-9020/
On Wed, Oct 04, 2023 at 08:32:39PM +0200, Michal Koutný wrote: > On Fri, Sep 29, 2023 at 11:00:50AM -0700, Roman Gushchin <roman.gushchin@linux.dev> wrote: > > This patchset improves the performance of accounted kernel memory allocations > > by ~30% as measured by a micro-benchmark [1]. The benchmark is very > > straightforward: 1M of 64 bytes-large kmalloc() allocations. > > Nice. Thanks! > Have you tried how these +34% compose with -34% reported way back [1] > when file lock accounting was added (because your benchmark and lock1 > sound quite similar)? No, I haven't. I'm kindly waiting for an automatic report here :) But if someone can run these tests manually, I'll appreciate it a lot. > (BTW Is that your motivation (too)?) Not really, it was on my todo list for a long time and I just got some spare cycles to figure out missing parts (mostly around targeted/remote charging). Also plan to try similar approach to speed up generic memcg charging. Thanks!