From patchwork Wed Sep 27 15:08:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 14615 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp2952087vqu; Wed, 27 Sep 2023 16:04:57 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEjgdQtrlMIQZqPg+Og/BPmmo380vNn/D0GdUHvpr3cT6Awc3TiJlK2zM/OC0xl/bg09y1P X-Received: by 2002:a25:214:0:b0:d78:5d61:ddee with SMTP id 20-20020a250214000000b00d785d61ddeemr3352848ybc.30.1695855896864; Wed, 27 Sep 2023 16:04:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695855896; cv=none; d=google.com; s=arc-20160816; b=YfsEyZipjXZaJAvARLM7qSUOHdmQuzbAthD9/Lj61v1CQ8JMp087cAQIZhzklqYcZq 1Gsaxn3x/adKw5HVq/+9rzQJcktwstfQ8YTXjOOrWKibIrL+aO64Du41woiTrYTv6eTJ 9UnjRwSPXdZH27v+uP+u5dYglf3jt5vXuiNneo+a1k/NHAF46iLXaedqTiSJbvmNsaht 7Gzzt6wQr6w8qmyx9H5xKu1jdkAkzdD6QpyRqXGSPLsarBmMGtxJhU6uXyuy5Ygg6iyw MKi/bI+0NvQ6WgEYS7FZV2L4vWXRqmxc7UnAxzep0fjzKHwvFtHYfpSnRGGh+j1Cnzt5 TtnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=Mx7CtH6cHJrUFBld37HGiK3WgeHY8tZGKc2c0Drcq2M=; fh=J+ThODwbvpIaijJ/a/gIBgC20RizoNlQ1sjxEf1nY4o=; b=VHdc3/JeKyAz1vQVoKLWldRS6JKJHTQA5czwoBo+h2ulgLVlJU8VSe1lAV0yDO2Sq7 Pdo5AHtj8q1gwhisp90PEtEDpEmxNcLpexxPGs5K+EH/Ls5Hg+tKJ911H1rCReNy/dTw DNlXXQIlOO7bK989YgzwpXWCmzWk9qA86d+Bd7RFjeOcu5jj9uDB62LvyJOS7Honilnt bGOUgqonZQO1V09gvpdMT1DnGJ461yZsQ++1UxnjYjSV1Gc6y5WLqA0SotMDlXB2ecPi CTAbQ0m6gbZw7Tgzi1C7WoAWG2VjOvFQO0OQBGWdPfKE9woKeyJUM28Br6expBYlQe0S cSJA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=aLkDt9Aw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id cn4-20020a056a020a8400b00578acf1e8a0si5361099pgb.471.2023.09.27.16.04.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Sep 2023 16:04:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=aLkDt9Aw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 5BBBD8051BAC; Wed, 27 Sep 2023 08:09:24 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232340AbjI0PJA (ORCPT + 19 others); Wed, 27 Sep 2023 11:09:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58876 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232332AbjI0PI6 (ORCPT ); Wed, 27 Sep 2023 11:08:58 -0400 Received: from out-197.mta1.migadu.com (out-197.mta1.migadu.com [IPv6:2001:41d0:203:375::c5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 16DEB12A for ; Wed, 27 Sep 2023 08:08:57 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1695827332; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=Mx7CtH6cHJrUFBld37HGiK3WgeHY8tZGKc2c0Drcq2M=; b=aLkDt9Aw9FniFt3rZ02X3yb2uiZuG/fL/+LQZnLtgdENgBt2TpkfsBmplfKZfgORJ+Car3 FEM2u00QMiFqgQB6NVNoz46iFA+hTFjpudzfEev+erU6A6CGIannFcgCjhBHXP22g1XPGb mkbfVuaA4mWkfj14/0wiM4V+WP8fH+g= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , Roman Gushchin Subject: [PATCH rfc 0/5] mm: improve performance of kernel memory accounting Date: Wed, 27 Sep 2023 08:08:27 -0700 Message-ID: <20230927150832.335132-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 27 Sep 2023 08:09:24 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778233792748234040 X-GMAIL-MSGID: 1778233792748234040 This patchset improves the performance of the kernel memory accounting by ~30% as measured by a micro-benchmark [1]. The benchmark is very straightforward: 1M of 64 bytes-large kmalloc() allocations. Below are results with the disabled kernel memory accounting, the original state and with this patchset applied. | | Kmem disabled | Original | Patched | Delta | |-------------+---------------+----------+---------+--------| | User cgroup | 29764 | 84435 | 59385 | -29.6% | | Root cgroup | 29742 | 48425 | 31573 | -34.8% | As we can see, the patchset removes the majority of the overhead when there is no actual accounting (a task belongs to the root memory cgroup) and almost halves the accounting overhead. Overall it improves the speed of accounted allocations by ~30%. The main idea is to get rid of unnecessary memcg->objcg conversions and switch to a scope-based protection of objcgs, which eliminates extra operations with objcg reference counters under a rcu read lock. More details are provided in individual commit descriptions. --- [1]: static int memory_alloc_test(struct seq_file *m, void *v) { unsigned long i, j; void **ptrs; ktime_t start, end; s64 delta, min_delta = LLONG_MAX; ptrs = kvmalloc(sizeof(void *) * 1000000, GFP_KERNEL); if (!ptrs) return -ENOMEM; for (j = 0; j < 100; j++) { start = ktime_get(); for (i = 0; i < 1000000; i++) ptrs[i] = kmalloc(64, GFP_KERNEL_ACCOUNT); end = ktime_get(); delta = ktime_us_delta(end, start); if (delta < min_delta) min_delta = delta; for (i = 0; i < 1000000; i++) kfree(ptrs[i]); } kvfree(ptrs); seq_printf(m, "%lld us\n", min_delta); return 0; } -- Signed-off-by: Roman Gushchin (Cruise) Roman Gushchin (5): mm: kmem: optimize get_obj_cgroup_from_current() mm: kmem: add direct objcg pointer to task_struct mm: kmem: make memcg keep a reference to the original objcg mm: kmem: scoped objcg protection percpu: scoped objcg protection include/linux/memcontrol.h | 24 ++++- include/linux/sched.h | 4 + mm/memcontrol.c | 178 ++++++++++++++++++++++++++++++++----- mm/percpu.c | 8 +- mm/slab.h | 10 +-- 5 files changed, 187 insertions(+), 37 deletions(-)