From patchwork Thu Oct 19 22:53:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 15643 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2010:b0:403:3b70:6f57 with SMTP id fe16csp698406vqb; Thu, 19 Oct 2023 15:54:35 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGgI7LfEFE0IDLToIeZXPhRzLAFAg2yZp5FKc0ekMKodLDFz52m0QYDusrHPjWvZ1aTzhnm X-Received: by 2002:a05:6a20:4415:b0:11f:4707:7365 with SMTP id ce21-20020a056a20441500b0011f47077365mr220180pzb.38.1697756074953; Thu, 19 Oct 2023 15:54:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697756074; cv=none; d=google.com; s=arc-20160816; b=ERlMdLKSDuww/DrmMJ5ywrxPmhrPiCRGg4KVPI4PqavgNcyl7WZ4WE1CSBfkDkGLMR H/aXPvP+GmgIiLJJSSZx6WyowHRPPkK8Artt8cA+QIJqyOqeHvC2OJhm2hf7UZezXhE1 +ECW3GU6OVlgRW2dJmSJR1GVaWm7iBY73IGiNNgnh+cZpGz6ziWLuVV66QG2JuIfk1xt hbb4uv5B844eMOyHo2mI8h5Hs2B/lnx2U6z0OHMejVn7alVA+AEd0tXYwXQjyXjtbCRi hBLt+f5mlbPDWhpS6+U8Ek9HxdPjoQnCiHsnvf8HDjKsJfDfPrvFuisokgR/fuGNmIxu Q0uA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=hC0qKkzA+IO3Kf7xsnGXiZfAqe7lXYsBdIx73dPy0OQ=; fh=0sD7u+6ZeVLabXSJ0/ta2yf8mUfUXWwh1sJBhHvFyso=; b=u+i780aoV6waSq0drcH+hWh4Ue+evsoLZbfCPqyIbUZnAVjMzipZcrGCtQ/rLs36Fx +HaW9FsL9ITdS6IXnRSE5XP+KXlpknr52HAa0FsO0PX+ujyG/OEdtkqm6rozQp9mhCw9 xtAyeNcZM0BthpOqgH9l0C/Bsj2gYONPSTkPQTM5CPhIBHmbwG4XED4Hk4Z2dCRwNTsc n5ly/0YXfi70hziaWV88KPeeX141AB/+XWLPZpUgIUtoqEebXnDT2wiTd58VUezBg0Nx /bo3mmcqHtfOGtGyrDDTMQ9I0o+wEv0/TI5CBZJ8hGd4rWaREbYHEaP+IEmYJZXN7BRE HskA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=uWIDVDNy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id by17-20020a056a02059100b005702257f32esi680212pgb.185.2023.10.19.15.54.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Oct 2023 15:54:34 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=uWIDVDNy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 123D58266398; Thu, 19 Oct 2023 15:54:29 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346695AbjJSWyR (ORCPT + 26 others); Thu, 19 Oct 2023 18:54:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346677AbjJSWyP (ORCPT ); Thu, 19 Oct 2023 18:54:15 -0400 Received: from out-191.mta0.migadu.com (out-191.mta0.migadu.com [91.218.175.191]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4CC7698 for ; Thu, 19 Oct 2023 15:54:13 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1697756051; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=hC0qKkzA+IO3Kf7xsnGXiZfAqe7lXYsBdIx73dPy0OQ=; b=uWIDVDNy/yIBGWJihzYBNfry/ngGZ8hzuQpgXFz7XZWYND5gcVp+ibHTEw7a8dFzMvky9r 36dCtnB3+Se3UWQvxRSVtmKOtErS/ef1zFuy8AVOzEl4Mgv7qpY8uMl0wtYsQq48ALCd6k 7Ly8205FxC6O2LonomBQV0KfyIggvBs= From: Roman Gushchin To: Andrew Morton Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , David Rientjes , Vlastimil Babka , Naresh Kamboju , Roman Gushchin Subject: [PATCH v5 0/6] mm: improve performance of accounted kernel memory allocations Date: Thu, 19 Oct 2023 15:53:40 -0700 Message-ID: <20231019225346.1822282-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Thu, 19 Oct 2023 15:54:29 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780226274101415650 X-GMAIL-MSGID: 1780226274101415650 This patchset improves the performance of accounted kernel memory allocations by ~30% as measured by a micro-benchmark [1]. The benchmark is very straightforward: 1M of 64 bytes-large kmalloc() allocations. Below are results with the disabled kernel memory accounting, the original state and with this patchset applied. | | Kmem disabled | Original | Patched | Delta | |-------------+---------------+----------+---------+--------| | User cgroup | 29764 | 84548 | 59078 | -30.0% | | Root cgroup | 29742 | 48342 | 31501 | -34.8% | As we can see, the patchset removes the majority of the overhead when there is no actual accounting (a task belongs to the root memory cgroup) and almost halves the accounting overhead otherwise. The main idea is to get rid of unnecessary memcg to objcg conversions and switch to a scope-based protection of objcgs, which eliminates extra operations with objcg reference counters under a rcu read lock. More details are provided in individual commit descriptions. v5: - fixed another refcnt bug spotted by Vlastimil - small refactoring of current_obj_cgroup() - added a patch for get_obj_cgroup() refactoring v4: - fixed a bug spotted by Vlastimil - cosmetic changes, per Vlastimil v3: - fixed a bug spotted by Shakeel - added some comments, per Shakeel v2: - fixed a bug discovered by Naresh Kamboju - code changes asked by Johannes (added comments, open-coded bit ops) - merged in a couple of small fixes v1: - made the objcg update fully lockless - fixed !CONFIG_MMU build issues rfc: https://lwn.net/Articles/945722/ --- [1]: static int memory_alloc_test(struct seq_file *m, void *v) { unsigned long i, j; void **ptrs; ktime_t start, end; s64 delta, min_delta = LLONG_MAX; ptrs = kvmalloc(sizeof(void *) * 1000000, GFP_KERNEL); if (!ptrs) return -ENOMEM; for (j = 0; j < 100; j++) { start = ktime_get(); for (i = 0; i < 1000000; i++) ptrs[i] = kmalloc(64, GFP_KERNEL_ACCOUNT); end = ktime_get(); delta = ktime_us_delta(end, start); if (delta < min_delta) min_delta = delta; for (i = 0; i < 1000000; i++) kfree(ptrs[i]); } kvfree(ptrs); seq_printf(m, "%lld us\n", min_delta); return 0; } -- Signed-off-by: Roman Gushchin (Cruise) Roman Gushchin (6): mm: kmem: optimize get_obj_cgroup_from_current() mm: kmem: add direct objcg pointer to task_struct mm: kmem: make memcg keep a reference to the original objcg mm: kmem: scoped objcg protection percpu: scoped objcg protection mm: kmem: reimplement get_obj_cgroup_from_current() include/linux/memcontrol.h | 28 +++++- include/linux/sched.h | 4 + include/linux/sched/mm.h | 4 + mm/memcontrol.c | 187 +++++++++++++++++++++++++++++++------ mm/percpu.c | 8 +- mm/slab.h | 15 +-- 6 files changed, 204 insertions(+), 42 deletions(-)