From patchwork Fri Sep 29 18:00:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roman Gushchin X-Patchwork-Id: 14748 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2a8e:b0:403:3b70:6f57 with SMTP id in14csp137786vqb; Fri, 29 Sep 2023 18:18:21 -0700 (PDT) X-Google-Smtp-Source: AGHT+IESk48MMnGxm8ywVEufDtGmJiVsPLXVAhzpFAuESHDILtE2r0LQzn8GziWYV+Lk4Hnrxmmk X-Received: by 2002:a17:90a:e48:b0:274:a241:a7a8 with SMTP id p8-20020a17090a0e4800b00274a241a7a8mr8900225pja.0.1696036701168; Fri, 29 Sep 2023 18:18:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696036701; cv=none; d=google.com; s=arc-20160816; b=quhyx8E2VhXhntV9tOrg3imaWvOQ+std1GmXZt47Ms+YMvqT1/WSibZ/J3NSiFQo9Q Wg2mRlEoBrZJFjFCmcstLWEAcXbKXl53Rb/6JNFHAxcMqkXCLn3EhTPMIp2eSkDI/OvS xYBdGUNHabyQu3TOjEhTxedYutVZT4Uqss0R2E/ju6yk9I72qAfTFoKSeBZRxuWkejSz EuLCvtslI2t9YVdFxBpdOLTqqOfYQ1ksGDI1/M0zK6+h2f09H2sU3/XwqM06Je1kRtF5 AWWNX7PndxM4PUu2x6ytPDDorP1bJMr2dr1OqQ9uh7HTYkVffNyq2NwRPLLl6+uRX4Ra FWKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=6tcTuWYRALIgJf3jd0QQAQe8tlt9e/aWb2Z4sY08mGQ=; fh=GrOh8lyhpWnv6fQ7PEuyh7hTNeBQucAMexY2PVem5IM=; b=TF9HspGdNgEt8sNWQque/lJ3fdlwuUsrMFh3GW7qnfOd/I3pclIEUkKtZKQGLyy5+F rMFrhweZB0HTOC0b52x6BJGit7pCdkLnbweyBQI4AuMxUBDb0jWSnCELNTDDIH/uHp9x fFgFIBaO6fS3kiazLCRHNJeYP8vx/4QKFziAD5k+jXFqCZ8v/qMrpiAlzB+C1Ww7/r5n gfGcsBV/1Q7N6KqtuDMXBS3lRJlZELr5BKIhS9Cff6g5R3+C2NYjf+sch9U13szsh0ye mPqQafYX7N1dz3goH2LSekekHmW6G0fhzlwF/F63DsaXF0UvfCbaPJ/cLrIHT9jjJrCy R+Ag== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=SkOuMpvS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id pc5-20020a17090b3b8500b0027740943ec2si2755326pjb.158.2023.09.29.18.18.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 29 Sep 2023 18:18:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=SkOuMpvS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id D691080ED9A7; Fri, 29 Sep 2023 11:01:58 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233353AbjI2SBV (ORCPT + 19 others); Fri, 29 Sep 2023 14:01:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232748AbjI2SBT (ORCPT ); Fri, 29 Sep 2023 14:01:19 -0400 Received: from out-195.mta0.migadu.com (out-195.mta0.migadu.com [91.218.175.195]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B6E119F for ; Fri, 29 Sep 2023 11:01:17 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1696010475; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=6tcTuWYRALIgJf3jd0QQAQe8tlt9e/aWb2Z4sY08mGQ=; b=SkOuMpvSIJQsX1dDkYHCau3cvj0O3IYvhitbxnrKAG2QyDLxJyA2P4OufwQXTxcFl3Nqf1 hBwl2Nq2rZ+L+tq8JEqSRm6O9MoulQjBqZEZYio1AlRAkoXBirXAhyCeetKPYbHVW7NG0V EW1XF9qCTWMH8bocXcrlnOOLzPABKJM= From: Roman Gushchin To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Dennis Zhou , Andrew Morton , David Rientjes , Vlastimil Babka , Roman Gushchin Subject: [PATCH v1 0/5] mm: improve performance of accounted kernel memory allocations Date: Fri, 29 Sep 2023 11:00:50 -0700 Message-ID: <20230929180056.1122002-1-roman.gushchin@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 29 Sep 2023 11:01:58 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778423379719696569 X-GMAIL-MSGID: 1778423379719696569 This patchset improves the performance of accounted kernel memory allocations by ~30% as measured by a micro-benchmark [1]. The benchmark is very straightforward: 1M of 64 bytes-large kmalloc() allocations. Below are results with the disabled kernel memory accounting, the original state and with this patchset applied. | | Kmem disabled | Original | Patched | Delta | |-------------+---------------+----------+---------+--------| | User cgroup | 29764 | 84548 | 59078 | -30.0% | | Root cgroup | 29742 | 48342 | 31501 | -34.8% | As we can see, the patchset removes the majority of the overhead when there is no actual accounting (a task belongs to the root memory cgroup) and almost halves the accounting overhead otherwise. The main idea is to get rid of unnecessary memcg to objcg conversions and switch to a scope-based protection of objcgs, which eliminates extra operations with objcg reference counters under a rcu read lock. More details are provided in individual commit descriptions. v1: - made the objcg update fully lockless - fixed !CONFIG_MMU build issues rfc: https://lwn.net/Articles/945722/ --- [1]: static int memory_alloc_test(struct seq_file *m, void *v) { unsigned long i, j; void **ptrs; ktime_t start, end; s64 delta, min_delta = LLONG_MAX; ptrs = kvmalloc(sizeof(void *) * 1000000, GFP_KERNEL); if (!ptrs) return -ENOMEM; for (j = 0; j < 100; j++) { start = ktime_get(); for (i = 0; i < 1000000; i++) ptrs[i] = kmalloc(64, GFP_KERNEL_ACCOUNT); end = ktime_get(); delta = ktime_us_delta(end, start); if (delta < min_delta) min_delta = delta; for (i = 0; i < 1000000; i++) kfree(ptrs[i]); } kvfree(ptrs); seq_printf(m, "%lld us\n", min_delta); return 0; } -- Signed-off-by: Roman Gushchin (Cruise) Roman Gushchin (5): mm: kmem: optimize get_obj_cgroup_from_current() mm: kmem: add direct objcg pointer to task_struct mm: kmem: make memcg keep a reference to the original objcg mm: kmem: scoped objcg protection percpu: scoped objcg protection include/linux/memcontrol.h | 24 ++++- include/linux/sched.h | 4 + mm/memcontrol.c | 184 ++++++++++++++++++++++++++++++++----- mm/percpu.c | 8 +- mm/slab.h | 10 +- 5 files changed, 192 insertions(+), 38 deletions(-)