From patchwork Mon Apr 17 11:02:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 84172 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2038253vqo; Mon, 17 Apr 2023 04:13:19 -0700 (PDT) X-Google-Smtp-Source: AKy350YQoG6hB+218GiLjO6Nhu6JWQG8BYBaVBqorEXITIUy9IV6UR/PGRCDo3PYfHiftEhSyhsP X-Received: by 2002:a05:6a20:3d81:b0:ef:b575:e55b with SMTP id s1-20020a056a203d8100b000efb575e55bmr4490883pzi.55.1681729999011; Mon, 17 Apr 2023 04:13:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1681729998; cv=none; d=google.com; s=arc-20160816; b=R1tBiJ70ZhKFjTU+Vj20UV1W3RdtM9HgPtJG9kI9d8ERqhARMIcP690L8RQNWnf0ld Cz2LUM3Z3hDIU+Pl8XGCPJnNAigZhQJvXCO2nFiAXrQj2+n4DES58fX4rr7X9XFGzlWL SCbCi7gl/igHpvxQPurKcSe77gLyBjOfWWx5XhMPuVzlxoMgUtguqGEoCVNFxBqwpVjl vTggf8n5gXXUDRm9eAlJbYICijlNlDqRisJSL6Q4grHQ4yqwW1YBjoNpma8cVz2q7nBs gHP+4/X/IlZaCOXY5MFLc3EbJCvQy21tYnnb1pkgwJwX2eqZwDBk1jVWUO+Dc++me45m n+tA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=AXWxF7DmGQlS+I9+P3uaxt1m3XBosotbfe+681ULx84=; b=QSP5HumNYX6tFOFD0abrH2L0tJmIKeRr0hBqWBiV7XW8rOHmikIs0U5shuPrfszDeX dQVm5M83LPX9NB5RwdyzSedMeJFCPBypj2VgyLrYH7Utt/1wG1q70dLK4RGZowxbNVe1 WpWWz35iP5mYxrRAyxiEMTv271CGw6F/5L/0Y6Yzz9zS5ibCcWKyz5WR5L/suZhF8XPs DtMvyFrr5/ByEtyZ50RwCDsYAbdYuGejQlABJfJ6RiGILTcOWefyhcgFGVBxzcDIhlJV CHeeh7Fb8Pt4kGGMwqBXhfvCzqS4rn5PDvbLLGdCPP8ArUXUJs88o6rIo154Pe+rOYxh cykA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=MzuSVH7Z; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u63-20020a638542000000b00513c817a392si9960641pgd.405.2023.04.17.04.13.06; Mon, 17 Apr 2023 04:13:18 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=MzuSVH7Z; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230414AbjDQLGg (ORCPT + 99 others); Mon, 17 Apr 2023 07:06:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33514 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229959AbjDQLGc (ORCPT ); Mon, 17 Apr 2023 07:06:32 -0400 Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B343E4EF2 for ; Mon, 17 Apr 2023 04:05:32 -0700 (PDT) Received: by mail-pl1-x62e.google.com with SMTP id i8so16232172plt.10 for ; Mon, 17 Apr 2023 04:05:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1681729435; x=1684321435; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=AXWxF7DmGQlS+I9+P3uaxt1m3XBosotbfe+681ULx84=; b=MzuSVH7Zo2sPbGWOQ47oeP4xLii2+aOg/66pgyJLKCz+X/rFQCZEZAlJXuDiRPX63L Cux+FDAbL2z7jkjdIeQ640L8J3LRvmFN5d/WR9Ny+AaK6EMbcWe5MWSCRdLkCXf100Ae 4S6Af4C5MkMixqMRH1kFCqPRf92TDYYP6Bans= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681729435; x=1684321435; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=AXWxF7DmGQlS+I9+P3uaxt1m3XBosotbfe+681ULx84=; b=haqPCf0VbV4J1YC5EFvts0UEGGS8y9h+mQvSZnV2q4faspjAReYocU8MEn7OyIwRo+ umV+lLB/iJI5N9XLqwnm+Yjkju2DqtVd+R4gyOD0cV1ysNDkpjMgYzdlgt5GnR/VClfJ IhwqwUK9SIpQv3OkhsKHQ5Wi7hLFE9oJHDG5aM53vA8hdJkj3sUHBPPbqCVec87ecbd5 k+tWiZE+CcsfEVd3rMt0mFXFD0qDreeYljUzudMcJtotBo2lCR8kUOCsD6kySB4whZg5 AJrczHNvEUVCQ6EY4tpWc0qMy4wTsPP6fJrFME3TZrteo91+P52fyOlzq22NPj7dBImW t1aw== X-Gm-Message-State: AAQBX9cGP9hxTcAEgM0aG3HjQFkEr3d5bcbkamNDDRoTc6SOeY9eZmqH CYjBL64mRcz+lY9BQDdoCbEyfw== X-Received: by 2002:a17:902:ac88:b0:1a6:5493:a3e8 with SMTP id h8-20020a170902ac8800b001a65493a3e8mr11813926plr.41.1681729435548; Mon, 17 Apr 2023 04:03:55 -0700 (PDT) Received: from tigerii.tok.corp.google.com ([2401:fa00:8f:203:97ac:e7c:d412:5251]) by smtp.gmail.com with ESMTPSA id c6-20020a170902aa4600b001a221d14179sm7258666plr.302.2023.04.17.04.03.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Apr 2023 04:03:55 -0700 (PDT) From: Sergey Senozhatsky To: Minchan Kim , Andrew Morton Cc: Yosry Ahmed , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Sergey Senozhatsky Subject: [PATCH] zsmalloc: allow only one active pool compaction context Date: Mon, 17 Apr 2023 20:02:59 +0900 Message-ID: <20230417110259.1737315-1-senozhatsky@chromium.org> X-Mailer: git-send-email 2.40.0.634.g4ca3ef3211-goog MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763421715384510599?= X-GMAIL-MSGID: =?utf-8?q?1763421715384510599?= zsmalloc pool can be compacted concurrently by many contexts, e.g. cc1 handle_mm_fault() do_anonymous_page() __alloc_pages_slowpath() try_to_free_pages() do_try_to_free_pages( lru_gen_shrink_node() shrink_slab() do_shrink_slab() zs_shrinker_scan() zs_compact() This creates unnecessary contention as all those processes compete for access to the same classes. A single compaction process is enough. Moreover contention that is created by multiple compaction processes impact other zsmalloc functions, e.g. zs_malloc(), since zsmalloc uses "global" pool->lock to synchronize access to pool. Introduce pool compaction mutex and permit only one compaction context at a time. /proc/lock-stat after make -j$((`nproc`+1)) linux kernel for &pool->lock#3: Base Patched -------------------------------------- con-bounces 9797655 8125860 contentions 11131185 9242153 waittime-min 0.09 0.10 waittime-max 4171695.76 3926258.74 waittime-total 506197629.16 417061026.20 waittime-avg 45.48 45.13 acq-bounces 13809103 11383480 acquisitions 21145155 18049364 holdtime-min 0.06 0.07 holdtime-max 7379928.80 3926274.89 holdtime-total 46273950.89 37279624.53 holdtime-avg 2.19 2.07 Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index cc81dfba05a0..0e036ec56c3c 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -264,6 +264,7 @@ struct zs_pool { struct work_struct free_work; #endif spinlock_t lock; + struct mutex compact_lock; }; struct zspage { @@ -2274,6 +2275,9 @@ unsigned long zs_compact(struct zs_pool *pool) struct size_class *class; unsigned long pages_freed = 0; + if (!mutex_trylock(&pool->compact_lock)) + return 0; + for (i = ZS_SIZE_CLASSES - 1; i >= 0; i--) { class = pool->size_class[i]; if (class->index != i) @@ -2281,6 +2285,7 @@ unsigned long zs_compact(struct zs_pool *pool) pages_freed += __zs_compact(pool, class); } atomic_long_add(pages_freed, &pool->stats.pages_compacted); + mutex_unlock(&pool->compact_lock); return pages_freed; } @@ -2388,6 +2393,7 @@ struct zs_pool *zs_create_pool(const char *name) init_deferred_free(pool); spin_lock_init(&pool->lock); + mutex_init(&pool->compact_lock); pool->name = kstrdup(name, GFP_KERNEL); if (!pool->name)