From patchwork Wed Oct 25 14:45:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 158141 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp1018vqb; Wed, 25 Oct 2023 07:46:52 -0700 (PDT) X-Google-Smtp-Source: AGHT+IERQJDLmWA3+lPUi7+/LlEJ7WjflWzVPHWdsdxfQYlN50YtCxB44S056CpTUE+AP9fmjVfb X-Received: by 2002:a81:4984:0:b0:579:f5c2:b16e with SMTP id w126-20020a814984000000b00579f5c2b16emr13803542ywa.31.1698245203023; Wed, 25 Oct 2023 07:46:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698245203; cv=none; d=google.com; s=arc-20160816; b=TXjzTuJSoOB/M3HKmfB7GeHvaADrgt1KhCWQFooVlq1qMUdqidBtKJ3lY50u5vv6HB lGrnUxhdAEbIQPx9tIqNNLsDaPhVcAqH3urAl/IIGR+pFh+AUwIf5yq9NhAyRhDEp88t s7+tv5IxQlQF6zIfUImFlYpD4x/kySGDSRyEc4wcJACc/b1I8dnaJekMICylmOmlubp2 D90S/r5Poc3zYQ+4Hw3TWgD72Kfo4IUgpTZVQ9a6ltrVcBCdqMImWOG2R3ctrZtuifgl oPvz/KQilGUziwN8NlV9yWk7GtKfO/K1Z5r0Fgg/GVHRJqYkcbL2HaKsXtfmgORokniz YQOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=VVek/o5yzsqrP73thiBmSPOq8G6wi+ra9xSsWuMFlN8=; fh=odHZZLn5aUrUhssH6KSiJJlFsZ7wTjfMLFlHN8avkCo=; b=QhGy7x3dfjkmOo244+4PUSJqw4uEO4XVUFUk5o7H5qv1gfn7o3AFY0uaixDKFBPSca XfjdKjxCobM4lZj1XUoT76uTrVMeyRMqJeEPL6WjiXDhUhOCwu2ygTXhw3/XV6KDkpiF DC4Du8QR8LOLAU3GEKsmFivdvpMFBpKUa0J/a7sRRWFI0C6gOiqbupEO4b4x5vVHwlqn k4jSmS/GxMvS4LUvSWOJiK087k9k38LUeFLt3es4Wz0zo94pKgh+czx2YEqV9w7BWqbS TAuGKx6lWqx2u9mFTZZ8Qr83rIVjNGfLAwWB75+DbRnMAAeqht7Rsav41FPvUlv3kWeo j8YA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id n136-20020a0dcb8e000000b005a7ddef0e41si10732325ywd.259.2023.10.25.07.46.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 07:46:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 5209C801C9C6; Wed, 25 Oct 2023 07:46:20 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343667AbjJYOqE (ORCPT + 26 others); Wed, 25 Oct 2023 10:46:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32822 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232800AbjJYOqA (ORCPT ); Wed, 25 Oct 2023 10:46:00 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 980C912F for ; Wed, 25 Oct 2023 07:45:58 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BB8E1C15; Wed, 25 Oct 2023 07:46:39 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 4EB863F64C; Wed, 25 Oct 2023 07:45:56 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , David Hildenbrand , Matthew Wilcox , Huang Ying , Gao Xiang , Yu Zhao , Yang Shi , Michal Hocko , Kefeng Wang Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags Date: Wed, 25 Oct 2023 15:45:43 +0100 Message-Id: <20231025144546.577640-2-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231025144546.577640-1-ryan.roberts@arm.com> References: <20231025144546.577640-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 25 Oct 2023 07:46:20 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780739162214520659 X-GMAIL-MSGID: 1780739162214520659 As preparation for supporting small-sized THP in the swap-out path, without first needing to split to order-0, Remove the CLUSTER_FLAG_HUGE, which, when present, always implies PMD-sized THP, which is the same as the cluster size. The only use of the flag was to determine whether a swap entry refers to a single page or a PMD-sized THP in swap_page_trans_huge_swapped(). Instead of relying on the flag, we now pass in nr_pages, which originates from the folio's number of pages. This allows the logic to work for folios of any order. The one snag is that one of the swap_page_trans_huge_swapped() call sites does not have the folio. But it was only being called there to avoid bothering to call __try_to_reclaim_swap() in some cases. __try_to_reclaim_swap() gets the folio and (via some other functions) calls swap_page_trans_huge_swapped(). So I've removed the problematic call site and believe the new logic should be equivalent. Removing CLUSTER_FLAG_HUGE also means we can remove split_swap_cluster() which used to be called during folio splitting, since split_swap_cluster()'s only job was to remove the flag. Signed-off-by: Ryan Roberts --- include/linux/swap.h | 10 ---------- mm/huge_memory.c | 3 --- mm/swapfile.c | 47 ++++++++------------------------------------ 3 files changed, 8 insertions(+), 52 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 19f30a29e1f1..a073366a227c 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -259,7 +259,6 @@ struct swap_cluster_info { }; #define CLUSTER_FLAG_FREE 1 /* This cluster is free */ #define CLUSTER_FLAG_NEXT_NULL 2 /* This cluster has no next cluster */ -#define CLUSTER_FLAG_HUGE 4 /* This cluster is backing a transparent huge page */ /* * We assign a cluster to each CPU, so each CPU can allocate swap entry from @@ -595,15 +594,6 @@ static inline int add_swap_extent(struct swap_info_struct *sis, } #endif /* CONFIG_SWAP */ -#ifdef CONFIG_THP_SWAP -extern int split_swap_cluster(swp_entry_t entry); -#else -static inline int split_swap_cluster(swp_entry_t entry) -{ - return 0; -} -#endif - #ifdef CONFIG_MEMCG static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) { diff --git a/mm/huge_memory.c b/mm/huge_memory.c index f31f02472396..b411dd4f1612 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2598,9 +2598,6 @@ static void __split_huge_page(struct page *page, struct list_head *list, shmem_uncharge(head->mapping->host, nr_dropped); remap_page(folio, nr); - if (folio_test_swapcache(folio)) - split_swap_cluster(folio->swap); - for (i = 0; i < nr; i++) { struct page *subpage = head + i; if (subpage == page) diff --git a/mm/swapfile.c b/mm/swapfile.c index e52f486834eb..b83ad77e04c0 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -342,18 +342,6 @@ static inline void cluster_set_null(struct swap_cluster_info *info) info->data = 0; } -static inline bool cluster_is_huge(struct swap_cluster_info *info) -{ - if (IS_ENABLED(CONFIG_THP_SWAP)) - return info->flags & CLUSTER_FLAG_HUGE; - return false; -} - -static inline void cluster_clear_huge(struct swap_cluster_info *info) -{ - info->flags &= ~CLUSTER_FLAG_HUGE; -} - static inline struct swap_cluster_info *lock_cluster(struct swap_info_struct *si, unsigned long offset) { @@ -1021,7 +1009,7 @@ static int swap_alloc_cluster(struct swap_info_struct *si, swp_entry_t *slot) offset = idx * SWAPFILE_CLUSTER; ci = lock_cluster(si, offset); alloc_cluster(si, idx); - cluster_set_count_flag(ci, SWAPFILE_CLUSTER, CLUSTER_FLAG_HUGE); + cluster_set_count(ci, SWAPFILE_CLUSTER); memset(si->swap_map + offset, SWAP_HAS_CACHE, SWAPFILE_CLUSTER); unlock_cluster(ci); @@ -1354,7 +1342,6 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry) ci = lock_cluster_or_swap_info(si, offset); if (size == SWAPFILE_CLUSTER) { - VM_BUG_ON(!cluster_is_huge(ci)); map = si->swap_map + offset; for (i = 0; i < SWAPFILE_CLUSTER; i++) { val = map[i]; @@ -1362,7 +1349,6 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry) if (val == SWAP_HAS_CACHE) free_entries++; } - cluster_clear_huge(ci); if (free_entries == SWAPFILE_CLUSTER) { unlock_cluster_or_swap_info(si, ci); spin_lock(&si->lock); @@ -1384,23 +1370,6 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry) unlock_cluster_or_swap_info(si, ci); } -#ifdef CONFIG_THP_SWAP -int split_swap_cluster(swp_entry_t entry) -{ - struct swap_info_struct *si; - struct swap_cluster_info *ci; - unsigned long offset = swp_offset(entry); - - si = _swap_info_get(entry); - if (!si) - return -EBUSY; - ci = lock_cluster(si, offset); - cluster_clear_huge(ci); - unlock_cluster(ci); - return 0; -} -#endif - static int swp_entry_cmp(const void *ent1, const void *ent2) { const swp_entry_t *e1 = ent1, *e2 = ent2; @@ -1508,22 +1477,23 @@ int swp_swapcount(swp_entry_t entry) } static bool swap_page_trans_huge_swapped(struct swap_info_struct *si, - swp_entry_t entry) + swp_entry_t entry, + unsigned int nr_pages) { struct swap_cluster_info *ci; unsigned char *map = si->swap_map; unsigned long roffset = swp_offset(entry); - unsigned long offset = round_down(roffset, SWAPFILE_CLUSTER); + unsigned long offset = round_down(roffset, nr_pages); int i; bool ret = false; ci = lock_cluster_or_swap_info(si, offset); - if (!ci || !cluster_is_huge(ci)) { + if (!ci || nr_pages == 1) { if (swap_count(map[roffset])) ret = true; goto unlock_out; } - for (i = 0; i < SWAPFILE_CLUSTER; i++) { + for (i = 0; i < nr_pages; i++) { if (swap_count(map[offset + i])) { ret = true; break; @@ -1545,7 +1515,7 @@ static bool folio_swapped(struct folio *folio) if (!IS_ENABLED(CONFIG_THP_SWAP) || likely(!folio_test_large(folio))) return swap_swapcount(si, entry) != 0; - return swap_page_trans_huge_swapped(si, entry); + return swap_page_trans_huge_swapped(si, entry, folio_nr_pages(folio)); } /** @@ -1606,8 +1576,7 @@ int free_swap_and_cache(swp_entry_t entry) p = _swap_info_get(entry); if (p) { count = __swap_entry_free(p, entry); - if (count == SWAP_HAS_CACHE && - !swap_page_trans_huge_swapped(p, entry)) + if (count == SWAP_HAS_CACHE) __try_to_reclaim_swap(p, swp_offset(entry), TTRS_UNMAPPED | TTRS_FULL); } From patchwork Wed Oct 25 14:45:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 158144 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp1241vqb; Wed, 25 Oct 2023 07:47:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGv1X+GfuTwxgYHLNP/AUd5pH4TWmfCM5NLmSdj6VxL7o0EeRjT2GF4m2pUhlGJtOup/xt6 X-Received: by 2002:a0d:ebd7:0:b0:59c:aea:d877 with SMTP id u206-20020a0debd7000000b0059c0aead877mr15975767ywe.40.1698245227500; Wed, 25 Oct 2023 07:47:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698245227; cv=none; d=google.com; s=arc-20160816; b=i2nnTrHtV3MHBk7rrr/iYy3z5U+oJ/UhBpy94JKAwil0ecKYEZIz4RkLg7xH80judT pKi1mKrlPpciyJxWs+zUCPFuflrSNtATU0Qr9v7Pb2IS+wJfU8BeLbYUJEuLuyRF6O3h kZfO7U8PNS9PNbRn8Qcn8KXUNxb6f/WznrzytOtKk2JS1sCVSK5yE9J3uElZKORuwkUO rbDK/x8JPAwq4cKLfC8Ss5PnUQSmXansnxxiAWRmED/fD9QMyzLd7FprD+mAgXnncwfM 51eD/u5YJNjD/kV4rrGKyzg2CwTJGPymfoFKSqjyuZH2U3FRGcWxkLOVWIVM2tQ81vUY w79A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=0Tpf03q3z+j3kVeYNfMMrhq2bga3CLxbv4cIDq7P5sE=; fh=odHZZLn5aUrUhssH6KSiJJlFsZ7wTjfMLFlHN8avkCo=; b=dO6S39INQ9/4qvbT5Q9awzjEoOcHLtdzY2Fgu72HgsdXzbEf7zmZ0nvmszdUAuHsvd 8AiqoX/VQ9NFFbIwLPzfSmkiA2lNE0rgQeXSROVtgmRnyqhc7Ko4jWnabujeZLsdMCjY AdftfIH14KCfmjmLaPuybS14QOmnSH66dTfLn5+mfPf8wEr2vSskBB/sKI4rMvOPnhsF MLshfeLyowNqGM8HXdgHJivz5Hc5WHxjH6BfWsq41Ffrof9A6TkReEonXhNjgCMDQ4mo 41xOpVDH8zGOwrkTaWZkstmmNe7eIX4IlVUlhDs+vv31tN86jhhonWlsis2YXs7Ft1Tf P5MA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id g207-20020a8152d8000000b0059bde0b55fbsi11264200ywb.24.2023.10.25.07.47.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 07:47:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 7681A80AEB3F; Wed, 25 Oct 2023 07:46:18 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343695AbjJYOqI (ORCPT + 26 others); Wed, 25 Oct 2023 10:46:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32846 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234655AbjJYOqC (ORCPT ); Wed, 25 Oct 2023 10:46:02 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id DAE91DC for ; Wed, 25 Oct 2023 07:45:59 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 30C9B1474; Wed, 25 Oct 2023 07:46:41 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 139993F7C5; Wed, 25 Oct 2023 07:45:57 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , David Hildenbrand , Matthew Wilcox , Huang Ying , Gao Xiang , Yu Zhao , Yang Shi , Michal Hocko , Kefeng Wang Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 2/4] mm: swap: Remove struct percpu_cluster Date: Wed, 25 Oct 2023 15:45:44 +0100 Message-Id: <20231025144546.577640-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231025144546.577640-1-ryan.roberts@arm.com> References: <20231025144546.577640-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 25 Oct 2023 07:46:18 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780739187546120204 X-GMAIL-MSGID: 1780739187546120204 struct percpu_cluster stores the index of cpu's current cluster and the offset of the next entry that will be allocated for the cpu. These two pieces of information are redundant because the cluster index is just (offset / SWAPFILE_CLUSTER). The only reason for explicitly keeping the cluster index is because the structure used for it also has a flag to indicate "no cluster". However this data structure also contains a spin lock, which is never used in this context, as a side effect the code copies the spinlock_t structure, which is questionable coding practice in my view. So let's clean this up and store only the next offset, and use a sentinal value (SWAP_NEXT_NULL) to indicate "no cluster". SWAP_NEXT_NULL is chosen to be 0, because 0 will never be seen legitimately; The first page in the swap file is the swap header, which is always marked bad to prevent it from being allocated as an entry. This also prevents the cluster to which it belongs being marked free, so it will never appear on the free list. This change saves 16 bytes per cpu. And given we are shortly going to extend this mechanism to be per-cpu-AND-per-order, we will end up saving 16 * 9 = 144 bytes per cpu, which adds up if you have 256 cpus in the system. Signed-off-by: Ryan Roberts --- include/linux/swap.h | 21 +++++++++++++-------- mm/swapfile.c | 43 +++++++++++++++++++------------------------ 2 files changed, 32 insertions(+), 32 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index a073366a227c..0ca8aaa098ba 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -261,14 +261,12 @@ struct swap_cluster_info { #define CLUSTER_FLAG_NEXT_NULL 2 /* This cluster has no next cluster */ /* - * We assign a cluster to each CPU, so each CPU can allocate swap entry from - * its own cluster and swapout sequentially. The purpose is to optimize swapout - * throughput. + * The first page in the swap file is the swap header, which is always marked + * bad to prevent it from being allocated as an entry. This also prevents the + * cluster to which it belongs being marked free. Therefore 0 is safe to use as + * a sentinel to indicate cpu_next is not valid in swap_info_struct. */ -struct percpu_cluster { - struct swap_cluster_info index; /* Current cluster index */ - unsigned int next; /* Likely next allocation offset */ -}; +#define SWAP_NEXT_NULL 0 struct swap_cluster_list { struct swap_cluster_info head; @@ -295,7 +293,14 @@ struct swap_info_struct { unsigned int cluster_next; /* likely index for next allocation */ unsigned int cluster_nr; /* countdown to next cluster search */ unsigned int __percpu *cluster_next_cpu; /*percpu index for next allocation */ - struct percpu_cluster __percpu *percpu_cluster; /* per cpu's swap location */ + unsigned int __percpu *cpu_next;/* + * Likely next allocation offset. We + * assign a cluster to each CPU, so each + * CPU can allocate swap entry from its + * own cluster and swapout sequentially. + * The purpose is to optimize swapout + * throughput. + */ struct rb_root swap_extent_root;/* root of the swap extent rbtree */ struct block_device *bdev; /* swap device or bdev of swap file */ struct file *swap_file; /* seldom referenced */ diff --git a/mm/swapfile.c b/mm/swapfile.c index b83ad77e04c0..617e34b8cdbe 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -591,7 +591,6 @@ static bool scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si, unsigned long offset) { - struct percpu_cluster *percpu_cluster; bool conflict; offset /= SWAPFILE_CLUSTER; @@ -602,8 +601,7 @@ scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si, if (!conflict) return false; - percpu_cluster = this_cpu_ptr(si->percpu_cluster); - cluster_set_null(&percpu_cluster->index); + *this_cpu_ptr(si->cpu_next) = SWAP_NEXT_NULL; return true; } @@ -614,16 +612,16 @@ scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si, static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, unsigned long *offset, unsigned long *scan_base) { - struct percpu_cluster *cluster; struct swap_cluster_info *ci; - unsigned long tmp, max; + unsigned int tmp, max; + unsigned int *cpu_next; new_cluster: - cluster = this_cpu_ptr(si->percpu_cluster); - if (cluster_is_null(&cluster->index)) { + cpu_next = this_cpu_ptr(si->cpu_next); + tmp = *cpu_next; + if (tmp == SWAP_NEXT_NULL) { if (!cluster_list_empty(&si->free_clusters)) { - cluster->index = si->free_clusters.head; - cluster->next = cluster_next(&cluster->index) * + tmp = cluster_next(&si->free_clusters.head) * SWAPFILE_CLUSTER; } else if (!cluster_list_empty(&si->discard_clusters)) { /* @@ -643,9 +641,8 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, * Other CPUs can use our cluster if they can't find a free cluster, * check if there is still free entry in the cluster */ - tmp = cluster->next; max = min_t(unsigned long, si->max, - (cluster_next(&cluster->index) + 1) * SWAPFILE_CLUSTER); + ALIGN_DOWN(tmp, SWAPFILE_CLUSTER) + SWAPFILE_CLUSTER); if (tmp < max) { ci = lock_cluster(si, tmp); while (tmp < max) { @@ -656,12 +653,13 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, unlock_cluster(ci); } if (tmp >= max) { - cluster_set_null(&cluster->index); + *cpu_next = SWAP_NEXT_NULL; goto new_cluster; } - cluster->next = tmp + 1; *offset = tmp; *scan_base = tmp; + tmp += 1; + *cpu_next = tmp < max ? tmp : SWAP_NEXT_NULL; return true; } @@ -2488,8 +2486,8 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) arch_swap_invalidate_area(p->type); zswap_swapoff(p->type); mutex_unlock(&swapon_mutex); - free_percpu(p->percpu_cluster); - p->percpu_cluster = NULL; + free_percpu(p->cpu_next); + p->cpu_next = NULL; free_percpu(p->cluster_next_cpu); p->cluster_next_cpu = NULL; vfree(swap_map); @@ -3073,16 +3071,13 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) for (ci = 0; ci < nr_cluster; ci++) spin_lock_init(&((cluster_info + ci)->lock)); - p->percpu_cluster = alloc_percpu(struct percpu_cluster); - if (!p->percpu_cluster) { + p->cpu_next = alloc_percpu(unsigned int); + if (!p->cpu_next) { error = -ENOMEM; goto bad_swap_unlock_inode; } - for_each_possible_cpu(cpu) { - struct percpu_cluster *cluster; - cluster = per_cpu_ptr(p->percpu_cluster, cpu); - cluster_set_null(&cluster->index); - } + for_each_possible_cpu(cpu) + per_cpu(*p->cpu_next, cpu) = SWAP_NEXT_NULL; } else { atomic_inc(&nr_rotate_swap); inced_nr_rotate_swap = true; @@ -3171,8 +3166,8 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) bad_swap_unlock_inode: inode_unlock(inode); bad_swap: - free_percpu(p->percpu_cluster); - p->percpu_cluster = NULL; + free_percpu(p->cpu_next); + p->cpu_next = NULL; free_percpu(p->cluster_next_cpu); p->cluster_next_cpu = NULL; if (inode && S_ISBLK(inode->i_mode) && p->bdev) { From patchwork Wed Oct 25 14:45:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 158145 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp1310vqb; Wed, 25 Oct 2023 07:47:23 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGkefkcbG0e3bDbx0I/kVEpXl0wHtVeNTOz4Ry4pQUMXNJ28DTa6u8AGuDuGfLM9SNdw/+1 X-Received: by 2002:a0d:f285:0:b0:5ad:4975:c860 with SMTP id b127-20020a0df285000000b005ad4975c860mr4850532ywf.39.1698245235993; Wed, 25 Oct 2023 07:47:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698245235; cv=none; d=google.com; s=arc-20160816; b=lCCfsG8+CapmYwb23MBvzk/ezJEeT/+I1VqEyHc5DBk8W7/nNbn+HR73nf5Mg89+GI S5nx6FXwFY2fYMYUlngyX+9cnWieY6/gnOriXbr6utLg/aJnX5PwdUX4Ce2YhkQch7Xe I0vPRTAYuibeXfm9rqyOxrLk/HdUHZYDe37aMl4FdKFkQs54IE9EKEoikF4rmCnOhDtl q0L2i/R9VFoQ8JkM4BpVDyBe3oX3vwn4Wm1yvwy61YSA9SsQyt9L/g/I+5fua51gbfAO kXaJEPV3y4ceDs6dS4PZaFwU/APDrQ4+qmImG0u1UoBxXT6e6r3g4kKauM3uV8cs3adf QvjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=D4xovi6VfiUHc7H1WexA1wOCaWXMFwz43AMZCLiNoqA=; fh=odHZZLn5aUrUhssH6KSiJJlFsZ7wTjfMLFlHN8avkCo=; b=WlrTlK1NH89vkVVPPdgm7QZIu5Wf3bbf7rpxkGzcbs4pBocU7RKr4TDERtAx0pYvje tcrO4CtsPMyHa9E//EA4duE5So3N4IPehwY6xPPJgm1exd0i8JMgDSa+WQBuaLoHmeNv v0XPaKWmDAYRl87HmhtfJI/jpFQD1c1E+/JbqyX/N9Go1Y8DFJf4HaG3ZGcAq+odo26h Hrbjp9i1OJiX1ylhOtS9YKynLrT+A5C3F282l8/ylKeiMRvayeD5vfuTll/TpbX6nkIo mGX5h+Gr0ZzUwALn/hvxEG0eRdpRAEVc4YKIAJiN6to/nnqSNKDByf15dhbnQuoxofRK uymA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id u136-20020a0deb8e000000b005a22a115e6csi11138536ywe.325.2023.10.25.07.47.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 07:47:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 2399D802D0C0; Wed, 25 Oct 2023 07:47:05 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344142AbjJYOqL (ORCPT + 26 others); Wed, 25 Oct 2023 10:46:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343659AbjJYOqE (ORCPT ); Wed, 25 Oct 2023 10:46:04 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 940F9B0 for ; Wed, 25 Oct 2023 07:46:01 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EB8F71477; Wed, 25 Oct 2023 07:46:42 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id CC0443F64C; Wed, 25 Oct 2023 07:45:59 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , David Hildenbrand , Matthew Wilcox , Huang Ying , Gao Xiang , Yu Zhao , Yang Shi , Michal Hocko , Kefeng Wang Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 3/4] mm: swap: Simplify ssd behavior when scanner steals entry Date: Wed, 25 Oct 2023 15:45:45 +0100 Message-Id: <20231025144546.577640-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231025144546.577640-1-ryan.roberts@arm.com> References: <20231025144546.577640-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 25 Oct 2023 07:47:05 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780739196493443263 X-GMAIL-MSGID: 1780739196493443263 When a CPU fails to reserve a cluster (due to free list exhaustion), we revert to the scanner to find a free entry somewhere in the swap file. This might cause an entry to be stolen from another CPU's reserved cluster. Upon noticing this, the CPU with the stolen entry would previously scan forward to the end of the cluster trying to find a free entry to use. If there were none, it would try to reserve a new pre-cpu cluster and allocate from that. This scanning behavior does not scale well to high-order allocations, which will be introduced in a future patch since would need to scan for a contiguous area that was naturally aligned. Given stealing is a rare occurrence, let's remove the scanning behavior from the ssd allocator and simply drop the cluster and try to allocate a new one. Given the purpose of the per-cpu cluster is to ensure a given task's pages are sequential on disk to aid readahead, allocating a new cluster at this point makes most sense. Furthermore, si->max will always be greater than or equal to the end of the last cluster because any partial cluster will never be put on the free cluster list. Therefore we can simplify this logic too. These changes make it simpler to generalize scan_swap_map_try_ssd_cluster() to handle any allocation order. Signed-off-by: Ryan Roberts --- mm/swapfile.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 617e34b8cdbe..94f7cc225eb9 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -639,27 +639,24 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, /* * Other CPUs can use our cluster if they can't find a free cluster, - * check if there is still free entry in the cluster + * check if the expected entry is still free. If not, drop it and + * reserve a new cluster. */ - max = min_t(unsigned long, si->max, - ALIGN_DOWN(tmp, SWAPFILE_CLUSTER) + SWAPFILE_CLUSTER); - if (tmp < max) { - ci = lock_cluster(si, tmp); - while (tmp < max) { - if (!si->swap_map[tmp]) - break; - tmp++; - } + ci = lock_cluster(si, tmp); + if (si->swap_map[tmp]) { unlock_cluster(ci); - } - if (tmp >= max) { *cpu_next = SWAP_NEXT_NULL; goto new_cluster; } + unlock_cluster(ci); + *offset = tmp; *scan_base = tmp; + + max = ALIGN_DOWN(tmp, SWAPFILE_CLUSTER) + SWAPFILE_CLUSTER; tmp += 1; *cpu_next = tmp < max ? tmp : SWAP_NEXT_NULL; + return true; } From patchwork Wed Oct 25 14:45:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 158142 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp1175vqb; Wed, 25 Oct 2023 07:47:07 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGJw4fw6e8xh6q0GeE7/I7brGz44UK5YA3PgBQV8P15jvXPkgTg/cb1WFFI564RJn2DM7do X-Received: by 2002:a25:7452:0:b0:d74:62df:e802 with SMTP id p79-20020a257452000000b00d7462dfe802mr16008482ybc.0.1698245220329; Wed, 25 Oct 2023 07:47:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698245220; cv=none; d=google.com; s=arc-20160816; b=L7rAK/KtwJ7MSYKJehTpZB6fB4BfaKCmBAfZeIwHCrb9RKY6Frq+PmemIVeh0LLItF vDroOSGxIFAGI2+cyea9kKPP+zjC/mmU7ZGjSoEoZhMe2fxhJY1v6xKIF3zVmXiPO3It BROWIwj8aHOQSoOfW3rPX4wYk1mREAtFCfztftbPIbQBVCw7RE8GKL+aNF4k241hwPIW htSLB3IW0mTLgxiGIV1Qt3jDvtAkdojKL7TE1YClRyThMd9snxCTIEoOY9Pq3kN5RmtY au8oN9aZ7mweXtlzIbdh+bghKbUlfSSc8k4rEWDqr/WwhbHJQUTbMsJefMq9hjVRxiCu jJaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=GX0qjfIMCrEhQnbOpdF6hDbPFvKVZiRSqG9m0ghIw9w=; fh=odHZZLn5aUrUhssH6KSiJJlFsZ7wTjfMLFlHN8avkCo=; b=uhkIhCr7HtMNGoD1XHEnkgEsA5oBYkYEskMCyGjPLwt1lTztGeaAZeOISdXXFXQUQo 6jinfi2vhLOx4gFlfwxAkXwQRcumQVzHiL+JCkjqEXfX34hCB9ZRUz+HKfykE38UlLY5 daTuA3848y5hSaIXO3mgvuk9/wCwvCSwGaZLHuXj1cicJWsDXtQAq7um38sERGamr407 VPmzBvtBluT+lDbanDWDanaAix4YtxkFyJoy8TlC75oZxD1piYL3s3cEFZ6a7PmSZyjO rHwAvbrbwri7uVmnAY6S540BIKXjlFWkO+r4kPzBv5BH4isILOjEY1n2WkQZonr+gXc4 uewA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id j141-20020a25d293000000b00d9ad3026c20si10144516ybg.355.2023.10.25.07.46.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 07:47:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 6AEDB801CBE7; Wed, 25 Oct 2023 07:46:51 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344151AbjJYOqO (ORCPT + 26 others); Wed, 25 Oct 2023 10:46:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344046AbjJYOqG (ORCPT ); Wed, 25 Oct 2023 10:46:06 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 6EFF6138 for ; Wed, 25 Oct 2023 07:46:03 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C7E2F1476; Wed, 25 Oct 2023 07:46:44 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 90E5C3F64C; Wed, 25 Oct 2023 07:46:01 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , David Hildenbrand , Matthew Wilcox , Huang Ying , Gao Xiang , Yu Zhao , Yang Shi , Michal Hocko , Kefeng Wang Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting Date: Wed, 25 Oct 2023 15:45:46 +0100 Message-Id: <20231025144546.577640-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231025144546.577640-1-ryan.roberts@arm.com> References: <20231025144546.577640-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 25 Oct 2023 07:46:51 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780739179682012096 X-GMAIL-MSGID: 1780739179682012096 The upcoming anonymous small-sized THP feature enables performance improvements by allocating large folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting. Therefore, solve this regression by adding support for swapping out small-sized THP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when CONFIG_THP_SWAP is enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support. Note that no attempt is made to swap-in THP here - this is still done page-by-page, like for PMD-sized THP. The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [1, (1 << PMD_ORDER)]. This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller falls back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback). The per-order current clusters are maintained per-cpu using the existing infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern. __scan_swap_map_try_ssd_cluster() is introduced to deal with arbitrary orders and scan_swap_map_try_ssd_cluster() is refactored as a wrapper for order-0. As is done for order-0 per-cpu clusters, the scanner now can steal order-0 entries from any per-cpu-per-order reserved cluster. This ensures that when the swap file is getting full, space doesn't get tied up in the per-cpu reserves. I've run the tests on Ampere Altra (arm64), set up with a 35G block ram device as the swap device and from inside a memcg limited to 40G memory. I've then run `usemem` from vm-scalability with 70 processes (each has its own core), each allocating and writing 1G of memory. I've repeated everything 5 times and taken the mean: Mean Performance Improvement vs 4K/baseline | alloc size | baseline | + this series | | | v6.6-rc4+anonfolio | | |:-----------|--------------------:|--------------------:| | 4K Page | 0.0% | 4.9% | | 64K THP | -44.1% | 10.7% | | 2M THP | 56.0% | 65.9% | So with this change, the regression for 64K swap performance goes away and 4K and 2M swap improves slightly too. Signed-off-by: Ryan Roberts Signed-off-by: Barry Song --- include/linux/swap.h | 10 +-- mm/swapfile.c | 149 +++++++++++++++++++++++++++++++------------ mm/vmscan.c | 10 +-- 3 files changed, 119 insertions(+), 50 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 0ca8aaa098ba..ccbca5db851b 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -295,11 +295,11 @@ struct swap_info_struct { unsigned int __percpu *cluster_next_cpu; /*percpu index for next allocation */ unsigned int __percpu *cpu_next;/* * Likely next allocation offset. We - * assign a cluster to each CPU, so each - * CPU can allocate swap entry from its - * own cluster and swapout sequentially. - * The purpose is to optimize swapout - * throughput. + * assign a cluster per-order to each + * CPU, so each CPU can allocate swap + * entry from its own cluster and + * swapout sequentially. The purpose is + * to optimize swapout throughput. */ struct rb_root swap_extent_root;/* root of the swap extent rbtree */ struct block_device *bdev; /* swap device or bdev of swap file */ diff --git a/mm/swapfile.c b/mm/swapfile.c index 94f7cc225eb9..b50bce50bed9 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -545,10 +545,12 @@ static void free_cluster(struct swap_info_struct *si, unsigned long idx) /* * The cluster corresponding to page_nr will be used. The cluster will be - * removed from free cluster list and its usage counter will be increased. + * removed from free cluster list and its usage counter will be increased by + * count. */ -static void inc_cluster_info_page(struct swap_info_struct *p, - struct swap_cluster_info *cluster_info, unsigned long page_nr) +static void add_cluster_info_page(struct swap_info_struct *p, + struct swap_cluster_info *cluster_info, unsigned long page_nr, + unsigned long count) { unsigned long idx = page_nr / SWAPFILE_CLUSTER; @@ -557,9 +559,19 @@ static void inc_cluster_info_page(struct swap_info_struct *p, if (cluster_is_free(&cluster_info[idx])) alloc_cluster(p, idx); - VM_BUG_ON(cluster_count(&cluster_info[idx]) >= SWAPFILE_CLUSTER); + VM_BUG_ON(cluster_count(&cluster_info[idx]) + count > SWAPFILE_CLUSTER); cluster_set_count(&cluster_info[idx], - cluster_count(&cluster_info[idx]) + 1); + cluster_count(&cluster_info[idx]) + count); +} + +/* + * The cluster corresponding to page_nr will be used. The cluster will be + * removed from free cluster list and its usage counter will be increased. + */ +static void inc_cluster_info_page(struct swap_info_struct *p, + struct swap_cluster_info *cluster_info, unsigned long page_nr) +{ + add_cluster_info_page(p, cluster_info, page_nr, 1); } /* @@ -588,8 +600,8 @@ static void dec_cluster_info_page(struct swap_info_struct *p, * cluster list. Avoiding such abuse to avoid list corruption. */ static bool -scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si, - unsigned long offset) +__scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si, + unsigned long offset, int order) { bool conflict; @@ -601,23 +613,36 @@ scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si, if (!conflict) return false; - *this_cpu_ptr(si->cpu_next) = SWAP_NEXT_NULL; + this_cpu_ptr(si->cpu_next)[order] = SWAP_NEXT_NULL; return true; } /* - * Try to get a swap entry from current cpu's swap entry pool (a cluster). This - * might involve allocating a new cluster for current CPU too. + * It's possible scan_swap_map_slots() uses a free cluster in the middle of free + * cluster list. Avoiding such abuse to avoid list corruption. */ -static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, - unsigned long *offset, unsigned long *scan_base) +static bool +scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si, + unsigned long offset) +{ + return __scan_swap_map_ssd_cluster_conflict(si, offset, 0); +} + +/* + * Try to get a swap entry (or size indicated by order) from current cpu's swap + * entry pool (a cluster). This might involve allocating a new cluster for + * current CPU too. + */ +static bool __scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, + unsigned long *offset, unsigned long *scan_base, int order) { struct swap_cluster_info *ci; - unsigned int tmp, max; + unsigned int tmp, max, i; unsigned int *cpu_next; + unsigned int nr_pages = 1 << order; new_cluster: - cpu_next = this_cpu_ptr(si->cpu_next); + cpu_next = &this_cpu_ptr(si->cpu_next)[order]; tmp = *cpu_next; if (tmp == SWAP_NEXT_NULL) { if (!cluster_list_empty(&si->free_clusters)) { @@ -643,10 +668,12 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, * reserve a new cluster. */ ci = lock_cluster(si, tmp); - if (si->swap_map[tmp]) { - unlock_cluster(ci); - *cpu_next = SWAP_NEXT_NULL; - goto new_cluster; + for (i = 0; i < nr_pages; i++) { + if (si->swap_map[tmp + i]) { + unlock_cluster(ci); + *cpu_next = SWAP_NEXT_NULL; + goto new_cluster; + } } unlock_cluster(ci); @@ -654,12 +681,22 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, *scan_base = tmp; max = ALIGN_DOWN(tmp, SWAPFILE_CLUSTER) + SWAPFILE_CLUSTER; - tmp += 1; + tmp += nr_pages; *cpu_next = tmp < max ? tmp : SWAP_NEXT_NULL; return true; } +/* + * Try to get a swap entry from current cpu's swap entry pool (a cluster). This + * might involve allocating a new cluster for current CPU too. + */ +static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, + unsigned long *offset, unsigned long *scan_base) +{ + return __scan_swap_map_try_ssd_cluster(si, offset, scan_base, 0); +} + static void __del_from_avail_list(struct swap_info_struct *p) { int nid; @@ -982,35 +1019,58 @@ static int scan_swap_map_slots(struct swap_info_struct *si, return n_ret; } -static int swap_alloc_cluster(struct swap_info_struct *si, swp_entry_t *slot) +static int swap_alloc_large(struct swap_info_struct *si, swp_entry_t *slot, + unsigned int nr_pages) { - unsigned long idx; struct swap_cluster_info *ci; - unsigned long offset; + unsigned long offset, scan_base; + int order = ilog2(nr_pages); + bool ret; /* - * Should not even be attempting cluster allocations when huge + * Should not even be attempting large allocations when huge * page swap is disabled. Warn and fail the allocation. */ - if (!IS_ENABLED(CONFIG_THP_SWAP)) { + if (!IS_ENABLED(CONFIG_THP_SWAP) || + nr_pages < 2 || nr_pages > SWAPFILE_CLUSTER || + !is_power_of_2(nr_pages)) { VM_WARN_ON_ONCE(1); return 0; } - if (cluster_list_empty(&si->free_clusters)) + /* + * Swapfile is not block device or not using clusters so unable to + * allocate large entries. + */ + if (!(si->flags & SWP_BLKDEV) || !si->cluster_info) return 0; - idx = cluster_list_first(&si->free_clusters); - offset = idx * SWAPFILE_CLUSTER; - ci = lock_cluster(si, offset); - alloc_cluster(si, idx); - cluster_set_count(ci, SWAPFILE_CLUSTER); +again: + /* + * __scan_swap_map_try_ssd_cluster() may drop si->lock during discard, + * so indicate that we are scanning to synchronise with swapoff. + */ + si->flags += SWP_SCANNING; + ret = __scan_swap_map_try_ssd_cluster(si, &offset, &scan_base, order); + si->flags -= SWP_SCANNING; + + /* + * If we failed to allocate or if swapoff is waiting for us (due to lock + * being dropped for discard above), return immediately. + */ + if (!ret || !(si->flags & SWP_WRITEOK)) + return 0; - memset(si->swap_map + offset, SWAP_HAS_CACHE, SWAPFILE_CLUSTER); + if (__scan_swap_map_ssd_cluster_conflict(si, offset, order)) + goto again; + + ci = lock_cluster(si, offset); + memset(si->swap_map + offset, SWAP_HAS_CACHE, nr_pages); + add_cluster_info_page(si, si->cluster_info, offset, nr_pages); unlock_cluster(ci); - swap_range_alloc(si, offset, SWAPFILE_CLUSTER); - *slot = swp_entry(si->type, offset); + swap_range_alloc(si, offset, nr_pages); + *slot = swp_entry(si->type, offset); return 1; } @@ -1036,7 +1096,7 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size) int node; /* Only single cluster request supported */ - WARN_ON_ONCE(n_goal > 1 && size == SWAPFILE_CLUSTER); + WARN_ON_ONCE(n_goal > 1 && size > 1); spin_lock(&swap_avail_lock); @@ -1073,14 +1133,13 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size) spin_unlock(&si->lock); goto nextsi; } - if (size == SWAPFILE_CLUSTER) { - if (si->flags & SWP_BLKDEV) - n_ret = swap_alloc_cluster(si, swp_entries); + if (size > 1) { + n_ret = swap_alloc_large(si, swp_entries, size); } else n_ret = scan_swap_map_slots(si, SWAP_HAS_CACHE, n_goal, swp_entries); spin_unlock(&si->lock); - if (n_ret || size == SWAPFILE_CLUSTER) + if (n_ret || size > 1) goto check_out; cond_resched(); @@ -3041,6 +3100,8 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) if (p->bdev && bdev_nonrot(p->bdev)) { int cpu; unsigned long ci, nr_cluster; + int nr_order; + int i; p->flags |= SWP_SOLIDSTATE; p->cluster_next_cpu = alloc_percpu(unsigned int); @@ -3068,13 +3129,19 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) for (ci = 0; ci < nr_cluster; ci++) spin_lock_init(&((cluster_info + ci)->lock)); - p->cpu_next = alloc_percpu(unsigned int); + nr_order = IS_ENABLED(CONFIG_THP_SWAP) ? PMD_ORDER + 1 : 1; + p->cpu_next = __alloc_percpu(sizeof(unsigned int) * nr_order, + __alignof__(unsigned int)); if (!p->cpu_next) { error = -ENOMEM; goto bad_swap_unlock_inode; } - for_each_possible_cpu(cpu) - per_cpu(*p->cpu_next, cpu) = SWAP_NEXT_NULL; + for_each_possible_cpu(cpu) { + unsigned int *cpu_next = per_cpu_ptr(p->cpu_next, cpu); + + for (i = 0; i < nr_order; i++) + cpu_next[i] = SWAP_NEXT_NULL; + } } else { atomic_inc(&nr_rotate_swap); inced_nr_rotate_swap = true; diff --git a/mm/vmscan.c b/mm/vmscan.c index 2cc0cb41fb32..ea19710aa4cd 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, if (!can_split_folio(folio, NULL)) goto activate_locked; /* - * Split folios without a PMD map right - * away. Chances are some or all of the - * tail pages can be freed without IO. + * Split PMD-mappable folios without a + * PMD map right away. Chances are some + * or all of the tail pages can be freed + * without IO. */ - if (!folio_entire_mapcount(folio) && + if (folio_test_pmd_mappable(folio) && + !folio_entire_mapcount(folio) && split_folio_to_list(folio, folio_list)) goto activate_locked;