From patchwork Thu Jan 18 11:10:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 189157 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2bc4:b0:101:a8e8:374 with SMTP id hx4csp266773dyb; Thu, 18 Jan 2024 03:13:10 -0800 (PST) X-Google-Smtp-Source: AGHT+IEPdVlv/p4HvCeEsWFVuJhonR0PmAydd3NiaWLcbABiQz0m6qgcq9CuzcBIc9Bbx/rSUHaV X-Received: by 2002:a05:6808:3c4f:b0:3bd:9387:98d1 with SMTP id gl15-20020a0568083c4f00b003bd938798d1mr1103793oib.38.1705576390479; Thu, 18 Jan 2024 03:13:10 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1705576390; cv=pass; d=google.com; s=arc-20160816; b=x/oeQV4oK2LUMGCShLq6ZovNwNVDjtI1U1kqKi9nvggfd2A2xmrj6hmFCriIVsrjRa vkZuPZ5KYrYAiHT54l2HC4kG08xUedk6pkIQWybBRhAfxD9/mrz5c+rYc1xNvJ0F4pn0 guhoWwAQ5ilORpMFobBXlrI61E2W69dB28aQ1Ps3KUE+hoWPmgcjAeuf/mnsKiGQzgrT 7PMXzUGqVwtte67fL6vC99LAiY93Pyt8E/XYueuCcP+MlK0uoBm/MqmyElrIwPbG64RV IBmGZwY7/eG8hqkweG/lNAD9tdYAmgupLyUPNftc7hR9FYB2EggnoEaLp3E242iycwp+ kgXg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=H8++HXKXEg2rLAGYvsGKqRlSZ467q1G3QH/l6LoLLoc=; fh=ZJLiB781/myatuwRmuahTy9Ut1y9dZk7a8omO6Pu6vE=; b=duVTqoMePwag710E8TPkEjZ7aVYVn0Jr/OEWhHv0waYd8A7BD7dgpDFqb3xEfofPng LRo327RK8UaQIhVGQzz3Y8xRj5aNDbr+H/JnCSZ4U2MqKAiwUvxBhUprZwBhUGnBbZ8O UJb7PnCEg0kKtaoU20ichr9RgJsoIupFgEGE0QKXHACQHks5Lng4aeB2Dv+Wy0nAq1TM enWzQt9SO8/iydSCS00zosOqQi5kIB0Jri4Vicb1L3j6NdI8vw5BRwaUXl3lu3yITJdD L3mE/0B2Kp9AKHwSVjkj9eMoTnuQxQMimOXMtTEi8UeghKbtenX+fgRGJnw6pLhLh3xQ z1kw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=CrCpw9Aa; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-30036-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-30036-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id y20-20020a05620a25d400b00783315792b4si14250408qko.319.2024.01.18.03.13.10 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jan 2024 03:13:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-30036-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=CrCpw9Aa; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-30036-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-30036-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 38C171C24CE6 for ; Thu, 18 Jan 2024 11:13:10 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 71E4724B3A; Thu, 18 Jan 2024 11:11:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CrCpw9Aa" Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 67E2F241E9 for ; Thu, 18 Jan 2024 11:11:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705576277; cv=none; b=ItQt6f7qSmpLkerPW8io73LD2zvUb+xOFqwRaxJYAcnsfg+bhn+Z/mxkTfkJitF3N712sTDlZAChABf8WC1VmQznqvv8rJPHk0ZxRrcq9GbUBUBM1TGRf5sdk56dIaBzSFzb7n3EVFRnSHuvh0/NYslJXKPLYAk+TD0F2wUIt30= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705576277; c=relaxed/simple; bh=D7RsOP+LIuhOPv0bwLxnFvu5o5GxOXsinfRwhRLItgk=; h=Received:DKIM-Signature:X-Google-DKIM-Signature: X-Gm-Message-State:X-Google-Smtp-Source:X-Received:Received:From: To:Cc:Subject:Date:Message-Id:X-Mailer:In-Reply-To:References: MIME-Version:Content-Transfer-Encoding; b=RPFL/a1vj9zbSaIC+Kn8+yH90ouOZ9JEEqhmP4fF0pG2/cf/WPBV6Wm2aaizru03YP+VsNLO0K1mheGTBPRWvv4zd987l4pkIgpPq/WtxQWhRYPCNDVyiL9WtDPFOt1PductVgjvRvBI+691AK74LMYuCgXLEsNaWd5pPPqK8M4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=CrCpw9Aa; arc=none smtp.client-ip=209.85.210.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-6db0c49e93eso9048771b3a.1 for ; Thu, 18 Jan 2024 03:11:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705576275; x=1706181075; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H8++HXKXEg2rLAGYvsGKqRlSZ467q1G3QH/l6LoLLoc=; b=CrCpw9AaBiMvmamXy2asi1RyF+jhyyrlT3YuH454IOQCZzYBAWm5UTM+MnfqjcaWrv r5cxJcxcsJXLVCvK9KIt3DT00g5DYtGYbBjxQ8B9Nk85M8LxNriRO5FQ3+LsD6FSSV0B dCTGRqF3FigS8yP+vY6M6kX3QASokF2aa6UZTIWQfL6ke8SylLvpbw0sL2gAxYvaxlgm 6XwSAZmdOHXqr3pjF0cwGjVAHjNFUpLzDC0VBvRnMwNL804lFdTelQu9+0gBvw0sleRo NIN2erPgtVY1UXP7yED4U/8v8YO9BnlTaaqccqBh5Wrk9pZS8vWYXqSUWjzrMxLrSZin d+sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705576275; x=1706181075; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H8++HXKXEg2rLAGYvsGKqRlSZ467q1G3QH/l6LoLLoc=; b=LhJDrIkdP1XrFkzKc7CShjrHbAsFRAPyZtSpSSar02glHEi6C89M6zjoR0sZrgZzsS GwL9ylIhPJ1xebnRPlnXuIwUBUP26YezYeYTaX99xkx7UU67uhah3EGYAjTu2o8EC/Mk g3B16o7MArvFJtr6deHLKrZPuEq8qPryL/1bw+vWKujYys/CzFRGcyo1z1L26eHBzEBA aVmwOC1rStqs5AisSgZ8WEkOmvLu0hviGHbKMKFwwj9Y/NvW7fIoc6/5qDKHkBExzvfO E6j6mM1WyVhmT3kwDwTPMejMQeRWuzj3Grh8YaPLs3HJG12ZYQt6b/a+B4xZnnMJed4l hSKA== X-Gm-Message-State: AOJu0YyvSBUC9UxGWlzGo3CO49I82hTnRkG2IRF2hlap5t2xJOoOOz3J NU19t1CwaUGWz9F6/1fvnLPCnZ23oZ/BTixASlF1J4d7UGCC8E2k X-Received: by 2002:a05:6a20:7354:b0:19b:90d8:2a11 with SMTP id v20-20020a056a20735400b0019b90d82a11mr617298pzc.69.1705576275572; Thu, 18 Jan 2024 03:11:15 -0800 (PST) Received: from barry-desktop.. (143.122.224.49.dyn.cust.vf.net.nz. [49.224.122.143]) by smtp.gmail.com with ESMTPSA id t19-20020a056a0021d300b006d9be753ac7sm3039107pfj.108.2024.01.18.03.11.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jan 2024 03:11:15 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: ryan.roberts@arm.com, akpm@linux-foundation.org, david@redhat.com, linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, mhocko@suse.com, shy828301@gmail.com, wangkefeng.wang@huawei.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yuzhao@google.com, surenb@google.com, steven.price@arm.com, Barry Song Subject: [PATCH RFC 1/6] arm64: mm: swap: support THP_SWAP on hardware with MTE Date: Fri, 19 Jan 2024 00:10:31 +1300 Message-Id: <20240118111036.72641-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240118111036.72641-1-21cnbao@gmail.com> References: <20231025144546.577640-1-ryan.roberts@arm.com> <20240118111036.72641-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788426468982764232 X-GMAIL-MSGID: 1788426468982764232 From: Barry Song Commit d0637c505f8a1 ("arm64: enable THP_SWAP for arm64") brings up THP_SWAP on ARM64, but it doesn't enable THP_SWP on hardware with MTE as the MTE code works with the assumption tags save/restore is always handling a folio with only one page. The limitation should be removed as more and more ARM64 SoCs have this feature. Co-existence of MTE and THP_SWAP becomes more and more important. This patch makes MTE tags saving support large folios, then we don't need to split large folios into base pages for swapping out on ARM64 SoCs with MTE any more. arch_prepare_to_swap() should take folio rather than page as parameter because we support THP swap-out as a whole. It saves tags for all pages in a large folio. As now we are restoring tags based-on folio, in arch_swap_restore(), we may increase some extra loops and early-exitings while refaulting a large folio which is still in swapcache in do_swap_page(). In case a large folio has nr pages, do_swap_page() will only set the PTE of the particular page which is causing the page fault. Thus do_swap_page() runs nr times, and each time, arch_swap_restore() will loop nr times for those subpages in the folio. So right now the algorithmic complexity becomes O(nr^2). Once we support mapping large folios in do_swap_page(), extra loops and early-exitings will decrease while not being completely removed as a large folio might get partially tagged in corner cases such as, 1. a large folio in swapcache can be partially unmapped, thus, MTE tags for the unmapped pages will be invalidated; 2. users might use mprotect() to set MTEs on a part of a large folio. arch_thp_swp_supported() is dropped since ARM64 MTE was the only one who needed it. Reviewed-by: Steven Price Signed-off-by: Barry Song Acked-by: Chris Li --- arch/arm64/include/asm/pgtable.h | 21 +++------------- arch/arm64/mm/mteswap.c | 42 ++++++++++++++++++++++++++++++++ include/linux/huge_mm.h | 12 --------- include/linux/pgtable.h | 2 +- mm/page_io.c | 2 +- mm/swap_slots.c | 2 +- 6 files changed, 49 insertions(+), 32 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 79ce70fbb751..9902395ca426 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1) #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline bool arch_thp_swp_supported(void) -{ - return !system_supports_mte(); -} -#define arch_thp_swp_supported arch_thp_swp_supported - /* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page @@ -1042,12 +1036,8 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, #ifdef CONFIG_ARM64_MTE #define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap(struct page *page) -{ - if (system_supports_mte()) - return mte_save_tags(page); - return 0; -} +#define arch_prepare_to_swap arch_prepare_to_swap +extern int arch_prepare_to_swap(struct folio *folio); #define __HAVE_ARCH_SWAP_INVALIDATE static inline void arch_swap_invalidate_page(int type, pgoff_t offset) @@ -1063,11 +1053,8 @@ static inline void arch_swap_invalidate_area(int type) } #define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) -{ - if (system_supports_mte()) - mte_restore_tags(entry, &folio->page); -} +#define arch_swap_restore arch_swap_restore +extern void arch_swap_restore(swp_entry_t entry, struct folio *folio); #endif /* CONFIG_ARM64_MTE */ diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index a31833e3ddc5..b9ca1b35902f 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -68,6 +68,13 @@ void mte_invalidate_tags(int type, pgoff_t offset) mte_free_tag_storage(tags); } +static inline void __mte_invalidate_tags(struct page *page) +{ + swp_entry_t entry = page_swap_entry(page); + + mte_invalidate_tags(swp_type(entry), swp_offset(entry)); +} + void mte_invalidate_tags_area(int type) { swp_entry_t entry = swp_entry(type, 0); @@ -83,3 +90,38 @@ void mte_invalidate_tags_area(int type) } xa_unlock(&mte_pages); } + +int arch_prepare_to_swap(struct folio *folio) +{ + int err; + long i; + + if (system_supports_mte()) { + long nr = folio_nr_pages(folio); + + for (i = 0; i < nr; i++) { + err = mte_save_tags(folio_page(folio, i)); + if (err) + goto out; + } + } + return 0; + +out: + while (i--) + __mte_invalidate_tags(folio_page(folio, i)); + return err; +} + +void arch_swap_restore(swp_entry_t entry, struct folio *folio) +{ + if (system_supports_mte()) { + long i, nr = folio_nr_pages(folio); + + entry.val -= swp_offset(entry) & (nr - 1); + for (i = 0; i < nr; i++) { + mte_restore_tags(entry, folio_page(folio, i)); + entry.val++; + } + } +} diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 5adb86af35fc..67219d2309dd 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -530,16 +530,4 @@ static inline int split_folio(struct folio *folio) return split_folio_to_list(folio, NULL); } -/* - * archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to - * limitations in the implementation like arm64 MTE can override this to - * false - */ -#ifndef arch_thp_swp_supported -static inline bool arch_thp_swp_supported(void) -{ - return true; -} -#endif - #endif /* _LINUX_HUGE_MM_H */ diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index f6d0e3513948..37fe83b0c358 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -925,7 +925,7 @@ static inline int arch_unmap_one(struct mm_struct *mm, * prototypes must be defined in the arch-specific asm/pgtable.h file. */ #ifndef __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap(struct page *page) +static inline int arch_prepare_to_swap(struct folio *folio) { return 0; } diff --git a/mm/page_io.c b/mm/page_io.c index ae2b49055e43..a9a7c236aecc 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -189,7 +189,7 @@ int swap_writepage(struct page *page, struct writeback_control *wbc) * Arch code may have to preserve more data than just the page * contents, e.g. memory tags. */ - ret = arch_prepare_to_swap(&folio->page); + ret = arch_prepare_to_swap(folio); if (ret) { folio_mark_dirty(folio); folio_unlock(folio); diff --git a/mm/swap_slots.c b/mm/swap_slots.c index 0bec1f705f8e..2325adbb1f19 100644 --- a/mm/swap_slots.c +++ b/mm/swap_slots.c @@ -307,7 +307,7 @@ swp_entry_t folio_alloc_swap(struct folio *folio) entry.val = 0; if (folio_test_large(folio)) { - if (IS_ENABLED(CONFIG_THP_SWAP) && arch_thp_swp_supported()) + if (IS_ENABLED(CONFIG_THP_SWAP)) get_swap_pages(1, &entry, folio_nr_pages(folio)); goto out; } From patchwork Thu Jan 18 11:10:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 189164 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2bc4:b0:101:a8e8:374 with SMTP id hx4csp267840dyb; Thu, 18 Jan 2024 03:15:19 -0800 (PST) X-Google-Smtp-Source: AGHT+IHeJ4vLu4SPEWIENniMIxgpML2l8vcwPsi7zJIBqS1A43PdjO7tdVwwLdjrjasZ8rYDuuIi X-Received: by 2002:a05:6359:6e89:b0:175:7be5:7355 with SMTP id ti9-20020a0563596e8900b001757be57355mr620476rwb.31.1705576519122; Thu, 18 Jan 2024 03:15:19 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1705576519; cv=pass; d=google.com; s=arc-20160816; b=R8kHpIeZijVHzuTmCHtph127EptgySxZwEEzWAKkSFvzAVLsGA6auLRPVf0nJjVphD MAZHtAJ/iXmP+9zaB/+ivkaeps2m1hVIXH0RCw49ZU10EbTUXXgnq7t/EMXYlkvTqLzN WOjG+vHMnIdzz8xfN3s+r5KxxAnk9AoN6iG1uZrciqT+Fku5lxWRDxBghQTKYbOrPgPR 7ZZw4YCbFgbCHlaj0wq3THK4diqkZ6gg6yUc1cHCXIV/hiCRlANiCUpiqsABqXqi+PQV X/dCw0sju/Y+WM0Gvc7At6kNU+bRQhOzjzEWlhqrR+6ulVDvunQYuuwzAGKrSQNXOIAr w/+Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=yrbTbn/Co4xZrxsx5gjY1Btb4+PFsZ1QZ40ExDZVluk=; fh=HoForU5yJzZ7+tbHNDgwWa4PuJoKilOkY6/CHbWKADs=; b=PJyXtQac88oADqtmL7ZVGdskSiH0yhZMgCRmd7Nu76aFjkGU/HJt9uCMOp836bOf3q eD3mGT9wbByVwOlM0bT7c+crNgyZbzDP0M0owtbd3OM7iSoLIeg/iLHufPDli90i7HYl Xzwxlgz0hZKTrCvN4B700gNOKRqnhstqtqTeguBCERtyk2IRint8nWs/mMIr3fz6mGEq gLro3aYIaXYIWaov/iEU4l/u7IJ518YlkGMiVbc3lAqjTi9j78WLvvGcefdvldTVMHwU X8isQ0j4riOIYOUKYUHLPkLfXnVNZM9BnAsKBbWGOitAYHQbffBNpNYC54vZtfPgaurg NB+A== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=lDhrubko; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-30037-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-30037-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id h36-20020a63f924000000b005cd77a0fd8dsi1286195pgi.484.2024.01.18.03.15.18 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jan 2024 03:15:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-30037-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=lDhrubko; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-30037-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-30037-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 9B092289F58 for ; Thu, 18 Jan 2024 11:13:22 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A275D24B5D; Thu, 18 Jan 2024 11:11:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lDhrubko" Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A31DB20B1C for ; Thu, 18 Jan 2024 11:11:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705576294; cv=none; b=ouFaID1an1gpGFRElCfyyiZBpJUXHvhVwW1bOb5o/oFjngYxgqt+8/3UUv+qyivcJpYmbXs6Hlroz+rIMDdMGR3o+ICfL/M1XL/EQsDhOlP/yE2nRiu6EdvrMi86ZuT2uGjXqcvJp46dCi2hExOwlz0E4KjhMFC3K0HAQDeYuKg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705576294; c=relaxed/simple; bh=W8f0NHx3Q000IPONH4BGK4nCqN6LfRPEfKrciV5t9a8=; h=Received:DKIM-Signature:X-Google-DKIM-Signature: X-Gm-Message-State:X-Google-Smtp-Source:X-Received:Received:From: To:Cc:Subject:Date:Message-Id:X-Mailer:In-Reply-To:References: MIME-Version:Content-Transfer-Encoding; b=CA16hygBrZArJwSYhgW8oJUKooXngzmcEnQeoIY57oQh3XrWWNXfjNke94P4vTAi+oHj1/yuAclUgKlbzdtsG+nsY7NDlyRNp76LsAkzH8WnZXJPsXum0M2oETg1LdzBCJDl6PpLSK3ulleMMN60AsZYlXXqiUorYO1zpctsui8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lDhrubko; arc=none smtp.client-ip=209.85.210.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-6db786df38dso3845147b3a.0 for ; Thu, 18 Jan 2024 03:11:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705576293; x=1706181093; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yrbTbn/Co4xZrxsx5gjY1Btb4+PFsZ1QZ40ExDZVluk=; b=lDhrubkoyTi2uAetPFp5DK+6HMUBjSh+L10SoBkZqz8LPj+BC+/oDSy2qLbpDdxj+z a0opjwTtHywfW7yevk8cO9FMR1CKSsRaKiCqTMeWFQ1TpgwJ1gXToh2USeDSE6RNbTS1 7YEyqYEXciBIJ2w9MdPkiIMvew4nfU9qRya8/N4Db5crr2CuW7cKm58n0UasS9URLbDK 8Q1wdghAzbGHyiuAPEh4qEkpqW7Ow0bwAeNRXZwsO/bcB3Sy1Apz9gxTXu3J+4UgvhGy +06CTOT9uVREgl28HSdXTgSGiE0b5AQu52ibpMGrGfLa5T6rc63PujDLJMNY28ujMIJ9 cE+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705576293; x=1706181093; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yrbTbn/Co4xZrxsx5gjY1Btb4+PFsZ1QZ40ExDZVluk=; b=uwe36LMprAQ16yn5+CtSRXD5YjgrJcS7cuLR2Xe9DAKw9wm5MlpChYyLhyqA3Re7MQ 8K1J3/canxBNtpIUzK0nH24Zm6hBma9REaTc0WCxkNvQ0lkkZPklsc4W43CTArLOmFR4 d6Yv3j7KKDEaQ2cOhDhrazhY5zUlGiz4PKloG8a9p2bc8tQfpx3G5Cu0SHUQqMvS0nhA Zbs+LKbDf0ziCQ04akT8d+E2UhM3cnPV0f3TCkMQL3HC3KW4oNq2qReVticcXD0FM7Si /g5m4X9dkGFH1ZldFmdBCNQ2re0jBbmf+7ILZysOUogvvA1OlVE8TWp0pRSIcG0qYvXF Q5UQ== X-Gm-Message-State: AOJu0Yw3NkiCHb+4nW2Rtk/tFl7xloTx0E3y+2ZXaGL8Oyk30rg175K8 1zguir9AC0eirLN/VHOAvyY3GW3fIHX6oYGdz6mcLkNtk/3Ixq5L X-Received: by 2002:a05:6a00:174e:b0:6d9:bd63:e3e5 with SMTP id j14-20020a056a00174e00b006d9bd63e3e5mr689678pfc.26.1705576292986; Thu, 18 Jan 2024 03:11:32 -0800 (PST) Received: from barry-desktop.. (143.122.224.49.dyn.cust.vf.net.nz. [49.224.122.143]) by smtp.gmail.com with ESMTPSA id t19-20020a056a0021d300b006d9be753ac7sm3039107pfj.108.2024.01.18.03.11.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jan 2024 03:11:32 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: ryan.roberts@arm.com, akpm@linux-foundation.org, david@redhat.com, linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, mhocko@suse.com, shy828301@gmail.com, wangkefeng.wang@huawei.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yuzhao@google.com, surenb@google.com, steven.price@arm.com, Chuanhua Han , Barry Song Subject: [PATCH RFC 2/6] mm: swap: introduce swap_nr_free() for batched swap_free() Date: Fri, 19 Jan 2024 00:10:32 +1300 Message-Id: <20240118111036.72641-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240118111036.72641-1-21cnbao@gmail.com> References: <20231025144546.577640-1-ryan.roberts@arm.com> <20240118111036.72641-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788426604097774403 X-GMAIL-MSGID: 1788426604097774403 From: Chuanhua Han While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song --- include/linux/swap.h | 6 ++++++ mm/swapfile.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/include/linux/swap.h b/include/linux/swap.h index 4db00ddad261..31a4ee2dcd1c 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -478,6 +478,7 @@ extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); extern void swap_free(swp_entry_t); +extern void swap_nr_free(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern int free_swap_and_cache(swp_entry_t); int swap_type_of(dev_t device, sector_t offset); @@ -553,6 +554,11 @@ static inline void swap_free(swp_entry_t swp) { } +void swap_nr_free(swp_entry_t entry, int nr_pages) +{ + +} + static inline void put_swap_folio(struct folio *folio, swp_entry_t swp) { } diff --git a/mm/swapfile.c b/mm/swapfile.c index 556ff7347d5f..6321bda96b77 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1335,6 +1335,35 @@ void swap_free(swp_entry_t entry) __swap_entry_free(p, entry); } +void swap_nr_free(swp_entry_t entry, int nr_pages) +{ + int i; + struct swap_cluster_info *ci; + struct swap_info_struct *p; + unsigned type = swp_type(entry); + unsigned long offset = swp_offset(entry); + DECLARE_BITMAP(usage, SWAPFILE_CLUSTER) = { 0 }; + + VM_BUG_ON(offset % SWAPFILE_CLUSTER + nr_pages > SWAPFILE_CLUSTER); + + if (nr_pages == 1) { + swap_free(entry); + return; + } + + p = _swap_info_get(entry); + + ci = lock_cluster(p, offset); + for (i = 0; i < nr_pages; i++) { + if (__swap_entry_free_locked(p, offset + i, 1)) + __bitmap_set(usage, i, 1); + } + unlock_cluster(ci); + + for_each_clear_bit(i, usage, nr_pages) + free_swap_slot(swp_entry(type, offset + i)); +} + /* * Called after dropping swapcache to decrease refcnt to swap entries. */ From patchwork Thu Jan 18 11:10:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 189165 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2bc4:b0:101:a8e8:374 with SMTP id hx4csp267886dyb; Thu, 18 Jan 2024 03:15:26 -0800 (PST) X-Google-Smtp-Source: AGHT+IHLZ/KtDioujlyj5TMRndV05rPSaw6E1skcw6wOInjdj0AOA5s1ltsIV/EBxtwcSfNtsZhH X-Received: by 2002:a05:6a20:da83:b0:19a:3f38:b180 with SMTP id iy3-20020a056a20da8300b0019a3f38b180mr776481pzb.31.1705576526540; Thu, 18 Jan 2024 03:15:26 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1705576526; cv=pass; d=google.com; s=arc-20160816; b=ay+ZDbqxAFmX4SbDBn4Mq2K1hF5j6BlW585gt3a15QiGpHQPca2z9OmWg0VtAX+/7B FMNfF/5gCtzlRV/LuIVIYQW8RXMONmO9LC7oZYRPBpUx6GI8R6x86IC6U50JYuwS0/Mu VHo81oUfyNidMhkK2A1fh7/r9y97Jar+CMFyoDh0E0iJ1MHjfK3WI/RoObdEuOUOLmyE UCXGzykfJmQFoHyVjVp7HXLVXulU2RuQrb57XntOlNaf6baY3zhneCMzXtcejDqVnye6 Uv1Opnobozdt5xuLS4PXH3BzfVOx1Nelw62WAR7LC7jzSrvtHdH+ZdkkYe7asCevafyU DIuw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=xZZoI8vzJz6uaOc3IP6SaeIVpVjWcCgJVsn6AMZhjDQ=; fh=HoForU5yJzZ7+tbHNDgwWa4PuJoKilOkY6/CHbWKADs=; b=L3S+JKvxA56vxpOfvF2TH9gULbx2xDKf/K0lnJVZxrgE7CFGfwcw0Bs5XNCAUOR7Ci Bugv+m0JVBtqx5nmWizTgyMF83GjrKUgizcKKsqrHvz4IQe2G4Q/ZmYGw8KsFwnxdq5h Uo/DlTTc+I43nQC386mwVzQCYsFxMpTZ8yejL8ipBacqcuojTMCQFSatn9n7u+Uc2heW Butz2oClRw+cwwFzRpgvG57ZzFLY03zEURWlY3rX2RUWKx3wSwTYv8AuyTf8rNESe48d KR841ZcnmJbtn9GAwOvPiGcWH30R7Ksog/SODSqruhMYV75WLDV85a0vH3ilvxJXPSF5 VgQQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=fjsLNuiI; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-30038-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-30038-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id o12-20020a656a4c000000b005cde435ff91si1319724pgu.878.2024.01.18.03.15.26 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jan 2024 03:15:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-30038-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=fjsLNuiI; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-30038-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-30038-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id E9E6CB24FAC for ; Thu, 18 Jan 2024 11:13:35 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1A34225550; Thu, 18 Jan 2024 11:11:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fjsLNuiI" Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D996250E1 for ; Thu, 18 Jan 2024 11:11:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705576309; cv=none; b=ab6PX7laIEaTjmXjWAhcqUczpDDqS2iNK+5kJ/gN07pXWh+pzUarCe82EmJwWpGz5T3JFYU7BfGgNX7FfbqPfcvgV5dYZHJ+wAIWr+wZHY8++oRvEHR7SW+1SLgeIvuYu5TrVYcbZng2yvuR7YLvuFlgtColXH9Gv6hw61Qvm28= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705576309; c=relaxed/simple; bh=EqW7jc7DVH+vVsBXp8KOGtXAxEdSJsfQ40/OG3oOCrg=; h=Received:DKIM-Signature:X-Google-DKIM-Signature: X-Gm-Message-State:X-Google-Smtp-Source:X-Received:Received:From: To:Cc:Subject:Date:Message-Id:X-Mailer:In-Reply-To:References: MIME-Version:Content-Transfer-Encoding; b=VTQBlpdumTZk82rwfYExWEdrR26MdQqscpE2Q6c1D7hcurOPiXUXQWq9QFbIAtoSw5kRG6BzxM3TAMjsJmS3PGw4lp+tqbe9CAStYM+1o9QZwJOonOLITn4ZCuKmX61NmWzjA3Up4Sqx5BXlC/eD8JIrw4DDZkjsVrsHqV7DjiU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fjsLNuiI; arc=none smtp.client-ip=209.85.215.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-pg1-f179.google.com with SMTP id 41be03b00d2f7-5cd8667c59eso8475997a12.2 for ; Thu, 18 Jan 2024 03:11:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705576307; x=1706181107; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xZZoI8vzJz6uaOc3IP6SaeIVpVjWcCgJVsn6AMZhjDQ=; b=fjsLNuiIdorbIqu6SnQ3DyJw7U7oLlq6K3NH8xkyvNr7EbUtKHLxKZtFPzQvQXk6ie hNYP56ku3FD4yacZGRyBYD0irVZGoVHkn4TwYQqtJjnpWAKCPstnvXtWpdZ2p//Myfrp O8VbdkPv6Qouo8pqUuj6u0Gr9JlEhxAjZWnI9OFfY+FDcnNRwV1K3WXrnubGfv8ZAEI6 8DdC2uTXJoZ4abzlinehG1UHztrrlfAz9vBTL2IowPeoyVbxfM/oKSTbSLVqyIl/APdd df+gzIJ10C4z+AN2a3tSs+/TlzWpP3/3+GGOEEfV+gsDHVHFnwknxoo36PoHcznk7Md2 3oKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705576307; x=1706181107; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xZZoI8vzJz6uaOc3IP6SaeIVpVjWcCgJVsn6AMZhjDQ=; b=ug8NfpIWSBO+Hc+Saq/YovAwY4YnecIiXYQyRdOXs+cuAUFWzWEumZgEMZWA1wQcy+ qsoqn51UM/jNKA5nR4NwBKM6972cjk0a+62sJazbo4/Pb2ys8bSVBwYq+qAyYjvwUXV4 GYLk3byj83AkQmQCVV72w/TiCh+IP3cWXDMRDDOJDgyKwijIb90rAENvO56AnKc6d0Mj /awFMa2R2R5ldkFTRo3/4BZbhkz6AnQ+mgWojWv8sEKggavr7av9HevJDl0/gPrrY9ig h06mGSU4gC26C6bmWLIca52oHjtk+iTKyjcDDAy0FtaIoYPLUvSEpzhAR9jbyiUDGe5R vZzw== X-Gm-Message-State: AOJu0YyLEGfPdKTvg8qdtue8jR76/J+r+IO0OvUMrCmHHylq3mT8VlUW 5V6o18qj9JqZP8CLzOzCRP2KZl4gNZEU/hozkWxvbfWoq0uBs9AJ X-Received: by 2002:a05:6a20:9f9b:b0:199:e4ab:691c with SMTP id mm27-20020a056a209f9b00b00199e4ab691cmr731200pzb.8.1705576307486; Thu, 18 Jan 2024 03:11:47 -0800 (PST) Received: from barry-desktop.. (143.122.224.49.dyn.cust.vf.net.nz. [49.224.122.143]) by smtp.gmail.com with ESMTPSA id t19-20020a056a0021d300b006d9be753ac7sm3039107pfj.108.2024.01.18.03.11.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jan 2024 03:11:47 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: ryan.roberts@arm.com, akpm@linux-foundation.org, david@redhat.com, linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, mhocko@suse.com, shy828301@gmail.com, wangkefeng.wang@huawei.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yuzhao@google.com, surenb@google.com, steven.price@arm.com, Chuanhua Han , Barry Song Subject: [PATCH RFC 3/6] mm: swap: make should_try_to_free_swap() support large-folio Date: Fri, 19 Jan 2024 00:10:33 +1300 Message-Id: <20240118111036.72641-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240118111036.72641-1-21cnbao@gmail.com> References: <20231025144546.577640-1-ryan.roberts@arm.com> <20240118111036.72641-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788426611847056426 X-GMAIL-MSGID: 1788426611847056426 From: Chuanhua Han should_try_to_free_swap() works with an assumption that swap-in is always done at normal page granularity, aka, folio_nr_pages = 1. To support large folio swap-in, this patch removes the assumption. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Acked-by: Chris Li --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 7e1f4849463a..f61a48929ba7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3714,7 +3714,7 @@ static inline bool should_try_to_free_swap(struct folio *folio, * reference only in case it's likely that we'll be the exlusive user. */ return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) && - folio_ref_count(folio) == 2; + folio_ref_count(folio) == (1 + folio_nr_pages(folio)); } static vm_fault_t pte_marker_clear(struct vm_fault *vmf) From patchwork Thu Jan 18 11:10:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 189166 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2bc4:b0:101:a8e8:374 with SMTP id hx4csp268022dyb; Thu, 18 Jan 2024 03:15:43 -0800 (PST) X-Google-Smtp-Source: AGHT+IFtNjd1x8BRJ1Ltv9rBc1j23ngiy8q4y+WBMarvnh+LJtwXgUkDJFbSLQJDx8UWTsF00wk8 X-Received: by 2002:a05:6871:58a2:b0:210:d06f:39dd with SMTP id ok34-20020a05687158a200b00210d06f39ddmr55609oac.26.1705576542979; Thu, 18 Jan 2024 03:15:42 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1705576542; cv=pass; d=google.com; s=arc-20160816; b=A/PYJUfWqCUBvQQ9kVpVKTORJWjtgi3T65cq++46f4L2TwGNmvCGfCRA5z8tRNH0HH FICh9TriSf6NK1hTamdPuYmSPTNhcqmCRZw89ymmyRRH+Ej1ujHh6rHOeVqTCIE/d/LN 4f2Fxymiro7jE1FC2Vm71hOs+2eVZeC69uWsDfpRo2UaW1xflW3ILnKHTqSo0DBXKtN8 WXbVuN0Al8l6C12hdNwRaVzjwmzcdltZ+yzd6k/Jg3z0rPqcGFm5ODfeLFxamj+f4Oup KXkVT2IVLmrSXxc8+jcBMVUJ2DJMyNUFMDvpt9ak8aLHBF6Zg8IqRdxcXrBgeSW0L4BD hKpA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=vxtV65B3xdBW2CsjvzUu/C8QzJUJxjQlgzUH6d2NXwk=; fh=HoForU5yJzZ7+tbHNDgwWa4PuJoKilOkY6/CHbWKADs=; b=R0NNl6yP4S/k8rht0hxn3/KeVUc4dos9yh6WIhLJro3QeE61pv9wjj0HWPR5W9Z7yL jIBWisOekDNVPRTkqCJIx2+ju0w86E+g/nWIQsTl/MZh433yaTL4JzVA+aBPTdcCSGiZ s1UXCYbf9Cy8SzOl2LimeMl+mC1q15GcmGs9UckBYb5RTymTJoqrMhlo0aBvjzPi1e7m W8NRlwOrAwYJIpQXIMGG6E4p7u6itxbjzDtt+SDFWIGzSGssDWYP4ACMI9P/Qzh63aca p7MM61vk4R4XAvi+MfvWmYMMJMlVLkn+FD+y/Gw7Lnt+3Gv1Tfl+OVk+4gp6+AT92gTE 1ftg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b="YzIVV/Kt"; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-30039-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-30039-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id bx26-20020a056a02051a00b00578ea9a0b93si1384319pgb.890.2024.01.18.03.15.42 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jan 2024 03:15:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-30039-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b="YzIVV/Kt"; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-30039-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-30039-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id AB6F6283A5C for ; Thu, 18 Jan 2024 11:13:47 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id EA7902577D; Thu, 18 Jan 2024 11:12:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YzIVV/Kt" Received: from mail-oo1-f51.google.com (mail-oo1-f51.google.com [209.85.161.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7778323768 for ; Thu, 18 Jan 2024 11:12:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705576327; cv=none; b=NCit6Lz5yq5QTDrwldvo8uZifsQa3pfpA4ZI+qQYMvGOyX3Kb36GgukSrZIGfSiJCgSyMOMs4U/SJKm0gYpm9sUnYwruOkQDtxQp/RIkauEJgJvX2nR+3Yw8/d914bh112gEfhpjH9D+DdoDGGpstQl+dUi+qq/93WS7vHZUEp0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705576327; c=relaxed/simple; bh=3iSuabJMCFOhjj90fjjIfmTTve7X6yPyjG4bw6oRZqE=; h=Received:DKIM-Signature:X-Google-DKIM-Signature: X-Gm-Message-State:X-Google-Smtp-Source:X-Received:Received:From: To:Cc:Subject:Date:Message-Id:X-Mailer:In-Reply-To:References: MIME-Version:Content-Transfer-Encoding; b=DyqlM3oXGMBFDMyEukKUPR3ypN/dKgBfgeTVLUOindaygf4dLrvFlixcn78kHNtAUrxSgp+SJETO/zZYwD3feVl9t8HisfUnRYy3Hk9QYRtNKIt1EwIRNbwAnBUd5B9a25NGWDI1GWJeKynbEvvI4uxVieVdmpfogdD9bDLgGng= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YzIVV/Kt; arc=none smtp.client-ip=209.85.161.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-oo1-f51.google.com with SMTP id 006d021491bc7-595d24ad466so6600513eaf.0 for ; Thu, 18 Jan 2024 03:12:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705576324; x=1706181124; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vxtV65B3xdBW2CsjvzUu/C8QzJUJxjQlgzUH6d2NXwk=; b=YzIVV/Ktr9rxO0oV++6pS+GCPLBOwHYC5hjFRWOon2FTHfP/34/4p/aoQ5i2HzG1Jp 6BHn5vD4xPqhTXJ1p25zK3hC69uCChayIUsFa+Hzc3F5NWtnLO5mDgr54KyjXPwhD2pm YUtEw3yRgK/kJlRz216vXUzCqiDXlxsHsubz33RWRJCMsI3juo85WYP0Pp+b/bca4m4/ nBuuLlMnjvBhLjH67a+VQlWFDoqi1zzuFpqozMwHgw33eq9RpIkc31EKmd3dLKt+Yzib 16bfBx3+z2NmOCxtluJkDyHonGHW4H+5lvQQ/7CQWIRS/zPWu2vk3dNq3jESCisYUT5Z bvTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705576324; x=1706181124; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vxtV65B3xdBW2CsjvzUu/C8QzJUJxjQlgzUH6d2NXwk=; b=tFcRvPp7AT0g4kiDo8z8u0rhm27VK0Mw2eA8ToyfLGBRmbg6NbwMjFskDLffmbqnZ0 qGdHMvwIDg/9tBlpqMa3l16wCmUCowRxYPxBthlPQ0jafCSXXJXrqPmeKsLB7D2x3lD3 T/6jumMk0iI9YczvNR55tkeC/084FGL600q48hcMDYr4jbqFYGNnPwusXCbK+fXIOEpO s/wSaLiY9j9UlZ5GnIaNFbm2LosrlnhW/mIWA1A1CLHqSCb5wviZBgzvDxLvGWIawzNZ iBKSdh9g8v09whjutdhFx9Eoz217p278zOSwYIJfhZ4TpBju2u2NmApdAgIq8Hz77BWB Nawg== X-Gm-Message-State: AOJu0YzuqTMmDq88SWsrm9umerkfbHoxU+7O1X9ICap4Kx3+HyY1xnbh EcKp91AoHLIM5exXWQr94IeyZBLpAoaEI88IYTCAM3HAE6/uIZNA9QN8JDo2 X-Received: by 2002:a05:6359:ba7:b0:175:5c8c:3ab with SMTP id gf39-20020a0563590ba700b001755c8c03abmr609446rwb.65.1705576324307; Thu, 18 Jan 2024 03:12:04 -0800 (PST) Received: from barry-desktop.. (143.122.224.49.dyn.cust.vf.net.nz. [49.224.122.143]) by smtp.gmail.com with ESMTPSA id t19-20020a056a0021d300b006d9be753ac7sm3039107pfj.108.2024.01.18.03.11.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jan 2024 03:12:03 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: ryan.roberts@arm.com, akpm@linux-foundation.org, david@redhat.com, linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, mhocko@suse.com, shy828301@gmail.com, wangkefeng.wang@huawei.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yuzhao@google.com, surenb@google.com, steven.price@arm.com, Chuanhua Han , Barry Song Subject: [PATCH RFC 4/6] mm: support large folios swapin as a whole Date: Fri, 19 Jan 2024 00:10:34 +1300 Message-Id: <20240118111036.72641-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240118111036.72641-1-21cnbao@gmail.com> References: <20231025144546.577640-1-ryan.roberts@arm.com> <20240118111036.72641-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788426629345746337 X-GMAIL-MSGID: 1788426629345746337 From: Chuanhua Han On an embedded system like Android, more than half of anon memory is actually in swap devices such as zRAM. For example, while an app is switched to back- ground, its most memory might be swapped-out. Now we have mTHP features, unfortunately, if we don't support large folios swap-in, once those large folios are swapped-out, we immediately lose the performance gain we can get through large folios and hardware optimization such as CONT-PTE. This patch brings up mTHP swap-in support. Right now, we limit mTHP swap-in to those contiguous swaps which were likely swapped out from mTHP as a whole. On the other hand, the current implementation only covers the SWAP_SYCHRONOUS case. It doesn't support swapin_readahead as large folios yet. Right now, we are re-faulting large folios which are still in swapcache as a whole, this can effectively decrease extra loops and early-exitings which we have increased in arch_swap_restore() while supporting MTE restore for folios rather than page. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song --- mm/memory.c | 108 +++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 94 insertions(+), 14 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index f61a48929ba7..928b3f542932 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -107,6 +107,8 @@ EXPORT_SYMBOL(mem_map); static vm_fault_t do_fault(struct vm_fault *vmf); static vm_fault_t do_anonymous_page(struct vm_fault *vmf); static bool vmf_pte_changed(struct vm_fault *vmf); +static struct folio *alloc_anon_folio(struct vm_fault *vmf, + bool (*pte_range_check)(pte_t *, int)); /* * Return true if the original pte was a uffd-wp pte marker (so the pte was @@ -3784,6 +3786,34 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf) return VM_FAULT_SIGBUS; } +static bool pte_range_swap(pte_t *pte, int nr_pages) +{ + int i; + swp_entry_t entry; + unsigned type; + pgoff_t start_offset; + + entry = pte_to_swp_entry(ptep_get_lockless(pte)); + if (non_swap_entry(entry)) + return false; + start_offset = swp_offset(entry); + if (start_offset % nr_pages) + return false; + + type = swp_type(entry); + for (i = 1; i < nr_pages; i++) { + entry = pte_to_swp_entry(ptep_get_lockless(pte + i)); + if (non_swap_entry(entry)) + return false; + if (swp_offset(entry) != start_offset + i) + return false; + if (swp_type(entry) != type) + return false; + } + + return true; +} + /* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked. @@ -3804,6 +3834,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte_t pte; vm_fault_t ret = 0; void *shadow = NULL; + int nr_pages = 1; + unsigned long start_address; + pte_t *start_pte; if (!pte_unmap_same(vmf)) goto out; @@ -3868,13 +3901,20 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1) { /* skip swapcache */ - folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, - vma, vmf->address, false); + folio = alloc_anon_folio(vmf, pte_range_swap); page = &folio->page; if (folio) { __folio_set_locked(folio); __folio_set_swapbacked(folio); + if (folio_test_large(folio)) { + unsigned long start_offset; + + nr_pages = folio_nr_pages(folio); + start_offset = swp_offset(entry) & ~(nr_pages - 1); + entry = swp_entry(swp_type(entry), start_offset); + } + if (mem_cgroup_swapin_charge_folio(folio, vma->vm_mm, GFP_KERNEL, entry)) { @@ -3980,6 +4020,39 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + + start_address = vmf->address; + start_pte = vmf->pte; + if (folio_test_large(folio)) { + unsigned long nr = folio_nr_pages(folio); + unsigned long addr = ALIGN_DOWN(vmf->address, nr * PAGE_SIZE); + pte_t *pte_t = vmf->pte - (vmf->address - addr) / PAGE_SIZE; + + /* + * case 1: we are allocating large_folio, try to map it as a whole + * iff the swap entries are still entirely mapped; + * case 2: we hit a large folio in swapcache, and all swap entries + * are still entirely mapped, try to map a large folio as a whole. + * otherwise, map only the faulting page within the large folio + * which is swapcache + */ + if (pte_range_swap(pte_t, nr)) { + start_address = addr; + start_pte = pte_t; + if (unlikely(folio == swapcache)) { + /* + * the below has been done before swap_read_folio() + * for case 1 + */ + nr_pages = nr; + entry = pte_to_swp_entry(ptep_get(start_pte)); + page = &folio->page; + } + } else if (nr_pages > 1) { /* ptes have changed for case 1 */ + goto out_nomap; + } + } + if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte))) goto out_nomap; @@ -4047,12 +4120,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free(entry); + swap_nr_free(entry, nr_pages); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); + folio_ref_add(folio, nr_pages - 1); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); + pte = mk_pte(page, vma->vm_page_prot); /* @@ -4062,14 +4137,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * exclusivity. */ if (!folio_test_ksm(folio) && - (exclusive || folio_ref_count(folio) == 1)) { + (exclusive || folio_ref_count(folio) == nr_pages)) { if (vmf->flags & FAULT_FLAG_WRITE) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE; } rmap_flags |= RMAP_EXCLUSIVE; } - flush_icache_page(vma, page); + flush_icache_pages(vma, page, nr_pages); if (pte_swp_soft_dirty(vmf->orig_pte)) pte = pte_mksoft_dirty(pte); if (pte_swp_uffd_wp(vmf->orig_pte)) @@ -4081,14 +4156,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) folio_add_new_anon_rmap(folio, vma, vmf->address); folio_add_lru_vma(folio, vma); } else { - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, start_address, rmap_flags); } VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); + set_ptes(vma->vm_mm, start_address, start_pte, pte, nr_pages); + + arch_do_swap_page(vma->vm_mm, vma, start_address, pte, vmf->orig_pte); folio_unlock(folio); if (folio != swapcache && swapcache) { @@ -4105,6 +4181,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } if (vmf->flags & FAULT_FLAG_WRITE) { + if (folio_test_large(folio) && nr_pages > 1) + vmf->orig_pte = ptep_get(vmf->pte); + ret |= do_wp_page(vmf); if (ret & VM_FAULT_ERROR) ret &= VM_FAULT_ERROR; @@ -4112,7 +4191,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } /* No need to invalidate - it was non-present before */ - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + update_mmu_cache_range(vmf, vma, start_address, start_pte, nr_pages); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); @@ -4148,7 +4227,8 @@ static bool pte_range_none(pte_t *pte, int nr_pages) return true; } -static struct folio *alloc_anon_folio(struct vm_fault *vmf) +static struct folio *alloc_anon_folio(struct vm_fault *vmf, + bool (*pte_range_check)(pte_t *, int)) { #ifdef CONFIG_TRANSPARENT_HUGEPAGE struct vm_area_struct *vma = vmf->vma; @@ -4190,7 +4270,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) order = highest_order(orders); while (orders) { addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order); - if (pte_range_none(pte + pte_index(addr), 1 << order)) + if (pte_range_check(pte + pte_index(addr), 1 << order)) break; order = next_order(&orders, order); } @@ -4269,7 +4349,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) if (unlikely(anon_vma_prepare(vma))) goto oom; /* Returns NULL on OOM or ERR_PTR(-EAGAIN) if we must retry the fault */ - folio = alloc_anon_folio(vmf); + folio = alloc_anon_folio(vmf, pte_range_none); if (IS_ERR(folio)) return 0; if (!folio) From patchwork Thu Jan 18 11:10:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 189160 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2bc4:b0:101:a8e8:374 with SMTP id hx4csp267147dyb; Thu, 18 Jan 2024 03:14:02 -0800 (PST) X-Google-Smtp-Source: AGHT+IFeDj/Ytr6+jNiJlnEGmrHmwS+SgTshor5mUi6bckbj8pvXk712Or5njiJ1hsB6KZx0B2MY X-Received: by 2002:a17:906:d148:b0:a26:ac91:143a with SMTP id br8-20020a170906d14800b00a26ac91143amr331072ejb.50.1705576441736; Thu, 18 Jan 2024 03:14:01 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1705576441; cv=pass; d=google.com; s=arc-20160816; b=yCb2UebEbcKGFBqW9MGUr54WCufmagjzMjgy2BHfraDoziX2KoDHr8ju1W/UrRIrNU T70+StLhU06K9bzpEr8IC9MC0a40TwUaQKu7F/7fCNgUMTluqhWYZ96A5mRUF654YCs7 YvskLh1yt8JdsCrDR4RGdIbEBNE2eAaZTQXXjzaFQm7pg0ccG6LjyZ8dzPtRnZxWS0RC GQOgC/EqGsqnJHDe5ZohBaeqI7+siBY2WZbE6PLP71wtXgW3QVC/4On2MZ/obstIDsyp z0gcTh8bU/bAT9mdueO2nwUUp/hsJbn2WhP0SZkUJZgRAgVIIlM+YkkxiOPvYHvsO/r5 CTDg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=lCBEGTZpNMmcOnq2Y92D+y0mof5icxn15UOhqT4EV4Y=; fh=E7SxvC7sqVqoE9/GOqzXQjoKC/Jt6DGUfGL6Wer1X8Y=; b=gZXg9E1Bz8E8rCTDk9SUJIhrvFlK645akMXSD7nW075igjEf8oWR0ww1zwluOmZ0gW u/plfBqdvzybeMN561VDmaxMNJLyOHdUHtIRWCa0XO/QRrggO/nr878ZFXshS5sGr0/i bAJnbCIF4mM7JbC2WGNgEtZXVPaINC9dLChRFoaGgycwfjWv5RjrUrwgmD7f5qngD1pF TrTDhNXdFvaMh2r5wGVB91/X8qq3wfWKtYozoV+o8xEFd+k3CAlWg/r886Bjjm1I4aAC svmYyFtfHc8Ee6hT79ZHzdw/mF7wD0HcWBmZ64m02YZ4LBVXu2jLy7e28mq5hqjqmcko gaEQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=jBAgfL1X; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-30040-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-30040-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id ks12-20020a170906f84c00b00a2ccf7acc62si5678743ejb.194.2024.01.18.03.14.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jan 2024 03:14:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-30040-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=jBAgfL1X; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-30040-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-30040-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 573E11F22E6A for ; Thu, 18 Jan 2024 11:14:01 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A9E0F25778; Thu, 18 Jan 2024 11:12:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jBAgfL1X" Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C972C25613 for ; Thu, 18 Jan 2024 11:12:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705576350; cv=none; b=is7Sq35kVnFslJVgmGs+6tyunjBsz9K0gLmkuyf+5fR3i5zOWqGwfsHrjivKeQX5sfoT2JfyYETv9WowLsNdj45cMb3dLIx+7cPd8sN2/K+TO4limQsRnF7arbdVTs9qijLnxrh+YPyNm1mzHXLcxjgmTk1l2afnqol13Cv3Ox0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705576350; c=relaxed/simple; bh=DwksO4pggIfrxQAN1+eTkGGLwNBL3ggzIFhofhqrICM=; h=Received:DKIM-Signature:X-Google-DKIM-Signature: X-Gm-Message-State:X-Google-Smtp-Source:X-Received:Received:From: To:Cc:Subject:Date:Message-Id:X-Mailer:In-Reply-To:References: MIME-Version:Content-Transfer-Encoding; b=ZYtbGxGi+rpwXR0WFKB5MP6n0l1fNXVbu+LvJVUD2qS/ZBBV5oxwHjp7wuJpGaqjChb3lMWPFoh/c9l+b3oZ8yfnnUhhImk82UoiN1B1Gq9Zusn3Xgta5Edj3Wnb6w7eCh1EAWFrfW/Sau7TYxEdZwXQZSfstNuYm66VJKIKqd8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=jBAgfL1X; arc=none smtp.client-ip=209.85.210.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-6dac8955af0so7015750b3a.0 for ; Thu, 18 Jan 2024 03:12:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705576348; x=1706181148; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lCBEGTZpNMmcOnq2Y92D+y0mof5icxn15UOhqT4EV4Y=; b=jBAgfL1X0U4H5TWVlcljiWvdoUKS51PKLX7xBDDKbJ6NUkfq3YK7kqOLNjZAjPanya vTBO5T5uZme5NrVLcSyrKuWYlu1gcsGq5cCC9gNXiCj/UNgy66MrdFKGRoi9VLijqYPn NcYHOthpSHctkNC64L7SKP++qwp5LppEn4ty25q35AcK0Bh+xnYo/xIhfL99eekPGEdN dLX3IZkNtg4S/1hQxRH62ujDJjznWuXPj0J2v/zG6lKp42M/LA/zpJSBQH3Ew/kNbY5i 1AFVnnlmTWLJ5H5ZSjLpiV5zEE5jFeAwk+Kl2StCmPK/OeBEvrpTc+qL5sOXLqRsGjKz z17A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705576348; x=1706181148; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lCBEGTZpNMmcOnq2Y92D+y0mof5icxn15UOhqT4EV4Y=; b=JvIEloHdCx5+EsDGk0uB6I6iYa4/n0WDY9E1svKC2CZiM4mNFLYR+EzsYiYjreJeJi FSU32B9F+U8EVCmMiEGL/dsBRwbiXRQFhJ7kX1ybOKpqYOVcLleQ7mCJGvl0c/CUu9zc Z7MI76c+r4PSgeEQeUNTRDJZ70pXRi40wmM1PcOpJvwaIRZAFeKv/ET7KIlS0w//QxL1 tUx3eXHnF8J6G/riv8bkKCDq3lz3Tck1IKATvyODM15vM9psn0WmGUaN+pzJ7L1u7MTh lr0WLEk862Y1yP4j8JTuPelpXR/wuU6EfKQ/8tIwjrcdOETJUwLmMsg+Ihnegtu1J3w8 pvcw== X-Gm-Message-State: AOJu0YxdA46+0GPJvCWb6FTA1A8Xjg7FM9wv0RX9yS9+GwWpvt4fYyCk YmCO058b5ya8AViI2a0cSrddW9mIe7mq44mRhBy5Llc5TKkWn4e9 X-Received: by 2002:a05:6a00:170c:b0:6db:6ed8:4e9f with SMTP id h12-20020a056a00170c00b006db6ed84e9fmr684987pfc.56.1705576348144; Thu, 18 Jan 2024 03:12:28 -0800 (PST) Received: from barry-desktop.. (143.122.224.49.dyn.cust.vf.net.nz. [49.224.122.143]) by smtp.gmail.com with ESMTPSA id t19-20020a056a0021d300b006d9be753ac7sm3039107pfj.108.2024.01.18.03.12.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jan 2024 03:12:27 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: ryan.roberts@arm.com, akpm@linux-foundation.org, david@redhat.com, linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, mhocko@suse.com, shy828301@gmail.com, wangkefeng.wang@huawei.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yuzhao@google.com, surenb@google.com, steven.price@arm.com, Barry Song , Chuanhua Han Subject: [PATCH RFC 5/6] mm: rmap: weaken the WARN_ON in __folio_add_anon_rmap() Date: Fri, 19 Jan 2024 00:10:35 +1300 Message-Id: <20240118111036.72641-6-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240118111036.72641-1-21cnbao@gmail.com> References: <20231025144546.577640-1-ryan.roberts@arm.com> <20240118111036.72641-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788426522821522650 X-GMAIL-MSGID: 1788426522821522650 From: Barry Song In do_swap_page(), while supporting large folio swap-in, we are using the helper folio_add_anon_rmap_ptes. This is triggerring a WARN_ON in __folio_add_anon_rmap. We can make the warning quiet by two ways 1. in do_swap_page, we call folio_add_new_anon_rmap() if we are sure the large folio is new allocated one; we call folio_add_anon_rmap_ptes() if we find the large folio in swapcache. 2. we always call folio_add_anon_rmap_ptes() in do_swap_page but weaken the WARN_ON in __folio_add_anon_rmap() by letting the WARN_ON less sensitive. Option 2 seems to be better for do_swap_page() as it can use unified code for all cases. Signed-off-by: Barry Song Tested-by: Chuanhua Han --- mm/rmap.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/rmap.c b/mm/rmap.c index f5d43edad529..469fcfd32317 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1304,7 +1304,10 @@ static __always_inline void __folio_add_anon_rmap(struct folio *folio, * page. */ VM_WARN_ON_FOLIO(folio_test_large(folio) && - level != RMAP_LEVEL_PMD, folio); + level != RMAP_LEVEL_PMD && + (!IS_ALIGNED(address, nr_pages * PAGE_SIZE) || + (folio_test_swapcache(folio) && !IS_ALIGNED(folio->index, nr_pages)) || + page != &folio->page), folio); __folio_set_anon(folio, vma, address, !!(flags & RMAP_EXCLUSIVE)); } else if (likely(!folio_test_ksm(folio))) { From patchwork Thu Jan 18 11:10:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 189161 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2bc4:b0:101:a8e8:374 with SMTP id hx4csp267256dyb; Thu, 18 Jan 2024 03:14:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IFYuKCPPBK/ItC56fnzp8jlny4vwyT+FBza053/wWpNL56I7j5yxkx3Ip7074fMCW/U2abA X-Received: by 2002:a05:620a:384:b0:783:47ff:34fe with SMTP id q4-20020a05620a038400b0078347ff34femr954777qkm.56.1705576453802; Thu, 18 Jan 2024 03:14:13 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1705576453; cv=pass; d=google.com; s=arc-20160816; b=QzPwrGRe69B+dFlCTHdhunCsWMWXuyOVgc4TS9IoGvtZngQaPhlSRfdL/JPyfLnR5y +pzk48u6hjZwOr+K2cbelmngHOrJw1Sd67lLIH+fBHbzJfFTPV2n9eDrYN9nH02lw5KQ bFU4XT7/FUa17m8pudpi4Tk2+eQREseV/Fpi3g64nRJhRwLd2X09HUkguPy3vkbBMNvw c3xe2wuzSN3oQ41QvdQ3igY5J1v/pHx0F/qm47zonCeOeJRUAMOr3xjxt6IVmc66bKUb ij3yCaLBdP+lWJphEUcicW4EMuP7ue8EMwhk1EiTXNLIs+dT3VNVlExd307fUR6LcfUL c4jA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=jPRbXml8IYd9XMbGDp/b9xmA+KUiJTFI45JekYQVHFk=; fh=HoForU5yJzZ7+tbHNDgwWa4PuJoKilOkY6/CHbWKADs=; b=L2L4DGHz++G5N4auPSlTMK00xXWWDVXu3j7oKotq1i4fn5CkVCABmCrf5wKIGeFd+b Sq3fVA6Vgyt18C49OdIicTLEv2UoIu5AFhoE89pq12JN0qIDyZxJ/1hgByLo+yCgVzES jjjS5HG8b1rjvwOzHlVFpWU/A9sdgGT1IaRfFn37njyACW4t1xOzQhG/JKUsxE5y8DZW rMda2C6AsaZBN8WIQlmqJkJaRb8lzFa6nKvg5J//ULHAyVzNsLocQZEr4/2rRDsThfVK vCulvW0uUAxcpngXMkK/HhY2Kab5Lwlv2YowDJCTEBnsZHZ/R5DqqIt7I/z4E55e0+22 FB5A== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=E6Zc89DB; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-30041-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-30041-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id qs7-20020a05620a394700b007816c894164si14745870qkn.370.2024.01.18.03.14.13 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jan 2024 03:14:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-30041-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20230601 header.b=E6Zc89DB; arc=pass (i=1 spf=pass spfdomain=gmail.com dkim=pass dkdomain=gmail.com dmarc=pass fromdomain=gmail.com); spf=pass (google.com: domain of linux-kernel+bounces-30041-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-30041-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 894221C24128 for ; Thu, 18 Jan 2024 11:14:13 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 65AA8286B4; Thu, 18 Jan 2024 11:12:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="E6Zc89DB" Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A05BC2375A for ; Thu, 18 Jan 2024 11:12:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705576367; cv=none; b=RJgC0rIF93fU42aT+Oi5zdcXf4L4+8hOHo/CktlHkMjE+erljvDi/LbvNidHwUQg/dlH+opSvsp+J3sUOCfCnb8gCBJa9xdg1PUdIMd3nZTIdLxMI1vXDlK8B1H/dYfKgJ1sPobCFGXuowi/+5efJBOCnqzsMs2XD93qTy4QQmw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705576367; c=relaxed/simple; bh=2rkgrzFdO6JMP0Th3YTAsnG21pW/6krZABLQ6XKuipI=; h=Received:DKIM-Signature:X-Google-DKIM-Signature: X-Gm-Message-State:X-Google-Smtp-Source:X-Received:Received:From: To:Cc:Subject:Date:Message-Id:X-Mailer:In-Reply-To:References: MIME-Version:Content-Transfer-Encoding; b=H2yZcbwUsFKHxfBzQJAGXrShR2lcX11zyrNW5UogYU+eSFysELfpKxSdg/hyseNcbG0xXOEhvO/SsrUI4LtnaaU0rwt4tJ1qUx0rHsptY+gAMVzuEH9LIdTyT9HrCTfHpJ1JwaszoCFMVVX1kHIwWhn306Lji5gejztX59noONs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=E6Zc89DB; arc=none smtp.client-ip=209.85.210.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-6d9b37f4804so393869b3a.1 for ; Thu, 18 Jan 2024 03:12:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705576365; x=1706181165; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jPRbXml8IYd9XMbGDp/b9xmA+KUiJTFI45JekYQVHFk=; b=E6Zc89DB+LpEirh8qGL7egVvFVf7h8DvkEwutnAf+0Fa9stFOkmSFxbMEW5O1I5F4C VShy7uqCZiqqlM9w3mIBVWZYBhne1utH6uXxipBQKWElab1S3uwpbrVcO6InSP3qKEMm kizI+eeZmgKoOu0TetJaDBXqTNy22gzzfR/5fhON3uVRzP9trVTuzGOOcb8ta+YzDIpF MxJgvLbTQDUoz/WvJ8P9Vj6HfJyJaGenWJMnmwWwW484RXKSk+28I8Gh6w4lVscv8SdP 8ly1Is6DW7toDfqXVVHNodEHmdhgR0D2I/zumcaRcjSG7BrNMndbT7bveHb7jofZX+Hd m6jQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705576365; x=1706181165; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jPRbXml8IYd9XMbGDp/b9xmA+KUiJTFI45JekYQVHFk=; b=Z7M5d4TN3dnu6kp+k8SQ2XowM762Aeo8sW0dcEbfo7pZf9vvjtXeeiMlgcwSUc/4mI j+InCj1j6t1i6rHjzrPYZdHYBCIykt5KKYY+aPHDs0WSBttQ1ds3r1WRuazmGSgjQlUi nbITmVed1QeatevGFRAe7b+rzEc+wNzRO2qBkUT5xhz4WHewSXFYDmBoV4/eregi7Uq/ fGv9HfKsmGiB1EgOfo507AH2XUR+q7opcECW8WCPv1QWv1WfsFENzGrJXwMlg9/+2I99 YlH47xKmV148OFJQ65CgeRCiBgVbbJtv5DRm6+YoIF9DrPjtB2rW0dTXW9ktOrWq+wbN 9JGw== X-Gm-Message-State: AOJu0YxPMCh6ue8UGID98rWVvn+mbXe3quvwBewVigokPNsDAIRwWydI Cb1YfEsmT08vFomEparx+lbr4ri/+aUzwv1h+13yzsJVJs+bLZEq X-Received: by 2002:a05:6a20:548b:b0:19a:fbfa:b16a with SMTP id i11-20020a056a20548b00b0019afbfab16amr1121941pzk.30.1705576364885; Thu, 18 Jan 2024 03:12:44 -0800 (PST) Received: from barry-desktop.. (143.122.224.49.dyn.cust.vf.net.nz. [49.224.122.143]) by smtp.gmail.com with ESMTPSA id t19-20020a056a0021d300b006d9be753ac7sm3039107pfj.108.2024.01.18.03.12.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jan 2024 03:12:44 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: ryan.roberts@arm.com, akpm@linux-foundation.org, david@redhat.com, linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, mhocko@suse.com, shy828301@gmail.com, wangkefeng.wang@huawei.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yuzhao@google.com, surenb@google.com, steven.price@arm.com, Chuanhua Han , Barry Song Subject: [PATCH RFC 6/6] mm: madvise: don't split mTHP for MADV_PAGEOUT Date: Fri, 19 Jan 2024 00:10:36 +1300 Message-Id: <20240118111036.72641-7-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240118111036.72641-1-21cnbao@gmail.com> References: <20231025144546.577640-1-ryan.roberts@arm.com> <20240118111036.72641-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788426535943928480 X-GMAIL-MSGID: 1788426535943928480 From: Chuanhua Han MADV_PAGEOUT and MADV_FREE are common cases in Android. Ryan's patchset has supported swapping large folios out as a whole for vmscan case. This patch extends the feature to madvise. If madvised range covers the whole large folio, we don't split it. Otherwise, we still need to split it. This patch doesn't depend on ARM64's CONT-PTE, alternatively, it defines one helper named pte_range_cont_mapped() to check if all PTEs are contiguously mapped to a large folio. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Signed-off-by: Barry Song --- include/asm-generic/tlb.h | 10 +++++++ include/linux/pgtable.h | 60 +++++++++++++++++++++++++++++++++++++++ mm/madvise.c | 48 +++++++++++++++++++++++++++++++ 3 files changed, 118 insertions(+) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 129a3a759976..f894e22da5d6 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -608,6 +608,16 @@ static inline void tlb_flush_p4d_range(struct mmu_gather *tlb, __tlb_remove_tlb_entry(tlb, ptep, address); \ } while (0) +#define tlb_remove_nr_tlb_entry(tlb, ptep, address, nr) \ + do { \ + int i; \ + tlb_flush_pte_range(tlb, address, \ + PAGE_SIZE * nr); \ + for (i = 0; i < nr; i++) \ + __tlb_remove_tlb_entry(tlb, ptep + i, \ + address + i * PAGE_SIZE); \ + } while (0) + #define tlb_remove_huge_tlb_entry(h, tlb, ptep, address) \ do { \ unsigned long _sz = huge_page_size(h); \ diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 37fe83b0c358..da0c1cf447e3 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -320,6 +320,42 @@ static inline pgd_t pgdp_get(pgd_t *pgdp) } #endif +#ifndef pte_range_cont_mapped +static inline bool pte_range_cont_mapped(unsigned long start_pfn, + pte_t *start_pte, + unsigned long start_addr, + int nr) +{ + int i; + pte_t pte_val; + + for (i = 0; i < nr; i++) { + pte_val = ptep_get(start_pte + i); + + if (pte_none(pte_val)) + return false; + + if (pte_pfn(pte_val) != (start_pfn + i)) + return false; + } + + return true; +} +#endif + +#ifndef pte_range_young +static inline bool pte_range_young(pte_t *start_pte, int nr) +{ + int i; + + for (i = 0; i < nr; i++) + if (pte_young(ptep_get(start_pte + i))) + return true; + + return false; +} +#endif + #ifndef __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, @@ -580,6 +616,23 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, } #endif +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR_RANGE_FULL +static inline pte_t ptep_get_and_clear_range_full(struct mm_struct *mm, + unsigned long start_addr, + pte_t *start_pte, + int nr, int full) +{ + int i; + pte_t pte; + + pte = ptep_get_and_clear_full(mm, start_addr, start_pte, full); + + for (i = 1; i < nr; i++) + ptep_get_and_clear_full(mm, start_addr + i * PAGE_SIZE, + start_pte + i, full); + + return pte; +} /* * If two threads concurrently fault at the same page, the thread that @@ -995,6 +1048,13 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) }) #endif +#ifndef pte_nr_addr_end +#define pte_nr_addr_end(addr, size, end) \ +({ unsigned long __boundary = ((addr) + size) & (~(size - 1)); \ + (__boundary - 1 < (end) - 1)? __boundary: (end); \ +}) +#endif + /* * When walking page tables, we usually want to skip any p?d_none entries; * and any p?d_bad entries - reporting the error before resetting to none. diff --git a/mm/madvise.c b/mm/madvise.c index 912155a94ed5..262460ac4b2e 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -452,6 +452,54 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, if (folio_test_large(folio)) { int err; + if (!folio_test_pmd_mappable(folio)) { + int nr_pages = folio_nr_pages(folio); + unsigned long folio_size = PAGE_SIZE * nr_pages; + unsigned long start_addr = ALIGN_DOWN(addr, nr_pages * PAGE_SIZE);; + unsigned long start_pfn = page_to_pfn(folio_page(folio, 0)); + pte_t *start_pte = pte - (addr - start_addr) / PAGE_SIZE; + unsigned long next = pte_nr_addr_end(addr, folio_size, end); + + if (!pte_range_cont_mapped(start_pfn, start_pte, start_addr, nr_pages)) + goto split; + + if (next - addr != folio_size) { + goto split; + } else { + /* Do not interfere with other mappings of this page */ + if (folio_estimated_sharers(folio) != 1) + goto skip; + + VM_BUG_ON(addr != start_addr || pte != start_pte); + + if (pte_range_young(start_pte, nr_pages)) { + ptent = ptep_get_and_clear_range_full(mm, start_addr, start_pte, + nr_pages, tlb->fullmm); + ptent = pte_mkold(ptent); + + set_ptes(mm, start_addr, start_pte, ptent, nr_pages); + tlb_remove_nr_tlb_entry(tlb, start_pte, start_addr, nr_pages); + } + + folio_clear_referenced(folio); + folio_test_clear_young(folio); + if (pageout) { + if (folio_isolate_lru(folio)) { + if (folio_test_unevictable(folio)) + folio_putback_lru(folio); + else + list_add(&folio->lru, &folio_list); + } + } else + folio_deactivate(folio); + } +skip: + pte += (next - PAGE_SIZE - (addr & PAGE_MASK))/PAGE_SIZE; + addr = next - PAGE_SIZE; + continue; + + } +split: if (folio_estimated_sharers(folio) != 1) break; if (pageout_anon_only_filter && !folio_test_anon(folio))