Message ID | 20231220224504.646757-29-david@redhat.com |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel+bounces-7590-ouuuleilei=gmail.com@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2483:b0:fb:cd0c:d3e with SMTP id q3csp62078dyi; Wed, 20 Dec 2023 15:05:39 -0800 (PST) X-Google-Smtp-Source: AGHT+IGVLRqu93VnoiHqvKR9zCAn2XAKaLRvA4KGCDyJkRe5vqkV/Rop08PzDLttwx3Wb2zm6L4p X-Received: by 2002:ac8:5f8b:0:b0:423:7b2b:173a with SMTP id j11-20020ac85f8b000000b004237b2b173amr28172987qta.50.1703113539143; Wed, 20 Dec 2023 15:05:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1703113539; cv=none; d=google.com; s=arc-20160816; b=sCpK1eVV/Bc7i+XlzLCTrroKIskN3lt8mLbe40dRuoNO1sWkXRSVk6z5c8Kx1u5q6w UqxfxhJTBGqUBrXzYAq3djFNs4GNiWKkHE1eCfcYI3X5NFJ03W2ZVsEdIRx6tCWEGiwB VuzTjxFV5O9srSMgbGjlXQoQ9UvvFdi+nq0E5QnuRF0TJufWkbpcpkb6bvPJomjdSTNo 6w+npWKMswY3pWslNd9mgQgUrN6iQ9bgFRXE4jjXZwPb2ELsFdj13tI/vCTiwob3GjDf cmEoC1vEtQfowzdkmdEtBsPElpUoDvIELPHlVHT3qo+CJd1+C39Y4h/GvG+SArPO3J5A AXaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=Io0fNOjJFUJN+cQn2T3HaexKpR7MvH4XyxQbyfgzgvU=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=Ja3ljRRdUcuCQdVT/cPWmhF+g3NcCCJsj5hMCcvc671aH5yJ4Mu+H+P8RY0d0YznlZ DVLRjUqn8PU45Acs/ovbG7MuRf7Ckz/zQkzbOWg+qz/lAcdx6bo5bLoR+9qili2xhAla SKCL9uRNuLmQihTnOgHOmVaDKtqk7lok9XzSOyadsnjefq/R6bVO2tETTgEAtxGJ2ZCC 4Zg4J4XzFyPNy5a6wFhvvVVjn9WGsiX+W4YNETA6EM+XCI856FHLPoB0fAKXTK2RL5Te UosWl0aZAyV447CFF32VFV80Vbo7KPARdOsHUeCO2G8mOzPa0UP7mANNtLIAXjxuKM30 tfKA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KMu5bPPM; spf=pass (google.com: domain of linux-kernel+bounces-7590-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-7590-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id w14-20020ac87e8e000000b0042790333fedsi774583qtj.474.2023.12.20.15.05.39 for <ouuuleilei@gmail.com> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Dec 2023 15:05:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-7590-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KMu5bPPM; spf=pass (google.com: domain of linux-kernel+bounces-7590-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-7590-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id EA08A1C21D1A for <ouuuleilei@gmail.com>; Wed, 20 Dec 2023 23:05:38 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id F19257408A; Wed, 20 Dec 2023 22:46:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="KMu5bPPM" X-Original-To: linux-kernel@vger.kernel.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D99A75644C for <linux-kernel@vger.kernel.org>; Wed, 20 Dec 2023 22:46:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703112384; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Io0fNOjJFUJN+cQn2T3HaexKpR7MvH4XyxQbyfgzgvU=; b=KMu5bPPME8GaUknG1Y/BE6CuqDPRV1bTWpApKe1UBlnUvjsw85aUrDL168z8FEeHtG3ZpN vNhUKnBX2lhPBpt3RZLU5qIlxZgPogbFrKoYoIG4ihubbKUrj0NqofPCYSAzkJNLawdrkw wZ4P5P4fXgLcJywmou5WSqKkD2XMSHw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-99-EEpn4mmfOKyp69_cUJHDvw-1; Wed, 20 Dec 2023 17:46:20 -0500 X-MC-Unique: EEpn4mmfOKyp69_cUJHDvw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9284985A588; Wed, 20 Dec 2023 22:46:19 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.192.101]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6B17A40C6EB9; Wed, 20 Dec 2023 22:46:17 +0000 (UTC) From: David Hildenbrand <david@redhat.com> To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand <david@redhat.com>, Andrew Morton <akpm@linux-foundation.org>, "Matthew Wilcox (Oracle)" <willy@infradead.org>, Hugh Dickins <hughd@google.com>, Ryan Roberts <ryan.roberts@arm.com>, Yin Fengwei <fengwei.yin@intel.com>, Mike Kravetz <mike.kravetz@oracle.com>, Muchun Song <muchun.song@linux.dev>, Peter Xu <peterx@redhat.com> Subject: [PATCH v2 28/40] mm/memory: page_remove_rmap() -> folio_remove_rmap_pte() Date: Wed, 20 Dec 2023 23:44:52 +0100 Message-ID: <20231220224504.646757-29-david@redhat.com> In-Reply-To: <20231220224504.646757-1-david@redhat.com> References: <20231220224504.646757-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: <linux-kernel.vger.kernel.org> List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org> List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785843982468326350 X-GMAIL-MSGID: 1785843982468326350 |
Series |
mm/rmap: interface overhaul
|
|
Commit Message
David Hildenbrand
Dec. 20, 2023, 10:44 p.m. UTC
Let's convert zap_pte_range() and closely-related
tlb_flush_rmap_batch(). While at it, perform some more folio conversion
in zap_pte_range().
Signed-off-by: David Hildenbrand <david@redhat.com>
---
mm/memory.c | 23 +++++++++++++----------
mm/mmu_gather.c | 2 +-
2 files changed, 14 insertions(+), 11 deletions(-)
Comments
On 20/12/2023 22:44, David Hildenbrand wrote: > Let's convert zap_pte_range() and closely-related > tlb_flush_rmap_batch(). While at it, perform some more folio conversion > in zap_pte_range(). > > Signed-off-by: David Hildenbrand <david@redhat.com> > --- > mm/memory.c | 23 +++++++++++++---------- > mm/mmu_gather.c | 2 +- > 2 files changed, 14 insertions(+), 11 deletions(-) > > diff --git a/mm/memory.c b/mm/memory.c > index 6552ea27b0bfa..eda2181275d9b 100644 > --- a/mm/memory.c > +++ b/mm/memory.c > @@ -1434,6 +1434,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, > arch_enter_lazy_mmu_mode(); > do { > pte_t ptent = ptep_get(pte); > + struct folio *folio; > struct page *page; > > if (pte_none(ptent)) > @@ -1459,21 +1460,22 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, > continue; > } > > + folio = page_folio(page); > delay_rmap = 0; > - if (!PageAnon(page)) { > + if (!folio_test_anon(folio)) { > if (pte_dirty(ptent)) { > - set_page_dirty(page); > + folio_set_dirty(folio); Is this foliation change definitely correct? I note that set_page_dirty() is defined as this: bool set_page_dirty(struct page *page) { return folio_mark_dirty(page_folio(page)); } And folio_mark_dirty() is doing more than just setting teh PG_dirty bit. In my equivalent change, as part of the contpte series, I've swapped set_page_dirty() for folio_mark_dirty(). > if (tlb_delay_rmap(tlb)) { > delay_rmap = 1; > force_flush = 1; > } > } > if (pte_young(ptent) && likely(vma_has_recency(vma))) > - mark_page_accessed(page); > + folio_mark_accessed(folio); > } > rss[mm_counter(page)]--; > if (!delay_rmap) { > - page_remove_rmap(page, vma, false); > + folio_remove_rmap_pte(folio, page, vma); > if (unlikely(page_mapcount(page) < 0)) > print_bad_pte(vma, addr, ptent, page); > } > @@ -1489,6 +1491,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, > if (is_device_private_entry(entry) || > is_device_exclusive_entry(entry)) { > page = pfn_swap_entry_to_page(entry); > + folio = page_folio(page); > if (unlikely(!should_zap_page(details, page))) > continue; > /* > @@ -1500,8 +1503,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, > WARN_ON_ONCE(!vma_is_anonymous(vma)); > rss[mm_counter(page)]--; > if (is_device_private_entry(entry)) > - page_remove_rmap(page, vma, false); > - put_page(page); > + folio_remove_rmap_pte(folio, page, vma); > + folio_put(folio); > } else if (!non_swap_entry(entry)) { > /* Genuine swap entry, hence a private anon page */ > if (!should_zap_cows(details)) > @@ -3220,10 +3223,10 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) > * threads. > * > * The critical issue is to order this > - * page_remove_rmap with the ptp_clear_flush above. > - * Those stores are ordered by (if nothing else,) > + * folio_remove_rmap_pte() with the ptp_clear_flush > + * above. Those stores are ordered by (if nothing else,) > * the barrier present in the atomic_add_negative > - * in page_remove_rmap. > + * in folio_remove_rmap_pte(); > * > * Then the TLB flush in ptep_clear_flush ensures that > * no process can access the old page before the > @@ -3232,7 +3235,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) > * mapcount is visible. So transitively, TLBs to > * old page will be flushed before it can be reused. > */ > - page_remove_rmap(vmf->page, vma, false); > + folio_remove_rmap_pte(old_folio, vmf->page, vma); > } > > /* Free the old page.. */ > diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c > index 4f559f4ddd217..604ddf08affed 100644 > --- a/mm/mmu_gather.c > +++ b/mm/mmu_gather.c > @@ -55,7 +55,7 @@ static void tlb_flush_rmap_batch(struct mmu_gather_batch *batch, struct vm_area_ > > if (encoded_page_flags(enc)) { > struct page *page = encoded_page_ptr(enc); > - page_remove_rmap(page, vma, false); > + folio_remove_rmap_pte(page_folio(page), page, vma); > } > } > }
On 22.01.24 17:58, Ryan Roberts wrote: > On 20/12/2023 22:44, David Hildenbrand wrote: >> Let's convert zap_pte_range() and closely-related >> tlb_flush_rmap_batch(). While at it, perform some more folio conversion >> in zap_pte_range(). >> >> Signed-off-by: David Hildenbrand <david@redhat.com> >> --- >> mm/memory.c | 23 +++++++++++++---------- >> mm/mmu_gather.c | 2 +- >> 2 files changed, 14 insertions(+), 11 deletions(-) >> >> diff --git a/mm/memory.c b/mm/memory.c >> index 6552ea27b0bfa..eda2181275d9b 100644 >> --- a/mm/memory.c >> +++ b/mm/memory.c >> @@ -1434,6 +1434,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, >> arch_enter_lazy_mmu_mode(); >> do { >> pte_t ptent = ptep_get(pte); >> + struct folio *folio; >> struct page *page; >> >> if (pte_none(ptent)) >> @@ -1459,21 +1460,22 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, >> continue; >> } >> >> + folio = page_folio(page); >> delay_rmap = 0; >> - if (!PageAnon(page)) { >> + if (!folio_test_anon(folio)) { >> if (pte_dirty(ptent)) { >> - set_page_dirty(page); >> + folio_set_dirty(folio); > > Is this foliation change definitely correct? I note that set_page_dirty() is > defined as this: > > bool set_page_dirty(struct page *page) > { > return folio_mark_dirty(page_folio(page)); > } > > And folio_mark_dirty() is doing more than just setting teh PG_dirty bit. In my > equivalent change, as part of the contpte series, I've swapped set_page_dirty() > for folio_mark_dirty(). Good catch, that should be folio_mark_dirty(). Let me send a fixup. (the difference in naming for both functions really is bad)
On Mon, Jan 22, 2024 at 06:01:58PM +0100, David Hildenbrand wrote: > > And folio_mark_dirty() is doing more than just setting teh PG_dirty bit. In my > > equivalent change, as part of the contpte series, I've swapped set_page_dirty() > > for folio_mark_dirty(). > > Good catch, that should be folio_mark_dirty(). Let me send a fixup. > > (the difference in naming for both functions really is bad) It really is, and I don't know what to do about it. We need a function that literally just sets the flag. For every other flag, that's folio_set_FLAG. We can't use __folio_set_flag because that means "set the flag non-atomically". We need a function that does all of the work involved with tracking dirty folios. I chose folio_mark_dirty() to align with folio_mark_uptodate() (ie mark is not just 'set" but also "do some extra work"). But because we're converting from set_page_dirty(), the OBVIOUS rename is to folio_set_dirty(), which is WRONG. So we're in the part of the design space where the consistent naming and the-obvious-thing-to-do-is-wrong are in collision, and I do not have a good answer. Maybe we can call the first function _folio_set_dirty(), and we don't have a folio_set_dirty() at all? We don't have a folio_set_uptodate(), so there's some precedent there.
On 22/01/2024 17:20, Matthew Wilcox wrote: > On Mon, Jan 22, 2024 at 06:01:58PM +0100, David Hildenbrand wrote: >>> And folio_mark_dirty() is doing more than just setting teh PG_dirty bit. In my >>> equivalent change, as part of the contpte series, I've swapped set_page_dirty() >>> for folio_mark_dirty(). >> >> Good catch, that should be folio_mark_dirty(). Let me send a fixup. >> >> (the difference in naming for both functions really is bad) > > It really is, and I don't know what to do about it. > > We need a function that literally just sets the flag. For every other > flag, that's folio_set_FLAG. We can't use __folio_set_flag because that > means "set the flag non-atomically". > > We need a function that does all of the work involved with tracking > dirty folios. I chose folio_mark_dirty() to align with > folio_mark_uptodate() (ie mark is not just 'set" but also "do some extra > work"). > > But because we're converting from set_page_dirty(), the OBVIOUS rename > is to folio_set_dirty(), which is WRONG. > > So we're in the part of the design space where the consistent naming and > the-obvious-thing-to-do-is-wrong are in collision, and I do not have a > good answer. > > Maybe we can call the first function _folio_set_dirty(), and we don't > have a folio_set_dirty() at all? We don't have a folio_set_uptodate(), > so there's some precedent there. Is there anything stopping us from renaming set_page_dirty() to mark_page_dirty() (or page_mark_dirty())? For me the folio naming is consistent, but the page names suck; presumably PageSetDirty() and set_page_dirty()... yuk.
On Mon, Jan 22, 2024 at 05:26:00PM +0000, Ryan Roberts wrote: > On 22/01/2024 17:20, Matthew Wilcox wrote: > > On Mon, Jan 22, 2024 at 06:01:58PM +0100, David Hildenbrand wrote: > >>> And folio_mark_dirty() is doing more than just setting teh PG_dirty bit. In my > >>> equivalent change, as part of the contpte series, I've swapped set_page_dirty() > >>> for folio_mark_dirty(). > >> > >> Good catch, that should be folio_mark_dirty(). Let me send a fixup. > >> > >> (the difference in naming for both functions really is bad) > > > > It really is, and I don't know what to do about it. > > > > We need a function that literally just sets the flag. For every other > > flag, that's folio_set_FLAG. We can't use __folio_set_flag because that > > means "set the flag non-atomically". > > > > We need a function that does all of the work involved with tracking > > dirty folios. I chose folio_mark_dirty() to align with > > folio_mark_uptodate() (ie mark is not just 'set" but also "do some extra > > work"). > > > > But because we're converting from set_page_dirty(), the OBVIOUS rename > > is to folio_set_dirty(), which is WRONG. > > > > So we're in the part of the design space where the consistent naming and > > the-obvious-thing-to-do-is-wrong are in collision, and I do not have a > > good answer. > > > > Maybe we can call the first function _folio_set_dirty(), and we don't > > have a folio_set_dirty() at all? We don't have a folio_set_uptodate(), > > so there's some precedent there. > > Is there anything stopping us from renaming set_page_dirty() to > mark_page_dirty() (or page_mark_dirty())? For me the folio naming is consistent, > but the page names suck; presumably PageSetDirty() and set_page_dirty()... yuk. Well, laziness. There's about 150 places where we mention set_page_dirty() and all of them need to be converted to folio_mark_dirty(). I don't particularly like converting code twice; I get the impression it annoys people. The important thing is what does it look like when someone writes a new filesystem in 2030. I fear that they may get confused and call folio_set_dirty(), not realising that they should be calling folio_mark_dirty(). It doesn't help that btrfs have decided to introduce btrfs_folio_set_dirty(). I think MM people can afford to add a leading '_' to folio_set_dirty() so that's my current favourite option for fixing this mess.
On 22.01.24 18:20, Matthew Wilcox wrote: > On Mon, Jan 22, 2024 at 06:01:58PM +0100, David Hildenbrand wrote: >>> And folio_mark_dirty() is doing more than just setting teh PG_dirty bit. In my >>> equivalent change, as part of the contpte series, I've swapped set_page_dirty() >>> for folio_mark_dirty(). >> >> Good catch, that should be folio_mark_dirty(). Let me send a fixup. >> >> (the difference in naming for both functions really is bad) > > It really is, and I don't know what to do about it. > > We need a function that literally just sets the flag. For every other > flag, that's folio_set_FLAG. We can't use __folio_set_flag because that > means "set the flag non-atomically". > > We need a function that does all of the work involved with tracking > dirty folios. I chose folio_mark_dirty() to align with > folio_mark_uptodate() (ie mark is not just 'set" but also "do some extra > work"). > > But because we're converting from set_page_dirty(), the OBVIOUS rename > is to folio_set_dirty(), which is WRONG. And I made the same mistake at least also in "mm/huge_memory: page_remove_rmap() -> folio_remove_rmap_pmd()". I better double check all these so-simple-looking conversions that just went upstream. Interestingly, __split_huge_pmd_locked() used SetPageReferenced() instead of > > So we're in the part of the design space where the consistent naming and > the-obvious-thing-to-do-is-wrong are in collision, and I do not have a > good answer. > > Maybe we can call the first function _folio_set_dirty(), and we don't > have a folio_set_dirty() at all? We don't have a folio_set_uptodate(), > so there's some precedent there. Good question. This mark vs. set is confusing. We want some way to highlight that folio_set_dirty() is the one that we usually do not want to use.
On 22.01.24 18:34, David Hildenbrand wrote: > On 22.01.24 18:20, Matthew Wilcox wrote: >> On Mon, Jan 22, 2024 at 06:01:58PM +0100, David Hildenbrand wrote: >>>> And folio_mark_dirty() is doing more than just setting teh PG_dirty bit. In my >>>> equivalent change, as part of the contpte series, I've swapped set_page_dirty() >>>> for folio_mark_dirty(). >>> >>> Good catch, that should be folio_mark_dirty(). Let me send a fixup. >>> >>> (the difference in naming for both functions really is bad) >> >> It really is, and I don't know what to do about it. >> >> We need a function that literally just sets the flag. For every other >> flag, that's folio_set_FLAG. We can't use __folio_set_flag because that >> means "set the flag non-atomically". >> >> We need a function that does all of the work involved with tracking >> dirty folios. I chose folio_mark_dirty() to align with >> folio_mark_uptodate() (ie mark is not just 'set" but also "do some extra >> work"). >> >> But because we're converting from set_page_dirty(), the OBVIOUS rename >> is to folio_set_dirty(), which is WRONG. > > And I made the same mistake at least also in "mm/huge_memory: > page_remove_rmap() -> folio_remove_rmap_pmd()". > > I better double check all these so-simple-looking conversions that just > went upstream. > > Interestingly, __split_huge_pmd_locked() used SetPageReferenced() > instead of Forgot to delete that sentence. Anyhow, it's all confusing. My replacement in 91b2978a34807 from SetPageDirty -> folio_set_dirty() was correct. It only operates on anon folios, likely that's why folio_set_dirty() is okay there. Oh my.
diff --git a/mm/memory.c b/mm/memory.c index 6552ea27b0bfa..eda2181275d9b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1434,6 +1434,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent = ptep_get(pte); + struct folio *folio; struct page *page; if (pte_none(ptent)) @@ -1459,21 +1460,22 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } + folio = page_folio(page); delay_rmap = 0; - if (!PageAnon(page)) { + if (!folio_test_anon(folio)) { if (pte_dirty(ptent)) { - set_page_dirty(page); + folio_set_dirty(folio); if (tlb_delay_rmap(tlb)) { delay_rmap = 1; force_flush = 1; } } if (pte_young(ptent) && likely(vma_has_recency(vma))) - mark_page_accessed(page); + folio_mark_accessed(folio); } rss[mm_counter(page)]--; if (!delay_rmap) { - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(folio, page, vma); if (unlikely(page_mapcount(page) < 0)) print_bad_pte(vma, addr, ptent, page); } @@ -1489,6 +1491,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (is_device_private_entry(entry) || is_device_exclusive_entry(entry)) { page = pfn_swap_entry_to_page(entry); + folio = page_folio(page); if (unlikely(!should_zap_page(details, page))) continue; /* @@ -1500,8 +1503,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, WARN_ON_ONCE(!vma_is_anonymous(vma)); rss[mm_counter(page)]--; if (is_device_private_entry(entry)) - page_remove_rmap(page, vma, false); - put_page(page); + folio_remove_rmap_pte(folio, page, vma); + folio_put(folio); } else if (!non_swap_entry(entry)) { /* Genuine swap entry, hence a private anon page */ if (!should_zap_cows(details)) @@ -3220,10 +3223,10 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * threads. * * The critical issue is to order this - * page_remove_rmap with the ptp_clear_flush above. - * Those stores are ordered by (if nothing else,) + * folio_remove_rmap_pte() with the ptp_clear_flush + * above. Those stores are ordered by (if nothing else,) * the barrier present in the atomic_add_negative - * in page_remove_rmap. + * in folio_remove_rmap_pte(); * * Then the TLB flush in ptep_clear_flush ensures that * no process can access the old page before the @@ -3232,7 +3235,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * mapcount is visible. So transitively, TLBs to * old page will be flushed before it can be reused. */ - page_remove_rmap(vmf->page, vma, false); + folio_remove_rmap_pte(old_folio, vmf->page, vma); } /* Free the old page.. */ diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 4f559f4ddd217..604ddf08affed 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -55,7 +55,7 @@ static void tlb_flush_rmap_batch(struct mmu_gather_batch *batch, struct vm_area_ if (encoded_page_flags(enc)) { struct page *page = encoded_page_ptr(enc); - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(page_folio(page), page, vma); } } }