From patchwork Mon Dec 11 15:56:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176751 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7151965vqy; Mon, 11 Dec 2023 07:57:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IHwEldFLXyNfLfr5G/b1ZbeaNjoj4MjK+nj25DBlEq3JIGMdGuUtUMcJKBfPbAKYxngXL5d X-Received: by 2002:a05:6358:7241:b0:16e:5ae3:30f2 with SMTP id i1-20020a056358724100b0016e5ae330f2mr3246321rwa.4.1702310249988; Mon, 11 Dec 2023 07:57:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310249; cv=none; d=google.com; s=arc-20160816; b=gXyVrfoECCYjfgQWrBSH1HAFRGoMHJTH2P4fKxVqyqhs2qcBtTT4v4JjrviHx7uyRN ZaQIezunONazwN9BQSfi5vMKw7Dtb/I85O8nQRVEuG8xE4TmC9+J46diEpCO6hltaOwc TiTpgofqiypy/9xt/CZdAhINgkdLNwVR+z+u/wI0tGLvtgxQDZPWmTlSMfy7CXt1vXr9 2vYxLAKeDyGo83c7RWa/kDYUnPGPkRvCitfZVZOIjC1tUWJnXHPEK2t6iac52/KsWGfL uu98hhBE+GH1Jlek/CPvsknBm+GUEANbvuaifTDDcUgtKWzrZy+E75RWtRCjfqyyrbub WYMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=W84BJOqs6odi3w10Q8+aDyiUulYqaQMW45jL1XIh9aE=; fh=zW2d2UaZUoEUxa6MVPed9mRARVtfplPxKZRc44Ak2yc=; b=qJ9FihcdUmQYLDsOJIFq1T6QVMeAJelKYdlz1Zi2kKXVoJq6/HJMzKLh8k9CmrkHGp 2QAw/1FwXI5+l56h6R3UFd2OhJhOdqekd7s1fHKN/SmSqUS6Kw3Z/49O+FlCLYwoLLVy 6cvdCgocaAEpeaJ6XZjWB3j0mfsnBPBG7kvAsgiaQZpmmXCACq+ZMzIuipGBMqC8teWq gl6NDbpNvTSDgZc/i51DJdDnUaRLy3M1dfkWhyit+b88ABSkitV393ouqM7OfIjBKvYP 96sr6kd8NQqdSGImSFPVJcnZPlxoJ1zZT8ASjSxM/nhUusD3p9KQaJC6RhQwfKFr/HZr bnoQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=VKlsiXXc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id i12-20020a65484c000000b005b93bfe91f3si6164423pgs.577.2023.12.11.07.57.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:57:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=VKlsiXXc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 6D965809E228; Mon, 11 Dec 2023 07:57:27 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344406AbjLKP5B (ORCPT + 99 others); Mon, 11 Dec 2023 10:57:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33300 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344397AbjLKP47 (ORCPT ); Mon, 11 Dec 2023 10:56:59 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A76ABD for ; Mon, 11 Dec 2023 07:57:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310224; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W84BJOqs6odi3w10Q8+aDyiUulYqaQMW45jL1XIh9aE=; b=VKlsiXXcGcU3uNtDAIJfvVyrbje/0ybjKefyLZPszC5fy20hCADsU/MsmLsd2A3w0k5j0o UX5K2cuWOhaHm9J0Wd40wC3Qd/Em643zJqKKdALyQdYuueCIV7usnp2CVXvcmPW5l9YmSI 4oAzbangN0R3CeAfgSkh1Kg497CwXuM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-518-De3rUVWSN9y0uphkMhN_QQ-1; Mon, 11 Dec 2023 10:56:59 -0500 X-MC-Unique: De3rUVWSN9y0uphkMhN_QQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9BC94870829; Mon, 11 Dec 2023 15:56:58 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 457E61121306; Mon, 11 Dec 2023 15:56:56 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu , Muchun Song Subject: [PATCH v1 01/39] mm/rmap: rename hugepage_add* to hugetlb_add* Date: Mon, 11 Dec 2023 16:56:14 +0100 Message-ID: <20231211155652.131054-2-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:57:27 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001672490063590 X-GMAIL-MSGID: 1785001672490063590 Let's just call it "hugetlb_". Yes, it's all already inconsistent and confusing because we have a lot of "hugepage_" functions for legacy reasons. But "hugetlb" cannot possibly be confused with transparent huge pages, and it matches "hugetlb.c" and "folio_test_hugetlb()". So let's minimize confusion in rmap code. Reviewed-by: Muchun Song Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts Reviewed-by: Matthew Wilcox (Oracle) --- include/linux/rmap.h | 4 ++-- mm/hugetlb.c | 8 ++++---- mm/migrate.c | 4 ++-- mm/rmap.c | 8 ++++---- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index af6a32b6f3e7..0bfea866f39b 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -208,9 +208,9 @@ void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr, void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); -void hugepage_add_anon_rmap(struct folio *, struct vm_area_struct *, +void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address, rmap_t flags); -void hugepage_add_new_anon_rmap(struct folio *, struct vm_area_struct *, +void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); static inline void __page_dup_rmap(struct page *page, bool compound) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 6feb3e0630d1..305f3ca1dee6 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5285,7 +5285,7 @@ hugetlb_install_folio(struct vm_area_struct *vma, pte_t *ptep, unsigned long add pte_t newpte = make_huge_pte(vma, &new_folio->page, 1); __folio_mark_uptodate(new_folio); - hugepage_add_new_anon_rmap(new_folio, vma, addr); + hugetlb_add_new_anon_rmap(new_folio, vma, addr); if (userfaultfd_wp(vma) && huge_pte_uffd_wp(old)) newpte = huge_pte_mkuffd_wp(newpte); set_huge_pte_at(vma->vm_mm, addr, ptep, newpte, sz); @@ -5988,7 +5988,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* Break COW or unshare */ huge_ptep_clear_flush(vma, haddr, ptep); page_remove_rmap(&old_folio->page, vma, true); - hugepage_add_new_anon_rmap(new_folio, vma, haddr); + hugetlb_add_new_anon_rmap(new_folio, vma, haddr); if (huge_pte_uffd_wp(pte)) newpte = huge_pte_mkuffd_wp(newpte); set_huge_pte_at(mm, haddr, ptep, newpte, huge_page_size(h)); @@ -6277,7 +6277,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, goto backout; if (anon_rmap) - hugepage_add_new_anon_rmap(folio, vma, haddr); + hugetlb_add_new_anon_rmap(folio, vma, haddr); else page_dup_file_rmap(&folio->page, true); new_pte = make_huge_pte(vma, &folio->page, ((vma->vm_flags & VM_WRITE) @@ -6732,7 +6732,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, if (folio_in_pagecache) page_dup_file_rmap(&folio->page, true); else - hugepage_add_new_anon_rmap(folio, dst_vma, dst_addr); + hugetlb_add_new_anon_rmap(folio, dst_vma, dst_addr); /* * For either: (1) CONTINUE on a non-shared VMA, or (2) UFFDIO_COPY diff --git a/mm/migrate.c b/mm/migrate.c index 35a88334bb3c..4cb849fa0dd2 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -249,8 +249,8 @@ static bool remove_migration_pte(struct folio *folio, pte = arch_make_huge_pte(pte, shift, vma->vm_flags); if (folio_test_anon(folio)) - hugepage_add_anon_rmap(folio, vma, pvmw.address, - rmap_flags); + hugetlb_add_anon_rmap(folio, vma, pvmw.address, + rmap_flags); else page_dup_file_rmap(new, true); set_huge_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte, diff --git a/mm/rmap.c b/mm/rmap.c index 846fc79f3ca9..80d42c31281a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2625,8 +2625,8 @@ void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc) * * RMAP_COMPOUND is ignored. */ -void hugepage_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, - unsigned long address, rmap_t flags) +void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, + unsigned long address, rmap_t flags) { VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); @@ -2637,8 +2637,8 @@ void hugepage_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, PageAnonExclusive(&folio->page), folio); } -void hugepage_add_new_anon_rmap(struct folio *folio, - struct vm_area_struct *vma, unsigned long address) +void hugetlb_add_new_anon_rmap(struct folio *folio, + struct vm_area_struct *vma, unsigned long address) { BUG_ON(address < vma->vm_start || address >= vma->vm_end); /* increment count (starts at -1) */ From patchwork Mon Dec 11 15:56:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176750 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7151876vqy; Mon, 11 Dec 2023 07:57:25 -0800 (PST) X-Google-Smtp-Source: AGHT+IHQRXorpm5XPxyXgAmM0HNV/hZxMtusLb1rRXTO2UBeHkhFoUMwThfb5Zbt3lgLr6HSAlQI X-Received: by 2002:a17:902:d58e:b0:1d0:b139:2c69 with SMTP id k14-20020a170902d58e00b001d0b1392c69mr2069050plh.64.1702310244327; Mon, 11 Dec 2023 07:57:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310244; cv=none; d=google.com; s=arc-20160816; b=Ib8vWnxnpRrurMkCUhTMBgnRC4KBeJRZD0H5OKZkRBPdrue6hsQVxH7NQu43TN5b0r 6cGCx6u0JUzBZ/d6YK/+AVEj9mB6KR0419O55iq/0j+sJ4JW8e/g4hN2BOWdfLHbyOS1 qx/Io41ncPOm8teLD7+r+aqWZdcIvE/vuiZoS9nax/wJbwMLpYw59rJvizYmmIfLvZfg +ooRnRfrnE80xdhDbpK6btFjXDI5chYRQQcaIIRnrJcKbtM8/S6jV54eiCiq6X3B3I3A dIryW6KdJcUgiRmxdFxmgIHzHCYXNpPOGr9FPj/POErGmICjow3RDvG4d0FGsCjqS1jM XDcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=K3NaOeaUC8VPVD4C/LE70HnwI+FiYJElEMKNgi9W/YQ=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=YysGSZ6UdMv6B6s1OZEHeQRvajJdLGSexbSWu77Qysy037z0Yfdsg1nsw034AeTpV9 1onujip3hIWHsIhtlppz8fRiY7VjfkjwCBzlrMeJMnF1tqaQDxFyNYPExTmKQ41PHn2f l0O04AEb5L//LJYp9+pBHFRV0IGnfDTer7s/5/sPckolR3d6a318hbaJMo4wBC9ynXDz 9Rl7RAQNvtIELoMdD2x6iHFlLVbJ5L17okoQtQOquydTE++i/S/O1D7rUhO699M7AL/1 83gX+rEW3IBCLrGPU1Jfzg9x4v9BtLZg2kGTnmATHmK7u/yOzuq6TTtVFJ7Wz0eL5oJD CdrQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=QR+woqSk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id pm6-20020a17090b3c4600b00278f6d616aasi6422651pjb.71.2023.12.11.07.57.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:57:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=QR+woqSk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id D049C809E233; Mon, 11 Dec 2023 07:57:18 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344417AbjLKP5D (ORCPT + 99 others); Mon, 11 Dec 2023 10:57:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33302 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344398AbjLKP5A (ORCPT ); Mon, 11 Dec 2023 10:57:00 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E8303B3 for ; Mon, 11 Dec 2023 07:57:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310226; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K3NaOeaUC8VPVD4C/LE70HnwI+FiYJElEMKNgi9W/YQ=; b=QR+woqSk+H4IhH5Wkj+Q4HW9MsCKASsU8nPaxWqOM05SM2NnzhppxMZAPRl+2Ag5EBfe7U vIz7EifORz6NWz5EnG1Z8O1AtyW6RrTZIQA+BJjfMQM7LLez2YHDioHiOFBQCCljukU0q9 oOdtLuQcSZOxdzoGM5CX1rwNfPp96Sk= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-88-dxPz7bySN2KzjJFO1gbBAg-1; Mon, 11 Dec 2023 10:57:02 -0500 X-MC-Unique: dxPz7bySN2KzjJFO1gbBAg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BF1A0280A9BB; Mon, 11 Dec 2023 15:57:01 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id EA3491121306; Mon, 11 Dec 2023 15:56:58 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 02/39] mm/rmap: introduce and use hugetlb_remove_rmap() Date: Mon, 11 Dec 2023 16:56:15 +0100 Message-ID: <20231211155652.131054-3-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:57:19 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001666451320931 X-GMAIL-MSGID: 1785001666451320931 hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. Let's introduce and use hugetlb_remove_rmap() and remove the hugetlb code from page_remove_rmap(). This effectively removes one check on the small-folio path as well. Note: all possible candidates that need care are page_remove_rmap() that pass compound=true. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts Reviewed-by: Matthew Wilcox (Oracle) --- include/linux/rmap.h | 5 +++++ mm/hugetlb.c | 4 ++-- mm/rmap.c | 17 ++++++++--------- 3 files changed, 15 insertions(+), 11 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 0bfea866f39b..d85bd1d4de04 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -213,6 +213,11 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +static inline void hugetlb_remove_rmap(struct folio *folio) +{ + atomic_dec(&folio->_entire_mapcount); +} + static inline void __page_dup_rmap(struct page *page, bool compound) { if (compound) { diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 305f3ca1dee6..ef48ae673890 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5676,7 +5676,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, make_pte_marker(PTE_MARKER_UFFD_WP), sz); hugetlb_count_sub(pages_per_huge_page(h), mm); - page_remove_rmap(page, vma, true); + hugetlb_remove_rmap(page_folio(page)); spin_unlock(ptl); tlb_remove_page_size(tlb, page, huge_page_size(h)); @@ -5987,7 +5987,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* Break COW or unshare */ huge_ptep_clear_flush(vma, haddr, ptep); - page_remove_rmap(&old_folio->page, vma, true); + hugetlb_remove_rmap(old_folio); hugetlb_add_new_anon_rmap(new_folio, vma, haddr); if (huge_pte_uffd_wp(pte)) newpte = huge_pte_mkuffd_wp(newpte); diff --git a/mm/rmap.c b/mm/rmap.c index 80d42c31281a..4e60c1f38eaa 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1482,13 +1482,6 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, VM_BUG_ON_PAGE(compound && !PageHead(page), page); - /* Hugetlb pages are not counted in NR_*MAPPED */ - if (unlikely(folio_test_hugetlb(folio))) { - /* hugetlb pages are always mapped with pmds */ - atomic_dec(&folio->_entire_mapcount); - return; - } - /* Is page being unmapped by PTE? Is this its last map to be removed? */ if (likely(!compound)) { last = atomic_add_negative(-1, &page->_mapcount); @@ -1846,7 +1839,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, dec_mm_counter(mm, mm_counter_file(&folio->page)); } discard: - page_remove_rmap(subpage, vma, folio_test_hugetlb(folio)); + if (unlikely(folio_test_hugetlb(folio))) + hugetlb_remove_rmap(folio); + else + page_remove_rmap(subpage, vma, false); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); @@ -2199,7 +2195,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, */ } - page_remove_rmap(subpage, vma, folio_test_hugetlb(folio)); + if (unlikely(folio_test_hugetlb(folio))) + hugetlb_remove_rmap(folio); + else + page_remove_rmap(subpage, vma, false); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); From patchwork Mon Dec 11 15:56:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176749 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7151920vqy; Mon, 11 Dec 2023 07:57:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IHZ+VrCmGpPMHYo4l0ULQWdpXNE0b99CeENTp5qnQjH8d59feoG1gdjA43Tx/1h7DB4dO0e X-Received: by 2002:a17:903:2307:b0:1d0:6ffd:610d with SMTP id d7-20020a170903230700b001d06ffd610dmr5465383plh.47.1702310248254; Mon, 11 Dec 2023 07:57:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310248; cv=none; d=google.com; s=arc-20160816; b=rKWCMyWq4zwlwFgjRM8HPW7ZI+QERzII/avl1stkxdS7TXSVEwrdKRWp+C8mPAVb73 ImCoIg5cY9UMKpBrLJQc/70imjaZX1hxoZKpuH419BTedTHwtTZEWL6myBEiJDU6ImBS Vvn6KJjTuDczfntdf77eOk9nx8zseEJjBVuN/BX8LaRcu/6gNvWdqIM6cLmZhEfQXFbL u5qG76OVb/dxA9QsA3Kc3gIOD3DA3pf4wtEzX/ku63n0l4fKQ0eo+IBXhlLkFUFNzTw/ rVWBMVSsG6NrNygnKO9q3BwGd/gHlHeq90rLDFLTg9p2z9vXzoYxVmzQZ91g5A+NVIO9 OfIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eQFoT8xrLd9LUZRih2ZF29493QEHiYzpAeVB57UisD0=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=k5nmVoHtQ6hkZBxbxFNOEdfYmfZlFDNpkefL8u3B9PW1dIh7ROhUmMQiq9afT0d53U SbC6ebWALoFJO8mSGgXPhhotgj3AgYNAu+tMU0lDWC20aL2SXhUitBbSS20jDjWSlbB8 vBePQxdOXkRjamNCJZd7qlNvN3L6XLHD4b7iiOe91mjGQWQgnfCP+tIXDXApdgqs6Did NYfiHFEOhveG4+j7ReIiTtQjf+v9WeY+89m4N4DLdtDZY8+JqHKirTwRMB+sJeff8DLr 8pO/D+opGCtNkpWj+8Y2YcKeFGb/gcwNbsDqGO/iBh459CDp5BvywADNSOFDG524irgi BFkw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="cK/oeP1o"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id j4-20020a170902690400b001d3156ab971si2234423plk.389.2023.12.11.07.57.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:57:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="cK/oeP1o"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 5599C8075287; Mon, 11 Dec 2023 07:57:21 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344463AbjLKP5M (ORCPT + 99 others); Mon, 11 Dec 2023 10:57:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55090 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344472AbjLKP5K (ORCPT ); Mon, 11 Dec 2023 10:57:10 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA6F6F3 for ; Mon, 11 Dec 2023 07:57:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310230; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eQFoT8xrLd9LUZRih2ZF29493QEHiYzpAeVB57UisD0=; b=cK/oeP1ouam/D+5+S4uhtaHIB7DBFVudnPTmX7aEjMpRgaClH695MluwewQAhP22ob9dVc GFPnwk9nS1q1Y7huvCaxx2lx//1oG1beysZJP1TcN6MFe1QsOAzCZrIlcwt7FSXYHMoy2j pKCjW4VFUrzdyRZFWKKxH+KecZ79IJw= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-620-xxnUSZcjMXupZYk_PaLjmg-1; Mon, 11 Dec 2023 10:57:05 -0500 X-MC-Unique: xxnUSZcjMXupZYk_PaLjmg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8D0001C01B3A; Mon, 11 Dec 2023 15:57:04 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2EA5E1121306; Mon, 11 Dec 2023 15:57:01 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 03/39] mm/rmap: introduce and use hugetlb_add_file_rmap() Date: Mon, 11 Dec 2023 16:56:16 +0100 Message-ID: <20231211155652.131054-4-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:57:21 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001670850957064 X-GMAIL-MSGID: 1785001670850957064 hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. Right now we're using page_dup_file_rmap() in some cases where "ordinary" rmap code would have used page_add_file_rmap(). So let's introduce and use hugetlb_add_file_rmap() instead. We won't be adding a "hugetlb_dup_file_rmap()" functon for the fork() case, as it would be doing the same: "dup" is just an optimization for "add". What remains is a single page_dup_file_rmap() call in fork() code. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- include/linux/rmap.h | 7 +++++++ mm/hugetlb.c | 6 +++--- mm/migrate.c | 2 +- 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index d85bd1d4de04..91178d1aa028 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -213,6 +213,13 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +static inline void hugetlb_add_file_rmap(struct folio *folio) +{ + VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); + + atomic_inc(&folio->_entire_mapcount); +} + static inline void hugetlb_remove_rmap(struct folio *folio) { atomic_dec(&folio->_entire_mapcount); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index ef48ae673890..57e898187931 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5408,7 +5408,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, * sleep during the process. */ if (!folio_test_anon(pte_folio)) { - page_dup_file_rmap(&pte_folio->page, true); + hugetlb_add_file_rmap(pte_folio); } else if (page_try_dup_anon_rmap(&pte_folio->page, true, src_vma)) { pte_t src_pte_old = entry; @@ -6279,7 +6279,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, if (anon_rmap) hugetlb_add_new_anon_rmap(folio, vma, haddr); else - page_dup_file_rmap(&folio->page, true); + hugetlb_add_file_rmap(folio); new_pte = make_huge_pte(vma, &folio->page, ((vma->vm_flags & VM_WRITE) && (vma->vm_flags & VM_SHARED))); /* @@ -6730,7 +6730,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, goto out_release_unlock; if (folio_in_pagecache) - page_dup_file_rmap(&folio->page, true); + hugetlb_add_file_rmap(folio); else hugetlb_add_new_anon_rmap(folio, dst_vma, dst_addr); diff --git a/mm/migrate.c b/mm/migrate.c index 4cb849fa0dd2..de9d94b99ab7 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -252,7 +252,7 @@ static bool remove_migration_pte(struct folio *folio, hugetlb_add_anon_rmap(folio, vma, pvmw.address, rmap_flags); else - page_dup_file_rmap(new, true); + hugetlb_add_file_rmap(folio); set_huge_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte, psize); } else From patchwork Mon Dec 11 15:56:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176755 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7152159vqy; Mon, 11 Dec 2023 07:57:49 -0800 (PST) X-Google-Smtp-Source: AGHT+IEpwo51Axa0DrO09OjHYZEELlZgUg3sjCdqZERDEguvzu8M2ASIS5axX+gqVczCEl0v/mXD X-Received: by 2002:aa7:8586:0:b0:6ce:f651:283f with SMTP id w6-20020aa78586000000b006cef651283fmr1147423pfn.25.1702310269365; Mon, 11 Dec 2023 07:57:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310269; cv=none; d=google.com; s=arc-20160816; b=EOqm0JUx+iflZFbJod0RmZFE9KaLCo+gA6gcwhj5ukZrnjMpC2sU54Usyy1nbfvo1U 0Q6GOXCaBrhcxVBpRSZnX68JLOi7G9IOR/SNryhoBfIbhtqwXm74B6O7ToSsrauvK6xw WEGD55RZYJtm3f1QGCOLPZUVWd4DF6NUSvq2dSOd6KD/q2GUdvURSKaL/aepGpX3bhKv IKTPQkvcIVoMHs117xdj6HM7sNxNBsZ+HMEJJOOFNtdZ+/43TSSrhW1DJfT94+GtZUWu Z/NXMFt5FYTChZnXrphylfP5ZHNemP5OXYwVu7Yty2dc+SNIjGakTuhI13NNgBvcrCfO OMFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=U+sz0aAg3eDeTtLeKtB884YEpFTON3wSVy4B1/EJUUg=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=bd609saHwG8nVmBFJZ1oVIIM/4Mf/kPNeMnkJ8hEuk1vwbwCGS4KEpm3jSojtDr+qx iP7qOWKpRxuDOPn8suhQuou59rYHh6bOkkVLVUmpCWdJapBYeV0AR0HcDyG0h+er6pp6 3jo9qvAAqBfX5ErsEEAlgJpcU7KecYABrcv7k4FRKb7w05QEl78fygpgi0HB2rBiPJ/F euqLTrlTqgCIlKlSxOHcAMSyJczBMtiHvGZP7yk+Of5CQDmEi3zQRppWjqyLF91fXtOz RbWsjbgtG4w3nfx7OmFY4ptV2Votstc8F5xkHfZIsu1U4vmHoNWl4rBZC4aFF+7bJbLv TyOg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Y4WJL66I; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id m24-20020a635818000000b005a1d8816535si6143433pgb.5.2023.12.11.07.57.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:57:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Y4WJL66I; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 211C280A1854; Mon, 11 Dec 2023 07:57:47 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344528AbjLKP5d (ORCPT + 99 others); Mon, 11 Dec 2023 10:57:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344469AbjLKP5U (ORCPT ); Mon, 11 Dec 2023 10:57:20 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 187ACB4 for ; Mon, 11 Dec 2023 07:57:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310246; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=U+sz0aAg3eDeTtLeKtB884YEpFTON3wSVy4B1/EJUUg=; b=Y4WJL66I/uvnjCdrUU8U2eVG7KPFI6qkvGDWid4xDaZEmR4r3dViBotRMkWD3imFLdlyGN icMzBoqhqs8ppNX8x2skhE3l87nxwBi8xxLAappdQ3doQX56Y9i7vGb1fJP4/W0btFHFl8 a6FhYHc+4h3Ywi61YiDaZ88uQ1umM8E= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-509-x5EZWnqnPL-8WFVv6DDwvA-1; Mon, 11 Dec 2023 10:57:23 -0500 X-MC-Unique: x5EZWnqnPL-8WFVv6DDwvA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9899D101E156; Mon, 11 Dec 2023 15:57:07 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id EAAC51121306; Mon, 11 Dec 2023 15:57:04 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 04/39] mm/rmap: introduce and use hugetlb_try_dup_anon_rmap() Date: Mon, 11 Dec 2023 16:56:17 +0100 Message-ID: <20231211155652.131054-5-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:57:47 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001693046785368 X-GMAIL-MSGID: 1785001693046785368 hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. So let's introduce and use hugetlb_try_dup_anon_rmap() to make all hugetlb handling use dedicated hugetlb_* rmap functions. Note that is_device_private_page() does not apply to hugetlb. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- include/linux/mm.h | 12 +++++++++--- include/linux/rmap.h | 15 +++++++++++++++ mm/hugetlb.c | 3 +-- 3 files changed, 25 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index b72bf25a45cf..ae547b62f325 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1964,15 +1964,21 @@ static inline bool page_maybe_dma_pinned(struct page *page) * * The caller has to hold the PT lock and the vma->vm_mm->->write_protect_seq. */ -static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma, - struct page *page) +static inline bool folio_needs_cow_for_dma(struct vm_area_struct *vma, + struct folio *folio) { VM_BUG_ON(!(raw_read_seqcount(&vma->vm_mm->write_protect_seq) & 1)); if (!test_bit(MMF_HAS_PINNED, &vma->vm_mm->flags)) return false; - return page_maybe_dma_pinned(page); + return folio_maybe_dma_pinned(folio); +} + +static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma, + struct page *page) +{ + return folio_needs_cow_for_dma(vma, page_folio(page)); } /** diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 91178d1aa028..ca42b3db5688 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -213,6 +213,21 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +/* See page_try_dup_anon_rmap() */ +static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, + struct vm_area_struct *vma) +{ + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + + if (PageAnonExclusive(&folio->page)) { + if (unlikely(folio_needs_cow_for_dma(vma, folio))) + return -EBUSY; + ClearPageAnonExclusive(&folio->page); + } + atomic_inc(&folio->_entire_mapcount); + return 0; +} + static inline void hugetlb_add_file_rmap(struct folio *folio) { VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 57e898187931..378e460a6ab4 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5409,8 +5409,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, */ if (!folio_test_anon(pte_folio)) { hugetlb_add_file_rmap(pte_folio); - } else if (page_try_dup_anon_rmap(&pte_folio->page, - true, src_vma)) { + } else if (hugetlb_try_dup_anon_rmap(pte_folio, src_vma)) { pte_t src_pte_old = entry; struct folio *new_folio; From patchwork Mon Dec 11 15:56:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176752 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7151956vqy; Mon, 11 Dec 2023 07:57:31 -0800 (PST) X-Google-Smtp-Source: AGHT+IGJx61IaPUEUL7pNSNl4R2BKw+x4rpfy3xO1yVW94INcY1rTn8p2cDYPJyLP0hy16BX0QkE X-Received: by 2002:a17:90a:ce14:b0:286:6f15:4bec with SMTP id f20-20020a17090ace1400b002866f154becmr2292045pju.54.1702310249678; Mon, 11 Dec 2023 07:57:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310249; cv=none; d=google.com; s=arc-20160816; b=ZisSqVOPiAz0OOO6oOWg2BQozRctrwjOw/9l+s1/uo9tS7utIM8daPdH0bvzZIHQ3b G0a5fOupO8e7EGmCu+CEgH7kq9ZG8x6Xj9VeRYuf3VI7Pl/JgEXFNfF/GW/KDTUPyFK2 KE2hFmLICc905guYqyenhlofuzgY5gfv5ItaAMhCHSrYvvyjTRXASUL32/bmTA02OIOa gcJ5hQVBY4KaFd+o9BaaJB1rLgwqwJ6Z3n9lEL7D6Zxx2hH0GKKpGzJlIi7FMH7ZGS2g krTtaDvArK9JRss4Nep+g7WUcmg5iIjNeuzE4b3QUbpeR6aV85HBDSwej/iXvYOzPa/l DXjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=NEVRdWv5MmhzBcZ7w5tNkRp7uojKujUjmq26NniU0lw=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=y4cysPVIt0e+13CiUvyUtJlhrJt1VUlVbUf4hbGEq+LkCEzGj0QIweL2wesvQV5QH2 3cYCtBX+ifvzSHWb+dJPd+hv9UXp7TORa/wdp9bAwwXRnfTrE6ZM1VfQzOQC7CWyRQlG +wtK1Qfap0bcb2grqpP6L01bhCJx9N1Gd+yTa7Od6L9rS+kT5jtqCAGSjQxzXF7Td66L u53A7LnoTpYNzPumpdLK8sMsVqCbZU6ponElcso03yF7CFNAXJGOCfM9zgHnR9LAxZ7j 7Z4wla0CxCRzv1pkakIbd/XgVE5F7rz96w7JiCAXF1LC3G0rW7IW8j86Jo5jkdGFis56 ESFg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bEMFaQXf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id q14-20020a17090a304e00b002839749088dsi6306959pjl.40.2023.12.11.07.57.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:57:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bEMFaQXf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 84311809FC83; Mon, 11 Dec 2023 07:57:28 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344462AbjLKP5O (ORCPT + 99 others); Mon, 11 Dec 2023 10:57:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55062 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344477AbjLKP5K (ORCPT ); Mon, 11 Dec 2023 10:57:10 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31CFB10B for ; Mon, 11 Dec 2023 07:57:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310235; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NEVRdWv5MmhzBcZ7w5tNkRp7uojKujUjmq26NniU0lw=; b=bEMFaQXfk0hztuuzstaaJvONUWUpQgUpBaijog4k4rH4OtdQTi4AMKwa+xzwuCEKreJVvR 1FyCLgS5Rz85I1SDOQ+0n/YDbXe3HXnngsZ2YGDc+DEJCJF80MUttMVC5x/JYXOECsUwD5 q+VmeCIylVb/jts0COGyhPB3cCUL4M0= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-173-qBz2N7eWPmuxhrmCuHZ-Dw-1; Mon, 11 Dec 2023 10:57:12 -0500 X-MC-Unique: qBz2N7eWPmuxhrmCuHZ-Dw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3226A28078D6; Mon, 11 Dec 2023 15:57:10 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 071301121306; Mon, 11 Dec 2023 15:57:07 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 05/39] mm/rmap: introduce and use hugetlb_try_share_anon_rmap() Date: Mon, 11 Dec 2023 16:56:18 +0100 Message-ID: <20231211155652.131054-6-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:57:28 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001672162068979 X-GMAIL-MSGID: 1785001672162068979 hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. So let's introduce and use hugetlb_try_dup_anon_rmap() to make all hugetlb handling use dedicated hugetlb_* rmap functions. Note that try_to_unmap_one() does not need care. Easy to spot because among all that nasty hugetlb special-casing in that function, we're not using set_huge_pte_at() on the anon path -- well, and that code assumes that we would want to swapout. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- include/linux/rmap.h | 23 +++++++++++++++++++++++ mm/rmap.c | 15 ++++++++++----- 2 files changed, 33 insertions(+), 5 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index ca42b3db5688..4c0650e9f6db 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -228,6 +228,29 @@ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, return 0; } +/* See page_try_share_anon_rmap() */ +static inline int hugetlb_try_share_anon_rmap(struct folio *folio) +{ + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + VM_WARN_ON_FOLIO(!PageAnonExclusive(&folio->page), folio); + + /* Paired with the memory barrier in try_grab_folio(). */ + if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) + smp_mb(); + + if (unlikely(folio_maybe_dma_pinned(folio))) + return -EBUSY; + ClearPageAnonExclusive(&folio->page); + + /* + * This is conceptually a smp_wmb() paired with the smp_rmb() in + * gup_must_unshare(). + */ + if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) + smp_mb__after_atomic(); + return 0; +} + static inline void hugetlb_add_file_rmap(struct folio *folio) { VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); diff --git a/mm/rmap.c b/mm/rmap.c index 4e60c1f38eaa..e210ac1b73de 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2147,13 +2147,18 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, !anon_exclusive, subpage); /* See page_try_share_anon_rmap(): clear PTE first. */ - if (anon_exclusive && - page_try_share_anon_rmap(subpage)) { - if (folio_test_hugetlb(folio)) + if (folio_test_hugetlb(folio)) { + if (anon_exclusive && + hugetlb_try_share_anon_rmap(folio)) { set_huge_pte_at(mm, address, pvmw.pte, pteval, hsz); - else - set_pte_at(mm, address, pvmw.pte, pteval); + ret = false; + page_vma_mapped_walk_done(&pvmw); + break; + } + } else if (anon_exclusive && + page_try_share_anon_rmap(subpage)) { + set_pte_at(mm, address, pvmw.pte, pteval); ret = false; page_vma_mapped_walk_done(&pvmw); break; From patchwork Mon Dec 11 15:56:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176756 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7152167vqy; Mon, 11 Dec 2023 07:57:52 -0800 (PST) X-Google-Smtp-Source: AGHT+IHnybJd8XlbA/FfRPuAHM1yy4+JYUCGFjIh6r+bZInkYVhilkbdafw8ydsqE+dmiRoXmQ0D X-Received: by 2002:a05:6a20:e121:b0:18f:e697:19d7 with SMTP id kr33-20020a056a20e12100b0018fe69719d7mr2850864pzb.55.1702310272071; Mon, 11 Dec 2023 07:57:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310272; cv=none; d=google.com; s=arc-20160816; b=oboQdK3l5WZruVi2/Gfl8Ky+ZSY8OXjLABD2x6oAF/M6VjR7ZGPnwJSSzL7L4bgkgg kN7TKTYjxZEinftdSku0MookQ4TpxOo6NQGqCXA9OJ9KCCFEGz/dad7NEA9v9BTpx/gl LN9f+AB3c5BGZyRuy1CU7ne5+SUj1t9s/iNziU5Sakiu8VWhpvWsudYhtH62LiHe3go8 1WPFpW0EHETQtff5f/64bL/zUfxaArGXAly7x3tvCIf0Nj2KzQMN7W8hlIyl4YDSCDP3 uv3pTXDYO/wAtrLeuZ2u25Sfox8gSWLzFWR7/syZvhL6aGthLKwm4GLNS6qUnbDUJVbh n0xg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=lrIUJZ6x1JeaI9Xj6nWMq9GTptSO/huW1R5CZYvfzvo=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=M16dHs4wbK+Cc/GpnqLspEcBVkKJ23gEphIpbMWq4IW+hmay7jVQJGZQ/6qkyq6pVR Ulr55dvEbt5e8J8nBNEQyTK3riqo/BRCV6q1VHuv8yruYJvE/6qtPXhLXjEF1t47rCKA p2Zyhcz2SEdYHzxPSOzyfoyEiN5MTU+d+OAx79XndGy39le+7j0Gqi9AcINi/ITYCpBx zwBgdooksXN/9050fhOzBkFlrrpROfLQBC0t9HmVYG1GbZlmp1gXY+h/Tb9VktpTpcPp RRwcoApxStM2M5ACDlfhraVwRUrLK2EFBRAQQyFW88GitZV/uiVqNXIGeESsAbpppvBZ PJWQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=MYMPNIJr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id j11-20020a056a00234b00b006cbf67abff9si6317315pfj.269.2023.12.11.07.57.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:57:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=MYMPNIJr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id E6D22809E8D2; Mon, 11 Dec 2023 07:57:46 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344489AbjLKP5X (ORCPT + 99 others); Mon, 11 Dec 2023 10:57:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51772 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344478AbjLKP5O (ORCPT ); Mon, 11 Dec 2023 10:57:14 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A5650E5 for ; Mon, 11 Dec 2023 07:57:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310239; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lrIUJZ6x1JeaI9Xj6nWMq9GTptSO/huW1R5CZYvfzvo=; b=MYMPNIJrFkWKlQjd4Nj5jE+WjyI259JNrFy53IVks/m3evAkrhw8lZTrUeX2z3exmK+kFw mL8VV33svvzlsfAZYwfEzhvu1ooSuyMNO0Ww0bqrOOlGDPM94FzW3o6mWYiW23H60mxywc 6yPabYI+wol5Q5Oj1nH9N5UTGhWBHtM= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-213-1tzqxM8YMK2PmJTvvwdWZQ-1; Mon, 11 Dec 2023 10:57:13 -0500 X-MC-Unique: 1tzqxM8YMK2PmJTvvwdWZQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 864033C28644; Mon, 11 Dec 2023 15:57:12 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6A5831121306; Mon, 11 Dec 2023 15:57:10 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 06/39] mm/rmap: add hugetlb sanity checks Date: Mon, 11 Dec 2023 16:56:19 +0100 Message-ID: <20231211155652.131054-7-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:57:47 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001696055580982 X-GMAIL-MSGID: 1785001696055580982 Let's make sure we end up with the right folios in the right functions. Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- include/linux/rmap.h | 7 +++++++ mm/rmap.c | 6 ++++++ 2 files changed, 13 insertions(+) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 4c0650e9f6db..e3857d26b944 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -217,6 +217,7 @@ void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, struct vm_area_struct *vma) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); if (PageAnonExclusive(&folio->page)) { @@ -231,6 +232,7 @@ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, /* See page_try_share_anon_rmap() */ static inline int hugetlb_try_share_anon_rmap(struct folio *folio) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); VM_WARN_ON_FOLIO(!PageAnonExclusive(&folio->page), folio); @@ -253,6 +255,7 @@ static inline int hugetlb_try_share_anon_rmap(struct folio *folio) static inline void hugetlb_add_file_rmap(struct folio *folio) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); atomic_inc(&folio->_entire_mapcount); @@ -260,11 +263,15 @@ static inline void hugetlb_add_file_rmap(struct folio *folio) static inline void hugetlb_remove_rmap(struct folio *folio) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); + atomic_dec(&folio->_entire_mapcount); } static inline void __page_dup_rmap(struct page *page, bool compound) { + VM_WARN_ON(folio_test_hugetlb(page_folio(page))); + if (compound) { struct folio *folio = (struct folio *)page; diff --git a/mm/rmap.c b/mm/rmap.c index e210ac1b73de..41597da14f26 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1343,6 +1343,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, { int nr = folio_nr_pages(folio); + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); VM_BUG_ON_VMA(address < vma->vm_start || address + (nr << PAGE_SHIFT) > vma->vm_end, vma); __folio_set_swapbacked(folio); @@ -1395,6 +1396,7 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, unsigned int nr_pmdmapped = 0, first; int nr = 0; + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(compound && !folio_test_pmd_mappable(folio), folio); /* Is page being mapped by PTE? Is this its first map to be added? */ @@ -1480,6 +1482,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, bool last; enum node_stat_item idx; + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); VM_BUG_ON_PAGE(compound && !PageHead(page), page); /* Is page being unmapped by PTE? Is this its last map to be removed? */ @@ -2632,6 +2635,7 @@ void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc) void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address, rmap_t flags) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); atomic_inc(&folio->_entire_mapcount); @@ -2644,6 +2648,8 @@ void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, void hugetlb_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); + BUG_ON(address < vma->vm_start || address >= vma->vm_end); /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); From patchwork Mon Dec 11 15:56:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176753 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7152105vqy; Mon, 11 Dec 2023 07:57:46 -0800 (PST) X-Google-Smtp-Source: AGHT+IEj7GetqXbLlUT0M3KBvF3MaMc0Vu+rIh7l0b3Acppmt7FY7P+v1Qj6BALBohq10vKmF2SP X-Received: by 2002:a17:90b:4b41:b0:286:6cc1:5fdb with SMTP id mi1-20020a17090b4b4100b002866cc15fdbmr2212268pjb.94.1702310265525; Mon, 11 Dec 2023 07:57:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310265; cv=none; d=google.com; s=arc-20160816; b=bPGZ+GrCGPbXZGurGKUs8zhE/sZTtZGT24pEMJvZeBAALAG2OqN4EAPOD6fPD8gNZa z1ISQrio+CDi/7hjMpKOEiP5gKqPz+mAg/qkWJcFkCX5awzIFfpbxw9yHZ4kqn4QUAtQ 38rjBFBCQOKVWhAmFUdwS9O2sntwgHSrAT0Mc9AQuF34FkafF9tzzLmenqKKhY/p+4sA T1rcGnwH6DyLEWfxg3EYyar6KxN01vnhMhLqjlBJGzrj92n3/xGYiSdLFx2376GnTMKy nt/1EpxE82phNhkKZjiT/F0iJMtZT6c+TwOoovNWdNM/CWxsdrUkQyWEI4ZJaznxqkP9 y9yA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=IYkHAajZElCwYCEwnUcbPY1wmlYkOiWYX0LP3vDIRKY=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=h9KHxkQ1sGjYiVNwp/0TQruEBFD5Eh18WLcB3dhGVr+/cDtyMZHzX6fKvdtgmVpck+ kkvLmTGnvjSJqv9APVtbpTldZlEKbxH9umf7B3pX7o2cWyWvAfvJlIR+zhKeD7ioSmM7 cWWeeEzIQ0qsQmdXcIwsyzUBdkC6eNEiE66UXd9F1aT4Gt2sAEShUIdANcJY+u2ykyIF ce2MITGBxZSpC/vdRCZp4EtEhw9eNlhz19K/8EXwcouVQz2t72oMFAZCq4KyFZZqH8Br Nv8RSxbqCN8uJashC9q07wKjtsfNa8eIPFmBs6sd+4f9nK2mMxldp8AakMRLDnNMnOuW cwjQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KWKhpbCC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id x23-20020a17090abc9700b00286c80ac075si7677878pjr.177.2023.12.11.07.57.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:57:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KWKhpbCC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 1A2038062090; Mon, 11 Dec 2023 07:57:40 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344465AbjLKP50 (ORCPT + 99 others); Mon, 11 Dec 2023 10:57:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344416AbjLKP5S (ORCPT ); Mon, 11 Dec 2023 10:57:18 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B0A14E8 for ; Mon, 11 Dec 2023 07:57:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310239; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IYkHAajZElCwYCEwnUcbPY1wmlYkOiWYX0LP3vDIRKY=; b=KWKhpbCCYQYSNrnF/x008JUiuE7WszyL2p972EGZNnyWjNABQbsw0VYI7Gh7p1fMxgVxPm sD+8mTAlYll0g4crKBffqoAp4E8fbDqg5xAzGmK6YIKp53FIIKJlhK7RqGQ9plm7cc6SVZ Br/TBRIyck62HeQns74s9YKitP7yvbE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-389-vLDkjjTeNrqXiJAcMx9kjQ-1; Mon, 11 Dec 2023 10:57:15 -0500 X-MC-Unique: vLDkjjTeNrqXiJAcMx9kjQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E97A4185A780; Mon, 11 Dec 2023 15:57:14 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id CEE511121306; Mon, 11 Dec 2023 15:57:12 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 07/39] mm/rmap: convert folio_add_file_rmap_range() into folio_add_file_rmap_[pte|ptes|pmd]() Date: Mon, 11 Dec 2023 16:56:20 +0100 Message-ID: <20231211155652.131054-8-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:57:40 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001689302838103 X-GMAIL-MSGID: 1785001689302838103 Let's get rid of the compound parameter and instead define implicitly which mappings we're adding. That is more future proof, easier to read and harder to mess up. Use an enum to express the granularity internally. Make the compiler always special-case on the granularity by using __always_inline. Replace the "compound" check by a switch-case that will be removed by the compiler completely. Add plenty of sanity checks with CONFIG_DEBUG_VM. Replace the folio_test_pmd_mappable() check by a config check in the caller and sanity checks. Convert the single user of folio_add_file_rmap_range(). This function design can later easily be extended to PUDs and to batch PMDs. Note that for now we don't support anything bigger than PMD-sized folios (as we cleanly separated hugetlb handling). Sanity checks will catch if that ever changes. Next up is removing page_remove_rmap() along with its "compound" parameter and smilarly converting all other rmap functions. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei Reviewed-by: Ryan Roberts --- include/linux/rmap.h | 47 +++++++++++++++++++++++++-- mm/memory.c | 2 +- mm/rmap.c | 75 +++++++++++++++++++++++++++++--------------- 3 files changed, 95 insertions(+), 29 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index e3857d26b944..1753900f4aed 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -191,6 +191,45 @@ typedef int __bitwise rmap_t; */ #define RMAP_COMPOUND ((__force rmap_t)BIT(1)) +/* + * Internally, we're using an enum to specify the granularity. Usually, + * we make the compiler create specialized variants for the different + * granularity. + */ +enum rmap_mode { + RMAP_MODE_PTE = 0, + RMAP_MODE_PMD, +}; + +static inline void __folio_rmap_sanity_checks(struct folio *folio, + struct page *page, int nr_pages, enum rmap_mode mode) +{ + /* hugetlb folios are handled separately. */ + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); + VM_WARN_ON_FOLIO(folio_test_large(folio) && + !folio_test_large_rmappable(folio), folio); + + VM_WARN_ON_ONCE(nr_pages <= 0); + VM_WARN_ON_FOLIO(page_folio(page) != folio, folio); + VM_WARN_ON_FOLIO(page_folio(page + nr_pages - 1) != folio, folio); + + switch (mode) { + case RMAP_MODE_PTE: + break; + case RMAP_MODE_PMD: + /* + * We don't support folios larger than a single PMD yet. So + * when RMAP_MODE_PMD is set, we assume that we are creating + * a single "entire" mapping of the folio. + */ + VM_WARN_ON_FOLIO(folio_nr_pages(folio) != HPAGE_PMD_NR, folio); + VM_WARN_ON_FOLIO(nr_pages != HPAGE_PMD_NR, folio); + break; + default: + VM_WARN_ON_ONCE(true); + } +} + /* * rmap interfaces called when adding or removing pte of page */ @@ -203,8 +242,12 @@ void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); -void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr, - struct vm_area_struct *, bool compound); +void folio_add_file_rmap_ptes(struct folio *, struct page *, int nr_pages, + struct vm_area_struct *); +#define folio_add_file_rmap_pte(folio, page, vma) \ + folio_add_file_rmap_ptes(folio, page, 1, vma) +void folio_add_file_rmap_pmd(struct folio *, struct page *, + struct vm_area_struct *); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); diff --git a/mm/memory.c b/mm/memory.c index 8f0b936b90b5..6a5540ba3c65 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4515,7 +4515,7 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio, folio_add_lru_vma(folio, vma); } else { add_mm_counter(vma->vm_mm, mm_counter_file(page), nr); - folio_add_file_rmap_range(folio, page, nr, vma, false); + folio_add_file_rmap_ptes(folio, page, nr, vma); } set_ptes(vma->vm_mm, addr, vmf->pte, entry, nr); diff --git a/mm/rmap.c b/mm/rmap.c index 41597da14f26..4f30930a1162 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1376,31 +1376,20 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, __lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr); } -/** - * folio_add_file_rmap_range - add pte mapping to page range of a folio - * @folio: The folio to add the mapping to - * @page: The first page to add - * @nr_pages: The number of pages which will be mapped - * @vma: the vm area in which the mapping is added - * @compound: charge the page as compound or small page - * - * The page range of folio is defined by [first_page, first_page + nr_pages) - * - * The caller needs to hold the pte lock. - */ -void folio_add_file_rmap_range(struct folio *folio, struct page *page, - unsigned int nr_pages, struct vm_area_struct *vma, - bool compound) +static __always_inline void __folio_add_file_rmap(struct folio *folio, + struct page *page, int nr_pages, struct vm_area_struct *vma, + enum rmap_mode mode) { atomic_t *mapped = &folio->_nr_pages_mapped; unsigned int nr_pmdmapped = 0, first; int nr = 0; - VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); - VM_WARN_ON_FOLIO(compound && !folio_test_pmd_mappable(folio), folio); + VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); /* Is page being mapped by PTE? Is this its first map to be added? */ - if (likely(!compound)) { + switch (mode) { + case RMAP_MODE_PTE: do { first = atomic_inc_and_test(&page->_mapcount); if (first && folio_test_large(folio)) { @@ -1411,9 +1400,8 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, if (first) nr++; } while (page++, --nr_pages > 0); - } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */ - + break; + case RMAP_MODE_PMD: first = atomic_inc_and_test(&folio->_entire_mapcount); if (first) { nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); @@ -1428,6 +1416,7 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, nr = 0; } } + break; } if (nr_pmdmapped) @@ -1441,6 +1430,43 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, mlock_vma_folio(folio, vma); } +/** + * folio_add_file_rmap_ptes - add PTE mappings to a page range of a folio + * @folio: The folio to add the mappings to + * @page: The first page to add + * @nr_pages: The number of pages that will be mapped using PTEs + * @vma: The vm area in which the mappings are added + * + * The page range of the folio is defined by [page, page + nr_pages) + * + * The caller needs to hold the page table lock. + */ +void folio_add_file_rmap_ptes(struct folio *folio, struct page *page, + int nr_pages, struct vm_area_struct *vma) +{ + __folio_add_file_rmap(folio, page, nr_pages, vma, RMAP_MODE_PTE); +} + +/** + * folio_add_file_rmap_pmd - add a PMD mapping to a page range of a folio + * @folio: The folio to add the mapping to + * @page: The first page to add + * @vma: The vm area in which the mapping is added + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock. + */ +void folio_add_file_rmap_pmd(struct folio *folio, struct page *page, + struct vm_area_struct *vma) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_add_file_rmap(folio, page, HPAGE_PMD_NR, vma, RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + /** * page_add_file_rmap - add pte mapping to a file page * @page: the page to add the mapping to @@ -1453,16 +1479,13 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, bool compound) { struct folio *folio = page_folio(page); - unsigned int nr_pages; VM_WARN_ON_ONCE_PAGE(compound && !PageTransHuge(page), page); if (likely(!compound)) - nr_pages = 1; + folio_add_file_rmap_pte(folio, page, vma); else - nr_pages = folio_nr_pages(folio); - - folio_add_file_rmap_range(folio, page, nr_pages, vma, compound); + folio_add_file_rmap_pmd(folio, page, vma); } /** From patchwork Mon Dec 11 15:56:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176754 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7152154vqy; Mon, 11 Dec 2023 07:57:49 -0800 (PST) X-Google-Smtp-Source: AGHT+IFk5vKg95dEBPoASl6iupjD5oc9p8685WCORs3mnNAonIk/Bdwpc93R2g3h02cZZor4n1Fd X-Received: by 2002:a05:6a20:1390:b0:18c:110e:c33d with SMTP id hn16-20020a056a20139000b0018c110ec33dmr2065766pzc.28.1702310269459; Mon, 11 Dec 2023 07:57:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310269; cv=none; d=google.com; s=arc-20160816; b=ax7X6B3ccxBuLiVapUeS0CIyZCcSSzV+pYrDivPLm9xb67uoXRFx0mAeFPC3fwlmZF fzXA0ve1PXDNiaHDxm3303OU/5j0O36GxBzGsryA2UAMngY1cgQlCUbJRGG9LNlMULZ0 hKo9OlPHFkuQ74i6oGNxSWKvVkFLRiZdnYC4zDC7S7yvz3foDhiUs9sCda7QGenrNq7H sbmQpHaWDKTiX1tL1IfoJYkFFMTi/cSh65WrlOcqv9g+fp9dad18//OENJvD9v3slRlB cZFfZWoNH/0tGLiFqdv8c/OiahCS5KIU/YAInQJlQqnC80Q1G4Mrvu/2Rmv2GgnBGZP0 t70Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=6j0c9ENvnxisPX95qRO4NJave2eFRdDuqpK0xgwx6DM=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=ePvpXc4uUvjajiX9ckfGK0XfcqnO5Kbu9yOTl9meRMVCMYCyb7gdRDSxfYmpe8Picm kBsTlgaVX730F21y9EUdmLPJIiHQouTaSpYc3SQ7DUZ4DQtvZvMHcBJNyL9NcMlqM7vB 1LlIfb8mkU5qH0t4A+LudqU/z8ikfP+4Nc3adJwXJsnkhgkjcTlmm1qs07hRog2VtLiG os3rJqJiw9DG4Dwgo7+q2Ugy6Gx2I87hcdKP4RL19vu5g8+P6TzIlh/Vh4HkhFi3KGmJ vC1jI2PZrsJI6CaucP3+yLKMBvWYTgqAZkXMVY+JE/d2wZ2t7MigIFIMmi2kkYrYxJ+p 5w1Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KpZeAeGJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id s35-20020a632163000000b005c66e60900csi6046202pgm.239.2023.12.11.07.57.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:57:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KpZeAeGJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 723A180569B8; Mon, 11 Dec 2023 07:57:44 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344478AbjLKP53 (ORCPT + 99 others); Mon, 11 Dec 2023 10:57:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51824 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344486AbjLKP5S (ORCPT ); Mon, 11 Dec 2023 10:57:18 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CECFEF4 for ; Mon, 11 Dec 2023 07:57:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310243; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6j0c9ENvnxisPX95qRO4NJave2eFRdDuqpK0xgwx6DM=; b=KpZeAeGJRDAN/jBkv/DGO11+O2YspIg2o3hKex3LIpd7FR1pDWBCX5d/MxtshL/sAmjO0R mw2I/gIv2/KjE3H+gCX/01xwcf7m1jwX2w+zZkZ3udCfiH711oVCghnNk3dmCFnU2RwqSU meU5O2PW3lnAamFgXAH3ulPGJpLB5Os= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-453-i-re_ooQPI6Z0iay6LlX8A-1; Mon, 11 Dec 2023 10:57:17 -0500 X-MC-Unique: i-re_ooQPI6Z0iay6LlX8A-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 321B73C2865F; Mon, 11 Dec 2023 15:57:17 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2D4961121306; Mon, 11 Dec 2023 15:57:15 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 08/39] mm/memory: page_add_file_rmap() -> folio_add_file_rmap_[pte|pmd]() Date: Mon, 11 Dec 2023 16:56:21 +0100 Message-ID: <20231211155652.131054-9-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:57:44 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001692718572182 X-GMAIL-MSGID: 1785001692718572182 Let's convert insert_page_into_pte_locked() and do_set_pmd(). While at it, perform some folio conversion. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- mm/memory.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 6a5540ba3c65..70754fd65788 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1859,12 +1859,14 @@ static int validate_page_before_insert(struct page *page) static int insert_page_into_pte_locked(struct vm_area_struct *vma, pte_t *pte, unsigned long addr, struct page *page, pgprot_t prot) { + struct folio *folio = page_folio(page); + if (!pte_none(ptep_get(pte))) return -EBUSY; /* Ok, finally just insert the thing.. */ - get_page(page); + folio_get(folio); inc_mm_counter(vma->vm_mm, mm_counter_file(page)); - page_add_file_rmap(page, vma, false); + folio_add_file_rmap_pte(folio, page, vma); set_pte_at(vma->vm_mm, addr, pte, mk_pte(page, prot)); return 0; } @@ -4409,6 +4411,7 @@ static void deposit_prealloc_pte(struct vm_fault *vmf) vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) { + struct folio *folio = page_folio(page); struct vm_area_struct *vma = vmf->vma; bool write = vmf->flags & FAULT_FLAG_WRITE; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; @@ -4418,8 +4421,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) if (!thp_vma_suitable_order(vma, haddr, PMD_ORDER)) return ret; - page = compound_head(page); - if (compound_order(page) != HPAGE_PMD_ORDER) + if (page != &folio->page || folio_order(folio) != HPAGE_PMD_ORDER) return ret; /* @@ -4428,7 +4430,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) * check. This kind of THP just can be PTE mapped. Access to * the corrupted subpage should trigger SIGBUS as expected. */ - if (unlikely(PageHasHWPoisoned(page))) + if (unlikely(folio_test_has_hwpoisoned(folio))) return ret; /* @@ -4452,7 +4454,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); add_mm_counter(vma->vm_mm, mm_counter_file(page), HPAGE_PMD_NR); - page_add_file_rmap(page, vma, true); + folio_add_file_rmap_pmd(folio, page, vma); /* * deposit and withdraw with pmd lock held From patchwork Mon Dec 11 15:56:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176757 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7152189vqy; Mon, 11 Dec 2023 07:57:53 -0800 (PST) X-Google-Smtp-Source: AGHT+IETstN9lGK+PqbIIcOUzzH+hdCpD0XBVN+N6GG+Hl5q4KJOz7ydbhujh6ggw8R/58NsV/IX X-Received: by 2002:a17:902:ab0d:b0:1d0:ce26:7ec2 with SMTP id ik13-20020a170902ab0d00b001d0ce267ec2mr4451290plb.6.1702310273516; Mon, 11 Dec 2023 07:57:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310273; cv=none; d=google.com; s=arc-20160816; b=Wwbn1uQUICGxyRl4DvCGDB6jGNi7FQftwPdfxu0a50cfsS+d9Pv0GIRQiI9pYwCHqN e1ZskjX1DfP+2GuPePjqz2X9RAwMTY8PRBYOJimgU77kQ9b7qpX4domWGYXQ4jfYrGS8 FRMWdKJxZoHFR3P8m/MruJibhq7pB3yY2kUiOS5LOnzu5IXz4mLFAF+eYmGK3vb9ZcgP IUtLDYpzfE2kEhWNtuehTjVcplfY5LvxQ1OgXwWXIUILCAhWunyoDY38CZch0dkoerit jwjeCigyZy+rEB3tl2BUlbtMT1bCKKXY2527Dp7MO3ZlJ5PvFIRjriWyNGvBjBPSlydJ QvPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=IFC4IqKosk2274JfAmIWQYXkgsQjdSCIu6Bum/spFuM=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=ScuVYoSDkA2RxTM1ZTNrcIn72pcPLSa0dpEzib/h0Ice33RFio+gxEtYG8wPElOb9q ElJcY6ISnDnClLRXTNuxVTIybvIqX0+TkEi8m0OPAHbdDR+xeDZhY6dZQxfQ5cxWuqfi eTvRioqynSrfqREuojx2e11mdy5j5d3mdaVH6zvqThQEIkix/A7T8+Gwh+3sEbern+0u V3l7QRsTBwquK61TpLbVw68knrcQdJd7YTeM9o2PiNUQpJ1h81iZrPJgMYcK01V/9rqv JgCbfD//miJ8uJfLJcLVZPQho0f5vMPFPJVBbOycX3xQcl+G47k5/KqNU600OZi2M0DH JrNg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=BSFDW9HY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id j15-20020a170903024f00b001d1db5e39absi6358281plh.320.2023.12.11.07.57.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:57:53 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=BSFDW9HY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 24CD1805ADCA; Mon, 11 Dec 2023 07:57:49 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344547AbjLKP5g (ORCPT + 99 others); Mon, 11 Dec 2023 10:57:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344491AbjLKP5S (ORCPT ); Mon, 11 Dec 2023 10:57:18 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A9FA4FF for ; Mon, 11 Dec 2023 07:57:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310243; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IFC4IqKosk2274JfAmIWQYXkgsQjdSCIu6Bum/spFuM=; b=BSFDW9HYPtLkYPg1P1jLd5BlFFZpPfadHrUDRrkiG1E3v1gpvFa4Ub12PpgnBeV74lEA2p r2SDtCSea7j84RuTnjX11onFxgYYyJk5jJDYY+Iy3bvmkl4LOO7WNOZLIruPwl8FZLrUQd 3hBEh8fW0+9PxVgI1CmN0uDCwhJ2nm0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-554-HfiNUATqMv-A-RIgHEr9kQ-1; Mon, 11 Dec 2023 10:57:20 -0500 X-MC-Unique: HfiNUATqMv-A-RIgHEr9kQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B132B870830; Mon, 11 Dec 2023 15:57:19 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8B9311121306; Mon, 11 Dec 2023 15:57:17 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 09/39] mm/huge_memory: page_add_file_rmap() -> folio_add_file_rmap_pmd() Date: Mon, 11 Dec 2023 16:56:22 +0100 Message-ID: <20231211155652.131054-10-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:57:49 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001696933023617 X-GMAIL-MSGID: 1785001696933023617 Let's convert remove_migration_pmd() and while at it, perform some folio conversion. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- mm/huge_memory.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3a387c6f18b6..1f5634b2f374 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3577,6 +3577,7 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) { + struct folio *folio = page_folio(new); struct vm_area_struct *vma = pvmw->vma; struct mm_struct *mm = vma->vm_mm; unsigned long address = pvmw->address; @@ -3588,7 +3589,7 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) return; entry = pmd_to_swp_entry(*pvmw->pmd); - get_page(new); + folio_get(folio); pmde = mk_huge_pmd(new, READ_ONCE(vma->vm_page_prot)); if (pmd_swp_soft_dirty(*pvmw->pmd)) pmde = pmd_mksoft_dirty(pmde); @@ -3599,10 +3600,10 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) if (!is_migration_entry_young(entry)) pmde = pmd_mkold(pmde); /* NOTE: this may contain setting soft-dirty on some archs */ - if (PageDirty(new) && is_migration_entry_dirty(entry)) + if (folio_test_dirty(folio) && is_migration_entry_dirty(entry)) pmde = pmd_mkdirty(pmde); - if (PageAnon(new)) { + if (folio_test_anon(folio)) { rmap_t rmap_flags = RMAP_COMPOUND; if (!is_readable_migration_entry(entry)) @@ -3610,9 +3611,9 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) page_add_anon_rmap(new, vma, haddr, rmap_flags); } else { - page_add_file_rmap(new, vma, true); + folio_add_file_rmap_pmd(folio, new, vma); } - VM_BUG_ON(pmd_write(pmde) && PageAnon(new) && !PageAnonExclusive(new)); + VM_BUG_ON(pmd_write(pmde) && folio_test_anon(folio) && !PageAnonExclusive(new)); set_pmd_at(mm, haddr, pvmw->pmd, pmde); /* No need to invalidate - it was non-present before */ From patchwork Mon Dec 11 15:56:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176785 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7155310vqy; Mon, 11 Dec 2023 08:01:30 -0800 (PST) X-Google-Smtp-Source: AGHT+IHrHxds6rM/+hHUr7MPsDss7OhAQoZGHZWVhNgzVlBOlN1yroTbVrFF3WWLeuRvHUaAmqG4 X-Received: by 2002:a05:6a00:2d84:b0:6cb:a60c:1d with SMTP id fb4-20020a056a002d8400b006cba60c001dmr2356512pfb.17.1702310490256; Mon, 11 Dec 2023 08:01:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310490; cv=none; d=google.com; s=arc-20160816; b=KobJqmTg17mis5AyEfQUF+KJtlDkNqtlQUTy80syHhXUHbzXh1bGSnlaiP+7TLQR3C I/HWJKlXi4dCgVd4JIx4yp8msPODCds7VqlGQ4VOSD+ZVT5CrWL9bPey5PQrrco/Xeuv FS4fZ1cPPPG9lKP7QbzPqF7kIA1+9KBvJVy0mnEmJ4giX4m4yCh0Q6tLeW1gkPybCKWq VQlHFgIK5Rl9C9cUJFoSRHImtE4+Jv51GE3sPF9R+3FlptxFqgHYdI5q4RuM14sX8cAp NZIgB1fv8nRM+Sv4RBh5HDSlWgQgTFbpGiM+E3OIwymlaOs2VDgR9laGdtowNcymxNOc 6U/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XZDdgML9rdzFWWmlqwVPDDDqGkwGAqFc+AWaxqMx8Tw=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=WobAtw8aX86LNAErrqbxeS2ynne/SqVT7+z7ILjbvF/kz6ZMSIpQxKxGU8HyfWv6Di TyfWD51BWWYLO4Zr7KHtKmsvrdz0rKTs8t/CuSqkSQsXeaHMFUkaWBhIB1lrWOTXHCPa FUIjKsVQQzdZQIxLYNOhZFt+QbqfHZ3Yl1S6vhw/+yfbzI9eRujTGnFjM3Q49zYptxkw 47Kd1qzsUYk3MU90hY9g7wuwXEMiDHlgRb41bkESzqZ7rbhq3YtyB4dd0nOFagflNgDH Qnj8+/qySpkP0zfEYbyM0uXkVN/HbUP5ASxou2cfvVcmcvfWFKagMIgk4WVLQLbKPcZp 0ZCw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=eaAA5Ctz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id d27-20020a631d5b000000b005bd42f6085fsi6223760pgm.344.2023.12.11.08.01.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 08:01:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=eaAA5Ctz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 5306280AFE9F; Mon, 11 Dec 2023 08:00:43 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344721AbjLKQAd (ORCPT + 99 others); Mon, 11 Dec 2023 11:00:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46658 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344731AbjLKQAH (ORCPT ); Mon, 11 Dec 2023 11:00:07 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1FD71BD7 for ; Mon, 11 Dec 2023 07:58:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310320; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XZDdgML9rdzFWWmlqwVPDDDqGkwGAqFc+AWaxqMx8Tw=; b=eaAA5CtzNPb3B1D+YZMIi4YeSdp9ykX2KQ+bN1NYOf1uh0ezm6UfvTOSI+fYGL76BoLyE+ 3DZluI6P00uXAAT1wgd7D68x69TMEQbnhux5B3GACKwha4d4zEgkoFOXS39hfAUseMTlvQ YKwYo5aZXd/RO/37qPLPYOVaD4EO5dQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-657-X45q_OfKOJOoxzulSDa8Qg-1; Mon, 11 Dec 2023 10:58:05 -0500 X-MC-Unique: X45q_OfKOJOoxzulSDa8Qg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 57BC9101A555; Mon, 11 Dec 2023 15:57:22 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 18B491121306; Mon, 11 Dec 2023 15:57:19 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 10/39] mm/migrate: page_add_file_rmap() -> folio_add_file_rmap_pte() Date: Mon, 11 Dec 2023 16:56:23 +0100 Message-ID: <20231211155652.131054-11-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 11 Dec 2023 08:00:44 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001924635897253 X-GMAIL-MSGID: 1785001924635897253 Let's convert remove_migration_pte(). Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- mm/migrate.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/migrate.c b/mm/migrate.c index de9d94b99ab7..efc19f53b05e 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -262,7 +262,7 @@ static bool remove_migration_pte(struct folio *folio, page_add_anon_rmap(new, vma, pvmw.address, rmap_flags); else - page_add_file_rmap(new, vma, false); + folio_add_file_rmap_pte(folio, new, vma); set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); } if (vma->vm_flags & VM_LOCKED) From patchwork Mon Dec 11 15:56:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176759 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7152261vqy; Mon, 11 Dec 2023 07:58:01 -0800 (PST) X-Google-Smtp-Source: AGHT+IGwHKGB+97v7z34kaHKWETK2/iRwXyy7cGta0QF1i+Qe1SGqm4XNyWvvJGNybJhf5Cznwnt X-Received: by 2002:a05:6a00:174a:b0:6ce:62d6:b59b with SMTP id j10-20020a056a00174a00b006ce62d6b59bmr2471623pfc.32.1702310278586; Mon, 11 Dec 2023 07:57:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310278; cv=none; d=google.com; s=arc-20160816; b=VpbRDL9HoTfEm4PBp42Yll5vj383w8s/iTuioL/EvjxJ+NUiL3ImB33n7/6zkyeCng LIj7wKfCf9hoUQg7pm/dURo3D4UKeRUSKfD43bvBTTgMzT1KXzszotHlKZfsG+DFpyzA /H3MgsBALPge9ogfWoevSUM1aEKjnW83Oyxea2dNR2fH8gdszLFM9kYcyNEzGwfh2/gW g/aLKV+9z1xrjpbEF6GYet1LkWnNZeUSpUJk57BqF6td9RO+4a0UF7vnxCLAjYu75lFE 0BZq4eCNdSayoi3HTYlFk6564Kmmlxtsu7MBtapo99yt4kXAocLtQWrdly5m2Luu0V2w Jy7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=L2Om8DYO6IhDTMNBMAtiLk55MQgBXdIwfXJDSy3Kty4=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=w/Mmy+kyWqSdz6sD2a7hsFMbQTvFNaLVgoDl073HPy0EHL8dbDFa5VfUj/C3M6n6cM tF0u9SiQVCTLPWNVwVVl2qjBUwJdFvEpDgfLFg7JnwvcvZE2LmHsqWKylemvA3aS9cz6 SsP29z1673J4cVomQzbsY9QusqApI395t1i7Qckc+E4rXhI9M/mn7gFgUKmFp2PwP/lo Sos32KJf+s85zuN41gv29QiO3UFDpdVAxdqCOtaITViA7nFfEtNKFKv661oAQ2tGMsTV rhaU7mqjJKoEBvHwDhil+Pj8ryAhHqcv1HOFZVpGcBWZ2xQr3w04XU0yBQGosjk7/ibF kLdA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bWIJHrCh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id o7-20020a656a47000000b005c1b323da1bsi6540517pgu.695.2023.12.11.07.57.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:57:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bWIJHrCh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 8580B805ADDD; Mon, 11 Dec 2023 07:57:53 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344555AbjLKP5j (ORCPT + 99 others); Mon, 11 Dec 2023 10:57:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46662 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344462AbjLKP5X (ORCPT ); Mon, 11 Dec 2023 10:57:23 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6CDE4BF for ; Mon, 11 Dec 2023 07:57:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310248; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L2Om8DYO6IhDTMNBMAtiLk55MQgBXdIwfXJDSy3Kty4=; b=bWIJHrChCcURhyhbAxqQ7JzeJkeumKNCCk6I31F5fzhG+IAu0gcPYtKHWqOqBBBLSISrzt 44LNFNSGFteppfycJb92CrKxgFvya4Ppkpn58u7e7GKOhGG5fvXPri0udG4det7/tnMUtu u7vZL+9uawYTIMtq5stfRkzimsohVi8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-203-TS2p9d5tO1ihFJdn6GaxsQ-1; Mon, 11 Dec 2023 10:57:24 -0500 X-MC-Unique: TS2p9d5tO1ihFJdn6GaxsQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 30A73870829; Mon, 11 Dec 2023 15:57:24 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 900321121306; Mon, 11 Dec 2023 15:57:22 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 11/39] mm/userfaultfd: page_add_file_rmap() -> folio_add_file_rmap_pte() Date: Mon, 11 Dec 2023 16:56:24 +0100 Message-ID: <20231211155652.131054-12-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:57:53 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001702750854865 X-GMAIL-MSGID: 1785001702750854865 Let's convert mfill_atomic_install_pte(). Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- mm/userfaultfd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 9ec814e47e99..330a481a1654 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -114,7 +114,7 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd, /* Usually, cache pages are already added to LRU */ if (newly_allocated) folio_add_lru(folio); - page_add_file_rmap(page, dst_vma, false); + folio_add_file_rmap_pte(folio, page, dst_vma); } else { page_add_new_anon_rmap(page, dst_vma, dst_addr); folio_add_lru_vma(folio, dst_vma); From patchwork Mon Dec 11 15:56:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176758 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7152230vqy; Mon, 11 Dec 2023 07:57:57 -0800 (PST) X-Google-Smtp-Source: AGHT+IEH5jlKVivj/owbESz4qT+bnTGukZSFKSjeUhM1oLVZ4b7G5uQ1tUT3cLmwbRJ9nOgB4vNI X-Received: by 2002:a17:902:ab57:b0:1d0:8988:cf93 with SMTP id ij23-20020a170902ab5700b001d08988cf93mr1949407plb.28.1702310277188; Mon, 11 Dec 2023 07:57:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310277; cv=none; d=google.com; s=arc-20160816; b=Yei56EJ8Yo5gxxu42FehCImlZVY0Sj8Txgi26sGoo1eldYaA9aZcyzCQOMe+bD9kxP Ly7h3DRyLgnRWYBGLrmnQVFSvqlaJrKIEIIuGVfscV5A5bproBPLpGMZvPaeLlWHtHFe aTHBEZQsFKSs/3+WJrSHAiD49lM5AGpLkwGhCAafzNTrDgN/wxXe51n/NEWlRQ3E3Xj+ d+1jHp5PxMJKVGZpFafYLPIwcnfs199u67odhOshA/6LCDinjSj0INastRaeD70EZg9Q /YGcRLwQ8q0mDE3EUoEhUbFuD6mT/S26dfepB7RFklM+nWHmCtK+n4Llv9dFeIF8K/kP nf2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=DP/TBY2FfVhMMZMh3t2xGvG39Fvp3GsRcAEgonPxhwA=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=AGxYaPIkt8Qj6NNZ4TnZDpx317XxMv7m9wNwMrMJsWwkWcjmnoYwE8IL5Q+YD78JFc 9vE0Jr5ljLvheL9CuWVBRXSGIk0oQcDPIncejEEKVO8/fQ2cn4k+gOUB90i1bkokLLEx YU/QTRC3kZspdGaQ2pmQIC9DH60bdL+h8VgEcGcT4i7NrTg8CSf+YPikrJXVsuoN4XKd XRimk+xVyHrHYZYlf1vFsrFTgGvbmfzrrY1N6ouFDUifHL38B3lPaHG/QJeyKjGbUHFO vyJVYoBGBZyjjf2gsgC8ck3Ij3j+6ExkfSZyH8zKiUzIityZmrACscaGyefxyZwN8vYO vn1g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Xzn5cTat; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id v3-20020a655683000000b005775e2a7951si6241324pgs.345.2023.12.11.07.57.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:57:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Xzn5cTat; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 7E111803D801; Mon, 11 Dec 2023 07:57:53 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344528AbjLKP5m (ORCPT + 99 others); Mon, 11 Dec 2023 10:57:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46722 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344509AbjLKP50 (ORCPT ); Mon, 11 Dec 2023 10:57:26 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3863110A for ; Mon, 11 Dec 2023 07:57:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310252; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DP/TBY2FfVhMMZMh3t2xGvG39Fvp3GsRcAEgonPxhwA=; b=Xzn5cTat5p+eejld5q+0GDwHOrr1gaA5SJeWrbTL0izXJqSM4fcSiUhswhyJtNb3nq3d7r bhYV4e5N50AUhiGZJyVAzlSjDq4l9i5Fbjxl4VobvYJRjCNaHHJ1ETQKJJmSyffs0MMlbf AAh2EEsbz+zkSodsG/NoPZ708vNxq1A= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-686-nYAYGRlYOBO4fpab-9MmDA-1; Mon, 11 Dec 2023 10:57:27 -0500 X-MC-Unique: nYAYGRlYOBO4fpab-9MmDA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7362810133A0; Mon, 11 Dec 2023 15:57:26 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9447E1121306; Mon, 11 Dec 2023 15:57:24 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 12/39] mm/rmap: remove page_add_file_rmap() Date: Mon, 11 Dec 2023 16:56:25 +0100 Message-ID: <20231211155652.131054-13-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:57:53 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001701401541184 X-GMAIL-MSGID: 1785001701401541184 All users are gone, let's remove it. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- include/linux/rmap.h | 2 -- mm/rmap.c | 21 --------------------- 2 files changed, 23 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 1753900f4aed..7198905dc8be 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -240,8 +240,6 @@ void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address); void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); -void page_add_file_rmap(struct page *, struct vm_area_struct *, - bool compound); void folio_add_file_rmap_ptes(struct folio *, struct page *, int nr_pages, struct vm_area_struct *); #define folio_add_file_rmap_pte(folio, page, vma) \ diff --git a/mm/rmap.c b/mm/rmap.c index 4f30930a1162..2ff2f11275e5 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1467,27 +1467,6 @@ void folio_add_file_rmap_pmd(struct folio *folio, struct page *page, #endif } -/** - * page_add_file_rmap - add pte mapping to a file page - * @page: the page to add the mapping to - * @vma: the vm area in which the mapping is added - * @compound: charge the page as compound or small page - * - * The caller needs to hold the pte lock. - */ -void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, - bool compound) -{ - struct folio *folio = page_folio(page); - - VM_WARN_ON_ONCE_PAGE(compound && !PageTransHuge(page), page); - - if (likely(!compound)) - folio_add_file_rmap_pte(folio, page, vma); - else - folio_add_file_rmap_pmd(folio, page, vma); -} - /** * page_remove_rmap - take down pte mapping from a page * @page: page to remove mapping from From patchwork Mon Dec 11 15:56:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176760 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7152327vqy; Mon, 11 Dec 2023 07:58:02 -0800 (PST) X-Google-Smtp-Source: AGHT+IGjvfoR3ztLTJfC9xWVLl9bq/GBB2wrlTmG8fJQl6tZWsX9VQwl/JPnaTNDNBtzL7KWq4Nn X-Received: by 2002:a17:902:988e:b0:1d0:6ffe:a2c with SMTP id s14-20020a170902988e00b001d06ffe0a2cmr3839574plp.138.1702310282662; Mon, 11 Dec 2023 07:58:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310282; cv=none; d=google.com; s=arc-20160816; b=sFvB/SwCjmR8XfJq2pIlLZf0C5sbTMGkRYYJPL/ms0MSvOh6xLQQpzsDshmj7/VA7v MCVWg3BXEJ88WA2FNWFvWqaRW4n2kXmfNNdSmM+KIAcOfIWwjwmMZM+smhHK3TgfYLVZ 6c5L3esUd6B2KvBzWxBIt3s5HJrlxmGD/sC68RWSZjzDVCef5QWCFMU5+enHNwIr0ZJC FR/IMv+M0FXxbDTDCSZQvYuzDUN6XrgrCJ4g/JPvWmMqUjfAiXDtGXNqnlsQ8KdPBJ8w LPQRZP8GIJ6U83lhHzA+P43Lbyniqm0OJsB66n6ska/nhb8G7RwTLIaXs8j+RCwvsROv gWXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=FoY+MxvyMEZDCAVG6BwT9WrYolbRNCvL6DZB18ZE48E=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=UZZEDE5A+s4RugVoprU2Tu6Cq7FIs2G+0yPDjUHE0iiEq98pAfTTNbBLVTtPd9ewt1 QDBnqi86WH4pMwCZJDB0uuXXsrcdvG6NEZIovT0Fj8mF7jGZTN0YFefqd3d3t2b3vOy0 Nd2nIg9VcT2qb7xpsNRIc6BdWqKIpk3DrFG3JmhkhUqH7rmKhq1QZ9u4mwlC3P8qp3G+ Fdh+CMaEZvc1HHh58w9pXIt9weQgff3d8d7syjcGKvCJYDajzFGNmJ+npNaerOjMpLiz PoCuhFP/y2R/21KW7P8umLBzUzjB8ViMqLvy/CpkhfTj1MMaV5pmn0gv/WojyDUWoGge +mCw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=hlziBPpc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id bh8-20020a170902a98800b001c9c83947d1si6076875plb.645.2023.12.11.07.58.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:58:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=hlziBPpc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id D351180A1876; Mon, 11 Dec 2023 07:58:00 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344585AbjLKP5s (ORCPT + 99 others); Mon, 11 Dec 2023 10:57:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46658 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344502AbjLKP5a (ORCPT ); Mon, 11 Dec 2023 10:57:30 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E324FEA for ; Mon, 11 Dec 2023 07:57:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310255; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FoY+MxvyMEZDCAVG6BwT9WrYolbRNCvL6DZB18ZE48E=; b=hlziBPpc8r1ZZO8+Jyjlj0lLR7Gs9viG31ZMTnbj6hltoT/6pZ8cs4T3TYB8ECSjDWeY19 sM/v4SK8LQwfU3yLpu/FgkJBLpU/zCrxwtDXGOdd2/YmhWRq/JrHzRLAdWp52pjisp21Xh j6sRGqQ6e14bqFLq8+BqYGRgWQ9nJjo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-632-2BLn_LvuPamqZYZJzhfDrg-1; Mon, 11 Dec 2023 10:57:29 -0500 X-MC-Unique: 2BLn_LvuPamqZYZJzhfDrg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1EEF9870822; Mon, 11 Dec 2023 15:57:29 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id D51861121306; Mon, 11 Dec 2023 15:57:26 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 13/39] mm/rmap: factor out adding folio mappings into __folio_add_rmap() Date: Mon, 11 Dec 2023 16:56:26 +0100 Message-ID: <20231211155652.131054-14-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:58:01 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001706547222605 X-GMAIL-MSGID: 1785001706547222605 Let's factor it out to prepare for reuse as we convert page_add_anon_rmap() to folio_add_anon_rmap_[pte|ptes|pmd](). Make the compiler always special-case on the granularity by using __always_inline. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand --- mm/rmap.c | 81 ++++++++++++++++++++++++++++++------------------------- 1 file changed, 45 insertions(+), 36 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 2ff2f11275e5..c5761986a411 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1157,6 +1157,49 @@ int folio_total_mapcount(struct folio *folio) return mapcount; } +static __always_inline unsigned int __folio_add_rmap(struct folio *folio, + struct page *page, int nr_pages, enum rmap_mode mode, + unsigned int *nr_pmdmapped) +{ + atomic_t *mapped = &folio->_nr_pages_mapped; + int first, nr = 0; + + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); + + /* Is page being mapped by PTE? Is this its first map to be added? */ + switch (mode) { + case RMAP_MODE_PTE: + do { + first = atomic_inc_and_test(&page->_mapcount); + if (first && folio_test_large(folio)) { + first = atomic_inc_return_relaxed(mapped); + first = (first < COMPOUND_MAPPED); + } + + if (first) + nr++; + } while (page++, --nr_pages > 0); + break; + case RMAP_MODE_PMD: + first = atomic_inc_and_test(&folio->_entire_mapcount); + if (first) { + nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); + if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { + *nr_pmdmapped = folio_nr_pages(folio); + nr = *nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); + /* Raced ahead of a remove and another add? */ + if (unlikely(nr < 0)) + nr = 0; + } else { + /* Raced ahead of a remove of COMPOUND_MAPPED */ + nr = 0; + } + } + break; + } + return nr; +} + /** * folio_move_anon_rmap - move a folio to our anon_vma * @folio: The folio to move to our anon_vma @@ -1380,45 +1423,11 @@ static __always_inline void __folio_add_file_rmap(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *vma, enum rmap_mode mode) { - atomic_t *mapped = &folio->_nr_pages_mapped; - unsigned int nr_pmdmapped = 0, first; - int nr = 0; + unsigned int nr, nr_pmdmapped = 0; VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); - __folio_rmap_sanity_checks(folio, page, nr_pages, mode); - - /* Is page being mapped by PTE? Is this its first map to be added? */ - switch (mode) { - case RMAP_MODE_PTE: - do { - first = atomic_inc_and_test(&page->_mapcount); - if (first && folio_test_large(folio)) { - first = atomic_inc_return_relaxed(mapped); - first = (first < COMPOUND_MAPPED); - } - - if (first) - nr++; - } while (page++, --nr_pages > 0); - break; - case RMAP_MODE_PMD: - first = atomic_inc_and_test(&folio->_entire_mapcount); - if (first) { - nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { - nr_pmdmapped = folio_nr_pages(folio); - nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); - /* Raced ahead of a remove and another add? */ - if (unlikely(nr < 0)) - nr = 0; - } else { - /* Raced ahead of a remove of COMPOUND_MAPPED */ - nr = 0; - } - } - break; - } + nr = __folio_add_rmap(folio, page, nr_pages, mode, &nr_pmdmapped); if (nr_pmdmapped) __lruvec_stat_mod_folio(folio, folio_test_swapbacked(folio) ? NR_SHMEM_PMDMAPPED : NR_FILE_PMDMAPPED, nr_pmdmapped); From patchwork Mon Dec 11 15:56:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176761 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7152487vqy; Mon, 11 Dec 2023 07:58:17 -0800 (PST) X-Google-Smtp-Source: AGHT+IHgBc1q5wEbZpxH69PPjgJEy46c3qKH29sYWoj6e/Nre/BRFcEhNOFZALlgR+I6jtvFhQkR X-Received: by 2002:a17:902:9a82:b0:1d0:9d64:6e3e with SMTP id w2-20020a1709029a8200b001d09d646e3emr1828509plp.117.1702310297167; Mon, 11 Dec 2023 07:58:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310297; cv=none; d=google.com; s=arc-20160816; b=P7OKZR5tteTtOOP2OJvkxAotiKfR3uJMvCnxGVdYdSkvhoz2hpnkUv4Oytjs8Gb/27 KxGtDWCbSiT2cJC2/XvJu870aOtxP0vUb1C6z4UWYxKUTiYA1eq35tOBFdOYjEsmRAcs aY3Kc3X8FUKznr/+C4M0L1+r+Y1YuR+62y/np6pYaC4rlnLlxSuBTc0hovkpAytCHi2F 9gSAm8OmaA3J+of2z6u/znUYBdWs99QsVSW4QQIMkhCJGaPcjZgPPEoqWWHTYQsmufYF pZJrBqbvrEOt8P2Z/u3MrvS8zWmuIOjDmcREs37gPDQ+Gb3J1Ch4bpNTxD2YWEqye9aI Lj3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=FvIn4PK1R1vqfE/cFFREYzckJVopW/NXl4TYpNX8kUk=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=rWsdeIoiuyPafDWPoKfQOpVQJx0P9shZjYRuK1GM92lsfLe1dl+qJgKpj+uGy+xxSd +6MjjLVg/g06R4E0G5ptd+zuD4DEonFRMb2qDWBF9wwAD2hIF3vDUwnxDXJnGo81nUYe yRCpNQ7m+w0xv//81/i0RfIVc7dF2o2YfIqgzakXvvwbJ1fVfMejNf9Tspf6RlTAMKrq BBouxPdqFxErXMgjez9stth6qMcXMzSOHlmP8YPYXBaWkPVmfSfO5YvzR6idLn6F+Lx3 9MWl5BD901UjvOp4oUkanzZlnLkMBY3Lx5uGvru0KxdF3f7WhC/QddgjrUx0xtX86h3t S3jw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="MkYo/Z9E"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id h9-20020a170902b94900b001b973681493si6188969pls.16.2023.12.11.07.58.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:58:17 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="MkYo/Z9E"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id C7C8480A6ED0; Mon, 11 Dec 2023 07:58:12 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344612AbjLKP56 (ORCPT + 99 others); Mon, 11 Dec 2023 10:57:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46708 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344506AbjLKP5a (ORCPT ); Mon, 11 Dec 2023 10:57:30 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 89AB210C for ; Mon, 11 Dec 2023 07:57:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310255; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FvIn4PK1R1vqfE/cFFREYzckJVopW/NXl4TYpNX8kUk=; b=MkYo/Z9E6GScHnS2rT58BD8uNQTw4AhokrXrRuUriv5AQ1KS3HAgRCzxzsX3MtTjmEhWuF hQtGXZqBgPre+a5hXPDdZ38Iu+p8SEeO2RoiLf9/o387br3tSGbWIk44WIWz2A/OD3C1Ye giK6FqmUb1PL0S52PlCDvRqztgFzTTc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-694-bp0xtxh7M-6waMAcw-G2oA-1; Mon, 11 Dec 2023 10:57:32 -0500 X-MC-Unique: bp0xtxh7M-6waMAcw-G2oA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AA129870820; Mon, 11 Dec 2023 15:57:31 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 583EB1121306; Mon, 11 Dec 2023 15:57:29 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 14/39] mm/rmap: introduce folio_add_anon_rmap_[pte|ptes|pmd]() Date: Mon, 11 Dec 2023 16:56:27 +0100 Message-ID: <20231211155652.131054-15-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:58:13 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001722288210964 X-GMAIL-MSGID: 1785001722288210964 Let's mimic what we did with folio_add_file_rmap_*() so we can similarly replace page_add_anon_rmap() next. Make the compiler always special-case on the granularity by using __always_inline. Note that the new functions ignore the RMAP_COMPOUND flag, which we will remove as soon as page_add_anon_rmap() is gone. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- include/linux/rmap.h | 6 +++ mm/rmap.c | 118 ++++++++++++++++++++++++++++++------------- 2 files changed, 88 insertions(+), 36 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 7198905dc8be..3b5357cb1c09 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -234,6 +234,12 @@ static inline void __folio_rmap_sanity_checks(struct folio *folio, * rmap interfaces called when adding or removing pte of page */ void folio_move_anon_rmap(struct folio *, struct vm_area_struct *); +void folio_add_anon_rmap_ptes(struct folio *, struct page *, int nr_pages, + struct vm_area_struct *, unsigned long address, rmap_t flags); +#define folio_add_anon_rmap_pte(folio, page, vma, address, flags) \ + folio_add_anon_rmap_ptes(folio, page, 1, vma, address, flags) +void folio_add_anon_rmap_pmd(struct folio *, struct page *, + struct vm_area_struct *, unsigned long address, rmap_t flags); void page_add_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, diff --git a/mm/rmap.c b/mm/rmap.c index c5761986a411..7787499fa2ad 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1300,38 +1300,20 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, unsigned long address, rmap_t flags) { struct folio *folio = page_folio(page); - atomic_t *mapped = &folio->_nr_pages_mapped; - int nr = 0, nr_pmdmapped = 0; - bool compound = flags & RMAP_COMPOUND; - bool first; - /* Is page being mapped by PTE? Is this its first map to be added? */ - if (likely(!compound)) { - first = atomic_inc_and_test(&page->_mapcount); - nr = first; - if (first && folio_test_large(folio)) { - nr = atomic_inc_return_relaxed(mapped); - nr = (nr < COMPOUND_MAPPED); - } - } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */ + if (likely(!(flags & RMAP_COMPOUND))) + folio_add_anon_rmap_pte(folio, page, vma, address, flags); + else + folio_add_anon_rmap_pmd(folio, page, vma, address, flags); +} - first = atomic_inc_and_test(&folio->_entire_mapcount); - if (first) { - nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { - nr_pmdmapped = folio_nr_pages(folio); - nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); - /* Raced ahead of a remove and another add? */ - if (unlikely(nr < 0)) - nr = 0; - } else { - /* Raced ahead of a remove of COMPOUND_MAPPED */ - nr = 0; - } - } - } +static __always_inline void __folio_add_anon_rmap(struct folio *folio, + struct page *page, int nr_pages, struct vm_area_struct *vma, + unsigned long address, rmap_t flags, enum rmap_mode mode) +{ + unsigned int i, nr, nr_pmdmapped = 0; + nr = __folio_add_rmap(folio, page, nr_pages, mode, &nr_pmdmapped); if (nr_pmdmapped) __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr_pmdmapped); if (nr) @@ -1345,18 +1327,34 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, * folio->index right when not given the address of the head * page. */ - VM_WARN_ON_FOLIO(folio_test_large(folio) && !compound, folio); + VM_WARN_ON_FOLIO(folio_test_large(folio) && + mode != RMAP_MODE_PMD, folio); __folio_set_anon(folio, vma, address, !!(flags & RMAP_EXCLUSIVE)); } else if (likely(!folio_test_ksm(folio))) { __page_check_anon_rmap(folio, page, vma, address); } - if (flags & RMAP_EXCLUSIVE) - SetPageAnonExclusive(page); - /* While PTE-mapping a THP we have a PMD and a PTE mapping. */ - VM_WARN_ON_FOLIO((atomic_read(&page->_mapcount) > 0 || - (folio_test_large(folio) && folio_entire_mapcount(folio) > 1)) && - PageAnonExclusive(page), folio); + + if (flags & RMAP_EXCLUSIVE) { + switch (mode) { + case RMAP_MODE_PTE: + for (i = 0; i < nr_pages; i++) + SetPageAnonExclusive(page + i); + break; + case RMAP_MODE_PMD: + SetPageAnonExclusive(page); + break; + } + } + for (i = 0; i < nr_pages; i++) { + struct page *cur_page = page + i; + + /* While PTE-mapping a THP we have a PMD and a PTE mapping. */ + VM_WARN_ON_FOLIO((atomic_read(&cur_page->_mapcount) > 0 || + (folio_test_large(folio) && + folio_entire_mapcount(folio) > 1)) && + PageAnonExclusive(cur_page), folio); + } /* * For large folio, only mlock it if it's fully mapped to VMA. It's @@ -1368,6 +1366,54 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, mlock_vma_folio(folio, vma); } +/** + * folio_add_anon_rmap_ptes - add PTE mappings to a page range of an anon folio + * @folio: The folio to add the mappings to + * @page: The first page to add + * @nr_pages: The number of pages which will be mapped + * @vma: The vm area in which the mappings are added + * @address: The user virtual address of the first page to map + * @flags: The rmap flags + * + * The page range of folio is defined by [first_page, first_page + nr_pages) + * + * The caller needs to hold the page table lock, and the page must be locked in + * the anon_vma case: to serialize mapping,index checking after setting, + * and to ensure that an anon folio is not being upgraded racily to a KSM folio + * (but KSM folios are never downgraded). + */ +void folio_add_anon_rmap_ptes(struct folio *folio, struct page *page, + int nr_pages, struct vm_area_struct *vma, unsigned long address, + rmap_t flags) +{ + __folio_add_anon_rmap(folio, page, nr_pages, vma, address, flags, + RMAP_MODE_PTE); +} + +/** + * folio_add_anon_rmap_pmd - add a PMD mapping to a page range of an anon folio + * @folio: The folio to add the mapping to + * @page: The first page to add + * @vma: The vm area in which the mapping is added + * @address: The user virtual address of the first page to map + * @flags: The rmap flags + * + * The page range of folio is defined by [first_page, first_page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock, and the page must be locked in + * the anon_vma case: to serialize mapping,index checking after setting. + */ +void folio_add_anon_rmap_pmd(struct folio *folio, struct page *page, + struct vm_area_struct *vma, unsigned long address, rmap_t flags) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_add_anon_rmap(folio, page, HPAGE_PMD_NR, vma, address, flags, + RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + /** * folio_add_new_anon_rmap - Add mapping to a new anonymous folio. * @folio: The folio to add the mapping to. From patchwork Mon Dec 11 15:56:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176763 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7152569vqy; Mon, 11 Dec 2023 07:58:25 -0800 (PST) X-Google-Smtp-Source: AGHT+IHvMCBs1dGKHky/NBOH+EOFfkqO/D+5htRfo22N6Ld1oMqTL7Oa2WgnJFghmEqdJjlQ+02F X-Received: by 2002:a17:902:e849:b0:1d0:c7c6:91a2 with SMTP id t9-20020a170902e84900b001d0c7c691a2mr2330897plg.100.1702310303846; Mon, 11 Dec 2023 07:58:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310303; cv=none; d=google.com; s=arc-20160816; b=m6TiAbnZwbtP8eS8xRU/LSMRX9+KxkfeKgKLMa1O/JIMLRHq3jpK6RwF44OTVtozI0 QjZ4+WUFzj6TjZjC52/7TygwW7YUZeD002oZbAL1RVrgfVuY5S2V+WLS1NnJODktRRJX +xF6T0XQl0L9hmlwVh1ZiYkDAaJBNFr1Mr12vCng/74XQtSz8+S7Ak4WyGyli7m0DfxJ 1iJG56Odjc6D39aQaEJw1jE/UEFqJaUpQvf6iRMTbrkVJxHFkF2RZowaD+grOOfNqEAp HL/xB7Cqq4uK77oj1McDosfw1VECHn4HaWqMBVdXtYIud63on5aAAwc8MwWr/zMvnrY9 g7WQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=gWnH4eIM4y9lGkdEQ6xWZD6LXOR6bYxTBmelfOqDsXI=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=UK+b4Ik82Wx0AEfU0LRgeDSnT4B13dM9xbInU2L2QkueWR3KMbNsmpYU8pBL+MIPbW tpDw6hnY4H6nRzS9VgcWd72QfCKbb41JXxZQoEgDO9I6LPwp6/v7WP9mw6iyKYC5CljV PDiicUa59h83Ifal1879Q0W4irtGUCwZhTtLH8IgbWqxWlSza7Z0yRFwvyBquKV3ui3O FYpT+1ApaoiihwGzfcDaBDvo/VICbnMGZ+nXgKnZcOzwyDXSmiSuEBu0e0kRi1FI01kO 5Jt3mn29Vx4K3oq8QKYTwGVvDaOTKS1fPL907rEuRk4W+x0se+yYHb3WxQIQWVJUtbc+ NMSQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JU+jXHi5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id h5-20020a170902f7c500b001d005be08c4si6235760plw.235.2023.12.11.07.58.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:58:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JU+jXHi5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 3C04780569B8; Mon, 11 Dec 2023 07:58:20 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344519AbjLKP6C (ORCPT + 99 others); Mon, 11 Dec 2023 10:58:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59300 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344529AbjLKP5d (ORCPT ); Mon, 11 Dec 2023 10:57:33 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5378A10A for ; Mon, 11 Dec 2023 07:57:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310258; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gWnH4eIM4y9lGkdEQ6xWZD6LXOR6bYxTBmelfOqDsXI=; b=JU+jXHi53Tz/FWXnxM4ejB5AzKvOp2hzWMbwsggpnvOwEn3S1VeDOaCIr24QRCkBnhAolz FDjEY94RfmiaS0exQoEBu9Ly6/6lf7yAkyOk4UgbRwEzVovgRSyG1LsyYferaPszjH93CD xZOqZhhi9FqnBXvVkxGrlJPkJa4ahuc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-255-Z3gSjZ9jPqqQxyKFZIs1PA-1; Mon, 11 Dec 2023 10:57:35 -0500 X-MC-Unique: Z3gSjZ9jPqqQxyKFZIs1PA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 742D1185A788; Mon, 11 Dec 2023 15:57:34 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id EF3641121306; Mon, 11 Dec 2023 15:57:31 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 15/39] mm/huge_memory: batch rmap operations in __split_huge_pmd_locked() Date: Mon, 11 Dec 2023 16:56:28 +0100 Message-ID: <20231211155652.131054-16-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:58:20 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001729169864662 X-GMAIL-MSGID: 1785001729169864662 Let's use folio_add_anon_rmap_ptes(), batching the rmap operations. While at it, use more folio operations (but only in the code branch we're touching), use VM_WARN_ON_FOLIO(), and pass RMAP_EXCLUSIVE instead of manually setting PageAnonExclusive. We should never see non-anon pages on that branch: otherwise, the existing page_add_anon_rmap() call would have been flawed already. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei Reviewed-by: Yin Fengwei --- mm/huge_memory.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 1f5634b2f374..82ad68fe0d12 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2398,6 +2398,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, unsigned long haddr, bool freeze) { struct mm_struct *mm = vma->vm_mm; + struct folio *folio; struct page *page; pgtable_t pgtable; pmd_t old_pmd, _pmd; @@ -2493,16 +2494,18 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, uffd_wp = pmd_swp_uffd_wp(old_pmd); } else { page = pmd_page(old_pmd); + folio = page_folio(page); if (pmd_dirty(old_pmd)) { dirty = true; - SetPageDirty(page); + folio_set_dirty(folio); } write = pmd_write(old_pmd); young = pmd_young(old_pmd); soft_dirty = pmd_soft_dirty(old_pmd); uffd_wp = pmd_uffd_wp(old_pmd); - VM_BUG_ON_PAGE(!page_count(page), page); + VM_WARN_ON_FOLIO(!folio_ref_count(folio), folio); + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); /* * Without "freeze", we'll simply split the PMD, propagating the @@ -2519,11 +2522,18 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, * * See page_try_share_anon_rmap(): invalidate PMD first. */ - anon_exclusive = PageAnon(page) && PageAnonExclusive(page); + anon_exclusive = PageAnonExclusive(page); if (freeze && anon_exclusive && page_try_share_anon_rmap(page)) freeze = false; - if (!freeze) - page_ref_add(page, HPAGE_PMD_NR - 1); + if (!freeze) { + rmap_t rmap_flags = RMAP_NONE; + + folio_ref_add(folio, HPAGE_PMD_NR - 1); + if (anon_exclusive) + rmap_flags |= RMAP_EXCLUSIVE; + folio_add_anon_rmap_ptes(folio, page, HPAGE_PMD_NR, + vma, haddr, rmap_flags); + } } /* @@ -2566,8 +2576,6 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = mk_pte(page + i, READ_ONCE(vma->vm_page_prot)); if (write) entry = pte_mkwrite(entry, vma); - if (anon_exclusive) - SetPageAnonExclusive(page + i); if (!young) entry = pte_mkold(entry); /* NOTE: this may set soft-dirty too on some archs */ @@ -2577,7 +2585,6 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = pte_mksoft_dirty(entry); if (uffd_wp) entry = pte_mkuffd_wp(entry); - page_add_anon_rmap(page + i, vma, addr, RMAP_NONE); } VM_BUG_ON(!pte_none(ptep_get(pte))); set_pte_at(mm, addr, pte, entry); From patchwork Mon Dec 11 15:56:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176762 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7152542vqy; Mon, 11 Dec 2023 07:58:22 -0800 (PST) X-Google-Smtp-Source: AGHT+IHSZKeB9MvfD14FsQbjnUkIvbbwtuOW0NhyB0BROnI4VMtYYmaM0cqVYMQmb2ZGJ+1QQPUC X-Received: by 2002:a05:6a00:244a:b0:6ce:7ef2:40a2 with SMTP id d10-20020a056a00244a00b006ce7ef240a2mr4873233pfj.42.1702310302568; Mon, 11 Dec 2023 07:58:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310302; cv=none; d=google.com; s=arc-20160816; b=J8UOoEPT8trLfO0wMASIfguvAzRC960KG5k3zt+ZSKCHR3epgXPi4Sag7b9jUUm2E7 dOWkYs7LiMv0DNnuTm5lcJrBbKPQPLSKA1NA3Ju2+ePzKNuykt0VOCLygX2VfT4+ifr6 ltel+xuWZNxP+AVb6jjPpRuid7y0BfwsrtwFmk9+GHmEmWTfCRTv3iCaIBKA/ooYYcgf vUUuYh17LMoVbzsx2ubgwdRdATbCm/HVuhhHOPBUTUTuJPWWys0f1nSh7JHzMhxkovEv FFjDDivI+iU07bHBa4PHSc/MReQWanRO6kiNEmGjzSJITQGCbj8BMFTtF0BfTu6w2Oij kuMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=UzjxFSmJorC6pNmLrufiiZpWu6JbL+HXYJ1oUv8JfJw=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=MdSUg74184mgg67qrON4wZOPrY1gka4k9Kp34aX5wyyNRX0aLZoCDXW6SQCFhZ0vA+ QPm7nAOo/C/M2ewnjgRnLPFuO7l2dtoHYybrF6jO1/J5Ibm6tgCAfbhzStSK/lbgXuk+ 6RHPVW4MQ18NfDowZugWCJ0Sgz0szp5EQirtPpNpRR13HcaiHWIbOjcUfRLe6jc8hd7r 6IPVHqjjuDAXyKaVpfIs5soEQkfVlwf8QkX+vasnSpgyi9MEFnWlsBYOfYQnniXjI3cN t3+PmaeGzw8ZwbmEFSsObctXKhvY03VuW8dqCL0gQsq/t3Tkxgs5q7f+fUKIzTrs7Okn 2g7A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZSv0bp5t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id d18-20020a056a00199200b006cb997a7799si6305017pfl.67.2023.12.11.07.58.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:58:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZSv0bp5t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id C66FD805ADD4; Mon, 11 Dec 2023 07:58:19 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344537AbjLKP6G (ORCPT + 99 others); Mon, 11 Dec 2023 10:58:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46744 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344565AbjLKP5j (ORCPT ); Mon, 11 Dec 2023 10:57:39 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 68297119 for ; Mon, 11 Dec 2023 07:57:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310260; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UzjxFSmJorC6pNmLrufiiZpWu6JbL+HXYJ1oUv8JfJw=; b=ZSv0bp5tpJK7KzwWeRRRi7MU0gLY0KSzNdnJvJxSNC4j6yNpBCw9M2VUZ99ubhtdt+umoS C1shLfUkgRKXYD1NpX+GZrDgEa3hYlfsef9bnOuOq3lxiyI4InQDmdYjaBiGSrIBlPQwLM DV4GEkvbXLSND6P0qDY7k+3waZX7TGI= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-180-2Y4Cj1kpO9CR1D_HN3qvZQ-1; Mon, 11 Dec 2023 10:57:37 -0500 X-MC-Unique: 2Y4Cj1kpO9CR1D_HN3qvZQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BB36238425B3; Mon, 11 Dec 2023 15:57:36 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id D12261121306; Mon, 11 Dec 2023 15:57:34 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 16/39] mm/huge_memory: page_add_anon_rmap() -> folio_add_anon_rmap_pmd() Date: Mon, 11 Dec 2023 16:56:29 +0100 Message-ID: <20231211155652.131054-17-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:58:19 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001727799169172 X-GMAIL-MSGID: 1785001727799169172 Let's convert remove_migration_pmd(). No need to set RMAP_COMPOUND, that we will remove soon. Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 82ad68fe0d12..b03374d1bb94 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3611,12 +3611,12 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) pmde = pmd_mkdirty(pmde); if (folio_test_anon(folio)) { - rmap_t rmap_flags = RMAP_COMPOUND; + rmap_t rmap_flags = RMAP_NONE; if (!is_readable_migration_entry(entry)) rmap_flags |= RMAP_EXCLUSIVE; - page_add_anon_rmap(new, vma, haddr, rmap_flags); + folio_add_anon_rmap_pmd(folio, new, vma, haddr, rmap_flags); } else { folio_add_file_rmap_pmd(folio, new, vma); } From patchwork Mon Dec 11 15:56:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176768 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7152960vqy; Mon, 11 Dec 2023 07:59:02 -0800 (PST) X-Google-Smtp-Source: AGHT+IHZjAcm7twwwoHvhj11IKeqhs8JlKb1LIjBExuEUWBmQiRXIi+2x4qrKWEzeXfBDfZFLvfv X-Received: by 2002:a17:90a:3049:b0:286:6cd8:ef12 with SMTP id q9-20020a17090a304900b002866cd8ef12mr6509931pjl.42.1702310338648; Mon, 11 Dec 2023 07:58:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310338; cv=none; d=google.com; s=arc-20160816; b=hiz7dXNspUlWZpftTtRfxFqsK5bWGQhWDFNTmKulvdonRQP6BOz5cvw3ZN09lHkSC2 ftIetKAWpbiayrDsrioMnwsDfvdTkt795u3X/Ox4JvGqEpcavN8h/CxSAUUx/ha4VF2q vsDOsL/sX86KOLItJyoeINe/PFx6L6o2rpUz5GSOtmgdWsi2UtSgbFJ+smh4UldULRHZ CbFLMx4/SACJFZ3NyPsyIcbi6sjj3OAch4NZ4+u6PvSOU4H6ITVk8dIz6qQjeacO2B1o afFU5DpVw8UGbUPTs8/ZI8umbR8cvH3XlVUVowzUnbBQmG78oeOm3J2PqQtLw/x0AWMh VUIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JUV4YjA/avU6lZdC2uH9xVMs5FHkuyGC756X6BS6p9Q=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=RruMuj73XdKwog4EKyVZ7Rh6+kAh1ozQPOMzChPG2V02N6Wiy212aHd6dljOVSGntq u0EspId42mJkf5PlrCyttuS+aF/7ci1bOBlHX4BoaiC/4MTwisv5WYuF4q/FGm1SA8S5 2k47Wu214lgWIaQmifZjO1HB1rcm4zexIeuqgBlp5PWNb/ZhI215LM19uhEarChgremV n8N3JtEH3S59Ctv5PU4+7FNS48ASxhGv6jymFuqVo1muPcih68nPNjd8Ogh2gb2vllKJ 1aDV0z8hIY+bx5HyC1VGI0xPZ+uFBXYStkPTdhfDLOdeSgBdExyIwzlVhYqE7w4dZh2z h6eQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=RQAaREd6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id pw9-20020a17090b278900b002867ec14e6fsi6381590pjb.153.2023.12.11.07.58.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:58:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=RQAaREd6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 5BC288046CB4; Mon, 11 Dec 2023 07:58:50 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344628AbjLKP6M (ORCPT + 99 others); Mon, 11 Dec 2023 10:58:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344593AbjLKP5z (ORCPT ); Mon, 11 Dec 2023 10:57:55 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F3B32127 for ; Mon, 11 Dec 2023 07:57:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310264; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JUV4YjA/avU6lZdC2uH9xVMs5FHkuyGC756X6BS6p9Q=; b=RQAaREd6AoShb0ZG4pIn6T5IlJig5g7Ub8H6GjVkU4GdVpxLkmYMzhzD0C8BWpH4FlRz28 ubxQN/APhgnIneSnxZrR+e9+GZrdq6XL9VdJGajoOg+JdQjvlaBL9S/+MS3bbMg4mJ+HVI m8r+XNn0IN67D+Kl+sppHeGC834F/G0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-501-MIk6dcdIMbiRE_u7hAYpUg-1; Mon, 11 Dec 2023 10:57:39 -0500 X-MC-Unique: MIk6dcdIMbiRE_u7hAYpUg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 134DA870831; Mon, 11 Dec 2023 15:57:39 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 012041121306; Mon, 11 Dec 2023 15:57:36 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 17/39] mm/migrate: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Mon, 11 Dec 2023 16:56:30 +0100 Message-ID: <20231211155652.131054-18-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:58:50 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001765698271365 X-GMAIL-MSGID: 1785001765698271365 Let's convert remove_migration_pte(). Signed-off-by: David Hildenbrand --- mm/migrate.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index efc19f53b05e..0e78680589bc 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -259,8 +259,8 @@ static bool remove_migration_pte(struct folio *folio, #endif { if (folio_test_anon(folio)) - page_add_anon_rmap(new, vma, pvmw.address, - rmap_flags); + folio_add_anon_rmap_pte(folio, new, vma, + pvmw.address, rmap_flags); else folio_add_file_rmap_pte(folio, new, vma); set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); From patchwork Mon Dec 11 15:56:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176772 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7153218vqy; Mon, 11 Dec 2023 07:59:25 -0800 (PST) X-Google-Smtp-Source: AGHT+IHP714Yal8SfmqKpyxZvwh1wTQBvCxua05sPOXk0waoG5EPULoh5onvFpU093p3vtNhQ9Jj X-Received: by 2002:a17:90a:f16:b0:286:6cc0:b8fc with SMTP id 22-20020a17090a0f1600b002866cc0b8fcmr1621139pjy.51.1702310363349; Mon, 11 Dec 2023 07:59:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310363; cv=none; d=google.com; s=arc-20160816; b=zZt2tzEbqxZcG9dWIZm+Nxrt1FG128wEua+JIIL5sd/08eSXt+z7KyhSrh+leM/Igh G6c0RqDax2x/vcCMAeNWZEnaK8PtfMDQ4eYR5+Af4R0qzrM1q0lZO0hAoDtrs2ff+8IJ k1etQeY8avFknkFx9QnyuwvhVcBtAXFfwKprHso67iTiUjwDmlZMcT8Z3WgFSuTgl1hu yH+oGPWSDkW8sVYv8FrIuaAK/g3csfPmL4htQ7Bnpau/h1IwEKSWeYcO1fToqpG8MlUd GSU09ACzrpA9anC0FWrzk/HPk3sXQ1+YvrCFqIVAnLsMpgLwPk4xOs/QzrfkIsz5E5jT kn+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=6EJB9NGtrAo1f2+F9yoIp/qkcy9rFrccif86+rqYixU=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=p1t148OMXUOGzNYlNUqfRf/lY2nCNP8K7lLNbe/veFeEycdkk5slR8VDcRys0cD1lz kH/LPkWRbfLGD92yO8t8m/sph/OWezN2eXXqoDJxaS8CmfYWfaUyHNoc+GX3Ya+omEqd 4prIeHxd/MOTcBa5HP+z6Nortd1oYUWqfQIkEJJrEmiT0ypCahpWDZVdIHrLlYj4uCKy LgzvHuRFoQtjMhgeAXmtII7aDPpto8xmq/HI8Sh7jMYvzTymgj7J8OWBvWFOpiRvtjtX 32JlXPbrr1YgRT9zwX2ekM33pYhW777XLhAgFF6SLG7ZRGnyT/EAtVWq8iuyAfj04ZaY 73MQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OZB2h8yN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id 22-20020a17090a0c1600b002864fceae0esi6346319pjs.162.2023.12.11.07.59.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:59:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OZB2h8yN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id A4141804C53C; Mon, 11 Dec 2023 07:59:11 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344554AbjLKP6R (ORCPT + 99 others); Mon, 11 Dec 2023 10:58:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39772 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344536AbjLKP55 (ORCPT ); Mon, 11 Dec 2023 10:57:57 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B435A184 for ; Mon, 11 Dec 2023 07:57:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310269; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6EJB9NGtrAo1f2+F9yoIp/qkcy9rFrccif86+rqYixU=; b=OZB2h8yNR6wjBuya4G/w2HVruSvog7S6Z3Zq5ttjb0EY3haMICk+hPf7zV2INahKI0+UqO HuT7kkLy365cPFUROgP3SAEZGn2YP9pkrzfKMaoq91Q5MDMxJMVKTpPO/95ntdjx4jmmf7 0R3s+s8SX5hA7pbR58y7leuokn1TsZ8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-418-_8Xjipn3N8OVVf8cQocN-w-1; Mon, 11 Dec 2023 10:57:42 -0500 X-MC-Unique: _8Xjipn3N8OVVf8cQocN-w-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9971B101E155; Mon, 11 Dec 2023 15:57:41 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 587D01121306; Mon, 11 Dec 2023 15:57:39 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 18/39] mm/ksm: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Mon, 11 Dec 2023 16:56:31 +0100 Message-ID: <20231211155652.131054-19-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:59:12 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001791555749311 X-GMAIL-MSGID: 1785001791555749311 Let's convert replace_page(). While at it, perform some folio conversion. Signed-off-by: David Hildenbrand --- mm/ksm.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index b93389a3780e..2b6888ad1470 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1199,6 +1199,7 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, static int replace_page(struct vm_area_struct *vma, struct page *page, struct page *kpage, pte_t orig_pte) { + struct folio *kfolio = page_folio(kpage); struct mm_struct *mm = vma->vm_mm; struct folio *folio; pmd_t *pmd; @@ -1238,15 +1239,16 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, goto out_mn; } VM_BUG_ON_PAGE(PageAnonExclusive(page), page); - VM_BUG_ON_PAGE(PageAnon(kpage) && PageAnonExclusive(kpage), kpage); + VM_BUG_ON_FOLIO(folio_test_anon(kfolio) && PageAnonExclusive(kpage), + kfolio); /* * No need to check ksm_use_zero_pages here: we can only have a * zero_page here if ksm_use_zero_pages was enabled already. */ if (!is_zero_pfn(page_to_pfn(kpage))) { - get_page(kpage); - page_add_anon_rmap(kpage, vma, addr, RMAP_NONE); + folio_get(kfolio); + folio_add_anon_rmap_pte(kfolio, kpage, vma, addr, RMAP_NONE); newpte = mk_pte(kpage, vma->vm_page_prot); } else { /* From patchwork Mon Dec 11 15:56:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176764 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7152645vqy; Mon, 11 Dec 2023 07:58:36 -0800 (PST) X-Google-Smtp-Source: AGHT+IFMS9WSK8tKyB5HzRDh+4L+R6fNi7XdS+kylLBoNHpTj2iktjI7SpIps/uIso9TEox6EEZz X-Received: by 2002:a17:902:780f:b0:1d3:1da3:4793 with SMTP id p15-20020a170902780f00b001d31da34793mr907437pll.133.1702310314477; Mon, 11 Dec 2023 07:58:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310314; cv=none; d=google.com; s=arc-20160816; b=0QW+dLSBwIunfRMCp9SqsfmcHMcEBX1Z350yWG/AKTqdVmegk/rzx6YeaDyJBM5eGG EiLgx5SnPea4uvGM0kYPvwiQHNh4kk1gZDcoaSXTkDmi5aXgBKMlTLOF7SD1oOv7WlfE k9zHCt74DPbSMItCvp20bbFQCKmJlW074Uud0WtmNtd5UnDkPt/h6oCkORb8NvmCmINU Y4EXGeyh87EoZKYzBAUU9td/5tFiIkNLUHRWJ8XHxS+324WUxraZN7mrJFl5qmI0NvrI RjOrRu/P0+TNOfYhvJfv8JXLcvyE/3l4NGosNQx6UOShxYjfPeZM7dObkxXcPUfONtqs B3tA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=g4JhnXoUG0c2uPae4pqIp1MlWaLrLx7AL/tRs/hDXm4=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=LXSZUOd4YB+/A5PnjQw8xxHHMPyqzIXnM4mqkgG/rlFR8LfVzp1DV69VOuhw1Ay6Nv 4IFvdvgg5ZgDdQ+1+FHGbWB4tUzLsgjbWYgC5huI7sej+q3y3qBaC107wNiC1IRILZ0V QVjZ+n18K9KURcIxO8evV5fxhqY5nZENRzYo2VkT3hi3w5noa2Yw3ELf9bA+ppjfCdt9 gv7Tl1XbQLLMgzAp6HShg7dJZKbUu+R+HdiZVdoULUYLZNp1TOwwOqFoTXREfc0ITosi 6SpN0MaR7XN/uLGxLZDXXCyQk/B3ka/DDWDXxlOHX1R/wk1IngpM+65DzdHqTircSVtd ++lQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="aOzALQX/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id t9-20020a170902e84900b001cc31c32e0fsi6423168plg.91.2023.12.11.07.58.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:58:34 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="aOzALQX/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id B3344803D811; Mon, 11 Dec 2023 07:58:26 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344607AbjLKP6O (ORCPT + 99 others); Mon, 11 Dec 2023 10:58:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39738 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344602AbjLKP54 (ORCPT ); Mon, 11 Dec 2023 10:57:56 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A8704130 for ; Mon, 11 Dec 2023 07:57:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310267; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g4JhnXoUG0c2uPae4pqIp1MlWaLrLx7AL/tRs/hDXm4=; b=aOzALQX/mrlBwXGmBW6nwpSZdZ3iCOloKmuf6bFZnVshbj4+pEf+8m/JeoMODPY3yKWJ7C X2geo8IGhdz4WIjkpCkSLSiLbNpwwDiE3nUOKUhGeTAEViAKNF8/6ibceRHl5S2CwUH3QZ S6YvN3rfpmhedIpsjpqyarJl7M9Vxrs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-350-BCpQV4YyOqmsm67nADb-VQ-1; Mon, 11 Dec 2023 10:57:45 -0500 X-MC-Unique: BCpQV4YyOqmsm67nADb-VQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 918BA848A64; Mon, 11 Dec 2023 15:57:44 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 03BE91121306; Mon, 11 Dec 2023 15:57:41 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 19/39] mm/swapfile: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Mon, 11 Dec 2023 16:56:32 +0100 Message-ID: <20231211155652.131054-20-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:58:26 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001740196437896 X-GMAIL-MSGID: 1785001740196437896 Let's convert unuse_pte(). Signed-off-by: David Hildenbrand --- mm/swapfile.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 8be70912e298..25f53bec5097 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1805,7 +1805,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, if (pte_swp_exclusive(old_pte)) rmap_flags |= RMAP_EXCLUSIVE; - page_add_anon_rmap(page, vma, addr, rmap_flags); + folio_add_anon_rmap_pte(folio, page, vma, addr, rmap_flags); } else { /* ksm created a completely new copy */ page_add_new_anon_rmap(page, vma, addr); lru_cache_add_inactive_or_unevictable(page, vma); From patchwork Mon Dec 11 15:56:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176765 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7152784vqy; Mon, 11 Dec 2023 07:58:45 -0800 (PST) X-Google-Smtp-Source: AGHT+IERwXLRhLkPMLUpk4s3Yu91PC+WALqCo/eJ8uNb9Kb8txOzj2WxfUujxskL2woPanQCRqlf X-Received: by 2002:a17:903:11c8:b0:1d0:a720:1fbc with SMTP id q8-20020a17090311c800b001d0a7201fbcmr4943412plh.102.1702310325251; Mon, 11 Dec 2023 07:58:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310325; cv=none; d=google.com; s=arc-20160816; b=E3E/+j9J5EnPeCELSMsp+JBjnefA8xYFkGtHqVXEaggXL5bQ0/9sTRQQpODp22xZpE dt/aK3thVNNhXpkoiyudk9WUEfXBfrwpSDtCxAgkFxfH7BTffTbljnfFVG7kDuwvyRTz Drzy1bKV0bALxWGvLEsOjOKo2L6TrvZiN1R4rTIbZANRA5LljVEfr+kf+7zRSsxKTbCr Sjc+ViOtYdm5NDoRORiVD6clW6zwGXUyYZ5aZ19yu18hWj6i/5FreBxESI3ND0YicMsv 7qmqs+mEwoSgU4JqfWlU2mqAop6XrTHjtx9+R+i2IhI/pekBBzfW6TUFsLSm1K7jnSbS nqjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ErbCAGluk3VXsqVDEiyD10ub89UxQEpB3+XojBc5SDY=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=Af9d5QY+M0f8d2i+C05jpNjsxSRdNUzSMLnUneeyJhflqKLItNYu37tpZWi3jlO2qc /bnBChCtpjBmxsJH4rZ6/K/GAPc6/mLtxYinZQlAwe30QaEVHew2T9CQ+Oe4+qsC9wdM FA3ViSlWBfiCtHsuogzjNwyuUeIdWVAkVQV23mSCPDq6CSbPCwppFnMI2tPEDlrY4vgK mkjm3Nb4OiZQEWJ7/+R3ceiSLKkJ4pK/sJocx6WS3NDyC+//Fxi6Xxawy42F8+zdBGdb ZrJxEGHQqixsyUWRAlaUGf0rGkhrzRVwlZq9XNn7lT3UihAufJN2FqSzncyXt1ZKEQAb JNMg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=d3ue1kfi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id r24-20020a170902be1800b001cf5c99a61csi6149600pls.180.2023.12.11.07.58.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:58:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=d3ue1kfi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 7B954805ADEA; Mon, 11 Dec 2023 07:58:42 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344537AbjLKP6U (ORCPT + 99 others); Mon, 11 Dec 2023 10:58:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39790 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344550AbjLKP57 (ORCPT ); Mon, 11 Dec 2023 10:57:59 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3657218B for ; Mon, 11 Dec 2023 07:57:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310271; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ErbCAGluk3VXsqVDEiyD10ub89UxQEpB3+XojBc5SDY=; b=d3ue1kfigJmW7Qyl3FBhDf1gxNbrsTPwuHZuEFMSXkJBWM/Qr3kV6QJhsDT1lDmScT51X6 OTgxkz1J6Ts4hGCXvRSFFxw4NEnP9OrAmIuSqlX3nkdDdG79hBthjM/7Uby8pM0WAD21gw KqbJRhk4WS0PhZZhfbgg86uzg1rDiE8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-690-mgQppAEyP5iFD7nuwA5GFg-1; Mon, 11 Dec 2023 10:57:47 -0500 X-MC-Unique: mgQppAEyP5iFD7nuwA5GFg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 57849870831; Mon, 11 Dec 2023 15:57:47 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id F2D511121306; Mon, 11 Dec 2023 15:57:44 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 20/39] mm/memory: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Mon, 11 Dec 2023 16:56:33 +0100 Message-ID: <20231211155652.131054-21-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:58:42 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001751538830042 X-GMAIL-MSGID: 1785001751538830042 Let's convert restore_exclusive_pte() and do_swap_page(). While at it, perform some folio conversion in restore_exclusive_pte(). Signed-off-by: David Hildenbrand --- mm/memory.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 70754fd65788..97e064883992 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -710,6 +710,7 @@ static void restore_exclusive_pte(struct vm_area_struct *vma, struct page *page, unsigned long address, pte_t *ptep) { + struct folio *folio = page_folio(page); pte_t orig_pte; pte_t pte; swp_entry_t entry; @@ -725,14 +726,15 @@ static void restore_exclusive_pte(struct vm_area_struct *vma, else if (is_writable_device_exclusive_entry(entry)) pte = maybe_mkwrite(pte_mkdirty(pte), vma); - VM_BUG_ON(pte_write(pte) && !(PageAnon(page) && PageAnonExclusive(page))); + VM_BUG_ON_FOLIO(pte_write(pte) && (!folio_test_anon(folio) && + PageAnonExclusive(page)), folio); /* * No need to take a page reference as one was already * created when the swap entry was made. */ - if (PageAnon(page)) - page_add_anon_rmap(page, vma, address, RMAP_NONE); + if (folio_test_anon(folio)) + folio_add_anon_rmap_pte(folio, page, vma, address, RMAP_NONE); else /* * Currently device exclusive access only supports anonymous @@ -4073,7 +4075,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) page_add_new_anon_rmap(page, vma, vmf->address); folio_add_lru_vma(folio, vma); } else { - page_add_anon_rmap(page, vma, vmf->address, rmap_flags); + folio_add_anon_rmap_pte(folio, page, vma, vmf->address, + rmap_flags); } VM_BUG_ON(!folio_test_anon(folio) || From patchwork Mon Dec 11 15:56:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176780 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7153669vqy; Mon, 11 Dec 2023 07:59:59 -0800 (PST) X-Google-Smtp-Source: AGHT+IHWaWzEnTb5C5ukpyZpRv4NdHo5ueofosGm824PD/++mpsccapAi9Aup51wyMoCM718Jb5s X-Received: by 2002:a05:6a20:3942:b0:190:4ca:8c05 with SMTP id r2-20020a056a20394200b0019004ca8c05mr2625598pzg.100.1702310392654; Mon, 11 Dec 2023 07:59:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310392; cv=none; d=google.com; s=arc-20160816; b=IkzSTe+5+/i8voSmz5MLpbDHB4VemNimg7xr5pdaZnrIRJoXWc5ww2XJy4X1LDgOrm epJSr0z05r0iRhJyc6ZSXyjZPDIbqhWbeow0RwOwHZnH36mwXBgzF1aYwjmITklUixS3 dgqHJ4uOWoCsjgKX9Wnb07VEds1TXlzxxU22fbEw2BuUg6GhydqLOhSnEhanhdQPDlvB 0x/YPM6USOPhAT57Ud04dfrUmfh6zLnKTmU7tATWcNPd4quxed9Uf8OWus7tHQH+QWIN J0lB+hb0FECMoQutq4+RFfCIKeAwXXXRzioAGNHZrx/07TML48B+/YVPjgNIU+c9WLbl CsUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=kqof2lbc18NuxxZ3avfpRF1Wh/FjZz2qWodEICCpfdI=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=k08c2hcrWSz2v2pnhqEuQxMLWK6zUagAmCnRm9mN3uizzWyhlIgLxZUUpPsOvhgJHA n1yhRhjGbBasPObH7uVTYTWar5ILr02+3EVJbAGrv5dd7z7KAhTm9Q3MR+cUt3asVwHt otR/Nt05t/9hTi41TK9Y3LVJ/2MMFeAaRvspg7EgQOZxS7PnJDpC99vhXxK8W4DCHxjN jKLQShzkcxSFen9NJhU75hl7KhFi2OQp9oHLm28XAUtZaosB1JPjEcX6pFbxt6QtDrXC VJRdHGcFwaOrIWHvsc/+qPqh6LNJ7LNy1GtI47xxXpfqFp4bdiAUlDyCxd6OPv0LaUPW Gshw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Dek82Fcf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id s2-20020a63f042000000b005c6859735fbsi5990551pgj.770.2023.12.11.07.59.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:59:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Dek82Fcf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 2B925804C213; Mon, 11 Dec 2023 07:59:41 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344789AbjLKP7b (ORCPT + 99 others); Mon, 11 Dec 2023 10:59:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39738 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344699AbjLKP6b (ORCPT ); Mon, 11 Dec 2023 10:58:31 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5E0B199 for ; Mon, 11 Dec 2023 07:58:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310308; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kqof2lbc18NuxxZ3avfpRF1Wh/FjZz2qWodEICCpfdI=; b=Dek82FcfP1ePG6KpQA36jOa71CLOX8vBP6c49VSUAwFAEGwvnTWYhOui8dnvFeadqhmPsb sheNBAJO1ADgXzLEuSZ2nJSes2PgmorlECakazllY+REaX+x4s5k0uNHXmZ4oUIT77bvzL GS8XUb3PE0UCDn6MqA4Q6ddZUuYmAc4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-146-nQ4jaL5vOtq2vnJawkD1nA-1; Mon, 11 Dec 2023 10:58:04 -0500 X-MC-Unique: nQ4jaL5vOtq2vnJawkD1nA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E265A85A58A; Mon, 11 Dec 2023 15:57:49 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id B71401121306; Mon, 11 Dec 2023 15:57:47 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 21/39] mm/rmap: remove page_add_anon_rmap() Date: Mon, 11 Dec 2023 16:56:34 +0100 Message-ID: <20231211155652.131054-22-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:59:41 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001822332632646 X-GMAIL-MSGID: 1785001822332632646 All users are gone, remove it and all traces. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 2 -- mm/rmap.c | 31 ++++--------------------------- 2 files changed, 4 insertions(+), 29 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 3b5357cb1c09..bd4edae4dbe7 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -240,8 +240,6 @@ void folio_add_anon_rmap_ptes(struct folio *, struct page *, int nr_pages, folio_add_anon_rmap_ptes(folio, page, 1, vma, address, flags) void folio_add_anon_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); -void page_add_anon_rmap(struct page *, struct vm_area_struct *, - unsigned long address, rmap_t flags); void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address); void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, diff --git a/mm/rmap.c b/mm/rmap.c index 7787499fa2ad..83cba8909848 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1271,7 +1271,7 @@ static void __page_check_anon_rmap(struct folio *folio, struct page *page, * The page's anon-rmap details (mapping and index) are guaranteed to * be set up correctly at this point. * - * We have exclusion against page_add_anon_rmap because the caller + * We have exclusion against folio_add_anon_rmap_*() because the caller * always holds the page locked. * * We have exclusion against page_add_new_anon_rmap because those pages @@ -1284,29 +1284,6 @@ static void __page_check_anon_rmap(struct folio *folio, struct page *page, page); } -/** - * page_add_anon_rmap - add pte mapping to an anonymous page - * @page: the page to add the mapping to - * @vma: the vm area in which the mapping is added - * @address: the user virtual address mapped - * @flags: the rmap flags - * - * The caller needs to hold the pte lock, and the page must be locked in - * the anon_vma case: to serialize mapping,index checking after setting, - * and to ensure that PageAnon is not being upgraded racily to PageKsm - * (but PageKsm is never downgraded to PageAnon). - */ -void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, - unsigned long address, rmap_t flags) -{ - struct folio *folio = page_folio(page); - - if (likely(!(flags & RMAP_COMPOUND))) - folio_add_anon_rmap_pte(folio, page, vma, address, flags); - else - folio_add_anon_rmap_pmd(folio, page, vma, address, flags); -} - static __always_inline void __folio_add_anon_rmap(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *vma, unsigned long address, rmap_t flags, enum rmap_mode mode) @@ -1420,7 +1397,7 @@ void folio_add_anon_rmap_pmd(struct folio *folio, struct page *page, * @vma: the vm area in which the mapping is added * @address: the user virtual address mapped * - * Like page_add_anon_rmap() but must only be called on *new* folios. + * Like folio_add_anon_rmap_*() but must only be called on *new* folios. * This means the inc-and-test can be bypassed. * The folio does not have to be locked. * @@ -1480,7 +1457,7 @@ static __always_inline void __folio_add_file_rmap(struct folio *folio, if (nr) __lruvec_stat_mod_folio(folio, NR_FILE_MAPPED, nr); - /* See comments in page_add_anon_rmap() */ + /* See comments in folio_add_anon_rmap_*() */ if (!folio_test_large(folio)) mlock_vma_folio(folio, vma); } @@ -1594,7 +1571,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, /* * It would be tidy to reset folio_test_anon mapping when fully - * unmapped, but that might overwrite a racing page_add_anon_rmap + * unmapped, but that might overwrite a racing folio_add_anon_rmap_*() * which increments mapcount after us but sets mapping before us: * so leave the reset to free_pages_prepare, and remember that * it's only reliable while mapped. From patchwork Mon Dec 11 15:56:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176766 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7152849vqy; Mon, 11 Dec 2023 07:58:50 -0800 (PST) X-Google-Smtp-Source: AGHT+IFTamp4W7bCpT+tY1u9YGIAVQXpt0QrQzLlyI7Gj5OLuFx5LWy5jAAuXk27jInF2mJRxO6A X-Received: by 2002:a05:6a21:18c:b0:190:6bc:adf0 with SMTP id le12-20020a056a21018c00b0019006bcadf0mr2541567pzb.50.1702310330381; Mon, 11 Dec 2023 07:58:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310330; cv=none; d=google.com; s=arc-20160816; b=SVYwwcNTlpFEeP2oEjub6qTHd+R16juzZDLkbSMYv/7mO1A35YigXIRzk/2Jrxrnto cbiZOKLkPn3Q0g6xNGmNZZz5TED4ma+xpVoh2fCKQqnTveoBmZ9kMi6NytGMUTmbLzTD 4evZ5n/hB6gzWQt9ITYEToCr19OXVC8krknfpj+j1wPDV5pe9D8D2Gr0Pf3OSAGG4+KO Ab6bZoU6kOPL8n7qUBnJl1Rb17PfXaK15evg2JbYCRwv0TfHTnrEKzslIq6Q1Zc+ysKy hpidzxUHaR2jxCYjzkgH+e04851xQ6cvZKtwONZQ4QiycPN1Vz5/kG9YkJSw8DbeNeh9 1aBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KuC5xwuqcP7rQWdUGs+5SfITvp1Oi5oe13QDRULS4WU=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=vSkAzFb4PcQi56sMSphFk3jmmciO7Nj2YcBjQ0VoHYVKAixrc8aYiXC5wyWXXqB/hG BkmJx9kAUFv8acJscuHRpVQz4oDjdDUOqSihEQaT4T6rS++Vwa9KqxVsVtsYokYvz5uH 9uu8L/V3dAvBTcoUceEns0436UX/4khtLNZPelYOWl7fbBLfMaSD3rs5lcqTmYrdNBAq 1F1IEK7lg4j/2ifLB4dwNUGD2U+iGEgDEForBfPMCJvzdanbcifas0PZOMSUcwMxoPVK 3E7BW6UgabZCFJVO4+IFDSWAUOOyqyzlMejENZ2rn+y6qBbTNYqJxyM40ZOOGiXkExmt 2a+w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=HVGBbowy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id p125-20020a634283000000b0059c0f9ef964si6100912pga.635.2023.12.11.07.58.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:58:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=HVGBbowy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 09FE2803D826; Mon, 11 Dec 2023 07:58:47 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344552AbjLKP60 (ORCPT + 99 others); Mon, 11 Dec 2023 10:58:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54534 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344570AbjLKP6B (ORCPT ); Mon, 11 Dec 2023 10:58:01 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D11F018F for ; Mon, 11 Dec 2023 07:57:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310277; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KuC5xwuqcP7rQWdUGs+5SfITvp1Oi5oe13QDRULS4WU=; b=HVGBbowyJeZXhiFjYSfHcHcDig0oaeQOMpoMNqRNrvNHxB+p+gOi9qcEBWMvgX+SpuV5M+ WgSGu556vNpqZoqW8PeSsNg5VXfCYCSp9Yutb4mMkrPJHu4tbclv5Ae6YqUoH2ZrmHEEUR hY3tMyiDlb/3GyuTsSmjHhA9sy0fdC8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-473-hqCqiiJtNeeYflAwvpNPtw-1; Mon, 11 Dec 2023 10:57:52 -0500 X-MC-Unique: hqCqiiJtNeeYflAwvpNPtw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D8E6585A597; Mon, 11 Dec 2023 15:57:51 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 25E491121306; Mon, 11 Dec 2023 15:57:50 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 22/39] mm/rmap: remove RMAP_COMPOUND Date: Mon, 11 Dec 2023 16:56:35 +0100 Message-ID: <20231211155652.131054-23-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:58:47 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001756726826257 X-GMAIL-MSGID: 1785001756726826257 No longer used, let's remove it and clarify RMAP_NONE/RMAP_EXCLUSIVE a bit. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 12 +++--------- mm/rmap.c | 2 -- 2 files changed, 3 insertions(+), 11 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index bd4edae4dbe7..0acebe41ab8e 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -177,20 +177,14 @@ struct anon_vma *folio_get_anon_vma(struct folio *folio); typedef int __bitwise rmap_t; /* - * No special request: if the page is a subpage of a compound page, it is - * mapped via a PTE. The mapped (sub)page is possibly shared between processes. + * No special request: A mapped anonymous (sub)page is possibly shared between + * processes. */ #define RMAP_NONE ((__force rmap_t)0) -/* The (sub)page is exclusive to a single process. */ +/* The anonymous (sub)page is exclusive to a single process. */ #define RMAP_EXCLUSIVE ((__force rmap_t)BIT(0)) -/* - * The compound page is not mapped via PTEs, but instead via a single PMD and - * should be accounted accordingly. - */ -#define RMAP_COMPOUND ((__force rmap_t)BIT(1)) - /* * Internally, we're using an enum to specify the granularity. Usually, * we make the compiler create specialized variants for the different diff --git a/mm/rmap.c b/mm/rmap.c index 83cba8909848..9212726268ba 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2663,8 +2663,6 @@ void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc) * The following two functions are for anonymous (private mapped) hugepages. * Unlike common anonymous pages, anonymous hugepages have no accounting code * and no lru code, because we handle hugepages differently from common pages. - * - * RMAP_COMPOUND is ignored. */ void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address, rmap_t flags) From patchwork Mon Dec 11 15:56:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176775 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7153395vqy; Mon, 11 Dec 2023 07:59:33 -0800 (PST) X-Google-Smtp-Source: AGHT+IHtmqwSCoCwcd5ITt5azze8rBDVEuSomz2/sq1ULertqcc7NFXYCFE4GW/pKV3rJF5Zc/A/ X-Received: by 2002:a17:902:6805:b0:1d0:488e:381d with SMTP id h5-20020a170902680500b001d0488e381dmr1660888plk.61.1702310371427; Mon, 11 Dec 2023 07:59:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310371; cv=none; d=google.com; s=arc-20160816; b=05SnNkIbt/SrNrfSTwoUpCOJRqGKP/elqj2I6asX90CyxVm3gJcpJTz5Z0/GHQRCwz coiJPyuc8MHO7FO1j46JGI9Srei04R6QPhXu8CRgOE5xAMSdTzFsT12/8QsEa2uQyxom k2icCL3GduZBedZLJDUaMjpWlThsb3YbyqSWGVyoLtAAb/wHtUV4ACr+55j09oCWNoLx CYFA5aZ6s63ezdXrQb82K9kBXWgUPRmAMQDF2pQUxBvJHlE/nyz2N3cukc7Y89GLtD+7 ud3VSLsSA6IMvHUGcetHpkOYQu5/H/3bfCYGhWUPKUDK+6fID/MeAAyJekCge+RkASAo 2+4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=OcVx4x0aUtY07dVPvydbU+uc8icopPtFroqSSpn7QWM=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=Yzdd23+fkItDqssXhcbSfJVXieC0jtgom5eOBxjtBuGRULwJYUZbepc4Ggozvwiwxe XTymp+s3a1xEVbWS+hUOitxOA5RdXvXUcIsaSP0sAcvJhb4gnaX4WRYBuZqePrcbsf17 FfpIkpqfxhgW55KXuaBvYFAjpBFWKgrWCTp/3lm4cHS3+mYHVTdau5OmfLc2cVfCYcQy s9cmgUXZXxyMDZ3kjKLWWTgGNtIgR+vdLLsrIZdcbcI9dY1c4hJXKSzCJiPZkILViDm1 Hy6KXvt9IQJFKdGzqjSR4es64byHu/eD/U+fHZEzF/lkUj5uqAM4pRLezWM5tI880Q0D v7dA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=cNvtRwPL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id p8-20020a170903248800b001d0c906f613si6095947plw.492.2023.12.11.07.59.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:59:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=cNvtRwPL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 83AF180448F6; Mon, 11 Dec 2023 07:59:22 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344607AbjLKP6a (ORCPT + 99 others); Mon, 11 Dec 2023 10:58:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344594AbjLKP6M (ORCPT ); Mon, 11 Dec 2023 10:58:12 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7720B1B2 for ; Mon, 11 Dec 2023 07:58:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310279; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OcVx4x0aUtY07dVPvydbU+uc8icopPtFroqSSpn7QWM=; b=cNvtRwPL3eFiW9oT7XnqoauygX8Wp/nkgWCFayufGsupZJlWy+PWrqNt8WWjAbuFWlWiQj Uvar4uVNKuSJn1s0R+dmBV8X38cVEvbZ8mgQoPxH6uB3pFkY+88RPKgD26QJr6NBvsLVeh bIgjFw05c0IQb4Dv6CyMaklC/aU+hZo= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-177-0UIjTO6mNmu_u3e5pNi68Q-1; Mon, 11 Dec 2023 10:57:54 -0500 X-MC-Unique: 0UIjTO6mNmu_u3e5pNi68Q-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3BFCA1C08B9E; Mon, 11 Dec 2023 15:57:54 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3E8C71121306; Mon, 11 Dec 2023 15:57:52 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 23/39] mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]() Date: Mon, 11 Dec 2023 16:56:36 +0100 Message-ID: <20231211155652.131054-24-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:59:23 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001800187391356 X-GMAIL-MSGID: 1785001800187391356 Let's mimic what we did with folio_add_file_rmap_*() and folio_add_anon_rmap_*() so we can similarly replace page_remove_rmap() next. Make the compiler always special-case on the granularity by using __always_inline. We're adding folio_remove_rmap_ptes() handling right away, as we want to use that soon for batching rmap operations when unmapping PTE-mapped large folios. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 6 ++++ mm/rmap.c | 79 ++++++++++++++++++++++++++++++++++++-------- 2 files changed, 71 insertions(+), 14 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 0acebe41ab8e..a266dc0ef99e 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -246,6 +246,12 @@ void folio_add_file_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); +void folio_remove_rmap_ptes(struct folio *, struct page *, int nr_pages, + struct vm_area_struct *); +#define folio_remove_rmap_pte(folio, page, vma) \ + folio_remove_rmap_ptes(folio, page, 1, vma) +void folio_remove_rmap_pmd(struct folio *, struct page *, + struct vm_area_struct *); void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address, rmap_t flags); diff --git a/mm/rmap.c b/mm/rmap.c index 9212726268ba..dc3be5807cee 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1511,25 +1511,38 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, bool compound) { struct folio *folio = page_folio(page); + + if (likely(!compound)) + folio_remove_rmap_pte(folio, page, vma); + else + folio_remove_rmap_pmd(folio, page, vma); +} + +static __always_inline void __folio_remove_rmap(struct folio *folio, + struct page *page, int nr_pages, struct vm_area_struct *vma, + enum rmap_mode mode) +{ atomic_t *mapped = &folio->_nr_pages_mapped; - int nr = 0, nr_pmdmapped = 0; - bool last; + int last, nr = 0, nr_pmdmapped = 0; enum node_stat_item idx; - VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); - VM_BUG_ON_PAGE(compound && !PageHead(page), page); + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); /* Is page being unmapped by PTE? Is this its last map to be removed? */ - if (likely(!compound)) { - last = atomic_add_negative(-1, &page->_mapcount); - nr = last; - if (last && folio_test_large(folio)) { - nr = atomic_dec_return_relaxed(mapped); - nr = (nr < COMPOUND_MAPPED); - } - } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */ + switch (mode) { + case RMAP_MODE_PTE: + do { + last = atomic_add_negative(-1, &page->_mapcount); + if (last && folio_test_large(folio)) { + last = atomic_dec_return_relaxed(mapped); + last = (last < COMPOUND_MAPPED); + } + if (last) + nr++; + } while (page++, --nr_pages > 0); + break; + case RMAP_MODE_PMD: last = atomic_add_negative(-1, &folio->_entire_mapcount); if (last) { nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); @@ -1544,6 +1557,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, nr = 0; } } + break; } if (nr_pmdmapped) { @@ -1565,7 +1579,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, * is still mapped. */ if (folio_test_large(folio) && folio_test_anon(folio)) - if (!compound || nr < nr_pmdmapped) + if (mode == RMAP_MODE_PTE || nr < nr_pmdmapped) deferred_split_folio(folio); } @@ -1580,6 +1594,43 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, munlock_vma_folio(folio, vma); } +/** + * folio_remove_rmap_ptes - remove PTE mappings from a page range of a folio + * @folio: The folio to remove the mappings from + * @page: The first page to remove + * @nr_pages: The number of pages that will be removed from the mapping + * @vma: The vm area from which the mappings are removed + * + * The page range of the folio is defined by [page, page + nr_pages) + * + * The caller needs to hold the page table lock. + */ +void folio_remove_rmap_ptes(struct folio *folio, struct page *page, + int nr_pages, struct vm_area_struct *vma) +{ + __folio_remove_rmap(folio, page, nr_pages, vma, RMAP_MODE_PTE); +} + +/** + * folio_remove_rmap_pmd - remove a PMD mapping from a page range of a folio + * @folio: The folio to remove the mapping from + * @page: The first page to remove + * @vma: The vm area from which the mapping is removed + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock. + */ +void folio_remove_rmap_pmd(struct folio *folio, struct page *page, + struct vm_area_struct *vma) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_remove_rmap(folio, page, HPAGE_PMD_NR, vma, RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + /* * @arg: enum ttu_flags will be passed to this argument */ From patchwork Mon Dec 11 15:56:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176769 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7153040vqy; Mon, 11 Dec 2023 07:59:05 -0800 (PST) X-Google-Smtp-Source: AGHT+IEHuOr+92AzuTqjknSVxofVGXILkYzj2rErEQKwY/MauDeewF+ff5xKsLNdFX4+/pBlu2pj X-Received: by 2002:a05:6a00:2d18:b0:6ce:2731:47b3 with SMTP id fa24-20020a056a002d1800b006ce273147b3mr5455040pfb.19.1702310344106; Mon, 11 Dec 2023 07:59:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310344; cv=none; d=google.com; s=arc-20160816; b=pSE8KYeEUSlwl+XcTvNhxBTZEKf/MQatYHYSBCp4EhiKRbheNjJzia3hBx/D0Mloa2 OBj5EXj7r3SX7Q0wqBz+0TpXTEt5/h0USHlOZGQXsvpbK8Il7StgQ09xojNn64Dm0QIN Bvwv6W3rWz+HaBHOaj52q/AQLGPy4f4w74QiXgtdh13X6kMsdHcPONTGwJZLeQffuQec jVVVciCiP4UjjN7cv+3l5l4oP5LB4WfWZ97ZhxNrakvuEWsXG5tJmBZil1fFI8ZPDgk9 0VSwHspeQmQXsukXa2paMsn64h2gIxBVwhdMYtZ44Xi27jz1T3/Lk8/xj7AoNgJ/QPT9 2ROw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jBiuDcFai57ywCoGHCEF1oxPQjyCBa6ek76+fUhJDqY=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=syTIA6zLdipKQldGYn1F0dWulUkpS96a5tzpA88z0qJ1rVk+OwBPF+c80kRYK18Bs+ EfGhnSN1EpdmAsSHAPuzzhRQ/aEM8v/H3UKB+eLVZJ7tBEz/cq7bTpX85JgbeKU/RStu D1XR3Lilhx7Xi3QiJvzyF4YvGhknX19GQqGqFaLj4hiicZ8x6PltE158RPKU1qz+Y7pt FfBbfkA0s57RJxe/bg8769IBHJi185v2UePWhk0nPIR7GIcckS+3EEO3zE5L0yipbi43 zC+CDyuWR+wJ3/h1sKRpgdkJ1M6DqUKfxwO/gJmwDPk369I/oTe1ol/NEn4viwvD07zP pv9Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=QZZooX0l; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id s13-20020a056a00194d00b006cef51f51bdsi4036338pfk.15.2023.12.11.07.59.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:59:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=QZZooX0l; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 7800A809E8C2; Mon, 11 Dec 2023 07:59:01 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344404AbjLKP6s (ORCPT + 99 others); Mon, 11 Dec 2023 10:58:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45590 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344569AbjLKP6R (ORCPT ); Mon, 11 Dec 2023 10:58:17 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 57858123 for ; Mon, 11 Dec 2023 07:58:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310288; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jBiuDcFai57ywCoGHCEF1oxPQjyCBa6ek76+fUhJDqY=; b=QZZooX0lnGE/RHF7aLRiUS4Vp1k3Kql7J+gSEXg/Greoz6L6y8Gx3yv44vCuo+mwHMgv/+ +VzEisip/hCPZacm2GtlKFAAmIbc5uDQqab4abGGfpPfvEsgc3U9epfh+lm8ZbU21QrnOX 8DaXS9iuTfDBGJQ2+h2hxk6sNXVImiQ= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-252-kgGExcclMx2ItCIeuAt4Eg-1; Mon, 11 Dec 2023 10:57:57 -0500 X-MC-Unique: kgGExcclMx2ItCIeuAt4Eg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F102F1C08BA2; Mon, 11 Dec 2023 15:57:56 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7E1691121306; Mon, 11 Dec 2023 15:57:54 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 24/39] kernel/events/uprobes: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 11 Dec 2023 16:56:37 +0100 Message-ID: <20231211155652.131054-25-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:59:01 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001771447968356 X-GMAIL-MSGID: 1785001771447968356 Let's convert __replace_page(). Signed-off-by: David Hildenbrand --- kernel/events/uprobes.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 435aac1d8c27..16731d240e16 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -198,7 +198,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, set_pte_at_notify(mm, addr, pvmw.pte, mk_pte(new_page, vma->vm_page_prot)); - page_remove_rmap(old_page, vma, false); + folio_remove_rmap_pte(old_folio, old_page, vma); if (!folio_mapped(old_folio)) folio_free_swap(old_folio); page_vma_mapped_walk_done(&pvmw); From patchwork Mon Dec 11 15:56:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176770 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7153127vqy; Mon, 11 Dec 2023 07:59:11 -0800 (PST) X-Google-Smtp-Source: AGHT+IEJX39b7bTqNBUo80Vcg9a//4uudk8SZnsLFeRwYKYFnEAmn8scvcd+wlMJbWuQc7+AvXuz X-Received: by 2002:a05:6a00:198f:b0:6ce:544f:f195 with SMTP id d15-20020a056a00198f00b006ce544ff195mr2148246pfl.60.1702310351319; Mon, 11 Dec 2023 07:59:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310351; cv=none; d=google.com; s=arc-20160816; b=y4oJcGlffWZ4ZRZMMf/YkVHG+ZuPReCLdARSHyZdVWh9zCzpnl9t2wOyf+GXOnE0ND 2vUc1rSFXuw86Ka3xjPpPIbWwdt4IssnMrDmliRttlHD6sBkLrIOyUPbb/v1OzG2kWaI 2dInpFL9kIL8Z+nVQWEr+lwKbBw0SVgDK9kR9aQtep6AFZtktPENWjqfAljYoQOO5+DA KEknlKWJUEsn5VwMw2mNdj39w5Ik5Orf2bS/7uewScv8onuqsyB6xKWdBUI/v8zkSdch fbEIDfO1lWs1OjW5e3uhoM5/w5Yrys5hctbPvHxxVhl4zlJYoy2H1MqCDS5I3mqAvskM Xv0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3iv72ytjW+obJugegMCmaB6S7PQkIGz8PWP7EAJvF9c=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=oGYGC8kXdqqut/T9mLJ68UaBDkn0u3b5k39bsPJ7qYpoBoT9xnxxdbaZcQE87a85g2 ylNWBaCNDroSkZVxERB3/293Fv8npRYPVwqjsgMhUW63QjwoCBC8PJYtKRPSwg4REDFg LBopCOAH8Z8TdJy1Wmr7/RgfJZ66Wy1qSltDBy7zd3rN6FBpWN4arcFDzdWnb4oPRWaj Dd2yt/Px5LUrR64abpkOvrXoklKLzf5SAySDDl87YyGvDxkmhHiwgukn7/v9qFt4sHIB 9r391oRomITuPxM2CDH662m6WE5TuYUe6qzpgF/Jx5F7V5g5BV/NbmN/px+sq4mnPIAP Lbqw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=VAHG4Ljx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id p9-20020a056a0026c900b006cbfbb64e00si6102903pfw.138.2023.12.11.07.59.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:59:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=VAHG4Ljx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 42145805AA00; Mon, 11 Dec 2023 07:59:08 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344609AbjLKP6k (ORCPT + 99 others); Mon, 11 Dec 2023 10:58:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344654AbjLKP6Q (ORCPT ); Mon, 11 Dec 2023 10:58:16 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B5DE511A for ; Mon, 11 Dec 2023 07:58:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310287; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3iv72ytjW+obJugegMCmaB6S7PQkIGz8PWP7EAJvF9c=; b=VAHG4LjxX/ttdv1nerhq1cbTydsDcfyUIomsJ5msHCZaPWUVWjskezoXzanY6CpiKEUfVj S/U35hVnJ8IGmTi0ipFGVGMbIjya1sFSDWyJFA8mFD2myARZ4tGBcTIcMEz/o0jA3qyI7w iAePnpUmH2UE49VCKEi1zJF4g/3jwig= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-328-jn5rk51mP9GOulr35P-yuw-1; Mon, 11 Dec 2023 10:58:00 -0500 X-MC-Unique: jn5rk51mP9GOulr35P-yuw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A2A9610133AE; Mon, 11 Dec 2023 15:57:59 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 35F591121306; Mon, 11 Dec 2023 15:57:57 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 25/39] mm/huge_memory: page_remove_rmap() -> folio_remove_rmap_pmd() Date: Mon, 11 Dec 2023 16:56:38 +0100 Message-ID: <20231211155652.131054-26-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:59:08 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001778721897157 X-GMAIL-MSGID: 1785001778721897157 Let's convert zap_huge_pmd() and set_pmd_migration_entry(). While at it, perform some more folio conversion. Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index b03374d1bb94..cfaa8b823015 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1898,7 +1898,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, if (pmd_present(orig_pmd)) { page = pmd_page(orig_pmd); - page_remove_rmap(page, vma, true); + folio_remove_rmap_pmd(page_folio(page), page, vma); VM_BUG_ON_PAGE(page_mapcount(page) < 0, page); VM_BUG_ON_PAGE(!PageHead(page), page); } else if (thp_migration_supported()) { @@ -2433,12 +2433,13 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, page = pfn_swap_entry_to_page(entry); } else { page = pmd_page(old_pmd); - if (!PageDirty(page) && pmd_dirty(old_pmd)) - set_page_dirty(page); - if (!PageReferenced(page) && pmd_young(old_pmd)) - SetPageReferenced(page); - page_remove_rmap(page, vma, true); - put_page(page); + folio = page_folio(page); + if (!folio_test_dirty(folio) && pmd_dirty(old_pmd)) + folio_set_dirty(folio); + if (!folio_test_referenced(folio) && pmd_young(old_pmd)) + folio_set_referenced(folio); + folio_remove_rmap_pmd(folio, page, vma); + folio_put(folio); } add_mm_counter(mm, mm_counter_file(page), -HPAGE_PMD_NR); return; @@ -2593,7 +2594,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, pte_unmap(pte - 1); if (!pmd_migration) - page_remove_rmap(page, vma, true); + folio_remove_rmap_pmd(folio, page, vma); if (freeze) put_page(page); @@ -3536,6 +3537,7 @@ late_initcall(split_huge_pages_debugfs); int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, struct page *page) { + struct folio *folio = page_folio(page); struct vm_area_struct *vma = pvmw->vma; struct mm_struct *mm = vma->vm_mm; unsigned long address = pvmw->address; @@ -3551,14 +3553,14 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, pmdval = pmdp_invalidate(vma, address, pvmw->pmd); /* See page_try_share_anon_rmap(): invalidate PMD first. */ - anon_exclusive = PageAnon(page) && PageAnonExclusive(page); + anon_exclusive = folio_test_anon(folio) && PageAnonExclusive(page); if (anon_exclusive && page_try_share_anon_rmap(page)) { set_pmd_at(mm, address, pvmw->pmd, pmdval); return -EBUSY; } if (pmd_dirty(pmdval)) - set_page_dirty(page); + folio_set_dirty(folio); if (pmd_write(pmdval)) entry = make_writable_migration_entry(page_to_pfn(page)); else if (anon_exclusive) @@ -3575,8 +3577,8 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, if (pmd_uffd_wp(pmdval)) pmdswp = pmd_swp_mkuffd_wp(pmdswp); set_pmd_at(mm, address, pvmw->pmd, pmdswp); - page_remove_rmap(page, vma, true); - put_page(page); + folio_remove_rmap_pmd(folio, page, vma); + folio_put(folio); trace_set_migration_pmd(address, pmd_val(pmdswp)); return 0; From patchwork Mon Dec 11 15:56:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176767 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7152947vqy; Mon, 11 Dec 2023 07:58:59 -0800 (PST) X-Google-Smtp-Source: AGHT+IGlPyPYcSKyASRaQ009NNF1RwLqgZobxWMy+K2GmqiZgqTyhSiuebzCJy3dZxhKb5uZL3gg X-Received: by 2002:a17:903:32cd:b0:1cc:5dd4:7ce5 with SMTP id i13-20020a17090332cd00b001cc5dd47ce5mr2232354plr.19.1702310338876; Mon, 11 Dec 2023 07:58:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310338; cv=none; d=google.com; s=arc-20160816; b=tS3UeuOrJmeY6zf0fxCD+7ju076HzzmhXMnG+SGBZEAmf5XtBcUHYkSM1k5QDbGn85 oDuh19EMwnYC29VIqyGKhPdpG2o9q77AeNKAmf3LwkUvetZHSG2LDav7DA6PE+VJm+9S 731Ep1tZzFiy8OcoU8bwZOyMj5/TMkIyEKHyUi1N1U+VidkMz7dg/XOupaOVpp1vy+pS 5hmFZVDcHBT9gf8lUi9LTBIExoKAosNfwvmOLC4QtQb8QeTVqgbMG39eHzi43pMH+PGa WlegeOPj/D/werzLw4ed6a0il2r8ReyyfM9S3w+KLndwWSeZur2HLsLRq2n4HEuf7oq9 gRjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=VER2ukZQXsfAHa/RpU4plqm2RzyVwEC40zuZNC17KhY=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=i0wDDFEAXTp2WduSej86bc+FnFwT1CWBBMkP5GF65eTSVSElf/RO+odzCIOKnLM0l/ 3tmYrDT25jZ6GGl+dn9+3LZ/8Y43wYoMORxUxmVZDd6VSEwZJdpqHyQgoqZ30Qw+EUEi 4l8d2Mig38IvgRttmV81Vg+Vm/kKBW64AOGZY/XzqDCvcRh+gZbDqm6WViaeWowkSU5H sUMX3HXu9qV4Zp/ctpqSdAns0CpU6EdsXlMGo44luZqEFB8hLXgCwHIVLFus33x+nbfz 9xfaw+v5tp0E15VfHE5St5ZtPGMNGPLfDVqnFjkoX56ST7f5fRPy5SYCP5auokN9ENQT QItA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=gXhLSx8w; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id f20-20020a631014000000b005bdfd3a269fsi6210617pgl.581.2023.12.11.07.58.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:58:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=gXhLSx8w; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 02C55809E239; Mon, 11 Dec 2023 07:58:56 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344537AbjLKP6n (ORCPT + 99 others); Mon, 11 Dec 2023 10:58:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59348 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344611AbjLKP6R (ORCPT ); Mon, 11 Dec 2023 10:58:17 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07CF111D for ; Mon, 11 Dec 2023 07:58:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310288; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VER2ukZQXsfAHa/RpU4plqm2RzyVwEC40zuZNC17KhY=; b=gXhLSx8wfWrx+SsqvBXh/5A6QnWb3MraItxEdms0AYXbBmLOn1F7PP4FbeiaWZO+HUp6tv uR6aQNCJiR6TP175pkE1MhfuTlR2u/3vQzmT+dpIarTC8P5b5CcwTeTzPxjmONpcyQc0nb z/bwRemW2NiHpt+wwtL53pr3CxdVAFE= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-554-O8xKFmHWPGuFDgNfrHONmg-1; Mon, 11 Dec 2023 10:58:04 -0500 X-MC-Unique: O8xKFmHWPGuFDgNfrHONmg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6E3723C28642; Mon, 11 Dec 2023 15:58:03 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 094C11121312; Mon, 11 Dec 2023 15:57:59 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 26/39] mm/khugepaged: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 11 Dec 2023 16:56:39 +0100 Message-ID: <20231211155652.131054-27-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:58:56 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001765675105643 X-GMAIL-MSGID: 1785001765675105643 Let's convert __collapse_huge_page_copy_succeeded() and collapse_pte_mapped_thp(). While at it, perform some more folio conversion in __collapse_huge_page_copy_succeeded(). We can get rid of release_pte_page(). Signed-off-by: David Hildenbrand --- mm/khugepaged.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index de174d049e71..4d90c9548ec9 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -494,11 +494,6 @@ static void release_pte_folio(struct folio *folio) folio_putback_lru(folio); } -static void release_pte_page(struct page *page) -{ - release_pte_folio(page_folio(page)); -} - static void release_pte_pages(pte_t *pte, pte_t *_pte, struct list_head *compound_pagelist) { @@ -687,6 +682,7 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, spinlock_t *ptl, struct list_head *compound_pagelist) { + struct folio *src_folio; struct page *src_page; struct page *tmp; pte_t *_pte; @@ -708,16 +704,17 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, } } else { src_page = pte_page(pteval); - if (!PageCompound(src_page)) - release_pte_page(src_page); + src_folio = page_folio(src_page); + if (!folio_test_large(src_folio)) + release_pte_folio(src_folio); /* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats - * inside page_remove_rmap(). + * inside folio_remove_rmap_pte(). */ spin_lock(ptl); ptep_clear(vma->vm_mm, address, _pte); - page_remove_rmap(src_page, vma, false); + folio_remove_rmap_pte(src_folio, src_page, vma); spin_unlock(ptl); free_page_and_swap_cache(src_page); } @@ -1624,7 +1621,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * PTE dirty? Shmem page is already dirty; file is read-only. */ ptep_clear(mm, addr, pte); - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(folio, page, vma); nr_ptes++; } From patchwork Mon Dec 11 15:56:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176774 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7153350vqy; Mon, 11 Dec 2023 07:59:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IGiUAmTeJwyIvlA2AVEqC6alkCOEOzNyscJ6Rmg6mOgw5dd2RUuSV/mi0ekedtDRDy451YY X-Received: by 2002:a05:6a20:918d:b0:18f:97c:384c with SMTP id v13-20020a056a20918d00b0018f097c384cmr5872570pzd.38.1702310368292; Mon, 11 Dec 2023 07:59:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310368; cv=none; d=google.com; s=arc-20160816; b=WH0Ct6zyeLMfDW5SLc1ApeelQ6Yta6X+Bu8G2YPTXYi+0qglR45pambM1ZwTraY+3q a/kChaOpkjn/uKWMTyxMwv0gtsERmzJX7NNMtYPmU/wiN62DOuzq+dq/FFOCi0vZ89Y0 oE4RdxBMUVoMRMB2x6NRPJQOGs5Zbmm0wZfZC0OF6Eg+zp1OMUH8J9z6X+/zbK1TuW8Q 8r8q8xuE0LBL4/zzgnUlVHSDwpkepfhcdSuseS5WpqzNykv5N+RzBLW5sl/dNe9+VFV0 aQgdTsT2arkZpiCg+Rkihre1MU0KLH2vT3ISbv5ft1N2KhmgSQHuPGKOKoNTo45a01qY aUXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QtgFnZdp8Ufq96QYGYH/M8uGtDulU5aA6agoBTCHJ/U=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=nOx3w3mdCJWlsj6Rv1OTxKl+YrTTWvkuCGUl5Vsb9YgIgpD7YtmIbuc26ynoGHbrQY i7hW/2z6kJeCy+rHV2ankRi+LiADoZGo+rB4r6UyjHFGQz8qCUJ/a8Rjz41Evxt/nuAw vKqir4qXy//CPKreyppvsZcDoR1prVf/Os2v59ZFwZKysb5er4HwNzUa3tcv2oQ7SptR 9dNjF1WNL6e/PEYDZGMDQDqQ5aHMheK9R5/YXmnziTuCNxbh67/w9rM6h2oBH8obOEEa hcBnvl65OYbNXrGvLnKhV2wPRgfCn3YLVqPsLr9ppGjz/opFC+kYphmgpNxF7yCUPyIc pNsQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ft72nQ3S; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id jc26-20020a056a006c9a00b006cda012bf20si6333322pfb.264.2023.12.11.07.59.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:59:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ft72nQ3S; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id C2735809E8EF; Mon, 11 Dec 2023 07:59:24 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344678AbjLKP7B (ORCPT + 99 others); Mon, 11 Dec 2023 10:59:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46700 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344675AbjLKP6X (ORCPT ); Mon, 11 Dec 2023 10:58:23 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91E98D64 for ; Mon, 11 Dec 2023 07:58:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310291; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QtgFnZdp8Ufq96QYGYH/M8uGtDulU5aA6agoBTCHJ/U=; b=ft72nQ3Se1iosYYxXxQwwUec2mj8WdFs9jlsx7pmbKArZdpz7Qr7HosB6r0+p/PqVTqx39 LxVzavZnk5jjj+yEeGsahqaPH5RF+TjXGDYAP2TtNWCACkWxHbfUqAAYkS+59n+YmsrWZm V5zgDSwPxffg9kE4wzQ5exWO/8yaJEw= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-578-Q5gG0FfpMgG6qcIyHy9Eng-1; Mon, 11 Dec 2023 10:58:06 -0500 X-MC-Unique: Q5gG0FfpMgG6qcIyHy9Eng-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 763472806408; Mon, 11 Dec 2023 15:58:05 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id A56501121306; Mon, 11 Dec 2023 15:58:03 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 27/39] mm/ksm: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 11 Dec 2023 16:56:40 +0100 Message-ID: <20231211155652.131054-28-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:59:25 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001796955328578 X-GMAIL-MSGID: 1785001796955328578 Let's convert replace_page(). Signed-off-by: David Hildenbrand --- mm/ksm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/ksm.c b/mm/ksm.c index 2b6888ad1470..b3d0cfaa2533 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1279,7 +1279,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, set_pte_at_notify(mm, addr, ptep, newpte); folio = page_folio(page); - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(folio, page, vma); if (!folio_mapped(folio)) folio_free_swap(folio); folio_put(folio); From patchwork Mon Dec 11 15:56:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176771 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7153174vqy; Mon, 11 Dec 2023 07:59:15 -0800 (PST) X-Google-Smtp-Source: AGHT+IGIbiKfq9iJ8stwBhlE50IZup2whkaQIsdZsbMNGuzC9UrUVQ/xCeapCQtzt8VOXZNiKms2 X-Received: by 2002:a17:90a:eb0d:b0:286:6cc0:b915 with SMTP id j13-20020a17090aeb0d00b002866cc0b915mr1577539pjz.76.1702310355206; Mon, 11 Dec 2023 07:59:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310355; cv=none; d=google.com; s=arc-20160816; b=JuH6EKjeLWHerGfB1j5CtFZ5rRnN3dlHxyX0uF+Ip/MZNv5rIz6tjm5hs0XLKw0BMg b/ODf2osysgi3u1Ff1bzvU9cQImMlEknvKR4rNxDdKwjc6KedANO5yeKkZ39dGFvKOyj yABv6aa7oiEETP/iLNIwe16ap0Pk8630SJERHPgdyqkI2WAdwif8AOfeQhSDjA+U7UaB o1z3FpUNOAXcI1E+ysgV5D/pGZioK+ts95dUmsMYuahM+zDUd8kwnTIUkZIR9N624r7/ kfM/Zmgaj2ScMHHl8f0GmAAwo4p1XWTjSWZAyDzveYjkJ6HKu/lBIYx5DV13xAcplhfm S0QA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=e8cPKSG16Q/IRVzSDalc66DZf+rAdIx9s1sCKVbyMk0=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=b1PekDpdqoGyrcfOLDOrkF7fSWd5mEXFs4opQeE1IwZGywPzjJz8UnbcT2Ts4FYqCQ iLXUIdgC+KvA988F20kEMYDI9JlSnIqCml5/WokYM47VvJnvrdfCv0hLshVsTjq2Awgr tvAn6j1ZONGWvoYPH6TpG5NYvZwpJSit42Lz9MuCvZG2y4HoZ5vxU5Msf8vIlZlb5sFz w4OyWVd+kdom/I4z9KZ3RAxVl0+QBHMzh3MKkrY3f8JSFdTgAvFc9ND8ytWtLB+hturb +rWjJUoK+2OdgM67OEz3GOJukza7HighDb/1Q2OMtaqLMCKlbp9SHqbxbkJINStAzudA IIWg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bkQgCc7i; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id n10-20020a17090ac68a00b00286a8518f0bsi7553869pjt.78.2023.12.11.07.59.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:59:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bkQgCc7i; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 78CA080620B8; Mon, 11 Dec 2023 07:59:07 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344729AbjLKP6v (ORCPT + 99 others); Mon, 11 Dec 2023 10:58:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42664 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344672AbjLKP6W (ORCPT ); Mon, 11 Dec 2023 10:58:22 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7FC19D63 for ; Mon, 11 Dec 2023 07:58:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310291; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e8cPKSG16Q/IRVzSDalc66DZf+rAdIx9s1sCKVbyMk0=; b=bkQgCc7iScm601TY+SP2/nIMXhGFSByz/BeCUoNQklTwje7J524ass4Py91+umjwUZAskU 36s1fkOsejZ3oR05R8+6I5xDKw+jdVLLvmLMa53LQBeg1miAs/6iNZTaiO6kqaiAbqdq4u CJOeOlC5Xoc7F1dfMIQ5NAQd8sfkl3c= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-611-Qc9Fji3rOuid9DMrJtMfFQ-1; Mon, 11 Dec 2023 10:58:08 -0500 X-MC-Unique: Qc9Fji3rOuid9DMrJtMfFQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CA5991C08BA2; Mon, 11 Dec 2023 15:58:07 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id BCEA41121306; Mon, 11 Dec 2023 15:58:05 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 28/39] mm/memory: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 11 Dec 2023 16:56:41 +0100 Message-ID: <20231211155652.131054-29-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:59:07 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001782841214643 X-GMAIL-MSGID: 1785001782841214643 Let's convert zap_pte_range() and closely-related tlb_flush_rmap_batch(). While at it, perform some more folio conversion in zap_pte_range(). Signed-off-by: David Hildenbrand --- mm/memory.c | 23 +++++++++++++---------- mm/mmu_gather.c | 2 +- 2 files changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 97e064883992..9a5724cf895f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1434,6 +1434,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent = ptep_get(pte); + struct folio *folio; struct page *page; if (pte_none(ptent)) @@ -1459,21 +1460,22 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } + folio = page_folio(page); delay_rmap = 0; - if (!PageAnon(page)) { + if (!folio_test_anon(folio)) { if (pte_dirty(ptent)) { - set_page_dirty(page); + folio_set_dirty(folio); if (tlb_delay_rmap(tlb)) { delay_rmap = 1; force_flush = 1; } } if (pte_young(ptent) && likely(vma_has_recency(vma))) - mark_page_accessed(page); + folio_mark_accessed(folio); } rss[mm_counter(page)]--; if (!delay_rmap) { - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(folio, page, vma); if (unlikely(page_mapcount(page) < 0)) print_bad_pte(vma, addr, ptent, page); } @@ -1489,6 +1491,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (is_device_private_entry(entry) || is_device_exclusive_entry(entry)) { page = pfn_swap_entry_to_page(entry); + folio = page_folio(page); if (unlikely(!should_zap_page(details, page))) continue; /* @@ -1500,8 +1503,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, WARN_ON_ONCE(!vma_is_anonymous(vma)); rss[mm_counter(page)]--; if (is_device_private_entry(entry)) - page_remove_rmap(page, vma, false); - put_page(page); + folio_remove_rmap_pte(folio, page, vma); + folio_put(folio); } else if (!non_swap_entry(entry)) { /* Genuine swap entry, hence a private anon page */ if (!should_zap_cows(details)) @@ -3220,10 +3223,10 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * threads. * * The critical issue is to order this - * page_remove_rmap with the ptp_clear_flush above. - * Those stores are ordered by (if nothing else,) + * folio_remove_rmap_pte() with the ptp_clear_flush + * above. Those stores are ordered by (if nothing else,) * the barrier present in the atomic_add_negative - * in page_remove_rmap. + * in folio_remove_rmap_pte(); * * Then the TLB flush in ptep_clear_flush ensures that * no process can access the old page before the @@ -3232,7 +3235,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * mapcount is visible. So transitively, TLBs to * old page will be flushed before it can be reused. */ - page_remove_rmap(vmf->page, vma, false); + folio_remove_rmap_pte(old_folio, vmf->page, vma); } /* Free the old page.. */ diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 4f559f4ddd21..604ddf08affe 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -55,7 +55,7 @@ static void tlb_flush_rmap_batch(struct mmu_gather_batch *batch, struct vm_area_ if (encoded_page_flags(enc)) { struct page *page = encoded_page_ptr(enc); - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(page_folio(page), page, vma); } } } From patchwork Mon Dec 11 15:56:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176773 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7153279vqy; Mon, 11 Dec 2023 07:59:26 -0800 (PST) X-Google-Smtp-Source: AGHT+IE5iMTmlcQGyKr97UQQ7531PRS5sNfPuHyO9L4SiBtSuMI5reILKfHRF0Otm4rpBwI3oX4C X-Received: by 2002:a17:902:8302:b0:1d1:e319:80c1 with SMTP id bd2-20020a170902830200b001d1e31980c1mr2137941plb.101.1702310365719; Mon, 11 Dec 2023 07:59:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310365; cv=none; d=google.com; s=arc-20160816; b=l4fMFOaLRH4z0DDw2ntAKuyZoi9PGidEqID0CiANt8iYlga3Ka7jCDLtcB92kZxwac iI40Sd5ToP5Xq/cTVlZPztyI8T7p6wRgunQA5/GIePkUMAVxnn7HCdVp5sls0WWl8ph1 WLNavYBJ3n6CrJG6wAf3Ovqi9efuIQFBPJfdNIII9xwGRbKvgYG6bSp2perykyuxy8/o mr+EBd0d5H4+TeWCLRz/VHVXCBLaJtQso8D8mfHla5BjNKGegdFIPzUyaaXaJVMeptih xR3johsH0vh/xA3sy85GXsAhdLFN5EvcS1DBQVY4iq201mFZcBsGkuUJoCryFsmNbJyG 9qnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=zZtDmTKZIWvKon5vN+gkyoFpv7PryiRHfxxjTiLIEgc=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=LQO+XIK00HqouALjBwQuYIQiajd+X6MFuVO1Pi8Qq4NEFRtmai7/unb78Mn4q74Sg5 NWJrjU2F/Pxpu3axpU0wlTHEpPQGy/LzA0RmWoAxJbGWyxBKuzyjeTd0rLL61nYRHvYG uW81ei9/K3VoReNvq7uAxuSar21Rcv/+GFvrCiK6XQf7gi5JlmXTjKdkSyh8BzbgnCwD V2+cT6VC3GCmS94oFKGz2dyJiGeE8xxi5af3cD6VQxjPRtvJoQwd8vObfec5ckIrcleA 96d+bfd4IfBRW1ic4M8uVsDRArW/IpxgCjM0g1NILuPiSK0adSzK/O2MPo8BEjZNoBp7 CUJQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ShBu6PgB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id q14-20020a170902eb8e00b001cfb3d25f2asi6177375plg.652.2023.12.11.07.59.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:59:25 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ShBu6PgB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 663AF804C203; Mon, 11 Dec 2023 07:59:16 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344506AbjLKP7F (ORCPT + 99 others); Mon, 11 Dec 2023 10:59:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344510AbjLKP6X (ORCPT ); Mon, 11 Dec 2023 10:58:23 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3DA2210C8 for ; Mon, 11 Dec 2023 07:58:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310295; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zZtDmTKZIWvKon5vN+gkyoFpv7PryiRHfxxjTiLIEgc=; b=ShBu6PgB5Fb2lh5ksyCJHeuf+Fr4tGiNs8ptYF7MiLDz8S9jFwH5578RfC4sgSem6EYchY SkeV4kwQ7UkPQg+gT1PJjKCpMxOpTHRDOYTCnTGqs6v7Ef/0l5MWiYm9T2UxXpXTLLeWBT V2JM4umVBc+JUyqzTGvURuxFDaiQRWQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-617-k9oBbGedOyagcPkWkippwQ-1; Mon, 11 Dec 2023 10:58:11 -0500 X-MC-Unique: k9oBbGedOyagcPkWkippwQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7A20A870835; Mon, 11 Dec 2023 15:58:10 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1F6FB1121306; Mon, 11 Dec 2023 15:58:07 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 29/39] mm/migrate_device: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 11 Dec 2023 16:56:42 +0100 Message-ID: <20231211155652.131054-30-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:59:16 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001794097219867 X-GMAIL-MSGID: 1785001794097219867 Let's convert migrate_vma_collect_pmd(). While at it, perform more folio conversion. Signed-off-by: David Hildenbrand --- mm/migrate_device.c | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 8ac1f79f754a..c51c99151ebb 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -107,6 +107,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, for (; addr < end; addr += PAGE_SIZE, ptep++) { unsigned long mpfn = 0, pfn; + struct folio *folio; struct page *page; swp_entry_t entry; pte_t pte; @@ -168,41 +169,43 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, } /* - * By getting a reference on the page we pin it and that blocks + * By getting a reference on the folio we pin it and that blocks * any kind of migration. Side effect is that it "freezes" the * pte. * - * We drop this reference after isolating the page from the lru - * for non device page (device page are not on the lru and thus + * We drop this reference after isolating the folio from the lru + * for non device folio (device folio are not on the lru and thus * can't be dropped from it). */ - get_page(page); + folio = page_folio(page); + folio_get(folio); /* - * We rely on trylock_page() to avoid deadlock between + * We rely on folio_trylock() to avoid deadlock between * concurrent migrations where each is waiting on the others - * page lock. If we can't immediately lock the page we fail this + * folio lock. If we can't immediately lock the folio we fail this * migration as it is only best effort anyway. * - * If we can lock the page it's safe to set up a migration entry - * now. In the common case where the page is mapped once in a + * If we can lock the folio it's safe to set up a migration entry + * now. In the common case where the folio is mapped once in a * single process setting up the migration entry now is an * optimisation to avoid walking the rmap later with * try_to_migrate(). */ - if (trylock_page(page)) { + if (folio_trylock(folio)) { bool anon_exclusive; pte_t swp_pte; flush_cache_page(vma, addr, pte_pfn(pte)); - anon_exclusive = PageAnon(page) && PageAnonExclusive(page); + anon_exclusive = folio_test_anon(folio) && + PageAnonExclusive(page); if (anon_exclusive) { pte = ptep_clear_flush(vma, addr, ptep); if (page_try_share_anon_rmap(page)) { set_pte_at(mm, addr, ptep, pte); - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); mpfn = 0; goto next; } @@ -214,7 +217,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, /* Set the dirty flag on the folio now the pte is gone. */ if (pte_dirty(pte)) - folio_mark_dirty(page_folio(page)); + folio_mark_dirty(folio); /* Setup special migration page table entry */ if (mpfn & MIGRATE_PFN_WRITE) @@ -248,16 +251,16 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, /* * This is like regular unmap: we remove the rmap and - * drop page refcount. Page won't be freed, as we took - * a reference just above. + * drop the folio refcount. The folio won't be freed, as + * we took a reference just above. */ - page_remove_rmap(page, vma, false); - put_page(page); + folio_remove_rmap_pte(folio, page, vma); + folio_put(folio); if (pte_present(pte)) unmapped++; } else { - put_page(page); + folio_put(folio); mpfn = 0; } From patchwork Mon Dec 11 15:56:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176777 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7153463vqy; Mon, 11 Dec 2023 07:59:38 -0800 (PST) X-Google-Smtp-Source: AGHT+IFUpyhAOBORQMXTpNFhC/9UrhJWkWtbqBhoGPZOWLueiBa79CipP14NBHs20Dcep4qqSO57 X-Received: by 2002:a05:6a20:a122:b0:18c:f9f9:4198 with SMTP id q34-20020a056a20a12200b0018cf9f94198mr7045858pzk.14.1702310377723; Mon, 11 Dec 2023 07:59:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310377; cv=none; d=google.com; s=arc-20160816; b=pwbZuEExLonAqls7jKP2q41MlGriLBKLhtpdeqU32PjdmMZZwDmQIDS8a4k3wlb62c hLgZeOiLLwo3RQ77fkYqF6t9YPgekeU7hcPI2h2md+7lOiKvJlC3d27HQSdqJKI7LQ6v l9G/NN0gYT1iY6G4mtiYpx9L4hrqCLWmXYL8VyQkdvERHjMCijF4mfCn7kVwrbZ4tT0b hOlVC2JgAB05Hh74oL4b6hVZvfq5yzeVt0fSmUk9M+ORmIJshRgAiJ8C1e88ezCKGra6 blFMFfoPtGJnBUZiA9Igb7nS51SNcGSl3szV7lsNLq1n9+fXiH6+x5K8bg8tfLBPyKGo aDhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eWMSqygXIPXuXZ+kMnrkRT/FVjt8oLCVIg9Vpm7UnEo=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=IP8m89yux9+JCBks/nUTwmJLwKSEJH05jMdlifE9qCKEampWL/w/IT81u4j0wucPRT 2/fXmxdJuCC1M6sghX//PiFJ7icuty3yX3AzkMQkg3nUdISejJQK3ODLw4t91Vu8DYLY VUd5QwRTIt5ZBVw17v58Ck8c9b+gPMhBKszGTQnPUEtTaykTAkjQuls1Rq09ihz/uzYZ TjtMTIUsLQDCFy+XUSMXT6ZjfeJLlhY2RvVmcXV2J01DQURQCoUnqS89z7TuwoHZt28h uKMmaI5BhktiJO87eShYvCIhbEW5Ud27jPK8iTVPqN5yz1S8DnFh8/IQ6hTNFRbD2cd8 bTSw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=EIb3kCxR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id x65-20020a636344000000b005b928e39429si6280952pgb.259.2023.12.11.07.59.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:59:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=EIb3kCxR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id D4E58809E8EE; Mon, 11 Dec 2023 07:59:34 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344686AbjLKP7K (ORCPT + 99 others); Mon, 11 Dec 2023 10:59:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344619AbjLKP6Z (ORCPT ); Mon, 11 Dec 2023 10:58:25 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 76624FF for ; Mon, 11 Dec 2023 07:58:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310299; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eWMSqygXIPXuXZ+kMnrkRT/FVjt8oLCVIg9Vpm7UnEo=; b=EIb3kCxROiZeOxHgPGObqYGgsVzY32aO8qGc03DOqyBzqtMgLKuvA/Qs5z1DmtqhkAV49C 0FbcMC8F4W+KsFuRhDQd3so/C0KwIq35JcGQZvmBLgK+WiBrBniwvAeYTUs+d6Peh5q4ky BEJ2xzNAuPISES6UwIqXsdDQJKbddq8= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-179-uhoPHFudP96VNqAzzD5Khw-1; Mon, 11 Dec 2023 10:58:13 -0500 X-MC-Unique: uhoPHFudP96VNqAzzD5Khw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 29BE038425AD; Mon, 11 Dec 2023 15:58:13 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id B38C31121306; Mon, 11 Dec 2023 15:58:10 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 30/39] mm/rmap: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 11 Dec 2023 16:56:43 +0100 Message-ID: <20231211155652.131054-31-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:59:35 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001806738258604 X-GMAIL-MSGID: 1785001806738258604 Let's convert try_to_unmap_one() and try_to_migrate_one(). Signed-off-by: David Hildenbrand --- mm/rmap.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index dc3be5807cee..233432f08e36 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1649,7 +1649,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, /* * When racing against e.g. zap_pte_range() on another cpu, - * in between its ptep_get_and_clear_full() and page_remove_rmap(), + * in between its ptep_get_and_clear_full() and folio_remove_rmap_*(), * try_to_unmap() may return before page_mapped() has become false, * if page table locking is skipped: use TTU_SYNC to wait for that. */ @@ -1930,7 +1930,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, if (unlikely(folio_test_hugetlb(folio))) hugetlb_remove_rmap(folio); else - page_remove_rmap(subpage, vma, false); + folio_remove_rmap_pte(folio, subpage, vma); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); @@ -1998,7 +1998,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, /* * When racing against e.g. zap_pte_range() on another cpu, - * in between its ptep_get_and_clear_full() and page_remove_rmap(), + * in between its ptep_get_and_clear_full() and folio_remove_rmap_*(), * try_to_migrate() may return before page_mapped() has become false, * if page table locking is skipped: use TTU_SYNC to wait for that. */ @@ -2291,7 +2291,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, if (unlikely(folio_test_hugetlb(folio))) hugetlb_remove_rmap(folio); else - page_remove_rmap(subpage, vma, false); + folio_remove_rmap_pte(folio, subpage, vma); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); @@ -2430,7 +2430,7 @@ static bool page_make_device_exclusive_one(struct folio *folio, * There is a reference on the page for the swap entry which has * been removed, so shouldn't take another. */ - page_remove_rmap(subpage, vma, false); + folio_remove_rmap_pte(folio, subpage, vma); } mmu_notifier_invalidate_range_end(&range); From patchwork Mon Dec 11 15:56:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176776 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7153432vqy; Mon, 11 Dec 2023 07:59:36 -0800 (PST) X-Google-Smtp-Source: AGHT+IEba7AeeygPdOhp3dOzN3fH8QCxH73JbwP3+0LhjGy8XVrNTsjhdVB8ruKLS0JnS7eMDXT5 X-Received: by 2002:a05:6a00:464b:b0:68f:a92a:8509 with SMTP id kp11-20020a056a00464b00b0068fa92a8509mr7209648pfb.7.1702310375703; Mon, 11 Dec 2023 07:59:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310375; cv=none; d=google.com; s=arc-20160816; b=ZpfEpMDUupX4JR3xn6t/EFzKk4vfFpbJ8zh449xg9gfMNjxg7FwRo1+BQytmOugYmP 85Is1eTQNV8jlF6BlLxq7GLwLbIViWl4UcVR3V4UVtivlj1k3tpRTy71ZxBxvwY/zIMh a40YbztNE8c9Vodc8WXnSrrkxq6FtPRGqyJOF+A+qxcIo0NfU9CbSLIT0mMTyLwiadtn ncgqbyjrYCq3FTq9Jy+UMPhcVoW6Zm/Bzg7sZI1OeWvttVRsn3gkQwEKnfnf2zdTADRO ciSJUDdcJVPumBW7pvH2dn1vsfzaJBthXiZlGRh6HC4xNzmz5tAXB5zQY0HmJUthttW9 2bKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=j7yPJXNweA3+MouJWc5Ro2+qKjmp/wEHVCvW0WWVjak=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=QjvOvh96I881eWjTH3IMSkVcNd7zx/fV0oXI3lEYIL9BlY7dWJ4h8IXh3hzD1WWT7q oGOcx1s5Slec/vZtBo2Xu0Nfqbwy50fc5xoLhoWtjoKPiVmhoAM7o1hmS1elqIbkvuEI lg9DFrPLr1dZJ9C1wMUe0dnQ6cZBmrsvyaP3i/Ki0d9kMRG4fReTnxKcTVNk2B7piTWF xnXKY3Os1rBUJYGplnw1/w+G9YS9vDBBTcMcyCdorQcWVmvHnYM1IbXfWa3vggXR1BqF 2CRBebvwlyPhwRPmz6oGX3Nn8x91sIzRnCY8dBlRxLXAQAStpS/iTAAsoGM8aDVWJ4Rr PwPg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=FE4aOR+B; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id z11-20020aa7888b000000b006ceb1a0fa60si6273359pfe.8.2023.12.11.07.59.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:59:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=FE4aOR+B; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 846FC804C201; Mon, 11 Dec 2023 07:59:26 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344619AbjLKP7P (ORCPT + 99 others); Mon, 11 Dec 2023 10:59:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42730 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344416AbjLKP6Z (ORCPT ); Mon, 11 Dec 2023 10:58:25 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8203010B for ; Mon, 11 Dec 2023 07:58:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310299; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j7yPJXNweA3+MouJWc5Ro2+qKjmp/wEHVCvW0WWVjak=; b=FE4aOR+B4pO1s+kwFrO36r+VBUDk1mmgPOd541AKtnHtBH6i8as87zUXpf+L422UvFd2s9 Ox58viPFEFDZPCNyd7wC186t2CERrQrtmy/iOQw0a9tlRqirAdXZ6nww/VoxJr/r9XlASb uctosC00XqZrtX7FR6taIMJsjbnui/0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-312-4EKUpxx8O7a6a8BNS2xAZA-1; Mon, 11 Dec 2023 10:58:17 -0500 X-MC-Unique: 4EKUpxx8O7a6a8BNS2xAZA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 07746848A6F; Mon, 11 Dec 2023 15:58:16 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 87CDF1121306; Mon, 11 Dec 2023 15:58:13 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 31/39] Documentation: stop referring to page_remove_rmap() Date: Mon, 11 Dec 2023 16:56:44 +0100 Message-ID: <20231211155652.131054-32-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:59:27 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001804476231321 X-GMAIL-MSGID: 1785001804476231321 Refer to folio_remove_rmap_*() instaed. Signed-off-by: David Hildenbrand --- Documentation/mm/transhuge.rst | 2 +- Documentation/mm/unevictable-lru.rst | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Documentation/mm/transhuge.rst b/Documentation/mm/transhuge.rst index 9a607059ea11..cf81272a6b8b 100644 --- a/Documentation/mm/transhuge.rst +++ b/Documentation/mm/transhuge.rst @@ -156,7 +156,7 @@ Partial unmap and deferred_split_folio() Unmapping part of THP (with munmap() or other way) is not going to free memory immediately. Instead, we detect that a subpage of THP is not in use -in page_remove_rmap() and queue the THP for splitting if memory pressure +in folio_remove_rmap_*() and queue the THP for splitting if memory pressure comes. Splitting will free up unused subpages. Splitting the page right away is not an option due to locking context in diff --git a/Documentation/mm/unevictable-lru.rst b/Documentation/mm/unevictable-lru.rst index 67f1338440a5..b6a07a26b10d 100644 --- a/Documentation/mm/unevictable-lru.rst +++ b/Documentation/mm/unevictable-lru.rst @@ -486,7 +486,7 @@ munlock the pages if we're removing the last VM_LOCKED VMA that maps the pages. Before the unevictable/mlock changes, mlocking did not mark the pages in any way, so unmapping them required no processing. -For each PTE (or PMD) being unmapped from a VMA, page_remove_rmap() calls +For each PTE (or PMD) being unmapped from a VMA, folio_remove_rmap_*() calls munlock_vma_folio(), which calls munlock_folio() when the VMA is VM_LOCKED (unless it was a PTE mapping of a part of a transparent huge page). @@ -511,7 +511,7 @@ userspace; truncation even unmaps and deletes any private anonymous pages which had been Copied-On-Write from the file pages now being truncated. Mlocked pages can be munlocked and deleted in this way: like with munmap(), -for each PTE (or PMD) being unmapped from a VMA, page_remove_rmap() calls +for each PTE (or PMD) being unmapped from a VMA, folio_remove_rmap_*() calls munlock_vma_folio(), which calls munlock_folio() when the VMA is VM_LOCKED (unless it was a PTE mapping of a part of a transparent huge page). From patchwork Mon Dec 11 15:56:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176781 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7153560vqy; Mon, 11 Dec 2023 07:59:52 -0800 (PST) X-Google-Smtp-Source: AGHT+IE3t9CUXMajeA4h7c0elxeXySmvoiJR/CPNV5wwWlkccTUhXdd/PMHtrd+qi/Vvn041oa2k X-Received: by 2002:a05:6a20:9492:b0:190:2ae6:c690 with SMTP id hs18-20020a056a20949200b001902ae6c690mr4834144pzb.66.1702310388060; Mon, 11 Dec 2023 07:59:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310388; cv=none; d=google.com; s=arc-20160816; b=M3TcxRNgII/8ySNX38uUJQH2e2B92qNd9v860fLGMvbfLRgaF5N/0MGE8TGOtlZLjJ 2DH/ciyzKTviONZAcbITstfDnGDETB8buRqFwQtMmmeCDVlxAWOrfZaT5lQuKnY7bhkd loWGkHnOTkzfnmeEDkzEBq653hl0rpM2QKq8bBUysNHVE1OusUa63GITMj8xM/NuguoN sdsHbsmbaZX2hnbi72nxgWyM5y7OZKp6+x9jMQwhN2C5h/qwlafPFhpkSQ2xU8l9ltu8 K6u6I9BoPoy5sBYLxofQigBpbivc8+gPzxfhmDxbyE0W3/PqQH40YImBTF6FXUY2h+QZ NyvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=yS0UIUzy1DjqGivo1HlVZbIDQrX5/8+loO66ukrqS7A=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=zKo+0hTK3tZ9rbzA6zGGnn9EO/+A5nL/cYVXT4I/YlOwqBS5sPsxtu0yd8OAIAjBCG MWYCMAoirzLc5MRlUqGrxcVyyUqMWelTeyHiaYOcoU5/MDdUTo1XPuZsqu56cC25SvPM MMVWmZEw26/TK7aSwprkGTc216lJHB1+cSnjiCDcrvwo/7Wv9CeR+BGyPenMt0SDlVHt GBLJpQa/VDEJMIQOuCPPMtW5D53WbUvtGdw/HWko74kk0xm5DE+IHFCAaLh5IC23yFr8 SLICaA6RunaLJ/bVMYqPBkbhOO8lBGusQdiclycyvNtt3TD8CqFb9dPS9gLmqavMDp3s lwPA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=fc43Z2yA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id f13-20020a056a001acd00b006cdce1d5ac7si6403440pfv.17.2023.12.11.07.59.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:59:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=fc43Z2yA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 532AE803C44C; Mon, 11 Dec 2023 07:59:36 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344698AbjLKP7T (ORCPT + 99 others); Mon, 11 Dec 2023 10:59:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46658 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344515AbjLKP6Z (ORCPT ); Mon, 11 Dec 2023 10:58:25 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1ED26118 for ; Mon, 11 Dec 2023 07:58:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310300; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yS0UIUzy1DjqGivo1HlVZbIDQrX5/8+loO66ukrqS7A=; b=fc43Z2yApnuURWYJD0nbLA/R4X3gN/hjCs6xzR1r+oDX2Fkv+bP8S5d4X7c/Gy1Ai8upxj P1uRK+F6oVcy7I6ts6rhD3PWOqCcL8HiZiXCysVbuszJZhwGmn1QYuKKLRY3dBc16OJER0 Lw6K7tIgM1mk/dLvVZoEz+POeEFoFE4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-346-rMmu2rhaOL6tNMECz-xHBQ-1; Mon, 11 Dec 2023 10:58:18 -0500 X-MC-Unique: rMmu2rhaOL6tNMECz-xHBQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 241BA848A83; Mon, 11 Dec 2023 15:58:18 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6AA331121306; Mon, 11 Dec 2023 15:58:16 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 32/39] mm/rmap: remove page_remove_rmap() Date: Mon, 11 Dec 2023 16:56:45 +0100 Message-ID: <20231211155652.131054-33-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:59:36 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001817576924921 X-GMAIL-MSGID: 1785001817576924921 All callers are gone, let's remove it and some leftover traces. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 4 +--- mm/filemap.c | 10 +++++----- mm/internal.h | 2 +- mm/memory-failure.c | 4 ++-- mm/rmap.c | 23 ++--------------------- 5 files changed, 11 insertions(+), 32 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index a266dc0ef99e..0f4eecd03bdc 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -244,8 +244,6 @@ void folio_add_file_rmap_ptes(struct folio *, struct page *, int nr_pages, folio_add_file_rmap_ptes(folio, page, 1, vma) void folio_add_file_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *); -void page_remove_rmap(struct page *, struct vm_area_struct *, - bool compound); void folio_remove_rmap_ptes(struct folio *, struct page *, int nr_pages, struct vm_area_struct *); #define folio_remove_rmap_pte(folio, page, vma) \ @@ -392,7 +390,7 @@ static inline int page_try_dup_anon_rmap(struct page *page, bool compound, * * This is similar to page_try_dup_anon_rmap(), however, not used during fork() * to duplicate a mapping, but instead to prepare for KSM or temporarily - * unmapping a page (swap, migration) via page_remove_rmap(). + * unmapping a page (swap, migration) via folio_remove_rmap_*(). * * Marking the page shared can only fail if the page may be pinned; device * private pages cannot get pinned and consequently this function cannot fail. diff --git a/mm/filemap.c b/mm/filemap.c index c0d7e1d7eea2..beff3865465a 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -113,11 +113,11 @@ * ->i_pages lock (try_to_unmap_one) * ->lruvec->lru_lock (follow_page->mark_page_accessed) * ->lruvec->lru_lock (check_pte_range->isolate_lru_page) - * ->private_lock (page_remove_rmap->set_page_dirty) - * ->i_pages lock (page_remove_rmap->set_page_dirty) - * bdi.wb->list_lock (page_remove_rmap->set_page_dirty) - * ->inode->i_lock (page_remove_rmap->set_page_dirty) - * ->memcg->move_lock (page_remove_rmap->folio_memcg_lock) + * ->private_lock (folio_remove_rmap_pte->set_page_dirty) + * ->i_pages lock (folio_remove_rmap_pte->set_page_dirty) + * bdi.wb->list_lock (folio_remove_rmap_pte->set_page_dirty) + * ->inode->i_lock (folio_remove_rmap_pte->set_page_dirty) + * ->memcg->move_lock (folio_remove_rmap_pte->folio_memcg_lock) * bdi.wb->list_lock (zap_pte_range->set_page_dirty) * ->inode->i_lock (zap_pte_range->set_page_dirty) * ->private_lock (zap_pte_range->block_dirty_folio) diff --git a/mm/internal.h b/mm/internal.h index 222e63b2dea4..a94355e70bd7 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -651,7 +651,7 @@ folio_within_vma(struct folio *folio, struct vm_area_struct *vma) * under page table lock for the pte/pmd being added or removed. * * mlock is usually called at the end of page_add_*_rmap(), munlock at - * the end of page_remove_rmap(); but new anon folios are managed by + * the end of folio_remove_rmap_*(); but new anon folios are managed by * folio_add_lru_vma() calling mlock_new_folio(). */ void mlock_folio(struct folio *folio); diff --git a/mm/memory-failure.c b/mm/memory-failure.c index d8c853b35dbb..01af9295c47c 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -2316,8 +2316,8 @@ int memory_failure(unsigned long pfn, int flags) * We use page flags to determine what action should be taken, but * the flags can be modified by the error containment action. One * example is an mlocked page, where PG_mlocked is cleared by - * page_remove_rmap() in try_to_unmap_one(). So to determine page status - * correctly, we save a copy of the page flags at this time. + * folio_remove_rmap_*() in try_to_unmap_one(). So to determine page + * status correctly, we save a copy of the page flags at this time. */ page_flags = p->flags; diff --git a/mm/rmap.c b/mm/rmap.c index 233432f08e36..b08dd7d6779d 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -470,7 +470,7 @@ void __init anon_vma_init(void) /* * Getting a lock on a stable anon_vma from a page off the LRU is tricky! * - * Since there is no serialization what so ever against page_remove_rmap() + * Since there is no serialization what so ever against folio_remove_rmap_*() * the best this function can do is return a refcount increased anon_vma * that might have been relevant to this page. * @@ -487,7 +487,7 @@ void __init anon_vma_init(void) * [ something equivalent to page_mapped_in_vma() ]. * * Since anon_vma's slab is SLAB_TYPESAFE_BY_RCU and we know from - * page_remove_rmap() that the anon_vma pointer from page->mapping is valid + * folio_remove_rmap_*() that the anon_vma pointer from page->mapping is valid * if there is a mapcount, we can dereference the anon_vma after observing * those. * @@ -1499,25 +1499,6 @@ void folio_add_file_rmap_pmd(struct folio *folio, struct page *page, #endif } -/** - * page_remove_rmap - take down pte mapping from a page - * @page: page to remove mapping from - * @vma: the vm area from which the mapping is removed - * @compound: uncharge the page as compound or small page - * - * The caller needs to hold the pte lock. - */ -void page_remove_rmap(struct page *page, struct vm_area_struct *vma, - bool compound) -{ - struct folio *folio = page_folio(page); - - if (likely(!compound)) - folio_remove_rmap_pte(folio, page, vma); - else - folio_remove_rmap_pmd(folio, page, vma); -} - static __always_inline void __folio_remove_rmap(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *vma, enum rmap_mode mode) From patchwork Mon Dec 11 15:56:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176787 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7155527vqy; Mon, 11 Dec 2023 08:01:41 -0800 (PST) X-Google-Smtp-Source: AGHT+IFhg6srNzP7kNeu9p2Fz2hRyIXrrSLruwfwH7apk6JzeIexgcB8/xJsSRlt51ge+rqb3iFt X-Received: by 2002:a17:90a:f692:b0:288:62f6:27f4 with SMTP id cl18-20020a17090af69200b0028862f627f4mr3672639pjb.8.1702310501043; Mon, 11 Dec 2023 08:01:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310501; cv=none; d=google.com; s=arc-20160816; b=ygijBit29DHM2anmu5KRM7xQgUnqiMICai21TiqrG2ts3yFfbgrqw2BqUtSgDqfBVt Ip39ynVDpRctx05f+qLIzxi9RNXCP5ThNGJjMq0c8X5QAo/qHXTPoUHDfQ+ET+b30aZ+ l8zYpQjZHTznSc6ZO0Y9OH1yOjVXta9eYct36mKUQPqfQe/nrAHRJYxJPf7xoFhSPIBw pAqWQQGruOhNLwahC1uFBH5AFWkcmfFdmGaO4LPG6HzW7iAK5b48iqvldC+4EJJOwfLh M013xR40hwNqhxMC7oPfslx8FoyOJ8F8RE0CNqhS+HNA/+aXQOUrv+/diDmuRcKr8MaX HYLg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=nX1pXuuGCFSgne2iWx/veTQmI+PIa0/pNoky6wKmMd0=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=ROoLTPGeut7pAyf29N6NOiDMrkfWm7q3FjpFl+e2j87wEsk3Tp6kWS6z9FJrNlIdUM 70kCQeOt8lq22W7zvEkk3TEtNU2bepVCCsct8GAhL0bpxfMdpiFMi7RIKdtY13B8r2D5 6Kc7sPX+NCbVGXWnRpjyu8AfaOzLI3v6k8d69AS2vU7EHfbZ8azq13SxFecLgtaR+onW Ahe6d6OjzNENhBqFTxMWBQi4B79lXEVF9R27DPgnZkWhnK79HmTnm283QC5vlb4wB1Xb qF8YaUXv+xx/o86Voxv1BAObmvmvyK1Fq0m0C2CrSnf9h2B8TFLRzjDA/B2OTibQRyd3 EkQg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=J7tFFTC7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id c22-20020a63da16000000b005c1b2d6aa3esi6123872pgh.355.2023.12.11.08.01.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 08:01:41 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=J7tFFTC7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 3F2E9807F946; Mon, 11 Dec 2023 08:00:16 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344729AbjLKP72 (ORCPT + 99 others); Mon, 11 Dec 2023 10:59:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344495AbjLKP60 (ORCPT ); Mon, 11 Dec 2023 10:58:26 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 619DCB3 for ; Mon, 11 Dec 2023 07:58:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310304; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nX1pXuuGCFSgne2iWx/veTQmI+PIa0/pNoky6wKmMd0=; b=J7tFFTC7KTwNnDMqe/WpYzR1ivyyVvJR13Fa0tKJr4gNiaJbwRd1NOLjg5G/kLe0QSkRoO PiAjn/NTdhCtgKerpkKFnI6kYb2Po3yY+8PLR9tBuiHe/UetzHjTL3ej6NedCz5s8LPqdk j/dRHr92mSAVyRDQT/HVMy+sQ55Iu18= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-515-6fAsG4HZNcWrnusNEcf0NQ-1; Mon, 11 Dec 2023 10:58:21 -0500 X-MC-Unique: 6fAsG4HZNcWrnusNEcf0NQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5105985A589; Mon, 11 Dec 2023 15:58:20 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5C4221121306; Mon, 11 Dec 2023 15:58:18 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 33/39] mm/rmap: convert page_dup_file_rmap() to folio_dup_file_rmap_[pte|ptes|pmd]() Date: Mon, 11 Dec 2023 16:56:46 +0100 Message-ID: <20231211155652.131054-34-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 11 Dec 2023 08:00:18 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001935869018760 X-GMAIL-MSGID: 1785001935869018760 Let's convert page_dup_file_rmap() like the other rmap functions. As there is only a single caller, convert that single caller right away and remove page_dup_file_rmap(). Add folio_dup_file_rmap_ptes() right away, we want to perform rmap baching during fork() soon. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 59 ++++++++++++++++++++++++++++++++++++++++---- mm/memory.c | 2 +- 2 files changed, 55 insertions(+), 6 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 0f4eecd03bdc..df60e44fecad 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -311,6 +311,60 @@ static inline void hugetlb_remove_rmap(struct folio *folio) atomic_dec(&folio->_entire_mapcount); } +static __always_inline void __folio_dup_file_rmap(struct folio *folio, + struct page *page, int nr_pages, enum rmap_mode mode) +{ + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); + + switch (mode) { + case RMAP_MODE_PTE: + do { + atomic_inc(&page->_mapcount); + } while (page++, --nr_pages > 0); + break; + case RMAP_MODE_PMD: + atomic_inc(&folio->_entire_mapcount); + break; + } +} + +/** + * folio_dup_file_rmap_ptes - duplicate PTE mappings of a page range of a folio + * @folio: The folio to duplicate the mappings of + * @page: The first page to duplicate the mappings of + * @nr_pages: The number of pages of which the mapping will be duplicated + * + * The page range of the folio is defined by [page, page + nr_pages) + * + * The caller needs to hold the page table lock. + */ +static inline void folio_dup_file_rmap_ptes(struct folio *folio, + struct page *page, int nr_pages) +{ + __folio_dup_file_rmap(folio, page, nr_pages, RMAP_MODE_PTE); +} +#define folio_dup_file_rmap_pte(folio, page) \ + folio_dup_file_rmap_ptes(folio, page, 1) + +/** + * folio_dup_file_rmap_pmd - duplicate a PMD mapping of a page range of a folio + * @folio: The folio to duplicate the mapping of + * @page: The first page to duplicate the mapping of + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock. + */ +static inline void folio_dup_file_rmap_pmd(struct folio *folio, + struct page *page) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_dup_file_rmap(folio, page, HPAGE_PMD_NR, RMAP_MODE_PTE); +#else + WARN_ON_ONCE(true); +#endif +} + static inline void __page_dup_rmap(struct page *page, bool compound) { VM_WARN_ON(folio_test_hugetlb(page_folio(page))); @@ -325,11 +379,6 @@ static inline void __page_dup_rmap(struct page *page, bool compound) } } -static inline void page_dup_file_rmap(struct page *page, bool compound) -{ - __page_dup_rmap(page, compound); -} - /** * page_try_dup_anon_rmap - try duplicating a mapping of an already mapped * anonymous page diff --git a/mm/memory.c b/mm/memory.c index 9a5724cf895f..42a0b7b41b86 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -965,7 +965,7 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, rss[MM_ANONPAGES]++; } else if (page) { folio_get(folio); - page_dup_file_rmap(page, false); + folio_dup_file_rmap_pte(folio, page); rss[mm_counter_file(page)]++; } From patchwork Mon Dec 11 15:56:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176779 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7153651vqy; Mon, 11 Dec 2023 07:59:58 -0800 (PST) X-Google-Smtp-Source: AGHT+IG190L91n+tSDCzIze72R2j0XUFft5ZrnqYAZXvKjhl2qhcbWapXYD+gVZEmKccydoL+mop X-Received: by 2002:a05:6a20:394a:b0:190:628e:bfe7 with SMTP id r10-20020a056a20394a00b00190628ebfe7mr6288836pzg.95.1702310392628; Mon, 11 Dec 2023 07:59:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310392; cv=none; d=google.com; s=arc-20160816; b=p4ueXyVPzWNNajcOt0IumDsAchM9b4glDOEOQGAgxyZ1In/0Cinc1i7101luF/wLvB G+Q25W3wUmAtOO2On1QgBEaD7KupJQaXt52VUjgZllUanLKWqWi9Aw6cTNAEX7nZDXou BxMjnbOpA6rhcPBaQ3Dm+wVuxo0dRE7PUx+22lqDICRTPhQ3zm6aag0miZgtu93nAnki Ycg0rOuJR0uOi0+YDEznhV85KxSFFUFQ0GIiruCC+M/j5BNeCp27wtnYcwM4YEKPsk8q 5VHL5buGbf2wsBr/SqTpuk6pgT95ME3uDXBGoNTlVMT0IUo6KNtYTI59RvMuX6Sjd2hD LDdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=hi7b3teT5yay/0ynlIbF8dSmjMX9HoAHjoRGBB4rTLU=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=dlYejXinQJt85SOl/JVSNYiOSC07bwqqfjdNh8XEAYc46tqY7/bY7o9I4jK+179HL5 /YRsuOmACBuFxsU7RaiZIPoOwE2EEulY8+bc3F+F9/Q3td8ylGEYpycR0XiXNkRKsoxc BGai1wPAQXCbDDv2DOLSi2sPO08gnfqWCMeEFb6/SvnnlbQy4dAj6BfesV5G4wV4pbrr 0wD24yN7FuU1GwHqyCPaGp+fEFPmsoZmdK+4GkA/mDdcKLufBfbN7y4rMBfgVI4g9GYT LlHlRFDV704enbpm+NJZj2regifySTiqCwjEUojiP3/jeHQ2biRJ/ZxRKv7XhU3EOWoB FHuw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=eczqDouv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id x123-20020a636381000000b005c6666dcce1si6329140pgb.583.2023.12.11.07.59.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:59:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=eczqDouv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id A3952803D837; Mon, 11 Dec 2023 07:59:37 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344633AbjLKP7Y (ORCPT + 99 others); Mon, 11 Dec 2023 10:59:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51082 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344582AbjLKP61 (ORCPT ); Mon, 11 Dec 2023 10:58:27 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE294136 for ; Mon, 11 Dec 2023 07:58:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310307; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hi7b3teT5yay/0ynlIbF8dSmjMX9HoAHjoRGBB4rTLU=; b=eczqDouvewc546nRfJRwY1yWGB/ezqT20jSSVGG6sYcQ2tQxnsANawyt3HfZotCFip2Eu2 QLRbLcbG8ghx0xUQU0QunXGPRhIEGOMZcgIEE1a7NdpPrZI8SMqmaGkqEqaKggUGlwSnxB c4ZsTenjfCQC71xdLPpe0n9PWo2E+pI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-645-vRbs79ACOCeF2HWItzmhcA-1; Mon, 11 Dec 2023 10:58:23 -0500 X-MC-Unique: vRbs79ACOCeF2HWItzmhcA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C070D86301D; Mon, 11 Dec 2023 15:58:22 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id B14EA1121306; Mon, 11 Dec 2023 15:58:20 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 34/39] mm/rmap: introduce folio_try_dup_anon_rmap_[pte|ptes|pmd]() Date: Mon, 11 Dec 2023 16:56:47 +0100 Message-ID: <20231211155652.131054-35-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:59:37 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001822392741956 X-GMAIL-MSGID: 1785001822392741956 The last user of page_needs_cow_for_dma() and __page_dup_rmap() are gone, remove them. Add folio_try_dup_anon_rmap_ptes() right away, we want to perform rmap baching during fork() soon. Signed-off-by: David Hildenbrand --- include/linux/mm.h | 6 -- include/linux/rmap.h | 150 ++++++++++++++++++++++++++++++------------- 2 files changed, 106 insertions(+), 50 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ae547b62f325..30edf3f7d1f3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1975,12 +1975,6 @@ static inline bool folio_needs_cow_for_dma(struct vm_area_struct *vma, return folio_maybe_dma_pinned(folio); } -static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma, - struct page *page) -{ - return folio_needs_cow_for_dma(vma, page_folio(page)); -} - /** * is_zero_page - Query if a page is a zero page * @page: The page to query diff --git a/include/linux/rmap.h b/include/linux/rmap.h index df60e44fecad..c6d8a02ecd56 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -365,68 +365,130 @@ static inline void folio_dup_file_rmap_pmd(struct folio *folio, #endif } -static inline void __page_dup_rmap(struct page *page, bool compound) +static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, + struct page *page, int nr_pages, struct vm_area_struct *src_vma, + enum rmap_mode mode) { - VM_WARN_ON(folio_test_hugetlb(page_folio(page))); + bool maybe_pinned; + int i; - if (compound) { - struct folio *folio = (struct folio *)page; + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); - VM_BUG_ON_PAGE(compound && !PageHead(page), page); + /* + * If this folio may have been pinned by the parent process, + * don't allow to duplicate the mappings but instead require to e.g., + * copy the subpage immediately for the child so that we'll always + * guarantee the pinned folio won't be randomly replaced in the + * future on write faults. + */ + maybe_pinned = likely(!folio_is_device_private(folio)) && + unlikely(folio_needs_cow_for_dma(src_vma, folio)); + + /* + * No need to check+clear for already shared PTEs/PMDs of the + * folio. But if any page is PageAnonExclusive, we must fallback to + * copying if the folio maybe pinned. + */ + switch (mode) { + case RMAP_MODE_PTE: + if (unlikely(maybe_pinned)) { + for (i = 0; i < nr_pages; i++) + if (PageAnonExclusive(page + i)) + return -EBUSY; + } + do { + if (PageAnonExclusive(page)) + ClearPageAnonExclusive(page); + atomic_inc(&page->_mapcount); + } while (page++, --nr_pages > 0); + break; + case RMAP_MODE_PMD: + if (PageAnonExclusive(page)) { + if (unlikely(maybe_pinned)) + return -EBUSY; + ClearPageAnonExclusive(page); + } atomic_inc(&folio->_entire_mapcount); - } else { - atomic_inc(&page->_mapcount); + break; } + return 0; } /** - * page_try_dup_anon_rmap - try duplicating a mapping of an already mapped - * anonymous page - * @page: the page to duplicate the mapping for - * @compound: the page is mapped as compound or as a small page - * @vma: the source vma + * folio_try_dup_anon_rmap_ptes - try duplicating PTE mappings of a page range + * of a folio + * @folio: The folio to duplicate the mappings of + * @page: The first page to duplicate the mappings of + * @nr_pages: The number of pages of which the mapping will be duplicated + * @src_vma: The vm area from which the mappings are duplicated + * + * The page range of the folio is defined by [page, page + nr_pages) * - * The caller needs to hold the PT lock and the vma->vma_mm->write_protect_seq. + * The caller needs to hold the page table lock and the + * vma->vma_mm->write_protect_seq. * - * Duplicating the mapping can only fail if the page may be pinned; device - * private pages cannot get pinned and consequently this function cannot fail. + * Duplicating the mappings can only fail if the folio may be pinned; device + * private folios cannot get pinned and consequently this function cannot fail + * for them. * - * If duplicating the mapping succeeds, the page has to be mapped R/O into - * the parent and the child. It must *not* get mapped writable after this call. + * If duplicating the mappings succeeded, the duplicated PTEs have to be R/O in + * the parent and the child. They must *not* be writable after this call + * succeeded. + * + * Returns 0 if duplicating the mappings succeeded. Returns -EBUSY otherwise. + */ +static inline int folio_try_dup_anon_rmap_ptes(struct folio *folio, + struct page *page, int nr_pages, struct vm_area_struct *src_vma) +{ + return __folio_try_dup_anon_rmap(folio, page, nr_pages, src_vma, + RMAP_MODE_PTE); +} +#define folio_try_dup_anon_rmap_pte(folio, page, vma) \ + folio_try_dup_anon_rmap_ptes(folio, page, 1, vma) + +/** + * folio_try_dup_anon_rmap_pmd - try duplicating a PMD mapping of a page range + * of a folio + * @folio: The folio to duplicate the mapping of + * @page: The first page to duplicate the mapping of + * @src_vma: The vm area from which the mapping is duplicated + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock and the + * vma->vma_mm->write_protect_seq. + * + * Duplicating the mapping can only fail if the folio may be pinned; device + * private folios cannot get pinned and consequently this function cannot fail + * for them. + * + * If duplicating the mapping succeeds, the duplicated PMD has to be R/O in + * the parent and the child. They must *not* be writable after this call + * succeeded. * * Returns 0 if duplicating the mapping succeeded. Returns -EBUSY otherwise. */ +static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, + struct page *page, struct vm_area_struct *src_vma) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + return __folio_try_dup_anon_rmap(folio, page, HPAGE_PMD_NR, src_vma, + RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); + return -EBUSY; +#endif +} + static inline int page_try_dup_anon_rmap(struct page *page, bool compound, struct vm_area_struct *vma) { - VM_BUG_ON_PAGE(!PageAnon(page), page); - - /* - * No need to check+clear for already shared pages, including KSM - * pages. - */ - if (!PageAnonExclusive(page)) - goto dup; + struct folio *folio = page_folio(page); - /* - * If this page may have been pinned by the parent process, - * don't allow to duplicate the mapping but instead require to e.g., - * copy the page immediately for the child so that we'll always - * guarantee the pinned page won't be randomly replaced in the - * future on write faults. - */ - if (likely(!is_device_private_page(page)) && - unlikely(page_needs_cow_for_dma(vma, page))) - return -EBUSY; - - ClearPageAnonExclusive(page); - /* - * It's okay to share the anon page between both processes, mapping - * the page R/O into both processes. - */ -dup: - __page_dup_rmap(page, compound); - return 0; + if (likely(!compound)) + return folio_try_dup_anon_rmap_pte(folio, page, vma); + return folio_try_dup_anon_rmap_pmd(folio, page, vma); } /** From patchwork Mon Dec 11 15:56:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176782 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7153882vqy; Mon, 11 Dec 2023 08:00:06 -0800 (PST) X-Google-Smtp-Source: AGHT+IEqr4i0NX2XSMAu+jyDzngCFsGPycBChsfgmglLb0bhrPvwG/ySeRPa2foJlSwTAc2TES78 X-Received: by 2002:a05:6a21:78a2:b0:191:50d7:aab1 with SMTP id bf34-20020a056a2178a200b0019150d7aab1mr505108pzc.37.1702310406365; Mon, 11 Dec 2023 08:00:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310406; cv=none; d=google.com; s=arc-20160816; b=ncV9+z+Lv7aUKxzssYN0k9NhKKPNoM0lsieaCyLNuz2rolk0+CrD8GadpYysM1VUyj y2f91iZ8wjqg3MKdgafHjvBvGtzLGSvvo5lWMptc85dapQ9dpAUW0qZ/m0cHjbuArmEu nLUn4AOWk6d0eJKM0ibASx+j2VzHsakqWvLSo5370Fkd1W7lGj4qYIoWmrUn1Uhxm0Xi TxisO0sGdF1bllXATnwBV1QHHAvz+ysxpmpZX3Q+k3LTS3iVRKd30oQX4JHRMepSnPXS dKXG+xMee+aUpIrmt1d02Y/+rzN9+1hPjBe4V8vdvsfSTzmKoB/ccajR0mjIaIN5nlvK /yOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=dpwgLE5BjdH1zvm+T58sLO1pe3sX/Qw4UhxJ1rV6Cd4=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=xakznLs9lLxYGK9iavQz0DZr5hEMOQQhWf0q2QyJANlWgDdia4S8iqoXl+sqv1U9ab yTq3rFgcXFElVKkT1FfLwI6bHikoHxVS7rthYjJkzF3RCTB6CSmuEU76J8s0uwSn2fQz VezRs7N/7ZfpdKibN7LKnYPuw/VXQke3zF0PDMixySJIrbuiB/94eJXawFSJXbIYpuTI ce1bOSmn0286pOiqLHMRSR+poOWbmvphJKf+2uU5xMKmiw48krtOmI1EU50Vakx79xMh eHBwd55o1BgFDa8DCtdtJj6P+zYs/Damgv7zTXsbcV3E+hroWXSoOZCH3L+LEF4ePt+S E40w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=hfgFALla; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id q6-20020a655246000000b005b92842d469si6246431pgp.62.2023.12.11.08.00.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 08:00:06 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=hfgFALla; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 17A9B803D838; Mon, 11 Dec 2023 08:00:01 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344508AbjLKP7q (ORCPT + 99 others); Mon, 11 Dec 2023 10:59:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45440 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344707AbjLKP6g (ORCPT ); Mon, 11 Dec 2023 10:58:36 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 784BC10C7 for ; Mon, 11 Dec 2023 07:58:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310312; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dpwgLE5BjdH1zvm+T58sLO1pe3sX/Qw4UhxJ1rV6Cd4=; b=hfgFALlaPgnmM2qhNzr+cidlXtSKePJmIkhix1LGSpma9Od5DPg+z11rcoAz+KBlyxKYuR p7bceMyZatmAJOhgbS+l/Bnb3nzEqboaP8UltPXB3RktWHDKZVU4xnTQs4b27YWp3OHl9s zUTIfs/6XCqekjB1jPH/Bsg876uhUK4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-410-c8kAgQI6P8SQgJ12PSpYrA-1; Mon, 11 Dec 2023 10:58:25 -0500 X-MC-Unique: c8kAgQI6P8SQgJ12PSpYrA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4CCFB870834; Mon, 11 Dec 2023 15:58:25 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2E9131121312; Mon, 11 Dec 2023 15:58:22 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 35/39] mm/huge_memory: page_try_dup_anon_rmap() -> folio_try_dup_anon_rmap_pmd() Date: Mon, 11 Dec 2023 16:56:48 +0100 Message-ID: <20231211155652.131054-36-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 11 Dec 2023 08:00:01 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001836865900502 X-GMAIL-MSGID: 1785001836865900502 Let's convert copy_huge_pmd() and fixup the comment in copy_huge_pud(). While at it, perform more folio conversion in copy_huge_pmd(). Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index cfaa8b823015..34f878916621 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1275,6 +1275,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, { spinlock_t *dst_ptl, *src_ptl; struct page *src_page; + struct folio *src_folio; pmd_t pmd; pgtable_t pgtable = NULL; int ret = -ENOMEM; @@ -1341,11 +1342,12 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, src_page = pmd_page(pmd); VM_BUG_ON_PAGE(!PageHead(src_page), src_page); + src_folio = page_folio(src_page); - get_page(src_page); - if (unlikely(page_try_dup_anon_rmap(src_page, true, src_vma))) { + folio_get(src_folio); + if (unlikely(folio_try_dup_anon_rmap_pmd(src_folio, src_page, src_vma))) { /* Page maybe pinned: split and retry the fault on PTEs. */ - put_page(src_page); + folio_put(src_folio); pte_free(dst_mm, pgtable); spin_unlock(src_ptl); spin_unlock(dst_ptl); @@ -1454,8 +1456,8 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, } /* - * TODO: once we support anonymous pages, use page_try_dup_anon_rmap() - * and split if duplicating fails. + * TODO: once we support anonymous pages, use + * folio_try_dup_anon_rmap_*() and split if duplicating fails. */ pudp_set_wrprotect(src_mm, addr, src_pud); pud = pud_mkold(pud_wrprotect(pud)); From patchwork Mon Dec 11 15:56:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176778 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7153676vqy; Mon, 11 Dec 2023 07:59:59 -0800 (PST) X-Google-Smtp-Source: AGHT+IEKTwFfO+P3mh+p+Cvn5tJP7Vj8kKzhtSlvYF7yeHa4cE9awpw9dn+DSR0+lbSIL312Hz3+ X-Received: by 2002:a17:90b:108:b0:286:cc10:f2c6 with SMTP id p8-20020a17090b010800b00286cc10f2c6mr3468292pjz.20.1702310399107; Mon, 11 Dec 2023 07:59:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310399; cv=none; d=google.com; s=arc-20160816; b=zzb2aUngz8JaeOgJn/RguHkOnQn24FLX+K+RPc/Bub6YpBS98JfsaN7qv6Jv5vQDSZ 5VEV+a3rTWjGSeDjCmIdtCbb6TYJV3E/cZ7ETNEo/k+AOLF6xF+srcdF2/yu3y4WlBt4 ox81z/0LUbvPSQNRBm1TVDbpfwDakzlK3Pw3iyXIxtYIlqkjTwLqIuXLQkbl2tBN2BT4 +WJ2TwEnJ9dC9Twcrn5ECPWNe5H5ftPBaBdRjGM5U02kIBl/kgxI+a+8AXkEwv+5MI+X sMjbLHxLzS+fj67ED/mE/QdwqR4KqSLf3ZtyLwk8CTiwLWcAAXwZnSN1qEWzE4M1CJVL G/1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XplrMRug9dV6L5N0sR7KY1Q40igMgZePuFWGdErJCwc=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=qzbbtUR3PeS1T2naVBK//f55VnRLynXzeWVpPnFjb7jq+4otOwAsaUnPWlWDvjbGff /7mJgUJeO9PSxuPF1RkcH6KTpIFgWb9Eub90g/rnRKgFnv0886Vlm4m5J1BnSq7PgEAx O0RfejWhGj3Kxw6dIzpB0wpj/X2N2qsltzSsKvsb9DSnUKrboDFA3xuNwTLl2M6Fu+5C JL9Js993dP09gzESwIxSwpN36LmqAtjOTAAiudJC0dc8xDmd5vN0jezrhcM42pdZ1V6K +mGk2oHcZB9QWjoeJTtmBQtmsXl/yDC9ehWBM04AWuhII73GnqQr+fTMeCj2jp6WYXFw /vYg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=TI6m4AFd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id x23-20020a17090abc9700b00286c80ac075si7680282pjr.177.2023.12.11.07.59.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 07:59:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=TI6m4AFd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 9131880620B0; Mon, 11 Dec 2023 07:59:46 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344795AbjLKP7e (ORCPT + 99 others); Mon, 11 Dec 2023 10:59:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344701AbjLKP6b (ORCPT ); Mon, 11 Dec 2023 10:58:31 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F3100198 for ; Mon, 11 Dec 2023 07:58:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310311; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XplrMRug9dV6L5N0sR7KY1Q40igMgZePuFWGdErJCwc=; b=TI6m4AFdzJn9HmI4z3Vb9JIPzbkp1reE8zTCuMmmGfDFtzrKIISkTXmJL5OdvvhGo1YQHo Sn0noarObFdXfop3Nclbx8lkLxGi0kZUCHrdeeB4ubY+AA1dlhuV/O5TIoFpsCalW2z7QT wBTY63qUIfk5ZHssbmEdOLe21Olf+4s= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-94-z8fIYRY_PHalWsMPzvXDSw-1; Mon, 11 Dec 2023 10:58:28 -0500 X-MC-Unique: z8fIYRY_PHalWsMPzvXDSw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3551C185A793; Mon, 11 Dec 2023 15:58:28 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9BFAB1121312; Mon, 11 Dec 2023 15:58:25 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 36/39] mm/memory: page_try_dup_anon_rmap() -> folio_try_dup_anon_rmap_pte() Date: Mon, 11 Dec 2023 16:56:49 +0100 Message-ID: <20231211155652.131054-37-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:59:46 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001828948477265 X-GMAIL-MSGID: 1785001828948477265 Let's convert copy_nonpresent_pte(). While at it, perform some more folio conversion. Signed-off-by: David Hildenbrand --- mm/memory.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 42a0b7b41b86..caaf4add6fa2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -785,6 +785,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, unsigned long vm_flags = dst_vma->vm_flags; pte_t orig_pte = ptep_get(src_pte); pte_t pte = orig_pte; + struct folio *folio; struct page *page; swp_entry_t entry = pte_to_swp_entry(orig_pte); @@ -829,6 +830,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, } } else if (is_device_private_entry(entry)) { page = pfn_swap_entry_to_page(entry); + folio = page_folio(page); /* * Update rss count even for unaddressable pages, as @@ -839,10 +841,10 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * for unaddressable pages, at some point. But for now * keep things as they are. */ - get_page(page); + folio_get(folio); rss[mm_counter(page)]++; /* Cannot fail as these pages cannot get pinned. */ - BUG_ON(page_try_dup_anon_rmap(page, false, src_vma)); + folio_try_dup_anon_rmap_pte(folio, page, src_vma); /* * We do not preserve soft-dirty information, because so @@ -956,7 +958,7 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, * future. */ folio_get(folio); - if (unlikely(page_try_dup_anon_rmap(page, false, src_vma))) { + if (unlikely(folio_try_dup_anon_rmap_pte(folio, page, src_vma))) { /* Page may be pinned, we have to copy. */ folio_put(folio); return copy_present_page(dst_vma, src_vma, dst_pte, src_pte, From patchwork Mon Dec 11 15:56:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176786 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7155362vqy; Mon, 11 Dec 2023 08:01:33 -0800 (PST) X-Google-Smtp-Source: AGHT+IEsLAi9lwHdEzVXnSw/5tYMdVBEKCdSIQVnNucVh5Ogicu0wbIsvXHBuvWEFPtJSofO1EpF X-Received: by 2002:a17:90a:5146:b0:286:a1be:c4e with SMTP id k6-20020a17090a514600b00286a1be0c4emr2072314pjm.9.1702310492898; Mon, 11 Dec 2023 08:01:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310492; cv=none; d=google.com; s=arc-20160816; b=pCT+lFRzKJV22eUWeZc57rbwmogKwQ1WfwHrYklIntd6cCXLUnIxmvui6uR20N4fgD llwxoJs/t/cK61fkZWToBqCUfk8HwcOQPv9Gaq/p5SqddAIMu3pxuWJapRQFe31ss4v4 MqvQUbGuDUQskHX6ORCd5l/WlukXXJkbPAPyUZgee8wfYWA11Lxd5Q7Te92RZGNRbLVK wapcjrROJ6g4bXo1fllgFAPb/Vn4YHbnR6KQ5YB7LPUt9c/0NovHRBlerFsA5YB6CggI Xv/pWrLaTDSw3zFB1B2IerkmCtwyg0CpbdfjtwehsYIaf9rkBpba7eU8Z7VEOCUqM88H 8Dbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Bl7kAOkMik26idrH9JyJnR9t8BI/kVxIDTOYx+vyhms=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=iuLczI8Op/CCqZzrQUx3vLG/vfQthaJhlPn2ZnR7UxthjatGkOiaSArfGnCkHbdvF7 DxL3TLVEaQjIeMRu2dYO1DY8JRQ+F+o1fyF7VqSntIIoeeBkK5L8L1GwLLy4O6zHxM1U oWnEjBP0KM9W7jRsTeZFn7uIfnUGyP9etczLbnsgAAJCHGDQPGcimEDkKG7JY+WKUJeZ 1qZyCKg22o/eWw3KCCQtRCyqOEz8RAdWgCRIhG0DrzndYgM/H3qMotfB6EN2ftLELQhG T5TettFFslYgiAbbd2/UHafPNna3iez98/DNxZNrZa9iODdzLCiXJyBzDjSHvmWVQzX7 UUsA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=iv5WN+IE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id u13-20020a17090ae00d00b00286b69f3ba7si7425645pjy.23.2023.12.11.08.01.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 08:01:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=iv5WN+IE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id AF417804C1BC; Mon, 11 Dec 2023 08:00:01 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344555AbjLKP7w (ORCPT + 99 others); Mon, 11 Dec 2023 10:59:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45590 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344726AbjLKP6p (ORCPT ); Mon, 11 Dec 2023 10:58:45 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 03A68171C for ; Mon, 11 Dec 2023 07:58:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310316; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Bl7kAOkMik26idrH9JyJnR9t8BI/kVxIDTOYx+vyhms=; b=iv5WN+IEbxhaG5qmVLgDrAs3JEuq4evZhbFX7EzjlT2Hf3aj2zEjJwCTNlaQLxMA/mADyi IZ+bufr2CgcGcwqpryeTP10e4Sm8i/fELlr6xgCTS33OrSC/PFjj7l8/pL+0JPpK7qUFk5 AXbPbEnIXwjRr/QkTcxNxNoO0oSumPY= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-382-bVYLacyzO7-p7IF_oXo34w-1; Mon, 11 Dec 2023 10:58:31 -0500 X-MC-Unique: bVYLacyzO7-p7IF_oXo34w-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D01CC280640D; Mon, 11 Dec 2023 15:58:30 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 945CB1121306; Mon, 11 Dec 2023 15:58:28 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 37/39] mm/rmap: remove page_try_dup_anon_rmap() Date: Mon, 11 Dec 2023 16:56:50 +0100 Message-ID: <20231211155652.131054-38-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 11 Dec 2023 08:00:02 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001927261444248 X-GMAIL-MSGID: 1785001927261444248 All users are gone, remove page_try_dup_anon_rmap() and any remaining traces. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index c6d8a02ecd56..1e37ee6ae0ba 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -256,7 +256,7 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); -/* See page_try_dup_anon_rmap() */ +/* See folio_try_dup_anon_rmap_*() */ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, struct vm_area_struct *vma) { @@ -481,16 +481,6 @@ static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, #endif } -static inline int page_try_dup_anon_rmap(struct page *page, bool compound, - struct vm_area_struct *vma) -{ - struct folio *folio = page_folio(page); - - if (likely(!compound)) - return folio_try_dup_anon_rmap_pte(folio, page, vma); - return folio_try_dup_anon_rmap_pmd(folio, page, vma); -} - /** * page_try_share_anon_rmap - try marking an exclusive anonymous page possibly * shared to prepare for KSM or temporary unmapping @@ -499,8 +489,8 @@ static inline int page_try_dup_anon_rmap(struct page *page, bool compound, * The caller needs to hold the PT lock and has to have the page table entry * cleared/invalidated. * - * This is similar to page_try_dup_anon_rmap(), however, not used during fork() - * to duplicate a mapping, but instead to prepare for KSM or temporarily + * This is similar to folio_try_dup_anon_rmap_*(), however, not used during + * fork() to duplicate a mapping, but instead to prepare for KSM or temporarily * unmapping a page (swap, migration) via folio_remove_rmap_*(). * * Marking the page shared can only fail if the page may be pinned; device From patchwork Mon Dec 11 15:56:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176783 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7154398vqy; Mon, 11 Dec 2023 08:00:35 -0800 (PST) X-Google-Smtp-Source: AGHT+IEqcdULvFWHq/KepdoN+LVG78KhqxQnwdjr5l5HlAgD2ANfp8UJbjE6OK7ClBQa+7HjtDng X-Received: by 2002:a17:902:ab51:b0:1d3:1afb:fe86 with SMTP id ij17-20020a170902ab5100b001d31afbfe86mr858696plb.105.1702310434297; Mon, 11 Dec 2023 08:00:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310434; cv=none; d=google.com; s=arc-20160816; b=kK+ggtqVNypLscFgBnWTiEBOUF4Rzxtw7nB+JudPcb0ulnunGkLv9BTkxfK6eNfKNt 53dPqSh7mpPer2GQUsSDB2wHMCJSuF1PJ+EY83hbJ9561m595yen6MqMWNCc7xhSA3sH AwH0UmABB8YxtK9BbPn0y0DGWUtdbepycGPVjnxcYQ7hXqwTAJh0h6Aq7U81u46szHGI xSWU4vba36X7GmhJ1AzmbT8+eQwpz6HNB46nmSmc7XwqD6fD1YQfnAzljpw91/gOcts5 Heo0IpZiaef7EYQDEd6F1l6ljIDdkvpkW7Lx3XNgc2UPk6znd7QfxokLjC3PsbBHvHRW d+0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=lJSSYac1nr+7xVv+4BgklrjvdDaeoEoSEBfa7pAELWg=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=0dmc0i7aWqiG59QTCSQ+xEBnmGuhP0LeTEfKoMy+x1o/mWg/OY/rmGPfYtGqredzR7 OQ209zjR84vvwm7kKCdWReJOYZJOBHqQSPo5LizJci7I2yGInvawUjq1twC/Ek/Y6B37 2pqOkNp918Vki/xGl0DfGw0FYOb7Hc3Xl822nAewt+wQMPjLtIRFTfr6jrLaf1S+Qd/V u4y2q5+jysewtLLb37MukZQATanv6cs5+bjqwMtWmAJIgmaxUJIIycrAP+Ay+EUtqgJW qTZ92mLL585OvX6Po2BMTPk/Da/CY6x/W5dZmxYmY2Vw0Sb6U0eoyLKboFqWr7wYqg0S kxdg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=NvdAjT3x; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id j12-20020a170903024c00b001cfc46abb07si6418962plh.128.2023.12.11.08.00.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 08:00:34 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=NvdAjT3x; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 45043807B4B1; Mon, 11 Dec 2023 08:00:07 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344669AbjLKP7y (ORCPT + 99 others); Mon, 11 Dec 2023 10:59:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39826 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344611AbjLKP6p (ORCPT ); Mon, 11 Dec 2023 10:58:45 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3E96710FC for ; Mon, 11 Dec 2023 07:58:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310315; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lJSSYac1nr+7xVv+4BgklrjvdDaeoEoSEBfa7pAELWg=; b=NvdAjT3xHBDI58QOlDRGBJtkg4gP6Vaz/Ig0DFi/o/769AEkg9ZSxyJu0f97H67sSC0Ik/ O/4vRufZKoFYq0KQsyM4VpKqd1MjyylBBSmQ27rZmpdhBNdrEeToPDtezfJLCGmaWuTp/c uJoYrOjj+KWjdNRg/8ItrhnKyTQxPSM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-616-cwKjNob0OqmmW5y0fjpgYA-1; Mon, 11 Dec 2023 10:58:33 -0500 X-MC-Unique: cwKjNob0OqmmW5y0fjpgYA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B6830185A787; Mon, 11 Dec 2023 15:58:32 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 14FA71121306; Mon, 11 Dec 2023 15:58:30 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 38/39] mm: convert page_try_share_anon_rmap() to folio_try_share_anon_rmap_[pte|pmd]() Date: Mon, 11 Dec 2023 16:56:51 +0100 Message-ID: <20231211155652.131054-39-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 11 Dec 2023 08:00:07 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001865773930467 X-GMAIL-MSGID: 1785001865773930467 Let's convert it like we converted all the other rmap functions. Don't introduce folio_try_share_anon_rmap_ptes() for now, as we don't have a user that wants rmap batching in sight. Pretty easy to add later. All users are easy to convert -- only ksm.c doesn't use folios yet but that is left for future work -- so let's just do it in a single shot. While at it, turn the BUG_ON into a WARN_ON_ONCE. Note that page_try_share_anon_rmap() so far didn't care about pte/pmd mappings (no compound parameter). We're changing that so we can perform better sanity checks and make the code actually more readable/consistent. For example, __folio_rmap_sanity_checks() will make sure that a PMD range actually falls completely into the folio. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 95 +++++++++++++++++++++++++++++++++----------- mm/gup.c | 2 +- mm/huge_memory.c | 9 +++-- mm/internal.h | 4 +- mm/ksm.c | 5 ++- mm/migrate_device.c | 2 +- mm/rmap.c | 11 ++--- 7 files changed, 89 insertions(+), 39 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 1e37ee6ae0ba..1e54a28cc884 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -272,7 +272,7 @@ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, return 0; } -/* See page_try_share_anon_rmap() */ +/* See folio_try_share_anon_rmap_*() */ static inline int hugetlb_try_share_anon_rmap(struct folio *folio) { VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); @@ -481,30 +481,15 @@ static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, #endif } -/** - * page_try_share_anon_rmap - try marking an exclusive anonymous page possibly - * shared to prepare for KSM or temporary unmapping - * @page: the exclusive anonymous page to try marking possibly shared - * - * The caller needs to hold the PT lock and has to have the page table entry - * cleared/invalidated. - * - * This is similar to folio_try_dup_anon_rmap_*(), however, not used during - * fork() to duplicate a mapping, but instead to prepare for KSM or temporarily - * unmapping a page (swap, migration) via folio_remove_rmap_*(). - * - * Marking the page shared can only fail if the page may be pinned; device - * private pages cannot get pinned and consequently this function cannot fail. - * - * Returns 0 if marking the page possibly shared succeeded. Returns -EBUSY - * otherwise. - */ -static inline int page_try_share_anon_rmap(struct page *page) +static __always_inline int __folio_try_share_anon_rmap(struct folio *folio, + struct page *page, int nr_pages, enum rmap_mode mode) { - VM_BUG_ON_PAGE(!PageAnon(page) || !PageAnonExclusive(page), page); + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + VM_WARN_ON_FOLIO(!PageAnonExclusive(page), folio); + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); - /* device private pages cannot get pinned via GUP. */ - if (unlikely(is_device_private_page(page))) { + /* device private folios cannot get pinned via GUP. */ + if (unlikely(folio_is_device_private(folio))) { ClearPageAnonExclusive(page); return 0; } @@ -555,7 +540,7 @@ static inline int page_try_share_anon_rmap(struct page *page) if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) smp_mb(); - if (unlikely(page_maybe_dma_pinned(page))) + if (unlikely(folio_maybe_dma_pinned(folio))) return -EBUSY; ClearPageAnonExclusive(page); @@ -568,6 +553,68 @@ static inline int page_try_share_anon_rmap(struct page *page) return 0; } +/** + * folio_try_share_anon_rmap_pte - try marking an exclusive anonymous page + * mapped by a PTE possibly shared to prepare + * for KSM or temporary unmapping + * @folio: The folio to share a mapping of + * @page: The mapped exclusive page + * + * The caller needs to hold the page table lock and has to have the page table + * entries cleared/invalidated. + * + * This is similar to folio_try_dup_anon_rmap_pte(), however, not used during + * fork() to duplicate mappings, but instead to prepare for KSM or temporarily + * unmapping parts of a folio (swap, migration) via folio_remove_rmap_pte(). + * + * Marking the mapped page shared can only fail if the folio maybe pinned; + * device private folios cannot get pinned and consequently this function cannot + * fail. + * + * Returns 0 if marking the mapped page possibly shared succeeded. Returns + * -EBUSY otherwise. + */ +static inline int folio_try_share_anon_rmap_pte(struct folio *folio, + struct page *page) +{ + return __folio_try_share_anon_rmap(folio, page, 1, RMAP_MODE_PTE); +} + +/** + * folio_try_share_anon_rmap_pmd - try marking an exclusive anonymous page + * range mapped by a PMD possibly shared to + * prepare for temporary unmapping + * @folio: The folio to share the mapping of + * @page: The first page to share the mapping of + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock and has to have the page table + * entries cleared/invalidated. + * + * This is similar to folio_try_dup_anon_rmap_pmd(), however, not used during + * fork() to duplicate a mapping, but instead to prepare for temporarily + * unmapping parts of a folio (swap, migration) via folio_remove_rmap_pmd(). + * + * Marking the mapped pages shared can only fail if the folio maybe pinned; + * device private folios cannot get pinned and consequently this function cannot + * fail. + * + * Returns 0 if marking the mapped pages possibly shared succeeded. Returns + * -EBUSY otherwise. + */ +static inline int folio_try_share_anon_rmap_pmd(struct folio *folio, + struct page *page) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + return __folio_try_share_anon_rmap(folio, page, HPAGE_PMD_NR, + RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); + return -EBUSY; +#endif +} + /* * Called from mm/vmscan.c to handle paging out */ diff --git a/mm/gup.c b/mm/gup.c index 0a5f0e91bfec..df83182ec72d 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -177,7 +177,7 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags) /* * Adjust the pincount before re-checking the PTE for changes. * This is essentially a smp_mb() and is paired with a memory - * barrier in page_try_share_anon_rmap(). + * barrier in folio_try_share_anon_rmap_*(). */ smp_mb__after_atomic(); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 34f878916621..c681296fa429 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2523,10 +2523,11 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, * In case we cannot clear PageAnonExclusive(), split the PMD * only and let try_to_migrate_one() fail later. * - * See page_try_share_anon_rmap(): invalidate PMD first. + * See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */ anon_exclusive = PageAnonExclusive(page); - if (freeze && anon_exclusive && page_try_share_anon_rmap(page)) + if (freeze && anon_exclusive && + folio_try_share_anon_rmap_pmd(folio, page)) freeze = false; if (!freeze) { rmap_t rmap_flags = RMAP_NONE; @@ -3554,9 +3555,9 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, flush_cache_range(vma, address, address + HPAGE_PMD_SIZE); pmdval = pmdp_invalidate(vma, address, pvmw->pmd); - /* See page_try_share_anon_rmap(): invalidate PMD first. */ + /* See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */ anon_exclusive = folio_test_anon(folio) && PageAnonExclusive(page); - if (anon_exclusive && page_try_share_anon_rmap(page)) { + if (anon_exclusive && folio_try_share_anon_rmap_pmd(folio, page)) { set_pmd_at(mm, address, pvmw->pmd, pmdval); return -EBUSY; } diff --git a/mm/internal.h b/mm/internal.h index a94355e70bd7..29589bc3f046 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1047,7 +1047,7 @@ enum { * * Ordinary GUP: Using the PT lock * * GUP-fast and fork(): mm->write_protect_seq * * GUP-fast and KSM or temporary unmapping (swap, migration): see - * page_try_share_anon_rmap() + * folio_try_share_anon_rmap_*() * * Must be called with the (sub)page that's actually referenced via the * page table entry, which might not necessarily be the head page for a @@ -1090,7 +1090,7 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma, return is_cow_mapping(vma->vm_flags); } - /* Paired with a memory barrier in page_try_share_anon_rmap(). */ + /* Paired with a memory barrier in folio_try_share_anon_rmap_*(). */ if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) smp_rmb(); diff --git a/mm/ksm.c b/mm/ksm.c index b3d0cfaa2533..be76e9dabf4f 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1161,8 +1161,9 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, goto out_unlock; } - /* See page_try_share_anon_rmap(): clear PTE first. */ - if (anon_exclusive && page_try_share_anon_rmap(page)) { + /* See folio_try_share_anon_rmap_pte(): clear PTE first. */ + if (anon_exclusive && + folio_try_share_anon_rmap_pte(page_folio(page), page)) { set_pte_at(mm, pvmw.address, pvmw.pte, entry); goto out_unlock; } diff --git a/mm/migrate_device.c b/mm/migrate_device.c index c51c99151ebb..9d0c1ad73722 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -202,7 +202,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, if (anon_exclusive) { pte = ptep_clear_flush(vma, addr, ptep); - if (page_try_share_anon_rmap(page)) { + if (folio_try_share_anon_rmap_pte(folio, page)) { set_pte_at(mm, addr, ptep, pte); folio_unlock(folio); folio_put(folio); diff --git a/mm/rmap.c b/mm/rmap.c index b08dd7d6779d..45296739236f 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1868,9 +1868,9 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, break; } - /* See page_try_share_anon_rmap(): clear PTE first. */ + /* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive && - page_try_share_anon_rmap(subpage)) { + folio_try_share_anon_rmap_pte(folio, subpage)) { swap_free(entry); set_pte_at(mm, address, pvmw.pte, pteval); ret = false; @@ -2144,7 +2144,8 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, pte_t swp_pte; if (anon_exclusive) - BUG_ON(page_try_share_anon_rmap(subpage)); + WARN_ON_ONCE(folio_try_share_anon_rmap_pte(folio, + subpage)); /* * Store the pfn of the page in a special migration @@ -2215,7 +2216,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, VM_BUG_ON_PAGE(pte_write(pteval) && folio_test_anon(folio) && !anon_exclusive, subpage); - /* See page_try_share_anon_rmap(): clear PTE first. */ + /* See folio_try_share_anon_rmap_pte(): clear PTE first. */ if (folio_test_hugetlb(folio)) { if (anon_exclusive && hugetlb_try_share_anon_rmap(folio)) { @@ -2226,7 +2227,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, break; } } else if (anon_exclusive && - page_try_share_anon_rmap(subpage)) { + folio_try_share_anon_rmap_pte(folio, subpage)) { set_pte_at(mm, address, pvmw.pte, pteval); ret = false; page_vma_mapped_walk_done(&pvmw); From patchwork Mon Dec 11 15:56:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 176784 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp7155001vqy; Mon, 11 Dec 2023 08:01:11 -0800 (PST) X-Google-Smtp-Source: AGHT+IE+19wH8qeaserKI8mpL28NJ1C17dVrSB2KEE3x6OCr3fnDpGzMMEExh+urVOkWR3dy2trz X-Received: by 2002:a05:6a00:391c:b0:6ce:2731:e864 with SMTP id fh28-20020a056a00391c00b006ce2731e864mr4975404pfb.43.1702310471455; Mon, 11 Dec 2023 08:01:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702310471; cv=none; d=google.com; s=arc-20160816; b=v4OnDs2QUp5xLrQaBW+c4bMcs6dNH/XBktyAMBXjvT7NJ6+1YSesbtWPGdHvogyOJA 3ugGevqtXAqpmwPetBJEVxYlM9WAXMtop+gdSpglsFCvxexX1Gq0+i/iAHp15MpiZh8N TtvuEaqVrzS4DYSN8bqoU9T2yrTbrhKWpzqulYoc/UbdBSwz9qcuvRz8PHH0m5+tYGTb F3JZPWgaJ/vS5r0Np+2BFjQTsd8VWKoRxv9BLk+N2kNFfa9QKBvCrgZid+yUjBg2ebrH dac0jcLm9Tf63Cz23YK+jQ7h/xnc624FrZvi3OU4O8IncVgeUlwwZUtlJDkSeYR/WOhM p3bA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=8kCOKQnn/YILhh4/y6F4VVgOwWNItKHStdMCkjwZ7+g=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=lJNgBWWVHzqpgN0kIJ5UpIpNRaL3WmXvFU95BV4ED9Fyu3QPs7hNRHzBQ2DW5ViHlK r9dHhgQvLXGyZDmFQwtz5922pmuu9vTLgY0Cm2MMOdEAl+BWrN+R/rKtw9JXdJOIWWAl IkHDsuY+gx74gEqSE5CGr8vfmGldVWSQU7MCSNJt1tSE4Im5zPhsHmcbgLr81qPHMeFi 1C4P11t6kG5YsQnGMzjtIxyr/ZUq29Z8xj81T4OEhux1UhU9tsi7a4BHehNG/YFKswSP 2jHVT+ZNH/6zL4voE57YsmRibG8DDubp/Jd/1QZ4BDnte/PmD4kCTV3ujBNWFaml8YyC 5jbQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=gNXV28fx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id a20-20020a656414000000b005c605cdc2c2si6093875pgv.854.2023.12.11.08.01.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Dec 2023 08:01:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=gNXV28fx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id D512780615CB; Mon, 11 Dec 2023 08:01:04 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344551AbjLKQAK (ORCPT + 99 others); Mon, 11 Dec 2023 11:00:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54534 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344754AbjLKP7D (ORCPT ); Mon, 11 Dec 2023 10:59:03 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9C980173B for ; Mon, 11 Dec 2023 07:58:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310318; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8kCOKQnn/YILhh4/y6F4VVgOwWNItKHStdMCkjwZ7+g=; b=gNXV28fxHJ0DMq3uCRbTfg/h02R+XOG4iZwBZH3CJEWWBHnA/KF3EUbdbHzheJuLmxXBrW dgIJEfH4Iw6oseOWOY6OvhCK8whw/OX2OJACyhVa7R6TfHUTnBJcINDT0sWqR8/ZMOnhh6 lpN/WM2ro3TK8Lbu65Bcfd+wwsln+lc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-461-PqgcZlcINPWhAdvqUvsCRQ-1; Mon, 11 Dec 2023 10:58:35 -0500 X-MC-Unique: PqgcZlcINPWhAdvqUvsCRQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E3F70848A68; Mon, 11 Dec 2023 15:58:34 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id F28F81121306; Mon, 11 Dec 2023 15:58:32 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 39/39] mm/rmap: rename COMPOUND_MAPPED to ENTIRELY_MAPPED Date: Mon, 11 Dec 2023 16:56:52 +0100 Message-ID: <20231211155652.131054-40-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Mon, 11 Dec 2023 08:01:05 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785001905150369260 X-GMAIL-MSGID: 1785001905150369260 We removed all "bool compound" and RMAP_COMPOUND parameters. Let's remove the remaining "compound" terminology by making COMPOUND_MAPPED match the "folio->_entire_mapcount" terminology, renaming it to ENTIRELY_MAPPED. ENTIRELY_MAPPED is only used when the whole folio is mapped using a single page table entry (e.g., a single PMD mapping a PMD-sized THP). For now, we don't support mapping any THP bigger than that, so ENTIRELY_MAPPED only applies to PMD-mapped PMD-sized THP only. Signed-off-by: David Hildenbrand --- Documentation/mm/transhuge.rst | 2 +- mm/internal.h | 6 +++--- mm/rmap.c | 18 +++++++++--------- 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/Documentation/mm/transhuge.rst b/Documentation/mm/transhuge.rst index cf81272a6b8b..93c9239b9ebe 100644 --- a/Documentation/mm/transhuge.rst +++ b/Documentation/mm/transhuge.rst @@ -117,7 +117,7 @@ pages: - map/unmap of a PMD entry for the whole THP increment/decrement folio->_entire_mapcount and also increment/decrement - folio->_nr_pages_mapped by COMPOUND_MAPPED when _entire_mapcount + folio->_nr_pages_mapped by ENTIRELY_MAPPED when _entire_mapcount goes from -1 to 0 or 0 to -1. - map/unmap of individual pages with PTE entry increment/decrement diff --git a/mm/internal.h b/mm/internal.h index 29589bc3f046..188807d2aebc 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -54,12 +54,12 @@ void page_writeback_init(void); /* * If a 16GB hugetlb folio were mapped by PTEs of all of its 4kB pages, - * its nr_pages_mapped would be 0x400000: choose the COMPOUND_MAPPED bit + * its nr_pages_mapped would be 0x400000: choose the ENTIRELY_MAPPED bit * above that range, instead of 2*(PMD_SIZE/PAGE_SIZE). Hugetlb currently * leaves nr_pages_mapped at 0, but avoid surprise if it participates later. */ -#define COMPOUND_MAPPED 0x800000 -#define FOLIO_PAGES_MAPPED (COMPOUND_MAPPED - 1) +#define ENTIRELY_MAPPED 0x800000 +#define FOLIO_PAGES_MAPPED (ENTIRELY_MAPPED - 1) /* * Flags passed to __show_mem() and show_free_areas() to suppress output in diff --git a/mm/rmap.c b/mm/rmap.c index 45296739236f..53753834a10d 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1173,7 +1173,7 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, first = atomic_inc_and_test(&page->_mapcount); if (first && folio_test_large(folio)) { first = atomic_inc_return_relaxed(mapped); - first = (first < COMPOUND_MAPPED); + first = (first < ENTIRELY_MAPPED); } if (first) @@ -1183,15 +1183,15 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, case RMAP_MODE_PMD: first = atomic_inc_and_test(&folio->_entire_mapcount); if (first) { - nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { + nr = atomic_add_return_relaxed(ENTIRELY_MAPPED, mapped); + if (likely(nr < ENTIRELY_MAPPED + ENTIRELY_MAPPED)) { *nr_pmdmapped = folio_nr_pages(folio); nr = *nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of a remove and another add? */ if (unlikely(nr < 0)) nr = 0; } else { - /* Raced ahead of a remove of COMPOUND_MAPPED */ + /* Raced ahead of a remove of ENTIRELY_MAPPED */ nr = 0; } } @@ -1434,7 +1434,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, } else { /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); - atomic_set(&folio->_nr_pages_mapped, COMPOUND_MAPPED); + atomic_set(&folio->_nr_pages_mapped, ENTIRELY_MAPPED); SetPageAnonExclusive(&folio->page); __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr); } @@ -1516,7 +1516,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, last = atomic_add_negative(-1, &page->_mapcount); if (last && folio_test_large(folio)) { last = atomic_dec_return_relaxed(mapped); - last = (last < COMPOUND_MAPPED); + last = (last < ENTIRELY_MAPPED); } if (last) @@ -1526,15 +1526,15 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, case RMAP_MODE_PMD: last = atomic_add_negative(-1, &folio->_entire_mapcount); if (last) { - nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED)) { + nr = atomic_sub_return_relaxed(ENTIRELY_MAPPED, mapped); + if (likely(nr < ENTIRELY_MAPPED)) { nr_pmdmapped = folio_nr_pages(folio); nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of another remove and an add? */ if (unlikely(nr < 0)) nr = 0; } else { - /* An add of COMPOUND_MAPPED raced ahead */ + /* An add of ENTIRELY_MAPPED raced ahead */ nr = 0; } }