From patchwork Mon Dec 4 14:21:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173364 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2794624vqy; Mon, 4 Dec 2023 06:22:11 -0800 (PST) X-Google-Smtp-Source: AGHT+IFOQxmCVR3aDc0g/DAa04Es8LZh5NoDfNZkIUlUapdF6zvL462vogpMG1GMg3PgpHtk8bd2 X-Received: by 2002:a05:6a20:7494:b0:18a:e003:21b2 with SMTP id p20-20020a056a20749400b0018ae00321b2mr1489346pzd.4.1701699731007; Mon, 04 Dec 2023 06:22:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699730; cv=none; d=google.com; s=arc-20160816; b=b1z1lQhnwVe7gG4DKWcb29qaJCoWz4pzX+Ac9hWnWaeFns1gisu4/xi7xU11rcpViy TGKK/1jj1gyHoxeRakCYiD3e/kmLviY80R2AUJIHEDKZ+GOuiJXucnyDqqjmE5O8wCVx linjwlqA+PNyhQ3So+tdrCFurOEkJ5OttsM2ccnCb7Mqz+c4aoLVwlUy6RwdEBQpwmEA g2DIVeUSiKv+qljAV6hcPofbChvrQuF6xqWBAK0lCmFiPBupFKrU1+lYT6bHyQAQIm1H Hesi6Mq9begyVeLKOsugT+HdBS8FG8uqkDBtoOUY5UWPVbJnHb7WA8jpU+/3v7Crrz7k xafQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=DtTuMDxg2eg7y+J7YOtz8EtHkmHKA1dRnnRsMSbHZZc=; fh=zW2d2UaZUoEUxa6MVPed9mRARVtfplPxKZRc44Ak2yc=; b=a61/6WzHMiSMZGDCJoC8OJF0iEvjA8wi9BIFFSuE9lQG1BaBibflxbLol3DTSY6PuF qFKjQWZH0h1JrIOdoe9VDvP5bdemC/ns408ki3GWpkQUVhIIg463LayXikVXLLL2mgXc dDkJG1OmZkc9nmwo9i5r69LhXLUyPHtHLYezb3jTsVeBE2rSrKa1FiPIMBtSRPgGYIfd QstNTKElZpGSKipHHEOTjS3ILIbMz+r1G4N1UuammelK8x8t/WNxY0+OpNg4Op/Gz2kG BtaOVgdy0dFUB6nqfJ/8XydNpkRGDyUyrAcxCGUnud6aRX/gHV8vOzC9uHsAgT/2IGm0 UWDw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=e91Q9iwt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id z14-20020a63e54e000000b005c21d7d1114si8279537pgj.818.2023.12.04.06.22.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:22:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=e91Q9iwt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 73F1E804B38C; Mon, 4 Dec 2023 06:22:04 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345060AbjLDOVx (ORCPT + 99 others); Mon, 4 Dec 2023 09:21:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33406 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235778AbjLDOVu (ORCPT ); Mon, 4 Dec 2023 09:21:50 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31D56AA for ; Mon, 4 Dec 2023 06:21:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699715; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DtTuMDxg2eg7y+J7YOtz8EtHkmHKA1dRnnRsMSbHZZc=; b=e91Q9iwtovq72RMythexJP4UBBJqfPRhHq4HWDpYRzTFQtbVQukrY3ctkkr1whBgLRWCbv J2fMcXy4iKu0XZmfg1RGQbM+Kur9vRaRNnlaxL+5LtDDnGeF7WOU1uLLNcIjD8OOruPbex niHxOfUi7t5Wr2isA79ESdVWLORXOLE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-158-Hayy6FKhOPqtL2ERdoY8Jw-1; Mon, 04 Dec 2023 09:21:51 -0500 X-MC-Unique: Hayy6FKhOPqtL2ERdoY8Jw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 899C08F9201; Mon, 4 Dec 2023 14:21:50 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id C71B12026D4C; Mon, 4 Dec 2023 14:21:48 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu , Muchun Song Subject: [PATCH RFC 01/39] mm/rmap: rename hugepage_add* to hugetlb_add* Date: Mon, 4 Dec 2023 15:21:08 +0100 Message-ID: <20231204142146.91437-2-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:22:04 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361497110357139 X-GMAIL-MSGID: 1784361497110357139 Let's just call it "hugetlb_". Yes, it's all already inconsistent and confusing because we have a lot of "hugepage_" functions for legacy reasons. But "hugetlb" cannot possibly be confused with transparent huge pages, and it matches "hugetlb.c" and "folio_test_hugetlb()". So let's minimize confusion in rmap code. Reviewed-by: Muchun Song Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 4 ++-- mm/hugetlb.c | 8 ++++---- mm/migrate.c | 4 ++-- mm/rmap.c | 8 ++++---- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index b26fe858fd444..4c5bfeb054636 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -203,9 +203,9 @@ void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr, void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); -void hugepage_add_anon_rmap(struct folio *, struct vm_area_struct *, +void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address, rmap_t flags); -void hugepage_add_new_anon_rmap(struct folio *, struct vm_area_struct *, +void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); static inline void __page_dup_rmap(struct page *page, bool compound) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1169ef2f2176f..4cfa0679661e2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5278,7 +5278,7 @@ hugetlb_install_folio(struct vm_area_struct *vma, pte_t *ptep, unsigned long add pte_t newpte = make_huge_pte(vma, &new_folio->page, 1); __folio_mark_uptodate(new_folio); - hugepage_add_new_anon_rmap(new_folio, vma, addr); + hugetlb_add_new_anon_rmap(new_folio, vma, addr); if (userfaultfd_wp(vma) && huge_pte_uffd_wp(old)) newpte = huge_pte_mkuffd_wp(newpte); set_huge_pte_at(vma->vm_mm, addr, ptep, newpte, sz); @@ -5981,7 +5981,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* Break COW or unshare */ huge_ptep_clear_flush(vma, haddr, ptep); page_remove_rmap(&old_folio->page, vma, true); - hugepage_add_new_anon_rmap(new_folio, vma, haddr); + hugetlb_add_new_anon_rmap(new_folio, vma, haddr); if (huge_pte_uffd_wp(pte)) newpte = huge_pte_mkuffd_wp(newpte); set_huge_pte_at(mm, haddr, ptep, newpte, huge_page_size(h)); @@ -6270,7 +6270,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, goto backout; if (anon_rmap) - hugepage_add_new_anon_rmap(folio, vma, haddr); + hugetlb_add_new_anon_rmap(folio, vma, haddr); else page_dup_file_rmap(&folio->page, true); new_pte = make_huge_pte(vma, &folio->page, ((vma->vm_flags & VM_WRITE) @@ -6725,7 +6725,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, if (folio_in_pagecache) page_dup_file_rmap(&folio->page, true); else - hugepage_add_new_anon_rmap(folio, dst_vma, dst_addr); + hugetlb_add_new_anon_rmap(folio, dst_vma, dst_addr); /* * For either: (1) CONTINUE on a non-shared VMA, or (2) UFFDIO_COPY diff --git a/mm/migrate.c b/mm/migrate.c index 35a88334bb3c2..4cb849fa0dd2c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -249,8 +249,8 @@ static bool remove_migration_pte(struct folio *folio, pte = arch_make_huge_pte(pte, shift, vma->vm_flags); if (folio_test_anon(folio)) - hugepage_add_anon_rmap(folio, vma, pvmw.address, - rmap_flags); + hugetlb_add_anon_rmap(folio, vma, pvmw.address, + rmap_flags); else page_dup_file_rmap(new, true); set_huge_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte, diff --git a/mm/rmap.c b/mm/rmap.c index 7a27a2b418021..112467c30b2c9 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2583,8 +2583,8 @@ void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc) * * RMAP_COMPOUND is ignored. */ -void hugepage_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, - unsigned long address, rmap_t flags) +void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, + unsigned long address, rmap_t flags) { VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); @@ -2595,8 +2595,8 @@ void hugepage_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, PageAnonExclusive(&folio->page), folio); } -void hugepage_add_new_anon_rmap(struct folio *folio, - struct vm_area_struct *vma, unsigned long address) +void hugetlb_add_new_anon_rmap(struct folio *folio, + struct vm_area_struct *vma, unsigned long address) { BUG_ON(address < vma->vm_start || address >= vma->vm_end); /* increment count (starts at -1) */ From patchwork Mon Dec 4 14:21:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173366 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2794722vqy; Mon, 4 Dec 2023 06:22:19 -0800 (PST) X-Google-Smtp-Source: AGHT+IEgGJ3Bs97WPgdiYaD8jqW2WpAmIwPayzJQ9uJuy9Pn5m+/vS+HSa7c+d0l/GLACW42P4aN X-Received: by 2002:a17:903:246:b0:1cc:3a8a:f19b with SMTP id j6-20020a170903024600b001cc3a8af19bmr1597218plh.14.1701699738508; Mon, 04 Dec 2023 06:22:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699738; cv=none; d=google.com; s=arc-20160816; b=qVh1LXUHfOK5amJcOeOwFqNL3QhauHDljbQvDSbE6x54xhj/eGgJ3pq1R+m0Zbqvuk ori1+KnBKLmoE5OBt2CDU1LSVFRkWQlrhpx3UL+uYwutbz+I/TXrfyKWje4JC94bO/aH jLDcIoUxbMeNOHTaxL8qvwjUN6T84FUdPvCeWlOU3Nk0IOH/FR9zdEfay7JUusnXa4Zh ECZtQOEv9aal4xwg/w5+D6DWTlNulb/otPJMRvHOX1VsUhIOTg8IIqBDG5cut3zkZK31 FdSkTT36hJhBJRepl6eeGH16kdZCWEUmsEHaB61z/hK8I3d7O1n8mFekNK7Qa4Jz1uZE Gwmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ShB35s+ZfhtJKsIKjmsv8EG/47pRc0gWRbXgiSuPDog=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=wMHOUizzIMacM00QMMn9PlYOtPB7sqE9FKA4YUbdmHrTvABEQ5Ehg4jfYF4nD9A9+M IpKUmMG8hCHsy3xZ0iGMkmmYDu+2PfF6Bcq8/f6WNJN1iyT+0pWmovheHpawb0xEnh2v doyEQszTx8RTwU5O+4LlYjgch/l7G0pQOuoOdQqcmNoLehQCFxBtoPUv6aNJ8aBqkKAe nO9SFuJ6jHqS7gT0Ts0Lq3N32xvUZIfwgIU8fEwpZFyCt63yLWrGA+5o0GGULpPqSg1L momzi4CdObOZJ58FxdHMzxVu3uquASbHoDnZiruZbATgN0YEREw5ajB2MY7dTHZuSuNE FKfA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=C3W7JgbO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id x18-20020a170902ec9200b001d0b3c280f9si999553plg.501.2023.12.04.06.22.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:22:18 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=C3W7JgbO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 36E94803BEE4; Mon, 4 Dec 2023 06:22:16 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345114AbjLDOV5 (ORCPT + 99 others); Mon, 4 Dec 2023 09:21:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52016 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345073AbjLDOVx (ORCPT ); Mon, 4 Dec 2023 09:21:53 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 23862A4 for ; Mon, 4 Dec 2023 06:21:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699718; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ShB35s+ZfhtJKsIKjmsv8EG/47pRc0gWRbXgiSuPDog=; b=C3W7JgbOW9Hx3dVHGdM2loPZ5nwXDS5luSyS6VlDarxz24LYD+TDZz+cwp7v/EkanNUe7V kEJlmdK/y/uUyHtvHTmEDequHnimXsxrn6XDKseocBXAUfueng9yovjet4WR4Mko2A2GEF 1++381fvYn0FkX4HqYRQKsJyxy9MWiY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-526-PsZ5JvICNICyMCT8BrmXEA-1; Mon, 04 Dec 2023 09:21:53 -0500 X-MC-Unique: PsZ5JvICNICyMCT8BrmXEA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 669068314F8; Mon, 4 Dec 2023 14:21:52 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id C83B62026D4C; Mon, 4 Dec 2023 14:21:50 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 02/39] mm/rmap: introduce and use hugetlb_remove_rmap() Date: Mon, 4 Dec 2023 15:21:09 +0100 Message-ID: <20231204142146.91437-3-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:22:16 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361504993589870 X-GMAIL-MSGID: 1784361504993589870 hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. Let's introduce and use hugetlb_remove_rmap() and remove the hugetlb code from page_remove_rmap(). This effectively removes one check on the small-folio path as well. Note: all possible candidates that need care are page_remove_rmap() that pass compound=true. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 5 +++++ mm/hugetlb.c | 4 ++-- mm/rmap.c | 17 ++++++++--------- 3 files changed, 15 insertions(+), 11 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 4c5bfeb054636..e8d1dc1d5361f 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -208,6 +208,11 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +static inline void hugetlb_remove_rmap(struct folio *folio) +{ + atomic_dec(&folio->_entire_mapcount); +} + static inline void __page_dup_rmap(struct page *page, bool compound) { if (compound) { diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4cfa0679661e2..d17bb53b19ff2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5669,7 +5669,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, make_pte_marker(PTE_MARKER_UFFD_WP), sz); hugetlb_count_sub(pages_per_huge_page(h), mm); - page_remove_rmap(page, vma, true); + hugetlb_remove_rmap(page_folio(page)); spin_unlock(ptl); tlb_remove_page_size(tlb, page, huge_page_size(h)); @@ -5980,7 +5980,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* Break COW or unshare */ huge_ptep_clear_flush(vma, haddr, ptep); - page_remove_rmap(&old_folio->page, vma, true); + hugetlb_remove_rmap(old_folio); hugetlb_add_new_anon_rmap(new_folio, vma, haddr); if (huge_pte_uffd_wp(pte)) newpte = huge_pte_mkuffd_wp(newpte); diff --git a/mm/rmap.c b/mm/rmap.c index 112467c30b2c9..5037581b79ec6 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1440,13 +1440,6 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, VM_BUG_ON_PAGE(compound && !PageHead(page), page); - /* Hugetlb pages are not counted in NR_*MAPPED */ - if (unlikely(folio_test_hugetlb(folio))) { - /* hugetlb pages are always mapped with pmds */ - atomic_dec(&folio->_entire_mapcount); - return; - } - /* Is page being unmapped by PTE? Is this its last map to be removed? */ if (likely(!compound)) { last = atomic_add_negative(-1, &page->_mapcount); @@ -1804,7 +1797,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, dec_mm_counter(mm, mm_counter_file(&folio->page)); } discard: - page_remove_rmap(subpage, vma, folio_test_hugetlb(folio)); + if (unlikely(folio_test_hugetlb(folio))) + hugetlb_remove_rmap(folio); + else + page_remove_rmap(subpage, vma, false); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); @@ -2157,7 +2153,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, */ } - page_remove_rmap(subpage, vma, folio_test_hugetlb(folio)); + if (unlikely(folio_test_hugetlb(folio))) + hugetlb_remove_rmap(folio); + else + page_remove_rmap(subpage, vma, false); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); From patchwork Mon Dec 4 14:21:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173365 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2794627vqy; Mon, 4 Dec 2023 06:22:11 -0800 (PST) X-Google-Smtp-Source: AGHT+IH45AKyoKp+w5JaaU8Aa3tDtdu91ItqA0lvmagi3dXAeqL9QP3GgueQnOjJec2u2Na6vMjs X-Received: by 2002:a05:6a00:8c13:b0:6ce:3bfb:b0d4 with SMTP id ih19-20020a056a008c1300b006ce3bfbb0d4mr3128912pfb.45.1701699731344; Mon, 04 Dec 2023 06:22:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699731; cv=none; d=google.com; s=arc-20160816; b=Ap8PdfpgypcLH65H9OlYgHLnc3EDy+IOR2Gr38zaLp1x7W0mCxQDIsOxzIMoeWt8yB ynW/HY4vdp1R84Qzk0PKbDli3yn/CH8l6M6Pqb2QQ4uNOhWzCrMYxkwlCzxuy4zpBDA/ zib+eTm+ciLr++jsuvh3Qm+UQMxCN+vFZEH8UkHNKaJMrP4myCHc5jbxT7ODfv7/Arpq Tmj0LmSsIz7YM3vHid8u42CVfT+KreHiErbaJ0uMdPODsYfHqtZPKrXRxQQLtS6guq3R SD/3BZvIr7a9tUfM1wOXxmqL+MdQ2gIurU+sPmT522Nfc1cYfmtInZmtFhstuq/w58w8 iwkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ED3P7Gmg7VM9PqzBG8DklxM7dI8K8CnTi9/+BO29yAM=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=YpqATL34wbzIa8Dy/XNlVOQAxoz45ivsTlkZbPRBpbL7QUlh09Lm86KnN7UfRavT7C y7zZq9qfdGcGvDeGlN4BvtPlwfMawgnkmg3bHQtqQ2Y25xTX3KuKQA5145yzjXmAlxe4 VX9m8DU6kLYjfC0lMJVcTXvxjsBcWgDBsRGov/CTTEdAXwsOb6URMA77kHoF8xd72CUb T9YOaNAYipov/xZNzvYi9xLHeqRRfPDnfUjolmzMia38rlwVWMKJreK3rz8RnIP8rRxE f3tR5Oe5eLaqRFTlSUGDYBMoNETmquLvUpkaLpYfS3He9hcOUO2G8nsQcgsp0G53chq6 fj8A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=DgMtRCae; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id fi5-20020a056a00398500b006cddefd2031si7961966pfb.37.2023.12.04.06.22.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:22:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=DgMtRCae; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 2810380A73EA; Mon, 4 Dec 2023 06:22:10 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345086AbjLDOVz (ORCPT + 99 others); Mon, 4 Dec 2023 09:21:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345067AbjLDOVx (ORCPT ); Mon, 4 Dec 2023 09:21:53 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E9814102 for ; Mon, 4 Dec 2023 06:21:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699718; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ED3P7Gmg7VM9PqzBG8DklxM7dI8K8CnTi9/+BO29yAM=; b=DgMtRCaecFSrXO0XC6touXlnpRUhPmG7WAfHI+OYVYp/L1atg/MaBM2G5vP9cgD7ioY0MZ qQpKD5tvxGbTBakeSalr+QyeAZAkgx+inxYJUFbBhG/aZl7CjGwn5L6WJbSBcITAx5v4tx 3oBsAw6NGN5+R3txrN0bXuCnd2QBrpE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-460-Qm87iSsWM7CFHWpPHaPMdA-1; Mon, 04 Dec 2023 09:21:55 -0500 X-MC-Unique: Qm87iSsWM7CFHWpPHaPMdA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 41BF2811E7E; Mon, 4 Dec 2023 14:21:54 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9EFFB2026D4C; Mon, 4 Dec 2023 14:21:52 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 03/39] mm/rmap: introduce and use hugetlb_add_file_rmap() Date: Mon, 4 Dec 2023 15:21:10 +0100 Message-ID: <20231204142146.91437-4-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:22:10 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361497096045837 X-GMAIL-MSGID: 1784361497096045837 hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. Right now we're using page_dup_file_rmap() in some cases where "ordinary" rmap code would have used page_add_file_rmap(). So let's introduce and use hugetlb_add_file_rmap() instead. We won't be adding a "hugetlb_dup_file_rmap()" functon for the fork() case, as it would be doing the same: "dup" is just an optimization for "add". What remains is a single page_dup_file_rmap() call in fork() code. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- include/linux/rmap.h | 7 +++++++ mm/hugetlb.c | 6 +++--- mm/migrate.c | 2 +- 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index e8d1dc1d5361f..0a81e8420a961 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -208,6 +208,13 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +static inline void hugetlb_add_file_rmap(struct folio *folio) +{ + VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); + + atomic_inc(&folio->_entire_mapcount); +} + static inline void hugetlb_remove_rmap(struct folio *folio) { atomic_dec(&folio->_entire_mapcount); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d17bb53b19ff2..541a8f38cfdc7 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5401,7 +5401,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, * sleep during the process. */ if (!folio_test_anon(pte_folio)) { - page_dup_file_rmap(&pte_folio->page, true); + hugetlb_add_file_rmap(pte_folio); } else if (page_try_dup_anon_rmap(&pte_folio->page, true, src_vma)) { pte_t src_pte_old = entry; @@ -6272,7 +6272,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, if (anon_rmap) hugetlb_add_new_anon_rmap(folio, vma, haddr); else - page_dup_file_rmap(&folio->page, true); + hugetlb_add_file_rmap(folio); new_pte = make_huge_pte(vma, &folio->page, ((vma->vm_flags & VM_WRITE) && (vma->vm_flags & VM_SHARED))); /* @@ -6723,7 +6723,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, goto out_release_unlock; if (folio_in_pagecache) - page_dup_file_rmap(&folio->page, true); + hugetlb_add_file_rmap(folio); else hugetlb_add_new_anon_rmap(folio, dst_vma, dst_addr); diff --git a/mm/migrate.c b/mm/migrate.c index 4cb849fa0dd2c..de9d94b99ab78 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -252,7 +252,7 @@ static bool remove_migration_pte(struct folio *folio, hugetlb_add_anon_rmap(folio, vma, pvmw.address, rmap_flags); else - page_dup_file_rmap(new, true); + hugetlb_add_file_rmap(folio); set_huge_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte, psize); } else From patchwork Mon Dec 4 14:21:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173368 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2794871vqy; Mon, 4 Dec 2023 06:22:30 -0800 (PST) X-Google-Smtp-Source: AGHT+IEVvR4a503GZ3pt0aNyYcZgN56P4JKwY/QJkNaxsJYrmMSg7uNmgLEkEXcH2hbV0WATTSin X-Received: by 2002:a05:6a00:84b:b0:6ce:2731:e875 with SMTP id q11-20020a056a00084b00b006ce2731e875mr4461047pfk.60.1701699750327; Mon, 04 Dec 2023 06:22:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699750; cv=none; d=google.com; s=arc-20160816; b=vHLR9RTf2YOqhZeN61bcHLX3sfpZfYT82GKZm/yPLQNVAijK446HVJqBWrDOwp3xIM Ti9/5HmYkyme5VkxrvQnXxiU1b3ooQRk9pw1sQXJnXXp63sl0pNt/e8RHWTvcimu2qfH j1a+bK48s1Yw6na0T/poIXGTB4+U9o2OnhhBEamYnln8QEb8h9zmW7w2sqTKvlFXFRNa 6C2c+VcNRErN3BUrH++3GplEav9hAKY23IONaLA29iAtqtx4/+Nigybltc1Pmmv9L4Yc jj3FI8iYRMiQop7CC4b2sJ5QWO9rQGIBiTqyXuBLdQJQAZHuuKYMoX35cOY+lTeWsFf/ Hlqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CldTAIMRJWc6rWIvSdlg20t+rIQF+KSKFiNVs+JtPQE=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=sR+ilI94nmGwyn3nup4YuMaGYvI0FlrYSAm7VkXBimIPEzmQT/TMyycbz3uOofvKRd mk6VIKRPtJU8ZoHODXKvadxhZBI/8Y2F2Y4bxrPqcwuFxHwTspTtZOmezvbZpxXQP3wT 14e6eiyS6euHmegq1TBrg/80TNT0IpkaAYFy4VIZX/TfTi/Nxd+fq2ISi7glq9OMbJUe N+tPdAgN7t2ZikPSEamJWwLNkLkpI8TIMIgvqSIISOchVuvuyDrG0N6SwMNVuluxjKV8 MWyobuOToqdiSw0D86LZD43VZproqT+yYFeLQ6k1s4yVpUxH9vAuECweBAa/0zsjrYVe VCqg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=LUzB0Tss; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id y41-20020a056a001ca900b006cbb1347e8esi7947211pfw.257.2023.12.04.06.22.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:22:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=LUzB0Tss; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 478FA80568E3; Mon, 4 Dec 2023 06:22:23 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345078AbjLDOWA (ORCPT + 99 others); Mon, 4 Dec 2023 09:22:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52120 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345095AbjLDOV4 (ORCPT ); Mon, 4 Dec 2023 09:21:56 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6703AA0 for ; Mon, 4 Dec 2023 06:22:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699721; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CldTAIMRJWc6rWIvSdlg20t+rIQF+KSKFiNVs+JtPQE=; b=LUzB0TssQaPwMcBy7PHK/8CHv/UUKZrB39+csy3bIPQG/xnlJS7fKoa5L7C2yS29qFbNMC ND+B4PkGyDpE2TiCPGj48iG0cDc29ZicSQm9gCeFbmF0J4XkJ1Qn6ti5W+/VqtCZ/l1Uri TmKDbKgQxTagGpkG4ucvqUGSU17NCGA= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-119-IrgCM3BPPOeYAKNDeamZgw-1; Mon, 04 Dec 2023 09:21:56 -0500 X-MC-Unique: IrgCM3BPPOeYAKNDeamZgw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1E5A73C0FCAD; Mon, 4 Dec 2023 14:21:56 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 79B642026D4C; Mon, 4 Dec 2023 14:21:54 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 04/39] mm/rmap: introduce and use hugetlb_try_dup_anon_rmap() Date: Mon, 4 Dec 2023 15:21:11 +0100 Message-ID: <20231204142146.91437-5-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:22:23 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361517107556594 X-GMAIL-MSGID: 1784361517107556594 hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. So let's introduce and use hugetlb_try_dup_anon_rmap() to make all hugetlb handling use dedicated hugetlb_* rmap functions. Note that is_device_private_page() does not apply to hugetlb. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- include/linux/mm.h | 12 +++++++++--- include/linux/rmap.h | 15 +++++++++++++++ mm/hugetlb.c | 3 +-- 3 files changed, 25 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 418d26608ece7..24c1c7c5a99c0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1953,15 +1953,21 @@ static inline bool page_maybe_dma_pinned(struct page *page) * * The caller has to hold the PT lock and the vma->vm_mm->->write_protect_seq. */ -static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma, - struct page *page) +static inline bool folio_needs_cow_for_dma(struct vm_area_struct *vma, + struct folio *folio) { VM_BUG_ON(!(raw_read_seqcount(&vma->vm_mm->write_protect_seq) & 1)); if (!test_bit(MMF_HAS_PINNED, &vma->vm_mm->flags)) return false; - return page_maybe_dma_pinned(page); + return folio_maybe_dma_pinned(folio); +} + +static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma, + struct page *page) +{ + return folio_needs_cow_for_dma(vma, page_folio(page)); } /** diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 0a81e8420a961..8068c332e2ce5 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -208,6 +208,21 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +/* See page_try_dup_anon_rmap() */ +static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, + struct vm_area_struct *vma) +{ + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + + if (PageAnonExclusive(&folio->page)) { + if (unlikely(folio_needs_cow_for_dma(vma, folio))) + return -EBUSY; + ClearPageAnonExclusive(&folio->page); + } + atomic_inc(&folio->_entire_mapcount); + return 0; +} + static inline void hugetlb_add_file_rmap(struct folio *folio) { VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 541a8f38cfdc7..d927f8b2893c0 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5402,8 +5402,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, */ if (!folio_test_anon(pte_folio)) { hugetlb_add_file_rmap(pte_folio); - } else if (page_try_dup_anon_rmap(&pte_folio->page, - true, src_vma)) { + } else if (hugetlb_try_dup_anon_rmap(pte_folio, src_vma)) { pte_t src_pte_old = entry; struct folio *new_folio; From patchwork Mon Dec 4 14:21:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173367 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2794719vqy; Mon, 4 Dec 2023 06:22:18 -0800 (PST) X-Google-Smtp-Source: AGHT+IF3beEVwTzbWCN+HGo6ad76PM2S3gZn/h5ONfy+fOPz94QRTY7kPQrSolXgfm7OZieTsgMg X-Received: by 2002:a05:6a00:174b:b0:6ce:2731:a08b with SMTP id j11-20020a056a00174b00b006ce2731a08bmr4929324pfc.58.1701699738340; Mon, 04 Dec 2023 06:22:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699738; cv=none; d=google.com; s=arc-20160816; b=DJ/+f8v2cwE1wtHDZt9gtSvqnbJvCV83a5mk5H00/VzIgBMvWiLT6vafoqwpcfYdir mq3X4yM+N8HG7Mix8N7+XEunQ+GppGhd0GrY1nmsXoPJogipOpkKMdWOHJHjX9FwoHgI cCL3aRb18JcZtBD4aWZYnEqOqwqdWmZPJ7+VpaRAsxChmnSEAZ4QxyK551Q6Aq9pICKp v9dm/hRRuoBgSV8G6O1kQd/hVN6VYWA1g8Nekkjs6KALjklZRQz1hseclDIWceUg2uWE qaeqde4n4i/6OwcrhhVMb7FG4pOhjfFvMHQXRMDDr47KzKejnq69m4yth5pU9veUlJup c7jA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4nUQXJdzLDxn5x/X9wL3ukpxzJ+76ZeroIzXMEVl8GM=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=fRAvugo5wi4xyzv8kxgKipfB1xiT7ad2LEhLwZLdy1lm3v2xCvZ3ROHaCAEqx+2bOj tlYNWwUNodlZ5Li2LqD2BT9Y2C0QJlUxw6JC6+I1AgYnzbiEXAAL0ThMzlUmqcgPNSRB 1MfTJMbFm/ZjBk8KsCF0+PZ3NaU+cFQHdOdET56gCXqndVCU/UeoC1s9Lu5YZb9NL1Lj CBH2OW0Sjdu9jRshHu+Dl4on+tR4howFqWwFgdFQE1mRCJkFnlaOoNdY7Upqv6rqXtH/ gQ3QEQddNwB8OZisXGbBDCgVAmgyUyeZtPx0XGf0f6RdwF9hByqdwUMdwUOEAHTws2Bz 5Akg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=IuZIPA1A; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id y29-20020a056a00181d00b006cdec3b3b1fsi7663682pfa.235.2023.12.04.06.22.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:22:18 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=IuZIPA1A; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id B0DA38051908; Mon, 4 Dec 2023 06:22:16 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345157AbjLDOWF (ORCPT + 99 others); Mon, 4 Dec 2023 09:22:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52188 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345112AbjLDOV6 (ORCPT ); Mon, 4 Dec 2023 09:21:58 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F7EBDF for ; Mon, 4 Dec 2023 06:22:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699723; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4nUQXJdzLDxn5x/X9wL3ukpxzJ+76ZeroIzXMEVl8GM=; b=IuZIPA1AcRPXNJDTi3f5+yulCD3ekcXjS5F2429w+2aJw4lTKEyAu9fRRPR1DaeHEjOm1c bv1LY16stx4u7+Yc1Y3C/D8PYDaShPgj4LOidJyRwCXI07h+nViAltMnIGH9OgXTSmaBy/ DX0nbVKZmNYpv3QIn3VnPdhuBbg9uUA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-9-ssjzReYIPVSCXays4yclSw-1; Mon, 04 Dec 2023 09:21:58 -0500 X-MC-Unique: ssjzReYIPVSCXays4yclSw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EA9F3101A590; Mon, 4 Dec 2023 14:21:57 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 572442026D4C; Mon, 4 Dec 2023 14:21:56 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 05/39] mm/rmap: introduce and use hugetlb_try_share_anon_rmap() Date: Mon, 4 Dec 2023 15:21:12 +0100 Message-ID: <20231204142146.91437-6-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:22:16 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361504830170849 X-GMAIL-MSGID: 1784361504830170849 hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. So let's introduce and use hugetlb_try_dup_anon_rmap() to make all hugetlb handling use dedicated hugetlb_* rmap functions. Note that try_to_unmap_one() does not need care. Easy to spot because among all that nasty hugetlb special-casing in that function, we're not using set_huge_pte_at() on the anon path -- well, and that code assumes that we we would want to swapout. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- include/linux/rmap.h | 20 ++++++++++++++++++++ mm/rmap.c | 15 ++++++++++----- 2 files changed, 30 insertions(+), 5 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 8068c332e2ce5..3f38141b53b9d 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -223,6 +223,26 @@ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, return 0; } +/* See page_try_share_anon_rmap() */ +static inline int hugetlb_try_share_anon_rmap(struct folio *folio) +{ + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + VM_WARN_ON_FOLIO(!PageAnonExclusive(&folio->page), folio); + + /* See page_try_share_anon_rmap() */ + if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) + smp_mb(); + + if (unlikely(folio_maybe_dma_pinned(folio))) + return -EBUSY; + ClearPageAnonExclusive(&folio->page); + + /* See page_try_share_anon_rmap() */ + if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) + smp_mb__after_atomic(); + return 0; +} + static inline void hugetlb_add_file_rmap(struct folio *folio) { VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); diff --git a/mm/rmap.c b/mm/rmap.c index 5037581b79ec6..2f1af3958e687 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2105,13 +2105,18 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, !anon_exclusive, subpage); /* See page_try_share_anon_rmap(): clear PTE first. */ - if (anon_exclusive && - page_try_share_anon_rmap(subpage)) { - if (folio_test_hugetlb(folio)) + if (folio_test_hugetlb(folio)) { + if (anon_exclusive && + hugetlb_try_share_anon_rmap(folio)) { set_huge_pte_at(mm, address, pvmw.pte, pteval, hsz); - else - set_pte_at(mm, address, pvmw.pte, pteval); + ret = false; + page_vma_mapped_walk_done(&pvmw); + break; + } + } else if (anon_exclusive && + page_try_share_anon_rmap(page)) { + set_pte_at(mm, address, pvmw.pte, pteval); ret = false; page_vma_mapped_walk_done(&pvmw); break; From patchwork Mon Dec 4 14:21:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173399 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2798291vqy; Mon, 4 Dec 2023 06:27:39 -0800 (PST) X-Google-Smtp-Source: AGHT+IEiZcnJl8dI2GYwME6iGhSxY+tRvMoNB6Dq/9x/2QSSZgStngc3OHF2XxpJ63JHhwz1QUyC X-Received: by 2002:a17:90b:3846:b0:286:c138:c2b7 with SMTP id nl6-20020a17090b384600b00286c138c2b7mr713694pjb.49.1701700059247; Mon, 04 Dec 2023 06:27:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701700059; cv=none; d=google.com; s=arc-20160816; b=ig78G2wDH+n0SF79cfoZynYac3feqU6edezG+GpL8lp4VDhbpPL9SOGM/zGErmAokb qtqE0pchgAubwo7DOmQ1JMF94wtF1jEdiF9SDx/CQ6rLBP87as2zkOYCsHSL37aETTQg 9uTwwosE7eJrsaqEhupXnEGEyMdrdhat0jpVjRxtdWLHUgR5H8ivqxf9NkFuNCn9RzDs nRJjx9WXhoQKXl2ucCGrQutl6Wzesb3WBHnbu5/DUfEoe+NOO7c7D73DNWIwndknoJU/ nxwA+vIAfwsHBDyGwc0TsS/+GQVw02gw+utPNq8pEND0nxZ5k2ciq/bNpg4kf8wzYf4K IX9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Py+BBg1WdhAyTKIO2vh7V9agn/F6NhuWPF4QbRDWhlI=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=ths9OL7ng8iihZ1zZiBsPst2l5vyJth3W83xOPsWLXlNlNTd82xGvsTNgmmX0pNifp lJ7Ltqg5/5aakA3bEomYLrHTN6crVeNmEFlxMvMKIAK9SclQ+QtcaUJAeH9ORkfvohTq kbPfqB+S6DFemQ3OFEnghDWCudMby0kWXi0wfeBLgV07XIVkFMRzud4E0hBJ1TmYYBxK UO+/A7WRumtUvRFCtvTxaBh2EF2Brw0tuwXQsrprfrToWmr1LM8IF0H0iI47LeLTfuC7 gLzqKovDm2aSRtztqdmzTDd/MNClck9g3uzUifRXW0HoOGiYRodhypSBlF+8k9CIsFgs ZVvA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=RTyBlxDR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id x12-20020a17090a970c00b00286a72405a3si1977786pjo.84.2023.12.04.06.27.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:27:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=RTyBlxDR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 61910803B349; Mon, 4 Dec 2023 06:27:35 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345157AbjLDO10 (ORCPT + 99 others); Mon, 4 Dec 2023 09:27:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52664 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235870AbjLDO1M (ORCPT ); Mon, 4 Dec 2023 09:27:12 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98BCF421B for ; Mon, 4 Dec 2023 06:24:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699870; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Py+BBg1WdhAyTKIO2vh7V9agn/F6NhuWPF4QbRDWhlI=; b=RTyBlxDRCWlMK4SwNUAeSqgEuKK55AXYyopeywo76uOJVUo6dt68wd3O3b5rDhF5JMDyaW crC92wC/EVAd/qCvlZxw2J9rCLI9tKCEMH9Na2wz7P0/IscxlUIB+sAP2iPmkzxryCRKD9 fpdR7TQGWjO4ZwdVPr3oOe8IS8iL9BM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-110-KmE7Bv2xNWyBuu4ga5J7Uw-1; Mon, 04 Dec 2023 09:24:16 -0500 X-MC-Unique: KmE7Bv2xNWyBuu4ga5J7Uw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CCFA88F9205; Mon, 4 Dec 2023 14:21:59 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 308C02026D68; Mon, 4 Dec 2023 14:21:58 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 06/39] mm/rmap: add hugetlb sanity checks Date: Mon, 4 Dec 2023 15:21:13 +0100 Message-ID: <20231204142146.91437-7-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:27:35 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361841541264834 X-GMAIL-MSGID: 1784361841541264834 Let's make sure we end up with the right folios in the right functions. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- include/linux/rmap.h | 7 +++++++ mm/rmap.c | 6 ++++++ 2 files changed, 13 insertions(+) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 3f38141b53b9d..77e336f86c72d 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -212,6 +212,7 @@ void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, struct vm_area_struct *vma) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); if (PageAnonExclusive(&folio->page)) { @@ -226,6 +227,7 @@ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, /* See page_try_share_anon_rmap() */ static inline int hugetlb_try_share_anon_rmap(struct folio *folio) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); VM_WARN_ON_FOLIO(!PageAnonExclusive(&folio->page), folio); @@ -245,6 +247,7 @@ static inline int hugetlb_try_share_anon_rmap(struct folio *folio) static inline void hugetlb_add_file_rmap(struct folio *folio) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); atomic_inc(&folio->_entire_mapcount); @@ -252,11 +255,15 @@ static inline void hugetlb_add_file_rmap(struct folio *folio) static inline void hugetlb_remove_rmap(struct folio *folio) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); + atomic_dec(&folio->_entire_mapcount); } static inline void __page_dup_rmap(struct page *page, bool compound) { + VM_WARN_ON(folio_test_hugetlb(page_folio(page))); + if (compound) { struct folio *folio = (struct folio *)page; diff --git a/mm/rmap.c b/mm/rmap.c index 2f1af3958e687..a735ecca47a81 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1313,6 +1313,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, { int nr; + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma); __folio_set_swapbacked(folio); @@ -1353,6 +1354,7 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, unsigned int nr_pmdmapped = 0, first; int nr = 0; + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(compound && !folio_test_pmd_mappable(folio), folio); /* Is page being mapped by PTE? Is this its first map to be added? */ @@ -1438,6 +1440,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, bool last; enum node_stat_item idx; + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); VM_BUG_ON_PAGE(compound && !PageHead(page), page); /* Is page being unmapped by PTE? Is this its last map to be removed? */ @@ -2590,6 +2593,7 @@ void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc) void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address, rmap_t flags) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); atomic_inc(&folio->_entire_mapcount); @@ -2602,6 +2606,8 @@ void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, void hugetlb_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); + BUG_ON(address < vma->vm_start || address >= vma->vm_end); /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); From patchwork Mon Dec 4 14:21:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173369 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2795054vqy; Mon, 4 Dec 2023 06:22:46 -0800 (PST) X-Google-Smtp-Source: AGHT+IGUwZGXIm3/B6r2CdBlDC1D+ecJfLkn16H4j3eOcwxx7gvtf7NxDgr5dntv1zhM5QaPselx X-Received: by 2002:a05:6a20:5653:b0:18f:97c:928f with SMTP id is19-20020a056a20565300b0018f097c928fmr1316899pzc.116.1701699766356; Mon, 04 Dec 2023 06:22:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699766; cv=none; d=google.com; s=arc-20160816; b=0+zUNZNQTS0nGoKJhCNjUQKWk321BYJ/S9WRqrQ8KRirUSnMu3gwVUNJYZJ81+h8bk tiBbOci7Wj3wZvhOehqo4CFbJDwXpDZt/QH/sEPQr39Sqz9KPO1KnLc38IDh6+e4yVNF 1LJhFSxc3fruJROo3Y7UEr+qV8FkXVfbnvLxK+p0IeNG6Kuei1YgpxnnwOCi7mCMFPTL 37jOWoZyExk9zdAmkJqjvW5SEjAdYM8blldUob+ddKpjsY/x9AEWR0NHCzVVpUylx4J5 UjqJPnry9cUtf6CoI94PmdPXNVU3eosADWoSRAsw7VloXWvrwiEB42ytO3qpr4DiT6kx eyVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Y8nY91srGGOwHNmHB2IEECEkScJep9NeXgCbzBUOTxo=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=YRFUZ87xzxsRCRqbHCFJIny5Pf01Vr94dymnd+Bo0Agn7YEdZ/yyb7DKCczv+6F9HG +6/icad1tYge/tPirL7EnH6OwkabeqWvhCorzt4HtniV33XwOaaA45FJG0OKApD5dXh8 PE722QifZROy7my5vp4zSRlgmRj0g3BtqykA6hb5jydOxXAcfOD38GOeoPEY3qKgNxYQ hEwN/nMF+6VBRxBx02LQAr3JCjQxmiwcO4iWLXx+PZRvqRXJ8jOif9GktLy2qc8ja4+r 1MIUB6FM176fCR25A5VINkLsyEAoRowlvGq3ehosEPqTjG/gLntBEq2VpxdrwolPVAYK hpsg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=BmVJAIfP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id q10-20020a056a00084a00b006ce5ba4300esi914216pfk.219.2023.12.04.06.22.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:22:46 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=BmVJAIfP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 679D080564D9; Mon, 4 Dec 2023 06:22:39 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345085AbjLDOWJ (ORCPT + 99 others); Mon, 4 Dec 2023 09:22:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52250 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235836AbjLDOWB (ORCPT ); Mon, 4 Dec 2023 09:22:01 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3E45011A for ; Mon, 4 Dec 2023 06:22:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699725; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y8nY91srGGOwHNmHB2IEECEkScJep9NeXgCbzBUOTxo=; b=BmVJAIfPcw8rV5BGoidQnofxE4E/A0/J/jJhmDh52n2XVTFrMcDXqQMaUiz+gzvkj3P51+ Ft+xOoDjefgMiIbOBsB4lC1IYY3fC0DcHmrjzAgQ2+4sonKBJ7z/y9dTILCBd38EQNlOSE DHFD2dZhkaEo8VS2dGUEVo7LMumlQpw= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-201-g2weQkhwMp-9jFxobmepBA-1; Mon, 04 Dec 2023 09:22:02 -0500 X-MC-Unique: g2weQkhwMp-9jFxobmepBA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AE9EB1C0514B; Mon, 4 Dec 2023 14:22:01 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 118622026D68; Mon, 4 Dec 2023 14:21:59 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 07/39] mm/rmap: convert folio_add_file_rmap_range() into folio_add_file_rmap_[pte|ptes|pmd]() Date: Mon, 4 Dec 2023 15:21:14 +0100 Message-ID: <20231204142146.91437-8-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:22:39 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361534418165901 X-GMAIL-MSGID: 1784361534418165901 Let's get rid of the compound parameter and instead define implicitly which mappings we're adding. That is more future proof, easier to read and harder to mess up. Use an enum to express the granularity internally. Make the compiler always special-case on the granularity by using __always_inline. Add plenty of sanity checks with CONFIG_DEBUG_VM. Replace the folio_test_pmd_mappable() check by a config check in the caller and sanity checks. Convert the single user of folio_add_file_rmap_range(). This function design can later easily be extended to PUDs and to batch PMDs. Note that for now we don't support anything bigger than PMD-sized folios (as we cleanly separated hugetlb handling). Sanity checks will catch if that ever changes. Next up is removing page_remove_rmap() along with its "compound" parameter and smilarly converting all other rmap functions. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 47 +++++++++++++++++++++++++++-- mm/memory.c | 2 +- mm/rmap.c | 72 ++++++++++++++++++++++++++++---------------- 3 files changed, 92 insertions(+), 29 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 77e336f86c72d..a4a30c361ac50 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -186,6 +186,45 @@ typedef int __bitwise rmap_t; */ #define RMAP_COMPOUND ((__force rmap_t)BIT(1)) +/* + * Internally, we're using an enum to specify the granularity. Usually, + * we make the compiler create specialized variants for the different + * granularity. + */ +enum rmap_mode { + RMAP_MODE_PTE = 0, + RMAP_MODE_PMD, +}; + +static inline void __folio_rmap_sanity_checks(struct folio *folio, + struct page *page, unsigned int nr_pages, enum rmap_mode mode) +{ + /* hugetlb folios are handled separately. */ + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); + VM_WARN_ON_FOLIO(folio_test_large(folio) && + !folio_test_large_rmappable(folio), folio); + + VM_WARN_ON_ONCE(!nr_pages || nr_pages > folio_nr_pages(folio)); + VM_WARN_ON_FOLIO(page_folio(page) != folio, folio); + VM_WARN_ON_FOLIO(page_folio(page + nr_pages - 1) != folio, folio); + + switch (mode) { + case RMAP_MODE_PTE: + break; + case RMAP_MODE_PMD: + /* + * We don't support folios larger than a single PMD yet. So + * when RMAP_MODE_PMD is set, we assume that we are creating + * a single "entire" mapping of the folio. + */ + VM_WARN_ON_FOLIO(folio_nr_pages(folio) != HPAGE_PMD_NR, folio); + VM_WARN_ON_FOLIO(nr_pages != HPAGE_PMD_NR, folio); + break; + default: + VM_WARN_ON_ONCE(true); + } +} + /* * rmap interfaces called when adding or removing pte of page */ @@ -198,8 +237,12 @@ void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); -void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr, - struct vm_area_struct *, bool compound); +void folio_add_file_rmap_ptes(struct folio *, struct page *, unsigned int nr, + struct vm_area_struct *); +#define folio_add_file_rmap_pte(folio, page, vma) \ + folio_add_file_rmap_ptes(folio, page, 1, vma) +void folio_add_file_rmap_pmd(struct folio *, struct page *, + struct vm_area_struct *); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); diff --git a/mm/memory.c b/mm/memory.c index 1f18ed4a54971..15325587cff01 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4414,7 +4414,7 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio, folio_add_lru_vma(folio, vma); } else { add_mm_counter(vma->vm_mm, mm_counter_file(page), nr); - folio_add_file_rmap_range(folio, page, nr, vma, false); + folio_add_file_rmap_ptes(folio, page, nr, vma); } set_ptes(vma->vm_mm, addr, vmf->pte, entry, nr); diff --git a/mm/rmap.c b/mm/rmap.c index a735ecca47a81..1614d98062948 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1334,31 +1334,19 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, SetPageAnonExclusive(&folio->page); } -/** - * folio_add_file_rmap_range - add pte mapping to page range of a folio - * @folio: The folio to add the mapping to - * @page: The first page to add - * @nr_pages: The number of pages which will be mapped - * @vma: the vm area in which the mapping is added - * @compound: charge the page as compound or small page - * - * The page range of folio is defined by [first_page, first_page + nr_pages) - * - * The caller needs to hold the pte lock. - */ -void folio_add_file_rmap_range(struct folio *folio, struct page *page, - unsigned int nr_pages, struct vm_area_struct *vma, - bool compound) +static __always_inline void __folio_add_file_rmap(struct folio *folio, + struct page *page, unsigned int nr_pages, + struct vm_area_struct *vma, enum rmap_mode mode) { atomic_t *mapped = &folio->_nr_pages_mapped; unsigned int nr_pmdmapped = 0, first; int nr = 0; - VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); - VM_WARN_ON_FOLIO(compound && !folio_test_pmd_mappable(folio), folio); + VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); /* Is page being mapped by PTE? Is this its first map to be added? */ - if (likely(!compound)) { + if (likely(mode == RMAP_MODE_PTE)) { do { first = atomic_inc_and_test(&page->_mapcount); if (first && folio_test_large(folio)) { @@ -1369,9 +1357,7 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, if (first) nr++; } while (page++, --nr_pages > 0); - } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */ - + } else if (mode == RMAP_MODE_PMD) { first = atomic_inc_and_test(&folio->_entire_mapcount); if (first) { nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); @@ -1399,6 +1385,43 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, mlock_vma_folio(folio, vma); } +/** + * folio_add_file_rmap_ptes - add PTE mappings to a page range of a folio + * @folio: The folio to add the mappings to + * @page: The first page to add + * @nr_pages: The number of pages that will be mapped using PTEs + * @vma: The vm area in which the mappings are added + * + * The page range of the folio is defined by [page, page + nr_pages) + * + * The caller needs to hold the page table lock. + */ +void folio_add_file_rmap_ptes(struct folio *folio, struct page *page, + unsigned int nr_pages, struct vm_area_struct *vma) +{ + __folio_add_file_rmap(folio, page, nr_pages, vma, RMAP_MODE_PTE); +} + +/** + * folio_add_file_rmap_pmd - add a PMD mapping to a page range of a folio + * @folio: The folio to add the mapping to + * @page: The first page to add + * @vma: The vm area in which the mapping is added + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock. + */ +void folio_add_file_rmap_pmd(struct folio *folio, struct page *page, + struct vm_area_struct *vma) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_add_file_rmap(folio, page, HPAGE_PMD_NR, vma, RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + /** * page_add_file_rmap - add pte mapping to a file page * @page: the page to add the mapping to @@ -1411,16 +1434,13 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, bool compound) { struct folio *folio = page_folio(page); - unsigned int nr_pages; VM_WARN_ON_ONCE_PAGE(compound && !PageTransHuge(page), page); if (likely(!compound)) - nr_pages = 1; + folio_add_file_rmap_pte(folio, page, vma); else - nr_pages = folio_nr_pages(folio); - - folio_add_file_rmap_range(folio, page, nr_pages, vma, compound); + folio_add_file_rmap_pmd(folio, page, vma); } /** From patchwork Mon Dec 4 14:21:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173373 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2795136vqy; Mon, 4 Dec 2023 06:22:55 -0800 (PST) X-Google-Smtp-Source: AGHT+IFdTydWT24d+9qFlTlE6BJ7485bQLOZobCGXuCZOMXk51PPo+D16YPLvMCWJTruiICKEa+Z X-Received: by 2002:a05:6a20:e103:b0:187:7678:f354 with SMTP id kr3-20020a056a20e10300b001877678f354mr2536314pzb.6.1701699774691; Mon, 04 Dec 2023 06:22:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699774; cv=none; d=google.com; s=arc-20160816; b=gfLfB32tTMuzonV2asasezoQoBHM14tvpfSvySgN7LYMFEBHCjJjYP32zpq/fx9Xwz ddH40tKguRQJgRoKzp6KKxvnsUewmORApyq/zI0H+fo/ch883GOzwfrh8J3AQA6+I5XS OB3U2E9IPB0uJdVp+WVuY5GndvtAX9hKFHg3Avi5A71IGsvL4mihcc9GCNItz+vjtAW5 5x8GHVqN9AOfbYI480q5dA33isOk71VOqN6o9WLg5yYLsKKuR68cGkJAliKlsQZ+SyPn GEm6pyHZyGgzOfvtf77uDuxi6l54IfsGSAt/vxzW8yoveiDcsVPnaShCoaY6LouB+V+C u1Xw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=rWanwqicBfazF+1iBEpyOQZ25LJEujBQqHyMDlWvhqA=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=HtbDCTZ3ink8AfOAla0BLs5fta54dLECaWjuVp7YzEa0AQ5AVeKqu5xLBAw5zoSiSb nxQgvN3L1l0h6cdUNId+IlGF0pj+tgoFpc6/lRE3lczSxcvT46XOjLL0EvqFQSe5rqmb zyU6KQQHVMSD9/EP4Ws1BNjEbPj7or1FbZT2W+hiobKjratMny87vlAutGLvtEc86RDG qWaCogeuxkyWKt35F+s4UgJz9r8RMoKmy1FGnsI2je+7A1jsM+t5tyM+3+bFmSpGyc0R Kxy1lzk04xaQpPRB1xNF1iBENPS9m8DMr9u6fmuONgomzFMo0auaxHpBtP8PNcLOMO2m 8P2A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=LAlUzxoT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id u24-20020a631418000000b00577e00c5ffasi2089805pgl.867.2023.12.04.06.22.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:22:54 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=LAlUzxoT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id BD492805AA14; Mon, 4 Dec 2023 06:22:46 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235852AbjLDOWW (ORCPT + 99 others); Mon, 4 Dec 2023 09:22:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52274 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235831AbjLDOWI (ORCPT ); Mon, 4 Dec 2023 09:22:08 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0E5B91A1 for ; Mon, 4 Dec 2023 06:22:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699728; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rWanwqicBfazF+1iBEpyOQZ25LJEujBQqHyMDlWvhqA=; b=LAlUzxoTyYLO0ukIM9henASMEmCRTRSh4aGJPKaCytwf4HCDnHzrRhaJXTryPDhxp3+QVN CJoyKXp8xctxkOMMePl2kFlSCCcxNMug8X7APeLBZPJcKg63S1G4ETcgewRLEwuIoFpP07 jF89SvjseSB7Ksw4MsjUQ4mliVFAx8s= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-31-zbIm8uHQNrW7J5QBtxdQDA-1; Mon, 04 Dec 2023 09:22:04 -0500 X-MC-Unique: zbIm8uHQNrW7J5QBtxdQDA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8C48C2825E93; Mon, 4 Dec 2023 14:22:03 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id EF5FB2026D68; Mon, 4 Dec 2023 14:22:01 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 08/39] mm/memory: page_add_file_rmap() -> folio_add_file_rmap_[pte|pmd]() Date: Mon, 4 Dec 2023 15:21:15 +0100 Message-ID: <20231204142146.91437-9-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:22:47 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361543146083020 X-GMAIL-MSGID: 1784361543146083020 Let's convert insert_page_into_pte_locked() and do_set_pmd(). While at it, perform some folio conversion. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- mm/memory.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 15325587cff01..be7fe58f7c297 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1845,12 +1845,14 @@ static int validate_page_before_insert(struct page *page) static int insert_page_into_pte_locked(struct vm_area_struct *vma, pte_t *pte, unsigned long addr, struct page *page, pgprot_t prot) { + struct folio *folio = page_folio(page); + if (!pte_none(ptep_get(pte))) return -EBUSY; /* Ok, finally just insert the thing.. */ - get_page(page); + folio_get(folio); inc_mm_counter(vma->vm_mm, mm_counter_file(page)); - page_add_file_rmap(page, vma, false); + folio_add_file_rmap_pte(folio, page, vma); set_pte_at(vma->vm_mm, addr, pte, mk_pte(page, prot)); return 0; } @@ -4308,6 +4310,7 @@ static void deposit_prealloc_pte(struct vm_fault *vmf) vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) { + struct folio *folio = page_folio(page); struct vm_area_struct *vma = vmf->vma; bool write = vmf->flags & FAULT_FLAG_WRITE; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; @@ -4317,8 +4320,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) if (!transhuge_vma_suitable(vma, haddr)) return ret; - page = compound_head(page); - if (compound_order(page) != HPAGE_PMD_ORDER) + if (page != &folio->page || folio_order(folio) != HPAGE_PMD_ORDER) return ret; /* @@ -4327,7 +4329,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) * check. This kind of THP just can be PTE mapped. Access to * the corrupted subpage should trigger SIGBUS as expected. */ - if (unlikely(PageHasHWPoisoned(page))) + if (unlikely(folio_test_has_hwpoisoned(folio))) return ret; /* @@ -4351,7 +4353,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); add_mm_counter(vma->vm_mm, mm_counter_file(page), HPAGE_PMD_NR); - page_add_file_rmap(page, vma, true); + folio_add_file_rmap_pmd(folio, page, vma); /* * deposit and withdraw with pmd lock held From patchwork Mon Dec 4 14:21:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173372 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2795119vqy; Mon, 4 Dec 2023 06:22:53 -0800 (PST) X-Google-Smtp-Source: AGHT+IGBHEp/N8qc6g5LFToraGVkdPHdlOmiKtkMz9n+Tmww33GG9TX117MgRlQrTmCohqUIgLAP X-Received: by 2002:a17:903:244d:b0:1d0:53b1:400 with SMTP id l13-20020a170903244d00b001d053b10400mr3801531pls.68.1701699773289; Mon, 04 Dec 2023 06:22:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699773; cv=none; d=google.com; s=arc-20160816; b=QArFOyD5HWdWR17331QDXCmWXxALf6+YoZISlCodaA1SkkbuXaveo6kuQxY1qqCbQ7 N4tSRaezv5VvhQL2S6ojLTROF4IudfQZzcwvpv38vTbgRGT4fjLXAlLGGMNB7iuYEuO7 DqW6nMNUQeMcAyaLCK1v34XTU5B7No9RXFG6jhL5LvaKL138W6c11RZATu0Vym/d/Ddc pgyWEcZgXT1AkDSsr1WiYR7TWfn0B5w7cFwL+Qv1fhUQ1Leva+RKBBllbpNy7fIpPhSk qei5EpWFxQjlAiPwZENOn4tgwlO3a7iHlGmipby3BMgzjc9yiuv/aZEq2QdmbVBztSOr AGxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=HFIwQ5rYAD0bMXQeuRR1LkqvEC8kZB0MG/s3u0E9e4Q=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=Yf/2RPQZf7N/r0nvhqMOfcuCJHs/js74eVHBNfKDg86cPD+73I2erE39LNyBS0v1LY iy6ZDxsLzH2u8X+GOhEVjixdU62SUPQQ7uMyT2Ik+I0xrMpf0YkNCncs65hRqnKSKdyA KCcAC49DuXevuGrUsE1fVDYjrQ7f1k4/j9T97mPJJiNphZiTaVVMsey/fNmtSzGkTJJT 3cXvg3NJvqMC+mbrKJoGPo4yMaYDugQfL33EckvTKUiPAGJ/6uHtk67fivIbF1grCxZs 32powYBqWs9Q6auiZz4RsxcTkOCa4oMNhNftXW2zT7/8g6qRZDEinl36k6sKOQz0ZWXN zKwQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ic4XyUL4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id iz22-20020a170902ef9600b001d05aa54912si5274091plb.504.2023.12.04.06.22.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:22:53 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ic4XyUL4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 2829980A9AB0; Mon, 4 Dec 2023 06:22:45 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235831AbjLDOW3 (ORCPT + 99 others); Mon, 4 Dec 2023 09:22:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52128 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345317AbjLDOWS (ORCPT ); Mon, 4 Dec 2023 09:22:18 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3AE02D48 for ; Mon, 4 Dec 2023 06:22:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699732; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HFIwQ5rYAD0bMXQeuRR1LkqvEC8kZB0MG/s3u0E9e4Q=; b=ic4XyUL4nnIHGu/JojvjbdG2KtBqobw8fF6hKPlY90nlCyW/ajgGxx0mOCHzqhldxFpIH4 mZbc0SSqZ4P5G2TbanbkkiiJNvxxuQ7Qb8+RLP6JSOpVCXjQr4Api43w2nYspERjHePFvR UX7J52rM1+M4niesNO1FOPOynH/9SrM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-253-KmX5I8pcNJiCMDlB3yQUQg-1; Mon, 04 Dec 2023 09:22:08 -0500 X-MC-Unique: KmX5I8pcNJiCMDlB3yQUQg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 524F58F9207; Mon, 4 Dec 2023 14:22:07 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id AE2382026D4C; Mon, 4 Dec 2023 14:22:05 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 10/39] mm/migrate: page_add_file_rmap() -> folio_add_file_rmap_pte() Date: Mon, 4 Dec 2023 15:21:17 +0100 Message-ID: <20231204142146.91437-11-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:22:45 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361541206273340 X-GMAIL-MSGID: 1784361541206273340 Let's convert remove_migration_pte(). Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- mm/migrate.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/migrate.c b/mm/migrate.c index de9d94b99ab78..efc19f53b05e6 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -262,7 +262,7 @@ static bool remove_migration_pte(struct folio *folio, page_add_anon_rmap(new, vma, pvmw.address, rmap_flags); else - page_add_file_rmap(new, vma, false); + folio_add_file_rmap_pte(folio, new, vma); set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); } if (vma->vm_flags & VM_LOCKED) From patchwork Mon Dec 4 14:21:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173371 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2795095vqy; Mon, 4 Dec 2023 06:22:51 -0800 (PST) X-Google-Smtp-Source: AGHT+IGNIxkzmmbwXTdjfMG87+KkXiF0M1T0uZNqaQBqMdBG0VM82V9lQ1PNJJnkxgLBu7MoBmNS X-Received: by 2002:a17:902:ce89:b0:1d0:a663:20fe with SMTP id f9-20020a170902ce8900b001d0a66320femr1130231plg.80.1701699770948; Mon, 04 Dec 2023 06:22:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699770; cv=none; d=google.com; s=arc-20160816; b=TWzf2J4dDIYaieqDltCa1FdmKwNs+xmCiERfuWZ+ktj7lNcloVG1/upVWuI1rYc9mK mNtbcqz6MlTOxZocpZGb522slSmGdNE0XDbvIy/faT4Tp4vpsIy++V6Un+1lJY/n1hZW s7mVJV7MYsWtByulNODox+rTN5fAesLvnfAwXzjYSq7MIG465hTajfye/OnzyBtoYEkO uTHxBTG26Wmf2gfNf0P+9FHOGxMq1Zp3mLg6OIbOki9uypBD150ftY60RfE0fFK7kn3w MFMY7+lSzcqInST97kutY84fMsKA2CerPalIAzrwZlQs/HpENP1rQd9t900jHeiXAbsV 0ZvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=DcozSwix+76TtkWexeYoWqf1En6dAEapeBtTC2bk/5E=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=L5jHF54uNjQu9dJpQeFSCtEUp6xQ6+MtJQEKo8JJNhPPagRAme3QERNaAx9ygOBrLb uPB00pogXddn5SVrB1OsG+R7ohLBKDtp4GKoFTtX78DCn+1n0FwgvXN2mFhX2PyCh2l5 j0C2f+k5IukubfnMaqXtDwwpC52tCx9aYxetrd6uZR+gy4sVSSVpI6B7Ip1gTecPprsn 3RQaDNduRb9zwVCCklllVrtpQjV9ZSnjj+xUGN5CkbFgd48eVgwyYmSd5NQpvxGol3N+ Adh3LoFHpocR9R5YF5bs3Xew2d+NYIbBFWAqZ3wZsVGhl0zcCULUswLt/vwS6ptOUMbl tiYQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Ik7hIMgf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id x9-20020a170902820900b001d05235181esi4616028pln.275.2023.12.04.06.22.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:22:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Ik7hIMgf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 7C6D0803B3AF; Mon, 4 Dec 2023 06:22:48 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345226AbjLDOWf (ORCPT + 99 others); Mon, 4 Dec 2023 09:22:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345142AbjLDOWT (ORCPT ); Mon, 4 Dec 2023 09:22:19 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 630ADD50 for ; Mon, 4 Dec 2023 06:22:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699733; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DcozSwix+76TtkWexeYoWqf1En6dAEapeBtTC2bk/5E=; b=Ik7hIMgfP2ulxvd0KMg8QH8/0DOwV62IgmkUV2kR2mPnQ/lO+OnYPxxgM/5wfKsrWacWzI AKSBnE3owc78rHYAgPQ/iQtblRFSgXmwYjZuAemD9tIoSwEoBkPZbm5ciX1eAy124w4jcA agMJ280xg8/Xwjvgw3BH4KmWCnnwgcE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-261--TlKEsHfM9OY2J5oZszMwg-1; Mon, 04 Dec 2023 09:22:09 -0500 X-MC-Unique: -TlKEsHfM9OY2J5oZszMwg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 290B08F9206; Mon, 4 Dec 2023 14:22:09 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8BF132026D68; Mon, 4 Dec 2023 14:22:07 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 11/39] mm/userfaultfd: page_add_file_rmap() -> folio_add_file_rmap_pte() Date: Mon, 4 Dec 2023 15:21:18 +0100 Message-ID: <20231204142146.91437-12-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:22:48 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361539281990041 X-GMAIL-MSGID: 1784361539281990041 Let's convert mfill_atomic_install_pte(). Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- mm/userfaultfd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 0b6ca553bebec..abf4c579d328a 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -114,7 +114,7 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd, /* Usually, cache pages are already added to LRU */ if (newly_allocated) folio_add_lru(folio); - page_add_file_rmap(page, dst_vma, false); + folio_add_file_rmap_pte(folio, page, dst_vma); } else { page_add_new_anon_rmap(page, dst_vma, dst_addr); folio_add_lru_vma(folio, dst_vma); From patchwork Mon Dec 4 14:21:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173370 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2795084vqy; Mon, 4 Dec 2023 06:22:50 -0800 (PST) X-Google-Smtp-Source: AGHT+IGc8TWQBjVhDIbh1oaaYV4XMBxNdjjkcGjFYfrokBtBx55A2HH9gkFo11rnPWeyQ4jG0rM/ X-Received: by 2002:a92:cc50:0:b0:35d:5995:7991 with SMTP id t16-20020a92cc50000000b0035d59957991mr5494930ilq.43.1701699769785; Mon, 04 Dec 2023 06:22:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699769; cv=none; d=google.com; s=arc-20160816; b=kKOgLwQsctpGvu6NdBY38Aq1hFzuv86/s0eiH+4t3rKAtRY1LMkJeYlMWdBy4+jYqc gGH2QyD+RGTKCLEmi3+b3/nQfI+or9tvA/EFXwgR5GW0WrG633XBc33rFDyUFUka7CdA fdvqpYoFMtL/0pjuMGZ0egVm/91TWskDY3IOUCeimwmUsWqBvLoTbv3lUejb+cboYIzS tgtDKJHjt1tzzKfVIi1Q2uT6ssNkN64u7jGdayoX8jGqut+XD9FbBp2q0PSIxfwwz+jP vxFWFUrcUCgtfXU3/P2HW3avLwqPX3S9UoultDyRXPRCbqXQ/MF0zLSpPgXIlpfr803J /nbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=O9+7fynWu7ZwbI+2ouLA8wkNah+BDZS2kzyGsyNToAE=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=kP478TLyeChIo4+XzG5xMtoaNezkDf2fDT53ePEjBxDB9OJq7MuAS6gH84NidLZOHl QngS/r/ntKi+qtfBnQgxsnNP8Z9POlBS2Viiq4uHez/Q8fs6aOlen37YrohOuSXsa7jc jLhlMBfeBXcKFq6NaoUuWmBvYjLv1SISLPr8MwlT7ElC93KF1V0+RgMr3mSsXE0fWd8T Yg2q4VN6Pat4ZL9agGzfAAovLW2JaVi31Qtr70mqUn9e3Y1fRcAeePWo8vCfr0vKtg3I 6SH004dc6ZUynKFdnoBn9nlIWYalOfMbvqpY3Nrl/qSpht2POR0GYW70dOl3IHnGhCGu 1h3Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="fNC1J8D/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id e14-20020a65688e000000b005c67a42a56dsi2708344pgt.741.2023.12.04.06.22.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:22:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="fNC1J8D/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id C29308051913; Mon, 4 Dec 2023 06:22:48 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235506AbjLDOWj (ORCPT + 99 others); Mon, 4 Dec 2023 09:22:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59614 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345241AbjLDOWU (ORCPT ); Mon, 4 Dec 2023 09:22:20 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 830F611A for ; Mon, 4 Dec 2023 06:22:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699734; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O9+7fynWu7ZwbI+2ouLA8wkNah+BDZS2kzyGsyNToAE=; b=fNC1J8D/0iuCGNB1IqlYZ6TXf75nLxeC1QVgjBvLFcW+cTsbaBkGvKXsPMsUsa3sVW4l77 UpT65iKhAkfTyuyjAJYNMSxan4ZqAjRKMi4MHqFJhrnb4xd0RbL105RP4VR8+kQQedJLB/ xYpq3CAw0kCkB0NaBCXU6/z/0PC7TqE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-518-ekuHhhc4MEqvFS8uO4ctFw-1; Mon, 04 Dec 2023 09:22:11 -0500 X-MC-Unique: ekuHhhc4MEqvFS8uO4ctFw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0E725811E7E; Mon, 4 Dec 2023 14:22:11 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 616822026D68; Mon, 4 Dec 2023 14:22:09 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 12/39] mm/rmap: remove page_add_file_rmap() Date: Mon, 4 Dec 2023 15:21:19 +0100 Message-ID: <20231204142146.91437-13-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:22:48 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361537588045449 X-GMAIL-MSGID: 1784361537588045449 All users are gone, let's remove it. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- include/linux/rmap.h | 2 -- mm/rmap.c | 21 --------------------- 2 files changed, 23 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index a4a30c361ac50..95f7b94a70295 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -235,8 +235,6 @@ void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address); void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); -void page_add_file_rmap(struct page *, struct vm_area_struct *, - bool compound); void folio_add_file_rmap_ptes(struct folio *, struct page *, unsigned int nr, struct vm_area_struct *); #define folio_add_file_rmap_pte(folio, page, vma) \ diff --git a/mm/rmap.c b/mm/rmap.c index 1614d98062948..53e2c653be99a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1422,27 +1422,6 @@ void folio_add_file_rmap_pmd(struct folio *folio, struct page *page, #endif } -/** - * page_add_file_rmap - add pte mapping to a file page - * @page: the page to add the mapping to - * @vma: the vm area in which the mapping is added - * @compound: charge the page as compound or small page - * - * The caller needs to hold the pte lock. - */ -void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, - bool compound) -{ - struct folio *folio = page_folio(page); - - VM_WARN_ON_ONCE_PAGE(compound && !PageTransHuge(page), page); - - if (likely(!compound)) - folio_add_file_rmap_pte(folio, page, vma); - else - folio_add_file_rmap_pmd(folio, page, vma); -} - /** * page_remove_rmap - take down pte mapping from a page * @page: page to remove mapping from From patchwork Mon Dec 4 14:21:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173376 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2795595vqy; Mon, 4 Dec 2023 06:23:38 -0800 (PST) X-Google-Smtp-Source: AGHT+IG7AiS8/6V2A14OYQZAqPDTmCfmq54CrGMbOi+CQD6PNXpwt5MfyJwRxq54U54lx4sNNZAQ X-Received: by 2002:a17:90b:5286:b0:286:6cc1:5fd9 with SMTP id si6-20020a17090b528600b002866cc15fd9mr1061642pjb.92.1701699817747; Mon, 04 Dec 2023 06:23:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699817; cv=none; d=google.com; s=arc-20160816; b=UKMYYxS4waxoD33fCWsuG8Gw3GL1H+zrmETr6UisX90hoDZaQ74iG02hg0jLj/y5Pk NRzlyGB+kIwl5q9geaVKL6EU8OUhlRZ6n7ICeMKORA1f1bijG16aVOMaelK+4oT51QIE h0DoDPx+YFzdDieuEZqh2VaVJ3+r/QDcBjI3og3cGzqTCn1YIJ5mnug0bVrb/pL+2hNo jpDoBSs0tT2W5eu7+gyCxcff0z33EMolcDBV9dPVYo7eSnglvisGp52Bc1BdvJCbZp/F 0J8CnXqmBC8thZgcqSIIrWG4YS9txGpAfnvYaQLMfE8uZIzhgO03MKwsUjt9RySmtHHP 5wMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=HK+wCLAanK8icsaUODtbkemdg+UtHwl4vMlZjDb9LtY=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=uMpu+QALvDH+XmjVbIw6zU6OloInWehiLDkCTzBL/PAEVa0EcpJfEo2m8P8r4b5RIz 2ZpTu49SJJMuCC9xsC5P3VURkCjk2LJObc5d9b1HQCjeKhXMC3oyAqdsaUNfpNI/t4cV tQXAOZTm6OTr2GFkgC+UbvWSrKcKmS4BYeZKUvcGLyunV03vwAU1U49mz0LIl1yV2MmB wBqftZgR2ROkdDE5qqulbrQigQ5VGkzfe6K6R8E0FooBOv3FgiHF5Tr6KI6Ob3Pup+Oa V+rieO3+EDl2yiUa7pjwXS65Y59FDtOwpuz6/NErmHOAcAYPRRgItays5FO7TBXex+3A xZxA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=LbDNQUwd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id g3-20020a17090ac30300b00286a4ce6474si2098058pjt.179.2023.12.04.06.23.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:23:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=LbDNQUwd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 706B98054C00; Mon, 4 Dec 2023 06:23:34 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345192AbjLDOW7 (ORCPT + 99 others); Mon, 4 Dec 2023 09:22:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235862AbjLDOWZ (ORCPT ); Mon, 4 Dec 2023 09:22:25 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DCF871721 for ; Mon, 4 Dec 2023 06:22:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699738; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HK+wCLAanK8icsaUODtbkemdg+UtHwl4vMlZjDb9LtY=; b=LbDNQUwdJBEAHZMfx+lR6r5w8iMrULA6uwC/6tX35g+cBE5972WwWne56a4OG9h1cY96L/ NhntiOHu29TjxTP7KkdcJzgHgeT7oYFFsPxfiNOMK+D9Ut16+I2BqJkEv3VDU8k6dQj7T5 wk0kLdB+n+gpxoS4uXJvSC1vHiGJUSM= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-441-tU-ty63bN-Gpvd547o2MWA-1; Mon, 04 Dec 2023 09:22:13 -0500 X-MC-Unique: tU-ty63bN-Gpvd547o2MWA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E0D0129AA3AE; Mon, 4 Dec 2023 14:22:12 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 477C72026D6E; Mon, 4 Dec 2023 14:22:11 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 13/39] mm/rmap: factor out adding folio mappings into __folio_add_rmap() Date: Mon, 4 Dec 2023 15:21:20 +0100 Message-ID: <20231204142146.91437-14-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:23:34 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361588238954836 X-GMAIL-MSGID: 1784361588238954836 Let's factor it out to prepare for reuse as we convert page_add_anon_rmap() to folio_add_anon_rmap_[pte|ptes|pmd](). Make the compiler always special-case on the granularity by using __always_inline. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- mm/rmap.c | 75 +++++++++++++++++++++++++++++++------------------------ 1 file changed, 42 insertions(+), 33 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 53e2c653be99a..c09b360402599 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1127,6 +1127,46 @@ int folio_total_mapcount(struct folio *folio) return mapcount; } +static __always_inline unsigned int __folio_add_rmap(struct folio *folio, + struct page *page, unsigned int nr_pages, enum rmap_mode mode, + int *nr_pmdmapped) +{ + atomic_t *mapped = &folio->_nr_pages_mapped; + int first, nr = 0; + + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); + + /* Is page being mapped by PTE? Is this its first map to be added? */ + if (likely(mode == RMAP_MODE_PTE)) { + do { + first = atomic_inc_and_test(&page->_mapcount); + if (first && folio_test_large(folio)) { + first = atomic_inc_return_relaxed(mapped); + first = (first < COMPOUND_MAPPED); + } + + if (first) + nr++; + } while (page++, --nr_pages > 0); + } else if (mode == RMAP_MODE_PMD) { + first = atomic_inc_and_test(&folio->_entire_mapcount); + if (first) { + nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); + if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { + *nr_pmdmapped = folio_nr_pages(folio); + nr = *nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); + /* Raced ahead of a remove and another add? */ + if (unlikely(nr < 0)) + nr = 0; + } else { + /* Raced ahead of a remove of COMPOUND_MAPPED */ + nr = 0; + } + } + } + return nr; +} + /** * folio_move_anon_rmap - move a folio to our anon_vma * @folio: The folio to move to our anon_vma @@ -1338,42 +1378,11 @@ static __always_inline void __folio_add_file_rmap(struct folio *folio, struct page *page, unsigned int nr_pages, struct vm_area_struct *vma, enum rmap_mode mode) { - atomic_t *mapped = &folio->_nr_pages_mapped; - unsigned int nr_pmdmapped = 0, first; - int nr = 0; + unsigned int nr, nr_pmdmapped = 0; VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); - __folio_rmap_sanity_checks(folio, page, nr_pages, mode); - - /* Is page being mapped by PTE? Is this its first map to be added? */ - if (likely(mode == RMAP_MODE_PTE)) { - do { - first = atomic_inc_and_test(&page->_mapcount); - if (first && folio_test_large(folio)) { - first = atomic_inc_return_relaxed(mapped); - first = (first < COMPOUND_MAPPED); - } - - if (first) - nr++; - } while (page++, --nr_pages > 0); - } else if (mode == RMAP_MODE_PMD) { - first = atomic_inc_and_test(&folio->_entire_mapcount); - if (first) { - nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { - nr_pmdmapped = folio_nr_pages(folio); - nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); - /* Raced ahead of a remove and another add? */ - if (unlikely(nr < 0)) - nr = 0; - } else { - /* Raced ahead of a remove of COMPOUND_MAPPED */ - nr = 0; - } - } - } + nr = __folio_add_rmap(folio, page, nr_pages, mode, &nr_pmdmapped); if (nr_pmdmapped) __lruvec_stat_mod_folio(folio, folio_test_swapbacked(folio) ? NR_SHMEM_PMDMAPPED : NR_FILE_PMDMAPPED, nr_pmdmapped); From patchwork Mon Dec 4 14:21:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173374 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2795279vqy; Mon, 4 Dec 2023 06:23:07 -0800 (PST) X-Google-Smtp-Source: AGHT+IEmAY4KPGHGetqZQFkzuiPH6U1Op9cbhoglNjvGv7tHTHtLSmYQeGDgMikXgU+h1kZs9GdT X-Received: by 2002:a17:903:2305:b0:1d0:6ffd:cea8 with SMTP id d5-20020a170903230500b001d06ffdcea8mr1974514plh.97.1701699787386; Mon, 04 Dec 2023 06:23:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699787; cv=none; d=google.com; s=arc-20160816; b=iey5JgMjx47FwBHXG54+KN4HFoVAIKLHrYMYsKhnJGP9idV0vzRTSqd53yoq5qvFcZ Cf9qImTZnGUeT5MsE3HRI9h2LC7qg3vDeomP1xr9AgclVhYKXivbodaKutEUSylSKzku lfduzwT+YAZXFivxW3k5/bbIEIdtH5AGN5FgLhwFKwsl7UZNkSuRXBRhZCfWdA8Xxr4F +WSAun8t0nj9l5LcyfOGExSWPKBkf/A+PCNjYHbudVUBtoUZ/WWQlWK+iukc5ULbNDQc HBpTdpTGqUFSvewkJ8qFiApAHsDz3yANrEI2gJB4SlzVAMblRmKw/20XP780FQBSfG9P s06Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5BTpbXGEFWsHEYKSa5/ZaE1vDQBM0C6WRkCwnDKka/8=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=BTPHJh7p6A622uFEMKjkmMlxDGz+NDCYKZPoShWU4O23EUsun8CdalYq1zRqVpLoap 6vnKyD1/isYAQp/2X7WstvFBcVC9kn25I/84D9hQ+XLqWTxE8hXuKw0yAjArEbUzkROe K18G+SVD67RL2Vee5oMaxMN42qoj9Pjp9v025phOdubygkA/KERQydIRkPRuWOF8aELv GkYIopxJtDSfrU00+corbFQdSal7nKEglcd1eZpweQK4z09rpoTAQ7V2/g9TwL2y/Zym zv8wk6LIrwUtOqE5Kw0ZsE4tHnIHr5xN/slvYsbTU/xJDkBtAY87r7Z+XAZ1mStUwCpZ JWCQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=EiAMLuZT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id g11-20020a170902934b00b001cfc0272b48si7704158plp.103.2023.12.04.06.23.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:23:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=EiAMLuZT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 5814C808204E; Mon, 4 Dec 2023 06:23:01 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345157AbjLDOWv (ORCPT + 99 others); Mon, 4 Dec 2023 09:22:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55728 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345429AbjLDOWY (ORCPT ); Mon, 4 Dec 2023 09:22:24 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5CE310F7 for ; Mon, 4 Dec 2023 06:22:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699736; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5BTpbXGEFWsHEYKSa5/ZaE1vDQBM0C6WRkCwnDKka/8=; b=EiAMLuZT7DBXqHkOVN44nEhYuAQH8stHPHRXWQ/OoJV3ONIld4g4zwZlgJnsUYSv5nq7a5 f9j+bNlWvVypwXO5sFvMEsFyJbslmEoVBdlqZ93OHd5K/w71fd1qyKtDPcWoPwk9NX006K WtKFtIo9FVM110cABdKdVIND4lLL2r4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-491-JZokyINqMGSj3FxNN-lSUQ-1; Mon, 04 Dec 2023 09:22:15 -0500 X-MC-Unique: JZokyINqMGSj3FxNN-lSUQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BE6A6185A78A; Mon, 4 Dec 2023 14:22:14 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 24C502026D4C; Mon, 4 Dec 2023 14:22:13 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 14/39] mm/rmap: introduce folio_add_anon_rmap_[pte|ptes|pmd]() Date: Mon, 4 Dec 2023 15:21:21 +0100 Message-ID: <20231204142146.91437-15-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:23:01 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361556636811192 X-GMAIL-MSGID: 1784361556636811192 Let's mimic what we did with folio_add_file_rmap_*() so we can similarly replace page_add_anon_rmap() next. Make the compiler always special-case on the granularity by using __always_inline. Note that the new functions ignore the RMAP_COMPOUND flag, which we will remove as soon as page_add_anon_rmap() is gone. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 6 +++ mm/rmap.c | 115 +++++++++++++++++++++++++++++-------------- 2 files changed, 85 insertions(+), 36 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 95f7b94a70295..9e1c197f50199 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -229,6 +229,12 @@ static inline void __folio_rmap_sanity_checks(struct folio *folio, * rmap interfaces called when adding or removing pte of page */ void folio_move_anon_rmap(struct folio *, struct vm_area_struct *); +void folio_add_anon_rmap_ptes(struct folio *, struct page *, unsigned int nr, + struct vm_area_struct *, unsigned long address, rmap_t flags); +#define folio_add_anon_rmap_pte(folio, page, vma, address, flags) \ + folio_add_anon_rmap_ptes(folio, page, 1, vma, address, flags) +void folio_add_anon_rmap_pmd(struct folio *, struct page *, + struct vm_area_struct *, unsigned long address, rmap_t flags); void page_add_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, diff --git a/mm/rmap.c b/mm/rmap.c index c09b360402599..85bea11e9266b 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1267,38 +1267,21 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, unsigned long address, rmap_t flags) { struct folio *folio = page_folio(page); - atomic_t *mapped = &folio->_nr_pages_mapped; - int nr = 0, nr_pmdmapped = 0; - bool compound = flags & RMAP_COMPOUND; - bool first; - /* Is page being mapped by PTE? Is this its first map to be added? */ - if (likely(!compound)) { - first = atomic_inc_and_test(&page->_mapcount); - nr = first; - if (first && folio_test_large(folio)) { - nr = atomic_inc_return_relaxed(mapped); - nr = (nr < COMPOUND_MAPPED); - } - } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */ + if (likely(!(flags & RMAP_COMPOUND))) + folio_add_anon_rmap_pte(folio, page, vma, address, flags); + else + folio_add_anon_rmap_pmd(folio, page, vma, address, flags); +} - first = atomic_inc_and_test(&folio->_entire_mapcount); - if (first) { - nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { - nr_pmdmapped = folio_nr_pages(folio); - nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); - /* Raced ahead of a remove and another add? */ - if (unlikely(nr < 0)) - nr = 0; - } else { - /* Raced ahead of a remove of COMPOUND_MAPPED */ - nr = 0; - } - } - } +static __always_inline void __folio_add_anon_rmap(struct folio *folio, + struct page *page, unsigned int nr_pages, + struct vm_area_struct *vma, unsigned long address, rmap_t flags, + enum rmap_mode mode) +{ + unsigned int i, nr, nr_pmdmapped = 0; + nr = __folio_add_rmap(folio, page, nr_pages, mode, &nr_pmdmapped); if (nr_pmdmapped) __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr_pmdmapped); if (nr) @@ -1312,18 +1295,30 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, * folio->index right when not given the address of the head * page. */ - VM_WARN_ON_FOLIO(folio_test_large(folio) && !compound, folio); + VM_WARN_ON_FOLIO(folio_test_large(folio) && + mode != RMAP_MODE_PMD, folio); __folio_set_anon(folio, vma, address, !!(flags & RMAP_EXCLUSIVE)); } else if (likely(!folio_test_ksm(folio))) { __page_check_anon_rmap(folio, page, vma, address); } - if (flags & RMAP_EXCLUSIVE) - SetPageAnonExclusive(page); - /* While PTE-mapping a THP we have a PMD and a PTE mapping. */ - VM_WARN_ON_FOLIO((atomic_read(&page->_mapcount) > 0 || - (folio_test_large(folio) && folio_entire_mapcount(folio) > 1)) && - PageAnonExclusive(page), folio); + + if (flags & RMAP_EXCLUSIVE) { + if (likely(nr_pages == 1 || mode != RMAP_MODE_PTE)) + SetPageAnonExclusive(page); + else + for (i = 0; i < nr_pages; i++) + SetPageAnonExclusive(page + i); + } + for (i = 0; i < nr_pages; i++) { + struct page *cur_page = page + i; + + /* While PTE-mapping a THP we have a PMD and a PTE mapping. */ + VM_WARN_ON_FOLIO((atomic_read(&cur_page->_mapcount) > 0 || + (folio_test_large(folio) && + folio_entire_mapcount(folio) > 1)) && + PageAnonExclusive(cur_page), folio); + } /* * For large folio, only mlock it if it's fully mapped to VMA. It's @@ -1335,6 +1330,54 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, mlock_vma_folio(folio, vma); } +/** + * folio_add_anon_rmap_ptes - add PTE mappings to a page range of an anon folio + * @folio: The folio to add the mappings to + * @page: The first page to add + * @nr_pages: The number of pages which will be mapped + * @vma: The vm area in which the mappings are added + * @address: The user virtual address of the first page to map + * @flags: The rmap flags + * + * The page range of folio is defined by [first_page, first_page + nr_pages) + * + * The caller needs to hold the page table lock, and the page must be locked in + * the anon_vma case: to serialize mapping,index checking after setting, + * and to ensure that an anon folio is not being upgraded racily to a KSM folio + * (but KSM folios are never downgraded). + */ +void folio_add_anon_rmap_ptes(struct folio *folio, struct page *page, + unsigned int nr_pages, struct vm_area_struct *vma, + unsigned long address, rmap_t flags) +{ + __folio_add_anon_rmap(folio, page, nr_pages, vma, address, flags, + RMAP_MODE_PTE); +} + +/** + * folio_add_anon_rmap_pmd - add a PMD mapping to a page range of an anon folio + * @folio: The folio to add the mapping to + * @page: The first page to add + * @vma: The vm area in which the mapping is added + * @address: The user virtual address of the first page to map + * @flags: The rmap flags + * + * The page range of folio is defined by [first_page, first_page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock, and the page must be locked in + * the anon_vma case: to serialize mapping,index checking after setting. + */ +void folio_add_anon_rmap_pmd(struct folio *folio, struct page *page, + struct vm_area_struct *vma, unsigned long address, rmap_t flags) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_add_anon_rmap(folio, page, HPAGE_PMD_NR, vma, address, flags, + RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + /** * folio_add_new_anon_rmap - Add mapping to a new anonymous folio. * @folio: The folio to add the mapping to. From patchwork Mon Dec 4 14:21:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173381 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2795852vqy; Mon, 4 Dec 2023 06:23:59 -0800 (PST) X-Google-Smtp-Source: AGHT+IF16v6yEgQ7gyek1mz40eOY1wv66odJ4ayKsnmqW/3MC/ARMd5yOrXAK27DDNdJ75j5m1yk X-Received: by 2002:a05:6a20:8e13:b0:18b:4632:af0b with SMTP id y19-20020a056a208e1300b0018b4632af0bmr5239263pzj.17.1701699839242; Mon, 04 Dec 2023 06:23:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699839; cv=none; d=google.com; s=arc-20160816; b=PGDYeYe2MAe05jY9MgvLLXnhHSeSRPEioX52gb2Oki159Xg4GZXph6+WyP591rdqEr wrL+PJjxnzb+gs+tjtA+b/7GV6epuchwacgu0JnUxP1Z7tKTvIMXuw1UpkPysUPL8oSq 9S3qfg8TzYIGP2vDIubUzWjZdwBPc+cmPBoSPuKHtx+o0t8JsPbReufV1QsZgchH+fqN RMr5GSlrfFGP4hoocO/+bnuMyQwx+hBggrPlGr2tylZC+k2mAI8VxJzKKwRlC0eW6QC4 Zy78hd/tj/HKdkhB5IcDB14p0XedJ8sCUFMl0csOTGmk+uRmaR3quVEgmhwXbxHKZaRD hzUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0rYG9NNO7tFTRIi4sOXhh6ZZDANHjsXpFzljN91NqwE=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=dpuHTVM827ZqK1McyHXOcjERyIhI4yj8kOxkWENbwpofFoGZi2kqwo4wj8Ii6nhLwx MhfZuKYmq4B/2/Nmy3xv+edmCt6BXmegvy+ceKujmYlaI9L2DkSC8VQ8W7hK9FDleL6i tcHwQ0kxWv4FtSmHnwn81GiHIF9kJ+WSiq8SCsnpmO9w/vpPGAZQUp5YyQ5u9b8TgqKk udBGHhCX04D5FRLlnY0pH8L9PfWVdYOSPqu1I9xyVM6RGokSI373v/WAIEUMbjz+Zma4 4USVUWPdL3/MGRDpLnu8QkuxCIRTQd07p/6Q0xqZCwJOi4IlF10dKwsGWQe3sy7r0hHj ok2w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bBBZr2L3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id by42-20020a056a0205aa00b005bdfd3a0d3esi2442926pgb.185.2023.12.04.06.23.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:23:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bBBZr2L3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 0F68D80ACFE6; Mon, 4 Dec 2023 06:23:48 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235922AbjLDOWz (ORCPT + 99 others); Mon, 4 Dec 2023 09:22:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52128 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235857AbjLDOWZ (ORCPT ); Mon, 4 Dec 2023 09:22:25 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CF7ED1720 for ; Mon, 4 Dec 2023 06:22:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699738; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0rYG9NNO7tFTRIi4sOXhh6ZZDANHjsXpFzljN91NqwE=; b=bBBZr2L3F/wfak2zVi+bJFcURglvRlJ2bxcPAtLLewO2Q8vr9HvnCnzLkagf/Nj50xE1hs oCGi9gVd1Lomi5nWKV4eSDX0YgiBWowlm3vNy5DJf2thjEfEU36Wfpd9nm6e19unUfjkTb 48rDdOV4oSlmYHzLV7tiRPoI9I7v2NU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-658-L-pBgpvKP7yCqqoQs9ALwA-1; Mon, 04 Dec 2023 09:22:17 -0500 X-MC-Unique: L-pBgpvKP7yCqqoQs9ALwA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EF0598314FC; Mon, 4 Dec 2023 14:22:16 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 052CE2026D4C; Mon, 4 Dec 2023 14:22:14 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 15/39] mm/huge_memory: batch rmap operations in __split_huge_pmd_locked() Date: Mon, 4 Dec 2023 15:21:22 +0100 Message-ID: <20231204142146.91437-16-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:23:48 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361610363944738 X-GMAIL-MSGID: 1784361610363944738 Let's use folio_add_anon_rmap_ptes(), batching the rmap operations. While at it, use more folio operations (but only in the code branch we're touching), use VM_WARN_ON_FOLIO(), and pass RMAP_COMPOUND instead of manually setting PageAnonExclusive. We should never see non-anon pages on that branch: otherwise, the existing page_add_anon_rmap() call would have been flawed already. Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index cb33c6e0404cf..2c037ab3f4916 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2099,6 +2099,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, unsigned long haddr, bool freeze) { struct mm_struct *mm = vma->vm_mm; + struct folio *folio; struct page *page; pgtable_t pgtable; pmd_t old_pmd, _pmd; @@ -2194,16 +2195,18 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, uffd_wp = pmd_swp_uffd_wp(old_pmd); } else { page = pmd_page(old_pmd); + folio = page_folio(page); if (pmd_dirty(old_pmd)) { dirty = true; - SetPageDirty(page); + folio_set_dirty(folio); } write = pmd_write(old_pmd); young = pmd_young(old_pmd); soft_dirty = pmd_soft_dirty(old_pmd); uffd_wp = pmd_uffd_wp(old_pmd); - VM_BUG_ON_PAGE(!page_count(page), page); + VM_WARN_ON_FOLIO(!folio_ref_count(folio), folio); + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); /* * Without "freeze", we'll simply split the PMD, propagating the @@ -2220,11 +2223,18 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, * * See page_try_share_anon_rmap(): invalidate PMD first. */ - anon_exclusive = PageAnon(page) && PageAnonExclusive(page); + anon_exclusive = PageAnonExclusive(page); if (freeze && anon_exclusive && page_try_share_anon_rmap(page)) freeze = false; - if (!freeze) - page_ref_add(page, HPAGE_PMD_NR - 1); + if (!freeze) { + rmap_t rmap_flags = RMAP_NONE; + + folio_ref_add(folio, HPAGE_PMD_NR - 1); + if (anon_exclusive) + rmap_flags = RMAP_EXCLUSIVE; + folio_add_anon_rmap_ptes(folio, page, HPAGE_PMD_NR, + vma, haddr, rmap_flags); + } } /* @@ -2267,8 +2277,6 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = mk_pte(page + i, READ_ONCE(vma->vm_page_prot)); if (write) entry = pte_mkwrite(entry, vma); - if (anon_exclusive) - SetPageAnonExclusive(page + i); if (!young) entry = pte_mkold(entry); /* NOTE: this may set soft-dirty too on some archs */ @@ -2278,7 +2286,6 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = pte_mksoft_dirty(entry); if (uffd_wp) entry = pte_mkuffd_wp(entry); - page_add_anon_rmap(page + i, vma, addr, RMAP_NONE); } VM_BUG_ON(!pte_none(ptep_get(pte))); set_pte_at(mm, addr, pte, entry); From patchwork Mon Dec 4 14:21:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173378 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2795751vqy; Mon, 4 Dec 2023 06:23:52 -0800 (PST) X-Google-Smtp-Source: AGHT+IFFDa215hzBtU750CW+MufUtxp4M/Hu+B3YoSRXPZaApt6fMDLoZCs3MVEZL1mm+IEfV6VY X-Received: by 2002:a05:6a20:9193:b0:187:dd5f:93b9 with SMTP id v19-20020a056a20919300b00187dd5f93b9mr1919873pzd.14.1701699831969; Mon, 04 Dec 2023 06:23:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699831; cv=none; d=google.com; s=arc-20160816; b=D35tMEJIYpd2vDhQeBzaisanHRtE7iWQNQuFd33UELmBtWngtpeNv5qQoJB0EP4QWV VZnuByterQwneelM4cfXdzNEZ9VgrOhA7tmJ/IYEtabClyJ2X8/U5e0hbLTeP/ddkWjb hDJBFuRXZaqb9hZv0Jjfo0S396TaH5Y38WtlkT03Q0Lan0fAUkHicRS+iLsCti+3jlV8 FDcRQmXCbwOw2XH1cSk8pe0r2h3WdNwSFOFM4LYu5iE+GUDsPzNPieIn6dTr/D2g0QX6 AZfL2G+nS19rLckKoLiMTvKF8omulAPleKmKBXX9ppsdB2caQwQS0jJAMfrQr9ZLLMTy Y2qQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=SP+gQ2Yl7ptxMmm0EN/M4/z6MAPTYZkvBQ5MoVKoj/U=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=yvJFfcHhoz6yR9aSY5/9uZe+Yb9Dr8lZKbatEQQzaC1tksSY/Nm74kmOqkirFOn1Vz +MUvsZl/cSaZTv3o4//aRJ95yU43xBq1rKbki6wU+8VERJceX1FIn7VPjSAgD+j3zF5Y JpJqupAfzSUoumC3BM4mF+LohrJ3Gm7e5JCx0jcxwicfICY1qfMnUim/sVfLLnhQc7BK Wdqfer3O6o5y/6gM2hK8EIxUvPKZCAHR+nLag41Z/lXHsWr6y80lrKcUXBSbR5CALv0g LYysr1dGyeI5ygYfGQeoqc03NdwIf0Bs6M1pFLScwXvHTHvP08/7GcjCtPcNHMRC1ehp U39A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JIOAxx9X; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id u24-20020a631418000000b005c685a3f4acsi2125874pgl.887.2023.12.04.06.23.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:23:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JIOAxx9X; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 5C9CA80B1E47; Mon, 4 Dec 2023 06:23:46 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235883AbjLDOXR (ORCPT + 99 others); Mon, 4 Dec 2023 09:23:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55712 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235887AbjLDOWt (ORCPT ); Mon, 4 Dec 2023 09:22:49 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F26D819A0 for ; Mon, 4 Dec 2023 06:22:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699747; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SP+gQ2Yl7ptxMmm0EN/M4/z6MAPTYZkvBQ5MoVKoj/U=; b=JIOAxx9XAGxW58tGaxmqu0YBcg2Yjeodr7xD3/UnF1MgoHhmjntBJg9d6tq6ZVm6G8zUxh 3lnZiE2WNltSUrgcBPHRDb6y7LWJ+m/WlnCLHIcehCUeCTIqEKZa+yZ9cL6QstgCSgcEIN fzWlDEXcDarZQ7bT+/fx1NunX77GwBQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-629-LlQ3-zByOwSqnbEF8yG6og-1; Mon, 04 Dec 2023 09:22:19 -0500 X-MC-Unique: LlQ3-zByOwSqnbEF8yG6og-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C80C18314FA; Mon, 4 Dec 2023 14:22:18 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3338B2026D4C; Mon, 4 Dec 2023 14:22:17 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 16/39] mm/huge_memory: page_add_anon_rmap() -> folio_add_anon_rmap_pmd() Date: Mon, 4 Dec 2023 15:21:23 +0100 Message-ID: <20231204142146.91437-17-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:23:46 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361603066506049 X-GMAIL-MSGID: 1784361603066506049 Let's convert remove_migration_pmd(). No need to set RMAP_COMPOUND, that we will remove soon. Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 2c037ab3f4916..332cb6cf99f38 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3310,12 +3310,12 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) pmde = pmd_mkdirty(pmde); if (folio_test_anon(folio)) { - rmap_t rmap_flags = RMAP_COMPOUND; + rmap_t rmap_flags = RMAP_NONE; if (!is_readable_migration_entry(entry)) rmap_flags |= RMAP_EXCLUSIVE; - page_add_anon_rmap(new, vma, haddr, rmap_flags); + folio_add_anon_rmap_pmd(folio, new, vma, haddr, rmap_flags); } else { folio_add_file_rmap_pmd(folio, new, vma); } From patchwork Mon Dec 4 14:21:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173375 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2795475vqy; Mon, 4 Dec 2023 06:23:25 -0800 (PST) X-Google-Smtp-Source: AGHT+IGzZ7lKB+CHffAk3CzTbggY6RqIvPMQmR0Pv4neyUnOQwiHZPg9j6zH+nQ/WYGkuTXSL0ra X-Received: by 2002:a05:6a20:748f:b0:18c:8fef:22cc with SMTP id p15-20020a056a20748f00b0018c8fef22ccmr1728000pzd.24.1701699805040; Mon, 04 Dec 2023 06:23:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699805; cv=none; d=google.com; s=arc-20160816; b=VXSCjZag5Du0Ce+B+YEtw6iALhgFmFf60UxncEq9AnBlTACBRZ+IZiDryoH1M1lhyx PT8RFLGNss7Uqj3ekekqgK3Zudk/oZoLfEcvmQE0wrgoJ12etxj+S3oIFrYPqGwx26u7 PPPLC/WMLRy70Y4Zr/DtYin7XRsY76OHQAFfGgpiwF+wV2nH73hF2+i77WjO6mxBIYzW hfPTWEJS+aYGjmxfIsBDF5Fl1bl5j7sI1mL4xcG19dU2shfWrFbDmHDH73tAGWypn3ga FUwAl8t3XdaMgAV2UAWoQg9cQJjOF0/Qmqlct6pyclXYDpydfH465Lj5aMRujP1t0cCH tc0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vhcry2rMTXBwyY+r1UEd/G15BeM6OBHeZJqknbajPVE=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=afy6eP/mf8neC3ow7GInmq6zpPjEWUQtaPXYF2OmFQt+DxRHTEhtCetXa5uVi6gufO h6aFO8AFuXRUVf7eKrCxSzRhK8jvBGE5abAsLbCE+snUVnhzo/O/YLwsFe+Z+ZwtLA4S LUe2iGarq0cr8kmyiuNIpqNgjzzbN8MOgjjEG50hV9Zi/WWDaCJKxfUEtOjfkU215PAP Y2//pnFf7AtFAIVwY+d+BX7c0qDq2QTFbrz9KPDFMt+dVDRr3r/nAwkrY1OcjbnuW+xN qMkUq7VPLsTbVfO0I0RKV2eSzUmiFb2bKJxiG78/HnsYL8lvfxe4duaT6LIplMGTLvHa VCmw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=EIrO8iVz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id dr1-20020a056a004a8100b006cb4d47e598si7646922pfb.141.2023.12.04.06.23.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:23:25 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=EIrO8iVz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id BC7E9803B3B3; Mon, 4 Dec 2023 06:23:22 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345426AbjLDOXG (ORCPT + 99 others); Mon, 4 Dec 2023 09:23:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59628 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345047AbjLDOWg (ORCPT ); Mon, 4 Dec 2023 09:22:36 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0190B1994 for ; Mon, 4 Dec 2023 06:22:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699744; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vhcry2rMTXBwyY+r1UEd/G15BeM6OBHeZJqknbajPVE=; b=EIrO8iVzv5bjb4oji3deYqexQozuete2XjPB3KR/4WzBVSejdY6AIFumPDl2u9atj1lKS+ RFZ96MJ8BkGGMEKz9pDp675dH8XItQcVLG8MXhzRd05+CHF7SiH24KEl2bW9gENmEc/hua Yj4ah+yn0d+gaDhWefGPi9lATDjF3yI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-688-ZWpJIdMyMpmAHNLnESmGHw-1; Mon, 04 Dec 2023 09:22:21 -0500 X-MC-Unique: ZWpJIdMyMpmAHNLnESmGHw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A76AF85A58B; Mon, 4 Dec 2023 14:22:20 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0C38F2026D4C; Mon, 4 Dec 2023 14:22:18 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 17/39] mm/migrate: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Mon, 4 Dec 2023 15:21:24 +0100 Message-ID: <20231204142146.91437-18-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:23:22 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361574823867458 X-GMAIL-MSGID: 1784361574823867458 Let's convert remove_migration_pte(). Signed-off-by: David Hildenbrand --- mm/migrate.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index efc19f53b05e6..0e78680589bcc 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -259,8 +259,8 @@ static bool remove_migration_pte(struct folio *folio, #endif { if (folio_test_anon(folio)) - page_add_anon_rmap(new, vma, pvmw.address, - rmap_flags); + folio_add_anon_rmap_pte(folio, new, vma, + pvmw.address, rmap_flags); else folio_add_file_rmap_pte(folio, new, vma); set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); From patchwork Mon Dec 4 14:21:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173379 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2795786vqy; Mon, 4 Dec 2023 06:23:54 -0800 (PST) X-Google-Smtp-Source: AGHT+IEtCrqFgcKkWnJ8kg+mMwkGCebH11eoAoyVBgZBorkE3TnrTFNpQ+FOLN67VUxSNBP0K1R/ X-Received: by 2002:a05:6a20:1591:b0:18d:c72:63b3 with SMTP id h17-20020a056a20159100b0018d0c7263b3mr5657487pzj.16.1701699834526; Mon, 04 Dec 2023 06:23:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699834; cv=none; d=google.com; s=arc-20160816; b=eFuEd21aWt5vL/vfRfvlP4MKh+v6SqC7BBNInI/bpXdBwyTP1IKkxC9mRiE8JVvJc1 Z+Vj8eaeUz807cnYZ15KWKRJMxJIHGcStOajEEhCJgvetpSfXo9MFTSabhOg8/oM1KT3 xk60w7voEaB6beTvnajhRLdBP0cuIb9X2sB6TUq9tmyy2llET3v620+Mr+jBePywF9jZ eS7Z5ZyaScBv/gZiPEGa6Bk2xl2Tezqpk6iugeviX4p9/WNB/sBpCP91WEYUnSns99PM hO7U2iBX+/fgar2N4AGUqK9cucdV074nS1Sv2o1OQFpj+oopBbd/hQ+NOn0lhlmcPpAe LLnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=MP2FAIOx2jsvMIhG8WePR/5GNOuXZxiB4Uil9+VePO8=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=vSu+FUEnmtno41WE7cD11+agJve3Dmqfw3/UOTV/tnEsucqxGPGKWCPNHvcREWi9g5 iWGBLLsQM2ly3pvcSwgtvaCP+Ohw0dCMqPqYToXHPp9kpuW/rJnXezN/0eX4iprwuJS8 6UeuzgH8xHYWIGZTHBWTCQYxgpRDRVSx7XyUbl6hlC7Buq9kos34Ip5cim8s5VHBp8jd MHgVjjc30zgdubTJk+rzBmd1yD6RFG6pijP/X8DyYR/BM6AHEXbqsud4QS/fxQ8Tcuiz lZuH/HWAAk606v37MOUTdu3B0TPGD+vr2zIzk4Li/WLl3IHMtYD1yTEHzFllvbWkrucb 6qyA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=dZnBkc4L; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id r9-20020a63d909000000b005b95ee3edc6si7949883pgg.628.2023.12.04.06.23.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:23:54 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=dZnBkc4L; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 01DAD808725E; Mon, 4 Dec 2023 06:23:45 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345546AbjLDOXV (ORCPT + 99 others); Mon, 4 Dec 2023 09:23:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59494 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235888AbjLDOWt (ORCPT ); Mon, 4 Dec 2023 09:22:49 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 672021A1 for ; Mon, 4 Dec 2023 06:22:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699747; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MP2FAIOx2jsvMIhG8WePR/5GNOuXZxiB4Uil9+VePO8=; b=dZnBkc4LXq1egqVzhauWpAlAlt4hPT62GGDRM4cZRZr87+nw41yaaOL2E8FraXLGKuxjeh ex7t/Rzw5AjdFBM//gHI5OjaYgqKKUv7wU0MV/G8tu6ovZ7UsLhF1/aiLUgRhJpDqNbm7l ayaYTeJAEaASw/ve0DjkwvAPmv5j6l8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-81-b962mDT3OiSU6I-NXCof5g-1; Mon, 04 Dec 2023 09:22:23 -0500 X-MC-Unique: b962mDT3OiSU6I-NXCof5g-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8B19389CA0A; Mon, 4 Dec 2023 14:22:22 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id E0BAE2026D4C; Mon, 4 Dec 2023 14:22:20 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 18/39] mm/ksm: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Mon, 4 Dec 2023 15:21:25 +0100 Message-ID: <20231204142146.91437-19-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:23:45 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361605531844852 X-GMAIL-MSGID: 1784361605531844852 Let's convert replace_page(). While at it, perform some folio conversion. Signed-off-by: David Hildenbrand --- mm/ksm.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index 6a831009b4cbf..357944588a9a9 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1199,6 +1199,7 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, static int replace_page(struct vm_area_struct *vma, struct page *page, struct page *kpage, pte_t orig_pte) { + struct folio *kfolio = page_folio(kpage); struct mm_struct *mm = vma->vm_mm; struct folio *folio; pmd_t *pmd; @@ -1238,15 +1239,16 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, goto out_mn; } VM_BUG_ON_PAGE(PageAnonExclusive(page), page); - VM_BUG_ON_PAGE(PageAnon(kpage) && PageAnonExclusive(kpage), kpage); + VM_BUG_ON_FOLIO(folio_test_anon(kfolio) && PageAnonExclusive(kpage), + kfolio); /* * No need to check ksm_use_zero_pages here: we can only have a * zero_page here if ksm_use_zero_pages was enabled already. */ if (!is_zero_pfn(page_to_pfn(kpage))) { - get_page(kpage); - page_add_anon_rmap(kpage, vma, addr, RMAP_NONE); + folio_get(kfolio); + folio_add_anon_rmap_pte(kfolio, kpage, vma, addr, RMAP_NONE); newpte = mk_pte(kpage, vma->vm_page_prot); } else { /* From patchwork Mon Dec 4 14:21:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173377 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2795734vqy; Mon, 4 Dec 2023 06:23:50 -0800 (PST) X-Google-Smtp-Source: AGHT+IE0l7vIiUv93Metcm4qyQYuaw1XDDy20qBwtae23FAK5XzWJx6ACO/QKfXN4tCpIGDTHayP X-Received: by 2002:a17:90b:145:b0:27d:12e1:7e20 with SMTP id em5-20020a17090b014500b0027d12e17e20mr3216713pjb.12.1701699830032; Mon, 04 Dec 2023 06:23:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699829; cv=none; d=google.com; s=arc-20160816; b=YEbgt/XdErLLZr9CaISu6Pw5Amh26O3gEl13LBPmxB9OsliGN1BK8UMA0qevN0fcbt yPWs0al21Nqlot2/qzSL0pV9cjAfa+0x/eUUin6hdAnHzDjiD4nbF6ewJveQEqLwV+wk vGqcP7WdLBJt5RPAIRxvpViY7lQpYhyjgEUfETCXka7ZZiEdeOU8+o63zcVX0K7NhDuj CMXzwsuE0yAM4kQLdlb1MdcrLth25bvVPaifuttkVoLGqd2w8Fcsgh9kf3TcXwe4B+Nu DsP0rOc8U897ThI7M91ns54i3rMeJ71vW5TvTyNtKWH/XjE2AfLTwgkMwAORJ7cggMlK Xulg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+HQj3RQXy3rfd5XJOTyn9grirPLdYd+avDtLGVmo9F4=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=HX5eFKbLx9ozkz8PMpDgT3uIJU1BvMgDFgHatz2v61aIgy83r0D55H8dfyb/NbZjJH bR97Zp2mq9tZzAtqbty+jZqgtQa/J+FpJBNmke0B2v9tKxD8gc1hROO4200KJx2awYoq anH1J+q1JlESPOcsgHVMJd1gQ0Ld46kcCuAMeZW4X+Cg8cEVk48/eMCuw4D1p5skZwG3 jciusCGh04djcnje/QurS7DFWrolwECwttr2MKRTrVGea/F52eBdX8Dgq0LUw9p/jKb/ 01aA1Q2ySfqdJRzAWsQ/p0LG4dilfQfVzCdCLvtug8RFVrn/gVeL7V8r4RK/m+o2jP2z 795Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="P/Y3ewnK"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id in24-20020a17090b439800b00286b48c954asi1548828pjb.11.2023.12.04.06.23.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:23:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="P/Y3ewnK"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 90CBD80A73E2; Mon, 4 Dec 2023 06:23:48 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345573AbjLDOX2 (ORCPT + 99 others); Mon, 4 Dec 2023 09:23:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60722 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235866AbjLDOWy (ORCPT ); Mon, 4 Dec 2023 09:22:54 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A1F9DF for ; Mon, 4 Dec 2023 06:22:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699750; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+HQj3RQXy3rfd5XJOTyn9grirPLdYd+avDtLGVmo9F4=; b=P/Y3ewnKOduZ3IwpUZpgzvJbXVEYRjc26ORaCJniyscFMCENyhnL6YARTI7JS/3PdMRSyP J/baf3cxnsf30r7TbJ8fR5ZvSqQKEw+gzUWTNk3uXhOMEh6oQj5reu+tNcH7f11FBtuff9 A1txQpg1Meoz/7/1cIbsshJ7/ZzTGZI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-695-AG-JYXFCP3KppGdqheHrmQ-1; Mon, 04 Dec 2023 09:22:24 -0500 X-MC-Unique: AG-JYXFCP3KppGdqheHrmQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6C7CF831506; Mon, 4 Dec 2023 14:22:24 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id C88092026D68; Mon, 4 Dec 2023 14:22:22 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 19/39] mm/swapfile: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Mon, 4 Dec 2023 15:21:26 +0100 Message-ID: <20231204142146.91437-20-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:23:48 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361601145587335 X-GMAIL-MSGID: 1784361601145587335 Let's convert unuse_pte(). Signed-off-by: David Hildenbrand --- mm/swapfile.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 4bc70f4591641..1ded3c150df95 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1805,7 +1805,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, if (pte_swp_exclusive(old_pte)) rmap_flags |= RMAP_EXCLUSIVE; - page_add_anon_rmap(page, vma, addr, rmap_flags); + folio_add_anon_rmap_pte(folio, page, vma, addr, rmap_flags); } else { /* ksm created a completely new copy */ page_add_new_anon_rmap(page, vma, addr); lru_cache_add_inactive_or_unevictable(page, vma); From patchwork Mon Dec 4 14:21:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173384 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2796046vqy; Mon, 4 Dec 2023 06:24:15 -0800 (PST) X-Google-Smtp-Source: AGHT+IGEmzQk00xgqdBq+fJ6BU/nPINVKtKrRlzyXwy3CiFPVLTlnBvedErjPUrUNglDxOdAMVcL X-Received: by 2002:a05:6a20:7346:b0:181:b87a:b8ba with SMTP id v6-20020a056a20734600b00181b87ab8bamr1604348pzc.53.1701699855673; Mon, 04 Dec 2023 06:24:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699855; cv=none; d=google.com; s=arc-20160816; b=SukbPbyVz7iTPBlTy8jpZE58ouE4PzzkUC/n3SOkzbpJwB/ur7Y9t8rMq1VBDdpvyG 2Yx9KeMl0wSrf/yLLduDGKWdZTYFzJYnResDQccEs7JKGdjHRFWAGahznwIaLXHI++Ry Jb7kljGtYrUt8kU7GDFeioJJrRmt4PjLImEASymUnYviubkyRSn4C+RFnAGUJ0t4zwZu i/P6tS8hMoxLAc9oQPFAf5aLGTj2QPvNMD9pY5FlR3mAj+oix3UcRCXIOY0K8lAPDP/f ZnBW8JoWYJlJWebOgdzS8tQguGraJiY/3JUFCMQS7XwzZQBIOeDtWKSmF4Im9HypW+qM YcAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=YcgEyKtjLabsj9Cw4VckjmMQxcp5kDVs71EjXlBqC+E=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=pw0IVSBTaQOwCPtbL/Ha1ZTHzMmKTczticK1LOeXlGMvfufFvlotxHn/lIR/GcvyfC F92jyOFwvNakT4Lu50Hccb97OBJS3JhNnfwVbz5Iina22Wu3E/Q/DxZU7GTnhKBFydvW QamSx1CZRBI2BXmzxUSAc5hMbtTahiX9IdzIeMunIQF5W0bZfFLtxR1vmVjNLw10vT0k RhBMSP/9QnOcIsVxh+MBmlKlfLR3sn7vmbiN9hexIkwF5NkG5rZb7iDhdhe3af/1Ixkv K7SZipwXHbY2I02DExkoh8NtYlcS5bpULSnmwJR1h/3/WGXg5y/4XxgLeUPFcn0ZDuEn /5uw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=BBNBL3QW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id j1-20020a056a00174100b006ce42ae90edsi2818890pfc.291.2023.12.04.06.24.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:24:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=BBNBL3QW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 6160080A4A20; Mon, 4 Dec 2023 06:24:13 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235880AbjLDOXe (ORCPT + 99 others); Mon, 4 Dec 2023 09:23:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35654 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235918AbjLDOWz (ORCPT ); Mon, 4 Dec 2023 09:22:55 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F59D107 for ; Mon, 4 Dec 2023 06:22:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699751; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YcgEyKtjLabsj9Cw4VckjmMQxcp5kDVs71EjXlBqC+E=; b=BBNBL3QWhxrNegllyxURZlMhK5hkYA9nbW2PMgkNBJ69gpjZm7Id24NOwODrJlj0ga+TRT UiF1OcKkhs12NSjlred/OFhWPPu3S5r0kiiVfp1bew20KTLlIJ8Ww9CN9X4jw4Yd+RkeH4 tGpjx8JxkVptn988OGeuOo1Yle7tEVA= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-600-YeNg8xp2Pg69b3ArcI-xHw-1; Mon, 04 Dec 2023 09:22:27 -0500 X-MC-Unique: YeNg8xp2Pg69b3ArcI-xHw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 464973C025D8; Mon, 4 Dec 2023 14:22:26 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id A5C672026D6E; Mon, 4 Dec 2023 14:22:24 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 20/39] mm/memory: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Mon, 4 Dec 2023 15:21:27 +0100 Message-ID: <20231204142146.91437-21-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:24:13 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361627749968388 X-GMAIL-MSGID: 1784361627749968388 Let's convert restore_exclusive_pte() and do_swap_page(). While at it, perform some folio conversion in restore_exclusive_pte(). Signed-off-by: David Hildenbrand --- mm/memory.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index be7fe58f7c297..9543b6e2b749b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -706,6 +706,7 @@ static void restore_exclusive_pte(struct vm_area_struct *vma, struct page *page, unsigned long address, pte_t *ptep) { + struct folio *folio = page_folio(page); pte_t orig_pte; pte_t pte; swp_entry_t entry; @@ -721,14 +722,15 @@ static void restore_exclusive_pte(struct vm_area_struct *vma, else if (is_writable_device_exclusive_entry(entry)) pte = maybe_mkwrite(pte_mkdirty(pte), vma); - VM_BUG_ON(pte_write(pte) && !(PageAnon(page) && PageAnonExclusive(page))); + VM_BUG_ON_FOLIO(pte_write(pte) && (!folio_test_anon(folio) && + PageAnonExclusive(page)), folio); /* * No need to take a page reference as one was already * created when the swap entry was made. */ - if (PageAnon(page)) - page_add_anon_rmap(page, vma, address, RMAP_NONE); + if (folio_test_anon(folio)) + folio_add_anon_rmap_pte(folio, page, vma, address, RMAP_NONE); else /* * Currently device exclusive access only supports anonymous @@ -4065,7 +4067,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) page_add_new_anon_rmap(page, vma, vmf->address); folio_add_lru_vma(folio, vma); } else { - page_add_anon_rmap(page, vma, vmf->address, rmap_flags); + folio_add_anon_rmap_pte(folio, page, vma, vmf->address, + rmap_flags); } VM_BUG_ON(!folio_test_anon(folio) || From patchwork Mon Dec 4 14:21:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173380 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2795827vqy; Mon, 4 Dec 2023 06:23:57 -0800 (PST) X-Google-Smtp-Source: AGHT+IEv3LogFki88F/xMsrGhrZpQAvD+hsYVV8nPFyDnuL+WUt1izhs87zRAk1B38fRit7tKEST X-Received: by 2002:a05:6a00:8d8a:b0:6ce:4018:a726 with SMTP id im10-20020a056a008d8a00b006ce4018a726mr3903511pfb.62.1701699837132; Mon, 04 Dec 2023 06:23:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699837; cv=none; d=google.com; s=arc-20160816; b=uKVY5IJTKEicVIt9W+7O3F/V5f4rFi/XhhpDfuong7D/1qfX4V7Beoj0ZAwKDKR+7g A5iiAPab2+lpArUqquuahbNUIWgF5G5YAF7xMr6iC5wzdGThhDXno1OpEK8KRsi9KMHi VtM4z7mtWf4x984f/3VgEAsVaZ7p0YnSQTgd3p/Xe17RRWToTe5UsyXYZrjAOKGV4R6z S8CiRwMT5EbLMJdAdU6mLmkf8YxTb2PmR8VMqTk9y52XARzK4oxFvBz0fPv+GdPWAyiP 2nbB+A830+fwa3DkQUwzjId27cCET9jSymJKskyWNl59EMXjf/krDQNgGtfnnhnx8TH9 g36A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=NQM7ysAYyclyPUNvOCqkfLGmI+LHJSTCFqaARoijbyI=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=ck11E3ONJ+KqFWdYz4hULfNGV5YSNFJCrxRfqMKDrcT4BOvU1fIm2fqDuOEIH/pQ30 y6cmo3I79Uw6/84qJaOUlkWQ+iGbZFMZzRjgeLJwD6qq5ccAAOaUlob5sp5fXllDnffQ tdCPEot3CUjjFdhU7+nD6Sir7Fh8CWYjAY0phsCtDrnx7eSgBc4q/vbKx5mdJPdQftnO qFBssgjXiq9XIA+7l3+dN7lvm5OtN197bpNpjpdTdo5MP3LajlHE/9/ewFIMSv+G6EyV oqypByGTGqf3chImDFrL6qrIv/YmBVsz5KxMrg2eV/qGmeI2mzHt8YybZIp3egNAj9qb On7A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=e6h5zBuA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id e15-20020a056a001a8f00b006c2cb9a884fsi7996168pfv.293.2023.12.04.06.23.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:23:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=e6h5zBuA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 1487980A7E76; Mon, 4 Dec 2023 06:23:56 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345200AbjLDOXj (ORCPT + 99 others); Mon, 4 Dec 2023 09:23:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235920AbjLDOWz (ORCPT ); Mon, 4 Dec 2023 09:22:55 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F8401BE for ; Mon, 4 Dec 2023 06:22:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699750; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NQM7ysAYyclyPUNvOCqkfLGmI+LHJSTCFqaARoijbyI=; b=e6h5zBuAAW7H7PK+BGyWpP0Qqlp1si+h8QuLO9BIzvs9bGkKlOjPIAXY2AleOxmEX22dsf 5dP79PdZzNNGxaFvfbyG7A3IueuPnwNd2ezMArmgcaJb9jfsVxq3PAFwIncTmABRmgVExf RozZn0pXlbWP49Pmr4vt3Kxuhav+Th0= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-220-6HiWY7_OOd6JGbhIhUP00g-1; Mon, 04 Dec 2023 09:22:28 -0500 X-MC-Unique: 6HiWY7_OOd6JGbhIhUP00g-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 207872825E9C; Mon, 4 Dec 2023 14:22:28 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7EADB2026D4C; Mon, 4 Dec 2023 14:22:26 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 21/39] mm/rmap: remove page_add_anon_rmap() Date: Mon, 4 Dec 2023 15:21:28 +0100 Message-ID: <20231204142146.91437-22-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:23:56 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361608423993508 X-GMAIL-MSGID: 1784361608423993508 All users are gone, remove it and all traces. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 2 -- mm/rmap.c | 31 ++++--------------------------- 2 files changed, 4 insertions(+), 29 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 9e1c197f50199..865d83148852d 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -235,8 +235,6 @@ void folio_add_anon_rmap_ptes(struct folio *, struct page *, unsigned int nr, folio_add_anon_rmap_ptes(folio, page, 1, vma, address, flags) void folio_add_anon_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); -void page_add_anon_rmap(struct page *, struct vm_area_struct *, - unsigned long address, rmap_t flags); void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address); void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, diff --git a/mm/rmap.c b/mm/rmap.c index 85bea11e9266b..4cb9d8b7d1d65 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1238,7 +1238,7 @@ static void __page_check_anon_rmap(struct folio *folio, struct page *page, * The page's anon-rmap details (mapping and index) are guaranteed to * be set up correctly at this point. * - * We have exclusion against page_add_anon_rmap because the caller + * We have exclusion against folio_add_anon_rmap_*() because the caller * always holds the page locked. * * We have exclusion against page_add_new_anon_rmap because those pages @@ -1251,29 +1251,6 @@ static void __page_check_anon_rmap(struct folio *folio, struct page *page, page); } -/** - * page_add_anon_rmap - add pte mapping to an anonymous page - * @page: the page to add the mapping to - * @vma: the vm area in which the mapping is added - * @address: the user virtual address mapped - * @flags: the rmap flags - * - * The caller needs to hold the pte lock, and the page must be locked in - * the anon_vma case: to serialize mapping,index checking after setting, - * and to ensure that PageAnon is not being upgraded racily to PageKsm - * (but PageKsm is never downgraded to PageAnon). - */ -void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, - unsigned long address, rmap_t flags) -{ - struct folio *folio = page_folio(page); - - if (likely(!(flags & RMAP_COMPOUND))) - folio_add_anon_rmap_pte(folio, page, vma, address, flags); - else - folio_add_anon_rmap_pmd(folio, page, vma, address, flags); -} - static __always_inline void __folio_add_anon_rmap(struct folio *folio, struct page *page, unsigned int nr_pages, struct vm_area_struct *vma, unsigned long address, rmap_t flags, @@ -1384,7 +1361,7 @@ void folio_add_anon_rmap_pmd(struct folio *folio, struct page *page, * @vma: the vm area in which the mapping is added * @address: the user virtual address mapped * - * Like page_add_anon_rmap() but must only be called on *new* folios. + * Like folio_add_anon_rmap_*() but must only be called on *new* folios. * This means the inc-and-test can be bypassed. * The folio does not have to be locked. * @@ -1432,7 +1409,7 @@ static __always_inline void __folio_add_file_rmap(struct folio *folio, if (nr) __lruvec_stat_mod_folio(folio, NR_FILE_MAPPED, nr); - /* See comments in page_add_anon_rmap() */ + /* See comments in folio_add_anon_rmap_*() */ if (!folio_test_large(folio)) mlock_vma_folio(folio, vma); } @@ -1546,7 +1523,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, /* * It would be tidy to reset folio_test_anon mapping when fully - * unmapped, but that might overwrite a racing page_add_anon_rmap + * unmapped, but that might overwrite a racing folio_add_anon_rmap_*() * which increments mapcount after us but sets mapping before us: * so leave the reset to free_pages_prepare, and remember that * it's only reliable while mapped. From patchwork Mon Dec 4 14:21:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173382 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2795868vqy; Mon, 4 Dec 2023 06:24:01 -0800 (PST) X-Google-Smtp-Source: AGHT+IG1pri+jD2JoY2oglXR34oMI922/L69cQi8BbagmkPYJEqSQ7uUZNKFEABh0Zcxy0VUShwN X-Received: by 2002:a17:902:da8f:b0:1d0:6d47:c3e3 with SMTP id j15-20020a170902da8f00b001d06d47c3e3mr1665618plx.17.1701699841000; Mon, 04 Dec 2023 06:24:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699840; cv=none; d=google.com; s=arc-20160816; b=dQ30fHKTsSxuoQi6e2yvsjonte8pBCviR/rCC6KwDyA2DWYi2bMa37k0pFTCUvyuUx KelEGnEMBPctdBssqcKi3o8F8LE1eDxAf4cxYnVOTjJdU/3SuWQIOjbCwtpLWd29yCFN L0tOeILCgve07n4k+XTVyw2YH+7kzikC7bHlL39ESspoTMyMT/sZ5jqDjpV2P7k/jQkl miwDUsBqBHPtqrwwF4PqIy4Ww/z9Hlv3zzMaMaO7nwJObbuUKVuS4xhoia0EKjXo3tOL jArT1H0eb/I0832/KcS8Qet1cyOAgLrjr5VXM4ALYI4sy/fjD/iTVup3izaVK4pqJZZg ow6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=yM8m6ebsRHp16rDmw00I1U0rPyCAW3pmLtjnA6MDqao=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=nJ/NFvCNdUrQ0/yWcmI/k+6uSwHgK4j9uS6XAz1DMb5wzcioVoH+GPvFcGKEUX3irb 4AwIiNDAPKfKbCd00oYw2ivKsSU3Q+NCr/p9j5HQZi248BGrTd6PKWMy0j2DBrvLPZ5c y7OJLBLeGdZOBcJoJSJgcyY4ZIOo6unCsnlNCXw1cH4IPXc9Yua96pZZzYJmOcYXJHth V9SMZAzf+M23zUo90Ldyo5wjJyYN9ga8M68mSNihXeD2Gd9T89c1fhsaBVtAKH5wuxud gDHHFI32uqvuqwGJXnyXoAXU4SfPkSimev5pQsgZy6AoomDzTfgR4DuBduyRzLBgiqXx 07mA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OfxSlGad; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id iz9-20020a170902ef8900b001d068dfe6e5si4674309plb.31.2023.12.04.06.24.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:24:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OfxSlGad; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 6EED980A7E76; Mon, 4 Dec 2023 06:23:59 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235920AbjLDOXm (ORCPT + 99 others); Mon, 4 Dec 2023 09:23:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345318AbjLDOXC (ORCPT ); Mon, 4 Dec 2023 09:23:02 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C069D19BC for ; Mon, 4 Dec 2023 06:22:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699754; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yM8m6ebsRHp16rDmw00I1U0rPyCAW3pmLtjnA6MDqao=; b=OfxSlGadlI4HtLxajMWXZbw3nuyK3JkFvFvj9sBje577HEXparIGrwYgrP7sOgC1e2e7xD 4/DVKx9aPOU7kSwRHIPOMyvikwYEp+ebr3SFAj3GoabwSFc93CMTZo3hbhCm+f2kPhk25q guLg1WsjjWzphOh+XA2U6jbXcnuw/Tc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-504-EoSRB73-NTeakXqzF60bBg-1; Mon, 04 Dec 2023 09:22:31 -0500 X-MC-Unique: EoSRB73-NTeakXqzF60bBg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F2B7C101A550; Mon, 4 Dec 2023 14:22:29 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 599D82026D4C; Mon, 4 Dec 2023 14:22:28 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 22/39] mm/rmap: remove RMAP_COMPOUND Date: Mon, 4 Dec 2023 15:21:29 +0100 Message-ID: <20231204142146.91437-23-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:23:59 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361611995384011 X-GMAIL-MSGID: 1784361611995384011 No longer used, let's remove it and clarify RMAP_NONE/RMAP_EXCLUSIVE a bit. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 12 +++--------- mm/rmap.c | 2 -- 2 files changed, 3 insertions(+), 11 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 865d83148852d..017b216915f19 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -172,20 +172,14 @@ struct anon_vma *folio_get_anon_vma(struct folio *folio); typedef int __bitwise rmap_t; /* - * No special request: if the page is a subpage of a compound page, it is - * mapped via a PTE. The mapped (sub)page is possibly shared between processes. + * No special request: A mapped anonymous (sub)page is possibly shared between + * processes. */ #define RMAP_NONE ((__force rmap_t)0) -/* The (sub)page is exclusive to a single process. */ +/* The anonymous (sub)page is exclusive to a single process. */ #define RMAP_EXCLUSIVE ((__force rmap_t)BIT(0)) -/* - * The compound page is not mapped via PTEs, but instead via a single PMD and - * should be accounted accordingly. - */ -#define RMAP_COMPOUND ((__force rmap_t)BIT(1)) - /* * Internally, we're using an enum to specify the granularity. Usually, * we make the compiler create specialized variants for the different diff --git a/mm/rmap.c b/mm/rmap.c index 4cb9d8b7d1d65..3587225055c5e 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2615,8 +2615,6 @@ void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc) * The following two functions are for anonymous (private mapped) hugepages. * Unlike common anonymous pages, anonymous hugepages have no accounting code * and no lru code, because we handle hugepages differently from common pages. - * - * RMAP_COMPOUND is ignored. */ void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address, rmap_t flags) From patchwork Mon Dec 4 14:21:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173383 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2795943vqy; Mon, 4 Dec 2023 06:24:07 -0800 (PST) X-Google-Smtp-Source: AGHT+IGBYIQrnO5kAgFgLi5duYwYlQbOgVV4Ao/FmtRHMJIFk150KFKhUkrspe5tZvp+ZiBBYgmd X-Received: by 2002:a17:902:7e4b:b0:1d0:5878:d4e3 with SMTP id a11-20020a1709027e4b00b001d05878d4e3mr3516113pln.9.1701699846184; Mon, 04 Dec 2023 06:24:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699846; cv=none; d=google.com; s=arc-20160816; b=eajy2NFIXX/7KRaMXSwTWW19EvRB9d+KDGsiu4IwrlzVpOCWrHQxHzta5tqmDrZJT8 Pu0g/IK1iDhPf8BzlQ3av8KeFfsFmf1TX9HGIsWVAzla5yoa/7wLfSBhJtS/W16i1mj7 3Ic/xoqPobIWZSuwU9ZeFqzfJxDBu0EDCGO8yTopkUx7qPom62H8nn+AzI9b2gvRU1f1 aORNblxCE5hkYXZH4nfHZwjOKijalN8tidN06TTvHi4CnvXqAIZPmJkdVyB7bURxW9j/ IND4kLTmjvesEu5adw5BGyNfD/7p6FVTCzwpDaa4E4CoxC8Qre+fOIestsglrIlvhhwh JOjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=MMp/rsTScJnGvEHSBNzvW50JXqbYQ0nspFlZU7tqJ/c=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=VHkdRcPrpgf5+BrMxtjq3B8v8V9/sSxvZeiOtLKEClZuJNcGV2G+ycBXKSjRwFw9M8 gmGmK2jN13yM0NiKNgXHHkn++wh3Pmiy23mrFLgu7RL35RC8x8yA7I/xpUfNHcgaFDkW CkPInELEdcaBLBuJEvZAByHvTrToWGblWJg+rMrWX6K/zsmnWrlBwbm83u/fJWDZW2J0 6LjE/tfjJLgzZ0STxCUHlRIoauCQFAwhg0anK0CBlIVYTqgyAUH+W9fOBQP1R4tNF8So BVY0svcY6ru63OyaZCvZKPAiePhEf5o/0e+AZUqKJlR39KeTFCBZQqkATrHGl8BZIOG8 B1MQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=PaCFXDen; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id n1-20020a1709026a8100b001d09e26a715si1864761plk.89.2023.12.04.06.24.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:24:06 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=PaCFXDen; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 10F4C80A7E76; Mon, 4 Dec 2023 06:24:05 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235884AbjLDOXw (ORCPT + 99 others); Mon, 4 Dec 2023 09:23:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345398AbjLDOXL (ORCPT ); Mon, 4 Dec 2023 09:23:11 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 34EBE1BD5 for ; Mon, 4 Dec 2023 06:22:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699757; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MMp/rsTScJnGvEHSBNzvW50JXqbYQ0nspFlZU7tqJ/c=; b=PaCFXDen46GeXpgUnGOLkf1/jSHuq7Tx0C02rdNeVRRBm79WRaQ9dI5KiJVCRIusdPYi7I bOnxKcEGu5PTMptePOuGSVb9gbQmx0YdcShGQh3+Mnj21KKT6oDj1M5ag/CZ9fcpW3HE8g kVknUCuH5xnXnBLC3/Dx33nnuLKzwlA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-494-KFAuLerwMeGWU5ALc3K4CQ-1; Mon, 04 Dec 2023 09:22:32 -0500 X-MC-Unique: KFAuLerwMeGWU5ALc3K4CQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4F04185A58C; Mon, 4 Dec 2023 14:22:32 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3A5802026D4C; Mon, 4 Dec 2023 14:22:30 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 23/39] mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]() Date: Mon, 4 Dec 2023 15:21:30 +0100 Message-ID: <20231204142146.91437-24-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:24:05 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361618240075385 X-GMAIL-MSGID: 1784361618240075385 Let's mimic what we did with folio_add_file_rmap_*() and folio_add_anon_rmap_*() so we can similarly replace page_remove_rmap() next. Make the compiler always special-case on the granularity by using __always_inline. We're adding folio_remove_rmap_ptes() handling right away, as we want to use that soon for batching rmap operations when unmapping PTE-mapped large folios. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 6 ++++ mm/rmap.c | 76 ++++++++++++++++++++++++++++++++++++-------- 2 files changed, 68 insertions(+), 14 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 017b216915f19..dd4ffb1d8ae04 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -241,6 +241,12 @@ void folio_add_file_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); +void folio_remove_rmap_ptes(struct folio *, struct page *, unsigned int nr, + struct vm_area_struct *); +#define folio_remove_rmap_pte(folio, page, vma) \ + folio_remove_rmap_ptes(folio, page, 1, vma) +void folio_remove_rmap_pmd(struct folio *, struct page *, + struct vm_area_struct *); void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address, rmap_t flags); diff --git a/mm/rmap.c b/mm/rmap.c index 3587225055c5e..50b6909157ac1 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1463,25 +1463,36 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, bool compound) { struct folio *folio = page_folio(page); + + if (likely(!compound)) + folio_remove_rmap_pte(folio, page, vma); + else + folio_remove_rmap_pmd(folio, page, vma); +} + +static __always_inline void __folio_remove_rmap(struct folio *folio, + struct page *page, unsigned int nr_pages, + struct vm_area_struct *vma, enum rmap_mode mode) +{ atomic_t *mapped = &folio->_nr_pages_mapped; - int nr = 0, nr_pmdmapped = 0; - bool last; + int last, nr = 0, nr_pmdmapped = 0; enum node_stat_item idx; - VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); - VM_BUG_ON_PAGE(compound && !PageHead(page), page); + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); /* Is page being unmapped by PTE? Is this its last map to be removed? */ - if (likely(!compound)) { - last = atomic_add_negative(-1, &page->_mapcount); - nr = last; - if (last && folio_test_large(folio)) { - nr = atomic_dec_return_relaxed(mapped); - nr = (nr < COMPOUND_MAPPED); - } - } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */ + if (likely(mode == RMAP_MODE_PTE)) { + do { + last = atomic_add_negative(-1, &page->_mapcount); + if (last && folio_test_large(folio)) { + last = atomic_dec_return_relaxed(mapped); + last = (last < COMPOUND_MAPPED); + } + if (last) + nr++; + } while (page++, --nr_pages > 0); + } else if (mode == RMAP_MODE_PMD) { last = atomic_add_negative(-1, &folio->_entire_mapcount); if (last) { nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); @@ -1517,7 +1528,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, * is still mapped. */ if (folio_test_pmd_mappable(folio) && folio_test_anon(folio)) - if (!compound || nr < nr_pmdmapped) + if (mode == RMAP_MODE_PTE || nr < nr_pmdmapped) deferred_split_folio(folio); } @@ -1532,6 +1543,43 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, munlock_vma_folio(folio, vma); } +/** + * folio_remove_rmap_ptes - remove PTE mappings from a page range of a folio + * @folio: The folio to remove the mappings from + * @page: The first page to remove + * @nr_pages: The number of pages that will be removed from the mapping + * @vma: The vm area from which the mappings are removed + * + * The page range of the folio is defined by [page, page + nr_pages) + * + * The caller needs to hold the page table lock. + */ +void folio_remove_rmap_ptes(struct folio *folio, struct page *page, + unsigned int nr_pages, struct vm_area_struct *vma) +{ + __folio_remove_rmap(folio, page, nr_pages, vma, RMAP_MODE_PTE); +} + +/** + * folio_remove_rmap_pmd - remove a PMD mapping from a page range of a folio + * @folio: The folio to remove the mapping from + * @page: The first page to remove + * @vma: The vm area from which the mapping is removed + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock. + */ +void folio_remove_rmap_pmd(struct folio *folio, struct page *page, + struct vm_area_struct *vma) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_remove_rmap(folio, page, HPAGE_PMD_NR, vma, RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + /* * @arg: enum ttu_flags will be passed to this argument */ From patchwork Mon Dec 4 14:21:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173385 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2796225vqy; Mon, 4 Dec 2023 06:24:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IGgMYN7HQF9QVRGO0xrEulgQ52OQhIeCg3fLf4yWTcll4MnLji74rKpuULHiWhr9aBJMWe3 X-Received: by 2002:a05:6a20:6a0e:b0:18a:7498:de11 with SMTP id p14-20020a056a206a0e00b0018a7498de11mr1741679pzk.4.1701699872538; Mon, 04 Dec 2023 06:24:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699872; cv=none; d=google.com; s=arc-20160816; b=ptx6sh8ADEMH8euulufuhWXYl9iIsuEGvLuvL1n/NfiWZeII+mHXCdkUSUgnjaD1pg piWIx3q5+OY2sCJnKGOe3qaD2XM8EbaeDbiE9QnFTq3t+tHJRoXA/lUx0YBZ3KKHTY2F R1HiRAmVxE8Ht19hQtaGDMGUOA0OSPLOZCAr31oUdQt5jkUQyTPjJoYT2HtjAkJxw6Oy qY0fiWoA6Mx0W7TcCtnRo+e5c9+awTDwjKzE91mL2l4AjYSwyM9STuMHXzoNlLUlkS8T p12gQeM6TfmZD4IfNglDlCuNrRTfNKGpEnR9sVTwxt0SqM3iQXuj8hq4ISEIQlXPhb9C N+XQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=nwrEIToy0DTNusea81H7rc7vo5MP+ocGWlNPVi4s7cw=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=ZWBZf0hal34IAA00rV9fqmKl5FDnlSePjG+3LnFdUn5wnz4VDHbB8HqSUZGRggFG+N jIKLfKBj9/DiVKLPaxSjJMZD9ngPkzvsmdyMXEuQH5w23XoUGaW1933Q0JlsTNYoax1G ylig5iz0EWLAYLdm8QLwUAGig9HyW782dA0NOrTkVAfmPF042mSFrf5v/lnVs2iQUKPj Z5FLhOf16upfA43OcJgHcnRYB1Mzqo7m+T5shbKv2IeDHaZEtYwJ7O2A1im71kGwAdvV 5zrSUo0CVCq80J6Lz6wMM6BzXg9KUGwjCIsXm2SNU3BEuwsNIugt4475K+Vmj5gGogKI ZPMw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=M7hYOejb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id r1-20020a6560c1000000b005c680fbab19si2322527pgv.513.2023.12.04.06.24.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:24:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=M7hYOejb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 275688096A71; Mon, 4 Dec 2023 06:24:29 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345069AbjLDOYI (ORCPT + 99 others); Mon, 4 Dec 2023 09:24:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52226 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235869AbjLDOXQ (ORCPT ); Mon, 4 Dec 2023 09:23:16 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2AC071BF2 for ; Mon, 4 Dec 2023 06:22:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699760; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nwrEIToy0DTNusea81H7rc7vo5MP+ocGWlNPVi4s7cw=; b=M7hYOejbkpOsNJGOtZjun0/TXZKWyfq7Lh2HCiH53JpCTLLiQe67nD9ABg2MJKLOOk21jT TZP009cltp4k4GxBeV+0zZDVAPnLtNWee35d1kB/+636Me/iRJSHDjHi2z9mHU5yCVbxSz YogfYVELWtLcji0MAVfD9UdCXAYbsKI= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-135-DF69DVHUPTOZMJ2zvuPJPA-1; Mon, 04 Dec 2023 09:22:34 -0500 X-MC-Unique: DF69DVHUPTOZMJ2zvuPJPA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 290303813BC9; Mon, 4 Dec 2023 14:22:34 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 87D722026D4C; Mon, 4 Dec 2023 14:22:32 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 24/39] kernel/events/uprobes: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 4 Dec 2023 15:21:31 +0100 Message-ID: <20231204142146.91437-25-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:24:29 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361645725848822 X-GMAIL-MSGID: 1784361645725848822 Let's convert __replace_page(). Signed-off-by: David Hildenbrand --- kernel/events/uprobes.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 435aac1d8c272..16731d240e169 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -198,7 +198,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, set_pte_at_notify(mm, addr, pvmw.pte, mk_pte(new_page, vma->vm_page_prot)); - page_remove_rmap(old_page, vma, false); + folio_remove_rmap_pte(old_folio, old_page, vma); if (!folio_mapped(old_folio)) folio_free_swap(old_folio); page_vma_mapped_walk_done(&pvmw); From patchwork Mon Dec 4 14:21:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173400 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2798664vqy; Mon, 4 Dec 2023 06:28:13 -0800 (PST) X-Google-Smtp-Source: AGHT+IG1q0MAZq1w29Co3UwlkV7aR22vDjO5/ApsJ6sUQux02oJEnvQkVu3Fuc0xJli5hS++XJXC X-Received: by 2002:a17:903:2348:b0:1d0:6ffd:8355 with SMTP id c8-20020a170903234800b001d06ffd8355mr1599926plh.96.1701700093268; Mon, 04 Dec 2023 06:28:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701700093; cv=none; d=google.com; s=arc-20160816; b=KeNjATT+GSzWmkT5ECjks5uZQqVKL3Z65GZiT69wAqp+cBCJQ0YP9NW8dHrRJeGByW Nfw+TAe9FXjKEnUn+iDjgyb0FD4WgEGW2fYUmjjr61cYcjy6ZfD9/7xR4CM5zYHYfJJX F9Sy/bde43+GRZujRZFVSeuzwZ6qsprbCjGo6lmSuqrffgnXhWCDg6L31QKjqtmDHm9v /RzliFek/LrdOEGRKhmxrm30fqYb/rUfa29V05WuVB/OX3aM4C+Nx8ulNfcYp/l4tdun iAFTb7ijp3s1xgzLAtsVGfmy+lhqMMVTjbJu5RBbp54ndtlS6uS/ANg+jJ0VSRjHHYM6 tZtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=K8HS/aZd56HSzREOp5VgtDWJgCULJ/ieYaN/iS/Tw6M=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=L9sDouHLuxmZNU51HOzAwctcg5tOhIfgkOcJGdQ2TKuxla7TbztBiZ9doDLJb0RVK+ cMPN5csrE9fyYQH8Y5SCI+nah3mS8goXkWBUv701zqMHexC08OVtzT0hvvcXFk3sjdUr KIrPOIe79ATHRx96RdVXfNRW96dkaJm+t7QebU3/N4Adpm0nU4A2RcTeevjJMR9BuJiF 9SpiGwHiSGyyc7DKdEqLB5OACeHZFtT/YkK+zU5V5OQhOblUDmTgqJOjFrijD2ePrGKX J5t/vhXxiUA1hoHmKR44fmzey9pvt5zHGUHUS19a5O9fflsgfoMD82XePt1aVUAjjA1i 2r+A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=dtKVN5+K; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id b2-20020a170903228200b001cfb1e21bcdsi1781631plh.539.2023.12.04.06.28.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:28:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=dtKVN5+K; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 2DACE8068120; Mon, 4 Dec 2023 06:28:09 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234345AbjLDO2A (ORCPT + 99 others); Mon, 4 Dec 2023 09:28:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52814 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235866AbjLDO1o (ORCPT ); Mon, 4 Dec 2023 09:27:44 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E08847B6 for ; Mon, 4 Dec 2023 06:25:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699905; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K8HS/aZd56HSzREOp5VgtDWJgCULJ/ieYaN/iS/Tw6M=; b=dtKVN5+KRnBbchXs2fYQjQYPf9X5DCzHN4Uho3AUMszXaEM7+anLJ/0YD8+i8RfNdBAJV+ JcHmBzP3aUIzRYFGUyXy+I63CE3ZG3rA1UnJc6JuD6nUVC+hn7Wuwmzw0dFeQ4TNhylYNt xn1e6Sa10rJmA7ISrc/Kdi7lJUBB08M= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-62-lIncdVQIMkCtbPQ0fZWxhw-1; Mon, 04 Dec 2023 09:23:56 -0500 X-MC-Unique: lIncdVQIMkCtbPQ0fZWxhw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0E0B41C06369; Mon, 4 Dec 2023 14:22:36 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 661A82026D4C; Mon, 4 Dec 2023 14:22:34 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 25/39] mm/huge_memory: page_remove_rmap() -> folio_remove_rmap_pmd() Date: Mon, 4 Dec 2023 15:21:32 +0100 Message-ID: <20231204142146.91437-26-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:28:09 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361876911441097 X-GMAIL-MSGID: 1784361876911441097 Let's convert zap_huge_pmd() and set_pmd_migration_entry(). While at it, perform some more folio conversion. Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 332cb6cf99f38..9376c28b0ad29 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1721,7 +1721,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, if (pmd_present(orig_pmd)) { page = pmd_page(orig_pmd); - page_remove_rmap(page, vma, true); + folio_remove_rmap_pmd(page_folio(page), page, vma); VM_BUG_ON_PAGE(page_mapcount(page) < 0, page); VM_BUG_ON_PAGE(!PageHead(page), page); } else if (thp_migration_supported()) { @@ -2134,12 +2134,13 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, page = pfn_swap_entry_to_page(entry); } else { page = pmd_page(old_pmd); - if (!PageDirty(page) && pmd_dirty(old_pmd)) - set_page_dirty(page); - if (!PageReferenced(page) && pmd_young(old_pmd)) - SetPageReferenced(page); - page_remove_rmap(page, vma, true); - put_page(page); + folio = page_folio(page); + if (!folio_test_dirty(folio) && pmd_dirty(old_pmd)) + folio_set_dirty(folio); + if (!folio_test_referenced(folio) && pmd_young(old_pmd)) + folio_set_referenced(folio); + folio_remove_rmap_pmd(folio, page, vma); + folio_put(folio); } add_mm_counter(mm, mm_counter_file(page), -HPAGE_PMD_NR); return; @@ -2294,7 +2295,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, pte_unmap(pte - 1); if (!pmd_migration) - page_remove_rmap(page, vma, true); + folio_remove_rmap_pmd(folio, page, vma); if (freeze) put_page(page); @@ -3235,6 +3236,7 @@ late_initcall(split_huge_pages_debugfs); int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, struct page *page) { + struct folio *folio = page_folio(page); struct vm_area_struct *vma = pvmw->vma; struct mm_struct *mm = vma->vm_mm; unsigned long address = pvmw->address; @@ -3250,14 +3252,14 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, pmdval = pmdp_invalidate(vma, address, pvmw->pmd); /* See page_try_share_anon_rmap(): invalidate PMD first. */ - anon_exclusive = PageAnon(page) && PageAnonExclusive(page); + anon_exclusive = folio_test_anon(folio) && PageAnonExclusive(page); if (anon_exclusive && page_try_share_anon_rmap(page)) { set_pmd_at(mm, address, pvmw->pmd, pmdval); return -EBUSY; } if (pmd_dirty(pmdval)) - set_page_dirty(page); + folio_set_dirty(folio); if (pmd_write(pmdval)) entry = make_writable_migration_entry(page_to_pfn(page)); else if (anon_exclusive) @@ -3274,8 +3276,8 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, if (pmd_uffd_wp(pmdval)) pmdswp = pmd_swp_mkuffd_wp(pmdswp); set_pmd_at(mm, address, pvmw->pmd, pmdswp); - page_remove_rmap(page, vma, true); - put_page(page); + folio_remove_rmap_pmd(folio, page, vma); + folio_put(folio); trace_set_migration_pmd(address, pmd_val(pmdswp)); return 0; From patchwork Mon Dec 4 14:21:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173386 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2796257vqy; Mon, 4 Dec 2023 06:24:36 -0800 (PST) X-Google-Smtp-Source: AGHT+IEEDnLO7ZQMnYUlra5A8on+jnXdxrfntNp3LWPI03vVJ9AXf93xpDX873bTQtSYbBOxsd/N X-Received: by 2002:a17:902:e74c:b0:1d0:5a43:2a48 with SMTP id p12-20020a170902e74c00b001d05a432a48mr1836013plf.31.1701699875850; Mon, 04 Dec 2023 06:24:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699875; cv=none; d=google.com; s=arc-20160816; b=BAYRUvfXs/4WLIpfhv1IA4ALLy1UHPi8GuMDAbw+/PngKMUR6ZghcTfY3EsYjIjKLy C/oZ0NAzH2v5iq1z+FRZwQeugFQfEKn0oEucLdf/216Ure8bzeNG/74tYBzzy3TaEIGR 6VtMrvm1FP00gjFeOSZQqLokGRsPF4FFLabmfJ/UQkRB7nROuYBLsNiPFvcMWfRzAmBR sdhGEgGrBjJRwNSiv97lJ7zWfjGXLvc53tafilm4Uj+gk/tGNvUrwKrbbbF33pDKu/kn qM2sUvrDzW/CMCJMNmOJMzypagDgCuKvc+HbImX8xuIDN+du1EK9GiiBCQ6a+0M6WIxB r+2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KbWS1WI33hCQDyuuf4OMcK1JcfvJiYhRWhxjjruqKZQ=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=lFx0cVU25zcj06Jc0dvEsriuW7KrM4FU6rg9K6SuycjgUaDxc4oRaiRbfb5M/2hH10 EQ9sHQWfnymGqWVMLLt/5MkviBjjXOFUIgCznfHCB7nbkVrVwMHF9PaVIcCyY2+GJ5hv jrwY9M+zS+tGlLjoI71Gdx2xOxk/uj2ry5SwQA3ANObHXNcPr7h+BickpHHe7mKN1JJY MtwAvwOLwoeJVzajP+pj9iosSk+NE9yn0WYJ0Gq6HkEfun0an69bWtv2y3TZEkdBmNKX crBdXi19kX8VCQZ206mkisi30FTqdCfanGseH8Hyx1zbgVvABWCvuEZj1OWBYbhpg+Db 3AEQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="NFVV1l/8"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id 8-20020a17090a08c800b0028681add0c4si3136334pjn.151.2023.12.04.06.24.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:24:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="NFVV1l/8"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 45D75803B344; Mon, 4 Dec 2023 06:24:33 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235888AbjLDOYO (ORCPT + 99 others); Mon, 4 Dec 2023 09:24:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35626 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345591AbjLDOXa (ORCPT ); Mon, 4 Dec 2023 09:23:30 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8701811A for ; Mon, 4 Dec 2023 06:22:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699761; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KbWS1WI33hCQDyuuf4OMcK1JcfvJiYhRWhxjjruqKZQ=; b=NFVV1l/8MKWY9ceMD+eUMGlWAvDSB8IwRFxOophuJWtMjG4F3TOk/aMUKyFRiuidJECV8X bgavYTx+mVv8FEZiwYC5OKUTq85kEIJaWqSPU1+59FoKLO/TLMFWBBuLQ7hN2gFrSKsfFt zLnQA12Pe34otTuespnW5T3pLydxs7k= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-516-d-d9o59fPPm40XM48Es9-g-1; Mon, 04 Dec 2023 09:22:38 -0500 X-MC-Unique: d-d9o59fPPm40XM48Es9-g-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D93A9101A529; Mon, 4 Dec 2023 14:22:37 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 460F42026D4C; Mon, 4 Dec 2023 14:22:36 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 26/39] mm/khugepaged: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 4 Dec 2023 15:21:33 +0100 Message-ID: <20231204142146.91437-27-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:24:33 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361648792783302 X-GMAIL-MSGID: 1784361648792783302 Let's convert __collapse_huge_page_copy_succeeded() and collapse_pte_mapped_thp(). While at it, perform some more folio conversion in __collapse_huge_page_copy_succeeded(). We can get rid of release_pte_page(). Signed-off-by: David Hildenbrand --- mm/khugepaged.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 064654717843e..c2d7438fac22d 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -493,11 +493,6 @@ static void release_pte_folio(struct folio *folio) folio_putback_lru(folio); } -static void release_pte_page(struct page *page) -{ - release_pte_folio(page_folio(page)); -} - static void release_pte_pages(pte_t *pte, pte_t *_pte, struct list_head *compound_pagelist) { @@ -686,6 +681,7 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, spinlock_t *ptl, struct list_head *compound_pagelist) { + struct folio *src_folio; struct page *src_page; struct page *tmp; pte_t *_pte; @@ -707,16 +703,17 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, } } else { src_page = pte_page(pteval); - if (!PageCompound(src_page)) - release_pte_page(src_page); + src_folio = page_folio(src_page); + if (!folio_test_large(src_folio)) + release_pte_folio(src_folio); /* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats - * inside page_remove_rmap(). + * inside folio_remove_rmap_pte(). */ spin_lock(ptl); ptep_clear(vma->vm_mm, address, _pte); - page_remove_rmap(src_page, vma, false); + folio_remove_rmap_pte(src_folio, src_page, vma); spin_unlock(ptl); free_page_and_swap_cache(src_page); } @@ -1619,7 +1616,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * PTE dirty? Shmem page is already dirty; file is read-only. */ ptep_clear(mm, addr, pte); - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(folio, page, vma); nr_ptes++; } From patchwork Mon Dec 4 14:21:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173388 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2796349vqy; Mon, 4 Dec 2023 06:24:43 -0800 (PST) X-Google-Smtp-Source: AGHT+IGIEKBQACHAhuahKCNVvzqqbzf6xrhMpj/oc9fNpXInlSG+6S+bozatLM3ve4FbPqhsLxVv X-Received: by 2002:a05:6a00:180d:b0:6ce:2731:c245 with SMTP id y13-20020a056a00180d00b006ce2731c245mr1540310pfa.52.1701699883414; Mon, 04 Dec 2023 06:24:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699883; cv=none; d=google.com; s=arc-20160816; b=sVaHlLoIgD4dJ++ti35Yn2aTtWBgs9c59W6dDitasX9EJw85E/8ESpAx5eXyU9gUqm Z9/tALWGM4F8mtwRetg6DegoxRX/rCZATweLNN2TrlhpO2qhP0nq2sn+WjDxedNrovbv RqekD3RIuvBq7CgFItCxA2WS7X9bMN2sNWUP6YYC0ARqWl+DHsX/ylPMhKyNrQJKNiM9 gXIe+i9XH/pbgMHz63iLJZlcr69fn9izNT9IpoRL/o9iU3JZFqqj/wkH+5bfSoBjZgy7 BuwSdyqB8ev3LGtm+YSjmaEZj8SsiznsbZ9DHC81OJTkEYct7KbfMAASjwQRqE+NcApU SG1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=IO1sF7V8N+lGKeYvOYG0Xi/4CGC4wNOlZZ1+xeF2jmE=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=NAacMyGBKK1ZvcFmwwYtXO1b50AtrOx8sXe6Hh/mRF43zWgJ0IV0NpmBBOrIAoCBry z5SPltLPN4I540cRHYK32A8TsSeK9VZFNlNa5w4/pO6W7K1169+46XCWD/b5PoIIqKAg CpnOceGrWr0mLSnivCDWaxcEW4XNIH1YZhhu0TB0zuvYM8c4WDi+0ZQnnsG8HQU55TYg DM2ke5fTvnqYMyrD+aF4z//lwq4Cl+123lqczXril/8LS0J+v6aO59AlwKJ2vAAOqYXG kVJfsj5+4pmKY6ntJ+DlNT4Ih8y7TkFTI7pWIru1W9UM3MfJo2nNOeRZMIOaL3HK3Ikt TW2Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=dlIijEOg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id eg10-20020a056a00800a00b006ce4e7a30casi1647566pfb.76.2023.12.04.06.24.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:24:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=dlIijEOg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 684AC8096A7A; Mon, 4 Dec 2023 06:24:39 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345600AbjLDOYV (ORCPT + 99 others); Mon, 4 Dec 2023 09:24:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35670 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235857AbjLDOXg (ORCPT ); Mon, 4 Dec 2023 09:23:36 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 64C3210E4 for ; Mon, 4 Dec 2023 06:22:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699765; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IO1sF7V8N+lGKeYvOYG0Xi/4CGC4wNOlZZ1+xeF2jmE=; b=dlIijEOgxDk40B2weiJNEqwQYsfVxJMvaiGQuCclUd/LumrZ+9yrMd0a8gQZaunCUXqZDM C/+uQPHhDU80t3x5OFEvxw1LhmHgrMasArQgXkRJ91fBfwn7zWYckqz8EofhQ/tXvFMVI9 EF3hfZ05ZeGUdON6pVS9u4EdNqpQPAA= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-662-EsHpcshiPLKnjsrxEa5ujA-1; Mon, 04 Dec 2023 09:22:40 -0500 X-MC-Unique: EsHpcshiPLKnjsrxEa5ujA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C3E941C03D95; Mon, 4 Dec 2023 14:22:39 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2055C2026D68; Mon, 4 Dec 2023 14:22:38 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 27/39] mm/ksm: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 4 Dec 2023 15:21:34 +0100 Message-ID: <20231204142146.91437-28-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:24:39 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361656877543329 X-GMAIL-MSGID: 1784361656877543329 Let's convert replace_page(). Signed-off-by: David Hildenbrand --- mm/ksm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/ksm.c b/mm/ksm.c index 357944588a9a9..c23aed4f1a344 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1279,7 +1279,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, set_pte_at_notify(mm, addr, ptep, newpte); folio = page_folio(page); - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(folio, page, vma); if (!folio_mapped(folio)) folio_free_swap(folio); folio_put(folio); From patchwork Mon Dec 4 14:21:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173387 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2796297vqy; Mon, 4 Dec 2023 06:24:39 -0800 (PST) X-Google-Smtp-Source: AGHT+IHoer95FPVHTb4ZYAvV6qe+u/i3c8lEYtQQQ2j9S2hH5eraTFSyx5WFdEN5Tt7QwUNnhyC3 X-Received: by 2002:a17:90a:8d06:b0:286:6cc0:885c with SMTP id c6-20020a17090a8d0600b002866cc0885cmr1383341pjo.73.1701699879582; Mon, 04 Dec 2023 06:24:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699879; cv=none; d=google.com; s=arc-20160816; b=ra4PtZP+uR8Vv5ksEBfxgvaH8xkB28WdBHZunmNHHFGtJ0Ck+1+NWKyAgpD8iFvO/J U1o064YsZyWvkkQWcAE+l0gdrRknUpmb6R8nr26oJXZyoILd5RhyUdy0tRQoQRo523CY 34h8cLaDCcXNir+lyzPFDNLz4AwKTchR89zzZlrwQbtjf1UC8GMOFkqX5RMmy5VguC8M CLzEcjJprsbgQHhO20yQRJlKASK5zQt/kDtsvMyo/pDM6Lx7VNjRF+5eH0uR+pUMlv4U 5SfgpHQfMvEpsVJeEGSIfd+aNUKWwSry22mvWfXFhB4+vYv7UvqoWHUu5GSF64/PFNx0 T2nw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=nWXNFAD1MN3p2bcOLf/LK1onshg4c1syDBwJJ/+GHnY=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=QXGuhD5YlQDJTOvOa7f1r+kcFvIQN95e2lQrqm24yqgqVip8TwBEm+8ntUnWqeKnC3 JNY5oZvq1ZAUT+r70vjE/1Mgxxe2p0PNqxenTzdVCt2h85Kl/HAtokfJZUiQgzVvXzjF c2IRF24WeFrNhR6vTpY2rIEcWByIk17RjUxdG7yJHqbAN4KDEqCYkAs9v1wC3thcNg1z Tbg7tfIPdL/5PQKLgMocngX8XCHjW89SjGAbzmPeEH+LH61rFoDw+y5jppE+MrAjq/MZ Y5Pl3KbTS5MusOierH7PQBSRR8Basky0MK9gDQvHtN9CKKEeA7f7/JZ+Su6wunU4yS7i jSdw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=dwT+6TRX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id v16-20020a17090a899000b002868673f28dsi3044372pjn.156.2023.12.04.06.24.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:24:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=dwT+6TRX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 6868D80A4A2F; Mon, 4 Dec 2023 06:24:36 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345085AbjLDOYT (ORCPT + 99 others); Mon, 4 Dec 2023 09:24:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59590 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235912AbjLDOXg (ORCPT ); Mon, 4 Dec 2023 09:23:36 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5B53B1B9 for ; Mon, 4 Dec 2023 06:22:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699764; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nWXNFAD1MN3p2bcOLf/LK1onshg4c1syDBwJJ/+GHnY=; b=dwT+6TRX6ItsxFnwBsyZx/EDPJpyZGHZPV2lZD7CgFlX+5ASdL7uw51ezPR1pCoaMUQ1o7 VjPV4ZGpTEkMvv/JRuKi/erQG2NgZgAlB3dPK1Lf0O4XrAZ82Ish6PH+w4SM7p9NtrST0X DbX8RRYyo5b36Ao/0vmcOQkZi8rIfYE= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-208-G1DqHr0HMdSb4ua9mh5Iag-1; Mon, 04 Dec 2023 09:22:42 -0500 X-MC-Unique: G1DqHr0HMdSb4ua9mh5Iag-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A7F1A29AA3AE; Mon, 4 Dec 2023 14:22:41 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0C88C2026D4C; Mon, 4 Dec 2023 14:22:39 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 28/39] mm/memory: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 4 Dec 2023 15:21:35 +0100 Message-ID: <20231204142146.91437-29-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:24:36 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361653043620510 X-GMAIL-MSGID: 1784361653043620510 Let's convert zap_pte_range() and closely-related tlb_flush_rmap_batch(). While at it, perform some more folio conversion in zap_pte_range(). Signed-off-by: David Hildenbrand --- mm/memory.c | 23 +++++++++++++---------- mm/mmu_gather.c | 2 +- 2 files changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 9543b6e2b749b..8c4f98bb617fa 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1425,6 +1425,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent = ptep_get(pte); + struct folio *folio; struct page *page; if (pte_none(ptent)) @@ -1450,21 +1451,22 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } + folio = page_folio(page); delay_rmap = 0; - if (!PageAnon(page)) { + if (!folio_test_anon(folio)) { if (pte_dirty(ptent)) { - set_page_dirty(page); + folio_set_dirty(folio); if (tlb_delay_rmap(tlb)) { delay_rmap = 1; force_flush = 1; } } if (pte_young(ptent) && likely(vma_has_recency(vma))) - mark_page_accessed(page); + folio_mark_accessed(folio); } rss[mm_counter(page)]--; if (!delay_rmap) { - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(folio, page, vma); if (unlikely(page_mapcount(page) < 0)) print_bad_pte(vma, addr, ptent, page); } @@ -1480,6 +1482,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (is_device_private_entry(entry) || is_device_exclusive_entry(entry)) { page = pfn_swap_entry_to_page(entry); + folio = page_folio(page); if (unlikely(!should_zap_page(details, page))) continue; /* @@ -1491,8 +1494,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, WARN_ON_ONCE(!vma_is_anonymous(vma)); rss[mm_counter(page)]--; if (is_device_private_entry(entry)) - page_remove_rmap(page, vma, false); - put_page(page); + folio_remove_rmap_pte(folio, page, vma); + folio_put(folio); } else if (!non_swap_entry(entry)) { /* Genuine swap entry, hence a private anon page */ if (!should_zap_cows(details)) @@ -3210,10 +3213,10 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * threads. * * The critical issue is to order this - * page_remove_rmap with the ptp_clear_flush above. - * Those stores are ordered by (if nothing else,) + * folio_remove_rmap_pte() with the ptp_clear_flush + * above. Those stores are ordered by (if nothing else,) * the barrier present in the atomic_add_negative - * in page_remove_rmap. + * in folio_remove_rmap_pte(); * * Then the TLB flush in ptep_clear_flush ensures that * no process can access the old page before the @@ -3222,7 +3225,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * mapcount is visible. So transitively, TLBs to * old page will be flushed before it can be reused. */ - page_remove_rmap(vmf->page, vma, false); + folio_remove_rmap_pte(old_folio, vmf->page, vma); } /* Free the old page.. */ diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 4f559f4ddd217..604ddf08affed 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -55,7 +55,7 @@ static void tlb_flush_rmap_batch(struct mmu_gather_batch *batch, struct vm_area_ if (encoded_page_flags(enc)) { struct page *page = encoded_page_ptr(enc); - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(page_folio(page), page, vma); } } } From patchwork Mon Dec 4 14:21:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173389 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2796542vqy; Mon, 4 Dec 2023 06:25:03 -0800 (PST) X-Google-Smtp-Source: AGHT+IGP8lSppMKTVV6aq56wopn03cdeGE40M5wLzxqnNLer7SewocU5N012iMfTk/nFSp8elVwC X-Received: by 2002:a05:6a20:2324:b0:18f:97c:617b with SMTP id n36-20020a056a20232400b0018f097c617bmr3830145pzc.120.1701699902727; Mon, 04 Dec 2023 06:25:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699902; cv=none; d=google.com; s=arc-20160816; b=mLfrsEVsR7TcdMeDqskUDoznbV4R4asDDjUVbJlcHUftlD7eu+5qu/N7xPZlYIeWDD jgqLvC8kMocc1RlAqWJ3Phvzu5SCSWTKrE5HbFhEE+YIGw4cqpydU2LU0fGegGoMWYdJ kNVljKitu8i4K0IAUhXH6l2SsptX1SJcJqc+znf1s++HFdD4Bg3HnDw13932+A1dxBTa 4enMqamRes/KerNH2K3wD+4pqhxMG6wCViyrmc5JB5X8LFsfOFLkq8CCq36urOZLWPaQ OZ7ZAqI1PSd0cElIQgoNM9eKMQFcOprzF5ffFWTPBIImpGDpd36cDWQQmO+PjfP+aUjG D8Ag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Nn9t1WIxRvDYJbnLiLQ2NH5amWMopIWzrUXghX6hcqw=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=GrPajMA+yVKfPn+HonPOUwg8dFsd54jsd5OlBKmCG3NGyD9i40Q0DQAewEubGQrJsB 1mCEa+VpAsdYwO/fYo882LFyakgjsgSn3r41gHvKim2Hc8gL2RxudYUB95LWxScqtGzw Ak5qFggV3D1DjrUeYsuhJ3UaQmlPohgdYHMPKUwpSpujIsNqcG1UkKIFc6jryN8gkuBs mg/FojiFM0zVyGMS7LTt9gWgXKKbiyfGHOxNvoKRBM/RrTilKkix051rYdPw8hps5F2B T7ajxPoZ5CAbrEygZ454y3+bDXBMOERHi+Bi/ezxm8m0Mi5R11res8GrO2Hyfhq0SO10 hgYA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="Ow/RMhuU"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id b10-20020a630c0a000000b005b7d9aace9dsi7780374pgl.46.2023.12.04.06.25.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:25:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="Ow/RMhuU"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 9E631805AA2E; Mon, 4 Dec 2023 06:24:57 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345157AbjLDOY0 (ORCPT + 99 others); Mon, 4 Dec 2023 09:24:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59672 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345176AbjLDOXj (ORCPT ); Mon, 4 Dec 2023 09:23:39 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 798F61BFC for ; Mon, 4 Dec 2023 06:22:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699766; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Nn9t1WIxRvDYJbnLiLQ2NH5amWMopIWzrUXghX6hcqw=; b=Ow/RMhuUyTobkGQavgMwZLbniXwskRrAB2zwxIm0MGm5WPB1sAhu0Ep+b0hYlcoS1pS4wC dPr/GRZtgAlUZ2Jz/HyGWg1pGf/uBVFui2uaff0XOBcO6+vokuzJ+0rJ/LdnSFSDEx0XC/ +kFUYfzKRfAV9fIADFvyx2l7EKWVSQM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-657-UIUYp5yyP5WVSRDqwEYaRg-1; Mon, 04 Dec 2023 09:22:44 -0500 X-MC-Unique: UIUYp5yyP5WVSRDqwEYaRg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7C4E68F9204; Mon, 4 Dec 2023 14:22:43 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id E1A552026D68; Mon, 4 Dec 2023 14:22:41 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 29/39] mm/migrate_device: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 4 Dec 2023 15:21:36 +0100 Message-ID: <20231204142146.91437-30-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:24:57 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361677312575476 X-GMAIL-MSGID: 1784361677312575476 Let's convert migrate_vma_collect_pmd(). While at it, perform more folio conversion. Signed-off-by: David Hildenbrand --- mm/migrate_device.c | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 8ac1f79f754a2..c51c99151ebb5 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -107,6 +107,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, for (; addr < end; addr += PAGE_SIZE, ptep++) { unsigned long mpfn = 0, pfn; + struct folio *folio; struct page *page; swp_entry_t entry; pte_t pte; @@ -168,41 +169,43 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, } /* - * By getting a reference on the page we pin it and that blocks + * By getting a reference on the folio we pin it and that blocks * any kind of migration. Side effect is that it "freezes" the * pte. * - * We drop this reference after isolating the page from the lru - * for non device page (device page are not on the lru and thus + * We drop this reference after isolating the folio from the lru + * for non device folio (device folio are not on the lru and thus * can't be dropped from it). */ - get_page(page); + folio = page_folio(page); + folio_get(folio); /* - * We rely on trylock_page() to avoid deadlock between + * We rely on folio_trylock() to avoid deadlock between * concurrent migrations where each is waiting on the others - * page lock. If we can't immediately lock the page we fail this + * folio lock. If we can't immediately lock the folio we fail this * migration as it is only best effort anyway. * - * If we can lock the page it's safe to set up a migration entry - * now. In the common case where the page is mapped once in a + * If we can lock the folio it's safe to set up a migration entry + * now. In the common case where the folio is mapped once in a * single process setting up the migration entry now is an * optimisation to avoid walking the rmap later with * try_to_migrate(). */ - if (trylock_page(page)) { + if (folio_trylock(folio)) { bool anon_exclusive; pte_t swp_pte; flush_cache_page(vma, addr, pte_pfn(pte)); - anon_exclusive = PageAnon(page) && PageAnonExclusive(page); + anon_exclusive = folio_test_anon(folio) && + PageAnonExclusive(page); if (anon_exclusive) { pte = ptep_clear_flush(vma, addr, ptep); if (page_try_share_anon_rmap(page)) { set_pte_at(mm, addr, ptep, pte); - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); mpfn = 0; goto next; } @@ -214,7 +217,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, /* Set the dirty flag on the folio now the pte is gone. */ if (pte_dirty(pte)) - folio_mark_dirty(page_folio(page)); + folio_mark_dirty(folio); /* Setup special migration page table entry */ if (mpfn & MIGRATE_PFN_WRITE) @@ -248,16 +251,16 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, /* * This is like regular unmap: we remove the rmap and - * drop page refcount. Page won't be freed, as we took - * a reference just above. + * drop the folio refcount. The folio won't be freed, as + * we took a reference just above. */ - page_remove_rmap(page, vma, false); - put_page(page); + folio_remove_rmap_pte(folio, page, vma); + folio_put(folio); if (pte_present(pte)) unmapped++; } else { - put_page(page); + folio_put(folio); mpfn = 0; } From patchwork Mon Dec 4 14:21:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173390 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2796559vqy; Mon, 4 Dec 2023 06:25:04 -0800 (PST) X-Google-Smtp-Source: AGHT+IHCdAIub9/s5E4tmYGgOkX26yeUP77/LzJv5CfOVhm3RsDWe/H3yXsEqwqKNg6f73Kw+kTR X-Received: by 2002:a05:6a20:431e:b0:189:2e8f:d34d with SMTP id h30-20020a056a20431e00b001892e8fd34dmr2742905pzk.42.1701699904358; Mon, 04 Dec 2023 06:25:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699904; cv=none; d=google.com; s=arc-20160816; b=YVK3NR+YwlGFebb85+oiatsW0iXlmahrSUPxikpAcGt8ZX/VW51jvJWD7ueH7JkRH8 p5loFEA9bWJ/K8P9LispkYYTisAmWbbLFi3Uc/DnagUkTDTJSd0aKHRvL+YmsJOq8t9J ueo3BUvse5SekDcRcazZukaFxgU/MPbYnhHh4BiHBf4WF74+Lz6xNvjxvcdb0iewfPm9 62UrghhIMjI4JabJT9rF5Y71ZDyHpdJfAp2Rk+tEy0y1+ioeRoKI+MwRK+zJFo+Up2z9 HuWMSOFHF8KIxFkI0e64M8xVQX+oBeFoGD99+rQaTb6jKC76nOgtCBFRXS8/Bx2t2nvc JwRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=wmC0HbMgxf2wn+iz2jDhVN1M/dlanF/535x+xcS2yyw=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=zipXurBzWzS2YP5ftG+9JD5ioJjZvTFigssJ6Krsv32tw7GZ93OKQL8UOlLTb2Xu2I jJOoV/NAxy1aeLvNGHYtA7CtYW+QbpA9EPBeCHKb0H1iuC/JSnUL/jo3JSu03mxy9/T8 uaJHUgUr8SdziKePUXBP5ScjOzkYe325t4YwOMVrAf+7R4KtxdFl/IjJRyGx4TTJS7dw 5Q+8H23MICWqiasfXlJHXNGN3YOgueVtJsWfXa7Lz/2PWRB5nx0iiO7P8Iaa20zOuAQf 0jQVujZOYMilDFVk+G2HVuZc0JbM5CjO9nywJJAH2QSXplD+TRVWnU4RizUdGstl36Ny 76Fw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=NqqW0uZE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id h130-20020a636c88000000b005c66a7d710bsi3353564pgc.459.2023.12.04.06.25.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:25:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=NqqW0uZE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 32945803B366; Mon, 4 Dec 2023 06:25:01 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235882AbjLDOYr (ORCPT + 99 others); Mon, 4 Dec 2023 09:24:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35626 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235870AbjLDOYN (ORCPT ); Mon, 4 Dec 2023 09:24:13 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0756B210A for ; Mon, 4 Dec 2023 06:22:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699773; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wmC0HbMgxf2wn+iz2jDhVN1M/dlanF/535x+xcS2yyw=; b=NqqW0uZEQlsoBadTPWHP/iGo1h4bNQCRSHnmK2iKLGnE0NSSOOkbWn+bSCTIKQtZeIR6Jw UqXNQYT8PJbOF4363mdtsh8GouYZLzhcITRpBUk9TjDypHAij6uXIzaKq4ciLz72AYQzWV IpEYAGnGTeljVlLsqrgvO0scSac/mAw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-668-0wmHc9idP7KPtoNc59ZDjA-1; Mon, 04 Dec 2023 09:22:47 -0500 X-MC-Unique: 0wmHc9idP7KPtoNc59ZDjA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8FEA085A5B5; Mon, 4 Dec 2023 14:22:45 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id BA6A22026D4C; Mon, 4 Dec 2023 14:22:43 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 30/39] mm/rmap: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 4 Dec 2023 15:21:37 +0100 Message-ID: <20231204142146.91437-31-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:25:01 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361678623547496 X-GMAIL-MSGID: 1784361678623547496 Let's convert try_to_unmap_one() and try_to_migrate_one(). Signed-off-by: David Hildenbrand --- mm/rmap.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 50b6909157ac1..4a0114d04ab48 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1598,7 +1598,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, /* * When racing against e.g. zap_pte_range() on another cpu, - * in between its ptep_get_and_clear_full() and page_remove_rmap(), + * in between its ptep_get_and_clear_full() and folio_remove_rmap_*(), * try_to_unmap() may return before page_mapped() has become false, * if page table locking is skipped: use TTU_SYNC to wait for that. */ @@ -1879,7 +1879,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, if (unlikely(folio_test_hugetlb(folio))) hugetlb_remove_rmap(folio); else - page_remove_rmap(subpage, vma, false); + folio_remove_rmap_pte(folio, subpage, vma); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); @@ -1947,7 +1947,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, /* * When racing against e.g. zap_pte_range() on another cpu, - * in between its ptep_get_and_clear_full() and page_remove_rmap(), + * in between its ptep_get_and_clear_full() and folio_remove_rmap_*(), * try_to_migrate() may return before page_mapped() has become false, * if page table locking is skipped: use TTU_SYNC to wait for that. */ @@ -2240,7 +2240,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, if (unlikely(folio_test_hugetlb(folio))) hugetlb_remove_rmap(folio); else - page_remove_rmap(subpage, vma, false); + folio_remove_rmap_pte(folio, subpage, vma); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); @@ -2379,7 +2379,7 @@ static bool page_make_device_exclusive_one(struct folio *folio, * There is a reference on the page for the swap entry which has * been removed, so shouldn't take another. */ - page_remove_rmap(subpage, vma, false); + folio_remove_rmap_pte(folio, subpage, vma); } mmu_notifier_invalidate_range_end(&range); From patchwork Mon Dec 4 14:21:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173391 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2796603vqy; Mon, 4 Dec 2023 06:25:06 -0800 (PST) X-Google-Smtp-Source: AGHT+IEiVxlZf0TmbRqq7wR9AaWcGATYNcifUrug2HiK8y5kXfTRNP1o3gnzeBv7YR1Id0LXI3lg X-Received: by 2002:a05:6a20:9694:b0:18f:97c:384b with SMTP id hp20-20020a056a20969400b0018f097c384bmr4080951pzc.37.1701699906477; Mon, 04 Dec 2023 06:25:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699906; cv=none; d=google.com; s=arc-20160816; b=PTdSWjFFG5+iSovu6Tn8IDYFJBTlo/YoLzSQ/GY6bVSkWocK3hAyiYcir8nsbHI80q 2zHRLllPJo7rD9KiHy2mxYeZFi8JQM/o/lRMElqePp0bfjbwgD2zeID23SRewawxGSEP 7Z4IIE9HDrO61WVCodgzVt5BN18nV6Y2nMzPjfA6aIv9C8t4RdarjGVR6Y6cPTyf6X4j xx2EkvXRcYG1xB2Jdzlv5BQWi8PPLJpYHFoBcs6ZGcyEEbuYCsglWvb6NIxDu1U1Kkjw J+5dorDq9QzHKdT08DPeYdnUtYxdtsyc/2Ccv2YQk8RM7RJlut+vSHpQePHiqfyjocom +8vQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=hz7uaUsseOxwoTHd/q5qxZ6Zu8cM7Yp8tdIii7iwU6Q=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=wC0n6mKhxzdylbIwOLSio1vQ9oCvNECFkSpq1FfDP89QqEpGh2H/ein64L8r6q5kLB Aq9gbe11JytELAHVRvPZ7bKgOVtYkTVAr3HGXjEMAM/csKSYqALMyK7TcKwmVDzLlScc io+UeoBhNSY3xF+47KlufEI52usPkNQkF8H1uUgEIZKwoCheIRlYcPsOGQqLUVJXG1/t x/M/D4M49mO2l9w4GkaszNR/OwRVodojn46Go6lulS05XHN6LOGAO7GODROL0IbCTCQn 7RXn0zl5piYyHhfvQ4xiYZ3C6IdxUaRGSqdqJB8W53S3qWTcmV2Qbv4wJ/xfrNrw0X4m kSmw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="H6z3m8e/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id t9-20020a056a00138900b006c4e73af9ebsi2977391pfg.176.2023.12.04.06.25.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:25:06 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="H6z3m8e/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id ACAA98096A5D; Mon, 4 Dec 2023 06:25:03 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345085AbjLDOYn (ORCPT + 99 others); Mon, 4 Dec 2023 09:24:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60736 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345266AbjLDOYK (ORCPT ); Mon, 4 Dec 2023 09:24:10 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7018F2105 for ; Mon, 4 Dec 2023 06:22:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699772; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hz7uaUsseOxwoTHd/q5qxZ6Zu8cM7Yp8tdIii7iwU6Q=; b=H6z3m8e/e40vfDeqZdC7U3FCCiHPCMX7LDc/oO8/KuJ6P3IEOCSfj8AD039JZ2oWoKM6CB vElj8bSoXa1Y8AMgHlqOAW9bZQy/zjV2OAX7XMlngvjNB23D59wuZt8cblWG8mhNWhu1pC J62Y6ioxuGAkYXI9HPcCRgRXFTCLZL8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-270-2PY8J7EQOv2OTifGixg-kQ-1; Mon, 04 Dec 2023 09:22:48 -0500 X-MC-Unique: 2PY8J7EQOv2OTifGixg-kQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9F0CD8314FB; Mon, 4 Dec 2023 14:22:47 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 01FC72026D68; Mon, 4 Dec 2023 14:22:45 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 31/39] Documentation: stop referring to page_remove_rmap() Date: Mon, 4 Dec 2023 15:21:38 +0100 Message-ID: <20231204142146.91437-32-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:25:03 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361680798879026 X-GMAIL-MSGID: 1784361680798879026 Refer to folio_remove_rmap_*() instaed. Signed-off-by: David Hildenbrand --- Documentation/mm/transhuge.rst | 2 +- Documentation/mm/unevictable-lru.rst | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Documentation/mm/transhuge.rst b/Documentation/mm/transhuge.rst index 9a607059ea11c..cf81272a6b8b6 100644 --- a/Documentation/mm/transhuge.rst +++ b/Documentation/mm/transhuge.rst @@ -156,7 +156,7 @@ Partial unmap and deferred_split_folio() Unmapping part of THP (with munmap() or other way) is not going to free memory immediately. Instead, we detect that a subpage of THP is not in use -in page_remove_rmap() and queue the THP for splitting if memory pressure +in folio_remove_rmap_*() and queue the THP for splitting if memory pressure comes. Splitting will free up unused subpages. Splitting the page right away is not an option due to locking context in diff --git a/Documentation/mm/unevictable-lru.rst b/Documentation/mm/unevictable-lru.rst index 67f1338440a50..b6a07a26b10d5 100644 --- a/Documentation/mm/unevictable-lru.rst +++ b/Documentation/mm/unevictable-lru.rst @@ -486,7 +486,7 @@ munlock the pages if we're removing the last VM_LOCKED VMA that maps the pages. Before the unevictable/mlock changes, mlocking did not mark the pages in any way, so unmapping them required no processing. -For each PTE (or PMD) being unmapped from a VMA, page_remove_rmap() calls +For each PTE (or PMD) being unmapped from a VMA, folio_remove_rmap_*() calls munlock_vma_folio(), which calls munlock_folio() when the VMA is VM_LOCKED (unless it was a PTE mapping of a part of a transparent huge page). @@ -511,7 +511,7 @@ userspace; truncation even unmaps and deletes any private anonymous pages which had been Copied-On-Write from the file pages now being truncated. Mlocked pages can be munlocked and deleted in this way: like with munmap(), -for each PTE (or PMD) being unmapped from a VMA, page_remove_rmap() calls +for each PTE (or PMD) being unmapped from a VMA, folio_remove_rmap_*() calls munlock_vma_folio(), which calls munlock_folio() when the VMA is VM_LOCKED (unless it was a PTE mapping of a part of a transparent huge page). From patchwork Mon Dec 4 14:21:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173392 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2796845vqy; Mon, 4 Dec 2023 06:25:29 -0800 (PST) X-Google-Smtp-Source: AGHT+IEEinRLEeKQYdO2OAq/XPKFUdrmsHLXF5/MSC6GNrdKNtGWqetLUBuSe4n4rIv59kmb2OZw X-Received: by 2002:a17:90b:3545:b0:286:d453:300a with SMTP id lt5-20020a17090b354500b00286d453300amr50339pjb.83.1701699928630; Mon, 04 Dec 2023 06:25:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699928; cv=none; d=google.com; s=arc-20160816; b=PXh6HgUpAF461On7HNeHh0mRfndXZU7zT4gU/lv256Ca9BNQezXUW+nOuKj3NT0hCo 6NV49faXJwsZXnxAcUw67LZe7LSXjd9bqJ/YkKlKx6yI4xhujDawV621EPt9P/Y5gugA RZzCe+P5I87dUBgFEUqNGS2eiftMvi6epZAO4bYZwMKZypcu+GYrD2tdsqjNwknFq67+ EXtmjseXfk6lLl/K24ITHcbgdd5VoDvuf5nuVLQhel7jsCBO/nhjGWMWG3KoojVkUTX+ ea9Mn1dbIIZb4IiyHg+FXRyOkJscFMujcY4125NSWsLkF+8UqMmstcI/UqOKAyBDnUZa 4sXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=nxWJm4ToHrNF66jwBix2CbkD6YpwBU1ELf60FTyZZ+4=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=HIdmC92Lkvgu7tvv+b+wPjvEPTOyQGIJJVZK9YNJpwGsKDCX52wDz+a9YhtmT57ia+ YcS6kzLXHZaDDbC50ww5GYx3zxxNNlo/nmJwJd0DfIWYE3E4nKGTI/0bnPHQb/wAZTd4 SOsyw9/2AUoBj/KKONFdzlsW9FXdBTv/W4cKqEjaE3qWix9nvF6UL7yhSm/qUj/PuoP8 LVq0o7ZqNzDdvhpauTSgsBGk0U1ljb1BAK4BUwV195r03YWrBcQP0aoxF961qAB1zvpw ur9Qk/SVfeao0+fjVAyyUI8q8Imq4BODsGoST4A+/0qIDzezeLtcjJO7u52ACB8uBHsP LgKA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ftV0rr66; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id w22-20020a17090a8a1600b002868d17b86esi2898183pjn.107.2023.12.04.06.25.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:25:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ftV0rr66; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id A50E480411A5; Mon, 4 Dec 2023 06:25:25 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345343AbjLDOYv (ORCPT + 99 others); Mon, 4 Dec 2023 09:24:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52264 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345490AbjLDOYP (ORCPT ); Mon, 4 Dec 2023 09:24:15 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 281B02114 for ; Mon, 4 Dec 2023 06:22:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699774; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nxWJm4ToHrNF66jwBix2CbkD6YpwBU1ELf60FTyZZ+4=; b=ftV0rr66scZ4ka3hzLm2lC+THPiAvYpmGaDqhawWFaiR9kBE7Ctp9AQrwGsGqEdZs/7sd2 o61qZi/8g3gE3wt3Tz7pL6DJV+rSJD23h00168NiHVd5dWEaDPz8kYG8p2zmjjLn0DxVyF 6A1BZI3iGKHaM3tu11okOUYMgH2vEfw= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-91-T-_XZXA9MVC5F6_MoTvAnQ-1; Mon, 04 Dec 2023 09:22:50 -0500 X-MC-Unique: T-_XZXA9MVC5F6_MoTvAnQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7448B3C100C8; Mon, 4 Dec 2023 14:22:49 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id D6C602026D4C; Mon, 4 Dec 2023 14:22:47 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 32/39] mm/rmap: remove page_remove_rmap() Date: Mon, 4 Dec 2023 15:21:39 +0100 Message-ID: <20231204142146.91437-33-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:25:25 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361704753365471 X-GMAIL-MSGID: 1784361704753365471 All callers are gone, let's remove it and some leftover traces. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 4 +--- mm/internal.h | 2 +- mm/memory-failure.c | 4 ++-- mm/rmap.c | 23 ++--------------------- 4 files changed, 6 insertions(+), 27 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index dd4ffb1d8ae04..bdbfb11638e4f 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -239,8 +239,6 @@ void folio_add_file_rmap_ptes(struct folio *, struct page *, unsigned int nr, folio_add_file_rmap_ptes(folio, page, 1, vma) void folio_add_file_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *); -void page_remove_rmap(struct page *, struct vm_area_struct *, - bool compound); void folio_remove_rmap_ptes(struct folio *, struct page *, unsigned int nr, struct vm_area_struct *); #define folio_remove_rmap_pte(folio, page, vma) \ @@ -384,7 +382,7 @@ static inline int page_try_dup_anon_rmap(struct page *page, bool compound, * * This is similar to page_try_dup_anon_rmap(), however, not used during fork() * to duplicate a mapping, but instead to prepare for KSM or temporarily - * unmapping a page (swap, migration) via page_remove_rmap(). + * unmapping a page (swap, migration) via folio_remove_rmap_*(). * * Marking the page shared can only fail if the page may be pinned; device * private pages cannot get pinned and consequently this function cannot fail. diff --git a/mm/internal.h b/mm/internal.h index b61034bd50f5f..43dca750c5afc 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -651,7 +651,7 @@ folio_within_vma(struct folio *folio, struct vm_area_struct *vma) * under page table lock for the pte/pmd being added or removed. * * mlock is usually called at the end of page_add_*_rmap(), munlock at - * the end of page_remove_rmap(); but new anon folios are managed by + * the end of folio_remove_rmap_*(); but new anon folios are managed by * folio_add_lru_vma() calling mlock_new_folio(). */ void mlock_folio(struct folio *folio); diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 660c21859118e..d0251cba84795 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -2317,8 +2317,8 @@ int memory_failure(unsigned long pfn, int flags) * We use page flags to determine what action should be taken, but * the flags can be modified by the error containment action. One * example is an mlocked page, where PG_mlocked is cleared by - * page_remove_rmap() in try_to_unmap_one(). So to determine page status - * correctly, we save a copy of the page flags at this time. + * folio_remove_rmap_*() in try_to_unmap_one(). So to determine page + * status correctly, we save a copy of the page flags at this time. */ page_flags = p->flags; diff --git a/mm/rmap.c b/mm/rmap.c index 4a0114d04ab48..8e86024953c03 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -470,7 +470,7 @@ void __init anon_vma_init(void) /* * Getting a lock on a stable anon_vma from a page off the LRU is tricky! * - * Since there is no serialization what so ever against page_remove_rmap() + * Since there is no serialization what so ever against folio_remove_rmap_*() * the best this function can do is return a refcount increased anon_vma * that might have been relevant to this page. * @@ -487,7 +487,7 @@ void __init anon_vma_init(void) * [ something equivalent to page_mapped_in_vma() ]. * * Since anon_vma's slab is SLAB_TYPESAFE_BY_RCU and we know from - * page_remove_rmap() that the anon_vma pointer from page->mapping is valid + * folio_remove_rmap_*() that the anon_vma pointer from page->mapping is valid * if there is a mapcount, we can dereference the anon_vma after observing * those. */ @@ -1451,25 +1451,6 @@ void folio_add_file_rmap_pmd(struct folio *folio, struct page *page, #endif } -/** - * page_remove_rmap - take down pte mapping from a page - * @page: page to remove mapping from - * @vma: the vm area from which the mapping is removed - * @compound: uncharge the page as compound or small page - * - * The caller needs to hold the pte lock. - */ -void page_remove_rmap(struct page *page, struct vm_area_struct *vma, - bool compound) -{ - struct folio *folio = page_folio(page); - - if (likely(!compound)) - folio_remove_rmap_pte(folio, page, vma); - else - folio_remove_rmap_pmd(folio, page, vma); -} - static __always_inline void __folio_remove_rmap(struct folio *folio, struct page *page, unsigned int nr_pages, struct vm_area_struct *vma, enum rmap_mode mode) From patchwork Mon Dec 4 14:21:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173397 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2797680vqy; Mon, 4 Dec 2023 06:26:43 -0800 (PST) X-Google-Smtp-Source: AGHT+IGdGSdTm73bamTBmGix9vBU/GrNbvP9Nm+cZLXdMKGxRu2/kJgnyBsXgQ6y3qYv9OHHRn0Q X-Received: by 2002:a05:6a20:4327:b0:187:4e8c:ac5c with SMTP id h39-20020a056a20432700b001874e8cac5cmr5213683pzk.1.1701700003231; Mon, 04 Dec 2023 06:26:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701700003; cv=none; d=google.com; s=arc-20160816; b=aytWJAJis9Db4RFWAChhTnn5KA9WxDb9YqBdXzup8Fjt9G43kzkbn9IWGAt4m3OhDZ rZ2WjWly+P7EOUmWYM0EQlOwTIV8/zFeRyB47u0dFY9c8U19l+7Ve55fydWnTW36Ul2Q MjrkKX78c0W4xnlYKKvbH7lvIfJdbHfQWXlRrjK8gKYI1bAg2JYx6nBaqCMN8p3tLkN7 JGCdB0no6R0+EJWzqhJCb5dD2BhukVNisBKDxYOf1H0Do0KdJ9RPRV4PwtZTztiyunxn vcvPJSaSYchemqKMICP1zZaSmqv2u9lWMRFilHpt6HSRTZqbi19Kdoi0qtyVd9T+QREp fg/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=fiJf6xaTCn9hTJUSpG0inAWeRQ8SaM/ZLU1i30jXpOI=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=ln2eTIXViLdxKLTzxBgatyF8VofnOX5iyBk3dveHk257Mpfv5GPC8E0iBaZGaRCMFz vRf/zkz8Z1P+Ewvsmnu/QogEPlRA7SUEgHne3jY4XCnM8rP/WXA7aZ31rpmEzhtWT6IX ddtXDRKfm2LizPJ4Q1Us5XbMutip64cyexlKsjQh5xk7C87lOzDqgyXVdGtEgPwZzTf2 A0dJ7pINi+on+ZpNPoQGXprcAI1xxp7RW7A3Z4vr1qv3yDp5+gHp9L0Tzln+bs0reiPP Eu1HZCDGzOhhOd3ZBnQ82mNPgGmY0o7N9uBTQVu2c1Cm6/CpZerYpDJSgl+zbIO9FQvc tqrA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=asngjzMl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id ei48-20020a056a0080f000b006cb901a87c4si3178821pfb.376.2023.12.04.06.26.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:26:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=asngjzMl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 2C0ED808D8EB; Mon, 4 Dec 2023 06:26:36 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234200AbjLDO01 (ORCPT + 99 others); Mon, 4 Dec 2023 09:26:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35704 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236044AbjLDO0B (ORCPT ); Mon, 4 Dec 2023 09:26:01 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A49F30C1 for ; Mon, 4 Dec 2023 06:23:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699814; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fiJf6xaTCn9hTJUSpG0inAWeRQ8SaM/ZLU1i30jXpOI=; b=asngjzMlhUU39DZx8hK0LeWM/KllccQY6HZzT294anYLK9LdIA71uEj3s5gGq8qD1epNdq MYnfJTpXsM+Lbc3/MquA/QILABOPLl+uMXI9HFcEpmjFHfffi/z3rVYQaEdOanYodWNjJh en8w42Ajh2Jq/HkqogoQnNp/q9YWQJE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-164-gWsSkqhfMSOvilQ5o3bMzQ-1; Mon, 04 Dec 2023 09:23:25 -0500 X-MC-Unique: gWsSkqhfMSOvilQ5o3bMzQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 53AF1811E82; Mon, 4 Dec 2023 14:22:51 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id AD95F2026D4C; Mon, 4 Dec 2023 14:22:49 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 33/39] mm/rmap: convert page_dup_file_rmap() to folio_dup_file_rmap_[pte|ptes|pmd]() Date: Mon, 4 Dec 2023 15:21:40 +0100 Message-ID: <20231204142146.91437-34-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:26:36 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361782665738445 X-GMAIL-MSGID: 1784361782665738445 Let's convert page_dup_file_rmap() like the other rmap functions. As there is only a single caller, convert that single caller right away and remove page_dup_file_rmap(). Add folio_dup_file_rmap_ptes() right away, we want to perform rmap baching during fork() soon. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 56 ++++++++++++++++++++++++++++++++++++++++---- mm/memory.c | 2 +- 2 files changed, 52 insertions(+), 6 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index bdbfb11638e4f..21d72cc602adc 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -303,6 +303,57 @@ static inline void hugetlb_remove_rmap(struct folio *folio) atomic_dec(&folio->_entire_mapcount); } +static inline void __folio_dup_rmap(struct folio *folio, struct page *page, + unsigned int nr_pages, enum rmap_mode mode) +{ + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); + + if (likely(mode == RMAP_MODE_PTE)) { + do { + atomic_inc(&page->_mapcount); + } while (page++, --nr_pages > 0); + } else if (mode == RMAP_MODE_PMD) { + atomic_inc(&folio->_entire_mapcount); + } +} + +/** + * folio_dup_file_rmap_ptes - duplicate PTE mappings of a page range of a folio + * @folio: The folio to duplicate the mappings of + * @page: The first page to duplicate the mappings of + * @nr_pages: The number of pages of which the mapping will be duplicated + * + * The page range of the folio is defined by [page, page + nr_pages) + * + * The caller needs to hold the page table lock. + */ +static inline void folio_dup_file_rmap_ptes(struct folio *folio, + struct page *page, unsigned int nr_pages) +{ + __folio_dup_rmap(folio, page, nr_pages, RMAP_MODE_PTE); +} +#define folio_dup_file_rmap_pte(folio, page) \ + folio_dup_file_rmap_ptes(folio, page, 1) + +/** + * folio_dup_file_rmap_pmd - duplicate a PMD mapping of a page range of a folio + * @folio: The folio to duplicate the mapping of + * @page: The first page to duplicate the mapping of + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock. + */ +static inline void folio_dup_file_rmap_pmd(struct folio *folio, + struct page *page) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_dup_rmap(folio, page, HPAGE_PMD_NR, RMAP_MODE_PTE); +#else + WARN_ON_ONCE(true); +#endif +} + static inline void __page_dup_rmap(struct page *page, bool compound) { VM_WARN_ON(folio_test_hugetlb(page_folio(page))); @@ -317,11 +368,6 @@ static inline void __page_dup_rmap(struct page *page, bool compound) } } -static inline void page_dup_file_rmap(struct page *page, bool compound) -{ - __page_dup_rmap(page, compound); -} - /** * page_try_dup_anon_rmap - try duplicating a mapping of an already mapped * anonymous page diff --git a/mm/memory.c b/mm/memory.c index 8c4f98bb617fa..eaab6a2e14eba 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -961,7 +961,7 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, rss[MM_ANONPAGES]++; } else if (page) { folio_get(folio); - page_dup_file_rmap(page, false); + folio_dup_file_rmap_pte(folio, page); rss[mm_counter_file(page)]++; } From patchwork Mon Dec 4 14:21:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173393 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2796889vqy; Mon, 4 Dec 2023 06:25:33 -0800 (PST) X-Google-Smtp-Source: AGHT+IErQvNCe3oUhXmvXMGxeGv+eHlgZy6UxKRZmVnwVo9s7DPTwIIw15aDvwbJwIAO0pDhL8Sz X-Received: by 2002:a05:6a20:160d:b0:138:2fb8:6c48 with SMTP id l13-20020a056a20160d00b001382fb86c48mr2437503pzj.8.1701699933136; Mon, 04 Dec 2023 06:25:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699933; cv=none; d=google.com; s=arc-20160816; b=ue705lLob/9g9DY5ySB8DixdpSX8SdyAulvXfTA07aCDfOOsNBOBpoXxgofX/G1Vq9 J9Ya6nUjgnUp9GcPO27s4OHcjEBdGls5eXqhGUZHekRUxfG1JC9iL/9gIEknliTBVNyU NcUA1hqk2MF9SqvjpVrYT9SSJC5onqQmrlIfkCfr6wgsRDhhQiVxujw9BiZRyI+tmX8F TPHx35droxuvlPJWvMB2i9v2KIJ2KcySOK9eIfnq6E/aXKRnnaMh/qOrU0JX3JOWfSiP vjdwt+meA5ffJTYvsOMMx3UZAy2fqxOfCo6yCFJfv6tSYf314tnM822LsLjf6N6EIHMJ g+XA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=8XTtIQTcL5nOGoGd5kvk/Mh9+sDV+bVrih5Rph4+Al0=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=td7j1/OkWEPgHG/tPLjMbK2hIGYhCVC6STBTNsFXkDHyLiw8GvX/0HI3bj2EA23gcM 3K7eg3wHMqQAKfpQcIAH6Vo9MsJfT4bYxUfpx/8jTOzo5qrxwCn7xId1NTZ81ZLXhJJp ncTolkcnNIMp0Ym/a/iG7Lt7jUjNxvaTnCkqTlFdMD3fqmAsLUuyvUMziKrTXSLFJPtz wSyUMEPPxbIaqpF98AaKijQKi8y+gn0nyPY/3kn3guHlM276ECc31IVSTs63uedrg6E0 5cBfeWhA4+be3o2NJ6z/OWbLtYv9i5hcV4YenlaTfTY6Sljl/dW45riNLzBhCHKuOPyv vJ4Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=fECgHZUM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id fb6-20020a056a002d8600b006cddc343ceesi7960157pfb.181.2023.12.04.06.25.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:25:33 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=fECgHZUM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id DE1BA80A8B55; Mon, 4 Dec 2023 06:25:31 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235851AbjLDOZV (ORCPT + 99 others); Mon, 4 Dec 2023 09:25:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235843AbjLDOYi (ORCPT ); Mon, 4 Dec 2023 09:24:38 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3F98C212B for ; Mon, 4 Dec 2023 06:23:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699780; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8XTtIQTcL5nOGoGd5kvk/Mh9+sDV+bVrih5Rph4+Al0=; b=fECgHZUMkKyLjXCDKRWCQKcgGRxWs9X6ptcTk9U1RKxMmTqEoSFNXF8Ntyqb9OTN41P0JV 1Y8prh6i4FShBXR0tEC/6jh5OLzdN/DP9dIdNn72KVTMfIczlzbicMneWhnRLIOermKb+N quTVG+uZ3wdpI0RJHebCATvHznS4Ip4= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-353-hT6yNEVjPHOWv0XESwmOZA-1; Mon, 04 Dec 2023 09:22:53 -0500 X-MC-Unique: hT6yNEVjPHOWv0XESwmOZA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2B9A53813BCC; Mon, 4 Dec 2023 14:22:53 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8CD092026D4C; Mon, 4 Dec 2023 14:22:51 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 34/39] mm/rmap: introduce folio_try_dup_anon_rmap_[pte|ptes|pmd]() Date: Mon, 4 Dec 2023 15:21:41 +0100 Message-ID: <20231204142146.91437-35-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:25:32 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361708733898725 X-GMAIL-MSGID: 1784361708733898725 The last user of page_needs_cow_for_dma() and __page_dup_rmap() are gone, remove them. Add folio_try_dup_anon_rmap_ptes() right away, we want to perform rmap baching during fork() soon. Signed-off-by: David Hildenbrand Signed-off-by: David Hildenbrand --- include/linux/mm.h | 6 -- include/linux/rmap.h | 145 +++++++++++++++++++++++++++++-------------- 2 files changed, 100 insertions(+), 51 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 24c1c7c5a99c0..f7565b35ae931 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1964,12 +1964,6 @@ static inline bool folio_needs_cow_for_dma(struct vm_area_struct *vma, return folio_maybe_dma_pinned(folio); } -static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma, - struct page *page) -{ - return folio_needs_cow_for_dma(vma, page_folio(page)); -} - /** * is_zero_page - Query if a page is a zero page * @page: The page to query diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 21d72cc602adc..84439f7720c62 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -354,68 +354,123 @@ static inline void folio_dup_file_rmap_pmd(struct folio *folio, #endif } -static inline void __page_dup_rmap(struct page *page, bool compound) +static inline int __folio_try_dup_anon_rmap(struct folio *folio, + struct page *page, unsigned int nr_pages, + struct vm_area_struct *src_vma, enum rmap_mode mode) { - VM_WARN_ON(folio_test_hugetlb(page_folio(page))); + int i; - if (compound) { - struct folio *folio = (struct folio *)page; + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); - VM_BUG_ON_PAGE(compound && !PageHead(page), page); - atomic_inc(&folio->_entire_mapcount); - } else { - atomic_inc(&page->_mapcount); + /* + * No need to check+clear for already shared PTEs/PMDs of the folio. + * This includes PTE mappings of (order-0) KSM folios. + */ + if (likely(mode == RMAP_MODE_PTE)) { + for (i = 0; i < nr_pages; i++) { + if (PageAnonExclusive(page + i)) + goto clear; + } + } else if (mode == RMAP_MODE_PMD) { + if (PageAnonExclusive(page)) + goto clear; } + goto dup; + +clear: + /* + * If this folio may have been pinned by the parent process, + * don't allow to duplicate the mappings but instead require to e.g., + * copy the subpage immediately for the child so that we'll always + * guarantee the pinned folio won't be randomly replaced in the + * future on write faults. + */ + if (likely(!folio_is_device_private(folio) && + unlikely(folio_needs_cow_for_dma(src_vma, folio)))) + return -EBUSY; + + if (likely(mode == RMAP_MODE_PTE)) { + for (i = 0; i < nr_pages; i++) + ClearPageAnonExclusive(page + i); + } else if (mode == RMAP_MODE_PMD) { + ClearPageAnonExclusive(page); + } + +dup: + __folio_dup_rmap(folio, page, nr_pages, mode); + return 0; } /** - * page_try_dup_anon_rmap - try duplicating a mapping of an already mapped - * anonymous page - * @page: the page to duplicate the mapping for - * @compound: the page is mapped as compound or as a small page - * @vma: the source vma + * folio_try_dup_anon_rmap_ptes - try duplicating PTE mappings of a page range + * of a folio + * @folio: The folio to duplicate the mappings of + * @page: The first page to duplicate the mappings of + * @nr_pages: The number of pages of which the mapping will be duplicated + * @src_vma: The vm area from which the mappings are duplicated * - * The caller needs to hold the PT lock and the vma->vma_mm->write_protect_seq. + * The page range of the folio is defined by [page, page + nr_pages) * - * Duplicating the mapping can only fail if the page may be pinned; device - * private pages cannot get pinned and consequently this function cannot fail. + * The caller needs to hold the page table lock and the + * vma->vma_mm->write_protect_seq. + * + * Duplicating the mappings can only fail if the folio may be pinned; device + * private folios cannot get pinned and consequently this function cannot fail. + * + * If duplicating the mappings succeeded, the duplicated PTEs have to be R/O in + * the parent and the child. They must *not* be writable after this call. + * + * Returns 0 if duplicating the mappings succeeded. Returns -EBUSY otherwise. + */ +static inline int folio_try_dup_anon_rmap_ptes(struct folio *folio, + struct page *page, unsigned int nr_pages, + struct vm_area_struct *src_vma) +{ + return __folio_try_dup_anon_rmap(folio, page, nr_pages, src_vma, + RMAP_MODE_PTE); +} +#define folio_try_dup_anon_rmap_pte(folio, page, vma) \ + folio_try_dup_anon_rmap_ptes(folio, page, 1, vma) + +/** + * folio_try_dup_anon_rmap_pmd - try duplicating a PMD mapping of a page range + * of a folio + * @folio: The folio to duplicate the mapping of + * @page: The first page to duplicate the mapping of + * @src_vma: The vm area from which the mapping is duplicated + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock and the + * vma->vma_mm->write_protect_seq. * - * If duplicating the mapping succeeds, the page has to be mapped R/O into - * the parent and the child. It must *not* get mapped writable after this call. + * Duplicating the mapping can only fail if the folio may be pinned; device + * private folios cannot get pinned and consequently this function cannot fail. + * + * If duplicating the mapping succeeds, the duplicated PMD has to be R/O in + * the parent and the child. They must *not* be writable after this call. * * Returns 0 if duplicating the mapping succeeded. Returns -EBUSY otherwise. */ +static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, + struct page *page, struct vm_area_struct *src_vma) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + return __folio_try_dup_anon_rmap(folio, page, HPAGE_PMD_NR, src_vma, + RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + static inline int page_try_dup_anon_rmap(struct page *page, bool compound, struct vm_area_struct *vma) { - VM_BUG_ON_PAGE(!PageAnon(page), page); - - /* - * No need to check+clear for already shared pages, including KSM - * pages. - */ - if (!PageAnonExclusive(page)) - goto dup; - - /* - * If this page may have been pinned by the parent process, - * don't allow to duplicate the mapping but instead require to e.g., - * copy the page immediately for the child so that we'll always - * guarantee the pinned page won't be randomly replaced in the - * future on write faults. - */ - if (likely(!is_device_private_page(page) && - unlikely(page_needs_cow_for_dma(vma, page)))) - return -EBUSY; + struct folio *folio = page_folio(page); - ClearPageAnonExclusive(page); - /* - * It's okay to share the anon page between both processes, mapping - * the page R/O into both processes. - */ -dup: - __page_dup_rmap(page, compound); - return 0; + if (likely(!compound)) + return folio_try_dup_anon_rmap_pte(folio, page, vma); + return folio_try_dup_anon_rmap_pmd(folio, page, vma); } /** From patchwork Mon Dec 4 14:21:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173394 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2796961vqy; Mon, 4 Dec 2023 06:25:40 -0800 (PST) X-Google-Smtp-Source: AGHT+IGm/qXkoTnaq5dOrVzBr5Lf6wND1K0epEsD9d9wIkb2ojGbre9M5i9fKxK/Tszcnil4Em0W X-Received: by 2002:a92:290f:0:b0:35d:59a2:2d7 with SMTP id l15-20020a92290f000000b0035d59a202d7mr5140492ilg.119.1701699940076; Mon, 04 Dec 2023 06:25:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699940; cv=none; d=google.com; s=arc-20160816; b=kMwO6u9HrzGDbZ6pS/47x4whrxA0CPfZ8N7XPaXrEykxwz6mhtKYLbt4W9FWJB1nyx GBxE0ttfEc9dlsze+egeR2pX4xRUVd0i8WsQpyR20m2JkXGV/VqBtnWUAAARvJzmvoMp 4HjprAbxgLQuGrt9FtM68h7aKTMqllkhRxYB/JWJKqU9YosukPiY3T31zmeySha7TLn4 ADVDQksA8e6CKBT5jQQwBhEtvyTNraHDVhBJg0Eehi7SU/4euiO6oNdlI3E43Jb89+PS 18WXaJyWHth6J3KfBbhpIi3izVLOqoy0FeBC2502QcTJ2bKO5wHQiSGrcV7ZtBYPWQkl pw3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QtlsU4Oj5pegnlPsv+IBcEoRKm4An7qodS6gdDA58Hg=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=EXEMr/30jTevO4R2fOMk8GcfSfLwhRXK9ytqtevJFpN2Zmyy9/xgIO01eq4qMVo4n6 9aucFhWErPTRsHc2xpGgXAqfHrD+hAfvkO6/Anr8EqP1gVDohsM8FEnVn10qnBHMEjmr hHCsT98/6WZSAadYtnCSLtfTyHKRNphEezbBWq6v0nQoMMuWf/Sl+i9c/HbBEISN7NeH f5mspQIhu4jD1Pt20xLjjv1Y9DroCzytR+sGUqOsPteuAH4ppveBR7xjERVLR6pM20/S 1R/kXpsCJm+30e0FcSIw3eaKrYwtp77SSFW8qd7gA+yMtl/yaPO8suA5f1J7xcULjHyx UHFw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=cYJIXjrH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id q20-20020a656854000000b00577960a815csi3699736pgt.188.2023.12.04.06.25.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:25:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=cYJIXjrH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 6BC5680A1E38; Mon, 4 Dec 2023 06:25:31 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345083AbjLDOZK (ORCPT + 99 others); Mon, 4 Dec 2023 09:25:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52188 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235910AbjLDOYa (ORCPT ); Mon, 4 Dec 2023 09:24:30 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B66711713 for ; Mon, 4 Dec 2023 06:22:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699779; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QtlsU4Oj5pegnlPsv+IBcEoRKm4An7qodS6gdDA58Hg=; b=cYJIXjrHSW88QPY+VPCYkDv3yAVyyv/HZGn3gWiR+XXJyfc8tOwoUES9YlsWzZEGtIY7mp XvN/g00743Uomd5q+rWiLOAeXkyE+tPMvpBYRkbUWT6wboP5G8h4ubYTYEH8z4bNzNj3g0 smyVAEhycuY19q5d8sD1vmVPzhgheAw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-451-GtUO8C-8PWedYvC2iAxb4A-1; Mon, 04 Dec 2023 09:22:55 -0500 X-MC-Unique: GtUO8C-8PWedYvC2iAxb4A-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 483868314FC; Mon, 4 Dec 2023 14:22:55 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6440D2026D4C; Mon, 4 Dec 2023 14:22:53 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 35/39] mm/huge_memory: page_try_dup_anon_rmap() -> folio_try_dup_anon_rmap_pmd() Date: Mon, 4 Dec 2023 15:21:42 +0100 Message-ID: <20231204142146.91437-36-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:25:31 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361716407300818 X-GMAIL-MSGID: 1784361716407300818 Let's convert copy_huge_pmd() and fixup the comment in copy_huge_pud(). While at it, perform more folio conversion in copy_huge_pmd(). Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9376c28b0ad29..138e1e62790be 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1098,6 +1098,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, { spinlock_t *dst_ptl, *src_ptl; struct page *src_page; + struct folio *src_folio; pmd_t pmd; pgtable_t pgtable = NULL; int ret = -ENOMEM; @@ -1164,11 +1165,12 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, src_page = pmd_page(pmd); VM_BUG_ON_PAGE(!PageHead(src_page), src_page); + src_folio = page_folio(src_page); - get_page(src_page); - if (unlikely(page_try_dup_anon_rmap(src_page, true, src_vma))) { + folio_get(src_folio); + if (unlikely(folio_try_dup_anon_rmap_pmd(src_folio, src_page, src_vma))) { /* Page maybe pinned: split and retry the fault on PTEs. */ - put_page(src_page); + folio_put(src_folio); pte_free(dst_mm, pgtable); spin_unlock(src_ptl); spin_unlock(dst_ptl); @@ -1277,8 +1279,8 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, } /* - * TODO: once we support anonymous pages, use page_try_dup_anon_rmap() - * and split if duplicating fails. + * TODO: once we support anonymous pages, use + * folio_try_dup_anon_rmap_*() and split if duplicating fails. */ pudp_set_wrprotect(src_mm, addr, src_pud); pud = pud_mkold(pud_wrprotect(pud)); From patchwork Mon Dec 4 14:21:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173395 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2797082vqy; Mon, 4 Dec 2023 06:25:49 -0800 (PST) X-Google-Smtp-Source: AGHT+IHfqShBmlH+JsJ7xj9BiObIPJCqIsbMiCEhY14QfY9xNP+SGQoY4Oq32HnPmQ6gvlbbcTCO X-Received: by 2002:a17:90b:164e:b0:286:9b7a:93c8 with SMTP id il14-20020a17090b164e00b002869b7a93c8mr1730449pjb.53.1701699949327; Mon, 04 Dec 2023 06:25:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699949; cv=none; d=google.com; s=arc-20160816; b=r6OpmFRLR4JHPnd/JYPLJNwRkaBTwBBJe6Og08GPeW161CKj1DoWu48HCcdD243Y1+ iE2SjNgxwymjpODFc8rLYU3LBA+PZK8A5bgJzi4ue9dWQdA26jfASOqLwv6ztSFrbAF9 xVUTbY8t9gbPojB6QWE72+YRX1Prg1S07NRuPETcXCxumfV3MzuoJkF+BE6mIHDCCEyb tDjpiNXsUafdVIzBZQtC+bFS8njyG2D2qkkAoUM2FuUl0mdGVwU7Kf5+q3Y20u1Jl8VY Wa3mla5W3iLq7ezxQt/SY1WVNCwehIH5RwM0NDW90HxdS7GJmro5WbSOHiWb4KKdGsg2 Qb+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eN0JAnAR2NtaQAP/A7ZBLwE/xmH/RpNbwyJWy6g296A=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=0o8vAh6qPsQlzFU6sYfspkRSgn30NyJ/P+psVjFMzjfoJJ619Y0C6pC3CIqvjQiIZX fUSO2BwSDtEua+hl6qwbK3xpIGXS+B36ImY2+cKjQDmD9HkNN+KPFFCItG2Mp8Dl7P02 3g+5HeZbcm+p57OG9tFgpi1cN2yfwqHyN/PI0Vv26NBWHK4neQ/T1HPapsgOV8dsWllN jtBCMYMQ5Ov7YEL8QfqacihK7Mpdu1EMqYfxdey8Rnm3cq/7ja7m3yHiYIovqmFqRXeW QB7932jYl83w0ukus75mfTAjeFUEELiLEfmh6dbAMhBfhnbGAcrCXKPuxmCVzZLKs5rx w5mw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=GmhkqdLt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id e12-20020a17090a728c00b00285c71e384csi5644601pjg.88.2023.12.04.06.25.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:25:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=GmhkqdLt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 529778078E38; Mon, 4 Dec 2023 06:25:37 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345197AbjLDOZY (ORCPT + 99 others); Mon, 4 Dec 2023 09:25:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55748 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235935AbjLDOYi (ORCPT ); Mon, 4 Dec 2023 09:24:38 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 05DF9172B for ; Mon, 4 Dec 2023 06:23:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699781; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eN0JAnAR2NtaQAP/A7ZBLwE/xmH/RpNbwyJWy6g296A=; b=GmhkqdLtKbk9dHaz8V9Ib4LrZwXk33b/eb2DaDMebvh3S3PB11IBHuuFN7qVZZK5tRaxaq L5LgojZRK7/gb5XzDdqzWy+dR3loy1CDtOHgNgCT6ig12kvi7ghOzIMgJ+EFBtj797juId VSeGSUkq9gcW6Iyy6OuFKxBY3c74J/4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-340-G29icLgwOhGGlLJZvf5FDw-1; Mon, 04 Dec 2023 09:22:58 -0500 X-MC-Unique: G29icLgwOhGGlLJZvf5FDw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2DD81101AA6F; Mon, 4 Dec 2023 14:22:57 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7EE922026D4C; Mon, 4 Dec 2023 14:22:55 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 36/39] mm/memory: page_try_dup_anon_rmap() -> folio_try_dup_anon_rmap_pte() Date: Mon, 4 Dec 2023 15:21:43 +0100 Message-ID: <20231204142146.91437-37-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:25:37 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361726266264104 X-GMAIL-MSGID: 1784361726266264104 Let's convert copy_nonpresent_pte(). While at it, perform some more folio conversion. Signed-off-by: David Hildenbrand --- mm/memory.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index eaab6a2e14eba..ad6da8168e461 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -781,6 +781,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, unsigned long vm_flags = dst_vma->vm_flags; pte_t orig_pte = ptep_get(src_pte); pte_t pte = orig_pte; + struct folio *folio; struct page *page; swp_entry_t entry = pte_to_swp_entry(orig_pte); @@ -825,6 +826,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, } } else if (is_device_private_entry(entry)) { page = pfn_swap_entry_to_page(entry); + folio = page_folio(page); /* * Update rss count even for unaddressable pages, as @@ -835,10 +837,10 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * for unaddressable pages, at some point. But for now * keep things as they are. */ - get_page(page); + folio_get(folio); rss[mm_counter(page)]++; /* Cannot fail as these pages cannot get pinned. */ - BUG_ON(page_try_dup_anon_rmap(page, false, src_vma)); + folio_try_dup_anon_rmap_pte(folio, page, src_vma); /* * We do not preserve soft-dirty information, because so @@ -952,7 +954,7 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, * future. */ folio_get(folio); - if (unlikely(page_try_dup_anon_rmap(page, false, src_vma))) { + if (unlikely(folio_try_dup_anon_rmap_pte(folio, page, src_vma))) { /* Page may be pinned, we have to copy. */ folio_put(folio); return copy_present_page(dst_vma, src_vma, dst_pte, src_pte, From patchwork Mon Dec 4 14:21:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173426 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2816651vqy; Mon, 4 Dec 2023 06:57:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IHaLwV/POUpzaL/DhL5E26OmpcZrRBp3TexgLbuJobC0vxlu8/vWuw6bEuC9JpadUDFyore X-Received: by 2002:a05:6a20:5497:b0:18c:3065:8048 with SMTP id i23-20020a056a20549700b0018c30658048mr1974338pzk.42.1701701852282; Mon, 04 Dec 2023 06:57:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701701852; cv=none; d=google.com; s=arc-20160816; b=NPKyE7ClPOnFeQQ1cEH7ZxSXQr6JbmRiFmluPpG6vOvoqI0nIaCs7hm8+ohherb02p aQaDkjZZeBgNMZeTVdBJdwmMTrGaXg5d9DA5Lb+lKmnL1+rJP62pw09RTr8TalrmPS4w gJsCzbzvq/qBC3DSctTP7GbzOKOJkILBeGhFRUZKuRTjmTRbuY4DboawOHHV9zpnBWlM nnOM3jP9WIebpZm1Yew2Q7m0SlDRX2cJ2h2om6uk5Ok7AzHXgnb1300O0jVP7oVXdi00 q7jUFvyq3UrEWJYnw67jU7P0QGnHgpcjv/WbvLp+ttDdUuYFCqddMihHVpVtlEOApQ/A 5uYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=v4HAiul36EO+vclRiaij572sHtWWM/DwfWC2VtVH9HE=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=jWkPZdDnfJRJEu/EPgp/Zkp1KvJRh0YlSPBOGvLq7aYIPGW7/3jK9MkvN3KYH9MHH+ T4L900Ye5rfoWzNb1zFXODozUmcywa5KqGJcW/QE4AtbshJ39ljU0PiMi7oFNu8fiXrH vYVjOJUg6m2P6dw8NTGedT6Juky7D1Y0qgyfb8srN4kY9AthDFRmgGME+MVU5+ENsbqI Eym/2/+24feDhPXYmdrEnNWpXrIu3wMHkfWHvKgCFRwdM2d4tBl966iOs0WqQJlJSCse jNGGrH+gadwhFxFMmiWmkVlSS1FG+fBAjqwxhIo6xtRZUWaJIS0zqk/cxJVwDtODKZLn aEHw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=a76vBY38; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id m27-20020a634c5b000000b005c2185da898si7915884pgl.832.2023.12.04.06.57.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:57:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=a76vBY38; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 568FB805B2D9; Mon, 4 Dec 2023 06:57:29 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233980AbjLDO4x (ORCPT + 99 others); Mon, 4 Dec 2023 09:56:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60722 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235947AbjLDOYl (ORCPT ); Mon, 4 Dec 2023 09:24:41 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BDFAB2130 for ; Mon, 4 Dec 2023 06:23:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699782; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v4HAiul36EO+vclRiaij572sHtWWM/DwfWC2VtVH9HE=; b=a76vBY38PqDSwobO96nI3/5poYp/A9ihfLT48/Y7WQS+BVYdwy3pozrFyougVaFzQN88D9 Jwv8YvIgN5AX91Mk/4ZeQAmHOecPBlZqbYp+8JZO6Ev63LRvRXkncO19PM1eMDnudLqQbx LnMM4vxKYFOhjecZ7cCkntwEGgOWwG8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-38-f9IMTYTNM3mqQeNw2EDP9A-1; Mon, 04 Dec 2023 09:22:59 -0500 X-MC-Unique: f9IMTYTNM3mqQeNw2EDP9A-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0378589CA00; Mon, 4 Dec 2023 14:22:59 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 661192026D4C; Mon, 4 Dec 2023 14:22:57 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 37/39] mm/rmap: remove page_try_dup_anon_rmap() Date: Mon, 4 Dec 2023 15:21:44 +0100 Message-ID: <20231204142146.91437-38-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:57:29 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784363721769601717 X-GMAIL-MSGID: 1784363721769601717 All users are gone, remove page_try_dup_anon_rmap() and any remaining traces. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 84439f7720c62..3c1df8e020188 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -251,7 +251,7 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); -/* See page_try_dup_anon_rmap() */ +/* See folio_try_dup_anon_rmap_*() */ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, struct vm_area_struct *vma) { @@ -463,16 +463,6 @@ static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, #endif } -static inline int page_try_dup_anon_rmap(struct page *page, bool compound, - struct vm_area_struct *vma) -{ - struct folio *folio = page_folio(page); - - if (likely(!compound)) - return folio_try_dup_anon_rmap_pte(folio, page, vma); - return folio_try_dup_anon_rmap_pmd(folio, page, vma); -} - /** * page_try_share_anon_rmap - try marking an exclusive anonymous page possibly * shared to prepare for KSM or temporary unmapping @@ -481,8 +471,8 @@ static inline int page_try_dup_anon_rmap(struct page *page, bool compound, * The caller needs to hold the PT lock and has to have the page table entry * cleared/invalidated. * - * This is similar to page_try_dup_anon_rmap(), however, not used during fork() - * to duplicate a mapping, but instead to prepare for KSM or temporarily + * This is similar to folio_try_dup_anon_rmap_*(), however, not used during + * fork() to duplicate a mapping, but instead to prepare for KSM or temporarily * unmapping a page (swap, migration) via folio_remove_rmap_*(). * * Marking the page shared can only fail if the page may be pinned; device From patchwork Mon Dec 4 14:21:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173396 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2797154vqy; Mon, 4 Dec 2023 06:25:56 -0800 (PST) X-Google-Smtp-Source: AGHT+IFmFjSpPjZRjoDN589Mau/kcLKL8b++V+2h62N3gMPdNGT8YocfBO22cbYZNaot6q2hWUPH X-Received: by 2002:a05:6a20:970f:b0:18f:97c:9784 with SMTP id hr15-20020a056a20970f00b0018f097c9784mr3833907pzc.108.1701699956070; Mon, 04 Dec 2023 06:25:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701699956; cv=none; d=google.com; s=arc-20160816; b=gxU6LFhdFWIhwNmwf/7TPdjWrmcdU07ivpLxAms4sEbwiDo42S96BCPurAp+3M29eH y9f3tCIkCGGa2T08yqDSwip/TaxdMgz5I2ktEj+39rj4S9kMVezC+yusO5mdbR23Ke0X Dvc+flEeOUB5WpdQMLMsYG99Qi8ol++0hiBKqEs3qSYufwFGai2/zSJQPKGfqvfzYXPP 7uDwtNOQ2nB7eHf0UAO9eC39/Hn+rpDI4h4IGdgh+JklOVhCL77xqrE9Q9RWH5Dntir4 4oGzbZCmmxFJFzfkIRGE5DDzSPsGYMLybyPeprpXih8GusvzJdgw95ySFGjMNYRZ23o8 AmNQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=TrXZDuex4z9dO0cUmH/YoAM+lWjEkvkLNiyxPRViLRQ=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=oAee4VWMcuwno5GC+P8lCAB6l1Xcfyjc8ngavleS0RVMfg3JKu1l3i4/OxiQBx0Nha Xr2ir7cKkUbETS22MJISuJh5OpxC582zqFItlTSeicEtIZy/6bCWbWRXgL9wSW53MWeC LjAynPpQvu2xojC+ny9Y13s9zxtg6q7PP0AHmHBcidseDEiWBgD7hZIGql0dCvWr/DQF 4IHiD5Bq8xyg5MPBkVFzvVEBUeZbV4ftbehbsabU7wNOHuwFtuph6XiyZc+niRn+w8Pw PDc9J8KgqCw0HMjs5snAaUpOGC7156ORoa86zaphehm9wPX4Ee7/+Bmxhx+hrRBOcZP9 LmFw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KDTRyq6B; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id c2-20020a631c42000000b005bd641c3614si7699153pgm.769.2023.12.04.06.25.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:25:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KDTRyq6B; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id EDA91809348C; Mon, 4 Dec 2023 06:25:40 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345534AbjLDOZ1 (ORCPT + 99 others); Mon, 4 Dec 2023 09:25:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235878AbjLDOYp (ORCPT ); Mon, 4 Dec 2023 09:24:45 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA326213D for ; Mon, 4 Dec 2023 06:23:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699784; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TrXZDuex4z9dO0cUmH/YoAM+lWjEkvkLNiyxPRViLRQ=; b=KDTRyq6BmV95WYbFntuv4EE1f6ZgsMqqardiyo24MUu6hrvmVbTUUhxL8bI4ndh0oQtUuh j5BYr2T+Sux3Hspx8M0foEoZ5UVzJDxVXCT+k7PQUsTX1lKp5jU2lTcCHyr4tvWJYia85a k+/rWFnnp1omkqd1uhznXQSkz1xpAaQ= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-295-Ef5PYdNvM-qrNNOKUuEk5A-1; Mon, 04 Dec 2023 09:23:01 -0500 X-MC-Unique: Ef5PYdNvM-qrNNOKUuEk5A-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D721E3813BC8; Mon, 4 Dec 2023 14:23:00 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3BB132026D4C; Mon, 4 Dec 2023 14:22:59 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 38/39] mm: convert page_try_share_anon_rmap() to folio_try_share_anon_rmap_[pte|pmd]() Date: Mon, 4 Dec 2023 15:21:45 +0100 Message-ID: <20231204142146.91437-39-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:25:41 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361732936665646 X-GMAIL-MSGID: 1784361732936665646 Let's convert it like we converted all the other rmap functions. Don't introduce folio_try_share_anon_rmap_ptes() for now, as we don't have a user that wants rmap batching in sight. Pretty easy to add later. All users are easy to convert -- only ksm.c doesn't use folios yet but that is left for future work -- so let's just do it in a single shot. While at it, turn the BUG_ON into a WARN_ON_ONCE. Note that page_try_share_anon_rmap() so far didn't care about pte/pmd mappings (no compound parameter). We're changing that so we can perform better sanity checks and make the code actually more readable/consistent. For example, __folio_rmap_sanity_checks() will make sure that a PMD range actually falls completely into the folio. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 98 ++++++++++++++++++++++++++++++++------------ mm/gup.c | 2 +- mm/huge_memory.c | 9 ++-- mm/internal.h | 4 +- mm/ksm.c | 5 ++- mm/migrate_device.c | 2 +- mm/rmap.c | 9 ++-- 7 files changed, 89 insertions(+), 40 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 3c1df8e020188..ab3ea4583d502 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -267,14 +267,14 @@ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, return 0; } -/* See page_try_share_anon_rmap() */ +/* See folio_try_share_anon_rmap_*() */ static inline int hugetlb_try_share_anon_rmap(struct folio *folio) { VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); VM_WARN_ON_FOLIO(!PageAnonExclusive(&folio->page), folio); - /* See page_try_share_anon_rmap() */ + /* See folio_try_share_anon_rmap_*() */ if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) smp_mb(); @@ -282,7 +282,7 @@ static inline int hugetlb_try_share_anon_rmap(struct folio *folio) return -EBUSY; ClearPageAnonExclusive(&folio->page); - /* See page_try_share_anon_rmap() */ + /* See folio_try_share_anon_rmap_*() */ if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) smp_mb__after_atomic(); return 0; @@ -463,30 +463,15 @@ static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, #endif } -/** - * page_try_share_anon_rmap - try marking an exclusive anonymous page possibly - * shared to prepare for KSM or temporary unmapping - * @page: the exclusive anonymous page to try marking possibly shared - * - * The caller needs to hold the PT lock and has to have the page table entry - * cleared/invalidated. - * - * This is similar to folio_try_dup_anon_rmap_*(), however, not used during - * fork() to duplicate a mapping, but instead to prepare for KSM or temporarily - * unmapping a page (swap, migration) via folio_remove_rmap_*(). - * - * Marking the page shared can only fail if the page may be pinned; device - * private pages cannot get pinned and consequently this function cannot fail. - * - * Returns 0 if marking the page possibly shared succeeded. Returns -EBUSY - * otherwise. - */ -static inline int page_try_share_anon_rmap(struct page *page) +static inline int __folio_try_share_anon_rmap(struct folio *folio, + struct page *page, unsigned int nr_pages, enum rmap_mode mode) { - VM_BUG_ON_PAGE(!PageAnon(page) || !PageAnonExclusive(page), page); + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + VM_WARN_ON_FOLIO(!PageAnonExclusive(page), folio); + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); - /* device private pages cannot get pinned via GUP. */ - if (unlikely(is_device_private_page(page))) { + /* device private folios cannot get pinned via GUP. */ + if (unlikely(folio_is_device_private(folio))) { ClearPageAnonExclusive(page); return 0; } @@ -537,7 +522,7 @@ static inline int page_try_share_anon_rmap(struct page *page) if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) smp_mb(); - if (unlikely(page_maybe_dma_pinned(page))) + if (unlikely(folio_maybe_dma_pinned(folio))) return -EBUSY; ClearPageAnonExclusive(page); @@ -550,6 +535,67 @@ static inline int page_try_share_anon_rmap(struct page *page) return 0; } +/** + * folio_try_share_anon_rmap_pte - try marking an exclusive anonymous page + * mapped by a PTE possibly shared to prepare + * for KSM or temporary unmapping + * @folio: The folio to share a mapping of + * @page: The mapped exclusive page + * + * The caller needs to hold the page table lock and has to have the page table + * entries cleared/invalidated. + * + * This is similar to folio_try_dup_anon_rmap_pte(), however, not used during + * fork() to duplicate mappings, but instead to prepare for KSM or temporarily + * unmapping parts of a folio (swap, migration) via folio_remove_rmap_pte(). + * + * Marking the mapped page shared can only fail if the folio maybe pinned; + * device private folios cannot get pinned and consequently this function cannot + * fail. + * + * Returns 0 if marking the mapped page possibly shared succeeded. Returns + * -EBUSY otherwise. + */ +static inline int folio_try_share_anon_rmap_pte(struct folio *folio, + struct page *page) +{ + return __folio_try_share_anon_rmap(folio, page, 1, RMAP_MODE_PTE); +} + +/** + * folio_try_share_anon_rmap_pmd - try marking an exclusive anonymous page + * range mapped by a PMD possibly shared to + * prepare for temporary unmapping + * @folio: The folio to share the mapping of + * @page: The first page to share the mapping of + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock and has to have the page table + * entries cleared/invalidated. + * + * This is similar to folio_try_dup_anon_rmap_pmd(), however, not used during + * fork() to duplicate a mapping, but instead to prepare for temporarily + * unmapping parts of a folio (swap, migration) via folio_remove_rmap_pmd(). + * + * Marking the mapped pages shared can only fail if the folio maybe pinned; + * device private folios cannot get pinned and consequently this function cannot + * fail. + * + * Returns 0 if marking the mapped pages possibly shared succeeded. Returns + * -EBUSY otherwise. + */ +static inline int folio_try_share_anon_rmap_pmd(struct folio *folio, + struct page *page) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + return __folio_try_share_anon_rmap(folio, page, HPAGE_PMD_NR, + RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + /* * Called from mm/vmscan.c to handle paging out */ diff --git a/mm/gup.c b/mm/gup.c index 231711efa390d..49f32411c68da 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -177,7 +177,7 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags) /* * Adjust the pincount before re-checking the PTE for changes. * This is essentially a smp_mb() and is paired with a memory - * barrier in page_try_share_anon_rmap(). + * barrier in folio_try_share_anon_rmap_*(). */ smp_mb__after_atomic(); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 138e1e62790be..ebbf5ee6192e7 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2224,10 +2224,11 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, * In case we cannot clear PageAnonExclusive(), split the PMD * only and let try_to_migrate_one() fail later. * - * See page_try_share_anon_rmap(): invalidate PMD first. + * See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */ anon_exclusive = PageAnonExclusive(page); - if (freeze && anon_exclusive && page_try_share_anon_rmap(page)) + if (freeze && anon_exclusive && + folio_try_share_anon_rmap_pmd(folio, page)) freeze = false; if (!freeze) { rmap_t rmap_flags = RMAP_NONE; @@ -3253,9 +3254,9 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, flush_cache_range(vma, address, address + HPAGE_PMD_SIZE); pmdval = pmdp_invalidate(vma, address, pvmw->pmd); - /* See page_try_share_anon_rmap(): invalidate PMD first. */ + /* See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */ anon_exclusive = folio_test_anon(folio) && PageAnonExclusive(page); - if (anon_exclusive && page_try_share_anon_rmap(page)) { + if (anon_exclusive && folio_try_share_anon_rmap_pmd(folio, page)) { set_pmd_at(mm, address, pvmw->pmd, pmdval); return -EBUSY; } diff --git a/mm/internal.h b/mm/internal.h index 43dca750c5afc..b9b630717b9b2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1047,7 +1047,7 @@ enum { * * Ordinary GUP: Using the PT lock * * GUP-fast and fork(): mm->write_protect_seq * * GUP-fast and KSM or temporary unmapping (swap, migration): see - * page_try_share_anon_rmap() + * folio_try_share_anon_rmap_*() * * Must be called with the (sub)page that's actually referenced via the * page table entry, which might not necessarily be the head page for a @@ -1090,7 +1090,7 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma, return is_cow_mapping(vma->vm_flags); } - /* Paired with a memory barrier in page_try_share_anon_rmap(). */ + /* Paired with a memory barrier in folio_try_share_anon_rmap_*(). */ if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) smp_rmb(); diff --git a/mm/ksm.c b/mm/ksm.c index c23aed4f1a344..51f2d989be2be 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1161,8 +1161,9 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, goto out_unlock; } - /* See page_try_share_anon_rmap(): clear PTE first. */ - if (anon_exclusive && page_try_share_anon_rmap(page)) { + /* See folio_try_share_anon_rmap_pte(): clear PTE first. */ + if (anon_exclusive && + folio_try_share_anon_rmap_pte(page_folio(page), page)) { set_pte_at(mm, pvmw.address, pvmw.pte, entry); goto out_unlock; } diff --git a/mm/migrate_device.c b/mm/migrate_device.c index c51c99151ebb5..9d0c1ad737225 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -202,7 +202,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, if (anon_exclusive) { pte = ptep_clear_flush(vma, addr, ptep); - if (page_try_share_anon_rmap(page)) { + if (folio_try_share_anon_rmap_pte(folio, page)) { set_pte_at(mm, addr, ptep, pte); folio_unlock(folio); folio_put(folio); diff --git a/mm/rmap.c b/mm/rmap.c index 8e86024953c03..7bb3a174efc8d 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1817,9 +1817,9 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, break; } - /* See page_try_share_anon_rmap(): clear PTE first. */ + /* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive && - page_try_share_anon_rmap(subpage)) { + folio_try_share_anon_rmap_pte(folio, subpage)) { swap_free(entry); set_pte_at(mm, address, pvmw.pte, pteval); ret = false; @@ -2093,7 +2093,8 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, pte_t swp_pte; if (anon_exclusive) - BUG_ON(page_try_share_anon_rmap(subpage)); + WARN_ON_ONCE(folio_try_share_anon_rmap_pte(folio, + subpage)); /* * Store the pfn of the page in a special migration @@ -2175,7 +2176,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, break; } } else if (anon_exclusive && - page_try_share_anon_rmap(page)) { + folio_try_share_anon_rmap_pte(folio, subpage)) { set_pte_at(mm, address, pvmw.pte, pteval); ret = false; page_vma_mapped_walk_done(&pvmw); From patchwork Mon Dec 4 14:21:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 173398 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp2797748vqy; Mon, 4 Dec 2023 06:26:50 -0800 (PST) X-Google-Smtp-Source: AGHT+IGPJcQgVBPWFYt8m/VchdG0+uZJmot+PyHPN3DnoJdno4OL+y/MZ/CVGjCFZ5tHKqZm0P77 X-Received: by 2002:a17:902:efc1:b0:1cf:fc8f:3fb6 with SMTP id ja1-20020a170902efc100b001cffc8f3fb6mr3870821plb.17.1701700009937; Mon, 04 Dec 2023 06:26:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701700009; cv=none; d=google.com; s=arc-20160816; b=pfkpAOT5DG78T5+DLS0mTA1XtFhd1Vg8Ycn06z4al/31wIJA12g60ngnC7Cg5vvrcQ zwmAciM5/FW8pl29PqGNfFjn8Xv5WiXYFyfAzDbQPc43U3hBgTqmLYpLdQ4exOKQY+ZW jSuOJwzPgcoJ5/kTsanRS2nQD9eLrlXTR/qiTUSsoP2g9Adm+WoHOpbg9n/nregy7wnl UgWT9PBJtrjnljiw4c5c3yhJzL9a6X12tiDzI8+aHugKy5FVhNGl1KUGUCGpkWPBbz6z ep2zIJpiVofoCMq5uPyjzwOc+k8o4EsMK8LXnc7OYxQdm/aU+z5hRpJr4Z4tA0d9DbQC Z9Jg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=VzMmgnK/Xwl+ZoGTv7dPX8K+DyrKez8QvfXwl9HQj10=; fh=UlFXU+hKOMcm6idRzNfe3Pu+B5L1f7aeir3C6i/NWQA=; b=eNnS5EiG2lJqOA8e64xcG5k6/bZMgZSM52Se8wMGkE1unwgsRkwVGWTCJlkWgORxGv 2qSrsski6PEe5lrY7idBUD6Ark4ebrjqwYRBB9g+3fldsO/IeUo8ln7+/pzQ7TYxoVrI NqxsCgpMYNEw+RwV7GKqQPOisnV1CZHu6k8gWr1E8H9S/8phhjHcliMDDBS0ySRSqVR4 WY34FNNpRRiJiKGq5oumu0q2LZNx9Rv0qCHtqfIoo94v+P3GIZY27tWKKQttc95upTXx Ywl42mLjxqq3iX9XGWpvNOxs+juNCw84vuezPAUJ/Sgift1zYibeMz0RoefD2WLyf9bg WliA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=RasSKsTv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id j18-20020a170902c3d200b001d085a5fe37si2768510plj.582.2023.12.04.06.26.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 06:26:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=RasSKsTv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id F201A80793F3; Mon, 4 Dec 2023 06:25:45 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345573AbjLDOZa (ORCPT + 99 others); Mon, 4 Dec 2023 09:25:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52268 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345322AbjLDOYu (ORCPT ); Mon, 4 Dec 2023 09:24:50 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B8DE2682 for ; Mon, 4 Dec 2023 06:23:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1701699785; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VzMmgnK/Xwl+ZoGTv7dPX8K+DyrKez8QvfXwl9HQj10=; b=RasSKsTvfXoyLfdPdD8I3UhJqtrjTqVOCObRjtgO/8cBSNWirnfOT5ZcBW4c+NTEMW8SuC BahHb5rHuS04ML49YaUcYep3ew87KicoFYyv/m+J0Om9/Nzargghqyo8U7OPLhpqbC/ROK vjvZfStMCu2FYaOou3btzlFEnOD0ZFY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-489-6zp6WxYFOY2-2yvUwM7PyQ-1; Mon, 04 Dec 2023 09:23:03 -0500 X-MC-Unique: 6zp6WxYFOY2-2yvUwM7PyQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AFCFD89CA05; Mon, 4 Dec 2023 14:23:02 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.195.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1B80C2026D4C; Mon, 4 Dec 2023 14:23:00 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH RFC 39/39] mm/rmap: rename COMPOUND_MAPPED to ENTIRELY_MAPPED Date: Mon, 4 Dec 2023 15:21:46 +0100 Message-ID: <20231204142146.91437-40-david@redhat.com> In-Reply-To: <20231204142146.91437-1-david@redhat.com> References: <20231204142146.91437-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 04 Dec 2023 06:25:46 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784361789376281380 X-GMAIL-MSGID: 1784361789376281380 We removed all "bool compound" and RMAP_COMPOUND parameters. Let's remove the remaining "compound" terminology by making COMPOUND_MAPPED match the "folio->_entire_mapcount" terminology, renaming it to ENTIRELY_MAPPED. ENTIRELY_MAPPED is only used when the whole folio is mapped using a single page table entry (e.g., a single PMD mapping a PMD-sized THP). For now, we don't support mapping any THP bigger than that, so ENTIRELY_MAPPED only applies to PMD-mapped PMD-sized THP only. Signed-off-by: David Hildenbrand --- Documentation/mm/transhuge.rst | 2 +- mm/internal.h | 6 +++--- mm/rmap.c | 18 +++++++++--------- 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/Documentation/mm/transhuge.rst b/Documentation/mm/transhuge.rst index cf81272a6b8b6..93c9239b9ebe2 100644 --- a/Documentation/mm/transhuge.rst +++ b/Documentation/mm/transhuge.rst @@ -117,7 +117,7 @@ pages: - map/unmap of a PMD entry for the whole THP increment/decrement folio->_entire_mapcount and also increment/decrement - folio->_nr_pages_mapped by COMPOUND_MAPPED when _entire_mapcount + folio->_nr_pages_mapped by ENTIRELY_MAPPED when _entire_mapcount goes from -1 to 0 or 0 to -1. - map/unmap of individual pages with PTE entry increment/decrement diff --git a/mm/internal.h b/mm/internal.h index b9b630717b9b2..700b230666f87 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -54,12 +54,12 @@ void page_writeback_init(void); /* * If a 16GB hugetlb folio were mapped by PTEs of all of its 4kB pages, - * its nr_pages_mapped would be 0x400000: choose the COMPOUND_MAPPED bit + * its nr_pages_mapped would be 0x400000: choose the ENTIRELY_MAPPED bit * above that range, instead of 2*(PMD_SIZE/PAGE_SIZE). Hugetlb currently * leaves nr_pages_mapped at 0, but avoid surprise if it participates later. */ -#define COMPOUND_MAPPED 0x800000 -#define FOLIO_PAGES_MAPPED (COMPOUND_MAPPED - 1) +#define ENTIRELY_MAPPED 0x800000 +#define FOLIO_PAGES_MAPPED (ENTIRELY_MAPPED - 1) /* * Flags passed to __show_mem() and show_free_areas() to suppress output in diff --git a/mm/rmap.c b/mm/rmap.c index 7bb3a174efc8d..a8e3563182103 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1142,7 +1142,7 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, first = atomic_inc_and_test(&page->_mapcount); if (first && folio_test_large(folio)) { first = atomic_inc_return_relaxed(mapped); - first = (first < COMPOUND_MAPPED); + first = (first < ENTIRELY_MAPPED); } if (first) @@ -1151,15 +1151,15 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, } else if (mode == RMAP_MODE_PMD) { first = atomic_inc_and_test(&folio->_entire_mapcount); if (first) { - nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { + nr = atomic_add_return_relaxed(ENTIRELY_MAPPED, mapped); + if (likely(nr < ENTIRELY_MAPPED + ENTIRELY_MAPPED)) { *nr_pmdmapped = folio_nr_pages(folio); nr = *nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of a remove and another add? */ if (unlikely(nr < 0)) nr = 0; } else { - /* Raced ahead of a remove of COMPOUND_MAPPED */ + /* Raced ahead of a remove of ENTIRELY_MAPPED */ nr = 0; } } @@ -1384,7 +1384,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, } else { /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); - atomic_set(&folio->_nr_pages_mapped, COMPOUND_MAPPED); + atomic_set(&folio->_nr_pages_mapped, ENTIRELY_MAPPED); nr = folio_nr_pages(folio); __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr); } @@ -1467,7 +1467,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, last = atomic_add_negative(-1, &page->_mapcount); if (last && folio_test_large(folio)) { last = atomic_dec_return_relaxed(mapped); - last = (last < COMPOUND_MAPPED); + last = (last < ENTIRELY_MAPPED); } if (last) @@ -1476,15 +1476,15 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, } else if (mode == RMAP_MODE_PMD) { last = atomic_add_negative(-1, &folio->_entire_mapcount); if (last) { - nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED)) { + nr = atomic_sub_return_relaxed(ENTIRELY_MAPPED, mapped); + if (likely(nr < ENTIRELY_MAPPED)) { nr_pmdmapped = folio_nr_pages(folio); nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of another remove and an add? */ if (unlikely(nr < 0)) nr = 0; } else { - /* An add of COMPOUND_MAPPED raced ahead */ + /* An add of ENTIRELY_MAPPED raced ahead */ nr = 0; } }