From patchwork Tue Dec 19 07:55:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 180800 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:24d3:b0:fb:cd0c:d3e with SMTP id r19csp1778217dyi; Mon, 18 Dec 2023 23:56:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IGKzPiHeUymRPDtsP8/KmP5egOhFc6ZFEh8GZGfE9xnCGsLPwX6VLRJHismCw+wMNeWnyUt X-Received: by 2002:a05:622a:1708:b0:41e:1d15:69a6 with SMTP id h8-20020a05622a170800b0041e1d1569a6mr27240967qtk.31.1702972592628; Mon, 18 Dec 2023 23:56:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702972592; cv=none; d=google.com; s=arc-20160816; b=s7ckSs7asXUdyi4rcuVc7+VOMF/Lu9+TY1trMjCGRnvm2bIa9ZHRff961JPPA1dL2R 4R5FSJDZhk2K/ZnR1wLLwLOno4IB8rusF4Eg6O//zwk82bpi9aJ/hKNaOJf+KIP+IyKi JZJEIPfmBVnoUD5aRkTFR3wm1ri6th7/kqsmODO3Ec/NSK+7yiB17tbti62wJWFMUC7w PgikUfJDKpMdCKazbWQu0gbzu/4Kvl89wgTGeAUt2KHOMVF9xJgXC9ueIzl0sqLNbcFx /6RAt3cAiHXkBcXM02hRppQrq0finJwnj5aFTzSZa+2VHccpOUV7LO9whLbKTbeaGu+r 9Aqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=NRvgvssN6KVSq9rYZ03PlMu3BaOgDfLGQsqd5RQuPrM=; fh=3mpu6Yyr5hMzDKtkQvzcApl7ZSQGU4G8QyIY2BAU7VY=; b=ZEfuuVzHjNwkS80U8WUvU+O5Btk8gYuepoGwsIQQvoyysHx8SNfjJAaQdSdpql6CCR D0CzWovXiL2VNzhKXxDY0TYOGT04Ui76YsQPfuj9Db/sAOxHhdNsxpAWfF1w6jCux21R x4TNPQkMk3T2swgDuSnjGIRdmfquzmudjg2T5kB6fO3TQd4uc/2FVAOkarO/OwzSOdMX XCYTKLqJHi94vn/ly3pj9J9gBVPTg2wWvjXsdLzJkUU3hiaK6LIJozIywFEYUYfUlFmG O515KU3ntnCaMmPk5eY8P79sxcBfGSB8S9IdROkjaVEWe/zqXNTMklkmyOPMHbQAUwWY X1yw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KTl2MQc1; spf=pass (google.com: domain of linux-kernel+bounces-4834-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4834-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id bs24-20020ac86f18000000b0042378c526a8si27942531qtb.569.2023.12.18.23.56.32 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 23:56:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-4834-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KTl2MQc1; spf=pass (google.com: domain of linux-kernel+bounces-4834-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4834-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 62A5C1C2372C for ; Tue, 19 Dec 2023 07:56:32 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 53032134D2; Tue, 19 Dec 2023 07:56:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="KTl2MQc1" X-Original-To: linux-kernel@vger.kernel.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21A9B12B60 for ; Tue, 19 Dec 2023 07:56:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702972570; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NRvgvssN6KVSq9rYZ03PlMu3BaOgDfLGQsqd5RQuPrM=; b=KTl2MQc1qjJSDnZ+ZNXtzsi8B2d2fzd/vgol6a3tXXDBnugvGpmgFArhd90xNSXoIIpDla Xa7615j8a7XEUY+WVcK8hFx9KnVjgVhoLAV9AISiMw75yBjQZ3yDzIbuZHh2TnJMwR4vVe biMb1DGFi9jor0dpqrp6Aj+Gs+/xi0g= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-672-dipUc7OWPBCmQTMSma-p8A-1; Tue, 19 Dec 2023 02:56:06 -0500 X-MC-Unique: dipUc7OWPBCmQTMSma-p8A-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E23A729AA38D; Tue, 19 Dec 2023 07:56:04 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.117]) by smtp.corp.redhat.com (Postfix) with ESMTP id B4B6D2026D66; Tue, 19 Dec 2023 07:55:52 +0000 (UTC) From: peterx@redhat.com To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Christophe Leroy , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Mike Kravetz , Mike Rapoport , Christoph Hellwig , John Hubbard , Andrew Jones , linux-arm-kernel@lists.infradead.org, Michael Ellerman , "Kirill A . Shutemov" , linuxppc-dev@lists.ozlabs.org, Rik van Riel , linux-riscv@lists.infradead.org, Yang Shi , James Houghton , "Aneesh Kumar K . V" , Andrew Morton , Jason Gunthorpe , Andrea Arcangeli , peterx@redhat.com, Axel Rasmussen Subject: [PATCH 01/13] mm/Kconfig: CONFIG_PGTABLE_HAS_HUGE_LEAVES Date: Tue, 19 Dec 2023 15:55:26 +0800 Message-ID: <20231219075538.414708-2-peterx@redhat.com> In-Reply-To: <20231219075538.414708-1-peterx@redhat.com> References: <20231219075538.414708-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785696188976149927 X-GMAIL-MSGID: 1785696188976149927 From: Peter Xu Introduce a config option that will be selected as long as huge leaves are involved in pgtable (thp or hugetlbfs). It would be useful to mark any code with this new config that can process either hugetlb or thp pages in any level that is higher than pte level. Signed-off-by: Peter Xu --- mm/Kconfig | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 8f8b02e9c136..4ca97d959323 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -904,6 +904,9 @@ config READ_ONLY_THP_FOR_FS endif # TRANSPARENT_HUGEPAGE +config PGTABLE_HAS_HUGE_LEAVES + def_bool TRANSPARENT_HUGEPAGE || HUGETLB_PAGE + # # UP and nommu archs use km based percpu allocator # From patchwork Tue Dec 19 07:55:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 180814 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:24d3:b0:fb:cd0c:d3e with SMTP id r19csp1782845dyi; Tue, 19 Dec 2023 00:06:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IH4SfmdRsoq5knDd9dttyDcu/s3Ccx24JuqrkmtB5iSy7WZVshVUtDWao4l2lLatOpue9K+ X-Received: by 2002:a50:d697:0:b0:553:354b:18f1 with SMTP id r23-20020a50d697000000b00553354b18f1mr2279306edi.19.1702973191795; Tue, 19 Dec 2023 00:06:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702973191; cv=none; d=google.com; s=arc-20160816; b=TLGzaTIuYRqK4rF4Gn3NzZB0M0BnpCf7FNECIB0ZVRKmhK+bjMaG3vc0OL/ALsm4m5 2N32x8Vi+1QfIvC5K7ZeD4FzCZ2MvTQRQlglJ2Vk3pZjkyCwO++K4rtQcAZNezWofmp8 gSiszx/nMSjeU6V32muOmCrYyovPpxs5BGOG7tOHxJsJs0pZZ4P2rqgWKOvlLOSST73Y jKuCeWUd16/IIGFiJD5ADwiY7IUEXnISNuPxkMXMdJVsYGaf1SOBIsXIsGTl9AR6L9Oi 2aW+TshROw0/yzW42XC49mBpzv5thMKEBJv/rFZoVE1scfeIZOxqcnlIe1EIJWrjepq0 KF+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=75YlhRgpWvz7xroPDJtpCNniQimqN/967LqIgG1M+eM=; fh=3mpu6Yyr5hMzDKtkQvzcApl7ZSQGU4G8QyIY2BAU7VY=; b=UtX/pk8eh1hP0sT8PfMC+vTmIpBSLVAKIOHA2wYYsU4nPnCBvDkn3E9wHwI5bmp+oJ AXJvRR+BLnxWDem4bYD5+3HqW1tlP54mxnrm1yR4xjsOkwkhxtt87i60ws1IRmy2oNjX vOzFYdcBjNEawNUKL2wImipsBfwzFxfvaucSPcuXSKSMm/1ppyj3cVVN9M6T8w6RfDLz A8k5WYfjp9LiNlWu7qwu5AitImaaUPFTFl/CQSTdhlj/DxOwXlMQ8soFYB+PPrEOVWEu iN2RC443kuBsTDadPKzM6jW1SmHtuiQ4CC79KJhv4Zh7sYagN4L6anIUUr7Xbuh9QqRM cpmA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZI0L6sbL; spf=pass (google.com: domain of linux-kernel+bounces-4835-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4835-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id i28-20020a50871c000000b005539e083f7csi226027edb.535.2023.12.19.00.06.31 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 00:06:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-4835-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZI0L6sbL; spf=pass (google.com: domain of linux-kernel+bounces-4835-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4835-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 97F101F25B49 for ; Tue, 19 Dec 2023 07:56:48 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5952A14293; Tue, 19 Dec 2023 07:56:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ZI0L6sbL" X-Original-To: linux-kernel@vger.kernel.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 465C113FF9 for ; Tue, 19 Dec 2023 07:56:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702972582; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=75YlhRgpWvz7xroPDJtpCNniQimqN/967LqIgG1M+eM=; b=ZI0L6sbLTVFrGtkM+vcH5a+0jUqvkNkPr9K0tgOuyp9vXoDi8cBVG27Wm53nyinNMN/xVS V32BRHn1bNGcUF/lp8HfBlDFIzSsLWJ7fdLBsZV80fHI2H69vt0x8Jr3Nlq+z/Ee2b6hhW rJpr8bJicawuydJn8QwZ7KT/5UDZfjU= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-356-wW0mlPoONC2GJ2s_aRZ7hQ-1; Tue, 19 Dec 2023 02:56:18 -0500 X-MC-Unique: wW0mlPoONC2GJ2s_aRZ7hQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AE4F23813F24; Tue, 19 Dec 2023 07:56:16 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.117]) by smtp.corp.redhat.com (Postfix) with ESMTP id 993A82026D66; Tue, 19 Dec 2023 07:56:05 +0000 (UTC) From: peterx@redhat.com To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Christophe Leroy , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Mike Kravetz , Mike Rapoport , Christoph Hellwig , John Hubbard , Andrew Jones , linux-arm-kernel@lists.infradead.org, Michael Ellerman , "Kirill A . Shutemov" , linuxppc-dev@lists.ozlabs.org, Rik van Riel , linux-riscv@lists.infradead.org, Yang Shi , James Houghton , "Aneesh Kumar K . V" , Andrew Morton , Jason Gunthorpe , Andrea Arcangeli , peterx@redhat.com, Axel Rasmussen Subject: [PATCH 02/13] mm/hugetlb: Declare hugetlbfs_pagecache_present() non-static Date: Tue, 19 Dec 2023 15:55:27 +0800 Message-ID: <20231219075538.414708-3-peterx@redhat.com> In-Reply-To: <20231219075538.414708-1-peterx@redhat.com> References: <20231219075538.414708-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785696817523340400 X-GMAIL-MSGID: 1785696817523340400 From: Peter Xu It will be used outside hugetlb.c soon. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 9 +++++++++ mm/hugetlb.c | 4 ++-- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 236ec7b63c54..f8c5c174c8a6 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -174,6 +174,9 @@ u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx); pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, pud_t *pud); +bool hugetlbfs_pagecache_present(struct hstate *h, + struct vm_area_struct *vma, + unsigned long address); struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage); @@ -1221,6 +1224,12 @@ static inline void hugetlb_register_node(struct node *node) static inline void hugetlb_unregister_node(struct node *node) { } + +static inline bool hugetlbfs_pagecache_present( + struct hstate *h, struct vm_area_struct *vma, unsigned long address) +{ + return false; +} #endif /* CONFIG_HUGETLB_PAGE */ static inline spinlock_t *huge_pte_lock(struct hstate *h, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 6feb3e0630d1..29705e5c6f40 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6018,8 +6018,8 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* * Return whether there is a pagecache page to back given address within VMA. */ -static bool hugetlbfs_pagecache_present(struct hstate *h, - struct vm_area_struct *vma, unsigned long address) +bool hugetlbfs_pagecache_present(struct hstate *h, + struct vm_area_struct *vma, unsigned long address) { struct address_space *mapping = vma->vm_file->f_mapping; pgoff_t idx = linear_page_index(vma, address); From patchwork Tue Dec 19 07:55:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 180801 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:24d3:b0:fb:cd0c:d3e with SMTP id r19csp1778375dyi; Mon, 18 Dec 2023 23:57:05 -0800 (PST) X-Google-Smtp-Source: AGHT+IEMXJXVGFNhgdy1hssLbzJx5mrrzMwQu/NqAGLlFqOiNv29wzSvzxj2Oef8w2BKkFbJZdM9 X-Received: by 2002:a05:6870:4215:b0:203:f926:5b6c with SMTP id u21-20020a056870421500b00203f9265b6cmr25008oac.39.1702972625104; Mon, 18 Dec 2023 23:57:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702972625; cv=none; d=google.com; s=arc-20160816; b=xhtTxgXJRdfYRQ/ApryUXIPstkeCni6clvvJHtdQG+ceC6lJPKU1qzM/q5511GI+ZN 1X3yHuquGgJcfb9p2mgQAwKZ9P+uyhoYd3qQr5110v90yXd5le84l888qbgsOzjYvaUp 9KNj3qCsJWd83/1t8N3+uiEfF77VHH0FfPPjDGETb4QGTIgqa7t6LVB98dNyLQuhi7tR jZS7pVLEHh88ydP6fivlzzraLgiKE38mvCCDQ73yMVkiD6kpcDKyKOrWMytn618fHUC/ 3v/9lQyJEnOYn6gPPNSuZdF8iWa0D+aR5B/qbX22itN+Rpdk5E2KNbWc6trwPeI2QQSt pqsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=VdhTUt+FrTuSBSRKmaD9aOddOtS2xHAJn3bdMelwyoI=; fh=3mpu6Yyr5hMzDKtkQvzcApl7ZSQGU4G8QyIY2BAU7VY=; b=CxzMSZ8UjDuoj2X7DrLm3nNYlc+0ekoShjA+RT+AJhhSYYZD7TowbLZb31v9khyWUd Wpk0d6GhLXhwrtooWwA84MP+SLZYdsdV3vGL+cK+q4NedIEZoWchREcdXVBqvRz5khnw MOavzL9UyUfFErViaJryULbL4KK1B1xia3UWLzXuVaWA0QY/aqOEf4M7txouuvwBVZET lkqobcjmhVmvO1155FZYDVgxi3si/g3iutp39GcIPE0HN5fQwyauIhVrpjRQWT8KO7U/ Qu9FgvMJWBOD+BjE/90rVmVRLKrwyXsDUOjtDxRmoclCQhz4kmnI7d0krVDdQ7dyfdqc pQDA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=S5RPX8rj; spf=pass (google.com: domain of linux-kernel+bounces-4836-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4836-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id v4-20020a63f204000000b005cd813c23besi5490597pgh.415.2023.12.18.23.57.04 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 23:57:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-4836-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=S5RPX8rj; spf=pass (google.com: domain of linux-kernel+bounces-4836-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4836-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id BBC9AB240A6 for ; Tue, 19 Dec 2023 07:57:03 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0998914F78; Tue, 19 Dec 2023 07:56:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="S5RPX8rj" X-Original-To: linux-kernel@vger.kernel.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 86A7C14A87 for ; Tue, 19 Dec 2023 07:56:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702972593; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VdhTUt+FrTuSBSRKmaD9aOddOtS2xHAJn3bdMelwyoI=; b=S5RPX8rjfwaWxduNP4ARumN1dF0Mc6P09VrcIRPJjancgkznbPEiRTsoawzYM77AqpIJtt iWo9fhCmrEbF/tqpDWhxN8SphxsgnjPZSE4vvLGjbP6kwpp3tuyAiQcRPyrzq8fAU3onrC 8MLmu0WS3xik1xWOaimbrrtWtcTaV0A= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-443-zvB374S3NdqDENm7NImiSg-1; Tue, 19 Dec 2023 02:56:29 -0500 X-MC-Unique: zvB374S3NdqDENm7NImiSg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 298F1185A782; Tue, 19 Dec 2023 07:56:28 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.117]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7B65A2026D6F; Tue, 19 Dec 2023 07:56:17 +0000 (UTC) From: peterx@redhat.com To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Christophe Leroy , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Mike Kravetz , Mike Rapoport , Christoph Hellwig , John Hubbard , Andrew Jones , linux-arm-kernel@lists.infradead.org, Michael Ellerman , "Kirill A . Shutemov" , linuxppc-dev@lists.ozlabs.org, Rik van Riel , linux-riscv@lists.infradead.org, Yang Shi , James Houghton , "Aneesh Kumar K . V" , Andrew Morton , Jason Gunthorpe , Andrea Arcangeli , peterx@redhat.com, Axel Rasmussen Subject: [PATCH 03/13] mm: Provide generic pmd_thp_or_huge() Date: Tue, 19 Dec 2023 15:55:28 +0800 Message-ID: <20231219075538.414708-4-peterx@redhat.com> In-Reply-To: <20231219075538.414708-1-peterx@redhat.com> References: <20231219075538.414708-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785696223615684149 X-GMAIL-MSGID: 1785696223615684149 From: Peter Xu ARM defines pmd_thp_or_huge(), detecting either a THP or a huge PMD. It can be a helpful helper if we want to merge more THP and hugetlb code paths. Make it a generic default implementation, only exist when CONFIG_MMU. Arch can overwrite it by defining its own version. For example, ARM's pgtable-2level.h defines it to always return false. Keep the macro declared with all config, it should be optimized to a false anyway if !THP && !HUGETLB. Signed-off-by: Peter Xu --- include/linux/pgtable.h | 4 ++++ mm/gup.c | 3 +-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index af7639c3b0a3..6f2fa1977b8a 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1355,6 +1355,10 @@ static inline int pmd_write(pmd_t pmd) #endif /* pmd_write */ #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#ifndef pmd_thp_or_huge +#define pmd_thp_or_huge(pmd) (pmd_huge(pmd) || pmd_trans_huge(pmd)) +#endif + #ifndef pud_write static inline int pud_write(pud_t pud) { diff --git a/mm/gup.c b/mm/gup.c index 0a5f0e91bfec..efc9847e58fb 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -3004,8 +3004,7 @@ static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned lo if (!pmd_present(pmd)) return 0; - if (unlikely(pmd_trans_huge(pmd) || pmd_huge(pmd) || - pmd_devmap(pmd))) { + if (unlikely(pmd_thp_or_huge(pmd) || pmd_devmap(pmd))) { /* See gup_pte_range() */ if (pmd_protnone(pmd)) return 0; From patchwork Tue Dec 19 07:55:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 180802 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:24d3:b0:fb:cd0c:d3e with SMTP id r19csp1778445dyi; Mon, 18 Dec 2023 23:57:18 -0800 (PST) X-Google-Smtp-Source: AGHT+IEymgYisl0A9Woi0B3JNe1W2ybckSPYkMNVzRB9JiVn4hytAq0gd6IhwRDFsO1twfRKFB4g X-Received: by 2002:a05:620a:ecf:b0:77f:23e:4405 with SMTP id x15-20020a05620a0ecf00b0077f023e4405mr17165100qkm.62.1702972638550; Mon, 18 Dec 2023 23:57:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702972638; cv=none; d=google.com; s=arc-20160816; b=uSXXmrVrruhTMP0woTEi0rFsi4uJhaW3w2fYfTkd4ccxWkLb6I6iCS4ja5lmKW3IBY ueStzPCJoZ1VF+htoVqFLoEqiKZnCumn4C9EihH+IGo080s0qsl8rpaLuqjSrQa0DPGB 1uTyTMZk3r3MAiEfPBM19/9UorlwYsOIppdVPmDYTscqW3Wzvhb4o+LixWBYqMZPcDq1 hHyGo4Bm4lpNX/+LKhD3A3WJY/xYw2BFmCK4TDf64ZKi7v4f7qafdJZJdKYzaKixZthe Qqkrc8e5CA1EfGZRKvleocWx6fp4kNITl6sftO4VRwf9QdCLirUYxxDKjW79fkkafFY3 gKWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=W3mfDaRytJv6LOYv89LHjQVs33fumKlGjLMhNR5CFmw=; fh=3mpu6Yyr5hMzDKtkQvzcApl7ZSQGU4G8QyIY2BAU7VY=; b=out1xCweo97M2znSw4vfEA+8JLKGdsG93l3KW7oh0OaPDcm4tMYPaYR/QKZj5r7IzO fZQdsHeB94fCzVBsyCOHO8ae/LeP9owkgJex4Ezomn/PNrsZs8xNZqJinW6ms/grMLVj 40AdzlMHcY2vZve2CQTFm/F4C6uJfh1YGcono/7EdxOd8RNZqpqoH1OJ4rYkT9+7N1Ak PydwPiOmD/saKO2UR/o3h8UiM7WWtjpbf70OVH9sSypzYsb3OEN8JeEQL9nHnV9hcfTg U8uKS/gI2vdkiQ1u3JMaCTFavKfITDWLxT8OyVdBuA9P1AqEurk7uZee4dqAiRIUIwh2 rz8A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="TwZnA+/P"; spf=pass (google.com: domain of linux-kernel+bounces-4837-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4837-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id d18-20020a05620a241200b0077bcd1f26a0si27164185qkn.163.2023.12.18.23.57.18 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 23:57:18 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-4837-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="TwZnA+/P"; spf=pass (google.com: domain of linux-kernel+bounces-4837-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4837-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 57D931C236EB for ; Tue, 19 Dec 2023 07:57:18 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1466B15EBF; Tue, 19 Dec 2023 07:56:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="TwZnA+/P" X-Original-To: linux-kernel@vger.kernel.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B37EA154A2 for ; Tue, 19 Dec 2023 07:56:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702972608; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W3mfDaRytJv6LOYv89LHjQVs33fumKlGjLMhNR5CFmw=; b=TwZnA+/POS/8p5MX8b8NuTCJRqef+OuEAkm9Sk73SpR4gW4vspKqYa5kSMsZg7kIUWYIyf hChUrNjx6t0+VlEO/xlvuGvZM/HfzC+F0bXtCcVfxyO/yYb9gIeogwfxGKhpzlP3kjfd0C 1T5eRdIREvZQhjDxnzzBz0ca8pmgQFc= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-296-k0rHzqs9O9yPkXz_rzqp7g-1; Tue, 19 Dec 2023 02:56:41 -0500 X-MC-Unique: k0rHzqs9O9yPkXz_rzqp7g-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0C34B1C0513F; Tue, 19 Dec 2023 07:56:40 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.117]) by smtp.corp.redhat.com (Postfix) with ESMTP id E06102026D66; Tue, 19 Dec 2023 07:56:28 +0000 (UTC) From: peterx@redhat.com To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Christophe Leroy , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Mike Kravetz , Mike Rapoport , Christoph Hellwig , John Hubbard , Andrew Jones , linux-arm-kernel@lists.infradead.org, Michael Ellerman , "Kirill A . Shutemov" , linuxppc-dev@lists.ozlabs.org, Rik van Riel , linux-riscv@lists.infradead.org, Yang Shi , James Houghton , "Aneesh Kumar K . V" , Andrew Morton , Jason Gunthorpe , Andrea Arcangeli , peterx@redhat.com, Axel Rasmussen Subject: [PATCH 04/13] mm: Make HPAGE_PXD_* macros even if !THP Date: Tue, 19 Dec 2023 15:55:29 +0800 Message-ID: <20231219075538.414708-5-peterx@redhat.com> In-Reply-To: <20231219075538.414708-1-peterx@redhat.com> References: <20231219075538.414708-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785696237423617918 X-GMAIL-MSGID: 1785696237423617918 From: Peter Xu These macros can be helpful when we plan to merge hugetlb code into generic code. Move them out and define them even if !THP. We actually already defined HPAGE_PMD_NR for other reasons even if !THP. Reorganize these macros. Reviewed-by: Christoph Hellwig Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index fa7a38a30fc6..d335130e145f 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -64,9 +64,6 @@ ssize_t single_hugepage_flag_show(struct kobject *kobj, enum transparent_hugepage_flag flag); extern struct kobj_attribute shmem_enabled_attr; -#define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT) -#define HPAGE_PMD_NR (1< X-Patchwork-Id: 180815 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:24d3:b0:fb:cd0c:d3e with SMTP id r19csp1783328dyi; Tue, 19 Dec 2023 00:07:50 -0800 (PST) X-Google-Smtp-Source: AGHT+IFwZrHKsFbkpAtegQHlR52yMaLhQc7aRWPY8s+qHBUSvKWZnlka5DRxoCvKSZ7Z8Fdgr+ME X-Received: by 2002:a17:907:a0c6:b0:a17:ab92:3d55 with SMTP id hw6-20020a170907a0c600b00a17ab923d55mr10265950ejc.19.1702973270262; Tue, 19 Dec 2023 00:07:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702973270; cv=none; d=google.com; s=arc-20160816; b=opSTXhnseJkOHImcikil32pDY62/SojxLrChzYufR1tmudGJX3sF5Nin3q/O8Fatkm PpKx9rbhAS92EvoKGrvOpt1P2G8ySbNEq84GCkLNwz/1TdWcddqQ8ZAET4CMswGD1x8L cxWHKqJ8DS14D2VosvdeLF90dm1R5Pkz8l3rYB8Jgn4bsIcrNXmQBz7wa3av5+6vMjY5 /rGhAGwR9O23tex56WjVTY1KA5wXLj+GxwxOJImeyeA47YgfLKlSmaJa6MELe7/anIhM kw1x3tHISfUm5CU3QOlb2H25vEhgOTaD1knEBELhE8qKKij2c8dwDMMu2QK3KQ/5HEXX bApg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=cOaAbTjD+sxL1nd4c8n1i5s+aCyb/VfpADXT9rdcjmA=; fh=3mpu6Yyr5hMzDKtkQvzcApl7ZSQGU4G8QyIY2BAU7VY=; b=yZhhaXGZ0S7BjSqYobtufL+hAQqv9KGtaRe3zj/CHvwm5SOvEA27RotQBmXSbDJ7ck hGRXWI2KbMDseevI9yni/qiWQri+5gc+tCRe/ZQttDDnZbgVnRtxi96ELKof44rhDrQ9 3Q0FkG1/VxbffbVkNFJJ4tkZG2w1aOuhZoAqiFJvXsNuq3Q+vFaJPfvL92avnhqDd37l 3VV0DgHsTpIsimd0mj0JY2TzJH98wi0Q6ywaLEBYfEy9ODymWsMCk94H/mhiRA1HSPET 0MGEXe82n7d0yaF1PjsfGXtJ0lIIA9odTpAu1UkyB9syQ4h5YxAznTwVl1wQrRE3jyLD mUQQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=I33wxrN3; spf=pass (google.com: domain of linux-kernel+bounces-4839-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4839-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id s21-20020a1709066c9500b00a236d3072e5si733862ejr.984.2023.12.19.00.07.50 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 00:07:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-4839-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=I33wxrN3; spf=pass (google.com: domain of linux-kernel+bounces-4839-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4839-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 55AB41F2695A for ; Tue, 19 Dec 2023 07:57:57 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7F1061865C; Tue, 19 Dec 2023 07:57:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="I33wxrN3" X-Original-To: linux-kernel@vger.kernel.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 216EB168CB for ; Tue, 19 Dec 2023 07:57:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702972620; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cOaAbTjD+sxL1nd4c8n1i5s+aCyb/VfpADXT9rdcjmA=; b=I33wxrN3O9vPCiFjlesq0j6PIBNMuraGIScTkbYbvD126LJOerl7Qbv7lyJXAobbriJ5b2 PJE1iXoe277NG5Ay6Jh2t0hOd/pApRLH1V9Tj4fRgUEdtP4kZ+fKXS2veBY3VgWWGbNjpD kfNn/JDYU/BfLh5ML/ltkLYD4M7hhs4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-564-DMz-Hx_eOQmrxyRviaVqjQ-1; Tue, 19 Dec 2023 02:56:54 -0500 X-MC-Unique: DMz-Hx_eOQmrxyRviaVqjQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 26A2183B8E7; Tue, 19 Dec 2023 07:56:53 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.117]) by smtp.corp.redhat.com (Postfix) with ESMTP id CFE972026F95; Tue, 19 Dec 2023 07:56:40 +0000 (UTC) From: peterx@redhat.com To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Christophe Leroy , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Mike Kravetz , Mike Rapoport , Christoph Hellwig , John Hubbard , Andrew Jones , linux-arm-kernel@lists.infradead.org, Michael Ellerman , "Kirill A . Shutemov" , linuxppc-dev@lists.ozlabs.org, Rik van Riel , linux-riscv@lists.infradead.org, Yang Shi , James Houghton , "Aneesh Kumar K . V" , Andrew Morton , Jason Gunthorpe , Andrea Arcangeli , peterx@redhat.com, Axel Rasmussen Subject: [PATCH 05/13] mm: Introduce vma_pgtable_walk_{begin|end}() Date: Tue, 19 Dec 2023 15:55:30 +0800 Message-ID: <20231219075538.414708-6-peterx@redhat.com> In-Reply-To: <20231219075538.414708-1-peterx@redhat.com> References: <20231219075538.414708-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785696899587422617 X-GMAIL-MSGID: 1785696899587422617 From: Peter Xu Introduce per-vma begin()/end() helpers for pgtable walks. This is a preparation work to merge hugetlb pgtable walkers with generic mm. The helpers need to be called before and after a pgtable walk, will start to be needed if the pgtable walker code supports hugetlb pages. It's a hook point for any type of VMA, but for now only hugetlb uses it to stablize the pgtable pages from getting away (due to possible pmd unsharing). Reviewed-by: Christoph Hellwig Signed-off-by: Peter Xu Reviewed-by: Muchun Song --- include/linux/mm.h | 3 +++ mm/memory.c | 12 ++++++++++++ 2 files changed, 15 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index b72bf25a45cf..85e43775824b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4181,4 +4181,7 @@ static inline bool pfn_is_unaccepted_memory(unsigned long pfn) return range_contains_unaccepted_memory(paddr, paddr + PAGE_SIZE); } +void vma_pgtable_walk_begin(struct vm_area_struct *vma); +void vma_pgtable_walk_end(struct vm_area_struct *vma); + #endif /* _LINUX_MM_H */ diff --git a/mm/memory.c b/mm/memory.c index 1795aba53cf5..9ac6a9db971e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6270,3 +6270,15 @@ void ptlock_free(struct ptdesc *ptdesc) kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif + +void vma_pgtable_walk_begin(struct vm_area_struct *vma) +{ + if (is_vm_hugetlb_page(vma)) + hugetlb_vma_lock_read(vma); +} + +void vma_pgtable_walk_end(struct vm_area_struct *vma) +{ + if (is_vm_hugetlb_page(vma)) + hugetlb_vma_unlock_read(vma); +} From patchwork Tue Dec 19 07:55:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 180803 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:24d3:b0:fb:cd0c:d3e with SMTP id r19csp1778721dyi; Mon, 18 Dec 2023 23:58:12 -0800 (PST) X-Google-Smtp-Source: AGHT+IGQhqKvE9Ehnf8NmK4A8PvjZ9dcsfomcFPxRTS5lBRb7K+iLDmOzToHF66pgXifelZCPSho X-Received: by 2002:a05:620a:11a6:b0:780:dde7:43d7 with SMTP id c6-20020a05620a11a600b00780dde743d7mr4120951qkk.59.1702972692140; Mon, 18 Dec 2023 23:58:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702972692; cv=none; d=google.com; s=arc-20160816; b=MnfhUmGJEBWOhgALO30GTrjvhKTFSz+RRmbeKgYnUjDXwqbP0WroI7Ac8PoWCOsLON PbJOD0iaVU9EValernVRl8uRxNGdoA8gviCQZHIbZ6shfgUeEr+7OUh8wsRHHoZ/uuaR 7tzNjVbkdnMax3FN3pbsprL2RERF9N7F4GtMZ2t0Myvhs7sCCPIY2XBPZsAa1Uz71fmS X0XktBZVG++NHYwK5SMjtLhmj+xUTlYyHV2QTjxnxnTOy3gHf+jqL3OzoKdBIHcvdLMY HWAfIskY7aJpM3xLMiP0dUJ+nFTq2OD4F5a+tQC8Oj7kfbuUz/U6EDQWI/dLC0vpDmQx ocwA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=uyBT0nqGKaQsvTTr5mCIJmcBkZIho6YT9IXUJI5Nmzw=; fh=3mpu6Yyr5hMzDKtkQvzcApl7ZSQGU4G8QyIY2BAU7VY=; b=vkeNbStwRox6k6LPKMEyzfk4V8kfoPXe7o24sr8V9ur7tdnwC16HgVesPbKj8TpsfF zltVQXPebfbM+KxN7n4aA+mdfZ6GDOAKxatT2VNw/DbWz7fO4AemELW3Feemu2l/BVyE gf0Yi3znzBwTDly4yjsdWoMAn/ra4PXSOtv0/VUftqrKCH8iY0vAFvkiN4khCAAVeDKb o67kew/ivfY91K/llDaZtAYgnh1bcC9DvLuqyXMYg9iITNi848SkHCWPfgzhl3VaOIKg 48KpHziT9uds/UBfP7SD5XI3nRFW/lLUvpTkNVAdBZiUPJmwH74db4r140iD6knhy8/7 o6Sg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=NkviQZc1; spf=pass (google.com: domain of linux-kernel+bounces-4840-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4840-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id u15-20020a05620a430f00b0077d6a185ca5si19503748qko.683.2023.12.18.23.58.12 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 23:58:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-4840-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=NkviQZc1; spf=pass (google.com: domain of linux-kernel+bounces-4840-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4840-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id E493F1C23877 for ; Tue, 19 Dec 2023 07:58:11 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6589F1944E; Tue, 19 Dec 2023 07:57:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="NkviQZc1" X-Original-To: linux-kernel@vger.kernel.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9060718E10 for ; Tue, 19 Dec 2023 07:57:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702972629; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uyBT0nqGKaQsvTTr5mCIJmcBkZIho6YT9IXUJI5Nmzw=; b=NkviQZc12NmMFj4HgvIl2cAmddrWkG9ee0wuSQYz1ZfZcEPG/W4lyvV5qIxX8BaJmG/QjY YG/pse4Nc5vOwf+d1/mwZN2vBnPcHgKJowD9zvjc1AuyGqyz+FnySk5VSCZtY+aIGpeCAS e4mFA7rm15Uji9Khdq9uL+/XCRrdxPs= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-150-L9GGlqWjOR6XdFWT616TOg-1; Tue, 19 Dec 2023 02:57:06 -0500 X-MC-Unique: L9GGlqWjOR6XdFWT616TOg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E4B5F3813F20; Tue, 19 Dec 2023 07:57:04 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.117]) by smtp.corp.redhat.com (Postfix) with ESMTP id EAA8A2026D66; Tue, 19 Dec 2023 07:56:53 +0000 (UTC) From: peterx@redhat.com To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Christophe Leroy , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Mike Kravetz , Mike Rapoport , Christoph Hellwig , John Hubbard , Andrew Jones , linux-arm-kernel@lists.infradead.org, Michael Ellerman , "Kirill A . Shutemov" , linuxppc-dev@lists.ozlabs.org, Rik van Riel , linux-riscv@lists.infradead.org, Yang Shi , James Houghton , "Aneesh Kumar K . V" , Andrew Morton , Jason Gunthorpe , Andrea Arcangeli , peterx@redhat.com, Axel Rasmussen Subject: [PATCH 06/13] mm/gup: Drop folio_fast_pin_allowed() in hugepd processing Date: Tue, 19 Dec 2023 15:55:31 +0800 Message-ID: <20231219075538.414708-7-peterx@redhat.com> In-Reply-To: <20231219075538.414708-1-peterx@redhat.com> References: <20231219075538.414708-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785696293724138393 X-GMAIL-MSGID: 1785696293724138393 From: Peter Xu Hugepd format for GUP is only used in PowerPC with hugetlbfs. There are some kernel usage of hugepd (can refer to hugepd_populate_kernel() for PPC_8XX), however those pages are not candidates for GUP. Commit a6e79df92e4a ("mm/gup: disallow FOLL_LONGTERM GUP-fast writing to file-backed mappings") added a check to fail gup-fast if there's potential risk of violating GUP over writeback file systems. That should never apply to hugepd. Considering that hugepd is an old format (and even software-only), there's no plan to extend hugepd into other file typed memories that is prone to the same issue. Drop that check, not only because it'll never be true for hugepd per any known plan, but also it paves way for reusing the function outside fast-gup. To make sure we'll still remember this issue just in case hugepd will be extended to support non-hugetlbfs memories, add a rich comment above gup_huge_pd(), explaining the issue with proper references. Cc: Christoph Hellwig Cc: Lorenzo Stoakes Cc: Michael Ellerman Cc: linuxppc-dev@lists.ozlabs.org Signed-off-by: Peter Xu --- mm/gup.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index efc9847e58fb..bb5b7134f10b 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2820,11 +2820,6 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, return 0; } - if (!folio_fast_pin_allowed(folio, flags)) { - gup_put_folio(folio, refs, flags); - return 0; - } - if (!pte_write(pte) && gup_must_unshare(NULL, flags, &folio->page)) { gup_put_folio(folio, refs, flags); return 0; @@ -2835,6 +2830,14 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, return 1; } +/* + * NOTE: currently GUP for a hugepd is only possible on hugetlbfs file + * systems on Power, which does not have issue with folio writeback against + * GUP updates. When hugepd will be extended to support non-hugetlbfs or + * even anonymous memory, we need to do extra check as what we do with most + * of the other folios. See writable_file_mapping_allowed() and + * folio_fast_pin_allowed() for more information. + */ static int gup_huge_pd(hugepd_t hugepd, unsigned long addr, unsigned int pdshift, unsigned long end, unsigned int flags, struct page **pages, int *nr) From patchwork Tue Dec 19 07:55:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 180804 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:24d3:b0:fb:cd0c:d3e with SMTP id r19csp1778811dyi; Mon, 18 Dec 2023 23:58:27 -0800 (PST) X-Google-Smtp-Source: AGHT+IG6jlmOZ1ZgfYkdSWnp/uUqBjHOwHbaL4XhOg9GFynhXIfOFPkUc43VHnjcWkTTAGfkkEPR X-Received: by 2002:a05:6358:27a3:b0:170:17eb:9c45 with SMTP id l35-20020a05635827a300b0017017eb9c45mr22918661rwb.38.1702972707149; Mon, 18 Dec 2023 23:58:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702972707; cv=none; d=google.com; s=arc-20160816; b=1IapSHhu3OMaFVe7kA09NR7qwrIBOkzc7LyoJi2MAZDpXU/qpF1YS8/tuM/6+KU2gB vmVaBUHRdcd4Rc5e66KFvMvn2n/foT0cUYErGbb0334PfRh/HV1QFD/SdIBaNEeJ0g0X RMix0NHN0Jdgqm1Py4d68US0Pox2hFCJLDd9R58Ld20GTrhxf4iBvjYbpFCJ4RcKf7Gv 9qYNwaHb/SHDDfQIj54IoPPc7qLox9GNBcBeDyHsquWyQe5PQ7RPp5q7tuk+sejmGUtE BdfzBmMQIgd6cHFKs8ndsiZKaIVF1tfdzCR0rN913pr8+Uh764jxcXy8X7frJRdyBNv5 0m9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=4pxuQ+zeUrZD43W0p30SCM4P3ajCE4Euxhqhb6PE8fI=; fh=3mpu6Yyr5hMzDKtkQvzcApl7ZSQGU4G8QyIY2BAU7VY=; b=JyBd1v/wgx7hNwnX9lxU6dfgshGBfGPw1IqfDrZmck+zqtn08OMtZMqy0zwEkIDssl 5xcAgnIb3535Y1vS9Axv2c6XcZKjKPEK9a/p1hulPg8chzbkSo9sDff3GL1AUE9qc3uW 0JeECRthO7c5SJZhs0fvuugCdXxSdXYf5MNVRLD/BD3TqlY6/KqDHVLjtS53udJJzLml kA0J6R05fayRVv1m6t7Pds8cLQRhU/Uh1HSeyxROL+lP2GD/5iGV4xVOqBs9YOn8Ir/e +NvyfhNzkl4dlgbrtRdFslYLvE/Lit0THcDppCm0HbgY5yh8XWAKLK3tttuDoas5EePh HU5Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=HXSDbslq; spf=pass (google.com: domain of linux-kernel+bounces-4841-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4841-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id a7-20020a631a47000000b005bd335981e2si18819096pgm.678.2023.12.18.23.58.27 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 23:58:27 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-4841-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=HXSDbslq; spf=pass (google.com: domain of linux-kernel+bounces-4841-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4841-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id DAFCC286AF2 for ; Tue, 19 Dec 2023 07:58:26 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E7DA513ADB; Tue, 19 Dec 2023 07:57:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="HXSDbslq" X-Original-To: linux-kernel@vger.kernel.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 04D9C11C89 for ; Tue, 19 Dec 2023 07:57:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702972644; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4pxuQ+zeUrZD43W0p30SCM4P3ajCE4Euxhqhb6PE8fI=; b=HXSDbslqmmhmUOJ5i6ilK+Jw0AKj4MScBwMEn1RmCCcyqYNAV707cl/ip3hiRMhtABXnFn 1hQLOiuS+kfrd6oQ6SyBhF1aNm0LjdItJETu/4zd9oagNRNDQZnNbCkgJcDP/oem+LG7Ln BWC+cHNvuzMVlH5B1NPR7umvClwGshQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-308-O_LLgdEzO2yAjxB50bIomw-1; Tue, 19 Dec 2023 02:57:17 -0500 X-MC-Unique: O_LLgdEzO2yAjxB50bIomw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 94A54837221; Tue, 19 Dec 2023 07:57:16 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.117]) by smtp.corp.redhat.com (Postfix) with ESMTP id B40302026D66; Tue, 19 Dec 2023 07:57:05 +0000 (UTC) From: peterx@redhat.com To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Christophe Leroy , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Mike Kravetz , Mike Rapoport , Christoph Hellwig , John Hubbard , Andrew Jones , linux-arm-kernel@lists.infradead.org, Michael Ellerman , "Kirill A . Shutemov" , linuxppc-dev@lists.ozlabs.org, Rik van Riel , linux-riscv@lists.infradead.org, Yang Shi , James Houghton , "Aneesh Kumar K . V" , Andrew Morton , Jason Gunthorpe , Andrea Arcangeli , peterx@redhat.com, Axel Rasmussen Subject: [PATCH 07/13] mm/gup: Refactor record_subpages() to find 1st small page Date: Tue, 19 Dec 2023 15:55:32 +0800 Message-ID: <20231219075538.414708-8-peterx@redhat.com> In-Reply-To: <20231219075538.414708-1-peterx@redhat.com> References: <20231219075538.414708-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785696309380115709 X-GMAIL-MSGID: 1785696309380115709 From: Peter Xu All the fast-gup functions take a tail page to operate, always need to do page mask calculations before feeding that into record_subpages(). Merge that logic into record_subpages(), so that it will do the nth_page() calculation. Signed-off-by: Peter Xu --- mm/gup.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index bb5b7134f10b..82d28d517d0d 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2767,13 +2767,16 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long addr, - unsigned long end, struct page **pages) +static int record_subpages(struct page *page, unsigned long sz, + unsigned long addr, unsigned long end, + struct page **pages) { + struct page *start_page; int nr; + start_page = nth_page(page, (addr & (sz - 1)) >> PAGE_SHIFT); for (nr = 0; addr != end; nr++, addr += PAGE_SIZE) - pages[nr] = nth_page(page, nr); + pages[nr] = nth_page(start_page, nr); return nr; } @@ -2808,8 +2811,8 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); - page = nth_page(pte_page(pte), (addr & (sz - 1)) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pte_page(pte); + refs = record_subpages(page, sz, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2882,8 +2885,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, pages, nr); } - page = nth_page(pmd_page(orig), (addr & ~PMD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pmd_page(orig); + refs = record_subpages(page, PMD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2926,8 +2929,8 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, pages, nr); } - page = nth_page(pud_page(orig), (addr & ~PUD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pud_page(orig); + refs = record_subpages(page, PUD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2966,8 +2969,8 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, BUILD_BUG_ON(pgd_devmap(orig)); - page = nth_page(pgd_page(orig), (addr & ~PGDIR_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pgd_page(orig); + refs = record_subpages(page, PGDIR_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) From patchwork Tue Dec 19 07:55:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 180805 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:24d3:b0:fb:cd0c:d3e with SMTP id r19csp1778882dyi; Mon, 18 Dec 2023 23:58:43 -0800 (PST) X-Google-Smtp-Source: AGHT+IGXeJXzScg2VYWw2SazoQk80dDYfrbZIqsZD6Z9R1h/R8bWoDPPb1bGwWgXbN1FQ+OcmKeu X-Received: by 2002:a05:6a20:7f9b:b0:18f:97c:9272 with SMTP id d27-20020a056a207f9b00b0018f097c9272mr9839471pzj.87.1702972722645; Mon, 18 Dec 2023 23:58:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702972722; cv=none; d=google.com; s=arc-20160816; b=sSCq5MdQX9zjyUlmyfweU69nvza4cjrk6fmMUm2iJLHAwoEWEbFIj55Y7RgHxytXiE dCcHeA701yWzBq9zhdTybUOZUor/uKf/QCUUIoD2dN7bqH7kdLRnT65gYd6yCpzavz7k 7uXpCAdAnq5r/rb9+wS/MiMo+2dHjjxKDNsDJEb2nKNyz31lByxwdHSv/Zg9qWhE4sjG +lGiGHb0Iqe9qqeqICYwugkMtIydVhtoG3tkcktxIVrsObafhq4oD4g9RGx/aCIcsrvY 40saLimEnoR+6RcpJkkQZ9SD40Eqjxt4KmWFKpr0ZwgZ19d9cepOVxfkKumKvYPSnh/I JOzQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=WpRYnsdxCsgwENogJmMAQBWBGi+Y3txosu8YBUBJ2nU=; fh=3mpu6Yyr5hMzDKtkQvzcApl7ZSQGU4G8QyIY2BAU7VY=; b=CTKTvls7Halavy58RjnkHzqJFf6WAjN+Ua/b9pANvDlPaVONFk343xFJuWCkIqRF5G ssHk48GmqctG8Mf7AdlQpPO04jxUHpWP0YS0mqpvMGMBRK7z9hNpJAYHQcCjLe1WuVL/ 30VuryF8tbZcVm2a4o5YwxLHaw32jiLp/pzx/SKJYsdmk0SOaHDifoI1FjN2T9soyIsE dzvfJMKQU4FEBkeJHxxQ0BcQrM6Mph9hE8l31B16ngQW/R5VGzMHYUK4kB+8ZsCIue8m 1Ct+JRxjP9Zr2Ic1gNPdDIupj5F7uVPdD0EPGZJygnXMAwE42qer57qFncSWnegrOicP qRJg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=AwC4wL1G; spf=pass (google.com: domain of linux-kernel+bounces-4842-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4842-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id d14-20020a170903230e00b001d3dbb38f80si934371plh.340.2023.12.18.23.58.42 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 23:58:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-4842-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=AwC4wL1G; spf=pass (google.com: domain of linux-kernel+bounces-4842-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4842-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 68313286B89 for ; Tue, 19 Dec 2023 07:58:42 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8586014288; Tue, 19 Dec 2023 07:57:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="AwC4wL1G" X-Original-To: linux-kernel@vger.kernel.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C25413AD7 for ; Tue, 19 Dec 2023 07:57:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702972652; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WpRYnsdxCsgwENogJmMAQBWBGi+Y3txosu8YBUBJ2nU=; b=AwC4wL1GCYaJKmrhLgDOJRag36CuxxOas9GVRphz0FEJpkecZQNnOaQVlMRHf7/GuQsSR1 qCUdkvNy6ZM+2+6uwmpRFq1CboMaQHOsRgpRMkrOY8Mw8EIRyPz2AUKV9UmtOliog1naXA ZK79211VtHl7eIe/66XjTNseEBETzc4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-650-6W9nSAUnNQOFp9mej7E_hg-1; Tue, 19 Dec 2023 02:57:29 -0500 X-MC-Unique: 6W9nSAUnNQOFp9mej7E_hg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1341F83B8E5; Tue, 19 Dec 2023 07:57:28 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.117]) by smtp.corp.redhat.com (Postfix) with ESMTP id 672D22026D66; Tue, 19 Dec 2023 07:57:17 +0000 (UTC) From: peterx@redhat.com To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Christophe Leroy , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Mike Kravetz , Mike Rapoport , Christoph Hellwig , John Hubbard , Andrew Jones , linux-arm-kernel@lists.infradead.org, Michael Ellerman , "Kirill A . Shutemov" , linuxppc-dev@lists.ozlabs.org, Rik van Riel , linux-riscv@lists.infradead.org, Yang Shi , James Houghton , "Aneesh Kumar K . V" , Andrew Morton , Jason Gunthorpe , Andrea Arcangeli , peterx@redhat.com, Axel Rasmussen Subject: [PATCH 08/13] mm/gup: Handle hugetlb for no_page_table() Date: Tue, 19 Dec 2023 15:55:33 +0800 Message-ID: <20231219075538.414708-9-peterx@redhat.com> In-Reply-To: <20231219075538.414708-1-peterx@redhat.com> References: <20231219075538.414708-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785696325245109987 X-GMAIL-MSGID: 1785696325245109987 From: Peter Xu no_page_table() is not yet used for hugetlb code paths. Make it prepared. The major difference here is hugetlb will return -EFAULT as long as page cache does not exist, even if VM_SHARED. See hugetlb_follow_page_mask(). Pass "address" into no_page_table() too, as hugetlb will need it. Reviewed-by: Christoph Hellwig Signed-off-by: Peter Xu --- mm/gup.c | 44 ++++++++++++++++++++++++++------------------ 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 82d28d517d0d..6c0d82fa8cc7 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -501,19 +501,27 @@ static inline void mm_set_has_pinned_flag(unsigned long *mm_flags) #ifdef CONFIG_MMU static struct page *no_page_table(struct vm_area_struct *vma, - unsigned int flags) + unsigned int flags, unsigned long address) { + if (!(flags & FOLL_DUMP)) + return NULL; + /* - * When core dumping an enormous anonymous area that nobody - * has touched so far, we don't want to allocate unnecessary pages or + * When core dumping, we don't want to allocate unnecessary pages or * page tables. Return error instead of NULL to skip handle_mm_fault, * then get_dump_page() will return NULL to leave a hole in the dump. * But we can only make this optimization where a hole would surely * be zero-filled if handle_mm_fault() actually did handle it. */ - if ((flags & FOLL_DUMP) && - (vma_is_anonymous(vma) || !vma->vm_ops->fault)) + if (is_vm_hugetlb_page(vma)) { + struct hstate *h = hstate_vma(vma); + + if (!hugetlbfs_pagecache_present(h, vma, address)) + return ERR_PTR(-EFAULT); + } else if ((vma_is_anonymous(vma) || !vma->vm_ops->fault)) { return ERR_PTR(-EFAULT); + } + return NULL; } @@ -593,7 +601,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, ptep = pte_offset_map_lock(mm, pmd, address, &ptl); if (!ptep) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); pte = ptep_get(ptep); if (!pte_present(pte)) goto no_page; @@ -685,7 +693,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, pte_unmap_unlock(ptep, ptl); if (!pte_none(pte)) return NULL; - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); } static struct page *follow_pmd_mask(struct vm_area_struct *vma, @@ -701,27 +709,27 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, pmd = pmd_offset(pudp, address); pmdval = pmdp_get_lockless(pmd); if (pmd_none(pmdval)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); if (!pmd_present(pmdval)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); if (pmd_devmap(pmdval)) { ptl = pmd_lock(mm, pmd); page = follow_devmap_pmd(vma, address, pmd, flags, &ctx->pgmap); spin_unlock(ptl); if (page) return page; - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); } if (likely(!pmd_trans_huge(pmdval))) return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); if (pmd_protnone(pmdval) && !gup_can_follow_protnone(vma, flags)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); ptl = pmd_lock(mm, pmd); if (unlikely(!pmd_present(*pmd))) { spin_unlock(ptl); - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); } if (unlikely(!pmd_trans_huge(*pmd))) { spin_unlock(ptl); @@ -752,17 +760,17 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, pud = pud_offset(p4dp, address); if (pud_none(*pud)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); if (pud_devmap(*pud)) { ptl = pud_lock(mm, pud); page = follow_devmap_pud(vma, address, pud, flags, &ctx->pgmap); spin_unlock(ptl); if (page) return page; - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); } if (unlikely(pud_bad(*pud))) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); return follow_pmd_mask(vma, address, pud, flags, ctx); } @@ -776,10 +784,10 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma, p4d = p4d_offset(pgdp, address); if (p4d_none(*p4d)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); BUILD_BUG_ON(p4d_huge(*p4d)); if (unlikely(p4d_bad(*p4d))) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); return follow_pud_mask(vma, address, p4d, flags, ctx); } @@ -829,7 +837,7 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, pgd = pgd_offset(mm, address); if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); return follow_p4d_mask(vma, address, pgd, flags, ctx); } From patchwork Tue Dec 19 07:55:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 180806 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:24d3:b0:fb:cd0c:d3e with SMTP id r19csp1778920dyi; Mon, 18 Dec 2023 23:58:57 -0800 (PST) X-Google-Smtp-Source: AGHT+IHWAs7ZNinOZzdXrQ5k9HMc4SvUfKFF+bU4XLgBsd0438pLINIT2QNb87toKXlbnn9Dn6L5 X-Received: by 2002:a05:6358:591f:b0:170:ca20:6fd with SMTP id g31-20020a056358591f00b00170ca2006fdmr20512746rwf.62.1702972737113; Mon, 18 Dec 2023 23:58:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702972737; cv=none; d=google.com; s=arc-20160816; b=U8AiJntAZ+tw6+32QJt/RP02M5cXgEDA8itzNLJpYCn3CReQ671zyGcpH3sLHwfacY yxrsYOz3KlNastIAO7n45Mw+L9BbtH5ddrv7e19KxyrQNKEsjPqZKILQGTYC8PgkhVFU c4Y96PhzOINdpTrdWXYZ8erYw/ewfxJbqxtgQ4zM5lPWOKRxcck3W+xNSSkuzLEs+2Ac D3f0yuu/cmYfw4V7ZXAWpoKnzMziT/M7aMnzTG/0ut+3jcy9fU9cHlmK+8okScI3l/CG O/csFeZS1IMX2R+7kUeGb1wrfMMf+MGOGel7n9xkxu3BWWQgsYIqFVg7oizGsLAs//O9 XscQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=7UtlH+qJqHqzo7Y62yc3egVvRJlUYI/8xtDdmttQSUc=; fh=3mpu6Yyr5hMzDKtkQvzcApl7ZSQGU4G8QyIY2BAU7VY=; b=VuFt5FunACto/Kk7ybmdsBk35t4lpQKoFL3hMmnIL1OoIM0/SL/ZVozvNjlauRQZhP ygVd+kvjgwVnwqzXA91ZolsFPFBxKj9xxvy6ohWi4DHk9YjbOSs0W0Om6jGqBkvM5/7w RZbGBQQ7qpnvykWwsoNWjtUsVo15uHQoepRwQIIkaAKLnHE3g8sSY4ucUhFn4gpq6+oy HHeAoZ3xDyKk3/hNHp5ZvIocGtTuV83dMXERCmqR0lNvNX10Hn4iyjplJ5frbWVXKKIz OYVCR5ip9/WjdyY14a4tNDz/jpfxwH+fEa8FayvYoQ/k5lBBpC/HchhyuXgVv0e790QW 2b9w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Y5b7wywr; spf=pass (google.com: domain of linux-kernel+bounces-4843-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4843-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id a9-20020aa78e89000000b006d93838b44esi417173pfr.41.2023.12.18.23.58.56 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 23:58:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-4843-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Y5b7wywr; spf=pass (google.com: domain of linux-kernel+bounces-4843-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4843-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id D96F5286BC5 for ; Tue, 19 Dec 2023 07:58:56 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 519B214F69; Tue, 19 Dec 2023 07:57:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Y5b7wywr" X-Original-To: linux-kernel@vger.kernel.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 413501429A for ; Tue, 19 Dec 2023 07:57:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702972666; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7UtlH+qJqHqzo7Y62yc3egVvRJlUYI/8xtDdmttQSUc=; b=Y5b7wywrl16gpY3+tzbsVsuvPxD5fZXreYpvcGhI4HODV4jgYBqgD2MDW6GhvWJiEOdbc3 flgTQM1SNdI+rbNOjOc9rCpkh0EsCijIMtev+gG9mcLn2HnW+NZ4w1SSYyaR5XJfadxxHE KRe6IX2k5lPo69getGFoq8tln6Qpocg= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-197-HHPXr_gTMGKJxc1F8Y3Vvw-1; Tue, 19 Dec 2023 02:57:42 -0500 X-MC-Unique: HHPXr_gTMGKJxc1F8Y3Vvw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 13EAF1C0513F; Tue, 19 Dec 2023 07:57:41 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.117]) by smtp.corp.redhat.com (Postfix) with ESMTP id D699F2026D66; Tue, 19 Dec 2023 07:57:28 +0000 (UTC) From: peterx@redhat.com To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Christophe Leroy , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Mike Kravetz , Mike Rapoport , Christoph Hellwig , John Hubbard , Andrew Jones , linux-arm-kernel@lists.infradead.org, Michael Ellerman , "Kirill A . Shutemov" , linuxppc-dev@lists.ozlabs.org, Rik van Riel , linux-riscv@lists.infradead.org, Yang Shi , James Houghton , "Aneesh Kumar K . V" , Andrew Morton , Jason Gunthorpe , Andrea Arcangeli , peterx@redhat.com, Axel Rasmussen Subject: [PATCH 09/13] mm/gup: Cache *pudp in follow_pud_mask() Date: Tue, 19 Dec 2023 15:55:34 +0800 Message-ID: <20231219075538.414708-10-peterx@redhat.com> In-Reply-To: <20231219075538.414708-1-peterx@redhat.com> References: <20231219075538.414708-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785696341175676439 X-GMAIL-MSGID: 1785696341175676439 From: Peter Xu Introduce "pud_t pud" in the function, so the code won't dereference *pudp multiple time. Not only because that looks less straightforward, but also because if the dereference really happened, it's not clear whether there can be race to see different *pudp values if it's being modified at the same time. Signed-off-by: Peter Xu Acked-by: James Houghton --- mm/gup.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 6c0d82fa8cc7..97e87b7a15c3 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -753,26 +753,27 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, unsigned int flags, struct follow_page_context *ctx) { - pud_t *pud; + pud_t *pudp, pud; spinlock_t *ptl; struct page *page; struct mm_struct *mm = vma->vm_mm; - pud = pud_offset(p4dp, address); - if (pud_none(*pud)) + pudp = pud_offset(p4dp, address); + pud = *pudp; + if (pud_none(pud)) return no_page_table(vma, flags, address); - if (pud_devmap(*pud)) { - ptl = pud_lock(mm, pud); - page = follow_devmap_pud(vma, address, pud, flags, &ctx->pgmap); + if (pud_devmap(pud)) { + ptl = pud_lock(mm, pudp); + page = follow_devmap_pud(vma, address, pudp, flags, &ctx->pgmap); spin_unlock(ptl); if (page) return page; return no_page_table(vma, flags, address); } - if (unlikely(pud_bad(*pud))) + if (unlikely(pud_bad(pud))) return no_page_table(vma, flags, address); - return follow_pmd_mask(vma, address, pud, flags, ctx); + return follow_pmd_mask(vma, address, pudp, flags, ctx); } static struct page *follow_p4d_mask(struct vm_area_struct *vma, From patchwork Tue Dec 19 07:55:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 180807 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:24d3:b0:fb:cd0c:d3e with SMTP id r19csp1779362dyi; Tue, 19 Dec 2023 00:00:10 -0800 (PST) X-Google-Smtp-Source: AGHT+IEzn/Ai6dkqaKVMe4uQ6kJMUApAVbgQMp8witZ6iPQpAhVTR3K4DdhUCjSdyC/BIqRpsRdC X-Received: by 2002:a05:6a20:158b:b0:18f:d156:5b16 with SMTP id h11-20020a056a20158b00b0018fd1565b16mr10384249pzj.104.1702972809843; Tue, 19 Dec 2023 00:00:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702972809; cv=none; d=google.com; s=arc-20160816; b=x/0DFFqAZVsTe9+8VSYMWNqgwwm5ObsYMMe6f/P0A/m3aP0CfVTNChLWSBdJBcCUMc qtTIYRiusyekXJMxcmQwNNwUmAKbjIedQIKiYWhjQ6+zMDQmDXLLNKSgZjS6eKAbFqTa Vl0u+8ewI0RJo6AlwnXRofEWRPpkp+h9oSQ8v8bYXc/gLTL914vHty8WXvU+jtqWPwtN 51x6hw55Iuo8VuAL4YOlFlcyHH5K/M5343EGI3nbEzhAoxY46yiB75utgylamAZLQGWw jfesBMq8eIsddw+M39hERJ7MoDDDNSracQ6sHKVnMGLpBigBqxAF3QcBKKCKYN5VnzmO Swaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=5ijea/674RZI9IDvQT14lUfypWJKjpWiawe+yIIZEUw=; fh=3mpu6Yyr5hMzDKtkQvzcApl7ZSQGU4G8QyIY2BAU7VY=; b=MeO7XrWHe2s1gU8gxbvXZUq1n/7MMUNi4ydBkWqHExgLXh7y7scoeZeH1BFM1E4uu5 kQ5gbpNLgvwxAbsskAz3mvdxlIXE1dJSZkSFux3gNsOP26vmtvFu3IhE3hYRHI3jE7Xb 4TnpBz2EGjpuucs71wcIAnnKFPzJBaYMlZCSyYga2xhQZWsn0BuDBoSnaKO/Eb/I2029 Gs+FXSwzOPNiMs+3lqP7yhJCVnlNPcR+71gqXhDSM/3TD+BLc5Hs3ZaxV7Crh3+EDuOh lJf/w+7GoPL8i+nIS0qC10HibVTv2s8tkFEWNIS2TodgvZyh9/f0JdFmkNKe4tijb2j1 Gjcg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=GhwfpJAD; spf=pass (google.com: domain of linux-kernel+bounces-4844-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4844-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id iz5-20020a170902ef8500b001d3ab026b7bsi4102951plb.563.2023.12.19.00.00.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 00:00:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-4844-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=GhwfpJAD; spf=pass (google.com: domain of linux-kernel+bounces-4844-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4844-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id B8529284CC1 for ; Tue, 19 Dec 2023 07:59:17 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C767712B92; Tue, 19 Dec 2023 07:58:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="GhwfpJAD" X-Original-To: linux-kernel@vger.kernel.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7CFDF9E8 for ; Tue, 19 Dec 2023 07:58:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702972679; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5ijea/674RZI9IDvQT14lUfypWJKjpWiawe+yIIZEUw=; b=GhwfpJAD799sVupS2gFl1SLEI973BuMndT6WqbQzs2W++5vGSqrZruMGFni2LvATy+FCcF 9Fv3TOK7rsVLc3jSho9FtIk9bZVi79nJ4QknqCV1BL1vfXH/qNttK6c0urVIXNBlB7EtVU OXOIDNz7mfY+cs2TqzPrgw4XLjG92Ho= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-48-DALhkOU0MoKtInSHgoxvqA-1; Tue, 19 Dec 2023 02:57:54 -0500 X-MC-Unique: DALhkOU0MoKtInSHgoxvqA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1D1EE3806720; Tue, 19 Dec 2023 07:57:53 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.117]) by smtp.corp.redhat.com (Postfix) with ESMTP id BF70B2026D66; Tue, 19 Dec 2023 07:57:41 +0000 (UTC) From: peterx@redhat.com To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Christophe Leroy , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Mike Kravetz , Mike Rapoport , Christoph Hellwig , John Hubbard , Andrew Jones , linux-arm-kernel@lists.infradead.org, Michael Ellerman , "Kirill A . Shutemov" , linuxppc-dev@lists.ozlabs.org, Rik van Riel , linux-riscv@lists.infradead.org, Yang Shi , James Houghton , "Aneesh Kumar K . V" , Andrew Morton , Jason Gunthorpe , Andrea Arcangeli , peterx@redhat.com, Axel Rasmussen Subject: [PATCH 10/13] mm/gup: Handle huge pud for follow_pud_mask() Date: Tue, 19 Dec 2023 15:55:35 +0800 Message-ID: <20231219075538.414708-11-peterx@redhat.com> In-Reply-To: <20231219075538.414708-1-peterx@redhat.com> References: <20231219075538.414708-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785696417284405337 X-GMAIL-MSGID: 1785696417284405337 From: Peter Xu Teach follow_pud_mask() to be able to handle normal PUD pages like hugetlb. Rename follow_devmap_pud() to follow_huge_pud() so that it can process either huge devmap or hugetlb. Move it out of TRANSPARENT_HUGEPAGE_PUD and and huge_memory.c (which relies on CONFIG_THP). In the new follow_huge_pud(), taking care of possible CoR for hugetlb if necessary. touch_pud() needs to be moved out of huge_memory.c to be accessable from gup.c even if !THP. Since at it, optimize the non-present check by adding a pud_present() early check before taking the pgtable lock, failing the follow_page() early if PUD is not present: that is required by both devmap or hugetlb. Use pud_huge() to also cover the pud_devmap() case. One more trivial thing to mention is, introduce "pud_t pud" in the code paths along the way, so the code doesn't dereference *pudp multiple time. Not only because that looks less straightforward, but also because if the dereference really happened, it's not clear whether there can be race to see different *pudp values when it's being modified at the same time. Setting ctx->page_mask properly for a PUD entry. As a side effect, this patch should also be able to optimize devmap GUP on PUD to be able to jump over the whole PUD range, but not yet verified. Hugetlb already can do so prior to this patch. Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 8 ----- mm/gup.c | 70 +++++++++++++++++++++++++++++++++++++++-- mm/huge_memory.c | 47 ++------------------------- mm/internal.h | 2 ++ 4 files changed, 71 insertions(+), 56 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index d335130e145f..80f181d76f94 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -346,8 +346,6 @@ static inline bool folio_test_pmd_mappable(struct folio *folio) struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, int flags, struct dev_pagemap **pgmap); -struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, - pud_t *pud, int flags, struct dev_pagemap **pgmap); vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf); @@ -503,12 +501,6 @@ static inline struct page *follow_devmap_pmd(struct vm_area_struct *vma, return NULL; } -static inline struct page *follow_devmap_pud(struct vm_area_struct *vma, - unsigned long addr, pud_t *pud, int flags, struct dev_pagemap **pgmap) -{ - return NULL; -} - static inline bool thp_migration_supported(void) { return false; diff --git a/mm/gup.c b/mm/gup.c index 97e87b7a15c3..5b14f91d2f6b 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -525,6 +525,70 @@ static struct page *no_page_table(struct vm_area_struct *vma, return NULL; } +#ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES +static struct page *follow_huge_pud(struct vm_area_struct *vma, + unsigned long addr, pud_t *pudp, + int flags, struct follow_page_context *ctx) +{ + struct mm_struct *mm = vma->vm_mm; + struct page *page; + pud_t pud = *pudp; + unsigned long pfn = pud_pfn(pud); + int ret; + + assert_spin_locked(pud_lockptr(mm, pudp)); + + if ((flags & FOLL_WRITE) && !pud_write(pud)) + return NULL; + + if (!pud_present(pud)) + return NULL; + + pfn += (addr & ~PUD_MASK) >> PAGE_SHIFT; + +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD + if (pud_devmap(pud)) { + /* + * device mapped pages can only be returned if the caller + * will manage the page reference count. + * + * At least one of FOLL_GET | FOLL_PIN must be set, so + * assert that here: + */ + if (!(flags & (FOLL_GET | FOLL_PIN))) + return ERR_PTR(-EEXIST); + + if (flags & FOLL_TOUCH) + touch_pud(vma, addr, pudp, flags & FOLL_WRITE); + + ctx->pgmap = get_dev_pagemap(pfn, ctx->pgmap); + if (!ctx->pgmap) + return ERR_PTR(-EFAULT); + } +#endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ + page = pfn_to_page(pfn); + + if (!pud_devmap(pud) && !pud_write(pud) && + gup_must_unshare(vma, flags, page)) + return ERR_PTR(-EMLINK); + + ret = try_grab_page(page, flags); + if (ret) + page = ERR_PTR(ret); + else + ctx->page_mask = HPAGE_PUD_NR - 1; + + return page; +} +#else /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ +static struct page *follow_huge_pud(struct vm_area_struct *vma, + unsigned long addr, pud_t *pudp, + int flags, struct follow_page_context *ctx) +{ + return NULL; +} +#endif /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ + static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address, pte_t *pte, unsigned int flags) { @@ -760,11 +824,11 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, pudp = pud_offset(p4dp, address); pud = *pudp; - if (pud_none(pud)) + if (pud_none(pud) || !pud_present(pud)) return no_page_table(vma, flags, address); - if (pud_devmap(pud)) { + if (pud_huge(pud)) { ptl = pud_lock(mm, pudp); - page = follow_devmap_pud(vma, address, pudp, flags, &ctx->pgmap); + page = follow_huge_pud(vma, address, pudp, flags, ctx); spin_unlock(ptl); if (page) return page; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 6be1a380a298..def1dbe0d7e8 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1371,8 +1371,8 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, } #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD -static void touch_pud(struct vm_area_struct *vma, unsigned long addr, - pud_t *pud, bool write) +void touch_pud(struct vm_area_struct *vma, unsigned long addr, + pud_t *pud, bool write) { pud_t _pud; @@ -1384,49 +1384,6 @@ static void touch_pud(struct vm_area_struct *vma, unsigned long addr, update_mmu_cache_pud(vma, addr, pud); } -struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, - pud_t *pud, int flags, struct dev_pagemap **pgmap) -{ - unsigned long pfn = pud_pfn(*pud); - struct mm_struct *mm = vma->vm_mm; - struct page *page; - int ret; - - assert_spin_locked(pud_lockptr(mm, pud)); - - if (flags & FOLL_WRITE && !pud_write(*pud)) - return NULL; - - if (pud_present(*pud) && pud_devmap(*pud)) - /* pass */; - else - return NULL; - - if (flags & FOLL_TOUCH) - touch_pud(vma, addr, pud, flags & FOLL_WRITE); - - /* - * device mapped pages can only be returned if the - * caller will manage the page reference count. - * - * At least one of FOLL_GET | FOLL_PIN must be set, so assert that here: - */ - if (!(flags & (FOLL_GET | FOLL_PIN))) - return ERR_PTR(-EEXIST); - - pfn += (addr & ~PUD_MASK) >> PAGE_SHIFT; - *pgmap = get_dev_pagemap(pfn, *pgmap); - if (!*pgmap) - return ERR_PTR(-EFAULT); - page = pfn_to_page(pfn); - - ret = try_grab_page(page, flags); - if (ret) - page = ERR_PTR(ret); - - return page; -} - int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, pud_t *dst_pud, pud_t *src_pud, unsigned long addr, struct vm_area_struct *vma) diff --git a/mm/internal.h b/mm/internal.h index 222e63b2dea4..2fca14553d0f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1007,6 +1007,8 @@ int __must_check try_grab_page(struct page *page, unsigned int flags); /* * mm/huge_memory.c */ +void touch_pud(struct vm_area_struct *vma, unsigned long addr, + pud_t *pud, bool write); struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, unsigned int flags); From patchwork Tue Dec 19 07:55:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 180813 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:24d3:b0:fb:cd0c:d3e with SMTP id r19csp1782735dyi; Tue, 19 Dec 2023 00:06:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IHv0Nn+AtV4P4ahMrY/RPOXAx8BOcY+hCZfOtTKfzjr1ojV91/WFVXiFE/O+vw1DRRwubFa X-Received: by 2002:a05:651c:222:b0:2cc:7692:1008 with SMTP id z2-20020a05651c022200b002cc76921008mr1268614ljn.2.1702973174660; Tue, 19 Dec 2023 00:06:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702973174; cv=none; d=google.com; s=arc-20160816; b=ewzkLiLrBKtWjBCa1F+FdvkjGgO6UcJ+JwhxCVFs/AW2Mkg9VoYmbtaV8vhf86Eh0P yFgeDvoBWTbWyWjN1ic6h6WgXYm9P2h/7rYu9KSIOf0MGTncY8sQ9+IZtykcqJll7aNk 2x0fjvCQcgHyt2TdgYRL4p3nwTQVLUTkWzGPDfkL5r3lfbZXD5WIkdfvA4Paqa+AlA87 vT2UPPZTmbF31vqRIoIHhAFVVAySJMGYFNIdS04VTHxzKXI+J3OgttqEGR+zSWrXcmY5 0v9gseEqsvpr6WPCBrmQX3U9gumbGx58cuoPCwclOPEVCSoVzMlTe/1nJdPww4iKzlGh 8m3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=z+0dGl/nMUgxPM498YSXtipz9Gb3hgqO7wNjJOzkT+E=; fh=3mpu6Yyr5hMzDKtkQvzcApl7ZSQGU4G8QyIY2BAU7VY=; b=ltPOXNt+jzPlRTb7ThOwiNpxibsKeFTNqSIv15apBGxCwXtXD5Ijl7FhGX3vyhAFec fgOAvTJbH25cBYgDSHby7ZRAnAZ1X7d25ruC+ZIHujGXU1/a45RQu2ptXMeqNvW8mP80 4XN/da4+hq5HOEbIYLsTZUwoachqEhcxf+4HVyyDYYbrkoD0E0BaAEeINjEA7VLt0LGM a2qaXG35nFiCNCFboVs39kJJ32MZq4rIuX8c0ySt+fjsO/S8UQHK78g8SQN7JxuKxrkJ cAE4GX507iMAXbO+IE6RgaGGoV4rD2cdptvPyXuqmT4F3zpy+u/yplkOQoTtsaCXYIfi ALGw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=WJ5uv6AI; spf=pass (google.com: domain of linux-kernel+bounces-4845-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4845-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id 13-20020a0564021f4d00b0055297dd8703si4128551edz.447.2023.12.19.00.06.14 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 00:06:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-4845-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=WJ5uv6AI; spf=pass (google.com: domain of linux-kernel+bounces-4845-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4845-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 8189C1F275F8 for ; Tue, 19 Dec 2023 07:59:34 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id EBB6D1549C; Tue, 19 Dec 2023 07:58:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="WJ5uv6AI" X-Original-To: linux-kernel@vger.kernel.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1281314276 for ; Tue, 19 Dec 2023 07:58:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702972689; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z+0dGl/nMUgxPM498YSXtipz9Gb3hgqO7wNjJOzkT+E=; b=WJ5uv6AILeMmCTjstbftPVsgN9LHSPpHbjYi2ptjB0nC8o39tMeo4GfnAnQ2wJhhFQuqG/ xA4RohhIoz5tWuf6a6b3y7luB4KPXpKdYfn/rqtcSlsFxhxaX1FzYUJxrKE5610Ljz2t9V wlSWl3Cnda/sLwKss7seEWQ9httt7Rw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-678-9aIBjkYHNgawgyAMHjSpHg-1; Tue, 19 Dec 2023 02:58:05 -0500 X-MC-Unique: 9aIBjkYHNgawgyAMHjSpHg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 82F6C83BA88; Tue, 19 Dec 2023 07:58:04 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.117]) by smtp.corp.redhat.com (Postfix) with ESMTP id B7CD92026D66; Tue, 19 Dec 2023 07:57:53 +0000 (UTC) From: peterx@redhat.com To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Christophe Leroy , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Mike Kravetz , Mike Rapoport , Christoph Hellwig , John Hubbard , Andrew Jones , linux-arm-kernel@lists.infradead.org, Michael Ellerman , "Kirill A . Shutemov" , linuxppc-dev@lists.ozlabs.org, Rik van Riel , linux-riscv@lists.infradead.org, Yang Shi , James Houghton , "Aneesh Kumar K . V" , Andrew Morton , Jason Gunthorpe , Andrea Arcangeli , peterx@redhat.com, Axel Rasmussen Subject: [PATCH 11/13] mm/gup: Handle huge pmd for follow_pmd_mask() Date: Tue, 19 Dec 2023 15:55:36 +0800 Message-ID: <20231219075538.414708-12-peterx@redhat.com> In-Reply-To: <20231219075538.414708-1-peterx@redhat.com> References: <20231219075538.414708-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785696799191983162 X-GMAIL-MSGID: 1785696799191983162 From: Peter Xu Replace pmd_trans_huge() with pmd_thp_or_huge() to also cover pmd_huge() as long as enabled. FOLL_TOUCH and FOLL_SPLIT_PMD only apply to THP, not yet huge. Since now follow_trans_huge_pmd() can process hugetlb pages, renaming it into follow_huge_pmd() to match what it does. Move it into gup.c so not depend on CONFIG_THP. When at it, move the ctx->page_mask setup into follow_huge_pmd(), only set it when the page is valid. It was not a bug to set it before even if GUP failed (page==NULL), because follow_page_mask() callers always ignores page_mask if so. But doing so makes the code cleaner. Signed-off-by: Peter Xu --- mm/gup.c | 107 ++++++++++++++++++++++++++++++++++++++++++++--- mm/huge_memory.c | 86 +------------------------------------ mm/internal.h | 5 +-- 3 files changed, 105 insertions(+), 93 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 5b14f91d2f6b..080dff79b650 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -580,6 +580,93 @@ static struct page *follow_huge_pud(struct vm_area_struct *vma, return page; } + +/* FOLL_FORCE can write to even unwritable PMDs in COW mappings. */ +static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page, + struct vm_area_struct *vma, + unsigned int flags) +{ + /* If the pmd is writable, we can write to the page. */ + if (pmd_write(pmd)) + return true; + + /* Maybe FOLL_FORCE is set to override it? */ + if (!(flags & FOLL_FORCE)) + return false; + + /* But FOLL_FORCE has no effect on shared mappings */ + if (vma->vm_flags & (VM_MAYSHARE | VM_SHARED)) + return false; + + /* ... or read-only private ones */ + if (!(vma->vm_flags & VM_MAYWRITE)) + return false; + + /* ... or already writable ones that just need to take a write fault */ + if (vma->vm_flags & VM_WRITE) + return false; + + /* + * See can_change_pte_writable(): we broke COW and could map the page + * writable if we have an exclusive anonymous page ... + */ + if (!page || !PageAnon(page) || !PageAnonExclusive(page)) + return false; + + /* ... and a write-fault isn't required for other reasons. */ + if (vma_soft_dirty_enabled(vma) && !pmd_soft_dirty(pmd)) + return false; + return !userfaultfd_huge_pmd_wp(vma, pmd); +} + +static struct page *follow_huge_pmd(struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmd, + unsigned int flags, + struct follow_page_context *ctx) +{ + struct mm_struct *mm = vma->vm_mm; + pmd_t pmdval = *pmd; + struct page *page; + int ret; + + assert_spin_locked(pmd_lockptr(mm, pmd)); + + page = pmd_page(pmdval); + VM_BUG_ON_PAGE(!PageHead(page) && !is_zone_device_page(page), page); + + if ((flags & FOLL_WRITE) && + !can_follow_write_pmd(pmdval, page, vma, flags)) + return NULL; + + /* Avoid dumping huge zero page */ + if ((flags & FOLL_DUMP) && is_huge_zero_pmd(pmdval)) + return ERR_PTR(-EFAULT); + + if (pmd_protnone(*pmd) && !gup_can_follow_protnone(vma, flags)) + return NULL; + + if (!pmd_write(pmdval) && gup_must_unshare(vma, flags, page)) + return ERR_PTR(-EMLINK); + + VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && + !PageAnonExclusive(page), page); + + ret = try_grab_page(page, flags); + if (ret) + return ERR_PTR(ret); + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + if (pmd_trans_huge(pmdval) && (flags & FOLL_TOUCH)) + touch_pmd(vma, addr, pmd, flags & FOLL_WRITE); +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + + page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT; + ctx->page_mask = HPAGE_PMD_NR - 1; + VM_BUG_ON_PAGE(!PageCompound(page) && !is_zone_device_page(page), page); + + return page; +} + #else /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ static struct page *follow_huge_pud(struct vm_area_struct *vma, unsigned long addr, pud_t *pudp, @@ -587,6 +674,14 @@ static struct page *follow_huge_pud(struct vm_area_struct *vma, { return NULL; } + +static struct page *follow_huge_pmd(struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmd, + unsigned int flags, + struct follow_page_context *ctx) +{ + return NULL; +} #endif /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ static int follow_pfn_pte(struct vm_area_struct *vma, unsigned long address, @@ -784,31 +879,31 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, return page; return no_page_table(vma, flags, address); } - if (likely(!pmd_trans_huge(pmdval))) + if (likely(!pmd_thp_or_huge(pmdval))) return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); if (pmd_protnone(pmdval) && !gup_can_follow_protnone(vma, flags)) return no_page_table(vma, flags, address); ptl = pmd_lock(mm, pmd); - if (unlikely(!pmd_present(*pmd))) { + pmdval = *pmd; + if (unlikely(!pmd_present(pmdval))) { spin_unlock(ptl); return no_page_table(vma, flags, address); } - if (unlikely(!pmd_trans_huge(*pmd))) { + if (unlikely(!pmd_thp_or_huge(pmdval))) { spin_unlock(ptl); return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } - if (flags & FOLL_SPLIT_PMD) { + if (pmd_trans_huge(pmdval) && (flags & FOLL_SPLIT_PMD)) { spin_unlock(ptl); split_huge_pmd(vma, pmd, address); /* If pmd was left empty, stuff a page table in there quickly */ return pte_alloc(mm, pmd) ? ERR_PTR(-ENOMEM) : follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } - page = follow_trans_huge_pmd(vma, address, pmd, flags); + page = follow_huge_pmd(vma, address, pmd, flags, ctx); spin_unlock(ptl); - ctx->page_mask = HPAGE_PMD_NR - 1; return page; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index def1dbe0d7e8..930c59d7ceab 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1216,8 +1216,8 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write) EXPORT_SYMBOL_GPL(vmf_insert_pfn_pud); #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ -static void touch_pmd(struct vm_area_struct *vma, unsigned long addr, - pmd_t *pmd, bool write) +void touch_pmd(struct vm_area_struct *vma, unsigned long addr, + pmd_t *pmd, bool write) { pmd_t _pmd; @@ -1570,88 +1570,6 @@ static inline bool can_change_pmd_writable(struct vm_area_struct *vma, return pmd_dirty(pmd); } -/* FOLL_FORCE can write to even unwritable PMDs in COW mappings. */ -static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page, - struct vm_area_struct *vma, - unsigned int flags) -{ - /* If the pmd is writable, we can write to the page. */ - if (pmd_write(pmd)) - return true; - - /* Maybe FOLL_FORCE is set to override it? */ - if (!(flags & FOLL_FORCE)) - return false; - - /* But FOLL_FORCE has no effect on shared mappings */ - if (vma->vm_flags & (VM_MAYSHARE | VM_SHARED)) - return false; - - /* ... or read-only private ones */ - if (!(vma->vm_flags & VM_MAYWRITE)) - return false; - - /* ... or already writable ones that just need to take a write fault */ - if (vma->vm_flags & VM_WRITE) - return false; - - /* - * See can_change_pte_writable(): we broke COW and could map the page - * writable if we have an exclusive anonymous page ... - */ - if (!page || !PageAnon(page) || !PageAnonExclusive(page)) - return false; - - /* ... and a write-fault isn't required for other reasons. */ - if (vma_soft_dirty_enabled(vma) && !pmd_soft_dirty(pmd)) - return false; - return !userfaultfd_huge_pmd_wp(vma, pmd); -} - -struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, - unsigned long addr, - pmd_t *pmd, - unsigned int flags) -{ - struct mm_struct *mm = vma->vm_mm; - struct page *page; - int ret; - - assert_spin_locked(pmd_lockptr(mm, pmd)); - - page = pmd_page(*pmd); - VM_BUG_ON_PAGE(!PageHead(page) && !is_zone_device_page(page), page); - - if ((flags & FOLL_WRITE) && - !can_follow_write_pmd(*pmd, page, vma, flags)) - return NULL; - - /* Avoid dumping huge zero page */ - if ((flags & FOLL_DUMP) && is_huge_zero_pmd(*pmd)) - return ERR_PTR(-EFAULT); - - if (pmd_protnone(*pmd) && !gup_can_follow_protnone(vma, flags)) - return NULL; - - if (!pmd_write(*pmd) && gup_must_unshare(vma, flags, page)) - return ERR_PTR(-EMLINK); - - VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && - !PageAnonExclusive(page), page); - - ret = try_grab_page(page, flags); - if (ret) - return ERR_PTR(ret); - - if (flags & FOLL_TOUCH) - touch_pmd(vma, addr, pmd, flags & FOLL_WRITE); - - page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT; - VM_BUG_ON_PAGE(!PageCompound(page) && !is_zone_device_page(page), page); - - return page; -} - /* NUMA hinting page fault entry point for trans huge pmds */ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) { diff --git a/mm/internal.h b/mm/internal.h index 2fca14553d0f..c0e953a1eb62 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1009,9 +1009,8 @@ int __must_check try_grab_page(struct page *page, unsigned int flags); */ void touch_pud(struct vm_area_struct *vma, unsigned long addr, pud_t *pud, bool write); -struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, - unsigned long addr, pmd_t *pmd, - unsigned int flags); +void touch_pmd(struct vm_area_struct *vma, unsigned long addr, + pmd_t *pmd, bool write); /* * mm/mmap.c From patchwork Tue Dec 19 07:55:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 180808 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:24d3:b0:fb:cd0c:d3e with SMTP id r19csp1779549dyi; Tue, 19 Dec 2023 00:00:29 -0800 (PST) X-Google-Smtp-Source: AGHT+IG9EgUVhjIPXRef2c51hMXMXZ4ybDDMdA9xXZ0mVYSYzm22sTX+wIa9hM/1K8Ifw94lGxnQ X-Received: by 2002:a05:6214:401a:b0:67e:ff50:f2b1 with SMTP id kd26-20020a056214401a00b0067eff50f2b1mr12953769qvb.90.1702972828806; Tue, 19 Dec 2023 00:00:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702972828; cv=none; d=google.com; s=arc-20160816; b=TcsQf4J2SXwUFLuo2dVSJgae8XleITt2t+0PUJatwJ4bXdTmqlZpI7m84faq2ccvcU O1nvlhwKN6zNRXswO2PKHrwUuagKdHIuCtiJb6FJ0QD36I/sc0BiIJ11PYsbzgC79/ss U5WsRD2tgDM7U4m/ZQgPePiWTxmKdE1c6xOiQ6ZuPb3q3ZhPUM4s+UCYJ4WkOhy8+3Za n5adv088lC5/TcNA6biI1b25Di8dxtab7dBZ1bBmL351DvkRySp6L82VSZUV8F0CjLUJ HRPQ9AQu3WdM9ww4QCQekkI19W/iA/XZ8CcsVmkWRn80RmpfwAShGMLIzqyUGfuLk6GS IhCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=z9BJA067XqZpd3Nf7GBMDnnzZoziQMnRZVuO9/Z3c+Q=; fh=3mpu6Yyr5hMzDKtkQvzcApl7ZSQGU4G8QyIY2BAU7VY=; b=0IXUoQHxUBTdL0Cm00xEFheDtk7pVoVe9MC0/yS/AGv1kHbAIQiczC6+1+wlFb96iO whMuECqRHK0N57FWuRxSVWCtY5rMM3YIDNplDelmguQ+/vlmzf/FPri2L87gDaqE/Bhi 8j2NHXIrMfwbxTBVcxL2TQmWDmI/3MlvrsYBxIjI4oAmdfZUcF7etMQ3tgwWpCPoandg 9SSxVxVF9VxUQpBwGQzBhRN2o27sVLgQYHOQ5aly8GEcZCbHfVTPtY6cpG0iacuUhO3e xKl21T8UvEj6qzoOGXyiBSyesy1aywYLuHsarBBlQkYNUYOr93DLuk1T3nY4TWWEYLuK u5XA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OTbHzUOh; spf=pass (google.com: domain of linux-kernel+bounces-4846-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4846-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id f10-20020a0cbeca000000b0067eec318cb1si15829600qvj.288.2023.12.19.00.00.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 00:00:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-4846-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OTbHzUOh; spf=pass (google.com: domain of linux-kernel+bounces-4846-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4846-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 3F47C1C23AC9 for ; Tue, 19 Dec 2023 07:59:50 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id F3F5312B7D; Tue, 19 Dec 2023 07:58:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="OTbHzUOh" X-Original-To: linux-kernel@vger.kernel.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 44B11F9E8 for ; Tue, 19 Dec 2023 07:58:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702972702; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z9BJA067XqZpd3Nf7GBMDnnzZoziQMnRZVuO9/Z3c+Q=; b=OTbHzUOhD4SSDuCOhNfkUyCQ8I22WiCN9HszJvBQZiO2UTcyp1bji6bC0rr0/nUv/OVOjb T26ATUEOh0iCI0SC5IKmNUOQgU8ynCbdWm/z7Jmr93+qefu3fNRZLB4PvDMX7Mj3pNU6/C O479d6k+PFO20begR3/oXMxM4G3vG9w= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-175-dRyYBC1tOkeHiKu36YFQHw-1; Tue, 19 Dec 2023 02:58:18 -0500 X-MC-Unique: dRyYBC1tOkeHiKu36YFQHw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7D4B63806720; Tue, 19 Dec 2023 07:58:16 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.117]) by smtp.corp.redhat.com (Postfix) with ESMTP id 528C52026D66; Tue, 19 Dec 2023 07:58:04 +0000 (UTC) From: peterx@redhat.com To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Christophe Leroy , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Mike Kravetz , Mike Rapoport , Christoph Hellwig , John Hubbard , Andrew Jones , linux-arm-kernel@lists.infradead.org, Michael Ellerman , "Kirill A . Shutemov" , linuxppc-dev@lists.ozlabs.org, Rik van Riel , linux-riscv@lists.infradead.org, Yang Shi , James Houghton , "Aneesh Kumar K . V" , Andrew Morton , Jason Gunthorpe , Andrea Arcangeli , peterx@redhat.com, Axel Rasmussen Subject: [PATCH 12/13] mm/gup: Handle hugepd for follow_page() Date: Tue, 19 Dec 2023 15:55:37 +0800 Message-ID: <20231219075538.414708-13-peterx@redhat.com> In-Reply-To: <20231219075538.414708-1-peterx@redhat.com> References: <20231219075538.414708-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785696437056053352 X-GMAIL-MSGID: 1785696437056053352 From: Peter Xu Hugepd is only used in PowerPC so far on 4K page size kernels where hash mmu is used. follow_page_mask() used to leverage hugetlb APIs to access hugepd entries. Teach follow_page_mask() itself on hugepd. With previous refactors on fast-gup gup_huge_pd(), most of the code can be easily leveraged. There's something not needed for follow page, for example, gup_hugepte() tries to detect pgtable entry change which will never happen with slow gup (which has the pgtable lock held), but that's not a problem to check. Since follow_page() always only fetch one page, set the end to "address + PAGE_SIZE" should suffice. We will still do the pgtable walk once for each hugetlb page by setting ctx->page_mask properly. One thing worth mentioning is that some level of pgtable's _bad() helper will report is_hugepd() entries as TRUE on Power8 hash MMUs. I think it at least applies to PUD on Power8 with 4K pgsize. It means feeding a hugepd entry to pud_bad() will report a false positive. Let's leave that for now because it can be arch-specific where I am a bit declined to touch. In this patch it's not a problem as long as hugepd is detected before any bad pgtable entries. Signed-off-by: Peter Xu --- mm/gup.c | 78 +++++++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 69 insertions(+), 9 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 080dff79b650..14a7d13e7bd6 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -30,6 +30,11 @@ struct follow_page_context { unsigned int page_mask; }; +static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, + unsigned long addr, unsigned int pdshift, + unsigned int flags, + struct follow_page_context *ctx); + static inline void sanity_check_pinned_pages(struct page **pages, unsigned long npages) { @@ -871,6 +876,9 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, return no_page_table(vma, flags, address); if (!pmd_present(pmdval)) return no_page_table(vma, flags, address); + if (unlikely(is_hugepd(__hugepd(pmd_val(pmdval))))) + return follow_hugepd(vma, __hugepd(pmd_val(pmdval)), + address, PMD_SHIFT, flags, ctx); if (pmd_devmap(pmdval)) { ptl = pmd_lock(mm, pmd); page = follow_devmap_pmd(vma, address, pmd, flags, &ctx->pgmap); @@ -921,6 +929,9 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, pud = *pudp; if (pud_none(pud) || !pud_present(pud)) return no_page_table(vma, flags, address); + if (unlikely(is_hugepd(__hugepd(pud_val(pud))))) + return follow_hugepd(vma, __hugepd(pud_val(pud)), + address, PUD_SHIFT, flags, ctx); if (pud_huge(pud)) { ptl = pud_lock(mm, pudp); page = follow_huge_pud(vma, address, pudp, flags, ctx); @@ -940,13 +951,17 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma, unsigned int flags, struct follow_page_context *ctx) { - p4d_t *p4d; + p4d_t *p4d, p4dval; p4d = p4d_offset(pgdp, address); - if (p4d_none(*p4d)) - return no_page_table(vma, flags, address); - BUILD_BUG_ON(p4d_huge(*p4d)); - if (unlikely(p4d_bad(*p4d))) + p4dval = *p4d; + BUILD_BUG_ON(p4d_huge(p4dval)); + + if (unlikely(is_hugepd(__hugepd(p4d_val(p4dval))))) + return follow_hugepd(vma, __hugepd(p4d_val(p4dval)), + address, P4D_SHIFT, flags, ctx); + + if (p4d_none(p4dval) || unlikely(p4d_bad(p4dval))) return no_page_table(vma, flags, address); return follow_pud_mask(vma, address, p4d, flags, ctx); @@ -980,7 +995,7 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, struct follow_page_context *ctx) { - pgd_t *pgd; + pgd_t *pgd, pgdval; struct mm_struct *mm = vma->vm_mm; ctx->page_mask = 0; @@ -995,11 +1010,17 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, &ctx->page_mask); pgd = pgd_offset(mm, address); + pgdval = *pgd; - if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - return no_page_table(vma, flags, address); + if (unlikely(is_hugepd(__hugepd(pgd_val(pgdval))))) + page = follow_hugepd(vma, __hugepd(pgd_val(pgdval)), + address, PGDIR_SHIFT, flags, ctx); + else if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) + page = no_page_table(vma, flags, address); + else + page = follow_p4d_mask(vma, address, pgd, flags, ctx); - return follow_p4d_mask(vma, address, pgd, flags, ctx); + return page; } struct page *follow_page(struct vm_area_struct *vma, unsigned long address, @@ -3026,6 +3047,37 @@ static int gup_huge_pd(hugepd_t hugepd, unsigned long addr, return 1; } + +static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, + unsigned long addr, unsigned int pdshift, + unsigned int flags, + struct follow_page_context *ctx) +{ + struct page *page; + struct hstate *h; + spinlock_t *ptl; + int nr = 0, ret; + pte_t *ptep; + + /* Only hugetlb supports hugepd */ + if (WARN_ON_ONCE(!is_vm_hugetlb_page(vma))) + return ERR_PTR(-EFAULT); + + h = hstate_vma(vma); + ptep = hugepte_offset(hugepd, addr, pdshift); + ptl = huge_pte_lock(h, vma->vm_mm, ptep); + ret = gup_huge_pd(hugepd, addr, pdshift, addr + PAGE_SIZE, + flags, &page, &nr); + spin_unlock(ptl); + + if (ret) { + WARN_ON_ONCE(nr != 1); + ctx->page_mask = (1U << huge_page_order(h)) - 1; + return page; + } + + return NULL; +} #else static inline int gup_huge_pd(hugepd_t hugepd, unsigned long addr, unsigned int pdshift, unsigned long end, unsigned int flags, @@ -3033,6 +3085,14 @@ static inline int gup_huge_pd(hugepd_t hugepd, unsigned long addr, { return 0; } + +static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, + unsigned long addr, unsigned int pdshift, + unsigned int flags, + struct follow_page_context *ctx) +{ + return NULL; +} #endif /* CONFIG_ARCH_HAS_HUGEPD */ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, From patchwork Tue Dec 19 07:55:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 180809 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:24d3:b0:fb:cd0c:d3e with SMTP id r19csp1779716dyi; Tue, 19 Dec 2023 00:00:44 -0800 (PST) X-Google-Smtp-Source: AGHT+IHDxHGsES+oPYV3MrZUjTi6JxnQo7KxwghlT/x4VCNU/2MLj1zguc4PpRu7c72YUZRFMcZW X-Received: by 2002:a05:620a:4e1:b0:77e:fba3:9d34 with SMTP id b1-20020a05620a04e100b0077efba39d34mr20504456qkh.152.1702972844138; Tue, 19 Dec 2023 00:00:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702972844; cv=none; d=google.com; s=arc-20160816; b=TJjL55Wm0w9Ca9CkuMw61A2oV3xgEeoLAjOKOuCXDxsBPOnqFVXkwAq4J4BSW9a34B 4rQoyxgvoomhTjHmG7Zl2oEnpLAyOle0HBAjsBMu2JVkOzSj70Q0xAO2EU2tIkUPBtb9 zCyJZMjOr9mk2+2TVZP5kl1TU+3e9SzkphBPIBcyVK0qydbQpH7xevoIM6o5mpLSxWWu KNW90O4E7LRcVCbsoAlKsh5Vd1fV2esgLOnCidk5KGTUy+fPJ6zypN+Fsd1KcACqfyvO 1WpoUtd3dZQMOQZk/Nefp98WBh3rGR4BjqaGO5Vthc7KIZSSCfDvhi8XKpZN8iP2dWZz KvPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=C+SULC41WW+Kd90iko+LfCmzwEshCatJrP9IQa4Pxio=; fh=3mpu6Yyr5hMzDKtkQvzcApl7ZSQGU4G8QyIY2BAU7VY=; b=GH0Dc0NgKYxjfxs2eQW5bwno8iwIu91LlddhsPuc73a7TydHlzKzd9fGY6z97YEl8z Gl1RcKnu0ViPc2ckBUCv41l5WEFlb1POv6UP0q+Kos7LFHZpdwnDMP7BAZ4z/Epm1fUT a14HbjUT+KYO0IDSKBvSGY8yhKqWgoX8e0MRigvXC7V4j5Kuw6iCH5IJzWI5iirkfGEF UYFiSLEtvSCEB0jzbWefj7AiCPaET+TepPD5POPrE9M0j96khyuJayi86iwoXsU16Jtb aK8Q443h0PJE2F6UTNQbK/8naEMKo2fXEi/IZrc0jrsv/7ubwG/M3YuAD6epsR/8gDvt aTfg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=HzGYs5D1; spf=pass (google.com: domain of linux-kernel+bounces-4848-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4848-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id pb22-20020a05620a839600b00774173d6bb4si26261305qkn.190.2023.12.19.00.00.44 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 00:00:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-4848-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=HzGYs5D1; spf=pass (google.com: domain of linux-kernel+bounces-4848-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-4848-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 99D441C2403E for ; Tue, 19 Dec 2023 08:00:19 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 326AB168CB; Tue, 19 Dec 2023 07:58:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="HzGYs5D1" X-Original-To: linux-kernel@vger.kernel.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8F29E156EA for ; Tue, 19 Dec 2023 07:58:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702972714; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=C+SULC41WW+Kd90iko+LfCmzwEshCatJrP9IQa4Pxio=; b=HzGYs5D1vLbUuJVmAykZPFsCzxrnjFycfrLCcmIOKg+r0BCb/RhiwCkwN6v8l5rTBdiMMC D6OEbZqVnMsZ2kJmReZCP5it19ahGc5eynOSTCEz4EtKWAi7ihjgqZgln949+/3ffJevPl SQADVOSweryz+2NcZdc9DCHxOHUDue4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-443-hR7CaQnoO4OWgrV4n0G44w-1; Tue, 19 Dec 2023 02:58:29 -0500 X-MC-Unique: hR7CaQnoO4OWgrV4n0G44w-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 462A5185A782; Tue, 19 Dec 2023 07:58:28 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.117]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4D2F42026D66; Tue, 19 Dec 2023 07:58:16 +0000 (UTC) From: peterx@redhat.com To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Christophe Leroy , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Mike Kravetz , Mike Rapoport , Christoph Hellwig , John Hubbard , Andrew Jones , linux-arm-kernel@lists.infradead.org, Michael Ellerman , "Kirill A . Shutemov" , linuxppc-dev@lists.ozlabs.org, Rik van Riel , linux-riscv@lists.infradead.org, Yang Shi , James Houghton , "Aneesh Kumar K . V" , Andrew Morton , Jason Gunthorpe , Andrea Arcangeli , peterx@redhat.com, Axel Rasmussen Subject: [PATCH 13/13] mm/gup: Handle hugetlb in the generic follow_page_mask code Date: Tue, 19 Dec 2023 15:55:38 +0800 Message-ID: <20231219075538.414708-14-peterx@redhat.com> In-Reply-To: <20231219075538.414708-1-peterx@redhat.com> References: <20231219075538.414708-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785696453072306744 X-GMAIL-MSGID: 1785696453072306744 From: Peter Xu Now follow_page() is ready to handle hugetlb pages in whatever form, and over all architectures. Switch to the generic code path. Time to retire hugetlb_follow_page_mask(), following the previous retirement of follow_hugetlb_page() in 4849807114b8. There may be a slight difference of how the loops run when processing slow GUP over a large hugetlb range on cont_pte/cont_pmd supported archs: each loop of __get_user_pages() will resolve one pgtable entry with the patch applied, rather than relying on the size of hugetlb hstate, the latter may cover multiple entries in one loop. A quick performance test on an aarch64 VM on M1 chip shows 15% degrade over a tight loop of slow gup after the path switched. That shouldn't be a problem because slow-gup should not be a hot path for GUP in general: when page is commonly present, fast-gup will already succeed, while when the page is indeed missing and require a follow up page fault, the slow gup degrade will probably buried in the fault paths anyway. It also explains why slow gup for THP used to be very slow before 57edfcfd3419 ("mm/gup: accelerate thp gup even for "pages != NULL"") lands, the latter not part of a performance analysis but a side benefit. If the performance will be a concern, we can consider handle CONT_PTE in follow_page(). Before that is justified to be necessary, keep everything clean and simple. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 7 ---- mm/gup.c | 15 +++------ mm/hugetlb.c | 71 ----------------------------------------- 3 files changed, 5 insertions(+), 88 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index f8c5c174c8a6..8a352d577ebf 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -332,13 +332,6 @@ static inline void hugetlb_zap_end( { } -static inline struct page *hugetlb_follow_page_mask( - struct vm_area_struct *vma, unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ -} - static inline int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *dst_vma, diff --git a/mm/gup.c b/mm/gup.c index 14a7d13e7bd6..f34c0a912311 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -997,18 +997,11 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, { pgd_t *pgd, pgdval; struct mm_struct *mm = vma->vm_mm; + struct page *page; - ctx->page_mask = 0; - - /* - * Call hugetlb_follow_page_mask for hugetlb vmas as it will use - * special hugetlb page table walking code. This eliminates the - * need to check for hugetlb entries in the general walking code. - */ - if (is_vm_hugetlb_page(vma)) - return hugetlb_follow_page_mask(vma, address, flags, - &ctx->page_mask); + vma_pgtable_walk_begin(vma); + ctx->page_mask = 0; pgd = pgd_offset(mm, address); pgdval = *pgd; @@ -1020,6 +1013,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, else page = follow_p4d_mask(vma, address, pgd, flags, ctx); + vma_pgtable_walk_end(vma); + return page; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 29705e5c6f40..3013122a739f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6783,77 +6783,6 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, } #endif /* CONFIG_USERFAULTFD */ -struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - struct hstate *h = hstate_vma(vma); - struct mm_struct *mm = vma->vm_mm; - unsigned long haddr = address & huge_page_mask(h); - struct page *page = NULL; - spinlock_t *ptl; - pte_t *pte, entry; - int ret; - - hugetlb_vma_lock_read(vma); - pte = hugetlb_walk(vma, haddr, huge_page_size(h)); - if (!pte) - goto out_unlock; - - ptl = huge_pte_lock(h, mm, pte); - entry = huge_ptep_get(pte); - if (pte_present(entry)) { - page = pte_page(entry); - - if (!huge_pte_write(entry)) { - if (flags & FOLL_WRITE) { - page = NULL; - goto out; - } - - if (gup_must_unshare(vma, flags, page)) { - /* Tell the caller to do unsharing */ - page = ERR_PTR(-EMLINK); - goto out; - } - } - - page = nth_page(page, ((address & ~huge_page_mask(h)) >> PAGE_SHIFT)); - - /* - * Note that page may be a sub-page, and with vmemmap - * optimizations the page struct may be read only. - * try_grab_page() will increase the ref count on the - * head page, so this will be OK. - * - * try_grab_page() should always be able to get the page here, - * because we hold the ptl lock and have verified pte_present(). - */ - ret = try_grab_page(page, flags); - - if (WARN_ON_ONCE(ret)) { - page = ERR_PTR(ret); - goto out; - } - - *page_mask = (1U << huge_page_order(h)) - 1; - } -out: - spin_unlock(ptl); -out_unlock: - hugetlb_vma_unlock_read(vma); - - /* - * Fixup retval for dump requests: if pagecache doesn't exist, - * don't try to allocate a new page but just skip it. - */ - if (!page && (flags & FOLL_DUMP) && - !hugetlbfs_pagecache_present(h, vma, address)) - page = ERR_PTR(-EFAULT); - - return page; -} - long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags)