From patchwork Thu Nov 16 01:28:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 165638 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2924146vqg; Wed, 15 Nov 2023 17:29:32 -0800 (PST) X-Google-Smtp-Source: AGHT+IE2HzDgGOBECzKL3A2kXJ3SzuedTAKmTlcu3QzOrWgD9admUs/ENLiKdf3yYvAIPDaATofn X-Received: by 2002:a17:902:9304:b0:1cc:47e5:b4bf with SMTP id bc4-20020a170902930400b001cc47e5b4bfmr7254286plb.1.1700098172320; Wed, 15 Nov 2023 17:29:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700098172; cv=none; d=google.com; s=arc-20160816; b=pJb2fQVsY9TOsxiXAhyov/54fQavqSh1XTHFQg0+A6eXIaXqa2dlkeFTB3NPE7eF8N 5N62gu3bcrYbQS9tf+IgX4IEBHvKboLPwjZdIrdmCHTceo7QnV8KXlhs/tQvxB6/NBL6 S8Hvv9DZ41TMq4ItsOhUMfmWal+Pr4dy1HKHEWN1814dSiZtZEtR11C3QQ40+Rsl2fpi yk4hmB+cv892SXjk5HkFtzK/nMj2LukVr8sptg9nzZA1Kybl0yXRv9gRpogwMNyiVFD6 WziXme7qLiUDmYtqM8MegSSUrDvHDABy8iDqIjCADGavp5wo/A7LhH2JJg/0noJEP+xj FQ1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=j75oQuDXytIzx0fg9CVxcXlN+sYlHIe4GGC/3d4Gln4=; fh=o+FJD7UTEFgAOwGBZppViFyhRpBzPGRfoO/6xzRNnMs=; b=iIxw0wrEGCsjPNbG9KJ8AE+p1tWfn1J2WSL3+t6nMNtTKLXlGZngk2840slbjQxsn3 u9vHC73fD9bbs4rC7k6wDpZUif+NiN5lvOpxCRTif2DV5/EyCcszTB1UzX3Ui0Zk9Ryv S97mbMNfwjFebvwnwZwk2cHQKEWGwDF2asHbNGpIDQeSV5Mq+AaIMKA9w5fJR/z7EFxN Atl/J8Iufo7CCUOXoDK8nGb7HD8ZuLnedjeGNG0drfND6rNS0X0cL5IsksYzNDRQO/0e Sgc9jfMGgzuDWQhF30hhBR05jSykRKxccCkSNzLa/WNvrOBSOzciURsobB39hKwR4o5y ev1Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Ie72S6Vo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id jj2-20020a170903048200b001c9ad30e4b8si10560940plb.320.2023.11.15.17.29.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Ie72S6Vo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 467EE80FDDD4; Wed, 15 Nov 2023 17:29:31 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344286AbjKPB3V (ORCPT + 28 others); Wed, 15 Nov 2023 20:29:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54036 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229692AbjKPB3U (ORCPT ); Wed, 15 Nov 2023 20:29:20 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E40EF182 for ; Wed, 15 Nov 2023 17:29:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098156; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j75oQuDXytIzx0fg9CVxcXlN+sYlHIe4GGC/3d4Gln4=; b=Ie72S6VoYRKq1jIqB5k4Cg7EijAhC5kiEVGJJq6CW4edXDyTiofF9d7IVo7iaAL0p8csTV iCHLVk9yV4uFgp47JVd1Xk2fR1OtABnHnIhjQKjNP22ZkJQJc2kT+26RztJulSsk9rcwZ5 fvZYm5J7Zji6kKHgW1auw3r3X0Gnclg= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-644-vkNfz2qmMNeOn9C8UIiorA-1; Wed, 15 Nov 2023 20:29:13 -0500 X-MC-Unique: vkNfz2qmMNeOn9C8UIiorA-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-421ac2bad66so928371cf.0 for ; Wed, 15 Nov 2023 17:29:13 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098153; x=1700702953; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=j75oQuDXytIzx0fg9CVxcXlN+sYlHIe4GGC/3d4Gln4=; b=PVzn/HDuUtlLokOzRb3Y8cG9XAgFaD7I8VXmxOtHjdyg2LzV7y8KHIlieYq+uTf/As GeioEoN+0GVkdIjzJCPH+/ldPqG+6SWDGDST8qYiFNoviggtdW2NpyfdOnHSTZ0V6AS3 Jy+E+Top5Ih+s02E6E2KDCp/WTFRqZIY0uznzu4UzH0NhaXSIJLMF4LVITzWDpT63rdI ZlcJ9IbnOYQRA81/OSKTsLynvqv4jCmp+dynlf3vqyKrgU7ukfgNw248YhwXt2UPq+k/ kfFV9K53XhXH2ZuMixVtlnrYQ6CVS8WBklVvmdd+1h7zcYC9F+r/sv90gsRy2mQZJ1jd JGEw== X-Gm-Message-State: AOJu0Yw4tHR5e3n7TBF4zkfoCW+8qqr3yaQALyxaIkoPAwGMYytZRhQO KYAwJhIV/1lJPb/W0GGiUDzwdhG1Mh6FAoPp2PJxEp4ZxVfx/emRYgATX8v7wmrHPyfHmSaolVR juMhA2N9G5oH27NgeA4YCcr8khYMUmUpK3lfRyHSJ0dnRiuicyaDJbHJ4Ox4U9D1RTElNKslFCo 4wwjd/2A== X-Received: by 2002:ac8:7d13:0:b0:41e:4be2:d3eb with SMTP id g19-20020ac87d13000000b0041e4be2d3ebmr10348628qtb.1.1700098153180; Wed, 15 Nov 2023 17:29:13 -0800 (PST) X-Received: by 2002:ac8:7d13:0:b0:41e:4be2:d3eb with SMTP id g19-20020ac87d13000000b0041e4be2d3ebmr10348586qtb.1.1700098152747; Wed, 15 Nov 2023 17:29:12 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:12 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 01/12] mm/hugetlb: Export hugetlbfs_pagecache_present() Date: Wed, 15 Nov 2023 20:28:57 -0500 Message-ID: <20231116012908.392077-2-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 15 Nov 2023 17:29:31 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782682140926223685 X-GMAIL-MSGID: 1782682140926223685 It will be used outside hugetlb.c soon. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 3 +++ mm/hugetlb.c | 4 ++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 236ec7b63c54..bb07279b8991 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -174,6 +174,9 @@ u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx); pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, pud_t *pud); +bool hugetlbfs_pagecache_present(struct hstate *h, + struct vm_area_struct *vma, + unsigned long address); struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 6feb3e0630d1..29705e5c6f40 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6018,8 +6018,8 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* * Return whether there is a pagecache page to back given address within VMA. */ -static bool hugetlbfs_pagecache_present(struct hstate *h, - struct vm_area_struct *vma, unsigned long address) +bool hugetlbfs_pagecache_present(struct hstate *h, + struct vm_area_struct *vma, unsigned long address) { struct address_space *mapping = vma->vm_file->f_mapping; pgoff_t idx = linear_page_index(vma, address); From patchwork Thu Nov 16 01:28:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 165639 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2924159vqg; Wed, 15 Nov 2023 17:29:34 -0800 (PST) X-Google-Smtp-Source: AGHT+IFbVAxE12bY02miRR7wr2PYoKtfAidTk5A9coakNk8DeYdwGsmoVXzPd3WAm71Ggg3neLI9 X-Received: by 2002:a25:3617:0:b0:dae:263e:5007 with SMTP id d23-20020a253617000000b00dae263e5007mr14001602yba.52.1700098174222; Wed, 15 Nov 2023 17:29:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700098174; cv=none; d=google.com; s=arc-20160816; b=JfpPCTA4fSJaotlmidgFn7CYvz8ofwn0qlpoP9r+4VxziDJtfLxConEkg44IYRPyQH VrVKG7N/dheDokyUGXQchAgAmFfIu7Pzryblk1PCq9zGx3zZoQX6GJ1TQQ+9J1Y9uZxP aDrFcLCqw7trUTmXpolMiTVoq3veKCueQwLVIAQM3wARLMVLEZznebOAhuybeSvOCydu zWcqhyh3tk6H8pW0k84aDx+W90TtKSbpj2TZW/ex31aCygKbKCDOtMr3kG4aCrijKbOj 02HdbeA2a631gOue+lpf3cq6D6+zGH4XO/Xw2yxxYR2I2e7ypXamWK31RqsQFqXuicLN QVwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Rr9/qVNLgFGWDGzcAqjV00T9t0NzgxXDD6w387jcEYI=; fh=o+FJD7UTEFgAOwGBZppViFyhRpBzPGRfoO/6xzRNnMs=; b=Z2ugLTPJ24aif2ah0MgmfTqF/dsiaHI2SrcoSkcBv3AHU3VUOwNAgMQocwWnX41+ib SXnX0Y82wbk8wSRf1X01u/UZpo5Fk/4o8Bjwxt4YL6eQTCqnDs66zQc3PpktO7QxGx/s V9Ct4rkMIq5ahscUm2/sUxbbM7kZSqN5rshN8wykqLA8xbUfZhuDPu0SeVX8ejiVGA9e RFBgND9zQtg7dPYavw82jCkjb7jgU+kVqxzpVVLv7LSAKnj8WM/H0taOPxQv23b1E9CK 8bQUbWwD7XcePKCGeoVcn2MJgWu8zKR72fsk8UsFack62HBP1f8Jxd0efdxh+i7aGYv6 uBWA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="NpAcFyZ/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id eg8-20020a056a00800800b006c0587c1b22si11648761pfb.345.2023.11.15.17.29.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:34 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="NpAcFyZ/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 3B0F780FC738; Wed, 15 Nov 2023 17:29:33 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344339AbjKPB3X (ORCPT + 28 others); Wed, 15 Nov 2023 20:29:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54044 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232310AbjKPB3U (ORCPT ); Wed, 15 Nov 2023 20:29:20 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 630A8198 for ; Wed, 15 Nov 2023 17:29:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098156; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Rr9/qVNLgFGWDGzcAqjV00T9t0NzgxXDD6w387jcEYI=; b=NpAcFyZ/9LB3zx3oT/POa/0V2bySj9fzQ3llny5quZ0hpVWWU8UQvR7KYlxunEAezmlwtO I0S+oxN/XNvDnuzVJo5ZGx7xLRfFsohMFmNM8S1KFlAK2u7FAHL8pSBTv2l901oxbFj1CD B3qNCRouZ7dNzcuTKcW5l9ramCiuagk= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-648-k_u7GC7_NJ-Y33KqFAf5PA-1; Wed, 15 Nov 2023 20:29:15 -0500 X-MC-Unique: k_u7GC7_NJ-Y33KqFAf5PA-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-66d7b75c854so807456d6.1 for ; Wed, 15 Nov 2023 17:29:15 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098154; x=1700702954; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Rr9/qVNLgFGWDGzcAqjV00T9t0NzgxXDD6w387jcEYI=; b=BdPGsJeQu0Z0N2pfLtaUMJWeIHJtmzwICZki8zMTcpwuBZoZXBOWCCJkLuMExhbViZ BBN+fUb/AV2MkeUrvwZb1AYh3ur/+JMnevRCKD6E39QiGjX3PLhKKaqfmeKsHKZg2nd7 QwNFJw5lgTLcnsqDMVON9wJnd0Toc6r/4ulN1PkMnuPBKT3YpgVVVZtYychmM8vCxFFL Q0i0u6cKsOT2agxYmbLfQXWIcVLoSZEvQ3sQ0+0cRdDPlsjbAUn2DZUwhTARqmTniLJw ED72deEFFc92CSnKvfCbWWrekosI3d7nyYLh9Q4bHep9XLBh000m3Escdh1d7joeflic YMOw== X-Gm-Message-State: AOJu0YxCiidt9RGVCXI7Oa82aMBQjZsg/EVR0t0E1nsC4w7rQg5gM97k zXZ27y2OQSwMsg3vE8VniVHMMQckjRzhWYuvhlxnGgsxFZS7xhX/DB7kLPOLOfrYc9GRuGuxQpy zWjt6y8dOswCRi2Uy/hMjGsj+cYOcFDr6KJlGSC6eWQfZA0SY/m1zLjNMVtvv8CyWs+IqDSEhZO +qd+sHmg== X-Received: by 2002:ac8:5189:0:b0:421:c275:f917 with SMTP id c9-20020ac85189000000b00421c275f917mr7201803qtn.6.1700098154738; Wed, 15 Nov 2023 17:29:14 -0800 (PST) X-Received: by 2002:ac8:5189:0:b0:421:c275:f917 with SMTP id c9-20020ac85189000000b00421c275f917mr7201781qtn.6.1700098154444; Wed, 15 Nov 2023 17:29:14 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:14 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 02/12] mm: Provide generic pmd_thp_or_huge() Date: Wed, 15 Nov 2023 20:28:58 -0500 Message-ID: <20231116012908.392077-3-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 15 Nov 2023 17:29:33 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782682142871561439 X-GMAIL-MSGID: 1782682142871561439 ARM defines pmd_thp_or_huge(), detecting either a THP or a huge PMD. It can be a helpful helper if we want to merge more THP and hugetlb code paths. Make it a generic default implementation, only exist when CONFIG_MMU. Arch can overwrite it by defining its own version. For example, ARM's pgtable-2level.h defines it to always return false. Signed-off-by: Peter Xu --- include/linux/pgtable.h | 4 ++++ mm/gup.c | 3 +-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index af7639c3b0a3..6f2fa1977b8a 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1355,6 +1355,10 @@ static inline int pmd_write(pmd_t pmd) #endif /* pmd_write */ #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#ifndef pmd_thp_or_huge +#define pmd_thp_or_huge(pmd) (pmd_huge(pmd) || pmd_trans_huge(pmd)) +#endif + #ifndef pud_write static inline int pud_write(pud_t pud) { diff --git a/mm/gup.c b/mm/gup.c index 231711efa390..a8b73a8289ad 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -3002,8 +3002,7 @@ static int gup_pmd_range(pud_t *pudp, pud_t pud, unsigned long addr, unsigned lo if (!pmd_present(pmd)) return 0; - if (unlikely(pmd_trans_huge(pmd) || pmd_huge(pmd) || - pmd_devmap(pmd))) { + if (unlikely(pmd_thp_or_huge(pmd) || pmd_devmap(pmd))) { /* See gup_pte_range() */ if (pmd_protnone(pmd)) return 0; From patchwork Thu Nov 16 01:28:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 165642 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2924214vqg; Wed, 15 Nov 2023 17:29:44 -0800 (PST) X-Google-Smtp-Source: AGHT+IEawFDEdtEvt2u2YW5dEG6XdpMjQg65S4Biya9Hj2JRfJE8Lm+7QHF8AE0H8o4lJnPo3o6Z X-Received: by 2002:a05:6a20:7486:b0:15e:dc75:66a7 with SMTP id p6-20020a056a20748600b0015edc7566a7mr12917356pzd.24.1700098183852; Wed, 15 Nov 2023 17:29:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700098183; cv=none; d=google.com; s=arc-20160816; b=EhJ3qvhq5VAE3f+UfWAYBq/+PsJArkt5N1hg0rIvTB9LpgWwzezw/CKluDWrBBUrGP AuHTJseE4je/M2Bpcg/406qaQPtIm3uh7VQySDnZye2LwPGdzZBo7vJBBkP9/oibmA6d nShiO8h/x13CXfY8eiGK2HEVrQfkz432k8OmKX9nsc1zMznMWe72DZ2DfFXjS/Wl1bOG R7vXDmgdD5uNNpOXufxlbYC/x8n/oHSYAS8czdNb/oh+urIYuzf6MuUF/TVT1tPA+Y7W Ac0KThqDMVbdNVXeiOkOReBTBWsX7yuAiJkUSGUEgcArt7z2UAGkOOrbcyfj46fmgXIY W2TQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=PGiD5LrOPbNAVIZVL9CsLOYGNogun3F0ZNXBV5lOZSk=; fh=o+FJD7UTEFgAOwGBZppViFyhRpBzPGRfoO/6xzRNnMs=; b=cUYo+BGJgElN84MOF2ZA52CjikIZ+0Q9aRpbG7fke7N+s+luZvhQ5buqdYjjByQydd jyVD7YQ66B9/P21RBz8Vw8Pu1CUCNPgx1Zwg5AjO+mYmAFRZNz+//HlSth1HNupVGHU/ Wyg//bGmo2mD6dfoemsS/7ib4Ng1Xp6IGhyxK5qtyTxHhWh4fMG7vTltbRNRCtphbVKL +izJEdKnuuWLU8dkcXWyxm5DbSY8s3ahJOoXXpJv0PbAxq6eopH3XxlIhUirjM2omaEj Gr91IQX9oiyEBbVBQFTYAjtJQ0WHfWtOA874+J3BuRShmmQ5V6bUw1aJuycQ1JOc8P8q FlQA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=LgKVGdB6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id d10-20020a170903230a00b001cc45fcb81fsi11861166plh.334.2023.11.15.17.29.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=LgKVGdB6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 8DEE6811F115; Wed, 15 Nov 2023 17:29:42 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344475AbjKPB33 (ORCPT + 28 others); Wed, 15 Nov 2023 20:29:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59768 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344326AbjKPB3Y (ORCPT ); Wed, 15 Nov 2023 20:29:24 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4538419B for ; Wed, 15 Nov 2023 17:29:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098160; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PGiD5LrOPbNAVIZVL9CsLOYGNogun3F0ZNXBV5lOZSk=; b=LgKVGdB6I4hNe4lwEdpQGH/NWKuJhQm4TWd80F/drBP9Vq8+DJCrNjCFv/e2kuYEzj5+7d e3ZI6JJ9s8w+mluRgz+vQmVYzjLiY7p4QY+g3zRawMDOY2z+hgurdBfHgiSJViKCvNQYr6 S95Aih6Eb2bBC33DMba37QwRgM6LpgQ= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-633-w0VJCoTSNWCS79fGDUeRTw-1; Wed, 15 Nov 2023 20:29:17 -0500 X-MC-Unique: w0VJCoTSNWCS79fGDUeRTw-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-66ffa15d354so814626d6.0 for ; Wed, 15 Nov 2023 17:29:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098156; x=1700702956; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PGiD5LrOPbNAVIZVL9CsLOYGNogun3F0ZNXBV5lOZSk=; b=FJ8tWTHSk7IrYTtgVBxiuvJsBDwI9TAywTI9/KP81JFBMTD004U5/1PrtccqxC+Xkh IVwp9SRsLDKoD9ebHCG86graWF20A7ICcwfb3WEDaTLxNssqoQkr/vJOjGp3/fwubx+I Y0G0nJrmWrh/btWLcDrqxzGz7SCBh4YkJOBqwqbwvXYSBHQYajKPfs/31UOE1aY9AkxD CcTSYYjYqsmCWsknGO2rhtyTdan/VvVAxLq2SoCn9QqnFbyhFAYPSJCtyzQRYwLw5K+0 +WHpYhpqluAv6RQzV/tz0exP1bU+nGIrym8RIr9Tb3gAcOhWQWjoJzv8clJkmmHvyqe2 JAHw== X-Gm-Message-State: AOJu0Yx9GOtu0iC3/zK1za/xuyk4dpw1sTCc/EyS5n5pNgP9Od8BQ9R7 LVheTEt/84XV1vrY/26q7ORNrZ35BpDcpbIgxg9AszZGCRRsVmh4yB1v3agZ+rvVttMinK+Nqsl iNnh2E5gSwNI5u/By5Mj9jQOT5RNnvxqc/5UTqCqpLyeND4xEy9d+7pvDcOQh3cGvqqRwcIeMYB LZsw38+A== X-Received: by 2002:a05:622a:8008:b0:41e:a62b:3d28 with SMTP id jr8-20020a05622a800800b0041ea62b3d28mr6714107qtb.4.1700098156347; Wed, 15 Nov 2023 17:29:16 -0800 (PST) X-Received: by 2002:a05:622a:8008:b0:41e:a62b:3d28 with SMTP id jr8-20020a05622a800800b0041ea62b3d28mr6714076qtb.4.1700098155869; Wed, 15 Nov 2023 17:29:15 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:15 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 03/12] mm: Export HPAGE_PXD_* macros even if !THP Date: Wed, 15 Nov 2023 20:28:59 -0500 Message-ID: <20231116012908.392077-4-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 15 Nov 2023 17:29:42 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782682153243685301 X-GMAIL-MSGID: 1782682153243685301 These macros can be helpful when we plan to merge hugetlb code into generic code. Move them out and define them even if !THP. We actually already defined HPAGE_PMD_NR for other reasons even if !THP. Reorganize these macros. Signed-off-by: Peter Xu Reviewed-by: Christoph Hellwig --- include/linux/huge_mm.h | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index fa0350b0812a..ec463410aecc 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -64,17 +64,19 @@ ssize_t single_hugepage_flag_show(struct kobject *kobj, enum transparent_hugepage_flag flag); extern struct kobj_attribute shmem_enabled_attr; -#define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT) -#define HPAGE_PMD_NR (1< X-Patchwork-Id: 165640 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2924167vqg; Wed, 15 Nov 2023 17:29:36 -0800 (PST) X-Google-Smtp-Source: AGHT+IHcAzdqRHvN4Zh7GBeKs3n5T0niEKWGVdJ4Fc64HBRbtq0b0dwzsSaJbOfm5FfFCgwmpYTi X-Received: by 2002:a25:c048:0:b0:d9a:401d:f5da with SMTP id c69-20020a25c048000000b00d9a401df5damr13119239ybf.51.1700098176352; Wed, 15 Nov 2023 17:29:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700098176; cv=none; d=google.com; s=arc-20160816; b=fpo1WB3dU6mksiizEx+X2ZXeL5q04Xs2iYHUSCGzWWyk56X9jqqTnEi8PbNUvxOgo0 lElnA6/z3r66HxClWMnZctw1ZyOrFFEOzlWdkljWVslloU5G98X4V67Dojz3+nfpyXwI 8VPQNnCux83hhXcp58oey0pLJoUNczj1oViX4J/Zg/xjmw6YXf7VwViJpoGn0qFCOi96 kW5oby3PteIX/GDIlMrDK/wsoT8M+EyrJE1rIAWE8kOPkUcq4GF4WvXIlRD9kNdA65o2 6JUYKmiTGGwlEdHVwMlkLak/HTqQ96GlYDfAqGwAQGvyq+gz+oKCdJlb5efdsEMZW188 EQmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=i0M65PApL8h6gQcrG6fLgH2zAjMUA+gwn5/0n3MZLTE=; fh=o+FJD7UTEFgAOwGBZppViFyhRpBzPGRfoO/6xzRNnMs=; b=C7DZFMDhmvE+ZG4efWv7U2Wd5tm83NbPp+VHf8LgdWbCM4k7DD3AH4tqlMnS23vXLZ dgtNfj6LNfPYWT8H3SQkYprV0UI6fv5g1V6FFz0S8HIjINbwMs00+7zbJlu+r9UlI3Iq uH/CWEs9Zzw5D74cKdXuIvNbenWJDu9IhGYtP6l0vDS64/GAOX/7iFhYgKmXF8AzjL+f Q9Nvr7vG6oQ91oLljSblt/eSnreK2A9wSU7pD3RziZNMgaNQ+a/G6HM+ec5zeDBWyF78 y/c0rvh3/EsQFlmXbIGeHhsk2E9mfrgkJ0if15VexugF7M+xtQWlWYR0AR62Qvup3ME6 W7TA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=GHO1LSnr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id c17-20020a6566d1000000b005aad5164a40si11269516pgw.246.2023.11.15.17.29.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=GHO1LSnr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 6A8FE81068CD; Wed, 15 Nov 2023 17:29:34 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344467AbjKPB31 (ORCPT + 28 others); Wed, 15 Nov 2023 20:29:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59766 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344386AbjKPB3Y (ORCPT ); Wed, 15 Nov 2023 20:29:24 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0F17198 for ; Wed, 15 Nov 2023 17:29:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098160; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i0M65PApL8h6gQcrG6fLgH2zAjMUA+gwn5/0n3MZLTE=; b=GHO1LSnrqCHQL4ba81d/XBbF3oovWtWEpxVi2TuEFvGBrqihW9XTKVjbaPZFbp14xHUxy0 Dk8vPhvUzTwKpfcJy9VPfCzpZ4/5HzAuVUKzh89kkphD4hhVgNrj1Lb7JVw+Da5qgo+/Cy lrguGfv0epguiXO0OxiRIRrcW+H85AA= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-435-eANnE0BLNCCdCo4qnJWQMQ-1; Wed, 15 Nov 2023 20:29:18 -0500 X-MC-Unique: eANnE0BLNCCdCo4qnJWQMQ-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-421be23b19aso883451cf.1 for ; Wed, 15 Nov 2023 17:29:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098158; x=1700702958; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i0M65PApL8h6gQcrG6fLgH2zAjMUA+gwn5/0n3MZLTE=; b=P5TTnvB7rfwOaEe5bTi5o061054oohYajKyzUPHRYafLfGzAK0djKiM+lPj2dhZQjy s6auI3nDLA042DcaD3exMLMjiKg65YI04BLVggQ0BieXRFwmAz4y8f6W+A4z/jKh2G9s D9GB9QW6hk1GkSnrRMy3cYAe2XQqvHY8AYX+tn7jT56sjjw+Px1McMMSUlW4BcCvDXPZ i4TH+4T4TaqTgltJW11rXUkcBwAUDm2rsAhCyUj0k+MP5YWn3o+61CDnt6klbVQSrCGj vOYfsoGLyx6nlfb5xHx4Fxs7ByF5C64BBp7mI/Gu86fOebrDGB0f/p6kqZCLENlD27nk W4Mw== X-Gm-Message-State: AOJu0Yxp4VT51JC2W+SbGBgBHGveDZ/5wZ6MkF72b5WVg7juU+IhmZxM caBfE0EHYZK2W7Jh+xamQochu+oVWR4NrM2OD7Wh7udzwR5WdwQ8snjPZm59k6AfVgKVcYDmLHr gmEwistp+cj/syDC2arvQN588weosizYOZI4w8sJFxU4Oig5cTCv7GBC2qm0aX7edRnh0WcpduN pQbfDZEQ== X-Received: by 2002:ac8:4716:0:b0:421:abb7:1eac with SMTP id f22-20020ac84716000000b00421abb71eacmr7303886qtp.0.1700098157927; Wed, 15 Nov 2023 17:29:17 -0800 (PST) X-Received: by 2002:ac8:4716:0:b0:421:abb7:1eac with SMTP id f22-20020ac84716000000b00421abb71eacmr7303852qtp.0.1700098157463; Wed, 15 Nov 2023 17:29:17 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:16 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 04/12] mm: Introduce vma_pgtable_walk_{begin|end}() Date: Wed, 15 Nov 2023 20:29:00 -0500 Message-ID: <20231116012908.392077-5-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 15 Nov 2023 17:29:34 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782682145412544495 X-GMAIL-MSGID: 1782682145412544495 Introduce per-vma begin()/end() helpers for pgtable walks. This is a preparation work to merge hugetlb pgtable walkers with generic mm. The helpers need to be called before and after a pgtable walk, will start to be needed if the pgtable walker code supports hugetlb pages. It's a hook point for any type of VMA, but for now only hugetlb uses it to stablize the pgtable pages from getting away (due to possible pmd unsharing). Signed-off-by: Peter Xu Reviewed-by: Christoph Hellwig --- include/linux/mm.h | 3 +++ mm/memory.c | 12 ++++++++++++ 2 files changed, 15 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 64cd1ee4aacc..349232dd20fb 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4154,4 +4154,7 @@ static inline bool pfn_is_unaccepted_memory(unsigned long pfn) return range_contains_unaccepted_memory(paddr, paddr + PAGE_SIZE); } +void vma_pgtable_walk_begin(struct vm_area_struct *vma); +void vma_pgtable_walk_end(struct vm_area_struct *vma); + #endif /* _LINUX_MM_H */ diff --git a/mm/memory.c b/mm/memory.c index e27e2e5beb3f..3a6434b40d87 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6180,3 +6180,15 @@ void ptlock_free(struct ptdesc *ptdesc) kmem_cache_free(page_ptl_cachep, ptdesc->ptl); } #endif + +void vma_pgtable_walk_begin(struct vm_area_struct *vma) +{ + if (is_vm_hugetlb_page(vma)) + hugetlb_vma_lock_read(vma); +} + +void vma_pgtable_walk_end(struct vm_area_struct *vma) +{ + if (is_vm_hugetlb_page(vma)) + hugetlb_vma_unlock_read(vma); +} From patchwork Thu Nov 16 01:29:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 165641 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2924207vqg; Wed, 15 Nov 2023 17:29:42 -0800 (PST) X-Google-Smtp-Source: AGHT+IHQhj4Orx8d/tcZbxKVTJa2o0Yhfu73YXyPZ8GvLJqhrHrmmNnG2PfjHx/MpgfDgBgz4i2K X-Received: by 2002:a17:902:864c:b0:1cc:2771:6fbe with SMTP id y12-20020a170902864c00b001cc27716fbemr6491949plt.39.1700098182561; Wed, 15 Nov 2023 17:29:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700098182; cv=none; d=google.com; s=arc-20160816; b=dldBQtpxtg0RdTh245wm2d/ZIv+fYoQaMINlnIFq0NJnY3FBXn7mL3BVxfvvTHRDh6 B0dQGygDzbmVje4J3njGXs9MufRtolTB/zLDskPQoQ4ZzDLKEUaXGXAMeOsWq98Dv+HD 11rsN+RLUyGE9oYjKafooEtoJ6UBRVMLoyExlab3dyadB21PNXC2Sfdp1nr1oy64go4h M5esrpVpXVkoKeScmhWvMyqCZ4kDFEpDbMXWWcYo5/qNfbNhYJJTqXUJ7pqYJOohkZBM uqhuYSe3yLhIjTsyBi6OohDkrSznQE7ndM6JzOzBlvjYIiMcC4HdXtcOFVhCDqRb0HxF jz7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=iVPh5vUSg15R+EjRvyUTGLnvTLGftOOyrePNaUJdD/U=; fh=o+FJD7UTEFgAOwGBZppViFyhRpBzPGRfoO/6xzRNnMs=; b=BB9AmIoagSzOs3qhJSko8eZB8Lykcj3sjKC8xokeb9CcpIf1fi/moaPJswqxdMPFcy ZBNZbHd3lm5vsnhhNCQk/c4dmBRTnqIwTdP9Stz+ln0e/21gIxlPQVYg5z6cQh14RpDh MyOCOCLDGa6e/RoYs3kbHQIRuEjSfqZ/HDNwY4Gi2SgTIjvUHDV3ydIJ6tKGOTLlSEhg fyI9kwXBTVVaymVcR22ntpcES9HgX7j/8vhy8BrQKdECStQsuXKKZhnf0rV3NJkasYYp TZ209braIOZ8K1CFt/21xUd5RhU5wnWfE5Wrg3l7ve9Ku2mGNLHohzKPZvFeLjLDcXUi m3/Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JGtAcCe9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id c10-20020a170902aa4a00b001cc2ef72ab2si10489836plr.68.2023.11.15.17.29.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JGtAcCe9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id A7B3780E73DF; Wed, 15 Nov 2023 17:29:41 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344485AbjKPB3c (ORCPT + 28 others); Wed, 15 Nov 2023 20:29:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59808 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344454AbjKPB30 (ORCPT ); Wed, 15 Nov 2023 20:29:26 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1AA18199 for ; Wed, 15 Nov 2023 17:29:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098162; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iVPh5vUSg15R+EjRvyUTGLnvTLGftOOyrePNaUJdD/U=; b=JGtAcCe9WHOXOtrqPb64GnJVE6w2GU9IMp9LSE+DCMXMvb/Im37grqe9PWvvqtY4HcefFj WyIj0z31+EEpk2kdYStFkdrNIgI9U+cFwKMStyfJg7XNxvbpUGRqM2DvZHAThMl0jYI4L4 zPG1GL6D+HI1VFF8Be/T/EVW3xRIdMU= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-646-2VO4HHQ_NoW6PizvXzl5Nw-1; Wed, 15 Nov 2023 20:29:20 -0500 X-MC-Unique: 2VO4HHQ_NoW6PizvXzl5Nw-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-41eb42115e9so899351cf.1 for ; Wed, 15 Nov 2023 17:29:20 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098159; x=1700702959; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iVPh5vUSg15R+EjRvyUTGLnvTLGftOOyrePNaUJdD/U=; b=EVLJgYzbG9zKeP5DkNaiA2171uiCSv9C/hxeJUnOv70IgsOv6/gSE8Qm18Sofmlnig YYmOqMdC7mYAPoH+mV3HaYCr38Dr8Q4uQ9+XSBjn0/C6iNNDvo9VBLJiOiZF6pqc1JII Nc2w9qhuZvfMBvLBtg2HyxoOEKfy82HR2Ah39v3xtXJG+lUmihqPc84RW1nOePihrAMv ELrUhGJB+SAsnHuGEgkx2r8toH2TKSqVfKPu+TVuDQZCleMQmuO/D8J00fr/gFqBUd1T ia9y+LZpR60K7OhuHrkqPTIfX+AXL7347oK1mUP8igHeempXhLNXRoss8yKw7p0OsU6+ /aig== X-Gm-Message-State: AOJu0YxfDVtwfqnbd+Q5vdzrIoYyS+bt29fhzaD1F3Ib40eLhmBL9Pv7 qvs79cMkLCjx3/xe7UUrXXiIVyvd7TGON4B5915qqtxJZSriCCmna5h9Vu1UW071Tg7gPJGncI5 EmxBx3JOcFG/4VciACw+od0YKoZ/wHo5YacMD8I7bWtu5KLw1H66ypsWzA/AD6c6uEKcjLokER2 fUWjveEQ== X-Received: by 2002:ac8:4716:0:b0:421:b9be:d4a4 with SMTP id f22-20020ac84716000000b00421b9bed4a4mr6891661qtp.0.1700098159514; Wed, 15 Nov 2023 17:29:19 -0800 (PST) X-Received: by 2002:ac8:4716:0:b0:421:b9be:d4a4 with SMTP id f22-20020ac84716000000b00421b9bed4a4mr6891630qtp.0.1700098159208; Wed, 15 Nov 2023 17:29:19 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:18 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 05/12] mm/gup: Fix follow_devmap_p[mu]d() to return even if NULL Date: Wed, 15 Nov 2023 20:29:01 -0500 Message-ID: <20231116012908.392077-6-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 15 Nov 2023 17:29:41 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782682151779757430 X-GMAIL-MSGID: 1782682151779757430 This seems to be a bug not by any report but by code observations. When GUP sees a devpmd or devpud, it should return whatever value returned from follow_devmap_p[mu]d(). If page==NULL returned, it means a fault is probably required. Skipping return the NULL should allow the code to fall through, which can cause unexpected behavior. It was there at least before the follow page rework (080dbb618b) in 2017, so 6 years. Not yet digging for a Fixes, assuming it can hardly trigger even if the logical bug does exist. Signed-off-by: Peter Xu Reviewed-by: Christoph Hellwig --- mm/gup.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index a8b73a8289ad..0e00204761d2 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -708,8 +708,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, ptl = pmd_lock(mm, pmd); page = follow_devmap_pmd(vma, address, pmd, flags, &ctx->pgmap); spin_unlock(ptl); - if (page) - return page; + return page; } if (likely(!pmd_trans_huge(pmdval))) return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); @@ -756,8 +755,7 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, ptl = pud_lock(mm, pud); page = follow_devmap_pud(vma, address, pud, flags, &ctx->pgmap); spin_unlock(ptl); - if (page) - return page; + return page; } if (unlikely(pud_bad(*pud))) return no_page_table(vma, flags); From patchwork Thu Nov 16 01:29:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 165643 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2924228vqg; Wed, 15 Nov 2023 17:29:45 -0800 (PST) X-Google-Smtp-Source: AGHT+IHBf/NkXxMPlzQXI1WFP04SAP0tE08tbYyc2otgPiZiWHL7xNx3tuE+wvicdsIo9wuAenzd X-Received: by 2002:a05:6830:5:b0:6c0:ae30:671e with SMTP id c5-20020a056830000500b006c0ae30671emr8276065otp.20.1700098185676; Wed, 15 Nov 2023 17:29:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700098185; cv=none; d=google.com; s=arc-20160816; b=kz7UDpZrPOuF5nTS04XSFqHZlH3TQnhXYpXltIGLZsnJ5gC11YhUglDG1zQdbi9B8C GT8KS7OY5WKO2zkCE4u546/cjgsBWkJeRK9bh7crcgYCuGzTdYF8LT9KnWiwpc0uktOf H+oybIWTy8dhPYF2pbFvbYvETVe1w+bSr3CDwHr0a8JR4I9+6PxZ288rVPaqO1pSExJx 45OGd3TCRELtir4Ak6CfZMxXxJdx2O/tczzWCjmuE/RkHsn6DQKdANJUvraCuGEjOAvm FQvCyh8GNFrCpDk7Ia3gqVzsNxsshz4xw3BchMHKqqTwdEn2vAoWOSUgVuPJEBn+odRG PlUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=bUKDadnp+1aD4MuuhezpyppaSOr8lCSuMGXbTAXddpQ=; fh=nvfIpCvT5SnQ83HxwVfj1I4jmL6Kgaw3F1yj79JBUNM=; b=mJbbrZIyvPttWUk80fpaGyu7vqSHH0+Vl/vUtIIG4e1zrEHPF0RF+bQtKMm49TVCuE Q9Y9lCIDJYWGcz6FYtBp7kzxZg9zyT36f7qbs2W7K9VK+ErPUteiLFVPvfcNN/IXAMNp Yg/DiAmoazKwaQ2iGD7Rv0PdDADeLQL0tVWl8ckP2D0ia04oCFhiOk0OPqyCTd3XslsZ hMVXeVN6lS6dliVH6bTbj9RB+/S+Et/8HHmDKDfMEz7uoBDrNA+pIr0/aqq7XIHlyH8+ AozuU/JvDHepnEN47UTRbyO2mcoy/GrkrN13/PpsbKsKGCWJLhiF40OismRGvupd/AS0 t0VQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=EQ9DHPYo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id ck6-20020a056a02090600b005b8f24e6526si11870235pgb.234.2023.11.15.17.29.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=EQ9DHPYo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id EFF1C81121EB; Wed, 15 Nov 2023 17:29:44 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344450AbjKPB3h (ORCPT + 28 others); Wed, 15 Nov 2023 20:29:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50648 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344498AbjKPB3e (ORCPT ); Wed, 15 Nov 2023 20:29:34 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC8DB199 for ; Wed, 15 Nov 2023 17:29:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098163; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bUKDadnp+1aD4MuuhezpyppaSOr8lCSuMGXbTAXddpQ=; b=EQ9DHPYog5gjUZU2fkQMByjxMp4TytU1nvXgGzlu7Qx8kOBVGotuAfA4DZZg/X+HQpsfay R7uUAF/At2X+V0FiQeYu7Mk8XrSuxtK2iM41rRw8rcffqaBypE9rtFb+qEbE3/VdIt1me5 zy1k4uWq4Fx9WeH551GIij5CmWwD/Oc= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-184-4Qd8MIcmOk-Nvc_gpjxIoQ-1; Wed, 15 Nov 2023 20:29:21 -0500 X-MC-Unique: 4Qd8MIcmOk-Nvc_gpjxIoQ-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-41cbafdb4b6so931571cf.0 for ; Wed, 15 Nov 2023 17:29:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098161; x=1700702961; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bUKDadnp+1aD4MuuhezpyppaSOr8lCSuMGXbTAXddpQ=; b=k9eEFSbhhlgrcGtAikDFRan7CnSbGRU1FQkSi3E5dvBWBXKdmrfxSwbuIbJRqSJmOH 5VYE1YAu8LdMxpnZqB06FAWOpSTBEkS/BaL0zenC6ETYUeIYPijpodxApMBIzHRnw6+G qhD83jcuSiXTP0xiRhqY25OQXDFza/MQTQxONxh3DCppN3e8ERaHB/AVuKMu0hl8shmg 1Bqd2LyStajvfU78XJ3sMY/EEVhcfBgkQm9K7TmiACl/SeBr59TihLxwSwXT6nYpp7hM vgOwJOyn9/YnFhIpgq2/zYTHgpiVRF9BUHRAf1vFHiB3Ba9wsUDwvgv1zf5b3RUHVmkk HNEA== X-Gm-Message-State: AOJu0Yzh6vWbY0n/PDkqTVsRE04oxzCGL9W1hpHiO4npGoz7kPy92W8U gBgqUZ6BTQOaTMpgeB/8JR8cOlo0sg/5mdMO4bGrx1jCVSKen/8g3iLDWMyLgoeEW+lqyXQwncj bWzZNx43CXF8Ny7ajiBBifzp09uapLQEqqFK/twIyTjaZcNOzdBWRlS0XEowj9dRcVgL/gz0qbl NpiWFygw== X-Received: by 2002:a05:622a:810e:b0:41c:d433:6c86 with SMTP id jx14-20020a05622a810e00b0041cd4336c86mr6482576qtb.4.1700098161061; Wed, 15 Nov 2023 17:29:21 -0800 (PST) X-Received: by 2002:a05:622a:810e:b0:41c:d433:6c86 with SMTP id jx14-20020a05622a810e00b0041cd4336c86mr6482544qtb.4.1700098160768; Wed, 15 Nov 2023 17:29:20 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:20 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton , Michael Ellerman , linuxppc-dev@lists.ozlabs.org Subject: [PATCH RFC 06/12] mm/gup: Drop folio_fast_pin_allowed() in hugepd processing Date: Wed, 15 Nov 2023 20:29:02 -0500 Message-ID: <20231116012908.392077-7-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 15 Nov 2023 17:29:45 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782682155051627968 X-GMAIL-MSGID: 1782682155051627968 Hugepd format is only used in PowerPC with hugetlbfs. In commit a6e79df92e4a ("mm/gup: disallow FOLL_LONGTERM GUP-fast writing to file-backed mappings"), we added a check to fail gup-fast if there's potential risk of violating GUP over writeback file systems. That should never apply to hugepd. Drop that check, not only because it'll never be true for hugepd, but also it paves way for reusing the function outside fast-gup. Cc: Lorenzo Stoakes Cc: Michael Ellerman Cc: linuxppc-dev@lists.ozlabs.org Signed-off-by: Peter Xu --- mm/gup.c | 5 ----- 1 file changed, 5 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 0e00204761d2..424d45e1afb3 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2816,11 +2816,6 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, return 0; } - if (!folio_fast_pin_allowed(folio, flags)) { - gup_put_folio(folio, refs, flags); - return 0; - } - if (!pte_write(pte) && gup_must_unshare(NULL, flags, &folio->page)) { gup_put_folio(folio, refs, flags); return 0; From patchwork Thu Nov 16 01:29:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 165644 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2924261vqg; Wed, 15 Nov 2023 17:29:53 -0800 (PST) X-Google-Smtp-Source: AGHT+IH0TlHhZVj1f7GFC+yt6C3k5F95BVFglmYrjSqvIKmjsPYeIh0Vo+41hQFUFS8wnDpjUo9n X-Received: by 2002:a05:6a20:734f:b0:187:958b:993b with SMTP id v15-20020a056a20734f00b00187958b993bmr379517pzc.10.1700098193677; Wed, 15 Nov 2023 17:29:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700098193; cv=none; d=google.com; s=arc-20160816; b=brH/83R3xJbYXiKu2jT4aD1u+qRPgAbFCW+KKANn2qsaXvRlT3BqJonTeopeJG994s eRl+CeqwxLYNTdj8nO3rAxdPUvMSXxD+2XuDsB2ZIk/y5ER0EGijG67e5xPKogVt6rGm f3ZBoQKk1WXO7ZgXuPXqIzu0/BU6J1RTFExnu49E8chM54b4Sur7+Lw9DZgxOBNKJUZQ aZ3z1zOK7arNM/Vp3PEv7K0srq9tthnPfcyrXpH2Yr/IxIJSno3zvV0D4rLjl4PCjM6z pXxrZYiu/7UGZxeut5B3a6R3x6bEhrtBo3dqXL3lRAFhf2fSpui92WAqFWfJAgdAoMAW Kpyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/m4Nmkq0cr3qM8eU/2c8RM1XZmbD+yMVfcEDigP2nYM=; fh=o+FJD7UTEFgAOwGBZppViFyhRpBzPGRfoO/6xzRNnMs=; b=IiCm2i9edsf76efr7lp3soa5HvgGInD5wyzcOhZVgC5ceNrQcavxJKy2KSFi24PX70 sMgfW5R/CUYwuGPKTupPgtZmkXMD3EP9rXGXOD5HFxnhgQKTHuRIzuOEXXx47iO5So4r FinI6Gde4+lv1QPWzKox8atk3VRpb+mTbAE2dRwCeC9mITqgVkGuvTmhVgYiaZwuxk8s dULf58VlNIvGKRR/yd6u8WjhrBfKJsocciKqV8NNnPPEJPt3RL5uXCKk91LAOLd5MFid KJmYQSUV2ZIubKvGAMWIlmfxJWhS1eBKfeU78MIoAnL4mDsuHfb+CuYtIFapR9Bg9hPL qGfA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=MqOjT7Fc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id e2-20020a656882000000b005be09a118d3si11394189pgt.143.2023.11.15.17.29.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:53 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=MqOjT7Fc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 67E8A81121EB; Wed, 15 Nov 2023 17:29:52 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344514AbjKPB3p (ORCPT + 28 others); Wed, 15 Nov 2023 20:29:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50658 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344487AbjKPB3f (ORCPT ); Wed, 15 Nov 2023 20:29:35 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A70101BF for ; Wed, 15 Nov 2023 17:29:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098164; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/m4Nmkq0cr3qM8eU/2c8RM1XZmbD+yMVfcEDigP2nYM=; b=MqOjT7Fci5yrpWa33B2q5gKSg1vCmiVYiKxFz5mY56b/A3vh18DQRUnsJ3ZOA6Ppv34ngf SdtNNqi1wVt7JMBgHWV2fY/TX7MTBfYJAWMdeuS65/M/BkEoLjNB8lWBQpo907QZKdwokz RK8KXwvkkeq/f4Vggm/dtwXytIv5S9k= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-175-WEUvw7azPfWEV7Xuhjq34Q-1; Wed, 15 Nov 2023 20:29:23 -0500 X-MC-Unique: WEUvw7azPfWEV7Xuhjq34Q-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-420c7034fddso915891cf.0 for ; Wed, 15 Nov 2023 17:29:23 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098163; x=1700702963; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/m4Nmkq0cr3qM8eU/2c8RM1XZmbD+yMVfcEDigP2nYM=; b=DV0nlCQpnC2ux22Ay1mhabMQ5MnA5ZlP6AMDXiqLB9O4LG3tVmpAWcV7CU+v+kp4WS JXOCNUIBolIEOhtPTuFUsXgDo26ZH1aUwMg2+Xb3V6hIhD7+PrvO35Q7uVewUEYXKfhS xxqgRCJBT/Z3+p1NaSw3I+nadYR0/XQOnSS27dnUnhASmeEZOEZXQZPBTukLlPefjALc E4Ud/4Kg+ImdvXJYUinFjyiLEtjbDlrB9tLnaR2jZ766FjmTOuz0tlSYPOglyXj+eicp AmvWzy0WlPdKk68s8pbwQpvV36GOB8ACljJ6aQzPl3b9cHPtInUce6LucpeBpsS0c+0J P2eg== X-Gm-Message-State: AOJu0YygplLJBAkFgFsQPSr046zRK//umVntqA7h0xxk12sSxC/E8PJZ eWDdJsuqBhB1PRmWLQ7uJDSFt1dhZ0ya+BwAQh6kSJ3+lGvq2NoO5Wq36KI2P0Hj1KLwltPJQnL EisMjOgxCsWzkXYdT6VCx7BboG1QwGIfDfk9H1CDUAceKCEvIAu2/pSJxOfClNRP76jczcG8N7p 1RFZX8Rg== X-Received: by 2002:ac8:1496:0:b0:421:9fe5:6dc1 with SMTP id l22-20020ac81496000000b004219fe56dc1mr6484684qtj.3.1700098163039; Wed, 15 Nov 2023 17:29:23 -0800 (PST) X-Received: by 2002:ac8:1496:0:b0:421:9fe5:6dc1 with SMTP id l22-20020ac81496000000b004219fe56dc1mr6484654qtj.3.1700098162561; Wed, 15 Nov 2023 17:29:22 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:22 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 07/12] mm/gup: Refactor record_subpages() to find 1st small page Date: Wed, 15 Nov 2023 20:29:03 -0500 Message-ID: <20231116012908.392077-8-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 15 Nov 2023 17:29:52 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782682163448510062 X-GMAIL-MSGID: 1782682163448510062 All the fast-gup functions take a tail page to operate, always need to do page mask calculations before feeding that into record_subpages(). Merge that logic into record_subpages(), so that we always take a head page, and leave the rest calculation to record_subpages(). Signed-off-by: Peter Xu --- mm/gup.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 424d45e1afb3..69dae51f3eb1 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2763,11 +2763,14 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long addr, - unsigned long end, struct page **pages) +static int record_subpages(struct page *head, unsigned long sz, + unsigned long addr, unsigned long end, + struct page **pages) { + struct page *page; int nr; + page = nth_page(head, (addr & (sz - 1)) >> PAGE_SHIFT); for (nr = 0; addr != end; nr++, addr += PAGE_SIZE) pages[nr] = nth_page(page, nr); @@ -2804,8 +2807,8 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); - page = nth_page(pte_page(pte), (addr & (sz - 1)) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pte_page(pte); + refs = record_subpages(page, sz, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2870,8 +2873,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, pages, nr); } - page = nth_page(pmd_page(orig), (addr & ~PMD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pmd_page(orig); + refs = record_subpages(page, PMD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2914,8 +2917,8 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, pages, nr); } - page = nth_page(pud_page(orig), (addr & ~PUD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pud_page(orig); + refs = record_subpages(page, PUD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2954,8 +2957,8 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, BUILD_BUG_ON(pgd_devmap(orig)); - page = nth_page(pgd_page(orig), (addr & ~PGDIR_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pgd_page(orig); + refs = record_subpages(page, PGDIR_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) From patchwork Thu Nov 16 01:29:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 165646 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2924338vqg; Wed, 15 Nov 2023 17:30:05 -0800 (PST) X-Google-Smtp-Source: AGHT+IFgLCZXFyITBmb+hD+Ht/n2DuM9YZ/QHtY7u3p7pb6MukIG/PiwAoiVvojDmc1t6xwHvObN X-Received: by 2002:a05:6a00:451c:b0:690:1857:3349 with SMTP id cw28-20020a056a00451c00b0069018573349mr18007140pfb.25.1700098205015; Wed, 15 Nov 2023 17:30:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700098204; cv=none; d=google.com; s=arc-20160816; b=QApgNn1tcZQkZT0EiU62mk47TmUk9c9ApnRnsY6E4iK6lTNo+t+TLKxEILEwYYsk2R eeWdn/8x7Avzm5vbMw5bZ+/Z1w8Gdq/mMDSIMFKKewMcCh5sH9qJVnD+TldTPaAP/2YQ M30iy/jQdGiQ6jPUxrqWH++QFG3gJG7eis/kRcO4ce1DLwISpoobxha8QIkod19coONS UZbafqxTsjKpL/39n/4A7WNkGWKfbMf9QFK0UyCLreEFv9wDdG3bjBV4wiXzcBb0SiVh 9tp5/JAiA+5C6mb5xPlPHcZ/orEANMh0i6QJWIGKL4i83A/Hdzwb3KtuMpzLIyjCdak0 spqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=hxqjSAwi8aTfo4zdaLYLd0e+DlQCcl09BvXUJR2cui0=; fh=o+FJD7UTEFgAOwGBZppViFyhRpBzPGRfoO/6xzRNnMs=; b=VHIC3AgdBKlzaZjjwtUF+CdMPfnlOzfovJBillD2iQZt5mjObS4/YjVwmIyw73rjr1 9O675JFtqUIXz3seqn7AYTDp10xB4LxkRXOsQ0M2Y8/Loa3FiXqKmbijmTgET8TkybR4 kf4jkNCATeI/KIbBYdL0rUkpLRoOB83Z+LKBZLEnhPDfQ6bv6Q0nBJuaIo0uDgchbrjm vR7YyAIUw3ln4pamjgIFYiQSCRzSqDEC7kmXNgriy6mY1rhpwifdqmz6/NkUNbRfkLaP 4eJtpey9L1kkLQEbalufizI6thJX10PlHF1f5kYC+FgP//EIrlVVNsX+nOd2hXREUtEY D7FQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=hIEgXfZb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id m12-20020a656a0c000000b005be3683ec67si11532323pgu.596.2023.11.15.17.30.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:30:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=hIEgXfZb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 444D7811F91F; Wed, 15 Nov 2023 17:30:04 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344538AbjKPB35 (ORCPT + 28 others); Wed, 15 Nov 2023 20:29:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50690 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344331AbjKPB3o (ORCPT ); Wed, 15 Nov 2023 20:29:44 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2524AD4B for ; Wed, 15 Nov 2023 17:29:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098168; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hxqjSAwi8aTfo4zdaLYLd0e+DlQCcl09BvXUJR2cui0=; b=hIEgXfZbAOrqX4z98mch3jAAm23q23QDu+Aq8lyd13NZEJ1Q/SibbZQC3gXtWRm/FQbSMB xqmAClvuvk3/eymMzQ0fxsWnX8gZvPi54cKIiivOGHf+8M/beQQ9/vefR/Y23u9DdtsgKy X2J5mvgDswmKAX3LEEuKra+DpbDCeLM= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-271-c5saubBzPJWJI-o4qu_cBQ-1; Wed, 15 Nov 2023 20:29:26 -0500 X-MC-Unique: c5saubBzPJWJI-o4qu_cBQ-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-77a02ceef95so2904185a.0 for ; Wed, 15 Nov 2023 17:29:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098165; x=1700702965; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hxqjSAwi8aTfo4zdaLYLd0e+DlQCcl09BvXUJR2cui0=; b=ANwI1CkwDUj98nrauz5703pYnACz6tz/f14E7bzSKJJUkOI+Xlkqs37rZGcaGlg1nu 9yfToxOmMgDqjyEAsUELKbwT71XENoYOX63zoiSRrePmL//irgZmGrTQcmBdTa//qwTG zll3ly+5uTcbpNYRSTvoyQf6v8itw19kUw/zNra6x/wGBm30ceF+ZPee017oMFwXhN5m 8Fh1XkG6R0d7o3uc3yJL0+kdm+C/CrkeST8FR3GRARpZGO1DYfFumjge/adBmXwQkLKw rJBue0H7yyneT2wyoDb/n1b1hIurNaQGTMmSoxgAT9CB9SGUQcsJMGWEqIryPKJclOJ2 zDHA== X-Gm-Message-State: AOJu0Yxu16moxEGh2N3tVTzc0Fpydzdo6RdQVkZkMQJ2WMIvtkteNn+2 r13KyY+KSO5wEeJNPsy3cHtIQaTIICs6hlJ6JzWvFjSXjS/2YQE179CttWUx1KiVeBQKc1SDkAC IkW0Lz0QMaoJ6pzt0dYcgcdu7ThFznbIeicQApy40h7V9kdxFnxHks0CGreg0sE0iz1s4GqeUNz LdEXnrUg== X-Received: by 2002:a05:622a:4291:b0:41c:d528:6589 with SMTP id cr17-20020a05622a429100b0041cd5286589mr8639226qtb.4.1700098164758; Wed, 15 Nov 2023 17:29:24 -0800 (PST) X-Received: by 2002:a05:622a:4291:b0:41c:d528:6589 with SMTP id cr17-20020a05622a429100b0041cd5286589mr8639189qtb.4.1700098164258; Wed, 15 Nov 2023 17:29:24 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:23 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 08/12] mm/gup: Handle hugetlb for no_page_table() Date: Wed, 15 Nov 2023 20:29:04 -0500 Message-ID: <20231116012908.392077-9-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 15 Nov 2023 17:30:04 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782682175577297793 X-GMAIL-MSGID: 1782682175577297793 no_page_table() is not yet used for hugetlb code paths. Make it prepared. The major difference here is hugetlb will return -EFAULT as long as page cache does not exist, even if VM_SHARED. See hugetlb_follow_page_mask(). Pass "address" into no_page_table() too, as hugetlb will need it. Signed-off-by: Peter Xu Reviewed-by: Christoph Hellwig --- mm/gup.c | 40 ++++++++++++++++++++++++---------------- 1 file changed, 24 insertions(+), 16 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 69dae51f3eb1..89c1584d68f0 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -501,19 +501,27 @@ static inline void mm_set_has_pinned_flag(unsigned long *mm_flags) #ifdef CONFIG_MMU static struct page *no_page_table(struct vm_area_struct *vma, - unsigned int flags) + unsigned int flags, unsigned long address) { + if (!(flags & FOLL_DUMP)) + return NULL; + /* - * When core dumping an enormous anonymous area that nobody - * has touched so far, we don't want to allocate unnecessary pages or + * When core dumping, we don't want to allocate unnecessary pages or * page tables. Return error instead of NULL to skip handle_mm_fault, * then get_dump_page() will return NULL to leave a hole in the dump. * But we can only make this optimization where a hole would surely * be zero-filled if handle_mm_fault() actually did handle it. */ - if ((flags & FOLL_DUMP) && - (vma_is_anonymous(vma) || !vma->vm_ops->fault)) + if (is_vm_hugetlb_page(vma)) { + struct hstate *h = hstate_vma(vma); + + if (!hugetlbfs_pagecache_present(h, vma, address)) + return ERR_PTR(-EFAULT); + } else if ((vma_is_anonymous(vma) || !vma->vm_ops->fault)) { return ERR_PTR(-EFAULT); + } + return NULL; } @@ -593,7 +601,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, ptep = pte_offset_map_lock(mm, pmd, address, &ptl); if (!ptep) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); pte = ptep_get(ptep); if (!pte_present(pte)) goto no_page; @@ -685,7 +693,7 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, pte_unmap_unlock(ptep, ptl); if (!pte_none(pte)) return NULL; - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); } static struct page *follow_pmd_mask(struct vm_area_struct *vma, @@ -701,9 +709,9 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, pmd = pmd_offset(pudp, address); pmdval = pmdp_get_lockless(pmd); if (pmd_none(pmdval)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); if (!pmd_present(pmdval)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); if (pmd_devmap(pmdval)) { ptl = pmd_lock(mm, pmd); page = follow_devmap_pmd(vma, address, pmd, flags, &ctx->pgmap); @@ -714,12 +722,12 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); if (pmd_protnone(pmdval) && !gup_can_follow_protnone(vma, flags)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); ptl = pmd_lock(mm, pmd); if (unlikely(!pmd_present(*pmd))) { spin_unlock(ptl); - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); } if (unlikely(!pmd_trans_huge(*pmd))) { spin_unlock(ptl); @@ -750,7 +758,7 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, pud = pud_offset(p4dp, address); if (pud_none(*pud)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); if (pud_devmap(*pud)) { ptl = pud_lock(mm, pud); page = follow_devmap_pud(vma, address, pud, flags, &ctx->pgmap); @@ -758,7 +766,7 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, return page; } if (unlikely(pud_bad(*pud))) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); return follow_pmd_mask(vma, address, pud, flags, ctx); } @@ -772,10 +780,10 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma, p4d = p4d_offset(pgdp, address); if (p4d_none(*p4d)) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); BUILD_BUG_ON(p4d_huge(*p4d)); if (unlikely(p4d_bad(*p4d))) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); return follow_pud_mask(vma, address, p4d, flags, ctx); } @@ -825,7 +833,7 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, pgd = pgd_offset(mm, address); if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - return no_page_table(vma, flags); + return no_page_table(vma, flags, address); return follow_p4d_mask(vma, address, pgd, flags, ctx); } From patchwork Thu Nov 16 01:29:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 165645 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2924283vqg; Wed, 15 Nov 2023 17:29:56 -0800 (PST) X-Google-Smtp-Source: AGHT+IHw7MQOrZgjGUZ80fmG0EAoDQxo9/ZUpNj/uHeoewY4eLep2Wg/d8DXF9pusdfNN9UGs1di X-Received: by 2002:a17:903:2585:b0:1cc:4a66:7d83 with SMTP id jb5-20020a170903258500b001cc4a667d83mr7361370plb.2.1700098196294; Wed, 15 Nov 2023 17:29:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700098196; cv=none; d=google.com; s=arc-20160816; b=ZrfhSh2I5WjdYQTjJZjB13530dgHUqaUSQhmXPwAw/fx3AtQ9rTZiUfKzhMg2nTuNi ZzoWzvw8GqCx9bgGFOg5uXewGcx0hWOj4OtHfTiKqJqCB3d6vRHW5oIYYH7fudLs0tbL wKGYPZ3MPc7WfZxJQo1BO33TGra/UyW7OAYItwvUyWYL/SdmdzK4IOD7l2g8iGNcbvLx 4rwmu+eNZNWQvY0J29Ou+Zkern9qMIeLrj193aoKxD5iOmFge/tzBesbm53cV5CjloKl ITT8ek4mDzWGNCCjis+qJNXZ1sdWmjgT5/VUB1rT2y5fiyp+qHDOnJAc5KKjrZzwiCgK h91w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=gQy6k5CEHGKiOJI0ugKFVUkk8XVFLkY9zLGCNjIpOps=; fh=o+FJD7UTEFgAOwGBZppViFyhRpBzPGRfoO/6xzRNnMs=; b=ffac/wH20gLgegSVoChIrHhjkI9QmxxIZys+kR9KQl7bDaJNIdmeBqnuzqUFBkWUlv o9Fs7b+HMK4DN/+9eyd8R+VtajCB6L84z80hEBzyQ7sGdHEUc0YW2j8zddoGHaPQ7EhF IMHD9Tl+rLCouvxqYyouJHVUdcYQxCt9AmZT6ILwB0JfTlWgvXnhW2dBI0jQnnb1c+30 54ZjPagoOpK8HEQ2l4NrHuXLUmN4on2iJTUUiFhqesbP5JG5Y00inAs1n5qKbwZEG0nj Dr1Vwr26sjISmfV5IkSD1L90SlmQ9T+lmdKoH9b2xdHrcbuVr144Ya5PeDio61NKB3bv 4nxg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=PZVOBA2p; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id j8-20020a170903024800b001cc43785cb5si11832739plh.174.2023.11.15.17.29.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=PZVOBA2p; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 852C5811F128; Wed, 15 Nov 2023 17:29:55 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344525AbjKPB3t (ORCPT + 28 others); Wed, 15 Nov 2023 20:29:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50712 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344524AbjKPB3o (ORCPT ); Wed, 15 Nov 2023 20:29:44 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1958D7A for ; Wed, 15 Nov 2023 17:29:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098168; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gQy6k5CEHGKiOJI0ugKFVUkk8XVFLkY9zLGCNjIpOps=; b=PZVOBA2p58yqfiE/MULqsB2VuOZGQf8ep3XJx0mTdt8jVTAi4AnNPGZA8q0SYITqXi6BFg UIsfqiv3j5KOMzTcQxOJITXqX8Ij6+ootSU6rKJ/V0PibKoF+X4tRwBk1bjohVRxMQMcJO T9c+TxtooIQ7qKwZNrcvmG95MpWmXwQ= Received: from mail-vk1-f200.google.com (mail-vk1-f200.google.com [209.85.221.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-524-Wi3vBIUJP7mdtOhBrVfTNw-1; Wed, 15 Nov 2023 20:29:27 -0500 X-MC-Unique: Wi3vBIUJP7mdtOhBrVfTNw-1 Received: by mail-vk1-f200.google.com with SMTP id 71dfb90a1353d-4ac1cbd8a18so40423e0c.0 for ; Wed, 15 Nov 2023 17:29:27 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098166; x=1700702966; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gQy6k5CEHGKiOJI0ugKFVUkk8XVFLkY9zLGCNjIpOps=; b=A4FX+XrrRViz7svb6iS1cRp3L64oJML0O7rditnVL9rVCXrvi15YT94MsVJh4LyWBG BZHgVsyv9ylfFXWCAi976yHXiSOLawYuWpgvoaSPgC2qUGcscqAOsZYnMZEYPQLGXVND POYzSKKtsqXCwDNV0mofSLjqrMVnh+V9P7fMKf5EPELzEzPv15sJ/jA3X+GdvV1Ewy9Q FPEUcOsCUWjufevJ9i4z5L1imMCdyRNRLJ9V49/KRZR8g9ydHJdlRcSRSCAA1J9jobSx SqxS3Nk1TOY/mx+zuycBJCEChk1qk4QatGswPSeeMp31o2vTspWXZb1WPO6E0qRjRAOH wPuA== X-Gm-Message-State: AOJu0YwZWwVmj6eQXxo7PVulQhCR5HS+HKYruy2FiI9NbyIBBZef/R2Y wellOQ5TOHB0WpR3Bh5CaWAFwSAolKgy1uYV5FJeczgjvAzy8dTegRfJIX5XkNAQZC+Udqu3jn7 Yf5yVaNPxp0M28jL3efqlcMP815nqGipKQaXxHDB63773/aEVyMtqtbtQmQqeu/9ijPDcveEN1A VMBmdz6A== X-Received: by 2002:a05:6122:1484:b0:4ab:f831:98d4 with SMTP id z4-20020a056122148400b004abf83198d4mr7555133vkp.0.1700098166387; Wed, 15 Nov 2023 17:29:26 -0800 (PST) X-Received: by 2002:a05:6122:1484:b0:4ab:f831:98d4 with SMTP id z4-20020a056122148400b004abf83198d4mr7555100vkp.0.1700098165901; Wed, 15 Nov 2023 17:29:25 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:25 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 09/12] mm/gup: Handle huge pud for follow_pud_mask() Date: Wed, 15 Nov 2023 20:29:05 -0500 Message-ID: <20231116012908.392077-10-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 15 Nov 2023 17:29:55 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782682166282906740 X-GMAIL-MSGID: 1782682166282906740 Teach follow_pud_mask() to be able to handle normal PUD pages like hugetlb. Rename follow_devmap_pud() to follow_huge_pud(), move it out of config CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD, instead let that macro only covers the devmap special operations, like pgmap. In the new follow_huge_pud(), taking care of possible CoR for hugetlb if necessary. Since at it, optimize the non-present check by adding a pud_present() early check before taking the pgtable lock, failing the follow_page() early if PUD is not present: that is required by both devmap or hugetlb. Use pud_huge() to also cover the pud_devmap() case. We need to export "struct follow_page_context" along the way, so that huge_memory.c can understand it. One trivial more thing to mention is, introduce "pud_t pud" in the code paths along the way, so the code doesn't dereference *pudp multiple time. Not only because that looks less straightforward, but also because if the dereference really happened, it's not clear whether there can be race to see different *pudp values when it's being modified at the same time. Setting ctx->page_mask properly for a PUD entry. As a side effect, this should also be able to optimize devmap GUP on PUD to be able to jump over the whole PUD range, but not yet verified. Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 17 +++---- mm/gup.c | 22 ++++----- mm/huge_memory.c | 98 +++++++++++++++++++++++------------------ 3 files changed, 73 insertions(+), 64 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index ec463410aecc..84815012d3cf 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -7,6 +7,11 @@ #include /* only for vma_is_dax() */ +struct follow_page_context { + struct dev_pagemap *pgmap; + unsigned int page_mask; +}; + vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf); int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr, @@ -222,8 +227,6 @@ static inline bool folio_test_pmd_mappable(struct folio *folio) struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, int flags, struct dev_pagemap **pgmap); -struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, - pud_t *pud, int flags, struct dev_pagemap **pgmap); vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf); @@ -372,18 +375,16 @@ static inline struct page *follow_devmap_pmd(struct vm_area_struct *vma, return NULL; } -static inline struct page *follow_devmap_pud(struct vm_area_struct *vma, - unsigned long addr, pud_t *pud, int flags, struct dev_pagemap **pgmap) -{ - return NULL; -} - static inline bool thp_migration_supported(void) { return false; } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +struct page *follow_huge_pud(struct vm_area_struct *vma, unsigned long addr, + pud_t *pud, int flags, + struct follow_page_context *ctx); + static inline int split_folio_to_list(struct folio *folio, struct list_head *list) { diff --git a/mm/gup.c b/mm/gup.c index 89c1584d68f0..55a2ae55f00f 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -25,11 +25,6 @@ #include "internal.h" -struct follow_page_context { - struct dev_pagemap *pgmap; - unsigned int page_mask; -}; - static inline void sanity_check_pinned_pages(struct page **pages, unsigned long npages) { @@ -751,24 +746,25 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, unsigned int flags, struct follow_page_context *ctx) { - pud_t *pud; + pud_t *pudp, pud; spinlock_t *ptl; struct page *page; struct mm_struct *mm = vma->vm_mm; - pud = pud_offset(p4dp, address); - if (pud_none(*pud)) + pudp = pud_offset(p4dp, address); + pud = *pudp; + if (pud_none(pud) || !pud_present(pud)) return no_page_table(vma, flags, address); - if (pud_devmap(*pud)) { - ptl = pud_lock(mm, pud); - page = follow_devmap_pud(vma, address, pud, flags, &ctx->pgmap); + if (pud_huge(pud)) { + ptl = pud_lock(mm, pudp); + page = follow_huge_pud(vma, address, pudp, flags, ctx); spin_unlock(ptl); return page; } - if (unlikely(pud_bad(*pud))) + if (unlikely(pud_bad(pud))) return no_page_table(vma, flags, address); - return follow_pmd_mask(vma, address, pud, flags, ctx); + return follow_pmd_mask(vma, address, pudp, flags, ctx); } static struct page *follow_p4d_mask(struct vm_area_struct *vma, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 6eb55f97a3d2..6748ef5f3fd9 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1207,49 +1207,6 @@ static void touch_pud(struct vm_area_struct *vma, unsigned long addr, update_mmu_cache_pud(vma, addr, pud); } -struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, - pud_t *pud, int flags, struct dev_pagemap **pgmap) -{ - unsigned long pfn = pud_pfn(*pud); - struct mm_struct *mm = vma->vm_mm; - struct page *page; - int ret; - - assert_spin_locked(pud_lockptr(mm, pud)); - - if (flags & FOLL_WRITE && !pud_write(*pud)) - return NULL; - - if (pud_present(*pud) && pud_devmap(*pud)) - /* pass */; - else - return NULL; - - if (flags & FOLL_TOUCH) - touch_pud(vma, addr, pud, flags & FOLL_WRITE); - - /* - * device mapped pages can only be returned if the - * caller will manage the page reference count. - * - * At least one of FOLL_GET | FOLL_PIN must be set, so assert that here: - */ - if (!(flags & (FOLL_GET | FOLL_PIN))) - return ERR_PTR(-EEXIST); - - pfn += (addr & ~PUD_MASK) >> PAGE_SHIFT; - *pgmap = get_dev_pagemap(pfn, *pgmap); - if (!*pgmap) - return ERR_PTR(-EFAULT); - page = pfn_to_page(pfn); - - ret = try_grab_page(page, flags); - if (ret) - page = ERR_PTR(ret); - - return page; -} - int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, pud_t *dst_pud, pud_t *src_pud, unsigned long addr, struct vm_area_struct *vma) @@ -1305,6 +1262,61 @@ void huge_pud_set_accessed(struct vm_fault *vmf, pud_t orig_pud) } #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ +struct page *follow_huge_pud(struct vm_area_struct *vma, unsigned long addr, + pud_t *pudp, int flags, + struct follow_page_context *ctx) +{ + struct mm_struct *mm = vma->vm_mm; + struct page *page; + pud_t pud = *pudp; + unsigned long pfn = pud_pfn(pud); + int ret; + + assert_spin_locked(pud_lockptr(mm, pudp)); + + if ((flags & FOLL_WRITE) && !pud_write(pud)) + return NULL; + + if (!pud_present(pud)) + return NULL; + + pfn += (addr & ~PUD_MASK) >> PAGE_SHIFT; + +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD + if (pud_devmap(pud)) { + /* + * device mapped pages can only be returned if the caller + * will manage the page reference count. + * + * At least one of FOLL_GET | FOLL_PIN must be set, so + * assert that here: + */ + if (!(flags & (FOLL_GET | FOLL_PIN))) + return ERR_PTR(-EEXIST); + + if (flags & FOLL_TOUCH) + touch_pud(vma, addr, pudp, flags & FOLL_WRITE); + + ctx->pgmap = get_dev_pagemap(pfn, ctx->pgmap); + if (!ctx->pgmap) + return ERR_PTR(-EFAULT); + } +#endif + page = pfn_to_page(pfn); + + if (!pud_devmap(pud) && !pud_write(pud) && + gup_must_unshare(vma, flags, page)) + return ERR_PTR(-EMLINK); + + ret = try_grab_page(page, flags); + if (ret) + page = ERR_PTR(ret); + else + ctx->page_mask = HPAGE_PUD_NR - 1; + + return page; +} + void huge_pmd_set_accessed(struct vm_fault *vmf) { bool write = vmf->flags & FAULT_FLAG_WRITE; From patchwork Thu Nov 16 01:29:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 165647 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2924386vqg; Wed, 15 Nov 2023 17:30:10 -0800 (PST) X-Google-Smtp-Source: AGHT+IEo373o9b6+gqsCwCSxApz4t80k2Pwws+EyHwaRHfpPVm56nwQjjtZ9kt0c0jt7Lh+4y7pq X-Received: by 2002:a17:903:244b:b0:1cc:4a83:d839 with SMTP id l11-20020a170903244b00b001cc4a83d839mr440622pls.17.1700098209954; Wed, 15 Nov 2023 17:30:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700098209; cv=none; d=google.com; s=arc-20160816; b=otaDrxZKszdfyW6G8/8jY+dFJQIugvPoVhbRe10jXqIGqK7qkHTfDDZ19lO2kb+iSK 1jTIvKxXVadHaZ2LrKLeipvU6uB9VBaaxxTScfEbWj20aq5rI2MG2pTD9LNeaQGhBnLS JNb9fSwRPiJJTnFZx1Y6cC4tpj6/kA3xPNqeXlgM6C26r7UjPJf6auq7lJjfCFxU03jY muEhVgRc/H+0D3SKM7kTJxLGjz+/ulU2GBJhnP07p1fSSkXxgKVmxhTLoisOVbn6FBp2 BNnY+bEMeUZaecruaAMCweRdFHx/ettqdtq9htS37qPiL6ev8gVhkTu+anMmfG0YSt5j s8Xw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KEE21en6sv5Z4IJFnP9A/1kLw7BwwEfESnJrJ/3CqcU=; fh=o+FJD7UTEFgAOwGBZppViFyhRpBzPGRfoO/6xzRNnMs=; b=z3bN1oudjNTd4ieDQKJNLovKytFMuZLJlwqWem3Xe4Lm1IvdE9mO+jh+YxbvXPP2L9 SpfddwdcEaiX3ljviD/F4Cz6chs8ArxCLfVWybukFE8CPmDAWuKV5OAtw26Y3srbDxhO rAanLOmWGN++/q3Bwyt3J8z5kdi1/s721dq7rEZ/38/8h3neTZ1SHoMMRFsiU+kdOnRS eeCrzgftEhCivA2xe35ltt0IBQ/anexd1eTOi+os2U8Gbc6rkdFWgakSHyJTciS3hfvv MN08K2kQDF4sJtGNjsYLS/hWLqssIVvbY8xVG6yBCKmfypxM423e79TA+KD86AECkchT gTzA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=h7zlXrn8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id jj13-20020a170903048d00b001cc4fbe9281si10691258plb.582.2023.11.15.17.30.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:30:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=h7zlXrn8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id B99B3811F13A; Wed, 15 Nov 2023 17:30:08 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235707AbjKPBaA (ORCPT + 28 others); Wed, 15 Nov 2023 20:30:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51402 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344246AbjKPB3p (ORCPT ); Wed, 15 Nov 2023 20:29:45 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDBD61706 for ; Wed, 15 Nov 2023 17:29:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098170; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KEE21en6sv5Z4IJFnP9A/1kLw7BwwEfESnJrJ/3CqcU=; b=h7zlXrn8zrW74CJMkpadoUkRJcfR5gZpgT7jXdBD4voMj88YYiZ37923d7kM75THsruA01 TVlx3y9AHjUCHu2gjJKejocUshcM9L+BYtQ++GV48G+SVVL0U6Omnt0c3i3Yu1SYhnJQdd X5UarljeMDla+lOez5uIOJd5PRcampE= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-153-n_kWC-KfMnuTYLLrOvvnyg-1; Wed, 15 Nov 2023 20:29:28 -0500 X-MC-Unique: n_kWC-KfMnuTYLLrOvvnyg-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-421be23b19aso883911cf.1 for ; Wed, 15 Nov 2023 17:29:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098168; x=1700702968; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KEE21en6sv5Z4IJFnP9A/1kLw7BwwEfESnJrJ/3CqcU=; b=hihRItkr9GIqNlc44y2699XFRqY/xQFNB/VnCH4cVPlZn2p5q/XeOxahsNtb3PgJk9 ciJ80ki/Qpi3RJYyUbn9NLzHlxNeap+BJ1VYkrs5JwVRyiMFL+GmeMlPqjP8L4aeDoPf YFpztV01MnHwVnESLjfp77gpRlMBObd1KKcZ+BQ2bFpwQskJkVo1cE6HX+B+m7bGZxab rhO9C7QsbNv5VBmFUn3pOYczSKTDRdQW4y99tlT5la3NQmBxaHXHvw2lNOM1neUaYsXO RgifVEnMPymoKztLKh64wT2lK06lnY6LfGaNOK7mOODbI403TTZ7JDg1cMJJFJ+YLT3p 84pA== X-Gm-Message-State: AOJu0YzrkWFiUSzgXuN+zt+gMKnO5wQd2GCSck2Wfl3rPCwqTUvbeMRu Bfjr0IhdtRA9R8o659aNrBnqN88prlIz7EE24I793MHoDRLX0iIWDhr/CkywMQh4PRSPqHrsuQ3 ZcZcNAWwgvq+BTnwyPWqynrme6Uzp55Iq+0ERB3PS6oTyd0ndfb05GezyPhG3gnueVBvgi6gtwl RpTAAHVw== X-Received: by 2002:ac8:5254:0:b0:421:c7ad:bb08 with SMTP id y20-20020ac85254000000b00421c7adbb08mr8123216qtn.6.1700098167845; Wed, 15 Nov 2023 17:29:27 -0800 (PST) X-Received: by 2002:ac8:5254:0:b0:421:c7ad:bb08 with SMTP id y20-20020ac85254000000b00421c7adbb08mr8123183qtn.6.1700098167489; Wed, 15 Nov 2023 17:29:27 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:27 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 10/12] mm/gup: Handle huge pmd for follow_pmd_mask() Date: Wed, 15 Nov 2023 20:29:06 -0500 Message-ID: <20231116012908.392077-11-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 15 Nov 2023 17:30:08 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782682180627604592 X-GMAIL-MSGID: 1782682180627604592 Do proper replacement of pmd_trans_huge() by using pmd_thp_or_huge() to also cover pmd_huge() as long as enabled. FOLL_TOUCH and FOLL_SPLIT_PMD only apply to THP, not yet huge. Since now follow_trans_huge_pmd() can handle hugetlb pages, renaming it into follow_huge_pmd() to match what it does. When at it, move the ctx->page_mask setup into follow_huge_pmd(), only set it when the page is valid. It was not a bug to set it before even if GUP failed (page==NULL), because follow_page_mask() callers always ignores page_mask if so. But doing so makes the code cleaner. Signed-off-by: Peter Xu --- mm/gup.c | 12 ++++++------ mm/huge_memory.c | 19 ++++++++++--------- mm/internal.h | 6 +++--- 3 files changed, 19 insertions(+), 18 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 55a2ae55f00f..7c210206470f 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -713,31 +713,31 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, spin_unlock(ptl); return page; } - if (likely(!pmd_trans_huge(pmdval))) + if (likely(!pmd_thp_or_huge(pmdval))) return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); if (pmd_protnone(pmdval) && !gup_can_follow_protnone(vma, flags)) return no_page_table(vma, flags, address); ptl = pmd_lock(mm, pmd); - if (unlikely(!pmd_present(*pmd))) { + pmdval = *pmd; + if (unlikely(!pmd_present(pmdval))) { spin_unlock(ptl); return no_page_table(vma, flags, address); } - if (unlikely(!pmd_trans_huge(*pmd))) { + if (unlikely(!pmd_thp_or_huge(pmdval))) { spin_unlock(ptl); return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } - if (flags & FOLL_SPLIT_PMD) { + if (pmd_trans_huge(pmdval) && (flags & FOLL_SPLIT_PMD)) { spin_unlock(ptl); split_huge_pmd(vma, pmd, address); /* If pmd was left empty, stuff a page table in there quickly */ return pte_alloc(mm, pmd) ? ERR_PTR(-ENOMEM) : follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } - page = follow_trans_huge_pmd(vma, address, pmd, flags); + page = follow_huge_pmd(vma, address, pmd, flags, ctx); spin_unlock(ptl); - ctx->page_mask = HPAGE_PMD_NR - 1; return page; } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 6748ef5f3fd9..43fb81218c5e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1486,32 +1486,32 @@ static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page, return !userfaultfd_huge_pmd_wp(vma, pmd); } -struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, - unsigned long addr, - pmd_t *pmd, - unsigned int flags) +struct page *follow_huge_pmd(struct vm_area_struct *vma, unsigned long addr, + pmd_t *pmd, unsigned int flags, + struct follow_page_context *ctx) { struct mm_struct *mm = vma->vm_mm; + pmd_t pmdval = *pmd; struct page *page; int ret; assert_spin_locked(pmd_lockptr(mm, pmd)); - page = pmd_page(*pmd); + page = pmd_page(pmdval); VM_BUG_ON_PAGE(!PageHead(page) && !is_zone_device_page(page), page); if ((flags & FOLL_WRITE) && - !can_follow_write_pmd(*pmd, page, vma, flags)) + !can_follow_write_pmd(pmdval, page, vma, flags)) return NULL; /* Avoid dumping huge zero page */ - if ((flags & FOLL_DUMP) && is_huge_zero_pmd(*pmd)) + if ((flags & FOLL_DUMP) && is_huge_zero_pmd(pmdval)) return ERR_PTR(-EFAULT); if (pmd_protnone(*pmd) && !gup_can_follow_protnone(vma, flags)) return NULL; - if (!pmd_write(*pmd) && gup_must_unshare(vma, flags, page)) + if (!pmd_write(pmdval) && gup_must_unshare(vma, flags, page)) return ERR_PTR(-EMLINK); VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && @@ -1521,10 +1521,11 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, if (ret) return ERR_PTR(ret); - if (flags & FOLL_TOUCH) + if (pmd_trans_huge(pmdval) && (flags & FOLL_TOUCH)) touch_pmd(vma, addr, pmd, flags & FOLL_WRITE); page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT; + ctx->page_mask = HPAGE_PMD_NR - 1; VM_BUG_ON_PAGE(!PageCompound(page) && !is_zone_device_page(page), page); return page; diff --git a/mm/internal.h b/mm/internal.h index 8450562744cf..bf0dc896c274 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1007,9 +1007,9 @@ int __must_check try_grab_page(struct page *page, unsigned int flags); /* * mm/huge_memory.c */ -struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, - unsigned long addr, pmd_t *pmd, - unsigned int flags); +struct page *follow_huge_pmd(struct vm_area_struct *vma, unsigned long addr, + pmd_t *pmd, unsigned int flags, + struct follow_page_context *ctx); /* * mm/mmap.c From patchwork Thu Nov 16 01:29:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 165649 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2924957vqg; Wed, 15 Nov 2023 17:31:16 -0800 (PST) X-Google-Smtp-Source: AGHT+IF4fB95m1Syw0S5WwA42mEfeueNrPj2JnL09p4SpbObJSQpMKGlyvbD2/zpnrJafoX2r+6s X-Received: by 2002:a05:6a20:9384:b0:183:e7bb:590a with SMTP id x4-20020a056a20938400b00183e7bb590amr17861812pzh.49.1700098275769; Wed, 15 Nov 2023 17:31:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700098275; cv=none; d=google.com; s=arc-20160816; b=J882coHeHeg72TF9Ffrue3HSVt2qsZdFB6STgU/RSA71bFDSVTcvtpqnGQTscPo3JI 8j0OEd+4T9qRO9Uzzs3eeuFQidQAt6deq0232eA8Q7A1UQe/6RMp3w3RQ5qR08cZwehV dpx5oph0Hn3XCHESc3mzJ/ruBVvw+jwnbgyXl5Tw8DaOazCaUW9y6GlrF2ObdwQlC4LX s7li2Vu0cont4jYHJ5dvCuVSpQ3z4UJqcdN7aubi8jlIZyyC8FauSrYeRUMkjtBakIGB FxIGs2kvbx9nBPJj9YyH4s+ylMf4cOTJhwc9YqoA1HkcNc2iF+d0meZJyBj9j/QK8qLK YVOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ltEuMSvLIUou1PxvqED3QFklmTWz+odfLbJoVHfc4cY=; fh=o+FJD7UTEFgAOwGBZppViFyhRpBzPGRfoO/6xzRNnMs=; b=nrjCoAP1/wCVN1O3DmQZKXu2/TC0R5o8M+V2uMnQCrLSMpxx/BFJYJE1owLOVzBQrz KNc8U63bBnNeZyb4CG5nxSBRYmYFp064FXtmNp/LYCo35NxcE2BA3d+Smln1gKoyU1Tn 6W0PLpPVSAY7cKyQLlOA/gQydBAKe2+t3dYGozBQ4ka4CRQ2l3WunZYA7R9N2pshCMQv u5G/JOpqbeLvnmCHJO7eIp42NWjmj8Sx5yNOwGRXrAVMdW0q/1eipJABIQpuWHCcw48p BWmJ2rQWCXhUjw2BPl5OiHba6LlJDZ66JE/2RH4J78i6JCFI+9vd65dnd/SAQkkMjZZ7 B3Zg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=dwaBAIPJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id y1-20020a655b41000000b005bdfda8e044si11085514pgr.775.2023.11.15.17.31.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:31:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=dwaBAIPJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 289BA80221CD; Wed, 15 Nov 2023 17:31:11 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344734AbjKPBaE (ORCPT + 28 others); Wed, 15 Nov 2023 20:30:04 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51474 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344572AbjKPB3q (ORCPT ); Wed, 15 Nov 2023 20:29:46 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 24F9B1730 for ; Wed, 15 Nov 2023 17:29:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098171; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ltEuMSvLIUou1PxvqED3QFklmTWz+odfLbJoVHfc4cY=; b=dwaBAIPJz8s/GuNVAuAOY6anN5K7DDxzyrHzDfGhoPOPlsN9BgnA/xC+sClGscAgVsJubc zjaBYCukv2BVOYDg12/ecd7HzXoqlSVyPi49UxsC7LSEr+JzCWhg6DOt8LsunoexMqHmZY Nlha1nX3SNtBs5Rg3cyH5PMUUNW3Yvg= Received: from mail-vs1-f71.google.com (mail-vs1-f71.google.com [209.85.217.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-92-0vpsEUi9ObuiQU58p1qRsg-1; Wed, 15 Nov 2023 20:29:30 -0500 X-MC-Unique: 0vpsEUi9ObuiQU58p1qRsg-1 Received: by mail-vs1-f71.google.com with SMTP id ada2fe7eead31-46107bc7936so12677137.1 for ; Wed, 15 Nov 2023 17:29:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098169; x=1700702969; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ltEuMSvLIUou1PxvqED3QFklmTWz+odfLbJoVHfc4cY=; b=cXhbHvZBlO3I+KA5cSlgRjcp2da8/VS/7PgSvMi64WHqV06HSZ+PxsF3jhobftjYR5 WMidV0fdMrR5H7JlnbZ9cpfWrK797Q6bR479OEk+GQ21UcnHen7E48frq+qZMm9hdhA5 zXi05er4mayNKpUFzEMpGUFDtbuIGQfCPdSCvwYaznF4GB/C29BYpg4jOx61lJUcbX5W NfLMQ8hD79sQd6w9zch7nmh0bkL+zVJpRatWgryE4WA5h27qLulLmfvepDYjzcvC5eyr 8/9L+ki2Cvv6OtPxcZHOv9SH2dwnCiaYxilHmllnHwzwqTJYtr7csrji3xA/5kLEL82P VkKg== X-Gm-Message-State: AOJu0YyOT0LtXmDOnvJmXT0P7DI+4ZTq4aHqT3feQybrsnMqImGAmTOB ECSVBcSSwUztTMsJnIc0mjktGDvNCu31oInccZMb5QTRZ85Kug2Gu+RBT5qQUkHHUfuj7i+pifG yHdI/UAlMVl7o8SoZ0rD3UZtBYx/VmReoo0ll4KD3Wfov/2WdmGy1d0i/lpGV3gs3eGeNuQLNth HCTaH3fg== X-Received: by 2002:a1f:1f95:0:b0:4ac:3094:fc14 with SMTP id f143-20020a1f1f95000000b004ac3094fc14mr7205523vkf.1.1700098169418; Wed, 15 Nov 2023 17:29:29 -0800 (PST) X-Received: by 2002:a1f:1f95:0:b0:4ac:3094:fc14 with SMTP id f143-20020a1f1f95000000b004ac3094fc14mr7205497vkf.1.1700098169016; Wed, 15 Nov 2023 17:29:29 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:28 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 11/12] mm/gup: Handle hugepd for follow_page() Date: Wed, 15 Nov 2023 20:29:07 -0500 Message-ID: <20231116012908.392077-12-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=0.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SORBS_WEB,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 15 Nov 2023 17:31:11 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782682249407644668 X-GMAIL-MSGID: 1782682249407644668 Hugepd is only used in PowerPC's hugetlbfs. follow_page_mask() used to leverage hugetlb APIs to access hugepd entries. Teach follow_page_mask() itself on hugepd. With previous refactors on fast-gup gup_huge_pd(), most of the code can be easily leveraged. Since follow_page() always only fetch one page, set the end to "address + PAGE_SIZE" should suffice. Signed-off-by: Peter Xu --- mm/gup.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 69 insertions(+), 8 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 7c210206470f..e635278f65f9 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -25,6 +25,11 @@ #include "internal.h" +static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, + unsigned long addr, unsigned int pdshift, + unsigned int flags, + struct follow_page_context *ctx); + static inline void sanity_check_pinned_pages(struct page **pages, unsigned long npages) { @@ -713,6 +718,9 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, spin_unlock(ptl); return page; } + if (unlikely(is_hugepd(__hugepd(pmd_val(pmdval))))) + return follow_hugepd(vma, __hugepd(pmd_val(pmdval)), + address, PMD_SHIFT, flags, ctx); if (likely(!pmd_thp_or_huge(pmdval))) return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); @@ -764,6 +772,10 @@ static struct page *follow_pud_mask(struct vm_area_struct *vma, if (unlikely(pud_bad(pud))) return no_page_table(vma, flags, address); + if (unlikely(is_hugepd(__hugepd(pud_val(pud))))) + return follow_hugepd(vma, __hugepd(pud_val(pud)), + address, PUD_SHIFT, flags, ctx); + return follow_pmd_mask(vma, address, pudp, flags, ctx); } @@ -772,15 +784,19 @@ static struct page *follow_p4d_mask(struct vm_area_struct *vma, unsigned int flags, struct follow_page_context *ctx) { - p4d_t *p4d; + p4d_t *p4d, p4dval; p4d = p4d_offset(pgdp, address); - if (p4d_none(*p4d)) - return no_page_table(vma, flags, address); - BUILD_BUG_ON(p4d_huge(*p4d)); - if (unlikely(p4d_bad(*p4d))) + p4dval = *p4d; + BUILD_BUG_ON(p4d_huge(p4dval)); + + if (p4d_none(p4dval) || unlikely(p4d_bad(p4dval))) return no_page_table(vma, flags, address); + if (unlikely(is_hugepd(__hugepd(p4d_val(p4dval))))) + return follow_hugepd(vma, __hugepd(p4d_val(p4dval)), + address, P4D_SHIFT, flags, ctx); + return follow_pud_mask(vma, address, p4d, flags, ctx); } @@ -812,7 +828,7 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, struct follow_page_context *ctx) { - pgd_t *pgd; + pgd_t *pgd, pgdval; struct mm_struct *mm = vma->vm_mm; ctx->page_mask = 0; @@ -827,11 +843,17 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, &ctx->page_mask); pgd = pgd_offset(mm, address); + pgdval = *pgd; if (pgd_none(*pgd) || unlikely(pgd_bad(*pgd))) - return no_page_table(vma, flags, address); + page = no_page_table(vma, flags, address); + else if (unlikely(is_hugepd(__hugepd(pgd_val(pgdval))))) + page = follow_hugepd(vma, __hugepd(pgd_val(pgdval)), + address, PGDIR_SHIFT, flags, ctx); + else + page = follow_p4d_mask(vma, address, pgd, flags, ctx); - return follow_p4d_mask(vma, address, pgd, flags, ctx); + return page; } struct page *follow_page(struct vm_area_struct *vma, unsigned long address, @@ -2850,6 +2872,37 @@ static int gup_huge_pd(hugepd_t hugepd, unsigned long addr, return 1; } + +static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, + unsigned long addr, unsigned int pdshift, + unsigned int flags, + struct follow_page_context *ctx) +{ + struct page *page; + struct hstate *h; + spinlock_t *ptl; + int nr = 0, ret; + pte_t *ptep; + + /* Only hugetlb supports hugepd */ + if (WARN_ON_ONCE(!is_vm_hugetlb_page(vma))) + return ERR_PTR(-EFAULT); + + h = hstate_vma(vma); + ptep = hugepte_offset(hugepd, addr, pdshift); + ptl = huge_pte_lock(h, mm, ptep); + ret = gup_huge_pd(hugepd, addr, pdshift, addr + PAGE_SIZE, + flags, &page, &nr)) { + spin_unlock(ptl); + + if (ret) { + WARN_ON_ONCE(nr != 1); + ctx->page_mask = (1U << huge_page_order(h)) - 1; + return page; + } + + return NULL; +} #else static inline int gup_huge_pd(hugepd_t hugepd, unsigned long addr, unsigned int pdshift, unsigned long end, unsigned int flags, @@ -2857,6 +2910,14 @@ static inline int gup_huge_pd(hugepd_t hugepd, unsigned long addr, { return 0; } + +static struct page *follow_hugepd(struct vm_area_struct *vma, hugepd_t hugepd, + unsigned long addr, unsigned int pdshift, + unsigned int flags, + struct follow_page_context *ctx) +{ + return NULL; +} #endif /* CONFIG_ARCH_HAS_HUGEPD */ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, From patchwork Thu Nov 16 01:29:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 165648 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2924871vqg; Wed, 15 Nov 2023 17:31:03 -0800 (PST) X-Google-Smtp-Source: AGHT+IEImx5YCOAb91/gAkM1SIjuFhvzulkTLHZAnzBCUOWlt/b7ZrQtCoZ305ILPXnPu9U1jx/K X-Received: by 2002:a17:90b:17c9:b0:27f:f260:ceaf with SMTP id me9-20020a17090b17c900b0027ff260ceafmr11776969pjb.10.1700098263069; Wed, 15 Nov 2023 17:31:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700098263; cv=none; d=google.com; s=arc-20160816; b=F0dI4QuiIQpJTZzMRmEGpTqlvHMCZcgP6nN/PpKs3RK4Yox698MzUNh+ZfZJjCgwlx gadBZpjCtpbZbsOxf10C6hqjcQIOTcEBtepc59bu0c0zGCsoNAa54lLjlkwebVK3t2nP Iqnd0dIcaGz4U1uZczUPal5eBSuvAdlWxTTH2yA+YsIHuJI4uTF0GOf6GNl8WR13AkqC bSqPnI/lA1mr08SmH6YJhyObINs6Y7PCPyaRKxcA2aEK5VpjSggGLeUMvMVly/nV55la 8p9IpgCKjMlNlzwH1MxZ/xg4DjdRuA22MYRsHP58nQAfdxQEkY3ah0V8yY+Zx9TxFoiH ju7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=k0e00AejVp7ba16+GtnRW4DXLpdJhvAKN3LJp7S4MZU=; fh=o+FJD7UTEFgAOwGBZppViFyhRpBzPGRfoO/6xzRNnMs=; b=guHrG4Sru249bz4lTFWgv4tirTPeRqGmUSH7GP1hNNPE8/12XIbmGKG7q22CYgxIIL yDlOpPl7JeKd/5hFhWK9wUtm8KJni0ukjiuAQHdVlBAfl2KC3GuEiHQVCuIXq8eXXg1v 9gBckHCQmCQkfbvObVgNxvVnIHu1pItWS2PLR+GXUJO0e99lmXhT0g5+ND/GVET4gG5N DySCMCtk7E1CvE6115yxb/G7pbN4JY1bNFJ9CqCk3vHELK4sM5MfLJc2Qv7fY1/mrmtW n56fAIAFeBAJJgeDXQ69trY44NW42W2tOcZOXwSTNzzMa43fCMN6/rhXkzgvYykXAv/d Bdaw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=emhn3xMi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id q6-20020a17090a2e0600b002748c1bbd79si898837pjd.6.2023.11.15.17.31.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:31:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=emhn3xMi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 87DAB80C47B1; Wed, 15 Nov 2023 17:30:49 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344541AbjKPBaP (ORCPT + 28 others); Wed, 15 Nov 2023 20:30:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50774 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344522AbjKPB35 (ORCPT ); Wed, 15 Nov 2023 20:29:57 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CEEC01998 for ; Wed, 15 Nov 2023 17:29:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098174; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=k0e00AejVp7ba16+GtnRW4DXLpdJhvAKN3LJp7S4MZU=; b=emhn3xMiKf8ctbhmkRjms69Nrxdm/+sdzlZElYnyd4aFAdu44akvpFO/Pbl+JJDDUoLXqx 3lAQ7Bq4N3RcuVQef3m5CXh6zrl2od9uQrTpt8U9BXBMhpfk/zJsImNPlRtUoyIqfc2F7a L6Hho4CKRyk4XPsUd93H7nSRzO8qjFU= Received: from mail-vs1-f71.google.com (mail-vs1-f71.google.com [209.85.217.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-385-0dE_vhAMM6SblPENVDGvlw-1; Wed, 15 Nov 2023 20:29:33 -0500 X-MC-Unique: 0dE_vhAMM6SblPENVDGvlw-1 Received: by mail-vs1-f71.google.com with SMTP id ada2fe7eead31-46107bc7936so12697137.1 for ; Wed, 15 Nov 2023 17:29:33 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098171; x=1700702971; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=k0e00AejVp7ba16+GtnRW4DXLpdJhvAKN3LJp7S4MZU=; b=UyCkTtwkEgkbc3eMSRFpITGAXUFtSNRDgLZH0w6L8HfhPR7Vjl3SdjvKOFNGv6JyX8 ZAATQT7XOWLu1w4SUhFfZt9/7RhK/8yycZG/jMyuri53fZ7OtIr+X0EzL+dngRPxHnu1 dK/4LzSh1Ppb9XpBRhfhr6oZDGz0ZMKJ5t60Vgwe3E0Dr9dSW6zc+HjhSImFRyv2egCX HrCrueFi4FzIMWV9VCoruxIP8KbQtqXxjDnSkLvUL7SD8KexEySOC/ESwZ9+zL4O9FPK 0HrhEZUgL/5Bv4Ewf29f3wswWeM4w2LIMY+3f6zLcu5rgw/Z/phgUbjtPaaAjfKwWHVk 4hhQ== X-Gm-Message-State: AOJu0YzsEHgzsUQYqOPyxXRGPyFvX99qU8mMInbHnD8jaZSNoc9EJ4NL KsMX6KLdP37RY/ajGhd6+RMGBER8U4OVcY+o0pa7QgcVCLfcnI6GWh0sD6NMkWj8aC0HDlb7p1b g0b08DkNxYg83gyHktNE8JTGDzJl9a3Dtlx1mEwxHzUnFELmkScyUby1y/yrRmyeV1VD2Ybe32e fDRVonpg== X-Received: by 2002:a1f:a017:0:b0:4ac:3694:7011 with SMTP id j23-20020a1fa017000000b004ac36947011mr7269398vke.0.1700098171317; Wed, 15 Nov 2023 17:29:31 -0800 (PST) X-Received: by 2002:a1f:a017:0:b0:4ac:3694:7011 with SMTP id j23-20020a1fa017000000b004ac36947011mr7269371vke.0.1700098170969; Wed, 15 Nov 2023 17:29:30 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:30 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 12/12] mm/gup: Merge hugetlb into generic mm code Date: Wed, 15 Nov 2023 20:29:08 -0500 Message-ID: <20231116012908.392077-13-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=0.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SORBS_WEB,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 15 Nov 2023 17:30:49 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782682236317466704 X-GMAIL-MSGID: 1782682236317466704 Now follow_page() is ready to handle hugetlb pages in whatever form, and over all architectures. Switch to the generic code path. Time to retire hugetlb_follow_page_mask(), following the previous retirement of follow_hugetlb_page() in 4849807114b8. There may be a slight difference of how the loops run when processing GUP over a large hugetlb range on either ARM64 (e.g. CONT_PMD) or RISCV (mostly its Svnapot extension on 64K huge pages): each loop of __get_user_pages() will resolve one pgtable entry with the patch applied, rather than relying on the size of hugetlb hstate, the latter may cover multiple entries in one loop. However, the performance difference should hopefully not be a major concern, considering that GUP just yet got 57edfcfd3419 ("mm/gup: accelerate thp gup even for "pages != NULL""), and that's not part of a performance analysis but a side dish. If the performance will be a concern, we can consider handle CONT_PTE in follow_page(), for example. Before that is justified to be necessary, keep everything clean and simple. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 7 ---- mm/gup.c | 15 +++------ mm/hugetlb.c | 71 ----------------------------------------- 3 files changed, 5 insertions(+), 88 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index bb07279b8991..87630a185acf 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -332,13 +332,6 @@ static inline void hugetlb_zap_end( { } -static inline struct page *hugetlb_follow_page_mask( - struct vm_area_struct *vma, unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ -} - static inline int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *dst_vma, diff --git a/mm/gup.c b/mm/gup.c index e635278f65f9..23fcac5aa3db 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -830,18 +830,11 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, { pgd_t *pgd, pgdval; struct mm_struct *mm = vma->vm_mm; + struct page *page; - ctx->page_mask = 0; - - /* - * Call hugetlb_follow_page_mask for hugetlb vmas as it will use - * special hugetlb page table walking code. This eliminates the - * need to check for hugetlb entries in the general walking code. - */ - if (is_vm_hugetlb_page(vma)) - return hugetlb_follow_page_mask(vma, address, flags, - &ctx->page_mask); + vma_pgtable_walk_begin(vma); + ctx->page_mask = 0; pgd = pgd_offset(mm, address); pgdval = *pgd; @@ -853,6 +846,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, else page = follow_p4d_mask(vma, address, pgd, flags, ctx); + vma_pgtable_walk_end(vma); + return page; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 29705e5c6f40..3013122a739f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6783,77 +6783,6 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, } #endif /* CONFIG_USERFAULTFD */ -struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags, - unsigned int *page_mask) -{ - struct hstate *h = hstate_vma(vma); - struct mm_struct *mm = vma->vm_mm; - unsigned long haddr = address & huge_page_mask(h); - struct page *page = NULL; - spinlock_t *ptl; - pte_t *pte, entry; - int ret; - - hugetlb_vma_lock_read(vma); - pte = hugetlb_walk(vma, haddr, huge_page_size(h)); - if (!pte) - goto out_unlock; - - ptl = huge_pte_lock(h, mm, pte); - entry = huge_ptep_get(pte); - if (pte_present(entry)) { - page = pte_page(entry); - - if (!huge_pte_write(entry)) { - if (flags & FOLL_WRITE) { - page = NULL; - goto out; - } - - if (gup_must_unshare(vma, flags, page)) { - /* Tell the caller to do unsharing */ - page = ERR_PTR(-EMLINK); - goto out; - } - } - - page = nth_page(page, ((address & ~huge_page_mask(h)) >> PAGE_SHIFT)); - - /* - * Note that page may be a sub-page, and with vmemmap - * optimizations the page struct may be read only. - * try_grab_page() will increase the ref count on the - * head page, so this will be OK. - * - * try_grab_page() should always be able to get the page here, - * because we hold the ptl lock and have verified pte_present(). - */ - ret = try_grab_page(page, flags); - - if (WARN_ON_ONCE(ret)) { - page = ERR_PTR(ret); - goto out; - } - - *page_mask = (1U << huge_page_order(h)) - 1; - } -out: - spin_unlock(ptl); -out_unlock: - hugetlb_vma_unlock_read(vma); - - /* - * Fixup retval for dump requests: if pagecache doesn't exist, - * don't try to allocate a new page but just skip it. - */ - if (!page && (flags & FOLL_DUMP) && - !hugetlbfs_pagecache_present(h, vma, address)) - page = ERR_PTR(-EFAULT); - - return page; -} - long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags)