From patchwork Wed Jun 28 21:53:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 114084 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp9240542vqr; Wed, 28 Jun 2023 15:12:50 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5zY4wpUpvVTp5ghr7CDFVzK+7JKxV/IQtVllvHzJMgEQt4wDXuc+toZdgQPvhMwMuytfAG X-Received: by 2002:a05:6a00:1d8b:b0:67f:7403:1fe8 with SMTP id z11-20020a056a001d8b00b0067f74031fe8mr5134130pfw.3.1687990370538; Wed, 28 Jun 2023 15:12:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687990370; cv=none; d=google.com; s=arc-20160816; b=UxNUhZ2T8a8j9JgmHbPbumk8RRAccp994jhG4jpDFYPY9V//a1G+ansU/tuucyJhYZ tyTjXsArJi4DzXudz12TfqiWqUVc4NPPeuFVoErjBwV0W96xcuZ1hyZv9q2lxnKRXV2S VaP0f9B8nVgeS9+1869v1AOT6pzk5Xy4pWV1b2l2/LQyLVlQje8pKv7rcEVEtcO2YniJ Dx67mImiw0CYYbN8tvwIFqOXfcT/mh+VDB/Ijb1Rkr1+4XQUYH77CrUfdIhevtV0SqG2 r0SlnYjNbB9M6N4BnOnm6R/eiRvoA3qzERN0UUsxCZPO+TFaNQWfNBsBn4eOWwNOWydi BjQw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=d8FL5BW3box1F48wj3b8rnHOt4NGgVkomS4C4Z085kw=; fh=YRm1SqlPzCC/sbB3u8ugRd2MBYnp3vC9/g67nUn+DD4=; b=kvpkr81ibVNtjf16TrHCPzUJlanlGlZZMs4QGoUvRrJTs1AXmRZJbTfuNWy0ZXAgUa mE7IPusUtSWpXfecNutZiYtdxFxqzz1R5+gWiiRWe8EReSer9GbOMcHNQrPO7m1qoIeR UVmjDA6jZdD6bad9Vpe47jK6HCul+YwPWhRHKwctepUySS5I+ExTS6Jhg15b5P6zVtv2 YSYnJAeGJvKdAiYlmPhmFgW+EekC3KXVA5v2O052xhRosqm8Iidgdbc8PVbZkuOFW++m Rv9qO4CbZxkNEbLvxYQX9x3vLP5Ug/aOWfQpyD+9Mjlpl6gLxOxZdoJqauLZNvBdlAGz ukxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ic2L1rOp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b13-20020a63d80d000000b0054fe7b18671si9846072pgh.809.2023.06.28.15.12.29; Wed, 28 Jun 2023 15:12:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ic2L1rOp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231769AbjF1VyM (ORCPT + 99 others); Wed, 28 Jun 2023 17:54:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231447AbjF1VyG (ORCPT ); Wed, 28 Jun 2023 17:54:06 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 36C62210E for ; Wed, 28 Jun 2023 14:53:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989199; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d8FL5BW3box1F48wj3b8rnHOt4NGgVkomS4C4Z085kw=; b=ic2L1rOpJ0/6YWQyVinnSV+wo4gaMSAydTYHE4XfVtt0UfnxJEfMyvkTsq0flqMxvHHk6R iC4lQAKDvClQj7YsyioEOcXpwcn4q13Og/q6pjNHhlDyHh6ivzPNwQNtP3KEFfECpyHsCh tgZ4CXkR+emPaYQdk7tc8UV0Fg3fX30= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-278-XGuUAjBEOgOhU_ifJYkfFw-1; Wed, 28 Jun 2023 17:53:14 -0400 X-MC-Unique: XGuUAjBEOgOhU_ifJYkfFw-1 Received: by mail-qv1-f69.google.com with SMTP id 6a1803df08f44-635f8ae5f88so39216d6.0 for ; Wed, 28 Jun 2023 14:53:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989194; x=1690581194; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d8FL5BW3box1F48wj3b8rnHOt4NGgVkomS4C4Z085kw=; b=aEv07ZggtSsJ5Lgec8AApn2cWsSXopBZ8N9XCAGr6UgIdsr4l0lc0hN6cyiVp1Y8Gx u79mdSSIFXN9Dl+hGK3DwC/Y6JcWsl50PuxCtb7pKYnY0ZBeWhpiMhMc9MfIq/0TBcjV uUdaWj9WOcHGF/HgpSpMuKn/S8m+mdDGw5Q7yHoedROgi7cQs2vDxmI/FyKcO6vM+91h B2JrCKujx7X70lIeP0L0t6kXIaQxzr+yxW3SqOfXJw1uaga5uERQ23pnvkLTst33dZpN FqKuresk64z02rT3hZET57Z6ouEeHsqydAt1S7XulTW0coDj1brEbTWmG11eRWWIbqpM bKOw== X-Gm-Message-State: AC+VfDwJK+BG+EJ9Njfoy3VqwBtZttpDjKH1Rito48ikzMM8WhclqsbN aNO23qYmqAwlsFY9qVQ7uPqqGlKywAJ5Hlpvxkrktd6yWYNL/GFWmKFzlke4CuniE+OsNI74wBc T55TS2KlZamMZWsW/ODqhKwqG X-Received: by 2002:a05:6214:401a:b0:62d:fdc4:1e8b with SMTP id kd26-20020a056214401a00b0062dfdc41e8bmr42375749qvb.2.1687989194251; Wed, 28 Jun 2023 14:53:14 -0700 (PDT) X-Received: by 2002:a05:6214:401a:b0:62d:fdc4:1e8b with SMTP id kd26-20020a056214401a00b0062dfdc41e8bmr42375738qvb.2.1687989194020; Wed, 28 Jun 2023 14:53:14 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:13 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 1/8] mm/hugetlb: Handle FOLL_DUMP well in follow_page_mask() Date: Wed, 28 Jun 2023 17:53:03 -0400 Message-ID: <20230628215310.73782-2-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1769986190619365019?= X-GMAIL-MSGID: =?utf-8?q?1769986190619365019?= Firstly, the no_page_table() is meaningless for hugetlb which is a no-op there, because a hugetlb page always satisfies: - vma_is_anonymous() == false - vma->vm_ops->fault != NULL So we can already safely remove it in hugetlb_follow_page_mask(), alongside with the page* variable. Meanwhile, what we do in follow_hugetlb_page() actually makes sense for a dump: we try to fault in the page only if the page cache is already allocated. Let's do the same here for follow_page_mask() on hugetlb. It should so far has zero effect on real dumps, because that still goes into follow_hugetlb_page(). But this may start to influence a bit on follow_page() users who mimics a "dump page" scenario, but hopefully in a good way. This also paves way for unifying the hugetlb gup-slow. Reviewed-by: Mike Kravetz Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu --- mm/gup.c | 9 ++------- mm/hugetlb.c | 9 +++++++++ 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index a8336b39d6b5..06bc2e31f209 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -767,7 +767,6 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, struct follow_page_context *ctx) { pgd_t *pgd; - struct page *page; struct mm_struct *mm = vma->vm_mm; ctx->page_mask = 0; @@ -780,12 +779,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, * hugetlb_follow_page_mask is only for follow_page() handling here. * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ - if (is_vm_hugetlb_page(vma)) { - page = hugetlb_follow_page_mask(vma, address, flags); - if (!page) - page = no_page_table(vma, flags); - return page; - } + if (is_vm_hugetlb_page(vma)) + return hugetlb_follow_page_mask(vma, address, flags); pgd = pgd_offset(mm, address); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bce28cca73a1..d04ba5782fdd 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6498,6 +6498,15 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, spin_unlock(ptl); out_unlock: hugetlb_vma_unlock_read(vma); + + /* + * Fixup retval for dump requests: if pagecache doesn't exist, + * don't try to allocate a new page but just skip it. + */ + if (!page && (flags & FOLL_DUMP) && + !hugetlbfs_pagecache_present(h, vma, address)) + page = ERR_PTR(-EFAULT); + return page; } From patchwork Wed Jun 28 21:53:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 114087 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp9241663vqr; Wed, 28 Jun 2023 15:14:54 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4Lh/3PRk73T2e7qkfTa8W5wt+o4A+LbopyXb+LIRy8/h4PyKNxwUn8+imzSMcx2oOkbRA+ X-Received: by 2002:adf:e804:0:b0:313:f380:f35c with SMTP id o4-20020adfe804000000b00313f380f35cmr7190127wrm.6.1687990494563; Wed, 28 Jun 2023 15:14:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687990494; cv=none; d=google.com; s=arc-20160816; b=utCLZpzXpG6ZkyXFbMlDaU8R1mxnvPfXJoVY030MHFfi/rJkwKjI72/3Y1FGlxlYYI ES2Sp9j2AEkmvjnQ/hqLp78K6WSpxVktWWvJFiBlHlq0CxXXv+1fCk7ag6uQ5lxS7pAU n6LRuTDG24cGXoKGyd87wWipqjjtbrxK+g+2qDF5fwZ21ALeKNO1K+n7X3qhCzke92/W MDAd4prtWG1yGafm143TD+2DN2KZ3YLAXJaE61SnhZ0Ntr8WbFexEMge3bUUZ0AO7FTS vG92QLVYLNCOShwFsuJqivMn/5QuNXg4RCPuh2+01SuzyxBIxfbDfAqIZKyHOOY1jX9u 4I4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=v3NUxws11iZimZs89ga1+mchuciCcK5t8cj7qgaLP/g=; fh=YRm1SqlPzCC/sbB3u8ugRd2MBYnp3vC9/g67nUn+DD4=; b=yw1BMQNiCQkDcp6chfve2TXRnQbU+p8g26F8Ve9O3k0ydTFghsM7x55BPSLUg2LVJU Azz/gimoiU4PrAXfDU2j4xuN5gkM4TGAlRk2cZx1STDlc1Eh83G3cAzoTkUE9aU7bL/n TBaUbt9ys2jmq1jh7+CeD1PJxBLJAJnvNAhexUvxc8ntEUrkLww4jNcO1F3hsoJKN7MO Xlf/UTVMqyGzj7iaEOrFIJQOsx1EuRahxk+Q+FevM6Lvc5JBd3g4aefQm8w0jFHgjiOH kIugyqqO4h7VewoHCmQzy/KgjPDLxfQKtJMvixeFBfuBjwUdrwCuoJXWQUpnX5DZzGPw 623A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OEl983S6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i12-20020a05640200cc00b0051bf8f889dbsi5601001edu.448.2023.06.28.15.14.29; Wed, 28 Jun 2023 15:14:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OEl983S6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231716AbjF1VyV (ORCPT + 99 others); Wed, 28 Jun 2023 17:54:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37748 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231303AbjF1VyJ (ORCPT ); Wed, 28 Jun 2023 17:54:09 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B47E61FFB for ; Wed, 28 Jun 2023 14:53:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989197; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v3NUxws11iZimZs89ga1+mchuciCcK5t8cj7qgaLP/g=; b=OEl983S61xAqUc7taiXElasALKboGk6fgtZlBFgUd3JbyO63frx7h8JFAIq9/CMAz5m3Kd p6gr7hj052LFA+Milgu40T1D1LtvRkcc/GZnl9NwOl0bexv3vNv45p1tVaFCv/09zukdP2 aNVoUAp2Pqa47jUVLdeeflnZ8lC8zCg= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-558-JOaH0agJOf6-Ev5MPtQW0A-1; Wed, 28 Jun 2023 17:53:16 -0400 X-MC-Unique: JOaH0agJOf6-Ev5MPtQW0A-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-62dd79f63e0so60436d6.0 for ; Wed, 28 Jun 2023 14:53:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989195; x=1690581195; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v3NUxws11iZimZs89ga1+mchuciCcK5t8cj7qgaLP/g=; b=KtfDYZi4WJPdTgeral5vIpmT5bcvhfjPh57yKrTRUDfCUY2MeHnlYGdrtM/OGnuhvJ /fDaeZWKQkcwcawYT1aF6FgeRb5O+WSDd3VdbLQuqsqOgWuINYxLyVf7GJUoD2bNvL0F g04zGBKT7QwQaKGv9nPVD+4Pv/eLPIu9krWLUMkqoSvSalhgYBouT9QpemVTpAPPAcPI 1NrXMxBJDZ5RMy1QY6C2+0OKM9r47A6N4USodc2c1w3VwZA9U+YEXcIUT8t9gjO7nBlp REuAtz9jzoedQH+yRBJCk9jy+cj0cdMtQ2FkxbZ78ywAnxhcNjnSuK8rQ+PPrvyPz5pO Ou+w== X-Gm-Message-State: AC+VfDyueb1hZSCRtAOP/g9plnRY+BjX1WCFtuQhPr7TK8D7rYrgdCZl qkQKtRASk85m7RGw6WxU2AlL4qkR2VfR1u0A4xWDT7h3jRruk9Cb0dt0zC9+5Y8DbtwazBZ8iBj 9nG0Dr/0is829bfVXlIOzJy2t X-Received: by 2002:a05:6214:f2f:b0:62b:6c6f:b3e3 with SMTP id iw15-20020a0562140f2f00b0062b6c6fb3e3mr45427363qvb.3.1687989195524; Wed, 28 Jun 2023 14:53:15 -0700 (PDT) X-Received: by 2002:a05:6214:f2f:b0:62b:6c6f:b3e3 with SMTP id iw15-20020a0562140f2f00b0062b6c6fb3e3mr45427346qvb.3.1687989195254; Wed, 28 Jun 2023 14:53:15 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:15 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 2/8] mm/hugetlb: Prepare hugetlb_follow_page_mask() for FOLL_PIN Date: Wed, 28 Jun 2023 17:53:04 -0400 Message-ID: <20230628215310.73782-3-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1769986320772941665?= X-GMAIL-MSGID: =?utf-8?q?1769986320772941665?= follow_page() doesn't use FOLL_PIN, meanwhile hugetlb seems to not be the target of FOLL_WRITE either. However add the checks. Namely, either the need to CoW due to missing write bit, or proper unsharing on !AnonExclusive pages over R/O pins to reject the follow page. That brings this function closer to follow_hugetlb_page(). So we don't care before, and also for now. But we'll care if we switch over slow-gup to use hugetlb_follow_page_mask(). We'll also care when to return -EMLINK properly, as that's the gup internal api to mean "we should unshare". Not really needed for follow page path, though. When at it, switching the try_grab_page() to use WARN_ON_ONCE(), to be clear that it just should never fail. When error happens, instead of setting page==NULL, capture the errno instead. Reviewed-by: Mike Kravetz Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu --- mm/hugetlb.c | 33 ++++++++++++++++++++++----------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d04ba5782fdd..4410139cf890 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6462,13 +6462,7 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, struct page *page = NULL; spinlock_t *ptl; pte_t *pte, entry; - - /* - * FOLL_PIN is not supported for follow_page(). Ordinary GUP goes via - * follow_hugetlb_page(). - */ - if (WARN_ON_ONCE(flags & FOLL_PIN)) - return NULL; + int ret; hugetlb_vma_lock_read(vma); pte = hugetlb_walk(vma, haddr, huge_page_size(h)); @@ -6478,8 +6472,23 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, ptl = huge_pte_lock(h, mm, pte); entry = huge_ptep_get(pte); if (pte_present(entry)) { - page = pte_page(entry) + - ((address & ~huge_page_mask(h)) >> PAGE_SHIFT); + page = pte_page(entry); + + if (!huge_pte_write(entry)) { + if (flags & FOLL_WRITE) { + page = NULL; + goto out; + } + + if (gup_must_unshare(vma, flags, page)) { + /* Tell the caller to do unsharing */ + page = ERR_PTR(-EMLINK); + goto out; + } + } + + page += ((address & ~huge_page_mask(h)) >> PAGE_SHIFT); + /* * Note that page may be a sub-page, and with vmemmap * optimizations the page struct may be read only. @@ -6489,8 +6498,10 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, * try_grab_page() should always be able to get the page here, * because we hold the ptl lock and have verified pte_present(). */ - if (try_grab_page(page, flags)) { - page = NULL; + ret = try_grab_page(page, flags); + + if (WARN_ON_ONCE(ret)) { + page = ERR_PTR(ret); goto out; } } From patchwork Wed Jun 28 21:53:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 114091 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp9254097vqr; Wed, 28 Jun 2023 15:44:26 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6EX7m47sNl/aX/084XfwSt8/4nw9zSORlba8TI2o1HBS6o9NiMK0UR0X+QGYah0tHP2huO X-Received: by 2002:a05:6a20:4418:b0:125:213e:8b68 with SMTP id ce24-20020a056a20441800b00125213e8b68mr26597281pzb.0.1687992266237; Wed, 28 Jun 2023 15:44:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687992266; cv=none; d=google.com; s=arc-20160816; b=ZrOzYm1TtmHwwfQ58NvRNIuNyxYTXce07WpxT6AV+ZT+YwlzqSdeyZCvI8fOvq0mdZ ZBsTxpVu9X/bRdjSqVpEZW4Pm4XaSWXvzcx5/ilBPzJSZsClgYuRf8lT7SgvEoWNB1VM AFM0gSC5gD8uzIEkA5Vn78SPdrALx+7plF7Zbav3BA+XIjQjr+YaXtvTfVLJrcQNHSWI xkXkVD/P9MC2GMRca6pUxGeQs+8kWStSFNP9kyMJaD/TmnhAoSpoYQwOd0JWJksjbNDb AKwqPy+FCqSzhDTROOn/d4xfkMYLYIjv/zBk8UKyNZcYVThqvBD8bgAwzobBQoOnG5j4 DR4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=IlLdQZDWL4abQusa0fWM05NgtC5yViIwxz1MKIy8bPA=; fh=YRm1SqlPzCC/sbB3u8ugRd2MBYnp3vC9/g67nUn+DD4=; b=v5h+jYlrZunHIz4kATjRz2PDmau4ml+WJNZ0HkIxdIoub1hmt5HwZo8rKAIDypn3oL 3A/ilz9WunSKLTleX5pB6SPb4Y6yTEt8XzoCmqOzcLPLZzR1X1K+0ByF+DRo5iAJer7L OUVWZdCJMfeJ3enT0TOXhgNqCv2JOTTA4ApJi2cuVPznGZuNMHEfRboQO94BfRemEk+T Cp17hWhkRNWKvk448LcUaXJ27aFHreQWxsABTo7v+ieWI1KMI9snwCcMDtnJPGA/T4jY ckiRKFbN5DPPu9HfgYsxrkE+9s94/2kF8HPPfSBQ4H4329abL/zs1Tg94o201ZRPSyL/ Lnug== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZgkP7MNe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k69-20020a638448000000b0053422305c20si10344018pgd.14.2023.06.28.15.44.13; Wed, 28 Jun 2023 15:44:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZgkP7MNe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232137AbjF1Vy0 (ORCPT + 99 others); Wed, 28 Jun 2023 17:54:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231637AbjF1VyJ (ORCPT ); Wed, 28 Jun 2023 17:54:09 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D2022111 for ; Wed, 28 Jun 2023 14:53:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989199; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IlLdQZDWL4abQusa0fWM05NgtC5yViIwxz1MKIy8bPA=; b=ZgkP7MNeWTBFLqPLJ1hIGcTOfs+K3Ubo7sNdgL/x80kV3dU1HNoTsPwBEq4a96EaqY8tCR dSH0Q+PDkk7ZC2VgZCPR26esbSbwrjRjegUEEBPSXKLuuO2j86/RRZmYT9ZzclvGpFd5oj GD+DnvXwRDWbSVbiwt6+SsnzjIEsrzk= Received: from mail-vk1-f198.google.com (mail-vk1-f198.google.com [209.85.221.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-335-lg8yFQ42PUCqedzxscCwcA-1; Wed, 28 Jun 2023 17:53:17 -0400 X-MC-Unique: lg8yFQ42PUCqedzxscCwcA-1 Received: by mail-vk1-f198.google.com with SMTP id 71dfb90a1353d-4716b1f55b9so2889e0c.0 for ; Wed, 28 Jun 2023 14:53:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989197; x=1690581197; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IlLdQZDWL4abQusa0fWM05NgtC5yViIwxz1MKIy8bPA=; b=d078f08flziWSVLAsr/H1JDvkV+WYJ46hM6vzWIwJloj+G7lSGzKI1J2HTSZK4fHia V/M8ZAP6b+682BLj/C1lRA1qeEWIpfvPGqLm0NVIgRfXBrv+LXjb4QLkNY4Nh0q/Ao2L a5ulKt0adNY5f/oiXNOPGPFaNpue8HhmxjTNoCAqPdBV9dBRLcB69jDT/A/CR4GOOjxk RP9aTSGqHA9wOBp/kUd4iNVPrAis3jzo8B84UTdJcLf0EUDwFy+K1qDm8UaXquqQmmXo zohUF01NgX/Wl4i5jQOYWLrPN9e5jcl/oqcoommB6KXn0VgvSZA3vyuWo7QcAmr1oM7H 5fkg== X-Gm-Message-State: AC+VfDwb/zm9HJ5L/ZQRPccNPt9ezGEQv3PVXGdswDbL8qQ2b5pNgXbn Hfsb6+/zvwQV1vH7RY7jCzM4UYbQw6DTSZSFQ/28u5NgyQzKKh//b3VdcTPv61/iZh4VsAq58SB GgZeBsGHx/1+H3umIHSZeLM51 X-Received: by 2002:a05:6102:358d:b0:440:b763:a69d with SMTP id h13-20020a056102358d00b00440b763a69dmr12045541vsu.2.1687989196707; Wed, 28 Jun 2023 14:53:16 -0700 (PDT) X-Received: by 2002:a05:6102:358d:b0:440:b763:a69d with SMTP id h13-20020a056102358d00b00440b763a69dmr12045528vsu.2.1687989196447; Wed, 28 Jun 2023 14:53:16 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:16 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 3/8] mm/hugetlb: Add page_mask for hugetlb_follow_page_mask() Date: Wed, 28 Jun 2023 17:53:05 -0400 Message-ID: <20230628215310.73782-4-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1769988178041038936?= X-GMAIL-MSGID: =?utf-8?q?1769988178041038936?= follow_page() doesn't need it, but we'll start to need it when unifying gup for hugetlb. Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 8 +++++--- mm/gup.c | 3 ++- mm/hugetlb.c | 5 ++++- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ca3c8e10f24a..9f282f370d96 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -131,7 +131,8 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, int copy_hugetlb_page_range(struct mm_struct *, struct mm_struct *, struct vm_area_struct *, struct vm_area_struct *); struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags); + unsigned long address, unsigned int flags, + unsigned int *page_mask); long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, struct page **, unsigned long *, unsigned long *, long, unsigned int, int *); @@ -297,8 +298,9 @@ static inline void adjust_range_if_pmd_sharing_possible( { } -static inline struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags) +static inline struct page *hugetlb_follow_page_mask( + struct vm_area_struct *vma, unsigned long address, unsigned int flags, + unsigned int *page_mask) { BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ } diff --git a/mm/gup.c b/mm/gup.c index 06bc2e31f209..1e2e23084f3c 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -780,7 +780,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ if (is_vm_hugetlb_page(vma)) - return hugetlb_follow_page_mask(vma, address, flags); + return hugetlb_follow_page_mask(vma, address, flags, + &ctx->page_mask); pgd = pgd_offset(mm, address); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4410139cf890..15e82a8a2b76 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6454,7 +6454,8 @@ static inline bool __follow_hugetlb_must_fault(struct vm_area_struct *vma, } struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags) + unsigned long address, unsigned int flags, + unsigned int *page_mask) { struct hstate *h = hstate_vma(vma); struct mm_struct *mm = vma->vm_mm; @@ -6504,6 +6505,8 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, page = ERR_PTR(ret); goto out; } + + *page_mask = (1U << huge_page_order(h)) - 1; } out: spin_unlock(ptl); From patchwork Wed Jun 28 21:53:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 114089 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp9246940vqr; Wed, 28 Jun 2023 15:26:20 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4+J++LzrFJGqtx0PQiN2fRMLezNmSBeeFSUyFdfyOanU0XzpDzeMZyKODSwphNrJBcGDMY X-Received: by 2002:a17:902:c945:b0:1ac:40f7:8b5a with SMTP id i5-20020a170902c94500b001ac40f78b5amr42932714pla.3.1687991180219; Wed, 28 Jun 2023 15:26:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687991180; cv=none; d=google.com; s=arc-20160816; b=KvzRaMZjw9i1yxsiWFlVgk5/9Ca63krGAw3v7SZv9q8aAVDJ/F3oo8j5hnxR1AneoV EP7n5phBKSfnkmet1P9KpV0OxIK+v83NetvPZjnwdmQ2Gijd1mXwFi/Fee2Mnukij7Cm 9tyn/pJyKMCzGQH0/u+Y7soNa7gcBzYjWnMw7uJddq9lRqW2qimTEJl4T2MaHC+G1clh RZYCnBzY8CKzXW45dw9kCX7K3Kb0OdWx7I5sECPtUFnEWbGfrVJSY42WprkIdr1jQZ0s 11fV9ZlC02JDrw45oZ8Qof72CxJ3NE0rTdvuIPXZB9hUqHX61dLJdWZoWE0dw5HtWeU7 4yTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=C1A4ivN2AFC1lD1f3DRS9KX47rVe/ku79YHFPXkOfKI=; fh=YRm1SqlPzCC/sbB3u8ugRd2MBYnp3vC9/g67nUn+DD4=; b=AmdsXkAI2wv+0qOFIn35uIoAVisVusTUURTPJDkGMg9kVlcCOWmx7OgCRNXZtGRusz ybiv8YIjlCvCH/FxIhckLbTGF++vFni+ZZTJLs14RJKFpdS8ZBqEZ5kCbdmWcCmjgHvw gaRMfOSufWtx6QyDSEvmpmInXFmsdUZIL9aEy8X+0DrCAeDgqAVPggebW/dAfr6z+qIk XsrFaJ3Zhlf1SceAK9MT73mYjM8fjB2bsHHngYNgYeVOYlkjLCSuFat4X81aBxucMTRA m8bTsz8nW8Tpfzaxcku3jCJ1hQMZavFl4KCJRWSMBh2DT8lYzthUR4jpqaonkw+SHWxK jC/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Kt5ZAVDd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o19-20020a170902779300b001b69de1eac0si8985668pll.367.2023.06.28.15.26.06; Wed, 28 Jun 2023 15:26:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Kt5ZAVDd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232180AbjF1Vyb (ORCPT + 99 others); Wed, 28 Jun 2023 17:54:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37764 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231636AbjF1VyJ (ORCPT ); Wed, 28 Jun 2023 17:54:09 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B4CC2110 for ; Wed, 28 Jun 2023 14:53:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989199; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=C1A4ivN2AFC1lD1f3DRS9KX47rVe/ku79YHFPXkOfKI=; b=Kt5ZAVDdr/jVhGx1pHDLciGBS5vMnFB4Ut2NcAbS5uYT0z1LRk4hQLNBXKmPIjBybkDoLm AKdANP0+1OwJzS32vBeWGapRVMeCNTZFe2u/R+xtmIllhpxxvhar9qT+8Yz29M2ZzGkLZD yNeKT/bQwUxyPULf8pU1+t2udNgVMzA= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-244-Repg9S1sOki2fFtIypOy6g-1; Wed, 28 Jun 2023 17:53:18 -0400 X-MC-Unique: Repg9S1sOki2fFtIypOy6g-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-635df023844so9736d6.1 for ; Wed, 28 Jun 2023 14:53:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989198; x=1690581198; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C1A4ivN2AFC1lD1f3DRS9KX47rVe/ku79YHFPXkOfKI=; b=HWx37WgvLrFAFvxRb62f5CdxN3t01hlsq6cxG3AnCyKUUEpok9TcaPw1vwHxZG0R0o 77DjUbORdmvzfWWI2We12w/1k8+s4RDFlnTKf2W8vGrlMwQjx62zem4XoyBJz5Tpwbv7 rn1uWdKAOzxaP1kWMH/tAuKPfi7zX2c1MCiPxJjoeOst2e3xFEgHRf1IcRsZw2O5wi3c sbZBFEubftghBFpdTo2oc4Ub2XZmiZjuYOEr/3oMo36dpn5Pgc1dya/hzVxUrVRr4H2V DTq2XqOxWXDJyM6rI/IcS0Ioz2dfA6JLq7EPnFaq3j/fqoQJNVEvGYMZEVRrjKFug3rJ ltCw== X-Gm-Message-State: AC+VfDyRO6WFc7FoUjC8DN5IyZSrXHH+GuGoKX3yPLZzYGrsNkw/rJXa Dqr0SpJ/O6sZIi9Gp0jjPi9GV6N/UE0ltlTEjT/FqIBgSYipXRhF65K+C1vrri452OcBKvYw1Oi e0Sir6apH8lgLdGocPyuIWGOP X-Received: by 2002:a05:6214:301a:b0:635:da19:a67f with SMTP id ke26-20020a056214301a00b00635da19a67fmr13098204qvb.1.1687989198001; Wed, 28 Jun 2023 14:53:18 -0700 (PDT) X-Received: by 2002:a05:6214:301a:b0:635:da19:a67f with SMTP id ke26-20020a056214301a00b00635da19a67fmr13098180qvb.1.1687989197705; Wed, 28 Jun 2023 14:53:17 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:17 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 4/8] mm/gup: Cleanup next_page handling Date: Wed, 28 Jun 2023 17:53:06 -0400 Message-ID: <20230628215310.73782-5-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1769987039445284270?= X-GMAIL-MSGID: =?utf-8?q?1769987039445284270?= The only path that doesn't use generic "**pages" handling is the gate vma. Make it use the same path, meanwhile tune the next_page label upper to cover "**pages" handling. This prepares for THP handling for "**pages". Reviewed-by: Lorenzo Stoakes Acked-by: David Hildenbrand Signed-off-by: Peter Xu --- mm/gup.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 1e2e23084f3c..5af1be81390b 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1124,7 +1124,7 @@ static long __get_user_pages(struct mm_struct *mm, if (!vma && in_gate_area(mm, start)) { ret = get_gate_page(mm, start & PAGE_MASK, gup_flags, &vma, - pages ? &pages[i] : NULL); + pages ? &page : NULL); if (ret) goto out; ctx.page_mask = 0; @@ -1194,19 +1194,18 @@ static long __get_user_pages(struct mm_struct *mm, ret = PTR_ERR(page); goto out; } - - goto next_page; } else if (IS_ERR(page)) { ret = PTR_ERR(page); goto out; } +next_page: if (pages) { pages[i] = page; flush_anon_page(vma, page, start); flush_dcache_page(page); ctx.page_mask = 0; } -next_page: + page_increm = 1 + (~(start >> PAGE_SHIFT) & ctx.page_mask); if (page_increm > nr_pages) page_increm = nr_pages; From patchwork Wed Jun 28 21:53:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 114083 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp9240342vqr; Wed, 28 Jun 2023 15:12:26 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ54ANPto3Z3ZlJtqExVxM54VPfad5flGVP20oODRLUCACZgUsKU47VXNfAPXjGHc4Uyg2bs X-Received: by 2002:a17:902:d489:b0:1ae:7ba2:1a7e with SMTP id c9-20020a170902d48900b001ae7ba21a7emr42631336plg.6.1687990346084; Wed, 28 Jun 2023 15:12:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687990346; cv=none; d=google.com; s=arc-20160816; b=0yF3z7T3vYQYjEH5BIy627OxxDE0lZHsrCMeTg6DoJEaSp1gpm3O3bumm1PGxiJnEI htFPk8vNSHX5yVlQiEF3TjV4PLCQgOOCTnx/eRlKfEBJPXAfQ+Um9w9ngQfQsvt3Qqp/ nSdr1MwGmagZ357Z5iDemVt0QYwcNZ3FiY6sNHFtUa1x/cfi4tKEyXYPR0Pb5RUSNLsL oAlZTbH39dDoDkDNr+y3wrFxKjURssx1ol7J1gMKBH4YKGYWqRo9kCVg5QznvtUlXkFU 6+wcDpvtLSb70oBPpYx8DzckSi8IctEapn1AeVp1NWV1lgT+BJ7+BuiX/IHGBIWWZN6F /uqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=LErGcSwJladPWljtQDBZEDG1gs3DbaZNQ9dZknt9+SI=; fh=YRm1SqlPzCC/sbB3u8ugRd2MBYnp3vC9/g67nUn+DD4=; b=CVKuQw+5SHAD92K9ZlwBhd3TnGS3r2wCsM5htc1kojj08THVbnedSNZD2Nidjjx0x6 xxuGyxVWKir1lNVcbbqbgO5MlVEkzmANMWBagd3E1S+00orUuj73FA3UgufoI9CWCX+x BZ40QA6rZNCkXRwwxXLygCXEX/gOEjdWTqVO1KnMHWCyr6GA5nkRnsmb7Bzx7/uEkiAg 5yKhMLrP1aULnng1cFsWc4/ypudyHAIVTd2epD8invO7jmnIZgWgz6nkAwOVZr+6/53V N7BNGPIsy+TO2hbYQD/8LUxnkdW4/LQv7XCE0wUwAOtAy2kR8z47ZtwYQHABaX/0/aTw ckHA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bE1pmSAG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l5-20020a654485000000b0055398c3184esi9679448pgq.6.2023.06.28.15.12.13; Wed, 28 Jun 2023 15:12:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bE1pmSAG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232284AbjF1Vzc (ORCPT + 99 others); Wed, 28 Jun 2023 17:55:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232364AbjF1Vyv (ORCPT ); Wed, 28 Jun 2023 17:54:51 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4FD7F211D for ; Wed, 28 Jun 2023 14:53:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989203; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LErGcSwJladPWljtQDBZEDG1gs3DbaZNQ9dZknt9+SI=; b=bE1pmSAGRaHObDruWiFGWoU2a0fcU2xPfyl3oncN6oCiWwKj5aNjSaLYs8vu/QCXmkoDMU XuAbRgG1Lr38UfrRf6rgNWcBGL1AbhwSAH1unYIn/y0/mbsORqW2YK+7ZDVb4E4qN8MGlg zMRLGlppt+4IJVTOR4q7ncFFNSV0VEM= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-644-gJ9xuyYVOUOJFMofO_LFbA-1; Wed, 28 Jun 2023 17:53:19 -0400 X-MC-Unique: gJ9xuyYVOUOJFMofO_LFbA-1 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-635e2618aaeso42496d6.0 for ; Wed, 28 Jun 2023 14:53:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989199; x=1690581199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LErGcSwJladPWljtQDBZEDG1gs3DbaZNQ9dZknt9+SI=; b=cDV/hM+b43d+MbPRxVRdMMTBye2AcJ05aMtildRSxipnD1rl+9KaJ5FsFhFvcCdSY6 GzDYSzZ4t+RXBmpfyWLY9rdZfVz6khzhJLR1Ev37W70ypkU4/NKs1ZXJ3d7WQ6XAJ2PX M0wzC8gdZcyXz8t5uQHmZnbRZOkUcDkwhBlsnYHkJt8JliBae8zEGtMGgWELSB0ihEeb y2zIT3DPCV/+dUrftaamlS1OSOMfDZHQL/qta/H5gSUD+scs2ETWU28dn+n3vb4hwFpd aGM8XBddVWHuCNb142mlBhTWcOVL0gKxPk91xzvAvD/sd5IP88J5esECzn57l6ZTDxJp NAyQ== X-Gm-Message-State: AC+VfDwaJGSOGeCiqpuykHnkNht0NjWisg7doKMM8GBv2P9xuS1bZztL JYBbdiuL9gY1Vd/Wal7KrSAXPLgpqHJZ9qM/fYeJI+8dEbSP5zryfQxMrqLI1B/BQmnZApJ/WZF ZSGapHYUpetYVWYixoHgt9jak X-Received: by 2002:a05:6214:501d:b0:621:65de:f60c with SMTP id jo29-20020a056214501d00b0062165def60cmr6650320qvb.3.1687989199421; Wed, 28 Jun 2023 14:53:19 -0700 (PDT) X-Received: by 2002:a05:6214:501d:b0:621:65de:f60c with SMTP id jo29-20020a056214501d00b0062165def60cmr6650303qvb.3.1687989199149; Wed, 28 Jun 2023 14:53:19 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:18 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 5/8] mm/gup: Accelerate thp gup even for "pages != NULL" Date: Wed, 28 Jun 2023 17:53:07 -0400 Message-ID: <20230628215310.73782-6-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1769986165186122795?= X-GMAIL-MSGID: =?utf-8?q?1769986165186122795?= The acceleration of THP was done with ctx.page_mask, however it'll be ignored if **pages is non-NULL. The old optimization was introduced in 2013 in 240aadeedc4a ("mm: accelerate mm_populate() treatment of THP pages"). It didn't explain why we can't optimize the **pages non-NULL case. It's possible that at that time the major goal was for mm_populate() which should be enough back then. Optimize thp for all cases, by properly looping over each subpage, doing cache flushes, and boost refcounts / pincounts where needed in one go. This can be verified using gup_test below: # chrt -f 1 ./gup_test -m 512 -t -L -n 1024 -r 10 Before: 13992.50 ( +-8.75%) After: 378.50 (+-69.62%) Reviewed-by: Lorenzo Stoakes Signed-off-by: Peter Xu --- mm/gup.c | 51 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 44 insertions(+), 7 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 5af1be81390b..0e2b0ff1143a 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1199,16 +1199,53 @@ static long __get_user_pages(struct mm_struct *mm, goto out; } next_page: - if (pages) { - pages[i] = page; - flush_anon_page(vma, page, start); - flush_dcache_page(page); - ctx.page_mask = 0; - } - page_increm = 1 + (~(start >> PAGE_SHIFT) & ctx.page_mask); if (page_increm > nr_pages) page_increm = nr_pages; + + if (pages) { + struct page *subpage; + unsigned int j; + + /* + * This must be a large folio (and doesn't need to + * be the whole folio; it can be part of it), do + * the refcount work for all the subpages too. + * + * NOTE: here the page may not be the head page + * e.g. when start addr is not thp-size aligned. + * try_grab_folio() should have taken care of tail + * pages. + */ + if (page_increm > 1) { + struct folio *folio; + + /* + * Since we already hold refcount on the + * large folio, this should never fail. + */ + folio = try_grab_folio(page, page_increm - 1, + foll_flags); + if (WARN_ON_ONCE(!folio)) { + /* + * Release the 1st page ref if the + * folio is problematic, fail hard. + */ + gup_put_folio(page_folio(page), 1, + foll_flags); + ret = -EFAULT; + goto out; + } + } + + for (j = 0; j < page_increm; j++) { + subpage = nth_page(page, j); + pages[i + j] = subpage; + flush_anon_page(vma, subpage, start + j * PAGE_SIZE); + flush_dcache_page(subpage); + } + } + i += page_increm; start += page_increm * PAGE_SIZE; nr_pages -= page_increm; From patchwork Wed Jun 28 21:53:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 114088 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp9246103vqr; Wed, 28 Jun 2023 15:24:04 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5ZrV2d3CdJ/Gx0WW8MjCdo3G7dLfyw2M7DaEcLy+pYZC3BvZYHJiW91z8zn7jvlNUAldGW X-Received: by 2002:a17:902:d489:b0:1ae:7ba2:1a7e with SMTP id c9-20020a170902d48900b001ae7ba21a7emr42662947plg.6.1687991044370; Wed, 28 Jun 2023 15:24:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687991044; cv=none; d=google.com; s=arc-20160816; b=CW+4/iecjjOLUiwqRfDUyQAWybzrX/UFF4M1h0LbiOqOvokU6iYftYpW0+Od2293Rf beNZpzMX8uQMzWGMtk9SBGl3JBYV+CyfPrAhGNG1TVyfkI1GZyLEpj8UKJ1jXkdpgVex intQWO0EO9DrOidnHVElYvjDUXcFe1BWPNFjqKli1we4OP2NlLSxTMQjFbNNSJ86JZ20 JpK+yKgXqN+7hCsMwvISkt3XQVxEhdZyokpVHhvVs9IEBWodB7iXnzzc90pyGAJGfhDx F8T43rYRnJfTd6hP5pcxC0iRhJWkAmGM+/SP1PV8Gxf8hRkQ/+3Sc8IH92Jdo7iKg9yg XXog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=f1uXyVFMcTV1oTRkqq3pr4O9sWiRe8mnSq6RPVSEFpA=; fh=YRm1SqlPzCC/sbB3u8ugRd2MBYnp3vC9/g67nUn+DD4=; b=aBokwYlIafi02Snn0iwXC+iLniDmk5BQgJ+og9zlHZgPb1i/iev7ofgfbQmjais/0f KUAKHgn4DtFwz2xSQozxKGgHNZ7rkM4INJs+Hv5jWqhF2fWHcWVNkIGnAmTFURe02avK hBI+Rlw/8b0M/JzwoDw8J9qoBPkltTXZlDm6vo2t3WrMBtbSoziniAl8dgnfpdvkBhuk eDOyQiFs1+Kp4elKakIwWQ6Z0O2x6I2R1j3U1gXzOfm9A9mpZ0wglc44RL4wtbqflfUv 6wItk1RBTJQmiZWKsequllQAnfffi4UI8jr1f+9Oljkw7KW9tTgzJhKNKhyD4o21NbmP idGw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=SIeCXVT0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r9-20020a63ec49000000b0053fc3bc33e4si9772898pgj.795.2023.06.28.15.23.50; Wed, 28 Jun 2023 15:24:04 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=SIeCXVT0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230340AbjF1VzZ (ORCPT + 99 others); Wed, 28 Jun 2023 17:55:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232357AbjF1Vyu (ORCPT ); Wed, 28 Jun 2023 17:54:50 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 786BF2129 for ; Wed, 28 Jun 2023 14:53:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989204; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f1uXyVFMcTV1oTRkqq3pr4O9sWiRe8mnSq6RPVSEFpA=; b=SIeCXVT0iCAceyn7bh1j3lTmzls/+92QkmMJgQyBUplE0AfwSmijPQ4Bdgo5emPgsVdkk2 Q97P4ABvMY24v6nSrm1PVaWjtk+Oo65HkhjHMOUNC/xvnA8dP+g32R9uUa3/uvFoWMuw7s hnEyrB4SBEqE7hINEO6x8va7TmO6Gn8= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-187-n002aZ1hMY60RK_efr0tPQ-1; Wed, 28 Jun 2023 17:53:21 -0400 X-MC-Unique: n002aZ1hMY60RK_efr0tPQ-1 Received: by mail-qv1-f71.google.com with SMTP id 6a1803df08f44-635df023844so9866d6.1 for ; Wed, 28 Jun 2023 14:53:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989201; x=1690581201; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=f1uXyVFMcTV1oTRkqq3pr4O9sWiRe8mnSq6RPVSEFpA=; b=UKlDJYNga3hcmuBChk+RqkrZ+5oM2UD/jrmcbP73Z4ZHKyMrr99Ep8hZVnT/kPvlyf ce1Q9E2pkXH2ZldozUPddxqQuprg5DCmFD+2viP2K+WAapt0yAdFaH2sR7xlx9Eup70O w7n6mBhb36atakSjjMb8TfmXbj0jR7LjK63VRuQoIVim5ITKjB6+GzG05sEF63OB5V3T +X9oJPTswMDlb9WUvSrzdScmjBG+PGGigOWRgUifzjm9LuHNXHePur4f+hGyNObKsxPf RXncGoVksVjQyhxLsuK7XPN76Qc7QWefodsRwGBtU7ii6d30MifvAcfaIU70hOKnrfkC ZVkQ== X-Gm-Message-State: AC+VfDxQedFtMGe09Z95XEpvJUoyTgkIk3RUKGi5JI93mGo2AXGrzvyy r66IjQiWnFrlCzEvPonlCrn9J56M5O2Bu0s5Kc7E1tmv/uNx6s5zsaf3mBAi11J4LasJUa1PZ+h 7yjTyd/ZGq22CHi8k74XBRHMH X-Received: by 2002:a05:6214:c6c:b0:62f:f6ac:abf5 with SMTP id t12-20020a0562140c6c00b0062ff6acabf5mr43014705qvj.5.1687989200984; Wed, 28 Jun 2023 14:53:20 -0700 (PDT) X-Received: by 2002:a05:6214:c6c:b0:62f:f6ac:abf5 with SMTP id t12-20020a0562140c6c00b0062ff6acabf5mr43014693qvj.5.1687989200642; Wed, 28 Jun 2023 14:53:20 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:20 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 6/8] mm/gup: Retire follow_hugetlb_page() Date: Wed, 28 Jun 2023 17:53:08 -0400 Message-ID: <20230628215310.73782-7-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1769986897448246872?= X-GMAIL-MSGID: =?utf-8?q?1769986897448246872?= Now __get_user_pages() should be well prepared to handle thp completely, as long as hugetlb gup requests even without the hugetlb's special path. Time to retire follow_hugetlb_page(). Tweak misc comments to reflect reality of follow_hugetlb_page()'s removal. Acked-by: David Hildenbrand Signed-off-by: Peter Xu --- fs/userfaultfd.c | 2 +- include/linux/hugetlb.h | 12 --- mm/gup.c | 19 ---- mm/hugetlb.c | 224 ---------------------------------------- 4 files changed, 1 insertion(+), 256 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 7cecd49e078b..ae711f1d7a83 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -427,7 +427,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) * * We also don't do userfault handling during * coredumping. hugetlbfs has the special - * follow_hugetlb_page() to skip missing pages in the + * hugetlb_follow_page_mask() to skip missing pages in the * FOLL_DUMP case, anon memory also checks for FOLL_DUMP with * the no_page_table() helper in follow_page_mask(), but the * shmem_vm_ops->fault method is invoked even during diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 9f282f370d96..9bc3c2d71b71 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -133,9 +133,6 @@ int copy_hugetlb_page_range(struct mm_struct *, struct mm_struct *, struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, unsigned int *page_mask); -long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, - struct page **, unsigned long *, unsigned long *, - long, unsigned int, int *); void unmap_hugepage_range(struct vm_area_struct *, unsigned long, unsigned long, struct page *, zap_flags_t); @@ -305,15 +302,6 @@ static inline struct page *hugetlb_follow_page_mask( BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ } -static inline long follow_hugetlb_page(struct mm_struct *mm, - struct vm_area_struct *vma, struct page **pages, - unsigned long *position, unsigned long *nr_pages, - long i, unsigned int flags, int *nonblocking) -{ - BUG(); - return 0; -} - static inline int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *dst_vma, diff --git a/mm/gup.c b/mm/gup.c index 0e2b0ff1143a..a7c294de6ae5 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -775,9 +775,6 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, * Call hugetlb_follow_page_mask for hugetlb vmas as it will use * special hugetlb page table walking code. This eliminates the * need to check for hugetlb entries in the general walking code. - * - * hugetlb_follow_page_mask is only for follow_page() handling here. - * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ if (is_vm_hugetlb_page(vma)) return hugetlb_follow_page_mask(vma, address, flags, @@ -1138,22 +1135,6 @@ static long __get_user_pages(struct mm_struct *mm, ret = check_vma_flags(vma, gup_flags); if (ret) goto out; - - if (is_vm_hugetlb_page(vma)) { - i = follow_hugetlb_page(mm, vma, pages, - &start, &nr_pages, i, - gup_flags, locked); - if (!*locked) { - /* - * We've got a VM_FAULT_RETRY - * and we've lost mmap_lock. - * We must stop here. - */ - BUG_ON(gup_flags & FOLL_NOWAIT); - goto out; - } - continue; - } } retry: /* diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 15e82a8a2b76..2f12da409a19 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5721,7 +5721,6 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* * Return whether there is a pagecache page to back given address within VMA. - * Caller follow_hugetlb_page() holds page_table_lock so we cannot lock_page. */ static bool hugetlbfs_pagecache_present(struct hstate *h, struct vm_area_struct *vma, unsigned long address) @@ -6422,37 +6421,6 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, } #endif /* CONFIG_USERFAULTFD */ -static void record_subpages(struct page *page, struct vm_area_struct *vma, - int refs, struct page **pages) -{ - int nr; - - for (nr = 0; nr < refs; nr++) { - if (likely(pages)) - pages[nr] = nth_page(page, nr); - } -} - -static inline bool __follow_hugetlb_must_fault(struct vm_area_struct *vma, - unsigned int flags, pte_t *pte, - bool *unshare) -{ - pte_t pteval = huge_ptep_get(pte); - - *unshare = false; - if (is_swap_pte(pteval)) - return true; - if (huge_pte_write(pteval)) - return false; - if (flags & FOLL_WRITE) - return true; - if (gup_must_unshare(vma, flags, pte_page(pteval))) { - *unshare = true; - return true; - } - return false; -} - struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, unsigned int *page_mask) @@ -6524,198 +6492,6 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, return page; } -long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, - struct page **pages, unsigned long *position, - unsigned long *nr_pages, long i, unsigned int flags, - int *locked) -{ - unsigned long pfn_offset; - unsigned long vaddr = *position; - unsigned long remainder = *nr_pages; - struct hstate *h = hstate_vma(vma); - int err = -EFAULT, refs; - - while (vaddr < vma->vm_end && remainder) { - pte_t *pte; - spinlock_t *ptl = NULL; - bool unshare = false; - int absent; - struct page *page; - - /* - * If we have a pending SIGKILL, don't keep faulting pages and - * potentially allocating memory. - */ - if (fatal_signal_pending(current)) { - remainder = 0; - break; - } - - hugetlb_vma_lock_read(vma); - /* - * Some archs (sparc64, sh*) have multiple pte_ts to - * each hugepage. We have to make sure we get the - * first, for the page indexing below to work. - * - * Note that page table lock is not held when pte is null. - */ - pte = hugetlb_walk(vma, vaddr & huge_page_mask(h), - huge_page_size(h)); - if (pte) - ptl = huge_pte_lock(h, mm, pte); - absent = !pte || huge_pte_none(huge_ptep_get(pte)); - - /* - * When coredumping, it suits get_dump_page if we just return - * an error where there's an empty slot with no huge pagecache - * to back it. This way, we avoid allocating a hugepage, and - * the sparse dumpfile avoids allocating disk blocks, but its - * huge holes still show up with zeroes where they need to be. - */ - if (absent && (flags & FOLL_DUMP) && - !hugetlbfs_pagecache_present(h, vma, vaddr)) { - if (pte) - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - remainder = 0; - break; - } - - /* - * We need call hugetlb_fault for both hugepages under migration - * (in which case hugetlb_fault waits for the migration,) and - * hwpoisoned hugepages (in which case we need to prevent the - * caller from accessing to them.) In order to do this, we use - * here is_swap_pte instead of is_hugetlb_entry_migration and - * is_hugetlb_entry_hwpoisoned. This is because it simply covers - * both cases, and because we can't follow correct pages - * directly from any kind of swap entries. - */ - if (absent || - __follow_hugetlb_must_fault(vma, flags, pte, &unshare)) { - vm_fault_t ret; - unsigned int fault_flags = 0; - - if (pte) - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - - if (flags & FOLL_WRITE) - fault_flags |= FAULT_FLAG_WRITE; - else if (unshare) - fault_flags |= FAULT_FLAG_UNSHARE; - if (locked) { - fault_flags |= FAULT_FLAG_ALLOW_RETRY | - FAULT_FLAG_KILLABLE; - if (flags & FOLL_INTERRUPTIBLE) - fault_flags |= FAULT_FLAG_INTERRUPTIBLE; - } - if (flags & FOLL_NOWAIT) - fault_flags |= FAULT_FLAG_ALLOW_RETRY | - FAULT_FLAG_RETRY_NOWAIT; - if (flags & FOLL_TRIED) { - /* - * Note: FAULT_FLAG_ALLOW_RETRY and - * FAULT_FLAG_TRIED can co-exist - */ - fault_flags |= FAULT_FLAG_TRIED; - } - ret = hugetlb_fault(mm, vma, vaddr, fault_flags); - if (ret & VM_FAULT_ERROR) { - err = vm_fault_to_errno(ret, flags); - remainder = 0; - break; - } - if (ret & VM_FAULT_RETRY) { - if (locked && - !(fault_flags & FAULT_FLAG_RETRY_NOWAIT)) - *locked = 0; - *nr_pages = 0; - /* - * VM_FAULT_RETRY must not return an - * error, it will return zero - * instead. - * - * No need to update "position" as the - * caller will not check it after - * *nr_pages is set to 0. - */ - return i; - } - continue; - } - - pfn_offset = (vaddr & ~huge_page_mask(h)) >> PAGE_SHIFT; - page = pte_page(huge_ptep_get(pte)); - - VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && - !PageAnonExclusive(page), page); - - /* - * If subpage information not requested, update counters - * and skip the same_page loop below. - */ - if (!pages && !pfn_offset && - (vaddr + huge_page_size(h) < vma->vm_end) && - (remainder >= pages_per_huge_page(h))) { - vaddr += huge_page_size(h); - remainder -= pages_per_huge_page(h); - i += pages_per_huge_page(h); - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - continue; - } - - /* vaddr may not be aligned to PAGE_SIZE */ - refs = min3(pages_per_huge_page(h) - pfn_offset, remainder, - (vma->vm_end - ALIGN_DOWN(vaddr, PAGE_SIZE)) >> PAGE_SHIFT); - - if (pages) - record_subpages(nth_page(page, pfn_offset), - vma, refs, - likely(pages) ? pages + i : NULL); - - if (pages) { - /* - * try_grab_folio() should always succeed here, - * because: a) we hold the ptl lock, and b) we've just - * checked that the huge page is present in the page - * tables. If the huge page is present, then the tail - * pages must also be present. The ptl prevents the - * head page and tail pages from being rearranged in - * any way. As this is hugetlb, the pages will never - * be p2pdma or not longterm pinable. So this page - * must be available at this point, unless the page - * refcount overflowed: - */ - if (WARN_ON_ONCE(!try_grab_folio(pages[i], refs, - flags))) { - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - remainder = 0; - err = -ENOMEM; - break; - } - } - - vaddr += (refs << PAGE_SHIFT); - remainder -= refs; - i += refs; - - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - } - *nr_pages = remainder; - /* - * setting position is actually required only if remainder is - * not zero but it's faster not to add a "if (remainder)" - * branch. - */ - *position = vaddr; - - return i ? i : err; -} - long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags) From patchwork Wed Jun 28 21:53:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 114086 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp9241008vqr; Wed, 28 Jun 2023 15:13:41 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6J9nsCT+7oSW7PeLyRYykNU9vpCvHGFfDB6rwXHKe/gCf2+hDFnbvloUuT1LfgUS/POz0M X-Received: by 2002:a05:6402:274b:b0:51a:4d46:4026 with SMTP id z11-20020a056402274b00b0051a4d464026mr26383908edd.0.1687990420892; Wed, 28 Jun 2023 15:13:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687990420; cv=none; d=google.com; s=arc-20160816; b=tHyaDn2byhiiLsJ7SumDsMRZRIGB32Iv6l5dAB22BnZiaUdS1txCEqqgequfJQ04/H U0F3w8doiQ87Zje+QnANMqv8Fxno92+l4b8+WaUYFIkJJSVC1wHP39iWF245jWaA3xlD 3FVA2M5tNH5aeEXAM/nth0vu0dGlTc0vU8QlHhjuAxa3yYng18i5TYxKByhl7WH7wYNX DoCaW050DQKjoSaJsXTmGomldt59GUvN6A+BTyR2RWWkt51YhQfjv/Q69sYIod0q57by Ahcy4Wb98zPup11Om9xozwFEptuy+D+quSGv4KmGrvM1vQQWmqq2Toukch9PVE+TDHrE +RmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=8HfHdG/KQ5Uxfq5f0w/MDNpC44bP5ckORXtaAeg9+TI=; fh=YRm1SqlPzCC/sbB3u8ugRd2MBYnp3vC9/g67nUn+DD4=; b=u8Jp3vJSvgR1up/PaNl8oRmhfahEDOCBUSnDNpTctgL4HcpVyUNhkAaXKcE+9VD1FN Us6OhzN50MG2bdT/xJFvlbyBfa0+1ycpUniaIXVktc2BY6a2w0VmYS/j35ibIjX4PpoW acvmj0tJ/XNF2jtWzoJLqMHDMNX9fhC3E87hBPOR0nFh7aOuN/ak9kd/01U9X4KwZMZf /N/WIVtvGDd77QXlxWDwCDY91LzPGM6b8ftq080DaKmmsb2eUPSoQON51sRjoGxZL5EF tzPTlB66A+CRPqpu9Xp/M9oEV3jDRyKYiaEtFLmbu3t+Yh6k85zCL0lhggTxD32DllLF vFlA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZXrUgH2n; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a23-20020aa7d757000000b0051d96435b79si4168338eds.600.2023.06.28.15.13.16; Wed, 28 Jun 2023 15:13:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZXrUgH2n; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232026AbjF1Vz2 (ORCPT + 99 others); Wed, 28 Jun 2023 17:55:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37842 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232369AbjF1Vyv (ORCPT ); Wed, 28 Jun 2023 17:54:51 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E97492127 for ; Wed, 28 Jun 2023 14:53:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989204; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8HfHdG/KQ5Uxfq5f0w/MDNpC44bP5ckORXtaAeg9+TI=; b=ZXrUgH2nVSv0WUOunmRgqKL8X0seYS0+KTaECkZc8pfdt2tM/4mkOI0VGWm33gBhmhEUBJ HxD5NBJ+wNy81U//o49gLi8MAIPDLc59BOPC9JRG958q3y6SsitFnueHSCrSbmmRY7AHNl Ze9ZGr81znxkjQO4FUyiFH1HvNY7+sg= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-584-kSXfzjzJNNu6lqB3GXRxFA-1; Wed, 28 Jun 2023 17:53:22 -0400 X-MC-Unique: kSXfzjzJNNu6lqB3GXRxFA-1 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-635f8ae5f88so39436d6.0 for ; Wed, 28 Jun 2023 14:53:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989202; x=1690581202; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8HfHdG/KQ5Uxfq5f0w/MDNpC44bP5ckORXtaAeg9+TI=; b=GoDaY9wQRXOwQBbqFfNXAW8wQtt/fuanxzhap9543mwplBHrKLNQ5up4jvJXmhSyCG mgdQ+oWR2KBjdewXyXLVjjMZ1EcE7rk+vEHazpwD3dteLNZji/l4CWldQ1nsa5jbvqEB D4Zhh83r0FADAia/6qp4/X7Ei+hIZEfXkFYZUL0l++HGM/QPDmRcBIul1nRooEzhC3ma 0DxFScfvSCRveab7eZOVQwCoxINMii6j/+S9oZXoO+XzHd4ApbSB5vz++wpfrSQuMRAV fXHH1uBgEH2EjEzceqDAnp1c8BOXpdYazy6OudpMWAaw+SHs21qOFZhTRGBnk7a3HOGO xW3A== X-Gm-Message-State: AC+VfDxn4EqsJZ84td0o0bfpPXy652+wV5sEkLAqKi/aOwnNYv24wRAB Y1LxsoWazhZrq/TdXu2N4RaQzGuYKkPcV+I2J6w7bHZ2Yh3jJVkGrneP23HUq7tVOlhJjoW1dQt wLZrpyXImamUXtaiNNOAL1dVx X-Received: by 2002:a05:6214:260e:b0:626:2305:6073 with SMTP id gu14-20020a056214260e00b0062623056073mr6706614qvb.4.1687989202308; Wed, 28 Jun 2023 14:53:22 -0700 (PDT) X-Received: by 2002:a05:6214:260e:b0:626:2305:6073 with SMTP id gu14-20020a056214260e00b0062623056073mr6706604qvb.4.1687989202071; Wed, 28 Jun 2023 14:53:22 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:21 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 7/8] selftests/mm: Add -a to run_vmtests.sh Date: Wed, 28 Jun 2023 17:53:09 -0400 Message-ID: <20230628215310.73782-8-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1769986243324236738?= X-GMAIL-MSGID: =?utf-8?q?1769986243324236738?= Allows to specify optional tests in run_vmtests.sh, where we can run time consuming test matrix only when user specified "-a". Acked-by: David Hildenbrand Signed-off-by: Peter Xu --- tools/testing/selftests/mm/run_vmtests.sh | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh index 3f26f6e15b2a..824e651f62f4 100644 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -12,11 +12,14 @@ exitcode=0 usage() { cat <"] +usage: ${BASH_SOURCE[0]:-$0} [ options ] + + -a: run all tests, including extra ones -t: specify specific categories to tests to run -h: display this message -The default behavior is to run all tests. +The default behavior is to run required tests only. If -a is specified, +will run all tests. Alternatively, specific groups tests can be run by passing a string to the -t argument containing one or more of the following categories @@ -60,9 +63,11 @@ EOF exit 0 } +RUN_ALL=false -while getopts "ht:" OPT; do +while getopts "aht:" OPT; do case ${OPT} in + "a") RUN_ALL=true ;; "h") usage ;; "t") VM_SELFTEST_ITEMS=${OPTARG} ;; esac From patchwork Wed Jun 28 21:53:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 114085 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp9240627vqr; Wed, 28 Jun 2023 15:13:03 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ58eVcHlvJTSQZHLa37uYmOVkJEFIlPXSeVzT/BFxVM7djyCCCQUOknI18OgLrZZK6hcz/P X-Received: by 2002:a05:6a21:999f:b0:116:696f:1dd1 with SMTP id ve31-20020a056a21999f00b00116696f1dd1mr1112745pzb.4.1687990382906; Wed, 28 Jun 2023 15:13:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687990382; cv=none; d=google.com; s=arc-20160816; b=wBU9cnX4/2P6mgj+VQECAKtSjjk8u31hnyN17jaGEBjQ4kKxFCYop8qtgCDC5jgPSz a4081NAWXuPNsp3M+EGv3nKIqcGSJ474LSD15iISAi+ZahuqZuEf75iTQUnpV/H7HJ9V DNGWWQ19UV18UFoEHIKGfeMDYHv5WVVsIJLJkHQ/2Tcwb3Ez4pd1piiN0pMRNuhAiL6+ NoRVR62Ph6JoB3APzHoZTqGUnThNbuhJUzftm0/md+6huoTJ7CJ5B90qQrjiiVc6Ksup o3FgexKzZcwTj30jJi6q+fmTQ7YyyV+YcA52E9znAAR/ymL3q4yEZ97iqCNeYSAgYGQO hNfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=8+0w023mz5OfdmLI6Zdp3hWUHid6rdaL/9V50JZhDTo=; fh=YRm1SqlPzCC/sbB3u8ugRd2MBYnp3vC9/g67nUn+DD4=; b=T5V3syEVGdUemhWtm0qNfXPwEWOCeIxoZTUsecnt5L4seKBf8YkB9vLefSNsYd6pcg GobhgQEqhimmduB3TOnLWGKVyhZkQOdGRCcyUB/DfqqgIq7X8R7A8VVhd/zhs1lAQ+ad 0Cw5UphSsbIl9B3Mo2KCdtpKFUNUJe9PiEKNOPxoYRnS9Sq1i34Plj++XhA84wKTL73y NflvYVPy7+/ld3E84cEsAd1k+5NHjmPVibA5rbk2iBYglLN48XBC/0toCkTOGzQCY4lU EX3E4VwWUUtNgLtlojFpI2QZ/LFtAWX7iv/R+TssgVS1dtafTTTWl4xaKUkEVHmR76oX ZVDQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=WAVyg2vO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l5-20020a654485000000b0055398c3184esi9679448pgq.6.2023.06.28.15.12.48; Wed, 28 Jun 2023 15:13:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=WAVyg2vO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231134AbjF1VzW (ORCPT + 99 others); Wed, 28 Jun 2023 17:55:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232356AbjF1Vyu (ORCPT ); Wed, 28 Jun 2023 17:54:50 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6BE772132 for ; Wed, 28 Jun 2023 14:53:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687989205; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8+0w023mz5OfdmLI6Zdp3hWUHid6rdaL/9V50JZhDTo=; b=WAVyg2vOBMSZEI73gnmbY6XMoO98SXLZrWWEZIfILHHaF13dsnb2VkEfu2emXaQTBjyjLy e6ZnDWlGk+bECAub3yQAAQLtiqaYA2MZNs+klTvQlbMywmFNDNLzqgcZW/HaK8lQrt2LFk HCRRIRs0PcCtgDKFzPEqb5PqTh9BK7I= Received: from mail-vs1-f71.google.com (mail-vs1-f71.google.com [209.85.217.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-280-AIS6DH5mN5Gj7gPGGoNDYA-1; Wed, 28 Jun 2023 17:53:24 -0400 X-MC-Unique: AIS6DH5mN5Gj7gPGGoNDYA-1 Received: by mail-vs1-f71.google.com with SMTP id ada2fe7eead31-440bb217ba2so3077137.0 for ; Wed, 28 Jun 2023 14:53:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687989203; x=1690581203; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8+0w023mz5OfdmLI6Zdp3hWUHid6rdaL/9V50JZhDTo=; b=WChpdnOkd86hXPmfOcGe22/C95IcPqfROSj/z5ENpEl6Kcc/tqfO6N7gbXfoLFH4wu 7pkpjrMbZx2mqn4oqltvbklOxE5dvZI5y32Cm2iprvrbHQt3JDyXQJCQL/xHM9N0hCAv FF1WIT5IwFiIdsasrBz9HHKRIyWUXvfAkYkR1gBhp0YnvAawXY997PTuBSkdTq6K1yxV c9nmL22WY9QT6if75Cxfsss8Hng3T6O7jrdaU/kxS8TVbTy4TCNQMVT8dgFOV2xQaRzd dcDseFer8PuQmLXkSwP4s6Pe4UKIjpbJWQMerSovw4AAgug6hVD5utq6wIjiIqO+zwjC X93Q== X-Gm-Message-State: AC+VfDxZcwiCvY1hGwC6Mv1pXJSG2+a+FoL9Ovbs7iPnuqhLeTnEhxzr sdjxDPdLt6L43hIzgKfdcUdMZGZ0S4ycHh1PGNeZXKJw9ETNTrsiO2+Yjq89ZL62PrY+6h62C4C S0STajzGgTLoq0WKHZ0yMUO/F X-Received: by 2002:a05:6102:3e16:b0:440:9794:fefa with SMTP id j22-20020a0561023e1600b004409794fefamr960328vsv.0.1687989203594; Wed, 28 Jun 2023 14:53:23 -0700 (PDT) X-Received: by 2002:a05:6102:3e16:b0:440:9794:fefa with SMTP id j22-20020a0561023e1600b004409794fefamr960312vsv.0.1687989203304; Wed, 28 Jun 2023 14:53:23 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id p3-20020a0cfac3000000b00631fea4d5bcsm6277797qvo.95.2023.06.28.14.53.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Jun 2023 14:53:23 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , "Kirill A . Shutemov" , Andrew Morton , Andrea Arcangeli , Mike Rapoport , John Hubbard , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Yang Shi , James Houghton , Jason Gunthorpe , Lorenzo Stoakes , Hugh Dickins , peterx@redhat.com Subject: [PATCH v4 8/8] selftests/mm: Add gup test matrix in run_vmtests.sh Date: Wed, 28 Jun 2023 17:53:10 -0400 Message-ID: <20230628215310.73782-9-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230628215310.73782-1-peterx@redhat.com> References: <20230628215310.73782-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1769986203634432693?= X-GMAIL-MSGID: =?utf-8?q?1769986203634432693?= Add a matrix for testing gup based on the current gup_test. Only run the matrix when -a is specified because it's a bit slow. It covers: - Different types of huge pages: thp, hugetlb, or no huge page - Permissions: Write / Read-only - Fast-gup, with/without - Types of the GUP: pin / gup / longterm pins - Shared / Private memories - GUP size: 1 / 512 / random page sizes Acked-by: David Hildenbrand Signed-off-by: Peter Xu --- tools/testing/selftests/mm/run_vmtests.sh | 37 ++++++++++++++++++++--- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh index 824e651f62f4..9666c0c171ab 100644 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -90,6 +90,30 @@ test_selected() { fi } +run_gup_matrix() { + # -t: thp=on, -T: thp=off, -H: hugetlb=on + local hugetlb_mb=$(( needmem_KB / 1024 )) + + for huge in -t -T "-H -m $hugetlb_mb"; do + # -u: gup-fast, -U: gup-basic, -a: pin-fast, -b: pin-basic, -L: pin-longterm + for test_cmd in -u -U -a -b -L; do + # -w: write=1, -W: write=0 + for write in -w -W; do + # -S: shared + for share in -S " "; do + # -n: How many pages to fetch together? 512 is special + # because it's default thp size (or 2M on x86), 123 to + # just test partial gup when hit a huge in whatever form + for num in "-n 1" "-n 512" "-n 123"; do + CATEGORY="gup_test" run_test ./gup_test \ + $huge $test_cmd $write $share $num + done + done + done + done + done +} + # get huge pagesize and freepages from /proc/meminfo while read -r name size unit; do if [ "$name" = "HugePages_Free:" ]; then @@ -194,13 +218,16 @@ fi CATEGORY="mmap" run_test ./map_fixed_noreplace -# get_user_pages_fast() benchmark -CATEGORY="gup_test" run_test ./gup_test -u -# pin_user_pages_fast() benchmark -CATEGORY="gup_test" run_test ./gup_test -a +if $RUN_ALL; then + run_gup_matrix +else + # get_user_pages_fast() benchmark + CATEGORY="gup_test" run_test ./gup_test -u + # pin_user_pages_fast() benchmark + CATEGORY="gup_test" run_test ./gup_test -a +fi # Dump pages 0, 19, and 4096, using pin_user_pages: CATEGORY="gup_test" run_test ./gup_test -ct -F 0x1 0 19 0x1000 - CATEGORY="gup_test" run_test ./gup_longterm CATEGORY="userfaultfd" run_test ./uffd-unit-tests