From patchwork Thu May 25 15:51:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 99096 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp483135vqr; Thu, 25 May 2023 09:07:10 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7hvKg1eLCgXTqnXCQefZJRxC0oYqekcarZJv2y3fFhKi4Ia/wmZtizZjFdNiJWh3ztVR/G X-Received: by 2002:a05:6a20:7d86:b0:10e:d90f:35d5 with SMTP id v6-20020a056a207d8600b0010ed90f35d5mr5583584pzj.51.1685030830599; Thu, 25 May 2023 09:07:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685030830; cv=none; d=google.com; s=arc-20160816; b=BkKesJAkYbp3iENSuxES2enOGiVBFYLn9sbSdT3UqUFDhcj6vM0qDobOzkCMwpFaxO DdR6gyHaB6LtEuaG6YDsofk1Q0H5QO1qxzg9woH+5pEBp5VbGQxEdYF/WvyJ8vQanY4M W1h+1MMYnC2TgmHqdkDj5q5jfJtx+1nG7ktHk2NGSFe6mQu2Get5QAu7l9Fb52MLa/M5 d3RpWliDEbQuNOdHio8HiiXkkWpZOoAYNFn+O9//Ef+PZ9X3AeuXI3JZOnDpNDeq1ehD KTRLN0eJkS48Me3CVrnlnr0fk7Ajf2ykZ8vKmfrh6cQUJdJYpu4Ixn9ZwwlfKDtA8j3p Uw7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=VN2ZrpZ7x9fEyIq82GnRy+FAZ1eh7E2WAh2aAKifPL4=; b=DCws5h9hGmyGLDXaDP4kyyokwuQzpybQt0DZf/nFRDMXhaH/NryiPPEcUtAjwCcJLl qGtIXDJgN5tvpyg3I2S8y07rTQbUhhmo5a+9N7GcSyf9X3KH1caYbqhy+VZM24MzhTdA B7g0mdnVrntyCTKUTDqBou5JiTdkz7g7nk+Cf1BJx3v1yVSosNyDcYE3FInliE49qwrC YQhUyCOnONjUJvCv6yxrOc87+dkiWxW+qVpsvDyoFGJcXFoTkaiOM+WMYEka8gLPf/Ha cx2T5cvcTd2CP0BYNDs0f2tYHJ6DeI5f1Q5nnkMZqUeyd1leddg6J8SiBAH2S/u6IHJc Q6PA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JBd9IpDH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f6-20020a637546000000b0053427b6841dsi58698pgn.337.2023.05.25.09.06.54; Thu, 25 May 2023 09:07:10 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JBd9IpDH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241726AbjEYPwV (ORCPT + 99 others); Thu, 25 May 2023 11:52:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41406 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241592AbjEYPwG (ORCPT ); Thu, 25 May 2023 11:52:06 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 85AFB197 for ; Thu, 25 May 2023 08:51:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685029876; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VN2ZrpZ7x9fEyIq82GnRy+FAZ1eh7E2WAh2aAKifPL4=; b=JBd9IpDHLV+EuM1hF8Dgyn4tW0k1HXqy/FmZa7cTvJkRauIcaYkNTFhx2Gw1fjmxYKZtcW AQWKDQi31wEMxOz0SjwOf2X2SxvyGYidUPlgEbNraQF/KdFw1mfR1DAbXQpNLnkcjovie3 hA9nJJMCVeSXHBFMmGWtoB4XZ445yh0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-554-FDn4oHLSPcWua6K8xvmmRA-1; Thu, 25 May 2023 11:51:11 -0400 X-MC-Unique: FDn4oHLSPcWua6K8xvmmRA-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9B8FF85A5AA; Thu, 25 May 2023 15:51:10 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3BEA6492B0A; Thu, 25 May 2023 15:51:08 +0000 (UTC) From: David Howells To: Christoph Hellwig , David Hildenbrand Cc: David Howells , Jens Axboe , Al Viro , Matthew Wilcox , Jan Kara , Jeff Layton , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , Christian Brauner , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton Subject: [RFC PATCH 1/3] mm: Don't pin ZERO_PAGE in pin_user_pages() Date: Thu, 25 May 2023 16:51:00 +0100 Message-Id: <20230525155102.87353-2-dhowells@redhat.com> In-Reply-To: <20230525155102.87353-1-dhowells@redhat.com> References: <20230525155102.87353-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766882888017128645?= X-GMAIL-MSGID: =?utf-8?q?1766882888017128645?= Make pin_user_pages*() leave the ZERO_PAGE unpinned if it extracts a pointer to it from the page tables and make unpin_user_page*() correspondingly ignore the ZERO_PAGE when unpinning. We don't want to risk overrunning the zero page's refcount as we're only allowed ~2 million pins on it - something that userspace can conceivably trigger. Signed-off-by: David Howells cc: Christoph Hellwig cc: David Hildenbrand cc: Andrew Morton cc: Jens Axboe cc: Al Viro cc: Matthew Wilcox cc: Jan Kara cc: Jeff Layton cc: Jason Gunthorpe cc: Logan Gunthorpe cc: Hillf Danton cc: Christian Brauner cc: Linus Torvalds cc: linux-fsdevel@vger.kernel.org cc: linux-block@vger.kernel.org cc: linux-kernel@vger.kernel.org cc: linux-mm@kvack.org --- mm/gup.c | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/mm/gup.c b/mm/gup.c index bbe416236593..d2662aa8cf01 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -51,7 +51,8 @@ static inline void sanity_check_pinned_pages(struct page **pages, struct page *page = *pages; struct folio *folio = page_folio(page); - if (!folio_test_anon(folio)) + if (page == ZERO_PAGE(0) || + !folio_test_anon(folio)) continue; if (!folio_test_large(folio) || folio_test_hugetlb(folio)) VM_BUG_ON_PAGE(!PageAnonExclusive(&folio->page), page); @@ -131,6 +132,13 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags) else if (flags & FOLL_PIN) { struct folio *folio; + /* + * Don't take a pin on the zero page - it's not going anywhere + * and it is used in a *lot* of places. + */ + if (page == ZERO_PAGE(0)) + return page_folio(ZERO_PAGE(0)); + /* * Can't do FOLL_LONGTERM + FOLL_PIN gup fast path if not in a * right zone, so fail and let the caller fall back to the slow @@ -180,6 +188,8 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags) static void gup_put_folio(struct folio *folio, int refs, unsigned int flags) { if (flags & FOLL_PIN) { + if (folio == page_folio(ZERO_PAGE(0))) + return; node_stat_mod_folio(folio, NR_FOLL_PIN_RELEASED, refs); if (folio_test_large(folio)) atomic_sub(refs, &folio->_pincount); @@ -224,6 +234,13 @@ int __must_check try_grab_page(struct page *page, unsigned int flags) if (flags & FOLL_GET) folio_ref_inc(folio); else if (flags & FOLL_PIN) { + /* + * Don't take a pin on the zero page - it's not going anywhere + * and it is used in a *lot* of places. + */ + if (page == ZERO_PAGE(0)) + return 0; + /* * Similar to try_grab_folio(): be sure to *also* * increment the normal page refcount field at least once, @@ -3079,6 +3096,9 @@ EXPORT_SYMBOL_GPL(get_user_pages_fast); * * FOLL_PIN means that the pages must be released via unpin_user_page(). Please * see Documentation/core-api/pin_user_pages.rst for further details. + * + * Note that if the zero_page is amongst the returned pages, it will not have + * pins in it and unpin_user_page() will not remove pins from it. */ int pin_user_pages_fast(unsigned long start, int nr_pages, unsigned int gup_flags, struct page **pages) @@ -3161,6 +3181,9 @@ EXPORT_SYMBOL(pin_user_pages); * pin_user_pages_unlocked() is the FOLL_PIN variant of * get_user_pages_unlocked(). Behavior is the same, except that this one sets * FOLL_PIN and rejects FOLL_GET. + * + * Note that if the zero_page is amongst the returned pages, it will not have + * pins in it and unpin_user_page() will not remove pins from it. */ long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages, struct page **pages, unsigned int gup_flags) From patchwork Thu May 25 15:51:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 99097 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp483156vqr; Thu, 25 May 2023 09:07:12 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ63PYFgSQLmpXLqwiCY+gtVN3HX99qaNj5fHl9Ax0E3P+JdTYWSiFkThnRPjjvM9A82osq4 X-Received: by 2002:a05:6a00:2d8c:b0:64d:1215:a0f5 with SMTP id fb12-20020a056a002d8c00b0064d1215a0f5mr9419008pfb.34.1685030831767; Thu, 25 May 2023 09:07:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685030831; cv=none; d=google.com; s=arc-20160816; b=q+/j6pdsmwj5BuTPfGlbVY+yYoh4ZKUHqUlfswd+W+ncMKk2XTfN2AeP56kJAlwej3 CJgGzfJmGuTbGB6KN8iLkZa353TwsdYHIOptL+0gmibUNhASb2iYfE5n0U2J6mACp8xf QzUxHxk9Di2gjwLdefgM83YC1FDWKFuRItX73XkbRxR04kp3LuQnblc2kOgowE7K2mIB k4e/ggE0BHbHC6vJz7TCB5WBUuL/x2bfoSk9PcEqjvL52IBMnICRqMaE+2KKx4ObSUqq E7F7UyTV7M9fxsAaRASFIZ+92SKa8+Vj4SIktHBjbTH5RjNpaBZ7NY54FutseLo6o8yM Qb7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=BVMkJArYFah5gI2QNFMxWzaYnawUBl8R91cCRWuGcSI=; b=UNHef73NxuwXgaaOG7togw4QxJUergdExFtIDv5FX1YUkYwUQzyT18tlq8rf0qUpUl ucHomYiK85IGWBwkiIZHm3QeYN4d6hqDWq1XFKZxdxKi6CQnnAa0CEd2GwF382ilO+Be +MO3ydPPLQuP0fVRT/JdqgbVekxKV/fk6cGE1ux1pWXT/qlXilwvAGXTVJyq1yWQ137l ZVIjV+nMqApGiTMAj/ti4ksAIJDLz3clI8DN5pXRb/vTXjmuFY8IJS449gOHS7pDfLcs fuCFvSKgNyLEcqd3DjtFn61gcoyxAf5up7k5Xzfnayftjr4IOM8mvJ3k8aCZfj79EiQN aZ6w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="Fp08mMQ/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y5-20020aa79425000000b0063b82c0941fsi1853578pfo.36.2023.05.25.09.06.53; Thu, 25 May 2023 09:07:11 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="Fp08mMQ/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241660AbjEYPwS (ORCPT + 99 others); Thu, 25 May 2023 11:52:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241601AbjEYPwH (ORCPT ); Thu, 25 May 2023 11:52:07 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0BD541A1 for ; Thu, 25 May 2023 08:51:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685029878; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BVMkJArYFah5gI2QNFMxWzaYnawUBl8R91cCRWuGcSI=; b=Fp08mMQ/g8XkuMXTSaMpNU1r3oiMD+mK4fQUkVIJM2kKjlr0B9/sPlTeaIx1XFvf+8d/fN GKO/N1Wo/J8t38jbYDdBaadqz4OvYjQ3HYsSgFC8pQ5bLB5NCAr291dp0kkESHodSmAMCn IC3/szzm/Fy2teW2aPSdJosFja+hMFo= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-414-ellFKLpmOhqtIg-dAldJeQ-1; Thu, 25 May 2023 11:51:14 -0400 X-MC-Unique: ellFKLpmOhqtIg-dAldJeQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C448238294B6; Thu, 25 May 2023 15:51:13 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6EE15407DEC3; Thu, 25 May 2023 15:51:11 +0000 (UTC) From: David Howells To: Christoph Hellwig , David Hildenbrand Cc: David Howells , Jens Axboe , Al Viro , Matthew Wilcox , Jan Kara , Jeff Layton , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , Christian Brauner , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton Subject: [RFC PATCH 2/3] mm: Provide a function to get an additional pin on a page Date: Thu, 25 May 2023 16:51:01 +0100 Message-Id: <20230525155102.87353-3-dhowells@redhat.com> In-Reply-To: <20230525155102.87353-1-dhowells@redhat.com> References: <20230525155102.87353-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766882889525727329?= X-GMAIL-MSGID: =?utf-8?q?1766882889525727329?= Provide a function to get an additional pin on a page that we already have a pin on. This will be used in fs/direct-io.c when dispatching multiple bios to a page we've extracted from a user-backed iter rather than redoing the extraction. Signed-off-by: David Howells cc: Christoph Hellwig cc: David Hildenbrand cc: Andrew Morton cc: Jens Axboe cc: Al Viro cc: Matthew Wilcox cc: Jan Kara cc: Jeff Layton cc: Jason Gunthorpe cc: Logan Gunthorpe cc: Hillf Danton cc: Christian Brauner cc: Linus Torvalds cc: linux-fsdevel@vger.kernel.org cc: linux-block@vger.kernel.org cc: linux-kernel@vger.kernel.org cc: linux-mm@kvack.org --- include/linux/mm.h | 1 + mm/gup.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 27ce77080c79..931b75dae7ff 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2383,6 +2383,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages, unsigned int gup_flags, struct page **pages); int pin_user_pages_fast(unsigned long start, int nr_pages, unsigned int gup_flags, struct page **pages); +void page_get_additional_pin(struct page *page); int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc); int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc, diff --git a/mm/gup.c b/mm/gup.c index d2662aa8cf01..b1e55847ca13 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -275,6 +275,35 @@ void unpin_user_page(struct page *page) } EXPORT_SYMBOL(unpin_user_page); +/** + * page_get_additional_pin - Try to get an additional pin on a pinned page + * @page: The page to be pinned + * + * Get an additional pin on a page we already have a pin on. Makes no change + * if the page is the zero_page. + */ +void page_get_additional_pin(struct page *page) +{ + struct folio *folio = page_folio(page); + + if (page == ZERO_PAGE(0)) + return; + + /* + * Similar to try_grab_folio(): be sure to *also* increment the normal + * page refcount field at least once, so that the page really is + * pinned. + */ + if (folio_test_large(folio)) { + WARN_ON_ONCE(atomic_read(&folio->_pincount) < 1); + folio_ref_add(folio, 1); + atomic_add(1, &folio->_pincount); + } else { + WARN_ON_ONCE(folio_ref_count(folio) < GUP_PIN_COUNTING_BIAS); + folio_ref_add(folio, GUP_PIN_COUNTING_BIAS); + } +} + static inline struct folio *gup_folio_range_next(struct page *start, unsigned long npages, unsigned long i, unsigned int *ntails) { From patchwork Thu May 25 15:51:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 99089 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp477330vqr; Thu, 25 May 2023 09:00:55 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5XAl9Ig4Xx2gsp9gX+M2JxPrXmELiuByBxpuCOQKX0Jb0I4H6AvyaICrD6CO2BnBztXaXm X-Received: by 2002:a17:902:b902:b0:1ab:7fb:aac1 with SMTP id bf2-20020a170902b90200b001ab07fbaac1mr1988696plb.24.1685030455440; Thu, 25 May 2023 09:00:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685030455; cv=none; d=google.com; s=arc-20160816; b=D8nwIlO24QPOWmyFjmOc17mNCe2Z2wVwQrQHGVsJG7PLRwLyfKuFUaO9sLAyMHp2xx uz1MInuZNOl7b0l+L47LLi2OMvf24VPpBpZ7DkPzezoIiGe/SKt9Wik7R7Ib0uTVXhAK 9xsI8Nfe+Wa7H6sgfbppvGYhHUEYAedcdOTkq5jvtoToXEDl7iInXVZ/vG1Z5Gu7IvKl 93Ppx9zqlX6ePwttNjSYiwrdAQm5nHeykjckvycDvYZmszDTdqd3Pyqs53XatPbbL1ye g0w23cpt2Lw5SloaeyYY5SYDKB9rE4JoAOHU2MaaklAm6digsfRyXldDTeoB0Uo4qJh6 YmzQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=pxFfFsuy6kbRVs0LWrBWAlBLaVA5rUY1NByIba8Y/08=; b=soAy9w1fG+sEM7Aa6rht5LoWPIwEwr9b02zPL21KBN58d90CeASAtDiXZ8N7YAx+yn 1FeyV7/4jfY/YiJ9aWn1rDPGOonbsQoRbcu8Q8Alh7oBv/xD1WbO0Q03KvJ+9hM9R+fu 1BGdFKEIBg6jKLWKcjyO9YvxqaoK0EzFQv8tamfHWfHuY+Ys8nU9mE9HAy7bXmh62ToE 3nY99U/tmbBDBryPHuJYmew37O1jKeN0wQU43cODbcmFaW8k307UVqXPmUCmjIZ77SsC MwBJPeJwMsYxbMR60lSIqgh6m56U7j8ZTMfSvDVHSTh+exitj9FK5cCsyOMUPqk3grnZ POmg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=R9Gr2oQA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u12-20020a170903124c00b001ae1bb70a41si1828012plh.168.2023.05.25.09.00.36; Thu, 25 May 2023 09:00:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=R9Gr2oQA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241650AbjEYPwP (ORCPT + 99 others); Thu, 25 May 2023 11:52:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41460 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241558AbjEYPwE (ORCPT ); Thu, 25 May 2023 11:52:04 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A6C010B for ; Thu, 25 May 2023 08:51:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685029879; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pxFfFsuy6kbRVs0LWrBWAlBLaVA5rUY1NByIba8Y/08=; b=R9Gr2oQALsW8UR9K3V1zdqIvn3jQ9oeABbYhxTcqiuk5SPnQMpXVCjekWrOPy6AZ+DRDnk tJGl+OxqGTvQXyI/iWYjKOKA80SEClKTZ/Nw58aEb5E+br08gY5LMtaPz+99x2V7LwLbwu 6HSMS52s7djNDgwZlxHDR59RcByBwgg= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-601-yKzjrWDzM6Su1Tb4mISqqA-1; Thu, 25 May 2023 11:51:18 -0400 X-MC-Unique: yKzjrWDzM6Su1Tb4mISqqA-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DB73F3C17C65; Thu, 25 May 2023 15:51:16 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7BE4C492B0A; Thu, 25 May 2023 15:51:14 +0000 (UTC) From: David Howells To: Christoph Hellwig , David Hildenbrand Cc: David Howells , Jens Axboe , Al Viro , Matthew Wilcox , Jan Kara , Jeff Layton , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , Christian Brauner , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Andrew Morton Subject: [RFC PATCH 3/3] block: Use iov_iter_extract_pages() and page pinning in direct-io.c Date: Thu, 25 May 2023 16:51:02 +0100 Message-Id: <20230525155102.87353-4-dhowells@redhat.com> In-Reply-To: <20230525155102.87353-1-dhowells@redhat.com> References: <20230525155102.87353-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766882495003175041?= X-GMAIL-MSGID: =?utf-8?q?1766882495003175041?= Change the old block-based direct-I/O code to use iov_iter_extract_pages() to pin user pages or leave kernel pages unpinned rather than taking refs when submitting bios. This makes use of the preceding patches to not take pins on the zero page (thereby allowing insertion of zero pages in with pinned pages) and to get additional pins on pages, allowing an extracted page to be used in multiple bios without having to re-extract it. Signed-off-by: David Howells cc: Christoph Hellwig cc: David Hildenbrand cc: Andrew Morton cc: Jens Axboe cc: Al Viro cc: Matthew Wilcox cc: Jan Kara cc: Jeff Layton cc: Jason Gunthorpe cc: Logan Gunthorpe cc: Hillf Danton cc: Christian Brauner cc: Linus Torvalds cc: linux-fsdevel@vger.kernel.org cc: linux-block@vger.kernel.org cc: linux-kernel@vger.kernel.org cc: linux-mm@kvack.org --- fs/direct-io.c | 68 ++++++++++++++++++++++++++++++-------------------- 1 file changed, 41 insertions(+), 27 deletions(-) diff --git a/fs/direct-io.c b/fs/direct-io.c index ad20f3428bab..14049204cac8 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -42,8 +42,8 @@ #include "internal.h" /* - * How many user pages to map in one call to get_user_pages(). This determines - * the size of a structure in the slab cache + * How many user pages to map in one call to iov_iter_extract_pages(). This + * determines the size of a structure in the slab cache */ #define DIO_PAGES 64 @@ -121,12 +121,13 @@ struct dio { struct inode *inode; loff_t i_size; /* i_size when submitted */ dio_iodone_t *end_io; /* IO completion function */ + bool need_unpin; /* T if we need to unpin the pages */ void *private; /* copy from map_bh.b_private */ /* BIO completion state */ spinlock_t bio_lock; /* protects BIO fields below */ - int page_errors; /* errno from get_user_pages() */ + int page_errors; /* err from iov_iter_extract_pages() */ int is_async; /* is IO async ? */ bool defer_completion; /* defer AIO completion to workqueue? */ bool should_dirty; /* if pages should be dirtied */ @@ -165,14 +166,14 @@ static inline unsigned dio_pages_present(struct dio_submit *sdio) */ static inline int dio_refill_pages(struct dio *dio, struct dio_submit *sdio) { + struct page **pages = dio->pages; const enum req_op dio_op = dio->opf & REQ_OP_MASK; ssize_t ret; - ret = iov_iter_get_pages2(sdio->iter, dio->pages, LONG_MAX, DIO_PAGES, - &sdio->from); + ret = iov_iter_extract_pages(sdio->iter, &pages, LONG_MAX, + DIO_PAGES, 0, &sdio->from); if (ret < 0 && sdio->blocks_available && dio_op == REQ_OP_WRITE) { - struct page *page = ZERO_PAGE(0); /* * A memory fault, but the filesystem has some outstanding * mapped blocks. We need to use those blocks up to avoid @@ -180,8 +181,7 @@ static inline int dio_refill_pages(struct dio *dio, struct dio_submit *sdio) */ if (dio->page_errors == 0) dio->page_errors = ret; - get_page(page); - dio->pages[0] = page; + dio->pages[0] = ZERO_PAGE(0); sdio->head = 0; sdio->tail = 1; sdio->from = 0; @@ -201,9 +201,9 @@ static inline int dio_refill_pages(struct dio *dio, struct dio_submit *sdio) /* * Get another userspace page. Returns an ERR_PTR on error. Pages are - * buffered inside the dio so that we can call get_user_pages() against a - * decent number of pages, less frequently. To provide nicer use of the - * L1 cache. + * buffered inside the dio so that we can call iov_iter_extract_pages() + * against a decent number of pages, less frequently. To provide nicer use of + * the L1 cache. */ static inline struct page *dio_get_page(struct dio *dio, struct dio_submit *sdio) @@ -219,6 +219,18 @@ static inline struct page *dio_get_page(struct dio *dio, return dio->pages[sdio->head]; } +static void dio_pin_page(struct dio *dio, struct page *page) +{ + if (dio->need_unpin) + page_get_additional_pin(page); +} + +static void dio_unpin_page(struct dio *dio, struct page *page) +{ + if (dio->need_unpin) + unpin_user_page(page); +} + /* * dio_complete() - called when all DIO BIO I/O has been completed * @@ -444,8 +456,9 @@ static inline void dio_bio_submit(struct dio *dio, struct dio_submit *sdio) */ static inline void dio_cleanup(struct dio *dio, struct dio_submit *sdio) { - while (sdio->head < sdio->tail) - put_page(dio->pages[sdio->head++]); + if (dio->need_unpin) + unpin_user_pages(dio->pages + sdio->head, + sdio->tail - sdio->head); } /* @@ -676,7 +689,7 @@ static inline int dio_new_bio(struct dio *dio, struct dio_submit *sdio, * * Return zero on success. Non-zero means the caller needs to start a new BIO. */ -static inline int dio_bio_add_page(struct dio_submit *sdio) +static inline int dio_bio_add_page(struct dio *dio, struct dio_submit *sdio) { int ret; @@ -688,7 +701,7 @@ static inline int dio_bio_add_page(struct dio_submit *sdio) */ if ((sdio->cur_page_len + sdio->cur_page_offset) == PAGE_SIZE) sdio->pages_in_io--; - get_page(sdio->cur_page); + dio_pin_page(dio, sdio->cur_page); sdio->final_block_in_bio = sdio->cur_page_block + (sdio->cur_page_len >> sdio->blkbits); ret = 0; @@ -743,11 +756,11 @@ static inline int dio_send_cur_page(struct dio *dio, struct dio_submit *sdio, goto out; } - if (dio_bio_add_page(sdio) != 0) { + if (dio_bio_add_page(dio, sdio) != 0) { dio_bio_submit(dio, sdio); ret = dio_new_bio(dio, sdio, sdio->cur_page_block, map_bh); if (ret == 0) { - ret = dio_bio_add_page(sdio); + ret = dio_bio_add_page(dio, sdio); BUG_ON(ret != 0); } } @@ -804,13 +817,13 @@ submit_page_section(struct dio *dio, struct dio_submit *sdio, struct page *page, */ if (sdio->cur_page) { ret = dio_send_cur_page(dio, sdio, map_bh); - put_page(sdio->cur_page); + dio_unpin_page(dio, sdio->cur_page); sdio->cur_page = NULL; if (ret) return ret; } - get_page(page); /* It is in dio */ + dio_pin_page(dio, page); /* It is in dio */ sdio->cur_page = page; sdio->cur_page_offset = offset; sdio->cur_page_len = len; @@ -825,7 +838,7 @@ submit_page_section(struct dio *dio, struct dio_submit *sdio, struct page *page, ret = dio_send_cur_page(dio, sdio, map_bh); if (sdio->bio) dio_bio_submit(dio, sdio); - put_page(sdio->cur_page); + dio_unpin_page(dio, sdio->cur_page); sdio->cur_page = NULL; } return ret; @@ -926,7 +939,7 @@ static int do_direct_IO(struct dio *dio, struct dio_submit *sdio, ret = get_more_blocks(dio, sdio, map_bh); if (ret) { - put_page(page); + dio_unpin_page(dio, page); goto out; } if (!buffer_mapped(map_bh)) @@ -971,7 +984,7 @@ static int do_direct_IO(struct dio *dio, struct dio_submit *sdio, /* AKPM: eargh, -ENOTBLK is a hack */ if (dio_op == REQ_OP_WRITE) { - put_page(page); + dio_unpin_page(dio, page); return -ENOTBLK; } @@ -984,7 +997,7 @@ static int do_direct_IO(struct dio *dio, struct dio_submit *sdio, if (sdio->block_in_file >= i_size_aligned >> blkbits) { /* We hit eof */ - put_page(page); + dio_unpin_page(dio, page); goto out; } zero_user(page, from, 1 << blkbits); @@ -1024,7 +1037,7 @@ static int do_direct_IO(struct dio *dio, struct dio_submit *sdio, sdio->next_block_for_io, map_bh); if (ret) { - put_page(page); + dio_unpin_page(dio, page); goto out; } sdio->next_block_for_io += this_chunk_blocks; @@ -1039,8 +1052,8 @@ static int do_direct_IO(struct dio *dio, struct dio_submit *sdio, break; } - /* Drop the ref which was taken in get_user_pages() */ - put_page(page); + /* Drop the pin which was taken in get_user_pages() */ + dio_unpin_page(dio, page); } out: return ret; @@ -1135,6 +1148,7 @@ ssize_t __blockdev_direct_IO(struct kiocb *iocb, struct inode *inode, /* will be released by direct_io_worker */ inode_lock(inode); } + dio->need_unpin = iov_iter_extract_will_pin(iter); /* Once we sampled i_size check for reads beyond EOF */ dio->i_size = i_size_read(inode); @@ -1259,7 +1273,7 @@ ssize_t __blockdev_direct_IO(struct kiocb *iocb, struct inode *inode, ret2 = dio_send_cur_page(dio, &sdio, &map_bh); if (retval == 0) retval = ret2; - put_page(sdio.cur_page); + dio_unpin_page(dio, sdio.cur_page); sdio.cur_page = NULL; } if (sdio.bio)