From patchwork Mon Jan 23 17:29:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 47271 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1735973wrn; Mon, 23 Jan 2023 09:50:56 -0800 (PST) X-Google-Smtp-Source: AMrXdXsmLnXE6pGFKIutxoYWUOut/4J4tpgCSrYwXaCDGHsElL0rcxNg5fQWZcX5mK8BuZ8ILZRb X-Received: by 2002:a17:907:3a83:b0:84d:2f2a:b158 with SMTP id fh3-20020a1709073a8300b0084d2f2ab158mr33167288ejc.11.1674496256649; Mon, 23 Jan 2023 09:50:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674496256; cv=none; d=google.com; s=arc-20160816; b=eDTIGjfT+FTFyKQvBoPGAJmxBrBrAQIdKZvGdWVKtruzz5nxKUpUAkV1vOtvIRuioa OWCugmC7s2ULm1ZhBR3fe9OfJxcjDSxsVmjXldlfwEnOGYQ2bFnLmZrkEacHvF09ziMG ukASc7i9uAGJoBaaz/q6jGwgw6/DvIsmox/07KmLdnm6SsRHBtgJTBAiMLxEZ7gDStXV b3qWKx8DY+Qikegoh2PGuuaIaIcUJ1udzrfbossHYZ+57itc1G7xllBhDZIBJo/suaCp 9AtGQFCIeNY4bxtvvbr5fMRzB712TkdCFf3YRD3i+9aVSrybMApMeaWbLh2iZlimkiK+ c+qQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=wx/XTYvUJRqTE+Z1U7sVb1ZtZu4yEbfP0vm2HM2RFA0=; b=B3V0xoK1gwlkeph/Uofw2lLu7pkU3LgeX6OmXMDJlsHOQDZ8mXIam6WyOGPlz7Cqdg /4qhaHh4UDxtZOtM9Jiaz2G6Dvyc2hKCcOzI+OfMCtB6qpuPD9t++FM4sjrKtOPfWNDJ nenWL91Nw5uTcQTVeZU5lFD4WAnGbfiBVLCw81HZvZ7s0GxaLZAbuq5MVoqrNO++TsEu 2WA6Y8A8kARyqHUaQu6l3hVCcHOny5hmymX/R294ERaYcWADESRulQzrGsNNfTCSva7+ ZnxTRFxZZ1A/UOCaNaHYC0va56xvtTTd1fECt4uWLJFhQQrKjNhcpq+1FdvhDbT+unKd dpAA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JEW5Kbel; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cs19-20020a170906dc9300b0073d81935416si771263ejc.749.2023.01.23.09.50.33; Mon, 23 Jan 2023 09:50:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JEW5Kbel; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231757AbjAWRbG (ORCPT + 99 others); Mon, 23 Jan 2023 12:31:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38106 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232464AbjAWRbE (ORCPT ); Mon, 23 Jan 2023 12:31:04 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A87FE29408 for ; Mon, 23 Jan 2023 09:30:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674495017; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wx/XTYvUJRqTE+Z1U7sVb1ZtZu4yEbfP0vm2HM2RFA0=; b=JEW5Kbel5wKPS9J53JEgvq5z8+vgzFPDVRKK7JgNWdjZWgHZOFE9BXxVwNzvL9Mmv+hHVE FuBDATxO2/9ivDqrbAeNwKOn5BCCikppqgyBYVORqw+33aYFecz1bsI/UaXTWpwbnpzXvK YpUbPTALmE5ZXb2OVSGpJSt5U8Hf4aY= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-74-v1d4CE5OP7qEUBREYXMPZA-1; Mon, 23 Jan 2023 12:30:15 -0500 X-MC-Unique: v1d4CE5OP7qEUBREYXMPZA-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 18F241C0896B; Mon, 23 Jan 2023 17:30:15 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id C1FF1492C1B; Mon, 23 Jan 2023 17:30:13 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v8 01/10] iov_iter: Define flags to qualify page extraction. Date: Mon, 23 Jan 2023 17:29:58 +0000 Message-Id: <20230123173007.325544-2-dhowells@redhat.com> In-Reply-To: <20230123173007.325544-1-dhowells@redhat.com> References: <20230123173007.325544-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755836587098951554?= X-GMAIL-MSGID: =?utf-8?q?1755836587098951554?= Define flags to qualify page extraction to pass into iov_iter_*_pages*() rather than passing in FOLL_* flags. For now only a flag to allow peer-to-peer DMA is supported. Signed-off-by: David Howells cc: Al Viro cc: Christoph Hellwig cc: Jens Axboe cc: Logan Gunthorpe cc: linux-fsdevel@vger.kernel.org cc: linux-block@vger.kernel.org Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard --- Notes: ver #7) - Don't use FOLL_* as a parameter, but rather define constants specifically to use with iov_iter_*_pages*(). - Drop the I/O direction constants for now. block/bio.c | 6 +++--- block/blk-map.c | 8 ++++---- include/linux/uio.h | 7 +++++-- lib/iov_iter.c | 14 ++++++++------ 4 files changed, 20 insertions(+), 15 deletions(-) diff --git a/block/bio.c b/block/bio.c index ab59a491a883..a289bbff036f 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1249,7 +1249,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) unsigned short entries_left = bio->bi_max_vecs - bio->bi_vcnt; struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt; struct page **pages = (struct page **)bv; - unsigned int gup_flags = 0; + unsigned int extract_flags = 0; ssize_t size, left; unsigned len, i = 0; size_t offset, trim; @@ -1264,7 +1264,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) pages += entries_left * (PAGE_PTRS_PER_BVEC - 1); if (bio->bi_bdev && blk_queue_pci_p2pdma(bio->bi_bdev->bd_disk->queue)) - gup_flags |= FOLL_PCI_P2PDMA; + extract_flags |= ITER_ALLOW_P2PDMA; /* * Each segment in the iov is required to be a block size multiple. @@ -1275,7 +1275,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) */ size = iov_iter_get_pages(iter, pages, UINT_MAX - bio->bi_iter.bi_size, - nr_pages, &offset, gup_flags); + nr_pages, &offset, extract_flags); if (unlikely(size <= 0)) return size ? size : -EFAULT; diff --git a/block/blk-map.c b/block/blk-map.c index 19940c978c73..bc111261fc82 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -267,7 +267,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, { unsigned int max_sectors = queue_max_hw_sectors(rq->q); unsigned int nr_vecs = iov_iter_npages(iter, BIO_MAX_VECS); - unsigned int gup_flags = 0; + unsigned int extract_flags = 0; struct bio *bio; int ret; int j; @@ -280,7 +280,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, return -ENOMEM; if (blk_queue_pci_p2pdma(rq->q)) - gup_flags |= FOLL_PCI_P2PDMA; + extract_flags |= ITER_ALLOW_P2PDMA; while (iov_iter_count(iter)) { struct page **pages, *stack_pages[UIO_FASTIOV]; @@ -291,10 +291,10 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (nr_vecs <= ARRAY_SIZE(stack_pages)) { pages = stack_pages; bytes = iov_iter_get_pages(iter, pages, LONG_MAX, - nr_vecs, &offs, gup_flags); + nr_vecs, &offs, extract_flags); } else { bytes = iov_iter_get_pages_alloc(iter, &pages, - LONG_MAX, &offs, gup_flags); + LONG_MAX, &offs, extract_flags); } if (unlikely(bytes <= 0)) { ret = bytes ? bytes : -EFAULT; diff --git a/include/linux/uio.h b/include/linux/uio.h index 9f158238edba..46d5080314c6 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -252,12 +252,12 @@ void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray * loff_t start, size_t count); ssize_t iov_iter_get_pages(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start, - unsigned gup_flags); + unsigned extract_flags); ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start); ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start, - unsigned gup_flags); + unsigned extract_flags); ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start); int iov_iter_npages(const struct iov_iter *i, int maxpages); @@ -360,4 +360,7 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, }; } +/* Flags for iov_iter_get/extract_pages*() */ +#define ITER_ALLOW_P2PDMA 0x01 /* Allow P2PDMA on the extracted pages */ + #endif diff --git a/lib/iov_iter.c b/lib/iov_iter.c index f9a3ff37ecd1..fb04abe7d746 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1432,9 +1432,9 @@ static struct page *first_bvec_segment(const struct iov_iter *i, static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, unsigned int maxpages, size_t *start, - unsigned int gup_flags) + unsigned int extract_flags) { - unsigned int n; + unsigned int n, gup_flags = 0; if (maxsize > i->count) maxsize = i->count; @@ -1442,6 +1442,8 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, return 0; if (maxsize > MAX_RW_COUNT) maxsize = MAX_RW_COUNT; + if (extract_flags & ITER_ALLOW_P2PDMA) + gup_flags |= FOLL_PCI_P2PDMA; if (likely(user_backed_iter(i))) { unsigned long addr; @@ -1495,14 +1497,14 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, ssize_t iov_iter_get_pages(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, - size_t *start, unsigned gup_flags) + size_t *start, unsigned extract_flags) { if (!maxpages) return 0; BUG_ON(!pages); return __iov_iter_get_pages_alloc(i, &pages, maxsize, maxpages, - start, gup_flags); + start, extract_flags); } EXPORT_SYMBOL_GPL(iov_iter_get_pages); @@ -1515,14 +1517,14 @@ EXPORT_SYMBOL(iov_iter_get_pages2); ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, - size_t *start, unsigned gup_flags) + size_t *start, unsigned extract_flags) { ssize_t len; *pages = NULL; len = __iov_iter_get_pages_alloc(i, pages, maxsize, ~0U, start, - gup_flags); + extract_flags); if (len <= 0) { kvfree(*pages); *pages = NULL; From patchwork Mon Jan 23 17:29:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 47275 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1736503wrn; Mon, 23 Jan 2023 09:52:26 -0800 (PST) X-Google-Smtp-Source: AK7set/KZhlfQoyQLPHqV+6YU27h4aSoqFg7O5nW71cXryOlAkEiyXfs40bj9JwyA8rBdGiu4d2O X-Received: by 2002:a05:6402:b09:b0:49f:2100:2a2b with SMTP id bm9-20020a0564020b0900b0049f21002a2bmr632661edb.31.1674496346463; Mon, 23 Jan 2023 09:52:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674496346; cv=none; d=google.com; s=arc-20160816; b=dwRrKouYoTBwv30ZEWaO6zSC1dQ/Mp6wJcpU46Zhy6LJk26XyR7FaQkByxuirAxCnO 1LFCelC+lUjgVYD3N9IOq/I1HG1cw9xypT/VplO8cwrWxy51t04c6fgaFjECmx1ZxsAC xgmx3IATpmo/i4ed/gicrCGG+w5R8XN5L7Q/xIEdfYGWM1D4r7Q+HksWJGjJdj5CZtBw SXzvGqkDWvLvOJkkKWNBjCoanYb02wEsmc14LfAK261q801Swynw9LIkhS3fvXcSr4qw TOgdUr/M/mdcJrnO9WZZNqkyIeuiqs1H24+gU7PkJZQyam7dSvrUQn7N5ScmXhgnGU6R SR/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1BQhW3wHk53SQ0g+pPIstFzSMICN3HLKrueb/Jgd0sA=; b=q/QAAbJzy8mr92VWoFR28jEoB6RzZ+iMlK9eUVTEkbgZ6Vz+BWy0oKlQSM9IlXxXNZ 0D+HcAz0/2OlQbyXgywQRKVUcj3LbycY27TrvCDfOvQ/5hZRPJg2GAEQUhkyIouZHN7z jwvtFw43Vg5/GSosf61PdjCGEl2A2YO0jl0LyARrDML/+ywoYxfO/SihPvzA7RCvAqQV Cs/dqrEhou7nUalJbcPJSQyhndRUAXF+UNUKNl091CWHnin4skcPEJC+Idu2R1t5C/i2 FzvZMQsLDWT5D4rFsM/RtLQEif0erSRRzhOyKLErHRCygemElSrXpdGfS1M8pGlb4wBu XRAg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OzEpjyob; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id xg11-20020a170907320b00b007c0a9904d63si54674544ejb.742.2023.01.23.09.52.03; Mon, 23 Jan 2023 09:52:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OzEpjyob; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232414AbjAWRcF (ORCPT + 99 others); Mon, 23 Jan 2023 12:32:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38304 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233724AbjAWRbz (ORCPT ); Mon, 23 Jan 2023 12:31:55 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0F7A2C672 for ; Mon, 23 Jan 2023 09:30:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674495022; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1BQhW3wHk53SQ0g+pPIstFzSMICN3HLKrueb/Jgd0sA=; b=OzEpjyobGWaD5haH3fQz2ZnQA0zR8GSFpK9H3VM5/0J1T4xTFjBR3BdetkZdp+/YlZzb3c 4lAJGb+zacIDEHWmN0V88+w5nKCoaBH+bobvAffbR523AMaoVKn5Go+maL1WA8LYRzuBhO mc9GW/qvT3lZHjrbvXzU9wxofkbN9y4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-103-MGtzFN3AN8yceGHs8VyG8Q-1; Mon, 23 Jan 2023 12:30:18 -0500 X-MC-Unique: MGtzFN3AN8yceGHs8VyG8Q-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8CA7A18E6C41; Mon, 23 Jan 2023 17:30:17 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id B1CA61121330; Mon, 23 Jan 2023 17:30:15 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig , John Hubbard , David Hildenbrand , linux-mm@kvack.org Subject: [PATCH v8 02/10] iov_iter: Add a function to extract a page list from an iterator Date: Mon, 23 Jan 2023 17:29:59 +0000 Message-Id: <20230123173007.325544-3-dhowells@redhat.com> In-Reply-To: <20230123173007.325544-1-dhowells@redhat.com> References: <20230123173007.325544-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755836681034977945?= X-GMAIL-MSGID: =?utf-8?q?1755836681034977945?= Add a function, iov_iter_extract_pages(), to extract a list of pages from an iterator. The pages may be returned with a pin added or nothing, depending on the type of iterator. Add a second function, iov_iter_extract_mode(), to determine how the cleanup should be done. There are two cases: (1) ITER_IOVEC or ITER_UBUF iterator. Extracted pages will have pins (FOLL_PIN) obtained on them so that a concurrent fork() will forcibly copy the page so that DMA is done to/from the parent's buffer and is unavailable to/unaffected by the child process. iov_iter_extract_mode() will return FOLL_PIN for this case. The caller should use something like folio_put_unpin() to dispose of the page. (2) Any other sort of iterator. No refs or pins are obtained on the page, the assumption is made that the caller will manage page retention. iov_iter_extract_mode() will return 0. The pages don't need additional disposal. Signed-off-by: David Howells cc: Al Viro cc: Christoph Hellwig cc: John Hubbard cc: David Hildenbrand cc: Matthew Wilcox cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org Reviewed-by: Christoph Hellwig --- Notes: ver #8) - It seems that all DIO is supposed to be done under FOLL_PIN now, and not FOLL_GET, so switch to only using pin_user_pages() for user-backed iters. - Wrap an argument in brackets in the iov_iter_extract_mode() macro. - Drop the extract_flags argument to iov_iter_extract_mode() for now [hch]. ver #7) - Switch to passing in iter-specific flags rather than FOLL_* flags. - Drop the direction flags for now. - Use ITER_ALLOW_P2PDMA to request FOLL_PCI_P2PDMA. - Disallow use of ITER_ALLOW_P2PDMA with non-user-backed iter. - Add support for extraction from KVEC-type iters. - Use iov_iter_advance() rather than open-coding it. - Make BVEC- and KVEC-type skip over initial empty vectors. ver #6) - Add back the function to indicate the cleanup mode. - Drop the cleanup_mode return arg to iov_iter_extract_pages(). - Pass FOLL_SOURCE/DEST_BUF in gup_flags. Check this against the iter data_source. ver #4) - Use ITER_SOURCE/DEST instead of WRITE/READ. - Allow additional FOLL_* flags, such as FOLL_PCI_P2PDMA to be passed in. ver #3) - Switch to using EXPORT_SYMBOL_GPL to prevent indirect 3rd-party access to get/pin_user_pages_fast()[1]. include/linux/uio.h | 22 +++ lib/iov_iter.c | 320 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 342 insertions(+) diff --git a/include/linux/uio.h b/include/linux/uio.h index 46d5080314c6..a8165335f8da 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -363,4 +363,26 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, /* Flags for iov_iter_get/extract_pages*() */ #define ITER_ALLOW_P2PDMA 0x01 /* Allow P2PDMA on the extracted pages */ +ssize_t iov_iter_extract_pages(struct iov_iter *i, struct page ***pages, + size_t maxsize, unsigned int maxpages, + unsigned int extract_flags, size_t *offset0); + +/** + * iov_iter_extract_mode - Indicate how pages from the iterator will be retained + * @iter: The iterator + * + * Examine the iterator and indicate by returning FOLL_PIN or 0 as to how, if + * at all, pages extracted from the iterator will be retained by the extraction + * function. + * + * FOLL_PIN indicates that the pages will have a pin placed in them that the + * caller must unpin. This is must be done for DMA/async DIO to force fork() + * to forcibly copy a page for the child (the parent must retain the original + * page). + * + * 0 indicates that no measures are taken and that it's up to the caller to + * retain the pages. + */ +#define iov_iter_extract_mode(iter) (user_backed_iter(iter) ? FOLL_PIN : 0) + #endif diff --git a/lib/iov_iter.c b/lib/iov_iter.c index fb04abe7d746..57f3e9404160 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1916,3 +1916,323 @@ void iov_iter_restore(struct iov_iter *i, struct iov_iter_state *state) i->iov -= state->nr_segs - i->nr_segs; i->nr_segs = state->nr_segs; } + +/* + * Extract a list of contiguous pages from an ITER_PIPE iterator. This does + * not get references of its own on the pages, nor does it get a pin on them. + * If there's a partial page, it adds that first and will then allocate and add + * pages into the pipe to make up the buffer space to the amount required. + * + * The caller must hold the pipe locked and only transferring into a pipe is + * supported. + */ +static ssize_t iov_iter_extract_pipe_pages(struct iov_iter *i, + struct page ***pages, size_t maxsize, + unsigned int maxpages, + unsigned int extract_flags, + size_t *offset0) +{ + unsigned int nr, offset, chunk, j; + struct page **p; + size_t left; + + if (!sanity(i)) + return -EFAULT; + + offset = pipe_npages(i, &nr); + if (!nr) + return -EFAULT; + *offset0 = offset; + + maxpages = min_t(size_t, nr, maxpages); + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + p = *pages; + + left = maxsize; + for (j = 0; j < maxpages; j++) { + struct page *page = append_pipe(i, left, &offset); + if (!page) + break; + chunk = min_t(size_t, left, PAGE_SIZE - offset); + left -= chunk; + *p++ = page; + } + if (!j) + return -EFAULT; + return maxsize - left; +} + +/* + * Extract a list of contiguous pages from an ITER_XARRAY iterator. This does not + * get references on the pages, nor does it get a pin on them. + */ +static ssize_t iov_iter_extract_xarray_pages(struct iov_iter *i, + struct page ***pages, size_t maxsize, + unsigned int maxpages, + unsigned int extract_flags, + size_t *offset0) +{ + struct page *page, **p; + unsigned int nr = 0, offset; + loff_t pos = i->xarray_start + i->iov_offset; + pgoff_t index = pos >> PAGE_SHIFT; + XA_STATE(xas, i->xarray, index); + + offset = pos & ~PAGE_MASK; + *offset0 = offset; + + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + p = *pages; + + rcu_read_lock(); + for (page = xas_load(&xas); page; page = xas_next(&xas)) { + if (xas_retry(&xas, page)) + continue; + + /* Has the page moved or been split? */ + if (unlikely(page != xas_reload(&xas))) { + xas_reset(&xas); + continue; + } + + p[nr++] = find_subpage(page, xas.xa_index); + if (nr == maxpages) + break; + } + rcu_read_unlock(); + + maxsize = min_t(size_t, nr * PAGE_SIZE - offset, maxsize); + iov_iter_advance(i, maxsize); + return maxsize; +} + +/* + * Extract a list of contiguous pages from an ITER_BVEC iterator. This does + * not get references on the pages, nor does it get a pin on them. + */ +static ssize_t iov_iter_extract_bvec_pages(struct iov_iter *i, + struct page ***pages, size_t maxsize, + unsigned int maxpages, + unsigned int extract_flags, + size_t *offset0) +{ + struct page **p, *page; + size_t skip = i->iov_offset, offset; + int k; + + for (;;) { + if (i->nr_segs == 0) + return 0; + maxsize = min(maxsize, i->bvec->bv_len - skip); + if (maxsize) + break; + i->iov_offset = 0; + i->nr_segs--; + i->kvec++; + skip = 0; + } + + skip += i->bvec->bv_offset; + page = i->bvec->bv_page + skip / PAGE_SIZE; + offset = skip % PAGE_SIZE; + *offset0 = offset; + + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + p = *pages; + for (k = 0; k < maxpages; k++) + p[k] = page + k; + + maxsize = min_t(size_t, maxsize, maxpages * PAGE_SIZE - offset); + iov_iter_advance(i, maxsize); + return maxsize; +} + +/* + * Extract a list of virtually contiguous pages from an ITER_KVEC iterator. + * This does not get references on the pages, nor does it get a pin on them. + */ +static ssize_t iov_iter_extract_kvec_pages(struct iov_iter *i, + struct page ***pages, size_t maxsize, + unsigned int maxpages, + unsigned int extract_flags, + size_t *offset0) +{ + struct page **p, *page; + const void *kaddr; + size_t skip = i->iov_offset, offset, len; + int k; + + for (;;) { + if (i->nr_segs == 0) + return 0; + maxsize = min(maxsize, i->kvec->iov_len - skip); + if (maxsize) + break; + i->iov_offset = 0; + i->nr_segs--; + i->kvec++; + skip = 0; + } + + offset = skip % PAGE_SIZE; + *offset0 = offset; + kaddr = i->kvec->iov_base; + + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + p = *pages; + + kaddr -= offset; + len = offset + maxsize; + for (k = 0; k < maxpages; k++) { + size_t seg = min_t(size_t, len, PAGE_SIZE); + + if (is_vmalloc_or_module_addr(kaddr)) + page = vmalloc_to_page(kaddr); + else + page = virt_to_page(kaddr); + + p[k] = page; + len -= seg; + kaddr += PAGE_SIZE; + } + + maxsize = min_t(size_t, maxsize, maxpages * PAGE_SIZE - offset); + iov_iter_advance(i, maxsize); + return maxsize; +} + +/* + * Extract a list of contiguous pages from a user iterator and get a pin on + * each of them. This should only be used if the iterator is user-backed + * (IOBUF/UBUF). + * + * It does not get refs on the pages, but the pages must be unpinned by the + * caller once the transfer is complete. + * + * This is safe to be used where background IO/DMA *is* going to be modifying + * the buffer; using a pin rather than a ref makes forces fork() to give the + * child a copy of the page. + */ +static ssize_t iov_iter_extract_user_pages(struct iov_iter *i, + struct page ***pages, + size_t maxsize, + unsigned int maxpages, + unsigned int extract_flags, + size_t *offset0) +{ + unsigned long addr; + unsigned int gup_flags = FOLL_PIN; + size_t offset; + int res; + + if (i->data_source == ITER_DEST) + gup_flags |= FOLL_WRITE; + if (extract_flags & ITER_ALLOW_P2PDMA) + gup_flags |= FOLL_PCI_P2PDMA; + if (i->nofault) + gup_flags |= FOLL_NOFAULT; + + addr = first_iovec_segment(i, &maxsize); + *offset0 = offset = addr % PAGE_SIZE; + addr &= PAGE_MASK; + maxpages = want_pages_array(pages, maxsize, offset, maxpages); + if (!maxpages) + return -ENOMEM; + res = pin_user_pages_fast(addr, maxpages, gup_flags, *pages); + if (unlikely(res <= 0)) + return res; + maxsize = min_t(size_t, maxsize, res * PAGE_SIZE - offset); + iov_iter_advance(i, maxsize); + return maxsize; +} + +/** + * iov_iter_extract_pages - Extract a list of contiguous pages from an iterator + * @i: The iterator to extract from + * @pages: Where to return the list of pages + * @maxsize: The maximum amount of iterator to extract + * @maxpages: The maximum size of the list of pages + * @extract_flags: Flags to qualify request + * @offset0: Where to return the starting offset into (*@pages)[0] + * + * Extract a list of contiguous pages from the current point of the iterator, + * advancing the iterator. The maximum number of pages and the maximum amount + * of page contents can be set. + * + * If *@pages is NULL, a page list will be allocated to the required size and + * *@pages will be set to its base. If *@pages is not NULL, it will be assumed + * that the caller allocated a page list at least @maxpages in size and this + * will be filled in. + * + * @extract_flags can have ITER_ALLOW_P2PDMA set to request peer-to-peer DMA be + * allowed on the pages extracted. + * + * The iov_iter_extract_mode() function can be used to query how cleanup should + * be performed. + * + * Extra refs or pins on the pages may be obtained as follows: + * + * (*) If the iterator is user-backed (ITER_IOVEC/ITER_UBUF), pins will be + * added to the pages, but refs will not be taken. + * iov_iter_extract_mode() will return FOLL_PIN. + * + * (*) If the iterator is ITER_PIPE, this must describe a destination for the + * data. Additional pages may be allocated and added to the pipe (which + * will hold the refs), but pins will not be obtained for the caller. The + * caller must hold the pipe lock. iov_iter_extract_mode() will return 0. + * + * (*) If the iterator is ITER_KVEC, ITER_BVEC or ITER_XARRAY, the pages are + * merely listed; no extra refs or pins are obtained. + * iov_iter_extract_mode() will return 0. + * + * Note also: + * + * (*) Use with ITER_DISCARD is not supported as that has no content. + * + * On success, the function sets *@pages to the new pagelist, if allocated, and + * sets *offset0 to the offset into the first page. + * + * It may also return -ENOMEM and -EFAULT. + */ +ssize_t iov_iter_extract_pages(struct iov_iter *i, + struct page ***pages, + size_t maxsize, + unsigned int maxpages, + unsigned int extract_flags, + size_t *offset0) +{ + maxsize = min_t(size_t, min_t(size_t, maxsize, i->count), MAX_RW_COUNT); + if (!maxsize) + return 0; + + if (likely(user_backed_iter(i))) + return iov_iter_extract_user_pages(i, pages, maxsize, + maxpages, extract_flags, + offset0); + if (iov_iter_is_kvec(i)) + return iov_iter_extract_kvec_pages(i, pages, maxsize, + maxpages, extract_flags, + offset0); + if (iov_iter_is_bvec(i)) + return iov_iter_extract_bvec_pages(i, pages, maxsize, + maxpages, extract_flags, + offset0); + if (iov_iter_is_pipe(i)) + return iov_iter_extract_pipe_pages(i, pages, maxsize, + maxpages, extract_flags, + offset0); + if (iov_iter_is_xarray(i)) + return iov_iter_extract_xarray_pages(i, pages, maxsize, + maxpages, extract_flags, + offset0); + return -EFAULT; +} +EXPORT_SYMBOL_GPL(iov_iter_extract_pages); From patchwork Mon Jan 23 17:30:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 47273 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1736059wrn; Mon, 23 Jan 2023 09:51:13 -0800 (PST) X-Google-Smtp-Source: AMrXdXup3Qm2rZjiEEqh+v2lrjqrK3iplFneDAHX4zC6/yKAk2REVbdg3f0QVcCtg86aWM9jrbeK X-Received: by 2002:a17:907:62a8:b0:86a:d385:81df with SMTP id nd40-20020a17090762a800b0086ad38581dfmr37557135ejc.3.1674496273736; Mon, 23 Jan 2023 09:51:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674496273; cv=none; d=google.com; s=arc-20160816; b=D8wKKK3FQBdu2vdaTturAcL2EgoEbbFftoNTxKwk0oa+b9rrTK5I/OAjfWdp62OMTE dM3iY026CmK2uEBc0ghsT27jh27lf7SWVSCvTyVWVKQmfDiEvwY4jD1mXLs3iNMV0KUr ZP4ZFvB1sCxqQMxAY5PiYISr/8UpAJUta3KOj5vD+ChoS09Jp3z7Re4Oe+irShCbjGjv Z2yqu9gAizpIzyMBW0xZ5Ue5yz1fTJQuPdcRI9JsXVlV6Wsfc6LXPTf0veSzZfqQKVIo LhMHcgT1/iehpC/UgEVssFCw4ubzCxlGhucjYlKrRUN6ZO8lT8McamL8/qRfEk4v7o/Z gqaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0jN2ELsFFzyNgM8YEdU+VNSLzU8C5xV57PSPrlYR/T4=; b=puIcICGIYr8X1rjgRSLbRQKLdfoxP6td6n7gV0qkFRMovL/wbhYziQnDyTE/9bofSU o2PR53YOp+7WNfee1nq8TKakBeEQovRraVGXcX0T93FKM6xlLlbwE+IirNwV6FWmBDNV A/zXIYsTR6iXe04HiGh6JPYY3hAuKWLKZQC1FKnKwwx2H55tKcvNgdliuNokxRzLW+Ow b7jFp230F+9MzORw0Qnh6EdWUQhmmMDR6BIm8MXWPYOG3yKNyfdaos0weOy6oQ54eQRz eOAvFJr4mTHpmO7nufs1OqFnfy7auAjn2J95c8v+/cOXVBQt+O5sSUpOJPqXcnZHjsft 3y3g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Qnv+jEyW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cw5-20020a170906478500b00872448d966asi25628113ejc.84.2023.01.23.09.50.50; Mon, 23 Jan 2023 09:51:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Qnv+jEyW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233728AbjAWRbz (ORCPT + 99 others); Mon, 23 Jan 2023 12:31:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229891AbjAWRbv (ORCPT ); Mon, 23 Jan 2023 12:31:51 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 44F672CC6A for ; Mon, 23 Jan 2023 09:30:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674495023; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0jN2ELsFFzyNgM8YEdU+VNSLzU8C5xV57PSPrlYR/T4=; b=Qnv+jEyW3cZBEZskzxVnhJS8FF+yMytpw1mpQ/LegsbFWfMF1lQS444nXt4TrkA3tPp+Zi jVy1wMmr21Ga3oH8LzBx633hzDSJkRV0sSQ2OsVLl4gTfEoq/kv65KUGqm+wiKv0W1Jheg iQh0w9nzEXASYcIdVAW610JJ2A+map4= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-656-PNKYRVDkPJio9uZRskHSUg-1; Mon, 23 Jan 2023 12:30:20 -0500 X-MC-Unique: PNKYRVDkPJio9uZRskHSUg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C4A183C14859; Mon, 23 Jan 2023 17:30:19 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 49E8D2166B32; Mon, 23 Jan 2023 17:30:18 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig , linux-mm@kvack.org Subject: [PATCH v8 03/10] mm: Provide a helper to drop a pin/ref on a page Date: Mon, 23 Jan 2023 17:30:00 +0000 Message-Id: <20230123173007.325544-4-dhowells@redhat.com> In-Reply-To: <20230123173007.325544-1-dhowells@redhat.com> References: <20230123173007.325544-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755836604681706592?= X-GMAIL-MSGID: =?utf-8?q?1755836604681706592?= Provide a helper in the get_user_pages code to drop a pin or a ref on a page based on being given FOLL_GET or FOLL_PIN in its flags argument or do nothing if neither is set. Signed-off-by: David Howells cc: Al Viro cc: Christoph Hellwig cc: Matthew Wilcox cc: linux-fsdevel@vger.kernel.org cc: linux-block@vger.kernel.org cc: linux-mm@kvack.org Reviewed-by: Christoph Hellwig --- include/linux/mm.h | 3 +++ mm/gup.c | 22 ++++++++++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 8f857163ac89..3de9d88f8524 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1367,6 +1367,9 @@ static inline bool is_cow_mapping(vm_flags_t flags) #define SECTION_IN_PAGE_FLAGS #endif +void folio_put_unpin(struct folio *folio, unsigned int flags); +void page_put_unpin(struct page *page, unsigned int flags); + /* * The identification function is mainly used by the buddy allocator for * determining if two pages could be buddies. We are not really identifying diff --git a/mm/gup.c b/mm/gup.c index f45a3a5be53a..3ee4b4c7e0cb 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -191,6 +191,28 @@ static void gup_put_folio(struct folio *folio, int refs, unsigned int flags) folio_put_refs(folio, refs); } +/** + * folio_put_unpin - Unpin/put a folio as appropriate + * @folio: The folio to release + * @flags: gup flags indicating the mode of release (FOLL_*) + * + * Release a folio according to the flags. If FOLL_GET is set, the folio has a + * ref dropped; if FOLL_PIN is set, it is unpinned; otherwise it is left + * unaltered. + */ +void folio_put_unpin(struct folio *folio, unsigned int flags) +{ + if (flags & (FOLL_GET | FOLL_PIN)) + gup_put_folio(folio, 1, flags); +} +EXPORT_SYMBOL_GPL(folio_put_unpin); + +void page_put_unpin(struct page *page, unsigned int flags) +{ + folio_put_unpin(page_folio(page), flags); +} +EXPORT_SYMBOL_GPL(page_put_unpin); + /** * try_grab_page() - elevate a page's refcount by a flag-dependent amount * @page: pointer to page to be grabbed From patchwork Mon Jan 23 17:30:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 47274 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1736114wrn; Mon, 23 Jan 2023 09:51:23 -0800 (PST) X-Google-Smtp-Source: AMrXdXv444p9XF7Gs4As/xk0SoDMQN1QLsjUJKSfYunLB4OUublppFzTYuAPHjnO4mLZEzZx1egs X-Received: by 2002:aa7:c9c7:0:b0:499:cc32:6a52 with SMTP id i7-20020aa7c9c7000000b00499cc326a52mr37890408edt.16.1674496283023; Mon, 23 Jan 2023 09:51:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674496283; cv=none; d=google.com; s=arc-20160816; b=VP97R7ojwFsKYCbG6AEvtzDCjlNH4nQW/7On/pAkrxwlOSjPxgus67mLB45oTju2bA FBLmwDf4Kl98sRnIzwZvJzDrjrFC0oEKKoBRu26tStRgbNpQVgFeCRA2Ipc38BNZEd4R Me4+fIiVqbPz4L/jofbZ6AE7gv3TYqHmKESJ2iTL6OLm0jRP8keaGUFm1PIUosFkOZRk ekxvwVJYg8zTH4+YWzHAW7QDCVi7HYrtVYB8qaEHYqTuQmyte9pT/4hSGSJYSGPb282u dsluAHSB4C/kO+A3mzpSNiVxMraZzfTiqEapqKUVfmhss7j63VICwRB8Ra6zG6LFO9mU 0lSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=bxZsldujvKlYIKHvA4IAxT3TcGniFcgHJOVEtOWexok=; b=IoiI+lchQ+ZV48qrGjeVvlYpaNjkguK8D3Kl1SkN9jigoKGaRN75I8/o2MXjPbc6Uo 4RluzbvDCtljBcDzvOmNBwJndPvh1JLaS5AZIHh3IUlgbCYtORVfcABsCugPmKd44ZMw CrFtluHYPbl/HsE3w8hlb1+OhPeGPw82vryFwkSzvUIRPWvUr0VEUREEzA7s/CENq/E/ pVa5Lj8We2bLJeTzfnQZLHZS4//4YRxZaOp3rr0P6XDASGaa/GHGhRlYqUpatsWH/5H+ ZGUozugEDFxzmdBPoAaVGgExoxUk9kvPdSgnDKdQW+jmof80/HPJXbitUWDj3WFwesRw 8wFA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZeQ2IPIR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id wg10-20020a17090705ca00b007c4f75345e2si33928451ejb.431.2023.01.23.09.51.00; Mon, 23 Jan 2023 09:51:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZeQ2IPIR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232571AbjAWRb6 (ORCPT + 99 others); Mon, 23 Jan 2023 12:31:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38314 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233199AbjAWRbx (ORCPT ); Mon, 23 Jan 2023 12:31:53 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 042BF2DE54 for ; Mon, 23 Jan 2023 09:30:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674495027; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bxZsldujvKlYIKHvA4IAxT3TcGniFcgHJOVEtOWexok=; b=ZeQ2IPIRey/d9RrbN5JXxr+0cvrpm/Lj0qLeyYEHx2qUgvwPnJYy1AAFMn81fRl35LAM8V c3Tj7/7CLiuyZTPiMUK7fvoCKK0m8h7uGLDI5mWWEp82OAj/tazuUY/hpXsQOye+vNaniy beUJcUswUVruBWshSSZklpibPsrQAys= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-237--KJ79ehvM1ivgAVOHAwz0A-1; Mon, 23 Jan 2023 12:30:24 -0500 X-MC-Unique: -KJ79ehvM1ivgAVOHAwz0A-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CCB293C14859; Mon, 23 Jan 2023 17:30:21 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6A9E8492B02; Mon, 23 Jan 2023 17:30:20 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig Subject: [PATCH v8 04/10] iomap: don't get an reference on ZERO_PAGE for direct I/O block zeroing Date: Mon, 23 Jan 2023 17:30:01 +0000 Message-Id: <20230123173007.325544-5-dhowells@redhat.com> In-Reply-To: <20230123173007.325544-1-dhowells@redhat.com> References: <20230123173007.325544-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755836614148539492?= X-GMAIL-MSGID: =?utf-8?q?1755836614148539492?= From: Christoph Hellwig ZERO_PAGE can't go away, no need to hold an extra reference. Signed-off-by: Christoph Hellwig Signed-off-by: David Howells Reviewed-by: David Hildenbrand --- fs/iomap/direct-io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 9804714b1751..47db4ead1e74 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -202,7 +202,7 @@ static void iomap_dio_zero(const struct iomap_iter *iter, struct iomap_dio *dio, bio->bi_private = dio; bio->bi_end_io = iomap_dio_bio_end_io; - get_page(page); + bio_set_flag(bio, BIO_NO_PAGE_REF); __bio_add_page(bio, page, len, 0); iomap_dio_submit_bio(iter, dio, bio, pos); } From patchwork Mon Jan 23 17:30:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 47272 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1736049wrn; Mon, 23 Jan 2023 09:51:11 -0800 (PST) X-Google-Smtp-Source: AMrXdXuZsTgsbasfcIVstQIcMRe0APYEvvosyLHZe6F5kslgMFuFlAKucRxyltW2bmJAW5aftHcA X-Received: by 2002:a17:906:270f:b0:86c:fa4c:220e with SMTP id z15-20020a170906270f00b0086cfa4c220emr28712103ejc.3.1674496271429; Mon, 23 Jan 2023 09:51:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674496271; cv=none; d=google.com; s=arc-20160816; b=sKf1vBim2xooBQifFTV9wAZ6JmIKijk+xUZB8+ncv8gd221t8M1iRzFlFP1rr2kE+d CMpJ7tTJeHFWdAwVEHA02ooiWho7/UrlRhbZRb8tULvHTU+N2gNSWG/IUgQUyLviX22P kB0fxkE6n5MjZiI52M16+qjDqMBdjPAKflOeHfJLGPDQF8RRcfcr4E6vBVwlwN+qapJq i84f6DS/AFwJraNRKEwYyBNoHLgYAq37STewTMFK4c3WXjsrfqoevBENA2GUh0L/qBcf gAVTbxIM1D/k76yS7qRkm/vyfOU5wXHyn3m9/UjxSG/Fhk5+k67B4xbCSs/K3YfNBsbL ilAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=u/H+JLqqGtjE96rU8y7Ek7KRkRO4zR4fLWrnb7rCy6M=; b=CTvqwzpQLGHw1TGMLATP3tEp3HyiB4XZldD+Qxjr47oJPxjqwv+ITnKKxFgigAicL/ fPblaEkpboh9xBGyMzHSxFOwL4ZGnEZrNSo7IIV8vppA3MHn++iMFP2GnhxqcC6xfuwj AE5k2pHZO5m2K6B6jN6R2Np41g/3lKqKUKVyOI679X8z4Yn7RsKUC7ILV9anOK2msQQa 7yqjpffRhPWSGdcw19U1rJpASJgmvUfSdFzdre2/RZZ3xWjb+NkI3S+SiZtPZy7Xy7qO eD/DgmEk4PEm0ayHDtaJzLP20+0KNjilgebEOy1HX3fbZZNLcd829y7HEB+5NQ2w33rO 3l0Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=PD8Sr3pC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id sb42-20020a1709076daa00b0087118b6b5f4si30187727ejc.495.2023.01.23.09.50.47; Mon, 23 Jan 2023 09:51:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=PD8Sr3pC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233232AbjAWRbx (ORCPT + 99 others); Mon, 23 Jan 2023 12:31:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231745AbjAWRbw (ORCPT ); Mon, 23 Jan 2023 12:31:52 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E6EC2DE45 for ; Mon, 23 Jan 2023 09:30:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674495025; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=u/H+JLqqGtjE96rU8y7Ek7KRkRO4zR4fLWrnb7rCy6M=; b=PD8Sr3pCxyN0Xg9nAgRtgppBlK6W0JFgOmD5cpHgWnR9wlmhL4TKXhhHbuUeU+pCXPQskU 1cktcgdWwmdVXkg59k5hlQsM3ArTuadPwxpcYuSvsrZqx2BI4MXZxo0cd6OBhQ/uowSRmI caKCihfTxcA9m3O+BDlydelpF4T6KU0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-567-_9NZHcN4ObeJq18P563qSQ-1; Mon, 23 Jan 2023 12:30:24 -0500 X-MC-Unique: _9NZHcN4ObeJq18P563qSQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D222B18E6C40; Mon, 23 Jan 2023 17:30:23 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6C40C40C2004; Mon, 23 Jan 2023 17:30:22 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig Subject: [PATCH v8 05/10] block: Fix bio_flagged() so that gcc can better optimise it Date: Mon, 23 Jan 2023 17:30:02 +0000 Message-Id: <20230123173007.325544-6-dhowells@redhat.com> In-Reply-To: <20230123173007.325544-1-dhowells@redhat.com> References: <20230123173007.325544-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755836602306886364?= X-GMAIL-MSGID: =?utf-8?q?1755836602306886364?= Fix bio_flagged() so that multiple instances of it, such as: if (bio_flagged(bio, BIO_PAGE_REFFED) || bio_flagged(bio, BIO_PAGE_PINNED)) can be combined by the gcc optimiser into a single test in assembly (arguably, this is a compiler optimisation issue[1]). The missed optimisation stems from bio_flagged() comparing the result of the bitwise-AND to zero. This results in an out-of-line bio_release_page() being compiled to something like: <+0>: mov 0x14(%rdi),%eax <+3>: test $0x1,%al <+5>: jne 0xffffffff816dac53 <+7>: test $0x2,%al <+9>: je 0xffffffff816dac5c <+11>: movzbl %sil,%esi <+15>: jmp 0xffffffff816daba1 <__bio_release_pages> <+20>: jmp 0xffffffff81d0b800 <__x86_return_thunk> However, the test is superfluous as the return type is bool. Removing it results in: <+0>: testb $0x3,0x14(%rdi) <+4>: je 0xffffffff816e4af4 <+6>: movzbl %sil,%esi <+10>: jmp 0xffffffff816dab7c <__bio_release_pages> <+15>: jmp 0xffffffff81d0b7c0 <__x86_return_thunk> instead. Also, the MOVZBL instruction looks unnecessary[2] - I think it's just 're-booling' the mark_dirty parameter. Signed-off-by: David Howells Reviewed-by: Christoph Hellwig cc: Jens Axboe cc: linux-block@vger.kernel.org Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108370 [1] Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108371 [2] Link: https://lore.kernel.org/r/167391056756.2311931.356007731815807265.stgit@warthog.procyon.org.uk/ # v6 --- include/linux/bio.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/bio.h b/include/linux/bio.h index c1da63f6c808..10366b8bdb13 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -227,7 +227,7 @@ static inline void bio_cnt_set(struct bio *bio, unsigned int count) static inline bool bio_flagged(struct bio *bio, unsigned int bit) { - return (bio->bi_flags & (1U << bit)) != 0; + return bio->bi_flags & (1U << bit); } static inline void bio_set_flag(struct bio *bio, unsigned int bit) From patchwork Mon Jan 23 17:30:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 47278 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1737101wrn; Mon, 23 Jan 2023 09:53:53 -0800 (PST) X-Google-Smtp-Source: AMrXdXtFonQkqwjnXYGKjzLUi2oba/s7KYWg1IYNXuw9nMF/XIY6zME1ivCSuRABJfvmTstlBiPy X-Received: by 2002:a17:907:1751:b0:869:c5a4:e1a7 with SMTP id lf17-20020a170907175100b00869c5a4e1a7mr26554950ejc.62.1674496432956; Mon, 23 Jan 2023 09:53:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674496432; cv=none; d=google.com; s=arc-20160816; b=w6qDLCOhUTkN7X88IJyGoUsB3TPrtG8UtHzOVhucWKdy4G9zy1iWmkqjh2UALp0ph1 8krUaD9cHBm0QkRnGp4KGPfKKStEYJo6QDRlwCNcV8BXkbQ9viT+SB40ctXA2W/lYcmH SCFwuIj3T7M0+s5NAyLOCp84p++UkxL7ETwozjp6FYcyRYb8a+iO7FTCml6lW5yrsHJY 6nMQQqMyPn15aG9y5Ngsz87XgnrIsSAsTntd3PIzv+hFeEb84bZNvL60BzqD48hVGD6T V/A72DtqwOvyO+ZSCsdxRMRB08qVeouX4v2Oo9N91XJezi7cswKgR8UzZQiwVf175kM4 grsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ugscixnhJtESpfybZzPz/KvOkM8syYX59ti+iaMHON0=; b=QWZSQCT9CzC46GaPvW/Wa2l0obDd7iMYwJt+KQK5DynxhVTFKqw0MmAfkymWpzM8C6 inGqRd3z/HWQBTLP4dEM6y3AEcz8GoqrVcJxawW8EO+/hfZKLUQ3xKZhX9USk9mNhtCS XTqYkRsAinMaHm3xwtIc75Kbcl6Ktr9NDlSDyHcnwmV5zXkWhbE58iHvptzxxuhJJH/s p6kMYi0S26YFyZomZucN/EnPZMBSUji75rAXrX/lEsVSK0FqlziKLdtwmWwMN3irFWqD 3lulLYfcsNNT7srJFlNPS1StMCupLfvg4EOlZ1YjUAQ1oFyjMtUDlCpNcwH07435XvvO TZ3g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=DmDO2dT0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id xd9-20020a170907078900b007d6105e0acfsi54280202ejb.501.2023.01.23.09.53.29; Mon, 23 Jan 2023 09:53:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=DmDO2dT0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233754AbjAWRdK (ORCPT + 99 others); Mon, 23 Jan 2023 12:33:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233790AbjAWRcl (ORCPT ); Mon, 23 Jan 2023 12:32:41 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 055562E0CA for ; Mon, 23 Jan 2023 09:30:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674495029; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ugscixnhJtESpfybZzPz/KvOkM8syYX59ti+iaMHON0=; b=DmDO2dT0tqy54boTTV3H9WoLiIugtoGd9dITPTlvcpQGlAq00fFRPb2LnZQz6848MiwN1O CqAS5xQ1kW/jG7g+3ukPWtac/WzrV8/JY4dL57ZYMl9RQj3QDHZreMFHEIVNdTODCKWzte A9Xm+f6kKxg/qFlsBNbduGBHoWNdAuY= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-351-DtAAEAOYPxm6ObCRFW-W9g-1; Mon, 23 Jan 2023 12:30:26 -0500 X-MC-Unique: DtAAEAOYPxm6ObCRFW-W9g-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 17F803C14844; Mon, 23 Jan 2023 17:30:26 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9D041492C1B; Mon, 23 Jan 2023 17:30:24 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig Subject: [PATCH v8 06/10] block: Rename BIO_NO_PAGE_REF to BIO_PAGE_REFFED and invert the meaning Date: Mon, 23 Jan 2023 17:30:03 +0000 Message-Id: <20230123173007.325544-7-dhowells@redhat.com> In-Reply-To: <20230123173007.325544-1-dhowells@redhat.com> References: <20230123173007.325544-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755836771423440156?= X-GMAIL-MSGID: =?utf-8?q?1755836771423440156?= From: Christoph Hellwig Replace BIO_NO_PAGE_REF with a BIO_PAGE_REFFED flag that has the inverted meaning is only set when a page reference has been acquired that needs to be relesed by bio_release_pages. Signed-off-by: David Howells cc: Al Viro cc: Jens Axboe cc: Jan Kara cc: Matthew Wilcox cc: Logan Gunthorpe cc: linux-block@vger.kernel.org --- Notes: ver #8) - Don't default to BIO_PAGE_REFFED [hch]. ver #5) - Split from patch that uses iov_iter_extract_pages(). block/bio.c | 2 +- block/blk-map.c | 1 + fs/direct-io.c | 2 ++ fs/iomap/direct-io.c | 1 - include/linux/bio.h | 2 +- include/linux/blk_types.h | 2 +- 6 files changed, 6 insertions(+), 4 deletions(-) diff --git a/block/bio.c b/block/bio.c index a289bbff036f..40c2b01906da 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1198,7 +1198,6 @@ void bio_iov_bvec_set(struct bio *bio, struct iov_iter *iter) bio->bi_io_vec = (struct bio_vec *)iter->bvec; bio->bi_iter.bi_bvec_done = iter->iov_offset; bio->bi_iter.bi_size = size; - bio_set_flag(bio, BIO_NO_PAGE_REF); bio_set_flag(bio, BIO_CLONED); } @@ -1343,6 +1342,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) return 0; } + bio_set_flag(bio, BIO_PAGE_REFFED); do { ret = __bio_iov_iter_get_pages(bio, iter); } while (!ret && iov_iter_count(iter) && !bio_full(bio, 0)); diff --git a/block/blk-map.c b/block/blk-map.c index bc111261fc82..a4ada4389d5e 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -282,6 +282,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (blk_queue_pci_p2pdma(rq->q)) extract_flags |= ITER_ALLOW_P2PDMA; + bio_set_flag(bio, BIO_PAGE_REFFED); while (iov_iter_count(iter)) { struct page **pages, *stack_pages[UIO_FASTIOV]; ssize_t bytes; diff --git a/fs/direct-io.c b/fs/direct-io.c index 03d381377ae1..07810465fc9d 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -403,6 +403,8 @@ dio_bio_alloc(struct dio *dio, struct dio_submit *sdio, bio->bi_end_io = dio_bio_end_aio; else bio->bi_end_io = dio_bio_end_io; + /* for now require references for all pages */ + bio_set_flag(bio, BIO_PAGE_REFFED); sdio->bio = bio; sdio->logical_offset_in_bio = sdio->cur_page_fs_offset; } diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 47db4ead1e74..c0e75900e754 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -202,7 +202,6 @@ static void iomap_dio_zero(const struct iomap_iter *iter, struct iomap_dio *dio, bio->bi_private = dio; bio->bi_end_io = iomap_dio_bio_end_io; - bio_set_flag(bio, BIO_NO_PAGE_REF); __bio_add_page(bio, page, len, 0); iomap_dio_submit_bio(iter, dio, bio, pos); } diff --git a/include/linux/bio.h b/include/linux/bio.h index 10366b8bdb13..805957c99147 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -484,7 +484,7 @@ void zero_fill_bio(struct bio *bio); static inline void bio_release_pages(struct bio *bio, bool mark_dirty) { - if (!bio_flagged(bio, BIO_NO_PAGE_REF)) + if (bio_flagged(bio, BIO_PAGE_REFFED)) __bio_release_pages(bio, mark_dirty); } diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 99be590f952f..7daa261f4f98 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -318,7 +318,7 @@ struct bio { * bio flags */ enum { - BIO_NO_PAGE_REF, /* don't put release vec pages */ + BIO_PAGE_REFFED, /* put pages in bio_release_pages() */ BIO_CLONED, /* doesn't own data */ BIO_BOUNCED, /* bio is a bounce bio */ BIO_QUIET, /* Make BIO Quiet */ From patchwork Mon Jan 23 17:30:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 47279 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1737180wrn; Mon, 23 Jan 2023 09:54:03 -0800 (PST) X-Google-Smtp-Source: AMrXdXuC5rBHETT49J6R5P+YMgbOvxdH4B73nrg0yDKr9Hzu32IDwoX9wly3Ug/oF4b1U2jC02Zp X-Received: by 2002:a17:907:b9da:b0:872:21b2:9a1f with SMTP id xa26-20020a170907b9da00b0087221b29a1fmr26731427ejc.58.1674496442984; Mon, 23 Jan 2023 09:54:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674496442; cv=none; d=google.com; s=arc-20160816; b=eqBi/dVh4x0BfVq0ltaAl7pWWzBlpidVc7kLHLbvcxqQR4aEvI4zK54hs9KCAaftCr BpeTMMxt7Opl0PqqWVkrmqsZadgS+5cTQm+ubcV0GS+tsiAFQUlVFDFrg9vO7uvXFKdp EnvMXDn5OmGQ7Um7CcaCGlbqNpMz2BheheQERKFnuQakmv+YLPdSMgngrbxrd0Ty5Wwt 0q7Eu11WihrLmlLUjQpEtwaWHcrdcW3wMp44ubieFtGmyLXivEwL/3h8ER0MlSUB1jOv nUostv46lSIK3nOBxqESFImeUcCaQCQYOX32qPoUETYMeUbP4UOP5i1Qnf4spiWbrLCY 11YQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=b3uYY2aIEqIpj/7ZvCh9wufNUCWJVpbbJY5BO/kkjEo=; b=gxr63LV6YkZ3wO0KnVpFm995aBYb8b6bYtoB5/HVzh8fnD8n0SZsYRvpJo3QYr2HMD 94ULrKMnjgjiOrUqwiC+ve0h2rvCGBxzCAdGYL6qoAnjASCMt8CpufLx5B8f2iFkPEEQ EJf/BNYYFC55UYcV4ixdKFURgu+33zQrrCJ1dvWEk1kHf0o0MwRIGsCc0szR+PHODZUG 7Ag4B5CguzSIQgYjxCpjv8FKzcZd9JcyVgbnnWcjBssngfArW8ZSqvHB5ITUIqEZsEZ0 BzrOSpdTnUK2J6WwZLToUhEe/w5M25FYh3Uh4qYkzTiVejM0Fi6AL9o03JebJ2FBF6TG 5Hmw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=fgplkPXU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id tb24-20020a1709078b9800b00871d0c97cc3si23329736ejc.856.2023.01.23.09.53.40; Mon, 23 Jan 2023 09:54:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=fgplkPXU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233781AbjAWRdO (ORCPT + 99 others); Mon, 23 Jan 2023 12:33:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233641AbjAWRcm (ORCPT ); Mon, 23 Jan 2023 12:32:42 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 023C42E0E1 for ; Mon, 23 Jan 2023 09:30:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674495032; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=b3uYY2aIEqIpj/7ZvCh9wufNUCWJVpbbJY5BO/kkjEo=; b=fgplkPXUalQ+3T7MUbIHc1WEJqFMr5hsAv5F3Cf+IHIlg/DxFo3n8MbWat10AXYCzRH1No N0Rq++WXKXdGdS/DzGXUVMM46g154y+5rUecd/HuyiLesojAxBKwVPVB1KLMIWtnGxpRwI iSoP/L58apFtDPay+GCV63bLROrnNgM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-400-CefHwYL9NW-bCU8o7VVH-A-1; Mon, 23 Jan 2023 12:30:28 -0500 X-MC-Unique: CefHwYL9NW-bCU8o7VVH-A-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 18F0A101A521; Mon, 23 Jan 2023 17:30:28 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id AA784492C1B; Mon, 23 Jan 2023 17:30:26 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig Subject: [PATCH v8 07/10] block: Switch to pinning pages. Date: Mon, 23 Jan 2023 17:30:04 +0000 Message-Id: <20230123173007.325544-8-dhowells@redhat.com> In-Reply-To: <20230123173007.325544-1-dhowells@redhat.com> References: <20230123173007.325544-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755836782515268799?= X-GMAIL-MSGID: =?utf-8?q?1755836782515268799?= Add BIO_PAGE_PINNED to indicate that the pages in a bio are pinned (FOLL_PIN) and that the pin will need removing. Signed-off-by: David Howells cc: Al Viro cc: Jens Axboe cc: Jan Kara cc: Christoph Hellwig cc: Matthew Wilcox cc: Logan Gunthorpe cc: linux-block@vger.kernel.org Reviewed-by: Christoph Hellwig --- Notes: ver #8) - Move the infrastructure to clean up pinned pages to this patch [hch]. - Put BIO_PAGE_PINNED before BIO_PAGE_REFFED as the latter should probably be removed at some point. FOLL_PIN can then be renumbered first. block/bio.c | 7 ++++--- block/blk.h | 28 ++++++++++++++++++++++++++++ include/linux/bio.h | 3 ++- include/linux/blk_types.h | 1 + 4 files changed, 35 insertions(+), 4 deletions(-) diff --git a/block/bio.c b/block/bio.c index 40c2b01906da..6f98bcfc0c92 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1170,13 +1170,14 @@ bool bio_add_folio(struct bio *bio, struct folio *folio, size_t len, void __bio_release_pages(struct bio *bio, bool mark_dirty) { + unsigned int gup_flags = bio_to_gup_flags(bio); struct bvec_iter_all iter_all; struct bio_vec *bvec; bio_for_each_segment_all(bvec, bio, iter_all) { if (mark_dirty && !PageCompound(bvec->bv_page)) set_page_dirty_lock(bvec->bv_page); - put_page(bvec->bv_page); + page_put_unpin(bvec->bv_page, gup_flags); } } EXPORT_SYMBOL_GPL(__bio_release_pages); @@ -1496,8 +1497,8 @@ void bio_set_pages_dirty(struct bio *bio) * the BIO and re-dirty the pages in process context. * * It is expected that bio_check_pages_dirty() will wholly own the BIO from - * here on. It will run one put_page() against each page and will run one - * bio_put() against the BIO. + * here on. It will run one page_put_unpin() against each page and will run + * one bio_put() against the BIO. */ static void bio_dirty_fn(struct work_struct *work); diff --git a/block/blk.h b/block/blk.h index 4c3b3325219a..294044d696e0 100644 --- a/block/blk.h +++ b/block/blk.h @@ -425,6 +425,34 @@ int bio_add_hw_page(struct request_queue *q, struct bio *bio, struct page *page, unsigned int len, unsigned int offset, unsigned int max_sectors, bool *same_page); +/* + * Set the cleanup mode for a bio from an iterator and the extraction flags. + */ +static inline void bio_set_cleanup_mode(struct bio *bio, struct iov_iter *iter) +{ + unsigned int cleanup_mode = iov_iter_extract_mode(iter); + + if (cleanup_mode & FOLL_GET) + bio_set_flag(bio, BIO_PAGE_REFFED); + if (cleanup_mode & FOLL_PIN) + bio_set_flag(bio, BIO_PAGE_PINNED); +} + +static inline unsigned int bio_to_gup_flags(struct bio *bio) +{ + return (bio_flagged(bio, BIO_PAGE_REFFED) ? FOLL_GET : 0) | + (bio_flagged(bio, BIO_PAGE_PINNED) ? FOLL_PIN : 0); +} + +/* + * Clean up a page appropriately, where the page may be pinned, may have a + * ref taken on it or neither. + */ +static inline void bio_release_page(struct bio *bio, struct page *page) +{ + page_put_unpin(page, bio_to_gup_flags(bio)); +} + struct request_queue *blk_alloc_queue(int node_id); int disk_scan_partitions(struct gendisk *disk, fmode_t mode, void *owner); diff --git a/include/linux/bio.h b/include/linux/bio.h index 805957c99147..b2c09997d79c 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -484,7 +484,8 @@ void zero_fill_bio(struct bio *bio); static inline void bio_release_pages(struct bio *bio, bool mark_dirty) { - if (bio_flagged(bio, BIO_PAGE_REFFED)) + if (bio_flagged(bio, BIO_PAGE_REFFED) || + bio_flagged(bio, BIO_PAGE_PINNED)) __bio_release_pages(bio, mark_dirty); } diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 7daa261f4f98..a0e339ff3d09 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -318,6 +318,7 @@ struct bio { * bio flags */ enum { + BIO_PAGE_PINNED, /* Unpin pages in bio_release_pages() */ BIO_PAGE_REFFED, /* put pages in bio_release_pages() */ BIO_CLONED, /* doesn't own data */ BIO_BOUNCED, /* bio is a bounce bio */ From patchwork Mon Jan 23 17:30:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 47276 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1736879wrn; Mon, 23 Jan 2023 09:53:19 -0800 (PST) X-Google-Smtp-Source: AMrXdXte4gdKmbvR1DUehlidey1d1c2TE2k8+5Uo2wFqFKBP2SRq3Vu4q9+vgToWgBdpHivQQYch X-Received: by 2002:a05:6402:380d:b0:49e:5ffd:5be2 with SMTP id es13-20020a056402380d00b0049e5ffd5be2mr18860247edb.40.1674496399176; Mon, 23 Jan 2023 09:53:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674496399; cv=none; d=google.com; s=arc-20160816; b=TM5TxVotkxRVsgawhdAi4ZFATWBFmsAtKNU2I4eMoUBKQRph1Fc6GyeyEZlN770mKu xqAR1vyDX+X9BgV0ag6jtST5mZzUpegMGdCRih6EIsTw5te0tU9P12p9IvqTUzPlhJkT uN8+yi3OunQoxB+WaQkM3EzUJ+Bp6YRXqs77GuNK1c5fydgBIetFbEjsg3MXq/JoU5Ng aXyktyLAgNKwXWHVpWN89FneNTsx8vBbI6OyE9agUelvjCTK19M1EzQGZYXcqEV+xE2v 70+SKjT0v6U7rN6NuMI/83kF4Bb0h56yzsD9xYqbpXk83dAkWJZsfn1r1nCtvf2u12tE n4YA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=8R74HsEjH9XlBjVpyUw0TUK8fGpl//j53c7hN7ZbLp4=; b=fubOzBFle49mRn2o7ikFnyTwjkEAD43Tyhg+Qiapyx5zT0XCvTTGvN+XML5G5kqYSQ L/BRG9XE+OPNQrUNN27Jx+Raxc0UQovJXURBbICI0UgC0XYlad67jslA1o0m3pgGMihJ q5GA3fE4zjtbV4MJ5YKr901zmpADJHmb/cs4e7nFP11fU0pvrF0YuUlXH9NBy9h9yNXd j9H2BHI9/Lvafx3+V9m/UJ2MleNArVnAo5pJ4nK+KY7HAK6X0Eh66fjWIqN/9nUXGORc QRBxGpLcqTMa/1aEJLPqbE6B9CGUgdiWzwnDZ9MIAg1xi/Jfu7DAy180SSeyzpYJWjpK PxTA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=HVxQfEdc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h10-20020a50c38a000000b00499b2eb6a0csi966876edf.348.2023.01.23.09.52.55; Mon, 23 Jan 2023 09:53:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=HVxQfEdc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232498AbjAWRcy (ORCPT + 99 others); Mon, 23 Jan 2023 12:32:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38302 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232777AbjAWRck (ORCPT ); Mon, 23 Jan 2023 12:32:40 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2A4D42E0F7 for ; Mon, 23 Jan 2023 09:30:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674495034; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8R74HsEjH9XlBjVpyUw0TUK8fGpl//j53c7hN7ZbLp4=; b=HVxQfEdca+HkaTNg2kgjCfV5n5o1L3BuEOnFIn5yoQcLg8te8WOjBAZnXvrQABLyVZ1876 Qxj42jgtiRkMoFlcu+T061eF88uOT60tgvjNwqmsQhfz5btQaLaKytFbmCcbA0ZqzhTdQw 00taPFLqVBJ65cLUMLIzNothkubBKrU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-225-4jqdVvyUNmqUbKHciUyroQ-1; Mon, 23 Jan 2023 12:30:30 -0500 X-MC-Unique: 4jqdVvyUNmqUbKHciUyroQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1BF023806713; Mon, 23 Jan 2023 17:30:30 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id AAF44492C1B; Mon, 23 Jan 2023 17:30:28 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig Subject: [PATCH v8 08/10] block: Convert bio_iov_iter_get_pages to use iov_iter_extract_pages Date: Mon, 23 Jan 2023 17:30:05 +0000 Message-Id: <20230123173007.325544-9-dhowells@redhat.com> In-Reply-To: <20230123173007.325544-1-dhowells@redhat.com> References: <20230123173007.325544-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755836736271046424?= X-GMAIL-MSGID: =?utf-8?q?1755836736271046424?= This will pin pages or leave them unaltered rather than getting a ref on them as appropriate to the iterator. The pages need to be pinned for DIO rather than having refs taken on them to prevent VM copy-on-write from malfunctioning during a concurrent fork() (the result of the I/O could otherwise end up being affected by/visible to the child process). Signed-off-by: David Howells cc: Al Viro cc: Jens Axboe cc: Jan Kara cc: Christoph Hellwig cc: Matthew Wilcox cc: Logan Gunthorpe cc: linux-block@vger.kernel.org Reviewed-by: Christoph Hellwig --- Notes: ver #8) - Split the patch up a bit [hch]. - We should only be using pinned/non-pinned pages and not ref'd pages, so adjust the comments appropriately. ver #7) - Don't treat BIO_PAGE_REFFED/PINNED as being the same as FOLL_GET/PIN. ver #5) - Transcribe the FOLL_* flags returned by iov_iter_extract_pages() to BIO_* flags and got rid of bi_cleanup_mode. - Replaced BIO_NO_PAGE_REF to BIO_PAGE_REFFED in the preceding patch. block/bio.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/block/bio.c b/block/bio.c index 6f98bcfc0c92..6dc54bf3ed27 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1213,7 +1213,7 @@ static int bio_iov_add_page(struct bio *bio, struct page *page, } if (same_page) - put_page(page); + bio_release_page(bio, page); return 0; } @@ -1227,7 +1227,7 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, queue_max_zone_append_sectors(q), &same_page) != len) return -EINVAL; if (same_page) - put_page(page); + bio_release_page(bio, page); return 0; } @@ -1238,10 +1238,10 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, * @bio: bio to add pages to * @iter: iov iterator describing the region to be mapped * - * Pins pages from *iter and appends them to @bio's bvec array. The - * pages will have to be released using put_page() when done. - * For multi-segment *iter, this function only adds pages from the - * next non-empty segment of the iov iterator. + * Extracts pages from *iter and appends them to @bio's bvec array. The pages + * will have to be cleaned up in the way indicated by the BIO_PAGE_PINNED flag. + * For a multi-segment *iter, this function only adds pages from the next + * non-empty segment of the iov iterator. */ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) { @@ -1273,9 +1273,9 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) * result to ensure the bio's total size is correct. The remainder of * the iov data will be picked up in the next bio iteration. */ - size = iov_iter_get_pages(iter, pages, - UINT_MAX - bio->bi_iter.bi_size, - nr_pages, &offset, extract_flags); + size = iov_iter_extract_pages(iter, &pages, + UINT_MAX - bio->bi_iter.bi_size, + nr_pages, extract_flags, &offset); if (unlikely(size <= 0)) return size ? size : -EFAULT; @@ -1308,7 +1308,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) iov_iter_revert(iter, left); out: while (i < nr_pages) - put_page(pages[i++]); + bio_release_page(bio, pages[i++]); return ret; } @@ -1343,7 +1343,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) return 0; } - bio_set_flag(bio, BIO_PAGE_REFFED); + bio_set_cleanup_mode(bio, iter); do { ret = __bio_iov_iter_get_pages(bio, iter); } while (!ret && iov_iter_count(iter) && !bio_full(bio, 0)); From patchwork Mon Jan 23 17:30:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 47277 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1736881wrn; Mon, 23 Jan 2023 09:53:19 -0800 (PST) X-Google-Smtp-Source: AMrXdXtZTEnsZ4rr7q+8kP59SqptZzAXrLxPmgqoNIrHXfZ5Ac6mSo5Ox/mjDTAqtOOwLf9hiI/o X-Received: by 2002:a17:907:a804:b0:7a6:5b50:a32c with SMTP id vo4-20020a170907a80400b007a65b50a32cmr30752043ejc.12.1674496399427; Mon, 23 Jan 2023 09:53:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674496399; cv=none; d=google.com; s=arc-20160816; b=lssRqzsrW5rFzjU2npDyqpGNr/ynXIAsINSFbsNRKXwsHtR69EOBIMRtUEkF5obitz ThRxIa8/hsHzYUj8ZzrXdlltpbTrdMEYffQM2zJxJkXWOCGVmLMR6HdZ8UkC3jCW2lS0 PKC9SnSF3krixJvV0w+brGIDyzry/Z3ytpepuvPBvt8FzE60mBJaDcmOl+BhOwculVxk +24PiTMjOIbP5GYJXRdBxafijvuvAQQtt7EQXoEunA4zuCcdlV2i1+kW3+GD24oFKXZQ qB/FI9LtvMIk3198nHOBY1sjmdlZKnNzL7l5xpFo3M8E6GKswoUBbgBeeU6PElDu+ZFj kpVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=zi65h8kb+iIzTHBTD5yjY8npgp3PwB4fOrnEoPzlk1s=; b=mo7xhRb5Ur5uzoVUxEjWksEOVNHkvjyKNqbOAoOUUCX56h84ONB1TlA9DL0us5uqdi USZYeevcpAbFBoxk/xLkX60by1lwOSHYusgjl+yKHmyu0ihm4qfEPp3/D33eFuLyrIpB 4KomHn+I8qymf/vXDBldQfnFSM5+rDOJ1Au20E/lwB5NOj/Gm4eX2cRFe6HlJXcHvJ5p hsfuxRR/jzh2tYoq02Gif3hnFWXmLYWghlW3Z1Wjm28PeXZqT4EhvhJGFV1usJghY4Fa 8VpZB/p/V31SZatq5uLdauMY6x0rkVizEAjQtRap27DCbOTegTDjlTaAOKUwqLmKJ/hn NVtg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="iUoscgl/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id sb25-20020a1709076d9900b0086f19377a7bsi26298406ejc.502.2023.01.23.09.52.56; Mon, 23 Jan 2023 09:53:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="iUoscgl/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233752AbjAWRc5 (ORCPT + 99 others); Mon, 23 Jan 2023 12:32:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233158AbjAWRck (ORCPT ); Mon, 23 Jan 2023 12:32:40 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EF7502ED4B for ; Mon, 23 Jan 2023 09:30:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674495038; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zi65h8kb+iIzTHBTD5yjY8npgp3PwB4fOrnEoPzlk1s=; b=iUoscgl/quBZkUH7UL6Bye19nNd3XnfGcYbpfvont0ZYfsoTb0AkB+tGL/HNdkZtlnN9aa GxO2CMnE/Zv5DqmGtxprRf1YsSXlHU+zh9AwzScko7LHzB4nsnA0ZxVPkP5UpDwcXt7PQo SfOyNiwtWl6MvwH85bxPcpYbiikm4yM= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-659-FeZB6NtzO5eUR41mHESISA-1; Mon, 23 Jan 2023 12:30:33 -0500 X-MC-Unique: FeZB6NtzO5eUR41mHESISA-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4FD0B1C0896F; Mon, 23 Jan 2023 17:30:32 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id E0742492C1B; Mon, 23 Jan 2023 17:30:30 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig Subject: [PATCH v8 09/10] block: convert bio_map_user_iov to use iov_iter_extract_pages Date: Mon, 23 Jan 2023 17:30:06 +0000 Message-Id: <20230123173007.325544-10-dhowells@redhat.com> In-Reply-To: <20230123173007.325544-1-dhowells@redhat.com> References: <20230123173007.325544-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755836736684000008?= X-GMAIL-MSGID: =?utf-8?q?1755836736684000008?= This will pin pages or leave them unaltered rather than getting a ref on them as appropriate to the iterator. The pages need to be pinned for DIO rather than having refs taken on them to prevent VM copy-on-write from malfunctioning during a concurrent fork() (the result of the I/O could otherwise end up being visible to/affected by the child process). Signed-off-by: David Howells cc: Al Viro cc: Jens Axboe cc: Jan Kara cc: Christoph Hellwig cc: Matthew Wilcox cc: Logan Gunthorpe cc: linux-block@vger.kernel.org Reviewed-by: Christoph Hellwig --- Notes: ver #8) - Split the patch up a bit [hch]. - We should only be using pinned/non-pinned pages and not ref'd pages, so adjust the comments appropriately. ver #7) - Don't treat BIO_PAGE_REFFED/PINNED as being the same as FOLL_GET/PIN. ver #5) - Transcribe the FOLL_* flags returned by iov_iter_extract_pages() to BIO_* flags and got rid of bi_cleanup_mode. - Replaced BIO_NO_PAGE_REF to BIO_PAGE_REFFED in the preceding patch. block/blk-map.c | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/block/blk-map.c b/block/blk-map.c index a4ada4389d5e..b9b36af3c3f7 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -282,21 +282,19 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (blk_queue_pci_p2pdma(rq->q)) extract_flags |= ITER_ALLOW_P2PDMA; - bio_set_flag(bio, BIO_PAGE_REFFED); + bio_set_cleanup_mode(bio, iter); while (iov_iter_count(iter)) { - struct page **pages, *stack_pages[UIO_FASTIOV]; + struct page *stack_pages[UIO_FASTIOV]; + struct page **pages = stack_pages; ssize_t bytes; size_t offs; int npages; - if (nr_vecs <= ARRAY_SIZE(stack_pages)) { - pages = stack_pages; - bytes = iov_iter_get_pages(iter, pages, LONG_MAX, - nr_vecs, &offs, extract_flags); - } else { - bytes = iov_iter_get_pages_alloc(iter, &pages, - LONG_MAX, &offs, extract_flags); - } + if (nr_vecs > ARRAY_SIZE(stack_pages)) + pages = NULL; + + bytes = iov_iter_extract_pages(iter, &pages, LONG_MAX, + nr_vecs, extract_flags, &offs); if (unlikely(bytes <= 0)) { ret = bytes ? bytes : -EFAULT; goto out_unmap; @@ -318,7 +316,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, if (!bio_add_hw_page(rq->q, bio, page, n, offs, max_sectors, &same_page)) { if (same_page) - put_page(page); + bio_release_page(bio, page); break; } @@ -330,7 +328,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, * release the pages we didn't map into the bio, if any */ while (j < npages) - put_page(pages[j++]); + bio_release_page(bio, pages[j++]); if (pages != stack_pages) kvfree(pages); /* couldn't stuff something into bio? */ From patchwork Mon Jan 23 17:30:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 47280 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1737265wrn; Mon, 23 Jan 2023 09:54:11 -0800 (PST) X-Google-Smtp-Source: AMrXdXuwyY6jbWqANBDS7NIU4O5voz5jUzyoKpu77pPDcCWp7DuK/GPvpEWoDr+DlnWDC2TaF2x/ X-Received: by 2002:a17:907:b9cd:b0:877:7dba:f55c with SMTP id xa13-20020a170907b9cd00b008777dbaf55cmr17048230ejc.8.1674496451196; Mon, 23 Jan 2023 09:54:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674496451; cv=none; d=google.com; s=arc-20160816; b=LOOhsBphS4+RXpXMEr77Ph2FOHyI8nPTD2n5uqj0kxhzAhOU+szG6cNKDMlOYS7D/r 7PY5aTy74/wSayCGr8eStd9EPVC3kvb0Y1XVMOvfAz/yhyxLD6XO7nkoTVvgHp8KGQ4l D97V+P7IYXRzCTGXD36ZhNtViD+lS1x1UIhYIOdCI93mnx7I9vMAPAYlNl36cSyZqm6K JeelB2Eue+8fjGI6qZ+JeaZB6h8qrBZy8aS7Vp8gvhnJRtAw7dHaHM9ngLDqhvpYpl/L H3gCfpt81tasEfRgvtOYMrSz75Z4DspQlUC8xsmZ4UmuLBwQ7OJWWkYuLu9oHSCXA0/u cC+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JKkpA1QDyFR8sFnII4BY3UsDy9lIqmuw1mdMwZBIzUs=; b=mb6+0NoVd1Es8oeQdolh3kI5+oNlxFZCh06YilFKxdz2BwLT+rGaEvvqFVI9bDEbIj tikmGeeHjSu23XNg6TdA/HPP6RND/fyG2EVeTqbtm8GhzwN1AlgJAZ8HtZFr15Ms1ddS nrE0bCvW5NsmOzibRzv+mq2pU/kemAlmyIKyZEhlsJZKrWA7SEM5Ix3S5QsYt+LzztPH U6Y5pNGWiAu+1huHNOSQM0DJzFBJywNjiXHARAxbu+pWkDEBqomRcRmAvJirJ4wxwvFe m12acHljSgFkqWCJbLv7AA4ZLxgdR7YzGmv5RnrhbSC2ukEODVR9jSj4EgvUWQz2ttNm CotA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Fhc170Tj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qa15-20020a170907868f00b008704dba09ddsi27341421ejc.291.2023.01.23.09.53.47; Mon, 23 Jan 2023 09:54:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Fhc170Tj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233795AbjAWRdX (ORCPT + 99 others); Mon, 23 Jan 2023 12:33:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39044 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233759AbjAWRco (ORCPT ); Mon, 23 Jan 2023 12:32:44 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6FB92ED54 for ; Mon, 23 Jan 2023 09:30:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674495046; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JKkpA1QDyFR8sFnII4BY3UsDy9lIqmuw1mdMwZBIzUs=; b=Fhc170TjQpEQxFHnnjJobMJ8D9gvK14j7Uggie3Qj3N6IfCX7qrTxr4PS5Dvw1duZzy9OL Y3wajXvUf1y1BDybT6OxRGGU/q5CFhPRvGRpf5bV8uXy88Tem6B9YMcnQl77Hp4iqjIOUb dp5pp1FXDCrjHGTsIbysbiEM+yW5BFg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-639-Tr5hk-wmOh-yYRPR03ZbOg-1; Mon, 23 Jan 2023 12:30:35 -0500 X-MC-Unique: Tr5hk-wmOh-yYRPR03ZbOg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6A0DF858F09; Mon, 23 Jan 2023 17:30:34 +0000 (UTC) Received: from warthog.procyon.org.uk.com (unknown [10.33.36.97]) by smtp.corp.redhat.com (Postfix) with ESMTP id E23071121330; Mon, 23 Jan 2023 17:30:32 +0000 (UTC) From: David Howells To: Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jens Axboe , Jan Kara , Jeff Layton , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig , linux-mm@kvack.org Subject: [PATCH v8 10/10] mm: Renumber FOLL_PIN and FOLL_GET down Date: Mon, 23 Jan 2023 17:30:07 +0000 Message-Id: <20230123173007.325544-11-dhowells@redhat.com> In-Reply-To: <20230123173007.325544-1-dhowells@redhat.com> References: <20230123173007.325544-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755836790686391042?= X-GMAIL-MSGID: =?utf-8?q?1755836790686391042?= Renumber FOLL_PIN and FOLL_GET down to bit 0 and 1 respectively so that they are coincidentally the same as BIO_PAGE_PINNED and BIO_PAGE_REFFED and also so that they can be stored in the bottom two bits of a page pointer (something I'm looking at for zerocopy socket fragments). (Note that BIO_PAGE_REFFED should probably be got rid of at some point, hence why FOLL_PIN is at 0.) Also renumber down the other FOLL_* flags to close the gaps. Signed-off-by: David Howells cc: Al Viro cc: Christoph Hellwig cc: Matthew Wilcox cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org Reviewed-by: Christoph Hellwig Reviewed-by: John Hubbard --- Notes: ver #8) - Put FOLL_PIN at bit 0 and FOLL_GET at bit 1 to match BIO_PAGE_*. - Renumber the remaining flags down to fill in the gap. include/linux/mm.h | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 3de9d88f8524..c95bc4f77e8f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3074,26 +3074,28 @@ static inline vm_fault_t vmf_error(int err) struct page *follow_page(struct vm_area_struct *vma, unsigned long address, unsigned int foll_flags); -#define FOLL_WRITE 0x01 /* check pte is writable */ -#define FOLL_TOUCH 0x02 /* mark page accessed */ -#define FOLL_GET 0x04 /* do get_page on page */ -#define FOLL_DUMP 0x08 /* give error on hole if it would be zero */ -#define FOLL_FORCE 0x10 /* get_user_pages read/write w/o permission */ -#define FOLL_NOWAIT 0x20 /* if a disk transfer is needed, start the IO +#define FOLL_PIN 0x01 /* pages must be released via unpin_user_page */ +#define FOLL_GET 0x02 /* do get_page on page (equivalent to BIO_FOLL_GET) */ +#define FOLL_WRITE 0x04 /* check pte is writable */ +#define FOLL_TOUCH 0x08 /* mark page accessed */ +#define FOLL_DUMP 0x10 /* give error on hole if it would be zero */ +#define FOLL_FORCE 0x20 /* get_user_pages read/write w/o permission */ +#define FOLL_NOWAIT 0x40 /* if a disk transfer is needed, start the IO * and return without waiting upon it */ #define FOLL_NOFAULT 0x80 /* do not fault in pages */ #define FOLL_HWPOISON 0x100 /* check page is hwpoisoned */ -#define FOLL_TRIED 0x800 /* a retry, previous pass started an IO */ -#define FOLL_REMOTE 0x2000 /* we are working on non-current tsk/mm */ -#define FOLL_ANON 0x8000 /* don't do file mappings */ -#define FOLL_LONGTERM 0x10000 /* mapping lifetime is indefinite: see below */ -#define FOLL_SPLIT_PMD 0x20000 /* split huge pmd before returning */ -#define FOLL_PIN 0x40000 /* pages must be released via unpin_user_page */ -#define FOLL_FAST_ONLY 0x80000 /* gup_fast: prevent fall-back to slow gup */ -#define FOLL_PCI_P2PDMA 0x100000 /* allow returning PCI P2PDMA pages */ -#define FOLL_INTERRUPTIBLE 0x200000 /* allow interrupts from generic signals */ +#define FOLL_TRIED 0x200 /* a retry, previous pass started an IO */ +#define FOLL_REMOTE 0x400 /* we are working on non-current tsk/mm */ +#define FOLL_ANON 0x800 /* don't do file mappings */ +#define FOLL_LONGTERM 0x1000 /* mapping lifetime is indefinite: see below */ +#define FOLL_SPLIT_PMD 0x2000 /* split huge pmd before returning */ +#define FOLL_FAST_ONLY 0x4000 /* gup_fast: prevent fall-back to slow gup */ +#define FOLL_PCI_P2PDMA 0x8000 /* allow returning PCI P2PDMA pages */ +#define FOLL_INTERRUPTIBLE 0x10000 /* allow interrupts from generic signals */ /* + * Note that FOLL_PIN is sorted to bit 0 to be coincident with BIO_PAGE_PINNED. + * * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each * other. Here is what they mean, and how to use them: *