From patchwork Sat Jan 7 00:34:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 40287 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1096466wrt; Fri, 6 Jan 2023 16:37:24 -0800 (PST) X-Google-Smtp-Source: AMrXdXul37ODAEqxWqdeLRfgnD5E93i3mOKUXx+sXKnFoyLxAwkSfCpMalRCXh3TPhBpRIRgSVJx X-Received: by 2002:a05:6a20:c78d:b0:a3:878d:c126 with SMTP id hk13-20020a056a20c78d00b000a3878dc126mr76101062pzb.42.1673051843898; Fri, 06 Jan 2023 16:37:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673051843; cv=none; d=google.com; s=arc-20160816; b=n375xU9HSxtLrZzvGtWd9iqsKIlRN1GjlaESGHw6B5PftJIIzUvZBPTvHeQI8FEXOh 1w9koaVMpfJZmP4Tnxybunol43a4iw8xOv4JxHHlBNZmL2zJ+qIqefEAYRJjOvGzsbEq rmCx9n+5PbtDwbKu3BFqy3l8PGGT9Ql/e4V9bOM8VT0Xe8WyQYqnYpXII3mWejJBydYz vswxUg4cWlxnSsIBRB2oty6kSjA37WfNALWF4h1zU+JD15XRu+fop4I7F5M0a0wB1Px4 D7ACs6LJquI9FviRj0CaUQnSGkNHCSSHhGVl8yYrr0q4eNiGSgVvU6GepQXnj1csw6sj tNiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:cc:to:from :subject:organization:dkim-signature; bh=OhwJrX+/uggIQK7yrDCabQsdugOCpi4Tzwg2wYNTyQg=; b=yJulg10X9Pvt280j9haCf6XKnVF6KOrTqP6ZQuOpiCy7tbd81Iheffbr9dO39SMRQn K0MZ2pu9ScKZ1mfKbQcHmM9PiuH6I5ATo0/NvkgYVHd0CIWMixOLHCA3J3fqxAL4q2+Q 3IBFlD36BzVhZ7voO1gzzrPJvKtAP3Ukq3XoNu47Ze6PyAkX8tk/l5QMSP8My2OrLM0j thmOD6ZGGiTJA3Ps7qzFH5LzQl58JDiHcLk7Hs58EFTDxrTCy4cY2YK2XjML4gTMUTeJ 2nyi6cW/2mtzFW+I6Lm+AMDGtbI6NgHQptBi9vH7939/TWdECIIFjhN8/ENd5wE4JEcp 2CNw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="GeEcMfq/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r3-20020a056a00216300b00574b46d3b26si2497564pff.334.2023.01.06.16.37.10; Fri, 06 Jan 2023 16:37:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="GeEcMfq/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230201AbjAGAgc (ORCPT + 99 others); Fri, 6 Jan 2023 19:36:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43620 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236584AbjAGAfU (ORCPT ); Fri, 6 Jan 2023 19:35:20 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EBB6A84BC9 for ; Fri, 6 Jan 2023 16:34:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1673051668; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OhwJrX+/uggIQK7yrDCabQsdugOCpi4Tzwg2wYNTyQg=; b=GeEcMfq/mZi+5kdD4iJvuPJpg/Ri5mIo8fGNRLeMQ3sc6HI3XPrNV9twCpqUWQYOz0jTXh Z0XxsZHSL4FJd2SKZhBBcl8POcbvnv1xsqW4InPSIx8KgpeDUtTqeGCb2cUyzgTekrxZCw v9YXOKRgarvP9czZ5peJw2C5KVImY7o= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-26-Cddw6XzOOKa2Yet_lD6Qbw-1; Fri, 06 Jan 2023 19:34:23 -0500 X-MC-Unique: Cddw6XzOOKa2Yet_lD6Qbw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 640A7802D19; Sat, 7 Jan 2023 00:34:23 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.87]) by smtp.corp.redhat.com (Postfix) with ESMTP id 17AD1492B07; Sat, 7 Jan 2023 00:34:21 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH v4 7/7] iov_iter, block: Make bio structs pin pages rather than ref'ing if appropriate From: David Howells To: Al Viro Cc: Jens Axboe , Christoph Hellwig , Matthew Wilcox , Logan Gunthorpe , dhowells@redhat.com, Christoph Hellwig , Matthew Wilcox , Jens Axboe , Jeff Layton , Logan Gunthorpe , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 07 Jan 2023 00:34:21 +0000 Message-ID: <167305166150.1521586.10220949115402059720.stgit@warthog.procyon.org.uk> In-Reply-To: <167305160937.1521586.133299343565358971.stgit@warthog.procyon.org.uk> References: <167305160937.1521586.133299343565358971.stgit@warthog.procyon.org.uk> User-Agent: StGit/1.5 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754322010432749495?= X-GMAIL-MSGID: =?utf-8?q?1754322010432749495?= Convert the block layer's bio code to use iov_iter_extract_pages() instead of iov_iter_get_pages(). This will pin pages or leave them unaltered rather than getting a ref on them as appropriate to the source iterator. A field, bi_cleanup_mode, is added to the bio struct that gets set by iov_iter_extract_pages() with FOLL_* flags indicating what cleanup is necessary. FOLL_GET -> put_page(), FOLL_PIN -> unpin_user_page(). Other flags could also be used in future. Newly allocated bio structs have bi_cleanup_mode set to FOLL_GET to indicate that attached pages are ref'd by default. Cloning sets it to 0. __bio_iov_iter_get_pages() overrides it to what iov_iter_extract_pages() indicates. [!] Note that this is tested a bit with ext4, but nothing else. Signed-off-by: David Howells cc: Al Viro cc: Jens Axboe cc: Christoph Hellwig cc: Matthew Wilcox cc: Logan Gunthorpe Signed-off-by: David Howells Signed-off-by: David Howells --- block/bio.c | 47 +++++++++++++++++++++++++++++++++------------ include/linux/blk_types.h | 1 + 2 files changed, 35 insertions(+), 13 deletions(-) diff --git a/block/bio.c b/block/bio.c index 5f96fcae3f75..eafcbeba0bab 100644 --- a/block/bio.c +++ b/block/bio.c @@ -243,6 +243,11 @@ static void bio_free(struct bio *bio) * Users of this function have their own bio allocation. Subsequently, * they must remember to pair any call to bio_init() with bio_uninit() * when IO has completed, or when the bio is released. + * + * We set the initial assumption that pages attached to the bio will be + * released with put_page() by setting bi_cleanup_mode to FOLL_GET, but this + * should be set to FOLL_PIN if the page should be unpinned instead; if the + * pages should not be put or unpinned, this should be set to 0 */ void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table, unsigned short max_vecs, blk_opf_t opf) @@ -274,6 +279,7 @@ void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table, #ifdef CONFIG_BLK_DEV_INTEGRITY bio->bi_integrity = NULL; #endif + bio->bi_cleanup_mode = FOLL_GET; bio->bi_vcnt = 0; atomic_set(&bio->__bi_remaining, 1); @@ -302,6 +308,7 @@ void bio_reset(struct bio *bio, struct block_device *bdev, blk_opf_t opf) { bio_uninit(bio); memset(bio, 0, BIO_RESET_BYTES); + bio->bi_cleanup_mode = FOLL_GET; atomic_set(&bio->__bi_remaining, 1); bio->bi_bdev = bdev; if (bio->bi_bdev) @@ -814,6 +821,7 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp) bio_set_flag(bio, BIO_CLONED); bio->bi_ioprio = bio_src->bi_ioprio; bio->bi_iter = bio_src->bi_iter; + bio->bi_cleanup_mode = 0; if (bio->bi_bdev) { if (bio->bi_bdev == bio_src->bi_bdev && @@ -1168,6 +1176,18 @@ bool bio_add_folio(struct bio *bio, struct folio *folio, size_t len, return bio_add_page(bio, &folio->page, len, off) > 0; } +/* + * Clean up a page according to the mode indicated by iov_iter_extract_pages(), + * where the page is may be pinned or may have a ref taken on it. + */ +static void bio_release_page(struct bio *bio, struct page *page) +{ + if (bio->bi_cleanup_mode & FOLL_PIN) + unpin_user_page(page); + if (bio->bi_cleanup_mode & FOLL_GET) + put_page(page); +} + void __bio_release_pages(struct bio *bio, bool mark_dirty) { struct bvec_iter_all iter_all; @@ -1176,7 +1196,7 @@ void __bio_release_pages(struct bio *bio, bool mark_dirty) bio_for_each_segment_all(bvec, bio, iter_all) { if (mark_dirty && !PageCompound(bvec->bv_page)) set_page_dirty_lock(bvec->bv_page); - put_page(bvec->bv_page); + bio_release_page(bio, bvec->bv_page); } } EXPORT_SYMBOL_GPL(__bio_release_pages); @@ -1213,7 +1233,7 @@ static int bio_iov_add_page(struct bio *bio, struct page *page, } if (same_page) - put_page(page); + bio_release_page(bio, page); return 0; } @@ -1227,7 +1247,7 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, queue_max_zone_append_sectors(q), &same_page) != len) return -EINVAL; if (same_page) - put_page(page); + bio_release_page(bio, page); return 0; } @@ -1238,10 +1258,10 @@ static int bio_iov_add_zone_append_page(struct bio *bio, struct page *page, * @bio: bio to add pages to * @iter: iov iterator describing the region to be mapped * - * Pins pages from *iter and appends them to @bio's bvec array. The - * pages will have to be released using put_page() when done. - * For multi-segment *iter, this function only adds pages from the - * next non-empty segment of the iov iterator. + * Pins pages from *iter and appends them to @bio's bvec array. The pages will + * have to be released using put_page() or unpin_user_page() when done as + * according to bi_cleanup_mode. For multi-segment *iter, this function only + * adds pages from the next non-empty segment of the iov iterator. */ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) { @@ -1273,9 +1293,10 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) * result to ensure the bio's total size is correct. The remainder of * the iov data will be picked up in the next bio iteration. */ - size = iov_iter_get_pages(iter, pages, - UINT_MAX - bio->bi_iter.bi_size, - nr_pages, &offset, gup_flags); + size = iov_iter_extract_pages(iter, &pages, + UINT_MAX - bio->bi_iter.bi_size, + nr_pages, gup_flags, + &offset, &bio->bi_cleanup_mode); if (unlikely(size <= 0)) return size ? size : -EFAULT; @@ -1308,7 +1329,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) iov_iter_revert(iter, left); out: while (i < nr_pages) - put_page(pages[i++]); + bio_release_page(bio, pages[i++]); return ret; } @@ -1489,8 +1510,8 @@ void bio_set_pages_dirty(struct bio *bio) * the BIO and re-dirty the pages in process context. * * It is expected that bio_check_pages_dirty() will wholly own the BIO from - * here on. It will run one put_page() against each page and will run one - * bio_put() against the BIO. + * here on. It will run one put_page() or unpin_user_page() against each page + * and will run one bio_put() against the BIO. */ static void bio_dirty_fn(struct work_struct *work); diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 99be590f952f..883f873a01ef 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -289,6 +289,7 @@ struct bio { #endif }; + unsigned int bi_cleanup_mode; /* How to clean up pages */ unsigned short bi_vcnt; /* how many bio_vec's */ /*