From patchwork Tue Feb 14 08:37:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 56762 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2845813wrn; Tue, 14 Feb 2023 00:44:25 -0800 (PST) X-Google-Smtp-Source: AK7set/yk3xzhGgOTXGbBgFVip2S/ts8uHeZc0uUs0ZjsrA6b3RXrimv4ylct0rR4w8w3PWJPsxn X-Received: by 2002:a17:906:470a:b0:8af:f93f:2434 with SMTP id y10-20020a170906470a00b008aff93f2434mr2793735ejq.38.1676364265116; Tue, 14 Feb 2023 00:44:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676364265; cv=none; d=google.com; s=arc-20160816; b=A32AyTMwE9QhrhvQyt3kaYsc8LA53WmGkRcgK64o3Kam/tKlVfdDyixYo6/4aQ6MW0 wuclkT62J8rEjgvjhjO6ddWzIqMu7n/fhqRBVifj1qhkDmkoABK56NTgz8Xde3UtmdSC ze53mmMTwSNcV1DJTyOcc8W8jOOKIpg3etnc0klhlAzw9Q4e2jVAkxS13yV0o/QtA91n LM5S3Qk9Dab8nqWkCyqAp+3pzejCDgkCFA/mwhvfoYScQI7MpLe0NcSnlZm2I1d/cN8v ZToHofPja9KwRiE39CSTpXWWx0hlYrFn2sZPCzOti46XhSgzPZvTPK1jabD0u7O0WO/C HYVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jiirWQEzjOJm38ClQocxHDg9OYUUCbqX/zh69yaIxOk=; b=BXxY72F+5tzvQ1/ED4E5tjQAsS6N/wdnjx2gIFamgBzz8b/mO+0ScdOdIdZARv5cm/ ucBAqqOhvmXzZ0cl+l9ZEEZikcfHi8Dson+aHXSU72ggDb3tZrgtgUd09Bd+ziYUU35x lHfjbcYVCT8SYMCd0gcZBngWGKAj7jajpMNgqxzpSCZGI5Za6Tq5UfG0CLJZrE1iDe3l 94BAAcDYW5ftVJETy31xlT4vr3xWZHaaYoEUaQ+dOB4+WCdLGLOV61xGHJHgRFNq6Gm1 gUyDo13hvL/wthRNauOezwa+CUeuLdtLbVDsNYYUEKG6uCO3D+VpUwOtXhl6k0f7VmYq tHSA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Ho5J9FsM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 12-20020a170906018c00b0088acf6e8c97si23980544ejb.531.2023.02.14.00.44.02; Tue, 14 Feb 2023 00:44:25 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Ho5J9FsM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232049AbjBNIit (ORCPT + 99 others); Tue, 14 Feb 2023 03:38:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41226 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231779AbjBNIiq (ORCPT ); Tue, 14 Feb 2023 03:38:46 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 81B70E042 for ; Tue, 14 Feb 2023 00:37:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676363849; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jiirWQEzjOJm38ClQocxHDg9OYUUCbqX/zh69yaIxOk=; b=Ho5J9FsMx0n8O+FoBAFqSyxcgyt7JXSWs9X+wPDB9SJF9r8SZnLJ3jJzq0mYAgsH9vQyes ecoAUggzqq77Lg0c+xYm5GjbketMUswQzc9zQs1v0apdvvPsS1IWnSszJJ/diiKJt0Xybt LJ35fFOzyRwTIzo649mtsWffAltegbE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-593-FzkSCvMjPFy74fuTaPhQVA-1; Tue, 14 Feb 2023 03:37:23 -0500 X-MC-Unique: FzkSCvMjPFy74fuTaPhQVA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E5B48811E6E; Tue, 14 Feb 2023 08:37:22 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id E2E9EC15BA0; Tue, 14 Feb 2023 08:37:14 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v3 1/5] splice: Rename new splice functions Date: Tue, 14 Feb 2023 08:37:06 +0000 Message-Id: <20230214083710.2547248-2-dhowells@redhat.com> In-Reply-To: <20230214083710.2547248-1-dhowells@redhat.com> References: <20230214083710.2547248-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757795335792439262?= X-GMAIL-MSGID: =?utf-8?q?1757795335792439262?= Rename generic_file_buffered_splice_read() to filemap_splice_read(). Rename generic_file_direct_splice_read() to direct_splice_read(). Requested-by: Christoph Hellwig Signed-off-by: David Howells Reviewed-by: Christoph Hellwig cc: Jens Axboe cc: Al Viro cc: John Hubbard cc: David Hildenbrand cc: Matthew Wilcox cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- fs/splice.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 2717078949a2..91b9e2cb9e03 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -287,9 +287,9 @@ void splice_shrink_spd(struct splice_pipe_desc *spd) * Splice data from an O_DIRECT file into pages and then add them to the output * pipe. */ -static ssize_t generic_file_direct_splice_read(struct file *in, loff_t *ppos, - struct pipe_inode_info *pipe, - size_t len, unsigned int flags) +static ssize_t direct_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags) { struct iov_iter to; struct bio_vec *bv; @@ -417,10 +417,9 @@ static size_t splice_folio_into_pipe(struct pipe_inode_info *pipe, * Splice folios from the pagecache of a buffered (ie. non-O_DIRECT) file into * a pipe. */ -static ssize_t generic_file_buffered_splice_read(struct file *in, loff_t *ppos, - struct pipe_inode_info *pipe, - size_t len, - unsigned int flags) +static ssize_t filemap_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags) { struct folio_batch fbatch; size_t total_spliced = 0, used, npages; @@ -529,8 +528,8 @@ ssize_t generic_file_splice_read(struct file *in, loff_t *ppos, if (unlikely(!len)) return 0; if (in->f_flags & O_DIRECT) - return generic_file_direct_splice_read(in, ppos, pipe, len, flags); - return generic_file_buffered_splice_read(in, ppos, pipe, len, flags); + return direct_splice_read(in, ppos, pipe, len, flags); + return filemap_splice_read(in, ppos, pipe, len, flags); } EXPORT_SYMBOL(generic_file_splice_read); From patchwork Tue Feb 14 08:37:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 56767 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2846634wrn; Tue, 14 Feb 2023 00:46:27 -0800 (PST) X-Google-Smtp-Source: AK7set9UbDxHt5zixBErbU5zkIYBRaOnM070bowqQtfv0LZ/QIz/a4kXVYyMMMPhaS6jHsJpAeUq X-Received: by 2002:a50:d494:0:b0:4ac:d30d:c3a3 with SMTP id s20-20020a50d494000000b004acd30dc3a3mr1495050edi.32.1676364387728; Tue, 14 Feb 2023 00:46:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676364387; cv=none; d=google.com; s=arc-20160816; b=qTGf/5BffoNWI3ZupRSXJRjDFOM/Y0mkqGqCuMkUJoG6Zywut3M7ZclxSaqz81JHca 6XFlFNrPGDO6nPbRacR/Fm01y1RUq82n4W5mtPhFNTf5HgU2UqX/iTky5/q7tG8yXhbL T08/1Joa0UbC0A/nreWcl/GO1Ro+a4w8XSdAD/3pgeMu1OMmDuBN7LopNGWId2mu0iaj 5oUDgFUs+zETxz9PA1zxpuJ3vrCU6VMH4sIMimqw3tIwSrEwGkpUOZQJrFDBPat/JKf+ kYEFEg5ThNPiHw4nwrdVrwgz2VyGi6usl+/UYuTUy2w/7zXOVMScxb3mZ6UgHOxOVnK3 N8EQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+b2VMeGpZTyrheqYPKTaHyxDIQ3yZeSE4YqjD4ZyLaI=; b=CJwIIbMcN9h2kKK8krDJFHj/71HfCyriR7O3cgFLbs4Vv0Ml97xJ5K5umAqSrwOEat trgAzcE/tRDK6SolibR96IQakkUDIJah8+Arfw0AcP+tk/vBbPQP6n4obY9TRkdypfAE ZvLBUv3q11CztfcF6nUr9D05B5v3KFQDQtEe/pqb5WHBHzMWkhhfxlHKSsTmzhPQjyD4 /+OR6SZdFG4Sa1mueJKZeMFHp905T9t5IBwFe6do/VM980scaicfrBuZutiupVjGpN/f b42ETRa76YHiDO/3/4avOWJJgDAFeT9v7bFSjSFO2yDPNQWGYynEI+oVEDz4pRuWvwV5 Bx4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=B42Yq0mS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b20-20020aa7dc14000000b004aac4b55319si15896241edu.480.2023.02.14.00.46.04; Tue, 14 Feb 2023 00:46:27 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=B42Yq0mS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232059AbjBNIiw (ORCPT + 99 others); Tue, 14 Feb 2023 03:38:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41224 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232045AbjBNIir (ORCPT ); Tue, 14 Feb 2023 03:38:47 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6194CDBE8 for ; Tue, 14 Feb 2023 00:37:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676363847; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+b2VMeGpZTyrheqYPKTaHyxDIQ3yZeSE4YqjD4ZyLaI=; b=B42Yq0mS1jT2+5yCTMZ9rNQMBeM+wEBBt38IY7prjxI6UEAlX0OZNSvX0wbGyLlkxqscSs 6PdRQ7hXUqxui/Stgdc3aGGYz0Szc7ZAyiFnJpdzUrQpqxpguNsv2BAtnWfWpJtyI6dOQ6 NS4EHg+3qbv/BJQHiFM7x4vxOUns+RM= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-225-1sNGJShAP3avneZS4IyFGA-1; Tue, 14 Feb 2023 03:37:26 -0500 X-MC-Unique: 1sNGJShAP3avneZS4IyFGA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8859E3811F2C; Tue, 14 Feb 2023 08:37:25 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9B3BFC15BA0; Tue, 14 Feb 2023 08:37:23 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v3 2/5] splice: Provide pipe_head_buf() helper Date: Tue, 14 Feb 2023 08:37:07 +0000 Message-Id: <20230214083710.2547248-3-dhowells@redhat.com> In-Reply-To: <20230214083710.2547248-1-dhowells@redhat.com> References: <20230214083710.2547248-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757795464132267415?= X-GMAIL-MSGID: =?utf-8?q?1757795464132267415?= Provide a helper, pipe_head_buf(), to get the current head buffer from a pipe. Implement this as a wrapper around a more general function, pipe_buf(), that gets a specified buffer. Requested-by: Christoph Hellwig Signed-off-by: David Howells Reviewed-by: Christoph Hellwig cc: Jens Axboe cc: Al Viro cc: John Hubbard cc: David Hildenbrand cc: Matthew Wilcox cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- fs/splice.c | 9 +++------ include/linux/pipe_fs_i.h | 20 ++++++++++++++++++++ 2 files changed, 23 insertions(+), 6 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 91b9e2cb9e03..7c0ff187f87a 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -295,7 +295,6 @@ static ssize_t direct_splice_read(struct file *in, loff_t *ppos, struct bio_vec *bv; struct kiocb kiocb; struct page **pages; - unsigned int head; ssize_t ret; size_t used, npages, chunk, remain, reclaim; int i; @@ -358,9 +357,8 @@ static ssize_t direct_splice_read(struct file *in, loff_t *ppos, } /* Push the remaining pages into the pipe. */ - head = pipe->head; for (i = 0; i < npages; i++) { - struct pipe_buffer *buf = &pipe->bufs[head & (pipe->ring_size - 1)]; + struct pipe_buffer *buf = pipe_head_buf(pipe); chunk = min_t(size_t, remain, PAGE_SIZE); *buf = (struct pipe_buffer) { @@ -369,10 +367,9 @@ static ssize_t direct_splice_read(struct file *in, loff_t *ppos, .offset = 0, .len = chunk, }; - head++; + pipe->head++; remain -= chunk; } - pipe->head = head; kfree(bv); return ret; @@ -394,7 +391,7 @@ static size_t splice_folio_into_pipe(struct pipe_inode_info *pipe, while (spliced < size && !pipe_full(pipe->head, pipe->tail, pipe->max_usage)) { - struct pipe_buffer *buf = &pipe->bufs[pipe->head & (pipe->ring_size - 1)]; + struct pipe_buffer *buf = pipe_head_buf(pipe); size_t part = min_t(size_t, PAGE_SIZE - offset, size - spliced); *buf = (struct pipe_buffer) { diff --git a/include/linux/pipe_fs_i.h b/include/linux/pipe_fs_i.h index 6cb65df3e3ba..d2c3f16cf6b1 100644 --- a/include/linux/pipe_fs_i.h +++ b/include/linux/pipe_fs_i.h @@ -156,6 +156,26 @@ static inline bool pipe_full(unsigned int head, unsigned int tail, return pipe_occupancy(head, tail) >= limit; } +/** + * pipe_buf - Return the pipe buffer for the specified slot in the pipe ring + * @pipe: The pipe to access + * @slot: The slot of interest + */ +static inline struct pipe_buffer *pipe_buf(const struct pipe_inode_info *pipe, + unsigned int slot) +{ + return &pipe->bufs[slot & (pipe->ring_size - 1)]; +} + +/** + * pipe_head_buf - Return the pipe buffer at the head of the pipe ring + * @pipe: The pipe to access + */ +static inline struct pipe_buffer *pipe_head_buf(const struct pipe_inode_info *pipe) +{ + return pipe_buf(pipe, pipe->head); +} + /** * pipe_buf_get - get a reference to a pipe_buffer * @pipe: the pipe that the buffer belongs to From patchwork Tue Feb 14 08:37:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 56764 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2846019wrn; Tue, 14 Feb 2023 00:44:58 -0800 (PST) X-Google-Smtp-Source: AK7set8+IqWxfBEMUVcR3jqExFT/okng29YjicCFuvxAAwGOCPn2jtU9LfxrNBeOWPtD+MO8UXvZ X-Received: by 2002:a17:907:124e:b0:8aa:c090:a9ef with SMTP id wc14-20020a170907124e00b008aac090a9efmr1413502ejb.55.1676364297848; Tue, 14 Feb 2023 00:44:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676364297; cv=none; d=google.com; s=arc-20160816; b=MmOsE3j0DtBOALbGPCC7lDowvDaXY4JqHUJYZRPS0tp4cirHBQBd76evZzSo3Bn5Y+ UR/vWWhIZ8wEhKrO/hRl6YMmkfgoV9PMhAIb2VGqVEkqosQ4e2z0DCi9r2G87PuSDn9Z LW8gd/K8KIQOa7HHEz5wPR+iLGa0rUclx9oIFnx1gYLY2xujQ5+d1fdFViTWOrvX7NCp xBImVMVDnB9tC9UuYWXkIwnJ20YT3u6KN2TD6ER5+6gufh3JmX0x75KNLmTlXLoLszNi r1kYpT6HpCBPecM/Z+HzrR5x1I1OH7kwPXK0vQfTPIneg1/4uT9vm6gEOGu/BdVNkQY9 wQlA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=xJXt4KNNFIdMvSSPmBpCRCAi8g0JFMLHorMyvAgfics=; b=OBoL2f2lUBsAF0BH7jX/YQldyG6UJObsIeoNLOhUR3FoLQxPpKVS/7baljaVUATWWr kkJlkJoesWHA9Jz84mvselHEMW2CzYDFbz/PFjAQSS8Ru1C3nGhA5RSQsfFRHczvv9of zVBo+xah0HeSz+Af2YZlCuOn6JXmaTtxA0/a9bJOl/6/iQJatxQ8aqK5dsUQKPN6Tgld vFhEJ85s5adgnxoX4FWdDfvNbTtFFVyq0kY7l9XeSfUKr/oouIMx8yX+h8VmmdxCX1Vg N/DbuHAeUn88wHsEmqM3hKUhrKymuZYvySnmfnCN5P+kmvBiE0Q1lAIj3YN+/obPGwuQ 5FkA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=gIAH3LiN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fl19-20020a1709072a9300b00787abcb1ce0si16835473ejc.679.2023.02.14.00.44.35; Tue, 14 Feb 2023 00:44:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=gIAH3LiN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232052AbjBNIjO (ORCPT + 99 others); Tue, 14 Feb 2023 03:39:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41402 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232083AbjBNIjA (ORCPT ); Tue, 14 Feb 2023 03:39:00 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DC02BE386 for ; Tue, 14 Feb 2023 00:37:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676363852; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xJXt4KNNFIdMvSSPmBpCRCAi8g0JFMLHorMyvAgfics=; b=gIAH3LiNKLVWbWoHrDW/yTd2ncPan6UF/YqqXlD6xnh+d+Z4sCZ3ZUHOgr+dCwRdi6dRKI clXPvg+2orcE9vI9x11UiC2mWfMkZvlEjJQkht7QRpi1/HfWuDOV/uSIaZJXEGRRotlIBZ DBHV0U1xqjr8PvoyjZwGm3AA59g/dW4= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-647-4m-qRrFTP7OlNbKg3ZIAJA-1; Tue, 14 Feb 2023 03:37:28 -0500 X-MC-Unique: 4m-qRrFTP7OlNbKg3ZIAJA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 14FC72999B22; Tue, 14 Feb 2023 08:37:28 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2717F140EBF6; Tue, 14 Feb 2023 08:37:26 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v3 3/5] splice: Use init_sync_kiocb() in filemap_splice_read() Date: Tue, 14 Feb 2023 08:37:08 +0000 Message-Id: <20230214083710.2547248-4-dhowells@redhat.com> In-Reply-To: <20230214083710.2547248-1-dhowells@redhat.com> References: <20230214083710.2547248-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757795370265246364?= X-GMAIL-MSGID: =?utf-8?q?1757795370265246364?= Use init_sync_kiocb() in filemap_splice_read() rather than open coding it. Requested-by: Christoph Hellwig Signed-off-by: David Howells cc: Christoph Hellwig cc: Jens Axboe cc: Al Viro cc: John Hubbard cc: David Hildenbrand cc: Matthew Wilcox cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- Notes: ver #2) - Don't attempt to filter IOCB_* flags. fs/splice.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 7c0ff187f87a..4ea63d6a9040 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -419,15 +419,14 @@ static ssize_t filemap_splice_read(struct file *in, loff_t *ppos, size_t len, unsigned int flags) { struct folio_batch fbatch; + struct kiocb iocb; size_t total_spliced = 0, used, npages; loff_t isize, end_offset; bool writably_mapped; int i, error = 0; - struct kiocb iocb = { - .ki_filp = in, - .ki_pos = *ppos, - }; + init_sync_kiocb(&iocb, in); + iocb.ki_pos = *ppos; /* Work out how much data we can actually add into the pipe */ used = pipe_occupancy(pipe->head, pipe->tail); From patchwork Tue Feb 14 08:37:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 56763 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2845969wrn; Tue, 14 Feb 2023 00:44:50 -0800 (PST) X-Google-Smtp-Source: AK7set854MiAJJ7p+0IC/jox4A4UGrt6FT6nVUsWWPLrMp+9fNbrp3tkfOS67gW6qF/AlTl4fxeY X-Received: by 2002:a17:906:3a57:b0:86f:1227:7a48 with SMTP id a23-20020a1709063a5700b0086f12277a48mr1843768ejf.17.1676364290073; Tue, 14 Feb 2023 00:44:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676364290; cv=none; d=google.com; s=arc-20160816; b=r0GGvWGSFwI586aZS0jEoELZRcDIz/zgoOCeytjvmsTtnx/CjnP/exgiTevaNN8hVc +e+Xwxc7vSf+Id8KVX40oFkblQA7K7BsCzgTZ1sJF/4P+z/sJMAn5ZHqc8U5V3mgUecf SboZ0YBf9R/fQEW/FsJAC4brbp2Z6zHD0bYok3koG7Pyh7c6demVoFaE+6FslM5f/qE7 uDlKuAdenT7SSPp7feX0VWmgw8Z42XsVtOOAGPxP68ymvFY3CnExUhlaBKiSQ4sdhLE+ gy9NuPUX8u48VrSAFXPgK2ziDJwIj21rDyscLpQvLPKhn4CFiSnyZoN1Y+SvTNCLQai2 qHwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=49IJ7O8KaG9bUjSCVtwVGeadJYJCODKyAnH6L7XF9aY=; b=uWuUH1Mq20NM0UXtRcGnDGajcPQDcJw2aZqn0JLBGeTmPAAA9HWglPrpqm+Ld8cHtO ADzqyMyH8GBVOmwan23992aBGk7x0UDRO0XAga7G5CNqqVlTwYXFMFTKQGVf2bHlHoaS JVViw3tEhc0RrAUkofGXIeU3F7vCLX79EEe0R+xB2wytAgGvKMFBuLsGOmLjQMydiwqM yl+Xk4ON2QGuiYPDcxiSq9KqMnUS7BVFXEhnICOIx6oYcp8Fw01PypNzdPx2iyxyD0RM m5CLRf8Z295+RfZG/XWCf14RnOHC0tYBiwUpragO44KcN6wp4C4fYZ9+jZKDtQIvLXQQ 4CYw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ahlB1iGT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fv29-20020a170907509d00b008890e939a9dsi15550913ejc.143.2023.02.14.00.44.26; Tue, 14 Feb 2023 00:44:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ahlB1iGT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232130AbjBNIjL (ORCPT + 99 others); Tue, 14 Feb 2023 03:39:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41404 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232067AbjBNIi7 (ORCPT ); Tue, 14 Feb 2023 03:38:59 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 591D1E392 for ; Tue, 14 Feb 2023 00:37:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676363854; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=49IJ7O8KaG9bUjSCVtwVGeadJYJCODKyAnH6L7XF9aY=; b=ahlB1iGTKA0o2kLosQo4zANzb8ckHyUultiULQA2tKSd9wRm1NMAVGG6tXs/FtgwcWIvti z2nrhHe2mO7e7G4UCTWr+VjZlKgbHuKTXnWNuI6w//Hajz7DIv7f1QkCG6QbiHvlhfVfkS G2/eYYSB1T1c37L0TPFl4nSnKHfEhDo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-208-whlcj7bIP-qL1QdHi6m4cg-1; Tue, 14 Feb 2023 03:37:31 -0500 X-MC-Unique: whlcj7bIP-qL1QdHi6m4cg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B63CC811E6E; Tue, 14 Feb 2023 08:37:30 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id C013540C945A; Tue, 14 Feb 2023 08:37:28 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , John Hubbard Subject: [PATCH v3 4/5] splice: Move filemap_read_splice() to mm/filemap.c Date: Tue, 14 Feb 2023 08:37:09 +0000 Message-Id: <20230214083710.2547248-5-dhowells@redhat.com> In-Reply-To: <20230214083710.2547248-1-dhowells@redhat.com> References: <20230214083710.2547248-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757795362097581289?= X-GMAIL-MSGID: =?utf-8?q?1757795362097581289?= Move filemap_read_splice() to mm/filemap.c and make filemap_get_pages() static again. Requested-by: Christoph Hellwig Signed-off-by: David Howells Reviewed-by: Christoph Hellwig cc: Jens Axboe cc: Al Viro cc: John Hubbard cc: David Hildenbrand cc: Matthew Wilcox cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- fs/splice.c | 127 ------------------------------------- include/linux/pagemap.h | 2 - include/linux/splice.h | 4 ++ mm/filemap.c | 137 ++++++++++++++++++++++++++++++++++++++-- 4 files changed, 135 insertions(+), 135 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 4ea63d6a9040..341cd8fb47a8 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -375,133 +375,6 @@ static ssize_t direct_splice_read(struct file *in, loff_t *ppos, return ret; } -/* - * Splice subpages from a folio into a pipe. - */ -static size_t splice_folio_into_pipe(struct pipe_inode_info *pipe, - struct folio *folio, - loff_t fpos, size_t size) -{ - struct page *page; - size_t spliced = 0, offset = offset_in_folio(folio, fpos); - - page = folio_page(folio, offset / PAGE_SIZE); - size = min(size, folio_size(folio) - offset); - offset %= PAGE_SIZE; - - while (spliced < size && - !pipe_full(pipe->head, pipe->tail, pipe->max_usage)) { - struct pipe_buffer *buf = pipe_head_buf(pipe); - size_t part = min_t(size_t, PAGE_SIZE - offset, size - spliced); - - *buf = (struct pipe_buffer) { - .ops = &page_cache_pipe_buf_ops, - .page = page, - .offset = offset, - .len = part, - }; - folio_get(folio); - pipe->head++; - page++; - spliced += part; - offset = 0; - } - - return spliced; -} - -/* - * Splice folios from the pagecache of a buffered (ie. non-O_DIRECT) file into - * a pipe. - */ -static ssize_t filemap_splice_read(struct file *in, loff_t *ppos, - struct pipe_inode_info *pipe, - size_t len, unsigned int flags) -{ - struct folio_batch fbatch; - struct kiocb iocb; - size_t total_spliced = 0, used, npages; - loff_t isize, end_offset; - bool writably_mapped; - int i, error = 0; - - init_sync_kiocb(&iocb, in); - iocb.ki_pos = *ppos; - - /* Work out how much data we can actually add into the pipe */ - used = pipe_occupancy(pipe->head, pipe->tail); - npages = max_t(ssize_t, pipe->max_usage - used, 0); - len = min_t(size_t, len, npages * PAGE_SIZE); - - folio_batch_init(&fbatch); - - do { - cond_resched(); - - if (*ppos >= i_size_read(file_inode(in))) - break; - - iocb.ki_pos = *ppos; - error = filemap_get_pages(&iocb, len, &fbatch, true); - if (error < 0) - break; - - /* - * i_size must be checked after we know the pages are Uptodate. - * - * Checking i_size after the check allows us to calculate - * the correct value for "nr", which means the zero-filled - * part of the page is not copied back to userspace (unless - * another truncate extends the file - this is desired though). - */ - isize = i_size_read(file_inode(in)); - if (unlikely(*ppos >= isize)) - break; - end_offset = min_t(loff_t, isize, *ppos + len); - - /* - * Once we start copying data, we don't want to be touching any - * cachelines that might be contended: - */ - writably_mapped = mapping_writably_mapped(in->f_mapping); - - for (i = 0; i < folio_batch_count(&fbatch); i++) { - struct folio *folio = fbatch.folios[i]; - size_t n; - - if (folio_pos(folio) >= end_offset) - goto out; - folio_mark_accessed(folio); - - /* - * If users can be writing to this folio using arbitrary - * virtual addresses, take care of potential aliasing - * before reading the folio on the kernel side. - */ - if (writably_mapped) - flush_dcache_folio(folio); - - n = splice_folio_into_pipe(pipe, folio, *ppos, len); - if (!n) - goto out; - len -= n; - total_spliced += n; - *ppos += n; - in->f_ra.prev_pos = *ppos; - if (pipe_full(pipe->head, pipe->tail, pipe->max_usage)) - goto out; - } - - folio_batch_release(&fbatch); - } while (len); - -out: - folio_batch_release(&fbatch); - file_accessed(in); - - return total_spliced ? total_spliced : error; -} - /** * generic_file_splice_read - splice data from file to a pipe * @in: file to splice from diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 3a7bdb35acff..29e1f9e76eb6 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -748,8 +748,6 @@ struct page *read_cache_page(struct address_space *, pgoff_t index, filler_t *filler, struct file *file); extern struct page * read_cache_page_gfp(struct address_space *mapping, pgoff_t index, gfp_t gfp_mask); -int filemap_get_pages(struct kiocb *iocb, size_t count, - struct folio_batch *fbatch, bool need_uptodate); static inline struct page *read_mapping_page(struct address_space *mapping, pgoff_t index, struct file *file) diff --git a/include/linux/splice.h b/include/linux/splice.h index a55179fd60fc..691c44ef5c0b 100644 --- a/include/linux/splice.h +++ b/include/linux/splice.h @@ -67,6 +67,10 @@ typedef int (splice_actor)(struct pipe_inode_info *, struct pipe_buffer *, typedef int (splice_direct_actor)(struct pipe_inode_info *, struct splice_desc *); +ssize_t filemap_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags); + extern ssize_t splice_from_pipe(struct pipe_inode_info *, struct file *, loff_t *, size_t, unsigned int, splice_actor *); diff --git a/mm/filemap.c b/mm/filemap.c index 6970be64a3e0..e1ee267675d2 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -42,6 +42,8 @@ #include #include #include +#include +#include #include #include #include "internal.h" @@ -2576,12 +2578,8 @@ static int filemap_readahead(struct kiocb *iocb, struct file *file, return 0; } -/* - * Extract some folios from the pagecache of a file, reading those pages from - * the backing store if necessary and waiting for them. - */ -int filemap_get_pages(struct kiocb *iocb, size_t count, - struct folio_batch *fbatch, bool need_uptodate) +static int filemap_get_pages(struct kiocb *iocb, size_t count, + struct folio_batch *fbatch, bool need_uptodate) { struct file *filp = iocb->ki_filp; struct address_space *mapping = filp->f_mapping; @@ -2845,6 +2843,133 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) } EXPORT_SYMBOL(generic_file_read_iter); +/* + * Splice subpages from a folio into a pipe. + */ +static size_t splice_folio_into_pipe(struct pipe_inode_info *pipe, + struct folio *folio, + loff_t fpos, size_t size) +{ + struct page *page; + size_t spliced = 0, offset = offset_in_folio(folio, fpos); + + page = folio_page(folio, offset / PAGE_SIZE); + size = min(size, folio_size(folio) - offset); + offset %= PAGE_SIZE; + + while (spliced < size && + !pipe_full(pipe->head, pipe->tail, pipe->max_usage)) { + struct pipe_buffer *buf = pipe_head_buf(pipe); + size_t part = min_t(size_t, PAGE_SIZE - offset, size - spliced); + + *buf = (struct pipe_buffer) { + .ops = &page_cache_pipe_buf_ops, + .page = page, + .offset = offset, + .len = part, + }; + folio_get(folio); + pipe->head++; + page++; + spliced += part; + offset = 0; + } + + return spliced; +} + +/* + * Splice folios from the pagecache of a buffered (ie. non-O_DIRECT) file into + * a pipe. + */ +ssize_t filemap_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags) +{ + struct folio_batch fbatch; + struct kiocb iocb; + size_t total_spliced = 0, used, npages; + loff_t isize, end_offset; + bool writably_mapped; + int i, error = 0; + + init_sync_kiocb(&iocb, in); + iocb.ki_pos = *ppos; + + /* Work out how much data we can actually add into the pipe */ + used = pipe_occupancy(pipe->head, pipe->tail); + npages = max_t(ssize_t, pipe->max_usage - used, 0); + len = min_t(size_t, len, npages * PAGE_SIZE); + + folio_batch_init(&fbatch); + + do { + cond_resched(); + + if (*ppos >= i_size_read(file_inode(in))) + break; + + iocb.ki_pos = *ppos; + error = filemap_get_pages(&iocb, len, &fbatch, true); + if (error < 0) + break; + + /* + * i_size must be checked after we know the pages are Uptodate. + * + * Checking i_size after the check allows us to calculate + * the correct value for "nr", which means the zero-filled + * part of the page is not copied back to userspace (unless + * another truncate extends the file - this is desired though). + */ + isize = i_size_read(file_inode(in)); + if (unlikely(*ppos >= isize)) + break; + end_offset = min_t(loff_t, isize, *ppos + len); + + /* + * Once we start copying data, we don't want to be touching any + * cachelines that might be contended: + */ + writably_mapped = mapping_writably_mapped(in->f_mapping); + + for (i = 0; i < folio_batch_count(&fbatch); i++) { + struct folio *folio = fbatch.folios[i]; + size_t n; + + if (folio_pos(folio) >= end_offset) + goto out; + folio_mark_accessed(folio); + + /* + * If users can be writing to this folio using arbitrary + * virtual addresses, take care of potential aliasing + * before reading the folio on the kernel side. + */ + if (writably_mapped) + flush_dcache_folio(folio); + + n = splice_folio_into_pipe(pipe, folio, *ppos, len); + if (!n) + goto out; + len -= n; + total_spliced += n; + *ppos += n; + in->f_ra.prev_pos = *ppos; + if (pipe_full(pipe->head, pipe->tail, pipe->max_usage)) + goto out; + } + + folio_batch_release(&fbatch); + } while (len); + +out: + folio_batch_release(&fbatch); + file_accessed(in); + + return total_spliced ? total_spliced : error; +} + static inline loff_t folio_seek_hole_data(struct xa_state *xas, struct address_space *mapping, struct folio *folio, loff_t start, loff_t end, bool seek_data) From patchwork Tue Feb 14 08:37:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 56766 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2846597wrn; Tue, 14 Feb 2023 00:46:24 -0800 (PST) X-Google-Smtp-Source: AK7set/30OJ3x3sz/uIolqEf+5Ufkhcr3vfPYeGzlSTKJS1RFWk+99/vaKXUpXjKtG3mwnU8Og79 X-Received: by 2002:a05:6a00:1ca8:b0:5a8:d38a:c6c4 with SMTP id y40-20020a056a001ca800b005a8d38ac6c4mr2605680pfw.14.1676364383787; Tue, 14 Feb 2023 00:46:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676364383; cv=none; d=google.com; s=arc-20160816; b=cRl87DWH4NUOBg0AtiSjjndgks4bcEAyUZGh1suowVfIqAT4eQt3v6Kvk5BLliBw9B C5Arxd5LJvbTxepWOim3MmLf1OlP+HF01lUnEQ6/TskgOvLq/A0GyCvxMT9yOJ9Dduwx WEdZc9dRqg66aTyp4aNPYyyMGPRJ69uSc1XF1ZzZlz0J3ZdtJAGEyXWWz0JxyL1mV5qJ XYW3UdzE35g3djQkWrA8jXhtl5nbxlWmmgvF1/oSWlrWmhfQc3QRLUWQ5yraOZzQciT5 L20DzxvzBQqCLj/Al+04wTxAyvRsyF+buoJKvF3Kc9LazdXgwiUDL8RY9WUdq1ScG+Ue mz0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Aqt7Ie9m7VBTVoG66ytoaEfOzMGB0BP8eJc00dhMkTk=; b=FHmJ3A3IE6nr69UC05rSymZvMJ7c/CsrcvGUqO1ocOzREF5V9XPl+rBkd7FXroJiBA SAV138w4qoio6mv1pXdq6K+fZzv20lk9sMPyRhBVpjpZ0DXR9muosENN/VuKDN6aUVtc 7HeS34CfzWvRMkJb5AVy8491IHNzMR/dOWpfxYHUTJozLYazC25KM+n9w0I0OR2l2ciJ Y8hqeRgYvyhhJCh+iQJp6ADUdMAqVP+6nND22a4hYekrsx+B8QVq7zuIFDarWoY91bNR CAaZlYFqKk+G6gngy+PPp86qO7yuIgY2/Ve2/L68m9RLNk9JilrUjG871MGVjVv37KSb lqLQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=GV6+rLhJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w11-20020aa7954b000000b0058c800a75f6si1705146pfq.335.2023.02.14.00.46.11; Tue, 14 Feb 2023 00:46:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=GV6+rLhJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232060AbjBNIj4 (ORCPT + 99 others); Tue, 14 Feb 2023 03:39:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41680 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232140AbjBNIjb (ORCPT ); Tue, 14 Feb 2023 03:39:31 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A7825EB5E for ; Tue, 14 Feb 2023 00:37:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676363861; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Aqt7Ie9m7VBTVoG66ytoaEfOzMGB0BP8eJc00dhMkTk=; b=GV6+rLhJefKd7pvQva/llPo3E91TzS1xiFki1fFoxov4WzgzbFTKGZdaopOxlX4bA97LDY gH3OJ87ADE5rtN/Lk1Yq574n/21iyjC0qpgYtTkcTjdVYN23x3uhvadMQ6RTENuNdjzoU8 Zj0F7PpQbC7GjhTdu0sPry4Y6wqZLl8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-639-REqII07MME2i0C0sg7aRjw-1; Tue, 14 Feb 2023 03:37:36 -0500 X-MC-Unique: REqII07MME2i0C0sg7aRjw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 151B7857A9F; Tue, 14 Feb 2023 08:37:35 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.33.36.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5598F140EBF6; Tue, 14 Feb 2023 08:37:31 +0000 (UTC) From: David Howells To: Jens Axboe , Al Viro , Christoph Hellwig Cc: David Howells , Matthew Wilcox , Jan Kara , Jeff Layton , David Hildenbrand , Jason Gunthorpe , Logan Gunthorpe , Hillf Danton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Daniel Golle , Guenter Roeck , Christoph Hellwig , John Hubbard , Miklos Szeredi , Hugh Dickins , Jan Harkes , Arnd Bergmann , Greg Kroah-Hartman , coda@cs.cmu.edu, codalist@coda.cs.cmu.edu, linux-unionfs@vger.kernel.org Subject: [PATCH v3 5/5] shmem, overlayfs, coda, tty, proc, kernfs, random: Fix splice-read Date: Tue, 14 Feb 2023 08:37:10 +0000 Message-Id: <20230214083710.2547248-6-dhowells@redhat.com> In-Reply-To: <20230214083710.2547248-1-dhowells@redhat.com> References: <20230214083710.2547248-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757795460053989465?= X-GMAIL-MSGID: =?utf-8?q?1757795460053989465?= The new filemap_splice_read() has an implicit expectation via filemap_get_pages() that ->read_folio() exists if ->readahead() doesn't fully populate the pagecache of the file it is reading from[1], potentially leading to a jump to NULL if this doesn't exist. A filesystem or driver shouldn't suffer from this if: - It doesn't set ->splice_read() - It implements ->read_folio() - It implements its own ->splice_read() Note that some filesystems set generic_file_splice_read() and generic_file_read_iter() but don't set ->read_folio(). g_f_read_iter() will fall back to filemap_read_iter() which looks like it should suffer from the same issue. Certain drivers, can just use direct_splice_read() rather than generic_file_splice_read() as that creates an output buffer and then just calls their ->read_iter() function: - random & urandom - tty - kernfs - proc - proc_namespace Stacked filesystems just need to pass the operation down a layer: - coda - overlayfs And finally, there's shmem (used in tmpfs, ramfs, rootfs). This needs its own splice-read implementation, based on filemap_splice_read(), but able to paste in zero_page when there's a page missing. Fixes: d9722a475711 ("splice: Do splice read from a buffered file without using ITER_PIPE") Signed-off-by: David Howells cc: Daniel Golle cc: Guenter Roeck cc: Christoph Hellwig cc: Jens Axboe cc: Al Viro cc: John Hubbard cc: David Hildenbrand cc: Matthew Wilcox cc: Miklos Szeredi cc: Hugh Dickins cc: Jan Harkes cc: Arnd Bergmann cc: Greg Kroah-Hartman cc: coda@cs.cmu.edu cc: codalist@coda.cs.cmu.edu cc: linux-unionfs@vger.kernel.org cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org Link: https://lore.kernel.org/r/Y+pdHFFTk1TTEBsO@makrotopia.org/ [1] Acked-by: Greg Kroah-Hartman Tested-by: Daniel Golle --- drivers/char/random.c | 4 +- drivers/tty/tty_io.c | 4 +- fs/coda/file.c | 36 +++++++++++- fs/kernfs/file.c | 2 +- fs/overlayfs/file.c | 36 +++++++++++- fs/proc/inode.c | 4 +- fs/proc/proc_sysctl.c | 2 +- fs/proc_namespace.c | 6 +- fs/splice.c | 6 +- include/linux/fs.h | 6 ++ include/linux/splice.h | 4 -- mm/filemap.c | 5 +- mm/internal.h | 6 ++ mm/shmem.c | 124 ++++++++++++++++++++++++++++++++++++++++- 14 files changed, 221 insertions(+), 24 deletions(-) diff --git a/drivers/char/random.c b/drivers/char/random.c index ce3ccd172cc8..792713616ba8 100644 --- a/drivers/char/random.c +++ b/drivers/char/random.c @@ -1546,7 +1546,7 @@ const struct file_operations random_fops = { .compat_ioctl = compat_ptr_ioctl, .fasync = random_fasync, .llseek = noop_llseek, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, }; @@ -1557,7 +1557,7 @@ const struct file_operations urandom_fops = { .compat_ioctl = compat_ptr_ioctl, .fasync = random_fasync, .llseek = noop_llseek, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, }; diff --git a/drivers/tty/tty_io.c b/drivers/tty/tty_io.c index 3149114bf130..495678e9b95e 100644 --- a/drivers/tty/tty_io.c +++ b/drivers/tty/tty_io.c @@ -466,7 +466,7 @@ static const struct file_operations tty_fops = { .llseek = no_llseek, .read_iter = tty_read, .write_iter = tty_write, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, .poll = tty_poll, .unlocked_ioctl = tty_ioctl, @@ -481,7 +481,7 @@ static const struct file_operations console_fops = { .llseek = no_llseek, .read_iter = tty_read, .write_iter = redirected_tty_write, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, .poll = tty_poll, .unlocked_ioctl = tty_ioctl, diff --git a/fs/coda/file.c b/fs/coda/file.c index 3f3c81e6b1ab..33cd7880d30e 100644 --- a/fs/coda/file.c +++ b/fs/coda/file.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include "coda_psdev.h" @@ -94,6 +95,39 @@ coda_file_write_iter(struct kiocb *iocb, struct iov_iter *to) return ret; } +static ssize_t +coda_file_splice_read(struct file *coda_file, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags) +{ + struct inode *coda_inode = file_inode(coda_file); + struct coda_file_info *cfi = coda_ftoc(coda_file); + struct file *in = cfi->cfi_container; + loff_t ki_pos = *ppos; + ssize_t ret; + + if (!in->f_op->splice_read) + return -EINVAL; + + ret = rw_verify_area(READ, in, ppos, len); + if (unlikely(ret < 0)) + return ret; + + ret = venus_access_intent(coda_inode->i_sb, coda_i2f(coda_inode), + &cfi->cfi_access_intent, + len, ki_pos, CODA_ACCESS_TYPE_READ); + if (ret) + goto finish_read; + + ret = in->f_op->splice_read(in, ppos, pipe, len, flags); + +finish_read: + venus_access_intent(coda_inode->i_sb, coda_i2f(coda_inode), + &cfi->cfi_access_intent, + len, ki_pos, CODA_ACCESS_TYPE_READ_FINISH); + return ret; +} + static void coda_vm_open(struct vm_area_struct *vma) { @@ -302,5 +336,5 @@ const struct file_operations coda_file_operations = { .open = coda_open, .release = coda_release, .fsync = coda_fsync, - .splice_read = generic_file_splice_read, + .splice_read = coda_file_splice_read, }; diff --git a/fs/kernfs/file.c b/fs/kernfs/file.c index e4a50e4ff0d2..9d23b8141db7 100644 --- a/fs/kernfs/file.c +++ b/fs/kernfs/file.c @@ -1011,7 +1011,7 @@ const struct file_operations kernfs_file_fops = { .release = kernfs_fop_release, .poll = kernfs_fop_poll, .fsync = noop_fsync, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, }; diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c index c9d0c362c7ef..267b61df6fcd 100644 --- a/fs/overlayfs/file.c +++ b/fs/overlayfs/file.c @@ -419,6 +419,40 @@ static ssize_t ovl_write_iter(struct kiocb *iocb, struct iov_iter *iter) return ret; } +static ssize_t ovl_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, size_t len, + unsigned int flags) +{ + const struct cred *old_cred; + struct fd real; + ssize_t ret; + + ret = ovl_real_fdget(in, &real); + if (ret) + return ret; + + ret = -EINVAL; + if (in->f_flags & O_DIRECT && + !(real.file->f_mode & FMODE_CAN_ODIRECT)) + goto out_fdput; + if (!real.file->f_op->splice_read) + goto out_fdput; + + ret = rw_verify_area(READ, in, ppos, len); + if (unlikely(ret < 0)) + return ret; + + old_cred = ovl_override_creds(file_inode(in)->i_sb); + ret = real.file->f_op->splice_read(real.file, ppos, pipe, len, flags); + + revert_creds(old_cred); + ovl_file_accessed(in); +out_fdput: + fdput(real); + + return ret; +} + /* * Calling iter_file_splice_write() directly from overlay's f_op may deadlock * due to lock order inversion between pipe->mutex in iter_file_splice_write() @@ -695,7 +729,7 @@ const struct file_operations ovl_file_operations = { .fallocate = ovl_fallocate, .fadvise = ovl_fadvise, .flush = ovl_flush, - .splice_read = generic_file_splice_read, + .splice_read = ovl_splice_read, .splice_write = ovl_splice_write, .copy_file_range = ovl_copy_file_range, diff --git a/fs/proc/inode.c b/fs/proc/inode.c index f495fdb39151..711f12706469 100644 --- a/fs/proc/inode.c +++ b/fs/proc/inode.c @@ -591,7 +591,7 @@ static const struct file_operations proc_iter_file_ops = { .llseek = proc_reg_llseek, .read_iter = proc_reg_read_iter, .write = proc_reg_write, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .poll = proc_reg_poll, .unlocked_ioctl = proc_reg_unlocked_ioctl, .mmap = proc_reg_mmap, @@ -617,7 +617,7 @@ static const struct file_operations proc_reg_file_ops_compat = { static const struct file_operations proc_iter_file_ops_compat = { .llseek = proc_reg_llseek, .read_iter = proc_reg_read_iter, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .write = proc_reg_write, .poll = proc_reg_poll, .unlocked_ioctl = proc_reg_unlocked_ioctl, diff --git a/fs/proc/proc_sysctl.c b/fs/proc/proc_sysctl.c index 48f2d60bd78a..92533bd0e67b 100644 --- a/fs/proc/proc_sysctl.c +++ b/fs/proc/proc_sysctl.c @@ -869,7 +869,7 @@ static const struct file_operations proc_sys_file_operations = { .poll = proc_sys_poll, .read_iter = proc_sys_read, .write_iter = proc_sys_write, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .splice_write = iter_file_splice_write, .llseek = default_llseek, }; diff --git a/fs/proc_namespace.c b/fs/proc_namespace.c index 846f9455ae22..492abbbeff5e 100644 --- a/fs/proc_namespace.c +++ b/fs/proc_namespace.c @@ -324,7 +324,7 @@ static int mountstats_open(struct inode *inode, struct file *file) const struct file_operations proc_mounts_operations = { .open = mounts_open, .read_iter = seq_read_iter, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .llseek = seq_lseek, .release = mounts_release, .poll = mounts_poll, @@ -333,7 +333,7 @@ const struct file_operations proc_mounts_operations = { const struct file_operations proc_mountinfo_operations = { .open = mountinfo_open, .read_iter = seq_read_iter, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .llseek = seq_lseek, .release = mounts_release, .poll = mounts_poll, @@ -342,7 +342,7 @@ const struct file_operations proc_mountinfo_operations = { const struct file_operations proc_mountstats_operations = { .open = mountstats_open, .read_iter = seq_read_iter, - .splice_read = generic_file_splice_read, + .splice_read = direct_splice_read, .llseek = seq_lseek, .release = mounts_release, }; diff --git a/fs/splice.c b/fs/splice.c index 341cd8fb47a8..0708cf0d12b7 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -287,9 +287,9 @@ void splice_shrink_spd(struct splice_pipe_desc *spd) * Splice data from an O_DIRECT file into pages and then add them to the output * pipe. */ -static ssize_t direct_splice_read(struct file *in, loff_t *ppos, - struct pipe_inode_info *pipe, - size_t len, unsigned int flags) +ssize_t direct_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags) { struct iov_iter to; struct bio_vec *bv; diff --git a/include/linux/fs.h b/include/linux/fs.h index c1769a2c5d70..551c9403f9b3 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -3163,6 +3163,12 @@ ssize_t vfs_iocb_iter_write(struct file *file, struct kiocb *iocb, struct iov_iter *iter); /* fs/splice.c */ +ssize_t filemap_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags); +ssize_t direct_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags); extern ssize_t generic_file_splice_read(struct file *, loff_t *, struct pipe_inode_info *, size_t, unsigned int); extern ssize_t iter_file_splice_write(struct pipe_inode_info *, diff --git a/include/linux/splice.h b/include/linux/splice.h index 691c44ef5c0b..a55179fd60fc 100644 --- a/include/linux/splice.h +++ b/include/linux/splice.h @@ -67,10 +67,6 @@ typedef int (splice_actor)(struct pipe_inode_info *, struct pipe_buffer *, typedef int (splice_direct_actor)(struct pipe_inode_info *, struct splice_desc *); -ssize_t filemap_splice_read(struct file *in, loff_t *ppos, - struct pipe_inode_info *pipe, - size_t len, unsigned int flags); - extern ssize_t splice_from_pipe(struct pipe_inode_info *, struct file *, loff_t *, size_t, unsigned int, splice_actor *); diff --git a/mm/filemap.c b/mm/filemap.c index e1ee267675d2..c01bbcb9fa92 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2846,9 +2846,8 @@ EXPORT_SYMBOL(generic_file_read_iter); /* * Splice subpages from a folio into a pipe. */ -static size_t splice_folio_into_pipe(struct pipe_inode_info *pipe, - struct folio *folio, - loff_t fpos, size_t size) +size_t splice_folio_into_pipe(struct pipe_inode_info *pipe, + struct folio *folio, loff_t fpos, size_t size) { struct page *page; size_t spliced = 0, offset = offset_in_folio(folio, fpos); diff --git a/mm/internal.h b/mm/internal.h index bcf75a8b032d..6d4ca98f3844 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -794,6 +794,12 @@ struct migration_target_control { gfp_t gfp_mask; }; +/* + * mm/filemap.c + */ +size_t splice_folio_into_pipe(struct pipe_inode_info *pipe, + struct folio *folio, loff_t fpos, size_t size); + /* * mm/vmalloc.c */ diff --git a/mm/shmem.c b/mm/shmem.c index 0005ab2c29af..5a3cc74aba28 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2711,6 +2711,128 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) return retval ? retval : error; } +static bool zero_pipe_buf_try_steal(struct pipe_inode_info *pipe, + struct pipe_buffer *buf) +{ + return false; +} + +static const struct pipe_buf_operations zero_pipe_buf_ops = { + .release = generic_pipe_buf_release, + .try_steal = zero_pipe_buf_try_steal, + .get = generic_pipe_buf_get, +}; + +static size_t splice_zeropage_into_pipe(struct pipe_inode_info *pipe, + loff_t fpos, size_t size) +{ + size_t offset = fpos & ~PAGE_MASK; + + size = min(size, PAGE_SIZE - offset); + + if (!pipe_full(pipe->head, pipe->tail, pipe->max_usage)) { + struct pipe_buffer *buf = pipe_head_buf(pipe); + + *buf = (struct pipe_buffer) { + .ops = &zero_pipe_buf_ops, + .page = ZERO_PAGE(0), + .offset = offset, + .len = size, + }; + get_page(buf->page); + pipe->head++; + } + + return size; +} + +static ssize_t shmem_file_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, + size_t len, unsigned int flags) +{ + struct inode *inode = file_inode(in); + struct address_space *mapping = inode->i_mapping; + struct folio *folio = NULL; + size_t total_spliced = 0, used, npages, n, part; + loff_t isize; + int error = 0; + + /* Work out how much data we can actually add into the pipe */ + used = pipe_occupancy(pipe->head, pipe->tail); + npages = max_t(ssize_t, pipe->max_usage - used, 0); + len = min_t(size_t, len, npages * PAGE_SIZE); + + do { + if (*ppos >= i_size_read(inode)) + break; + + error = shmem_get_folio(inode, *ppos / PAGE_SIZE, &folio, SGP_READ); + if (error) { + if (error == -EINVAL) + error = 0; + break; + } + if (folio) { + folio_unlock(folio); + + if (folio_test_hwpoison(folio)) { + error = -EIO; + break; + } + } + + /* + * i_size must be checked after we know the pages are Uptodate. + * + * Checking i_size after the check allows us to calculate + * the correct value for "nr", which means the zero-filled + * part of the page is not copied back to userspace (unless + * another truncate extends the file - this is desired though). + */ + isize = i_size_read(inode); + if (unlikely(*ppos >= isize)) + break; + part = min_t(loff_t, isize - *ppos, len); + + if (folio) { + /* + * If users can be writing to this page using arbitrary + * virtual addresses, take care about potential aliasing + * before reading the page on the kernel side. + */ + if (mapping_writably_mapped(mapping)) + flush_dcache_folio(folio); + folio_mark_accessed(folio); + /* + * Ok, we have the page, and it's up-to-date, so we can + * now splice it into the pipe. + */ + n = splice_folio_into_pipe(pipe, folio, *ppos, part); + folio_put(folio); + folio = NULL; + } else { + n = splice_zeropage_into_pipe(pipe, *ppos, len); + } + + if (!n) + break; + len -= n; + total_spliced += n; + *ppos += n; + in->f_ra.prev_pos = *ppos; + if (pipe_full(pipe->head, pipe->tail, pipe->max_usage)) + break; + + cond_resched(); + } while (len); + + if (folio) + folio_put(folio); + + file_accessed(in); + return total_spliced ? total_spliced : error; +} + static loff_t shmem_file_llseek(struct file *file, loff_t offset, int whence) { struct address_space *mapping = file->f_mapping; @@ -3929,7 +4051,7 @@ static const struct file_operations shmem_file_operations = { .read_iter = shmem_file_read_iter, .write_iter = generic_file_write_iter, .fsync = noop_fsync, - .splice_read = generic_file_splice_read, + .splice_read = shmem_file_splice_read, .splice_write = iter_file_splice_write, .fallocate = shmem_fallocate, #endif