From patchwork Wed May 31 12:45:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 101418 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2852536vqr; Wed, 31 May 2023 05:50:17 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4tOiXcj3q6eHJ5YsMZiCrInsJtVNsb3DgmhIDOF/oHuFSA11DLzhLCDQ3wX10g97m7zpDl X-Received: by 2002:a05:6a20:2451:b0:10b:bf2d:71bb with SMTP id t17-20020a056a20245100b0010bbf2d71bbmr5829413pzc.27.1685537416271; Wed, 31 May 2023 05:50:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685537416; cv=none; d=google.com; s=arc-20160816; b=gXqCCRqnxZPNe9pLZTzNe6gKTKWMWrEyqXIZz2R99FFZvbzb4UXj6cnPzbfkShQFF5 AhKpQtN553/KuJnomZpXjQ6k1gV2p9NrrUtz3+k4Ka7pEtqhT9mRb2EpEUWJX0DBQCw5 BZwmn1EGsmd2g3lIvKNxwf3vHyFKs5CG2L0FeFZ7qLNPdIjGc892CU4bhfCLut/F8BdK 1s4E9EeTXIKXwuL4XSquHpblQeTtI0PF2uvb0o1tFR2VrX3OymOqbcflZWm05wvtVFnK VXr7k793CVUhKfEehszDHkFmC2sFw7x+Z4yROnujF1CQhDOAlwN8B7mjuupHLQaXMp4A 4IVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=VUrQef8DmIGGqxD5N5RNkU1BHyTMou9oFGzjwHHlMuE=; b=qQ573rgZw56n5J4RcWUR5GQzLuS2+yf64vOC5C3hR8dU+vQHaSEVdKa1gqzoKazSOM SvbHfzYV/ttzpXTs10hLDQMoMwHOkbVxBtnVDiPse6f/EGd7wzhdSxkcbZ4AmMc8FUE5 1ikMB2TU5OPM+CtgjUjFVrgDIdoaX9fwiQlKX5AbAx5a//2XsqEJfSD4Aa4CI5Z4wKoh 7nA+swb5Qegdq5MTPotWbA2QbOfUMyzVwu1+V0CMwoEQlP+Lnq98m3qcxFwXZfvMfSPj cz7tWAMOkwp91sAABGqJgCT7HqbKHwJ0JhoUO4r8mlO/0nWwAwoOl7MV5ScOE6XUs6Vf ax4A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=H5yRwNeV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r85-20020a632b58000000b0053fb1a847adsi897820pgr.802.2023.05.31.05.50.03; Wed, 31 May 2023 05:50:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=H5yRwNeV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232718AbjEaMq3 (ORCPT + 99 others); Wed, 31 May 2023 08:46:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235901AbjEaMq2 (ORCPT ); Wed, 31 May 2023 08:46:28 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 02DCE124 for ; Wed, 31 May 2023 05:45:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685537142; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VUrQef8DmIGGqxD5N5RNkU1BHyTMou9oFGzjwHHlMuE=; b=H5yRwNeVIjOnlS5NbLxWGa6hsALP2DgsXpmWB5byNmH/bj/eNw71uz3bZdOwWFtI/nvJhU 5rWTUaAGkSL09UJIWE0YSaiDFR5GScNUu1OziMoAHBDYKBhVEbvvAVgAwEk2q4no7YnUE/ IN+QKUgcOkPNSI68si7CnfvicpooxVk= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-479-MLbnjYjsMtCMKcbnjqfptw-1; Wed, 31 May 2023 08:45:38 -0400 X-MC-Unique: MLbnjYjsMtCMKcbnjqfptw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DE0311C0150B; Wed, 31 May 2023 12:45:37 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1CF7F20296C8; Wed, 31 May 2023 12:45:34 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Hellwig , Linus Torvalds , Al Viro , Jan Kara , Jeff Layton , David Hildenbrand , Christian Brauner , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org Subject: [PATCH net-next v2 1/6] splice, net: Fix MSG_MORE signalling in splice_direct_to_actor() Date: Wed, 31 May 2023 13:45:23 +0100 Message-ID: <20230531124528.699123-2-dhowells@redhat.com> In-Reply-To: <20230531124528.699123-1-dhowells@redhat.com> References: <20230531124528.699123-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Spam-Status: No, score=-2.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767414081835188043?= X-GMAIL-MSGID: =?utf-8?q?1767414081835188043?= splice_direct_to_actor() doesn't manage SPLICE_F_MORE correctly - and, as a result, incorrectly signals MSG_MORE when splicing to a socket. The problem happens when a short splice occurs because we got a short read due to hitting the EOF on a file. Because the length read (read_len) is less than the remaining size to be spliced (len), SPLICE_F_MORE is set. This causes MSG_MORE to be set by pipe_to_sendpage(), indicating to the network protocol that more data is to be expected. With the changes I want to make to switch from using sendpage to using sendmsg(MSG_SPLICE_PAGES), MSG_MORE needs to work properly. This was observed with the multi_chunk_sendfile tests in the tls kselftest program. Some of those tests would hang and time out when the last chunk of file was less than the sendfile request size. This has been observed before[1] and worked around in AF_TLS[2]. Fix this by checking to see if the source file is seekable if we get a short read and, if it is, checking to see if we hit the file size. This should also work for block devices. This won't help procfiles and suchlike as they're zero length files that can be read from[3]. To handle that, should splice make a zero-length call with SPLICE_F_MORE cleared (assuming it wasn't set by userspace via splice()) if it gets a zero-length read? Signed-off-by: David Howells cc: Jakub Kicinski cc: Jens Axboe cc: Christoph Hellwig cc: Linus Torvalds cc: Al Viro cc: Matthew Wilcox cc: Jan Kara cc: Jeff Layton cc: David Hildenbrand cc: Christian Brauner cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: linux-fsdevel@vger.kernel.org cc: linux-block@vger.kernel.org cc: linux-mm@kvack.org cc: netdev@vger.kernel.org Link: https://lore.kernel.org/netdev/1591392508-14592-1-git-send-email-pooja.trivedi@stackpath.com/ [1] Link: https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=d452d48b9f8b1a7f8152d33ef52cfd7fe1735b0a [2] Link: https://lore.kernel.org/r/CAHk-=wjDq5_wLWrapzFiJ3ZNn6aGFWeMJpAj5q+4z-Ok8DD9dA@mail.gmail.com/ [3] --- fs/splice.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 3e06611d19ae..a7cf216c02a7 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -982,10 +982,21 @@ ssize_t splice_direct_to_actor(struct file *in, struct splice_desc *sd, * If this is the last data and SPLICE_F_MORE was not set * initially, clears it. */ - if (read_len < len) - sd->flags |= SPLICE_F_MORE; - else if (!more) + if (read_len < len) { + struct inode *ii = in->f_mapping->host; + + if (ii->i_fop->llseek != noop_llseek && + pos >= i_size_read(ii)) { + if (!more) + sd->flags &= ~SPLICE_F_MORE; + } else { + sd->flags |= SPLICE_F_MORE; + } + + } else if (!more) { sd->flags &= ~SPLICE_F_MORE; + } + /* * NOTE: nonblocking mode only applies to the input. We * must not do the output in nonblocking mode as then we From patchwork Wed May 31 12:45:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 101413 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2852169vqr; Wed, 31 May 2023 05:49:34 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7L6Ef8OYf9ktBu1qHjOy78PjM7YIu4eKLeClXwpdGLcjC7XHgu99YDYtSu2opBt0Du/Su8 X-Received: by 2002:a05:6a00:2191:b0:643:857d:87a3 with SMTP id h17-20020a056a00219100b00643857d87a3mr5210351pfi.11.1685537374287; Wed, 31 May 2023 05:49:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685537374; cv=none; d=google.com; s=arc-20160816; b=P5mZ8II0dDpUM2t3tqWb8tdLd6kFri9KnqXz9Sti2jvwRUTRYbtxqHbu098iB9LYN+ FWJZuLzgM7jJ2dLvejzz2WF2UESvV0mTz2VvX9vuvVSw5SS36VR9qbz1laQAiWAxQpNi cjGe2YXFhfRAls8gObw+xKysZoWu57QHRawgy+1LJKiWNrjdI8DnIks/E1aDBAKMSV05 DcSykOuy0CYOyeL8ApJMw3LuuRJyrrCICiKwF+4aWmkcoZ4//Fw9KkSsTiD0qTXcbsBf ETCAVG5iWV3Enecid/LKPFyg+GVZvWmpw7XhfKkyhq73Mq/rTv8Hc+xDKeeJTxY61GzV VwlQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=704Gul1es5vKB8f3fNlqLP0xfed1dEhsuWb+SLm3klg=; b=bD7SG57JImrURYFCGXiYbRwvTFJm60p3HeWTQZcqvGnYlXX5pXipbe0NeT+wVPMlAl VlfWOJfU2ZfT2IwXwz3BlKmg0qXVKCF13QAE8lmLRNE8ER6g1EX5dC7+JoEhFtef97OK q/s8OqBAYVb5CYDXvJhJVgGpzh8xToBNh+YFG0b6rmzKSA825+D2cj6vcZmSX7GBjtcW HsjxHUKUQiqkcAo7S/VIWmwHbyOiGISsGQZnEZ7zKFC88nrl/bBv1XnQoQfKNlvyfTiH lNZ/MPubaxJuNYme/VciVWj8vwlF37zAkoHg8XiL36FGY/N1UMI+CV/rIapfYetZXDiW z24w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="WEbs/vgq"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b124-20020a62cf82000000b0064f6a52105csi3527948pfg.378.2023.05.31.05.49.21; Wed, 31 May 2023 05:49:34 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="WEbs/vgq"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235944AbjEaMqf (ORCPT + 99 others); Wed, 31 May 2023 08:46:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43160 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235909AbjEaMqa (ORCPT ); Wed, 31 May 2023 08:46:30 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE7B7128 for ; Wed, 31 May 2023 05:45:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685537144; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=704Gul1es5vKB8f3fNlqLP0xfed1dEhsuWb+SLm3klg=; b=WEbs/vgqtdc7UsAEh8O4M0ghNHBj1PiB9QnqdNiMATXe6KuZ8WDckAVxnrIYpEeGtRZpVz Dgg/KbMZeVf2+T/0e7BHrIPzac6BBLQbKaYG70TI6YRnEFVgKwlbEQiY+ZZqfCwLCr/t0y 9hwdKHHc0uU/gAIZpQIyaU3KoQxhhLU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-625-u-3ocbAEPfKvWCDqcrmGEg-1; Wed, 31 May 2023 08:45:41 -0400 X-MC-Unique: u-3ocbAEPfKvWCDqcrmGEg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A721F3C0C897; Wed, 31 May 2023 12:45:40 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id B74EE112132E; Wed, 31 May 2023 12:45:38 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next v2 2/6] net: Block MSG_SENDPAGE_* from being passed to sendmsg() by userspace Date: Wed, 31 May 2023 13:45:24 +0100 Message-ID: <20230531124528.699123-3-dhowells@redhat.com> In-Reply-To: <20230531124528.699123-1-dhowells@redhat.com> References: <20230531124528.699123-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767414037981866008?= X-GMAIL-MSGID: =?utf-8?q?1767414037981866008?= It is necessary to allow MSG_SENDPAGE_* to be passed into ->sendmsg() to allow sendmsg(MSG_SPLICE_PAGES) to replace ->sendpage(). Unblocking them in the network protocol, however, allows these flags to be passed in by userspace too[1]. Fix this by marking MSG_SENDPAGE_NOPOLICY, MSG_SENDPAGE_NOTLAST and MSG_SENDPAGE_DECRYPTED as internal flags, which causes sendmsg() to object if they are passed to sendmsg() by userspace. Network protocol ->sendmsg() implementations can then allow them through. Note that it should be possible to remove MSG_SENDPAGE_NOTLAST once sendpage is removed as a whole slew of pages will be passed in in one go by splice through sendmsg, with MSG_MORE being set if it has more data waiting in the pipe. Signed-off-by: David Howells cc: Jakub Kicinski cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org Link: https://lore.kernel.org/r/20230526181338.03a99016@kernel.org/ [1] --- include/linux/socket.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/socket.h b/include/linux/socket.h index bd1cc3238851..3fd3436bc09f 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -339,7 +339,9 @@ struct ucred { #endif /* Flags to be cleared on entry by sendmsg and sendmmsg syscalls */ -#define MSG_INTERNAL_SENDMSG_FLAGS (MSG_SPLICE_PAGES) +#define MSG_INTERNAL_SENDMSG_FLAGS \ + (MSG_SPLICE_PAGES | MSG_SENDPAGE_NOPOLICY | MSG_SENDPAGE_NOTLAST | \ + MSG_SENDPAGE_DECRYPTED) /* Setsockoptions(2) level. Thanks to BSD these must match IPPROTO_xxx */ #define SOL_IP 0 From patchwork Wed May 31 12:45:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 101416 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2852443vqr; Wed, 31 May 2023 05:50:07 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7aAmrqG3XjYg5SicccRLNFJiI0cNa5C8ed/WrTUd/ptBFwTsP8ytbsz+QMflzytDCTvhOE X-Received: by 2002:a05:6a00:b50:b0:64c:a554:f577 with SMTP id p16-20020a056a000b5000b0064ca554f577mr6983835pfo.11.1685537407383; Wed, 31 May 2023 05:50:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685537407; cv=none; d=google.com; s=arc-20160816; b=DqUoEzPZBmJEnHv/bpBhkq0mrURFnXtjI0ntKyd+OaZ/dOExFWcwkEJNHmBldWCJt4 bFe7gGhNVxyjvOPYon4SzSJQWO3NyXLr5ilWXUnyZYMUwGPgVpA6Jx6v9ddpC5ieQSdC uQoMNhILOOl69Db6JekkbRA7+Xnyl8fUxAy6xppMwpFDxcbPy2LAGdvXH6C62XwilRX6 mgbDU+/rwtxOXcLzH6gKehsnRglncqbNCBvKd3aqkNO8k67zEsdClNDLPYfhLI1qI6TD O6ek9I1nIXpyV/qEVadnW3nwdzSFIthV5Z9nQQo81cSTqCXIaH2r13A5TMKoryLFURLZ oaQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=xFpJiBMaV6haKLFUiEEfF2pWt3wFFqyfTnw9cLbfIuI=; b=VoQvtMvNEerWXOcwkzKU0Fon4x/7PrYuJE8BAQFrKfTq7I1uKK/ZqhX5A7++E5bAwS A4WYr9gsTz8pHSpWv15tNpgoOWj5bbryMEUzxdpWnWkuYm/f4zCgbZXDS4bleRXZKZBW 3i2NqfA+2dUFQ37zNLGcAUSxXUxs+zMwXLXeLI42bGizfaKlA78lUmdllksH/rg+UUY6 8JkMG1FfvjnyxK8a3tLVMbcF2N0khIMVmoR8Kx7wGrTvvRlwX0A6UArGvPiQ3BXyzYsD xSDvGC1DywwL+q3csNOcWlQCVVm/3f7ixHbhjSlR0Wl1j5D7UG/fL61OVCPBLD5CGwtO dmGw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZE7fg6BI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n9-20020aa79849000000b0063d3aed5eb3si3508086pfq.197.2023.05.31.05.49.55; Wed, 31 May 2023 05:50:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZE7fg6BI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235946AbjEaMrL (ORCPT + 99 others); Wed, 31 May 2023 08:47:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43308 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235902AbjEaMrI (ORCPT ); Wed, 31 May 2023 08:47:08 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8BA9311F for ; Wed, 31 May 2023 05:45:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685537148; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xFpJiBMaV6haKLFUiEEfF2pWt3wFFqyfTnw9cLbfIuI=; b=ZE7fg6BIXSpMhLnW1clDpdZMGuwFlPPwrT4GfLeLtte4aZ40GZam6apjFQMk3M7vdDCbRm jokXXHV8C7jkofiVAF5OjRWlU2lDfl7umRAJ/aJ9On2aZLMxxM7Q3/OJbW5bArFCq4XdiR uqZQsMD3dYPzpLirFrtrRmS4DoOe+h8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-116-ZBHr42RPMCeBYoDi-5JaaA-1; Wed, 31 May 2023 08:45:44 -0400 X-MC-Unique: ZBHr42RPMCeBYoDi-5JaaA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C5504811E7F; Wed, 31 May 2023 12:45:43 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 99826112132C; Wed, 31 May 2023 12:45:41 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next v2 3/6] tls/sw: Support MSG_SPLICE_PAGES Date: Wed, 31 May 2023 13:45:25 +0100 Message-ID: <20230531124528.699123-4-dhowells@redhat.com> In-Reply-To: <20230531124528.699123-1-dhowells@redhat.com> References: <20230531124528.699123-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767414072360465417?= X-GMAIL-MSGID: =?utf-8?q?1767414072360465417?= Make TLS's sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #2) - "rls_" should be "tls_". net/tls/tls_sw.c | 46 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index 6e6a7c37d685..60d807b63f75 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -929,6 +929,38 @@ static int tls_sw_push_pending_record(struct sock *sk, int flags) &copied, flags); } +static int tls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg, + struct sk_msg *msg_pl, size_t try_to_copy, + ssize_t *copied) +{ + struct page *page = NULL, **pages = &page; + + do { + ssize_t part; + size_t off; + bool put = false; + + part = iov_iter_extract_pages(&msg->msg_iter, &pages, + try_to_copy, 1, 0, &off); + if (part <= 0) + return part ?: -EIO; + + if (WARN_ON_ONCE(!sendpage_ok(page))) { + iov_iter_revert(&msg->msg_iter, part); + return -EIO; + } + + sk_msg_page_add(msg_pl, page, part, off); + sk_mem_charge(sk, part); + if (put) + put_page(page); + *copied += part; + try_to_copy -= part; + } while (try_to_copy && !sk_msg_full(msg_pl)); + + return 0; +} + int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) { long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); @@ -1018,6 +1050,17 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) full_record = true; } + if (try_to_copy && (msg->msg_flags & MSG_SPLICE_PAGES)) { + ret = tls_sw_sendmsg_splice(sk, msg, msg_pl, + try_to_copy, &copied); + if (ret < 0) + goto send_end; + tls_ctx->pending_open_record_frags = true; + if (full_record || eor || sk_msg_full(msg_pl)) + goto copied; + continue; + } + if (!is_kvec && (full_record || eor) && !async_capable) { u32 first = msg_pl->sg.end; @@ -1080,8 +1123,9 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) /* Open records defined only if successfully copied, otherwise * we would trim the sg but not reset the open record frags. */ - tls_ctx->pending_open_record_frags = true; copied += try_to_copy; +copied: + tls_ctx->pending_open_record_frags = true; if (full_record || eor) { ret = bpf_exec_tx_verdict(msg_pl, sk, full_record, record_type, &copied, From patchwork Wed May 31 12:45:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 101419 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2852878vqr; Wed, 31 May 2023 05:50:52 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6wYwahlM0Y9uGNz+Tqr0i6ebDxSNwPEJaebGkng9GkF6LHsqQ6CuiRqSzDUWYy/8flMK0y X-Received: by 2002:a92:c6c3:0:b0:325:d689:6cd7 with SMTP id v3-20020a92c6c3000000b00325d6896cd7mr2381150ilm.16.1685537452697; Wed, 31 May 2023 05:50:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685537452; cv=none; d=google.com; s=arc-20160816; b=fJd3orLYDPnbHRfnaskl0YGoLBt71nPrqQtNgJP7LEUKJDtwGU1+4kloEZPDt3REBN arH+6IT4ItQONOVaqYk5hhxmlQK5UufV3KOh5jZauZjmLFmIzlsQE8mf3GxtJUqGgIWQ 3c4csktuEnEJ99DqsuFZrM31XVoNyVBqQ9ZZdqSz7LrqJ2OAxBNbY+epMqdXTcoHj1uW 8y4P5OEuZCRutIIIh2AZt5uBF5cfzG7fs8aOJdJOub8fDf1iYMp9/4JkjLTPNm6dhuJJ jky2esglvYN3bLl6DZlDNLtF2ObVcr0eVbh5+kvdiXVMNeCAQzbYThRvaLQSjLt6T+/k a3uA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=W8aEdMrYnuprZg7eZCUzLy+kKHthZDT31VQbMyJsAn8=; b=ASfXGm9fIucGj0XvSJFKJh5X9yqR4pXP5FJN49pLdFD8/TkB35PHTJ/tLnuCebc5G5 jNNMb82EK91VfFTdgwHaRbAqP7+JkUGksNCtme3oI2hw35XE92LCB5f/QKLS1u2uz10R x1/XcIdO0FoBEvk4aBEopzYuEwOmNk6B0XHYpvvzgk/KSO0y63XVGXFALzG7PFR85yw5 ps9NWYAcUFBDYYO2pa9FoH0+f4ngB4ABPOUA0ojbFfhgAWmtaBeZVyG71VS10lp91lmv CbwmWCPZVyObYhzB34QANNqppmho2k2iLU9ZacG9HGLvbUTs6E6X8nOFaaBnPJtYBKE7 HyvA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ShjY5812; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r85-20020a632b58000000b0053fb1a847adsi897820pgr.802.2023.05.31.05.50.40; Wed, 31 May 2023 05:50:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ShjY5812; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235963AbjEaMrP (ORCPT + 99 others); Wed, 31 May 2023 08:47:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233059AbjEaMrK (ORCPT ); Wed, 31 May 2023 08:47:10 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 87707129 for ; Wed, 31 May 2023 05:45:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685537153; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W8aEdMrYnuprZg7eZCUzLy+kKHthZDT31VQbMyJsAn8=; b=ShjY5812qP/ExB4WDJqrXmXVKXQ/Gk0bs0OguC20oTpz4LcvF7IDDvHpzJ5RQe2o6EMDZQ rd7KuxwTMiG/w9ouCAjaNB2VETPyNuJnqYHUeJxQLPfBGhTpHLn1MIRPfPJ2P5jrVvZkDB X7wuH6NdWEpvT4yOZ5vvTRPy61ki28Q= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-370-LbK9yftePiayyMeYHhVc5g-1; Wed, 31 May 2023 08:45:48 -0400 X-MC-Unique: LbK9yftePiayyMeYHhVc5g-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D0DF53825BA9; Wed, 31 May 2023 12:45:47 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id F3555140E963; Wed, 31 May 2023 12:45:44 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH net-next v2 4/6] tls/sw: Convert tls_sw_sendpage() to use MSG_SPLICE_PAGES Date: Wed, 31 May 2023 13:45:26 +0100 Message-ID: <20230531124528.699123-5-dhowells@redhat.com> In-Reply-To: <20230531124528.699123-1-dhowells@redhat.com> References: <20230531124528.699123-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Spam-Status: No, score=-2.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767414120183371175?= X-GMAIL-MSGID: =?utf-8?q?1767414120183371175?= Convert tls_sw_sendpage() and tls_sw_sendpage_locked() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. [!] Note that tls_sw_sendpage_locked() appears to have the wrong locking upstream. I think the caller will only hold the socket lock, but it should hold tls_ctx->tx_lock too. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: bpf@vger.kernel.org --- net/tls/tls_sw.c | 165 +++++++++-------------------------------------- 1 file changed, 31 insertions(+), 134 deletions(-) diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index 60d807b63f75..f63e4405cf34 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -961,7 +961,8 @@ static int tls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg, return 0; } -int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) +static int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, + size_t size) { long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); struct tls_context *tls_ctx = tls_get_ctx(sk); @@ -984,15 +985,6 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) int ret = 0; int pending; - if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | - MSG_CMSG_COMPAT)) - return -EOPNOTSUPP; - - ret = mutex_lock_interruptible(&tls_ctx->tx_lock); - if (ret) - return ret; - lock_sock(sk); - if (unlikely(msg->msg_controllen)) { ret = tls_process_cmsg(sk, msg, &record_type); if (ret) { @@ -1193,157 +1185,62 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) send_end: ret = sk_stream_error(sk, msg->msg_flags, ret); - - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); return copied > 0 ? copied : ret; } -static int tls_sw_do_sendpage(struct sock *sk, struct page *page, - int offset, size_t size, int flags) +int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) { - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); struct tls_context *tls_ctx = tls_get_ctx(sk); - struct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx); - struct tls_prot_info *prot = &tls_ctx->prot_info; - unsigned char record_type = TLS_RECORD_TYPE_DATA; - struct sk_msg *msg_pl; - struct tls_rec *rec; - int num_async = 0; - ssize_t copied = 0; - bool full_record; - int record_room; - int ret = 0; - bool eor; - - eor = !(flags & MSG_SENDPAGE_NOTLAST); - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); - - /* Call the sk_stream functions to manage the sndbuf mem. */ - while (size > 0) { - size_t copy, required_size; - - if (sk->sk_err) { - ret = -sk->sk_err; - goto sendpage_end; - } - - if (ctx->open_rec) - rec = ctx->open_rec; - else - rec = ctx->open_rec = tls_get_rec(sk); - if (!rec) { - ret = -ENOMEM; - goto sendpage_end; - } - - msg_pl = &rec->msg_plaintext; - - full_record = false; - record_room = TLS_MAX_PAYLOAD_SIZE - msg_pl->sg.size; - copy = size; - if (copy >= record_room) { - copy = record_room; - full_record = true; - } - - required_size = msg_pl->sg.size + copy + prot->overhead_size; - - if (!sk_stream_memory_free(sk)) - goto wait_for_sndbuf; -alloc_payload: - ret = tls_alloc_encrypted_msg(sk, required_size); - if (ret) { - if (ret != -ENOSPC) - goto wait_for_memory; - - /* Adjust copy according to the amount that was - * actually allocated. The difference is due - * to max sg elements limit - */ - copy -= required_size - msg_pl->sg.size; - full_record = true; - } - - sk_msg_page_add(msg_pl, page, copy, offset); - sk_mem_charge(sk, copy); - - offset += copy; - size -= copy; - copied += copy; - - tls_ctx->pending_open_record_frags = true; - if (full_record || eor || sk_msg_full(msg_pl)) { - ret = bpf_exec_tx_verdict(msg_pl, sk, full_record, - record_type, &copied, flags); - if (ret) { - if (ret == -EINPROGRESS) - num_async++; - else if (ret == -ENOMEM) - goto wait_for_memory; - else if (ret != -EAGAIN) { - if (ret == -ENOSPC) - ret = 0; - goto sendpage_end; - } - } - } - continue; -wait_for_sndbuf: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); -wait_for_memory: - ret = sk_stream_wait_memory(sk, &timeo); - if (ret) { - if (ctx->open_rec) - tls_trim_both_msgs(sk, msg_pl->sg.size); - goto sendpage_end; - } + int ret; - if (ctx->open_rec) - goto alloc_payload; - } + if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | + MSG_CMSG_COMPAT | MSG_SPLICE_PAGES | + MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY)) + return -EOPNOTSUPP; - if (num_async) { - /* Transmit if any encryptions have completed */ - if (test_and_clear_bit(BIT_TX_SCHEDULED, &ctx->tx_bitmask)) { - cancel_delayed_work(&ctx->tx_work.work); - tls_tx_records(sk, flags); - } - } -sendpage_end: - ret = sk_stream_error(sk, flags, ret); - return copied > 0 ? copied : ret; + ret = mutex_lock_interruptible(&tls_ctx->tx_lock); + if (ret) + return ret; + lock_sock(sk); + ret = tls_sw_sendmsg_locked(sk, msg, size); + release_sock(sk); + mutex_unlock(&tls_ctx->tx_lock); + return ret; } int tls_sw_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags) { + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + if (flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY | MSG_NO_SHARED_FRAGS)) return -EOPNOTSUPP; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - return tls_sw_do_sendpage(sk, page, offset, size, flags); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_sw_sendmsg_locked(sk, &msg, size); } int tls_sw_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tls_context *tls_ctx = tls_get_ctx(sk); - int ret; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; if (flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY)) return -EOPNOTSUPP; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - ret = mutex_lock_interruptible(&tls_ctx->tx_lock); - if (ret) - return ret; - lock_sock(sk); - ret = tls_sw_do_sendpage(sk, page, offset, size, flags); - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); - return ret; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_sw_sendmsg(sk, &msg, size); } static int From patchwork Wed May 31 12:45:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 101417 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2852529vqr; Wed, 31 May 2023 05:50:16 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ51EnwLrXgF9uTSkMcB6lVmaKrmHR4d/f3wVcesWRvMD6q8T8wODC9INdiwy5hAtiMLligh X-Received: by 2002:a17:903:1c5:b0:1b0:1564:39d7 with SMTP id e5-20020a17090301c500b001b0156439d7mr4318115plh.50.1685537415732; Wed, 31 May 2023 05:50:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685537415; cv=none; d=google.com; s=arc-20160816; b=Qhb7mhIQI7S5RnF/l298wEnjjhP2jVZSLgTZs3ttFCiaXneouwkMv29bCxaHwOdMOA OhGahmViQIAVpVzC0mzQZn0v7opFQErOza0ka/49AUr0FWKy+kSBCsFVOOa/ig+XiBJz IsJl3IlwlB9WhsC5Hx1+WU/4APM8jloZzt+WROVlk8xv3o0WUnkIQypplHBvBs/8+XVz ftd7m5/so7TtYppv0O8ESTGpFcUZsnnxp/NUL3jIuMG6A/k+nbrjQYiCKsQXYeqSN6Nw ocW/yJoFH7ZvCUhkm8EXOnf0BQIJan84OZvJ7Hj9iEz7zFRB5YI/HWBsyArNPQvVGesn oKAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=uuYRuRqPrqYrWBy2myOfoGGw2Qio81ibrygQkKyKemI=; b=v2xuFxHX01v+cxbkYM1SQcWID1d95zyH78bJEuNwx+N/GZW/6VFnWOjpfQIjGehkDo bjVam8el4VI5osv/x3SHdBGg+HFU7M6HBjyrqO5W6N+qYmuspOI3xTYLKvXbv6lV4357 StubTBxyDeAa7pOUPNTuhXRbR/9tdbjspOLBCTPOPQBXxa41hB+zfOoKWfwPzjCTMmqt 4CJ6HotIX0N5bZ/9d0DDlYmEfaLsmoVsK7X7kInDgDntmHLLLpYSqMMWkjgy5TYicrD7 Z8vTdkQpDd9nAJHObo+BFgzhb+IKSWVci/64zCCxiJgfM+azV4wMdm68ILzBBP3zw3RD +5Kg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=G09xyAbt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x8-20020a170902ec8800b001ac6a6f2dd3si813030plg.555.2023.05.31.05.50.03; Wed, 31 May 2023 05:50:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=G09xyAbt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235905AbjEaMrJ (ORCPT + 99 others); Wed, 31 May 2023 08:47:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43310 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233059AbjEaMrI (ORCPT ); Wed, 31 May 2023 08:47:08 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 10EF812E for ; Wed, 31 May 2023 05:45:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685537154; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uuYRuRqPrqYrWBy2myOfoGGw2Qio81ibrygQkKyKemI=; b=G09xyAbtaLd0YusgR+cF8uHNuSVjQQSuw1TsOzPj7hQdntCUi0JadTv4rYgXGARJSuDioR Fr5gGLzY1KcR0F9XnZMXtw9/gsFVvV8/zECc+g1yiEKURxPLYzvKPOR/yQCO+5YZp3S0Sw gbuUMpzuiIX8dPNmfV9Fuomq/deQrh0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-54-R6gsZvE5P7aGqYDd_4hZXw-1; Wed, 31 May 2023 08:45:51 -0400 X-MC-Unique: R6gsZvE5P7aGqYDd_4hZXw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 650F3811E7F; Wed, 31 May 2023 12:45:50 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 994162166B25; Wed, 31 May 2023 12:45:48 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next v2 5/6] tls/device: Support MSG_SPLICE_PAGES Date: Wed, 31 May 2023 13:45:27 +0100 Message-ID: <20230531124528.699123-6-dhowells@redhat.com> In-Reply-To: <20230531124528.699123-1-dhowells@redhat.com> References: <20230531124528.699123-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Spam-Status: No, score=-2.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767414081584082901?= X-GMAIL-MSGID: =?utf-8?q?1767414081584082901?= Make TLS's device sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls_device.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index a959572a816f..221fb30cba51 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -508,6 +508,29 @@ static int tls_push_data(struct sock *sk, tls_append_frag(record, &zc_pfrag, copy); iter_offset.offset += copy; + } else if (copy && (flags & MSG_SPLICE_PAGES)) { + struct page_frag zc_pfrag; + struct page **pages = &zc_pfrag.page; + size_t off; + + rc = iov_iter_extract_pages(iter_offset.msg_iter, + &pages, copy, 1, 0, &off); + if (rc <= 0) { + if (rc == 0) + rc = -EIO; + goto handle_error; + } + copy = rc; + + if (WARN_ON_ONCE(!sendpage_ok(zc_pfrag.page))) { + iov_iter_revert(iter_offset.msg_iter, copy); + rc = -EIO; + goto handle_error; + } + + zc_pfrag.offset = off; + zc_pfrag.size = copy; + tls_append_frag(record, &zc_pfrag, copy); } else if (copy) { copy = min_t(size_t, copy, pfrag->size - pfrag->offset); @@ -571,6 +594,9 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) union tls_iter_offset iter; int rc; + if (!tls_ctx->zerocopy_sendfile) + msg->msg_flags &= ~MSG_SPLICE_PAGES; + mutex_lock(&tls_ctx->tx_lock); lock_sock(sk); From patchwork Wed May 31 12:45:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 101424 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2855857vqr; Wed, 31 May 2023 05:56:30 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4xdvIuQ3ktRRpom+KbXnADDuXW6QDTX3a6s2hiLMwGzwRzfumn5PEFdgo4aC4zJmhyVMBT X-Received: by 2002:a05:6358:e48a:b0:127:ae14:200 with SMTP id by10-20020a056358e48a00b00127ae140200mr2188006rwb.1.1685537789702; Wed, 31 May 2023 05:56:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685537789; cv=none; d=google.com; s=arc-20160816; b=FqsJYb8SUKVjI9M8iY9Vm8INGSiQgPvmsLpD8zYGYPyeBGSLMckvKCn3O9mzmsKF1d gj1b5ZQhqBjhuwz/dZcyqSQK8p7m/OkcbIDbzHbeH6HMHitumqwySuo6fyB+dFJ0AqZR FlXs82G9JxFvJ00KvWn5vlbMJ9+vEUhbADCG2mEVhLvF7ujXcs2yyBixVWuNFF2zEJ33 iuSCTEfWPctr5A8syIsKxWVD5jd743rk25wU5DChxabe9ZI7bXfVHgIxUdrT9kHFJNz5 i5M8wOBZDd9/iyl7Y4uhJtnyzUAw0VIMJbfKITUX0Ya0EXdEWHk3uN7mhZSvEsnK13LW jGcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jiNOYIAVcjsuvSMtZGlGjCJlFKnRRxqCgJAQxTqAqH0=; b=CeX/FqJJ6XLsn8vJTzGiElkbKpO08dBykAOw3zPwZGuRnfhlQb1XkjyFdQ1sTIbL+d BNwMZde3gIu0/sc6AcJqF4dXMBEBC+um7eV3Eyqj6aDxgEq3ZxYeoffgrQcwvGxjv9hn phxSyF2+5VJH7uKUOzTjsmXCJgX3jBcekxczNxk3A3NGvNOO9Im/JiMnDWpzPqa+/kyi DCuZEmVB8v6amJzM9R3R04qaOIvrfUY/3CfOdyy7KSgOUenjNu3v07CYlpbmsq3Q/oiH EHHSLlZGqV4yduak54KdRbi4Trp9Jo6fK/OT7NWd7AZBwtP+pAP338frQGrp2URd6IPE srLg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=P2rFeRnD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x12-20020aa79a4c000000b0063b7645b97asi3528517pfj.380.2023.05.31.05.56.14; Wed, 31 May 2023 05:56:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=P2rFeRnD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235958AbjEaMrZ (ORCPT + 99 others); Wed, 31 May 2023 08:47:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43320 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235952AbjEaMrS (ORCPT ); Wed, 31 May 2023 08:47:18 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80415134 for ; Wed, 31 May 2023 05:45:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685537155; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jiNOYIAVcjsuvSMtZGlGjCJlFKnRRxqCgJAQxTqAqH0=; b=P2rFeRnDBXTO8rq+tGSr2ui7YwqxORfzS2p2woTWB5rktf2HKwYkRZjUW3MjzBUCla925K i61pGochNNWVPHAYMDQufTkufD9AhENE6pBySiKrRZuOXhzUMNnU967Hm1gBO+XAxt0GoR Ho5F3V9Tz8boDrzjhHxZdb9nJq/bf4o= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-198-cKdTb43SN-mmwECm3okGRg-1; Wed, 31 May 2023 08:45:54 -0400 X-MC-Unique: cKdTb43SN-mmwECm3okGRg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3AE7B8032E4; Wed, 31 May 2023 12:45:53 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2F12040C6EC4; Wed, 31 May 2023 12:45:51 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next v2 6/6] tls/device: Convert tls_device_sendpage() to use MSG_SPLICE_PAGES Date: Wed, 31 May 2023 13:45:28 +0100 Message-ID: <20230531124528.699123-7-dhowells@redhat.com> In-Reply-To: <20230531124528.699123-1-dhowells@redhat.com> References: <20230531124528.699123-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Spam-Status: No, score=-2.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767414473332221563?= X-GMAIL-MSGID: =?utf-8?q?1767414473332221563?= Convert tls_device_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. With that, the tls_iter_offset union is no longer necessary and can be replaced with an iov_iter pointer and the zc_page argument to tls_push_data() can also be removed. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls_device.c | 84 +++++++++++--------------------------------- 1 file changed, 20 insertions(+), 64 deletions(-) diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index 221fb30cba51..684171363dd9 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -422,16 +422,10 @@ static int tls_device_copy_data(void *addr, size_t bytes, struct iov_iter *i) return 0; } -union tls_iter_offset { - struct iov_iter *msg_iter; - int offset; -}; - static int tls_push_data(struct sock *sk, - union tls_iter_offset iter_offset, + struct iov_iter *iter, size_t size, int flags, - unsigned char record_type, - struct page *zc_page) + unsigned char record_type) { struct tls_context *tls_ctx = tls_get_ctx(sk); struct tls_prot_info *prot = &tls_ctx->prot_info; @@ -499,22 +493,13 @@ static int tls_push_data(struct sock *sk, record = ctx->open_record; copy = min_t(size_t, size, max_open_record_len - record->len); - if (copy && zc_page) { - struct page_frag zc_pfrag; - - zc_pfrag.page = zc_page; - zc_pfrag.offset = iter_offset.offset; - zc_pfrag.size = copy; - tls_append_frag(record, &zc_pfrag, copy); - - iter_offset.offset += copy; - } else if (copy && (flags & MSG_SPLICE_PAGES)) { + if (copy && (flags & MSG_SPLICE_PAGES)) { struct page_frag zc_pfrag; struct page **pages = &zc_pfrag.page; size_t off; - rc = iov_iter_extract_pages(iter_offset.msg_iter, - &pages, copy, 1, 0, &off); + rc = iov_iter_extract_pages(iter, &pages, + copy, 1, 0, &off); if (rc <= 0) { if (rc == 0) rc = -EIO; @@ -523,7 +508,7 @@ static int tls_push_data(struct sock *sk, copy = rc; if (WARN_ON_ONCE(!sendpage_ok(zc_pfrag.page))) { - iov_iter_revert(iter_offset.msg_iter, copy); + iov_iter_revert(iter, copy); rc = -EIO; goto handle_error; } @@ -536,7 +521,7 @@ static int tls_push_data(struct sock *sk, rc = tls_device_copy_data(page_address(pfrag->page) + pfrag->offset, copy, - iter_offset.msg_iter); + iter); if (rc) goto handle_error; tls_append_frag(record, pfrag, copy); @@ -591,7 +576,6 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) { unsigned char record_type = TLS_RECORD_TYPE_DATA; struct tls_context *tls_ctx = tls_get_ctx(sk); - union tls_iter_offset iter; int rc; if (!tls_ctx->zerocopy_sendfile) @@ -606,8 +590,8 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) goto out; } - iter.msg_iter = &msg->msg_iter; - rc = tls_push_data(sk, iter, size, msg->msg_flags, record_type, NULL); + rc = tls_push_data(sk, &msg->msg_iter, size, msg->msg_flags, + record_type); out: release_sock(sk); @@ -618,44 +602,18 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) int tls_device_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tls_context *tls_ctx = tls_get_ctx(sk); - union tls_iter_offset iter_offset; - struct iov_iter msg_iter; - char *kaddr; - struct kvec iov; - int rc; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; - - mutex_lock(&tls_ctx->tx_lock); - lock_sock(sk); + msg.msg_flags |= MSG_MORE; - if (flags & MSG_OOB) { - rc = -EOPNOTSUPP; - goto out; - } - - if (tls_ctx->zerocopy_sendfile) { - iter_offset.offset = offset; - rc = tls_push_data(sk, iter_offset, size, - flags, TLS_RECORD_TYPE_DATA, page); - goto out; - } - - kaddr = kmap(page); - iov.iov_base = kaddr + offset; - iov.iov_len = size; - iov_iter_kvec(&msg_iter, ITER_SOURCE, &iov, 1, size); - iter_offset.msg_iter = &msg_iter; - rc = tls_push_data(sk, iter_offset, size, flags, TLS_RECORD_TYPE_DATA, - NULL); - kunmap(page); + if (flags & MSG_OOB) + return -EOPNOTSUPP; -out: - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); - return rc; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_device_sendmsg(sk, &msg, size); } struct tls_record_info *tls_get_record(struct tls_offload_context_tx *context, @@ -720,12 +678,10 @@ EXPORT_SYMBOL(tls_get_record); static int tls_device_push_pending_record(struct sock *sk, int flags) { - union tls_iter_offset iter; - struct iov_iter msg_iter; + struct iov_iter iter; - iov_iter_kvec(&msg_iter, ITER_SOURCE, NULL, 0, 0); - iter.msg_iter = &msg_iter; - return tls_push_data(sk, iter, 0, flags, TLS_RECORD_TYPE_DATA, NULL); + iov_iter_kvec(&iter, ITER_SOURCE, NULL, 0, 0); + return tls_push_data(sk, &iter, 0, flags, TLS_RECORD_TYPE_DATA); } void tls_device_write_space(struct sock *sk, struct tls_context *ctx)