From patchwork Mon Jun 5 12:45:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 103241 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2661304vqr; Mon, 5 Jun 2023 05:52:33 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7IPhqa1t3avWgQm0CBdw+h+4zDwNOOXP/E5eT3XdBsEfF2Qa8YVd1y/LcP4KO/mCfFcz5F X-Received: by 2002:a17:90b:374a:b0:256:23f:abc5 with SMTP id ne10-20020a17090b374a00b00256023fabc5mr2923786pjb.10.1685969553153; Mon, 05 Jun 2023 05:52:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685969553; cv=none; d=google.com; s=arc-20160816; b=jUY3O6pKNkRPvSzQYSpXj7U88xBibLzH/k8L7JVFvoYnWzfvYpxAzfraZxdTDlQkmN O8uCf/jaMhade0yuzCWu0fdPVbaoUEGcM4OUNBX3+vWBZPromFPtf/LkmnYo9kazfYC8 8C/OwtOBPwngqERiJHcH8jg1RxTYG7dQQuWr9Mtj1RDEQV32tYeurcxrKYQ3ELbqipLB qCrOWitRzMW2JwYZfRVVDiygFSjCMICPewhk0tn7kUIpmkoaIinGC/IG6IXrT3UiG41C crK6a1V6Fh9asTso7sOfj/hv0HAwZ8jyBOG6fO1Z2vQDu3d+XHfB2AgmqXcv+/tafh2R 553Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=704Gul1es5vKB8f3fNlqLP0xfed1dEhsuWb+SLm3klg=; b=POAh7sSBOsQofDmEI6/Qdvv4oq79K5FC08DkMn5ErFbWuESXwg7DRkpi1zWvAWwBxL w9Rtp1vabcq6uKz+xl8JvJYpElHSzXm3G48w+m2Nx2CC4nHQsVl4AjJB8nI2Ep138ObC Yzb3+GpcHyR9OuxbKYoVg+o6ndzxNjopN4zA59gb8kGkHJyQ0zSXNrnZptseRz8CgEoq wyKmE3Ku81cwhXG16OyOc4DjQPJ8Xf7i1DFH8mWQOFDAJabTS1p8OsgKm5oXiRIyBp+o /dgh4L9IqBtlPlNZwXmynNOXjjJE2/2q14pFjGRcXweEwguUOt9M3KXYdZiWkWlzzMqD FVHw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=CHr9+w62; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h15-20020a63384f000000b005321c340960si5485424pgn.811.2023.06.05.05.52.19; Mon, 05 Jun 2023 05:52:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=CHr9+w62; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232564AbjFEMrF (ORCPT + 99 others); Mon, 5 Jun 2023 08:47:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39346 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232896AbjFEMrB (ORCPT ); Mon, 5 Jun 2023 08:47:01 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 491CB98 for ; Mon, 5 Jun 2023 05:46:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685969172; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=704Gul1es5vKB8f3fNlqLP0xfed1dEhsuWb+SLm3klg=; b=CHr9+w62TEpawCNwoKX4Pc8i7mGlzNcBlA0w9umac9MspqPJaSQe9c2uXe0H2GUNWh6iI1 +nSev6Y7jtoQRFj9r3XG435qkwQDnZSx6WVdXKH17dAyUCUooRlDVMQoSmWkWRBVmkJbvX ekXHZt4XML5/N+RqZt+XrLFJD95czvo= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-638-MPVBXKHIMYqM4_90ThZprA-1; Mon, 05 Jun 2023 08:46:09 -0400 X-MC-Unique: MPVBXKHIMYqM4_90ThZprA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 67A5C1C0878A; Mon, 5 Jun 2023 12:46:08 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 682677AE4; Mon, 5 Jun 2023 12:46:06 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org, Linus Torvalds Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next v4 01/11] net: Block MSG_SENDPAGE_* from being passed to sendmsg() by userspace Date: Mon, 5 Jun 2023 13:45:50 +0100 Message-ID: <20230605124600.1722160-2-dhowells@redhat.com> In-Reply-To: <20230605124600.1722160-1-dhowells@redhat.com> References: <20230605124600.1722160-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767867210225523455?= X-GMAIL-MSGID: =?utf-8?q?1767867210225523455?= It is necessary to allow MSG_SENDPAGE_* to be passed into ->sendmsg() to allow sendmsg(MSG_SPLICE_PAGES) to replace ->sendpage(). Unblocking them in the network protocol, however, allows these flags to be passed in by userspace too[1]. Fix this by marking MSG_SENDPAGE_NOPOLICY, MSG_SENDPAGE_NOTLAST and MSG_SENDPAGE_DECRYPTED as internal flags, which causes sendmsg() to object if they are passed to sendmsg() by userspace. Network protocol ->sendmsg() implementations can then allow them through. Note that it should be possible to remove MSG_SENDPAGE_NOTLAST once sendpage is removed as a whole slew of pages will be passed in in one go by splice through sendmsg, with MSG_MORE being set if it has more data waiting in the pipe. Signed-off-by: David Howells cc: Jakub Kicinski cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org Link: https://lore.kernel.org/r/20230526181338.03a99016@kernel.org/ [1] --- include/linux/socket.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/socket.h b/include/linux/socket.h index bd1cc3238851..3fd3436bc09f 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -339,7 +339,9 @@ struct ucred { #endif /* Flags to be cleared on entry by sendmsg and sendmmsg syscalls */ -#define MSG_INTERNAL_SENDMSG_FLAGS (MSG_SPLICE_PAGES) +#define MSG_INTERNAL_SENDMSG_FLAGS \ + (MSG_SPLICE_PAGES | MSG_SENDPAGE_NOPOLICY | MSG_SENDPAGE_NOTLAST | \ + MSG_SENDPAGE_DECRYPTED) /* Setsockoptions(2) level. Thanks to BSD these must match IPPROTO_xxx */ #define SOL_IP 0 From patchwork Mon Jun 5 12:45:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 103240 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2661049vqr; Mon, 5 Jun 2023 05:52:08 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7NYIKoCtrFp5qYozcEw5lP+fFxuc11QiYAgAOREjDa2Fy2vx6EQjJ/f6er36/Boq4HtjBm X-Received: by 2002:a05:6a21:118a:b0:10b:8221:3348 with SMTP id oj10-20020a056a21118a00b0010b82213348mr6243143pzb.50.1685969527950; Mon, 05 Jun 2023 05:52:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685969527; cv=none; d=google.com; s=arc-20160816; b=NwOhJ744eBcpUSdSVIuip5GrUyP6SUkRq6d2JNNprcfQD0Xk7O96LbXYKNtqZ7dQhT 6+lDBBscwfBuJGDwrfjVl35t0DZk7IX/pfFw5g6QdC0M9YX3mh1tTMIIU1FFiMIA8bT9 OgAzZ9SuEN23NZscx2AN04t97qxTFc8Jyqo4sEEouEo857TVqMTiepunGx0VR7O35x73 IRct/+1rAZQSxTgh/7EpAygrr+aos4rCsv023ccIbh+YZ7lgRgqM8aGFZAPI9Mzvaq6p 9UdtnUampmxf7Q4Ve2qb7he3ByjrunO8kFsa1bnYFklhBjbThRAL8rrb5C2yAs7aVtfy KkXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=GIUHoDHkYJ6R7Kj3TX2onke+6WDgXd+OHlld+dEFkxk=; b=iFRdxJanFEV7ZykrTlTsRR2VtVyddB03+5P7zrJ7pWmUamHH9wwmvdk/2UcGWuWjde jLP8HkMvxwLKzJo+E/WNbTZytHpvBebAIaAZxHXq6IyIjlf14RXEOQ0WH6OHE4LJTg3m d2KACnVtIgGsnBEtI1eptGlS1av2w2PaGKxywXYUOWYhLQwLanuohhkQsvR3ddv3Z/wB wPEU+lUeDkDp2y7/PVroY8DrzaFtmew1iam5QJQ8j2qSPxDUsne3lpoJ18K5fv7cRnVx BmIhGDOPoMuFQgl0UjUveF2PvYc50tfTf4HhDySHgQNttVslhbDWZfwEduyRaxUldgG4 YzBw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=hpvtNxd6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l2-20020a632502000000b0053b99f632c9si5508213pgl.842.2023.06.05.05.51.53; Mon, 05 Jun 2023 05:52:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=hpvtNxd6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233238AbjFEMrD (ORCPT + 99 others); Mon, 5 Jun 2023 08:47:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39394 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232564AbjFEMrB (ORCPT ); Mon, 5 Jun 2023 08:47:01 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96AB8CD for ; Mon, 5 Jun 2023 05:46:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685969174; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GIUHoDHkYJ6R7Kj3TX2onke+6WDgXd+OHlld+dEFkxk=; b=hpvtNxd6wF5AhCckUhdfIT3KIfAXCmv6N5LnfDE0zPR2MZ/0nZAF/dBJSv0yTMFOksk2VQ nSR+jJFqMGVClj72jOeraaORBa9yu0NrNaQ/AAtwzUc3MNGg8jVxrQJeDIXlkfzA/6a/dM pbOqW4gTGqX8GMr9JEvY0+Jo2MI2qtw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-317-SQ2IsWf4OIK8YfiAO4St3w-1; Mon, 05 Jun 2023 08:46:11 -0400 X-MC-Unique: SQ2IsWf4OIK8YfiAO4St3w-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1DB5285A5BA; Mon, 5 Jun 2023 12:46:11 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 243AF140E954; Mon, 5 Jun 2023 12:46:09 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org, Linus Torvalds Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next v4 02/11] tls: Allow MSG_SPLICE_PAGES but treat it as normal sendmsg Date: Mon, 5 Jun 2023 13:45:51 +0100 Message-ID: <20230605124600.1722160-3-dhowells@redhat.com> In-Reply-To: <20230605124600.1722160-1-dhowells@redhat.com> References: <20230605124600.1722160-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767867183845071425?= X-GMAIL-MSGID: =?utf-8?q?1767867183845071425?= Allow MSG_SPLICE_PAGES to be specified to sendmsg() but treat it as normal sendmsg for now. This means the data will just be copied until MSG_SPLICE_PAGES is handled. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls_device.c | 3 ++- net/tls/tls_sw.c | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index a959572a816f..9ef766e41c7a 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -447,7 +447,8 @@ static int tls_push_data(struct sock *sk, long timeo; if (flags & - ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST)) + ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | + MSG_SPLICE_PAGES)) return -EOPNOTSUPP; if (unlikely(sk->sk_err)) diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index 6e6a7c37d685..cac1adc968e8 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -953,7 +953,7 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) int pending; if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | - MSG_CMSG_COMPAT)) + MSG_CMSG_COMPAT | MSG_SPLICE_PAGES)) return -EOPNOTSUPP; ret = mutex_lock_interruptible(&tls_ctx->tx_lock); From patchwork Mon Jun 5 12:45:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 103250 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2670353vqr; Mon, 5 Jun 2023 06:05:37 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4eUbXCedkg27hhtnhfYeQVnd73tdMmtVrXAJePDK/zpC+Am1vwO8HhG9CpYq2TYJSQAaiA X-Received: by 2002:a05:6a21:108e:b0:10b:a464:a662 with SMTP id nl14-20020a056a21108e00b0010ba464a662mr2165279pzb.30.1685970336913; Mon, 05 Jun 2023 06:05:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685970336; cv=none; d=google.com; s=arc-20160816; b=nm/ptYg2CJeaa8OF0ejhzWs+QI7aMk791e+yi7SgB0JzcIUN3Dkcjmb23z3YVZ8mvC KJFaAC/zsmh+p0binPwt6H5sVIL+EifELddiwxCqfG1YaTC5t+meCxgUlcu5jGclbvS+ TGzXYrmhdH9BQw0CK1FdPMsOXTHi8zCGKCUqA6XImuPBdyEiUY0XiOrrgM7+BwV+49d3 zFnXEKtJPhDiZsjmVjF5TXgnjqHAyFrhDKUmYLncmJx5Awy7uIRoqiRutq58310X93nn mN6rtQ1tVLOpEs90lb9dBi5Sr2MvSBBdF5m1KLj0AeoMXwBj2EL06FmYEQ+rrwnrxqPa vKlQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=C5MeH3bkXFVlppirWEf/JAhtMhZSTtHegYn7rkL31b8=; b=StBLmvrgDSnWtsr1q0S57QtHpP2wqBe9M6+nUraHW3IM0+zpOqgHr9nrt+X/Js2LpT 7WrFSDGcUrCkX/RMcNhLfNqrlxEyMWpszQpIiafb3pYKh1CCzKs87m+qh4HGXz8VGlx8 prDVHQRe5OYQ0AGWg76PE0kI5AJXCuLy2txnEaJUKzz4mooLaMf6hfVNjgrb6aEu8sTJ MDRBaLC+210da1CJHEN8XPDrYbGXtKNBChVMJ6f+rRM0F0P0z0MUROaGFs4E0hYN8yQv w6op0A7ddF4BXt8l1bijzAftCuyuUkRXPKGXfN4AhwCkkRMP2bDGcfrQGiwULDLiIGH+ kQjQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=XN3QCrvg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z6-20020a63b046000000b0053fc68bc1b6si5358458pgo.42.2023.06.05.06.05.22; Mon, 05 Jun 2023 06:05:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=XN3QCrvg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233038AbjFEMrM (ORCPT + 99 others); Mon, 5 Jun 2023 08:47:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233461AbjFEMrH (ORCPT ); Mon, 5 Jun 2023 08:47:07 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91985A1 for ; Mon, 5 Jun 2023 05:46:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685969180; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=C5MeH3bkXFVlppirWEf/JAhtMhZSTtHegYn7rkL31b8=; b=XN3QCrvgbmhKKJn7AABwIx38A5l8fyJhfj7ecX/Qx+XK5T9J8IIfSqh1fw7PRp91c3Tp59 wpLOCCzK+hitbuxS0mMd93d9WaKP9WordXd/dtbxxUMBLpp8fc00ttOqu7qFZk1FTSyrRb +9b/C3dLpr9CK3pnBJC1HxmtgK6tAmk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-286-bdPEAlfdMymvJ-XIJ90puw-1; Mon, 05 Jun 2023 08:46:14 -0400 X-MC-Unique: bdPEAlfdMymvJ-XIJ90puw-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DFE8B85A5A8; Mon, 5 Jun 2023 12:46:13 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id D2BE6400F4E; Mon, 5 Jun 2023 12:46:11 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org, Linus Torvalds Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next v4 03/11] splice, net: Use sendmsg(MSG_SPLICE_PAGES) rather than ->sendpage() Date: Mon, 5 Jun 2023 13:45:52 +0100 Message-ID: <20230605124600.1722160-4-dhowells@redhat.com> In-Reply-To: <20230605124600.1722160-1-dhowells@redhat.com> References: <20230605124600.1722160-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767868032166686852?= X-GMAIL-MSGID: =?utf-8?q?1767868032166686852?= Replace generic_splice_sendpage() + splice_from_pipe + pipe_to_sendpage() with a net-specific handler, splice_to_socket(), that calls sendmsg() with MSG_SPLICE_PAGES set instead of calling ->sendpage(). MSG_MORE is used to indicate if the sendmsg() is expected to be followed with more data. This allows multiple pipe-buffer pages to be passed in a single call in a BVEC iterator, allowing the processing to be pushed down to a loop in the protocol driver. This helps pave the way for passing multipage folios down too. Protocols that haven't been converted to handle MSG_SPLICE_PAGES yet should just ignore it and do a normal sendmsg() for now - although that may be a bit slower as it may copy everything. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- fs/splice.c | 158 +++++++++++++++++++++++++++++++++-------- include/linux/fs.h | 2 - include/linux/splice.h | 2 + net/socket.c | 26 +------ 4 files changed, 131 insertions(+), 57 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 3e06611d19ae..9b1d43c0c562 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -33,6 +33,7 @@ #include #include #include +#include #include #include @@ -448,30 +449,6 @@ const struct pipe_buf_operations nosteal_pipe_buf_ops = { }; EXPORT_SYMBOL(nosteal_pipe_buf_ops); -/* - * Send 'sd->len' bytes to socket from 'sd->file' at position 'sd->pos' - * using sendpage(). Return the number of bytes sent. - */ -static int pipe_to_sendpage(struct pipe_inode_info *pipe, - struct pipe_buffer *buf, struct splice_desc *sd) -{ - struct file *file = sd->u.file; - loff_t pos = sd->pos; - int more; - - if (!likely(file->f_op->sendpage)) - return -EINVAL; - - more = (sd->flags & SPLICE_F_MORE) ? MSG_MORE : 0; - - if (sd->len < sd->total_len && - pipe_occupancy(pipe->head, pipe->tail) > 1) - more |= MSG_SENDPAGE_NOTLAST; - - return file->f_op->sendpage(file, buf->page, buf->offset, - sd->len, &pos, more); -} - static void wakeup_pipe_writers(struct pipe_inode_info *pipe) { smp_mb(); @@ -652,7 +629,7 @@ static void splice_from_pipe_end(struct pipe_inode_info *pipe, struct splice_des * Description: * This function does little more than loop over the pipe and call * @actor to do the actual moving of a single struct pipe_buffer to - * the desired destination. See pipe_to_file, pipe_to_sendpage, or + * the desired destination. See pipe_to_file, pipe_to_sendmsg, or * pipe_to_user. * */ @@ -833,8 +810,9 @@ iter_file_splice_write(struct pipe_inode_info *pipe, struct file *out, EXPORT_SYMBOL(iter_file_splice_write); +#ifdef CONFIG_NET /** - * generic_splice_sendpage - splice data from a pipe to a socket + * splice_to_socket - splice data from a pipe to a socket * @pipe: pipe to splice from * @out: socket to write to * @ppos: position in @out @@ -846,13 +824,131 @@ EXPORT_SYMBOL(iter_file_splice_write); * is involved. * */ -ssize_t generic_splice_sendpage(struct pipe_inode_info *pipe, struct file *out, - loff_t *ppos, size_t len, unsigned int flags) +ssize_t splice_to_socket(struct pipe_inode_info *pipe, struct file *out, + loff_t *ppos, size_t len, unsigned int flags) { - return splice_from_pipe(pipe, out, ppos, len, flags, pipe_to_sendpage); -} + struct socket *sock = sock_from_file(out); + struct bio_vec bvec[16]; + struct msghdr msg = {}; + ssize_t ret; + size_t spliced = 0; + bool need_wakeup = false; + + pipe_lock(pipe); + + while (len > 0) { + unsigned int head, tail, mask, bc = 0; + size_t remain = len; + + /* + * Check for signal early to make process killable when there + * are always buffers available + */ + ret = -ERESTARTSYS; + if (signal_pending(current)) + break; + + while (pipe_empty(pipe->head, pipe->tail)) { + ret = 0; + if (!pipe->writers) + goto out; + + if (spliced) + goto out; + + ret = -EAGAIN; + if (flags & SPLICE_F_NONBLOCK) + goto out; + + ret = -ERESTARTSYS; + if (signal_pending(current)) + goto out; + + if (need_wakeup) { + wakeup_pipe_writers(pipe); + need_wakeup = false; + } + + pipe_wait_readable(pipe); + } + + head = pipe->head; + tail = pipe->tail; + mask = pipe->ring_size - 1; + + while (!pipe_empty(head, tail)) { + struct pipe_buffer *buf = &pipe->bufs[tail & mask]; + size_t seg; -EXPORT_SYMBOL(generic_splice_sendpage); + if (!buf->len) { + tail++; + continue; + } + + seg = min_t(size_t, remain, buf->len); + seg = min_t(size_t, seg, PAGE_SIZE); + + ret = pipe_buf_confirm(pipe, buf); + if (unlikely(ret)) { + if (ret == -ENODATA) + ret = 0; + break; + } + + bvec_set_page(&bvec[bc++], buf->page, seg, buf->offset); + remain -= seg; + if (seg >= buf->len) + tail++; + if (bc >= ARRAY_SIZE(bvec)) + break; + } + + if (!bc) + break; + + msg.msg_flags = MSG_SPLICE_PAGES; + if (flags & SPLICE_F_MORE) + msg.msg_flags |= MSG_MORE; + if (remain && pipe_occupancy(pipe->head, tail) > 0) + msg.msg_flags |= MSG_MORE; + + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, bvec, bc, + len - remain); + ret = sock_sendmsg(sock, &msg); + if (ret <= 0) + break; + + spliced += ret; + len -= ret; + tail = pipe->tail; + while (ret > 0) { + struct pipe_buffer *buf = &pipe->bufs[tail & mask]; + size_t seg = min_t(size_t, ret, buf->len); + + buf->offset += seg; + buf->len -= seg; + ret -= seg; + + if (!buf->len) { + pipe_buf_release(pipe, buf); + tail++; + } + } + + if (tail != pipe->tail) { + pipe->tail = tail; + if (pipe->files) + need_wakeup = true; + } + } + +out: + pipe_unlock(pipe); + if (need_wakeup) + wakeup_pipe_writers(pipe); + return spliced ?: ret; +} +#endif static int warn_unsupported(struct file *file, const char *op) { diff --git a/include/linux/fs.h b/include/linux/fs.h index 21a981680856..f8254c3acf83 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2759,8 +2759,6 @@ extern ssize_t generic_file_splice_read(struct file *, loff_t *, struct pipe_inode_info *, size_t, unsigned int); extern ssize_t iter_file_splice_write(struct pipe_inode_info *, struct file *, loff_t *, size_t, unsigned int); -extern ssize_t generic_splice_sendpage(struct pipe_inode_info *pipe, - struct file *out, loff_t *, size_t len, unsigned int flags); extern long do_splice_direct(struct file *in, loff_t *ppos, struct file *out, loff_t *opos, size_t len, unsigned int flags); diff --git a/include/linux/splice.h b/include/linux/splice.h index a55179fd60fc..991ae318b6eb 100644 --- a/include/linux/splice.h +++ b/include/linux/splice.h @@ -84,6 +84,8 @@ extern long do_splice(struct file *in, loff_t *off_in, extern long do_tee(struct file *in, struct file *out, size_t len, unsigned int flags); +extern ssize_t splice_to_socket(struct pipe_inode_info *pipe, struct file *out, + loff_t *ppos, size_t len, unsigned int flags); /* * for dynamic pipe sizing diff --git a/net/socket.c b/net/socket.c index 3df96e9ba4e2..c4d9104418c8 100644 --- a/net/socket.c +++ b/net/socket.c @@ -57,6 +57,7 @@ #include #include #include +#include #include #include #include @@ -126,8 +127,6 @@ static long compat_sock_ioctl(struct file *file, unsigned int cmd, unsigned long arg); #endif static int sock_fasync(int fd, struct file *filp, int on); -static ssize_t sock_sendpage(struct file *file, struct page *page, - int offset, size_t size, loff_t *ppos, int more); static ssize_t sock_splice_read(struct file *file, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags); @@ -162,8 +161,7 @@ static const struct file_operations socket_file_ops = { .mmap = sock_mmap, .release = sock_close, .fasync = sock_fasync, - .sendpage = sock_sendpage, - .splice_write = generic_splice_sendpage, + .splice_write = splice_to_socket, .splice_read = sock_splice_read, .show_fdinfo = sock_show_fdinfo, }; @@ -1066,26 +1064,6 @@ int kernel_recvmsg(struct socket *sock, struct msghdr *msg, } EXPORT_SYMBOL(kernel_recvmsg); -static ssize_t sock_sendpage(struct file *file, struct page *page, - int offset, size_t size, loff_t *ppos, int more) -{ - struct socket *sock; - int flags; - int ret; - - sock = file->private_data; - - flags = (file->f_flags & O_NONBLOCK) ? MSG_DONTWAIT : 0; - /* more is a combination of MSG_MORE and MSG_SENDPAGE_NOTLAST */ - flags |= more; - - ret = kernel_sendpage(sock, page, offset, size, flags); - - if (trace_sock_send_length_enabled()) - call_trace_sock_send_length(sock->sk, ret, 0); - return ret; -} - static ssize_t sock_splice_read(struct file *file, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags) From patchwork Mon Jun 5 12:45:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 103242 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2661762vqr; Mon, 5 Jun 2023 05:53:29 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7dnpMurLDS8da+G1ZsfsqoY9Y2FCoFg1L2q2mT0e/V0d/VWbH5ItS0LO3zcjOv4EaRRSm2 X-Received: by 2002:a05:6359:2e9a:b0:129:c26d:e5ee with SMTP id rp26-20020a0563592e9a00b00129c26de5eemr3423474rwb.7.1685969609106; Mon, 05 Jun 2023 05:53:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685969609; cv=none; d=google.com; s=arc-20160816; b=v+d/wArz0zGnIccFmqUBc4sGGfWWr0zs/emvpPyhybd0x1cVgtyZ+Au78LTJ1z6wOV +lBq9V0kxB2bjzC2+VTa+ND1tj7FZrDlkaZZ3wlcka7n7OxDPKHTvv5VNxeV7Fly64pg CamfcxdDDXOY6OPPPz1y9mLGVWlJ4/dGzS6bjQrExAFziM5c3SXh1y4k/Kn/OHiHXd8P fempMgExKo165HA+PzzpXGv0hBXT3qVv1BWBxLcMdslq5UqyjDBW7GsBw6I5f6KlLAlg una3s9EFmnaFcq7WiqkuzEhdDedoHdau6/a1jLXkCRpGnjCVN8NVMCtFdjnTi2DJCZvJ veuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=nTV++T1qFJV7bEuDjQHfHExm7YDjffLohFjYna0MPZ0=; b=aBvzsdNA6AlkU17LfFtd043aSVJ1/CJnDJs9uegFl9Sz1bdtIVnFcxeDC7wh81owai LGDiQt0+roVuykn9BbEe2gWi1ooETBbOas648XXYatKSLB2S396kB5gAWFBZ6ySIpS12 OrFX/2rPIjv7vvHwqBPh9TdtD0sKS7XNJVRj4Rbq9zdzHba9MJFfvPgG+DUVAvk+A8C8 io9KcD3GHipupkxnS3hWFMxuqeEs5z98GvhJ+nXR1xuxNuFZJjsXZctHRmYlVTdnrJPZ 58vwgyneeacKrGJ0eMb7oZVOCNSKUlrotFiLWK5kRSAw0iCDgCCBOUODvhm6SRieFjTG 5AWA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="KUD2uP/C"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b18-20020a639312000000b00517ab01bb8fsi5668028pge.100.2023.06.05.05.53.13; Mon, 05 Jun 2023 05:53:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="KUD2uP/C"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233673AbjFEMrS (ORCPT + 99 others); Mon, 5 Jun 2023 08:47:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233567AbjFEMrJ (ORCPT ); Mon, 5 Jun 2023 08:47:09 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B0FB1C7 for ; Mon, 5 Jun 2023 05:46:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685969182; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nTV++T1qFJV7bEuDjQHfHExm7YDjffLohFjYna0MPZ0=; b=KUD2uP/CPkw7Ol4rEB8/O4sVtGnJdQAgX9GvpoWTznXDqxBhJ1JufwgZI9al8AZNyW6G1z jO2d17ZSzl/PrL/wK/Otai6U3SqRD0HRJtz0sHfdkca8z7frExO55tpIyKbXPvardO4FHS k6efHOMm/SEQ+ZtALR2esg7aId2xWRA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-601-gzBNOFPnODqMmUTrqVdFFg-1; Mon, 05 Jun 2023 08:46:18 -0400 X-MC-Unique: gzBNOFPnODqMmUTrqVdFFg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C07C3811E93; Mon, 5 Jun 2023 12:46:17 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id DBA884087C62; Mon, 5 Jun 2023 12:46:14 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org, Linus Torvalds Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Hellwig , Al Viro , Jan Kara , Jeff Layton , David Hildenbrand , Christian Brauner , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org Subject: [PATCH net-next v4 04/11] splice, net: Add a splice_eof op to file-ops and socket-ops Date: Mon, 5 Jun 2023 13:45:53 +0100 Message-ID: <20230605124600.1722160-5-dhowells@redhat.com> In-Reply-To: <20230605124600.1722160-1-dhowells@redhat.com> References: <20230605124600.1722160-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767867268450963701?= X-GMAIL-MSGID: =?utf-8?q?1767867268450963701?= Add an optional method, ->splice_eof(), to allow splice to indicate the premature termination of a splice to struct file_operations and struct proto_ops. This is called if sendfile() or splice() encounters all of the following conditions inside splice_direct_to_actor(): (1) the user did not set SPLICE_F_MORE (splice only), and (2) an EOF condition occurred (->splice_read() returned 0), and (3) we haven't read enough to fulfill the request (ie. len > 0 still), and (4) we have already spliced at least one byte. A further patch will modify the behaviour of SPLICE_F_MORE to always be passed to the actor if either the user set it or we haven't yet read sufficient data to fulfill the request. Suggested-by: Linus Torvalds Link: https://lore.kernel.org/r/CAHk-=wh=V579PDYvkpnTobCLGczbgxpMgGmmhqiTyE34Cpi5Gg@mail.gmail.com/ Signed-off-by: David Howells cc: Jakub Kicinski cc: Jens Axboe cc: Christoph Hellwig cc: Al Viro cc: Matthew Wilcox cc: Jan Kara cc: Jeff Layton cc: David Hildenbrand cc: Christian Brauner cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: linux-fsdevel@vger.kernel.org cc: linux-block@vger.kernel.org cc: linux-mm@kvack.org cc: netdev@vger.kernel.org --- fs/splice.c | 31 ++++++++++++++++++++++++++++++- include/linux/fs.h | 1 + include/linux/net.h | 1 + include/linux/splice.h | 1 + include/net/sock.h | 1 + net/socket.c | 10 ++++++++++ 6 files changed, 44 insertions(+), 1 deletion(-) diff --git a/fs/splice.c b/fs/splice.c index 9b1d43c0c562..3063f9a3ba62 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -969,6 +969,17 @@ static long do_splice_from(struct pipe_inode_info *pipe, struct file *out, return out->f_op->splice_write(pipe, out, ppos, len, flags); } +/* + * Indicate to the caller that there was a premature EOF when reading from the + * source and the caller didn't indicate they would be sending more data after + * this. + */ +static void do_splice_eof(struct splice_desc *sd) +{ + if (sd->splice_eof) + sd->splice_eof(sd); +} + /* * Attempt to initiate a splice from a file to a pipe. */ @@ -1068,7 +1079,7 @@ ssize_t splice_direct_to_actor(struct file *in, struct splice_desc *sd, ret = do_splice_to(in, &pos, pipe, len, flags); if (unlikely(ret <= 0)) - goto out_release; + goto read_failure; read_len = ret; sd->total_len = read_len; @@ -1108,6 +1119,15 @@ ssize_t splice_direct_to_actor(struct file *in, struct splice_desc *sd, file_accessed(in); return bytes; +read_failure: + /* + * If the user did *not* set SPLICE_F_MORE *and* we didn't hit that + * "use all of len" case that cleared SPLICE_F_MORE, *and* we did a + * "->splice_in()" that returned EOF (ie zero) *and* we have sent at + * least 1 byte *then* we will also do the ->splice_eof() call. + */ + if (ret == 0 && !more && len > 0 && bytes) + do_splice_eof(sd); out_release: /* * If we did an incomplete transfer we must release @@ -1136,6 +1156,14 @@ static int direct_splice_actor(struct pipe_inode_info *pipe, sd->flags); } +static void direct_file_splice_eof(struct splice_desc *sd) +{ + struct file *file = sd->u.file; + + if (file->f_op->splice_eof) + file->f_op->splice_eof(file); +} + /** * do_splice_direct - splices data directly between two files * @in: file to splice from @@ -1161,6 +1189,7 @@ long do_splice_direct(struct file *in, loff_t *ppos, struct file *out, .flags = flags, .pos = *ppos, .u.file = out, + .splice_eof = direct_file_splice_eof, .opos = opos, }; long ret; diff --git a/include/linux/fs.h b/include/linux/fs.h index f8254c3acf83..e393f2550300 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1796,6 +1796,7 @@ struct file_operations { int (*flock) (struct file *, int, struct file_lock *); ssize_t (*splice_write)(struct pipe_inode_info *, struct file *, loff_t *, size_t, unsigned int); ssize_t (*splice_read)(struct file *, loff_t *, struct pipe_inode_info *, size_t, unsigned int); + void (*splice_eof)(struct file *file); int (*setlease)(struct file *, long, struct file_lock **, void **); long (*fallocate)(struct file *file, int mode, loff_t offset, loff_t len); diff --git a/include/linux/net.h b/include/linux/net.h index b73ad8e3c212..8defc8f1d82e 100644 --- a/include/linux/net.h +++ b/include/linux/net.h @@ -210,6 +210,7 @@ struct proto_ops { int offset, size_t size, int flags); ssize_t (*splice_read)(struct socket *sock, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags); + void (*splice_eof)(struct socket *sock); int (*set_peek_off)(struct sock *sk, int val); int (*peek_len)(struct socket *sock); diff --git a/include/linux/splice.h b/include/linux/splice.h index 991ae318b6eb..4fab18a6e371 100644 --- a/include/linux/splice.h +++ b/include/linux/splice.h @@ -38,6 +38,7 @@ struct splice_desc { struct file *file; /* file to read/write */ void *data; /* cookie */ } u; + void (*splice_eof)(struct splice_desc *sd); /* Unexpected EOF handler */ loff_t pos; /* file position */ loff_t *opos; /* sendfile: output position */ size_t num_spliced; /* number of bytes already spliced */ diff --git a/include/net/sock.h b/include/net/sock.h index 656ea89f60ff..330b9c24ef70 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1267,6 +1267,7 @@ struct proto { size_t len, int flags, int *addr_len); int (*sendpage)(struct sock *sk, struct page *page, int offset, size_t size, int flags); + void (*splice_eof)(struct socket *sock); int (*bind)(struct sock *sk, struct sockaddr *addr, int addr_len); int (*bind_add)(struct sock *sk, diff --git a/net/socket.c b/net/socket.c index c4d9104418c8..b778fc03c6e0 100644 --- a/net/socket.c +++ b/net/socket.c @@ -130,6 +130,7 @@ static int sock_fasync(int fd, struct file *filp, int on); static ssize_t sock_splice_read(struct file *file, loff_t *ppos, struct pipe_inode_info *pipe, size_t len, unsigned int flags); +static void sock_splice_eof(struct file *file); #ifdef CONFIG_PROC_FS static void sock_show_fdinfo(struct seq_file *m, struct file *f) @@ -163,6 +164,7 @@ static const struct file_operations socket_file_ops = { .fasync = sock_fasync, .splice_write = splice_to_socket, .splice_read = sock_splice_read, + .splice_eof = sock_splice_eof, .show_fdinfo = sock_show_fdinfo, }; @@ -1076,6 +1078,14 @@ static ssize_t sock_splice_read(struct file *file, loff_t *ppos, return sock->ops->splice_read(sock, ppos, pipe, len, flags); } +static void sock_splice_eof(struct file *file) +{ + struct socket *sock = file->private_data; + + if (sock->ops->splice_eof) + sock->ops->splice_eof(sock); +} + static ssize_t sock_read_iter(struct kiocb *iocb, struct iov_iter *to) { struct file *file = iocb->ki_filp; From patchwork Mon Jun 5 12:45:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 103256 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2674321vqr; Mon, 5 Jun 2023 06:10:27 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ61iNGZGZ+QWTinsdZ6oXPZUxvBSEY5PeFIgvpv+ympk671Aw73rkevRI73BdrVJD0hmDGY X-Received: by 2002:a17:90b:f8d:b0:256:5d2b:1c48 with SMTP id ft13-20020a17090b0f8d00b002565d2b1c48mr2789550pjb.43.1685970627402; Mon, 05 Jun 2023 06:10:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685970627; cv=none; d=google.com; s=arc-20160816; b=pguKpMwiBCm4R6fo1Ame6Uvz7hftRtBPIuca2bhvVYggl9OY/3spPvMKGkpf8cnUT7 dwadRtaCfbNBFP+seTvfWqj8aWsy8yRz8KookRa81xZHxD7ZXvXshvLAr3HT8/Ccby+D Tv+2q7NX+2sPhTMf92JQMl716wmZnlt4AlCtVuYJDZmQ9WVbeTQ2CZi2jl7Fr+B36ukq lxpu9XpVkctqYmxEjhrLlkflzi5236krMQfWQZ4Ow1aojje404m3wiQ6iq+pztaKN/Lb fRqrzM60TqguEpz+J2K7Bnst8nob7ONBnfgjwipZHahZkY3L8BuTxyHr6p9zlQ8smyUq XjXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=uzwGRGnSqTM78sqFFubXIMZwBUumsiZD0dJpdzPncUQ=; b=p/gW81l2ssxjLI7bbBKfB2M3yeCPv8UmdyoMrxx73YFztVNvb7yodKaJKYkmnMDZA+ a5OsVzlvDr7OEh5F/NJm2nmdG2xVoEV+GB1SA+K+/QhRW971pgLMercrB03+ZIj3+Rzc 1MAwMxoFFQShbhJuZQCe/o/FmhuhPSjvsZqeBmcQi7flW9Cc8Wo9k5qlO/rmctsQI+8O 55aVehGdZ9S+DVi9sTwK3t3RmzYuoj5W7MjetaJiPuzGWL5BRLcUCtxuunkpHNQ1PSBM 22hfgw+ny8kak4egczBb7vv4ymAyN/zhblhheytvCa3I0yk0vW/1/Xugd2HyIKu0ChM3 SLhg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=BZShh1dS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 28-20020a17090a195c00b00256620f78d6si7094694pjh.189.2023.06.05.06.10.14; Mon, 05 Jun 2023 06:10:27 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=BZShh1dS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232157AbjFEMrO (ORCPT + 99 others); Mon, 5 Jun 2023 08:47:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39460 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233383AbjFEMrJ (ORCPT ); Mon, 5 Jun 2023 08:47:09 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0DDE7EC for ; Mon, 5 Jun 2023 05:46:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685969184; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uzwGRGnSqTM78sqFFubXIMZwBUumsiZD0dJpdzPncUQ=; b=BZShh1dSD14MjRLBrE/zWKfUNBFLfgKHepVJofaFIKmx6wTDvw4AmH3zgb4wDPUEFB1wmm w0i4dBZ1xrXFcM3l/QWWhRLjCtxsodb3DIqoYwIQ3Gukm0SqGIA/O170RDQQM5FmKZN/ih Z2liT8LzfD8qGZy576Xo74cBMjHX2no= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-356-G11wNmOlO3am4T5WkNjXTA-1; Mon, 05 Jun 2023 08:46:21 -0400 X-MC-Unique: G11wNmOlO3am4T5WkNjXTA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 48E47858F14; Mon, 5 Jun 2023 12:46:20 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6DAD82026D49; Mon, 5 Jun 2023 12:46:18 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org, Linus Torvalds Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next v4 05/11] tls/sw: Use splice_eof() to flush Date: Mon, 5 Jun 2023 13:45:54 +0100 Message-ID: <20230605124600.1722160-6-dhowells@redhat.com> In-Reply-To: <20230605124600.1722160-1-dhowells@redhat.com> References: <20230605124600.1722160-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767868336368334625?= X-GMAIL-MSGID: =?utf-8?q?1767868336368334625?= Allow splice to end a TLS record after prematurely ending a splice/sendfile due to getting an EOF condition (->splice_read() returned 0) after splice had called TLS with a sendmsg() with MSG_MORE set when the user didn't set MSG_MORE. Suggested-by: Linus Torvalds Link: https://lore.kernel.org/r/CAHk-=wh=V579PDYvkpnTobCLGczbgxpMgGmmhqiTyE34Cpi5Gg@mail.gmail.com/ Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls.h | 1 + net/tls/tls_main.c | 2 ++ net/tls/tls_sw.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 77 insertions(+) diff --git a/net/tls/tls.h b/net/tls/tls.h index 0672acab2773..4922668fefaa 100644 --- a/net/tls/tls.h +++ b/net/tls/tls.h @@ -97,6 +97,7 @@ void tls_update_rx_zc_capable(struct tls_context *tls_ctx); void tls_sw_strparser_arm(struct sock *sk, struct tls_context *ctx); void tls_sw_strparser_done(struct tls_context *tls_ctx); int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size); +void tls_sw_splice_eof(struct socket *sock); int tls_sw_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags); int tls_sw_sendpage(struct sock *sk, struct page *page, diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c index 3d45fdb5c4e9..83fa15e52af6 100644 --- a/net/tls/tls_main.c +++ b/net/tls/tls_main.c @@ -924,6 +924,7 @@ static void build_proto_ops(struct proto_ops ops[TLS_NUM_CONFIG][TLS_NUM_CONFIG] ops[TLS_BASE][TLS_BASE] = *base; ops[TLS_SW ][TLS_BASE] = ops[TLS_BASE][TLS_BASE]; + ops[TLS_SW ][TLS_BASE].splice_eof = tls_sw_splice_eof; ops[TLS_SW ][TLS_BASE].sendpage_locked = tls_sw_sendpage_locked; ops[TLS_BASE][TLS_SW ] = ops[TLS_BASE][TLS_BASE]; @@ -992,6 +993,7 @@ static void build_protos(struct proto prot[TLS_NUM_CONFIG][TLS_NUM_CONFIG], prot[TLS_SW][TLS_BASE] = prot[TLS_BASE][TLS_BASE]; prot[TLS_SW][TLS_BASE].sendmsg = tls_sw_sendmsg; + prot[TLS_SW][TLS_BASE].splice_eof = tls_sw_splice_eof; prot[TLS_SW][TLS_BASE].sendpage = tls_sw_sendpage; prot[TLS_BASE][TLS_SW] = prot[TLS_BASE][TLS_BASE]; diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index cac1adc968e8..7a6bb670073f 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -1155,6 +1155,80 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) return copied > 0 ? copied : ret; } +/* + * Handle unexpected EOF during splice without SPLICE_F_MORE set. + */ +void tls_sw_splice_eof(struct socket *sock) +{ + struct sock *sk = sock->sk; + struct tls_context *tls_ctx = tls_get_ctx(sk); + struct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx); + struct tls_rec *rec; + struct sk_msg *msg_pl; + ssize_t copied = 0; + bool retrying = false; + int ret = 0; + int pending; + + if (!ctx->open_rec) + return; + + mutex_lock(&tls_ctx->tx_lock); + lock_sock(sk); + +retry: + rec = ctx->open_rec; + if (!rec) + goto unlock; + + msg_pl = &rec->msg_plaintext; + + /* Check the BPF advisor and perform transmission. */ + ret = bpf_exec_tx_verdict(msg_pl, sk, false, TLS_RECORD_TYPE_DATA, + &copied, 0); + switch (ret) { + case 0: + case -EAGAIN: + if (retrying) + goto unlock; + retrying = true; + goto retry; + case -EINPROGRESS: + break; + default: + goto unlock; + } + + /* Wait for pending encryptions to get completed */ + spin_lock_bh(&ctx->encrypt_compl_lock); + ctx->async_notify = true; + + pending = atomic_read(&ctx->encrypt_pending); + spin_unlock_bh(&ctx->encrypt_compl_lock); + if (pending) + crypto_wait_req(-EINPROGRESS, &ctx->async_wait); + else + reinit_completion(&ctx->async_wait.completion); + + /* There can be no concurrent accesses, since we have no pending + * encrypt operations + */ + WRITE_ONCE(ctx->async_notify, false); + + if (ctx->async_wait.err) + goto unlock; + + /* Transmit if any encryptions have completed */ + if (test_and_clear_bit(BIT_TX_SCHEDULED, &ctx->tx_bitmask)) { + cancel_delayed_work(&ctx->tx_work.work); + tls_tx_records(sk, 0); + } + +unlock: + release_sock(sk); + mutex_unlock(&tls_ctx->tx_lock); +} + static int tls_sw_do_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { From patchwork Mon Jun 5 12:45:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 103258 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2674747vqr; Mon, 5 Jun 2023 06:10:59 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4JgzX9F7LwZ2IqzkON6a1gEfY+/p4eh6hX83lcNjzkPSejm9Vg8b8ixKv4uSBZya04ohvO X-Received: by 2002:a05:6a21:998e:b0:10b:91c5:45a8 with SMTP id ve14-20020a056a21998e00b0010b91c545a8mr7158749pzb.15.1685970659214; Mon, 05 Jun 2023 06:10:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685970659; cv=none; d=google.com; s=arc-20160816; b=HIHJ9VEavCCYYF9Yaexnqmt36ocry8i2WkxwCMnFJMm2bH5Mt+HEgvqmuHl2nbH35q QMfNgytNxHl+ZDwFE1AEYqvv0T3G1axu2/asGx58s7YuGsWRR0oDspVYOpzWmxn23AUj 8t8PKGinKZbiyFjzQZbxGz+2eD8caPPvTMVbb5+WGK2JIlEZLuGxeFY9Ngx+TxWb6OeK SrZE81U+K+bSTCjk0TknjmMPJgtLZQzH/B1vlukaGVd+XRlmTTngEoxBD/A6eOVKNUqq eqEw8qt6aNth1ecJfejLTsLuTNVdPhZb6kQDKt0cNI/djdjBIimoCLSWjou0dqeHysXa IhHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=uXvpsAJrK5w03LYiEcU7w+tO1rfwJo4A6Bo4opbn2SA=; b=YtxMfdAuCf+hyw5VVyue4umfVJcIBQUijxrI2pK2A7WRLcq5cUAXAabNDlp5hgRmot KxVxfbHwQjZTo9ocpHXFkmcM0nweJ+vp7F2xwShzAr2lD3xuHnv+DfI6S9ZR+X3k6CHT JO8JZKsJ4ylhAXzBZkb2Ip4bvk4KbCW/J3W4T7tNcWN5/LU5NCCtpeS+51Wy7U6/aZF2 4yJ2tV8OOoWlzGLacVHe32TP4E44XWtG/f5XVdLMSO7atUi5tu8dvjvNMgcLo8sqlAp6 Rd/LH53EN3zXFKvhLfqVJmTQLumNPchBv0Iv/Q8a/1uPCVNOidHjzJX6MMdq2V3zSofA +1xg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=XnB9Lj+f; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 125-20020a620683000000b0065171ffb053si5288309pfg.248.2023.06.05.06.10.46; Mon, 05 Jun 2023 06:10:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=XnB9Lj+f; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230324AbjFEMv3 (ORCPT + 99 others); Mon, 5 Jun 2023 08:51:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43188 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232237AbjFEMvW (ORCPT ); Mon, 5 Jun 2023 08:51:22 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 78119F3 for ; Mon, 5 Jun 2023 05:50:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685969416; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uXvpsAJrK5w03LYiEcU7w+tO1rfwJo4A6Bo4opbn2SA=; b=XnB9Lj+fZPCLK8aJdCPZh4KVFMtM1X+Ova7lPJ2Ky4huynMYQ0ynLvxom8CNOp/CtGdCAd qtVgGG5/gRow8Kfdt7fKZnbqPGQh0k1YQhUNSJjoK9ISxt+WzK8PHTnK7l8UDmuAFoYGXG Inz+fVcCg7JiEH+SBF+rgTijbi8YmL4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-407-n2hFhH1TMHWqAqgk0i2fyg-1; Mon, 05 Jun 2023 08:48:53 -0400 X-MC-Unique: n2hFhH1TMHWqAqgk0i2fyg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 112E1101B040; Mon, 5 Jun 2023 12:46:23 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0EDA1C154D1; Mon, 5 Jun 2023 12:46:20 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org, Linus Torvalds Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next v4 06/11] tls/device: Use splice_eof() to flush Date: Mon, 5 Jun 2023 13:45:55 +0100 Message-ID: <20230605124600.1722160-7-dhowells@redhat.com> In-Reply-To: <20230605124600.1722160-1-dhowells@redhat.com> References: <20230605124600.1722160-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767868369628808824?= X-GMAIL-MSGID: =?utf-8?q?1767868369628808824?= Allow splice to end a TLS record after prematurely ending a splice/sendfile due to getting an EOF condition (->splice_read() returned 0) after splice had called TLS with a sendmsg() with MSG_MORE set when the user didn't set MSG_MORE. Suggested-by: Linus Torvalds Link: https://lore.kernel.org/r/CAHk-=wh=V579PDYvkpnTobCLGczbgxpMgGmmhqiTyE34Cpi5Gg@mail.gmail.com/ Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls.h | 1 + net/tls/tls_device.c | 23 +++++++++++++++++++++++ net/tls/tls_main.c | 2 ++ 3 files changed, 26 insertions(+) diff --git a/net/tls/tls.h b/net/tls/tls.h index 4922668fefaa..d002c3af1966 100644 --- a/net/tls/tls.h +++ b/net/tls/tls.h @@ -116,6 +116,7 @@ ssize_t tls_sw_splice_read(struct socket *sock, loff_t *ppos, size_t len, unsigned int flags); int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size); +void tls_device_splice_eof(struct socket *sock); int tls_device_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags); int tls_tx_records(struct sock *sk, int flags); diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index 9ef766e41c7a..439be833dcf9 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -590,6 +590,29 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) return rc; } +void tls_device_splice_eof(struct socket *sock) +{ + struct sock *sk = sock->sk; + struct tls_context *tls_ctx = tls_get_ctx(sk); + union tls_iter_offset iter; + struct iov_iter iov_iter = {}; + + if (!tls_is_partially_sent_record(tls_ctx)) + return; + + mutex_lock(&tls_ctx->tx_lock); + lock_sock(sk); + + if (tls_is_partially_sent_record(tls_ctx)) { + iov_iter_bvec(&iov_iter, ITER_SOURCE, NULL, 0, 0); + iter.msg_iter = &iov_iter; + tls_push_data(sk, iter, 0, 0, TLS_RECORD_TYPE_DATA, NULL); + } + + release_sock(sk); + mutex_unlock(&tls_ctx->tx_lock); +} + int tls_device_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c index 83fa15e52af6..113f3ff91d6c 100644 --- a/net/tls/tls_main.c +++ b/net/tls/tls_main.c @@ -1009,10 +1009,12 @@ static void build_protos(struct proto prot[TLS_NUM_CONFIG][TLS_NUM_CONFIG], #ifdef CONFIG_TLS_DEVICE prot[TLS_HW][TLS_BASE] = prot[TLS_BASE][TLS_BASE]; prot[TLS_HW][TLS_BASE].sendmsg = tls_device_sendmsg; + prot[TLS_HW][TLS_BASE].splice_eof = tls_device_splice_eof; prot[TLS_HW][TLS_BASE].sendpage = tls_device_sendpage; prot[TLS_HW][TLS_SW] = prot[TLS_BASE][TLS_SW]; prot[TLS_HW][TLS_SW].sendmsg = tls_device_sendmsg; + prot[TLS_HW][TLS_SW].splice_eof = tls_device_splice_eof; prot[TLS_HW][TLS_SW].sendpage = tls_device_sendpage; prot[TLS_BASE][TLS_HW] = prot[TLS_BASE][TLS_SW]; From patchwork Mon Jun 5 12:45:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 103249 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2670331vqr; Mon, 5 Jun 2023 06:05:36 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ48OfhJfLJ4WXBkYo5WiuxtaPn4r/KjT4Q9h5f4N+ZZBmmMXb5a7OfZohJuvKNCD7IWUFYb X-Received: by 2002:a05:6a21:788d:b0:116:d935:f742 with SMTP id bf13-20020a056a21788d00b00116d935f742mr2582549pzc.20.1685970335613; Mon, 05 Jun 2023 06:05:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685970335; cv=none; d=google.com; s=arc-20160816; b=wwWaxjG0akkEdsBDSxmsAimcyWPIwSg2CQcaSqMRfKw9BYJD+NOGQ/+o4nBAMTgwNL jYx9TX8ac6Dc4NHtTxyPYjSjDKSAL8mEFSk3cPaHtPOoiNiPx9I4TdH8DVglyjQ0Pq/S t026D+f02Wjrdf0v2WbmhR6C3DhImZwQ2/eTuU9/fyYBXgPT/MrRnUEnSLXOw9pnrBGa UO97OIwQKDgxtWbw5vIcdrlvztJdPas2l1jt+Dody0KhRSZ4rruLgOoXbnQGUKgIKN24 Bmk6WnsbJABjyrC9WvmAJtlVwOR6mzNAfl1Z/lsNhxQaiPQOEdEAPG2rYrjXyf5Ie6Lf /CLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JUlHVh8MNfDhhBvWQPpsitslIPkAozPn8dJTCurx9p4=; b=c++R7HlLP9D+41b4SWHRfy+1Pzultl/EZtx6qewRGuxBuGKu4dSmAyrOP4eRXPAH0x xqIbZDFerWmfC+1Q0ok5S47h0e1ACmLHk+h6EJDHlrCJ0lx0ca+awEXfPEqRcpRIv3AT k4gQfC461/TR88qwYnNSkmnWBEnYOGYHm828Gl4gXhCmdfkfwlHWlyb0eiTrYK77bDhV 1lMEBpF4YM0bF7fuNvVSdiQG92o0s8ZTLzrqRSW2K76Crg91O/iThojWYVGTMnG/dRgI ufWkL+UptWFjrgu3PAUz6y/PLxu4vXc4SQN5V9cUTK9UPKj9qEVGoWi6soKgaGUyFx53 MZ8Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=EiWgDQDk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 20-20020a631854000000b0053ef518d7ffsi5421700pgy.303.2023.06.05.06.05.20; Mon, 05 Jun 2023 06:05:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=EiWgDQDk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233779AbjFEMri (ORCPT + 99 others); Mon, 5 Jun 2023 08:47:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233680AbjFEMrT (ORCPT ); Mon, 5 Jun 2023 08:47:19 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C16FE8 for ; Mon, 5 Jun 2023 05:46:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685969193; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JUlHVh8MNfDhhBvWQPpsitslIPkAozPn8dJTCurx9p4=; b=EiWgDQDkIPI+omQQ8ZAOxVrdPTS2HDIXITemFP8d4ZwVTGTmAs5bpu+sdofAVTaZJJDGfo bBI7zIhCdFUxSecrAw6PZfhd81t6g8VYAtO2uK+Tlnwcy/mCgL7StWSUD3Ow94lvvPdG6m o2gin3GdH5YEzdLkhNLdvXy7WKO/mfQ= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-502-xgRayiKmOY-kMAeSedZ3cg-1; Mon, 05 Jun 2023 08:46:28 -0400 X-MC-Unique: xgRayiKmOY-kMAeSedZ3cg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 92C5438294AB; Mon, 5 Jun 2023 12:46:26 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id A634CC1603B; Mon, 5 Jun 2023 12:46:23 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org, Linus Torvalds Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Hellwig , Al Viro , Jan Kara , Jeff Layton , David Hildenbrand , Christian Brauner , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org Subject: [PATCH net-next v4 07/11] splice, net: Fix SPLICE_F_MORE signalling in splice_direct_to_actor() Date: Mon, 5 Jun 2023 13:45:56 +0100 Message-ID: <20230605124600.1722160-8-dhowells@redhat.com> In-Reply-To: <20230605124600.1722160-1-dhowells@redhat.com> References: <20230605124600.1722160-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767868030520404077?= X-GMAIL-MSGID: =?utf-8?q?1767868030520404077?= splice_direct_to_actor() doesn't manage SPLICE_F_MORE correctly[1] - and, as a result, it incorrectly signals/fails to signal MSG_MORE when splicing to a socket. The problem I'm seeing happens when a short splice occurs because we got a short read due to hitting the EOF on a file: as the length read (read_len) is less than the remaining size to be spliced (len), SPLICE_F_MORE (and thus MSG_MORE) is set. The issue is that, for the moment, we have no way to know *why* the short read occurred and so can't make a good decision on whether we *should* keep MSG_MORE set. MSG_SENDPAGE_NOTLAST was added to work around this, but that is also set incorrectly under some circumstances - for example if a short read fills a single pipe_buffer, but the next read would return more (seqfile can do this). This was observed with the multi_chunk_sendfile tests in the tls kselftest program. Some of those tests would hang and time out when the last chunk of file was less than the sendfile request size: build/kselftest/net/tls -r tls.12_aes_gcm.multi_chunk_sendfile This has been observed before[2] and worked around in AF_TLS[3]. Fix this by making splice_direct_to_actor() always signal SPLICE_F_MORE if we haven't yet hit the requested operation size. SPLICE_F_MORE remains signalled if the user passed it in to splice() but otherwise gets cleared when we've read sufficient data to fulfill the request. If, however, we get a premature EOF from ->splice_read(), have sent at least one byte and SPLICE_F_MORE was not set by the caller, ->splice_eof() will be invoked. Signed-off-by: David Howells cc: Linus Torvalds cc: Jakub Kicinski cc: Jens Axboe cc: Christoph Hellwig cc: Al Viro cc: Matthew Wilcox cc: Jan Kara cc: Jeff Layton cc: David Hildenbrand cc: Christian Brauner cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: linux-fsdevel@vger.kernel.org cc: linux-block@vger.kernel.org cc: linux-mm@kvack.org cc: netdev@vger.kernel.org Link: https://lore.kernel.org/r/499791.1685485603@warthog.procyon.org.uk/ [1] Link: https://lore.kernel.org/r/1591392508-14592-1-git-send-email-pooja.trivedi@stackpath.com/ [2] Link: https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=d452d48b9f8b1a7f8152d33ef52cfd7fe1735b0a [3] --- Notes: ver #4) - Use ->splice_eof() to signal a premature EOF to the splice output. fs/splice.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 3063f9a3ba62..1be3ba622b0c 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -1063,13 +1063,17 @@ ssize_t splice_direct_to_actor(struct file *in, struct splice_desc *sd, */ bytes = 0; len = sd->total_len; + + /* Don't block on output, we have to drain the direct pipe. */ flags = sd->flags; + sd->flags &= ~SPLICE_F_NONBLOCK; /* - * Don't block on output, we have to drain the direct pipe. + * We signal MORE until we've read sufficient data to fulfill the + * request and we keep signalling it if the caller set it. */ - sd->flags &= ~SPLICE_F_NONBLOCK; more = sd->flags & SPLICE_F_MORE; + sd->flags |= SPLICE_F_MORE; WARN_ON_ONCE(!pipe_empty(pipe->head, pipe->tail)); @@ -1085,14 +1089,12 @@ ssize_t splice_direct_to_actor(struct file *in, struct splice_desc *sd, sd->total_len = read_len; /* - * If more data is pending, set SPLICE_F_MORE - * If this is the last data and SPLICE_F_MORE was not set - * initially, clears it. + * If we now have sufficient data to fulfill the request then + * we clear SPLICE_F_MORE if it was not set initially. */ - if (read_len < len) - sd->flags |= SPLICE_F_MORE; - else if (!more) + if (read_len >= len && !more) sd->flags &= ~SPLICE_F_MORE; + /* * NOTE: nonblocking mode only applies to the input. We * must not do the output in nonblocking mode as then we From patchwork Mon Jun 5 12:45:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 103255 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2674332vqr; Mon, 5 Jun 2023 06:10:28 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4WXx08I9DSV3CD9H3BQlODv3QQiaJXiGjSGrqUfPP8tDs8ueEvGVKt8GDcF9zTRQIz45U2 X-Received: by 2002:a05:6a21:100b:b0:10e:d3a0:cddf with SMTP id nk11-20020a056a21100b00b0010ed3a0cddfmr6756412pzb.12.1685970628392; Mon, 05 Jun 2023 06:10:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685970628; cv=none; d=google.com; s=arc-20160816; b=LfGJ8nt09nWS5g4XpwZ/mU6liVeVd7fft0W460We21RCciRHgngJOtvA5s26XcKVbw +HKGBrFqRvQr71EoUavbZ7wXkWnR2pcf3O0hCOFADb7P/1M1l6eKqmSVRxWBoUDmcLCF Yij9ZMV0Epz+lNJCdjs73DkxW9bAIfgMSFiEf1xvHwEh3129Die1bV83xWz4Sva+vePw uzCD8vGCVr07dNLnkLKPN0GK0BiWDtl1r42VJrlHSCv3zIHrVMl5GbNjMreBJh9PAJQx W2YLCI+7JJunWEkY9ZotbDiIwj4Wvj++bdAHSYD+ecLwBrXMEdDbNejgNJ3Tm1z7I85M WO5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=W0FWRRqlHW2dml6zeyANP94ZNOMoghLMR9ZqTCxdb24=; b=D8W6w0TQ7olYeYKm6fSbVvoUyqaQNDMDUqkLa3o0FTcEWGuzUQ8GmE9rEookFL8EEc RLZRgOZVatbwvOFcdR8Ho/WzNrgjKxL6G33JZVbGYgpyrE8iJM4Nf7b5NcyNUjLoHIfI Bb+E0PxCni0hX4+NTRz4ru/BGKSr7kPFSi9sAFsbs5B8iWLsLZAt8nIa1W5atDqxdo1X L0GIDCfu7crbsRz74zH0IYSjzO/vHWyw6vE0BIq4vnLYGAxS5xz0NDsuxhnnccagULFf k2O+mAbVVz3LD6AcZBEH4W4R3856KtXpYuPc35RAjAtP0jgh8vQ/3nG3IZl9/pqwH1Hz x/SQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZmXjLkUq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s22-20020a639256000000b00534882d5325si5652184pgn.96.2023.06.05.06.10.15; Mon, 05 Jun 2023 06:10:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZmXjLkUq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233771AbjFEMre (ORCPT + 99 others); Mon, 5 Jun 2023 08:47:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39560 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233657AbjFEMrO (ORCPT ); Mon, 5 Jun 2023 08:47:14 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3878BFD for ; Mon, 5 Jun 2023 05:46:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685969193; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W0FWRRqlHW2dml6zeyANP94ZNOMoghLMR9ZqTCxdb24=; b=ZmXjLkUqIViZ1v3uV3oYe8ThAujivfwlJagek8G7dsR9R8g+t3auPBG/YqZsxin5phOrnb 55iW89JTkh4AJKikZwehYGVZ3N+LkTdw/sALiFS7JL4f/kHM+rgkSJ/eBq9+cEnaDCM1+z QLPk5VHX62ha2DPCuhKipJpHXi9nGaU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-217-qmCbG5JcO_-fBCTGdBFm8w-1; Mon, 05 Jun 2023 08:46:30 -0400 X-MC-Unique: qmCbG5JcO_-fBCTGdBFm8w-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9153980120A; Mon, 5 Jun 2023 12:46:29 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 73B754087C62; Mon, 5 Jun 2023 12:46:27 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org, Linus Torvalds Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next v4 08/11] tls/sw: Support MSG_SPLICE_PAGES Date: Mon, 5 Jun 2023 13:45:57 +0100 Message-ID: <20230605124600.1722160-9-dhowells@redhat.com> In-Reply-To: <20230605124600.1722160-1-dhowells@redhat.com> References: <20230605124600.1722160-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767868337537260760?= X-GMAIL-MSGID: =?utf-8?q?1767868337537260760?= Make TLS's sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #2) - "rls_" should be "tls_". net/tls/tls_sw.c | 46 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index 7a6bb670073f..b85f92be7c9d 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -929,6 +929,38 @@ static int tls_sw_push_pending_record(struct sock *sk, int flags) &copied, flags); } +static int tls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg, + struct sk_msg *msg_pl, size_t try_to_copy, + ssize_t *copied) +{ + struct page *page = NULL, **pages = &page; + + do { + ssize_t part; + size_t off; + bool put = false; + + part = iov_iter_extract_pages(&msg->msg_iter, &pages, + try_to_copy, 1, 0, &off); + if (part <= 0) + return part ?: -EIO; + + if (WARN_ON_ONCE(!sendpage_ok(page))) { + iov_iter_revert(&msg->msg_iter, part); + return -EIO; + } + + sk_msg_page_add(msg_pl, page, part, off); + sk_mem_charge(sk, part); + if (put) + put_page(page); + *copied += part; + try_to_copy -= part; + } while (try_to_copy && !sk_msg_full(msg_pl)); + + return 0; +} + int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) { long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); @@ -1018,6 +1050,17 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) full_record = true; } + if (try_to_copy && (msg->msg_flags & MSG_SPLICE_PAGES)) { + ret = tls_sw_sendmsg_splice(sk, msg, msg_pl, + try_to_copy, &copied); + if (ret < 0) + goto send_end; + tls_ctx->pending_open_record_frags = true; + if (full_record || eor || sk_msg_full(msg_pl)) + goto copied; + continue; + } + if (!is_kvec && (full_record || eor) && !async_capable) { u32 first = msg_pl->sg.end; @@ -1080,8 +1123,9 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) /* Open records defined only if successfully copied, otherwise * we would trim the sg but not reset the open record frags. */ - tls_ctx->pending_open_record_frags = true; copied += try_to_copy; +copied: + tls_ctx->pending_open_record_frags = true; if (full_record || eor) { ret = bpf_exec_tx_verdict(msg_pl, sk, full_record, record_type, &copied, From patchwork Mon Jun 5 12:45:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 103243 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2662230vqr; Mon, 5 Jun 2023 05:54:28 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7dspKs4jFvI06MlEty10SpDVk2PGmj7XmXCEMjBgLSXoP7gBpDDAZWqutIWdolymyPb9rO X-Received: by 2002:a05:6a20:8e19:b0:ff:a017:2b07 with SMTP id y25-20020a056a208e1900b000ffa0172b07mr3600925pzj.20.1685969668236; Mon, 05 Jun 2023 05:54:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685969668; cv=none; d=google.com; s=arc-20160816; b=fex+edEn4Q6Ctzdm6LmqOel5miIsWNcFJXFmOfEikFWg9r6sKQ2juBv9VNeNXYDGNm hTbg/UA3wOjzY5NfPyompGse+meKkzTGUauUMTG7pqEAitwwvni5Ro35mXPJLCIU5zM5 /XqJXL2OL6lHCAtc3PAGL5Ub1EHg2pxxcatOsRev4chIN+HypFdxTScojDVw2RooPIQC KB+utElEW+Rqglbb9xjCTjl5JU6ooM4wNVh9YqTxCsAYDKY8QgH6gOPUGQYl93PJvqxk GRD1A8Pcf6mlcXIjaUmZme9+a6KbZ6oJL70QgfWwJ/779ruOZkj9I5SbRJo4DBJYtdc9 cM1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=AoX0zr5g/7kYp0xKccyyKWT5aRpif55zj/+ZZKuK9zI=; b=ccOUbO2U4BhcteMhPvWxD1V26bkh8Ffg2/Phw/wDnMGmT7AEkWWm0LBznmO7K8ZuYC leSVHXkiQsD5BtRsqPUt+YYpHHTb0Wg/xqtf6BTXoxliWXkTHXUOMeAgrEiBdbgLudc/ zXSfL/METlpwaPRs7oRGzK499kbUGB0ElJyqFnu14T8P/llVR27CMim47ntYNrqejLmA oCATVtTJEeryTvrLCBLK78MqdoTTclFLJX/oKwWNMj/tlBulDf1mXjIJlqjklzFKsi3b ZNMiS4w7b0VslxYFzO7zRkiAb4gJfzvEUn9LOgYUGUkY6EO04LTNSrt6VnvIM5/3i+Sf qNJg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=UmSsXQ7S; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p24-20020a63f458000000b0051f17a517adsi5560684pgk.492.2023.06.05.05.54.13; Mon, 05 Jun 2023 05:54:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=UmSsXQ7S; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233790AbjFEMrk (ORCPT + 99 others); Mon, 5 Jun 2023 08:47:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39578 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233690AbjFEMrY (ORCPT ); Mon, 5 Jun 2023 08:47:24 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 84F6CDC for ; Mon, 5 Jun 2023 05:46:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685969199; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AoX0zr5g/7kYp0xKccyyKWT5aRpif55zj/+ZZKuK9zI=; b=UmSsXQ7SQH4bfV5NmFGjtH1D1iQN7TC/CyyA4c25EYcYN2MG5BsbpIZCBiifqY2KoSxyb3 XEmysS95mbVjMjyCC8fne2/doE1nnGUdh/cTwE61yMiUj9RWguFixvz4PFCg69qnotb7W4 tXlksINlYEnVQYAlmYrZxvIsErQT1ow= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-169-FYm9rYGUOayGBbwg_JM_LQ-1; Mon, 05 Jun 2023 08:46:33 -0400 X-MC-Unique: FYm9rYGUOayGBbwg_JM_LQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D75DF811E92; Mon, 5 Jun 2023 12:46:32 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 368DA4087C68; Mon, 5 Jun 2023 12:46:30 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org, Linus Torvalds Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH net-next v4 09/11] tls/sw: Convert tls_sw_sendpage() to use MSG_SPLICE_PAGES Date: Mon, 5 Jun 2023 13:45:58 +0100 Message-ID: <20230605124600.1722160-10-dhowells@redhat.com> In-Reply-To: <20230605124600.1722160-1-dhowells@redhat.com> References: <20230605124600.1722160-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767867330907233300?= X-GMAIL-MSGID: =?utf-8?q?1767867330907233300?= Convert tls_sw_sendpage() and tls_sw_sendpage_locked() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. [!] Note that tls_sw_sendpage_locked() appears to have the wrong locking upstream. I think the caller will only hold the socket lock, but it should hold tls_ctx->tx_lock too. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: bpf@vger.kernel.org --- net/tls/tls_sw.c | 173 ++++++++++------------------------------------- 1 file changed, 35 insertions(+), 138 deletions(-) diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index b85f92be7c9d..5ffb8de862f6 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -961,7 +961,8 @@ static int tls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg, return 0; } -int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) +static int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, + size_t size) { long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); struct tls_context *tls_ctx = tls_get_ctx(sk); @@ -984,15 +985,6 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) int ret = 0; int pending; - if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | - MSG_CMSG_COMPAT | MSG_SPLICE_PAGES)) - return -EOPNOTSUPP; - - ret = mutex_lock_interruptible(&tls_ctx->tx_lock); - if (ret) - return ret; - lock_sock(sk); - if (unlikely(msg->msg_controllen)) { ret = tls_process_cmsg(sk, msg, &record_type); if (ret) { @@ -1193,10 +1185,27 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) send_end: ret = sk_stream_error(sk, msg->msg_flags, ret); + return copied > 0 ? copied : ret; +} +int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) +{ + struct tls_context *tls_ctx = tls_get_ctx(sk); + int ret; + + if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | + MSG_CMSG_COMPAT | MSG_SPLICE_PAGES | + MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY)) + return -EOPNOTSUPP; + + ret = mutex_lock_interruptible(&tls_ctx->tx_lock); + if (ret) + return ret; + lock_sock(sk); + ret = tls_sw_sendmsg_locked(sk, msg, size); release_sock(sk); mutex_unlock(&tls_ctx->tx_lock); - return copied > 0 ? copied : ret; + return ret; } /* @@ -1273,151 +1282,39 @@ void tls_sw_splice_eof(struct socket *sock) mutex_unlock(&tls_ctx->tx_lock); } -static int tls_sw_do_sendpage(struct sock *sk, struct page *page, - int offset, size_t size, int flags) -{ - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); - struct tls_context *tls_ctx = tls_get_ctx(sk); - struct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx); - struct tls_prot_info *prot = &tls_ctx->prot_info; - unsigned char record_type = TLS_RECORD_TYPE_DATA; - struct sk_msg *msg_pl; - struct tls_rec *rec; - int num_async = 0; - ssize_t copied = 0; - bool full_record; - int record_room; - int ret = 0; - bool eor; - - eor = !(flags & MSG_SENDPAGE_NOTLAST); - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); - - /* Call the sk_stream functions to manage the sndbuf mem. */ - while (size > 0) { - size_t copy, required_size; - - if (sk->sk_err) { - ret = -sk->sk_err; - goto sendpage_end; - } - - if (ctx->open_rec) - rec = ctx->open_rec; - else - rec = ctx->open_rec = tls_get_rec(sk); - if (!rec) { - ret = -ENOMEM; - goto sendpage_end; - } - - msg_pl = &rec->msg_plaintext; - - full_record = false; - record_room = TLS_MAX_PAYLOAD_SIZE - msg_pl->sg.size; - copy = size; - if (copy >= record_room) { - copy = record_room; - full_record = true; - } - - required_size = msg_pl->sg.size + copy + prot->overhead_size; - - if (!sk_stream_memory_free(sk)) - goto wait_for_sndbuf; -alloc_payload: - ret = tls_alloc_encrypted_msg(sk, required_size); - if (ret) { - if (ret != -ENOSPC) - goto wait_for_memory; - - /* Adjust copy according to the amount that was - * actually allocated. The difference is due - * to max sg elements limit - */ - copy -= required_size - msg_pl->sg.size; - full_record = true; - } - - sk_msg_page_add(msg_pl, page, copy, offset); - sk_mem_charge(sk, copy); - - offset += copy; - size -= copy; - copied += copy; - - tls_ctx->pending_open_record_frags = true; - if (full_record || eor || sk_msg_full(msg_pl)) { - ret = bpf_exec_tx_verdict(msg_pl, sk, full_record, - record_type, &copied, flags); - if (ret) { - if (ret == -EINPROGRESS) - num_async++; - else if (ret == -ENOMEM) - goto wait_for_memory; - else if (ret != -EAGAIN) { - if (ret == -ENOSPC) - ret = 0; - goto sendpage_end; - } - } - } - continue; -wait_for_sndbuf: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); -wait_for_memory: - ret = sk_stream_wait_memory(sk, &timeo); - if (ret) { - if (ctx->open_rec) - tls_trim_both_msgs(sk, msg_pl->sg.size); - goto sendpage_end; - } - - if (ctx->open_rec) - goto alloc_payload; - } - - if (num_async) { - /* Transmit if any encryptions have completed */ - if (test_and_clear_bit(BIT_TX_SCHEDULED, &ctx->tx_bitmask)) { - cancel_delayed_work(&ctx->tx_work.work); - tls_tx_records(sk, flags); - } - } -sendpage_end: - ret = sk_stream_error(sk, flags, ret); - return copied > 0 ? copied : ret; -} - int tls_sw_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags) { + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + if (flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY | MSG_NO_SHARED_FRAGS)) return -EOPNOTSUPP; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - return tls_sw_do_sendpage(sk, page, offset, size, flags); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_sw_sendmsg_locked(sk, &msg, size); } int tls_sw_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tls_context *tls_ctx = tls_get_ctx(sk); - int ret; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; if (flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY)) return -EOPNOTSUPP; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - ret = mutex_lock_interruptible(&tls_ctx->tx_lock); - if (ret) - return ret; - lock_sock(sk); - ret = tls_sw_do_sendpage(sk, page, offset, size, flags); - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); - return ret; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_sw_sendmsg(sk, &msg, size); } static int From patchwork Mon Jun 5 12:45:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 103253 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2673895vqr; Mon, 5 Jun 2023 06:09:53 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4PMfghGoCmiV5px2vC8XpVX7/uD7OR5CgBaH4/kTzF7YlcJ+F874k+TPoycNYyqVJwHZiu X-Received: by 2002:a17:902:728b:b0:1ae:2e0d:b38c with SMTP id d11-20020a170902728b00b001ae2e0db38cmr2797767pll.12.1685970593673; Mon, 05 Jun 2023 06:09:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685970593; cv=none; d=google.com; s=arc-20160816; b=yiWswkZpd4zxWv4m9MTrHId2+d4l2/Vw0SFRRtOggnyCK+NdlSV/J7ihfjv94lWXyJ LJNuNhNkh//JtpoK2t816Oj8wfLgDXesyhjW/VXXqgYWccR4gbuIjmxp8I0HYNWjrCGS bC611YOcsTtf0uVOyUUoGO09C11NPdlYbaoZlqoRiaVUpp9KnBU9rznp/cYqp2MIFnUU vhcfFWEZkx1qOWrtH5KbeW8wPzcisUo/tWgJ3/0q15Oxalhf7VgQkybdM0KXDNDNEGol 8XRN3PY1P1UYmqXfuJsQjDQPDDmWayNDpYmwRSo3j851UjWUWjcsGR+sELx3SQoj6KQo RpjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/RO1yxPz5vv1zCyaq/Wi9xLZBTYHqYC8FIFy+4PrNrg=; b=PKZpkhNo5NnElSBnOo+h3VyBAQD5A8IukPpI+YcDUJpUZsYart6mH4cVXx0hDRN1LG +O/8zmXtr4gRCGR+5/8zd9+vlroo3ZIysP27GLodUHoiaR3g44wcbeP67UrmOhPMg5Kz 84NEYyv4hEh56Zken2BdYC0Q2pxkkIrXHWfvQb7T/8+pPmjGa9GyV4GkXEzCdvVDnNdi 57CyRVoQ8QY8CwSl0mFZ4vUUHnl2rifFN5WdjF980fgpEAU7bc2tjEmQuKJNYu/Hmxff 43l+ARruV8G0Pwi8lbdtPlke2QH+O6OxZNMIw3HXKLEl5tRJygxUwa0o0wEWGdrS9kBW jZCg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=HuItWBqq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l8-20020a170902f68800b001ae5fe35b6dsi5731570plg.177.2023.06.05.06.09.40; Mon, 05 Jun 2023 06:09:53 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=HuItWBqq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233801AbjFEMro (ORCPT + 99 others); Mon, 5 Jun 2023 08:47:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233692AbjFEMrZ (ORCPT ); Mon, 5 Jun 2023 08:47:25 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 872FBEA for ; Mon, 5 Jun 2023 05:46:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685969202; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/RO1yxPz5vv1zCyaq/Wi9xLZBTYHqYC8FIFy+4PrNrg=; b=HuItWBqqcmi8V1eZ49EJ/UODFJNLjMDtLynE+CgjA5ad3PsM8JwFzqO/SApvOLG0ZadZjF ZXCys4h918alAR2/h2twBCokwuQEl2RspaoG7ph4CV9iSYZvkqtRYmxoSVJSrJmcWIp2c/ 53Qx5NVyf3Ktr3jK2213UROfr7R+3uI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-280-n91873ZbOZGlKJe05F1TyQ-1; Mon, 05 Jun 2023 08:46:36 -0400 X-MC-Unique: n91873ZbOZGlKJe05F1TyQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BBF8285A5A8; Mon, 5 Jun 2023 12:46:35 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id A0877140E954; Mon, 5 Jun 2023 12:46:33 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org, Linus Torvalds Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next v4 10/11] tls/device: Support MSG_SPLICE_PAGES Date: Mon, 5 Jun 2023 13:45:59 +0100 Message-ID: <20230605124600.1722160-11-dhowells@redhat.com> In-Reply-To: <20230605124600.1722160-1-dhowells@redhat.com> References: <20230605124600.1722160-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767868301254465950?= X-GMAIL-MSGID: =?utf-8?q?1767868301254465950?= Make TLS's device sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls_device.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index 439be833dcf9..bb3bb523544e 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -509,6 +509,29 @@ static int tls_push_data(struct sock *sk, tls_append_frag(record, &zc_pfrag, copy); iter_offset.offset += copy; + } else if (copy && (flags & MSG_SPLICE_PAGES)) { + struct page_frag zc_pfrag; + struct page **pages = &zc_pfrag.page; + size_t off; + + rc = iov_iter_extract_pages(iter_offset.msg_iter, + &pages, copy, 1, 0, &off); + if (rc <= 0) { + if (rc == 0) + rc = -EIO; + goto handle_error; + } + copy = rc; + + if (WARN_ON_ONCE(!sendpage_ok(zc_pfrag.page))) { + iov_iter_revert(iter_offset.msg_iter, copy); + rc = -EIO; + goto handle_error; + } + + zc_pfrag.offset = off; + zc_pfrag.size = copy; + tls_append_frag(record, &zc_pfrag, copy); } else if (copy) { copy = min_t(size_t, copy, pfrag->size - pfrag->offset); @@ -572,6 +595,9 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) union tls_iter_offset iter; int rc; + if (!tls_ctx->zerocopy_sendfile) + msg->msg_flags &= ~MSG_SPLICE_PAGES; + mutex_lock(&tls_ctx->tx_lock); lock_sock(sk); From patchwork Mon Jun 5 12:46:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 103252 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2673378vqr; Mon, 5 Jun 2023 06:09:15 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5J3+fyTzBsLMEhN8cUIUN9KTTzAyasYaBObjMY54evmv7X+Gz+PkU4rtvUt/CM1nwA+7Wh X-Received: by 2002:a05:6a20:e619:b0:10a:be5c:6e2d with SMTP id my25-20020a056a20e61900b0010abe5c6e2dmr1815182pzb.39.1685970555407; Mon, 05 Jun 2023 06:09:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685970555; cv=none; d=google.com; s=arc-20160816; b=Yk1knbg8sOFrRO+uyNzthlfWmrHkntl1II9vEj0nrNwOeHmAI4857gsmCJffLlipo7 ewyKimPdDO3ylb1IvBZ7xphl8noPC6LCt8o392d9ZDXImg05LJ51AZNZSMXL8m544Uol rwHE2PfpaAKiQs9LMpUYN9xf09fncZIlsNh7sI2lGfRId8EivbnNy4+9DC6LQetOGxWT sEVcs5WFtVHzsftgGJRle36lS3xklOdmZtKF7AQE1vhKmABe9UckckB7SGZTWaqpyh2T plzh26xqOSeagHuyom/y/Q6U3RpapILHWqc+Xt1AcPROEwDUj0FKowa+JNlOAqvCWLaV 6rag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=fV2I/xZhOL8TFCqO/iJxKQbBwABt3P8CReEOZMBw48A=; b=YF7VY/4+ugh7KvDF4hwoyFvIg3BjcIR7vnwXqJj9zg82iH6juvg0Pqc+AB1Ku8yBQm iHodg/7DEq2eWS7m0KdNWD7X6eiE2bdNNXPpp2N5TcMt4cCY4GuEAUbcC8/V3e3MedY7 qoNeU4/gB/xGmZeFUp8/M/FAEKQ7xD6p8Lp4yCWXgts/TDNy0G1bxOPPBtoSMhk515BX ErwNeMZBYMCCDJrTWKHq+Jk9seOdaMu3Dt+sgAn/N/Yk78+Nb6LbbHQO4JJL5M4avJJh 3Y88walA7/b0anmv4DR86HN6JN9sB2XsMcLiYdkM3GQ+WVmU/g1rz9BRCYrbsXkusbVq SOkQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=YOWB5TKp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i185-20020a636dc2000000b00542649c0f4esi5367038pgc.43.2023.06.05.06.09.02; Mon, 05 Jun 2023 06:09:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=YOWB5TKp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233716AbjFEMrs (ORCPT + 99 others); Mon, 5 Jun 2023 08:47:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39604 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233695AbjFEMrZ (ORCPT ); Mon, 5 Jun 2023 08:47:25 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED561E6 for ; Mon, 5 Jun 2023 05:46:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685969202; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fV2I/xZhOL8TFCqO/iJxKQbBwABt3P8CReEOZMBw48A=; b=YOWB5TKpqX5b1EFmEOLp+lVfM9z9/muaZFGyxMm8AvN2g579M500HF+NoIchOfDUbCHffw YmgPd/cP+qcU20FG2mSQ/hpod5cVi4hfn1x9kBQyz6fAzicGNKTJPvOFnoUo9GbAFPRuP5 oF53SOpiChqysqkIDbthvr1+AWxOfPA= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-494-Y1UVt5u7MwKDkWoplay2iw-1; Mon, 05 Jun 2023 08:46:39 -0400 X-MC-Unique: Y1UVt5u7MwKDkWoplay2iw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 38D8A1C05EC8; Mon, 5 Jun 2023 12:46:38 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6A3987AE4; Mon, 5 Jun 2023 12:46:36 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org, Linus Torvalds Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next v4 11/11] tls/device: Convert tls_device_sendpage() to use MSG_SPLICE_PAGES Date: Mon, 5 Jun 2023 13:46:00 +0100 Message-ID: <20230605124600.1722160-12-dhowells@redhat.com> In-Reply-To: <20230605124600.1722160-1-dhowells@redhat.com> References: <20230605124600.1722160-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767868260556482275?= X-GMAIL-MSGID: =?utf-8?q?1767868260556482275?= Convert tls_device_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. With that, the tls_iter_offset union is no longer necessary and can be replaced with an iov_iter pointer and the zc_page argument to tls_push_data() can also be removed. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/tls/tls_device.c | 92 +++++++++++--------------------------------- 1 file changed, 23 insertions(+), 69 deletions(-) diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index bb3bb523544e..b4864d55900f 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -422,16 +422,10 @@ static int tls_device_copy_data(void *addr, size_t bytes, struct iov_iter *i) return 0; } -union tls_iter_offset { - struct iov_iter *msg_iter; - int offset; -}; - static int tls_push_data(struct sock *sk, - union tls_iter_offset iter_offset, + struct iov_iter *iter, size_t size, int flags, - unsigned char record_type, - struct page *zc_page) + unsigned char record_type) { struct tls_context *tls_ctx = tls_get_ctx(sk); struct tls_prot_info *prot = &tls_ctx->prot_info; @@ -500,22 +494,13 @@ static int tls_push_data(struct sock *sk, record = ctx->open_record; copy = min_t(size_t, size, max_open_record_len - record->len); - if (copy && zc_page) { - struct page_frag zc_pfrag; - - zc_pfrag.page = zc_page; - zc_pfrag.offset = iter_offset.offset; - zc_pfrag.size = copy; - tls_append_frag(record, &zc_pfrag, copy); - - iter_offset.offset += copy; - } else if (copy && (flags & MSG_SPLICE_PAGES)) { + if (copy && (flags & MSG_SPLICE_PAGES)) { struct page_frag zc_pfrag; struct page **pages = &zc_pfrag.page; size_t off; - rc = iov_iter_extract_pages(iter_offset.msg_iter, - &pages, copy, 1, 0, &off); + rc = iov_iter_extract_pages(iter, &pages, + copy, 1, 0, &off); if (rc <= 0) { if (rc == 0) rc = -EIO; @@ -524,7 +509,7 @@ static int tls_push_data(struct sock *sk, copy = rc; if (WARN_ON_ONCE(!sendpage_ok(zc_pfrag.page))) { - iov_iter_revert(iter_offset.msg_iter, copy); + iov_iter_revert(iter, copy); rc = -EIO; goto handle_error; } @@ -537,7 +522,7 @@ static int tls_push_data(struct sock *sk, rc = tls_device_copy_data(page_address(pfrag->page) + pfrag->offset, copy, - iter_offset.msg_iter); + iter); if (rc) goto handle_error; tls_append_frag(record, pfrag, copy); @@ -592,7 +577,6 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) { unsigned char record_type = TLS_RECORD_TYPE_DATA; struct tls_context *tls_ctx = tls_get_ctx(sk); - union tls_iter_offset iter; int rc; if (!tls_ctx->zerocopy_sendfile) @@ -607,8 +591,8 @@ int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) goto out; } - iter.msg_iter = &msg->msg_iter; - rc = tls_push_data(sk, iter, size, msg->msg_flags, record_type, NULL); + rc = tls_push_data(sk, &msg->msg_iter, size, msg->msg_flags, + record_type); out: release_sock(sk); @@ -620,8 +604,7 @@ void tls_device_splice_eof(struct socket *sock) { struct sock *sk = sock->sk; struct tls_context *tls_ctx = tls_get_ctx(sk); - union tls_iter_offset iter; - struct iov_iter iov_iter = {}; + struct iov_iter iter = {}; if (!tls_is_partially_sent_record(tls_ctx)) return; @@ -630,9 +613,8 @@ void tls_device_splice_eof(struct socket *sock) lock_sock(sk); if (tls_is_partially_sent_record(tls_ctx)) { - iov_iter_bvec(&iov_iter, ITER_SOURCE, NULL, 0, 0); - iter.msg_iter = &iov_iter; - tls_push_data(sk, iter, 0, 0, TLS_RECORD_TYPE_DATA, NULL); + iov_iter_bvec(&iter, ITER_SOURCE, NULL, 0, 0); + tls_push_data(sk, &iter, 0, 0, TLS_RECORD_TYPE_DATA); } release_sock(sk); @@ -642,44 +624,18 @@ void tls_device_splice_eof(struct socket *sock) int tls_device_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tls_context *tls_ctx = tls_get_ctx(sk); - union tls_iter_offset iter_offset; - struct iov_iter msg_iter; - char *kaddr; - struct kvec iov; - int rc; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; - - mutex_lock(&tls_ctx->tx_lock); - lock_sock(sk); + msg.msg_flags |= MSG_MORE; - if (flags & MSG_OOB) { - rc = -EOPNOTSUPP; - goto out; - } - - if (tls_ctx->zerocopy_sendfile) { - iter_offset.offset = offset; - rc = tls_push_data(sk, iter_offset, size, - flags, TLS_RECORD_TYPE_DATA, page); - goto out; - } - - kaddr = kmap(page); - iov.iov_base = kaddr + offset; - iov.iov_len = size; - iov_iter_kvec(&msg_iter, ITER_SOURCE, &iov, 1, size); - iter_offset.msg_iter = &msg_iter; - rc = tls_push_data(sk, iter_offset, size, flags, TLS_RECORD_TYPE_DATA, - NULL); - kunmap(page); + if (flags & MSG_OOB) + return -EOPNOTSUPP; -out: - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); - return rc; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_device_sendmsg(sk, &msg, size); } struct tls_record_info *tls_get_record(struct tls_offload_context_tx *context, @@ -744,12 +700,10 @@ EXPORT_SYMBOL(tls_get_record); static int tls_device_push_pending_record(struct sock *sk, int flags) { - union tls_iter_offset iter; - struct iov_iter msg_iter; + struct iov_iter iter; - iov_iter_kvec(&msg_iter, ITER_SOURCE, NULL, 0, 0); - iter.msg_iter = &msg_iter; - return tls_push_data(sk, iter, 0, flags, TLS_RECORD_TYPE_DATA, NULL); + iov_iter_kvec(&iter, ITER_SOURCE, NULL, 0, 0); + return tls_push_data(sk, &iter, 0, flags, TLS_RECORD_TYPE_DATA); } void tls_device_write_space(struct sock *sk, struct tls_context *ctx)