From patchwork Thu May 18 13:06:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95875 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp492997vqo; Thu, 18 May 2023 06:29:26 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4WSpIZYnis8yRhLYA9O9qaQEr24ROfoJqqqxDv9MJtvqCvA8WVY523wW9hL6o8ICjxEkN+ X-Received: by 2002:a05:6a00:2d1a:b0:636:e52f:631e with SMTP id fa26-20020a056a002d1a00b00636e52f631emr4934972pfb.1.1684416566464; Thu, 18 May 2023 06:29:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684416566; cv=none; d=google.com; s=arc-20160816; b=UwvqTwijceKdLTcEe4a5YWFBFVSJOb8+/a+5P8MfAx4mS+QzncAHwe+IUV8iu4ZWn6 cdsb9NtF/dLDMRYFur6+nF4bhVbc3uSNDif+Fug5z9f3GAwIo101zluXr1acoaVR1ai0 FANye1hfUwK3igJbc1fDvMgPWT+GaZjKHYiGIPxh32OXZrbRWVUIkwupdYzUg+Xy+zF9 m/SM0E0qR8YzoXUcZMMrZo//L25WtYj5EGOSkUiIxLQA+Xq7iuGd1RnRrGpoZrbVjA95 drFxQLUnbWfJ8AA5qkVeM9p4JGPoOVNyFtnOjaZ4rSgIPACbT4/WxGJt5pdBh+f2Ln5M MmpQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=zo53+MxtcExCaAvP1E3d2cbKIX8PzAGUnvJuN4UlUmE=; b=HTDoLi4mO7IaNuN/RldO3TFCKKJz3q/+VFGq0x4WKWLbfem8eDscjZaRTnYfvnTmuW sOEEBbwqM2BZx6Mb8/6DO/3VD9JaH+np+DPDfD/tJPXGJmZLUwzkimLUrywoiUf6ayRr //UD1NNP6fEWDzak2VbYir7UsY2gGNTusmQx2HAlETFmpYV9BwxK/lgAy2+4x+5B7sgc dp5hRTEZZN1qBeAcxbFDMe7EBq3EI9ZWHgrI6QbTquQYiFBcuuBguJbfIAq2S5UXDyje atNJIaAPEBctljmDemktUfLLtjuZbAYKQsMfN/UcKrjU+GSOdm0Aea5Xpyg7O5c8q/au GX+w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=fgYijiPJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g26-20020aa79f1a000000b006452ad4ae32si1646284pfr.237.2023.05.18.06.29.06; Thu, 18 May 2023 06:29:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=fgYijiPJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230302AbjERNIf (ORCPT + 99 others); Thu, 18 May 2023 09:08:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41130 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230436AbjERNI1 (ORCPT ); Thu, 18 May 2023 09:08:27 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96EDF1718 for ; Thu, 18 May 2023 06:07:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415260; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zo53+MxtcExCaAvP1E3d2cbKIX8PzAGUnvJuN4UlUmE=; b=fgYijiPJzL1PAt9hhLgjTmeH5REJkwRVHn5+EZKTNTcQoI9+EX0FasaUMgsbLEic2POr2Q 6HW3F7w7A6Xi225PuBYr5BXYRWwwvqBqkVGq/wzqNkJXRdu4TNw+eshgOLxvABJLVwqiyd jRlyvrXy1TC1A+B8kBzS878G2KmVgvI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-209-DwdtkeJQPymicpdEc0OY1w-1; Thu, 18 May 2023 09:07:24 -0400 X-MC-Unique: DwdtkeJQPymicpdEc0OY1w-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 673338026E8; Thu, 18 May 2023 13:07:23 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 73135C15BA0; Thu, 18 May 2023 13:07:20 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn , io-uring@vger.kernel.org Subject: [PATCH net-next v9 01/16] net: Declare MSG_SPLICE_PAGES internal sendmsg() flag Date: Thu, 18 May 2023 14:06:58 +0100 Message-Id: <20230518130713.1515729-2-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766238785435925030?= X-GMAIL-MSGID: =?utf-8?q?1766238785435925030?= Declare MSG_SPLICE_PAGES, an internal sendmsg() flag, that hints to a network protocol that it should splice pages from the source iterator rather than copying the data if it can. This flag is added to a list that is cleared by sendmsg syscalls on entry. This is intended as a replacement for the ->sendpage() op, allowing a way to splice in several multipage folios in one go. Signed-off-by: David Howells Reviewed-by: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: io-uring@vger.kernel.org cc: netdev@vger.kernel.org --- Notes: ver #7) - In ____sys_sendmsg(), clear internal flags before setting msg_flags. - Clear internal flags in uring io_send{,_zc}(). include/linux/socket.h | 3 +++ io_uring/net.c | 2 ++ net/socket.c | 2 ++ 3 files changed, 7 insertions(+) diff --git a/include/linux/socket.h b/include/linux/socket.h index 13c3a237b9c9..bd1cc3238851 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -327,6 +327,7 @@ struct ucred { */ #define MSG_ZEROCOPY 0x4000000 /* Use user data in kernel path */ +#define MSG_SPLICE_PAGES 0x8000000 /* Splice the pages from the iterator in sendmsg() */ #define MSG_FASTOPEN 0x20000000 /* Send data in TCP SYN */ #define MSG_CMSG_CLOEXEC 0x40000000 /* Set close_on_exec for file descriptor received through @@ -337,6 +338,8 @@ struct ucred { #define MSG_CMSG_COMPAT 0 /* We never have 32 bit fixups */ #endif +/* Flags to be cleared on entry by sendmsg and sendmmsg syscalls */ +#define MSG_INTERNAL_SENDMSG_FLAGS (MSG_SPLICE_PAGES) /* Setsockoptions(2) level. Thanks to BSD these must match IPPROTO_xxx */ #define SOL_IP 0 diff --git a/io_uring/net.c b/io_uring/net.c index 89e839013837..f7cbb3c7a575 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -389,6 +389,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) if (flags & MSG_WAITALL) min_ret = iov_iter_count(&msg.msg_iter); + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; msg.msg_flags = flags; ret = sock_sendmsg(sock, &msg); if (ret < min_ret) { @@ -1136,6 +1137,7 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) msg_flags |= MSG_DONTWAIT; if (msg_flags & MSG_WAITALL) min_ret = iov_iter_count(&msg.msg_iter); + msg_flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; msg.msg_flags = msg_flags; msg.msg_ubuf = &io_notif_to_data(zc->notif)->uarg; diff --git a/net/socket.c b/net/socket.c index b7e01d0fe082..3df96e9ba4e2 100644 --- a/net/socket.c +++ b/net/socket.c @@ -2138,6 +2138,7 @@ int __sys_sendto(int fd, void __user *buff, size_t len, unsigned int flags, msg.msg_name = (struct sockaddr *)&address; msg.msg_namelen = addr_len; } + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; if (sock->file->f_flags & O_NONBLOCK) flags |= MSG_DONTWAIT; msg.msg_flags = flags; @@ -2483,6 +2484,7 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys, msg_sys->msg_control = ctl_buf; msg_sys->msg_control_is_user = false; } + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; msg_sys->msg_flags = flags; if (sock->file->f_flags & O_NONBLOCK) From patchwork Thu May 18 13:06:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95860 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp480036vqo; Thu, 18 May 2023 06:09:48 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6Qgrj2OVbaIMNtZHfliIO8vruqBu8FhqH0wqcZXtv0HTtoUL0NzH+F4CFcRfrqGLo9uJoA X-Received: by 2002:a17:903:4293:b0:1ab:19db:f2b with SMTP id ju19-20020a170903429300b001ab19db0f2bmr2636815plb.36.1684415387912; Thu, 18 May 2023 06:09:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684415387; cv=none; d=google.com; s=arc-20160816; b=q48M4de5sh4hxP67wLSKHJ/oTnWkja5ruMsI67m/BupbnqrEP3UPPCXs9Rrc46WxPX M0Go968iFYv+7IS9gVSmay9og4FddiZVBArVjP3aYCKlV32eljMqYg4Cy4gwUU66HDgN AeHMXrn1AtamUjwtzwGbytItBf+LhgeU11+gxzV+wX9Y5zOE7w011jpE6EbDtYMz08Y7 x40Fk6V7Qiip8EUoiMFG6w0ym7xwGlJgEa+6ZSvj9PkL8ExIr3mMUtWbVRS1jdOFFjrK 4eIPaVIEvxD6/cSK46AJho6NQNQoxhqFgKxVQvb/rD6cGdfRNCl8jOjrGoUskkdwlk4C 7BcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=dMBJPtbzh4IjFjJ1HeY2h7yLKLc/0VkfcRg1OdFJq1c=; b=W+nwzKoIUQXN/ctTEcL0F7U1VedU3l81VtkAllQ1iuK5wva1itn02TJnDnhfiv/5X4 MJrCszfPzzKoQgrFMNzsdJeK2z8qqqRME80oodbLWAoJnqR8F/QofVRF39DaZ+eK5nqQ 7EYSCVp1r25big9CD+aeQFR4ZzOSwwIWo7sFWhmYNv9VUN8fDe9D7CLtCfZP7pyNoPiX 3sazsxqOAa6K7TxmUeaV1KvqZ87HjOqCS/NGqk+1rAJE9DsIAjNP6cP2HDg2aRxsqHX7 /mbDsqTLR2oMumPzQZ05motLUAgRx8iGAGo8TLZc0ugNgemgZsRPtGYJnuKe90cm1jTD /kFQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=M5GStmbP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id jj2-20020a170903048200b001ab259969e7si1279098plb.31.2023.05.18.06.09.28; Thu, 18 May 2023 06:09:47 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=M5GStmbP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230453AbjERNIW (ORCPT + 99 others); Thu, 18 May 2023 09:08:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230436AbjERNIQ (ORCPT ); Thu, 18 May 2023 09:08:16 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B52D0EA for ; Thu, 18 May 2023 06:07:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415254; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dMBJPtbzh4IjFjJ1HeY2h7yLKLc/0VkfcRg1OdFJq1c=; b=M5GStmbP5ZaOKl16U6FVUYpB2lPvlM5Eutb/AL3SHfpySFrLL9474u0sBWPkM3iaYvhMOp XPDaDXkfSnM3ZUnbYb2n/WEqPGqjrFCvBebtd2NMZOaZXPXyqUwdDAfzkxVr2wtO34TteI 7NZn4YiSjEocjizoB59HUvy6QkjjEH8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-659-VyatmphiPrqaOCvVPK9LmQ-1; Thu, 18 May 2023 09:07:29 -0400 X-MC-Unique: VyatmphiPrqaOCvVPK9LmQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D8A64185A7A4; Thu, 18 May 2023 13:07:28 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3AAD02026DFD; Thu, 18 May 2023 13:07:25 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 02/16] net: Pass max frags into skb_append_pagefrags() Date: Thu, 18 May 2023 14:06:59 +0100 Message-Id: <20230518130713.1515729-3-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766237549665660151?= X-GMAIL-MSGID: =?utf-8?q?1766237549665660151?= Pass the maximum number of fragments into skb_append_pagefrags() rather than using MAX_SKB_FRAGS so that it can be used from code that wants to specify sysctl_max_skb_frags. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/linux/skbuff.h | 2 +- net/core/skbuff.c | 4 ++-- net/ipv4/ip_output.c | 3 ++- net/unix/af_unix.c | 2 +- 4 files changed, 6 insertions(+), 5 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 8cff3d817131..15011408c47c 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -1383,7 +1383,7 @@ static inline int skb_pad(struct sk_buff *skb, int pad) #define dev_kfree_skb(a) consume_skb(a) int skb_append_pagefrags(struct sk_buff *skb, struct page *page, - int offset, size_t size); + int offset, size_t size, size_t max_frags); struct skb_seq_state { __u32 lower_offset; diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 6724a84ebb09..7f53dcb26ad3 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -4188,13 +4188,13 @@ unsigned int skb_find_text(struct sk_buff *skb, unsigned int from, EXPORT_SYMBOL(skb_find_text); int skb_append_pagefrags(struct sk_buff *skb, struct page *page, - int offset, size_t size) + int offset, size_t size, size_t max_frags) { int i = skb_shinfo(skb)->nr_frags; if (skb_can_coalesce(skb, i, page, offset)) { skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], size); - } else if (i < MAX_SKB_FRAGS) { + } else if (i < max_frags) { skb_zcopy_downgrade_managed(skb); get_page(page); skb_fill_page_desc_noacc(skb, i, page, offset, size); diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 61892268e8a6..52fc840898d8 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1450,7 +1450,8 @@ ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, if (len > size) len = size; - if (skb_append_pagefrags(skb, page, offset, len)) { + if (skb_append_pagefrags(skb, page, offset, len, + MAX_SKB_FRAGS)) { err = -EMSGSIZE; goto error; } diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index cc695c9f09ec..dd55506b4632 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2349,7 +2349,7 @@ static ssize_t unix_stream_sendpage(struct socket *socket, struct page *page, newskb = NULL; } - if (skb_append_pagefrags(skb, page, offset, size)) { + if (skb_append_pagefrags(skb, page, offset, size, MAX_SKB_FRAGS)) { tail = skb; goto alloc_skb; } From patchwork Thu May 18 13:07:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95877 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp494251vqo; Thu, 18 May 2023 06:31:18 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4LIN01BJOw1JhsfYMyxfnv1HteiR3uHJEQfpMp5qZQ4u4+SERKNspfDKMtTrjhifDShE6R X-Received: by 2002:a05:6a00:2e15:b0:645:834c:f521 with SMTP id fc21-20020a056a002e1500b00645834cf521mr3691491pfb.17.1684416678006; Thu, 18 May 2023 06:31:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684416677; cv=none; d=google.com; s=arc-20160816; b=mQ10Yfcz2dKOKlnPfmwvXxJ+woT8jbZb5znbRaZKwslUUlyKiXVIEMicRZ401NBhKw YoDCzM22ApQfSqLy3HnnxMy8UyvYoAyYoyqNbHhmIoSqIFVXHK/9vzNsP3pBx/tMb+y8 XM6tuGXgGW8F2fsuSAam2H28uFN1GJv/APq1qZ1t4tPXDEvCx6T8r9Rgph54dCF+tqxX 7Y18AfeH3bS4nyuEvFmFQub+RmOvJhkB/pFeWRaMNg7P60Wtb87CvEpRRQ6e1jmaRoYC Sln5aSIL4mkKdtuuV4OAA6yFrX/q6ZpgCRfMEiADXTt4xQbdV0GAYWg++b1GJhomwgqa ylJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Fl+2zC2cOWTcM7FjMckfIDo5NeTUJ6H1d1XKuMzp1Mk=; b=FvBu7ZXRHS1vMyOticy4nwK0riF7X0LnVzM64bv9QwSq+o1Ktw7VMs7w4K/tIGkXQZ 4RFO0bwVvV0uQmYzLvS45Zh9RI+jMGwy8jpi9i6tDAOF7mHNnzEJ/+3EXPsv9msec33h lllN3iDR672RUaJ5SRfhqTJKccvDgo2zTRf0NbQnwTHr6+Xhunj5ufk4qdbcQ92BMis6 MiyQjQbUdwfOJjBEODMVJYd5YESozhtwOQwslyV2A3ZC4AaIORDbJKO2Szgw7rntDl0A 3KdPX7jDncr8z2ty1YETyCskJVSgW3n17xvobKI5YvARuhKwHJk0VUuF0waQBVMFpHiC V0JA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=K4ux9D60; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w19-20020a627b13000000b0062550a9236csi1663779pfc.248.2023.05.18.06.31.00; Thu, 18 May 2023 06:31:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=K4ux9D60; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230491AbjERNI3 (ORCPT + 99 others); Thu, 18 May 2023 09:08:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41102 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230290AbjERNI0 (ORCPT ); Thu, 18 May 2023 09:08:26 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1AB33F3 for ; Thu, 18 May 2023 06:07:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415258; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fl+2zC2cOWTcM7FjMckfIDo5NeTUJ6H1d1XKuMzp1Mk=; b=K4ux9D60JqWru9tjJ6abW99MWch/v98KUSKDQKYYvTnQUd/iOzt0Fd4nR5zPY+zuA/LKP/ zWOeG2KYkmci6Dk6VzrUoow7XHh7UJiRaBqVFFIqjEbzAsaS4YN1wHe6aVV2/VcQZHhZcW oamB9To8jFImu4+B1UfYgywe72+SNCI= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-595-Nim3qSxMPQ2L4dUXo6bQVg-1; Thu, 18 May 2023 09:07:34 -0400 X-MC-Unique: Nim3qSxMPQ2L4dUXo6bQVg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D7F9E28237C8; Thu, 18 May 2023 13:07:33 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 650961121314; Thu, 18 May 2023 13:07:30 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 03/16] net: Add a function to splice pages into an skbuff for MSG_SPLICE_PAGES Date: Thu, 18 May 2023 14:07:00 +0100 Message-Id: <20230518130713.1515729-4-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766238902773211244?= X-GMAIL-MSGID: =?utf-8?q?1766238902773211244?= Add a function to handle MSG_SPLICE_PAGES being passed internally to sendmsg(). Pages are spliced into the given socket buffer if possible and copied in if not (e.g. they're slab pages or have a zero refcount). Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Al Viro cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #8) - Order local variables in reverse xmas tree order. - Remove duplicate coalescence check. - Warn if sendpage_ok() fails. ver #7) - Export function. - Never copy data, return -EIO if sendpage_ok() returns false. include/linux/skbuff.h | 3 ++ net/core/skbuff.c | 88 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 15011408c47c..1b2ebf6113e0 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -5097,5 +5097,8 @@ static inline void skb_mark_for_recycle(struct sk_buff *skb) #endif } +ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, + ssize_t maxsize, gfp_t gfp); + #endif /* __KERNEL__ */ #endif /* _LINUX_SKBUFF_H */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 7f53dcb26ad3..f4a5b51aed22 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -6892,3 +6892,91 @@ nodefer: __kfree_skb(skb); if (unlikely(kick) && !cmpxchg(&sd->defer_ipi_scheduled, 0, 1)) smp_call_function_single_async(cpu, &sd->defer_csd); } + +static void skb_splice_csum_page(struct sk_buff *skb, struct page *page, + size_t offset, size_t len) +{ + const char *kaddr; + __wsum csum; + + kaddr = kmap_local_page(page); + csum = csum_partial(kaddr + offset, len, 0); + kunmap_local(kaddr); + skb->csum = csum_block_add(skb->csum, csum, skb->len); +} + +/** + * skb_splice_from_iter - Splice (or copy) pages to skbuff + * @skb: The buffer to add pages to + * @iter: Iterator representing the pages to be added + * @maxsize: Maximum amount of pages to be added + * @gfp: Allocation flags + * + * This is a common helper function for supporting MSG_SPLICE_PAGES. It + * extracts pages from an iterator and adds them to the socket buffer if + * possible, copying them to fragments if not possible (such as if they're slab + * pages). + * + * Returns the amount of data spliced/copied or -EMSGSIZE if there's + * insufficient space in the buffer to transfer anything. + */ +ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, + ssize_t maxsize, gfp_t gfp) +{ + size_t frag_limit = READ_ONCE(sysctl_max_skb_frags); + struct page *pages[8], **ppages = pages; + ssize_t spliced = 0, ret = 0; + unsigned int i; + + while (iter->count > 0) { + ssize_t space, nr; + size_t off, len; + + ret = -EMSGSIZE; + space = frag_limit - skb_shinfo(skb)->nr_frags; + if (space < 0) + break; + + /* We might be able to coalesce without increasing nr_frags */ + nr = clamp_t(size_t, space, 1, ARRAY_SIZE(pages)); + + len = iov_iter_extract_pages(iter, &ppages, maxsize, nr, 0, &off); + if (len <= 0) { + ret = len ?: -EIO; + break; + } + + i = 0; + do { + struct page *page = pages[i++]; + size_t part = min_t(size_t, PAGE_SIZE - off, len); + + ret = -EIO; + if (WARN_ON_ONCE(!sendpage_ok(page))) + goto out; + + ret = skb_append_pagefrags(skb, page, off, part, + frag_limit); + if (ret < 0) { + iov_iter_revert(iter, len); + goto out; + } + + if (skb->ip_summed == CHECKSUM_NONE) + skb_splice_csum_page(skb, page, off, part); + + off = 0; + spliced += part; + maxsize -= part; + len -= part; + } while (len > 0); + + if (maxsize <= 0) + break; + } + +out: + skb_len_add(skb, spliced); + return spliced ?: ret; +} +EXPORT_SYMBOL(skb_splice_from_iter); From patchwork Thu May 18 13:07:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95886 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp496086vqo; Thu, 18 May 2023 06:33:19 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6O157GVOsB5E+UomAuZAufa6ZIPSihy7CyBMj5/1y+csoGAz05FSJcQZmJ4aZfgb/KtNkF X-Received: by 2002:a17:902:ab51:b0:1ad:fa2e:17f8 with SMTP id ij17-20020a170902ab5100b001adfa2e17f8mr2295970plb.12.1684416798734; Thu, 18 May 2023 06:33:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684416798; cv=none; d=google.com; s=arc-20160816; b=jBmhTrEpate5QwESGW/jPsP/63kUYZUDYgEfp/Bnf76MEBBQxZy7EBHFvx65FDMVlT 3OnzFHuDGLF5nWGH4COZGZ2Oe5rZvJ6ykaRTq0/qgoK+h32eg5opfVmPFoYFDIMKSDWY 1BtIkhqfTppn4VcfRE7SDIRbcvgcAl5YJJM9mzWuxFOeWGXfviH4VCnm/q3XXSY8WQBq mgw0yYVKl1y7o17G0Ver8i6qGajY4RgcKguSfefwLOKFKAXSElidzp11yiWqmMrp+8LG SkhIUbONQKVfsm4U+RBAHFgr9EEradEeVn57yozacTv4hykjsttMjLFZ7p8EYId4CFyW Mn5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CbEUQCnChP/PzTidzzQWL6iO0yyE2e0IQp2l4Z4U9vY=; b=XWoe+XX4x+uwmTDs78JAn5Y+0rwZUHyDihqRzL+xVuuGAQBFlsdeRx6RMuF9MMX7p3 fUjU6hPROV+KvsWAluv09+6gxcyW4TtkiDY85JzaSJU2noBp91+ABAViWJYltQoUmRCA 1cE2oMaTxfbxeSi7PckjaglZ9dMWrcbf2IQO0TRW3ep9mQMLO1hgE3RNX0rFKvM2t4wR D9dFSat5JBGcpYEfQWtHwbwmAifB1Nf4Hx6C1tU8dY8NsWeYLX8cga9eE8jvtpCPWrxa wggxS4Pf14TB7012haJ969McW6qnZ/e5WzB3yy/gLwuRdsthV/kr6BrGVy59zLTvohRE fIDA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=CJjD5nHa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b5-20020a170902e94500b001ae5fb1436asi1203863pll.184.2023.05.18.06.32.57; Thu, 18 May 2023 06:33:18 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=CJjD5nHa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230510AbjERNI4 (ORCPT + 99 others); Thu, 18 May 2023 09:08:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41242 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231178AbjERNIt (ORCPT ); Thu, 18 May 2023 09:08:49 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 902FBDA for ; Thu, 18 May 2023 06:07:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415261; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CbEUQCnChP/PzTidzzQWL6iO0yyE2e0IQp2l4Z4U9vY=; b=CJjD5nHa/H/BRCvEQdE6BvzzpAeiMLAhRSc7iDHPF6JH/GjamVg5UdPg1p4xDeb/G5Cve6 T4PnHLmf7Ktpcg9ezewsp9omoQWg06y1pjTYq+sOu0H1rJFV3T6fzkYoO8ydtNSkNs2SyP DG0a5W3F+3Md/f8u3xpCWyBJPjriA8A= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-355-v9bQtabbOrGx9iDPjoFLmg-1; Thu, 18 May 2023 09:07:39 -0400 X-MC-Unique: v9bQtabbOrGx9iDPjoFLmg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 322A184AF35; Thu, 18 May 2023 13:07:38 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id B53DA40C6EC4; Thu, 18 May 2023 13:07:34 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 04/16] tcp: Support MSG_SPLICE_PAGES Date: Thu, 18 May 2023 14:07:01 +0100 Message-Id: <20230518130713.1515729-5-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766239029012229167?= X-GMAIL-MSGID: =?utf-8?q?1766239029012229167?= Make TCP's sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced or copied (if it cannot be spliced) from the source iterator. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #9) - Fix a merge conflict with commit eea96a3e2c909. ver #7) - Missed a "zc = 1" in tcp_sendmsg_locked(). ver #6) - Set zc to 0/MSG_ZEROCOPY/MSG_SPLICE_PAGES rather than 0/1/2. - Use common helper. net/ipv4/tcp.c | 43 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 3d18e295bb2f..2d61150d01f1 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1223,7 +1223,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) int flags, err, copied = 0; int mss_now = 0, size_goal, copied_syn = 0; int process_backlog = 0; - bool zc = false; + int zc = 0; long timeo; flags = msg->msg_flags; @@ -1231,7 +1231,8 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if ((flags & MSG_ZEROCOPY) && size) { if (msg->msg_ubuf) { uarg = msg->msg_ubuf; - zc = sk->sk_route_caps & NETIF_F_SG; + if (sk->sk_route_caps & NETIF_F_SG) + zc = MSG_ZEROCOPY; } else if (sock_flag(sk, SOCK_ZEROCOPY)) { skb = tcp_write_queue_tail(sk); uarg = msg_zerocopy_realloc(sk, size, skb_zcopy(skb)); @@ -1239,10 +1240,14 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) err = -ENOBUFS; goto out_err; } - zc = sk->sk_route_caps & NETIF_F_SG; - if (!zc) + if (sk->sk_route_caps & NETIF_F_SG) + zc = MSG_ZEROCOPY; + else uarg_to_msgzc(uarg)->zerocopy = 0; } + } else if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES) && size) { + if (sk->sk_route_caps & NETIF_F_SG) + zc = MSG_SPLICE_PAGES; } if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect) && @@ -1305,7 +1310,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) goto do_error; while (msg_data_left(msg)) { - int copy = 0; + ssize_t copy = 0; skb = tcp_write_queue_tail(sk); if (skb) @@ -1346,7 +1351,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (copy > msg_data_left(msg)) copy = msg_data_left(msg); - if (!zc) { + if (zc == 0) { bool merge = true; int i = skb_shinfo(skb)->nr_frags; struct page_frag *pfrag = sk_page_frag(sk); @@ -1391,7 +1396,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) page_ref_inc(pfrag->page); } pfrag->offset += copy; - } else { + } else if (zc == MSG_ZEROCOPY) { /* First append to a fragless skb builds initial * pure zerocopy skb */ @@ -1412,6 +1417,30 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (err < 0) goto do_error; copy = err; + } else if (zc == MSG_SPLICE_PAGES) { + /* Splice in data if we can; copy if we can't. */ + if (tcp_downgrade_zcopy_pure(sk, skb)) + goto wait_for_space; + copy = tcp_wmem_schedule(sk, copy); + if (!copy) + goto wait_for_space; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) { + if (err == -EMSGSIZE) { + tcp_mark_push(tp, skb); + goto new_segment; + } + goto do_error; + } + copy = err; + + if (!(flags & MSG_NO_SHARED_FRAGS)) + skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; + + sk_wmem_queued_add(sk, copy); + sk_mem_charge(sk, copy); } if (!copied) From patchwork Thu May 18 13:07:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95879 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp494882vqo; Thu, 18 May 2023 06:32:02 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5SSTh7NfpjPFqeNNc3zcmMdMN0qWcbLlu7lVEkz7ezd8+6GsjYVKEHPTrgKbFyF/dBtptf X-Received: by 2002:a05:6a20:9184:b0:107:1805:fef5 with SMTP id v4-20020a056a20918400b001071805fef5mr2666071pzd.28.1684416721910; Thu, 18 May 2023 06:32:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684416721; cv=none; d=google.com; s=arc-20160816; b=GwfIBa//Z/1ozZTbeqi8OH6m7a+1foqky93CIQAZtX86qrpKfmjLX3czGitJuOe1/r TF6ObIjUGh46CLCJAB8Ye8bRrbVhvdXjYxKDb9DAW3J7Fkk5LHGZybq/+D2tkZ8FLBkv xfofAF8KuNTlZBgJpzbpUOCfO7WqML9YfdIod48XQybCFMmhT7fbljHXbppI5btNRLKR c9Ymn5DcmjjIcYjQE7SHXfOVKS9QyJyNcqdWdLUueVd5L2jn1lhQtsroQ3TRRY6cIHis LGjx/p/Xl55s1/CzQ5garXSNQ6UiI6YHhUFdoKN/4ZVLwy8hWDp0MOXUQ53HblflXMlj DrJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=79TjdqGEZWzMR9284kh0SsbKnhgq/mMmLDGgzzPhuR4=; b=CkdLjc5une0QtA8u3yPYYN/qjHzL9tEa+6LatTvC+s11ZFwvGHsj6d+U15kwOQHrGQ fR9AkjO4EvN4TTWNoj9wgXeTQGqTZ+mXMs/J/puokUIwxyw2ivE7vnNcCajjzzEB+W9j C2T2sWhyOmKT3XzAlOjR3x7dTllhx+GOEFjcfj9mg8+6PMFrChP05D/ilWc7yvNV1GpT r9e/eLYBLQu5msCbr1HglFqxj7xJ4avYy+w1IV8hMnw6s28DMqsxw6ABLyiS/KIYPj3C St0w8azxPAOTRODOWkmcPVOJ64qMtJduXEXY+b5/BxSPscwJyB+5nQh0W0wvbEgDySVr kfug== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=AUJSO9xu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g22-20020a633756000000b00524eef20da6si1438420pgn.642.2023.05.18.06.31.43; Thu, 18 May 2023 06:32:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=AUJSO9xu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231314AbjERNJL (ORCPT + 99 others); Thu, 18 May 2023 09:09:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41272 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231156AbjERNI6 (ORCPT ); Thu, 18 May 2023 09:08:58 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C1A7C1720 for ; Thu, 18 May 2023 06:07:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415275; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=79TjdqGEZWzMR9284kh0SsbKnhgq/mMmLDGgzzPhuR4=; b=AUJSO9xu7OrroOvpYtiQ6O95qgc38erYpIwqJMm5RXw5yYCNKaOWEsoJqB/RIdGCJ1wyTH 1cc3hmEzj7gdUY93NnezMSvymDr+79KOAC4amVrOMvGbP0WbL7exFVEI6j2IVP8XrAAWAx gT4u9eDy0qsF0ys+maTn8zVstwslEEk= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-247-KanpYTY4OHmNeX6aG_XJaQ-1; Thu, 18 May 2023 09:07:44 -0400 X-MC-Unique: KanpYTY4OHmNeX6aG_XJaQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 65A833815F77; Thu, 18 May 2023 13:07:43 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 51BE440C6EC4; Thu, 18 May 2023 13:07:40 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 05/16] tcp: Convert do_tcp_sendpages() to use MSG_SPLICE_PAGES Date: Thu, 18 May 2023 14:07:02 +0100 Message-Id: <20230518130713.1515729-6-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766238948364101356?= X-GMAIL-MSGID: =?utf-8?q?1766238948364101356?= Convert do_tcp_sendpages() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. do_tcp_sendpages() can then be inlined in subsequent patches into its callers. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 158 +++---------------------------------------------- 1 file changed, 7 insertions(+), 151 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 2d61150d01f1..f3a0c02678e0 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -974,163 +974,19 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -static struct sk_buff *tcp_build_frag(struct sock *sk, int size_goal, int flags, - struct page *page, int offset, size_t *size) -{ - struct sk_buff *skb = tcp_write_queue_tail(sk); - struct tcp_sock *tp = tcp_sk(sk); - bool can_coalesce; - int copy, i; - - if (!skb || (copy = size_goal - skb->len) <= 0 || - !tcp_skb_can_collapse_to(skb)) { -new_segment: - if (!sk_stream_memory_free(sk)) - return NULL; - - skb = tcp_stream_alloc_skb(sk, 0, sk->sk_allocation, - tcp_rtx_and_write_queues_empty(sk)); - if (!skb) - return NULL; - -#ifdef CONFIG_TLS_DEVICE - skb->decrypted = !!(flags & MSG_SENDPAGE_DECRYPTED); -#endif - tcp_skb_entail(sk, skb); - copy = size_goal; - } - - if (copy > *size) - copy = *size; - - i = skb_shinfo(skb)->nr_frags; - can_coalesce = skb_can_coalesce(skb, i, page, offset); - if (!can_coalesce && i >= READ_ONCE(sysctl_max_skb_frags)) { - tcp_mark_push(tp, skb); - goto new_segment; - } - if (tcp_downgrade_zcopy_pure(sk, skb)) - return NULL; - - copy = tcp_wmem_schedule(sk, copy); - if (!copy) - return NULL; - - if (can_coalesce) { - skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); - } else { - get_page(page); - skb_fill_page_desc_noacc(skb, i, page, offset, copy); - } - - if (!(flags & MSG_NO_SHARED_FRAGS)) - skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; - - skb->len += copy; - skb->data_len += copy; - skb->truesize += copy; - sk_wmem_queued_add(sk, copy); - sk_mem_charge(sk, copy); - WRITE_ONCE(tp->write_seq, tp->write_seq + copy); - TCP_SKB_CB(skb)->end_seq += copy; - tcp_skb_pcount_set(skb, 0); - - *size = copy; - return skb; -} - ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tcp_sock *tp = tcp_sk(sk); - int mss_now, size_goal; - int err; - ssize_t copied; - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); - - if (IS_ENABLED(CONFIG_DEBUG_VM) && - WARN_ONCE(!sendpage_ok(page), - "page must not be a Slab one and have page_count > 0")) - return -EINVAL; - - /* Wait for a connection to finish. One exception is TCP Fast Open - * (passive side) where data is allowed to be sent before a connection - * is fully established. - */ - if (((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) && - !tcp_passive_fastopen(sk)) { - err = sk_stream_wait_connect(sk, &timeo); - if (err != 0) - goto out_err; - } + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - mss_now = tcp_send_mss(sk, &size_goal, flags); - copied = 0; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - err = -EPIPE; - if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) - goto out_err; - - while (size > 0) { - struct sk_buff *skb; - size_t copy = size; - - skb = tcp_build_frag(sk, size_goal, flags, page, offset, ©); - if (!skb) - goto wait_for_space; - - if (!copied) - TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; - - copied += copy; - offset += copy; - size -= copy; - if (!size) - goto out; - - if (skb->len < size_goal || (flags & MSG_OOB)) - continue; - - if (forced_push(tp)) { - tcp_mark_push(tp, skb); - __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); - } else if (skb == tcp_send_head(sk)) - tcp_push_one(sk, mss_now); - continue; - -wait_for_space: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); - tcp_push(sk, flags & ~MSG_MORE, mss_now, - TCP_NAGLE_PUSH, size_goal); - - err = sk_stream_wait_memory(sk, &timeo); - if (err != 0) - goto do_error; - - mss_now = tcp_send_mss(sk, &size_goal, flags); - } - -out: - if (copied) { - tcp_tx_timestamp(sk, sk->sk_tsflags); - if (!(flags & MSG_SENDPAGE_NOTLAST)) - tcp_push(sk, flags, mss_now, tp->nonagle, size_goal); - } - return copied; - -do_error: - tcp_remove_empty_skb(sk); - if (copied) - goto out; -out_err: - /* make sure we wake any epoll edge trigger waiter */ - if (unlikely(tcp_rtx_and_write_queues_empty(sk) && err == -EAGAIN)) { - sk->sk_write_space(sk); - tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED); - } - return sk_stream_error(sk, flags, err); + return tcp_sendmsg_locked(sk, &msg, size); } EXPORT_SYMBOL_GPL(do_tcp_sendpages); From patchwork Thu May 18 13:07:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95890 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp496664vqo; Thu, 18 May 2023 06:34:06 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4kWU70v+k/FXR4pHTpNS8B9WS2YOOzqw4K4J8jevSUg/Ngv3YJ4KQAnR7b1KZ7YRNLpSMb X-Received: by 2002:aa7:88c6:0:b0:643:9cc0:a3be with SMTP id k6-20020aa788c6000000b006439cc0a3bemr4257717pff.5.1684416846243; Thu, 18 May 2023 06:34:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684416846; cv=none; d=google.com; s=arc-20160816; b=0Dk9nns2N+OWjfWATZJ6tRNS/vXGHWkdo/Cz71AYsjrA0vimSAGIpESOKwUHhNSXL4 f5PTGILhF/gjtWqO8EPMChiJjtn1qTEybDyQ941k3HsH/u+q3gbKBv0KvGz73Fm8ZI85 B0soWxfl5kXS8uEzO/nx0Pc2S3PUQhQxWQeL3jxFlYgoaHZk2wySAhhtgmhWs6Z/uAL2 JyTnvMh9nCBEfiVo0A+W7RjUED/C5b5XUdTYPfId9XDv2MZU89c7OHwILVy43iC08mD3 OY3YSHOyz24nps1raRu/qqvy7Uu7qTpo70FXFMExAhHGanxlaM0v2ilEAlJDJ11kUBgb 8Smw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2EXLlBUI8HMrlq1LdzuePmnE9vo0jChMg019VHUJEgs=; b=s0HLYuUB2jLzLV5RsMAvU4xOq5QsowkLC8sg0290GoDxuJ2hPW1YuelOPfdSSMt31z Y5EeGqxxcJ/4WxoavJrWGd0rXFTjVJDCL+jbNpE7/u/c4i9p2dsSGP/AWz9i4utTLQ2R N6ZLk0LwlhWuJKSeMIOGpdlfD8o9BJJ23dQF23ki/mXWKUQeFfYmFPVpJxcjZfcDlt7t gYaas4kjng+0vFCi3ebNvPSjN5Co36tZ+k8yJ/dS3UZijqZn7IWkqF19+dmwWkKU5mpM X7PVIDH/q0HwHWMY3hj7UE/AOvgWsOGzioZE2sVmsDRz9v5VtcyvKNxQvl18pNK/tVSM qUWw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=EkC7VsvO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j71-20020a63804a000000b00530b6228f91si383999pgd.895.2023.05.18.06.33.41; Thu, 18 May 2023 06:34:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=EkC7VsvO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231178AbjERNJI (ORCPT + 99 others); Thu, 18 May 2023 09:09:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41278 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230268AbjERNI5 (ORCPT ); Thu, 18 May 2023 09:08:57 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 231D41733 for ; Thu, 18 May 2023 06:07:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415278; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2EXLlBUI8HMrlq1LdzuePmnE9vo0jChMg019VHUJEgs=; b=EkC7VsvOGxY/5VzhnL5xHTuZWV1Yjk5RIv5FqtbIFQtBncdSf41MgsMBNaIGyCZf842Rbf Gd0chgxEBOQhXDC3e9WjL1gKVYQgs6GUqUCzXAZf0vvgGKHUfrb2QLRjZnrNGSKEVEXOtA KqqMkRBZL1spgHaE3QwUTe4ZQe01Ja8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-209-s4xticGiMR-62gp7XbuBGA-1; Thu, 18 May 2023 09:07:52 -0400 X-MC-Unique: s4xticGiMR-62gp7XbuBGA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 38B6985C07A; Thu, 18 May 2023 13:07:48 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 155802026D25; Thu, 18 May 2023 13:07:43 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Fastabend , Jakub Sitnicki , bpf@vger.kernel.org Subject: [PATCH net-next v9 06/16] tcp_bpf: Inline do_tcp_sendpages as it's now a wrapper around tcp_sendmsg Date: Thu, 18 May 2023 14:07:03 +0100 Message-Id: <20230518130713.1515729-7-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766239078960086571?= X-GMAIL-MSGID: =?utf-8?q?1766239078960086571?= do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: John Fastabend cc: Jakub Sitnicki cc: "David S. Miller" cc: Eric Dumazet cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: bpf@vger.kernel.org --- net/ipv4/tcp_bpf.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c index 2e9547467edb..0291d15acd19 100644 --- a/net/ipv4/tcp_bpf.c +++ b/net/ipv4/tcp_bpf.c @@ -72,11 +72,13 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, { bool apply = apply_bytes; struct scatterlist *sge; + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct page *page; int size, ret = 0; u32 off; while (1) { + struct bio_vec bvec; bool has_tx_ulp; sge = sk_msg_elem(msg, msg->sg.start); @@ -88,16 +90,18 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, tcp_rate_check_app_limited(sk); retry: has_tx_ulp = tls_sw_has_ctx_tx(sk); - if (has_tx_ulp) { - flags |= MSG_SENDPAGE_NOPOLICY; - ret = kernel_sendpage_locked(sk, - page, off, size, flags); - } else { - ret = do_tcp_sendpages(sk, page, off, size, flags); - } + if (has_tx_ulp) + msghdr.msg_flags |= MSG_SENDPAGE_NOPOLICY; + if (flags & MSG_SENDPAGE_NOTLAST) + msghdr.msg_flags |= MSG_MORE; + + bvec_set_page(&bvec, page, size, off); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret <= 0) return ret; + if (apply) apply_bytes -= ret; msg->sg.size -= ret; @@ -404,7 +408,7 @@ static int tcp_bpf_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) long timeo; int flags; - /* Don't let internal do_tcp_sendpages() flags through */ + /* Don't let internal sendpage flags through */ flags = (msg->msg_flags & ~MSG_SENDPAGE_DECRYPTED); flags |= MSG_NO_SHARED_FRAGS; From patchwork Thu May 18 13:07:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95870 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp490674vqo; Thu, 18 May 2023 06:25:21 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ49TjE5SiKyG/euR/dFjs8P0wyR24aJQ6+BKLgVf2YtLZtcZi58MS3juo8L14LNu3XxseBC X-Received: by 2002:a05:6a00:b83:b0:640:f313:efba with SMTP id g3-20020a056a000b8300b00640f313efbamr5077020pfj.19.1684416320948; Thu, 18 May 2023 06:25:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684416320; cv=none; d=google.com; s=arc-20160816; b=UFNa22pI0QfgqWI9Ff3Ospe4Rvqw8AfSjcUR4pteo12IZFQFkop9eDKdh16e+4eliN MbQ/LTrgbIUtO3S932LabKhHRyK3W9DrLiZaDq/PxZgT7JP0cI+Bsn7OoBFX4zKd2BvT Vx5n7ya4nRl1MFwSPEj1+HVSn6515E+fx9QlqSVOQXZsV9Y/sG8HWrYY0OTWhsCSJxym Nm1gjFsa6QapI/svfSHZR/3ILlsh0ECc1ueHCj5P70XY0znI9moOiKqgG5dBuwcWKuoS HTKq0PaiwOTRbAC8I60+9WSKZos6R4pE1i6eRV1V7XmstiX4mr34JQ7wOzSd47+RD37t 4VOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=tYeoTX9B8dad49qWPHePVLYzX6oyR1jBlVymL6Q5Zsc=; b=QHPlrHM/OJ94XeIJ/yX1kIHl8d17f7h0nwxsKLvw4YXc1Pu9ywUTy/0DrwqLDKOPsL M15x1yxmNc1LA71Axnv4RNx8Ui0W5tWIsCiOUgWMmtidIzOLUSZ0SWPkn676Tv2Lw1YP OuGp+sdK9VkKzmrDTWvPR7wxNmoOXdkTzOniufk5BA05/w/AP/0gkh8/ulYjzsYLch+H tkZwVmSkGsMWaVh+YqLAPmhqQ4UwnuxJ1dfPje0txxG+RDp9TwzEq1nn/CHuWttq5/Bz pMXV5PkuRMsO4jRHZs0iDNF2KiFPJBm/Fkm9/NK8+6Ci9hprR75UluH1uAacDY6cQsvi p06Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KMNieTEP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g26-20020aa79f1a000000b006452ad4ae32si1646284pfr.237.2023.05.18.06.25.04; Thu, 18 May 2023 06:25:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KMNieTEP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231394AbjERNJ3 (ORCPT + 99 others); Thu, 18 May 2023 09:09:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231265AbjERNJF (ORCPT ); Thu, 18 May 2023 09:09:05 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70A25172D for ; Thu, 18 May 2023 06:07:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415278; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tYeoTX9B8dad49qWPHePVLYzX6oyR1jBlVymL6Q5Zsc=; b=KMNieTEPSx2wiKDuDJljpaXJ601hrQnO096t8JplIZmBf7dE2GxpQ9KsLi/JYRAwsmIoYi OwwY2hAYcnEehDlRy9BZ4MyHtkfqQaK8n5etSwD4+QlKpgJcFCyUaa25woVWfFaHSA/EiT mx/zvn4x21qVtW0Mi7hpn+9i9encr2A= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-539-6BlD0X51NFmY4rfARQLhyQ-1; Thu, 18 May 2023 09:07:53 -0400 X-MC-Unique: 6BlD0X51NFmY4rfARQLhyQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6BC7328237C1; Thu, 18 May 2023 13:07:52 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 14D062026D16; Thu, 18 May 2023 13:07:48 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Steffen Klassert , Herbert Xu Subject: [PATCH net-next v9 07/16] espintcp: Inline do_tcp_sendpages() Date: Thu, 18 May 2023 14:07:04 +0100 Message-Id: <20230518130713.1515729-8-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766238527719425695?= X-GMAIL-MSGID: =?utf-8?q?1766238527719425695?= do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Steffen Klassert cc: Herbert Xu cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/xfrm/espintcp.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/net/xfrm/espintcp.c b/net/xfrm/espintcp.c index 872b80188e83..3504925babdb 100644 --- a/net/xfrm/espintcp.c +++ b/net/xfrm/espintcp.c @@ -205,14 +205,16 @@ static int espintcp_sendskb_locked(struct sock *sk, struct espintcp_msg *emsg, static int espintcp_sendskmsg_locked(struct sock *sk, struct espintcp_msg *emsg, int flags) { + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct sk_msg *skmsg = &emsg->skmsg; struct scatterlist *sg; int done = 0; int ret; - flags |= MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags |= MSG_SENDPAGE_NOTLAST; sg = &skmsg->sg.data[skmsg->sg.start]; do { + struct bio_vec bvec; size_t size = sg->length - emsg->offset; int offset = sg->offset + emsg->offset; struct page *p; @@ -220,11 +222,13 @@ static int espintcp_sendskmsg_locked(struct sock *sk, emsg->offset = 0; if (sg_is_last(sg)) - flags &= ~MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags &= ~MSG_SENDPAGE_NOTLAST; p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret < 0) { emsg->offset = offset - sg->offset; skmsg->sg.start += done; From patchwork Thu May 18 13:07:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95889 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp496660vqo; Thu, 18 May 2023 06:34:06 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5b5U8fSmnY3CKfIGzeXNR9NVUfxCWLrqeKThxpgjzDHiCgB0TjyfEwbnEvm3NblLY4+seO X-Received: by 2002:a05:6a21:3998:b0:101:5743:fd01 with SMTP id ad24-20020a056a21399800b001015743fd01mr2535514pzc.25.1684416845541; Thu, 18 May 2023 06:34:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684416845; cv=none; d=google.com; s=arc-20160816; b=sc17M8ZnziOmY4dyZaE1DhDMkOnKOp/NLUMT+qCiNMENmpD/QyXKx0P475O1nyeD7W jWiSzIRcfrDXDwIvrvzYglCmS053sA3KmQKdBGB8T7IvEwZuSvXoS4Cjkb9Y+j8Q5+Xa BJ/jESt1mzCHPOpsB5dseZXoioCnqjmgWTjlWdUUoq+wv7LxX/wjygnoxau+bbwYiPHP gvShKXRfP3jMm6u2iLJbPOWJc0a12zYByXdZvpajLwz3uoykZfptIJ0mpJumMFnqFDlf irLvJtTj6iD77VGXb0Wz+gbaLC6Fdkr7V/hEbJuO2nuqds9gTvH1XD3EPqmngvOkBErC JpfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JUrQ0rsOJEKe2VauAy4RSjWfQDMPveXienxXCaL8A+w=; b=d/cGU7pNpjAY5RcRKyh3+/RTaWMfoWNUBQ8Urlt01CSj2JamNtvCS/zrhfy+RaNHbB Z9M/GUlA+XF2pbaMxQEA6rYQLL+l62iU0WHfJFUMEYsMX4zA2qRAZ2azUym9epwtacgv gD5GoQtqoSLksvYQPpyogwZXpW9BdON9lUJ2kR6Tew5W+vpP0GinkSqMJNW2WvCTg0QO +gMOZLJ1GR5xwJnMSbb47TxD78SSSc1dU0AFbkRcLQFnjRlmH5vFOPFZ4/d63ags71J2 AOOwDlc+VJPhGpcEwLNOhQQ0CrWNZ7H1mUyzU19d4e+S66rwF56AyfLUTVJn2dPNyetQ swbw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ha4hbdrT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k11-20020a633d0b000000b00524f8b0f257si1482027pga.102.2023.05.18.06.33.37; Thu, 18 May 2023 06:34:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ha4hbdrT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231207AbjERNJ1 (ORCPT + 99 others); Thu, 18 May 2023 09:09:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41340 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231260AbjERNJE (ORCPT ); Thu, 18 May 2023 09:09:04 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB8191727 for ; Thu, 18 May 2023 06:08:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415282; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JUrQ0rsOJEKe2VauAy4RSjWfQDMPveXienxXCaL8A+w=; b=ha4hbdrTDchZHFApnNGs5mebuFQA72rAqrYiQJCu6qBzXKMedE3wBoiYDSyrsLv8Haa7JZ GjtLVvCEn60Yppok9O5XgiWLA1FeIh9vgouESUHgSfS21K1deO6rjLdq+a/yoQ5SRfaIan epQkS+WAGxV/Cwc4lubE/uFnayKmW1Y= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-551-cLgcPFrJO0us3MfDI4Owqw-1; Thu, 18 May 2023 09:07:57 -0400 X-MC-Unique: cLgcPFrJO0us3MfDI4Owqw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 540B585C075; Thu, 18 May 2023 13:07:56 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 36E3E40C2072; Thu, 18 May 2023 13:07:53 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Boris Pismenny , John Fastabend Subject: [PATCH net-next v9 08/16] tls: Inline do_tcp_sendpages() Date: Thu, 18 May 2023 14:07:05 +0100 Message-Id: <20230518130713.1515729-9-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766239078157728758?= X-GMAIL-MSGID: =?utf-8?q?1766239078157728758?= do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: "David S. Miller" cc: Eric Dumazet cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tls.h | 2 +- net/tls/tls_main.c | 24 +++++++++++++++--------- 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/include/net/tls.h b/include/net/tls.h index 6056ce5a2aa5..5791ca7a189c 100644 --- a/include/net/tls.h +++ b/include/net/tls.h @@ -258,7 +258,7 @@ struct tls_context { struct scatterlist *partially_sent_record; u16 partially_sent_offset; - bool in_tcp_sendpages; + bool splicing_pages; bool pending_open_record_frags; struct mutex tx_lock; /* protects partially_sent_* fields and diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c index f2e7302a4d96..3d45fdb5c4e9 100644 --- a/net/tls/tls_main.c +++ b/net/tls/tls_main.c @@ -125,7 +125,10 @@ int tls_push_sg(struct sock *sk, u16 first_offset, int flags) { - int sendpage_flags = flags | MSG_SENDPAGE_NOTLAST; + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SENDPAGE_NOTLAST | MSG_SPLICE_PAGES | flags, + }; int ret = 0; struct page *p; size_t size; @@ -134,16 +137,19 @@ int tls_push_sg(struct sock *sk, size = sg->length - offset; offset += sg->offset; - ctx->in_tcp_sendpages = true; + ctx->splicing_pages = true; while (1) { if (sg_is_last(sg)) - sendpage_flags = flags; + msg.msg_flags = flags; /* is sending application-limited? */ tcp_rate_check_app_limited(sk); p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, sendpage_flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + + ret = tcp_sendmsg_locked(sk, &msg, size); if (ret != size) { if (ret > 0) { @@ -155,7 +161,7 @@ int tls_push_sg(struct sock *sk, offset -= sg->offset; ctx->partially_sent_offset = offset; ctx->partially_sent_record = (void *)sg; - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return ret; } @@ -169,7 +175,7 @@ int tls_push_sg(struct sock *sk, size = sg->length; } - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return 0; } @@ -247,11 +253,11 @@ static void tls_write_space(struct sock *sk) { struct tls_context *ctx = tls_get_ctx(sk); - /* If in_tcp_sendpages call lower protocol write space handler + /* If splicing_pages call lower protocol write space handler * to ensure we wake up any waiting operations there. For example - * if do_tcp_sendpages where to call sk_wait_event. + * if splicing pages where to call sk_wait_event. */ - if (ctx->in_tcp_sendpages) { + if (ctx->splicing_pages) { ctx->sk_write_space(sk); return; } From patchwork Thu May 18 13:07:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95880 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp495015vqo; Thu, 18 May 2023 06:32:11 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4nmYX0cdhRHOCmDBX+Ht+DhGgwBlT956jVseNM+rwJR5y7vBytsJLpEybgL5ljW2B64Ttl X-Received: by 2002:a05:6a20:5493:b0:100:85e2:26c9 with SMTP id i19-20020a056a20549300b0010085e226c9mr2713620pzk.13.1684416730984; Thu, 18 May 2023 06:32:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684416730; cv=none; d=google.com; s=arc-20160816; b=VY354ELLfjbY1RL1WBwwPlUm3Jaj9zWJjuqElMKejNDyqyJrkDAQ9Hzw1+Eu2FZ0xQ A9yL1n3Ocr90HQ919X38J1kiXj2NtAI9btcUWxn2CbMADN0qy3tBdC5jgz88da1PkUlM 5WQRILzDRFaIGVXw56LXsXiIEuKEIUXXlxVIDHhd+1wuFdsEyitW40p+N71YgesQhG+b YPufh7ya0OHEkZoYOUTwoATFZrOrnPKQ3j6+bD8M33s7cQnE4eUrVxG0SKwSMOYSaFf7 HNtozQY+y8+bGHD7/q6IozSKgdu4/CoIoxTOJfBm0DuSCRFtjQ+rVL4/ni9rfkcuh6Zq sjAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=tEe6KsGIUg5Ug9eymZUK4cq5L1lxhWlwpIXxPsGo3ck=; b=VmoyEHb+LMLJYcF5V94KAoPmWtrG+Omi4mpa82RlHOEVPbZOJAkfY6tt+aFUoDOypD rb+b49TgK7WQasuHAb6u7BeqOOvfkwP3td4BPLnqj33grtea04wxVQbXLV3x3WL3mNNH z+yWJidcR+DLKAdx/aiWtSlmu552Z5+jr42A8zEFvwqF0Fn7DX4xIDQUVNm4JiVzE5aU U5sPP5IpqqYXNPWckZsAEiIFhVIi8M4m1nSdC6jBeQYcvqshLU7eznRCaSyGxr+3rjkj zsmcwL4BfE1ewIniZZxlc8qD+S5akZ3qRnbqfyNT1VCmKbpf+zHeGvCBqVXr/DKO+3lk 20ew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=INZcY7WI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b7-20020a63d807000000b0053468421910si1051919pgh.620.2023.05.18.06.31.55; Thu, 18 May 2023 06:32:10 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=INZcY7WI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231468AbjERNJr (ORCPT + 99 others); Thu, 18 May 2023 09:09:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41460 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231334AbjERNJN (ORCPT ); Thu, 18 May 2023 09:09:13 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D83D173A for ; Thu, 18 May 2023 06:08:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415287; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tEe6KsGIUg5Ug9eymZUK4cq5L1lxhWlwpIXxPsGo3ck=; b=INZcY7WItzY8HAdslSVXUWkgoKvQoEzEl1qVYbdzmB+OGKieC1qK+WSE6fjCsUxiGYwu3n A6ixJzr1vb4+ih+/pWmTrn0YyODjbqWCrHef6DMEYaqcmISnBRqOPl/Dl8r9EgRQTp18z8 j2F/cSkNYSRoA7Qk8SzOKLDBWsEU6HE= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-159-rWJ6E0svM2ScwbkcEaLtzw-1; Thu, 18 May 2023 09:08:03 -0400 X-MC-Unique: rWJ6E0svM2ScwbkcEaLtzw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CE5503815F79; Thu, 18 May 2023 13:08:01 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5DF3B492B01; Thu, 18 May 2023 13:07:57 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Bernard Metzler , Tom Talpey , Jason Gunthorpe , Leon Romanovsky , linux-rdma@vger.kernel.org Subject: [PATCH net-next v9 09/16] siw: Inline do_tcp_sendpages() Date: Thu, 18 May 2023 14:07:06 +0100 Message-Id: <20230518130713.1515729-10-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766238958103315943?= X-GMAIL-MSGID: =?utf-8?q?1766238958103315943?= do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells Reviewed-by: Bernard Metzler Reviewed-by: Tom Talpey cc: Jason Gunthorpe cc: Leon Romanovsky cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: netdev@vger.kernel.org --- Notes: ver #6) - Don't clear MSG_SPLICE_PAGES on the last page. drivers/infiniband/sw/siw/siw_qp_tx.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_qp_tx.c b/drivers/infiniband/sw/siw/siw_qp_tx.c index 4b292e0504f1..ffb16beb6c30 100644 --- a/drivers/infiniband/sw/siw/siw_qp_tx.c +++ b/drivers/infiniband/sw/siw/siw_qp_tx.c @@ -312,7 +312,7 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, } /* - * 0copy TCP transmit interface: Use do_tcp_sendpages. + * 0copy TCP transmit interface: Use MSG_SPLICE_PAGES. * * Using sendpage to push page by page appears to be less efficient * than using sendmsg, even if data are copied. @@ -323,20 +323,27 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, static int siw_tcp_sendpages(struct socket *s, struct page **page, int offset, size_t size) { + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = (MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST | + MSG_SPLICE_PAGES), + }; struct sock *sk = s->sk; - int i = 0, rv = 0, sent = 0, - flags = MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST; + int i = 0, rv = 0, sent = 0; while (size) { size_t bytes = min_t(size_t, PAGE_SIZE - offset, size); if (size + offset <= PAGE_SIZE) - flags = MSG_MORE | MSG_DONTWAIT; + msg.msg_flags &= ~MSG_SENDPAGE_NOTLAST; tcp_rate_check_app_limited(sk); + bvec_set_page(&bvec, page[i], bytes, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + try_page_again: lock_sock(sk); - rv = do_tcp_sendpages(sk, page[i], offset, bytes, flags); + rv = tcp_sendmsg_locked(sk, &msg, size); release_sock(sk); if (rv > 0) { From patchwork Thu May 18 13:07:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95862 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp481103vqo; Thu, 18 May 2023 06:11:12 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6LO64o7kVRNOQIb9oSFNjkpxBH6QNe1pwIM4WyWqCf5n6e8fKd//bxU7yeE1ctas+zCQqk X-Received: by 2002:a05:6a20:72a3:b0:e9:5b0a:deff with SMTP id o35-20020a056a2072a300b000e95b0adeffmr2709427pzk.22.1684415471938; Thu, 18 May 2023 06:11:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684415471; cv=none; d=google.com; s=arc-20160816; b=LCLqifw32xAQBhWhzDClO2p2OMYQGxpNQsfTLt6uFmuHMZLM6gU28laSkmOYBb8+AM Z4JTooEagjDVhJfDW25ZL29yPopYLhtoPPIG+c+IPxJM7OEtblA4EMQ58+s92QMMfSRx sp5Y0q5aaSb4oICcpL9s11QSSeCD90twBPVo54nYvR3ffMx53IV3x+Q4A4pCD34r4OFG 4Pobe0a/l+d2kF/AB9RO8TeuN40fxu9cp3sED7yUWNESY6Ih0a/5b+wM+0fqEn1MOnii 5PmcQN/yxio9ohmRK3fUgNegmFYiaI0HnZL7Cvp7+tMt8vgk227G+oIW5ojmdvP8zL4J t5ZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=FkLIsuWcxLdm5m/RLKw/6bFEvudw3Dbf8/jbJKlIa0s=; b=g7D1LFWX6ZqoHV69m+7mGx++1pxHrG3ri/GMg8nTjduy6BmGowiArs7mWnsOJo2nnO CRJ3CnylQ0zAEbySOPz5lumYASV78sS+JtJUCx+vfaKLfxinN8hVfxp5VX8HCsXXLd3s V8AlUBK03JMY4TucuAIk+YFOq5pgOF6uX2EcJ4RPwdeBGHL7TNtuN8dkRx7GgV2gF5Dq rt01boYxlxAQXZydxXtyB+60T4VTGT84XxKW4KgN+trEFJRYvVqgkhMoNWi8yuBcuXrw 3kf3GnREtQ+ZzZhx7FZzWudKTfwfMYgUehmwv76Hqj5F/IE4Td74nCUf8h47qEzCWwWw lQHQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=M7VpMFAB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b124-20020a633482000000b005346e1b7611si1339951pga.857.2023.05.18.06.10.56; Thu, 18 May 2023 06:11:11 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=M7VpMFAB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231343AbjERNJu (ORCPT + 99 others); Thu, 18 May 2023 09:09:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41126 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231338AbjERNJO (ORCPT ); Thu, 18 May 2023 09:09:14 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6AE111721 for ; Thu, 18 May 2023 06:08:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415292; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FkLIsuWcxLdm5m/RLKw/6bFEvudw3Dbf8/jbJKlIa0s=; b=M7VpMFABjBxC5i5STrvWa6eucx+QXxSfVQZp+A4ahShtb6V8MViJBzQiAc8Vf7JJGAK96B 7P8ZGYLH/lCJu7hlJO8Eogu6llO3FZL/TUy85Ipkn3KBxGBuM5hutsT+xXYemp14+Rwjb/ EspAZFD5evr/sB7b9jIO7siqzCJ6qAk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-459-_LTtSiUiMLObar1uKYQXhg-1; Thu, 18 May 2023 09:08:08 -0400 X-MC-Unique: _LTtSiUiMLObar1uKYQXhg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 594D0857DD7; Thu, 18 May 2023 13:08:07 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id A93852026D16; Thu, 18 May 2023 13:08:04 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 10/16] tcp: Fold do_tcp_sendpages() into tcp_sendpage_locked() Date: Thu, 18 May 2023 14:07:07 +0100 Message-Id: <20230518130713.1515729-11-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766237637956202013?= X-GMAIL-MSGID: =?utf-8?q?1766237637956202013?= Fold do_tcp_sendpages() into its last remaining caller, tcp_sendpage_locked(). Signed-off-by: David Howells cc: Eric Dumazet cc: David Ahern cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tcp.h | 2 -- net/ipv4/tcp.c | 21 +++++++-------------- 2 files changed, 7 insertions(+), 16 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index 04a31643cda3..02a6cff1827e 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -333,8 +333,6 @@ int tcp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags); int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags); -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags); int tcp_send_mss(struct sock *sk, int *size_goal, int flags); void tcp_push(struct sock *sk, int flags, int mss_now, int nonagle, int size_goal); diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index f3a0c02678e0..e9506cebecce 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -974,12 +974,17 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags) +int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, + size_t size, int flags) { struct bio_vec bvec; struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + if (!(sk->sk_route_caps & NETIF_F_SG)) + return sock_no_sendpage_locked(sk, page, offset, size, flags); + + tcp_rate_check_app_limited(sk); /* is sending application-limited? */ + bvec_set_page(&bvec, page, size, offset); iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); @@ -988,18 +993,6 @@ ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, return tcp_sendmsg_locked(sk, &msg, size); } -EXPORT_SYMBOL_GPL(do_tcp_sendpages); - -int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, - size_t size, int flags) -{ - if (!(sk->sk_route_caps & NETIF_F_SG)) - return sock_no_sendpage_locked(sk, page, offset, size, flags); - - tcp_rate_check_app_limited(sk); /* is sending application-limited? */ - - return do_tcp_sendpages(sk, page, offset, size, flags); -} EXPORT_SYMBOL_GPL(tcp_sendpage_locked); int tcp_sendpage(struct sock *sk, struct page *page, int offset, From patchwork Thu May 18 13:07:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95863 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp481545vqo; Thu, 18 May 2023 06:11:40 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4AjXPiYIhuggS3huKJWNYrC+fQTlH8VI3lDQcrsUg8lqSK2O4VBe6WRXabah0WCGN4yTDd X-Received: by 2002:a17:902:e5ce:b0:1ad:e3a8:3bf with SMTP id u14-20020a170902e5ce00b001ade3a803bfmr3105534plf.21.1684415499849; Thu, 18 May 2023 06:11:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684415499; cv=none; d=google.com; s=arc-20160816; b=YeBnPCSJBOLbWar9dLoDQAG8b7iEHWrXlQ1gPJNV5Yky9gVzPkQhuPwLd/MQCRkEo8 Kq8bDopK8W08Aqv1+KecKKq0yb2aMi5GHtlCsY3TZ4/+2ChkSNLWFCun1sOLqLPNcttE fILdxvCgk+wJldGfBpIZ7LQwgM6OlfBE9GUb+Wq6ywzJIZf/adJUU1N8rKjS2cycDXym KmTpnzG4ARBo5/+tC/s9tqt91m/xJ41s2p73FdezIj2aOK5OLipnCf9lAaVN0u+wY/6S CEbSPB9LSjYhNSdNnqLhsh9CqT2w4lBIvO+xk2/rnwOTcuOiBlTqXUdNTRYh7edqOd5U tR7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Di14zlocOEDECmKOzgXOEyS4sjRLATonTy1WJdIXAUU=; b=ymuBREOEy0C7iG5JEFQob3D/fyFZGGEIhOiIInWqx+M3mKYamzys1Vt5x6sOrAtryf ISRTlFRgbOzJcPw0h3gvUazt1OsfP00+TYqyg5TPM7Bj94upQIKQnCOS+zR9po+NflYA eAYD6aQNehnCH8gy5owfpg1FGLc0D3WejlTmpJctK8N2D/EKpe1tCHyK0W6dpFMLi8tB n4WeBFOPiu53S3hvLAFCcUhqzTvb1KSV8crh19DeTN2K9UkVJv7Ie1GAjEy63lZEUFOG XpnY93n4zEUtU5KjcUCKjFNBcrQ5GpJu2YRAOMF5FIv0kxadJc+oLh3SIjJJ/IZtcmSN PoRg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=IZPfQphn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id jj2-20020a170903048200b001ab259969e7si1279098plb.31.2023.05.18.06.11.21; Thu, 18 May 2023 06:11:39 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=IZPfQphn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231493AbjERNJw (ORCPT + 99 others); Thu, 18 May 2023 09:09:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41492 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231351AbjERNJS (ORCPT ); Thu, 18 May 2023 09:09:18 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3028173D for ; Thu, 18 May 2023 06:08:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415295; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Di14zlocOEDECmKOzgXOEyS4sjRLATonTy1WJdIXAUU=; b=IZPfQphnxvTxMtTbxTtjCHZJUQrTcR7Dv8VJuWWvOzvw7GkbZ4AOnljsqz4quz8vfHCiSd IN64I9ELnjCxoQUPnlclu6phuNyXDseh9aygXcQoh6glXYz0YyStdNpRhmFNeqQzXVluGr QrA5KqJZ2DQRwHgCcl2bNMEfxpEO4iE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-255-IupZrLRDOgiU6UQOydG1Gw-1; Thu, 18 May 2023 09:08:12 -0400 X-MC-Unique: IupZrLRDOgiU6UQOydG1Gw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2621484AF2F; Thu, 18 May 2023 13:08:11 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0967F40C2063; Thu, 18 May 2023 13:08:07 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 11/16] ip, udp: Support MSG_SPLICE_PAGES Date: Thu, 18 May 2023 14:07:08 +0100 Message-Id: <20230518130713.1515729-12-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766237667124973668?= X-GMAIL-MSGID: =?utf-8?q?1766237667124973668?= Make IP/UDP sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/ipv4/ip_output.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 52fc840898d8..c7db973b5d29 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1048,6 +1048,14 @@ static int __ip_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + if (inet->hdrincl) + return -EPERM; + if (rt->dst.dev->features & NETIF_F_SG) + /* We need an empty buffer to attach stuff to */ + paged = true; + else + flags &= ~MSG_SPLICE_PAGES; } cork->length += length; @@ -1207,6 +1215,15 @@ static int __ip_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + struct msghdr *msg = from; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) + goto error; + copy = err; + wmem_alloc_delta += copy; } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Thu May 18 13:07:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95861 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp480919vqo; Thu, 18 May 2023 06:10:57 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5WhnPKPE65cavOaRO8R23TwcWji1sn4FupnetF7k52tuqvm2QyKJ/uqtzmgzZfZkIxM3OY X-Received: by 2002:a05:6a00:1885:b0:63b:8dcc:84de with SMTP id x5-20020a056a00188500b0063b8dcc84demr3737737pfh.4.1684415457534; Thu, 18 May 2023 06:10:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684415457; cv=none; d=google.com; s=arc-20160816; b=siXCuuIOXy4E7fVTP0k6PzsNwi9tvVPQRefSl24AXwx8ucTS1ClrcYyBcDPpcV2WjH 3GigPM2GwIcdL2uQg72tFpnWYKtIPq3L4BJWURFQHSgb8axMEDJBQ3/MeLBMsEjWJFv0 TULOcysGKYT8G7ci87ILe3iWaui1hMAEDeHAYHCOtWsh/xU5mnY0F229qCKaMnEbHAaM ZUA02WoGcoQX+ej2SV30xFFbzUGSzpPcqSEAgNVAFapfxmMuAHl6f536cRYVEH14IysG slddMoDO5WFLAaEvgZDEbq5/yiBVNHslp3Y4BBFeCdDw5aSJp1M/q44AHIspmfjD1grE UYwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5dEuCyEYbAeTLbptDWG+lpG3TePaGyDY6iKA6gIr8Gk=; b=t9nCnUEKw93VjaLEPt6XUrBHwYAcqoZ5qS1RD3I+7XaYeyFbIJ5BeRPrAcRt6s6e+F DlA4p3Mu7oDvFZ1f+4cmtbHCButUioat2iG6FePZ4vu9gyi4O/mW1ljgaMOcZnXgGbat L9m93HoKIXH8HwmbUUDdjiB0YEX5tc7z2rF7Gll1RU6Af9YLysrBn1xq0pr/k+z+kw/D URZK2UxMSq0UR/Tav+vl3gsrPL+yzxFDYmk1WbRwmlMFWHFhNQqb0C4q3AAyVD3pZcnV NxPzzuINxq2ZD8Ng11Mx8c0MBW9uxJoNekKWWa8GD5AKUASPbaGzI8jzH/mg8Z6d1JYg 4hCw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=c3wExOHS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x23-20020aa78f17000000b0063b7bb98251si1575939pfr.216.2023.05.18.06.10.34; Thu, 18 May 2023 06:10:57 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=c3wExOHS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231434AbjERNJo (ORCPT + 99 others); Thu, 18 May 2023 09:09:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41130 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231329AbjERNJM (ORCPT ); Thu, 18 May 2023 09:09:12 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80960198A for ; Thu, 18 May 2023 06:08:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415299; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5dEuCyEYbAeTLbptDWG+lpG3TePaGyDY6iKA6gIr8Gk=; b=c3wExOHSMc8Frx8NgXUiqkVRaxzvkh9VxHV/Ox9dDrs/0O3w7GCaIE0HTOij6BlXjJ/FuZ 8uOGCQxOOc1xDKumaGnjWw0zVOMIE9IUYtGLG6yl0QJ2lKeGdv6Q2YDBlU5v822Y3O7Bbc bqqdUAu0oERjvCmXwc+0djpDvpEuy7E= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-638-lV58l2MBPkWSA52mh6FUSg-1; Thu, 18 May 2023 09:08:16 -0400 X-MC-Unique: lV58l2MBPkWSA52mh6FUSg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2F4D128237C8; Thu, 18 May 2023 13:08:15 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 397681121318; Thu, 18 May 2023 13:08:12 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 12/16] ip6, udp6: Support MSG_SPLICE_PAGES Date: Thu, 18 May 2023 14:07:09 +0100 Message-Id: <20230518130713.1515729-13-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766237622921110307?= X-GMAIL-MSGID: =?utf-8?q?1766237622921110307?= Make IP6/UDP6 sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible, copying the data if not. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/ipv6/ip6_output.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 9554cf46ed88..c722cb881b2d 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1589,6 +1589,14 @@ static int __ip6_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + if (inet_sk(sk)->hdrincl) + return -EPERM; + if (rt->dst.dev->features & NETIF_F_SG) + /* We need an empty buffer to attach stuff to */ + paged = true; + else + flags &= ~MSG_SPLICE_PAGES; } /* @@ -1778,6 +1786,15 @@ static int __ip6_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + struct msghdr *msg = from; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) + goto error; + copy = err; + wmem_alloc_delta += copy; } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Thu May 18 13:07:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95893 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp497363vqo; Thu, 18 May 2023 06:35:07 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7ugUk6aJus/bbDct/sg/GLT1Ws+A3IzMVuQdwkxhQeLVo1Pa5wn9cB7itv3EZ23Tn3Liea X-Received: by 2002:a17:90a:6641:b0:24e:3a0a:9b48 with SMTP id f1-20020a17090a664100b0024e3a0a9b48mr2687332pjm.21.1684416907180; Thu, 18 May 2023 06:35:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684416907; cv=none; d=google.com; s=arc-20160816; b=Qy6yexafUQlye9fkXC4PLiZwkzfTwRs0GG5sWE68e2IGSkW35OCleObBK4HGFkjPjC /YRnazJH4egxwefSrC3pxR5oxdKJE1dgGSfTbYLK+mIMU6O5ob8u9VUN8ZTN3syNoLth bkpEPUHl04tIdZORPTDqQ+uupsYRF/Sr+LrCzRBD9qDsHidQOy+8cxngZouPWVrxVVa3 GfhGCRtuebx6wMC9eNfjRvJkkmyV4LEoBORplfrpXcBlUNbQNBhRvFMIqh96M+CNZS5x 7gntx9G3Y85etROTfFOGGWzbHFhVREsU6QnPYZ67ROtjxcrqvGvmrVZLUBFFKoXonh+E aMWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=arPLIYOMzcdUPWLjuF22tHo0vP4jVzFFThikgCzP+CE=; b=x+O61mNYb/0hAzlWMl9WZVfzeP8Fj34m9ceu1R1cKuZ0ravOO7QSxEjAEbBQa6RxYT MxLZw65v4MY8y008/EWjDa1ykcDx7zBvdj6R6l/OMz3k2TM+gPvHluTBnSnWzQgIc0Ti flvGmW/pEOFONhgt92HDEeq+NeYLIEy6q/ann54ZnJ5BX7DxPN7M7ZoBWPnZHIm2mfa6 xdi98UIpLDgH8Ae5uEBcsExSESnoFHL8YaTwUQkpTKDJMk86Uj3RcW7kSZxvDVSa3Lm+ 8SDHeh8INhsIjlrM4Bo0HYKPWKBY/dtpt2xHUXqV1bc4B0dVF/ByWW43mF70XvEOwDAg B/zA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=LYzg1Z8M; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n17-20020a17090a929100b00250bf29b6d6si4156764pjo.165.2023.05.18.06.34.48; Thu, 18 May 2023 06:35:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=LYzg1Z8M; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231513AbjERNKQ (ORCPT + 99 others); Thu, 18 May 2023 09:10:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41510 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230422AbjERNJf (ORCPT ); Thu, 18 May 2023 09:09:35 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F163B19A5 for ; Thu, 18 May 2023 06:08:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415302; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=arPLIYOMzcdUPWLjuF22tHo0vP4jVzFFThikgCzP+CE=; b=LYzg1Z8M7W1ijB1N4wPMu1mUCFV+AOkQl37EnTSvrB+XeEM7ceBYfqQcw/dUEODqxWMVjm SVt+JMj+pqzFYyuBU0OHPExmuL6mPVS6hYz9XSUa0o6Tz3JEDxTDZqW98HfRfdGG1wCbHI m6tHu2qfzZIgNt6E2lsx6GGKtqRGY/s= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-122-MLznaae9M3inxRR4K92iug-1; Thu, 18 May 2023 09:08:19 -0400 X-MC-Unique: MLznaae9M3inxRR4K92iug-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7E93928237C3; Thu, 18 May 2023 13:08:18 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 18F1E1121315; Thu, 18 May 2023 13:08:15 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 13/16] udp: Convert udp_sendpage() to use MSG_SPLICE_PAGES Date: Thu, 18 May 2023 14:07:10 +0100 Message-Id: <20230518130713.1515729-14-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766239142578424245?= X-GMAIL-MSGID: =?utf-8?q?1766239142578424245?= Convert udp_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - udp_sendpage() shouldn't lock the socket around udp_sendpage(). - udp_sendpage() should only set MSG_MORE if MSG_SENDPAGE_NOTLAST is set. net/ipv4/udp.c | 51 ++++++-------------------------------------------- 1 file changed, 6 insertions(+), 45 deletions(-) diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index aa32afd871ee..2879dc6d66ea 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -1332,54 +1332,15 @@ EXPORT_SYMBOL(udp_sendmsg); int udp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct inet_sock *inet = inet_sk(sk); - struct udp_sock *up = udp_sk(sk); - int ret; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES }; if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; - - if (!up->pending) { - struct msghdr msg = { .msg_flags = flags|MSG_MORE }; - - /* Call udp_sendmsg to specify destination address which - * sendpage interface can't pass. - * This will succeed only when the socket is connected. - */ - ret = udp_sendmsg(sk, &msg, 0); - if (ret < 0) - return ret; - } - - lock_sock(sk); + msg.msg_flags |= MSG_MORE; - if (unlikely(!up->pending)) { - release_sock(sk); - - net_dbg_ratelimited("cork failed\n"); - return -EINVAL; - } - - ret = ip_append_page(sk, &inet->cork.fl.u.ip4, - page, offset, size, flags); - if (ret == -EOPNOTSUPP) { - release_sock(sk); - return sock_no_sendpage(sk->sk_socket, page, offset, - size, flags); - } - if (ret < 0) { - udp_flush_pending_frames(sk); - goto out; - } - - up->len += size; - if (!(READ_ONCE(up->corkflag) || (flags&MSG_MORE))) - ret = udp_push_pending_frames(sk); - if (!ret) - ret = size; -out: - release_sock(sk); - return ret; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return udp_sendmsg(sk, &msg, size); } #define UDP_SKB_IS_STATELESS 0x80000000 From patchwork Thu May 18 13:07:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95868 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp487596vqo; Thu, 18 May 2023 06:20:13 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5C3HUa26jyDKS2ONgHyL+mynBf9O/vlVCwVW85hCvDzePeJzfwZQ361Kbuwuu3F9bu13Os X-Received: by 2002:a17:902:e5c8:b0:1ac:4fb3:1693 with SMTP id u8-20020a170902e5c800b001ac4fb31693mr2782926plf.52.1684416013452; Thu, 18 May 2023 06:20:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684416013; cv=none; d=google.com; s=arc-20160816; b=Uoq4EsdLGVZ+8O95u3VS7WTX6DSAJdcCYkVLQIfPMkLwTjoXAWk8KDC4xdpvZvSABW PEaa9wYG6t8o4c8+FNaI/HKw1e7Y+VAdqDaMQH9WxrdXXsI+Yhb/XhjGYmJ836PKyYQm 0YM9cuBaBqMp6Fvo6RVdpcHedjaoP86J/IgTKrpzDG7mhkvgfbudMSqhcTOLNhOi+mnv SKx6kSepUexzDzLkQj7B6/EJBga67t8majTiF8LdPT07Nqv+I3Ny1Rc5C/Cmm4aUtPsv JNWN6ExfJ4DKkb6b66bD4PHBF7RHlC2LElx7gj9PqpGyUl+NMz6MJRoPD3Ry85MbhJu9 mO0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vVs0CsGrBKFATDm79TCVboT9gSnZlC7XDeXTGZfAJzM=; b=Gid0dnddaikIsPKoavGv/2VhJaXUkJWtRHDUicS/dXUIJ3xvmPm+SxFSmz9JhJFw0b HPXYJVow+AM7pFvuES5p4E4BlgPwa6ydN954XgtBihNnTB0xg6AiEBYmMcjDnCSvGIG1 fgaPqoMo4zqoHhGBH0UKZpa2/TwRdH7ewT7GAaQLFyo07aQK506pvciqUS45Q30Umdp9 BavbkKV8ZixVnQCyAAKZKRWEb6KAeLsAsGFFFTYdWx6Ain/WQ6nn7LOB+5yu87MUXZl9 ka+io+ReYcTe1hdHFnMBf0J3U+rdBAYkdDBF9USqyEjNV+Gr6rMtnlvbMtOSf7fp+L1i yAAA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=BzIykeKi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d8-20020a170902654800b001ab286e740bsi1216434pln.328.2023.05.18.06.19.58; Thu, 18 May 2023 06:20:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=BzIykeKi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231424AbjERNKW (ORCPT + 99 others); Thu, 18 May 2023 09:10:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41830 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231405AbjERNJh (ORCPT ); Thu, 18 May 2023 09:09:37 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5F2AE1738 for ; Thu, 18 May 2023 06:08:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415308; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vVs0CsGrBKFATDm79TCVboT9gSnZlC7XDeXTGZfAJzM=; b=BzIykeKiTe1rT9Hb3uHNw4AFpenLRMjOTmB8zXuUPu/lywVenGLVnb7b4Ua6p2M6kKZXbk H+18kfXFURwvbIuR42+gT+mHZAwasRc510sZDuiaQOLWcRZewiuXOiILimNq32/WrWspoH KgQtzBQ7yqgxI+c2/dNr+GoW53YBhyA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-433-p1Z1QuysOXekhrovt4BGFQ-1; Thu, 18 May 2023 09:08:24 -0400 X-MC-Unique: p1Z1QuysOXekhrovt4BGFQ-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3684385A5B1; Thu, 18 May 2023 13:08:22 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7D030492C13; Thu, 18 May 2023 13:08:19 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v9 14/16] ip: Remove ip_append_page() Date: Thu, 18 May 2023 14:07:11 +0100 Message-Id: <20230518130713.1515729-15-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766238205597644811?= X-GMAIL-MSGID: =?utf-8?q?1766238205597644811?= ip_append_page() is no longer used with the removal of udp_sendpage(), so remove it. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #7) - Remove now-unused csum_page(). include/net/ip.h | 2 - net/ipv4/ip_output.c | 148 ++----------------------------------------- 2 files changed, 4 insertions(+), 146 deletions(-) diff --git a/include/net/ip.h b/include/net/ip.h index c3fffaa92d6e..7627a4df893b 100644 --- a/include/net/ip.h +++ b/include/net/ip.h @@ -220,8 +220,6 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4, unsigned int flags); int ip_generic_getfrag(void *from, char *to, int offset, int len, int odd, struct sk_buff *skb); -ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, - int offset, size_t size, int flags); struct sk_buff *__ip_make_skb(struct sock *sk, struct flowi4 *fl4, struct sk_buff_head *queue, struct inet_cork *cork); diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index c7db973b5d29..553c740a6bfb 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -946,17 +946,6 @@ ip_generic_getfrag(void *from, char *to, int offset, int len, int odd, struct sk } EXPORT_SYMBOL(ip_generic_getfrag); -static inline __wsum -csum_page(struct page *page, int offset, int copy) -{ - char *kaddr; - __wsum csum; - kaddr = kmap(page); - csum = csum_partial(kaddr + offset, copy, 0); - kunmap(page); - return csum; -} - static int __ip_append_data(struct sock *sk, struct flowi4 *fl4, struct sk_buff_head *queue, @@ -1327,10 +1316,10 @@ static int ip_setup_cork(struct sock *sk, struct inet_cork *cork, } /* - * ip_append_data() and ip_append_page() can make one large IP datagram - * from many pieces of data. Each pieces will be holded on the socket - * until ip_push_pending_frames() is called. Each piece can be a page - * or non-page data. + * ip_append_data() can make one large IP datagram from many pieces of + * data. Each piece will be held on the socket until + * ip_push_pending_frames() is called. Each piece can be a page or + * non-page data. * * Not only UDP, other transport protocols - e.g. raw sockets - can use * this interface potentially. @@ -1363,135 +1352,6 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4, from, length, transhdrlen, flags); } -ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, - int offset, size_t size, int flags) -{ - struct inet_sock *inet = inet_sk(sk); - struct sk_buff *skb; - struct rtable *rt; - struct ip_options *opt = NULL; - struct inet_cork *cork; - int hh_len; - int mtu; - int len; - int err; - unsigned int maxfraglen, fragheaderlen, fraggap, maxnonfragsize; - - if (inet->hdrincl) - return -EPERM; - - if (flags&MSG_PROBE) - return 0; - - if (skb_queue_empty(&sk->sk_write_queue)) - return -EINVAL; - - cork = &inet->cork.base; - rt = (struct rtable *)cork->dst; - if (cork->flags & IPCORK_OPT) - opt = cork->opt; - - if (!(rt->dst.dev->features & NETIF_F_SG)) - return -EOPNOTSUPP; - - hh_len = LL_RESERVED_SPACE(rt->dst.dev); - mtu = cork->gso_size ? IP_MAX_MTU : cork->fragsize; - - fragheaderlen = sizeof(struct iphdr) + (opt ? opt->optlen : 0); - maxfraglen = ((mtu - fragheaderlen) & ~7) + fragheaderlen; - maxnonfragsize = ip_sk_ignore_df(sk) ? 0xFFFF : mtu; - - if (cork->length + size > maxnonfragsize - fragheaderlen) { - ip_local_error(sk, EMSGSIZE, fl4->daddr, inet->inet_dport, - mtu - (opt ? opt->optlen : 0)); - return -EMSGSIZE; - } - - skb = skb_peek_tail(&sk->sk_write_queue); - if (!skb) - return -EINVAL; - - cork->length += size; - - while (size > 0) { - /* Check if the remaining data fits into current packet. */ - len = mtu - skb->len; - if (len < size) - len = maxfraglen - skb->len; - - if (len <= 0) { - struct sk_buff *skb_prev; - int alloclen; - - skb_prev = skb; - fraggap = skb_prev->len - maxfraglen; - - alloclen = fragheaderlen + hh_len + fraggap + 15; - skb = sock_wmalloc(sk, alloclen, 1, sk->sk_allocation); - if (unlikely(!skb)) { - err = -ENOBUFS; - goto error; - } - - /* - * Fill in the control structures - */ - skb->ip_summed = CHECKSUM_NONE; - skb->csum = 0; - skb_reserve(skb, hh_len); - - /* - * Find where to start putting bytes. - */ - skb_put(skb, fragheaderlen + fraggap); - skb_reset_network_header(skb); - skb->transport_header = (skb->network_header + - fragheaderlen); - if (fraggap) { - skb->csum = skb_copy_and_csum_bits(skb_prev, - maxfraglen, - skb_transport_header(skb), - fraggap); - skb_prev->csum = csum_sub(skb_prev->csum, - skb->csum); - pskb_trim_unique(skb_prev, maxfraglen); - } - - /* - * Put the packet on the pending queue. - */ - __skb_queue_tail(&sk->sk_write_queue, skb); - continue; - } - - if (len > size) - len = size; - - if (skb_append_pagefrags(skb, page, offset, len, - MAX_SKB_FRAGS)) { - err = -EMSGSIZE; - goto error; - } - - if (skb->ip_summed == CHECKSUM_NONE) { - __wsum csum; - csum = csum_page(page, offset, len); - skb->csum = csum_block_add(skb->csum, csum, skb->len); - } - - skb_len_add(skb, len); - refcount_add(len, &sk->sk_wmem_alloc); - offset += len; - size -= len; - } - return 0; - -error: - cork->length -= size; - IP_INC_STATS(sock_net(sk), IPSTATS_MIB_OUTDISCARDS); - return err; -} - static void ip_cork_release(struct inet_cork *cork) { cork->flags &= ~IPCORK_OPT; From patchwork Thu May 18 13:07:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95897 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp497962vqo; Thu, 18 May 2023 06:36:00 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ404frTC0GnAUctVKpcEPOmA097pCFy4j1xs1qjkQxGPF6QBfxwEi3Gvg3RrY44rsRyXuB4 X-Received: by 2002:aa7:8885:0:b0:643:59ed:5dc9 with SMTP id z5-20020aa78885000000b0064359ed5dc9mr5098242pfe.12.1684416960618; Thu, 18 May 2023 06:36:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684416960; cv=none; d=google.com; s=arc-20160816; b=wbYVpjwOsJ6A4iPv61b3/S5SNqP47Kn6SjfcNwAFTEz32ZfrNuJWp1jHgKLLTztkAJ R2puLP5BQhkrocyb37q4peB3nB+QMVkyFLaTjZan//xpWeIi/6s3yTfhL9ZFasVwZ0UF Qp5Z5dnW/jCicYQAztTOsJxEnejtofD30LX4ZSWeU1GhKM7cJNB70JVXfJuj8C4rOUQm NaQGpaXccY9tAPU1wt4wHVNvtuPZDicOPsk4145rbpgHuVKDPYEdQQUcjWSGwVhNRS/8 LMtAhAtC4/9B61xKbE4yH9eCSzQpOycpoOKb3NDLnTq3PZy+1JY9Xx9eqb2eZQ/RlDw9 SQ9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4sbDm+4sgaynAjmiQFVbDl+OyfXvG2vWGJtRH+IAST4=; b=kU8EsNEP//e/Qppb/keYGefoaBmobthPoFzC7oxg51T2WKy7RoMSJe6jc9iBX3+jCL vbSQokOSKmR3gpNNnyeg2FjGDHWgAkXpmHmKmXAX+RBqgUMk/NMr2DXKs4Jau65Fatks 9l+hYk4RInC0JdRHvenCT2rZABa8xAuSqUUiOB50Qw0ssFtg/xwhuII5vULPSuosxT9x 9Aup16/J7PxMQa51q30f2QkRHFnIA8p4ZH0OfIS+zWcXMUZGg5oTGkAaguI1l9yUdiOT vV7+Qd4mJvQX3Y7fh+zL+ehZvu+oWdU09Cn74Ohm/g8sUCzFDYcckyL19dGgsLjTTt4E jsPw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Ew3cNGPh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f4-20020aa79d84000000b0063d33dc49e8si1666506pfq.363.2023.05.18.06.35.45; Thu, 18 May 2023 06:36:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Ew3cNGPh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231236AbjERNKU (ORCPT + 99 others); Thu, 18 May 2023 09:10:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231288AbjERNJg (ORCPT ); Thu, 18 May 2023 09:09:36 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B84F1BE3 for ; Thu, 18 May 2023 06:08:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415310; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4sbDm+4sgaynAjmiQFVbDl+OyfXvG2vWGJtRH+IAST4=; b=Ew3cNGPhr85+EitiVJQzD1UN9vWQ2ttjdOEQBi9EJlo3LyMe6ylqFDaDelAdkT6HTCuk1S YxWpznZ/bfTLxJIw76u0d+9J636Yd0oHh1m4XyacEDaWLGWPWVTf91Mbk8lyn65WcmNshC W1qkNt2qvumOTPVcCt4h56GFwfrJN4U= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-189-jUFF3OsoPBixAzERtA42UA-1; Thu, 18 May 2023 09:08:28 -0400 X-MC-Unique: jUFF3OsoPBixAzERtA42UA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1BD65101A551; Thu, 18 May 2023 13:08:27 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2DFAB63F5F; Thu, 18 May 2023 13:08:23 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Kuniyuki Iwashima Subject: [PATCH net-next v9 15/16] af_unix: Support MSG_SPLICE_PAGES Date: Thu, 18 May 2023 14:07:12 +0100 Message-Id: <20230518130713.1515729-16-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766239199322240129?= X-GMAIL-MSGID: =?utf-8?q?1766239199322240129?= Make AF_UNIX sendmsg() support MSG_SPLICE_PAGES, splicing in pages from the source iterator if possible and copying the data in otherwise. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Kuniyuki Iwashima cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/unix/af_unix.c | 49 +++++++++++++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 16 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index dd55506b4632..976bc1c5e11b 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2200,19 +2200,25 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, while (sent < len) { size = len - sent; - /* Keep two messages in the pipe so it schedules better */ - size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + skb = sock_alloc_send_pskb(sk, 0, 0, + msg->msg_flags & MSG_DONTWAIT, + &err, 0); + } else { + /* Keep two messages in the pipe so it schedules better */ + size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); - /* allow fallback to order-0 allocations */ - size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); + /* allow fallback to order-0 allocations */ + size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); - data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); + data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); - data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); + data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); - skb = sock_alloc_send_pskb(sk, size - data_len, data_len, - msg->msg_flags & MSG_DONTWAIT, &err, - get_order(UNIX_SKB_FRAGS_SZ)); + skb = sock_alloc_send_pskb(sk, size - data_len, data_len, + msg->msg_flags & MSG_DONTWAIT, &err, + get_order(UNIX_SKB_FRAGS_SZ)); + } if (!skb) goto out_err; @@ -2224,13 +2230,24 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, } fds_sent = true; - skb_put(skb, size - data_len); - skb->data_len = data_len; - skb->len = size; - err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); - if (err) { - kfree_skb(skb); - goto out_err; + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + err = skb_splice_from_iter(skb, &msg->msg_iter, size, + sk->sk_allocation); + if (err < 0) { + kfree_skb(skb); + goto out_err; + } + size = err; + refcount_add(size, &sk->sk_wmem_alloc); + } else { + skb_put(skb, size - data_len); + skb->data_len = data_len; + skb->len = size; + err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); + if (err) { + kfree_skb(skb); + goto out_err; + } } unix_state_lock(other); From patchwork Thu May 18 13:07:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95878 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp494869vqo; Thu, 18 May 2023 06:32:01 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ56TDerAiG6Ln/XKCp4n0ZP3rs2bq1i7ZscEK5+zaPsa0H/g4BpXWVA9cfjpezxuP2UwMVU X-Received: by 2002:a05:6a20:3c8d:b0:103:8c8b:c689 with SMTP id b13-20020a056a203c8d00b001038c8bc689mr2469230pzj.51.1684416720788; Thu, 18 May 2023 06:32:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684416720; cv=none; d=google.com; s=arc-20160816; b=xyHiCbmnDKID+b8CPjC8YkAPIQU62fU3mVIg5QPmAQOSHBY3IPwV4rDG0i3CUiQCjo NduHBM7giCNd5FaV+Ne7Zp4xI7f8hMYlvY9T8TaUGOZMqmL1ts+yKDyXNtl+ICZ/IC37 4qApR3+kpiWXRY97zoLrbhfpZ0ceorUae6qXU5DfKC7qjeYpJ6rY6G+5kQotx4vREkTR Bw3mXg3HB+fv/VhSLG23tNoxI+I/zPQ9Nr1P/D67U7ysnHeWGFbDFS/RNIUmynDGFCKC RbHekRKNP/juvlqg5ul0gSRjsCrUdSH1ePMNJtKBVaQ95yGaCGOOkt5S6pFS/5Cpv622 DO9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=wF+2b4fBEfy00AN3VDBr6eYpnbX7vBgnsz0rKmJdTGo=; b=0/cyKPmQYp3idaLah4K1rkOF5xZtNiW01CZW4c+YNoxPMzbEusTSwzWwXVvu84O+d7 0NEa2nf4aCBO2cf3smNRTYap3CtD0ST8GU1Rq+3Auh+X4USete9a2yFqRvMJvfQ0RQyN Seltafe8soCXIeuhrPrJbm9mhBlqiy7yMPRCewDqo9g/MYNT1tD1F8hZA8kqO6l4pHEi 74rs69Pne0ltzEqIFwmscXp9C40IBynKF/MXFrBdYZixbIkIUUDA3HkCgafi81QGFWvI k5MiFvqUIY6jHliUsyHvub4vVzsHpQ0nSK203oLpxFgunZ9KSfIvjw1lYrvsybpuFqS5 z25g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=UxE+nE6r; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l185-20020a6388c2000000b005073e3342eesi1294737pgd.143.2023.05.18.06.31.45; Thu, 18 May 2023 06:32:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=UxE+nE6r; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231341AbjERNKk (ORCPT + 99 others); Thu, 18 May 2023 09:10:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231347AbjERNJs (ORCPT ); Thu, 18 May 2023 09:09:48 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 982AE1FE7 for ; Thu, 18 May 2023 06:08:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684415315; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wF+2b4fBEfy00AN3VDBr6eYpnbX7vBgnsz0rKmJdTGo=; b=UxE+nE6r/pcY4E/4fVcQlBlZkY4oLYaDZYf3L4YS271fvIaYcu4iywEcRE8TFQI3pam/G3 n7MW3vai2Z3PRjsSl6Sy6743WUWRV8S5nrrJWzEfWMWILG0tfsRv49ETXGv5YSa2sdFqsG KlnZ+XmMxEAjHREuvtMd92N4HPtgnrs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-434-1jG8JoGMMZqLc75xYcIWrQ-1; Thu, 18 May 2023 09:08:32 -0400 X-MC-Unique: 1jG8JoGMMZqLc75xYcIWrQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2CE1384AF32; Thu, 18 May 2023 13:08:31 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 09D6E1121314; Thu, 18 May 2023 13:08:27 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Kuniyuki Iwashima Subject: [PATCH net-next v9 16/16] unix: Convert udp_sendpage() to use MSG_SPLICE_PAGES Date: Thu, 18 May 2023 14:07:13 +0100 Message-Id: <20230518130713.1515729-17-dhowells@redhat.com> In-Reply-To: <20230518130713.1515729-1-dhowells@redhat.com> References: <20230518130713.1515729-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766238947368892580?= X-GMAIL-MSGID: =?utf-8?q?1766238947368892580?= Convert unix_stream_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Kuniyuki Iwashima cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/unix/af_unix.c | 134 +++------------------------------------------ 1 file changed, 7 insertions(+), 127 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index 976bc1c5e11b..115436ce1f8a 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -1839,24 +1839,6 @@ static void maybe_add_creds(struct sk_buff *skb, const struct socket *sock, } } -static int maybe_init_creds(struct scm_cookie *scm, - struct socket *socket, - const struct sock *other) -{ - int err; - struct msghdr msg = { .msg_controllen = 0 }; - - err = scm_send(socket, &msg, scm, false); - if (err) - return err; - - if (unix_passcred_enabled(socket, other)) { - scm->pid = get_pid(task_tgid(current)); - current_uid_gid(&scm->creds.uid, &scm->creds.gid); - } - return err; -} - static bool unix_skb_scm_eq(struct sk_buff *skb, struct scm_cookie *scm) { @@ -2292,117 +2274,15 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, static ssize_t unix_stream_sendpage(struct socket *socket, struct page *page, int offset, size_t size, int flags) { - int err; - bool send_sigpipe = false; - bool init_scm = true; - struct scm_cookie scm; - struct sock *other, *sk = socket->sk; - struct sk_buff *skb, *newskb = NULL, *tail = NULL; - - if (flags & MSG_OOB) - return -EOPNOTSUPP; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES }; - other = unix_peer(sk); - if (!other || sk->sk_state != TCP_ESTABLISHED) - return -ENOTCONN; - - if (false) { -alloc_skb: - unix_state_unlock(other); - mutex_unlock(&unix_sk(other)->iolock); - newskb = sock_alloc_send_pskb(sk, 0, 0, flags & MSG_DONTWAIT, - &err, 0); - if (!newskb) - goto err; - } - - /* we must acquire iolock as we modify already present - * skbs in the sk_receive_queue and mess with skb->len - */ - err = mutex_lock_interruptible(&unix_sk(other)->iolock); - if (err) { - err = flags & MSG_DONTWAIT ? -EAGAIN : -ERESTARTSYS; - goto err; - } - - if (sk->sk_shutdown & SEND_SHUTDOWN) { - err = -EPIPE; - send_sigpipe = true; - goto err_unlock; - } - - unix_state_lock(other); + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - if (sock_flag(other, SOCK_DEAD) || - other->sk_shutdown & RCV_SHUTDOWN) { - err = -EPIPE; - send_sigpipe = true; - goto err_state_unlock; - } - - if (init_scm) { - err = maybe_init_creds(&scm, socket, other); - if (err) - goto err_state_unlock; - init_scm = false; - } - - skb = skb_peek_tail(&other->sk_receive_queue); - if (tail && tail == skb) { - skb = newskb; - } else if (!skb || !unix_skb_scm_eq(skb, &scm)) { - if (newskb) { - skb = newskb; - } else { - tail = skb; - goto alloc_skb; - } - } else if (newskb) { - /* this is fast path, we don't necessarily need to - * call to kfree_skb even though with newskb == NULL - * this - does no harm - */ - consume_skb(newskb); - newskb = NULL; - } - - if (skb_append_pagefrags(skb, page, offset, size, MAX_SKB_FRAGS)) { - tail = skb; - goto alloc_skb; - } - - skb->len += size; - skb->data_len += size; - skb->truesize += size; - refcount_add(size, &sk->sk_wmem_alloc); - - if (newskb) { - err = unix_scm_to_skb(&scm, skb, false); - if (err) - goto err_state_unlock; - spin_lock(&other->sk_receive_queue.lock); - __skb_queue_tail(&other->sk_receive_queue, newskb); - spin_unlock(&other->sk_receive_queue.lock); - } - - unix_state_unlock(other); - mutex_unlock(&unix_sk(other)->iolock); - - other->sk_data_ready(other); - scm_destroy(&scm); - return size; - -err_state_unlock: - unix_state_unlock(other); -err_unlock: - mutex_unlock(&unix_sk(other)->iolock); -err: - kfree_skb(newskb); - if (send_sigpipe && !(flags & MSG_NOSIGNAL)) - send_sig(SIGPIPE, current, 0); - if (!init_scm) - scm_destroy(&scm); - return err; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return unix_stream_sendmsg(socket, &msg, size); } static int unix_seqpacket_sendmsg(struct socket *sock, struct msghdr *msg,