From patchwork Mon May 15 09:33:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 94015 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp6789841vqo; Mon, 15 May 2023 02:42:22 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6IX0wo9ZcbbyzMvLpwDJdMLpKsefdaqozaKO7PKaaCeFD462AaR4p9rhp7sti4xmYziPzj X-Received: by 2002:a05:6a21:6d8a:b0:104:8045:c966 with SMTP id wl10-20020a056a216d8a00b001048045c966mr13147577pzb.36.1684143742119; Mon, 15 May 2023 02:42:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684143742; cv=none; d=google.com; s=arc-20160816; b=s1EBDJQjp0YLHNPmNv/AuybInf+FNeLmOyuyuXhHpWiW3bf2C1Uyk5eDAj1NDJa7hd W9cXfvzMgw+iZAUo/uvbWwZ4HaRlXK0nr4pGNqKRoy/la2DjVRwSF1GvS1TTUAFWzF77 nN/XtKcd3juhXtY6D7agop/pwGC5NClkUFcKQ4TCWzNnkThr2FJdkBs5uc9CWmKFcYeo pT/hzrYeraVhQMXHKCuLiBmBxO+vYw8/ms/VhkqCAe8IPjc+64h00YrJmMig4uyyiXMg f4/G+WP1biRT6v+UmCDSaRts+/bAjtgv+oHEXP404MEwHPgi05HGCQGd8uU1DzzuhVna TmrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=b0te0jKR5ygwGnEzoKAl4ELcEmhdRmxdQq46C0lqTV4=; b=Tz+DCjSq5LIcGxjYA1VkUvqGdB8/Un8SKGAU0RGoHlCkdctiLVKJ+7+XftrW3HZB9a crDJIeMnyb7FTb4WhoiPZ0J2rQecAg1psrknnzbkpPWqh8xEzwagP4VkxDrO9nN8Bs9L 5JEHoN7HW2tAe9hUvgR3XALBhSuP6AG5ucOKGNYXcT0FLedqIXWykh2p+IeKx67sC1ob Ax/yjVBYEN6oOFOSVSgAPZVT0wu/C92m2WdW224Ey312qXtIh9XksOLLBNVoyaw+F33m he6Gm25nbhTY9SVv3MKp5Z70zvLGrrhgu32LdVLrS+5ogfd2lZA1C97MU++YfVCai3qE lQ6A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=aaFuKOUL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v10-20020a63f20a000000b005340840c0c7si3122212pgh.476.2023.05.15.02.42.09; Mon, 15 May 2023 02:42:22 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=aaFuKOUL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238763AbjEOJfF (ORCPT + 99 others); Mon, 15 May 2023 05:35:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238325AbjEOJet (ORCPT ); Mon, 15 May 2023 05:34:49 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 74350A7 for ; Mon, 15 May 2023 02:34:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684143240; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=b0te0jKR5ygwGnEzoKAl4ELcEmhdRmxdQq46C0lqTV4=; b=aaFuKOUL5mHRLETW8pqUL6lW/y2P/BhCyE+2wO5xVJxh3FQLh8hKGrQA0el7Y46c45EkGi H99uq5rZIpQWn1gl9g/zxIOQylijrLNsM7RagqnWUISVnD+WlwpxI45Aidzz2J5upe/HAm g0sRGTilGoDnDe5jquFGPhaTobSXkTY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-497-0XilkyABPe2SO0dMlnANnw-1; Mon, 15 May 2023 05:33:57 -0400 X-MC-Unique: 0XilkyABPe2SO0dMlnANnw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 571E487082A; Mon, 15 May 2023 09:33:56 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4DF7E2026D16; Mon, 15 May 2023 09:33:53 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn Subject: [PATCH net-next v7 01/16] net: Declare MSG_SPLICE_PAGES internal sendmsg() flag Date: Mon, 15 May 2023 10:33:30 +0100 Message-Id: <20230515093345.396978-2-dhowells@redhat.com> In-Reply-To: <20230515093345.396978-1-dhowells@redhat.com> References: <20230515093345.396978-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765952708852649389?= X-GMAIL-MSGID: =?utf-8?q?1765952708852649389?= Declare MSG_SPLICE_PAGES, an internal sendmsg() flag, that hints to a network protocol that it should splice pages from the source iterator rather than copying the data if it can. This flag is added to a list that is cleared by sendmsg syscalls on entry. This is intended as a replacement for the ->sendpage() op, allowing a way to splice in several multipage folios in one go. Signed-off-by: David Howells Reviewed-by: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #7) - In ____sys_sendmsg(), clear internal flags before setting msg_flags. - Clear internal flags in uring io_send{,_zc}(). include/linux/socket.h | 3 +++ io_uring/net.c | 2 ++ net/socket.c | 2 ++ 3 files changed, 7 insertions(+) diff --git a/include/linux/socket.h b/include/linux/socket.h index 13c3a237b9c9..bd1cc3238851 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -327,6 +327,7 @@ struct ucred { */ #define MSG_ZEROCOPY 0x4000000 /* Use user data in kernel path */ +#define MSG_SPLICE_PAGES 0x8000000 /* Splice the pages from the iterator in sendmsg() */ #define MSG_FASTOPEN 0x20000000 /* Send data in TCP SYN */ #define MSG_CMSG_CLOEXEC 0x40000000 /* Set close_on_exec for file descriptor received through @@ -337,6 +338,8 @@ struct ucred { #define MSG_CMSG_COMPAT 0 /* We never have 32 bit fixups */ #endif +/* Flags to be cleared on entry by sendmsg and sendmmsg syscalls */ +#define MSG_INTERNAL_SENDMSG_FLAGS (MSG_SPLICE_PAGES) /* Setsockoptions(2) level. Thanks to BSD these must match IPPROTO_xxx */ #define SOL_IP 0 diff --git a/io_uring/net.c b/io_uring/net.c index 89e839013837..f7cbb3c7a575 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -389,6 +389,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) if (flags & MSG_WAITALL) min_ret = iov_iter_count(&msg.msg_iter); + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; msg.msg_flags = flags; ret = sock_sendmsg(sock, &msg); if (ret < min_ret) { @@ -1136,6 +1137,7 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) msg_flags |= MSG_DONTWAIT; if (msg_flags & MSG_WAITALL) min_ret = iov_iter_count(&msg.msg_iter); + msg_flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; msg.msg_flags = msg_flags; msg.msg_ubuf = &io_notif_to_data(zc->notif)->uarg; diff --git a/net/socket.c b/net/socket.c index b7e01d0fe082..3df96e9ba4e2 100644 --- a/net/socket.c +++ b/net/socket.c @@ -2138,6 +2138,7 @@ int __sys_sendto(int fd, void __user *buff, size_t len, unsigned int flags, msg.msg_name = (struct sockaddr *)&address; msg.msg_namelen = addr_len; } + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; if (sock->file->f_flags & O_NONBLOCK) flags |= MSG_DONTWAIT; msg.msg_flags = flags; @@ -2483,6 +2484,7 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys, msg_sys->msg_control = ctl_buf; msg_sys->msg_control_is_user = false; } + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; msg_sys->msg_flags = flags; if (sock->file->f_flags & O_NONBLOCK) From patchwork Mon May 15 09:33:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 94035 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp6791773vqo; Mon, 15 May 2023 02:47:08 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5NItScRlMkS5wPHVpIlNmIfz4c+po/+GVGQCGAdHYXdVsnfmm2Amh0hlVl6mfn2Y6tvZwg X-Received: by 2002:a17:902:bf4a:b0:19f:3797:d8de with SMTP id u10-20020a170902bf4a00b0019f3797d8demr34147428pls.9.1684144028368; Mon, 15 May 2023 02:47:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684144028; cv=none; d=google.com; s=arc-20160816; b=O6+trlYoVWF9tQqfu9XjqBhRfpt723nrecQjEBFmUe7mycjZcMiibSQRTTkSK2c4ks cZms1MuLzS5uGkNaDLhl7etdU1wHCoIpwcyzvCPBcY/XmCVPw9kqz+wnkyKo66fwkV0G roQNQW6VjaiA5F6gJcgr/1U1cQe5WHs4rJPqIP3nOxE4VDLPbKkcjsiYTXq1GIayMreX oPPqUIknBIc2Psef9KvF6KSKUuI/AFos72bdpbWAcY9gaoyEHYCnXkRA3Kf8hSETibLM QmKyUvvBiK+HIMDST8tvtQmNHuVkCCkpl7MDS1yVoPH0nGcG3ojmjdUkt/iz/fpBv+Rk 8C2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Oms7mbQShTpY01zvjwYgDwHWlSZDYU/BDFTf+CWwguM=; b=J5Y21VQxZUBPHn31mm9Xw1b58YB6xmwcyKOrTuzqymLCcP//hru4wsaAA/lFyGtHj4 dhOUkZ9YKIM9cxIAjsYEj0CIQncqTMMOIR44bom+LzavyeKPWHsOBjDj5fk0g8RcFtcY sI7NreWFwisR1LgM0Xm7YFef2qXWVy9cENt1ApBqStE/zoUHL544PyU9BUKb51tL+OK7 P0KO2BKGXO68Y5zEk+eKL9ntgiAgbQEmI9iRRKuIUwO5HF9e+O/YbPwW/Oao+O1MagEh lm4OibVD/LeVGm06oiZCkK0VVvuBUVBI1za0iHVZETpOsmQYrjHd6J1edu7+EqMf/3wM 8Uvg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=MBZrvF71; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ij24-20020a170902ab5800b001ab21df595esi15432461plb.274.2023.05.15.02.46.56; Mon, 15 May 2023 02:47:08 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=MBZrvF71; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232730AbjEOJfR (ORCPT + 99 others); Mon, 15 May 2023 05:35:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39524 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233814AbjEOJe5 (ORCPT ); Mon, 15 May 2023 05:34:57 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 89F6AE69 for ; Mon, 15 May 2023 02:34:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684143246; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Oms7mbQShTpY01zvjwYgDwHWlSZDYU/BDFTf+CWwguM=; b=MBZrvF71aEDWbry85MpSUSI8afO85N357wnAuFKz4GAmDmh1gRLN9l5HGWg2iyDSpWdJ9h ZQ0NnPtPakMck4KFCobd0tpV08eyF4oUw3//pL313hBcI8r4IzBs3Ygo3rmsWw+rosu2/n oyJalDh0f4Y+GAgOzpLgNdEhs9tqYkY= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-445-AkM32TIKNTGBQP3iQjlccw-1; Mon, 15 May 2023 05:34:00 -0400 X-MC-Unique: AkM32TIKNTGBQP3iQjlccw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B58403C0CF00; Mon, 15 May 2023 09:33:59 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 08F5040C206F; Mon, 15 May 2023 09:33:56 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v7 02/16] net: Pass max frags into skb_append_pagefrags() Date: Mon, 15 May 2023 10:33:31 +0100 Message-Id: <20230515093345.396978-3-dhowells@redhat.com> In-Reply-To: <20230515093345.396978-1-dhowells@redhat.com> References: <20230515093345.396978-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765953008927257715?= X-GMAIL-MSGID: =?utf-8?q?1765953008927257715?= Pass the maximum number of fragments into skb_append_pagefrags() rather than using MAX_SKB_FRAGS so that it can be used from code that wants to specify sysctl_max_skb_frags. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/linux/skbuff.h | 2 +- net/core/skbuff.c | 4 ++-- net/ipv4/ip_output.c | 3 ++- net/unix/af_unix.c | 2 +- 4 files changed, 6 insertions(+), 5 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 00e8c435fa1a..4c0ad48e38ca 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -1383,7 +1383,7 @@ static inline int skb_pad(struct sk_buff *skb, int pad) #define dev_kfree_skb(a) consume_skb(a) int skb_append_pagefrags(struct sk_buff *skb, struct page *page, - int offset, size_t size); + int offset, size_t size, size_t max_frags); struct skb_seq_state { __u32 lower_offset; diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 6724a84ebb09..7f53dcb26ad3 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -4188,13 +4188,13 @@ unsigned int skb_find_text(struct sk_buff *skb, unsigned int from, EXPORT_SYMBOL(skb_find_text); int skb_append_pagefrags(struct sk_buff *skb, struct page *page, - int offset, size_t size) + int offset, size_t size, size_t max_frags) { int i = skb_shinfo(skb)->nr_frags; if (skb_can_coalesce(skb, i, page, offset)) { skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], size); - } else if (i < MAX_SKB_FRAGS) { + } else if (i < max_frags) { skb_zcopy_downgrade_managed(skb); get_page(page); skb_fill_page_desc_noacc(skb, i, page, offset, size); diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 61892268e8a6..52fc840898d8 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1450,7 +1450,8 @@ ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, if (len > size) len = size; - if (skb_append_pagefrags(skb, page, offset, len)) { + if (skb_append_pagefrags(skb, page, offset, len, + MAX_SKB_FRAGS)) { err = -EMSGSIZE; goto error; } diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index cc695c9f09ec..dd55506b4632 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2349,7 +2349,7 @@ static ssize_t unix_stream_sendpage(struct socket *socket, struct page *page, newskb = NULL; } - if (skb_append_pagefrags(skb, page, offset, size)) { + if (skb_append_pagefrags(skb, page, offset, size, MAX_SKB_FRAGS)) { tail = skb; goto alloc_skb; } From patchwork Mon May 15 09:33:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 94036 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp6791783vqo; Mon, 15 May 2023 02:47:10 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6UUHTM0bW89QNtWiS8lWv0pJgzv0f5btbO7wIl2JFEqRH5Tgk3ySEN6VY9GvjqJimGKzXc X-Received: by 2002:a05:6a00:15c6:b0:635:d9e1:8e1e with SMTP id o6-20020a056a0015c600b00635d9e18e1emr48126808pfu.11.1684144030428; Mon, 15 May 2023 02:47:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684144030; cv=none; d=google.com; s=arc-20160816; b=nApbJWyqi90xx5LmOygZOC8oeOm8GqdIUfIHvtN4FbVvBAeMjJmq4Nka9oS+AerD92 PwMjuPs4xc764ck6sdfqN/1Lxdm+shGkawkFjVNLaPRjN6kSRGk3A/bI5VI84ZWImctJ SQiLb4t1lNclw4VCJoTFGzAMcjCjRvLzD9BlytbJshJLo7thsFAd8Y3zdR4Wy/3oVeXK WwwDl2T9BCqg17NvxvvT/nsWRvM3kp2ZBW90ZCEO/h9ujfZC9LasvL08DZqptI91yXCk XQ21x20Zb5q72NKuWlFCPCXCBiiwHs/vxLKfhmB4rI8QxlgHEZ3Tz7/3Hhhf491PCzCE LavA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=rQETt65RGaUyqHGF2SRFOY3ON9kc871DZcLjRArLuXs=; b=YXhz2OYv2pAOSWwp7re1V6cp7VfkNelDpDADQfveA9JbaNFC1PblsHYaOjsF7rSaDI HZQpiredPETyK+WDA+Sl5OMQgKJlw61GUW/Ks2IOMkaFJUGQ0Z9zU2mjblCMcoSTnGh3 nAvkkyYjWjgz7MmaMQhvKjqEvAC9eDpvPZRBCd1S5et18o7qWpd7rvCGU2Jg31G4x22V 4TSUTlrA+jPLtSZv3Y3ObbheYl+arUR/U95BaGj90ntbx3kP9uUleZZwEaxZgaS3CnQz myGDgKUHUjnLo8KhZialFYenb8zkpqc+gaAsHocCmmSR3dqetf5NtTDU/vtPobP7O6dS VyIw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=eTfvkF9c; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x27-20020aa78f1b000000b0064697515ef6si16413892pfr.361.2023.05.15.02.46.58; Mon, 15 May 2023 02:47:10 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=eTfvkF9c; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239215AbjEOJfO (ORCPT + 99 others); Mon, 15 May 2023 05:35:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39526 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239119AbjEOJe4 (ORCPT ); Mon, 15 May 2023 05:34:56 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3606DF for ; Mon, 15 May 2023 02:34:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684143249; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rQETt65RGaUyqHGF2SRFOY3ON9kc871DZcLjRArLuXs=; b=eTfvkF9cEDJ9lcmE1hxeOwN6Dgf/Rex+/TTXbT9Yovjnp/VqyE/OdHpT/nL5D1YNJec3KY tfmGU4PNzXIAfNMZ1txv4T+cSr7Qg7fS4yPiTtfOg5S20w7AU4TW22r+KxjdlFNtdf0tlD K0BYzVu7Zpwj2jdPt07X3K/9xuShQ/g= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-516-ea5lFs8TOwOYnV30iV30Iw-1; Mon, 15 May 2023 05:34:05 -0400 X-MC-Unique: ea5lFs8TOwOYnV30iV30Iw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B2060185A79C; Mon, 15 May 2023 09:34:04 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1B4DF63F84; Mon, 15 May 2023 09:34:00 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v7 03/16] net: Add a function to splice pages into an skbuff for MSG_SPLICE_PAGES Date: Mon, 15 May 2023 10:33:32 +0100 Message-Id: <20230515093345.396978-4-dhowells@redhat.com> In-Reply-To: <20230515093345.396978-1-dhowells@redhat.com> References: <20230515093345.396978-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765953010761569549?= X-GMAIL-MSGID: =?utf-8?q?1765953010761569549?= Add a function to handle MSG_SPLICE_PAGES being passed internally to sendmsg(). Pages are spliced into the given socket buffer if possible and copied in if not (e.g. they're slab pages or have a zero refcount). Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Al Viro cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #7) - Export function. - Never copy data, return -EIO if sendpage_ok() returns false. include/linux/skbuff.h | 3 ++ net/core/skbuff.c | 95 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 98 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 4c0ad48e38ca..1c5f0ac6f8c3 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -5097,5 +5097,8 @@ static inline void skb_mark_for_recycle(struct sk_buff *skb) #endif } +ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, + ssize_t maxsize, gfp_t gfp); + #endif /* __KERNEL__ */ #endif /* _LINUX_SKBUFF_H */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 7f53dcb26ad3..56d629ea2f3d 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -6892,3 +6892,98 @@ nodefer: __kfree_skb(skb); if (unlikely(kick) && !cmpxchg(&sd->defer_ipi_scheduled, 0, 1)) smp_call_function_single_async(cpu, &sd->defer_csd); } + +static void skb_splice_csum_page(struct sk_buff *skb, struct page *page, + size_t offset, size_t len) +{ + const char *kaddr; + __wsum csum; + + kaddr = kmap_local_page(page); + csum = csum_partial(kaddr + offset, len, 0); + kunmap_local(kaddr); + skb->csum = csum_block_add(skb->csum, csum, skb->len); +} + +/** + * skb_splice_from_iter - Splice (or copy) pages to skbuff + * @skb: The buffer to add pages to + * @iter: Iterator representing the pages to be added + * @maxsize: Maximum amount of pages to be added + * @gfp: Allocation flags + * + * This is a common helper function for supporting MSG_SPLICE_PAGES. It + * extracts pages from an iterator and adds them to the socket buffer if + * possible, copying them to fragments if not possible (such as if they're slab + * pages). + * + * Returns the amount of data spliced/copied or -EMSGSIZE if there's + * insufficient space in the buffer to transfer anything. + */ +ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, + ssize_t maxsize, gfp_t gfp) +{ + struct page *pages[8], **ppages = pages; + unsigned int i; + ssize_t spliced = 0, ret = 0; + size_t frag_limit = READ_ONCE(sysctl_max_skb_frags); + + while (iter->count > 0) { + ssize_t space, nr; + size_t off, len; + + ret = -EMSGSIZE; + space = frag_limit - skb_shinfo(skb)->nr_frags; + if (space < 0) + break; + + /* We might be able to coalesce without increasing nr_frags */ + nr = clamp_t(size_t, space, 1, ARRAY_SIZE(pages)); + + len = iov_iter_extract_pages(iter, &ppages, maxsize, nr, 0, &off); + if (len <= 0) { + ret = len ?: -EIO; + break; + } + + if (space == 0 && + !skb_can_coalesce(skb, skb_shinfo(skb)->nr_frags, + pages[0], off)) { + iov_iter_revert(iter, len); + break; + } + + i = 0; + do { + struct page *page = pages[i++]; + size_t part = min_t(size_t, PAGE_SIZE - off, len); + + ret = -EIO; + if (!sendpage_ok(page)) + goto out; + + ret = skb_append_pagefrags(skb, page, off, part, + frag_limit); + if (ret < 0) { + iov_iter_revert(iter, len); + goto out; + } + + if (skb->ip_summed == CHECKSUM_NONE) + skb_splice_csum_page(skb, page, off, part); + + off = 0; + spliced += part; + maxsize -= part; + len -= part; + } while (len > 0); + + if (maxsize <= 0) + break; + } + +out: + skb_len_add(skb, spliced); + return spliced ?: ret; +} +EXPORT_SYMBOL(skb_splice_from_iter); From patchwork Mon May 15 09:33:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 94025 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp6791134vqo; Mon, 15 May 2023 02:45:40 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7qY7A9v+LWAo6rT+Trhi0masc9SLde+Yp17BPPCc5k86SgX/0MgM62Zxnaah1ou4NGhVx7 X-Received: by 2002:a05:6a20:7488:b0:106:47d1:63bc with SMTP id p8-20020a056a20748800b0010647d163bcmr2121475pzd.36.1684143940408; Mon, 15 May 2023 02:45:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684143940; cv=none; d=google.com; s=arc-20160816; b=XMbp0HDcJRP6yGkCJAJj5yq6oocZkWEZpQ3D3EYW0poHSdYmfbh6MBrXTJbJQwvNqg tWl6Czk8aqgz1TxlRk5n998ugYsPccp94S3q0oov4apPKFGpNi2RyMB3NhyxpPM+lX4B 5JQJR4nOQoq/8ALhxiw0kJaIyYbPe2dAm4nN/g8igDM9taD1GnGICw+BpEw5U5F+xYnP sfWwT4CB89WpFk0+P+PikxKLsCKp2wqXla933KPXEDMR1tgXR1cAQ9M0XpSLQINfyqeX NMF4g68N7OsuLo/Fx6aiMoiWkizICmUoHjJzljXfIXhjXLC/rwtlzvPY77e4sCybmLu7 ZRCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=DZXfiar5CKohUwsPymjzwnCAT+wFqXG5Kam/6y6M8pg=; b=0lEDJQ5R8FmVVPzA9BzQjLAfXk5WcMy79eMTB1NcejpGN2PBhpkgqa/r2ZDFPxGKK1 Dwg09Lus3TymVJCD6qbCqBMbvznfFlzlo2GJ3MKh4HS8Kqc9K/O4LQ9ei2jinenypq71 Sn+cNCzEBqCPLruAGOTY+b+hj6b+BYkjSmYMP84gcPIjM3BYzllBs466KZ8fXPCiZcqE l66sw/kr9OuBm2eyYBgjWUcEuF+dMTaGvo9HmAp1tW/RlQdfblR8eBQS9l7dm4QcjfQN LE8qy0yULJLD12+NxsRmhF/RD4bBeMJhLNScbMs7ZdTdZbu0sdC4X5XoI58d2x4Ge+/l 8N7Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=CQ6BNhs3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p22-20020a639516000000b005302f7ec8c4si16081729pgd.845.2023.05.15.02.45.28; Mon, 15 May 2023 02:45:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=CQ6BNhs3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238623AbjEOJfb (ORCPT + 99 others); Mon, 15 May 2023 05:35:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238384AbjEOJfB (ORCPT ); Mon, 15 May 2023 05:35:01 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12AE41AD for ; Mon, 15 May 2023 02:34:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684143252; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DZXfiar5CKohUwsPymjzwnCAT+wFqXG5Kam/6y6M8pg=; b=CQ6BNhs3fUU9eJ2xniiys9NVsyn7wyrgk+6p8F7aCsBG8JaKRinrgfqMXEM1e9BJM1TShG q6v+qeGyEApVCxqlYaPxUuomc9hPd0eeIg5Wl+UgC3LmnWXR4IL3gsuDczurMvrH5t4yf+ 20E1DG0jGzwRa6dO3W81he2BKEAkS+U= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-365-QGYT3avTOnGhGIbiGpk5qg-1; Mon, 15 May 2023 05:34:09 -0400 X-MC-Unique: QGYT3avTOnGhGIbiGpk5qg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 360B3811E7C; Mon, 15 May 2023 09:34:08 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8F615C15BA0; Mon, 15 May 2023 09:34:05 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v7 04/16] tcp: Support MSG_SPLICE_PAGES Date: Mon, 15 May 2023 10:33:33 +0100 Message-Id: <20230515093345.396978-5-dhowells@redhat.com> In-Reply-To: <20230515093345.396978-1-dhowells@redhat.com> References: <20230515093345.396978-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765952916323730923?= X-GMAIL-MSGID: =?utf-8?q?1765952916323730923?= Make TCP's sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced or copied (if it cannot be spliced) from the source iterator. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #7) - Missed a "zc = 1" in tcp_sendmsg_locked(). ver #6) - Set zc to 0/MSG_ZEROCOPY/MSG_SPLICE_PAGES rather than 0/1/2. - Use common helper. net/ipv4/tcp.c | 43 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 4d6392c16b7a..026f483f42e3 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1223,7 +1223,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) int flags, err, copied = 0; int mss_now = 0, size_goal, copied_syn = 0; int process_backlog = 0; - bool zc = false; + int zc = 0; long timeo; flags = msg->msg_flags; @@ -1234,17 +1234,22 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (msg->msg_ubuf) { uarg = msg->msg_ubuf; net_zcopy_get(uarg); - zc = sk->sk_route_caps & NETIF_F_SG; + if (sk->sk_route_caps & NETIF_F_SG) + zc = MSG_ZEROCOPY; } else if (sock_flag(sk, SOCK_ZEROCOPY)) { uarg = msg_zerocopy_realloc(sk, size, skb_zcopy(skb)); if (!uarg) { err = -ENOBUFS; goto out_err; } - zc = sk->sk_route_caps & NETIF_F_SG; - if (!zc) + if (sk->sk_route_caps & NETIF_F_SG) + zc = MSG_ZEROCOPY; + else uarg_to_msgzc(uarg)->zerocopy = 0; } + } else if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES) && size) { + if (sk->sk_route_caps & NETIF_F_SG) + zc = MSG_SPLICE_PAGES; } if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect) && @@ -1307,7 +1312,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) goto do_error; while (msg_data_left(msg)) { - int copy = 0; + ssize_t copy = 0; skb = tcp_write_queue_tail(sk); if (skb) @@ -1348,7 +1353,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (copy > msg_data_left(msg)) copy = msg_data_left(msg); - if (!zc) { + if (zc == 0) { bool merge = true; int i = skb_shinfo(skb)->nr_frags; struct page_frag *pfrag = sk_page_frag(sk); @@ -1393,7 +1398,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) page_ref_inc(pfrag->page); } pfrag->offset += copy; - } else { + } else if (zc == MSG_ZEROCOPY) { /* First append to a fragless skb builds initial * pure zerocopy skb */ @@ -1414,6 +1419,30 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (err < 0) goto do_error; copy = err; + } else if (zc == MSG_SPLICE_PAGES) { + /* Splice in data if we can; copy if we can't. */ + if (tcp_downgrade_zcopy_pure(sk, skb)) + goto wait_for_space; + copy = tcp_wmem_schedule(sk, copy); + if (!copy) + goto wait_for_space; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) { + if (err == -EMSGSIZE) { + tcp_mark_push(tp, skb); + goto new_segment; + } + goto do_error; + } + copy = err; + + if (!(flags & MSG_NO_SHARED_FRAGS)) + skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; + + sk_wmem_queued_add(sk, copy); + sk_mem_charge(sk, copy); } if (!copied) From patchwork Mon May 15 09:33:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 94019 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp6790716vqo; Mon, 15 May 2023 02:44:39 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5Ev4D4mBVUxlYGBUNsjx5w35hSCORsLzb1wUHyMEKS3kStd4445SVIV5mXs5CHiVAYbyDn X-Received: by 2002:a05:6a20:1443:b0:101:a459:bb0d with SMTP id a3-20020a056a20144300b00101a459bb0dmr27087316pzi.2.1684143879310; Mon, 15 May 2023 02:44:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684143879; cv=none; d=google.com; s=arc-20160816; b=jAwhFfwyz1ok/FSgxkArjazedsGl8AENjc9t7Rori6b9U4Q+YKh+mky/gxvxzEjrFR 0118sl0L/vJbEUGnC6iQmqUTVSEAoItEcXFJ0AAmYTRwMSIcAGlH0gxiSBYOdP/BqWJK RWuht9WuFQh9X0LCL1zMPXJEqln/2Q9G9bcK2gB7dIiH254reL0lFSFO5A/qjnReWGGd IVGwuOHCufMqZQ5S2TqPckDyqS+HXbK0B6XpNYv2YimUVXUIYiy2U/xrgc/83PlAl8iO NxwPvsC/CINHpEoN53haWCB8281YG8/Nis114Kl0GcV8QfxrPulgYGO/NWC9CZkct73Q qdfg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9s33rFBaxABfh3fue+YSREMQMkxGRapWweS5v7j7klU=; b=MAXGip9xgwU3PpLQXab5RorClzTWyqwpLv5mYzyDEri1LRWkTI4w8YaDa+b+FjuVDv FRmcyV+uaSIaQyroA0fmLq3WWBJKw0vg+A4w0Q02QI1flA+yDb6L11NMNl/JMvjlrZsv K/BOUdc457/TEl393tkj+HIpYIMS/Eut8hR9rUa6S+QEPoxqRAcfQkkusGFPjMqFc3c9 xYv2JxlWVLBEUPV+7LyO5WhvNzl1oAmn1CnNpdBlj/RrIOhYBZZachAAJQrKWxIBz7wK Ve1Z8Oqe/jtw8aHfTSFNuv1zgPmYGYzpcVPVUEJDgKliUR1W8lSYb6l4C+ODWCkylGl4 YxLw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JCL1HuSA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u184-20020a6385c1000000b005303b490638si17009942pgd.466.2023.05.15.02.44.26; Mon, 15 May 2023 02:44:39 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JCL1HuSA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230080AbjEOJfh (ORCPT + 99 others); Mon, 15 May 2023 05:35:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39658 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238471AbjEOJfD (ORCPT ); Mon, 15 May 2023 05:35:03 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5D594E68 for ; Mon, 15 May 2023 02:34:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684143255; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9s33rFBaxABfh3fue+YSREMQMkxGRapWweS5v7j7klU=; b=JCL1HuSAyvpgl/H3k+tiCxzmuyRmiw+LbZgRDLC3yUm5ydyARIvNJGAS8UrOOIc+GhXUkC lTdc2Yahk2ta637cK2j/ofwhP/Ktf9IB29gmEnFZjr5uQPIn1omuVWx4U5jnKiRD7omYkB moJJlJo7nFsrDVqiw50ptg0d8TpQxPs= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-638-ctaAd9Y3OV6hQ5fXV43Iug-1; Mon, 15 May 2023 05:34:12 -0400 X-MC-Unique: ctaAd9Y3OV6hQ5fXV43Iug-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 60F4338149AE; Mon, 15 May 2023 09:34:11 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id E0534C15BA0; Mon, 15 May 2023 09:34:08 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v7 05/16] tcp: Convert do_tcp_sendpages() to use MSG_SPLICE_PAGES Date: Mon, 15 May 2023 10:33:34 +0100 Message-Id: <20230515093345.396978-6-dhowells@redhat.com> In-Reply-To: <20230515093345.396978-1-dhowells@redhat.com> References: <20230515093345.396978-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765952852598629811?= X-GMAIL-MSGID: =?utf-8?q?1765952852598629811?= Convert do_tcp_sendpages() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. do_tcp_sendpages() can then be inlined in subsequent patches into its callers. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 158 +++---------------------------------------------- 1 file changed, 7 insertions(+), 151 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 026f483f42e3..28231e503af9 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -974,163 +974,19 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -static struct sk_buff *tcp_build_frag(struct sock *sk, int size_goal, int flags, - struct page *page, int offset, size_t *size) -{ - struct sk_buff *skb = tcp_write_queue_tail(sk); - struct tcp_sock *tp = tcp_sk(sk); - bool can_coalesce; - int copy, i; - - if (!skb || (copy = size_goal - skb->len) <= 0 || - !tcp_skb_can_collapse_to(skb)) { -new_segment: - if (!sk_stream_memory_free(sk)) - return NULL; - - skb = tcp_stream_alloc_skb(sk, 0, sk->sk_allocation, - tcp_rtx_and_write_queues_empty(sk)); - if (!skb) - return NULL; - -#ifdef CONFIG_TLS_DEVICE - skb->decrypted = !!(flags & MSG_SENDPAGE_DECRYPTED); -#endif - tcp_skb_entail(sk, skb); - copy = size_goal; - } - - if (copy > *size) - copy = *size; - - i = skb_shinfo(skb)->nr_frags; - can_coalesce = skb_can_coalesce(skb, i, page, offset); - if (!can_coalesce && i >= READ_ONCE(sysctl_max_skb_frags)) { - tcp_mark_push(tp, skb); - goto new_segment; - } - if (tcp_downgrade_zcopy_pure(sk, skb)) - return NULL; - - copy = tcp_wmem_schedule(sk, copy); - if (!copy) - return NULL; - - if (can_coalesce) { - skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); - } else { - get_page(page); - skb_fill_page_desc_noacc(skb, i, page, offset, copy); - } - - if (!(flags & MSG_NO_SHARED_FRAGS)) - skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; - - skb->len += copy; - skb->data_len += copy; - skb->truesize += copy; - sk_wmem_queued_add(sk, copy); - sk_mem_charge(sk, copy); - WRITE_ONCE(tp->write_seq, tp->write_seq + copy); - TCP_SKB_CB(skb)->end_seq += copy; - tcp_skb_pcount_set(skb, 0); - - *size = copy; - return skb; -} - ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tcp_sock *tp = tcp_sk(sk); - int mss_now, size_goal; - int err; - ssize_t copied; - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); - - if (IS_ENABLED(CONFIG_DEBUG_VM) && - WARN_ONCE(!sendpage_ok(page), - "page must not be a Slab one and have page_count > 0")) - return -EINVAL; - - /* Wait for a connection to finish. One exception is TCP Fast Open - * (passive side) where data is allowed to be sent before a connection - * is fully established. - */ - if (((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) && - !tcp_passive_fastopen(sk)) { - err = sk_stream_wait_connect(sk, &timeo); - if (err != 0) - goto out_err; - } + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - mss_now = tcp_send_mss(sk, &size_goal, flags); - copied = 0; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - err = -EPIPE; - if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) - goto out_err; - - while (size > 0) { - struct sk_buff *skb; - size_t copy = size; - - skb = tcp_build_frag(sk, size_goal, flags, page, offset, ©); - if (!skb) - goto wait_for_space; - - if (!copied) - TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; - - copied += copy; - offset += copy; - size -= copy; - if (!size) - goto out; - - if (skb->len < size_goal || (flags & MSG_OOB)) - continue; - - if (forced_push(tp)) { - tcp_mark_push(tp, skb); - __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); - } else if (skb == tcp_send_head(sk)) - tcp_push_one(sk, mss_now); - continue; - -wait_for_space: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); - tcp_push(sk, flags & ~MSG_MORE, mss_now, - TCP_NAGLE_PUSH, size_goal); - - err = sk_stream_wait_memory(sk, &timeo); - if (err != 0) - goto do_error; - - mss_now = tcp_send_mss(sk, &size_goal, flags); - } - -out: - if (copied) { - tcp_tx_timestamp(sk, sk->sk_tsflags); - if (!(flags & MSG_SENDPAGE_NOTLAST)) - tcp_push(sk, flags, mss_now, tp->nonagle, size_goal); - } - return copied; - -do_error: - tcp_remove_empty_skb(sk); - if (copied) - goto out; -out_err: - /* make sure we wake any epoll edge trigger waiter */ - if (unlikely(tcp_rtx_and_write_queues_empty(sk) && err == -EAGAIN)) { - sk->sk_write_space(sk); - tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED); - } - return sk_stream_error(sk, flags, err); + return tcp_sendmsg_locked(sk, &msg, size); } EXPORT_SYMBOL_GPL(do_tcp_sendpages); From patchwork Mon May 15 09:33:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 94011 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp6788564vqo; Mon, 15 May 2023 02:39:04 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6qgQwbDIY5c7v8tvftdIGkHGDUy5cIvqxaP2UaHBvid+AJBw7mk3RLBLd6BP0Mln/odXFv X-Received: by 2002:a17:902:d34b:b0:1aa:dd14:da98 with SMTP id l11-20020a170902d34b00b001aadd14da98mr34682983plk.50.1684143543689; Mon, 15 May 2023 02:39:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684143543; cv=none; d=google.com; s=arc-20160816; b=yBno4Jl4PJI3Yzg9Or7A2ypOHU1hYXY33T6KVTAjBQuQkOUMfI81X7UST+q++LfrFw sTapf1oO6z9tX7ZnxwBhvswRKnqRpOQcPRDCRGiYXsvGEOLyNRmBEQx3f0CSxY+2KGZS qJsN7Qk5qZtO9qO+C/RNIim/R6rqaOrQbF/wjemQw8AISwtoCHiiAyyNWKMn9HFUp+Dc vRVJWyA6DE98SnwZ/CBH3+nQrarqNI59Ufu9yHm2hCuXMgYnCVOqnkUY7XvWHfXbrSho fUDhQgjeoP3ZaeWVUe/NdJ7PdmH4bMHku+4yt2v2ilAGyzkHQZWMTpGYBZTrlcpnN/ob WS4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2EXLlBUI8HMrlq1LdzuePmnE9vo0jChMg019VHUJEgs=; b=IeFLUprF3qb2OcpVrYLh4XHF8PM4lbuiuhrWfCBQnZDn5jAWnwVjHVKXWIKltBko0+ kXqGe2mHRNMBF6iUhFz9BGI65kJqIx69tn7VPIs5nLLQARsVtbGnA6wQepfllrwE+ZeJ hs6YCTSeRwZ7QEBTcaVzdVW/n5s+Ae6nF+15w8ntA0yyhvpBoqJsgn8tLLZZL9gxKaxe oEo0yhSJPbHMsmjRH0fWxfqvTk4p44buNtGJNm0c0DreAqMzaLyf8KnB302QgX9RNpRd ll6xFgJaPcIKyeTzs+eMpCe0c3s4i58o0uc9SlfrnAWWWAetodVw2BCDTkzuOZXzHJTM a8uQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=SgEnjXer; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gg24-20020a17090b0a1800b0024df4dee56bsi1119334pjb.138.2023.05.15.02.38.51; Mon, 15 May 2023 02:39:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=SgEnjXer; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239969AbjEOJfn (ORCPT + 99 others); Mon, 15 May 2023 05:35:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39716 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238860AbjEOJfI (ORCPT ); Mon, 15 May 2023 05:35:08 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CAE0BE61 for ; Mon, 15 May 2023 02:34:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684143260; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2EXLlBUI8HMrlq1LdzuePmnE9vo0jChMg019VHUJEgs=; b=SgEnjXer37414Cpo9EFCtmPkWdLMxV828uu9nzDyzTn7/bTJheFk3n4lQKGgnBl+rdQXru 4L4XO2slDo4SHDtGRjDDm6mfekGxPTzo3DfVnmzoZ/v9HzVxvWlHrw+IZ6VFy1orb6Ss1Q tuNYCmk2v0vga2j9kH5GJ0SI7J6+1eM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-515-ekVa9ADbMTarb4EWYMOq1Q-1; Mon, 15 May 2023 05:34:16 -0400 X-MC-Unique: ekVa9ADbMTarb4EWYMOq1Q-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BF541185A790; Mon, 15 May 2023 09:34:15 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9E68B2026D16; Mon, 15 May 2023 09:34:12 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Fastabend , Jakub Sitnicki , bpf@vger.kernel.org Subject: [PATCH net-next v7 06/16] tcp_bpf: Inline do_tcp_sendpages as it's now a wrapper around tcp_sendmsg Date: Mon, 15 May 2023 10:33:35 +0100 Message-Id: <20230515093345.396978-7-dhowells@redhat.com> In-Reply-To: <20230515093345.396978-1-dhowells@redhat.com> References: <20230515093345.396978-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765952500240666286?= X-GMAIL-MSGID: =?utf-8?q?1765952500240666286?= do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: John Fastabend cc: Jakub Sitnicki cc: "David S. Miller" cc: Eric Dumazet cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: bpf@vger.kernel.org --- net/ipv4/tcp_bpf.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c index 2e9547467edb..0291d15acd19 100644 --- a/net/ipv4/tcp_bpf.c +++ b/net/ipv4/tcp_bpf.c @@ -72,11 +72,13 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, { bool apply = apply_bytes; struct scatterlist *sge; + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct page *page; int size, ret = 0; u32 off; while (1) { + struct bio_vec bvec; bool has_tx_ulp; sge = sk_msg_elem(msg, msg->sg.start); @@ -88,16 +90,18 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, tcp_rate_check_app_limited(sk); retry: has_tx_ulp = tls_sw_has_ctx_tx(sk); - if (has_tx_ulp) { - flags |= MSG_SENDPAGE_NOPOLICY; - ret = kernel_sendpage_locked(sk, - page, off, size, flags); - } else { - ret = do_tcp_sendpages(sk, page, off, size, flags); - } + if (has_tx_ulp) + msghdr.msg_flags |= MSG_SENDPAGE_NOPOLICY; + if (flags & MSG_SENDPAGE_NOTLAST) + msghdr.msg_flags |= MSG_MORE; + + bvec_set_page(&bvec, page, size, off); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret <= 0) return ret; + if (apply) apply_bytes -= ret; msg->sg.size -= ret; @@ -404,7 +408,7 @@ static int tcp_bpf_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) long timeo; int flags; - /* Don't let internal do_tcp_sendpages() flags through */ + /* Don't let internal sendpage flags through */ flags = (msg->msg_flags & ~MSG_SENDPAGE_DECRYPTED); flags |= MSG_NO_SHARED_FRAGS; From patchwork Mon May 15 09:33:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 94031 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp6791487vqo; Mon, 15 May 2023 02:46:32 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5hFEZjqQBYtuhk8aLtq6axrMtWsmBK8t+jaQDKdvqd+trfRWQncuhkh5T7Ymc8/nasqllF X-Received: by 2002:a05:6a00:168b:b0:63d:2d7d:b6f2 with SMTP id k11-20020a056a00168b00b0063d2d7db6f2mr43844529pfc.4.1684143991739; Mon, 15 May 2023 02:46:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684143991; cv=none; d=google.com; s=arc-20160816; b=xqfAYgTMcxN2i32hsu2jqzVlGpx15C32egRTmbs2clAa1e1WQOSppDwfO4r2OhbOwX tQRs0+9uho9LdE7Hg0jmRG6WouYgsus7SSrIOcgq+jiuwGN+l+uWVcORIKz232nX6AwV NAsb5EKo6g4Ef4Nme7IVx6LxeSxPwhEdznT67BIuMHfnwqzC6UrVPm9J2FhISF2bF9sh wNMw2UW2zo4RH+wuIrfv0nCadZ7N8mxgXjOgmDnsxDTK1bYJVRiUF3hmEku1QncjUMa7 Pe7sY1gYjyicSPcbRh4BYI/t/r3PX4uEulejqtfOcE5dsphPVRB+C42E9lrkf62pyEMK zvUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=tYeoTX9B8dad49qWPHePVLYzX6oyR1jBlVymL6Q5Zsc=; b=mD7cdcdrZNR9R0BvnOqqBYlRMT9LWM45XPLmdtD77shUk/wG/Ey1szI94/z1UKkI63 mubR4g+zMFT7bw0BQzFBzWLLdt+f5BgeVGggnoXgpndhqsMtp5s/eXKcIrCcazLNlXLH K37g0SMt/CcpTHVrEUmXXgJ2uQShvDGAKMm17+abA/E9at5Ebhuzq6apsOY9C2wGq3UW uOLo10hTzFl/SftEm4i3B2PW1pUOEVkHVEeDRV7W90G/lsNRqHif9wpfv7902uqHqT2M jhIQp0Y7dkYi+c5Jw+XmMR3h3m4MlJEC/QuAOD+DDLO5xIEuRiQFpRJ0nBKo5if4a6NQ nFqQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ebtoFHUE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l191-20020a6391c8000000b00524ecf898a2si14957842pge.359.2023.05.15.02.46.19; Mon, 15 May 2023 02:46:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ebtoFHUE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240313AbjEOJfx (ORCPT + 99 others); Mon, 15 May 2023 05:35:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238780AbjEOJfL (ORCPT ); Mon, 15 May 2023 05:35:11 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6591E51 for ; Mon, 15 May 2023 02:34:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684143265; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tYeoTX9B8dad49qWPHePVLYzX6oyR1jBlVymL6Q5Zsc=; b=ebtoFHUEUwWYqmdixBBOfQNOz1rh//p0kJZPIQVAGTYIGh62n2y6w1n8opm24/fe8zUyAT JZLmS7j9n9Au+qWzzjdbhtIkn43lGWzPDw1UJlmdEnEFEpqwI6Vb65un9Cn3ubxpiMz0Bd a+FfVJU/bjLoSqQibgrpW+h6jYhPn6Q= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-640-1nXxZzOvNO2kG1IIBR_aRA-1; Mon, 15 May 2023 05:34:20 -0400 X-MC-Unique: 1nXxZzOvNO2kG1IIBR_aRA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BFE803C0D858; Mon, 15 May 2023 09:34:19 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id B9CE063F3D; Mon, 15 May 2023 09:34:16 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Steffen Klassert , Herbert Xu Subject: [PATCH net-next v7 07/16] espintcp: Inline do_tcp_sendpages() Date: Mon, 15 May 2023 10:33:36 +0100 Message-Id: <20230515093345.396978-8-dhowells@redhat.com> In-Reply-To: <20230515093345.396978-1-dhowells@redhat.com> References: <20230515093345.396978-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765952970196586955?= X-GMAIL-MSGID: =?utf-8?q?1765952970196586955?= do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Steffen Klassert cc: Herbert Xu cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/xfrm/espintcp.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/net/xfrm/espintcp.c b/net/xfrm/espintcp.c index 872b80188e83..3504925babdb 100644 --- a/net/xfrm/espintcp.c +++ b/net/xfrm/espintcp.c @@ -205,14 +205,16 @@ static int espintcp_sendskb_locked(struct sock *sk, struct espintcp_msg *emsg, static int espintcp_sendskmsg_locked(struct sock *sk, struct espintcp_msg *emsg, int flags) { + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct sk_msg *skmsg = &emsg->skmsg; struct scatterlist *sg; int done = 0; int ret; - flags |= MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags |= MSG_SENDPAGE_NOTLAST; sg = &skmsg->sg.data[skmsg->sg.start]; do { + struct bio_vec bvec; size_t size = sg->length - emsg->offset; int offset = sg->offset + emsg->offset; struct page *p; @@ -220,11 +222,13 @@ static int espintcp_sendskmsg_locked(struct sock *sk, emsg->offset = 0; if (sg_is_last(sg)) - flags &= ~MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags &= ~MSG_SENDPAGE_NOTLAST; p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret < 0) { emsg->offset = offset - sg->offset; skmsg->sg.start += done; From patchwork Mon May 15 09:33:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 94029 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp6791283vqo; Mon, 15 May 2023 02:46:03 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4wxGq0s9rrSbTgswkDhH6racBD1ZLLu1b2OaRmV1ZbT8rEKf9MSY78z+7n6Uk0j7RiQZ/W X-Received: by 2002:a17:90a:8d14:b0:24d:e273:8863 with SMTP id c20-20020a17090a8d1400b0024de2738863mr34423284pjo.3.1684143962821; Mon, 15 May 2023 02:46:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684143962; cv=none; d=google.com; s=arc-20160816; b=e65/FAUigY7mclab4tLF3q9OLPU+1qn2+8t4fJkhFjuCq49VTCoHIYacWvGExBMdPM U2/fg2BGdlxluhv7SjVL1ym0UdesX7Y1p9u+jktkpg9VnUjdq8vTU6wgvqvoiVYKYy2a EYvk8ySaYP3OrgLhHUtqICD3Hn5FNnciyRBuaKpeW/rnwF5z6xTd35wGzktWrPbvij0K 9uv0tA6u0so627VyprzekNUB2MWcXLKUmGth/BlHK+7QaYECDTaC8pclfuJLwZVHEAWg /F7+KmW697eWumHZRUK1VaquzDOMQrb96bzTf2ZKG9E5Zm7jCniQsWVJU/u56AtczLzq eerQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JUrQ0rsOJEKe2VauAy4RSjWfQDMPveXienxXCaL8A+w=; b=WvuVy4G09Vjfb1EQ75kIntgdKGrPmXpOMGDPoVniE+itM4NpTYvbUxAEeTlKWFhMhT CBBJOUP26I0ccc41BMECphLSd8szAP2TbIxVMAEyMhR3EMo/uvA68/kd0B0lQdCeUizp s2IFeptpP7oHNOqzVrxabtzGGo6NBIDCiKMA7gYCtjTOZoEcyRVishEyjCG+VJg5tz7n +Ji3uncje6JZYjK//Cx9lCXED9xbVH5jPy9jJSiwVqecZmbuOe8OAVJZslbGzFAWPtMj p3/b0Xm9orGvaH+Cm8tZuH29uPW9nLIFRWZi5aFzL9hf1jOBwzaUKqeu1eJDRdNfzqNH ZFOA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=H9kHFhe1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p20-20020a17090a429400b00252873f1500si10949927pjg.138.2023.05.15.02.45.48; Mon, 15 May 2023 02:46:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=H9kHFhe1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240368AbjEOJgH (ORCPT + 99 others); Mon, 15 May 2023 05:36:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39776 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232865AbjEOJfS (ORCPT ); Mon, 15 May 2023 05:35:18 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0A3BE7B for ; Mon, 15 May 2023 02:34:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684143268; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JUrQ0rsOJEKe2VauAy4RSjWfQDMPveXienxXCaL8A+w=; b=H9kHFhe1FRhYsn+cRuwnXOgtg/xX18AbrjRniHG/MrBnFm846oP8Kc8/3pTorKCQgzAVz1 SyfGolSV58bJvu6KM/Ozp+Kjc3+Q32bMX/kZDTQvNd6iDplTaVIbv5S2i6qsxAvfFSmUD9 DxLUpPfCcsEwW4tCLEBKhfwBPboTc0Q= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-654-_VdNKKjjMc2ew_BgJPdgKA-1; Mon, 15 May 2023 05:34:25 -0400 X-MC-Unique: _VdNKKjjMc2ew_BgJPdgKA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 781491C0950C; Mon, 15 May 2023 09:34:24 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id E5861C15BA0; Mon, 15 May 2023 09:34:20 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Boris Pismenny , John Fastabend Subject: [PATCH net-next v7 08/16] tls: Inline do_tcp_sendpages() Date: Mon, 15 May 2023 10:33:37 +0100 Message-Id: <20230515093345.396978-9-dhowells@redhat.com> In-Reply-To: <20230515093345.396978-1-dhowells@redhat.com> References: <20230515093345.396978-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765952940264987629?= X-GMAIL-MSGID: =?utf-8?q?1765952940264987629?= do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: "David S. Miller" cc: Eric Dumazet cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tls.h | 2 +- net/tls/tls_main.c | 24 +++++++++++++++--------- 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/include/net/tls.h b/include/net/tls.h index 6056ce5a2aa5..5791ca7a189c 100644 --- a/include/net/tls.h +++ b/include/net/tls.h @@ -258,7 +258,7 @@ struct tls_context { struct scatterlist *partially_sent_record; u16 partially_sent_offset; - bool in_tcp_sendpages; + bool splicing_pages; bool pending_open_record_frags; struct mutex tx_lock; /* protects partially_sent_* fields and diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c index f2e7302a4d96..3d45fdb5c4e9 100644 --- a/net/tls/tls_main.c +++ b/net/tls/tls_main.c @@ -125,7 +125,10 @@ int tls_push_sg(struct sock *sk, u16 first_offset, int flags) { - int sendpage_flags = flags | MSG_SENDPAGE_NOTLAST; + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SENDPAGE_NOTLAST | MSG_SPLICE_PAGES | flags, + }; int ret = 0; struct page *p; size_t size; @@ -134,16 +137,19 @@ int tls_push_sg(struct sock *sk, size = sg->length - offset; offset += sg->offset; - ctx->in_tcp_sendpages = true; + ctx->splicing_pages = true; while (1) { if (sg_is_last(sg)) - sendpage_flags = flags; + msg.msg_flags = flags; /* is sending application-limited? */ tcp_rate_check_app_limited(sk); p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, sendpage_flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + + ret = tcp_sendmsg_locked(sk, &msg, size); if (ret != size) { if (ret > 0) { @@ -155,7 +161,7 @@ int tls_push_sg(struct sock *sk, offset -= sg->offset; ctx->partially_sent_offset = offset; ctx->partially_sent_record = (void *)sg; - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return ret; } @@ -169,7 +175,7 @@ int tls_push_sg(struct sock *sk, size = sg->length; } - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return 0; } @@ -247,11 +253,11 @@ static void tls_write_space(struct sock *sk) { struct tls_context *ctx = tls_get_ctx(sk); - /* If in_tcp_sendpages call lower protocol write space handler + /* If splicing_pages call lower protocol write space handler * to ensure we wake up any waiting operations there. For example - * if do_tcp_sendpages where to call sk_wait_event. + * if splicing pages where to call sk_wait_event. */ - if (ctx->in_tcp_sendpages) { + if (ctx->splicing_pages) { ctx->sk_write_space(sk); return; } From patchwork Mon May 15 09:33:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 94013 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp6789186vqo; Mon, 15 May 2023 02:40:32 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6VUNI22OYU39mSWXrrgzn/+yJVfeeVgDzr5Bbx2vjOmklEdv1hu1grEt/8egQbpHNf8b0T X-Received: by 2002:a17:90a:a0c:b0:24d:e956:f4df with SMTP id o12-20020a17090a0a0c00b0024de956f4dfmr33351897pjo.11.1684143632667; Mon, 15 May 2023 02:40:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684143632; cv=none; d=google.com; s=arc-20160816; b=SWkI8iG9j6CnTOXMDwDL/pGPU4HDUSYWinTHbGlcgAO1HFwBPo0gtTZzwBdhPIPbPH qFtLhEUyw93iHQCE5bRbGaWstRdI4y9n1SXyVYvaHC68kwAU80WxYYO0qiFpkNv8I95/ WTMOYxzmLKOhyWJMKB0OOynKVsZoI6RQVhDaU6RLSIK1iZeHzYZYX1UWyEPJ2a8cxq4r iCrE6OdYHFYBkq6ue0cGHCIJy6vLc4w1aZA3M6QkxeKVd8v0NYHUXki1wlPmReBM/K1j RGPH2PxynQ+NBDeTWWmSizKkH8BC2miYbrtg4lFT2+4znhTtirrNbixlL5Z1KoaLg2gL czcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=tEe6KsGIUg5Ug9eymZUK4cq5L1lxhWlwpIXxPsGo3ck=; b=RTSQNgFwDndf2sRD0qwFemyLQc0ipL7Vj78H9Xq3eOGI4ODfF7KtfUIwqbSEaczMIb tChJakvfXX5j98T/ZRFVh/W5H3Vp+8E+6yf8GntHKqw7o8NEMxRwA7TAqEAMgH7YuznS dD5CkpDpxjjuuODk0kGG0bRWABOwXBzUECJBiA53DEVlH5jFP+dOFQhfqg3WbRymikPb Q8waIBy/Oa70zsumjpUfXDAPrUY8RPTQeAv/NGNASxw3suLySYSjUWwLR1mAzZtX2N9f JYkncJuqkyBU8v8CBo4jUI+p5EjAdPy8yQQ9nmNYy2G3vncHdTGniUOK+eJaArj6aAzD Wzww== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OdPxdO8D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t28-20020a63b25c000000b0053423d8bf11si1591935pgo.448.2023.05.15.02.40.19; Mon, 15 May 2023 02:40:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OdPxdO8D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240359AbjEOJgC (ORCPT + 99 others); Mon, 15 May 2023 05:36:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39820 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238141AbjEOJfZ (ORCPT ); Mon, 15 May 2023 05:35:25 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D35AE52 for ; Mon, 15 May 2023 02:34:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684143274; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tEe6KsGIUg5Ug9eymZUK4cq5L1lxhWlwpIXxPsGo3ck=; b=OdPxdO8DFhPRe+KQNXGqPz0goIuh6Cz24y23ZnczbWxojkORWecMZaWZjrXSgqAl2Akw0m AuoA1yKaNgU3IwilzM5kExlNAObSlvd2FMJyLdcUimszQRSlAOOqHRvdW7UWtg715DoP/B BpGpSdjXIwBEOS8pLlVPYJX2g1f4LR0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-428-lT4Ao4tgPxOf3GaRnh_sfA-1; Mon, 15 May 2023 05:34:31 -0400 X-MC-Unique: lT4Ao4tgPxOf3GaRnh_sfA-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 02F81185A790; Mon, 15 May 2023 09:34:30 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 270BD492B00; Mon, 15 May 2023 09:34:25 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Bernard Metzler , Tom Talpey , Jason Gunthorpe , Leon Romanovsky , linux-rdma@vger.kernel.org Subject: [PATCH net-next v7 09/16] siw: Inline do_tcp_sendpages() Date: Mon, 15 May 2023 10:33:38 +0100 Message-Id: <20230515093345.396978-10-dhowells@redhat.com> In-Reply-To: <20230515093345.396978-1-dhowells@redhat.com> References: <20230515093345.396978-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765952593812068377?= X-GMAIL-MSGID: =?utf-8?q?1765952593812068377?= do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells Reviewed-by: Bernard Metzler Reviewed-by: Tom Talpey cc: Jason Gunthorpe cc: Leon Romanovsky cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: netdev@vger.kernel.org --- Notes: ver #6) - Don't clear MSG_SPLICE_PAGES on the last page. drivers/infiniband/sw/siw/siw_qp_tx.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_qp_tx.c b/drivers/infiniband/sw/siw/siw_qp_tx.c index 4b292e0504f1..ffb16beb6c30 100644 --- a/drivers/infiniband/sw/siw/siw_qp_tx.c +++ b/drivers/infiniband/sw/siw/siw_qp_tx.c @@ -312,7 +312,7 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, } /* - * 0copy TCP transmit interface: Use do_tcp_sendpages. + * 0copy TCP transmit interface: Use MSG_SPLICE_PAGES. * * Using sendpage to push page by page appears to be less efficient * than using sendmsg, even if data are copied. @@ -323,20 +323,27 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, static int siw_tcp_sendpages(struct socket *s, struct page **page, int offset, size_t size) { + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = (MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST | + MSG_SPLICE_PAGES), + }; struct sock *sk = s->sk; - int i = 0, rv = 0, sent = 0, - flags = MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST; + int i = 0, rv = 0, sent = 0; while (size) { size_t bytes = min_t(size_t, PAGE_SIZE - offset, size); if (size + offset <= PAGE_SIZE) - flags = MSG_MORE | MSG_DONTWAIT; + msg.msg_flags &= ~MSG_SENDPAGE_NOTLAST; tcp_rate_check_app_limited(sk); + bvec_set_page(&bvec, page[i], bytes, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + try_page_again: lock_sock(sk); - rv = do_tcp_sendpages(sk, page[i], offset, bytes, flags); + rv = tcp_sendmsg_locked(sk, &msg, size); release_sock(sk); if (rv > 0) { From patchwork Mon May 15 09:33:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 94027 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp6791200vqo; Mon, 15 May 2023 02:45:50 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7D8uq5jSGyG8zSsRLin7PZCJ8kXP7BAOs7JgrBjrzbwuNmWhM08Ny89ftfnMs1txoLpOrD X-Received: by 2002:a05:6a00:140f:b0:63d:24d0:2c32 with SMTP id l15-20020a056a00140f00b0063d24d02c32mr39392958pfu.33.1684143950406; Mon, 15 May 2023 02:45:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684143950; cv=none; d=google.com; s=arc-20160816; b=OXHOC0JHgvnsodMT3et9rz0QjXhf9bNhMurMrZvuF21cnq8AYiyxiTM3CHBQkzl2YK c7+LcbUMMOzBKxVZ5JA4Cb8ut1BDsFgW0iZ0JaXpc9tDDy6oULR7B4R5mbzduZFygS9N CXrYrMnDaibKOcHRe9UElsd3YpxJIydQwonPnYK3zCObqVBBbJTeLuYaj6FMkuICA7iw k3/U3EVRbj+Fk6is12WYMbD7KNwfH2hZtJ0V51NXK0fL4c0SND+j0aGGfeWzKro3Vzvr dNME3JKclRk+3Tolx5MESwqdLu5AdMw1/5iAX6CjPj21Gsv/d0rVZbbmstqUvGdiTx68 60Yg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=enBN5jT1muKlfAJKNsWEtZoec3VPKG1S27Z+IZtkOMs=; b=ljWmXGL4n4W8WfVaqUyzOZ2srIHziIoELNl2BiVcb+tPQL7NaVDXnr+hUlacqvBk1J rpTkmSwTLW3MRL7B5Pm7F3L5RWGi9JG86cbm5YtW2NwWouFwukQk1rSgLirfxddgyhub LsrUw0aL/wXiMyf3WdVP7ObKzvTbF1tcbfOMfdIsPdSvCDCKbs1eYjslGN9bB1/EQas0 QZDaaQsdXV7lCVGHzn0uhwbqyFUVBZXKEQJHQA/kgg0zSbTZxZHE3/u64q1XLxpIVsaO VQz6qv57xiyR0XiisQm/ND+pa3jxq877NtV05HtP0ZrNcYd+eT5WaYdDj0aKOAtD+yt/ RMrw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=dR4d6N7b; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v124-20020a626182000000b00647b9ed07c7si13950278pfb.128.2023.05.15.02.45.37; Mon, 15 May 2023 02:45:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=dR4d6N7b; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238019AbjEOJgm (ORCPT + 99 others); Mon, 15 May 2023 05:36:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39900 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239874AbjEOJfi (ORCPT ); Mon, 15 May 2023 05:35:38 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96CA4E63 for ; Mon, 15 May 2023 02:34:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684143281; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=enBN5jT1muKlfAJKNsWEtZoec3VPKG1S27Z+IZtkOMs=; b=dR4d6N7b2kctBeQ4OGLQuMn6lKc2G6nxLpU9Lq4FIyLjCXxK0v/OIKV22dra0CevjdfNYT 9nAZVlhsDVeczArgKPoUyYhAEUa1mUQJdPYwubHx95qmfGXQHSZWwDvR2/P0fBjlGlDPkE EZgAySH4AIE1CKrggqn5g+DJnoh8HDE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-115-U5XToGkmMZCD_W7I6Y32Ng-1; Mon, 15 May 2023 05:34:35 -0400 X-MC-Unique: U5XToGkmMZCD_W7I6Y32Ng-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B8275891750; Mon, 15 May 2023 09:34:34 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 27EA32026D25; Mon, 15 May 2023 09:34:31 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v7 10/16] tcp: Fold do_tcp_sendpages() into tcp_sendpage_locked() Date: Mon, 15 May 2023 10:33:39 +0100 Message-Id: <20230515093345.396978-11-dhowells@redhat.com> In-Reply-To: <20230515093345.396978-1-dhowells@redhat.com> References: <20230515093345.396978-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765952926961492944?= X-GMAIL-MSGID: =?utf-8?q?1765952926961492944?= Fold do_tcp_sendpages() into its last remaining caller, tcp_sendpage_locked(). Signed-off-by: David Howells cc: Eric Dumazet cc: David Ahern cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tcp.h | 2 -- net/ipv4/tcp.c | 21 +++++++-------------- 2 files changed, 7 insertions(+), 16 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index 04a31643cda3..02a6cff1827e 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -333,8 +333,6 @@ int tcp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags); int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags); -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags); int tcp_send_mss(struct sock *sk, int *size_goal, int flags); void tcp_push(struct sock *sk, int flags, int mss_now, int nonagle, int size_goal); diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 28231e503af9..5e4752131583 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -974,12 +974,17 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags) +int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, + size_t size, int flags) { struct bio_vec bvec; struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + if (!(sk->sk_route_caps & NETIF_F_SG)) + return sock_no_sendpage_locked(sk, page, offset, size, flags); + + tcp_rate_check_app_limited(sk); /* is sending application-limited? */ + bvec_set_page(&bvec, page, size, offset); iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); @@ -988,18 +993,6 @@ ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, return tcp_sendmsg_locked(sk, &msg, size); } -EXPORT_SYMBOL_GPL(do_tcp_sendpages); - -int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, - size_t size, int flags) -{ - if (!(sk->sk_route_caps & NETIF_F_SG)) - return sock_no_sendpage_locked(sk, page, offset, size, flags); - - tcp_rate_check_app_limited(sk); /* is sending application-limited? */ - - return do_tcp_sendpages(sk, page, offset, size, flags); -} EXPORT_SYMBOL_GPL(tcp_sendpage_locked); int tcp_sendpage(struct sock *sk, struct page *page, int offset, From patchwork Mon May 15 09:33:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 94039 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp6792091vqo; Mon, 15 May 2023 02:47:57 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4gWm4dJ7aa5MJaisC1OOAPq7x0qVvW3KTL+i/w6XVExVFNloQvlH+VwB19GRMyiqHglQaR X-Received: by 2002:a05:6a00:148b:b0:63b:7a55:ae89 with SMTP id v11-20020a056a00148b00b0063b7a55ae89mr46052785pfu.27.1684144076799; Mon, 15 May 2023 02:47:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684144076; cv=none; d=google.com; s=arc-20160816; b=SH6XSOGIWtND0bLDSYC11sah9sg+LfPJzRotVMYUivT/jU9ko8Raj5JXCaWofdbb1l 1i5NSbeIRCX1vMQ81UPQXDuBnEfjlEOH8O7YyRojKULLEzAx0IaA3vW8U0i1Yt9/SPAd Wdcq+BOAAVtSWVWfYg2oD8qkyIbAM+HxU8nFYzBCzm8Y20Db82JDM20XKjZLoJ7R0LPr 2wvSzVka8redRJ94z3Uv0kr/TnoR9BXzWgpV/SesDVKVnUU0q0XE6H60WFfLHXKMrzSn ceoLdTpMJljvNr5D7LiYA/1Ug7q9cpm98T32kqteTJ+CoAAf13JFqRgvhBYYOIVeGhhH zb3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Di14zlocOEDECmKOzgXOEyS4sjRLATonTy1WJdIXAUU=; b=Eq4mvKSuNDWHa2zwLNq1JuaWByUtKXP1IDBVFXpoW11iMnurctLnPhEDa6EvzDJnMn ZVMTWwsFvvt95jWhjH+WzPiIWtwY2/Xc5lc49OfolLUNFt7Iz2y+ljHJ4FsYDN2ViUOl +/dIB2SJBMFMbjS6w9Op0C1bqeH/Vw64QATuTBgosOVYFDvXQvvgswoIVRclZKwLU8YH N3u07qP3hLdv6B1hj46goFB6dqXrTG+S7rv6rxnGlljfYU5Z+um4hw1/lYe3Jf0UAn3k LYeP9t4H9Ri1GCWa+Fxb4zPTbFPCdLBTqEDlHQnLooX3iTEl2Whk4eO/EfPxqMWJ7TJu n0aw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=QmCtWFz2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j20-20020a633c14000000b0051f8817a307si16630318pga.869.2023.05.15.02.47.44; Mon, 15 May 2023 02:47:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=QmCtWFz2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240455AbjEOJg1 (ORCPT + 99 others); Mon, 15 May 2023 05:36:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39898 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238384AbjEOJfe (ORCPT ); Mon, 15 May 2023 05:35:34 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5917010D5 for ; Mon, 15 May 2023 02:34:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684143282; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Di14zlocOEDECmKOzgXOEyS4sjRLATonTy1WJdIXAUU=; b=QmCtWFz2sv4+4ExIDaBlxn9dlcRDT2+WnXn2Sn+ga/JB15WodCzxoJaazbksITpDA1XecD EmpbdVx3VFsgjHkTPcG/dndupWOV1YNlHEQnlJxmcoYaZZfDMmh3Gf/mQKIrdcYFZ1WYRl R9NXjVRySgzEW+xLbR/xMeqVyshqTcQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-675-DqP9KNUYOT-s0mTetspL_g-1; Mon, 15 May 2023 05:34:39 -0400 X-MC-Unique: DqP9KNUYOT-s0mTetspL_g-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6ABAF891750; Mon, 15 May 2023 09:34:38 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8BC60C15BA0; Mon, 15 May 2023 09:34:35 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v7 11/16] ip, udp: Support MSG_SPLICE_PAGES Date: Mon, 15 May 2023 10:33:40 +0100 Message-Id: <20230515093345.396978-12-dhowells@redhat.com> In-Reply-To: <20230515093345.396978-1-dhowells@redhat.com> References: <20230515093345.396978-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765953058945144948?= X-GMAIL-MSGID: =?utf-8?q?1765953058945144948?= Make IP/UDP sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/ipv4/ip_output.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 52fc840898d8..c7db973b5d29 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1048,6 +1048,14 @@ static int __ip_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + if (inet->hdrincl) + return -EPERM; + if (rt->dst.dev->features & NETIF_F_SG) + /* We need an empty buffer to attach stuff to */ + paged = true; + else + flags &= ~MSG_SPLICE_PAGES; } cork->length += length; @@ -1207,6 +1215,15 @@ static int __ip_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + struct msghdr *msg = from; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) + goto error; + copy = err; + wmem_alloc_delta += copy; } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Mon May 15 09:33:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 94028 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp6791239vqo; Mon, 15 May 2023 02:45:55 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7X24FQHnorxMTqmCst0773buKWdMmdBXQ/bEQlWP12x4rHnCi67eH1fERMMXaQt3XoK8tG X-Received: by 2002:a05:6a20:9184:b0:100:d4ee:ba80 with SMTP id v4-20020a056a20918400b00100d4eeba80mr21931815pzd.27.1684143955456; Mon, 15 May 2023 02:45:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684143955; cv=none; d=google.com; s=arc-20160816; b=TVdwpWZFADo2jiQnEhu9lM+A8eFWE/FqbZKEWEnEZ3OmH/JwJxMbm1J3cAk+5wwQBu vC/PBSclqXfDfyod4kvXNrpudU1oozZ++eum8x9WSEu4hLUZY+OpK7DBINmRaieZKVij cgjEOV5NPnb2VPw5uEMCloC3RQrvpZ0VGpJnvL7jmswS007MxRtdf0Oxp5eDIPRYD3do gmJ7lj5x+e8Dhhvm8Yq4al3iolw0jfiThDMPzKRJZWGqhpuMLQPHMizota62wMREKUqI 2yroG2RsL5k3owWFAwdyVDOcj8fnn9xcSMbtXIm9/J4/wyCKpFQnjBXaC3bIPL9ll68A A7JA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5dEuCyEYbAeTLbptDWG+lpG3TePaGyDY6iKA6gIr8Gk=; b=Nz/cKLswDYPX8XsL281GRh8VrprML1V6SwUmElaRGFD0OJ1Jcjx5+MGHDTWQ7POT07 tSphGxVXh6OxTPPEJUnR+rpfgDCOpgGZHxzRWhvnDQxEeMkmESmgYL/f6F2B3ufeputM 34JE0w+i98aF0L38eYNyyf1hMXOhFhOacCVcjIbIfTzQ+MhZUKwtPlqKNC2n8EKFFOeg 3lH6VbbMAnOzVt3M1SwcHtYcYLqWSAYFvvF6zRl8PAq4uMP4AvTOKvyIjVfT6jamSDVU YYbC4qf+BfC3ptYVKR/V9ROceyr/8gmMpBdXtoyzA3z4bmJ7ZY6251qaDFgiPkddj4eL Uo/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=gqV2PDOd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c20-20020a6566d4000000b005303a7ca052si15985060pgw.491.2023.05.15.02.45.42; Mon, 15 May 2023 02:45:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=gqV2PDOd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230186AbjEOJgt (ORCPT + 99 others); Mon, 15 May 2023 05:36:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39904 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239907AbjEOJfi (ORCPT ); Mon, 15 May 2023 05:35:38 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB1C510CE for ; Mon, 15 May 2023 02:34:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684143287; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5dEuCyEYbAeTLbptDWG+lpG3TePaGyDY6iKA6gIr8Gk=; b=gqV2PDOdSOtvhYwZyarkp9am8JU9GeNJMEcOlijk/7lq1oLv4KBqTjL8RmMHNL56hLbehO rJD+gz9VnGietzea0BN0UYvCrBmx7zzbILA5G1Whshr0EugBkfDtN/UBUK+x6DXGBm1p4Q dIg6OI2HFeZu/eOsaoxiw4pmP71REfs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-196-7yv7CbjeNdu0aC-6W8LDyw-1; Mon, 15 May 2023 05:34:43 -0400 X-MC-Unique: 7yv7CbjeNdu0aC-6W8LDyw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7E350870820; Mon, 15 May 2023 09:34:42 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id AED6840C6EC4; Mon, 15 May 2023 09:34:39 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v7 12/16] ip6, udp6: Support MSG_SPLICE_PAGES Date: Mon, 15 May 2023 10:33:41 +0100 Message-Id: <20230515093345.396978-13-dhowells@redhat.com> In-Reply-To: <20230515093345.396978-1-dhowells@redhat.com> References: <20230515093345.396978-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765952931808060335?= X-GMAIL-MSGID: =?utf-8?q?1765952931808060335?= Make IP6/UDP6 sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible, copying the data if not. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/ipv6/ip6_output.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 9554cf46ed88..c722cb881b2d 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1589,6 +1589,14 @@ static int __ip6_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + if (inet_sk(sk)->hdrincl) + return -EPERM; + if (rt->dst.dev->features & NETIF_F_SG) + /* We need an empty buffer to attach stuff to */ + paged = true; + else + flags &= ~MSG_SPLICE_PAGES; } /* @@ -1778,6 +1786,15 @@ static int __ip6_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + struct msghdr *msg = from; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) + goto error; + copy = err; + wmem_alloc_delta += copy; } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Mon May 15 09:33:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 94023 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp6790822vqo; Mon, 15 May 2023 02:44:57 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7rs+db+dB3L6nORkfP9zfc8lG0Zp5ikJ3du0L+hBmU3jR6hdSDqNRZPCHkGCLBrMQiZF+t X-Received: by 2002:a05:6a20:7f9c:b0:105:7857:e4dc with SMTP id d28-20020a056a207f9c00b001057857e4dcmr6106820pzj.7.1684143896800; Mon, 15 May 2023 02:44:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684143896; cv=none; d=google.com; s=arc-20160816; b=RcehtB9wkoIlOPi6z2vC70eX9s6c688h6HrFV/+UIQ377SAMb4RDXEST3LXCPzHzLZ vfSL6mfnMcNa174nSejdOIxP+XvNf+nYvbVpDRqeZ8+ptTitWdqgMiVqOkhfQfHL0nYO io6308sPk0PD5R9MmpxhvK7wnclz6ecULRuZNMQrSXESk+U3VsCU1ZBuVNyyTBfAHq83 GDhPPCx+m96mkylTylL+7NibWbKSxyfq2DkP/QUUZk+tTsLYkwFzZISt+10n3N5Kj3HO LOE07VzO/A1uCKGXDav7GggC83A1G7ovbvC4rpoDYUQ10e5AUkY1ZBetCk3mmpdmezte pR/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=arPLIYOMzcdUPWLjuF22tHo0vP4jVzFFThikgCzP+CE=; b=QaQVUCJPNtCiM87TpbaMZdVykfntMqzC4+zXxR5qkAmxXgSGvR2GiBk7SWiRHM8NTu 4ilBEckOwC37czxiAnKHfaA8m2HhwPZg0GKToi+7gqOo3rPJOI4ubcOdWQn7+iP3v14m QvY+SAQ7QdM0vatMep8QtZNImirx+juyXPmA4BLi6ZXYRKP+siITLVATGohZjG9rM6gG thnHMRPnQHKXnYKk/tcNXNEuM/NncublO+W5Nh49V3oX1Cm56V3caAYa3/iZU6xXIJGR 9+SI5bPTNL0yFojNMlSOi2oz7XqWiMZYwehkiuG0rXy+wYMrl9YTuem10gTnq3BA3Pil vBdQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bhDDoqoK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j13-20020a63cf0d000000b005139e5df68asi15419132pgg.433.2023.05.15.02.44.44; Mon, 15 May 2023 02:44:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bhDDoqoK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240656AbjEOJhR (ORCPT + 99 others); Mon, 15 May 2023 05:37:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239962AbjEOJfk (ORCPT ); Mon, 15 May 2023 05:35:40 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D7A26A7 for ; Mon, 15 May 2023 02:34:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684143292; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=arPLIYOMzcdUPWLjuF22tHo0vP4jVzFFThikgCzP+CE=; b=bhDDoqoKD8Vs2fOKJAsl3ERnapdNksqIpTXwiLqWGZkNVlfQpeZWWRnv9q6pUAD8mnhtz+ L6Zp6ifvWu5MmAjUHw7MJtetaxIVjFqsq66PDKkNV9NOCcgh8/e9taN460HiU2gS1owJwc UE3adqywb4jhsHO2H+bJlquXDEPZ32g= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-66-s4PfbxzJO6eXzsD79oKorw-1; Mon, 15 May 2023 05:34:47 -0400 X-MC-Unique: s4PfbxzJO6eXzsD79oKorw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 23A9387082F; Mon, 15 May 2023 09:34:46 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 40B9963F5F; Mon, 15 May 2023 09:34:43 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v7 13/16] udp: Convert udp_sendpage() to use MSG_SPLICE_PAGES Date: Mon, 15 May 2023 10:33:42 +0100 Message-Id: <20230515093345.396978-14-dhowells@redhat.com> In-Reply-To: <20230515093345.396978-1-dhowells@redhat.com> References: <20230515093345.396978-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765952870806214807?= X-GMAIL-MSGID: =?utf-8?q?1765952870806214807?= Convert udp_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - udp_sendpage() shouldn't lock the socket around udp_sendpage(). - udp_sendpage() should only set MSG_MORE if MSG_SENDPAGE_NOTLAST is set. net/ipv4/udp.c | 51 ++++++-------------------------------------------- 1 file changed, 6 insertions(+), 45 deletions(-) diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index aa32afd871ee..2879dc6d66ea 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -1332,54 +1332,15 @@ EXPORT_SYMBOL(udp_sendmsg); int udp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct inet_sock *inet = inet_sk(sk); - struct udp_sock *up = udp_sk(sk); - int ret; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES }; if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; - - if (!up->pending) { - struct msghdr msg = { .msg_flags = flags|MSG_MORE }; - - /* Call udp_sendmsg to specify destination address which - * sendpage interface can't pass. - * This will succeed only when the socket is connected. - */ - ret = udp_sendmsg(sk, &msg, 0); - if (ret < 0) - return ret; - } - - lock_sock(sk); + msg.msg_flags |= MSG_MORE; - if (unlikely(!up->pending)) { - release_sock(sk); - - net_dbg_ratelimited("cork failed\n"); - return -EINVAL; - } - - ret = ip_append_page(sk, &inet->cork.fl.u.ip4, - page, offset, size, flags); - if (ret == -EOPNOTSUPP) { - release_sock(sk); - return sock_no_sendpage(sk->sk_socket, page, offset, - size, flags); - } - if (ret < 0) { - udp_flush_pending_frames(sk); - goto out; - } - - up->len += size; - if (!(READ_ONCE(up->corkflag) || (flags&MSG_MORE))) - ret = udp_push_pending_frames(sk); - if (!ret) - ret = size; -out: - release_sock(sk); - return ret; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return udp_sendmsg(sk, &msg, size); } #define UDP_SKB_IS_STATELESS 0x80000000 From patchwork Mon May 15 09:33:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 94026 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp6791135vqo; Mon, 15 May 2023 02:45:40 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ438Twuo+tZr+huKSTmdameUhiQ08tqzOPvnLvl8q6yDksR7gmQzSKrrI1wyJe6MZDHL8sR X-Received: by 2002:a17:902:e549:b0:1ab:d6f:51b0 with SMTP id n9-20020a170902e54900b001ab0d6f51b0mr41840209plf.18.1684143940536; Mon, 15 May 2023 02:45:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684143940; cv=none; d=google.com; s=arc-20160816; b=ZfnW9/t8DRSFivAYNJxaJ5cn3olArICPYz7xwSz7iGP+1yvkXrXT1zsh29s+2my1Pe A0DPzhOG4VReLyffQiux3Hxm+/lfxxq8l4F3f1bs6jl3fdogMOUTXYX2MQB+BjzkQBa/ kTdOwBq1scjYK3V0C+oN/Ge79SpaKISMx95M/P6De3RrKFhlxqm5q7ey7QgYeV3HPnOh 1XUApAf+6vPa0c7v3u4EELTp7T5eNvO+ycaqEOpJFD+Z7oHUNbKatilmWrneQmuZl2uC OSrh59xmDF9mV2Ciz8ztUtWeWg2HsY4jaEVVNWqxlgu6yruycJn4YbXNAQIvwYlsJUNn 0AVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vVs0CsGrBKFATDm79TCVboT9gSnZlC7XDeXTGZfAJzM=; b=lWXMbizJyVxQtNEVmbvUI8rvkwvySBE1KWJwYMtrTK8gxzIH3eEn3kldvPVDsA+jvK ZnB+Df3G0AqqW80b2zuxpxwSMECSvaQhumSGTJUzLkG3MhXxc3V9q4/PtI5As5RPcePO ZzPc34lFGaPKMmoKswy6a9ZT1Ui7VXeCltW+Mb3E99csYR0yvdvJacpu1GyMXom0xHUz Wbt6dgYtGIxwSfrAZ8KCy4ICaaApLyLVSUfSpq5ZwU1ajoJTtfqamfz8OAesjFpBT1RU W4koEVDjKFaUgGViCr76FAQrB6/TB2l3MIyDor4PoPiF2w3NpJtzuoziJpjGhr9AAnLX vTIQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=B8EtneKn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q17-20020a170902dad100b001990520ebccsi16885007plx.587.2023.05.15.02.45.28; Mon, 15 May 2023 02:45:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=B8EtneKn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240305AbjEOJhK (ORCPT + 99 others); Mon, 15 May 2023 05:37:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39972 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238860AbjEOJfq (ORCPT ); Mon, 15 May 2023 05:35:46 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6DE6CF1 for ; Mon, 15 May 2023 02:34:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684143294; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vVs0CsGrBKFATDm79TCVboT9gSnZlC7XDeXTGZfAJzM=; b=B8EtneKnfazipX5zzGu2VJXt3pLn33aPFx8p9MM5c7e9gOB/ldFCf/PZt4LgVPwK29juqX pImO40SZxRFB5Zg+6VN3rwbveY5+6T7nhX5s+/OtL/7OCMm6NZYJaIzwKw6j6ZCHlbeko1 khp8GGBn5tCJ7hD9lejqMDNt38q2J/w= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-447-17qNAQVlPLmO5920f3R9FQ-1; Mon, 15 May 2023 05:34:51 -0400 X-MC-Unique: 17qNAQVlPLmO5920f3R9FQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 83AFB80080E; Mon, 15 May 2023 09:34:50 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1799A40C206F; Mon, 15 May 2023 09:34:46 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v7 14/16] ip: Remove ip_append_page() Date: Mon, 15 May 2023 10:33:43 +0100 Message-Id: <20230515093345.396978-15-dhowells@redhat.com> In-Reply-To: <20230515093345.396978-1-dhowells@redhat.com> References: <20230515093345.396978-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765952916138940895?= X-GMAIL-MSGID: =?utf-8?q?1765952916138940895?= ip_append_page() is no longer used with the removal of udp_sendpage(), so remove it. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #7) - Remove now-unused csum_page(). include/net/ip.h | 2 - net/ipv4/ip_output.c | 148 ++----------------------------------------- 2 files changed, 4 insertions(+), 146 deletions(-) diff --git a/include/net/ip.h b/include/net/ip.h index c3fffaa92d6e..7627a4df893b 100644 --- a/include/net/ip.h +++ b/include/net/ip.h @@ -220,8 +220,6 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4, unsigned int flags); int ip_generic_getfrag(void *from, char *to, int offset, int len, int odd, struct sk_buff *skb); -ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, - int offset, size_t size, int flags); struct sk_buff *__ip_make_skb(struct sock *sk, struct flowi4 *fl4, struct sk_buff_head *queue, struct inet_cork *cork); diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index c7db973b5d29..553c740a6bfb 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -946,17 +946,6 @@ ip_generic_getfrag(void *from, char *to, int offset, int len, int odd, struct sk } EXPORT_SYMBOL(ip_generic_getfrag); -static inline __wsum -csum_page(struct page *page, int offset, int copy) -{ - char *kaddr; - __wsum csum; - kaddr = kmap(page); - csum = csum_partial(kaddr + offset, copy, 0); - kunmap(page); - return csum; -} - static int __ip_append_data(struct sock *sk, struct flowi4 *fl4, struct sk_buff_head *queue, @@ -1327,10 +1316,10 @@ static int ip_setup_cork(struct sock *sk, struct inet_cork *cork, } /* - * ip_append_data() and ip_append_page() can make one large IP datagram - * from many pieces of data. Each pieces will be holded on the socket - * until ip_push_pending_frames() is called. Each piece can be a page - * or non-page data. + * ip_append_data() can make one large IP datagram from many pieces of + * data. Each piece will be held on the socket until + * ip_push_pending_frames() is called. Each piece can be a page or + * non-page data. * * Not only UDP, other transport protocols - e.g. raw sockets - can use * this interface potentially. @@ -1363,135 +1352,6 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4, from, length, transhdrlen, flags); } -ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, - int offset, size_t size, int flags) -{ - struct inet_sock *inet = inet_sk(sk); - struct sk_buff *skb; - struct rtable *rt; - struct ip_options *opt = NULL; - struct inet_cork *cork; - int hh_len; - int mtu; - int len; - int err; - unsigned int maxfraglen, fragheaderlen, fraggap, maxnonfragsize; - - if (inet->hdrincl) - return -EPERM; - - if (flags&MSG_PROBE) - return 0; - - if (skb_queue_empty(&sk->sk_write_queue)) - return -EINVAL; - - cork = &inet->cork.base; - rt = (struct rtable *)cork->dst; - if (cork->flags & IPCORK_OPT) - opt = cork->opt; - - if (!(rt->dst.dev->features & NETIF_F_SG)) - return -EOPNOTSUPP; - - hh_len = LL_RESERVED_SPACE(rt->dst.dev); - mtu = cork->gso_size ? IP_MAX_MTU : cork->fragsize; - - fragheaderlen = sizeof(struct iphdr) + (opt ? opt->optlen : 0); - maxfraglen = ((mtu - fragheaderlen) & ~7) + fragheaderlen; - maxnonfragsize = ip_sk_ignore_df(sk) ? 0xFFFF : mtu; - - if (cork->length + size > maxnonfragsize - fragheaderlen) { - ip_local_error(sk, EMSGSIZE, fl4->daddr, inet->inet_dport, - mtu - (opt ? opt->optlen : 0)); - return -EMSGSIZE; - } - - skb = skb_peek_tail(&sk->sk_write_queue); - if (!skb) - return -EINVAL; - - cork->length += size; - - while (size > 0) { - /* Check if the remaining data fits into current packet. */ - len = mtu - skb->len; - if (len < size) - len = maxfraglen - skb->len; - - if (len <= 0) { - struct sk_buff *skb_prev; - int alloclen; - - skb_prev = skb; - fraggap = skb_prev->len - maxfraglen; - - alloclen = fragheaderlen + hh_len + fraggap + 15; - skb = sock_wmalloc(sk, alloclen, 1, sk->sk_allocation); - if (unlikely(!skb)) { - err = -ENOBUFS; - goto error; - } - - /* - * Fill in the control structures - */ - skb->ip_summed = CHECKSUM_NONE; - skb->csum = 0; - skb_reserve(skb, hh_len); - - /* - * Find where to start putting bytes. - */ - skb_put(skb, fragheaderlen + fraggap); - skb_reset_network_header(skb); - skb->transport_header = (skb->network_header + - fragheaderlen); - if (fraggap) { - skb->csum = skb_copy_and_csum_bits(skb_prev, - maxfraglen, - skb_transport_header(skb), - fraggap); - skb_prev->csum = csum_sub(skb_prev->csum, - skb->csum); - pskb_trim_unique(skb_prev, maxfraglen); - } - - /* - * Put the packet on the pending queue. - */ - __skb_queue_tail(&sk->sk_write_queue, skb); - continue; - } - - if (len > size) - len = size; - - if (skb_append_pagefrags(skb, page, offset, len, - MAX_SKB_FRAGS)) { - err = -EMSGSIZE; - goto error; - } - - if (skb->ip_summed == CHECKSUM_NONE) { - __wsum csum; - csum = csum_page(page, offset, len); - skb->csum = csum_block_add(skb->csum, csum, skb->len); - } - - skb_len_add(skb, len); - refcount_add(len, &sk->sk_wmem_alloc); - offset += len; - size -= len; - } - return 0; - -error: - cork->length -= size; - IP_INC_STATS(sock_net(sk), IPSTATS_MIB_OUTDISCARDS); - return err; -} - static void ip_cork_release(struct inet_cork *cork) { cork->flags &= ~IPCORK_OPT; From patchwork Mon May 15 09:33:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 94060 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp6795366vqo; Mon, 15 May 2023 02:56:51 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5vqjY58lQZjmLaY4EPB4X4ovYQ6mUD9cl7KmC4a/WlKleMbRCx5U9mG2z1Pv2OD2AcOiKC X-Received: by 2002:a17:90a:608f:b0:250:2384:120d with SMTP id z15-20020a17090a608f00b002502384120dmr31852381pji.19.1684144611071; Mon, 15 May 2023 02:56:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684144611; cv=none; d=google.com; s=arc-20160816; b=O6tBovZZxXlz7sf+jtnX7HqxTHtBsReNHqMRchQ9HH6yuJu0gnXtgJX4Sl9vcwoXQ2 gDSjKZwx++ns89Cyujmv3aKpQSnDT6vfLKqJpx0CS8HxZyx4DRchWN9FmU9ScI/M5oTe JTv0snAQMEiebJiCROHcXQKYp1XEaXiq/bN5OOlMxTdRs03ZmiwJpa3/XQtHuyvqUkgP LqWaMuF5mt/NbWOknJBE20nxr8ch0NdavTtGmP/pg/ncp4WAbnfYH+UTIryaJvP4xiwl RpOScABDkhXNuwF2UlE7vxCQSLXRlwjKGmC4ndnN3G35daApUxqt8NIQ+ec6EdRoQEMS mtkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4sbDm+4sgaynAjmiQFVbDl+OyfXvG2vWGJtRH+IAST4=; b=ttYxXhkXsdL6QtxuTOwGhhVJLYNDbexa5/anvuak9vWaZ+YYbY8JRlhFe8bIKVyQX9 3FxVq/8bUBJ0nlptfw29KRLCha9uCL0SQnEes1Yc/dr6pyymwPfFKwQEbv3S0tqS0N+A idUGV1JZyAdTBJqTo/g3QfGvEJW2LjS68jD0SKqXnOFZl7gB8Usn0sceMhbMujjXt9gV EDd0/JasVGczYsk7C/DlwvnKG/K6lyI8HdjeJN34+/OjjQZ5+5oSqKum5Xw4oBX6gj9J Phu7rfnRQprPlILYbBHytnKpej04yrnS8fT3rLSbU6L0yduT99+UTiuk7cwf+wGYpW4Z 8K6g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=QU2ijg83; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h184-20020a636cc1000000b005303ae2e7fasi15643721pgc.640.2023.05.15.02.56.38; Mon, 15 May 2023 02:56:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=QU2ijg83; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240243AbjEOJhD (ORCPT + 99 others); Mon, 15 May 2023 05:37:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39524 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240029AbjEOJfr (ORCPT ); Mon, 15 May 2023 05:35:47 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC6AAE69 for ; Mon, 15 May 2023 02:34:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684143298; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4sbDm+4sgaynAjmiQFVbDl+OyfXvG2vWGJtRH+IAST4=; b=QU2ijg83FwnKDPEVz7xr0B/AOVthu0sUNFXTZCkU7KDorppCHdrAnYDj07KdPFUzRTYwsh maLpWV/vpr/erE0sbXkUnsvxIqlI6zMUMwvMB7r1Y1Dfgq1aYXbfq/zAgoT6UqRBQTsvAK ieiKsN3sz3aI7gdf88RaweV+nxcZdYM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-351-oLogL-wgMESbZ_pbZl0Txg-1; Mon, 15 May 2023 05:34:55 -0400 X-MC-Unique: oLogL-wgMESbZ_pbZl0Txg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DE904185A79C; Mon, 15 May 2023 09:34:54 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id E86781121314; Mon, 15 May 2023 09:34:51 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Kuniyuki Iwashima Subject: [PATCH net-next v7 15/16] af_unix: Support MSG_SPLICE_PAGES Date: Mon, 15 May 2023 10:33:44 +0100 Message-Id: <20230515093345.396978-16-dhowells@redhat.com> In-Reply-To: <20230515093345.396978-1-dhowells@redhat.com> References: <20230515093345.396978-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765953619753410766?= X-GMAIL-MSGID: =?utf-8?q?1765953619753410766?= Make AF_UNIX sendmsg() support MSG_SPLICE_PAGES, splicing in pages from the source iterator if possible and copying the data in otherwise. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Kuniyuki Iwashima cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/unix/af_unix.c | 49 +++++++++++++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 16 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index dd55506b4632..976bc1c5e11b 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2200,19 +2200,25 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, while (sent < len) { size = len - sent; - /* Keep two messages in the pipe so it schedules better */ - size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + skb = sock_alloc_send_pskb(sk, 0, 0, + msg->msg_flags & MSG_DONTWAIT, + &err, 0); + } else { + /* Keep two messages in the pipe so it schedules better */ + size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); - /* allow fallback to order-0 allocations */ - size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); + /* allow fallback to order-0 allocations */ + size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); - data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); + data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); - data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); + data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); - skb = sock_alloc_send_pskb(sk, size - data_len, data_len, - msg->msg_flags & MSG_DONTWAIT, &err, - get_order(UNIX_SKB_FRAGS_SZ)); + skb = sock_alloc_send_pskb(sk, size - data_len, data_len, + msg->msg_flags & MSG_DONTWAIT, &err, + get_order(UNIX_SKB_FRAGS_SZ)); + } if (!skb) goto out_err; @@ -2224,13 +2230,24 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, } fds_sent = true; - skb_put(skb, size - data_len); - skb->data_len = data_len; - skb->len = size; - err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); - if (err) { - kfree_skb(skb); - goto out_err; + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + err = skb_splice_from_iter(skb, &msg->msg_iter, size, + sk->sk_allocation); + if (err < 0) { + kfree_skb(skb); + goto out_err; + } + size = err; + refcount_add(size, &sk->sk_wmem_alloc); + } else { + skb_put(skb, size - data_len); + skb->data_len = data_len; + skb->len = size; + err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); + if (err) { + kfree_skb(skb); + goto out_err; + } } unix_state_lock(other); From patchwork Mon May 15 09:33:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 94022 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp6790802vqo; Mon, 15 May 2023 02:44:54 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ40xZgo+NFISWW9XHpnRvFnwBNdnRMuf0qiq3KLEIoYRsdYTW8/XjkH5IuOJ37ozAF+QWnE X-Received: by 2002:a17:903:185:b0:1aa:db0f:1aba with SMTP id z5-20020a170903018500b001aadb0f1abamr41075603plg.47.1684143894640; Mon, 15 May 2023 02:44:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684143894; cv=none; d=google.com; s=arc-20160816; b=vIRTpWukSWjPkG/3DepJiDgL5vGsZpqM6ckADNNDV2lAW0BPoAUCAxClvwtNJi3yxj NMm4p7SqXxVaV4KhEnlt8yavBdatSPGS4boloiFxpzySOlb4WkbV5Km221J2pY6OuKLI stMKRgAfvAg0fYnZtirjLMICQ/rjV+y0IwyhxWL1nq+SaoVlTQjXG84O9qbjM7w7c97v qVoflFIXYXagN/lPxz89Yxh03AxMlALVoAh1lzCwkV+8e2wTpp0U8eu7bkdUEXy4fx9O wlAblgJVOyr7ZRzWAwuFR2ygT26c9HF7iDocsAw+EQkEADq4eW7q6yDll8cBVGQvuW4K b8Vg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=wF+2b4fBEfy00AN3VDBr6eYpnbX7vBgnsz0rKmJdTGo=; b=l4VPNfLL4ndXlcMadu1cyICG556FZOTIvqVPjx+Ic1R3XFXMGHDR0P4M2kzXoEILRg zZPoc9bODao8aFqxmhInh6LUn/MzGv8hEJ4BKuzyqM9iAHQ3yO9z+MxfRccYQjM8m9PH qMIdBB5mvZ8+I+ES3pYsxbxNhS7O/2lf83lggEoq/958xXGEUR7KUK6IYyG8IdLYOaSt 5IxKduXptAixKYW7cPzk+YEB8IsXMm+LcESUoBX4GrFNeHPASOkZIxoUxq+PeGBTmMWZ /P+yvd8SYImXUcCt0tlE3R/nv5ng0Q7TZI4KKLegFurYipNsykHuJmn15TlT4TlDynIY /nVg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZQmILozE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r134-20020a632b8c000000b005288509b88asi15729659pgr.681.2023.05.15.02.44.42; Mon, 15 May 2023 02:44:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZQmILozE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240196AbjEOJg5 (ORCPT + 99 others); Mon, 15 May 2023 05:36:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40174 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240033AbjEOJfs (ORCPT ); Mon, 15 May 2023 05:35:48 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D60C319AB for ; Mon, 15 May 2023 02:35:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684143304; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wF+2b4fBEfy00AN3VDBr6eYpnbX7vBgnsz0rKmJdTGo=; b=ZQmILozE5fw8BZWKXiXaBBnuX6H0rKpH/7F8B3lRLudFq6AW+bf18Cbzph1cRKiaHo91Tp T0vv4dF2pCPS7uRmvGaD1opsqc/K9Sx5bdwsnKENi6Qq5Cbig1sYi7xO9E32S/xxzdqE+w ZT87YFDs/Q+TfGF4AHZM3vqeKfT/XuM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-616-mv3QcuzZMBiItf3Uupm36w-1; Mon, 15 May 2023 05:34:59 -0400 X-MC-Unique: mv3QcuzZMBiItf3Uupm36w-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6FEA6101A551; Mon, 15 May 2023 09:34:58 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 880FB4021C6; Mon, 15 May 2023 09:34:55 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Kuniyuki Iwashima Subject: [PATCH net-next v7 16/16] unix: Convert udp_sendpage() to use MSG_SPLICE_PAGES Date: Mon, 15 May 2023 10:33:45 +0100 Message-Id: <20230515093345.396978-17-dhowells@redhat.com> In-Reply-To: <20230515093345.396978-1-dhowells@redhat.com> References: <20230515093345.396978-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765952868558831767?= X-GMAIL-MSGID: =?utf-8?q?1765952868558831767?= Convert unix_stream_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Kuniyuki Iwashima cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/unix/af_unix.c | 134 +++------------------------------------------ 1 file changed, 7 insertions(+), 127 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index 976bc1c5e11b..115436ce1f8a 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -1839,24 +1839,6 @@ static void maybe_add_creds(struct sk_buff *skb, const struct socket *sock, } } -static int maybe_init_creds(struct scm_cookie *scm, - struct socket *socket, - const struct sock *other) -{ - int err; - struct msghdr msg = { .msg_controllen = 0 }; - - err = scm_send(socket, &msg, scm, false); - if (err) - return err; - - if (unix_passcred_enabled(socket, other)) { - scm->pid = get_pid(task_tgid(current)); - current_uid_gid(&scm->creds.uid, &scm->creds.gid); - } - return err; -} - static bool unix_skb_scm_eq(struct sk_buff *skb, struct scm_cookie *scm) { @@ -2292,117 +2274,15 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, static ssize_t unix_stream_sendpage(struct socket *socket, struct page *page, int offset, size_t size, int flags) { - int err; - bool send_sigpipe = false; - bool init_scm = true; - struct scm_cookie scm; - struct sock *other, *sk = socket->sk; - struct sk_buff *skb, *newskb = NULL, *tail = NULL; - - if (flags & MSG_OOB) - return -EOPNOTSUPP; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES }; - other = unix_peer(sk); - if (!other || sk->sk_state != TCP_ESTABLISHED) - return -ENOTCONN; - - if (false) { -alloc_skb: - unix_state_unlock(other); - mutex_unlock(&unix_sk(other)->iolock); - newskb = sock_alloc_send_pskb(sk, 0, 0, flags & MSG_DONTWAIT, - &err, 0); - if (!newskb) - goto err; - } - - /* we must acquire iolock as we modify already present - * skbs in the sk_receive_queue and mess with skb->len - */ - err = mutex_lock_interruptible(&unix_sk(other)->iolock); - if (err) { - err = flags & MSG_DONTWAIT ? -EAGAIN : -ERESTARTSYS; - goto err; - } - - if (sk->sk_shutdown & SEND_SHUTDOWN) { - err = -EPIPE; - send_sigpipe = true; - goto err_unlock; - } - - unix_state_lock(other); + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - if (sock_flag(other, SOCK_DEAD) || - other->sk_shutdown & RCV_SHUTDOWN) { - err = -EPIPE; - send_sigpipe = true; - goto err_state_unlock; - } - - if (init_scm) { - err = maybe_init_creds(&scm, socket, other); - if (err) - goto err_state_unlock; - init_scm = false; - } - - skb = skb_peek_tail(&other->sk_receive_queue); - if (tail && tail == skb) { - skb = newskb; - } else if (!skb || !unix_skb_scm_eq(skb, &scm)) { - if (newskb) { - skb = newskb; - } else { - tail = skb; - goto alloc_skb; - } - } else if (newskb) { - /* this is fast path, we don't necessarily need to - * call to kfree_skb even though with newskb == NULL - * this - does no harm - */ - consume_skb(newskb); - newskb = NULL; - } - - if (skb_append_pagefrags(skb, page, offset, size, MAX_SKB_FRAGS)) { - tail = skb; - goto alloc_skb; - } - - skb->len += size; - skb->data_len += size; - skb->truesize += size; - refcount_add(size, &sk->sk_wmem_alloc); - - if (newskb) { - err = unix_scm_to_skb(&scm, skb, false); - if (err) - goto err_state_unlock; - spin_lock(&other->sk_receive_queue.lock); - __skb_queue_tail(&other->sk_receive_queue, newskb); - spin_unlock(&other->sk_receive_queue.lock); - } - - unix_state_unlock(other); - mutex_unlock(&unix_sk(other)->iolock); - - other->sk_data_ready(other); - scm_destroy(&scm); - return size; - -err_state_unlock: - unix_state_unlock(other); -err_unlock: - mutex_unlock(&unix_sk(other)->iolock); -err: - kfree_skb(newskb); - if (send_sigpipe && !(flags & MSG_NOSIGNAL)) - send_sig(SIGPIPE, current, 0); - if (!init_scm) - scm_destroy(&scm); - return err; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return unix_stream_sendmsg(socket, &msg, size); } static int unix_seqpacket_sendmsg(struct socket *sock, struct msghdr *msg,