From patchwork Mon May 22 12:11:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 97380 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1409272vqo; Mon, 22 May 2023 05:28:00 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4Nb6dFfwUTBOfZw42GTbCg7M4jczRTeUrifL+k2j08JoORggRiLSmj1nBERA7TYAojyeqS X-Received: by 2002:a17:902:ced0:b0:1af:b47e:7892 with SMTP id d16-20020a170902ced000b001afb47e7892mr3298292plg.67.1684758480300; Mon, 22 May 2023 05:28:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684758480; cv=none; d=google.com; s=arc-20160816; b=bnxU4V7TwxjqCfAu1cowcd7eHuY8fgEYnMCjIlXxgZmg5x1FIqHSbiWcFN1NdgDyv4 qBEc8ON4cCg3vemr3KLBpM+JsZ/LdY7ub7SFQ/ZJvuSJoP7yOq90Vg2tQcBCuUP5BXKP rH+pD6PQywqRoElK7CCONHeGNeyvA+oYIrPRbLJFIgUzGjVac/STD4+E25uTchrQnDhd E8l6jbPfBd0OPXf5b+wlaK54BO9uC4vyHBV2U1rqejDUPmxVmhw3IJ6fNY37DN6jYF2r YCOvl+Ik9EZx+PRdsyAPCTFZ8Bz1UpWa860tAX3QIfF2VTKxKtuT0p5CCGCuvjMdQiwo R1Jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=zo53+MxtcExCaAvP1E3d2cbKIX8PzAGUnvJuN4UlUmE=; b=yn7MJ7fSkUgirwcTrWSYchvfQ1w9+33xX81GYY4es8c7kh/jvkonvtOW1Cegez9acv BQe3r0tw09Cs7cD1vKqgf6MnRJAp1SSMxnp9LVK1+5jhoP/n/7k3HxJSGaWADv2pRMD0 yE7H0k78rEfvEvqM7Ry2OY9g+Z6tA27bh7qKj78dkt0QP3NHI2x4OsPfbQ/WFsVzdJxe 5R1Xo2xostZxmx8nllZKVrZvEnPGwoItvLwUad1JNL8Hj0qqGBeDYHfXAa10vNwPlRZT Kyu0ykAj7Vfd1O5IQr+KNaYbhSrFwxWA3FktnKbgNQkHKCZtpI1AeEACx6nhucA7Cpz3 g6qA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=cWm5s77X; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x4-20020a170902ea8400b001ac5896c746si4309409plb.329.2023.05.22.05.27.48; Mon, 22 May 2023 05:28:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=cWm5s77X; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233256AbjEVMMb (ORCPT + 99 others); Mon, 22 May 2023 08:12:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233221AbjEVMM0 (ORCPT ); Mon, 22 May 2023 08:12:26 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F769A2 for ; Mon, 22 May 2023 05:11:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684757498; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zo53+MxtcExCaAvP1E3d2cbKIX8PzAGUnvJuN4UlUmE=; b=cWm5s77XtGqo27sxpLyTWLrmOe4XoB/2V4fNOHcaK88hwCs3P045plsHGayumDdi6rMFMr cZznylM5QtQzsF2moq6qoH3cuJsjobEvUpwGvNaKTNuIibK4fxFIcXk6YeKr2GAMYAOYaM u4f35MpHR/CH7A17OQ4q8R24JsBdgV0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-645-LAZUC1NYOiyGQR7ae9mZ2Q-1; Mon, 22 May 2023 08:11:34 -0400 X-MC-Unique: LAZUC1NYOiyGQR7ae9mZ2Q-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6FA7E185A78F; Mon, 22 May 2023 12:11:33 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id A54FE140E95D; Mon, 22 May 2023 12:11:30 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn , io-uring@vger.kernel.org Subject: [PATCH net-next v10 01/16] net: Declare MSG_SPLICE_PAGES internal sendmsg() flag Date: Mon, 22 May 2023 13:11:10 +0100 Message-Id: <20230522121125.2595254-2-dhowells@redhat.com> In-Reply-To: <20230522121125.2595254-1-dhowells@redhat.com> References: <20230522121125.2595254-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766597308072100959?= X-GMAIL-MSGID: =?utf-8?q?1766597308072100959?= Declare MSG_SPLICE_PAGES, an internal sendmsg() flag, that hints to a network protocol that it should splice pages from the source iterator rather than copying the data if it can. This flag is added to a list that is cleared by sendmsg syscalls on entry. This is intended as a replacement for the ->sendpage() op, allowing a way to splice in several multipage folios in one go. Signed-off-by: David Howells Reviewed-by: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: io-uring@vger.kernel.org cc: netdev@vger.kernel.org --- Notes: ver #7) - In ____sys_sendmsg(), clear internal flags before setting msg_flags. - Clear internal flags in uring io_send{,_zc}(). include/linux/socket.h | 3 +++ io_uring/net.c | 2 ++ net/socket.c | 2 ++ 3 files changed, 7 insertions(+) diff --git a/include/linux/socket.h b/include/linux/socket.h index 13c3a237b9c9..bd1cc3238851 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -327,6 +327,7 @@ struct ucred { */ #define MSG_ZEROCOPY 0x4000000 /* Use user data in kernel path */ +#define MSG_SPLICE_PAGES 0x8000000 /* Splice the pages from the iterator in sendmsg() */ #define MSG_FASTOPEN 0x20000000 /* Send data in TCP SYN */ #define MSG_CMSG_CLOEXEC 0x40000000 /* Set close_on_exec for file descriptor received through @@ -337,6 +338,8 @@ struct ucred { #define MSG_CMSG_COMPAT 0 /* We never have 32 bit fixups */ #endif +/* Flags to be cleared on entry by sendmsg and sendmmsg syscalls */ +#define MSG_INTERNAL_SENDMSG_FLAGS (MSG_SPLICE_PAGES) /* Setsockoptions(2) level. Thanks to BSD these must match IPPROTO_xxx */ #define SOL_IP 0 diff --git a/io_uring/net.c b/io_uring/net.c index 89e839013837..f7cbb3c7a575 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -389,6 +389,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) if (flags & MSG_WAITALL) min_ret = iov_iter_count(&msg.msg_iter); + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; msg.msg_flags = flags; ret = sock_sendmsg(sock, &msg); if (ret < min_ret) { @@ -1136,6 +1137,7 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) msg_flags |= MSG_DONTWAIT; if (msg_flags & MSG_WAITALL) min_ret = iov_iter_count(&msg.msg_iter); + msg_flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; msg.msg_flags = msg_flags; msg.msg_ubuf = &io_notif_to_data(zc->notif)->uarg; diff --git a/net/socket.c b/net/socket.c index b7e01d0fe082..3df96e9ba4e2 100644 --- a/net/socket.c +++ b/net/socket.c @@ -2138,6 +2138,7 @@ int __sys_sendto(int fd, void __user *buff, size_t len, unsigned int flags, msg.msg_name = (struct sockaddr *)&address; msg.msg_namelen = addr_len; } + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; if (sock->file->f_flags & O_NONBLOCK) flags |= MSG_DONTWAIT; msg.msg_flags = flags; @@ -2483,6 +2484,7 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys, msg_sys->msg_control = ctl_buf; msg_sys->msg_control_is_user = false; } + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; msg_sys->msg_flags = flags; if (sock->file->f_flags & O_NONBLOCK) From patchwork Mon May 22 12:11:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 97358 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1408089vqo; Mon, 22 May 2023 05:25:58 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4kEONgmC2Kl7Sg2FJiB1lF+/JZ0/jJpGGHDkMM2BcqWsS4oiV4Jqw8OiBpEkkOldIhCW+E X-Received: by 2002:a17:90a:9301:b0:255:4635:830c with SMTP id p1-20020a17090a930100b002554635830cmr5144827pjo.40.1684758358452; Mon, 22 May 2023 05:25:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684758358; cv=none; d=google.com; s=arc-20160816; b=s96kiN7N5y5pctXw6qan34W/CZb5jYCG9rU21c2MoXxmvPS98jS4CZ6zckmfhoHela mh+aUedmK3d43BE9MkgFWp9+FuzOhb59fXIDhwylE77sJSJIIZO/z9Le4+WmDxK/QJPT dzwQH6TeQAVoRN9SlIrmTm1gaK82bm9JewEfcCoZpPp8xtrj3xrAa/Q5SliQR04hWaq4 exyUsgl2m/4YHs7aZ3jEzchhBejTpqb292/acsktM/g4BhNQKGmN9tQCx2ltRiGftoJP JB4x8upiOwlpexWuo0CB7N7dIkwOtR9Ki/afS5IBbBh3KcYK/apM9NMG7FgsrKljUXna 3dhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=dMBJPtbzh4IjFjJ1HeY2h7yLKLc/0VkfcRg1OdFJq1c=; b=OatsX/NJJ+UalL77XH+G1/eOaqX3hIw+7ZUlAF/bpn7Z2EkEftPNtmNNJC4riQUQVS QvjU3RhLie8ZkmVa4eIodKWyyWfg7QA3yzQle4PxDFzNx2zQ6G8YGwJZofiE/+KxbP5L TW7A7MyQ7VaeUNCyTTdoYheRfjJMaAF3pIQ9jNCLhmi5P2HTX4E1FizY0zNuj1JTn75b lYh4ec8gcoiTmPjd5O7UQ0M2cMdMRFHETBvWpqO8WN5RlJfJOboPvdQKJDbi4/cEPvGO OBOkThEKrZv9CgCPSvI1zRoonQOodejZjOeNhBIssFfUzekpiTwk5IxuxSbcuTdErviN YZnw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="fBY/C8Hp"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i190-20020a6387c7000000b0053b8f80c631si2082018pge.195.2023.05.22.05.25.43; Mon, 22 May 2023 05:25:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="fBY/C8Hp"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232890AbjEVMMT (ORCPT + 99 others); Mon, 22 May 2023 08:12:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46388 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229577AbjEVMMR (ORCPT ); Mon, 22 May 2023 08:12:17 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C38C79B for ; Mon, 22 May 2023 05:11:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684757501; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dMBJPtbzh4IjFjJ1HeY2h7yLKLc/0VkfcRg1OdFJq1c=; b=fBY/C8HpOJeKnd5SWjguBlniacaIEyMUrRRiYZ5FMqGcVN0vwSOPPrTNt4clFp8K0BE9if HOx4KSGpybXeKylY4zzpUXJS2wv22VlC1A2za25IJVrB9cvQyg9EoNRYNmwB+qzdfzqew2 2AdlM3m/ewe86QV1pWRlg3Hs7BeXCKI= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-652-UlqM_PkRPIidB5yyeKr35w-1; Mon, 22 May 2023 08:11:37 -0400 X-MC-Unique: UlqM_PkRPIidB5yyeKr35w-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D8CBB3800E87; Mon, 22 May 2023 12:11:36 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5FD694F2DC5; Mon, 22 May 2023 12:11:34 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v10 02/16] net: Pass max frags into skb_append_pagefrags() Date: Mon, 22 May 2023 13:11:11 +0100 Message-Id: <20230522121125.2595254-3-dhowells@redhat.com> In-Reply-To: <20230522121125.2595254-1-dhowells@redhat.com> References: <20230522121125.2595254-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766597180107996319?= X-GMAIL-MSGID: =?utf-8?q?1766597180107996319?= Pass the maximum number of fragments into skb_append_pagefrags() rather than using MAX_SKB_FRAGS so that it can be used from code that wants to specify sysctl_max_skb_frags. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/linux/skbuff.h | 2 +- net/core/skbuff.c | 4 ++-- net/ipv4/ip_output.c | 3 ++- net/unix/af_unix.c | 2 +- 4 files changed, 6 insertions(+), 5 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 8cff3d817131..15011408c47c 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -1383,7 +1383,7 @@ static inline int skb_pad(struct sk_buff *skb, int pad) #define dev_kfree_skb(a) consume_skb(a) int skb_append_pagefrags(struct sk_buff *skb, struct page *page, - int offset, size_t size); + int offset, size_t size, size_t max_frags); struct skb_seq_state { __u32 lower_offset; diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 6724a84ebb09..7f53dcb26ad3 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -4188,13 +4188,13 @@ unsigned int skb_find_text(struct sk_buff *skb, unsigned int from, EXPORT_SYMBOL(skb_find_text); int skb_append_pagefrags(struct sk_buff *skb, struct page *page, - int offset, size_t size) + int offset, size_t size, size_t max_frags) { int i = skb_shinfo(skb)->nr_frags; if (skb_can_coalesce(skb, i, page, offset)) { skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], size); - } else if (i < MAX_SKB_FRAGS) { + } else if (i < max_frags) { skb_zcopy_downgrade_managed(skb); get_page(page); skb_fill_page_desc_noacc(skb, i, page, offset, size); diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 61892268e8a6..52fc840898d8 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1450,7 +1450,8 @@ ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, if (len > size) len = size; - if (skb_append_pagefrags(skb, page, offset, len)) { + if (skb_append_pagefrags(skb, page, offset, len, + MAX_SKB_FRAGS)) { err = -EMSGSIZE; goto error; } diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index cc695c9f09ec..dd55506b4632 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2349,7 +2349,7 @@ static ssize_t unix_stream_sendpage(struct socket *socket, struct page *page, newskb = NULL; } - if (skb_append_pagefrags(skb, page, offset, size)) { + if (skb_append_pagefrags(skb, page, offset, size, MAX_SKB_FRAGS)) { tail = skb; goto alloc_skb; } From patchwork Mon May 22 12:11:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 97387 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1409564vqo; Mon, 22 May 2023 05:28:29 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ68SMh1Xumzw+K0g+o22euNKGWCRkXaf3dPrdSOPWasFh3TBJxJhxn6UqnXb5hOvfJkz9Fn X-Received: by 2002:a05:6a20:394a:b0:105:1b07:a6c0 with SMTP id r10-20020a056a20394a00b001051b07a6c0mr12993127pzg.52.1684758508697; Mon, 22 May 2023 05:28:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684758508; cv=none; d=google.com; s=arc-20160816; b=cSE6XWguUbPQi7mBLxz5m61LinaEWIQrvp9pShifl3XxjgYQWHWM3yFd0ZdW9bq7m0 Oq7Xm2mG47OtX0IRXwDpIGVaIAqkoSoPTHZplHjcF4jnSN3bcqqi51GA+Kdb4UStOhRk ZyluNfXiF+U2ln/YdAWFyjg050yqiOvIpWrLFOC5J9r2ISJkJH3tJGKFVTyWRyx7Z22C ltKzjEkE/5chhphSI6OWNASvaRBHwJYGFAMAwX8MrNwqHwLUavp2skw0lP3JfmeSj3Mq lTjcDpz6Az1Kbpvoci2zD//At9Nw1FPh3yMt2zDcGrtTACKBUioaOtr1Q+av9tAZYV7L MvKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Fl+2zC2cOWTcM7FjMckfIDo5NeTUJ6H1d1XKuMzp1Mk=; b=Pkeaz+r0j/jj83/MhCtyuRqaKstrcM1G0fZ5xyDUGXOK9j/6ffodss33yXKb/CUIf+ Juue6faRvrKtl2hKbYjyFga51bvcSxuJpszq9yDLgoT9i+V9Qo4SM37EgtJfhb44epiV PctD8mKapiGcXREqDlo97T5TQg1uMflQXIGzu0y3cSTh84A9W3uoKJoPCSWqvMi9H4mp Gc/7HBCtMDq68LL30yWk2DE3x5nYa21xEExe0LKhjvOLu7pokwEtvGPj0W0+hZbsot5g 8wDn0J4pL3UHXHKH5KqvU5alwcewbEWn4IBfS1IvMdCdDIDuALJermiemeVrADp+CQLf 1XYQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OLD3rVgG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i13-20020a17090acf8d00b0024799a3324dsi236732pju.162.2023.05.22.05.28.14; Mon, 22 May 2023 05:28:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OLD3rVgG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233664AbjEVMMi (ORCPT + 99 others); Mon, 22 May 2023 08:12:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46426 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232845AbjEVMMb (ORCPT ); Mon, 22 May 2023 08:12:31 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 18971B9 for ; Mon, 22 May 2023 05:11:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684757504; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fl+2zC2cOWTcM7FjMckfIDo5NeTUJ6H1d1XKuMzp1Mk=; b=OLD3rVgGbPPElNzSBQR4vEKCR2APkAP90YdwE5m9wWPqT1i7UJR1i9txorm91IuOQYETaj VBiqDeHH0XFyzOtbtHwdFBkph2tZnlkOx8+9LKmRNa1/M4FVpykCIAxe7x+v77r4/V+Yzc 1UHI7YP77Zpir1T80qfeeAAMrwuQT2w= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-569-AAbKytGoOYy6qpGKlO0Rjg-1; Mon, 22 May 2023 08:11:41 -0400 X-MC-Unique: AAbKytGoOYy6qpGKlO0Rjg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 12CA4185A78F; Mon, 22 May 2023 12:11:40 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 90F96140E95D; Mon, 22 May 2023 12:11:37 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v10 03/16] net: Add a function to splice pages into an skbuff for MSG_SPLICE_PAGES Date: Mon, 22 May 2023 13:11:12 +0100 Message-Id: <20230522121125.2595254-4-dhowells@redhat.com> In-Reply-To: <20230522121125.2595254-1-dhowells@redhat.com> References: <20230522121125.2595254-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766597338378079250?= X-GMAIL-MSGID: =?utf-8?q?1766597338378079250?= Add a function to handle MSG_SPLICE_PAGES being passed internally to sendmsg(). Pages are spliced into the given socket buffer if possible and copied in if not (e.g. they're slab pages or have a zero refcount). Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Al Viro cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #8) - Order local variables in reverse xmas tree order. - Remove duplicate coalescence check. - Warn if sendpage_ok() fails. ver #7) - Export function. - Never copy data, return -EIO if sendpage_ok() returns false. include/linux/skbuff.h | 3 ++ net/core/skbuff.c | 88 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 15011408c47c..1b2ebf6113e0 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -5097,5 +5097,8 @@ static inline void skb_mark_for_recycle(struct sk_buff *skb) #endif } +ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, + ssize_t maxsize, gfp_t gfp); + #endif /* __KERNEL__ */ #endif /* _LINUX_SKBUFF_H */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 7f53dcb26ad3..f4a5b51aed22 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -6892,3 +6892,91 @@ nodefer: __kfree_skb(skb); if (unlikely(kick) && !cmpxchg(&sd->defer_ipi_scheduled, 0, 1)) smp_call_function_single_async(cpu, &sd->defer_csd); } + +static void skb_splice_csum_page(struct sk_buff *skb, struct page *page, + size_t offset, size_t len) +{ + const char *kaddr; + __wsum csum; + + kaddr = kmap_local_page(page); + csum = csum_partial(kaddr + offset, len, 0); + kunmap_local(kaddr); + skb->csum = csum_block_add(skb->csum, csum, skb->len); +} + +/** + * skb_splice_from_iter - Splice (or copy) pages to skbuff + * @skb: The buffer to add pages to + * @iter: Iterator representing the pages to be added + * @maxsize: Maximum amount of pages to be added + * @gfp: Allocation flags + * + * This is a common helper function for supporting MSG_SPLICE_PAGES. It + * extracts pages from an iterator and adds them to the socket buffer if + * possible, copying them to fragments if not possible (such as if they're slab + * pages). + * + * Returns the amount of data spliced/copied or -EMSGSIZE if there's + * insufficient space in the buffer to transfer anything. + */ +ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, + ssize_t maxsize, gfp_t gfp) +{ + size_t frag_limit = READ_ONCE(sysctl_max_skb_frags); + struct page *pages[8], **ppages = pages; + ssize_t spliced = 0, ret = 0; + unsigned int i; + + while (iter->count > 0) { + ssize_t space, nr; + size_t off, len; + + ret = -EMSGSIZE; + space = frag_limit - skb_shinfo(skb)->nr_frags; + if (space < 0) + break; + + /* We might be able to coalesce without increasing nr_frags */ + nr = clamp_t(size_t, space, 1, ARRAY_SIZE(pages)); + + len = iov_iter_extract_pages(iter, &ppages, maxsize, nr, 0, &off); + if (len <= 0) { + ret = len ?: -EIO; + break; + } + + i = 0; + do { + struct page *page = pages[i++]; + size_t part = min_t(size_t, PAGE_SIZE - off, len); + + ret = -EIO; + if (WARN_ON_ONCE(!sendpage_ok(page))) + goto out; + + ret = skb_append_pagefrags(skb, page, off, part, + frag_limit); + if (ret < 0) { + iov_iter_revert(iter, len); + goto out; + } + + if (skb->ip_summed == CHECKSUM_NONE) + skb_splice_csum_page(skb, page, off, part); + + off = 0; + spliced += part; + maxsize -= part; + len -= part; + } while (len > 0); + + if (maxsize <= 0) + break; + } + +out: + skb_len_add(skb, spliced); + return spliced ?: ret; +} +EXPORT_SYMBOL(skb_splice_from_iter); From patchwork Mon May 22 12:11:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 97364 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1408256vqo; Mon, 22 May 2023 05:26:15 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4QMF9z5IdxbLplCQO+y0A7trNgSKoz83GNreAa/TmIYOYoECD0VM0+1Mon+QWw6n+wFRUJ X-Received: by 2002:a17:90a:9cf:b0:250:bb6:47ed with SMTP id 73-20020a17090a09cf00b002500bb647edmr9515989pjo.33.1684758375349; Mon, 22 May 2023 05:26:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684758375; cv=none; d=google.com; s=arc-20160816; b=qY5JUGX8NPLEJOnNRFWirhlsM3bbOnYPzfHp7YVv9Mb4pvvmwTMjG7vSqRU8+dpXIG CtCK8CwalfWL7FjT40MQZrAdCsb1BjRZnSCkuUXZvaahM9qJflcA1mBtCuZVBmWXIIwd 93Jm+2YqrdJk41PGkBfJwV0FVO63XA7IkfJLO1M35/rgQeVwD/7ZIblwwsGgv73mqIYl Itd28jOz++/Qkkc78bdBbNq6nRSUyEYCDFhkS1by4pR55WBTYxEGXX4MAaMKGTwJsMdT ddV3nffLeFi8J7rRpzWNj351Ol29D+csQk6MtCGEz9Efl5ptk0XBWIF5krfuw/V9CZoi a+vA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CbEUQCnChP/PzTidzzQWL6iO0yyE2e0IQp2l4Z4U9vY=; b=ze5zvqQL5aBNVPiXq2okvalT+TB9O0Qwk/lWxU95UQqRlCPFiWnrBIA8gm/13tl3a3 dJN7I79Yws9UHvi+WGL0CXq+jFDFYLmDWoLKmFdbwlrBIXjT+x3fBDFh9iS68dHZ+DTP J2UyTIjYgDI72YeT5hyJZwYyEzAbsSbyvkInRrcXQt5qKBVlh7rBiTZPK+N3jCf0krLB 87xhJwSjKZzp09YTeGBETpi9Q2iPlywk+hDZNCMK2V1UDPv9n7bzJGTaEYHD/iACi0/4 7nsEtXG2NWjuLuoTmpM2p3DdcwkBeifi0eEsoSg5WBPsmAUtrw8EyxUMyfKZTw8aXsdP LOHQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=UYi0EFZE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id oc9-20020a17090b1c0900b0024de687d6e3si2439952pjb.111.2023.05.22.05.25.56; Mon, 22 May 2023 05:26:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=UYi0EFZE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233730AbjEVMNR (ORCPT + 99 others); Mon, 22 May 2023 08:13:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46646 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232411AbjEVMNF (ORCPT ); Mon, 22 May 2023 08:13:05 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 64AE3AC for ; Mon, 22 May 2023 05:11:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684757507; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CbEUQCnChP/PzTidzzQWL6iO0yyE2e0IQp2l4Z4U9vY=; b=UYi0EFZE4QxwXjQDBqAuzaGiN+HWO0+we82QVPIXmVmRdrxCx1Lz9LGT+UIdUTqMr26iRQ xUXIOIEA8GhLAlwmCAH0YxV8NgCAajPtMQo8Y6dcMWhhsf7jnF7FubOqUV2p163WAUyYJI Ispigw0oSPdP0DXhIcSXFPOaruDkkoc= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-316-9JSwXOCGNNaJscCJXa4Eqg-1; Mon, 22 May 2023 08:11:44 -0400 X-MC-Unique: 9JSwXOCGNNaJscCJXa4Eqg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5FC812A5956E; Mon, 22 May 2023 12:11:43 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id C96347B7C; Mon, 22 May 2023 12:11:40 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v10 04/16] tcp: Support MSG_SPLICE_PAGES Date: Mon, 22 May 2023 13:11:13 +0100 Message-Id: <20230522121125.2595254-5-dhowells@redhat.com> In-Reply-To: <20230522121125.2595254-1-dhowells@redhat.com> References: <20230522121125.2595254-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766597198214350247?= X-GMAIL-MSGID: =?utf-8?q?1766597198214350247?= Make TCP's sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced or copied (if it cannot be spliced) from the source iterator. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #9) - Fix a merge conflict with commit eea96a3e2c909. ver #7) - Missed a "zc = 1" in tcp_sendmsg_locked(). ver #6) - Set zc to 0/MSG_ZEROCOPY/MSG_SPLICE_PAGES rather than 0/1/2. - Use common helper. net/ipv4/tcp.c | 43 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 3d18e295bb2f..2d61150d01f1 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1223,7 +1223,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) int flags, err, copied = 0; int mss_now = 0, size_goal, copied_syn = 0; int process_backlog = 0; - bool zc = false; + int zc = 0; long timeo; flags = msg->msg_flags; @@ -1231,7 +1231,8 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if ((flags & MSG_ZEROCOPY) && size) { if (msg->msg_ubuf) { uarg = msg->msg_ubuf; - zc = sk->sk_route_caps & NETIF_F_SG; + if (sk->sk_route_caps & NETIF_F_SG) + zc = MSG_ZEROCOPY; } else if (sock_flag(sk, SOCK_ZEROCOPY)) { skb = tcp_write_queue_tail(sk); uarg = msg_zerocopy_realloc(sk, size, skb_zcopy(skb)); @@ -1239,10 +1240,14 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) err = -ENOBUFS; goto out_err; } - zc = sk->sk_route_caps & NETIF_F_SG; - if (!zc) + if (sk->sk_route_caps & NETIF_F_SG) + zc = MSG_ZEROCOPY; + else uarg_to_msgzc(uarg)->zerocopy = 0; } + } else if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES) && size) { + if (sk->sk_route_caps & NETIF_F_SG) + zc = MSG_SPLICE_PAGES; } if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect) && @@ -1305,7 +1310,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) goto do_error; while (msg_data_left(msg)) { - int copy = 0; + ssize_t copy = 0; skb = tcp_write_queue_tail(sk); if (skb) @@ -1346,7 +1351,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (copy > msg_data_left(msg)) copy = msg_data_left(msg); - if (!zc) { + if (zc == 0) { bool merge = true; int i = skb_shinfo(skb)->nr_frags; struct page_frag *pfrag = sk_page_frag(sk); @@ -1391,7 +1396,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) page_ref_inc(pfrag->page); } pfrag->offset += copy; - } else { + } else if (zc == MSG_ZEROCOPY) { /* First append to a fragless skb builds initial * pure zerocopy skb */ @@ -1412,6 +1417,30 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (err < 0) goto do_error; copy = err; + } else if (zc == MSG_SPLICE_PAGES) { + /* Splice in data if we can; copy if we can't. */ + if (tcp_downgrade_zcopy_pure(sk, skb)) + goto wait_for_space; + copy = tcp_wmem_schedule(sk, copy); + if (!copy) + goto wait_for_space; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) { + if (err == -EMSGSIZE) { + tcp_mark_push(tp, skb); + goto new_segment; + } + goto do_error; + } + copy = err; + + if (!(flags & MSG_NO_SHARED_FRAGS)) + skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; + + sk_wmem_queued_add(sk, copy); + sk_mem_charge(sk, copy); } if (!copied) From patchwork Mon May 22 12:11:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 97382 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1409348vqo; Mon, 22 May 2023 05:28:06 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ75BtcZ/gwWj6KOu9X9CyEsLvwpgmQ5CBtmyreYYTSUTImZpLAz6Usyqdu4qXXRIGz/WT7j X-Received: by 2002:a17:903:11cd:b0:1a6:84be:a08f with SMTP id q13-20020a17090311cd00b001a684bea08fmr11268146plh.64.1684758486438; Mon, 22 May 2023 05:28:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684758486; cv=none; d=google.com; s=arc-20160816; b=lNjJ14yqaBjN4SoEbMkSkj4eL9yMz3J7rBu0K9uwt+fB2wDkgusD3d0D7Nu4Vfdred dW12p0+ZBDFXYujCDZMyHuUiMpuZ62CETFkUzVnYlpiOFQPYR8UjB+PFKzHlX25rhEwc 6Inibu2IYcj/urQkRZNbFIcJgHCKmFiXUtjBHzZYDtU79buH88L110+iCKwKAgYvtoQO barY2Up2doiWvnj5za4oT4zfZYBig0f+gsNlqhIO/l5NJHiuYPBELmlmyTVwiYtsKkzb giM6iQvS6ewe7+f9LYCdTARUVPQa0R4k6mVGKx0jf9h67oVzG1tsZoURSAvK8TkzP7OL KPCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=79TjdqGEZWzMR9284kh0SsbKnhgq/mMmLDGgzzPhuR4=; b=ReRmThl29fR/c0YMd3lmCYvaaskFmvKv2ZDoA8If0vgFtVKfbb8p3IVZUsQkvzrTnK lormLlU8UnWmW74q9ejlulAzVwtv8VrGfiOD+T9sWxqP7GC3vjpAQvJf87Su7mPt2VMn edeVotpPN6DMdjT0BHgvvw9n+v9HZEqbBewgigUOIgq+uXuT7KE+BrNhM2FamezsqQNz z6pnhyY4WfNZAVDR+txopf5S6HY+3s2/+v2P5xsI2aEUWpn33vtv5kOURGF6YW0SMZI2 UBGbOz6yuV87fGWkVs9mjkwArMo//YZXkxaNkSFwdbeuNRVndhCDue9WwiXWCjf/BF4b wTOA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=TSjUuf9o; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l2-20020a170902ec0200b0019e9c8e51f4si4327245pld.165.2023.05.22.05.27.53; Mon, 22 May 2023 05:28:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=TSjUuf9o; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229987AbjEVMNV (ORCPT + 99 others); Mon, 22 May 2023 08:13:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230312AbjEVMNI (ORCPT ); Mon, 22 May 2023 08:13:08 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 85D95BF for ; Mon, 22 May 2023 05:11:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684757509; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=79TjdqGEZWzMR9284kh0SsbKnhgq/mMmLDGgzzPhuR4=; b=TSjUuf9oNzSeS0ef1c0KwwMkY4D0pvdIwPbA1CmuNOuNzHacK0RodGJLUL5aNvdWcoG5ru 2UYNKHX7fubNkLREmZwGQTYzXyZ7MBMYtA7YYIwYwmyINae3W7qpX/QhKawzTPF4mPMGN/ Kfk51xdBAooLtod/+6dcqavAuoxWvW4= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-13-s8VTVl0_ND6DLvjr5Pd7CA-1; Mon, 22 May 2023 08:11:48 -0400 X-MC-Unique: s8VTVl0_ND6DLvjr5Pd7CA-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 87AEA3800E87; Mon, 22 May 2023 12:11:46 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 14AA7492B0A; Mon, 22 May 2023 12:11:43 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v10 05/16] tcp: Convert do_tcp_sendpages() to use MSG_SPLICE_PAGES Date: Mon, 22 May 2023 13:11:14 +0100 Message-Id: <20230522121125.2595254-6-dhowells@redhat.com> In-Reply-To: <20230522121125.2595254-1-dhowells@redhat.com> References: <20230522121125.2595254-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766597314288937871?= X-GMAIL-MSGID: =?utf-8?q?1766597314288937871?= Convert do_tcp_sendpages() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. do_tcp_sendpages() can then be inlined in subsequent patches into its callers. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 158 +++---------------------------------------------- 1 file changed, 7 insertions(+), 151 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 2d61150d01f1..f3a0c02678e0 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -974,163 +974,19 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -static struct sk_buff *tcp_build_frag(struct sock *sk, int size_goal, int flags, - struct page *page, int offset, size_t *size) -{ - struct sk_buff *skb = tcp_write_queue_tail(sk); - struct tcp_sock *tp = tcp_sk(sk); - bool can_coalesce; - int copy, i; - - if (!skb || (copy = size_goal - skb->len) <= 0 || - !tcp_skb_can_collapse_to(skb)) { -new_segment: - if (!sk_stream_memory_free(sk)) - return NULL; - - skb = tcp_stream_alloc_skb(sk, 0, sk->sk_allocation, - tcp_rtx_and_write_queues_empty(sk)); - if (!skb) - return NULL; - -#ifdef CONFIG_TLS_DEVICE - skb->decrypted = !!(flags & MSG_SENDPAGE_DECRYPTED); -#endif - tcp_skb_entail(sk, skb); - copy = size_goal; - } - - if (copy > *size) - copy = *size; - - i = skb_shinfo(skb)->nr_frags; - can_coalesce = skb_can_coalesce(skb, i, page, offset); - if (!can_coalesce && i >= READ_ONCE(sysctl_max_skb_frags)) { - tcp_mark_push(tp, skb); - goto new_segment; - } - if (tcp_downgrade_zcopy_pure(sk, skb)) - return NULL; - - copy = tcp_wmem_schedule(sk, copy); - if (!copy) - return NULL; - - if (can_coalesce) { - skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); - } else { - get_page(page); - skb_fill_page_desc_noacc(skb, i, page, offset, copy); - } - - if (!(flags & MSG_NO_SHARED_FRAGS)) - skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; - - skb->len += copy; - skb->data_len += copy; - skb->truesize += copy; - sk_wmem_queued_add(sk, copy); - sk_mem_charge(sk, copy); - WRITE_ONCE(tp->write_seq, tp->write_seq + copy); - TCP_SKB_CB(skb)->end_seq += copy; - tcp_skb_pcount_set(skb, 0); - - *size = copy; - return skb; -} - ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tcp_sock *tp = tcp_sk(sk); - int mss_now, size_goal; - int err; - ssize_t copied; - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); - - if (IS_ENABLED(CONFIG_DEBUG_VM) && - WARN_ONCE(!sendpage_ok(page), - "page must not be a Slab one and have page_count > 0")) - return -EINVAL; - - /* Wait for a connection to finish. One exception is TCP Fast Open - * (passive side) where data is allowed to be sent before a connection - * is fully established. - */ - if (((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) && - !tcp_passive_fastopen(sk)) { - err = sk_stream_wait_connect(sk, &timeo); - if (err != 0) - goto out_err; - } + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - mss_now = tcp_send_mss(sk, &size_goal, flags); - copied = 0; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - err = -EPIPE; - if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) - goto out_err; - - while (size > 0) { - struct sk_buff *skb; - size_t copy = size; - - skb = tcp_build_frag(sk, size_goal, flags, page, offset, ©); - if (!skb) - goto wait_for_space; - - if (!copied) - TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; - - copied += copy; - offset += copy; - size -= copy; - if (!size) - goto out; - - if (skb->len < size_goal || (flags & MSG_OOB)) - continue; - - if (forced_push(tp)) { - tcp_mark_push(tp, skb); - __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); - } else if (skb == tcp_send_head(sk)) - tcp_push_one(sk, mss_now); - continue; - -wait_for_space: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); - tcp_push(sk, flags & ~MSG_MORE, mss_now, - TCP_NAGLE_PUSH, size_goal); - - err = sk_stream_wait_memory(sk, &timeo); - if (err != 0) - goto do_error; - - mss_now = tcp_send_mss(sk, &size_goal, flags); - } - -out: - if (copied) { - tcp_tx_timestamp(sk, sk->sk_tsflags); - if (!(flags & MSG_SENDPAGE_NOTLAST)) - tcp_push(sk, flags, mss_now, tp->nonagle, size_goal); - } - return copied; - -do_error: - tcp_remove_empty_skb(sk); - if (copied) - goto out; -out_err: - /* make sure we wake any epoll edge trigger waiter */ - if (unlikely(tcp_rtx_and_write_queues_empty(sk) && err == -EAGAIN)) { - sk->sk_write_space(sk); - tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED); - } - return sk_stream_error(sk, flags, err); + return tcp_sendmsg_locked(sk, &msg, size); } EXPORT_SYMBOL_GPL(do_tcp_sendpages); From patchwork Mon May 22 12:11:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 97351 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1401813vqo; Mon, 22 May 2023 05:15:38 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6tlUGV1NxB8F/eeOKA1j9PlS2R/oNFWVUS4K2x9cDxC3H+SvgPTcPIjLDaovXy6V4m7+kN X-Received: by 2002:a05:6a00:134c:b0:64d:2b06:5949 with SMTP id k12-20020a056a00134c00b0064d2b065949mr14364081pfu.8.1684757738252; Mon, 22 May 2023 05:15:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684757738; cv=none; d=google.com; s=arc-20160816; b=V3ZTZewOFQ3l2/Cm+XSKJsFICn25CyvPAcbNccKYvEgWMTNlE4oC4saaI8CO5wI9t/ 9g7yS9nmjKq41dYqWITLD6gd+rkaVDND5BjLuogsEQlQoT2G6klDtCfwPcjPjNBvR4F6 bnCLz1A2etoQq4Fggoh5yGDEfMyrg1/iMeK5e/hT04/iGVsctZElp+0IfW3uwflIcLcX En3hCdg3RWXYvw1S51zMDb6gMZKQ/k38rA/YlR7BVDEQatJsFe5XAeRzUhvfriJy+6Im zcx2al2GPAibDBDlVj7sQ8qkLnRT03lfiTsy8hUPr0UNMb+u9PXKUJKKfLYPFN+sxphy dGoA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2EXLlBUI8HMrlq1LdzuePmnE9vo0jChMg019VHUJEgs=; b=FOEaUAlTxdd4S7fIvlSnyGyiuBq2BRfRPkQ4iSEud8Pbuq1U+70Zga3UZoZpN3JPaT pTYy77P7N3Cz8Y1k40nNTKV7/E4uKgZwnULWXjm0ez/jwt7VaSDNJdBhwPQPiMpn7V/P SrZeaqidY1F4xSNV5Dws+vZEgWz2IvDIYE7K0LthmYyF3B8OQXT8nNrsYLIcoku93BZB THVrf6MLj5ytpxlhVNZl6dEU47OuKVDd2o6lX+BzOPj1+cQoBQy6hd2Jt/61z6XXhSFx fCoxhg/hpuJiqdfeh1+IYsM2L8DSuU6sTvgFpnXCHml5pU9+mJhGA8JZGU6lT74nA5vh D2Pw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=e8SDArFG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y3-20020aa79ae3000000b00643aa8d2548si471342pfp.120.2023.05.22.05.15.23; Mon, 22 May 2023 05:15:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=e8SDArFG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233783AbjEVMNm (ORCPT + 99 others); Mon, 22 May 2023 08:13:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46752 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232133AbjEVMNN (ORCPT ); Mon, 22 May 2023 08:13:13 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F5C2CE for ; Mon, 22 May 2023 05:11:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684757516; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2EXLlBUI8HMrlq1LdzuePmnE9vo0jChMg019VHUJEgs=; b=e8SDArFGbawiESCCnOF3nppfyh+Ho2U5yL0I1J1mHKCnuslG4fsujcmxTOA+whwCeu6Wrp DzzYiq5w00N5dOzqaVuaZAiG5frNOcU06XR4AxeOuin7WR3PBvzn2sD2t+3BUKB70STk0c ChZUU4R3UbENqzL40KuGZeyYEphSOmE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-447-fpogOg9rPfSKKoQ1z-hkZA-1; Mon, 22 May 2023 08:11:51 -0400 X-MC-Unique: fpogOg9rPfSKKoQ1z-hkZA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 298F3802355; Mon, 22 May 2023 12:11:50 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5AD417C52; Mon, 22 May 2023 12:11:47 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Fastabend , Jakub Sitnicki , bpf@vger.kernel.org Subject: [PATCH net-next v10 06/16] tcp_bpf: Inline do_tcp_sendpages as it's now a wrapper around tcp_sendmsg Date: Mon, 22 May 2023 13:11:15 +0100 Message-Id: <20230522121125.2595254-7-dhowells@redhat.com> In-Reply-To: <20230522121125.2595254-1-dhowells@redhat.com> References: <20230522121125.2595254-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766596530192572499?= X-GMAIL-MSGID: =?utf-8?q?1766596530192572499?= do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: John Fastabend cc: Jakub Sitnicki cc: "David S. Miller" cc: Eric Dumazet cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: bpf@vger.kernel.org --- net/ipv4/tcp_bpf.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c index 2e9547467edb..0291d15acd19 100644 --- a/net/ipv4/tcp_bpf.c +++ b/net/ipv4/tcp_bpf.c @@ -72,11 +72,13 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, { bool apply = apply_bytes; struct scatterlist *sge; + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct page *page; int size, ret = 0; u32 off; while (1) { + struct bio_vec bvec; bool has_tx_ulp; sge = sk_msg_elem(msg, msg->sg.start); @@ -88,16 +90,18 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, tcp_rate_check_app_limited(sk); retry: has_tx_ulp = tls_sw_has_ctx_tx(sk); - if (has_tx_ulp) { - flags |= MSG_SENDPAGE_NOPOLICY; - ret = kernel_sendpage_locked(sk, - page, off, size, flags); - } else { - ret = do_tcp_sendpages(sk, page, off, size, flags); - } + if (has_tx_ulp) + msghdr.msg_flags |= MSG_SENDPAGE_NOPOLICY; + if (flags & MSG_SENDPAGE_NOTLAST) + msghdr.msg_flags |= MSG_MORE; + + bvec_set_page(&bvec, page, size, off); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret <= 0) return ret; + if (apply) apply_bytes -= ret; msg->sg.size -= ret; @@ -404,7 +408,7 @@ static int tcp_bpf_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) long timeo; int flags; - /* Don't let internal do_tcp_sendpages() flags through */ + /* Don't let internal sendpage flags through */ flags = (msg->msg_flags & ~MSG_SENDPAGE_DECRYPTED); flags |= MSG_NO_SHARED_FRAGS; From patchwork Mon May 22 12:11:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 97383 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1409471vqo; Mon, 22 May 2023 05:28:20 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4A+oV4B5HJq/JRWWNzb1R70qo/gI5BzM+AqNjUxeGKbGVHW1KOyh24wRsgn3IQBJosLvWq X-Received: by 2002:a05:6a21:7889:b0:10b:e88f:5983 with SMTP id bf9-20020a056a21788900b0010be88f5983mr1909050pzc.43.1684758500233; Mon, 22 May 2023 05:28:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684758500; cv=none; d=google.com; s=arc-20160816; b=NpkQcfhAR3YTJjBLqemtpKeeVfQYg0YXPkS6mw/lEnC2WtlEjzspw2XCVttj+QdaWA I15mJ/aOcBMZVk6kVJo91SVOO234HhH9qOll+eyuoZ3Ib9+fH7noS2yH5FyWpM12JH3f jLf1i31nNv0VDp0FYAnr//2wyjmrHRBOJURNVR+Zq5snNw0Z8fltkQXCo1OEcdogvvba Je4aOOvMOko31bObXMWD1Kq076b7HhURaWNIVcCPKx18BKFV+im/Mv3fp76FadHjtulx FFiz1hlxIvPjh/Fdm3LesyW1pm87r3enpSTf0+NV94R27RmyVLAMPhbbjdIQv3J0Xt6w iOfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=tYeoTX9B8dad49qWPHePVLYzX6oyR1jBlVymL6Q5Zsc=; b=aW/KVhcbr2/YYs851lKZ/esWZLjs+QgYL7O/pMSjI2wIB7Sbj2PCTscqpKU5vgVVxi DDnhDny1foOKa64ny/F23eK+Yj3yOIzD0pevvvyVEOATc5Wt5hZ1zRvxhioQuG6dJNv9 2W7k4iTmeUTBmDzn1z7Jx+7acRk5Mo/2L2XEhQ4l9UXL7IeltUF6Oouf7qbzXks2swMG 0WuqF+5KnpOFVjIL1no0nOccyG+MlNGFUqDvlDdeTSahuopjF3wNTDwnefuTzXn+Utia fiLPCAB1ZHBt+hrygvCBzIWUWL8e/t+a0oJFry4uSz9WLavGy9jPT/Kq/V3h8ChtZP8g ekGw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=U+3QQm+v; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b5-20020aa78ec5000000b0064d6ca382a1si1827306pfr.342.2023.05.22.05.28.07; Mon, 22 May 2023 05:28:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=U+3QQm+v; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230051AbjEVMNk (ORCPT + 99 others); Mon, 22 May 2023 08:13:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233469AbjEVMNO (ORCPT ); Mon, 22 May 2023 08:13:14 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BD55FDC for ; Mon, 22 May 2023 05:11:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684757519; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tYeoTX9B8dad49qWPHePVLYzX6oyR1jBlVymL6Q5Zsc=; b=U+3QQm+v312ByLRBWyREcfENqCS583v+cFK+oa83ifmyvdb62SuqMx9uS3/FaejO7F+PKC eFTyVYJ2SZsIEG/YYiFZ6Ie1iKYGz/fb/x7EzL75GbQn6O3wgPbit16hd/VUMmXgM3bkZ0 rIMqJw2CoVrCBUoa8bqhKXrhWZBbJSM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-329-z_k6b1dFNWyCPfMJlHKjGw-1; Mon, 22 May 2023 08:11:55 -0400 X-MC-Unique: z_k6b1dFNWyCPfMJlHKjGw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 096DF811E8F; Mon, 22 May 2023 12:11:54 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id D9261C54F80; Mon, 22 May 2023 12:11:50 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Steffen Klassert , Herbert Xu Subject: [PATCH net-next v10 07/16] espintcp: Inline do_tcp_sendpages() Date: Mon, 22 May 2023 13:11:16 +0100 Message-Id: <20230522121125.2595254-8-dhowells@redhat.com> In-Reply-To: <20230522121125.2595254-1-dhowells@redhat.com> References: <20230522121125.2595254-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766597329060370103?= X-GMAIL-MSGID: =?utf-8?q?1766597329060370103?= do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Steffen Klassert cc: Herbert Xu cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/xfrm/espintcp.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/net/xfrm/espintcp.c b/net/xfrm/espintcp.c index 872b80188e83..3504925babdb 100644 --- a/net/xfrm/espintcp.c +++ b/net/xfrm/espintcp.c @@ -205,14 +205,16 @@ static int espintcp_sendskb_locked(struct sock *sk, struct espintcp_msg *emsg, static int espintcp_sendskmsg_locked(struct sock *sk, struct espintcp_msg *emsg, int flags) { + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct sk_msg *skmsg = &emsg->skmsg; struct scatterlist *sg; int done = 0; int ret; - flags |= MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags |= MSG_SENDPAGE_NOTLAST; sg = &skmsg->sg.data[skmsg->sg.start]; do { + struct bio_vec bvec; size_t size = sg->length - emsg->offset; int offset = sg->offset + emsg->offset; struct page *p; @@ -220,11 +222,13 @@ static int espintcp_sendskmsg_locked(struct sock *sk, emsg->offset = 0; if (sg_is_last(sg)) - flags &= ~MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags &= ~MSG_SENDPAGE_NOTLAST; p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret < 0) { emsg->offset = offset - sg->offset; skmsg->sg.start += done; From patchwork Mon May 22 12:11:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 97352 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1401926vqo; Mon, 22 May 2023 05:15:45 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5i9TTYQ5mvRcAcQDQrOw6GKy2mj5rP8T2vr3kfbNz2MMNckT4iEepybse0ZjAU3kOU/CGY X-Received: by 2002:a17:90b:1d03:b0:24d:fba9:80e9 with SMTP id on3-20020a17090b1d0300b0024dfba980e9mr10929674pjb.23.1684757745460; Mon, 22 May 2023 05:15:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684757745; cv=none; d=google.com; s=arc-20160816; b=VADeiD39vLnzrYm34CEkKo9N2elH4iLF3ez4KXuLt/psuCrXlwm3/zssqHxQ+rYThi OhUllvVJT0WIw2bDbFG4gvLWyuP0NHATe6CYbStRVavShPxS+5Dv7YO/86cJUYIz+X6A Vj4t3HKzULzh/nJuzvKMoU6jO001iektIx5F/IBblaoVVEJZWlFFBAySjuBTmToXNodz ZuaVkCUfUaeaSJaEnDTKZE8Xyi9pqjdrD4DqwrxSwgpIXGxXFCdGdpp02Jj88mq+wHIX PYIOZSdXXA9Y4cBt02k2sWV0qm0tZdEHH2BQDnBkfmpZ/rCMMQN+K/LLekHhnweARB1J Pg2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JUrQ0rsOJEKe2VauAy4RSjWfQDMPveXienxXCaL8A+w=; b=SIA46/AePAIg1HVit1zNhmIRD0QpS6dB0zQ9mm7n5hDnACQtX1Df6Wakyr0MuyKfic ygeHvPDQIX1lJprCRTdE0xwxrrTnkwFJRWbwj/a/EZyzf0sprqcdn4bO8vD09VA5LzUJ Qe+2U+1AugV3vtTxMKUsQn+BLcnjPYyi1pFVvyquTn+IqRqSIusETsZ08cDm7Dz1CfgW WJReEHhFqyQucWA1zXPAZa+ZpggWmiCN4pQRWIBlzI8dNkH3/hwaA7ZIOZdxkA9oCuY6 kz28KIf+O123i50c3frVOmVcHJuplJANV66++vyG9IH+THBJfTZaE3pzj9muejf931wM j3TQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=BpgwDkDc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v6-20020a17090abb8600b00246b1a9630csi2611257pjr.130.2023.05.22.05.15.30; Mon, 22 May 2023 05:15:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=BpgwDkDc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233791AbjEVMNq (ORCPT + 99 others); Mon, 22 May 2023 08:13:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233714AbjEVMNQ (ORCPT ); Mon, 22 May 2023 08:13:16 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D27DED for ; Mon, 22 May 2023 05:12:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684757523; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JUrQ0rsOJEKe2VauAy4RSjWfQDMPveXienxXCaL8A+w=; b=BpgwDkDcl3Pndc3JOHamlETALJg+zqoL+69qT4hkl/XqiDAWZ2iiG2oHKjR+WY/9KdyINf shBEqcOkiudMaIcNAcSlYxtEPAxa40kyefxsZI8BLu2zdbpBBIzI9/dCmSX0C7Sy8H9PpC 0/VRJvFGCiue2puCvYAVXU9vBc2X4xQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-587-GZXCZZECNuaQheY_clprmg-1; Mon, 22 May 2023 08:12:00 -0400 X-MC-Unique: GZXCZZECNuaQheY_clprmg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 73454802A55; Mon, 22 May 2023 12:11:59 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id B74D740E6A4E; Mon, 22 May 2023 12:11:56 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Boris Pismenny , John Fastabend Subject: [PATCH net-next v10 08/16] tls: Inline do_tcp_sendpages() Date: Mon, 22 May 2023 13:11:17 +0100 Message-Id: <20230522121125.2595254-9-dhowells@redhat.com> In-Reply-To: <20230522121125.2595254-1-dhowells@redhat.com> References: <20230522121125.2595254-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766596537878051484?= X-GMAIL-MSGID: =?utf-8?q?1766596537878051484?= do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: "David S. Miller" cc: Eric Dumazet cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org Signed-off-by: Jakub Kicinski Tested-by: Tariq Toukan --- include/net/tls.h | 2 +- net/tls/tls_main.c | 24 +++++++++++++++--------- 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/include/net/tls.h b/include/net/tls.h index 6056ce5a2aa5..5791ca7a189c 100644 --- a/include/net/tls.h +++ b/include/net/tls.h @@ -258,7 +258,7 @@ struct tls_context { struct scatterlist *partially_sent_record; u16 partially_sent_offset; - bool in_tcp_sendpages; + bool splicing_pages; bool pending_open_record_frags; struct mutex tx_lock; /* protects partially_sent_* fields and diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c index f2e7302a4d96..3d45fdb5c4e9 100644 --- a/net/tls/tls_main.c +++ b/net/tls/tls_main.c @@ -125,7 +125,10 @@ int tls_push_sg(struct sock *sk, u16 first_offset, int flags) { - int sendpage_flags = flags | MSG_SENDPAGE_NOTLAST; + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SENDPAGE_NOTLAST | MSG_SPLICE_PAGES | flags, + }; int ret = 0; struct page *p; size_t size; @@ -134,16 +137,19 @@ int tls_push_sg(struct sock *sk, size = sg->length - offset; offset += sg->offset; - ctx->in_tcp_sendpages = true; + ctx->splicing_pages = true; while (1) { if (sg_is_last(sg)) - sendpage_flags = flags; + msg.msg_flags = flags; /* is sending application-limited? */ tcp_rate_check_app_limited(sk); p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, sendpage_flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + + ret = tcp_sendmsg_locked(sk, &msg, size); if (ret != size) { if (ret > 0) { @@ -155,7 +161,7 @@ int tls_push_sg(struct sock *sk, offset -= sg->offset; ctx->partially_sent_offset = offset; ctx->partially_sent_record = (void *)sg; - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return ret; } @@ -169,7 +175,7 @@ int tls_push_sg(struct sock *sk, size = sg->length; } - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return 0; } @@ -247,11 +253,11 @@ static void tls_write_space(struct sock *sk) { struct tls_context *ctx = tls_get_ctx(sk); - /* If in_tcp_sendpages call lower protocol write space handler + /* If splicing_pages call lower protocol write space handler * to ensure we wake up any waiting operations there. For example - * if do_tcp_sendpages where to call sk_wait_event. + * if splicing pages where to call sk_wait_event. */ - if (ctx->in_tcp_sendpages) { + if (ctx->splicing_pages) { ctx->sk_write_space(sk); return; } From patchwork Mon May 22 12:11:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 97393 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1410011vqo; Mon, 22 May 2023 05:29:10 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6DmqfmpbOhU+OaeIR2xJSRuC7s/Rz1cC1wsd89WYagsi4C4E9bWKeM/yfZ7Q/BeAmQ/NfD X-Received: by 2002:a05:6a00:2484:b0:64d:5c4c:7e20 with SMTP id c4-20020a056a00248400b0064d5c4c7e20mr6453091pfv.26.1684758549788; Mon, 22 May 2023 05:29:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684758549; cv=none; d=google.com; s=arc-20160816; b=voUW+/JEnJpdUWxazuISwVWaZH6PlTbVVKKSQMqjSiJ2tkcwLQDtrQJU1nOv4KstTM 3s3pX4MQoaqX3lreBt/bDcdPaiGDFP40s0BhrkPxBGO9HrgV9FJeQ75soIzboRGYM7oc VBTndu/K9GNv0/FRUI0U1G3aIYPRRgD2mxbJkLfAa6V7zK4DDgJO8WgXvURHyxs+kUWm z0OQQBqUqO7LPCEaxADhnS2/MyU1y0Az2IIp0hp5DmTXUHkCofZqTR9GwvwJMclflREu We1csimYr/FAsAd0GAKwlEjFAOfUsWOobYEw5wCgDHgionT+n53UTKQn1abEk2QErT4t kfsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=tEe6KsGIUg5Ug9eymZUK4cq5L1lxhWlwpIXxPsGo3ck=; b=Ngj92XBxdi0dOAhXhdz3bfkhulUPEE2Lmfxo9NB3NK+CWtTCaPs6BhrUFu+rVmMEED VNUuHuMXuEwxbxTWwKoIRgZhQ8T7bO5brdcHtgPmbU4DcZzyglCBgWsy0Eirhr1KOLdP CsujJgsKyeLeJAbJMBPFAMeuavYCDel/u4i6CgKJrXCMEZSDnIrIwWk+N4K+XPVdPAEU 5/PR62RrJRi7z42MHM9+Pg2ps15Ipkd0iBPCUYuajxB8NmgvVGZPox9QqNywaPxCSQU1 zelcG7tll+e2T5q7n9CBvJf3zjl5WE9d2h/lYRml7zMrH8AWODL7b95WIDp8K/y4l+91 aMRg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=LFjjVKi4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e30-20020a63745e000000b0050bf22172d3si610578pgn.490.2023.05.22.05.28.57; Mon, 22 May 2023 05:29:09 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=LFjjVKi4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233483AbjEVMNh (ORCPT + 99 others); Mon, 22 May 2023 08:13:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46786 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232966AbjEVMNM (ORCPT ); Mon, 22 May 2023 08:13:12 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B576FA for ; Mon, 22 May 2023 05:12:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684757528; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tEe6KsGIUg5Ug9eymZUK4cq5L1lxhWlwpIXxPsGo3ck=; b=LFjjVKi46ASWKBNz4jZ9s06cwiSmNcmkCaHROR8tTjXRUxYCpittLDSGYI+MOW0rew5eUj OwxxzNos3P7D8Qe/v5QJGOInjelKQBEuUNDRVJPp/KtJPaDUgk4CDRnoL7Ji+KKx5XCYz5 6ZCoX0grfDTBg427BrpK6Y4bibbeK0Y= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-474-MWYfU8vlNOi-xXlJkHtgaw-1; Mon, 22 May 2023 08:12:04 -0400 X-MC-Unique: MWYfU8vlNOi-xXlJkHtgaw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6F58E858F14; Mon, 22 May 2023 12:12:03 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2BC277B7C; Mon, 22 May 2023 12:12:00 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Bernard Metzler , Tom Talpey , Jason Gunthorpe , Leon Romanovsky , linux-rdma@vger.kernel.org Subject: [PATCH net-next v10 09/16] siw: Inline do_tcp_sendpages() Date: Mon, 22 May 2023 13:11:18 +0100 Message-Id: <20230522121125.2595254-10-dhowells@redhat.com> In-Reply-To: <20230522121125.2595254-1-dhowells@redhat.com> References: <20230522121125.2595254-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766597381202774562?= X-GMAIL-MSGID: =?utf-8?q?1766597381202774562?= do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells Reviewed-by: Bernard Metzler Reviewed-by: Tom Talpey cc: Jason Gunthorpe cc: Leon Romanovsky cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: netdev@vger.kernel.org --- Notes: ver #6) - Don't clear MSG_SPLICE_PAGES on the last page. drivers/infiniband/sw/siw/siw_qp_tx.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_qp_tx.c b/drivers/infiniband/sw/siw/siw_qp_tx.c index 4b292e0504f1..ffb16beb6c30 100644 --- a/drivers/infiniband/sw/siw/siw_qp_tx.c +++ b/drivers/infiniband/sw/siw/siw_qp_tx.c @@ -312,7 +312,7 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, } /* - * 0copy TCP transmit interface: Use do_tcp_sendpages. + * 0copy TCP transmit interface: Use MSG_SPLICE_PAGES. * * Using sendpage to push page by page appears to be less efficient * than using sendmsg, even if data are copied. @@ -323,20 +323,27 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, static int siw_tcp_sendpages(struct socket *s, struct page **page, int offset, size_t size) { + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = (MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST | + MSG_SPLICE_PAGES), + }; struct sock *sk = s->sk; - int i = 0, rv = 0, sent = 0, - flags = MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST; + int i = 0, rv = 0, sent = 0; while (size) { size_t bytes = min_t(size_t, PAGE_SIZE - offset, size); if (size + offset <= PAGE_SIZE) - flags = MSG_MORE | MSG_DONTWAIT; + msg.msg_flags &= ~MSG_SENDPAGE_NOTLAST; tcp_rate_check_app_limited(sk); + bvec_set_page(&bvec, page[i], bytes, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + try_page_again: lock_sock(sk); - rv = do_tcp_sendpages(sk, page[i], offset, bytes, flags); + rv = tcp_sendmsg_locked(sk, &msg, size); release_sock(sk); if (rv > 0) { From patchwork Mon May 22 12:11:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 97365 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1408294vqo; Mon, 22 May 2023 05:26:19 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7NyBD/6TcUcoZ1L94/inZI6bsM1Q5oyouPaPv4iHApIiOd4+3YS3Y1hiuSHS8ppElNYyRh X-Received: by 2002:a17:90b:1295:b0:255:47fd:9ab2 with SMTP id fw21-20020a17090b129500b0025547fd9ab2mr4851381pjb.12.1684758379455; Mon, 22 May 2023 05:26:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684758379; cv=none; d=google.com; s=arc-20160816; b=aDKqGbMrswHkmDTzr2Iqb8s5IKrnX/SfLgJrTdCCLcIBs/In/vvXbdtsIZhIrcYXeW JbQkJHmhblaViE93FxqEoonEYFhkDUzFVgIh2E0ofiu8oolVd6uB5hCk8MYEP7wTHBZX ctq+VfdA27aXEibAJOVp0eU0QX7TZKAaPNPGJmEj3oxPhMwYRFKZ94XR8VT1BIs9NGEj 0oF8wSPelnDlW/0E0vnDI7jvj0Jm7Z8Jp/B4HIgHY4DSPJ0za8h4pKvO+OXI7I92hGhQ KfMlUdQSFyP6H5c9u1hKvEwB8NsEA9x6nwzjYzroGyTKUipP9RFFIM4F+wa+Cd1qGgZQ CHbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=FkLIsuWcxLdm5m/RLKw/6bFEvudw3Dbf8/jbJKlIa0s=; b=SMVXLr2BZTFuEqdTNAe0SzAhgV0T2n5Xje1KVRg/0kLplDNf5ntAwVuItpLenVz4Lp ioLgc6KPvvPoQ6LRiIHnVaUr/g0IwX3ozqyckRaGVzrbZ+fC27fvfox51d5NHD4MWfGr +CuKiqVwKL4qYhsdMAkWMGr7kMmWany2Sf9tNpx4hjAlrGY2Rt6wwocyV0H2DETzvwul SSOXy5ByTWCt/HSPGa7VSw31mjLdusaApi6pD4WGROe0uvORuHtl7VKVs8KD3OjGk5He 7WJ9bXvLXG9KGVCDz7vOjeMWM3U9ePwptvVeuxVnISNUAPL0nZHMPE6BVPpRA5QGwB49 hRCw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=SFO9r++p; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b30-20020a63931e000000b005348f910e65si4616937pge.824.2023.05.22.05.26.06; Mon, 22 May 2023 05:26:19 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=SFO9r++p; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232305AbjEVMNu (ORCPT + 99 others); Mon, 22 May 2023 08:13:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46402 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232741AbjEVMNQ (ORCPT ); Mon, 22 May 2023 08:13:16 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D55ABFF for ; Mon, 22 May 2023 05:12:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684757535; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FkLIsuWcxLdm5m/RLKw/6bFEvudw3Dbf8/jbJKlIa0s=; b=SFO9r++p9NbqLMvSoZKfVcB0EuNtjTD7ik+MhoCTR3sU/axb4tkPwIaMYQL79PpUz+Xa19 gGV9YXrNiHnR+vVLOdW1aFb3TN1XuX2tAo8jJyJxkxJdZ+OZCF9ipr5Uy4U3ITV5Xle6zB wCMIAvNF0VSVuBAfwhXjYAuNBO2nNqg= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-54-cIwQpXAwOju38rvcwCBp5g-1; Mon, 22 May 2023 08:12:07 -0400 X-MC-Unique: cIwQpXAwOju38rvcwCBp5g-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CCA9C3800E8C; Mon, 22 May 2023 12:12:06 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 57ECF7B7C; Mon, 22 May 2023 12:12:04 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v10 10/16] tcp: Fold do_tcp_sendpages() into tcp_sendpage_locked() Date: Mon, 22 May 2023 13:11:19 +0100 Message-Id: <20230522121125.2595254-11-dhowells@redhat.com> In-Reply-To: <20230522121125.2595254-1-dhowells@redhat.com> References: <20230522121125.2595254-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766597202080243688?= X-GMAIL-MSGID: =?utf-8?q?1766597202080243688?= Fold do_tcp_sendpages() into its last remaining caller, tcp_sendpage_locked(). Signed-off-by: David Howells cc: Eric Dumazet cc: David Ahern cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tcp.h | 2 -- net/ipv4/tcp.c | 21 +++++++-------------- 2 files changed, 7 insertions(+), 16 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index 04a31643cda3..02a6cff1827e 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -333,8 +333,6 @@ int tcp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags); int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags); -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags); int tcp_send_mss(struct sock *sk, int *size_goal, int flags); void tcp_push(struct sock *sk, int flags, int mss_now, int nonagle, int size_goal); diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index f3a0c02678e0..e9506cebecce 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -974,12 +974,17 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags) +int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, + size_t size, int flags) { struct bio_vec bvec; struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + if (!(sk->sk_route_caps & NETIF_F_SG)) + return sock_no_sendpage_locked(sk, page, offset, size, flags); + + tcp_rate_check_app_limited(sk); /* is sending application-limited? */ + bvec_set_page(&bvec, page, size, offset); iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); @@ -988,18 +993,6 @@ ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, return tcp_sendmsg_locked(sk, &msg, size); } -EXPORT_SYMBOL_GPL(do_tcp_sendpages); - -int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, - size_t size, int flags) -{ - if (!(sk->sk_route_caps & NETIF_F_SG)) - return sock_no_sendpage_locked(sk, page, offset, size, flags); - - tcp_rate_check_app_limited(sk); /* is sending application-limited? */ - - return do_tcp_sendpages(sk, page, offset, size, flags); -} EXPORT_SYMBOL_GPL(tcp_sendpage_locked); int tcp_sendpage(struct sock *sk, struct page *page, int offset, From patchwork Mon May 22 12:11:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 97353 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1402582vqo; Mon, 22 May 2023 05:16:48 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4owxiPs6RJiDGqWu21NWr1iIIoqfS9O9esK/kcuZHkdVuSJG+9IFMrlJ/507jJGlG8sfhx X-Received: by 2002:a05:6a00:134c:b0:64d:2b06:5949 with SMTP id k12-20020a056a00134c00b0064d2b065949mr14368781pfu.8.1684757807726; Mon, 22 May 2023 05:16:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684757807; cv=none; d=google.com; s=arc-20160816; b=e21mPfFqtRzfR850ngjoor4HqG1V57Jwwb3KqYX0aI69tYD9EhDG5VN3n9OTs09JK2 B1kkp0osD2Wa1HiNMbGv/6+auC/epm3lfvdyzDFs49J4J1xgJ0LCRfYvk26ukuE5RdNP YbW1WzTuKxgUvA3RH1FEYVd01QwCJOgpkP9C5tUb9G8OaMFteyQikDrtZ25XuOkYIOlm 2v5/ev1dxnFBX8TL8cXa7vtr+76ynxuPUne2tj5cq28JiTU78CrPFEvBcw+5Iu4B7KJ3 fPVqUER/bpdoGMJCvAKjPZ4EQSKrlgtl/Nn64nt1/phxZ75GzuYEDoRfVSCPnsGBeFLo 0rgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Di14zlocOEDECmKOzgXOEyS4sjRLATonTy1WJdIXAUU=; b=tm2vob2yQnzin+UeyctfqRVHlkuCXLee0BNjD0qObrSeZRWC37piF4lvZHCBW16Gr3 mSDWu831Iubx35FRdY2WhdXMxCML71+qebTv49AH59JfgdWcLVtZgP/zpzJfv2vx7To7 vrJHwLuJPjbRjYjQk501rOgmz/pbgcQ1kZR2WNL3KQLmCOpfDm3eBaUDI3H/jaVZ7nrY fLJsSLd6gM0/swJHzqwOXGN5zTx2Iznn+vD2zv2zGvhQWOMwrVadFuUnWn5dQiOBYr+r fLSeW9ecQ4sw4qQZhx7mIfsyxMmkA7zX2K3x4gnoPPtAahEC8l2TzQWqb5AIAZrqbu61 KBtw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=cul1hH2Z; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g3-20020aa79f03000000b00625c6092a12si4702548pfr.215.2023.05.22.05.16.32; Mon, 22 May 2023 05:16:47 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=cul1hH2Z; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233733AbjEVMN6 (ORCPT + 99 others); Mon, 22 May 2023 08:13:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233346AbjEVMNS (ORCPT ); Mon, 22 May 2023 08:13:18 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1786FA3 for ; Mon, 22 May 2023 05:12:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684757538; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Di14zlocOEDECmKOzgXOEyS4sjRLATonTy1WJdIXAUU=; b=cul1hH2ZjcACPvgRvbC7yUyba812B7cG0nlVmVZxlpIWux6lQdyw1drqQXvTRCrNx8qBKI VWtv1Q5ELX1Oo5k6013eHW/2PwmW9oIlK21QVZHjdHNgF6zRrQ39P7VPMsSaLcpch2xaPr 0Vv+/6Q6EGDHvvameWjEzSZ/XJ5mUhU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-389-i0ekLMi1P3uITGz1Ox_XFQ-1; Mon, 22 May 2023 08:12:11 -0400 X-MC-Unique: i0ekLMi1P3uITGz1Ox_XFQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2602780349B; Mon, 22 May 2023 12:12:10 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 856907B7C; Mon, 22 May 2023 12:12:07 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v10 11/16] ip, udp: Support MSG_SPLICE_PAGES Date: Mon, 22 May 2023 13:11:20 +0100 Message-Id: <20230522121125.2595254-12-dhowells@redhat.com> In-Reply-To: <20230522121125.2595254-1-dhowells@redhat.com> References: <20230522121125.2595254-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766596603134894669?= X-GMAIL-MSGID: =?utf-8?q?1766596603134894669?= Make IP/UDP sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/ipv4/ip_output.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 52fc840898d8..c7db973b5d29 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1048,6 +1048,14 @@ static int __ip_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + if (inet->hdrincl) + return -EPERM; + if (rt->dst.dev->features & NETIF_F_SG) + /* We need an empty buffer to attach stuff to */ + paged = true; + else + flags &= ~MSG_SPLICE_PAGES; } cork->length += length; @@ -1207,6 +1215,15 @@ static int __ip_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + struct msghdr *msg = from; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) + goto error; + copy = err; + wmem_alloc_delta += copy; } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Mon May 22 12:11:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 97381 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1409314vqo; Mon, 22 May 2023 05:28:03 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ58n9GxeXSrMVhrOza/rxaOuVnONZ+Lg3C03xZrMf4yhL3gwYB1IoO7BvzcWDFl9FVY4pes X-Received: by 2002:a17:90a:4a07:b0:247:2874:a6a6 with SMTP id e7-20020a17090a4a0700b002472874a6a6mr10064451pjh.2.1684758483558; Mon, 22 May 2023 05:28:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684758483; cv=none; d=google.com; s=arc-20160816; b=qPLzV6WxIXs9NYvpTQW3zdV7USQUte6fOvUXCbSIHxbmlfu3q/Y3I0DG9BBVhuYAha Q2mMHxFZIgDYpaB+rAfx72mVAGeJrMuaaFO/T6Ehs/oJKtT7ZUat2n4GbLIeQpxrA5C2 HNYcR/tR5D98XgHIVCDQj+KL9PNfmPF/Jy37TCDr1SGmyXDx6uIPVIyENo9J8jRr1J8i qCqk2O2RRzBjv4zM78BXLrAbRn4iG7qhZsKXGNHJ/iAKm5Tfsp43zeJwlyc7xecqJM8U iJxbd1VzyL2QpVSSbtUtK77Rm+4stMyGSp21DynYYugjOrk7BRngA8qBUIFAI9hF32N1 O+tA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5dEuCyEYbAeTLbptDWG+lpG3TePaGyDY6iKA6gIr8Gk=; b=ffd3YawaEPueAPXBl1MQC14SZ3eQhu85Pa0T2P1YDChOpR75bmkCMN46y06omwrq44 dh9to8Bxe9CFnyzv7km4FGyIWw5wTtxeJX9vuuTQAMZr0ChJruQzu5pdevjCtmqRXGPN vOIEEVncOTUSrbD7UZsePmA9iJmj3ETO0FoZ9NdUrKJb/fUD74UEdeiFj6cW5IRUxYOO DXxmN9RBoFZcT3iGm7tm6jcyqyBoyM0fT1Hg1HCSuUjsJNmBOtqJKLcB3QrtWMGuHqI8 HS3BraKdsl/0mRK13u0NKh3Hguwf6HySqw7dWR06zZukCS+AYISGYhLc++AAW4/FBhdw RM3Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=f9MGfhu+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b134-20020a63348c000000b005289d0633e6si4753725pga.733.2023.05.22.05.27.49; Mon, 22 May 2023 05:28:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=f9MGfhu+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233767AbjEVMNc (ORCPT + 99 others); Mon, 22 May 2023 08:13:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46804 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232889AbjEVMNM (ORCPT ); Mon, 22 May 2023 08:13:12 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8D29410C for ; Mon, 22 May 2023 05:12:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684757538; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5dEuCyEYbAeTLbptDWG+lpG3TePaGyDY6iKA6gIr8Gk=; b=f9MGfhu+kYRZ553CRO+KZiiRJvo9JTj5p0xr8Q8GLJMsnwVOKxCdi28aY8N4IbIqnLU79H Csh03QSQtP4XJJDuOp9mbWGMYfr3hcdi30/MJeoxHtwbZCXBY98JtCbf3tumoI9Mq8+GdW q8EIlvJRZd+n5Z5XYnLU06d/gaea6aY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-569-bsmo5ELVM-2TXNBTHt6zHg-1; Mon, 22 May 2023 08:12:14 -0400 X-MC-Unique: bsmo5ELVM-2TXNBTHt6zHg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8B87B85A5A8; Mon, 22 May 2023 12:12:13 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0EFBB1121314; Mon, 22 May 2023 12:12:10 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v10 12/16] ip6, udp6: Support MSG_SPLICE_PAGES Date: Mon, 22 May 2023 13:11:21 +0100 Message-Id: <20230522121125.2595254-13-dhowells@redhat.com> In-Reply-To: <20230522121125.2595254-1-dhowells@redhat.com> References: <20230522121125.2595254-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766597311938387304?= X-GMAIL-MSGID: =?utf-8?q?1766597311938387304?= Make IP6/UDP6 sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible, copying the data if not. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/ipv6/ip6_output.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 9554cf46ed88..c722cb881b2d 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1589,6 +1589,14 @@ static int __ip6_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + if (inet_sk(sk)->hdrincl) + return -EPERM; + if (rt->dst.dev->features & NETIF_F_SG) + /* We need an empty buffer to attach stuff to */ + paged = true; + else + flags &= ~MSG_SPLICE_PAGES; } /* @@ -1778,6 +1786,15 @@ static int __ip6_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + struct msghdr *msg = from; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) + goto error; + copy = err; + wmem_alloc_delta += copy; } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Mon May 22 12:11:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 97354 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1403444vqo; Mon, 22 May 2023 05:18:14 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5NQfAXt8/egoc1XcySxCyvdJuNt9cFTbkvtZnre+DffLvxZcG9ggF0z0m/EXkSimxpCJiD X-Received: by 2002:a17:90a:1385:b0:250:50c5:cabc with SMTP id i5-20020a17090a138500b0025050c5cabcmr15367411pja.3.1684757894480; Mon, 22 May 2023 05:18:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684757894; cv=none; d=google.com; s=arc-20160816; b=zr7QyJDpHJdLTkQuCtaNSnmHl4hLBogdl4x2qz47sCEBIx39PbVQBqL2BtooO/nNJI gc78FYvijbpWKkUmgWSN+sEwt2jXX/dmrSHPXhxL7crK5/lCBefXG6d97Dcb4tuGOv58 IhCfKoVDurTy3sujuZPC5sBBTbBwnsw7YuUkh0Lv+S6mgdjnI5nj9SxMYrXfihfinx51 mh3+JeZSAm1C7VHfRFTxLv5eZqY1Y6042ZkshOkCiqD9d/o9f/DHVadiCe5AeFR2RNSB rDIaLGXbSk9G3+c+iJONVhVGbAbWuoQYiyjMVdkINa394cj+X/tovS6Yl5se3iUDIMoN ohKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=arPLIYOMzcdUPWLjuF22tHo0vP4jVzFFThikgCzP+CE=; b=YySm5kj3xE5R9D9zKiDsr9d7VGmM0e1ybNMHqVGh/aJC7JvH0OYFpyCD5OerlQ+i2t ngTzyct94uggdiXh/R5qDFDlsRpuiFqNhriYvVnwKvrVy/MNhqy4V+zEfS9ZMQPlNeP9 OLhg1eOKXbILSD9nMD09X1RNvr3YRwvYgXluBUP7pwKhw3DxY8b02KxwhiYQYeN2qGrv 8k3grdDJcgBEf+FSJ7q7Mdk94fjUCRgpeMzfcQKEADzEjXw+jKOd7emyCTxXPqMJqZcf Pal7rjtobxlS/6/tvlf8z8V2mN6F2YGQjWRX4zBely2Nk47ZUJSq21VzgAchICGJYSSq TR8g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=gNEBrb2+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ie2-20020a17090b400200b0024666c4e8d8si65500pjb.187.2023.05.22.05.17.58; Mon, 22 May 2023 05:18:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=gNEBrb2+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233747AbjEVMOV (ORCPT + 99 others); Mon, 22 May 2023 08:14:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46772 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230365AbjEVMNU (ORCPT ); Mon, 22 May 2023 08:13:20 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D7C3E118 for ; Mon, 22 May 2023 05:12:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684757539; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=arPLIYOMzcdUPWLjuF22tHo0vP4jVzFFThikgCzP+CE=; b=gNEBrb2+7QCk6oqJRxymmU2zNQ1qUWvTbpjt4XQnlR97+GayVv49U4ZLO94yy6Vsxn0Vzu dE2e3/jN1bYNV+SXQaP9xlBlH8CLuLTQuILSXCI6aoz91yf3VxrPVhsHiucuIvd3uOMKf5 Yv1TbqZ7ZCI65CYoXM8EpicRj6Whbgg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-407-eMUMHyWdN1qeVHDjTVvlzg-1; Mon, 22 May 2023 08:12:17 -0400 X-MC-Unique: eMUMHyWdN1qeVHDjTVvlzg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AF505185A78B; Mon, 22 May 2023 12:12:16 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4439120296C6; Mon, 22 May 2023 12:12:14 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v10 13/16] udp: Convert udp_sendpage() to use MSG_SPLICE_PAGES Date: Mon, 22 May 2023 13:11:22 +0100 Message-Id: <20230522121125.2595254-14-dhowells@redhat.com> In-Reply-To: <20230522121125.2595254-1-dhowells@redhat.com> References: <20230522121125.2595254-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766596693876925711?= X-GMAIL-MSGID: =?utf-8?q?1766596693876925711?= Convert udp_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - udp_sendpage() shouldn't lock the socket around udp_sendpage(). - udp_sendpage() should only set MSG_MORE if MSG_SENDPAGE_NOTLAST is set. net/ipv4/udp.c | 51 ++++++-------------------------------------------- 1 file changed, 6 insertions(+), 45 deletions(-) diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index aa32afd871ee..2879dc6d66ea 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -1332,54 +1332,15 @@ EXPORT_SYMBOL(udp_sendmsg); int udp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct inet_sock *inet = inet_sk(sk); - struct udp_sock *up = udp_sk(sk); - int ret; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES }; if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; - - if (!up->pending) { - struct msghdr msg = { .msg_flags = flags|MSG_MORE }; - - /* Call udp_sendmsg to specify destination address which - * sendpage interface can't pass. - * This will succeed only when the socket is connected. - */ - ret = udp_sendmsg(sk, &msg, 0); - if (ret < 0) - return ret; - } - - lock_sock(sk); + msg.msg_flags |= MSG_MORE; - if (unlikely(!up->pending)) { - release_sock(sk); - - net_dbg_ratelimited("cork failed\n"); - return -EINVAL; - } - - ret = ip_append_page(sk, &inet->cork.fl.u.ip4, - page, offset, size, flags); - if (ret == -EOPNOTSUPP) { - release_sock(sk); - return sock_no_sendpage(sk->sk_socket, page, offset, - size, flags); - } - if (ret < 0) { - udp_flush_pending_frames(sk); - goto out; - } - - up->len += size; - if (!(READ_ONCE(up->corkflag) || (flags&MSG_MORE))) - ret = udp_push_pending_frames(sk); - if (!ret) - ret = size; -out: - release_sock(sk); - return ret; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return udp_sendmsg(sk, &msg, size); } #define UDP_SKB_IS_STATELESS 0x80000000 From patchwork Mon May 22 12:11:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 97355 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1403803vqo; Mon, 22 May 2023 05:18:46 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7+qysDDyQRK737Up1jV+/Ep9UF3I9Tn/7UprG2o1+hpmoKBFb8wVG5YrPRuydJCJaEH2zm X-Received: by 2002:a05:6a00:1389:b0:623:8592:75c4 with SMTP id t9-20020a056a00138900b00623859275c4mr15904157pfg.29.1684757926065; Mon, 22 May 2023 05:18:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684757926; cv=none; d=google.com; s=arc-20160816; b=wh+hR9xdYvftK3GNZHN6SqMkmcD8nE+u11ovvOGSO9EONO3IeLdhkND6Wts8cbGKbr 9dNlhri6/uagXI5Yftw84KGEgsqLDJNaeHLqiB9IIe3sYxH+ukuxuNl3566qDtaPsbaL Ql80GpglwqfhwEDbLMBi1PPp18PgZ1SSquUR5GtS8r0+tPBmw68eIIZ4U63okN5A274b +B6p5XdLPl5VklxlNAONsIn72vNwmnjofSQXGFP685UsWRBveQjgJyBa5AclaSjgtF7Z K0+SKY7cl5+JlxAiPT29BV96bY0YIm2s7ftoc3Lh8FrUDYrjtOtgQorAjgT6r8fdkQnT 3IvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vVs0CsGrBKFATDm79TCVboT9gSnZlC7XDeXTGZfAJzM=; b=eJzIGK6zmVqeEOvZC3+WDJekFmd19DqgDa7y+ZpHXPa0Y7f3yIotheZWxB3e9I194G BywZ0UF+GwxBwUMRb1w39UJpp2tsfyKxsKnFT9cye4oKcfEzsPuW75a0SJ8NZESlQ6Zo lqgHRKUsQ1CZM5A6GgSNnFBH3iyjlXAODC6lh6tiLFZEXC0c+1qovSJWWmEoQeIbrIUO a3LVhB3GY/cVorfNdGhygBgQWyJwFIt+lpevauGwGsZRqn7k7Ri5DmhILMax/6Q7p2Bd iB2BuXBMp2NRTvgbsQ4SQmNA9j86JhQglz7RZxhcjn5tnQ+sVqyJvGXk3IlFnl/seBXe wP6g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=UbtONZjS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y3-20020aa79ae3000000b00643aa8d2548si471342pfp.120.2023.05.22.05.18.31; Mon, 22 May 2023 05:18:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=UbtONZjS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233784AbjEVMOi (ORCPT + 99 others); Mon, 22 May 2023 08:14:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230363AbjEVMNc (ORCPT ); Mon, 22 May 2023 08:13:32 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 486AB133 for ; Mon, 22 May 2023 05:12:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684757544; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vVs0CsGrBKFATDm79TCVboT9gSnZlC7XDeXTGZfAJzM=; b=UbtONZjSlqYaw5tuOmwMSHGYPuwr1YMBexKN50cdDg6T8/pVj18TkMp/4oXFjpfwMKcjUu y/Qw8vbCO7XfjZ2dzZKy1xZ7LJHHXeymzKIRzhW55wotxif24l9lYxnIzh5CeucZGHMmtY sa16khsj8TPEJRJWRHVN90f9CFzmWCc= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-296-LPYcjbL_PmKPilJ00U_9eQ-1; Mon, 22 May 2023 08:12:21 -0400 X-MC-Unique: LPYcjbL_PmKPilJ00U_9eQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 544FB3C0BE2F; Mon, 22 May 2023 12:12:20 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id ABF042029F6D; Mon, 22 May 2023 12:12:17 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v10 14/16] ip: Remove ip_append_page() Date: Mon, 22 May 2023 13:11:23 +0100 Message-Id: <20230522121125.2595254-15-dhowells@redhat.com> In-Reply-To: <20230522121125.2595254-1-dhowells@redhat.com> References: <20230522121125.2595254-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766596726964249490?= X-GMAIL-MSGID: =?utf-8?q?1766596726964249490?= ip_append_page() is no longer used with the removal of udp_sendpage(), so remove it. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #7) - Remove now-unused csum_page(). include/net/ip.h | 2 - net/ipv4/ip_output.c | 148 ++----------------------------------------- 2 files changed, 4 insertions(+), 146 deletions(-) diff --git a/include/net/ip.h b/include/net/ip.h index c3fffaa92d6e..7627a4df893b 100644 --- a/include/net/ip.h +++ b/include/net/ip.h @@ -220,8 +220,6 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4, unsigned int flags); int ip_generic_getfrag(void *from, char *to, int offset, int len, int odd, struct sk_buff *skb); -ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, - int offset, size_t size, int flags); struct sk_buff *__ip_make_skb(struct sock *sk, struct flowi4 *fl4, struct sk_buff_head *queue, struct inet_cork *cork); diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index c7db973b5d29..553c740a6bfb 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -946,17 +946,6 @@ ip_generic_getfrag(void *from, char *to, int offset, int len, int odd, struct sk } EXPORT_SYMBOL(ip_generic_getfrag); -static inline __wsum -csum_page(struct page *page, int offset, int copy) -{ - char *kaddr; - __wsum csum; - kaddr = kmap(page); - csum = csum_partial(kaddr + offset, copy, 0); - kunmap(page); - return csum; -} - static int __ip_append_data(struct sock *sk, struct flowi4 *fl4, struct sk_buff_head *queue, @@ -1327,10 +1316,10 @@ static int ip_setup_cork(struct sock *sk, struct inet_cork *cork, } /* - * ip_append_data() and ip_append_page() can make one large IP datagram - * from many pieces of data. Each pieces will be holded on the socket - * until ip_push_pending_frames() is called. Each piece can be a page - * or non-page data. + * ip_append_data() can make one large IP datagram from many pieces of + * data. Each piece will be held on the socket until + * ip_push_pending_frames() is called. Each piece can be a page or + * non-page data. * * Not only UDP, other transport protocols - e.g. raw sockets - can use * this interface potentially. @@ -1363,135 +1352,6 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4, from, length, transhdrlen, flags); } -ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, - int offset, size_t size, int flags) -{ - struct inet_sock *inet = inet_sk(sk); - struct sk_buff *skb; - struct rtable *rt; - struct ip_options *opt = NULL; - struct inet_cork *cork; - int hh_len; - int mtu; - int len; - int err; - unsigned int maxfraglen, fragheaderlen, fraggap, maxnonfragsize; - - if (inet->hdrincl) - return -EPERM; - - if (flags&MSG_PROBE) - return 0; - - if (skb_queue_empty(&sk->sk_write_queue)) - return -EINVAL; - - cork = &inet->cork.base; - rt = (struct rtable *)cork->dst; - if (cork->flags & IPCORK_OPT) - opt = cork->opt; - - if (!(rt->dst.dev->features & NETIF_F_SG)) - return -EOPNOTSUPP; - - hh_len = LL_RESERVED_SPACE(rt->dst.dev); - mtu = cork->gso_size ? IP_MAX_MTU : cork->fragsize; - - fragheaderlen = sizeof(struct iphdr) + (opt ? opt->optlen : 0); - maxfraglen = ((mtu - fragheaderlen) & ~7) + fragheaderlen; - maxnonfragsize = ip_sk_ignore_df(sk) ? 0xFFFF : mtu; - - if (cork->length + size > maxnonfragsize - fragheaderlen) { - ip_local_error(sk, EMSGSIZE, fl4->daddr, inet->inet_dport, - mtu - (opt ? opt->optlen : 0)); - return -EMSGSIZE; - } - - skb = skb_peek_tail(&sk->sk_write_queue); - if (!skb) - return -EINVAL; - - cork->length += size; - - while (size > 0) { - /* Check if the remaining data fits into current packet. */ - len = mtu - skb->len; - if (len < size) - len = maxfraglen - skb->len; - - if (len <= 0) { - struct sk_buff *skb_prev; - int alloclen; - - skb_prev = skb; - fraggap = skb_prev->len - maxfraglen; - - alloclen = fragheaderlen + hh_len + fraggap + 15; - skb = sock_wmalloc(sk, alloclen, 1, sk->sk_allocation); - if (unlikely(!skb)) { - err = -ENOBUFS; - goto error; - } - - /* - * Fill in the control structures - */ - skb->ip_summed = CHECKSUM_NONE; - skb->csum = 0; - skb_reserve(skb, hh_len); - - /* - * Find where to start putting bytes. - */ - skb_put(skb, fragheaderlen + fraggap); - skb_reset_network_header(skb); - skb->transport_header = (skb->network_header + - fragheaderlen); - if (fraggap) { - skb->csum = skb_copy_and_csum_bits(skb_prev, - maxfraglen, - skb_transport_header(skb), - fraggap); - skb_prev->csum = csum_sub(skb_prev->csum, - skb->csum); - pskb_trim_unique(skb_prev, maxfraglen); - } - - /* - * Put the packet on the pending queue. - */ - __skb_queue_tail(&sk->sk_write_queue, skb); - continue; - } - - if (len > size) - len = size; - - if (skb_append_pagefrags(skb, page, offset, len, - MAX_SKB_FRAGS)) { - err = -EMSGSIZE; - goto error; - } - - if (skb->ip_summed == CHECKSUM_NONE) { - __wsum csum; - csum = csum_page(page, offset, len); - skb->csum = csum_block_add(skb->csum, csum, skb->len); - } - - skb_len_add(skb, len); - refcount_add(len, &sk->sk_wmem_alloc); - offset += len; - size -= len; - } - return 0; - -error: - cork->length -= size; - IP_INC_STATS(sock_net(sk), IPSTATS_MIB_OUTDISCARDS); - return err; -} - static void ip_cork_release(struct inet_cork *cork) { cork->flags &= ~IPCORK_OPT; From patchwork Mon May 22 12:11:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 97378 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1409044vqo; Mon, 22 May 2023 05:27:38 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5CePoWG6lzatVMnNsgiN6JRUziK0l6sYzgQ5RhnSCU+6NeBaGPXMnv9yqJv4bcZTx0FwTw X-Received: by 2002:a05:6a20:840e:b0:102:d2fa:d707 with SMTP id c14-20020a056a20840e00b00102d2fad707mr12229907pzd.52.1684758457873; Mon, 22 May 2023 05:27:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684758457; cv=none; d=google.com; s=arc-20160816; b=LiGLjSwaLIQO4xvuhf8K1BZXTlJexgpUO9Fpb7PeKeviYETac9Zla4xtxtaBV1f+aj FD6kzHDoBF34xy9W4YWbmNJMLOzaXqFEbnw4X+lA/E7kiduYbb+rUfmril4BTyTX88T2 KzrEN+cwW5tuy2xpvNH9vl4+3h6kX/hdzajthuIf4HQ5v5MJf/sWHurWDHjfkFx887TU t8iuU4KmyrVm/XuCdFMaxPwjjQjppfx0UKALk6rxjgiAPZCAh4VjINq3pfREDXp9nOJd GWgEPoJ4fL9LVtOmhnf9M8quWVeh32/LrPlIC075zE28Hl1mnuFVfsnrceHlRu2Hs91m Sbkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4sbDm+4sgaynAjmiQFVbDl+OyfXvG2vWGJtRH+IAST4=; b=PUPnpf6W8GDUPPZFwytsFt9P44kN6UqcfYKqezUxiBbw1aMb87qJ9h6iEj9AoXkG4D VWNNJ2XNP8GezRiR28xJsu5QNJSSWwMoYbQNVvRxeQlyPlZEri/K3O+54oXk9nnqz7kR /SlRnSQV4bEQmQ5WXK1l5dPT8BkZ22BZJmv4FY3CE2sdPi60RtHV5uMKqtfCW6Jkro7z UKpmyN04m8QXELH3d8L9NgRa81f9ecjlyJPJOl9yeI6xxDlOf1+y9F3dokgXnC5kxVRW KhVo9d6L9dPKynzWne61a7/QKWn4L4U3brHzUpCx/FxKeNhRavGYNsR022GdXLSzJ9m+ sWRw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="G/CKFMnq"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j6-20020a633c06000000b0053482f39396si4539199pga.778.2023.05.22.05.27.25; Mon, 22 May 2023 05:27:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="G/CKFMnq"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233829AbjEVMOm (ORCPT + 99 others); Mon, 22 May 2023 08:14:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233773AbjEVMNh (ORCPT ); Mon, 22 May 2023 08:13:37 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 86690B7 for ; Mon, 22 May 2023 05:12:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684757550; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4sbDm+4sgaynAjmiQFVbDl+OyfXvG2vWGJtRH+IAST4=; b=G/CKFMnqzvP1STpQK6HprlhUmBsCn2PIwUrB8a8+Bc1OuV2NxV61Fw4/huDzDB442cyE/X 3Vbl6/jCWQ5cXGFQL0OuNW580yKQmh5GjaWqR9k8LXTv4+RVzC7TpoRBvRPKjLlUpqatlN ++d/mkPAcOZn6gnR2jfaV4zfLZxHUbE= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-160-SjSudd0MOWalkYvBaLq9kg-1; Mon, 22 May 2023 08:12:24 -0400 X-MC-Unique: SjSudd0MOWalkYvBaLq9kg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B4B991C04602; Mon, 22 May 2023 12:12:23 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id 112762166B26; Mon, 22 May 2023 12:12:20 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Kuniyuki Iwashima Subject: [PATCH net-next v10 15/16] af_unix: Support MSG_SPLICE_PAGES Date: Mon, 22 May 2023 13:11:24 +0100 Message-Id: <20230522121125.2595254-16-dhowells@redhat.com> In-Reply-To: <20230522121125.2595254-1-dhowells@redhat.com> References: <20230522121125.2595254-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766597284396513313?= X-GMAIL-MSGID: =?utf-8?q?1766597284396513313?= Make AF_UNIX sendmsg() support MSG_SPLICE_PAGES, splicing in pages from the source iterator if possible and copying the data in otherwise. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Kuniyuki Iwashima cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/unix/af_unix.c | 49 +++++++++++++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 16 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index dd55506b4632..976bc1c5e11b 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2200,19 +2200,25 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, while (sent < len) { size = len - sent; - /* Keep two messages in the pipe so it schedules better */ - size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + skb = sock_alloc_send_pskb(sk, 0, 0, + msg->msg_flags & MSG_DONTWAIT, + &err, 0); + } else { + /* Keep two messages in the pipe so it schedules better */ + size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); - /* allow fallback to order-0 allocations */ - size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); + /* allow fallback to order-0 allocations */ + size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); - data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); + data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); - data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); + data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); - skb = sock_alloc_send_pskb(sk, size - data_len, data_len, - msg->msg_flags & MSG_DONTWAIT, &err, - get_order(UNIX_SKB_FRAGS_SZ)); + skb = sock_alloc_send_pskb(sk, size - data_len, data_len, + msg->msg_flags & MSG_DONTWAIT, &err, + get_order(UNIX_SKB_FRAGS_SZ)); + } if (!skb) goto out_err; @@ -2224,13 +2230,24 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, } fds_sent = true; - skb_put(skb, size - data_len); - skb->data_len = data_len; - skb->len = size; - err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); - if (err) { - kfree_skb(skb); - goto out_err; + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + err = skb_splice_from_iter(skb, &msg->msg_iter, size, + sk->sk_allocation); + if (err < 0) { + kfree_skb(skb); + goto out_err; + } + size = err; + refcount_add(size, &sk->sk_wmem_alloc); + } else { + skb_put(skb, size - data_len); + skb->data_len = data_len; + skb->len = size; + err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); + if (err) { + kfree_skb(skb); + goto out_err; + } } unix_state_lock(other); From patchwork Mon May 22 12:11:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 97370 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1408599vqo; Mon, 22 May 2023 05:26:53 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4eHhAXqi4oc+cK9kcll0a/8eoVP18kf1IAOTU/1Mf040q/EJzSFC6UyqOO2DENRNQNVOHK X-Received: by 2002:a05:6a20:258b:b0:105:5800:c51b with SMTP id k11-20020a056a20258b00b001055800c51bmr11295440pzd.30.1684758413055; Mon, 22 May 2023 05:26:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684758413; cv=none; d=google.com; s=arc-20160816; b=CiLQuWffFQmg2yK9KFuKD3OEF8Ei4Ur/DudDRanDtA5VUBniXC41jLJ4R42lX9YaNJ 2tRx6sgpjqytmq6dc3hSMNF8JAu+68DJiAXDOrd41JeNq+AHQB1bTeM6sqa0SiEDrBYb A2TVW5i06rt67ZVpNUBmf27jvz5VkUuBt6g8XOrMxSeVmRinIG9Jurzy+fmmk9g0i3kZ SRWtDcdZBYloqWVHp6c92c/yzCes57FBoKeppkNsAcoDOWm0bJpO1M94cv4qhw2O0xRB R50vNioHOhOj+w4sYwBfp4BI7TEoW7oRQkiZVr6Qec1lyeXE/8qPumdMmwazZ2vk5oWu GJ+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4nmIdUwXs8xUZa8LHr6SjLu9qXp25hoa6tjiHzVF7uA=; b=Akh1nKksu2IpAvLoZsCfw9cllq5EhDoG7kysqoMeBtsEzzPMFsYxxlLPA8wNaSrcGG Djz4J9IeyAQhUqapAmHiRdqd0SDObt2OIK3fXGU4+B2cjb0YOC1gO89RCmVtEHTO7/wq GhT/BJSOctzEYWmsYrnG62QbVNGl0sKFuc2CPTqTPbNg5wyaWJY8COk4M4cSBUAIfVXV hYZ6IxBJJocODhGc3qiOE7R/n4dL0itRYG3ZQgLbKgITKkyky+b9p6j1Mz9cZh0fd11c /MirH6m94/XtTLyUG7FAVELqmYXAB/xM4zMLb9zFJV3+g8nIubKo5B2PMnGXDO/DV/4I ThhQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=S02zCYjB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 185-20020a6301c2000000b005004b2acf6dsi1391512pgb.810.2023.05.22.05.26.38; Mon, 22 May 2023 05:26:53 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=S02zCYjB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233836AbjEVMOp (ORCPT + 99 others); Mon, 22 May 2023 08:14:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47218 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232921AbjEVMNh (ORCPT ); Mon, 22 May 2023 08:13:37 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 48C141A8 for ; Mon, 22 May 2023 05:12:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684757553; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4nmIdUwXs8xUZa8LHr6SjLu9qXp25hoa6tjiHzVF7uA=; b=S02zCYjBj2eSxU3JR+nCF+w5NpsRZd6XDz1Okh7EibaO0D7FRW1zNSYLViPAC1V0ybegBb Mo5q/84+kaIS8pN4qZ5RzjYj2W85x072SKjbmT6E6qCkBmFPnfCivnR2HnO02x3/UmJPRj 6KgLAFv4LSqfy3yV3wluqP+//8k1nIs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-576-SCXcFoZWMeW_c6Kl9rP7Pg-1; Mon, 22 May 2023 08:12:28 -0400 X-MC-Unique: SCXcFoZWMeW_c6Kl9rP7Pg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4357E185A792; Mon, 22 May 2023 12:12:27 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.39.192.68]) by smtp.corp.redhat.com (Postfix) with ESMTP id A11807B7C; Mon, 22 May 2023 12:12:24 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Kuniyuki Iwashima Subject: [PATCH net-next v10 16/16] unix: Convert unix_stream_sendpage() to use MSG_SPLICE_PAGES Date: Mon, 22 May 2023 13:11:25 +0100 Message-Id: <20230522121125.2595254-17-dhowells@redhat.com> In-Reply-To: <20230522121125.2595254-1-dhowells@redhat.com> References: <20230522121125.2595254-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766597237638015755?= X-GMAIL-MSGID: =?utf-8?q?1766597237638015755?= Convert unix_stream_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Kuniyuki Iwashima cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #10) - Fix subject to refer to unix_stream_sendpage() not udp_sendpage(). net/unix/af_unix.c | 134 +++------------------------------------------ 1 file changed, 7 insertions(+), 127 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index 976bc1c5e11b..115436ce1f8a 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -1839,24 +1839,6 @@ static void maybe_add_creds(struct sk_buff *skb, const struct socket *sock, } } -static int maybe_init_creds(struct scm_cookie *scm, - struct socket *socket, - const struct sock *other) -{ - int err; - struct msghdr msg = { .msg_controllen = 0 }; - - err = scm_send(socket, &msg, scm, false); - if (err) - return err; - - if (unix_passcred_enabled(socket, other)) { - scm->pid = get_pid(task_tgid(current)); - current_uid_gid(&scm->creds.uid, &scm->creds.gid); - } - return err; -} - static bool unix_skb_scm_eq(struct sk_buff *skb, struct scm_cookie *scm) { @@ -2292,117 +2274,15 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, static ssize_t unix_stream_sendpage(struct socket *socket, struct page *page, int offset, size_t size, int flags) { - int err; - bool send_sigpipe = false; - bool init_scm = true; - struct scm_cookie scm; - struct sock *other, *sk = socket->sk; - struct sk_buff *skb, *newskb = NULL, *tail = NULL; - - if (flags & MSG_OOB) - return -EOPNOTSUPP; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES }; - other = unix_peer(sk); - if (!other || sk->sk_state != TCP_ESTABLISHED) - return -ENOTCONN; - - if (false) { -alloc_skb: - unix_state_unlock(other); - mutex_unlock(&unix_sk(other)->iolock); - newskb = sock_alloc_send_pskb(sk, 0, 0, flags & MSG_DONTWAIT, - &err, 0); - if (!newskb) - goto err; - } - - /* we must acquire iolock as we modify already present - * skbs in the sk_receive_queue and mess with skb->len - */ - err = mutex_lock_interruptible(&unix_sk(other)->iolock); - if (err) { - err = flags & MSG_DONTWAIT ? -EAGAIN : -ERESTARTSYS; - goto err; - } - - if (sk->sk_shutdown & SEND_SHUTDOWN) { - err = -EPIPE; - send_sigpipe = true; - goto err_unlock; - } - - unix_state_lock(other); + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - if (sock_flag(other, SOCK_DEAD) || - other->sk_shutdown & RCV_SHUTDOWN) { - err = -EPIPE; - send_sigpipe = true; - goto err_state_unlock; - } - - if (init_scm) { - err = maybe_init_creds(&scm, socket, other); - if (err) - goto err_state_unlock; - init_scm = false; - } - - skb = skb_peek_tail(&other->sk_receive_queue); - if (tail && tail == skb) { - skb = newskb; - } else if (!skb || !unix_skb_scm_eq(skb, &scm)) { - if (newskb) { - skb = newskb; - } else { - tail = skb; - goto alloc_skb; - } - } else if (newskb) { - /* this is fast path, we don't necessarily need to - * call to kfree_skb even though with newskb == NULL - * this - does no harm - */ - consume_skb(newskb); - newskb = NULL; - } - - if (skb_append_pagefrags(skb, page, offset, size, MAX_SKB_FRAGS)) { - tail = skb; - goto alloc_skb; - } - - skb->len += size; - skb->data_len += size; - skb->truesize += size; - refcount_add(size, &sk->sk_wmem_alloc); - - if (newskb) { - err = unix_scm_to_skb(&scm, skb, false); - if (err) - goto err_state_unlock; - spin_lock(&other->sk_receive_queue.lock); - __skb_queue_tail(&other->sk_receive_queue, newskb); - spin_unlock(&other->sk_receive_queue.lock); - } - - unix_state_unlock(other); - mutex_unlock(&unix_sk(other)->iolock); - - other->sk_data_ready(other); - scm_destroy(&scm); - return size; - -err_state_unlock: - unix_state_unlock(other); -err_unlock: - mutex_unlock(&unix_sk(other)->iolock); -err: - kfree_skb(newskb); - if (send_sigpipe && !(flags & MSG_NOSIGNAL)) - send_sig(SIGPIPE, current, 0); - if (!init_scm) - scm_destroy(&scm); - return err; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return unix_stream_sendmsg(socket, &msg, size); } static int unix_seqpacket_sendmsg(struct socket *sock, struct msghdr *msg,