From patchwork Thu May 18 11:34:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95817 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp423284vqo; Thu, 18 May 2023 04:41:28 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ711Bpv6n6rtWMIctgxnN9kUMWstCORpjkxwf0mCTf04dZt1ZwM8AKDoXsTc8hiS2qlIxLh X-Received: by 2002:a17:902:f683:b0:1ab:29bc:bd87 with SMTP id l3-20020a170902f68300b001ab29bcbd87mr2432698plg.35.1684410088653; Thu, 18 May 2023 04:41:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684410088; cv=none; d=google.com; s=arc-20160816; b=0xG1Xk/vbGmk+5v/MkP02ulu0akav+FDKkstxPyqrtmI2Z6XSEeKXHnYr3cowcgXMH uRh+7AV19DJF+b9oolSz2dvfzBN3IxrcDwDUAtzJBRQEG7sVDzyef3e7Ro8gpQvyE1na DTys10jbrQfrDofqjNy9AZPA5RU4t1jYqH1Ggt8CVBw7v/due/U7Zm/muJlDkXAqNoJh RHIzHsJV5koiwOs/Y0SZLW6CtxWYPow1/EVpWzedrdg0GezYKUVAuRz0LGQxDsG/Ionl Nn8yFgQIKdpTmewrZsAJOZDZFSTCPqKK6KMgdliyFYPJOT8IKfh1rgTnkfM586/ysCxy foig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=zo53+MxtcExCaAvP1E3d2cbKIX8PzAGUnvJuN4UlUmE=; b=tLpgd7AHV4HTYTnrH/0g2V3vpnGbX3hcs4yarILMf6lddcKvnNCQ31FE58WYTOCzyc JzxoWLE0sOzCuQkX6++JROt/NeuE9b6PjJ8f2etDrM+PCaDjibiPy5ipAmDSp71SE39J KC2WmBrcyCcePKzWz61F71EWQzdpl4QOCy8f0X3b5kI46LcBjvLt2ZqAX15FcK/+w++5 16zQ6VrMvSoKML2QsxdtvOo5SnaU+2Gi2s1DylavH9xyDY/gpb6sVLraEDIrSWauIrax Razva4ar4foYOQ1Dv16aoXsYSiiGpo3WjD+xnidukZc4ByXX7VDhi48p2J47bSj6PfUZ Wbig== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bJxBtKtF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id iw13-20020a170903044d00b001ae45f0c55esi1122275plb.14.2023.05.18.04.41.11; Thu, 18 May 2023 04:41:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bJxBtKtF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231338AbjERLgI (ORCPT + 99 others); Thu, 18 May 2023 07:36:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59704 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231301AbjERLgD (ORCPT ); Thu, 18 May 2023 07:36:03 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4FEE31BC6 for ; Thu, 18 May 2023 04:35:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684409707; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zo53+MxtcExCaAvP1E3d2cbKIX8PzAGUnvJuN4UlUmE=; b=bJxBtKtF+FyWlVLIQaYo5SnmIp0knfvSmXfAPeUPAUQ/cIeiyMqerVAzF/6HJ8bWyToJRq 2wcq5dMoK746qyoDavsj5VG0LMhCuSVMKHKKWpyfbWNzkLqvuPcswxTcVrL1wIy4xrpEVH ae8RK+zh0DemSd2ZRxEiv1a221rBy20= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-122-Q1oqsoVdOXek48f82BcBpA-1; Thu, 18 May 2023 07:35:03 -0400 X-MC-Unique: Q1oqsoVdOXek48f82BcBpA-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0B82786C60A; Thu, 18 May 2023 11:35:02 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id A3A93492B01; Thu, 18 May 2023 11:34:58 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Willem de Bruijn , io-uring@vger.kernel.org Subject: [PATCH net-next v8 01/16] net: Declare MSG_SPLICE_PAGES internal sendmsg() flag Date: Thu, 18 May 2023 12:34:38 +0100 Message-Id: <20230518113453.1350757-2-dhowells@redhat.com> In-Reply-To: <20230518113453.1350757-1-dhowells@redhat.com> References: <20230518113453.1350757-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766231992940692441?= X-GMAIL-MSGID: =?utf-8?q?1766231992940692441?= Declare MSG_SPLICE_PAGES, an internal sendmsg() flag, that hints to a network protocol that it should splice pages from the source iterator rather than copying the data if it can. This flag is added to a list that is cleared by sendmsg syscalls on entry. This is intended as a replacement for the ->sendpage() op, allowing a way to splice in several multipage folios in one go. Signed-off-by: David Howells Reviewed-by: Willem de Bruijn cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: io-uring@vger.kernel.org cc: netdev@vger.kernel.org --- Notes: ver #7) - In ____sys_sendmsg(), clear internal flags before setting msg_flags. - Clear internal flags in uring io_send{,_zc}(). include/linux/socket.h | 3 +++ io_uring/net.c | 2 ++ net/socket.c | 2 ++ 3 files changed, 7 insertions(+) diff --git a/include/linux/socket.h b/include/linux/socket.h index 13c3a237b9c9..bd1cc3238851 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -327,6 +327,7 @@ struct ucred { */ #define MSG_ZEROCOPY 0x4000000 /* Use user data in kernel path */ +#define MSG_SPLICE_PAGES 0x8000000 /* Splice the pages from the iterator in sendmsg() */ #define MSG_FASTOPEN 0x20000000 /* Send data in TCP SYN */ #define MSG_CMSG_CLOEXEC 0x40000000 /* Set close_on_exec for file descriptor received through @@ -337,6 +338,8 @@ struct ucred { #define MSG_CMSG_COMPAT 0 /* We never have 32 bit fixups */ #endif +/* Flags to be cleared on entry by sendmsg and sendmmsg syscalls */ +#define MSG_INTERNAL_SENDMSG_FLAGS (MSG_SPLICE_PAGES) /* Setsockoptions(2) level. Thanks to BSD these must match IPPROTO_xxx */ #define SOL_IP 0 diff --git a/io_uring/net.c b/io_uring/net.c index 89e839013837..f7cbb3c7a575 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -389,6 +389,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) if (flags & MSG_WAITALL) min_ret = iov_iter_count(&msg.msg_iter); + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; msg.msg_flags = flags; ret = sock_sendmsg(sock, &msg); if (ret < min_ret) { @@ -1136,6 +1137,7 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) msg_flags |= MSG_DONTWAIT; if (msg_flags & MSG_WAITALL) min_ret = iov_iter_count(&msg.msg_iter); + msg_flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; msg.msg_flags = msg_flags; msg.msg_ubuf = &io_notif_to_data(zc->notif)->uarg; diff --git a/net/socket.c b/net/socket.c index b7e01d0fe082..3df96e9ba4e2 100644 --- a/net/socket.c +++ b/net/socket.c @@ -2138,6 +2138,7 @@ int __sys_sendto(int fd, void __user *buff, size_t len, unsigned int flags, msg.msg_name = (struct sockaddr *)&address; msg.msg_namelen = addr_len; } + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; if (sock->file->f_flags & O_NONBLOCK) flags |= MSG_DONTWAIT; msg.msg_flags = flags; @@ -2483,6 +2484,7 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys, msg_sys->msg_control = ctl_buf; msg_sys->msg_control_is_user = false; } + flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; msg_sys->msg_flags = flags; if (sock->file->f_flags & O_NONBLOCK) From patchwork Thu May 18 11:34:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95813 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp423032vqo; Thu, 18 May 2023 04:41:01 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5+WH86LdvccxUpHD0+hjs/SWRT22L/EH57lvZYfRnV0AFgwlS2uCMHksQftfb+PrHufB3j X-Received: by 2002:a17:902:d48c:b0:1a6:ebc1:c54c with SMTP id c12-20020a170902d48c00b001a6ebc1c54cmr2929367plg.1.1684410060954; Thu, 18 May 2023 04:41:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684410060; cv=none; d=google.com; s=arc-20160816; b=V/Bvob26aaIn+4a7ZiBAfvPTAumcp6vofCuiz5owxPBCjE9Guaa5G6CtuOwU5RD+rM jgnlsnt6PyApfCvyWpQQExGsHXT2B2u0DDHvHnph6WD8moL5bSj2+PM17ZSQBVZe/Xr9 G07UtX840E0kybc5e1j8fKbG/lG2XZu22jRsq4PzGuP+KLST4MCubmfFqaojtpA7Vsk/ bxEg5fryHVskId4rH3eRs2BDGm/y/alZe7B5kElvH+xGKn7Rt2RCBFzI0OVCq605qSpZ yQyS+k7vI+/eF5/ocdrNHD4dhGhESWTzjxYcOGJ3yKohra+tAqYiLQwqH2ErG9esw/92 XY+w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Oms7mbQShTpY01zvjwYgDwHWlSZDYU/BDFTf+CWwguM=; b=OW+UJIUzJL7Q8yXp0cWbim+5pbutno6eczsR7JbIlT6ZRsN4UQgsOHW+2esntSGmuz TzTMgxLBJ+Op4Cm89r0N5h/WoZ172JDGnNQ4J5kRALL9LFhdG+noyDj1NAsNtsR4PXz4 dW4uT6LbnJaN9mOy/RzS1gT5KbwfoqIW3ePKwFaQ6p4VCHAkE6WH3ncdZ8154rI+nG+0 CHlbXSxuBPAZr//gxPmOeCigU/9cBcRcIaRJRZhigP/AJhZicqpYk8/vpm7/wpPMt2Zp bf6Z2Gkbgr9Z75fYn0ZxH0TqKkDDc0erLbBdRL1vvLeKhKOSNAr4+OfiBGMN53txCMEs nkUg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=W4YKlTMs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c3-20020a170902d90300b001a21fd27407si1006209plz.469.2023.05.18.04.40.45; Thu, 18 May 2023 04:41:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=W4YKlTMs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231362AbjERLgV (ORCPT + 99 others); Thu, 18 May 2023 07:36:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59824 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231346AbjERLgL (ORCPT ); Thu, 18 May 2023 07:36:11 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A40F619B1 for ; Thu, 18 May 2023 04:35:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684409717; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Oms7mbQShTpY01zvjwYgDwHWlSZDYU/BDFTf+CWwguM=; b=W4YKlTMsHdMdIN2Zi7RAq//Pxfd87EG8rpo2rsXv+a9gB7MTPFxA6ueq8jcUmiwxjvwfvV lCqqKlvmfezeXHaCF4rPRpaZIE7Jtd3Cxt2DOIlGwnhbGfax33pGb4m4/2nxk14VRmO+gQ 4o+Nwa89jS6XJ1DoAPGGQxef3j7JhTY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-61-ELmJWgkyMLiJ2SkyRnHWsQ-1; Thu, 18 May 2023 07:35:13 -0400 X-MC-Unique: ELmJWgkyMLiJ2SkyRnHWsQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9157910146FC; Thu, 18 May 2023 11:35:12 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id AB5E214171C0; Thu, 18 May 2023 11:35:06 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v8 02/16] net: Pass max frags into skb_append_pagefrags() Date: Thu, 18 May 2023 12:34:39 +0100 Message-Id: <20230518113453.1350757-3-dhowells@redhat.com> In-Reply-To: <20230518113453.1350757-1-dhowells@redhat.com> References: <20230518113453.1350757-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766231964435478832?= X-GMAIL-MSGID: =?utf-8?q?1766231964435478832?= Pass the maximum number of fragments into skb_append_pagefrags() rather than using MAX_SKB_FRAGS so that it can be used from code that wants to specify sysctl_max_skb_frags. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/linux/skbuff.h | 2 +- net/core/skbuff.c | 4 ++-- net/ipv4/ip_output.c | 3 ++- net/unix/af_unix.c | 2 +- 4 files changed, 6 insertions(+), 5 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 00e8c435fa1a..4c0ad48e38ca 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -1383,7 +1383,7 @@ static inline int skb_pad(struct sk_buff *skb, int pad) #define dev_kfree_skb(a) consume_skb(a) int skb_append_pagefrags(struct sk_buff *skb, struct page *page, - int offset, size_t size); + int offset, size_t size, size_t max_frags); struct skb_seq_state { __u32 lower_offset; diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 6724a84ebb09..7f53dcb26ad3 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -4188,13 +4188,13 @@ unsigned int skb_find_text(struct sk_buff *skb, unsigned int from, EXPORT_SYMBOL(skb_find_text); int skb_append_pagefrags(struct sk_buff *skb, struct page *page, - int offset, size_t size) + int offset, size_t size, size_t max_frags) { int i = skb_shinfo(skb)->nr_frags; if (skb_can_coalesce(skb, i, page, offset)) { skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], size); - } else if (i < MAX_SKB_FRAGS) { + } else if (i < max_frags) { skb_zcopy_downgrade_managed(skb); get_page(page); skb_fill_page_desc_noacc(skb, i, page, offset, size); diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 61892268e8a6..52fc840898d8 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1450,7 +1450,8 @@ ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, if (len > size) len = size; - if (skb_append_pagefrags(skb, page, offset, len)) { + if (skb_append_pagefrags(skb, page, offset, len, + MAX_SKB_FRAGS)) { err = -EMSGSIZE; goto error; } diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index cc695c9f09ec..dd55506b4632 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2349,7 +2349,7 @@ static ssize_t unix_stream_sendpage(struct socket *socket, struct page *page, newskb = NULL; } - if (skb_append_pagefrags(skb, page, offset, size)) { + if (skb_append_pagefrags(skb, page, offset, size, MAX_SKB_FRAGS)) { tail = skb; goto alloc_skb; } From patchwork Thu May 18 11:34:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95815 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp423116vqo; Thu, 18 May 2023 04:41:10 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4MnyoVgGcOLNQtM9VN5nN3IgheR68ZxvZ+3Wbpua/lenB3Ds5JguSjakuMpSfrQA3fJmNI X-Received: by 2002:a05:6a00:1820:b0:63b:7fc0:a4af with SMTP id y32-20020a056a00182000b0063b7fc0a4afmr4572821pfa.26.1684410070580; Thu, 18 May 2023 04:41:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684410070; cv=none; d=google.com; s=arc-20160816; b=UvywHmBTD0f6+D8q1LrZ4IRLQeGC1KHpp5FH2LWIznibca4PK+FXUq+rWKqjF/ejA9 opLqTYBQoITwpBQ+H82+bmRQxEQuODtgFAtU0BrxZjsoMYCy5jPwLq0S3icbjsLciV0n pKjKlOUIUQ766SQBLZuB40e6EVpQWZy31+Km18tP/dLZnpeSQ1TrZMSYlN+aWPtjurz4 u0TYijOjmvBsVNH+iOki8Lb2CHMxemhVVB/GgjajURheCBsEBkIbuS3qVCIa1oUIp9sk x2AEWcvRZb6YpZi/hrbegKjTJw/psuQGDrCVDyxjKcFjAzILwbAdBEuXJbwd/oikKqAR S2RQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jACcDubg9VuPG9obSLLRy2+X1qFqLp435QyVtyfFY1I=; b=kJr+OE8mINn9sUMED04CbEKXJDzvUr6Xf+g2BvscQPI+IBRsKdszSHyVoJlOMEnkZn /afplvfBG87G+2w4ljPKLyElAaJ7ddH0bdNcx76iJBchS6grMdZ2II8ak4eVRjsCa+kM COxJtaqF4k940mOUU+gUz2hKPK9VB+JsOK3FNSLTbpHdWyT3xq0aepJBFprA7Rp8Sltn ErdQg+XwRLl9+uMr5V5l6KlHCTOnA7aE68yDbeJyYgmvjxgpFB47/IpyUZdwSgQgo+OV TCn6qv4ggVqvnkBiT9XvCtALm531HFSMRyp8FeOQJjqaHi/O4BZ3Sd4jYGqHMl6vbieZ sGBw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=WQq4Va8D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z128-20020a633386000000b0051322a8d2aesi1148113pgz.110.2023.05.18.04.40.54; Thu, 18 May 2023 04:41:10 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=WQq4Va8D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231372AbjERLgY (ORCPT + 99 others); Thu, 18 May 2023 07:36:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59762 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231190AbjERLgQ (ORCPT ); Thu, 18 May 2023 07:36:16 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0953BD for ; Thu, 18 May 2023 04:35:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684409722; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jACcDubg9VuPG9obSLLRy2+X1qFqLp435QyVtyfFY1I=; b=WQq4Va8DhIAOlHOuLNgCg+xiRLa7lUYGg48EsLxCbfBlov7ZwbzuUnxGgtkAsZRccRa/A9 B/qjnj6EqmD31+usONJFL3I8vo/eBn6/yi1Lvu/yR+skVCADHC1CJaRN/c1MxwTeJw7RZE 1Ojy7qsKJZGs1/BUFk9zM1IXOqCtorw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-70-TOUdyjn5N_CVtCdOeDCHHg-1; Thu, 18 May 2023 07:35:16 -0400 X-MC-Unique: TOUdyjn5N_CVtCdOeDCHHg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A16C71805221; Thu, 18 May 2023 11:35:15 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7B00414171C0; Thu, 18 May 2023 11:35:13 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v8 03/16] net: Add a function to splice pages into an skbuff for MSG_SPLICE_PAGES Date: Thu, 18 May 2023 12:34:40 +0100 Message-Id: <20230518113453.1350757-4-dhowells@redhat.com> In-Reply-To: <20230518113453.1350757-1-dhowells@redhat.com> References: <20230518113453.1350757-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766231974525616623?= X-GMAIL-MSGID: =?utf-8?q?1766231974525616623?= Add a function to handle MSG_SPLICE_PAGES being passed internally to sendmsg(). Pages are spliced into the given socket buffer if possible and copied in if not (e.g. they're slab pages or have a zero refcount). Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Al Viro cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #8) - Order local variables in reverse xmas tree order. - Remove duplicate coalescence check. - Warn if sendpage_ok() fails. ver #7) - Export function. - Never copy data, return -EIO if sendpage_ok() returns false. include/linux/skbuff.h | 3 ++ net/core/skbuff.c | 88 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 4c0ad48e38ca..1c5f0ac6f8c3 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -5097,5 +5097,8 @@ static inline void skb_mark_for_recycle(struct sk_buff *skb) #endif } +ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, + ssize_t maxsize, gfp_t gfp); + #endif /* __KERNEL__ */ #endif /* _LINUX_SKBUFF_H */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 7f53dcb26ad3..f4a5b51aed22 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -6892,3 +6892,91 @@ nodefer: __kfree_skb(skb); if (unlikely(kick) && !cmpxchg(&sd->defer_ipi_scheduled, 0, 1)) smp_call_function_single_async(cpu, &sd->defer_csd); } + +static void skb_splice_csum_page(struct sk_buff *skb, struct page *page, + size_t offset, size_t len) +{ + const char *kaddr; + __wsum csum; + + kaddr = kmap_local_page(page); + csum = csum_partial(kaddr + offset, len, 0); + kunmap_local(kaddr); + skb->csum = csum_block_add(skb->csum, csum, skb->len); +} + +/** + * skb_splice_from_iter - Splice (or copy) pages to skbuff + * @skb: The buffer to add pages to + * @iter: Iterator representing the pages to be added + * @maxsize: Maximum amount of pages to be added + * @gfp: Allocation flags + * + * This is a common helper function for supporting MSG_SPLICE_PAGES. It + * extracts pages from an iterator and adds them to the socket buffer if + * possible, copying them to fragments if not possible (such as if they're slab + * pages). + * + * Returns the amount of data spliced/copied or -EMSGSIZE if there's + * insufficient space in the buffer to transfer anything. + */ +ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, + ssize_t maxsize, gfp_t gfp) +{ + size_t frag_limit = READ_ONCE(sysctl_max_skb_frags); + struct page *pages[8], **ppages = pages; + ssize_t spliced = 0, ret = 0; + unsigned int i; + + while (iter->count > 0) { + ssize_t space, nr; + size_t off, len; + + ret = -EMSGSIZE; + space = frag_limit - skb_shinfo(skb)->nr_frags; + if (space < 0) + break; + + /* We might be able to coalesce without increasing nr_frags */ + nr = clamp_t(size_t, space, 1, ARRAY_SIZE(pages)); + + len = iov_iter_extract_pages(iter, &ppages, maxsize, nr, 0, &off); + if (len <= 0) { + ret = len ?: -EIO; + break; + } + + i = 0; + do { + struct page *page = pages[i++]; + size_t part = min_t(size_t, PAGE_SIZE - off, len); + + ret = -EIO; + if (WARN_ON_ONCE(!sendpage_ok(page))) + goto out; + + ret = skb_append_pagefrags(skb, page, off, part, + frag_limit); + if (ret < 0) { + iov_iter_revert(iter, len); + goto out; + } + + if (skb->ip_summed == CHECKSUM_NONE) + skb_splice_csum_page(skb, page, off, part); + + off = 0; + spliced += part; + maxsize -= part; + len -= part; + } while (len > 0); + + if (maxsize <= 0) + break; + } + +out: + skb_len_add(skb, spliced); + return spliced ?: ret; +} +EXPORT_SYMBOL(skb_splice_from_iter); From patchwork Thu May 18 11:34:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95823 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp425210vqo; Thu, 18 May 2023 04:45:27 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4QvZ4gDYWCQFLCHwCVKObd3zxO5IzgAk76XetQGbco9KMSGruqp8KxEU9RzVXgL/OUSzNE X-Received: by 2002:a17:90a:a886:b0:24d:ebf8:b228 with SMTP id h6-20020a17090aa88600b0024debf8b228mr2276454pjq.19.1684410327491; Thu, 18 May 2023 04:45:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684410327; cv=none; d=google.com; s=arc-20160816; b=is7lbNTuCKTmxfreWx9Vgt368+J0lMirTWFkUyZ7YM+uo+rf6N/vllFTUHXjTjMVn2 m3BWnaoBC4D6C3TxmywCuKdgAei7ZPKH30UUQtfY1DRdSnpsZY3PIPNkp4V/0mYKprXv 9P/vbgNWdcJH9EgaKzsaWdcLzpNLgfGUhKO0uiSNgfgotjY+J0egtaBgK1tIZ3fB/Fr0 swB5Ft5oFUx439bX+OHyWOY3VwlmyZzShVjo9Vxh2RPg4E+5RCci56xUTosDFn6Gkbq+ OGtKqYcE3zlv//eyRg2MXYvsggPH+E+YbBQ1JkkHNwqWq7tdYMwZRGEWXVjpNbhod7gM 8kEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=DZXfiar5CKohUwsPymjzwnCAT+wFqXG5Kam/6y6M8pg=; b=Eb34xmhy573UaYVI59/KsqNIbr8IEd3mZz1cuGe1chUZjt1iJ4KNjdnachdMLgrQOJ LbHtZ3PdnJfV2dI/K/V4Ag9qj92KfQXwokDsLkJS3FFO6kvAgitqUCq68m5m/HuIUEAe Cb/hOFOrI+LCqhnR/N30S03tC70i9CqzPuAcp+c2ap4pi/uAC/VgOl2hcGNmonQTTPGb FXjqO9dSfyxxG0KmcZVwqrUMirGK1sUumefmYqOZMBAK99uAfvTP+w1lOgKkHFEZo6EX rGipPekgjTpq4QgbuRAZS40cdBdeFhEve21jdnSAY7+g0dLBaEHzphpwW+IuK32z0YND E12Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Uo9imcSK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l70-20020a638849000000b00528595a1390si1209255pgd.588.2023.05.18.04.45.12; Thu, 18 May 2023 04:45:27 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Uo9imcSK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231389AbjERLg2 (ORCPT + 99 others); Thu, 18 May 2023 07:36:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59764 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231334AbjERLgS (ORCPT ); Thu, 18 May 2023 07:36:18 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 02A00F4 for ; Thu, 18 May 2023 04:35:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684409726; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DZXfiar5CKohUwsPymjzwnCAT+wFqXG5Kam/6y6M8pg=; b=Uo9imcSKOxGHt04h/Z0yvufWqa5Ya8500vmKhGDOmW9uZJ4Mn90aXM5CX4kvgudpD8RcL0 ALJxWlGFHIF5hLwXFfPkMoaSpKodEHC3/cJgIaXVf/SwZ3LhmBsVCWRHb22fJWO+cqmTBq EXwWPEU6sj3LZV6YLZA2PcdKKqtLTmc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-206-lw-qVOlBOruNJ3l5NNmWVQ-1; Thu, 18 May 2023 07:35:21 -0400 X-MC-Unique: lw-qVOlBOruNJ3l5NNmWVQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D7B0C10146FE; Thu, 18 May 2023 11:35:20 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 78A2B492B01; Thu, 18 May 2023 11:35:16 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v8 04/16] tcp: Support MSG_SPLICE_PAGES Date: Thu, 18 May 2023 12:34:41 +0100 Message-Id: <20230518113453.1350757-5-dhowells@redhat.com> In-Reply-To: <20230518113453.1350757-1-dhowells@redhat.com> References: <20230518113453.1350757-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766232243133791494?= X-GMAIL-MSGID: =?utf-8?q?1766232243133791494?= Make TCP's sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced or copied (if it cannot be spliced) from the source iterator. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #7) - Missed a "zc = 1" in tcp_sendmsg_locked(). ver #6) - Set zc to 0/MSG_ZEROCOPY/MSG_SPLICE_PAGES rather than 0/1/2. - Use common helper. net/ipv4/tcp.c | 43 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 36 insertions(+), 7 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 4d6392c16b7a..026f483f42e3 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1223,7 +1223,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) int flags, err, copied = 0; int mss_now = 0, size_goal, copied_syn = 0; int process_backlog = 0; - bool zc = false; + int zc = 0; long timeo; flags = msg->msg_flags; @@ -1234,17 +1234,22 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (msg->msg_ubuf) { uarg = msg->msg_ubuf; net_zcopy_get(uarg); - zc = sk->sk_route_caps & NETIF_F_SG; + if (sk->sk_route_caps & NETIF_F_SG) + zc = MSG_ZEROCOPY; } else if (sock_flag(sk, SOCK_ZEROCOPY)) { uarg = msg_zerocopy_realloc(sk, size, skb_zcopy(skb)); if (!uarg) { err = -ENOBUFS; goto out_err; } - zc = sk->sk_route_caps & NETIF_F_SG; - if (!zc) + if (sk->sk_route_caps & NETIF_F_SG) + zc = MSG_ZEROCOPY; + else uarg_to_msgzc(uarg)->zerocopy = 0; } + } else if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES) && size) { + if (sk->sk_route_caps & NETIF_F_SG) + zc = MSG_SPLICE_PAGES; } if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect) && @@ -1307,7 +1312,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) goto do_error; while (msg_data_left(msg)) { - int copy = 0; + ssize_t copy = 0; skb = tcp_write_queue_tail(sk); if (skb) @@ -1348,7 +1353,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (copy > msg_data_left(msg)) copy = msg_data_left(msg); - if (!zc) { + if (zc == 0) { bool merge = true; int i = skb_shinfo(skb)->nr_frags; struct page_frag *pfrag = sk_page_frag(sk); @@ -1393,7 +1398,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) page_ref_inc(pfrag->page); } pfrag->offset += copy; - } else { + } else if (zc == MSG_ZEROCOPY) { /* First append to a fragless skb builds initial * pure zerocopy skb */ @@ -1414,6 +1419,30 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (err < 0) goto do_error; copy = err; + } else if (zc == MSG_SPLICE_PAGES) { + /* Splice in data if we can; copy if we can't. */ + if (tcp_downgrade_zcopy_pure(sk, skb)) + goto wait_for_space; + copy = tcp_wmem_schedule(sk, copy); + if (!copy) + goto wait_for_space; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) { + if (err == -EMSGSIZE) { + tcp_mark_push(tp, skb); + goto new_segment; + } + goto do_error; + } + copy = err; + + if (!(flags & MSG_NO_SHARED_FRAGS)) + skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; + + sk_wmem_queued_add(sk, copy); + sk_mem_charge(sk, copy); } if (!copied) From patchwork Thu May 18 11:34:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95829 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp426917vqo; Thu, 18 May 2023 04:48:37 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ65O16lQWOEk3bCD+/9z4zDG6rCTVFZAc7R2MuJ9xVfzh9TUBJ52dC9c031dQVAm8Ckp1nQ X-Received: by 2002:a17:902:f7c2:b0:1aa:fbaa:ee09 with SMTP id h2-20020a170902f7c200b001aafbaaee09mr1821045plw.49.1684410517313; Thu, 18 May 2023 04:48:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684410517; cv=none; d=google.com; s=arc-20160816; b=uIxt/vvMKq6nGE1iV3zgnLWQ9013V1KaerHR2zgBFtpGn7oh2AVTIU0Xy/eK2hiBEg 5912gbA6Ni3E6LetVyvGrudtbYeTpPy5V6H8OPF32uj5mjF+j7gjHLYUWPX8Yucs+B0p c9WXTVfcg5jZL/qs927iEYdXHwF21w6Q6xMouClYN9TiSSDXaNgDYV7zU0kHCK1y9LES gye469fHDtXmzUZiFy1dEd/UOqwFqfqo9SJoVpgAhZd0QHgNwCnZSax7KTg8tIYbRpWF zMVxAIPpOjo2TeFD05KhciOQIvlFf5fX4RQ/vAIf/NMe1Bl4y48QoI9yfejtOg8TUt+A NO3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9s33rFBaxABfh3fue+YSREMQMkxGRapWweS5v7j7klU=; b=QHz/FqsueM2DSt6URD/UOOxGnN+rQQdiEaC4tA2IgfUeC0f8hwYl6oRn8mME+saFB3 AAYOAzqmbjHwvRNP4gOvVBtrWJNlTinKn5+vtLTLE5ccYdrO86DWiXFZUtF5vOoxg6cl EBWRgSW/C9q8NLldlbQVsslIVkQnWdS17RtY3nXMD6tMY9Vqxd/gxvON5tkrcXfmbZ3P aN+Qz167nmI6A95aTD0xkhVsHaDJbUtJ4IfpLiUd+65H8SysMwuVUrrtf+/jgZdC8aHK gToae0OYAkwwwRavDhzVhoOperw5VM+uzTqJgrAaJUQUq8N7D8xDd9hdrJk9t1jrZZ4C Pm8Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="S3YSy/vN"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u8-20020a17090341c800b001aafea6f85esi1141004ple.435.2023.05.18.04.48.21; Thu, 18 May 2023 04:48:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="S3YSy/vN"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231370AbjERLgn (ORCPT + 99 others); Thu, 18 May 2023 07:36:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59888 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231365AbjERLgX (ORCPT ); Thu, 18 May 2023 07:36:23 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 81F90C1 for ; Thu, 18 May 2023 04:35:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684409731; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9s33rFBaxABfh3fue+YSREMQMkxGRapWweS5v7j7klU=; b=S3YSy/vNTD4QzdIaoPvB/HLDy4M2H/NX/ntLqyZdSlYQJJs6FLf9bmTV4LM3oBVjfl+owE 3q8pfTlzdChC8NPzW568xwri9wCWNzWZmFK4oFMzSSK75cCOa34YrpqNh5NklZRC6oEZzd zue3jGzUjj69mFjt0NIWhvSHC5tYt7I= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-474-w1z0J2L2NUqr5n53vZ04mA-1; Thu, 18 May 2023 07:35:26 -0400 X-MC-Unique: w1z0J2L2NUqr5n53vZ04mA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 75BFF86C60A; Thu, 18 May 2023 11:35:25 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 44FBAC15BA0; Thu, 18 May 2023 11:35:22 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v8 05/16] tcp: Convert do_tcp_sendpages() to use MSG_SPLICE_PAGES Date: Thu, 18 May 2023 12:34:42 +0100 Message-Id: <20230518113453.1350757-6-dhowells@redhat.com> In-Reply-To: <20230518113453.1350757-1-dhowells@redhat.com> References: <20230518113453.1350757-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766232442337115450?= X-GMAIL-MSGID: =?utf-8?q?1766232442337115450?= Convert do_tcp_sendpages() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. do_tcp_sendpages() can then be inlined in subsequent patches into its callers. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/ipv4/tcp.c | 158 +++---------------------------------------------- 1 file changed, 7 insertions(+), 151 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 026f483f42e3..28231e503af9 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -974,163 +974,19 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -static struct sk_buff *tcp_build_frag(struct sock *sk, int size_goal, int flags, - struct page *page, int offset, size_t *size) -{ - struct sk_buff *skb = tcp_write_queue_tail(sk); - struct tcp_sock *tp = tcp_sk(sk); - bool can_coalesce; - int copy, i; - - if (!skb || (copy = size_goal - skb->len) <= 0 || - !tcp_skb_can_collapse_to(skb)) { -new_segment: - if (!sk_stream_memory_free(sk)) - return NULL; - - skb = tcp_stream_alloc_skb(sk, 0, sk->sk_allocation, - tcp_rtx_and_write_queues_empty(sk)); - if (!skb) - return NULL; - -#ifdef CONFIG_TLS_DEVICE - skb->decrypted = !!(flags & MSG_SENDPAGE_DECRYPTED); -#endif - tcp_skb_entail(sk, skb); - copy = size_goal; - } - - if (copy > *size) - copy = *size; - - i = skb_shinfo(skb)->nr_frags; - can_coalesce = skb_can_coalesce(skb, i, page, offset); - if (!can_coalesce && i >= READ_ONCE(sysctl_max_skb_frags)) { - tcp_mark_push(tp, skb); - goto new_segment; - } - if (tcp_downgrade_zcopy_pure(sk, skb)) - return NULL; - - copy = tcp_wmem_schedule(sk, copy); - if (!copy) - return NULL; - - if (can_coalesce) { - skb_frag_size_add(&skb_shinfo(skb)->frags[i - 1], copy); - } else { - get_page(page); - skb_fill_page_desc_noacc(skb, i, page, offset, copy); - } - - if (!(flags & MSG_NO_SHARED_FRAGS)) - skb_shinfo(skb)->flags |= SKBFL_SHARED_FRAG; - - skb->len += copy; - skb->data_len += copy; - skb->truesize += copy; - sk_wmem_queued_add(sk, copy); - sk_mem_charge(sk, copy); - WRITE_ONCE(tp->write_seq, tp->write_seq + copy); - TCP_SKB_CB(skb)->end_seq += copy; - tcp_skb_pcount_set(skb, 0); - - *size = copy; - return skb; -} - ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tcp_sock *tp = tcp_sk(sk); - int mss_now, size_goal; - int err; - ssize_t copied; - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); - - if (IS_ENABLED(CONFIG_DEBUG_VM) && - WARN_ONCE(!sendpage_ok(page), - "page must not be a Slab one and have page_count > 0")) - return -EINVAL; - - /* Wait for a connection to finish. One exception is TCP Fast Open - * (passive side) where data is allowed to be sent before a connection - * is fully established. - */ - if (((1 << sk->sk_state) & ~(TCPF_ESTABLISHED | TCPF_CLOSE_WAIT)) && - !tcp_passive_fastopen(sk)) { - err = sk_stream_wait_connect(sk, &timeo); - if (err != 0) - goto out_err; - } + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); - mss_now = tcp_send_mss(sk, &size_goal, flags); - copied = 0; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - err = -EPIPE; - if (sk->sk_err || (sk->sk_shutdown & SEND_SHUTDOWN)) - goto out_err; - - while (size > 0) { - struct sk_buff *skb; - size_t copy = size; - - skb = tcp_build_frag(sk, size_goal, flags, page, offset, ©); - if (!skb) - goto wait_for_space; - - if (!copied) - TCP_SKB_CB(skb)->tcp_flags &= ~TCPHDR_PSH; - - copied += copy; - offset += copy; - size -= copy; - if (!size) - goto out; - - if (skb->len < size_goal || (flags & MSG_OOB)) - continue; - - if (forced_push(tp)) { - tcp_mark_push(tp, skb); - __tcp_push_pending_frames(sk, mss_now, TCP_NAGLE_PUSH); - } else if (skb == tcp_send_head(sk)) - tcp_push_one(sk, mss_now); - continue; - -wait_for_space: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); - tcp_push(sk, flags & ~MSG_MORE, mss_now, - TCP_NAGLE_PUSH, size_goal); - - err = sk_stream_wait_memory(sk, &timeo); - if (err != 0) - goto do_error; - - mss_now = tcp_send_mss(sk, &size_goal, flags); - } - -out: - if (copied) { - tcp_tx_timestamp(sk, sk->sk_tsflags); - if (!(flags & MSG_SENDPAGE_NOTLAST)) - tcp_push(sk, flags, mss_now, tp->nonagle, size_goal); - } - return copied; - -do_error: - tcp_remove_empty_skb(sk); - if (copied) - goto out; -out_err: - /* make sure we wake any epoll edge trigger waiter */ - if (unlikely(tcp_rtx_and_write_queues_empty(sk) && err == -EAGAIN)) { - sk->sk_write_space(sk); - tcp_chrono_stop(sk, TCP_CHRONO_SNDBUF_LIMITED); - } - return sk_stream_error(sk, flags, err); + return tcp_sendmsg_locked(sk, &msg, size); } EXPORT_SYMBOL_GPL(do_tcp_sendpages); From patchwork Thu May 18 11:34:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95821 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp424280vqo; Thu, 18 May 2023 04:43:34 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ53fddghQ4remwMI7ROuUJnkQJnNWYDBp6tlGOJhhex0J25Z5vYi2Owl4A8j0hsvo5iMTNZ X-Received: by 2002:a05:6a20:c907:b0:104:ffd0:2338 with SMTP id gx7-20020a056a20c90700b00104ffd02338mr1667690pzb.26.1684410213553; Thu, 18 May 2023 04:43:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684410213; cv=none; d=google.com; s=arc-20160816; b=dTjowmYORPlzDMVh4Ci0Nrf5oWxh4ewh0n9cmQiNNBlJVpdkLOxyEvgKxLUrOiBTfk Lgm+1ddvds0nX1/VvzGy3DGD7WmbGLdQUnfLghQvVwmGLbm7GMKAExGIPikqavr/B4OT 4i8cZctwt4GU/DN2quqQlQDTXzfj2SEyGznzLAXbzvjiD1LIx8U7weahXLUQa/QP03bY HWtW3QFdxN2afhlViqA5ns3YK3ciJtwj7lcK6Qx81r/cILnDrqdS42zkzb0KYHlpzUji jhvVlq2Uyi4koDSPMvAnIi7GrscCyvQVvbqFcI38w2fXs93MGuViQq4BI7VDxUMc3GXT 5CHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2EXLlBUI8HMrlq1LdzuePmnE9vo0jChMg019VHUJEgs=; b=lwzQ1mhpnrS3tODnzOL7IaOWgxwvZBbI5CDFK6BU6M3tNiqVcyDEngV8ogRFUrGtYB TvxjGQGoOYYX5x0G6S06afhxo73oC9DjP80c8e6TveIgIP4I0AX/KfepWgww8tSIXw8M IKfpld2UDmrQD2R6/nwJpCNeBMF8KGV0gMzUsS81KX241plzXcHes9q0W8hhSbWB+9FV SLoO7mozYGcEUivONg31ckZ6fm4Cwg/Fd2pIrg7aDT51ummY9px579at7jOYTenFpOGd NRCFCu5sP/6L7aUcHts760QFs8kxUZT+p3Oqhv8YFsAf/3/SVNAu66vx96ZC71Cw1ZHk baRw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=QwPdJ3Pq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b6-20020a637146000000b00528c2cfc421si1201119pgn.880.2023.05.18.04.43.18; Thu, 18 May 2023 04:43:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=QwPdJ3Pq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231383AbjERLgo (ORCPT + 99 others); Thu, 18 May 2023 07:36:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231340AbjERLgY (ORCPT ); Thu, 18 May 2023 07:36:24 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 936C5100 for ; Thu, 18 May 2023 04:35:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684409733; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2EXLlBUI8HMrlq1LdzuePmnE9vo0jChMg019VHUJEgs=; b=QwPdJ3PqCF66n1nSTJtH3dnojHaceQppcgdF438zTiJnVGPlLrj5bS+2zEd1PFzV+djhMt qfYoLWtmI7dSkKzNQLCbA24qlwQ9OJ2+atiPyDjO5jORgCmAtEuho3wiHMLPb/i7d8H+hE 5bzOsyOqVaweQdUfnvgMiMDkg7+U45Q= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-84-Mvf0PKrvOkyxjyMevBqNLQ-1; Thu, 18 May 2023 07:35:29 -0400 X-MC-Unique: Mvf0PKrvOkyxjyMevBqNLQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C1BF83C10221; Thu, 18 May 2023 11:35:28 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 27987492B01; Thu, 18 May 2023 11:35:26 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, John Fastabend , Jakub Sitnicki , bpf@vger.kernel.org Subject: [PATCH net-next v8 06/16] tcp_bpf: Inline do_tcp_sendpages as it's now a wrapper around tcp_sendmsg Date: Thu, 18 May 2023 12:34:43 +0100 Message-Id: <20230518113453.1350757-7-dhowells@redhat.com> In-Reply-To: <20230518113453.1350757-1-dhowells@redhat.com> References: <20230518113453.1350757-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766232123922147619?= X-GMAIL-MSGID: =?utf-8?q?1766232123922147619?= do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: John Fastabend cc: Jakub Sitnicki cc: "David S. Miller" cc: Eric Dumazet cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: bpf@vger.kernel.org --- net/ipv4/tcp_bpf.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c index 2e9547467edb..0291d15acd19 100644 --- a/net/ipv4/tcp_bpf.c +++ b/net/ipv4/tcp_bpf.c @@ -72,11 +72,13 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, { bool apply = apply_bytes; struct scatterlist *sge; + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct page *page; int size, ret = 0; u32 off; while (1) { + struct bio_vec bvec; bool has_tx_ulp; sge = sk_msg_elem(msg, msg->sg.start); @@ -88,16 +90,18 @@ static int tcp_bpf_push(struct sock *sk, struct sk_msg *msg, u32 apply_bytes, tcp_rate_check_app_limited(sk); retry: has_tx_ulp = tls_sw_has_ctx_tx(sk); - if (has_tx_ulp) { - flags |= MSG_SENDPAGE_NOPOLICY; - ret = kernel_sendpage_locked(sk, - page, off, size, flags); - } else { - ret = do_tcp_sendpages(sk, page, off, size, flags); - } + if (has_tx_ulp) + msghdr.msg_flags |= MSG_SENDPAGE_NOPOLICY; + if (flags & MSG_SENDPAGE_NOTLAST) + msghdr.msg_flags |= MSG_MORE; + + bvec_set_page(&bvec, page, size, off); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret <= 0) return ret; + if (apply) apply_bytes -= ret; msg->sg.size -= ret; @@ -404,7 +408,7 @@ static int tcp_bpf_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) long timeo; int flags; - /* Don't let internal do_tcp_sendpages() flags through */ + /* Don't let internal sendpage flags through */ flags = (msg->msg_flags & ~MSG_SENDPAGE_DECRYPTED); flags |= MSG_NO_SHARED_FRAGS; From patchwork Thu May 18 11:34:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95825 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp426484vqo; Thu, 18 May 2023 04:47:44 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5+liOb1UPdrc7lCnsTnAC9eopau8m+7B0Vfe9RvZOKY7xx5EM/gjseB9KN/gD+r3+p7O1z X-Received: by 2002:a05:6a20:9385:b0:102:a593:a162 with SMTP id x5-20020a056a20938500b00102a593a162mr2299064pzh.4.1684410463998; Thu, 18 May 2023 04:47:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684410463; cv=none; d=google.com; s=arc-20160816; b=wCaUnjrE0o9tug8RFQXvBBS4P+MpXgfUPMYji8JuWo/u7E0xuvMfbnb/tmIU4E9C1/ rATCUEE40u4x1+IQLA4gDkacB5eNPDisMhWRNnLOLrwjzQR3FlU3jUydH2WjqRhCWacH vPy3sMRPNhmHOKvFpxYMoEnjOH43g8RhmgPwSm9lEmED3aUh/mnpW2u2Kp98YEHfiLWk m21x0Ns60elsGCyDYBO1WGZLsY1FdDF+r9SR/x3VmfV4CrT0HRjgtxhCvLZluexW1oTL vozfExcaNIgp4Nrb58ICblEhM0VdGlR9TU+vTSYFpY9KfLZ0etAfcKAeboE0DATCbDXm 8tSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=tYeoTX9B8dad49qWPHePVLYzX6oyR1jBlVymL6Q5Zsc=; b=BE+SuEY+mLyGcSXbjApmWoOVKnEyxWSaci3Pt6N280mO7k1WdusrNjp7fct80RuGZ9 ktbVGxjUK302Wp4tI3NpeDgkGDUk1xoZkdWrFG/66E18l96RMwzEpxSN8Uovura8HXTm U5hPdMLo3ajg2FBfqEzAZiJRchDDs4SwSBc24DS1zpBpioAPljUDgQaauMBEypJ2xJ/K DSc0fmMqbgDiIYfXB1pnrWdvL4Oa0IiJvJqj06feEo/UzGMxmaxHQ8PLj2RQk0fMJCX7 VM88/LCJS/d0KdhHI1sreauDmVYJQWq/sWQnAZFGGMyCE2kkQst/1rb9q20FG/5ukEAg Dn6A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=IkpdP4dY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b6-20020a637146000000b00528c2cfc421si1201119pgn.880.2023.05.18.04.47.28; Thu, 18 May 2023 04:47:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=IkpdP4dY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231397AbjERLg5 (ORCPT + 99 others); Thu, 18 May 2023 07:36:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59962 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231392AbjERLga (ORCPT ); Thu, 18 May 2023 07:36:30 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6167C1A7 for ; Thu, 18 May 2023 04:35:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684409738; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tYeoTX9B8dad49qWPHePVLYzX6oyR1jBlVymL6Q5Zsc=; b=IkpdP4dYySEBDFP3NmdBkEPmbop7YFSufs50b6Qepgd+hJPxvCq+tGwbuNQE4iOxh9EVCa bXY1c9ycz85dxWDOBmiM6095FIj+gSbXevzZ0ve70uCFDgO6NRnekG/zDtxwu0FPVEmMkD JLe/EpN4CqtEYN7a8QT9VIE+C0yhU4A= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-214-HshzOJ6CNxe9-YY1obDoFw-1; Thu, 18 May 2023 07:35:35 -0400 X-MC-Unique: HshzOJ6CNxe9-YY1obDoFw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 724DA1C09512; Thu, 18 May 2023 11:35:34 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id E91B3492B01; Thu, 18 May 2023 11:35:29 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Steffen Klassert , Herbert Xu Subject: [PATCH net-next v8 07/16] espintcp: Inline do_tcp_sendpages() Date: Thu, 18 May 2023 12:34:44 +0100 Message-Id: <20230518113453.1350757-8-dhowells@redhat.com> In-Reply-To: <20230518113453.1350757-1-dhowells@redhat.com> References: <20230518113453.1350757-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766232386452590751?= X-GMAIL-MSGID: =?utf-8?q?1766232386452590751?= do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Steffen Klassert cc: Herbert Xu cc: Eric Dumazet cc: "David S. Miller" cc: David Ahern cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/xfrm/espintcp.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/net/xfrm/espintcp.c b/net/xfrm/espintcp.c index 872b80188e83..3504925babdb 100644 --- a/net/xfrm/espintcp.c +++ b/net/xfrm/espintcp.c @@ -205,14 +205,16 @@ static int espintcp_sendskb_locked(struct sock *sk, struct espintcp_msg *emsg, static int espintcp_sendskmsg_locked(struct sock *sk, struct espintcp_msg *emsg, int flags) { + struct msghdr msghdr = { .msg_flags = flags | MSG_SPLICE_PAGES, }; struct sk_msg *skmsg = &emsg->skmsg; struct scatterlist *sg; int done = 0; int ret; - flags |= MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags |= MSG_SENDPAGE_NOTLAST; sg = &skmsg->sg.data[skmsg->sg.start]; do { + struct bio_vec bvec; size_t size = sg->length - emsg->offset; int offset = sg->offset + emsg->offset; struct page *p; @@ -220,11 +222,13 @@ static int espintcp_sendskmsg_locked(struct sock *sk, emsg->offset = 0; if (sg_is_last(sg)) - flags &= ~MSG_SENDPAGE_NOTLAST; + msghdr.msg_flags &= ~MSG_SENDPAGE_NOTLAST; p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msghdr.msg_iter, ITER_SOURCE, &bvec, 1, size); + ret = tcp_sendmsg_locked(sk, &msghdr, size); if (ret < 0) { emsg->offset = offset - sg->offset; skmsg->sg.start += done; From patchwork Thu May 18 11:34:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95820 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp424074vqo; Thu, 18 May 2023 04:43:08 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4oRPZ9P9raoHd7CZCzxN76Lw0uxbfdjwcOTpzfKjUM+lW0YICUoIEkjLO1Jm4ZjWeiaIUS X-Received: by 2002:a05:6a00:24d0:b0:64c:ecf7:f492 with SMTP id d16-20020a056a0024d000b0064cecf7f492mr4710734pfv.4.1684410187921; Thu, 18 May 2023 04:43:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684410187; cv=none; d=google.com; s=arc-20160816; b=C7o5rxASmmeQaAr1YJl/d7ZfuuisbRhF5YTmuXZcQNEKSwRI2tc/BO1vP+h7M4reSx jINXecCzudwzU0Yii6Lo+oRtiDuebSZq10J6n5PT4upnnXbxM+DmSmI6xekKKljCZhJ3 XZ8qur6OUaKTiZR1E5LxQp6uqVb3GjpD7yFCkJ3IwktZC2erOMuYMozVJLBIM0gtDXgG 7ilTGK2uMmghyJ+t26DNiEw/CKcysKPElr9z65weGyBx6axcQhwSJemAr1UCe65+HrhJ P0fx3TU9UlN+slJaIxWF2vc3uD9mJwmTOgBdswCXo4IsXiEePkkXaBOk3xHN5MsCJIsL 4Dpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JUrQ0rsOJEKe2VauAy4RSjWfQDMPveXienxXCaL8A+w=; b=pp3nK/QpgHcX9rg2OYDgoITZYy/a8CC3+NGgvvwYuz2BxxHZNjyh/HN6vtYO2Kz1sN RugeB7gQ1KwIPKHFk7kllBtdpvW06nL2+lbMonUKPjigAeXWjBXi4dx0w4u/jsWLhWU4 ri59RMK4gVdQfDhZSLd9trRAu2vRqyPQW/OTjcEJWAPYiapdOH3H6oBJq+eLDbR5tTFt rn+Dtd7VEQoqYCf3H5pUF2Syy6RoTBRI3UbEo1QF7du1k40eb58mDNFwqaC4/qC6/Vv/ /1OUGoK9UHml3+3B9Lw+3rMICO+vOntrpJ1Ngf87s1ryFNWcO9hs38vkXio9MKgSbI9s v2YA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=TsXxBTU1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a24-20020aa794b8000000b0063806e9892fsi1471453pfl.243.2023.05.18.04.42.52; Thu, 18 May 2023 04:43:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=TsXxBTU1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231473AbjERLhD (ORCPT + 99 others); Thu, 18 May 2023 07:37:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231395AbjERLge (ORCPT ); Thu, 18 May 2023 07:36:34 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 60CBAF9 for ; Thu, 18 May 2023 04:35:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684409743; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JUrQ0rsOJEKe2VauAy4RSjWfQDMPveXienxXCaL8A+w=; b=TsXxBTU1hH9CI00NjsUwKx7nau6dVC5OfcLGkyq4ooRuVarY/3ZBbzwrstkq25xianot5Y SEtFWCg3tcEhm0+rDqooHoyNpWPLZBk9dC6QhK7FEoNxMgJjxPpE/nXD3IfmWreHqC0BkI lwXKl1e1uxCWiIh/JV2rqHaHZ6mzcWU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-575-Of9dg0BOOn66lre7Opvf8A-1; Thu, 18 May 2023 07:35:39 -0400 X-MC-Unique: Of9dg0BOOn66lre7Opvf8A-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5D25829DD980; Thu, 18 May 2023 11:35:38 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id D496C2166B31; Thu, 18 May 2023 11:35:35 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Boris Pismenny , John Fastabend Subject: [PATCH net-next v8 08/16] tls: Inline do_tcp_sendpages() Date: Thu, 18 May 2023 12:34:45 +0100 Message-Id: <20230518113453.1350757-9-dhowells@redhat.com> In-Reply-To: <20230518113453.1350757-1-dhowells@redhat.com> References: <20230518113453.1350757-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766232097335903391?= X-GMAIL-MSGID: =?utf-8?q?1766232097335903391?= do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: "David S. Miller" cc: Eric Dumazet cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tls.h | 2 +- net/tls/tls_main.c | 24 +++++++++++++++--------- 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/include/net/tls.h b/include/net/tls.h index 6056ce5a2aa5..5791ca7a189c 100644 --- a/include/net/tls.h +++ b/include/net/tls.h @@ -258,7 +258,7 @@ struct tls_context { struct scatterlist *partially_sent_record; u16 partially_sent_offset; - bool in_tcp_sendpages; + bool splicing_pages; bool pending_open_record_frags; struct mutex tx_lock; /* protects partially_sent_* fields and diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c index f2e7302a4d96..3d45fdb5c4e9 100644 --- a/net/tls/tls_main.c +++ b/net/tls/tls_main.c @@ -125,7 +125,10 @@ int tls_push_sg(struct sock *sk, u16 first_offset, int flags) { - int sendpage_flags = flags | MSG_SENDPAGE_NOTLAST; + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = MSG_SENDPAGE_NOTLAST | MSG_SPLICE_PAGES | flags, + }; int ret = 0; struct page *p; size_t size; @@ -134,16 +137,19 @@ int tls_push_sg(struct sock *sk, size = sg->length - offset; offset += sg->offset; - ctx->in_tcp_sendpages = true; + ctx->splicing_pages = true; while (1) { if (sg_is_last(sg)) - sendpage_flags = flags; + msg.msg_flags = flags; /* is sending application-limited? */ tcp_rate_check_app_limited(sk); p = sg_page(sg); retry: - ret = do_tcp_sendpages(sk, p, offset, size, sendpage_flags); + bvec_set_page(&bvec, p, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + + ret = tcp_sendmsg_locked(sk, &msg, size); if (ret != size) { if (ret > 0) { @@ -155,7 +161,7 @@ int tls_push_sg(struct sock *sk, offset -= sg->offset; ctx->partially_sent_offset = offset; ctx->partially_sent_record = (void *)sg; - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return ret; } @@ -169,7 +175,7 @@ int tls_push_sg(struct sock *sk, size = sg->length; } - ctx->in_tcp_sendpages = false; + ctx->splicing_pages = false; return 0; } @@ -247,11 +253,11 @@ static void tls_write_space(struct sock *sk) { struct tls_context *ctx = tls_get_ctx(sk); - /* If in_tcp_sendpages call lower protocol write space handler + /* If splicing_pages call lower protocol write space handler * to ensure we wake up any waiting operations there. For example - * if do_tcp_sendpages where to call sk_wait_event. + * if splicing pages where to call sk_wait_event. */ - if (ctx->in_tcp_sendpages) { + if (ctx->splicing_pages) { ctx->sk_write_space(sk); return; } From patchwork Thu May 18 11:34:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95810 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp422646vqo; Thu, 18 May 2023 04:40:16 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4sV4xFH/XJXFslKPS5OmM1W1/5OTagbJXKXVFzszK6YQhRcyptv53Y1GqkkVCISfdPvJfl X-Received: by 2002:a05:6a20:3d19:b0:109:bdaf:6b45 with SMTP id y25-20020a056a203d1900b00109bdaf6b45mr877864pzi.5.1684410016300; Thu, 18 May 2023 04:40:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684410016; cv=none; d=google.com; s=arc-20160816; b=pw2oeyMHqDIGZr6QPV6BQah+wAywF/AQ9w87Kaj8SRGfpY3xCvB2MghlpxtHUcF/Wu pHA0EGK8PANifyOk2EzA86gNl/wqi8wWdeX3gX8XA7Us6B8kg95G7Bo3Z5rKl0fwOhGW Mswp9aGKQSpkXwmEn88H4a9xaPx/5rztR0xpgL1M0qEbqrRkZ+qaQXbBL07ll+v2Ht/e imDaWchjx1xkBeYpLf98pgLq6XSKMab6e6ISDxSi2bl973oYbOWFv5UpyLHyU+ZTzt3Y QLBPL5bLN/BVZJWNzZlOsqqHiRvPpfe0F5cijgne79rcP0QKtbHKFNq/BkhKDdCZYvwn nDmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=tEe6KsGIUg5Ug9eymZUK4cq5L1lxhWlwpIXxPsGo3ck=; b=tiThXVNgb+ZB/QQuGrQvxToZYU+W0d3Jd5MxLFm/F1IyDWJ/UZ3YWaT4nFrTK8U0X5 9dDpa3Wdmr/ys7iQLNXsJOehA2C7Wc73+34yk8JL8DeIEtM5ptHRt4ohJ5XODEWFJweu t+Lfn1u2ZDkZcI+vUUFXMnNGTBQ7tQuK9TlE4bMy0XZxQVsjiPZPFdG2R8XbX/94+G9+ h1/b3JMalKGmHd16au5z5DVzXaNCOM4lSr5dZ04tkXRXusmWO6QI9q9CH7ztbTXYsgyq Y6jxRd+n662Fk1UcOWeVc+mRGvf0OUs/l2E44z48oM8I5NsO9ZOULZXLwlK8oDfbSJme gTTw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=VmBYGGsC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id lp15-20020a17090b4a8f00b0024666c4e8d8si2376939pjb.187.2023.05.18.04.40.00; Thu, 18 May 2023 04:40:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=VmBYGGsC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231348AbjERLhA (ORCPT + 99 others); Thu, 18 May 2023 07:37:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60098 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231394AbjERLge (ORCPT ); Thu, 18 May 2023 07:36:34 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D593B19A for ; Thu, 18 May 2023 04:35:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684409749; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tEe6KsGIUg5Ug9eymZUK4cq5L1lxhWlwpIXxPsGo3ck=; b=VmBYGGsCtwey0ieiDzNGTTLoxEM++51zyOWBJIjmttnbEMWZ3Kz3ucsC874e3PyOjOCCwr uR1F7MD0371tgPISw/XKB7T4usF7bTAN1yAavJuAVqUnhTs3eZr+hkgMe8LYwKUCX4f4In MEIQ7oNFZ+3RanCRP2jpOEzIkA4ocLs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-146-D-Rd9ENnNDKPveHNDxPUbw-1; Thu, 18 May 2023 07:35:43 -0400 X-MC-Unique: D-Rd9ENnNDKPveHNDxPUbw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B38E010146F8; Thu, 18 May 2023 11:35:42 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1878240C2063; Thu, 18 May 2023 11:35:39 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Bernard Metzler , Tom Talpey , Jason Gunthorpe , Leon Romanovsky , linux-rdma@vger.kernel.org Subject: [PATCH net-next v8 09/16] siw: Inline do_tcp_sendpages() Date: Thu, 18 May 2023 12:34:46 +0100 Message-Id: <20230518113453.1350757-10-dhowells@redhat.com> In-Reply-To: <20230518113453.1350757-1-dhowells@redhat.com> References: <20230518113453.1350757-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766231917511707575?= X-GMAIL-MSGID: =?utf-8?q?1766231917511707575?= do_tcp_sendpages() is now just a small wrapper around tcp_sendmsg_locked(), so inline it, allowing do_tcp_sendpages() to be removed. This is part of replacing ->sendpage() with a call to sendmsg() with MSG_SPLICE_PAGES set. Signed-off-by: David Howells Reviewed-by: Bernard Metzler Reviewed-by: Tom Talpey cc: Jason Gunthorpe cc: Leon Romanovsky cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: linux-rdma@vger.kernel.org cc: netdev@vger.kernel.org --- Notes: ver #6) - Don't clear MSG_SPLICE_PAGES on the last page. drivers/infiniband/sw/siw/siw_qp_tx.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_qp_tx.c b/drivers/infiniband/sw/siw/siw_qp_tx.c index 4b292e0504f1..ffb16beb6c30 100644 --- a/drivers/infiniband/sw/siw/siw_qp_tx.c +++ b/drivers/infiniband/sw/siw/siw_qp_tx.c @@ -312,7 +312,7 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, } /* - * 0copy TCP transmit interface: Use do_tcp_sendpages. + * 0copy TCP transmit interface: Use MSG_SPLICE_PAGES. * * Using sendpage to push page by page appears to be less efficient * than using sendmsg, even if data are copied. @@ -323,20 +323,27 @@ static int siw_tx_ctrl(struct siw_iwarp_tx *c_tx, struct socket *s, static int siw_tcp_sendpages(struct socket *s, struct page **page, int offset, size_t size) { + struct bio_vec bvec; + struct msghdr msg = { + .msg_flags = (MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST | + MSG_SPLICE_PAGES), + }; struct sock *sk = s->sk; - int i = 0, rv = 0, sent = 0, - flags = MSG_MORE | MSG_DONTWAIT | MSG_SENDPAGE_NOTLAST; + int i = 0, rv = 0, sent = 0; while (size) { size_t bytes = min_t(size_t, PAGE_SIZE - offset, size); if (size + offset <= PAGE_SIZE) - flags = MSG_MORE | MSG_DONTWAIT; + msg.msg_flags &= ~MSG_SENDPAGE_NOTLAST; tcp_rate_check_app_limited(sk); + bvec_set_page(&bvec, page[i], bytes, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + try_page_again: lock_sock(sk); - rv = do_tcp_sendpages(sk, page[i], offset, bytes, flags); + rv = tcp_sendmsg_locked(sk, &msg, size); release_sock(sk); if (rv > 0) { From patchwork Thu May 18 11:34:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95826 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp426751vqo; Thu, 18 May 2023 04:48:14 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ57/ciGRkMO6oXwzL1uY3bLnetT8l7agD9t8xnvFsAoeV5HK8iKgyZjoPxNYlLuJ7mfLxEj X-Received: by 2002:a17:902:c40a:b0:1a5:f9b:27bd with SMTP id k10-20020a170902c40a00b001a50f9b27bdmr2823680plk.34.1684410494732; Thu, 18 May 2023 04:48:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684410494; cv=none; d=google.com; s=arc-20160816; b=bRiRKK3b8Ez3UPuIYVi2AZaQvXQ3z16U9aVEumfBlrHhKNJuwbWiM0y2MWYabXtq67 pDLmfr6mYg/V4rb8hpCxNG0p9C0Ysk26EYm87EWRvO8naYbtmezmAnURE8dcliR6PSvH 7E+UM8fos/2onIu+CPP0yiEhuVP3VbxMPs9ZjAuYuFZFM70dj4jIZzLMAV4HwAzd4Xan 6zNRgBCqWM8OnWjekFSmoN0ajxiK+H8vpEjPiCpftRkVEYLgUStjATOrldiUEPO6zWWq kkV0k4iJh8igVD/67shiROy7KtXyepYLNnJ1kqiB0Dh8UOwos+4AMoeC7JeclMLn8PRr smQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=enBN5jT1muKlfAJKNsWEtZoec3VPKG1S27Z+IZtkOMs=; b=oJy0dXPT8ARNmVtsW05NfMgdnxJUEyxPvi25UkHlYGLG9Sf2+P9zh6lQc2XBZ24T1I 7b2OF/VzHaGkTgFG2CKlTabr7ZXUz3KciqJpD60S88BqQ1oUQrFtirqA+ivKj55k2jSX FFzrPxsFe7W3fE41/u3AgLv+eRgVjDbHlIxOwnjOQ7pDQTHEOLFKAKFefKM4b+4p6pN/ 0OZvB9C5O5MKd8q5uO7k1NqiRXkJYdW/RjKnO3Eovo/TFqtZts4DGxziRQNzS3Np+fQo j1xeCx9cJ27dqS8nKDH/mBaA54nicAx/NfUwHz6mC0Mn+NjAl5j0xRIS6omh2770fS7d ueUw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=C+fVtVmO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l6-20020a170903244600b001ac94b7f2f0si1113910pls.523.2023.05.18.04.47.59; Thu, 18 May 2023 04:48:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=C+fVtVmO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231509AbjERLhe (ORCPT + 99 others); Thu, 18 May 2023 07:37:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60208 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231437AbjERLgq (ORCPT ); Thu, 18 May 2023 07:36:46 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC68019B4 for ; Thu, 18 May 2023 04:35:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684409755; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=enBN5jT1muKlfAJKNsWEtZoec3VPKG1S27Z+IZtkOMs=; b=C+fVtVmOFxo1nHGvgoXlMjwJvKN9Lyb8vDqBijHVjGjGMCdyC6pdG4AjW3hoy0hj0Twayn ySk9De/CjxZEPhZ1wC2xTCBYC1q2DOoPxlt6Y8RHpI4Qw9BeBTwlQOUMa2MCOhOyDeJdOD /8enmPdUjgFMxuzm4t48fzRp+cB5N3I= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-669-zHr-wG-8MFKIw9HgrsEr2A-1; Thu, 18 May 2023 07:35:47 -0400 X-MC-Unique: zHr-wG-8MFKIw9HgrsEr2A-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EAFF61014702; Thu, 18 May 2023 11:35:46 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id DADA640C2063; Thu, 18 May 2023 11:35:43 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v8 10/16] tcp: Fold do_tcp_sendpages() into tcp_sendpage_locked() Date: Thu, 18 May 2023 12:34:47 +0100 Message-Id: <20230518113453.1350757-11-dhowells@redhat.com> In-Reply-To: <20230518113453.1350757-1-dhowells@redhat.com> References: <20230518113453.1350757-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766232418655188338?= X-GMAIL-MSGID: =?utf-8?q?1766232418655188338?= Fold do_tcp_sendpages() into its last remaining caller, tcp_sendpage_locked(). Signed-off-by: David Howells cc: Eric Dumazet cc: David Ahern cc: "David S. Miller" cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- include/net/tcp.h | 2 -- net/ipv4/tcp.c | 21 +++++++-------------- 2 files changed, 7 insertions(+), 16 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index 04a31643cda3..02a6cff1827e 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -333,8 +333,6 @@ int tcp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags); int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags); -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags); int tcp_send_mss(struct sock *sk, int *size_goal, int flags); void tcp_push(struct sock *sk, int flags, int mss_now, int nonagle, int size_goal); diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 28231e503af9..5e4752131583 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -974,12 +974,17 @@ static int tcp_wmem_schedule(struct sock *sk, int copy) return min(copy, sk->sk_forward_alloc); } -ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, - size_t size, int flags) +int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, + size_t size, int flags) { struct bio_vec bvec; struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + if (!(sk->sk_route_caps & NETIF_F_SG)) + return sock_no_sendpage_locked(sk, page, offset, size, flags); + + tcp_rate_check_app_limited(sk); /* is sending application-limited? */ + bvec_set_page(&bvec, page, size, offset); iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); @@ -988,18 +993,6 @@ ssize_t do_tcp_sendpages(struct sock *sk, struct page *page, int offset, return tcp_sendmsg_locked(sk, &msg, size); } -EXPORT_SYMBOL_GPL(do_tcp_sendpages); - -int tcp_sendpage_locked(struct sock *sk, struct page *page, int offset, - size_t size, int flags) -{ - if (!(sk->sk_route_caps & NETIF_F_SG)) - return sock_no_sendpage_locked(sk, page, offset, size, flags); - - tcp_rate_check_app_limited(sk); /* is sending application-limited? */ - - return do_tcp_sendpages(sk, page, offset, size, flags); -} EXPORT_SYMBOL_GPL(tcp_sendpage_locked); int tcp_sendpage(struct sock *sk, struct page *page, int offset, From patchwork Thu May 18 11:34:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95809 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp421723vqo; Thu, 18 May 2023 04:38:17 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7dvLBocoFFtNBzl6Y0L2OQk19w9TfkMtS03s+UPYykD1opnMuTGTicQf5FBRU50uY5TZO+ X-Received: by 2002:a17:90a:eb16:b0:253:342b:a14e with SMTP id j22-20020a17090aeb1600b00253342ba14emr2233394pjz.21.1684409897116; Thu, 18 May 2023 04:38:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684409897; cv=none; d=google.com; s=arc-20160816; b=0BROcvzLwlFW0B1R/eFVqh6Ramvjgy1E1LihgqiMHQGVSIkmVzV+zFDsdGt0ibrofr QuU7HAlwu1sy+81FtHeQsE6kUR0cY6w59vCBrHzHsMmJfZ2NbceiUiOURtlS8Bzdcbog VmoIKF3tA8XXD+aZukZUKYOwSMaDHIlksEONITUXS0KKlpl4X/jmjVG43gSiUu+6UoLQ Bqa8DrlUqcnZd517OY/rs843cb7I9lAhk9CCnYue0SNUz/PD3k68Z0+ChPteqYm3nVdG TWUNH5xv5oe4xmCPT7QwoCPF9CvvQRtJ4lWU6Yp/oz5UnvDEQfV8ahiBM5aRUnznTgEG XIFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Di14zlocOEDECmKOzgXOEyS4sjRLATonTy1WJdIXAUU=; b=BUaee+H5OHkr5SXFP2CzmLFvAjXdtbRJY6uaAnqc8nkQRDYW80MUZNOg8/8p/Cv3OV fCzxgSivZuHC3U2enVJx6CsDj4uLnhW7vMqyIyHnBvoQZPXKUKElvRMI52aM8D/QD5Y6 pznIo2XqlGLN1Yr7YWCK1KX9bOTZ300hCwB9lFValFwGlDXjBeg08o7FeeKK1SNe/Ud5 qml/VV+3IIWhJ1iDP+Vf++A3y2iTHdZrWuT6AEdRkJ1x3cqhNvwW7QNcLJ543g1w28dc C1wiyzMtZpPg2rG8faZji1lZpifRkgGL30vV8LWCoWFfT9dimNZadovjrLSQC53iWxy/ /5Tw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Jq9glSlO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bg9-20020a17090b0d8900b0024e2d70d28asi1464226pjb.140.2023.05.18.04.38.01; Thu, 18 May 2023 04:38:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Jq9glSlO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231496AbjERLhc (ORCPT + 99 others); Thu, 18 May 2023 07:37:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60250 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231439AbjERLgr (ORCPT ); Thu, 18 May 2023 07:36:47 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1996D199F for ; Thu, 18 May 2023 04:35:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684409757; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Di14zlocOEDECmKOzgXOEyS4sjRLATonTy1WJdIXAUU=; b=Jq9glSlO8Ix+w7um7FZaNiMJSODyIuLvfSHFzXYifu3vUTqdkCHYifqG/sdBXk2QjGwqEY O8XBYUorLh/9VGnF9zZ6dHiQdZsKN+8XLKEEpr8r3Z/5A5UB9PEQiqwK4CeR3Eh9s3ZQW8 1yvqiXtv6lLLz9lRE6rNyNBr+M8yAHo= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-175-D_mli70IPguE5JHBJeMmUg-1; Thu, 18 May 2023 07:35:51 -0400 X-MC-Unique: D_mli70IPguE5JHBJeMmUg-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 260A93814597; Thu, 18 May 2023 11:35:50 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id AB20B492B02; Thu, 18 May 2023 11:35:47 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v8 11/16] ip, udp: Support MSG_SPLICE_PAGES Date: Thu, 18 May 2023 12:34:48 +0100 Message-Id: <20230518113453.1350757-12-dhowells@redhat.com> In-Reply-To: <20230518113453.1350757-1-dhowells@redhat.com> References: <20230518113453.1350757-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766231792193904476?= X-GMAIL-MSGID: =?utf-8?q?1766231792193904476?= Make IP/UDP sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/ipv4/ip_output.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 52fc840898d8..c7db973b5d29 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1048,6 +1048,14 @@ static int __ip_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + if (inet->hdrincl) + return -EPERM; + if (rt->dst.dev->features & NETIF_F_SG) + /* We need an empty buffer to attach stuff to */ + paged = true; + else + flags &= ~MSG_SPLICE_PAGES; } cork->length += length; @@ -1207,6 +1215,15 @@ static int __ip_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + struct msghdr *msg = from; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) + goto error; + copy = err; + wmem_alloc_delta += copy; } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Thu May 18 11:34:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95814 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp423112vqo; Thu, 18 May 2023 04:41:09 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7KuziRIK8sUlIZeGkjzNCm56GN40EkVAJz1t5YXxWB/jSsSOK1O4heO+5JAzXZZLLTN7uH X-Received: by 2002:a17:90a:1181:b0:24e:4a1a:3994 with SMTP id e1-20020a17090a118100b0024e4a1a3994mr2274880pja.3.1684410069460; Thu, 18 May 2023 04:41:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684410069; cv=none; d=google.com; s=arc-20160816; b=sD8uvFMX6hJ5ynIbdlS2JvJcJUI9yHsOSDa0M4nw7bULejT+PqWKnZAthcXQKjKMcC q/MzydNN346tEL/n1WkwDmcyw2dk38QfttPNUj1mo6d5dHJsC7Zx6zVHGUxzkHBFCGmE P3lAvj7vcv2lG7FODwltNXB/0riatuW/4fpmhfA4Wn9ugbJboB2wGhR3BCZhHpKIl+fe 1z9mLk+bgYNHQaen1WvdjDS56odRi+7KljDPqRsZ1tWrzczchmExSj7QBnvT+uy0t4o2 EpDr8RfwT6l8v5gzS7jF/DkNwKvPH2qUed4zSLmvf4uZQGzqQvZNSjP1p+x4kdJ7eJhd nc3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5dEuCyEYbAeTLbptDWG+lpG3TePaGyDY6iKA6gIr8Gk=; b=vXj7K8IXfesW1/87tEp21lxSiAVK/OHo202ZSov7izfL7IlLdc3s+z586bCFEl4sFd tVIeeQsGEkk042l88z5phxsSlvkQbGqZ+iaK6eIiYdB0QDJSXy7upWRiZlBxiB5rMUeC z4LVeSNYvUwnwnf5CtvUipHcmh50bhtF/PFnd3XVDMK0Tnm+Tw+rpRXf1USiuyRD8+Ow /miIac4lMgQGyZauVGvpZRR4uTOxv4WVX4ZTZ4LuC8LNj593lycfwYDFmPE0MJKiuEHL OhZh4Hf8amQAMBPvlTMkeOKX/a1CZS1StiVtx6s9j0i9/NKRwxMUBANfAOYxMt6pXX5E h8ww== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=WggRO0We; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id mi16-20020a17090b4b5000b002475b063d35si4494320pjb.142.2023.05.18.04.40.53; Thu, 18 May 2023 04:41:09 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=WggRO0We; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231514AbjERLhh (ORCPT + 99 others); Thu, 18 May 2023 07:37:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59344 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231440AbjERLgs (ORCPT ); Thu, 18 May 2023 07:36:48 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2EABF19A3 for ; Thu, 18 May 2023 04:36:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684409759; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5dEuCyEYbAeTLbptDWG+lpG3TePaGyDY6iKA6gIr8Gk=; b=WggRO0WeP/jLRWS4qJcthCtGl5EnVDsPr7ZeDxf2eUtHCkibfunl7zHUzk3Vh5oUNbOf5I zKvpF+RjKEVz5Ytqy51sSRM8MVr5e1qgR2wW6u/+4b8VE/nMbY0Oy3EtXSq3u1OK6Rk+41 Ss0zy4q9aa/LZc7PHUS2d/5MEMPYjdU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-668-jwZ3dLuJNq-_aRCVShqxqQ-1; Thu, 18 May 2023 07:35:54 -0400 X-MC-Unique: jwZ3dLuJNq-_aRCVShqxqQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7959E86C60C; Thu, 18 May 2023 11:35:53 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3D35614171C0; Thu, 18 May 2023 11:35:51 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v8 12/16] ip6, udp6: Support MSG_SPLICE_PAGES Date: Thu, 18 May 2023 12:34:49 +0100 Message-Id: <20230518113453.1350757-13-dhowells@redhat.com> In-Reply-To: <20230518113453.1350757-1-dhowells@redhat.com> References: <20230518113453.1350757-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766231972736918683?= X-GMAIL-MSGID: =?utf-8?q?1766231972736918683?= Make IP6/UDP6 sendmsg() support MSG_SPLICE_PAGES. This causes pages to be spliced from the source iterator if possible, copying the data if not. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/ipv6/ip6_output.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 9554cf46ed88..c722cb881b2d 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1589,6 +1589,14 @@ static int __ip6_append_data(struct sock *sk, skb_zcopy_set(skb, uarg, &extra_uref); } } + } else if ((flags & MSG_SPLICE_PAGES) && length) { + if (inet_sk(sk)->hdrincl) + return -EPERM; + if (rt->dst.dev->features & NETIF_F_SG) + /* We need an empty buffer to attach stuff to */ + paged = true; + else + flags &= ~MSG_SPLICE_PAGES; } /* @@ -1778,6 +1786,15 @@ static int __ip6_append_data(struct sock *sk, err = -EFAULT; goto error; } + } else if (flags & MSG_SPLICE_PAGES) { + struct msghdr *msg = from; + + err = skb_splice_from_iter(skb, &msg->msg_iter, copy, + sk->sk_allocation); + if (err < 0) + goto error; + copy = err; + wmem_alloc_delta += copy; } else if (!zc) { int i = skb_shinfo(skb)->nr_frags; From patchwork Thu May 18 11:34:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95812 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp422999vqo; Thu, 18 May 2023 04:40:56 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5ff5uJlc879JzGFPbhcD4EMJWwTsusJe2JZT5jDW+hp25nDymzAtAyibLrgFoSteLZ8y3w X-Received: by 2002:a17:903:41c3:b0:1ae:622c:e745 with SMTP id u3-20020a17090341c300b001ae622ce745mr2975844ple.1.1684410056532; Thu, 18 May 2023 04:40:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684410056; cv=none; d=google.com; s=arc-20160816; b=IC60ftvvYigPQtMir3AlbAXTqibry5r9e+xnrWSDnncGBnctF2U3yB24h9pGf7nnsf 6RqxsBMsFiRl1MxF4jxnvXBathYpS8RSoAfgs6uHuqL7cyXlRFhkitSxW1et1qf3c/RX g6sfBOhXiaGvettlwidrT2SEla+yGNLKkVO8jLl9wjds+yz5O4ilTJpY4fCjIuJ+6Bwb aPjEGq/GIzq0pk03cDW4cEk6MIs7JeRslWUOVI9T+8DlGimUzsetyWyXuUlyjY2BNE76 TTTGme03wqdP2oTrgbY4gQx6fGL8pBonpIFo5oPhPfYNVA62nx4ELbGa44IxYYqs+PVj 1LhA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=arPLIYOMzcdUPWLjuF22tHo0vP4jVzFFThikgCzP+CE=; b=oK/ttDOYbmxli0mVHS9rgVkyUONL27Et2XU3TK/2+fQGbAFZ8t1PuGHfH35/fsb4kq bbVlx1OkFfmrPOr0cLGOWoBovt/3q3o07VfIgdc5zF9iRLLu9R0m6bfLp2OG1dYx2ZVR hLoF+Ibgsj6LQ/qI8RvYnWmoQpVqiQKKDYFdPhI7Crznu/VVdR5TRUXc7001ZdCYyEKA tBfaRe09hL8it4514dd2FE6zwp2DE7EBYoqEWnJNVgZoPOenzPoX0oJ5CDcY2FRNSzRT KjuHIEKTHZun0IqGvsGPLwN+eB/vl7l3y/4GPdj+xkp5srMGiJQiGD7V28WFMwaRXbtM 16FA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=iMZCKANK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p5-20020a170902b08500b001a20c983d21si1146603plr.51.2023.05.18.04.40.41; Thu, 18 May 2023 04:40:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=iMZCKANK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231530AbjERLho (ORCPT + 99 others); Thu, 18 May 2023 07:37:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60350 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231444AbjERLgy (ORCPT ); Thu, 18 May 2023 07:36:54 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 554EFE57 for ; Thu, 18 May 2023 04:36:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684409762; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=arPLIYOMzcdUPWLjuF22tHo0vP4jVzFFThikgCzP+CE=; b=iMZCKANKH5bQn4i7wapSqaXjZsq/K9buAe3PDGIoLJPEiVBDjjnt7amfYEZao3NGTkm0R2 SmH3fa9zC9/yGF3Hs/07ViI/l2nh2obgNQ83R7LsyW9vjVQhD3NgHGcw4tKwxY8gYORmKw vTu4gTiwcnhp64MNCVs5trUSW1lql+A= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-253-zr9gaF3vOCeYO5wuwbnLTQ-1; Thu, 18 May 2023 07:35:58 -0400 X-MC-Unique: zr9gaF3vOCeYO5wuwbnLTQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 19F6086C60D; Thu, 18 May 2023 11:35:57 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6844440C206F; Thu, 18 May 2023 11:35:54 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v8 13/16] udp: Convert udp_sendpage() to use MSG_SPLICE_PAGES Date: Thu, 18 May 2023 12:34:50 +0100 Message-Id: <20230518113453.1350757-14-dhowells@redhat.com> In-Reply-To: <20230518113453.1350757-1-dhowells@redhat.com> References: <20230518113453.1350757-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766231959135395543?= X-GMAIL-MSGID: =?utf-8?q?1766231959135395543?= Convert udp_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - udp_sendpage() shouldn't lock the socket around udp_sendpage(). - udp_sendpage() should only set MSG_MORE if MSG_SENDPAGE_NOTLAST is set. net/ipv4/udp.c | 51 ++++++-------------------------------------------- 1 file changed, 6 insertions(+), 45 deletions(-) diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index aa32afd871ee..2879dc6d66ea 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -1332,54 +1332,15 @@ EXPORT_SYMBOL(udp_sendmsg); int udp_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct inet_sock *inet = inet_sk(sk); - struct udp_sock *up = udp_sk(sk); - int ret; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES }; if (flags & MSG_SENDPAGE_NOTLAST) - flags |= MSG_MORE; - - if (!up->pending) { - struct msghdr msg = { .msg_flags = flags|MSG_MORE }; - - /* Call udp_sendmsg to specify destination address which - * sendpage interface can't pass. - * This will succeed only when the socket is connected. - */ - ret = udp_sendmsg(sk, &msg, 0); - if (ret < 0) - return ret; - } - - lock_sock(sk); + msg.msg_flags |= MSG_MORE; - if (unlikely(!up->pending)) { - release_sock(sk); - - net_dbg_ratelimited("cork failed\n"); - return -EINVAL; - } - - ret = ip_append_page(sk, &inet->cork.fl.u.ip4, - page, offset, size, flags); - if (ret == -EOPNOTSUPP) { - release_sock(sk); - return sock_no_sendpage(sk->sk_socket, page, offset, - size, flags); - } - if (ret < 0) { - udp_flush_pending_frames(sk); - goto out; - } - - up->len += size; - if (!(READ_ONCE(up->corkflag) || (flags&MSG_MORE))) - ret = udp_push_pending_frames(sk); - if (!ret) - ret = size; -out: - release_sock(sk); - return ret; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return udp_sendmsg(sk, &msg, size); } #define UDP_SKB_IS_STATELESS 0x80000000 From patchwork Thu May 18 11:34:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95834 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp427440vqo; Thu, 18 May 2023 04:49:31 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5jsbKQrkiEr5NBpbJ1tqQJfqzGZgwrTBHPyIDo+UeK3/FQFv8g8nKmploe5H0slR1hkJXJ X-Received: by 2002:a05:6a20:7d94:b0:100:a6ba:ba1b with SMTP id v20-20020a056a207d9400b00100a6baba1bmr1958899pzj.51.1684410570983; Thu, 18 May 2023 04:49:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684410570; cv=none; d=google.com; s=arc-20160816; b=mUXb/w2kIXVc3pjTnaQ1/fOBivC1Bj6+mZVHPJQARYxNFfttslLJLyvU681ykuzqqp pB7tghM/Y7+m2WUnfHSjErOHWBB8uKF4SU92Nkwca6EfljSroAsnBs0CfRneiQ0/xzs0 D1aIOMEZm2Dr+q4xMXBjLiTPR0kGXVuofjk2d1YMh6sziMO26Lc/dbWU/IF2npWtOkZD ixe2i6+9TVRL3SmQCOePzG7eYW55oLI6qrzL8Y0h+EdwZna4t7vt+YGiWpUY2hHbPxZp HwMvPkQM1YbARCY/6bJuPRiN2IGB2GvBxM0C33oEa2EuQRsQ/DujlneSS/+XjzrF+rsy kaog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vVs0CsGrBKFATDm79TCVboT9gSnZlC7XDeXTGZfAJzM=; b=Or+Zu9iIO7LDTQmFTjwwRttoMX0gL22Nrl+pyCFrw1wMziHMbe4fefiukeeKQZjtFz 9hO92NbjgTaaWREGG6iPEIrZK6t/U6CJ7anzmmxwn/xVmt6UD2WMYVPz29P5aOCZGYQU 038HULaipIzw3jUqp0MOEK+KppobJr0mq+XkDqz7LJ4A8xZfQojzKW5gpWWz/DNIK7KI H5vOBU6G1pIfQBdncYGclTxr3wCyo+IkM/CC/Ip44UXvK4ayBRo39jsjAcqAoRCC1YkI Mf16j/p37/uNJyE0Mb6jxhvfWiRGoVzBq6JeG0n49iTfwfcghJidYxH80Wes7TwjOzQX 4gzw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=IMrHobwu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k62-20020a638441000000b0051423af249fsi354486pgd.304.2023.05.18.04.49.16; Thu, 18 May 2023 04:49:30 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=IMrHobwu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231520AbjERLhl (ORCPT + 99 others); Thu, 18 May 2023 07:37:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59452 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231442AbjERLgw (ORCPT ); Thu, 18 May 2023 07:36:52 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F04B19B3 for ; Thu, 18 May 2023 04:36:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684409765; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vVs0CsGrBKFATDm79TCVboT9gSnZlC7XDeXTGZfAJzM=; b=IMrHobwuwL0OdVOO8/wULcxQnoc3GDU6fTkdLJ0WR4D/jJ3cYeuaoNhr63cA5EMrX8fOqp QKeLfzYAP5qn0CoR9iP0sej4kzZOutzIKw2ZOqQHBfgvww65YmnKvVwIaB7Ezai5VS8TbF 7+ih65QqcJEkA5weUc0mJHoNsZ7Ie0A= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-266-miPRkfAKPhyxWJ8vCer5_A-1; Thu, 18 May 2023 07:36:01 -0400 X-MC-Unique: miPRkfAKPhyxWJ8vCer5_A-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C56AD10146F8; Thu, 18 May 2023 11:36:00 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4A1FB2166B31; Thu, 18 May 2023 11:35:58 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH net-next v8 14/16] ip: Remove ip_append_page() Date: Thu, 18 May 2023 12:34:51 +0100 Message-Id: <20230518113453.1350757-15-dhowells@redhat.com> In-Reply-To: <20230518113453.1350757-1-dhowells@redhat.com> References: <20230518113453.1350757-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766232499024200646?= X-GMAIL-MSGID: =?utf-8?q?1766232499024200646?= ip_append_page() is no longer used with the removal of udp_sendpage(), so remove it. Signed-off-by: David Howells cc: Willem de Bruijn cc: David Ahern cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #7) - Remove now-unused csum_page(). include/net/ip.h | 2 - net/ipv4/ip_output.c | 148 ++----------------------------------------- 2 files changed, 4 insertions(+), 146 deletions(-) diff --git a/include/net/ip.h b/include/net/ip.h index c3fffaa92d6e..7627a4df893b 100644 --- a/include/net/ip.h +++ b/include/net/ip.h @@ -220,8 +220,6 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4, unsigned int flags); int ip_generic_getfrag(void *from, char *to, int offset, int len, int odd, struct sk_buff *skb); -ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, - int offset, size_t size, int flags); struct sk_buff *__ip_make_skb(struct sock *sk, struct flowi4 *fl4, struct sk_buff_head *queue, struct inet_cork *cork); diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index c7db973b5d29..553c740a6bfb 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -946,17 +946,6 @@ ip_generic_getfrag(void *from, char *to, int offset, int len, int odd, struct sk } EXPORT_SYMBOL(ip_generic_getfrag); -static inline __wsum -csum_page(struct page *page, int offset, int copy) -{ - char *kaddr; - __wsum csum; - kaddr = kmap(page); - csum = csum_partial(kaddr + offset, copy, 0); - kunmap(page); - return csum; -} - static int __ip_append_data(struct sock *sk, struct flowi4 *fl4, struct sk_buff_head *queue, @@ -1327,10 +1316,10 @@ static int ip_setup_cork(struct sock *sk, struct inet_cork *cork, } /* - * ip_append_data() and ip_append_page() can make one large IP datagram - * from many pieces of data. Each pieces will be holded on the socket - * until ip_push_pending_frames() is called. Each piece can be a page - * or non-page data. + * ip_append_data() can make one large IP datagram from many pieces of + * data. Each piece will be held on the socket until + * ip_push_pending_frames() is called. Each piece can be a page or + * non-page data. * * Not only UDP, other transport protocols - e.g. raw sockets - can use * this interface potentially. @@ -1363,135 +1352,6 @@ int ip_append_data(struct sock *sk, struct flowi4 *fl4, from, length, transhdrlen, flags); } -ssize_t ip_append_page(struct sock *sk, struct flowi4 *fl4, struct page *page, - int offset, size_t size, int flags) -{ - struct inet_sock *inet = inet_sk(sk); - struct sk_buff *skb; - struct rtable *rt; - struct ip_options *opt = NULL; - struct inet_cork *cork; - int hh_len; - int mtu; - int len; - int err; - unsigned int maxfraglen, fragheaderlen, fraggap, maxnonfragsize; - - if (inet->hdrincl) - return -EPERM; - - if (flags&MSG_PROBE) - return 0; - - if (skb_queue_empty(&sk->sk_write_queue)) - return -EINVAL; - - cork = &inet->cork.base; - rt = (struct rtable *)cork->dst; - if (cork->flags & IPCORK_OPT) - opt = cork->opt; - - if (!(rt->dst.dev->features & NETIF_F_SG)) - return -EOPNOTSUPP; - - hh_len = LL_RESERVED_SPACE(rt->dst.dev); - mtu = cork->gso_size ? IP_MAX_MTU : cork->fragsize; - - fragheaderlen = sizeof(struct iphdr) + (opt ? opt->optlen : 0); - maxfraglen = ((mtu - fragheaderlen) & ~7) + fragheaderlen; - maxnonfragsize = ip_sk_ignore_df(sk) ? 0xFFFF : mtu; - - if (cork->length + size > maxnonfragsize - fragheaderlen) { - ip_local_error(sk, EMSGSIZE, fl4->daddr, inet->inet_dport, - mtu - (opt ? opt->optlen : 0)); - return -EMSGSIZE; - } - - skb = skb_peek_tail(&sk->sk_write_queue); - if (!skb) - return -EINVAL; - - cork->length += size; - - while (size > 0) { - /* Check if the remaining data fits into current packet. */ - len = mtu - skb->len; - if (len < size) - len = maxfraglen - skb->len; - - if (len <= 0) { - struct sk_buff *skb_prev; - int alloclen; - - skb_prev = skb; - fraggap = skb_prev->len - maxfraglen; - - alloclen = fragheaderlen + hh_len + fraggap + 15; - skb = sock_wmalloc(sk, alloclen, 1, sk->sk_allocation); - if (unlikely(!skb)) { - err = -ENOBUFS; - goto error; - } - - /* - * Fill in the control structures - */ - skb->ip_summed = CHECKSUM_NONE; - skb->csum = 0; - skb_reserve(skb, hh_len); - - /* - * Find where to start putting bytes. - */ - skb_put(skb, fragheaderlen + fraggap); - skb_reset_network_header(skb); - skb->transport_header = (skb->network_header + - fragheaderlen); - if (fraggap) { - skb->csum = skb_copy_and_csum_bits(skb_prev, - maxfraglen, - skb_transport_header(skb), - fraggap); - skb_prev->csum = csum_sub(skb_prev->csum, - skb->csum); - pskb_trim_unique(skb_prev, maxfraglen); - } - - /* - * Put the packet on the pending queue. - */ - __skb_queue_tail(&sk->sk_write_queue, skb); - continue; - } - - if (len > size) - len = size; - - if (skb_append_pagefrags(skb, page, offset, len, - MAX_SKB_FRAGS)) { - err = -EMSGSIZE; - goto error; - } - - if (skb->ip_summed == CHECKSUM_NONE) { - __wsum csum; - csum = csum_page(page, offset, len); - skb->csum = csum_block_add(skb->csum, csum, skb->len); - } - - skb_len_add(skb, len); - refcount_add(len, &sk->sk_wmem_alloc); - offset += len; - size -= len; - } - return 0; - -error: - cork->length -= size; - IP_INC_STATS(sock_net(sk), IPSTATS_MIB_OUTDISCARDS); - return err; -} - static void ip_cork_release(struct inet_cork *cork) { cork->flags &= ~IPCORK_OPT; From patchwork Thu May 18 11:34:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95827 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp426885vqo; Thu, 18 May 2023 04:48:33 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4vDCMEd8h7ufi/yvxuFUBe9w1VizOxsjrhEfI5mmfqBcI/lFtxBCy+WYWLIRyLEQiza3MA X-Received: by 2002:a05:6a00:ad5:b0:64d:22db:1023 with SMTP id c21-20020a056a000ad500b0064d22db1023mr1497040pfl.22.1684410513059; Thu, 18 May 2023 04:48:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684410513; cv=none; d=google.com; s=arc-20160816; b=wKFkUXkjPlRKxy26vqN/ifwDd0ER74ZDK2aktIxdVfKsNr2WqAwyccHgE9aTo1nb3h 2Gslic33xS+2piiR2mcCBHmHz/S2ZyLjKFUzWUjjWWwNEfHlRoDSjzNoevsi3X97HDHb vxugGDQ+K+O2UsOVJh8VDVYSc43PcVH9M3BLZOHud/w7psvBFaD0wfaIwJAWeGDbBtyu geSK2jYuR/YvqsNBPM/5dMBxlyl3StCmQItU4rNh245YMh8fGYJiaHWUCTW3I6BHIboH IG4tuzjhD6YCD/Ka3jcpSji2v/S6ZuvtiJV3NvG5PXcOHDYF8I4yVODvSTlJC3pURBSi zMwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4sbDm+4sgaynAjmiQFVbDl+OyfXvG2vWGJtRH+IAST4=; b=s3cX1BnvKRm8BPfuvO0dMINyaxwbJrgHP06dQlw1/AvKcRCkM/yzuCdyv7+mXG3pl5 u50yOotdlpgHgyDlMNA3fCUGozqiGjxLz/GoQXalrXn95AwvBJgBfBBMzbiqnVEuP67p Fj0OvynqRkC1v6FZd+kOhsMC2daE34cPXPWNCUl2gHKaztJaHx8C/qG92K1skI3msywY Mkha1+2qngWrxrHgO5VvyTM6CIXisbHwXkgMVspRuPIoKdwuiN5+dTCIZ3GC1wK9jvKa 0nZvVk14qoUnmdQC+auD90eB/0wspV5ytEZSVsStXJVbG2BCbaZ+EVlQX7WrJCpinF2A SJfg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=hKQv8dVJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i123-20020a625481000000b006438c116ebcsi1440092pfb.384.2023.05.18.04.48.17; Thu, 18 May 2023 04:48:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=hKQv8dVJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231350AbjERLiA (ORCPT + 99 others); Thu, 18 May 2023 07:38:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60442 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231405AbjERLhB (ORCPT ); Thu, 18 May 2023 07:37:01 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B3531FC2 for ; Thu, 18 May 2023 04:36:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684409768; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4sbDm+4sgaynAjmiQFVbDl+OyfXvG2vWGJtRH+IAST4=; b=hKQv8dVJzsTl4XqPsNN2GZzMeEb+4w4Mi75eSy41RMeIeYX7ly/RRIQP5L41TVKqpfp2Tt ziLCWvJWsnYqGXibcKHuubpsKdI0YGvHEyjYnEkZoQC2SRFCWkI902oVnkR0GCI9WDRTvw vDch7egDiqaCDfmxcHBbZ0dJZJa/7qQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-497-sU7nZ7tANKyeN_LLALrjqw-1; Thu, 18 May 2023 07:36:05 -0400 X-MC-Unique: sU7nZ7tANKyeN_LLALrjqw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 121C787DC27; Thu, 18 May 2023 11:36:04 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8346BC0297C; Thu, 18 May 2023 11:36:01 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Kuniyuki Iwashima Subject: [PATCH net-next v8 15/16] af_unix: Support MSG_SPLICE_PAGES Date: Thu, 18 May 2023 12:34:52 +0100 Message-Id: <20230518113453.1350757-16-dhowells@redhat.com> In-Reply-To: <20230518113453.1350757-1-dhowells@redhat.com> References: <20230518113453.1350757-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766232438049887934?= X-GMAIL-MSGID: =?utf-8?q?1766232438049887934?= Make AF_UNIX sendmsg() support MSG_SPLICE_PAGES, splicing in pages from the source iterator if possible and copying the data in otherwise. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Kuniyuki Iwashima cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- Notes: ver #6) - Use common helper. net/unix/af_unix.c | 49 +++++++++++++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 16 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index dd55506b4632..976bc1c5e11b 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2200,19 +2200,25 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, while (sent < len) { size = len - sent; - /* Keep two messages in the pipe so it schedules better */ - size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + skb = sock_alloc_send_pskb(sk, 0, 0, + msg->msg_flags & MSG_DONTWAIT, + &err, 0); + } else { + /* Keep two messages in the pipe so it schedules better */ + size = min_t(int, size, (sk->sk_sndbuf >> 1) - 64); - /* allow fallback to order-0 allocations */ - size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); + /* allow fallback to order-0 allocations */ + size = min_t(int, size, SKB_MAX_HEAD(0) + UNIX_SKB_FRAGS_SZ); - data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); + data_len = max_t(int, 0, size - SKB_MAX_HEAD(0)); - data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); + data_len = min_t(size_t, size, PAGE_ALIGN(data_len)); - skb = sock_alloc_send_pskb(sk, size - data_len, data_len, - msg->msg_flags & MSG_DONTWAIT, &err, - get_order(UNIX_SKB_FRAGS_SZ)); + skb = sock_alloc_send_pskb(sk, size - data_len, data_len, + msg->msg_flags & MSG_DONTWAIT, &err, + get_order(UNIX_SKB_FRAGS_SZ)); + } if (!skb) goto out_err; @@ -2224,13 +2230,24 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, } fds_sent = true; - skb_put(skb, size - data_len); - skb->data_len = data_len; - skb->len = size; - err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); - if (err) { - kfree_skb(skb); - goto out_err; + if (unlikely(msg->msg_flags & MSG_SPLICE_PAGES)) { + err = skb_splice_from_iter(skb, &msg->msg_iter, size, + sk->sk_allocation); + if (err < 0) { + kfree_skb(skb); + goto out_err; + } + size = err; + refcount_add(size, &sk->sk_wmem_alloc); + } else { + skb_put(skb, size - data_len); + skb->data_len = data_len; + skb->len = size; + err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size); + if (err) { + kfree_skb(skb); + goto out_err; + } } unix_state_lock(other); From patchwork Thu May 18 11:34:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 95811 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp422976vqo; Thu, 18 May 2023 04:40:54 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ70SWEp3nAwZ7C28yCeYifah0GtqY/WPCx1XUpK6yr/4YcHYCVEQMV+sBP0aHdkN1H3aRut X-Received: by 2002:a17:903:2786:b0:1ae:4ba0:af86 with SMTP id jw6-20020a170903278600b001ae4ba0af86mr1720414plb.64.1684410054097; Thu, 18 May 2023 04:40:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684410054; cv=none; d=google.com; s=arc-20160816; b=04RWgwQys3tcFQsj1f1lqVOSxpkn4S0aJ0k65oYD7ZPcpDJparCuJn9Lobz8Ey8/BT x4M7oprJEBTJhJpRknQZQfExDzke2c7j0ZOGCJgMGXkGL+2MbiUhJe8JPwmiT2Po30Dc 7mACl2kkWLg4ecR6Y4Z5GB+B723h5wUQbOhRn1vqq4l+XItmVN3uuKhAZiwUh9gMFACD j+Y+XTFG00PnMaA4UWIogg2hPWuP0OwHrOO2JKD6PJ3gzg/5TH9DxeZT+84xNsyFzLFN 1hRoI7a/mBPu46H9p7UOuvTAlGx+dG22aMz4omtLEklxVcZZ6cSa6XRc2frxvKXMvqfT kQnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=wF+2b4fBEfy00AN3VDBr6eYpnbX7vBgnsz0rKmJdTGo=; b=qv50XsOkHfXMcnFmhPX/h3K8aQo7JfY/EVSSdd5D7ItRAOUqdIA5DmaF8vVAs0UAug cMX0Uk8wOCZY6fwa2YCU9Il284HyQzGIq7jHcLjFWtS3t0Hkrk+PKjaWL5XpshnDxbyK IzNqWtgFywqAN8n6dH1lHrj10IUsZi/jXQY6qKWArAFXLhb0S3tSSJuyBXBBE/dPDDb4 j7ULQetZVTxVyStCcvBrNtC61T1kz3Dq1z1L7UZYLbplHzV859RTXKPefEE5oSnUWrs6 WnlSsQhP86yYxbK7Dw9WA2qMUgWAFLERi3oTbHCz/5Ns9XZgXHBJH7/eYlwZYhq8WfAY uWOg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OQFbrdB6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id jc13-20020a17090325cd00b001a1d76e7214si1117642plb.111.2023.05.18.04.40.37; Thu, 18 May 2023 04:40:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OQFbrdB6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231551AbjERLiF (ORCPT + 99 others); Thu, 18 May 2023 07:38:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60664 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231407AbjERLhE (ORCPT ); Thu, 18 May 2023 07:37:04 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 610421FFC for ; Thu, 18 May 2023 04:36:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684409772; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wF+2b4fBEfy00AN3VDBr6eYpnbX7vBgnsz0rKmJdTGo=; b=OQFbrdB62k69JS6ELvbpo6WWRt2paN3EKEhLMBpB9cxZw7CHXqyIr1JQvJo7kYquFHfGCi EegYMUt1vELoGaKHoW2HG3GD/io37brwgjM3MiqkMn9ZeXpVNQF6Y4hQlrAQX4CpNnFLBa qcVso5R8B4nFy9BZeILXfURkTk/odjY= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-321-vqjt2jLQMMKuS89fOjFPaQ-1; Thu, 18 May 2023 07:36:09 -0400 X-MC-Unique: vqjt2jLQMMKuS89fOjFPaQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F189729DD98B; Thu, 18 May 2023 11:36:07 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.221]) by smtp.corp.redhat.com (Postfix) with ESMTP id 371ACC15BA0; Thu, 18 May 2023 11:36:05 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Kuniyuki Iwashima Subject: [PATCH net-next v8 16/16] unix: Convert udp_sendpage() to use MSG_SPLICE_PAGES Date: Thu, 18 May 2023 12:34:53 +0100 Message-Id: <20230518113453.1350757-17-dhowells@redhat.com> In-Reply-To: <20230518113453.1350757-1-dhowells@redhat.com> References: <20230518113453.1350757-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766231956655498962?= X-GMAIL-MSGID: =?utf-8?q?1766231956655498962?= Convert unix_stream_sendpage() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: Kuniyuki Iwashima cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org --- net/unix/af_unix.c | 134 +++------------------------------------------ 1 file changed, 7 insertions(+), 127 deletions(-) diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index 976bc1c5e11b..115436ce1f8a 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -1839,24 +1839,6 @@ static void maybe_add_creds(struct sk_buff *skb, const struct socket *sock, } } -static int maybe_init_creds(struct scm_cookie *scm, - struct socket *socket, - const struct sock *other) -{ - int err; - struct msghdr msg = { .msg_controllen = 0 }; - - err = scm_send(socket, &msg, scm, false); - if (err) - return err; - - if (unix_passcred_enabled(socket, other)) { - scm->pid = get_pid(task_tgid(current)); - current_uid_gid(&scm->creds.uid, &scm->creds.gid); - } - return err; -} - static bool unix_skb_scm_eq(struct sk_buff *skb, struct scm_cookie *scm) { @@ -2292,117 +2274,15 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, static ssize_t unix_stream_sendpage(struct socket *socket, struct page *page, int offset, size_t size, int flags) { - int err; - bool send_sigpipe = false; - bool init_scm = true; - struct scm_cookie scm; - struct sock *other, *sk = socket->sk; - struct sk_buff *skb, *newskb = NULL, *tail = NULL; - - if (flags & MSG_OOB) - return -EOPNOTSUPP; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES }; - other = unix_peer(sk); - if (!other || sk->sk_state != TCP_ESTABLISHED) - return -ENOTCONN; - - if (false) { -alloc_skb: - unix_state_unlock(other); - mutex_unlock(&unix_sk(other)->iolock); - newskb = sock_alloc_send_pskb(sk, 0, 0, flags & MSG_DONTWAIT, - &err, 0); - if (!newskb) - goto err; - } - - /* we must acquire iolock as we modify already present - * skbs in the sk_receive_queue and mess with skb->len - */ - err = mutex_lock_interruptible(&unix_sk(other)->iolock); - if (err) { - err = flags & MSG_DONTWAIT ? -EAGAIN : -ERESTARTSYS; - goto err; - } - - if (sk->sk_shutdown & SEND_SHUTDOWN) { - err = -EPIPE; - send_sigpipe = true; - goto err_unlock; - } - - unix_state_lock(other); + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - if (sock_flag(other, SOCK_DEAD) || - other->sk_shutdown & RCV_SHUTDOWN) { - err = -EPIPE; - send_sigpipe = true; - goto err_state_unlock; - } - - if (init_scm) { - err = maybe_init_creds(&scm, socket, other); - if (err) - goto err_state_unlock; - init_scm = false; - } - - skb = skb_peek_tail(&other->sk_receive_queue); - if (tail && tail == skb) { - skb = newskb; - } else if (!skb || !unix_skb_scm_eq(skb, &scm)) { - if (newskb) { - skb = newskb; - } else { - tail = skb; - goto alloc_skb; - } - } else if (newskb) { - /* this is fast path, we don't necessarily need to - * call to kfree_skb even though with newskb == NULL - * this - does no harm - */ - consume_skb(newskb); - newskb = NULL; - } - - if (skb_append_pagefrags(skb, page, offset, size, MAX_SKB_FRAGS)) { - tail = skb; - goto alloc_skb; - } - - skb->len += size; - skb->data_len += size; - skb->truesize += size; - refcount_add(size, &sk->sk_wmem_alloc); - - if (newskb) { - err = unix_scm_to_skb(&scm, skb, false); - if (err) - goto err_state_unlock; - spin_lock(&other->sk_receive_queue.lock); - __skb_queue_tail(&other->sk_receive_queue, newskb); - spin_unlock(&other->sk_receive_queue.lock); - } - - unix_state_unlock(other); - mutex_unlock(&unix_sk(other)->iolock); - - other->sk_data_ready(other); - scm_destroy(&scm); - return size; - -err_state_unlock: - unix_state_unlock(other); -err_unlock: - mutex_unlock(&unix_sk(other)->iolock); -err: - kfree_skb(newskb); - if (send_sigpipe && !(flags & MSG_NOSIGNAL)) - send_sig(SIGPIPE, current, 0); - if (!init_scm) - scm_destroy(&scm); - return err; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return unix_stream_sendmsg(socket, &msg, size); } static int unix_seqpacket_sendmsg(struct socket *sock, struct msghdr *msg,