From patchwork Sat Jul 29 21:49:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 128181 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp1276249vqg; Sat, 29 Jul 2023 18:02:48 -0700 (PDT) X-Google-Smtp-Source: APBJJlEmlaS9wvkDG66KpfH8cNWvoKDSMxQubLBi6CMFWmux68LplQZb6gym7mNgDn81RW8elmNg X-Received: by 2002:a17:907:a087:b0:99c:c5b:8c13 with SMTP id hu7-20020a170907a08700b0099c0c5b8c13mr378261ejc.52.1690678968169; Sat, 29 Jul 2023 18:02:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690678968; cv=none; d=google.com; s=arc-20160816; b=DdSCpox/TcaWrtec5v8/SAUEnp2eau3oTFDkKcv1jWQoc8/zG4yshOU6KrFlKFv7Ix 11G2SZx6tL3N029IzzdghSzQnjtwBT5zBN4p8/fEV3AF32U19ACzBOs8u41JeYh/WwUj ZjMNwEhHmaCwBsnygjFmGMgjFMvNwEy28BJv6jZwJ4drIFGchr+s5IkN8bK96oX4bw95 SjqwsaYjLhmmrgTPUUFArgbYBg8EE8BFFZ6lBLnnZ9a1KyAbsHPGy2dIqWABwxdgTSPT dBu18e+ZzCA7l3kgdXQBpkz6sROPH0LRt39iwNOYWOJCYbmDAQwKWS1+tY/Q+v8ckFuz jV5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:message-id:date:content-transfer-encoding :content-id:mime-version:subject:cc:to:references:in-reply-to:from :organization:dkim-signature; bh=9xoC7L2bOOoVu4fgnru97jFTPx3UfbeG/oW1+7jnOPk=; fh=odR6ePhgG3efvLnOC6esgVziFfxnbwc3+i/FBVzCEco=; b=r+poGgpJTXJOB7Keo/z48WOTKUlCgRbL4ugtZH0lUwgNIDCJimEZlwlA6yEIlhkfkE +3GXT1QPriX8BrBAKmB/Mns2ONVYx9JEJ5eSFfDyTlW17GXaenSOTMlPgn/TPCbZVPa7 HarmF9Znq5D2flPUyGVLOxZUJiINWAfhHRNGGj8CftkTp9ip1Uu3fJAiA65ANN95b301 +arKbnzBWQrqU5WRs3NMeEpiNzZaDJeg0sRnVZfDIs+iiVqrk/QSjhzzE0KVsOadA6HA 3FQzzxi/IfSp2yQIObXGKyIX0yrVcIZUwCRalzoROufqUiFZRdVVOt7IfCnnQbDpGt0C 2mxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ELu93K+r; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id lu10-20020a170906faca00b0099bd646e72asi4927567ejb.309.2023.07.29.18.02.16; Sat, 29 Jul 2023 18:02:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ELu93K+r; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229653AbjG2Vu3 (ORCPT + 99 others); Sat, 29 Jul 2023 17:50:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38972 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229619AbjG2VuW (ORCPT ); Sat, 29 Jul 2023 17:50:22 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B4E432736 for ; Sat, 29 Jul 2023 14:49:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1690667375; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9xoC7L2bOOoVu4fgnru97jFTPx3UfbeG/oW1+7jnOPk=; b=ELu93K+rYkj8lEp8bk08vwvwe4K+e4NGzYU5+4grJRu+TWo3IJct4jcteeeKB8dTKDTj5D MTbBTdpnt3alfTqzKj0bemcnxTl14uRjTGKxN0Pf84hlVxPaBEY57Au4WMJIgod7NxYm9L 0vqUczGW7BZd1hDENlTwKUPuIcX/WD4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-662-hWbhkitSMLSaI_dUnS9x_w-1; Sat, 29 Jul 2023 17:49:30 -0400 X-MC-Unique: hWbhkitSMLSaI_dUnS9x_w-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 98B08858F1E; Sat, 29 Jul 2023 21:49:29 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.131]) by smtp.corp.redhat.com (Postfix) with ESMTP id D50992166B25; Sat, 29 Jul 2023 21:49:27 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 From: David Howells In-Reply-To: <20230718160737.52c68c73@kernel.org> References: <20230718160737.52c68c73@kernel.org> <000000000000881d0606004541d1@google.com> <0000000000001416bb06004ebf53@google.com> To: Jakub Kicinski , Willem de Bruijn Cc: dhowells@redhat.com, syzbot , bpf@vger.kernel.org, brauner@kernel.org, davem@davemloft.net, dsahern@kernel.org, edumazet@google.com, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, pabeni@redhat.com, syzkaller-bugs@googlegroups.com, viro@zeniv.linux.org.uk Subject: Endless loop in udp with MSG_SPLICE_READ - Re: [syzbot] [fs?] INFO: task hung in pipe_release (4) MIME-Version: 1.0 Content-ID: <792237.1690667367.1@warthog.procyon.org.uk> Date: Sat, 29 Jul 2023 22:49:27 +0100 Message-ID: <792238.1690667367@warthog.procyon.org.uk> X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772805389432821767 X-GMAIL-MSGID: 1772805389432821767 Hi Jakub, Willem, I think I'm going to need your help with this one. > > syzbot has bisected this issue to: > > > > commit 7ac7c987850c3ec617c778f7bd871804dc1c648d > > Author: David Howells > > Date: Mon May 22 12:11:22 2023 +0000 > > > > udp: Convert udp_sendpage() to use MSG_SPLICE_PAGES > > > > bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=15853bcaa80000 > > start commit: 3f01e9fed845 Merge tag 'linux-watchdog-6.5-rc2' of git://w.. > > git tree: upstream > > final oops: https://syzkaller.appspot.com/x/report.txt?x=17853bcaa80000 > > console output: https://syzkaller.appspot.com/x/log.txt?x=13853bcaa80000 > > kernel config: https://syzkaller.appspot.com/x/.config?x=150188feee7071a7 > > dashboard link: https://syzkaller.appspot.com/bug?extid=f527b971b4bdc8e79f9e > > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=12a86682a80000 > > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=1520ab6ca80000 > > > > Reported-by: syzbot+f527b971b4bdc8e79f9e@syzkaller.appspotmail.com > > Fixes: 7ac7c987850c ("udp: Convert udp_sendpage() to use MSG_SPLICE_PAGES") > > > > For information about bisection process see: https://goo.gl/tpsmEJ#bisection The issue that syzbot is triggering seems to be something to do with the calculations in the "if (copy <= 0) { ... }" chunk in __ip_append_data() when MSG_SPLICE_PAGES is in operation. What seems to happen is that the test program uses sendmsg() + MSG_MORE to loads a UDP packet with 1406 bytes of data to the MTU size (1434) and then splices in 8 extra bytes. r3 = socket$inet_udp(0x2, 0x2, 0x0) setsockopt$sock_int(r3, 0x1, 0x6, &(0x7f0000000140)=0x32, 0x4) bind$inet(r3, &(0x7f0000000000)={0x2, 0x0, @dev={0xac, 0x14, 0x14, 0x15}}, 0x10) connect$inet(r3, &(0x7f0000000200)={0x2, 0x0, @broadcast}, 0x10) sendmmsg(r3, &(0x7f0000000180)=[{{0x0, 0x0, 0x0}}, {{0x0, 0xfffffffffffffed3, &(0x7f0000000940)=[{&(0x7f00000006c0)='O', 0x57e}], 0x1}}], 0x4000000000003bd, 0x8800) write$binfmt_misc(r1, &(0x7f0000000440)=ANY=[], 0x8) splice(r0, 0x0, r2, 0x0, 0x4ffe0, 0x0) This results in some negative intermediate values turning up in the calculations - and this results in the remaining length being made longer from 8 to 14. I added some printks (patch attached), resulting in the attached tracelines: ==>splice_to_socket() 7099 udp_sendmsg(8,8) __ip_append_data(copy=-6,len=8, mtu=1434 skblen=1434 maxfl=1428) pagedlen 14 = 14 - 0 copy -6 = 14 - 0 - 6 - 14 length 8 -= -6 + 0 __ip_append_data(copy=1414,len=14, mtu=1434 skblen=20 maxfl=1428) copy=1414 len=14 skb_splice_from_iter(8,14) __ip_append_data(copy=1406,len=6, mtu=1434 skblen=28 maxfl=1428) copy=1406 len=6 skb_splice_from_iter(0,6) __ip_append_data(copy=1406,len=6, mtu=1434 skblen=28 maxfl=1428) copy=1406 len=6 skb_splice_from_iter(0,6) __ip_append_data(copy=1406,len=6, mtu=1434 skblen=28 maxfl=1428) copy=1406 len=6 skb_splice_from_iter(0,6) __ip_append_data(copy=1406,len=6, mtu=1434 skblen=28 maxfl=1428) copy=1406 len=6 skb_splice_from_iter(0,6) copy=1406 len=6 skb_splice_from_iter(0,6) ... 'copy' gets calculated as -6 because the maxfraglen (maxfl=1428) is 8 bytes less than the amount of data then in the packet (skblen=1434). 'copy' gets recalculated part way down as -6 from datalen (14) - transhdrlen (0) - fraggap (6) - pagedlen (14). datalen is 14 because it was length (8) + fraggap (6). Inside skb_splice_from_iter(), we eventually end up in an enless loop in which msg_iter.count is 0 and the length to be copied is 6. It always returns 0 because there's nothing to copy, and so __ip_append_data() cycles round the loop endlessly. Any suggestion as to how to fix this? Thanks, David --- Debug hang in pipe_release's pipe_lock --- fs/splice.c | 3 +++ net/core/skbuff.c | 7 +++++++ net/ipv4/ip_output.c | 24 ++++++++++++++++++++++++ net/ipv4/udp.c | 3 +++ 4 files changed, 37 insertions(+) diff --git a/fs/splice.c b/fs/splice.c index 004eb1c4ce31..9ee82b818bd6 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -801,6 +801,8 @@ ssize_t splice_to_socket(struct pipe_inode_info *pipe, struct file *out, size_t spliced = 0; bool need_wakeup = false; + printk("==>splice_to_socket() %u\n", current->pid); + pipe_lock(pipe); while (len > 0) { @@ -911,6 +913,7 @@ ssize_t splice_to_socket(struct pipe_inode_info *pipe, struct file *out, pipe_unlock(pipe); if (need_wakeup) wakeup_pipe_writers(pipe); + printk("<==splice_to_socket() = %zd\n", spliced ?: ret); return spliced ?: ret; } #endif diff --git a/net/core/skbuff.c b/net/core/skbuff.c index a298992060e6..c3d60da9e3f7 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -6801,6 +6801,13 @@ ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, ssize_t spliced = 0, ret = 0; unsigned int i; + static int __pcount; + + if (__pcount < 6) { + printk("skb_splice_from_iter(%zu,%zd)\n", iter->count, maxsize); + __pcount++; + } + while (iter->count > 0) { ssize_t space, nr, len; size_t off; diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 6e70839257f7..8c84a7d13627 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1066,6 +1066,14 @@ static int __ip_append_data(struct sock *sk, copy = mtu - skb->len; if (copy < length) copy = maxfraglen - skb->len; + if (flags & MSG_SPLICE_PAGES) { + static int __pcount; + if (__pcount < 6) { + printk("__ip_append_data(copy=%d,len=%d, mtu=%d skblen=%d maxfl=%d)\n", + copy, length, mtu, skb->len, maxfraglen); + __pcount++; + } + } if (copy <= 0) { char *data; unsigned int datalen; @@ -1112,6 +1120,10 @@ static int __ip_append_data(struct sock *sk, else { alloclen = fragheaderlen + transhdrlen; pagedlen = datalen - transhdrlen; + if (flags & MSG_SPLICE_PAGES) { + printk("pagedlen %d = %d - %d\n", + pagedlen, datalen, transhdrlen); + } } alloclen += alloc_extra; @@ -1158,6 +1170,9 @@ static int __ip_append_data(struct sock *sk, } copy = datalen - transhdrlen - fraggap - pagedlen; + if (flags & MSG_SPLICE_PAGES) + printk("copy %d = %d - %d - %d - %d\n", + copy, datalen, transhdrlen, fraggap, pagedlen); if (copy > 0 && getfrag(from, data + transhdrlen, offset, copy, fraggap, skb) < 0) { err = -EFAULT; kfree_skb(skb); @@ -1165,6 +1180,8 @@ static int __ip_append_data(struct sock *sk, } offset += copy; + if (flags & MSG_SPLICE_PAGES) + printk("length %d -= %d + %d\n", length, copy, transhdrlen); length -= copy + transhdrlen; transhdrlen = 0; exthdrlen = 0; @@ -1192,6 +1209,13 @@ static int __ip_append_data(struct sock *sk, continue; } + if (flags & MSG_SPLICE_PAGES) { + static int __qcount; + if (__qcount < 6) { + printk("copy=%d len=%d\n", copy, length); + __qcount++; + } + } if (copy > length) copy = length; diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index 42a96b3547c9..bd3f4e62574b 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -1081,6 +1081,9 @@ int udp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) if (msg->msg_flags & MSG_OOB) /* Mirror BSD error message compatibility */ return -EOPNOTSUPP; + if (msg->msg_flags & MSG_SPLICE_PAGES) + printk("udp_sendmsg(%zx,%zx)\n", msg->msg_iter.count, len); + getfrag = is_udplite ? udplite_getfrag : ip_generic_getfrag; fl4 = &inet->cork.fl.u.ip4;