Message ID | 20230121085521.9566-1-kerneljasonxing@gmail.com |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp641481wrn; Sat, 21 Jan 2023 01:14:57 -0800 (PST) X-Google-Smtp-Source: AMrXdXtwBiVPHN+NClaNE7HUGxcfrXWaCaibEYQZjPk3rfPnB+Yukhe9waMz/UMcLdfOX6vZFtr4 X-Received: by 2002:a17:902:cf11:b0:185:441e:2d7a with SMTP id i17-20020a170902cf1100b00185441e2d7amr17333925plg.17.1674292497556; Sat, 21 Jan 2023 01:14:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674292497; cv=none; d=google.com; s=arc-20160816; b=qxj2KY9VpO5rhh5ac75B6bs00pk1BHX0CKu6Z/OQZU1yUPr/q9nWw4NHJ6ugwnLMZP b/z5L9k6KSj6cUx9W+ytW6AtaQj4ma2XrqrynRpkF0Yu21xqpi7p2MkK1iCb90YKsEbY 8y3m3aS0GBpuS0LPNmT66e+jAJW2jjfN970Px6n8kYUYr4q04vvnpD+9nzUJe7nw5rZ7 5LrOKCntu6WdjVIh6YwmSSVVAC2fbzMy0pq/O5pee9864Qe+66j9epeTwZEbP+FFwOCz cz0W/0TuPmpNTvWy0kYXDbBVZ4mNHNcCs6PcnV6csKijL9aMPULop1NF1ZscdSX0cVN9 hI0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=PrnGpbTnMcSAdRfQebkBMLK0G9v/WsMM46OOzL4dzDA=; b=O+3iNC1lKsrqzuELGBsJdTD0o0ZYS3yWINviWG10Xo0gjm3cWHtMPOggSpUEeaLZh+ uMsEjWTyiylgmzWkB5wrQME299Se5o3/OCMb2Im88uSmqXIUuhQjLntykJkkQxJpbZRw Mu2kH+9aIh2j1wYiwwSKcQrufZvzkTTPfvSu3u5r8kzg1jiyyrPbQguF351vVQfGLiJ+ 7zFLhQj4rM3GRVyjuWHzcm025UxAF0FcE2wCqyl4cmvRvAEeeg89mIF7+pFHVk2V4Mzz YyF8AIyg0ZSC+2I4ysqYRP4hLzfuIcCZSlZ8uONbHFSiRkbv6+Anhf+RnjE2NJzgDbkV 1aag== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=SDgmsuHE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b4-20020a170902d50400b001928641c19fsi47769079plg.286.2023.01.21.01.14.41; Sat, 21 Jan 2023 01:14:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=SDgmsuHE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229789AbjAUIzc (ORCPT <rfc822;forouhar.linux@gmail.com> + 99 others); Sat, 21 Jan 2023 03:55:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39720 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229735AbjAUIza (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Sat, 21 Jan 2023 03:55:30 -0500 Received: from mail-pg1-x532.google.com (mail-pg1-x532.google.com [IPv6:2607:f8b0:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8AD4965F00; Sat, 21 Jan 2023 00:55:29 -0800 (PST) Received: by mail-pg1-x532.google.com with SMTP id g68so5776958pgc.11; Sat, 21 Jan 2023 00:55:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=PrnGpbTnMcSAdRfQebkBMLK0G9v/WsMM46OOzL4dzDA=; b=SDgmsuHEoebuFkYmBvrFkLKy8cp1wMRSw6dwPIcLHUVNjujn6pQvfS04Kd66UAr6jg gQHORYgc8k4DFh4N1th30dsYb4lEeIXxHlYylXeF+eWgTRB/YGd/LxJpoWSQQ48ftFmL WTlEIypkkTJgri1WauSjeLaTDPWocpJuCA/8x2lZ/VrUV9ireiy2Jba9WmKZxGWT3tYV X//7ZskP9FB8Vg8q68Axf0WsNxQzFpc/5ZLcVbVF4PeO0pACd5KaM1kPK364Sfg9HT3n cehY8/32fDbNBVNwZhkW9AKZLGNMEHTLnnXnsocO13r5AWLSKm4wU4B3EYK4YHbnTLpy o3uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=PrnGpbTnMcSAdRfQebkBMLK0G9v/WsMM46OOzL4dzDA=; b=FGCAq+A/9G2nsSQTcXfzastImQFkALWLunpvV0y3leeabIpughZdcf8JmrJbBGRobP 2sk2dH7o8wDVTPMlJ6lQdxpPJE0dDs6vjlfYuIbQPEQIyINnd3K4SC+HdU7onkUyQ+xE Ycwxe5rmWrZhnvDi4o7ThY8r7ZRClXfladTwZdez7n1SB0vzaGl3Hb+i8AreuS26ddOp jr/H+0nbzEx4NkcHBSN/70nmpnD+Lenan+aVcvntsBiREIoFC8Mk8awJLqJsCeCQrFOk YzJPZiaetNvsfNmGuzo0HmWjDfRkWMgu9rjvx/pTumwmMDbMhCJopjSA9JxVuJOxjoJn GvzQ== X-Gm-Message-State: AFqh2kqPA8tfogde2FLOv3B06GdRNYNz3dLYIigT/pHX2AneI6clV5ic juiE3JBR0t/8JE9eg6c/nok= X-Received: by 2002:a62:148d:0:b0:58b:ca43:9c05 with SMTP id 135-20020a62148d000000b0058bca439c05mr16399291pfu.16.1674291328948; Sat, 21 Jan 2023 00:55:28 -0800 (PST) Received: from KERNELXING-MB0.tencent.com ([114.253.32.172]) by smtp.gmail.com with ESMTPSA id z4-20020aa79484000000b005823b7da05asm11927073pfk.122.2023.01.21.00.55.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Jan 2023 00:55:28 -0800 (PST) From: Jason Xing <kerneljasonxing@gmail.com> To: jesse.brandeburg@intel.com, anthony.l.nguyen@intel.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, richardcochran@gmail.com, ast@kernel.org, daniel@iogearbox.net, hawk@kernel.org, john.fastabend@gmail.co Cc: intel-wired-lan@lists.osuosl.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kerneljasonxing@gmail.com, Jason Xing <kernelxing@tencent.com> Subject: [PATCH net] ixgbe: allow to increase MTU to some extent with XDP enalbed Date: Sat, 21 Jan 2023 16:55:21 +0800 Message-Id: <20230121085521.9566-1-kerneljasonxing@gmail.com> X-Mailer: git-send-email 2.33.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755622929945738302?= X-GMAIL-MSGID: =?utf-8?q?1755622929945738302?= |
Series |
[net] ixgbe: allow to increase MTU to some extent with XDP enalbed
|
|
Commit Message
Jason Xing
Jan. 21, 2023, 8:55 a.m. UTC
From: Jason Xing <kernelxing@tencent.com> I encountered one case where I cannot increase the MTU size with XDP enabled if the server is equipped with IXGBE card, which happened on thousands of servers. I noticed it was prohibited from 2017[1] and added size checks[2] if allowed soon after the previous patch. Interesting part goes like this: 1) Changing MTU directly from 1500 (default value) to 2000 doesn't work because the driver finds out that 'new_frame_size > ixgbe_rx_bufsz(ring)' in ixgbe_change_mtu() function. 2) However, if we change MTU to 1501 then change from 1501 to 2000, it does work, because the driver sets __IXGBE_RX_3K_BUFFER when MTU size is converted to 1501, which later size check policy allows. The default MTU value for most servers is 1500 which cannot be adjusted directly to the value larger than IXGBE_MAX_2K_FRAME_BUILD_SKB (1534 or 1536) if it loads XDP. After I do a quick study on the manner of i40E driver allowing two kinds of buffer size (one is 2048 while another is 3072) to support XDP mode in i40e_max_xdp_frame_size(), I believe the default MTU size is possibly not satisfied in XDP mode when IXGBE driver is in use, we sometimes need to insert a new header, say, vxlan header. So setting the 3K-buffer flag could solve the issue. [1] commit 38b7e7f8ae82 ("ixgbe: Do not allow LRO or MTU change with XDP") [2] commit fabf1bce103a ("ixgbe: Prevent unsupported configurations with XDP") Signed-off-by: Jason Xing <kernelxing@tencent.com> --- drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 3 +++ 1 file changed, 3 insertions(+)
Comments
Dear Jason, Thank you for your patch. Am 21.01.23 um 09:55 schrieb Jason Xing: > From: Jason Xing <kernelxing@tencent.com> There is a small typo in the summary: ena*bl*ed > I encountered one case where I cannot increase the MTU size with XDP > enabled if the server is equipped with IXGBE card, which happened on > thousands of servers. I noticed it was prohibited from 2017[1] and That’s included since Linux 4.19-rc1. > added size checks[2] if allowed soon after the previous patch. > > Interesting part goes like this: > 1) Changing MTU directly from 1500 (default value) to 2000 doesn't > work because the driver finds out that 'new_frame_size > > ixgbe_rx_bufsz(ring)' in ixgbe_change_mtu() function. > 2) However, if we change MTU to 1501 then change from 1501 to 2000, it > does work, because the driver sets __IXGBE_RX_3K_BUFFER when MTU size > is converted to 1501, which later size check policy allows. > > The default MTU value for most servers is 1500 which cannot be adjusted > directly to the value larger than IXGBE_MAX_2K_FRAME_BUILD_SKB (1534 or > 1536) if it loads XDP. > > After I do a quick study on the manner of i40E driver allowing two kinds > of buffer size (one is 2048 while another is 3072) to support XDP mode in > i40e_max_xdp_frame_size(), I believe the default MTU size is possibly not > satisfied in XDP mode when IXGBE driver is in use, we sometimes need to > insert a new header, say, vxlan header. So setting the 3K-buffer flag > could solve the issue. What card did you test with exactly? > [1] commit 38b7e7f8ae82 ("ixgbe: Do not allow LRO or MTU change with XDP") > [2] commit fabf1bce103a ("ixgbe: Prevent unsupported configurations with > XDP") I’d say to not break the line in references. > Signed-off-by: Jason Xing <kernelxing@tencent.com> > --- > drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > index ab8370c413f3..dc016582f91e 100644 > --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > @@ -4313,6 +4313,9 @@ static void ixgbe_set_rx_buffer_len(struct ixgbe_adapter *adapter) > if (IXGBE_2K_TOO_SMALL_WITH_PADDING || > (max_frame > (ETH_FRAME_LEN + ETH_FCS_LEN))) > set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state); > + > + if (ixgbe_enabled_xdp_adapter(adapter)) > + set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state); > #endif > } > } Kind regards, Paul
On Mon, Jan 23, 2023 at 4:21 AM Paul Menzel <pmenzel@molgen.mpg.de> wrote: > > Dear Jason, > > > Thank you for your patch. Hello Paul, Thanks for the review. > > Am 21.01.23 um 09:55 schrieb Jason Xing: > > From: Jason Xing <kernelxing@tencent.com> > > There is a small typo in the summary: ena*bl*ed > Sure it is. That's my fault. > > I encountered one case where I cannot increase the MTU size with XDP > > enabled if the server is equipped with IXGBE card, which happened on > > thousands of servers. I noticed it was prohibited from 2017[1] and > > That’s included since Linux 4.19-rc1. > > > added size checks[2] if allowed soon after the previous patch. > > > > Interesting part goes like this: > > 1) Changing MTU directly from 1500 (default value) to 2000 doesn't > > work because the driver finds out that 'new_frame_size > > > ixgbe_rx_bufsz(ring)' in ixgbe_change_mtu() function. > > 2) However, if we change MTU to 1501 then change from 1501 to 2000, it > > does work, because the driver sets __IXGBE_RX_3K_BUFFER when MTU size > > is converted to 1501, which later size check policy allows. > > > > The default MTU value for most servers is 1500 which cannot be adjusted > > directly to the value larger than IXGBE_MAX_2K_FRAME_BUILD_SKB (1534 or > > 1536) if it loads XDP. > > > > After I do a quick study on the manner of i40E driver allowing two kinds > > of buffer size (one is 2048 while another is 3072) to support XDP mode in > > i40e_max_xdp_frame_size(), I believe the default MTU size is possibly not > > satisfied in XDP mode when IXGBE driver is in use, we sometimes need to > > insert a new header, say, vxlan header. So setting the 3K-buffer flag > > could solve the issue. > > What card did you test with exactly? > It is the IXGBE driver that has such an issue. The I40E driver I mentioned here is only for contrast. It's not that proper from my point of view if the IXGBE driver cannot directly adjust to 2000. Thus I would like more reviews and suggestions. Thanks, Jason > > [1] commit 38b7e7f8ae82 ("ixgbe: Do not allow LRO or MTU change with XDP") > > [2] commit fabf1bce103a ("ixgbe: Prevent unsupported configurations with > > XDP") > > I’d say to not break the line in references. > > > Signed-off-by: Jason Xing <kernelxing@tencent.com> > > --- > > drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 3 +++ > > 1 file changed, 3 insertions(+) > > > > diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > > index ab8370c413f3..dc016582f91e 100644 > > --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > > +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > > @@ -4313,6 +4313,9 @@ static void ixgbe_set_rx_buffer_len(struct ixgbe_adapter *adapter) > > if (IXGBE_2K_TOO_SMALL_WITH_PADDING || > > (max_frame > (ETH_FRAME_LEN + ETH_FCS_LEN))) > > set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state); > > + > > + if (ixgbe_enabled_xdp_adapter(adapter)) > > + set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state); > > #endif > > } > > } > > > Kind regards, > > Paul
I'm sorry. I just noticed that I sent it to the wrong email address of john.fastabend previously. So I corrected it here. Thanks, Jason On Sat, Jan 21, 2023 at 4:55 PM Jason Xing <kerneljasonxing@gmail.com> wrote: > > From: Jason Xing <kernelxing@tencent.com> > > I encountered one case where I cannot increase the MTU size with XDP > enabled if the server is equipped with IXGBE card, which happened on > thousands of servers. I noticed it was prohibited from 2017[1] and > added size checks[2] if allowed soon after the previous patch. > > Interesting part goes like this: > 1) Changing MTU directly from 1500 (default value) to 2000 doesn't > work because the driver finds out that 'new_frame_size > > ixgbe_rx_bufsz(ring)' in ixgbe_change_mtu() function. > 2) However, if we change MTU to 1501 then change from 1501 to 2000, it > does work, because the driver sets __IXGBE_RX_3K_BUFFER when MTU size > is converted to 1501, which later size check policy allows. > > The default MTU value for most servers is 1500 which cannot be adjusted > directly to the value larger than IXGBE_MAX_2K_FRAME_BUILD_SKB (1534 or > 1536) if it loads XDP. > > After I do a quick study on the manner of i40E driver allowing two kinds > of buffer size (one is 2048 while another is 3072) to support XDP mode in > i40e_max_xdp_frame_size(), I believe the default MTU size is possibly not > satisfied in XDP mode when IXGBE driver is in use, we sometimes need to > insert a new header, say, vxlan header. So setting the 3K-buffer flag > could solve the issue. > > [1] commit 38b7e7f8ae82 ("ixgbe: Do not allow LRO or MTU change with XDP") > [2] commit fabf1bce103a ("ixgbe: Prevent unsupported configurations with > XDP") > > Signed-off-by: Jason Xing <kernelxing@tencent.com> > --- > drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > index ab8370c413f3..dc016582f91e 100644 > --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > @@ -4313,6 +4313,9 @@ static void ixgbe_set_rx_buffer_len(struct ixgbe_adapter *adapter) > if (IXGBE_2K_TOO_SMALL_WITH_PADDING || > (max_frame > (ETH_FRAME_LEN + ETH_FCS_LEN))) > set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state); > + > + if (ixgbe_enabled_xdp_adapter(adapter)) > + set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state); > #endif > } > } > -- > 2.37.3 >
On Sat, Jan 21, 2023 at 04:55:21PM +0800, Jason Xing wrote: > From: Jason Xing <kernelxing@tencent.com> > > I encountered one case where I cannot increase the MTU size with XDP > enabled if the server is equipped with IXGBE card, which happened on > thousands of servers. I noticed it was prohibited from 2017[1] and > added size checks[2] if allowed soon after the previous patch. > > Interesting part goes like this: > 1) Changing MTU directly from 1500 (default value) to 2000 doesn't > work because the driver finds out that 'new_frame_size > > ixgbe_rx_bufsz(ring)' in ixgbe_change_mtu() function. > 2) However, if we change MTU to 1501 then change from 1501 to 2000, it > does work, because the driver sets __IXGBE_RX_3K_BUFFER when MTU size > is converted to 1501, which later size check policy allows. > > The default MTU value for most servers is 1500 which cannot be adjusted > directly to the value larger than IXGBE_MAX_2K_FRAME_BUILD_SKB (1534 or > 1536) if it loads XDP. > > After I do a quick study on the manner of i40E driver allowing two kinds > of buffer size (one is 2048 while another is 3072) to support XDP mode in > i40e_max_xdp_frame_size(), I believe the default MTU size is possibly not > satisfied in XDP mode when IXGBE driver is in use, we sometimes need to > insert a new header, say, vxlan header. So setting the 3K-buffer flag > could solve the issue. > > [1] commit 38b7e7f8ae82 ("ixgbe: Do not allow LRO or MTU change with XDP") > [2] commit fabf1bce103a ("ixgbe: Prevent unsupported configurations with > XDP") > > Signed-off-by: Jason Xing <kernelxing@tencent.com> > --- > drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > index ab8370c413f3..dc016582f91e 100644 > --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > @@ -4313,6 +4313,9 @@ static void ixgbe_set_rx_buffer_len(struct ixgbe_adapter *adapter) > if (IXGBE_2K_TOO_SMALL_WITH_PADDING || > (max_frame > (ETH_FRAME_LEN + ETH_FCS_LEN))) > set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state); > + > + if (ixgbe_enabled_xdp_adapter(adapter)) > + set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state); This will result with unnecessary overhead for 1500 MTU because you will be working on order-1 pages. Instead I would focus on fixing ixgbe_change_mtu() and stop relying on ixgbe_rx_bufsz() in there. You can check what we do on ice/i40e sides. I'm not looking actively into ixgbe internals but I don't think that there is anything that stops us from using 3k buffers with XDP. > #endif > } > } > -- > 2.37.3 >
From: Maciej Fijalkowski <maciej.fijalkowski@intel.com> Date: Thu, 26 Jan 2023 13:17:20 +0100 > On Sat, Jan 21, 2023 at 04:55:21PM +0800, Jason Xing wrote: >> From: Jason Xing <kernelxing@tencent.com> >> >> I encountered one case where I cannot increase the MTU size with XDP >> enabled if the server is equipped with IXGBE card, which happened on >> thousands of servers. I noticed it was prohibited from 2017[1] and >> added size checks[2] if allowed soon after the previous patch. >> >> Interesting part goes like this: >> 1) Changing MTU directly from 1500 (default value) to 2000 doesn't >> work because the driver finds out that 'new_frame_size > >> ixgbe_rx_bufsz(ring)' in ixgbe_change_mtu() function. >> 2) However, if we change MTU to 1501 then change from 1501 to 2000, it >> does work, because the driver sets __IXGBE_RX_3K_BUFFER when MTU size >> is converted to 1501, which later size check policy allows. >> >> The default MTU value for most servers is 1500 which cannot be adjusted >> directly to the value larger than IXGBE_MAX_2K_FRAME_BUILD_SKB (1534 or >> 1536) if it loads XDP. >> >> After I do a quick study on the manner of i40E driver allowing two kinds >> of buffer size (one is 2048 while another is 3072) to support XDP mode in >> i40e_max_xdp_frame_size(), I believe the default MTU size is possibly not >> satisfied in XDP mode when IXGBE driver is in use, we sometimes need to >> insert a new header, say, vxlan header. So setting the 3K-buffer flag >> could solve the issue. >> >> [1] commit 38b7e7f8ae82 ("ixgbe: Do not allow LRO or MTU change with XDP") >> [2] commit fabf1bce103a ("ixgbe: Prevent unsupported configurations with >> XDP") >> >> Signed-off-by: Jason Xing <kernelxing@tencent.com> >> --- >> drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 3 +++ >> 1 file changed, 3 insertions(+) >> >> diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c >> index ab8370c413f3..dc016582f91e 100644 >> --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c >> +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c >> @@ -4313,6 +4313,9 @@ static void ixgbe_set_rx_buffer_len(struct ixgbe_adapter *adapter) >> if (IXGBE_2K_TOO_SMALL_WITH_PADDING || >> (max_frame > (ETH_FRAME_LEN + ETH_FCS_LEN))) >> set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state); >> + >> + if (ixgbe_enabled_xdp_adapter(adapter)) >> + set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state); > > This will result with unnecessary overhead for 1500 MTU because you will > be working on order-1 pages. Instead I would focus on fixing > ixgbe_change_mtu() and stop relying on ixgbe_rx_bufsz() in there. You can > check what we do on ice/i40e sides. > > I'm not looking actively into ixgbe internals but I don't think that there > is anything that stops us from using 3k buffers with XDP. I think it uses the same logics as the rest of drivers: splits a 4k page into two 2k buffers when MTU is <= 1536, otherwise uses order-1 pages and uses 3k buffers. OTOH ixgbe is not fully correct in terms how it calculates Rx headroom, but the main problem is how it calculates the maximum MTU available when XDP is on. Our usual MTU supported when XDP is on is 3046 bytes. For MTU <= 1536, 2k buffers are used even for XDP, so the fix is not correct. Maciej is right that i40e and ice do that way better and don't have such issue. > >> #endif >> } >> } >> -- >> 2.37.3 >> Thanks, Olek
On Thu, Jan 26, 2023 at 11:56 PM Alexander Lobakin <alexandr.lobakin@intel.com> wrote: > > From: Maciej Fijalkowski <maciej.fijalkowski@intel.com> > Date: Thu, 26 Jan 2023 13:17:20 +0100 > > > On Sat, Jan 21, 2023 at 04:55:21PM +0800, Jason Xing wrote: > >> From: Jason Xing <kernelxing@tencent.com> > >> > >> I encountered one case where I cannot increase the MTU size with XDP > >> enabled if the server is equipped with IXGBE card, which happened on > >> thousands of servers. I noticed it was prohibited from 2017[1] and > >> added size checks[2] if allowed soon after the previous patch. > >> > >> Interesting part goes like this: > >> 1) Changing MTU directly from 1500 (default value) to 2000 doesn't > >> work because the driver finds out that 'new_frame_size > > >> ixgbe_rx_bufsz(ring)' in ixgbe_change_mtu() function. > >> 2) However, if we change MTU to 1501 then change from 1501 to 2000, it > >> does work, because the driver sets __IXGBE_RX_3K_BUFFER when MTU size > >> is converted to 1501, which later size check policy allows. > >> > >> The default MTU value for most servers is 1500 which cannot be adjusted > >> directly to the value larger than IXGBE_MAX_2K_FRAME_BUILD_SKB (1534 or > >> 1536) if it loads XDP. > >> > >> After I do a quick study on the manner of i40E driver allowing two kinds > >> of buffer size (one is 2048 while another is 3072) to support XDP mode in > >> i40e_max_xdp_frame_size(), I believe the default MTU size is possibly not > >> satisfied in XDP mode when IXGBE driver is in use, we sometimes need to > >> insert a new header, say, vxlan header. So setting the 3K-buffer flag > >> could solve the issue. > >> > >> [1] commit 38b7e7f8ae82 ("ixgbe: Do not allow LRO or MTU change with XDP") > >> [2] commit fabf1bce103a ("ixgbe: Prevent unsupported configurations with > >> XDP") > >> > >> Signed-off-by: Jason Xing <kernelxing@tencent.com> > >> --- > >> drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 3 +++ > >> 1 file changed, 3 insertions(+) > >> > >> diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > >> index ab8370c413f3..dc016582f91e 100644 > >> --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > >> +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c > >> @@ -4313,6 +4313,9 @@ static void ixgbe_set_rx_buffer_len(struct ixgbe_adapter *adapter) > >> if (IXGBE_2K_TOO_SMALL_WITH_PADDING || > >> (max_frame > (ETH_FRAME_LEN + ETH_FCS_LEN))) > >> set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state); > >> + > >> + if (ixgbe_enabled_xdp_adapter(adapter)) > >> + set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state); > > > > This will result with unnecessary overhead for 1500 MTU because you will > > be working on order-1 pages. Instead I would focus on fixing > > ixgbe_change_mtu() and stop relying on ixgbe_rx_bufsz() in there. You can > > check what we do on ice/i40e sides. Well, now I see the commit 23b44513c3e6f in 2019. Thanks, Maciej. > > > > I'm not looking actively into ixgbe internals but I don't think that there > > is anything that stops us from using 3k buffers with XDP. > > I think it uses the same logics as the rest of drivers: splits a 4k page > into two 2k buffers when MTU is <= 1536, otherwise uses order-1 pages > and uses 3k buffers. > > OTOH ixgbe is not fully correct in terms how it calculates Rx headroom, > but the main problem is how it calculates the maximum MTU available when > XDP is on. Our usual MTU supported when XDP is on is 3046 bytes. > For MTU <= 1536, 2k buffers are used even for XDP, so the fix is not > correct. Maciej is right that i40e and ice do that way better and don't > have such issue. Thank you for the detailed explanation. And yes, I checked this part in the ice/i40e driver which introduces ice/i40e_max_xdp_frame_size() to test if we can change MTU size when the driver is loading the XDP program. I will rewrite the patch as the i40e/ice does in the next submission. Thanks, Jason > > > > >> #endif > >> } > >> } > >> -- > >> 2.37.3 > >> > > Thanks, > Olek
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c index ab8370c413f3..dc016582f91e 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c @@ -4313,6 +4313,9 @@ static void ixgbe_set_rx_buffer_len(struct ixgbe_adapter *adapter) if (IXGBE_2K_TOO_SMALL_WITH_PADDING || (max_frame > (ETH_FRAME_LEN + ETH_FCS_LEN))) set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state); + + if (ixgbe_enabled_xdp_adapter(adapter)) + set_bit(__IXGBE_RX_3K_BUFFER, &rx_ring->state); #endif } }