From patchwork Tue Aug 8 03:19:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?6buE5p2w?= X-Patchwork-Id: 132756 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c44e:0:b0:3f2:4152:657d with SMTP id w14csp2331052vqr; Tue, 8 Aug 2023 12:00:15 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHwHo+8Il7OQOe7OzTeTu1M/CmC7+46FxMre4aINKIbK/Ko7JOw5iKH6/0m23u9FV5z/o2X X-Received: by 2002:a05:6a00:9a3:b0:67c:db:c2f4 with SMTP id u35-20020a056a0009a300b0067c00dbc2f4mr13725947pfg.4.1691521214634; Tue, 08 Aug 2023 12:00:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691521214; cv=none; d=google.com; s=arc-20160816; b=bX0ZvIU8p2+JAym9uR+SZ5Tu1Ud+LgYTJd+7QkFRAZlbSIVDINwqEh0RSxEX7BKhS/ 1t2Dt5Z+UehunhBBiipgmMRIfNpv8rz/zYTd3GAZ7uKCgSmeCcO9km2LqQzLV2G59JAb vGfk8vEkV3cs9HxhixtkqHGkVP6zrYuzlB/5meGvV2PKBgeSdXPGrZHujNFfXf2S9j9t rg7xJuRWwolA8Pf1W5UJgGC15G56cwJvaZJIiUrMGtyBmTziTo/oxrNJ30xD7APbb312 COy6//0DGrtHmhkK3yC6G5/R0Jf8rshdRQooCzQdu50jKnDAKHAeMPsYN7ofuJlr9T/C MqqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=uLQYdihKrZnqzA8pyJFR0KmRfev22Y9Nk5IK1CsNWSQ=; fh=d52JIWYEkePcj18BLr16Py79ZIzTYL7SzTuYhl8wvHo=; b=cDMvVDuymnNCwhGVfvtp73o5f0b2XV0rbbw2DR1lgpxge93ehnnr+lYdCUy38NqT0L l14RiQxnIIErrUbPwaq9UR4xaMC5mkscfFJ7CsaMRhAQYYcz+LoGvwlsEQIorSl5aG+Y fRy5lH+gUHFVZZMZ6aWrNxuIHGJU6eOUeoXcVPS5pbJq69XuKVFR3fkIgAD8boWLxAqA mP95sHScvC3Y5XOkKbzZxBBiC45LoeTM4b60omKk1WMFYd0cWVoMQGhGD6TjGo1+8auE 2iezCHRtzEzvFv0HyO9Xpy8DxNP9yA0WxBqlkl7T24iw9ON6IEavLM4KcHSXUw5kYTmT idFw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=Kl4mmZ6w; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u17-20020a056a00099100b0068254b9108bsi7762128pfg.71.2023.08.08.12.00.01; Tue, 08 Aug 2023 12:00:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=Kl4mmZ6w; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231235AbjHHDVA (ORCPT + 99 others); Mon, 7 Aug 2023 23:21:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45460 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229454AbjHHDU4 (ORCPT ); Mon, 7 Aug 2023 23:20:56 -0400 Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A405310F0 for ; Mon, 7 Aug 2023 20:20:28 -0700 (PDT) Received: by mail-pl1-x629.google.com with SMTP id d9443c01a7336-1bba54f7eefso40768345ad.1 for ; Mon, 07 Aug 2023 20:20:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1691464828; x=1692069628; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uLQYdihKrZnqzA8pyJFR0KmRfev22Y9Nk5IK1CsNWSQ=; b=Kl4mmZ6wC30QI0KYvWfqwsTsY4ihiWC63YfNOwTtHsCTiPRlVcI029sp+MJOROKfoo 2ZSjwgqwyuqMHwmhreAo6j9KGFeqiP4JTgT+5eIfWSoSekMcFAsja8NYdATAxJRAzZha LgHGf3BWqBvWvn5Ibdk2tZLiEfEVgU7EokhG1/IOeN7a4Al8TjQt7K1s0XC5mN196l4D ObsT7B1ydnVq2hD/R+PJ3GIdyMv70DAsQoTfRJJk0ZiyITCwrdg2BpYb2NUERKzwD37q 7WVSfkeOaG8pIqpShLoRNDtTBASpWi86pCdSd4Sb5itRrV6POLS4x2CKUFvjHQ28tKtd jk2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691464828; x=1692069628; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uLQYdihKrZnqzA8pyJFR0KmRfev22Y9Nk5IK1CsNWSQ=; b=fc1qJZKqx8Era/X7nlTfEb0JCHScaCLd6yjIIWiYB0C1LzCutcR+qf0O8rC62l7wSP nu9XFrUNm3Tnwn11KTRXjzh/jBo1ACkDbEJ9vbwf2vO98n75b7fQvSEDIw/YlEi4zxzA /SK3SPt0XPsng7CJ8s2LCkpKlzrnKFTeNZA1YSOycpb7Qjj90g6qlQlsalqfQb88dT2P IApRvEedZh0tEvoOA3ul1oHas9YL2h54vYmBozuuUGDwalgHmW3Wd/SgwurAvUs0vm6g TOoVQGn/Y0Bb2mJNGggtBxnQE8dBDqcKr5G97S/pkKEH5FfWhfscInQGzEHmGSCmU5oz 5hUQ== X-Gm-Message-State: AOJu0YwuUb3FsS7kih4CIfjoQrrJkzETNehPLGLiPRsrlIgEQt8wU2Zk r43tgmZU+XEtJJ45XeDveVXfKw== X-Received: by 2002:a17:902:d4c8:b0:1bc:1e17:6d70 with SMTP id o8-20020a170902d4c800b001bc1e176d70mr11761596plg.24.1691464827688; Mon, 07 Aug 2023 20:20:27 -0700 (PDT) Received: from C02FG34NMD6R.bytedance.net ([2408:8656:30f8:e020::b]) by smtp.gmail.com with ESMTPSA id 13-20020a170902c10d00b001b896686c78sm7675800pli.66.2023.08.07.20.20.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Aug 2023 20:20:27 -0700 (PDT) From: Albert Huang To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: Albert Huang , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Pavel Begunkov , Yunsheng Lin , Kees Cook , Richard Gobert , "open list:NETWORKING DRIVERS" , open list , "open list:XDP (eXpress Data Path)" Subject: [RFC v3 Optimizing veth xsk performance 1/9] veth: Implement ethtool's get_ringparam() callback Date: Tue, 8 Aug 2023 11:19:05 +0800 Message-Id: <20230808031913.46965-2-huangjie.albert@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230808031913.46965-1-huangjie.albert@bytedance.com> References: <20230808031913.46965-1-huangjie.albert@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773688549433752801 X-GMAIL-MSGID: 1773688549433752801 some xsk library calls get_ringparam() API to get the queue length to init the xsk umem. Implement that in veth so those scenarios can work properly. Signed-off-by: Albert Huang --- drivers/net/veth.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 614f3e3efab0..77e12d52ca2b 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -255,6 +255,17 @@ static void veth_get_channels(struct net_device *dev, static int veth_set_channels(struct net_device *dev, struct ethtool_channels *ch); +static void veth_get_ringparam(struct net_device *dev, + struct ethtool_ringparam *ring, + struct kernel_ethtool_ringparam *kernel_ring, + struct netlink_ext_ack *extack) +{ + ring->rx_max_pending = VETH_RING_SIZE; + ring->tx_max_pending = VETH_RING_SIZE; + ring->rx_pending = VETH_RING_SIZE; + ring->tx_pending = VETH_RING_SIZE; +} + static const struct ethtool_ops veth_ethtool_ops = { .get_drvinfo = veth_get_drvinfo, .get_link = ethtool_op_get_link, @@ -265,6 +276,7 @@ static const struct ethtool_ops veth_ethtool_ops = { .get_ts_info = ethtool_op_get_ts_info, .get_channels = veth_get_channels, .set_channels = veth_set_channels, + .get_ringparam = veth_get_ringparam, }; /* general routines */ From patchwork Tue Aug 8 03:19:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?6buE5p2w?= X-Patchwork-Id: 132535 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c44e:0:b0:3f2:4152:657d with SMTP id w14csp2213194vqr; Tue, 8 Aug 2023 08:50:35 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGCrd9WjDD1ZWXGOYY6JcK/9EszqNGT/3UVV8e89mhpF1u6ikHEw+EM7WU1eEfouCBo2tKe X-Received: by 2002:a17:90b:fce:b0:269:1860:c3bd with SMTP id gd14-20020a17090b0fce00b002691860c3bdmr9192041pjb.19.1691509834743; Tue, 08 Aug 2023 08:50:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691509834; cv=none; d=google.com; s=arc-20160816; b=gifWm9mp2vaEcpz+RFvLNhbMq2SYflA2Fq40F/2dJKgZDksvFGVwtEOr1QY77euWmu frGnf8Jy6AD4yTiAtqSHHj9Vy1+H7k6fGYOfo7r98zAqxbyKshPKrv3ul3XBwtGRIDhg 5kLfUb/aEjWVKBQ+qI8Z67JMrTSQO1fwu+VnctjQ4B/+LoZSDtIAsXmLcd3AA8sALvwm lPQf3EaqEwjurlKJ2FLS+XER7p7gd3LwjYJU73pS+Rlcwoc7tfoIDW78P5tnaECpfMGC +5NsicQf4KhwG5FxOsKM8NbabhFcJEb7vC9UlaolKWd+Gv2Q5XJYVc7G+X5hAmVZeQs4 FJjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CSkaRhEoax/kU+S7BeIv84prtFC3VRpkmwXx3cyNhW4=; fh=d52JIWYEkePcj18BLr16Py79ZIzTYL7SzTuYhl8wvHo=; b=FO1qUEMSaqnQ6wj5L5jUvR32zjGdA5XEq08YKOsz1lCHlqvgIfc6BpjdexxhcyLgBo 7Pn1K19LxyhcNkFoUsUQPRDPkr+QApevetmzS8BEFTZNTh1HbITHuL8ihDvrHiSgMazQ Q+LGE8oomg0LzueKyvSnNBMCVWgIR8Oqht/eIrtckuZIsPhSaLGNo0NoPYntw75+zJ3+ 8xYMI/DB0of/zvSnE5f8YgcZ5My4VNNxrkBHvlMgQNWInIBPk+g95rdNsjcBeoltemaV bpTfgw2fb9BB941W2bREsTDaZ6MTj+1+eDy8kVab5mCsXrLdtW8DWBegkLPccxb2vEp9 5gYQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=d6NExLM8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id kk17-20020a17090b4a1100b002692eb69014si6007474pjb.20.2023.08.08.08.50.21; Tue, 08 Aug 2023 08:50:34 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=d6NExLM8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231393AbjHHDVN (ORCPT + 99 others); Mon, 7 Aug 2023 23:21:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45492 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230128AbjHHDVA (ORCPT ); Mon, 7 Aug 2023 23:21:00 -0400 Received: from mail-pl1-x62c.google.com (mail-pl1-x62c.google.com [IPv6:2607:f8b0:4864:20::62c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6EE110DE for ; Mon, 7 Aug 2023 20:20:36 -0700 (PDT) Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-1bbf8cb61aeso33502795ad.2 for ; Mon, 07 Aug 2023 20:20:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1691464836; x=1692069636; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CSkaRhEoax/kU+S7BeIv84prtFC3VRpkmwXx3cyNhW4=; b=d6NExLM8tFgWdgX9G9vZowhLgc1yvXqs7IkLMLHXraFw++XbkXxgetWnrYgoDnH2PL bKoG+iR2qZdyLwagLCZt6X9V4gshueZ4JBNkk7hSH9GzbbXtE6DV8cFw2t6L291TW3Bl RkrxAU1mOtbcr70fK4bljLdaAEYWd4A5hr9wZ96t2zuuh/S5KsRZqo4l/VpAuTJk/0th pUM4uhplW0joZl356oAtpSoeLbDUed2TyxXYGQvRA20cg/tISFQFjAtUbYl0jgSapqG8 fJpW3L2C0NbOqpLGIqYRwXCjCJ31gazngAztABsx8PEOY9U9Ou7Yy/RRiZiEIIQEcmLC ZkHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691464836; x=1692069636; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CSkaRhEoax/kU+S7BeIv84prtFC3VRpkmwXx3cyNhW4=; b=cPnlSH3mizTEb1yjV+yo1dY5z9svjcqHKTYRqxyeepvVUlZrf2a1X59kzZAlnSWZ0S 31CNjw0g8xfzvsDz/ToxZ2nAc7gKebTTKBM4hwuHItHqmj/JEoT121nwJZYLBYSgswhF hGtHiXMP10SoEmhavrkcZi7lpRYotgMiPecOgFigMzs5Kd6oq1BiUM/OSfdBLUPdA1Gc ch/J60teq4sqd7V07Unbl9JUy+qX/eebAl2Crb/vwrpFDJMf5mLtEzVcAPILAfCnYc2/ y4J2Wi1kFQfSc6GADEug81ADyR94uNNK2AynX0OLN4aCUOsmpKcjeHnmoNvg4ZUAtRQE zSow== X-Gm-Message-State: AOJu0YzD+3d+6mlIuvanxVbnZWENqGDocCd93GnM9R1M7B6ReIGoKpz7 ijwe61dha3qFt2zDOiXpk7gM1g== X-Received: by 2002:a17:902:d48c:b0:1bc:1df2:4c07 with SMTP id c12-20020a170902d48c00b001bc1df24c07mr11625675plg.63.1691464836135; Mon, 07 Aug 2023 20:20:36 -0700 (PDT) Received: from C02FG34NMD6R.bytedance.net ([2408:8656:30f8:e020::b]) by smtp.gmail.com with ESMTPSA id 13-20020a170902c10d00b001b896686c78sm7675800pli.66.2023.08.07.20.20.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Aug 2023 20:20:35 -0700 (PDT) From: Albert Huang To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: Albert Huang , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Pavel Begunkov , Yunsheng Lin , Kees Cook , Richard Gobert , "open list:NETWORKING DRIVERS" , open list , "open list:XDP (eXpress Data Path)" Subject: [RFC v3 Optimizing veth xsk performance 2/9] xsk: add dma_check_skip for skipping dma check Date: Tue, 8 Aug 2023 11:19:06 +0800 Message-Id: <20230808031913.46965-3-huangjie.albert@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230808031913.46965-1-huangjie.albert@bytedance.com> References: <20230808031913.46965-1-huangjie.albert@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773676616358262202 X-GMAIL-MSGID: 1773676616358262202 for the virtual net device such as veth, there is no need to do dma check if we support zero copy. add this flag after unaligned. beacause there is 4 bytes hole pahole -V ./net/xdp/xsk_buff_pool.o: ----------- ... /* --- cacheline 3 boundary (192 bytes) --- */ u32 chunk_size; /* 192 4 */ u32 frame_len; /* 196 4 */ u8 cached_need_wakeup; /* 200 1 */ bool uses_need_wakeup; /* 201 1 */ bool dma_need_sync; /* 202 1 */ bool unaligned; /* 203 1 */ /* XXX 4 bytes hole, try to pack */ void * addrs; /* 208 8 */ spinlock_t cq_lock; /* 216 4 */ ... ----------- Signed-off-by: Albert Huang --- include/net/xsk_buff_pool.h | 1 + net/xdp/xsk_buff_pool.c | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/net/xsk_buff_pool.h b/include/net/xsk_buff_pool.h index b0bdff26fc88..fe31097dc11b 100644 --- a/include/net/xsk_buff_pool.h +++ b/include/net/xsk_buff_pool.h @@ -81,6 +81,7 @@ struct xsk_buff_pool { bool uses_need_wakeup; bool dma_need_sync; bool unaligned; + bool dma_check_skip; void *addrs; /* Mutual exclusion of the completion ring in the SKB mode. Two cases to protect: * NAPI TX thread and sendmsg error paths in the SKB destructor callback and when diff --git a/net/xdp/xsk_buff_pool.c b/net/xdp/xsk_buff_pool.c index b3f7b310811e..ed251b8e8773 100644 --- a/net/xdp/xsk_buff_pool.c +++ b/net/xdp/xsk_buff_pool.c @@ -85,6 +85,7 @@ struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs, XDP_PACKET_HEADROOM; pool->umem = umem; pool->addrs = umem->addrs; + pool->dma_check_skip = false; INIT_LIST_HEAD(&pool->free_list); INIT_LIST_HEAD(&pool->xskb_list); INIT_LIST_HEAD(&pool->xsk_tx_list); @@ -202,7 +203,7 @@ int xp_assign_dev(struct xsk_buff_pool *pool, if (err) goto err_unreg_pool; - if (!pool->dma_pages) { + if (!pool->dma_pages && !pool->dma_check_skip) { WARN(1, "Driver did not DMA map zero-copy buffers"); err = -EINVAL; goto err_unreg_xsk; From patchwork Tue Aug 8 03:19:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?6buE5p2w?= X-Patchwork-Id: 132531 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c44e:0:b0:3f2:4152:657d with SMTP id w14csp2207527vqr; Tue, 8 Aug 2023 08:41:04 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGv+rCQP09EpY/6NpDnMTGY15yUIh7pYJvBx9U2vZujveN09bXSs2dc6Qr0Tw51Q7dR8zd3 X-Received: by 2002:a17:902:f807:b0:1bb:a55d:c6d7 with SMTP id ix7-20020a170902f80700b001bba55dc6d7mr25859plb.66.1691509264230; Tue, 08 Aug 2023 08:41:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691509264; cv=none; d=google.com; s=arc-20160816; b=d0b+J5XSwb/d2E9WmeRAx43t6BPF6WCzNoHj7+UbNpE+S4xv92pL8UmgCZzl8xGpE5 gVW81zEZe0w5mpFHOxPnqF4B5tXNNISeBhvTcHyQ+HTNIsin7aYngmMiDwO/qyyG+gc8 0dKhC/IU1Xum6jA1pj7KBwNcrLIRQpawUtWjGo+lv7nQfjq9lhW2DCq7yTH7fO6WovjO izjy8/SQ6WjdwBkP8I7qIMuqmpMERltTW+Prn/0VSF3ii7YtXXk34M5B1sSfOluP4XMO LEFsrVtKrzawj+dF++fIr0hN0d/g5ofHQ8rYfkF9Ct5LjqZLh58wi4Zt6s+o3F2RUPpC AbDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9C7bHaMSd+SY+pPB7Vtx4/6UZIJ72tWmfxoauVOSVx0=; fh=d52JIWYEkePcj18BLr16Py79ZIzTYL7SzTuYhl8wvHo=; b=yoLiH0Eo915g23Cy0N3z50HBMr5PIDqKHlTfK/v09lsXEqMNDPCUnpWEnyUjjP0ThK 6ViH+R9hngBdRhEdNojk1ewBRkDy3h42anML85mV6bwJn+p5UGM9OO69ZA1o+cpzCZFm 8eNKJMBulZnd3ZCaVaOjrOdTASw+aimUPG99w+Fydl1iN2tljXRPESHaRaI1twhRNtpM 1ZwyZyxG4EB2h7RrcGgLmEiyxegcJvehLy9qErD4yrCJavCp2ZAu6UOBrVPDp/qyUYeJ sWR/nQGSv3TkSh91shAvNwCYN7RwZ8tCZHyyB9oW0Q4I78aCpNWZsLQba796UctOTGhs gqkw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=VHfxDg5Q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k11-20020a170902c40b00b001bc553d16e0si5853030plk.507.2023.08.08.08.40.51; Tue, 08 Aug 2023 08:41:04 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=VHfxDg5Q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231210AbjHHDUy (ORCPT + 99 others); Mon, 7 Aug 2023 23:20:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231157AbjHHDUp (ORCPT ); Mon, 7 Aug 2023 23:20:45 -0400 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C358A10DC for ; Mon, 7 Aug 2023 20:20:43 -0700 (PDT) Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-1bc73a2b0easo8964765ad.0 for ; Mon, 07 Aug 2023 20:20:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1691464843; x=1692069643; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9C7bHaMSd+SY+pPB7Vtx4/6UZIJ72tWmfxoauVOSVx0=; b=VHfxDg5QSz1PqjRWt9B9x6mNaS6RhCwH3kbVmHYf16gDy0fMcTjbQfLdJ1bUO9PgyF e858P0e+77eKT6XZAaPITT05KvwS75X5CChow28CxQevaNfwHO/a/b/Tjth6QhYmamUl D2koPxHSfUnNqjewXP7wEAFJPTs3dyQoX1v5vV3qDUxrlvdVl0KOQKfXzW86ewoVN7mA /80D4Vn2cmagGQlwfUnU6NadEKweg09+IK+04wQqqwm6DaEtzrp97grfGS8he+jLvx/k eWU21VDBsw4THoGugTPD0TDkYAr8PyGh2PllRIUx5Bm4PqGbt8R1AGICgsH1UV/Qg33Y IKPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691464843; x=1692069643; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9C7bHaMSd+SY+pPB7Vtx4/6UZIJ72tWmfxoauVOSVx0=; b=Z0GK9sJ93Bj4mxmCK8FbBdK98zsvZp/9F2IpYptBe/fYoPdCYy2Uw86oHYHaZ/MhH0 JWwlk837gooaXPbwQ0rt6lMfaidwbuHsr44Eoe7J+B5y2iK8kiDWJm4vUF33mxWSoMRY midOoAHBHJNuOp6iSLEEAq/50elAXcV6+jRTEio6MlwxjvYmUrSWCz8OfUR/LfQEXJqT iXvGLr0ZbP8U9L4h964zfYLVygkiCL7IbxIzskb4PrbVyPQgWCyoukycEMO3njiSkNPc h16PmFBzKG0PAmyO8NeTVO5u5Ny4+pi+EV8P0avQNsTooRsbDeDzZGIoiXFsNFsVj0Ve whWg== X-Gm-Message-State: AOJu0Yy5sdKo4MUWzeB1DrxjPe0zH4uK8jFilz3EKEwnLXJ94GD/yvyf v67zeKIT/iypg1r3rVZSTwdRPA== X-Received: by 2002:a17:902:c404:b0:1bb:a367:a77 with SMTP id k4-20020a170902c40400b001bba3670a77mr11633095plk.31.1691464843297; Mon, 07 Aug 2023 20:20:43 -0700 (PDT) Received: from C02FG34NMD6R.bytedance.net ([2408:8656:30f8:e020::b]) by smtp.gmail.com with ESMTPSA id 13-20020a170902c10d00b001b896686c78sm7675800pli.66.2023.08.07.20.20.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Aug 2023 20:20:42 -0700 (PDT) From: Albert Huang To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: Albert Huang , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Pavel Begunkov , Yunsheng Lin , Kees Cook , Richard Gobert , "open list:NETWORKING DRIVERS" , open list , "open list:XDP (eXpress Data Path)" Subject: [RFC v3 Optimizing veth xsk performance 3/9] veth: add support for send queue Date: Tue, 8 Aug 2023 11:19:07 +0800 Message-Id: <20230808031913.46965-4-huangjie.albert@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230808031913.46965-1-huangjie.albert@bytedance.com> References: <20230808031913.46965-1-huangjie.albert@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773676017891864229 X-GMAIL-MSGID: 1773676017891864229 in order to support native af_xdp for veth. we need support for send queue for napi tx. the upcoming patch will make use of it. Signed-off-by: Albert Huang --- drivers/net/veth.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 77e12d52ca2b..25faba879505 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -56,6 +56,11 @@ struct veth_rq_stats { struct u64_stats_sync syncp; }; +struct veth_sq_stats { + struct veth_stats vs; + struct u64_stats_sync syncp; +}; + struct veth_rq { struct napi_struct xdp_napi; struct napi_struct __rcu *napi; /* points to xdp_napi when the latter is initialized */ @@ -69,11 +74,25 @@ struct veth_rq { struct page_pool *page_pool; }; +struct veth_sq { + struct napi_struct xdp_napi; + struct net_device *dev; + struct xdp_mem_info xdp_mem; + struct veth_sq_stats stats; + u32 queue_index; + /* for xsk */ + struct { + struct xsk_buff_pool __rcu *pool; + u32 last_cpu; + } xsk; +}; + struct veth_priv { struct net_device __rcu *peer; atomic64_t dropped; struct bpf_prog *_xdp_prog; struct veth_rq *rq; + struct veth_sq *sq; unsigned int requested_headroom; }; @@ -1495,6 +1514,15 @@ static int veth_alloc_queues(struct net_device *dev) u64_stats_init(&priv->rq[i].stats.syncp); } + priv->sq = kcalloc(dev->num_tx_queues, sizeof(*priv->sq), GFP_KERNEL); + if (!priv->sq) + return -ENOMEM; + + for (i = 0; i < dev->num_tx_queues; i++) { + priv->sq[i].dev = dev; + u64_stats_init(&priv->sq[i].stats.syncp); + } + return 0; } @@ -1503,6 +1531,7 @@ static void veth_free_queues(struct net_device *dev) struct veth_priv *priv = netdev_priv(dev); kfree(priv->rq); + kfree(priv->sq); } static int veth_dev_init(struct net_device *dev) From patchwork Tue Aug 8 03:19:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?6buE5p2w?= X-Patchwork-Id: 132530 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c44e:0:b0:3f2:4152:657d with SMTP id w14csp2207355vqr; Tue, 8 Aug 2023 08:40:46 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFimMnujPwNajObqjDKqRM0GHQtDvcjoLp92xypCoAi9Gvjwqu4pR2HQh1ioDL14iehhTUL X-Received: by 2002:a05:6a21:279f:b0:105:6d0e:c046 with SMTP id rn31-20020a056a21279f00b001056d0ec046mr11794578pzb.26.1691509246235; Tue, 08 Aug 2023 08:40:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691509246; cv=none; d=google.com; s=arc-20160816; b=diST5gNtjUni2/ez7V+8hfYrSUCMePeAWGUNSWP/plOiAbCRgYZxKynh+/oBTOF6N6 pDK9ciIOcBG3LBrnQYDrHeayhM2+j28tayd7/swVE7OC9sE2nvnUl7tDhgrC8qr7nq7Z FdrU/DTf4r7eYpkWMZYtkgcyd6Odo8TzwsIRW9vgDXv6QKg2RW0YfVEwa87wEaEi3kyF QPoXoj5APcFeleuvb/eJsuqBQg19j7YK+13oaNy2CDO90TlooD6EOOMbmkNvdnK6NQRY fySBpMlT28vD9NNV4RhAbCZHl3Alz8HFQ2hix5McyPDXABk+wDkGSRtigeOSSWVjPO9v wfBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=nhqzuKs8acODjWqieOr60TzQPrln+0aaO9utMxKEwXY=; fh=d52JIWYEkePcj18BLr16Py79ZIzTYL7SzTuYhl8wvHo=; b=N9xhZBjqNRFn/r2vNx3HdLOxjwel7xz6URi5dOzxoz54UQrG/AGEETZCYF7lfpb6/s tV8TIn+fBYCQVEYFSpLJIf0kcTg6ZO5ZhRQXW8gnt3d5tqDxcgz+CRjLDjNmUf9XTLPX s0+1dFYeEDDUL+uR4IrcENtsvJs/L8C+O0bFUXDel5I2zKrwLM+7d+0ersb8fg/5DuW9 q96NHOWozw9B43bRaWbIiFS7FsthSVF7rDZlk/b1hpcYZx/0lujypFqxlVNfdTR3fmjN 4wVjZwtbeSapqhMjW9ypj54B6gtLY87XMqoxE+CyphTLWkLwzUyAgK4KsB0K2KrRS2+h Vw+g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=OJL9LrbS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 81-20020a630154000000b005645fcc15absi7357941pgb.266.2023.08.08.08.40.32; Tue, 08 Aug 2023 08:40:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=OJL9LrbS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229722AbjHHDVt (ORCPT + 99 others); Mon, 7 Aug 2023 23:21:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229998AbjHHDVO (ORCPT ); Mon, 7 Aug 2023 23:21:14 -0400 Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B7DB10D2 for ; Mon, 7 Aug 2023 20:20:50 -0700 (PDT) Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-1bc63ef9959so19864395ad.2 for ; Mon, 07 Aug 2023 20:20:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1691464850; x=1692069650; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nhqzuKs8acODjWqieOr60TzQPrln+0aaO9utMxKEwXY=; b=OJL9LrbSgQt70QJMij8iZC+JU41pG03VeJ0y7Nl922F2ZaU7U87A5B753UGGc0Yzx1 /bOWzzKWiz5MHCq3MgepwcOP88c3QaVKgVkTsFPHfIKsB1CIEy7ISR3PzFTQvfTVch3s WnjevnnH9SsiyAEF6GrygCCohR3vtxY6qtYmJVOnIJAzQC5P3SlBXIP0HPbiUcyU/Tpg WzO3Noh9RFrUXrGlfzBi0PbKEYMSibgslfmMf83UoYgMOmjvVh7T/nHKLh9NspmzZ0bp ZmzvtRNd6WNniQsh/hzTpgIkY/Il5uuKqN/T/P4pQ1UI1p7bI63YzkuOWTSh/Dn//l3M qjZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691464850; x=1692069650; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nhqzuKs8acODjWqieOr60TzQPrln+0aaO9utMxKEwXY=; b=LtgxD1WjbXT1PaQt5uGg7O1Psggg9QbDwI8fZNx08fjEiggSK5l40v5uXaw5ZVHdzV i9nRv6HApq5kFj4fmk5mjLzSaYid4RYerLLVZDW2bE8uJ9+uV9KRvfCoFfErlpgsDPRI jHqznJmtLyG9BiPW0w7ansVbUeyP89dVVY+kUw/hW2JN1ETMM+zDuC4MjirR577wHJUh tNTP73suKGK9keKx5aGYGy5cZLOS+Rl0nrWmBqsyiCtNZoe6NcIirnD7WjJSjZRN1rMJ PxpDv2+w8n0uuRdxKrxNd8QcdfesJXattDnIQoSCHvftZn01s1hJhG8UB6Hj5LbOs+TC MLGA== X-Gm-Message-State: AOJu0YzNzLuIzdseg3Twzz8qe3hkn5WparVl0jv0h12A178TIqdje2aE e5jb7jnr1JWMxt3IuAuXqWz5qA== X-Received: by 2002:a17:902:eccc:b0:1b8:1335:b775 with SMTP id a12-20020a170902eccc00b001b81335b775mr13965313plh.0.1691464850307; Mon, 07 Aug 2023 20:20:50 -0700 (PDT) Received: from C02FG34NMD6R.bytedance.net ([2408:8656:30f8:e020::b]) by smtp.gmail.com with ESMTPSA id 13-20020a170902c10d00b001b896686c78sm7675800pli.66.2023.08.07.20.20.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Aug 2023 20:20:49 -0700 (PDT) From: Albert Huang To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: Albert Huang , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Pavel Begunkov , Yunsheng Lin , Kees Cook , Richard Gobert , "open list:NETWORKING DRIVERS" , open list , "open list:XDP (eXpress Data Path)" Subject: [RFC v3 Optimizing veth xsk performance 4/9] xsk: add xsk_tx_completed_addr function Date: Tue, 8 Aug 2023 11:19:08 +0800 Message-Id: <20230808031913.46965-5-huangjie.albert@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230808031913.46965-1-huangjie.albert@bytedance.com> References: <20230808031913.46965-1-huangjie.albert@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773675999156440070 X-GMAIL-MSGID: 1773675999156440070 Return desc to the cq by using the descriptor address. Signed-off-by: Albert Huang --- include/net/xdp_sock_drv.h | 5 +++++ net/xdp/xsk.c | 6 ++++++ net/xdp/xsk_queue.h | 10 ++++++++++ 3 files changed, 21 insertions(+) diff --git a/include/net/xdp_sock_drv.h b/include/net/xdp_sock_drv.h index 1f6fc8c7a84c..de82c596e48f 100644 --- a/include/net/xdp_sock_drv.h +++ b/include/net/xdp_sock_drv.h @@ -15,6 +15,7 @@ #ifdef CONFIG_XDP_SOCKETS void xsk_tx_completed(struct xsk_buff_pool *pool, u32 nb_entries); +void xsk_tx_completed_addr(struct xsk_buff_pool *pool, u64 addr); bool xsk_tx_peek_desc(struct xsk_buff_pool *pool, struct xdp_desc *desc); u32 xsk_tx_peek_release_desc_batch(struct xsk_buff_pool *pool, u32 max); void xsk_tx_release(struct xsk_buff_pool *pool); @@ -188,6 +189,10 @@ static inline void xsk_tx_completed(struct xsk_buff_pool *pool, u32 nb_entries) { } +static inline void xsk_tx_completed_addr(struct xsk_buff_pool *pool, u64 addr) +{ +} + static inline bool xsk_tx_peek_desc(struct xsk_buff_pool *pool, struct xdp_desc *desc) { diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index 4f1e0599146e..b2b8aa7b0bcf 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -396,6 +396,12 @@ void xsk_tx_completed(struct xsk_buff_pool *pool, u32 nb_entries) } EXPORT_SYMBOL(xsk_tx_completed); +void xsk_tx_completed_addr(struct xsk_buff_pool *pool, u64 addr) +{ + xskq_prod_submit_addr(pool->cq, addr); +} +EXPORT_SYMBOL(xsk_tx_completed_addr); + void xsk_tx_release(struct xsk_buff_pool *pool) { struct xdp_sock *xs; diff --git a/net/xdp/xsk_queue.h b/net/xdp/xsk_queue.h index 13354a1e4280..3a5e26a81dc2 100644 --- a/net/xdp/xsk_queue.h +++ b/net/xdp/xsk_queue.h @@ -428,6 +428,16 @@ static inline void __xskq_prod_submit(struct xsk_queue *q, u32 idx) smp_store_release(&q->ring->producer, idx); /* B, matches C */ } +static inline void xskq_prod_submit_addr(struct xsk_queue *q, u64 addr) +{ + struct xdp_umem_ring *ring = (struct xdp_umem_ring *)q->ring; + u32 idx = q->ring->producer; + + ring->desc[idx++ & q->ring_mask] = addr; + + __xskq_prod_submit(q, idx); +} + static inline void xskq_prod_submit(struct xsk_queue *q) { __xskq_prod_submit(q, q->cached_prod); From patchwork Tue Aug 8 03:19:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?6buE5p2w?= X-Patchwork-Id: 132529 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c44e:0:b0:3f2:4152:657d with SMTP id w14csp2205870vqr; Tue, 8 Aug 2023 08:38:25 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGyR0Wi/8ptwpqQeP4omMV46WXRwcfBSzghJfIITLly6CFx9pNnLydDQD1YM83YhC4iv4ms X-Received: by 2002:a17:90b:30c2:b0:268:220a:7080 with SMTP id hi2-20020a17090b30c200b00268220a7080mr12639963pjb.2.1691509105471; Tue, 08 Aug 2023 08:38:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691509105; cv=none; d=google.com; s=arc-20160816; b=ExLgrvrICegEkT87M6ioMeJTqSYrUXJItNl2WT1iprnYypdusdlFqMraJ3tF/JSm+I U8DdTCqaPEJ70c0lbdna9H/qyR+9YxvbXbGm1yRmq0tUttvEV8Oku1X6ZWia0rDG4aWA I8aewrujIXf6AzE+CmhMVEJjbP0VgyIKsYX4bmq5FR6yNEpIOqmEsNu1sHivzxVquoMe YSemIHFBwDvTvwW4YbfEWtzuYxSi5SrMtJPQkicN297U/Ep8YsKbEY0kGG7IJU+PHhKK lLZR9y7Pn8DbhA+npXIdlTU6TZM2UViwgLoQyT4fiuSZCu+FW5905EcS85pbPKdFfWhq WEAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=tzvycUSp6fj3lhy6v0s47NdlhPCC8jzqygliYQ4HaF8=; fh=d52JIWYEkePcj18BLr16Py79ZIzTYL7SzTuYhl8wvHo=; b=wPLpmSZbaLO4ts9mH92tmzCu6UBcleV8EgrZiuROFyAv1ReW9mji//xUFbkTLopphA hWBgV2GgFK1Cj+E/x7cgh/dSaC1Gv3QIus6GXFFx2mohaBWQcixBNqOXVGICVdlkSYqo 0M//lDVvY1EtCPXTUsPFoyG1bmLQNybygbmZ+VVhkWGPHa4Kxz9jOO0af4r5SvPPnsn7 P1FOccPJITQyW7sjfyK5iON1+5JnblGCcRcKqXfsu1XGLIvVPDZ5R5ZEG2VpfbQ64l1c ked5uMp4aOGWK2RsQC3/2N5MVmZwj7zHhb5LlwxhcGiG2K0H7CQahLEmxN6AGDQ1Jdcm x+YA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="NvVq2h/3"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q16-20020a17090aa01000b0025bae7f2b10si10809832pjp.13.2023.08.08.08.38.11; Tue, 08 Aug 2023 08:38:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="NvVq2h/3"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231365AbjHHDVG (ORCPT + 99 others); Mon, 7 Aug 2023 23:21:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231225AbjHHDVA (ORCPT ); Mon, 7 Aug 2023 23:21:00 -0400 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DBD4D138 for ; Mon, 7 Aug 2023 20:20:57 -0700 (PDT) Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-1bba54f7eefso40769765ad.1 for ; Mon, 07 Aug 2023 20:20:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1691464857; x=1692069657; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tzvycUSp6fj3lhy6v0s47NdlhPCC8jzqygliYQ4HaF8=; b=NvVq2h/3LxshBlFB6gD30775uo3/1fXs1KQJKJFoXl/OGNxuF83Tu9AIGUcdDtSWix TbsEnaXloQxMxlKaCVptTVgdab0tdTLokOycU/saAgyxbajhxKhjG0JFKl+S2ntaQDEi H/BkS3UzYJXQTbydd1Fh8rJ28tHVdjdslTXSLoKRAaE6VlD7bwEbCCHTjH2Yl/CtlOgq qE8YFjdsbQs2aSzfcriCwjrjvYqmwhvsFV5aopNlV9BEMdlZ9Dcqo9riMIPW6sSi4gWL WHT5FQQxnzB4mD3NT1+4b1bHu+7RUbCzfXxMfDLL5kYnoz3Gbztq3ildDRlo8jDGdh0u QzSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691464857; x=1692069657; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tzvycUSp6fj3lhy6v0s47NdlhPCC8jzqygliYQ4HaF8=; b=jE6sKLnFQJ2QArLBT4BsmxgYKKAtM7Y1p1hs4AqUjSUC6bu1h2uVJdR9L412w/fr+J YxrwQ/WL/MKlZlu65eM7WwwfpjrzKgBDZakoGhy8eeI/99RjoRyeml/M8NoTPDDO1LTp FVddDnI6fMBue5UWT6FFkX+oTL3amY9tFTrHK44dgGkJdgmpFHPT7uwE/lyMyZRz+926 vbVm2irHwMgxo6ebAThxI1PeYE5zehi14Otl/xhna+b9uIChAhMifvMCntpWB+ujRHRq Ij1saahAf3hH0tGo+cWkIaB+wyszS6G/Ffr7GGdq4p0lAGT+WlfddNYJ+5GvQ6aHFUDQ jiiw== X-Gm-Message-State: AOJu0YxsJkho3ATmRro6VjH5Q16SzlmXWG9lVfmbijsYKrDOYdabtiKi RxlNa2OpJuQW3JDRDmDkbzITmw== X-Received: by 2002:a17:902:6bc1:b0:1b6:6a14:3734 with SMTP id m1-20020a1709026bc100b001b66a143734mr8363404plt.29.1691464857325; Mon, 07 Aug 2023 20:20:57 -0700 (PDT) Received: from C02FG34NMD6R.bytedance.net ([2408:8656:30f8:e020::b]) by smtp.gmail.com with ESMTPSA id 13-20020a170902c10d00b001b896686c78sm7675800pli.66.2023.08.07.20.20.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Aug 2023 20:20:56 -0700 (PDT) From: Albert Huang To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: Albert Huang , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Pavel Begunkov , Yunsheng Lin , Kees Cook , Richard Gobert , "open list:NETWORKING DRIVERS" , open list , "open list:XDP (eXpress Data Path)" Subject: [RFC v3 Optimizing veth xsk performance 5/9] veth: use send queue tx napi to xmit xsk tx desc Date: Tue, 8 Aug 2023 11:19:09 +0800 Message-Id: <20230808031913.46965-6-huangjie.albert@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230808031913.46965-1-huangjie.albert@bytedance.com> References: <20230808031913.46965-1-huangjie.albert@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773675851967737719 X-GMAIL-MSGID: 1773675851967737719 use send queue tx napi to xmit xsk tx desc Signed-off-by: Albert Huang --- drivers/net/veth.c | 230 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 229 insertions(+), 1 deletion(-) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 25faba879505..28b891dd8dc9 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -27,6 +27,8 @@ #include #include #include +#include +#include #define DRV_NAME "veth" #define DRV_VERSION "1.0" @@ -1061,6 +1063,141 @@ static int veth_poll(struct napi_struct *napi, int budget) return done; } +static struct sk_buff *veth_build_skb(void *head, int headroom, int len, + int buflen) +{ + struct sk_buff *skb; + + skb = build_skb(head, buflen); + if (!skb) + return NULL; + + skb_reserve(skb, headroom); + skb_put(skb, len); + + return skb; +} + +static int veth_xsk_tx_xmit(struct veth_sq *sq, struct xsk_buff_pool *xsk_pool, int budget) +{ + struct veth_priv *priv, *peer_priv; + struct net_device *dev, *peer_dev; + struct veth_stats stats = {}; + struct sk_buff *skb = NULL; + struct veth_rq *peer_rq; + struct xdp_desc desc; + int done = 0; + + dev = sq->dev; + priv = netdev_priv(dev); + peer_dev = priv->peer; + peer_priv = netdev_priv(peer_dev); + + /* todo: queue index must set before this */ + peer_rq = &peer_priv->rq[sq->queue_index]; + + /* set xsk wake up flag, to do: where to disable */ + if (xsk_uses_need_wakeup(xsk_pool)) + xsk_set_tx_need_wakeup(xsk_pool); + + while (budget-- > 0) { + unsigned int truesize = 0; + struct page *page; + void *vaddr; + void *addr; + + if (!xsk_tx_peek_desc(xsk_pool, &desc)) + break; + + addr = xsk_buff_raw_get_data(xsk_pool, desc.addr); + + /* can not hold all data in a page */ + truesize = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); + truesize += desc.len + xsk_pool->headroom; + if (truesize > PAGE_SIZE) { + xsk_tx_completed_addr(xsk_pool, desc.addr); + stats.xdp_drops++; + break; + } + + page = dev_alloc_page(); + if (!page) { + xsk_tx_completed_addr(xsk_pool, desc.addr); + stats.xdp_drops++; + break; + } + vaddr = page_to_virt(page); + + memcpy(vaddr + xsk_pool->headroom, addr, desc.len); + xsk_tx_completed_addr(xsk_pool, desc.addr); + + skb = veth_build_skb(vaddr, xsk_pool->headroom, desc.len, PAGE_SIZE); + if (!skb) { + put_page(page); + stats.xdp_drops++; + break; + } + skb->protocol = eth_type_trans(skb, peer_dev); + napi_gro_receive(&peer_rq->xdp_napi, skb); + + stats.xdp_bytes += desc.len; + done++; + } + + /* release, move consumer，and wakeup the producer */ + if (done) { + napi_schedule(&peer_rq->xdp_napi); + xsk_tx_release(xsk_pool); + } + + u64_stats_update_begin(&sq->stats.syncp); + sq->stats.vs.xdp_packets += done; + sq->stats.vs.xdp_bytes += stats.xdp_bytes; + sq->stats.vs.xdp_drops += stats.xdp_drops; + u64_stats_update_end(&sq->stats.syncp); + + return done; +} + +static int veth_poll_tx(struct napi_struct *napi, int budget) +{ + struct veth_sq *sq = container_of(napi, struct veth_sq, xdp_napi); + struct xsk_buff_pool *pool; + int done = 0; + + sq->xsk.last_cpu = smp_processor_id(); + + /* xmit for tx queue */ + rcu_read_lock(); + pool = rcu_dereference(sq->xsk.pool); + if (pool) + done = veth_xsk_tx_xmit(sq, pool, budget); + + rcu_read_unlock(); + + if (done < budget) { + /* if done < budget, the tx ring is no buffer */ + napi_complete_done(napi, done); + } + + return done; +} + +static int veth_napi_add_tx(struct net_device *dev) +{ + struct veth_priv *priv = netdev_priv(dev); + int i; + + for (i = 0; i < dev->real_num_rx_queues; i++) { + struct veth_sq *sq = &priv->sq[i]; + + netif_napi_add(dev, &sq->xdp_napi, veth_poll_tx); + napi_enable(&sq->xdp_napi); + } + + return 0; +} + static int veth_create_page_pool(struct veth_rq *rq) { struct page_pool_params pp_params = { @@ -1153,6 +1290,19 @@ static void veth_napi_del_range(struct net_device *dev, int start, int end) } } +static void veth_napi_del_tx(struct net_device *dev) +{ + struct veth_priv *priv = netdev_priv(dev); + int i; + + for (i = 0; i < dev->real_num_rx_queues; i++) { + struct veth_sq *sq = &priv->sq[i]; + + napi_disable(&sq->xdp_napi); + __netif_napi_del(&sq->xdp_napi); + } +} + static void veth_napi_del(struct net_device *dev) { veth_napi_del_range(dev, 0, dev->real_num_rx_queues); @@ -1360,7 +1510,7 @@ static void veth_set_xdp_features(struct net_device *dev) struct veth_priv *priv_peer = netdev_priv(peer); xdp_features_t val = NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT | - NETDEV_XDP_ACT_RX_SG; + NETDEV_XDP_ACT_RX_SG | NETDEV_XDP_ACT_XSK_ZEROCOPY; if (priv_peer->_xdp_prog || veth_gro_requested(peer)) val |= NETDEV_XDP_ACT_NDO_XMIT | @@ -1737,11 +1887,89 @@ static int veth_xdp_set(struct net_device *dev, struct bpf_prog *prog, return err; } +static int veth_xsk_pool_enable(struct net_device *dev, struct xsk_buff_pool *pool, u16 qid) +{ + struct veth_priv *peer_priv; + struct veth_priv *priv = netdev_priv(dev); + struct net_device *peer_dev = priv->peer; + int err = 0; + + if (qid >= dev->real_num_tx_queues) + return -EINVAL; + + if (!peer_dev) + return -EINVAL; + + /* no dma, so we just skip dma skip in xsk zero copy */ + pool->dma_check_skip = true; + + peer_priv = netdev_priv(peer_dev); + + /* enable peer tx xdp here, this side + * xdp is enable by veth_xdp_set + * to do: we need to check whther this side is already enable xdp + * maybe it do not have xdp prog + */ + if (!(peer_priv->_xdp_prog) && (!veth_gro_requested(peer_dev))) { + /* peer should enable napi*/ + err = veth_napi_enable(peer_dev); + if (err) + return err; + } + + /* Here is already protected by rtnl_lock, so rcu_assign_pointer + * is safe. + */ + rcu_assign_pointer(priv->sq[qid].xsk.pool, pool); + + veth_napi_add_tx(dev); + + return err; +} + +static int veth_xsk_pool_disable(struct net_device *dev, u16 qid) +{ + struct veth_priv *peer_priv; + struct veth_priv *priv = netdev_priv(dev); + struct net_device *peer_dev = priv->peer; + int err = 0; + + if (qid >= dev->real_num_tx_queues) + return -EINVAL; + + if (!peer_dev) + return -EINVAL; + + peer_priv = netdev_priv(peer_dev); + + /* to do: this may be failed */ + if (!(peer_priv->_xdp_prog) && (!veth_gro_requested(peer_dev))) { + /* disable peer napi */ + veth_napi_del(peer_dev); + } + + veth_napi_del_tx(dev); + + rcu_assign_pointer(priv->sq[qid].xsk.pool, NULL); + return err; +} + +/* this is for setup xdp */ +static int veth_xsk_pool_setup(struct net_device *dev, struct netdev_bpf *xdp) +{ + if (xdp->xsk.pool) + return veth_xsk_pool_enable(dev, xdp->xsk.pool, xdp->xsk.queue_id); + else + return veth_xsk_pool_disable(dev, xdp->xsk.queue_id); +} + static int veth_xdp(struct net_device *dev, struct netdev_bpf *xdp) { switch (xdp->command) { case XDP_SETUP_PROG: return veth_xdp_set(dev, xdp->prog, xdp->extack); + case XDP_SETUP_XSK_POOL: + return veth_xsk_pool_setup(dev, xdp); default: return -EINVAL; } From patchwork Tue Aug 8 03:19:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?6buE5p2w?= X-Patchwork-Id: 132534 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c44e:0:b0:3f2:4152:657d with SMTP id w14csp2212193vqr; Tue, 8 Aug 2023 08:48:43 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFfRHEIxKAPDHJhRfVk9JFmaJhn2HzdwEytc7dYXeexyc0JorRGpmBV+dJGtqSj+ajIKtDK X-Received: by 2002:a17:902:b193:b0:1b3:fb76:215b with SMTP id s19-20020a170902b19300b001b3fb76215bmr52526plr.48.1691509723593; Tue, 08 Aug 2023 08:48:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691509723; cv=none; d=google.com; s=arc-20160816; b=lm9/ru69TV2HpoBX8e4ywOY0yF0gELPAP3Rjr9r4Pmnu4kOWsPtOrlIIc7/QUyuJ34 8Q/zBq8rXMBGlN3pAh4XyRjYAxhhNIW42phq/5APr7LKkOMkpHlLMqfhthuSUS/c45G+ H33uxgkWXacn1xugTWVM97UgbkyhWOTKr82OJHJqdJATOPIIeOqpXLty2b8tdxfq1BPL OeQFsoqwGMQ2Z+wXQWgUV0tAcXIKHXXNsM7ZRitmn52XwI0GtBN+DpPUfF0fFgLPnsqc 3aqOxqtzh7kM5v2Ps2VoLK0KGrsUo9NU4n3VVZH8fM0g/+MSA9geiSnKNOCJIyu5fszX zfuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ICV3oSZrEepyfkdFh294C2oc/HDgt3q4tnvL6u1XOSo=; fh=d52JIWYEkePcj18BLr16Py79ZIzTYL7SzTuYhl8wvHo=; b=KF3qNAZPoNLWoCnOHfEpox9rczZYFZlls2Fb9RL0Tt2+VK+3HsXobg4AUU1d/9SJUO obqmWLxN89/Vvx6QrR+F/6YQbVg7XdLAzBvCYIJt40YOcAug4bafFVfvfbzQlbHBmsMB 3fNthcjVuYmOxpRn6Ria6yMjjibycEX6cxWeU8Zo5UQ+yUU9S/1hFuLfe3C7H7+k9mSe G5gwT4FMh4VeLg7n9/VDFICqHj4DK0NFiE3Wf8Ofoqzn2lVJKOauCZtukH9Wjq7JM7qa OS3spksLU/f96BvXe/yKgmAkol5iYQu1qTs9iu1jM0MKbERuILc6WBL0awKmgS+gcAdn 7UOQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=lutZ0A2W; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id lm7-20020a170903298700b001bba7713547si7687091plb.151.2023.08.08.08.48.30; Tue, 08 Aug 2023 08:48:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=lutZ0A2W; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231428AbjHHDV3 (ORCPT + 99 others); Mon, 7 Aug 2023 23:21:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231370AbjHHDVG (ORCPT ); Mon, 7 Aug 2023 23:21:06 -0400 Received: from mail-pg1-x530.google.com (mail-pg1-x530.google.com [IPv6:2607:f8b0:4864:20::530]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 047281729 for ; Mon, 7 Aug 2023 20:21:05 -0700 (PDT) Received: by mail-pg1-x530.google.com with SMTP id 41be03b00d2f7-53482b44007so3016567a12.2 for ; Mon, 07 Aug 2023 20:21:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1691464864; x=1692069664; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ICV3oSZrEepyfkdFh294C2oc/HDgt3q4tnvL6u1XOSo=; b=lutZ0A2WljE0PHMq69IriO2rZ3mTGdwt4yVVFZOwx7KaXGHtsYthZEnorDptzuoR9Z 3bTfCPeVS8fpSvafHRcMLxrDNgnA2loby72Wgu9cfloNQRxmHCbPK3vKdpTZfK8jUlmF 7p0hYB/3nnDjWEtIceXVAL8kl4qqmuLDT5CgZQQRARE9KRnFvExGQzyuneXXfHttAgdL zsKB0Oh+uxe0BPwxqJaQMOPzn/nTgtz9LdZMlVg0RTJdKhq0bRAT7rJsnC7zn+ynGm5n Lymc0VkgXf3C5i+q0ZNqVquZH5V2gVRVSKb9Yn7GpEFzEVlv3u68qZNnVPavuo/G/XFK RNwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691464864; x=1692069664; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ICV3oSZrEepyfkdFh294C2oc/HDgt3q4tnvL6u1XOSo=; b=OqbWEPWagk7j7me7n4HvNtO0vOTtZbHqulCu8PGlR4gJ7tWy+doWvPAMKrLIHwWde8 wK8gnkDxy6c9ZBoKlWPV6DltHAG4aC6BHNuQuNHqBFHY03tbaJn8zwuLz3ekXQNznojE 25/IaMNvpv4junlTnmmZ0TTA1J+tu7lj4MlaWY17j8PJhCnchdcYRVIjfTryk8zw3U/b pW2RKqU2KkavmrfXsG0sg6aH5eRAPlzPUdZoCXmxVHa6KbuhLZpA9bsRNvXj1YcWd39h IaGqmv8l5RbTtRMHQPIUR8wrzQ7kzC9albHhPUafURtLwkFsxUPmduhkUaXoi7KdeImJ fr5Q== X-Gm-Message-State: AOJu0YwYrKRa849FSqDMigPx3vEG013aFqpbfwX6McIQ8h8sovGNQQMQ A/wDReLqryiH2Z2c3968wD4uAg== X-Received: by 2002:a05:6a21:2713:b0:133:2974:d31a with SMTP id rm19-20020a056a21271300b001332974d31amr8482956pzb.17.1691464864453; Mon, 07 Aug 2023 20:21:04 -0700 (PDT) Received: from C02FG34NMD6R.bytedance.net ([2408:8656:30f8:e020::b]) by smtp.gmail.com with ESMTPSA id 13-20020a170902c10d00b001b896686c78sm7675800pli.66.2023.08.07.20.20.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Aug 2023 20:21:04 -0700 (PDT) From: Albert Huang To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: Albert Huang , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Pavel Begunkov , Yunsheng Lin , Kees Cook , Richard Gobert , "open list:NETWORKING DRIVERS" , open list , "open list:XDP (eXpress Data Path)" Subject: [RFC v3 Optimizing veth xsk performance 6/9] veth: add ndo_xsk_wakeup callback for veth Date: Tue, 8 Aug 2023 11:19:10 +0800 Message-Id: <20230808031913.46965-7-huangjie.albert@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230808031913.46965-1-huangjie.albert@bytedance.com> References: <20230808031913.46965-1-huangjie.albert@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773676499931360485 X-GMAIL-MSGID: 1773676499931360485 add ndo_xsk_wakeup callback for veth, this is used to wakeup napi tx. Signed-off-by: Albert Huang --- drivers/net/veth.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 28b891dd8dc9..ac78d6a87416 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -1805,6 +1805,44 @@ static void veth_set_rx_headroom(struct net_device *dev, int new_hr) rcu_read_unlock(); } +static void veth_xsk_remote_trigger_napi(void *info) +{ + struct veth_sq *sq = info; + + napi_schedule(&sq->xdp_napi); +} + +static int veth_xsk_wakeup(struct net_device *dev, u32 qid, u32 flag) +{ + struct veth_priv *priv; + struct veth_sq *sq; + u32 last_cpu, cur_cpu; + + if (!netif_running(dev)) + return -ENETDOWN; + + if (qid >= dev->real_num_rx_queues) + return -EINVAL; + + priv = netdev_priv(dev); + sq = &priv->sq[qid]; + + if (napi_if_scheduled_mark_missed(&sq->xdp_napi)) + return 0; + + last_cpu = sq->xsk.last_cpu; + cur_cpu = get_cpu(); + + /* raise a napi */ + if (last_cpu == cur_cpu) + napi_schedule(&sq->xdp_napi); + else + smp_call_function_single(last_cpu, veth_xsk_remote_trigger_napi, sq, true); + + put_cpu(); + return 0; +} + static int veth_xdp_set(struct net_device *dev, struct bpf_prog *prog, struct netlink_ext_ack *extack) { @@ -2019,6 +2057,7 @@ static const struct net_device_ops veth_netdev_ops = { .ndo_set_rx_headroom = veth_set_rx_headroom, .ndo_bpf = veth_xdp, .ndo_xdp_xmit = veth_ndo_xdp_xmit, + .ndo_xsk_wakeup = veth_xsk_wakeup, .ndo_get_peer_dev = veth_peer_dev, }; From patchwork Tue Aug 8 03:19:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?6buE5p2w?= X-Patchwork-Id: 132544 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c44e:0:b0:3f2:4152:657d with SMTP id w14csp2232777vqr; Tue, 8 Aug 2023 09:20:55 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFCoQwoP4q4lZZ+iUd27a+0Vso2reD7RPI//SoDgANpLErGaKs6uW+Oh6ZMz9ZocvPmk4Cf X-Received: by 2002:a05:6a20:a128:b0:12d:10b6:98f9 with SMTP id q40-20020a056a20a12800b0012d10b698f9mr16641018pzk.56.1691511655022; Tue, 08 Aug 2023 09:20:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691511655; cv=none; d=google.com; s=arc-20160816; b=CTpd25Fska+MOJKttWopqwSas8Dx1byOQkWmIbotrohh2AHf1uiHP47d29DlrcrJe8 phtffOiyG5gS0m/q1qjzeJ6xnaj38ryihR2w7DFyWMfEp4YUE9oQaWNISDC/t84u1HRm Cb897heqgU68LhTeIpPlVD4UAB5oqm1mjkieyF/v3AHpsPVxYaUMlgttwjI9TbiKN4MO ayianI9W0rwr6uhbfmswnFVSke57sSCCZJmW+SK0zx0Xrb/hW3treKmwGSAzTHAOXd3S CNB+c5xxEYd1dDd0h9Bll6AS3LWhoqztw/xAtl/rupzY+coSV0dSFmtghliQsh+nZUrG Oe+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vNrm/+UjyC6jP3361KhAbnju37DBvc3v6PBY22LsM1M=; fh=d52JIWYEkePcj18BLr16Py79ZIzTYL7SzTuYhl8wvHo=; b=WAhRdYyAIL6PxahCQqLNZnLO8X4W6jmynmiBrHootBoHqoTvdoxkNp2HAlllX4nk5y vqYDVRMkeyFcBkyEFzcaYfp2kLj9cNwdR57U9vGE2GSP3Hp0bmtqKHmS9HgMJRnm7ICS btwMZIzEoUcQvasQ9oeCdKJrg+9oCKeGFzQ/ZeE7NmDxNVTanIGiTTJlkzf1cv2HpYZC xMW3VNp329goi/gCc17tuQJssppCdlPD15/qHxMC74+EdSoPIp1n+nGtPSuonAHhptjO 1w7Up8cPj09m9xk3dQn2ZV97R01wW40thTrV60gPbGY39xvbhcomlFnonV7Tn83ej7BJ p8Dw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=khfL3k+d; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s196-20020a6377cd000000b005648b1d63aasi7327676pgc.553.2023.08.08.09.20.38; Tue, 08 Aug 2023 09:20:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=khfL3k+d; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229882AbjHHDWH (ORCPT + 99 others); Mon, 7 Aug 2023 23:22:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231204AbjHHDV2 (ORCPT ); Mon, 7 Aug 2023 23:21:28 -0400 Received: from mail-pl1-x636.google.com (mail-pl1-x636.google.com [IPv6:2607:f8b0:4864:20::636]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96B821BF5 for ; Mon, 7 Aug 2023 20:21:12 -0700 (PDT) Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-1bba04b9df3so45869755ad.0 for ; Mon, 07 Aug 2023 20:21:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1691464872; x=1692069672; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vNrm/+UjyC6jP3361KhAbnju37DBvc3v6PBY22LsM1M=; b=khfL3k+d/CGznwaO5bu9+cRGuRQ6fYxlH9gn4KFfrTBbf9tNYQ6PkS2QLz2ndsXr7u ZZFyAYrq4pkWZ2egL0T8+F1oRE2BdqG2AchQKzk5rpmHiJn7brKTCQ8ATfQT7sFIWFge +wJcsLzJca4mkXRRBfJXZfKyNTBPlv+eL9czPa03R4sohsTcUYnfsQxbhSXxU25FemRt iejLrUhpyeo72CMnzwLsp9SuIrXP3m4KaE4DHI1lDQX6YY5lFuWPo3Vo9fSW3SuO+lAg gxB6SIK1coQKqnReEqyCG/Z1l38wetEMXmr5Avzcf+u4KBASxQT7KEpJWvcaZOhS10vC fb8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691464872; x=1692069672; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vNrm/+UjyC6jP3361KhAbnju37DBvc3v6PBY22LsM1M=; b=IL9/TDscc15cJUqsiI109PGIQxKuybvEbLZxx4RtkUSaieN6R9mWBk7sP0IgmAmRl5 3i8pyr0wcwFCFoj/e/H3a9HXmjJ63PN0FQoBv3xTecfjI+SzEfzuoL1apXQAfPpTCDxq z01oT9dta/a4/80/2icWLgNVmujoDpG5UVhe4NrccWIzQMjPMILgKnbqVc1hb0i78fqk oz/IEYSi2ZgHJoKn/Trutlth6rrezWn0L5qq+LkBc5rQRc5Cvzy/hPR3w4CGs3CLBEDP YJHRIBdmSNv1reZU2EOoNlQ6PdFMTl8Hs3auY70RSw8MaFqo48uFE/EnZVaz1OU2Los7 e9Vg== X-Gm-Message-State: AOJu0YwxrylECoJ+mGIpZG5Fy0qriRSCJEKqRe1yliGlX38shn51NUyI OQ62vP7UEWWJmWbVjMWPrnKRvQ== X-Received: by 2002:a17:902:6909:b0:1b8:9461:6729 with SMTP id j9-20020a170902690900b001b894616729mr10585584plk.2.1691464871774; Mon, 07 Aug 2023 20:21:11 -0700 (PDT) Received: from C02FG34NMD6R.bytedance.net ([2408:8656:30f8:e020::b]) by smtp.gmail.com with ESMTPSA id 13-20020a170902c10d00b001b896686c78sm7675800pli.66.2023.08.07.20.21.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Aug 2023 20:21:11 -0700 (PDT) From: Albert Huang To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: Albert Huang , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Pavel Begunkov , Yunsheng Lin , Kees Cook , Richard Gobert , "open list:NETWORKING DRIVERS" , open list , "open list:XDP (eXpress Data Path)" Subject: [RFC v3 Optimizing veth xsk performance 7/9] sk_buff: add destructor_arg_xsk_pool for zero copy Date: Tue, 8 Aug 2023 11:19:11 +0800 Message-Id: <20230808031913.46965-8-huangjie.albert@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230808031913.46965-1-huangjie.albert@bytedance.com> References: <20230808031913.46965-1-huangjie.albert@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773678525539099644 X-GMAIL-MSGID: 1773678525539099644 this member is add for dummy dev to suppot zero copy Signed-off-by: Albert Huang --- include/linux/skbuff.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 16a49ba534e4..db999056022e 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -592,6 +592,8 @@ struct skb_shared_info { /* Intermediate layers must ensure that destructor_arg * remains valid until skb destructor */ void * destructor_arg; + /* just for dummy device xsk zero copy */ + void *destructor_arg_xsk_pool; /* must be last field, see pskb_expand_head() */ skb_frag_t frags[MAX_SKB_FRAGS]; From patchwork Tue Aug 8 03:19:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?6buE5p2w?= X-Patchwork-Id: 132532 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c44e:0:b0:3f2:4152:657d with SMTP id w14csp2208683vqr; Tue, 8 Aug 2023 08:42:54 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFcJ1oSy4ebGB0fVz4nnaSBFtRdhNHtclGuKm1NRkku9Xy8AM3N52QhsrAB4btTd+GHGSJ9 X-Received: by 2002:a17:907:2cea:b0:993:f127:2391 with SMTP id hz10-20020a1709072cea00b00993f1272391mr226987ejc.32.1691509374406; Tue, 08 Aug 2023 08:42:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691509374; cv=none; d=google.com; s=arc-20160816; b=w6ZO2n03ZnDTRQvvqUFEVEnjpqCa4L1jkgqfAMgXSirsOGy2ar6GEjdtrMazmEC/lQ OV2KYlQQLZOobqoE+crHxhOmOLwpyNKSe0pHGTWEVhNYxC+f8owfJFzzEAMPYJYW7HWR zL2Spvhd0T7EmfFjcSyxgpH8/YZ0htzK2T9uxV0sgFqiltRqfeasn8c3Kqbz25ph9QcJ lUnfGa5l88+G+z1CbhYGwXhQcC5syT1eBL1TOV3b/Qs6nF7L5u8oglBfNm/x2rElfMv4 1SK0J7BU/nSttD4QDGH7Y8tySbMXUauJuRAQcZ18sqwtG27/js/lyg+mlNpwdBwHzC1T pVQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=VNxERTokHn2QOzCO7nu1QH5j2GU8N7cGLZ9USDe1a7I=; fh=d52JIWYEkePcj18BLr16Py79ZIzTYL7SzTuYhl8wvHo=; b=Q0Ce1CGo35f73tE1M3RVDAIrejMoXTtj03ZF49QH5sx2VkkZp1oPTqMMatnq+pPLHN G/8flu7XxU0qvvVHb0gd4fmUmvu+CYHKsTc5wgKAU8jguAOygA7yYyRa48yszUrS48id jkqBTD8FxpEwcdHsLxTBjlS+mczx45qVN5ViBT2H6fzaqc3xr+ORPTK2QmBI2DZu0Jf5 HRlGyTFkykIigaQHAmD016bOX89xU6f6drlgBtaBWEGanH9KYekHTEoUjtLNgSfWv0RO IurF3aVlqgEG4aSshKIVUUQnIjGQVcZCHyx3bOjl4vzBkwVb4KQzSBK7hrcEaF7acXMr WxmQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=C4mBcxo9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x2-20020a1709060a4200b0099bc55749c1si1383ejf.349.2023.08.08.08.42.30; Tue, 08 Aug 2023 08:42:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=C4mBcxo9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231397AbjHHDWk (ORCPT + 99 others); Mon, 7 Aug 2023 23:22:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46664 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231394AbjHHDVu (ORCPT ); Mon, 7 Aug 2023 23:21:50 -0400 Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96E1D1FFC for ; Mon, 7 Aug 2023 20:21:19 -0700 (PDT) Received: by mail-pg1-x536.google.com with SMTP id 41be03b00d2f7-563e21a6011so3822560a12.0 for ; Mon, 07 Aug 2023 20:21:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1691464879; x=1692069679; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VNxERTokHn2QOzCO7nu1QH5j2GU8N7cGLZ9USDe1a7I=; b=C4mBcxo9HLIzgPmkZwvVwMe2Y8+oOUbmzCrCDJqkOO4K6EXNq7sy0sXwURk4dnYWN/ nqw603ksEycjKPz+q7Gb7cwNFSYIK+vVrjiyNyqw3L/+v1thar9dzpcpeGKEFR3o4wLH S6xLfrbCC72IxnIkr5RURV71aSuBQadDIjBi3PHym4cpsgU/YZrISBPUzd4jceT+L/0a 307O/xTRmtCcwfALjTjmrGApGmrIdSdjks+lyuqttA5hf1liWdxzdc0Cqo6COCriBkaS QPLzDbZufpr2FBVXchXRpN1W/43dNgJJikedWzqrHrtBXTwNFXaAoiDaBXhaBmaMmOlA CRuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691464879; x=1692069679; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VNxERTokHn2QOzCO7nu1QH5j2GU8N7cGLZ9USDe1a7I=; b=GftnFg7Kn/I7llgRSeCKiDaji6HG56QA0JGB7pil67rOun4dMc13GEQ7xsZ8k1kRfD C4OpOAcFQIcFWDa1tCpJ4EYu+z4/rHbo263LYq6hFInKS9DsgXbMovpn1K/z5NUZHVUd xJc7H19jj+cgLQNrzjoP4EYGJc/4Q0PGrNN18Egt6VoeeFLKVNyBPBR+NPpfzVOAGYJi 7FJDLGV3O3bCl5JS4PnzJV/DTLsLfwwvPsroZLkiTXFzuf2+6HtRMCsV6htFtTLpAOwx 50+bpm3t9ceLgGfuCCm1Z2e71MKiV7L7EZl0UNUZhAOIKd81UoobLPAEfutwd1qd3DlT /5hQ== X-Gm-Message-State: AOJu0YxMFgJN9fNkUNTOeORbxziJEkQOqtxhyTxx6Jw12YSgKOpx6HKR NNtdBZ9/t+/36WoxwBuE0n8TQw== X-Received: by 2002:a05:6a20:3206:b0:138:2fb8:6a14 with SMTP id hl6-20020a056a20320600b001382fb86a14mr11332615pzc.3.1691464878793; Mon, 07 Aug 2023 20:21:18 -0700 (PDT) Received: from C02FG34NMD6R.bytedance.net ([2408:8656:30f8:e020::b]) by smtp.gmail.com with ESMTPSA id 13-20020a170902c10d00b001b896686c78sm7675800pli.66.2023.08.07.20.21.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Aug 2023 20:21:18 -0700 (PDT) From: Albert Huang To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: Albert Huang , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Pavel Begunkov , Yunsheng Lin , Kees Cook , Richard Gobert , "open list:NETWORKING DRIVERS" , open list , "open list:XDP (eXpress Data Path)" Subject: [RFC v3 Optimizing veth xsk performance 8/9] veth: af_xdp tx batch support for ipv4 udp Date: Tue, 8 Aug 2023 11:19:12 +0800 Message-Id: <20230808031913.46965-9-huangjie.albert@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230808031913.46965-1-huangjie.albert@bytedance.com> References: <20230808031913.46965-1-huangjie.albert@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773676133442400328 X-GMAIL-MSGID: 1773676133442400328 A typical topology is shown below: veth<--------veth-peer 1 | |2 | bridge<------->eth0(such as mlnx5 NIC) If you use af_xdp to send packets from veth to a physical NIC, it needs to go through some software paths, so we can refer to the implementation of kernel GSO. When af_xdp sends packets out from veth, consider aggregating packets and send a large packet from the veth virtual NIC to the physical NIC. performance:(test weth libxdp lib) AF_XDP without batch : 480 Kpps (with ksoftirqd 100% cpu) AF_XDP with batch : 1.5 Mpps (with ksoftirqd 15% cpu) With af_xdp batch, the libxdp user-space program reaches a bottleneck. Therefore, the softirq did not reach the limit. Signed-off-by: Albert Huang --- drivers/net/veth.c | 408 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 387 insertions(+), 21 deletions(-) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index ac78d6a87416..70489d017b51 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -29,6 +29,7 @@ #include #include #include +#include #define DRV_NAME "veth" #define DRV_VERSION "1.0" @@ -103,6 +104,23 @@ struct veth_xdp_tx_bq { unsigned int count; }; +struct veth_batch_tuple { + __u8 protocol; + __be32 saddr; + __be32 daddr; + __be16 source; + __be16 dest; + __be16 batch_size; + __be16 batch_segs; + bool batch_enable; + bool batch_flush; +}; + +struct veth_seg_info { + u32 segs; + u64 desc[] ____cacheline_aligned_in_smp; +}; + /* * ethtool interface */ @@ -1078,11 +1096,340 @@ static struct sk_buff *veth_build_skb(void *head, int headroom, int len, return skb; } +static void veth_xsk_destruct_skb(struct sk_buff *skb) +{ + struct skb_shared_info *si = skb_shinfo(skb); + struct xsk_buff_pool *pool = (struct xsk_buff_pool *)si->destructor_arg_xsk_pool; + struct veth_seg_info *seg_info = (struct veth_seg_info *)si->destructor_arg; + unsigned long flags; + u32 index = 0; + u64 addr; + + /* release cq */ + spin_lock_irqsave(&pool->cq_lock, flags); + for (index = 0; index < seg_info->segs; index++) { + addr = (u64)(long)seg_info->desc[index]; + xsk_tx_completed_addr(pool, addr); + } + spin_unlock_irqrestore(&pool->cq_lock, flags); + + kfree(seg_info); + si->destructor_arg = NULL; + si->destructor_arg_xsk_pool = NULL; +} + +static struct sk_buff *veth_build_gso_head_skb(struct net_device *dev, + char *buff, u32 tot_len, + u32 headroom, u32 iph_len, + u32 th_len) +{ + struct sk_buff *skb = NULL; + int err = 0; + + skb = alloc_skb(tot_len, GFP_KERNEL); + if (unlikely(!skb)) + return NULL; + + /* header room contains the eth header */ + skb_reserve(skb, headroom - ETH_HLEN); + skb_put(skb, ETH_HLEN + iph_len + th_len); + skb_shinfo(skb)->gso_segs = 0; + + err = skb_store_bits(skb, 0, buff, ETH_HLEN + iph_len + th_len); + if (unlikely(err)) { + kfree_skb(skb); + return NULL; + } + + skb->protocol = eth_type_trans(skb, dev); + skb->network_header = skb->mac_header + ETH_HLEN; + skb->transport_header = skb->network_header + iph_len; + skb->ip_summed = CHECKSUM_PARTIAL; + + return skb; +} + +/* only ipv4 udp match + * to do: tcp and ipv6 + */ +static inline bool veth_segment_match(struct veth_batch_tuple *tuple, + struct iphdr *iph, struct udphdr *udph) +{ + if (tuple->protocol == iph->protocol && + tuple->saddr == iph->saddr && + tuple->daddr == iph->daddr && + tuple->source == udph->source && + tuple->dest == udph->dest && + tuple->batch_size == ntohs(udph->len)) { + tuple->batch_flush = false; + return true; + } + + tuple->batch_flush = true; + return false; +} + +static inline void veth_tuple_init(struct veth_batch_tuple *tuple, + struct iphdr *iph, struct udphdr *udph) +{ + tuple->protocol = iph->protocol; + tuple->saddr = iph->saddr; + tuple->daddr = iph->daddr; + tuple->source = udph->source; + tuple->dest = udph->dest; + tuple->batch_flush = false; + tuple->batch_size = ntohs(udph->len); + tuple->batch_segs = 0; +} + +static inline bool veth_batch_ip_check_v4(struct iphdr *iph, u32 len) +{ + if (len <= (ETH_HLEN + sizeof(*iph))) + return false; + + if (iph->ihl < 5 || iph->version != 4 || len < (iph->ihl * 4 + ETH_HLEN)) + return false; + + return true; +} + +static struct sk_buff *veth_build_skb_batch_udp(struct net_device *dev, + struct xsk_buff_pool *pool, + struct xdp_desc *desc, + struct veth_batch_tuple *tuple, + struct sk_buff *prev_skb) +{ + u32 hr, len, ts, index, iph_len, th_len, data_offset, data_len, tot_len; + struct veth_seg_info *seg_info; + void *buffer; + struct udphdr *udph; + struct iphdr *iph; + struct sk_buff *skb; + struct page *page; + u32 seg_len = 0; + int hh_len = 0; + u64 addr; + + addr = desc->addr; + len = desc->len; + + /* l2 reserved len */ + hh_len = LL_RESERVED_SPACE(dev); + hr = max(NET_SKB_PAD, L1_CACHE_ALIGN(hh_len)); + + /* data points to eth header */ + buffer = (unsigned char *)xsk_buff_raw_get_data(pool, addr); + + iph = (struct iphdr *)(buffer + ETH_HLEN); + iph_len = iph->ihl * 4; + + udph = (struct udphdr *)(buffer + ETH_HLEN + iph_len); + th_len = sizeof(struct udphdr); + + if (tuple->batch_flush) + veth_tuple_init(tuple, iph, udph); + + ts = pool->unaligned ? len : pool->chunk_size; + + data_offset = offset_in_page(buffer) + ETH_HLEN + iph_len + th_len; + data_len = len - (ETH_HLEN + iph_len + th_len); + + /* head is null or this is a new 5 tuple */ + if (!prev_skb || !veth_segment_match(tuple, iph, udph)) { + tot_len = hr + iph_len + th_len; + skb = veth_build_gso_head_skb(dev, buffer, tot_len, hr, iph_len, th_len); + if (!skb) { + /* to do: handle here for skb */ + return NULL; + } + + /* store information for gso */ + seg_len = struct_size(seg_info, desc, MAX_SKB_FRAGS); + seg_info = kmalloc(seg_len, GFP_KERNEL); + if (!seg_info) { + /* to do */ + kfree_skb(skb); + return NULL; + } + } else { + skb = prev_skb; + skb_shinfo(skb)->gso_type = SKB_GSO_UDP_L4 | SKB_GSO_PARTIAL; + skb_shinfo(skb)->gso_size = data_len; + skb->ip_summed = CHECKSUM_PARTIAL; + + /* max segment is MAX_SKB_FRAGS */ + if (skb_shinfo(skb)->gso_segs >= MAX_SKB_FRAGS - 1) + tuple->batch_flush = true; + + seg_info = (struct veth_seg_info *)skb_shinfo(skb)->destructor_arg; + } + + /* offset in umem pool buffer */ + addr = buffer - pool->addrs; + + /* get the page of the desc */ + page = pool->umem->pgs[addr >> PAGE_SHIFT]; + + /* in order to avoid to get freed by kfree_skb */ + get_page(page); + + /* desc.data can not hold in two */ + skb_fill_page_desc(skb, skb_shinfo(skb)->gso_segs, page, data_offset, data_len); + + skb->len += data_len; + skb->data_len += data_len; + skb->truesize += ts; + skb->dev = dev; + + /* later we will support gso for this */ + index = skb_shinfo(skb)->gso_segs; + seg_info->desc[index] = desc->addr; + seg_info->segs = ++index; + skb_shinfo(skb)->gso_segs++; + + skb_shinfo(skb)->destructor_arg = (void *)(long)seg_info; + skb_shinfo(skb)->destructor_arg_xsk_pool = (void *)(long)pool; + skb->destructor = veth_xsk_destruct_skb; + + /* to do: + * add skb to sock. may be there is no need to do for this + * and this might be multiple xsk sockets involved, so it's + * difficult to determine which socket is sending the data. + * refcount_add(ts, &xs->sk.sk_wmem_alloc); + */ + return skb; +} + +static inline struct sk_buff *veth_build_skb_def(struct net_device *dev, + struct xsk_buff_pool *pool, struct xdp_desc *desc) +{ + struct sk_buff *skb = NULL; + struct page *page; + void *buffer; + void *vaddr; + + page = dev_alloc_page(); + if (!page) + return NULL; + + buffer = (unsigned char *)xsk_buff_raw_get_data(pool, desc->addr); + + vaddr = page_to_virt(page); + memcpy(vaddr + pool->headroom, buffer, desc->len); + skb = veth_build_skb(vaddr, pool->headroom, desc->len, PAGE_SIZE); + if (!skb) { + put_page(page); + return NULL; + } + + skb->protocol = eth_type_trans(skb, dev); + + return skb; +} + +/* To call the following function, the following conditions must be met: + * 1.The data packet must be a standard Ethernet data packet + * 2. Data packets support batch sending + */ +static inline struct sk_buff *veth_build_skb_batch_v4(struct net_device *dev, + struct xsk_buff_pool *pool, + struct xdp_desc *desc, + struct veth_batch_tuple *tuple, + struct sk_buff *prev_skb) +{ + struct iphdr *iph; + void *buffer; + u64 addr; + + addr = desc->addr; + buffer = (unsigned char *)xsk_buff_raw_get_data(pool, addr); + iph = (struct iphdr *)(buffer + ETH_HLEN); + if (!veth_batch_ip_check_v4(iph, desc->len)) + goto normal; + + switch (iph->protocol) { + case IPPROTO_UDP: + return veth_build_skb_batch_udp(dev, pool, desc, tuple, prev_skb); + default: + break; + } +normal: + tuple->batch_enable = false; + return veth_build_skb_def(dev, pool, desc); +} + +/* Zero copy needs to meet the following conditions： + * 1. The data content of tx desc must be within one page + * 2、the tx desc must support batch xmit, which seted by userspace + */ +static inline bool veth_batch_desc_check(void *buff, u32 len) +{ + u32 offset; + + offset = offset_in_page(buff); + if (PAGE_SIZE - offset < len) + return false; + + return true; +} + +/* here must be a ipv4 or ipv6 packet */ +static inline struct sk_buff *veth_build_skb_batch(struct net_device *dev, + struct xsk_buff_pool *pool, + struct xdp_desc *desc, + struct veth_batch_tuple *tuple, + struct sk_buff *prev_skb) +{ + const struct ethhdr *eth; + void *buffer; + + buffer = xsk_buff_raw_get_data(pool, desc->addr); + if (!veth_batch_desc_check(buffer, desc->len)) + goto normal; + + eth = (struct ethhdr *)buffer; + switch (ntohs(eth->h_proto)) { + case ETH_P_IP: + tuple->batch_enable = true; + return veth_build_skb_batch_v4(dev, pool, desc, tuple, prev_skb); + /* to do: not support yet, just build skb, no batch */ + case ETH_P_IPV6: + fallthrough; + default: + break; + } + +normal: + tuple->batch_flush = false; + tuple->batch_enable = false; + return veth_build_skb_def(dev, pool, desc); +} + +/* just support ipv4 udp batch + * to do: ipv4 tcp and ipv6 + */ +static inline void veth_skb_batch_checksum(struct sk_buff *skb) +{ + struct iphdr *iph = ip_hdr(skb); + struct udphdr *uh = udp_hdr(skb); + int ip_tot_len = skb->len; + int udp_len = skb->len - (skb->transport_header - skb->network_header); + + iph->tot_len = htons(ip_tot_len); + ip_send_check(iph); + uh->len = htons(udp_len); + uh->check = 0; + + udp4_hwcsum(skb, iph->saddr, iph->daddr); +} + static int veth_xsk_tx_xmit(struct veth_sq *sq, struct xsk_buff_pool *xsk_pool, int budget) { struct veth_priv *priv, *peer_priv; struct net_device *dev, *peer_dev; + struct veth_batch_tuple tuple; struct veth_stats stats = {}; + struct sk_buff *prev_skb = NULL; struct sk_buff *skb = NULL; struct veth_rq *peer_rq; struct xdp_desc desc; @@ -1093,24 +1440,23 @@ static int veth_xsk_tx_xmit(struct veth_sq *sq, struct xsk_buff_pool *xsk_pool, peer_dev = priv->peer; peer_priv = netdev_priv(peer_dev); - /* todo: queue index must set before this */ + /* queue_index set in napi enable + * to do:may be we should select rq by 5-tuple or hash + */ peer_rq = &peer_priv->rq[sq->queue_index]; + memset(&tuple, 0, sizeof(tuple)); + /* set xsk wake up flag, to do: where to disable */ if (xsk_uses_need_wakeup(xsk_pool)) xsk_set_tx_need_wakeup(xsk_pool); while (budget-- > 0) { unsigned int truesize = 0; - struct page *page; - void *vaddr; - void *addr; if (!xsk_tx_peek_desc(xsk_pool, &desc)) break; - addr = xsk_buff_raw_get_data(xsk_pool, desc.addr); - /* can not hold all data in a page */ truesize = SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); truesize += desc.len + xsk_pool->headroom; @@ -1120,30 +1466,50 @@ static int veth_xsk_tx_xmit(struct veth_sq *sq, struct xsk_buff_pool *xsk_pool, break; } - page = dev_alloc_page(); - if (!page) { + skb = veth_build_skb_batch(peer_dev, xsk_pool, &desc, &tuple, prev_skb); + if (!skb) { + stats.rx_drops++; xsk_tx_completed_addr(xsk_pool, desc.addr); - stats.xdp_drops++; - break; + if (prev_skb != skb) { + napi_gro_receive(&peer_rq->xdp_napi, prev_skb); + prev_skb = NULL; + } + continue; } - vaddr = page_to_virt(page); - - memcpy(vaddr + xsk_pool->headroom, addr, desc.len); - xsk_tx_completed_addr(xsk_pool, desc.addr); - skb = veth_build_skb(vaddr, xsk_pool->headroom, desc.len, PAGE_SIZE); - if (!skb) { - put_page(page); - stats.xdp_drops++; - break; + if (!tuple.batch_enable) { + xsk_tx_completed_addr(xsk_pool, desc.addr); + /* flush the prev skb first to avoid out of order */ + if (prev_skb != skb && prev_skb) { + veth_skb_batch_checksum(prev_skb); + napi_gro_receive(&peer_rq->xdp_napi, prev_skb); + prev_skb = NULL; + } + napi_gro_receive(&peer_rq->xdp_napi, skb); + skb = NULL; + } else { + if (prev_skb && tuple.batch_flush) { + veth_skb_batch_checksum(prev_skb); + napi_gro_receive(&peer_rq->xdp_napi, prev_skb); + if (prev_skb == skb) + prev_skb = skb = NULL; + else + prev_skb = skb; + } else { + prev_skb = skb; + } } - skb->protocol = eth_type_trans(skb, peer_dev); - napi_gro_receive(&peer_rq->xdp_napi, skb); stats.xdp_bytes += desc.len; done++; } + /* means there is a skb need to send to peer_rq (batch)*/ + if (skb) { + veth_skb_batch_checksum(skb); + napi_gro_receive(&peer_rq->xdp_napi, skb); + } + /* release, move consumer，and wakeup the producer */ if (done) { napi_schedule(&peer_rq->xdp_napi); From patchwork Tue Aug 8 03:19:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?6buE5p2w?= X-Patchwork-Id: 132634 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c44e:0:b0:3f2:4152:657d with SMTP id w14csp2292412vqr; Tue, 8 Aug 2023 10:51:43 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGGCzCxfZjfMq9mlK/BH6uiJJ7ANlY0km/eYfyuk9y1CaToDRq5L8ItumAUpFTpQR76DnlZ X-Received: by 2002:a17:903:22cc:b0:1bc:6707:f451 with SMTP id y12-20020a17090322cc00b001bc6707f451mr434990plg.42.1691517102669; Tue, 08 Aug 2023 10:51:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691517102; cv=none; d=google.com; s=arc-20160816; b=fyYqb6PK8KrXqzbMNWQs3xxowN8K55GfBWSUSag+32cd5BqBiU5SccJ25n1B9vHXyU H+hfO+/fX2USzFqdw/fzGElMcr1/aRO84doXS7aMGzty9s2p2pe/GuRK6QikQ2sB/xWO WaVGbK2+K8guqePE0g1CU7uQ2NI4RjMRQKCt83XEim9hNR48/JbqGxbKHdTLYsodBKae rbf+uCzurnQdXJKPGLqJkGu2lg8kRCIMPrs77axkkNtGC4mS9H1iQC8ISCukQjsPCa8H zFSfpNFADYqxsjeIMTtK7N8D05q7B+P4h2ZYGjwXgkcARL4X1LPW9h4xs8VSvx1TlEKP VRVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=DZBR/L7n+c0K4+2jvj2qF9IKHrGGr2P5oMHLWIibsdA=; fh=d52JIWYEkePcj18BLr16Py79ZIzTYL7SzTuYhl8wvHo=; b=XictnDV2IXqRH5v5meSy4FvnAylknVoUjasSui+y1oDQEWz9yhYp3EqcZjPQHgzjeE o/gx8mGUrmu9+GT2hgV2W9RUKulE/yhkN/tSPZhJnJRsUTF4qZtV6C0Os7YBp4kwUQ1e e/k/qPVR5Vm62BdltgU9EfH5CmHZ4Klfbm/Pw3NGgrqff7uoDs2lt+tWAJS6sw9yWVvZ Ywh27a1weF0e3uO8KCDAym4ovRDrfryh4CqukWXzxMMuNDwSlbt+De3g2iHQY6KuJlMv otUzf3dC2eUcrdEiucfrznkTCJrNViEzYU09XTRe+7ipLtCOTp/VfAo6tHY0nGIwQp1G fZbQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="P/V9ljiS"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id kx11-20020a170902f94b00b001b9e82a6beesi7569422plb.548.2023.08.08.10.51.29; Tue, 08 Aug 2023 10:51:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="P/V9ljiS"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231460AbjHHDXc (ORCPT + 99 others); Mon, 7 Aug 2023 23:23:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45972 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231350AbjHHDWr (ORCPT ); Mon, 7 Aug 2023 23:22:47 -0400 Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4FDDD213F for ; Mon, 7 Aug 2023 20:21:28 -0700 (PDT) Received: by mail-pf1-x431.google.com with SMTP id d2e1a72fcca58-68783b2e40bso3598456b3a.3 for ; Mon, 07 Aug 2023 20:21:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1691464888; x=1692069688; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DZBR/L7n+c0K4+2jvj2qF9IKHrGGr2P5oMHLWIibsdA=; b=P/V9ljiS+RZV06sSAc2COBIOCp251sZvy3FmMZoG/OkNXoyMAnbNe24EKZNrbifKu0 TMw9/bIBHlaHnzbtUtWzPHTqYsy2sFGtkRs+xt8vbZzsiS0x7Rc9mKFj2RktzhmElS+2 SIeHhQhwHgkVPUcIXzdkakUQs/PkxTbSAUA2zU2DOYbE+GTmiLeGsM6DB34oY7IioJoX u8Jqfm2ETyE34gqRs7naQ2o/4eL8Sh9ZBIOH/9/XmQWQmnyKbBt5QKI6jrWVjEbTDSob NWueoCvGAVQgsnFEvyZJztfQr+58ypf9d4mbmvIy2JWUjfWc7DNBUg9/MfRu1XPR5mvw xldg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691464888; x=1692069688; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DZBR/L7n+c0K4+2jvj2qF9IKHrGGr2P5oMHLWIibsdA=; b=OCa4MA9DgRDH/apv7O2ve7JZrjxfhZnJroy5CaxhAq8CyAanvDiCtp/C9rBtWpIOHg 77InA7nLPCOsfqL9eJQBii94VN4a5Y/Fbv8Uy9BlffLiuSzk7X0FANNTA8fJPek/7+jm 3N5BKBxAe4ymqHoEwDa1HCngoMT4qdHMdvUrENxQK6Mk8xfofRPejqPwoIIjAKHq4YcG tvdJaq6YaOJJxShDIf2iNrWKtgshbxmbOxvLiL98q7bawiA25eA3MBZnh6x0oIerJKdB kyYMzzaIf2Dv4Iy6po48r2XeOvpRymA2aaIqDyy0YLYP8Mi1Nz9/fWpn/EAuBi8a+dgb WOvQ== X-Gm-Message-State: AOJu0YyZx9ZxiWDodKmb0u2xQgkBtBfZj+O00fIYRlEzt8b9dpuFPwxx 37vZVS1L6SEe0qr4aQsi3UG3/Q== X-Received: by 2002:a05:6a20:9193:b0:140:d536:d424 with SMTP id v19-20020a056a20919300b00140d536d424mr6753584pzd.53.1691464887689; Mon, 07 Aug 2023 20:21:27 -0700 (PDT) Received: from C02FG34NMD6R.bytedance.net ([2408:8656:30f8:e020::b]) by smtp.gmail.com with ESMTPSA id 13-20020a170902c10d00b001b896686c78sm7675800pli.66.2023.08.07.20.21.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Aug 2023 20:21:27 -0700 (PDT) From: Albert Huang To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: Albert Huang , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Pavel Begunkov , Yunsheng Lin , Kees Cook , Richard Gobert , "open list:NETWORKING DRIVERS" , open list , "open list:XDP (eXpress Data Path)" Subject: [RFC v3 Optimizing veth xsk performance 9/9] veth: add support for AF_XDP tx need_wakup feature Date: Tue, 8 Aug 2023 11:19:13 +0800 Message-Id: <20230808031913.46965-10-huangjie.albert@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230808031913.46965-1-huangjie.albert@bytedance.com> References: <20230808031913.46965-1-huangjie.albert@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773684237372171787 X-GMAIL-MSGID: 1773684237372171787 this patch only support for tx need_wakup feature. Signed-off-by: Albert Huang --- drivers/net/veth.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 70489d017b51..7c60c64ef10b 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -1447,9 +1447,9 @@ static int veth_xsk_tx_xmit(struct veth_sq *sq, struct xsk_buff_pool *xsk_pool, memset(&tuple, 0, sizeof(tuple)); - /* set xsk wake up flag, to do: where to disable */ + /* clear xsk wake up flag */ if (xsk_uses_need_wakeup(xsk_pool)) - xsk_set_tx_need_wakeup(xsk_pool); + xsk_clear_tx_need_wakeup(xsk_pool); while (budget-- > 0) { unsigned int truesize = 0; @@ -1539,12 +1539,15 @@ static int veth_poll_tx(struct napi_struct *napi, int budget) if (pool) done = veth_xsk_tx_xmit(sq, pool, budget); - rcu_read_unlock(); - if (done < budget) { + /* set xsk wake up flag */ + if (xsk_uses_need_wakeup(pool)) + xsk_set_tx_need_wakeup(pool); + /* if done < budget, the tx ring is no buffer */ napi_complete_done(napi, done); } + rcu_read_unlock(); return done; }