From patchwork Mon Oct 23 11:52:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?6buE5p2w?= X-Patchwork-Id: 156849 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce89:0:b0:403:3b70:6f57 with SMTP id p9csp1238864vqx; Mon, 23 Oct 2023 04:54:05 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEUYY9suScOcelfS7JVz7iRDRrvRG8vkOHihRyykkYjjoHcBe9wjluky8oQmqPalrvj4Ggt X-Received: by 2002:a17:902:d483:b0:1c8:90bf:4234 with SMTP id c3-20020a170902d48300b001c890bf4234mr10953245plg.61.1698062044832; Mon, 23 Oct 2023 04:54:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698062044; cv=none; d=google.com; s=arc-20160816; b=yEs7Unwm6ipiKE32s48/lBpBZuHUKblUVCDld+zvrhKhuVpvBwrQxGF/0Mg4gKvcpO VlT5U/AY4DvzCapUZ9Yd2Bt0QEApB6QQG6wDwt5XGtObKh4lrxm6Id1SVr0M9j9lQSGb 2hzL61Lzp8mIjF5g7+NyZ0nRcw64ZEvAe1g0/k1K9Iya2q0XsIX8VE0sZrk6ezIv+rZ2 lluOx267xxl6Rfn0TjiTc4cxkVPVlIP6CwxmkATd6yVfYN9B/BgJnewto39deqyWHiN2 2u+J9i7A2JYqWApSLptUrF4+r/WevqsQNtUAzO64EPkyRnoYAjx6fMBNDqWGaqlcWKVe zTwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=5OpOO/ujxo7qV6VnFj/vbc7gRpnZ29/Hl135s51sMGA=; fh=Eie8tHu2cRFnZDYB9E51vR95MKD8+2R/65UWVHq4AYM=; b=JwFK3qk7+9Wy49t4d2citQoY3AXVcdOPKe58nW9dPQzvL0NRgn//tCA8i5lqn4lAbq RX/Ttngvuj5veCsh0uJh6+jqK+wjwNDAh9i/yVU2iNhxSvEARNaIgUgNQuC8+HrHnwKT W4p8gPzC+/P4Ue+CdNfCPRm15cULzbtlxUkwG3G8sOuvD/zbhvWJDz8yTIaQzkf4jRNg Om4UgtxLjZ9gqmwrCVpkjrLS+bASR82Or2LpaLmIuOdKZC99r5ahIuOldQELWc3oiyL3 lzFmcAsvVfqbBH6yNTnpbzMSyWW0qvmu9/2o/db3ObshKllEjX8wZl5rdGPK4LomCqj6 cYWw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=AgFW9mzk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id w13-20020a1709027b8d00b001bf0e15c0a9si6013982pll.269.2023.10.23.04.54.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Oct 2023 04:54:04 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=AgFW9mzk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id EEF6E80A28D9; Mon, 23 Oct 2023 04:53:56 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233272AbjJWLxj (ORCPT + 27 others); Mon, 23 Oct 2023 07:53:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59910 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231132AbjJWLxh (ORCPT ); Mon, 23 Oct 2023 07:53:37 -0400 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6077E4 for ; Mon, 23 Oct 2023 04:53:12 -0700 (PDT) Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-1c9d922c039so25935465ad.3 for ; Mon, 23 Oct 2023 04:53:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1698061992; x=1698666792; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=5OpOO/ujxo7qV6VnFj/vbc7gRpnZ29/Hl135s51sMGA=; b=AgFW9mzkVt5FKvoQ/pxyV5mJZJjs+7GrsRwDsUEtNH4aHIjgwT2TP+eCfbdxkrK3mW I1NzBQImL3xztARwgZMBzJi4aiIjN+17FAndZVaDTHMwPvdBcg2M1Nz+N9e9j0gRJZIY NMeS8D/HbCjpDJA2jV3e8uk2Q5ZF9TZaxQ5j6ImFga90Bz9fvxdweQ3Zg4OWd6vVwFJV kb3OumdXJS5wl+tF5D4aHeSJWyvbjc+uOLBJpj2Sx0+2rscTdrjZArVyzxtiF44Pp54g oV3A5H/WoS4eAgmKV3qPzkKhv3pEeMPECvSP2EEZo+d+Qi4u8wIb1n/UbFsBFtC+5WDX 4ibA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698061992; x=1698666792; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=5OpOO/ujxo7qV6VnFj/vbc7gRpnZ29/Hl135s51sMGA=; b=l6fSsWqkmpny8yY2T28xeUrrwHTTQlenfSHP39QbHFiNBzeeLQnkw2qu1awv4sqjyd 8mGKMfi6f4X7VXD/VttAoG7N8skquhl9qhJysL0ScKTsgipvUCpKd7ZHPH6oGPHIIcVl g3E3bURTkIY6jTke3tGuv9045EellfsjbA3FGl5Dy13Yvj3LxBTAozI6Uuo3Qfi3mdmN i7j2r0YGhWyDjTa7toZrbfdxXA8d/QAak1VmzhGjogZKlAxGZrm7Unr2uvJskWQp9QhC vR0PL30ILOhcsomJSMmFKFwIQ9eVOCBl62ftYUfGqYAG3dd6cp1LOecPhG53Ln9+CuS/ LSzA== X-Gm-Message-State: AOJu0YyZuYKbun9znMuhzJr+LKwZTbNZKc9UJLYzk/ftf7nX3I5oPS57 vE+/uFos5qwtU2NUrkvMndCGYA== X-Received: by 2002:a17:902:d506:b0:1ca:abe:a090 with SMTP id b6-20020a170902d50600b001ca0abea090mr9388853plg.62.1698061992110; Mon, 23 Oct 2023 04:53:12 -0700 (PDT) Received: from C02FG34NMD6R.bytedance.net ([240e:6b1:c0:120::2:3]) by smtp.gmail.com with ESMTPSA id l20-20020a170903005400b001bba7aab822sm5835142pla.5.2023.10.23.04.53.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Oct 2023 04:53:11 -0700 (PDT) From: Albert Huang To: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon Cc: Albert Huang , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 net-next] xsk: avoid starving the xsk further down the list Date: Mon, 23 Oct 2023 19:52:54 +0800 Message-Id: <20231023115255.76934-1-huangjie.albert@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Mon, 23 Oct 2023 04:53:57 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780547106771080572 X-GMAIL-MSGID: 1780547106771080572 In the previous implementation, when multiple xsk sockets were associated with a single xsk_buff_pool, a situation could arise where the xsk_tx_list maintained data at the front for one xsk socket while starving the xsk sockets at the back of the list. This could result in issues such as the inability to transmit packets, increased latency, and jitter. To address this problem, we introduced a new variable called tx_budget_cache, which limits each xsk to transmit a maximum of MAX_PER_SOCKET_BUDGET tx descriptors. This allocation ensures equitable opportunities for subsequent xsk sockets to send tx descriptors. The value of MAX_PER_SOCKET_BUDGET is temporarily set to TX_BATCH_SIZE(32). Signed-off-by: Albert Huang --- include/net/xdp_sock.h | 5 +++++ net/xdp/xsk.c | 19 +++++++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/include/net/xdp_sock.h b/include/net/xdp_sock.h index 69b472604b86..08cbdf6fca85 100644 --- a/include/net/xdp_sock.h +++ b/include/net/xdp_sock.h @@ -63,6 +63,11 @@ struct xdp_sock { struct xsk_queue *tx ____cacheline_aligned_in_smp; struct list_head tx_list; + /* Record the actual number of times xsk has transmitted a tx + * descriptor, with a maximum limit not exceeding MAX_PER_SOCKET_BUDGET + */ + u32 tx_budget_cache; + /* Protects generic receive. */ spinlock_t rx_lock; diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index f5e96e0d6e01..fd0d54b7c046 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -33,6 +33,7 @@ #include "xsk.h" #define TX_BATCH_SIZE 32 +#define MAX_PER_SOCKET_BUDGET (TX_BATCH_SIZE) static DEFINE_PER_CPU(struct list_head, xskmap_flush_list); @@ -413,16 +414,25 @@ EXPORT_SYMBOL(xsk_tx_release); bool xsk_tx_peek_desc(struct xsk_buff_pool *pool, struct xdp_desc *desc) { + bool xsk_cache_full = false; struct xdp_sock *xs; rcu_read_lock(); +again: list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list) { + if (xs->tx_budget_cache >= MAX_PER_SOCKET_BUDGET) { + xsk_cache_full = true; + continue; + } + if (!xskq_cons_peek_desc(xs->tx, desc, pool)) { if (xskq_has_descs(xs->tx)) xskq_cons_release(xs->tx); continue; } + xs->tx_budget_cache++; + /* This is the backpressure mechanism for the Tx path. * Reserve space in the completion queue and only proceed * if there is space in it. This avoids having to implement @@ -436,6 +446,14 @@ bool xsk_tx_peek_desc(struct xsk_buff_pool *pool, struct xdp_desc *desc) return true; } + if (xsk_cache_full) { + list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list) { + xs->tx_budget_cache = 0; + } + xsk_cache_full = false; + goto again; + } + out: rcu_read_unlock(); return false; @@ -1230,6 +1248,7 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len) xs->zc = xs->umem->zc; xs->sg = !!(xs->umem->flags & XDP_UMEM_SG_FLAG); xs->queue_id = qid; + xs->tx_budget_cache = 0; xp_add_xsk(xs->pool, xs); out_unlock: