From patchwork Fri Oct 13 06:33:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?6buE5p2w?= X-Patchwork-Id: 152336 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp1697903vqb; Thu, 12 Oct 2023 23:34:33 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFXh1zFk8OPJm5IFV2ozxWQNZfwnNF8x7AQ2Uv1Ly+k0yt1f0Wgjhw31hrGWbdEGqCb6hPQ X-Received: by 2002:a05:6a00:4783:b0:6b3:c72d:b01 with SMTP id dh3-20020a056a00478300b006b3c72d0b01mr453211pfb.1.1697178872979; Thu, 12 Oct 2023 23:34:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697178872; cv=none; d=google.com; s=arc-20160816; b=fEwS0j7aEe2BOrHUiqnB1UvTX2xyiFpox6q3G07u7R3BXkX9Nh1AOqvRysAGzS+NDD rfAKHdIH96l5QeBiZ6Zmgx7RJoWLabyGvfTw+6NegjOerWraOR8U2hz8hWLNdy5rQVU+ 5gycWij7W6hpFPRpIkhCILD8NIJg49WybMXsKrQjCWsPSLqCcg/LFjTFZqH8eKCTHW4z CBFRH8zER//GBSZmzo0aquEgz4MPDBq2NJGUjCGTRMRHoBXtxZhzcZbeGslGKAasma4P E9OLqdcF97VkomuAhXE7Dsb0StQ45DQS6dc1CtGCf8ocoOgguw33vKzRzj4dsuj5QqaV TFbg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=Wrh5eOkTQdzzRaA0+eIlvA+uCT3v/wg9JTwG/iC1Neo=; fh=Eie8tHu2cRFnZDYB9E51vR95MKD8+2R/65UWVHq4AYM=; b=eJCHFHTis/+X4l36gc1VISkNeA9vv1/if4wmxlySCwv+ajPBXSRjp4qO8eh6ehQHU2 bBqoEmNWI1A+0JcjRzHO2qpuWTmhBXZYZz2bpFEY9NyQBF00uwSOyakQC5drUmvKFdWD JcrdL8QHmozH7A9TvZx6lAnFyRgMyjIjZqc0rEWZTwtODM8GT+jz49MEso/xqaCJWvCh cvP9uwy4h7Ht64omF6Z4R/H9PNUBhSd5Xl/efXgfFgi67Ayrxx4hPzEeKum6/Sh5qRxK /+rrFZzdru6QM/o3BdBDmprXLT4WeILoFH4gZsKigg/uxb/OJ9rl95BwQljABYr8xXyO DfVQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=K7NPfzoo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id eb23-20020a056a004c9700b00686e01db946si15972705pfb.64.2023.10.12.23.34.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Oct 2023 23:34:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=K7NPfzoo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 8FE2A832044B; Thu, 12 Oct 2023 23:34:30 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229761AbjJMGeV (ORCPT + 19 others); Fri, 13 Oct 2023 02:34:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50898 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229703AbjJMGeT (ORCPT ); Fri, 13 Oct 2023 02:34:19 -0400 Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E927BB for ; Thu, 12 Oct 2023 23:33:55 -0700 (PDT) Received: by mail-pl1-x62b.google.com with SMTP id d9443c01a7336-1c9daca2b85so13224715ad.1 for ; Thu, 12 Oct 2023 23:33:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1697178835; x=1697783635; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Wrh5eOkTQdzzRaA0+eIlvA+uCT3v/wg9JTwG/iC1Neo=; b=K7NPfzooF7r4is5QPvNz6ZF9N5qSbl4ZDyxdUxN69jQ/La+IZl/RcFIMLn7N7bhYkH O+06GoWJzVdkrh8m6gFdMlB18kSawVpHD0eM2WrvqnFLIatjzguJYibZrqDKMKj+bCdM ZHZkFHJWysufwdeiKDKrkIjJPUjBbDsFKrjMwPFKokZhVqmwRuk8DAN20NfZcbUth0Ti NdX+pw2TV/aSD9cFAd/0eDlQVrSLaSQgsuYMjSVVkHwQpaq6zQWggISJyy2bzsBDG/4s mmHHWMg5NNsyvHL/PEa++0a/Bf9jxsFU0QSQs5MsxB14JF715rENMSjhidRaQGI1YucR HHpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697178835; x=1697783635; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Wrh5eOkTQdzzRaA0+eIlvA+uCT3v/wg9JTwG/iC1Neo=; b=BRQ0NBcqdNC2ax3El2p9VB0Ab3ixB5HUXTQgtEx1C3gWhZowQOwM/sspyxyWg9N1kO 58KnWSGWtmino/fTLWLE6IStX86EVNxlmA47C7NAnepVkMdLXuD0UlYqGJPPimEMVOF7 qL2+qGk8UCgcBhluLbw82eDaYg+E9xMpBtItQAU3g/zwb11nTKi4zO70eIEcmGjhcQbw pBMU++mJO9QOlQbVWv/1DtiSPc3wJwBf7YDH9NWf54xMVhSGIcdWEnMrbo+EbG6ckKrq v2hf8uwJDGWFxGp4WD5AROIsYlijtFnpYpS713kqRnBKfj/x5O56dz0nNB1qTNZNtvmY nQ5g== X-Gm-Message-State: AOJu0YwljWSLZMGpkyrD1gL5xZRLhlOkOf8Qjt+/ewsD5BSeE9waRX51 vpf/bolYLfwLWuSKPS2l+phuDMbvWbRVwCpESUA= X-Received: by 2002:a17:903:22d0:b0:1c7:22ae:4080 with SMTP id y16-20020a17090322d000b001c722ae4080mr35178736plg.0.1697178834760; Thu, 12 Oct 2023 23:33:54 -0700 (PDT) Received: from C02FG34NMD6R.bytedance.net ([203.208.189.11]) by smtp.gmail.com with ESMTPSA id x6-20020a170902ec8600b001c44c8d857esm3042304plg.120.2023.10.12.23.33.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Oct 2023 23:33:54 -0700 (PDT) From: Albert Huang To: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon Cc: Albert Huang , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH net-next] xsk: Avoid starving xsk at the end of the list Date: Fri, 13 Oct 2023 14:33:31 +0800 Message-Id: <20231013063332.38189-1-huangjie.albert@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Thu, 12 Oct 2023 23:34:30 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779621033967852179 X-GMAIL-MSGID: 1779621033967852179 In the previous implementation, when multiple xsk sockets were associated with a single xsk_buff_pool, a situation could arise where the xsk_tx_list maintained data at the front for one xsk socket while starving the xsk sockets at the back of the list. This could result in issues such as the inability to transmit packets, increased latency, and jitter. To address this problem, we introduced a new variable called tx_budget_cache, which limits each xsk to transmit a maximum of MAX_XSK_TX_BUDGET tx descriptors. This allocation ensures equitable opportunities for subsequent xsk sockets to send tx descriptors. The value of MAX_XSK_TX_BUDGET is temporarily set to 16. Signed-off-by: Albert Huang --- include/net/xdp_sock.h | 6 ++++++ net/xdp/xsk.c | 17 +++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/include/net/xdp_sock.h b/include/net/xdp_sock.h index 69b472604b86..f617ff54e38c 100644 --- a/include/net/xdp_sock.h +++ b/include/net/xdp_sock.h @@ -44,6 +44,7 @@ struct xsk_map { struct xdp_sock __rcu *xsk_map[]; }; +#define MAX_XSK_TX_BUDGET 16 struct xdp_sock { /* struct sock must be the first member of struct xdp_sock */ struct sock sk; @@ -63,6 +64,11 @@ struct xdp_sock { struct xsk_queue *tx ____cacheline_aligned_in_smp; struct list_head tx_list; + /* Record the actual number of times xsk has transmitted a tx + * descriptor, with a maximum limit not exceeding MAX_XSK_TX_BUDGET + */ + u32 tx_budget_cache; + /* Protects generic receive. */ spinlock_t rx_lock; diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index f5e96e0d6e01..bf964456e9b1 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -413,16 +413,25 @@ EXPORT_SYMBOL(xsk_tx_release); bool xsk_tx_peek_desc(struct xsk_buff_pool *pool, struct xdp_desc *desc) { + u32 xsk_full_count = 0; struct xdp_sock *xs; rcu_read_lock(); +again: list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list) { + if (xs->tx_budget_cache >= MAX_XSK_TX_BUDGET) { + xsk_full_count++; + continue; + } + if (!xskq_cons_peek_desc(xs->tx, desc, pool)) { if (xskq_has_descs(xs->tx)) xskq_cons_release(xs->tx); continue; } + xs->tx_budget_cache++; + /* This is the backpressure mechanism for the Tx path. * Reserve space in the completion queue and only proceed * if there is space in it. This avoids having to implement @@ -436,6 +445,13 @@ bool xsk_tx_peek_desc(struct xsk_buff_pool *pool, struct xdp_desc *desc) return true; } + if (unlikely(xsk_full_count > 0)) { + list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list) { + xs->tx_budget_cache = 0; + } + goto again; + } + out: rcu_read_unlock(); return false; @@ -1230,6 +1246,7 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len) xs->zc = xs->umem->zc; xs->sg = !!(xs->umem->flags & XDP_UMEM_SG_FLAG); xs->queue_id = qid; + xs->tx_budget_cache = 0; xp_add_xsk(xs->pool, xs); out_unlock: