From patchwork Mon Oct 23 12:57:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?b?6buE5p2w?= X-Patchwork-Id: 156863 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce89:0:b0:403:3b70:6f57 with SMTP id p9csp1275682vqx; Mon, 23 Oct 2023 05:58:31 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGcDg/OuOvQ6rzbV4dPzAXMIhBp5amwUxQjDTxvLbH7mPSxKpizWuIRkaHGuZpc+ARS9nJI X-Received: by 2002:a17:903:1109:b0:1c3:c687:478c with SMTP id n9-20020a170903110900b001c3c687478cmr10406520plh.8.1698065911089; Mon, 23 Oct 2023 05:58:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698065911; cv=none; d=google.com; s=arc-20160816; b=hlCGo6ob5Grw0/WUauziECZbo6do3DYhI/WA6RaRo4+iPv/+9wG6dTDECpAm83r9UE tG2aOBEttLgEHqvrA3YPqpIqIM5Tlphqm5bjdxJHZ6nPYH4znbIU/AflqTlZqjPgVgLw J/h3Q27qQwsLNkkgxcwJBNVitpC6/Q/sGqF7F9s+DYHOK0WpKo17Y5ck0n54tuKmCWng Hud/F6d0EIv3+FtfWP9G+UGXuYtax7lvt3KZMb14/I3eSaEDcSkJF3ldQrD/Hc6FA7H9 JeEzQReSmiLzo+t+j4u/c0ZfxItWPm9CVfe2pk2bd4l6MNkiMQzOGgUmaEK9KCGY9n5c A0VQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=yCAXAY3fgSa7Kb0MKqcgqBNDm1vRIcXYYGEzF/D8jjI=; fh=Eie8tHu2cRFnZDYB9E51vR95MKD8+2R/65UWVHq4AYM=; b=hXv8BZ9toXtoz/hMKxDFnZ58sQUvxLnOOOGAjhzF1154uoRqiIV3CM42l7VNxi+Q94 tdWKXhNp1wwt12PMd7YNZPDSvtoAuE2ipVQjpEP2OmzoLo4xyKGBduR8HBFjiYczmdcn E0JYbFGXEU0bA6/IPDszVeM9FfbgSEpWto+OREpWhCk/iBx/z2tvBjB0V9JqmWiWS3Eb zIqeEiYpeb973r8skU+WMvWTrpwvqUyB6krcsVfrXjIliBHjTMKdU2TG+R6b+KM2tf1y YNncR71afpouE/J4jk+yDw5mKi3kTHmx5UOA1EmyoSu2B+gs5nv8ZZ7aszlgq1a6kBo4 FM1g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=cn5eYQgo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id 6-20020a170902ee4600b001ca35cb0dbfsi6228545plo.613.2023.10.23.05.58.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Oct 2023 05:58:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=cn5eYQgo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 8129D805F5D2; Mon, 23 Oct 2023 05:58:28 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229939AbjJWM6M (ORCPT + 27 others); Mon, 23 Oct 2023 08:58:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43300 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229572AbjJWM6L (ORCPT ); Mon, 23 Oct 2023 08:58:11 -0400 Received: from mail-pf1-x434.google.com (mail-pf1-x434.google.com [IPv6:2607:f8b0:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5942310B for ; Mon, 23 Oct 2023 05:57:47 -0700 (PDT) Received: by mail-pf1-x434.google.com with SMTP id d2e1a72fcca58-6b5e6301a19so3099849b3a.0 for ; Mon, 23 Oct 2023 05:57:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1698065867; x=1698670667; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=yCAXAY3fgSa7Kb0MKqcgqBNDm1vRIcXYYGEzF/D8jjI=; b=cn5eYQgoZwYoVrloc6pgtjiTXhKXt/b4gJTG6YWyaYjCPEW3JeXcGuvbt084Jxeb8h M+fOAzw6Am89vWd0VishxaqzzqqQYCtSWHOWE9JwE6dow6D0jDl/cKXdIFyf1YY2Lz24 btW++4KfL4GQEAfrzygkr95BTn5LC3DRnxeUiJMPYkRUvjvC8wnMwMFSX1COCuK871Ph s5LV2/rBbujfR63tRGVEI945BKdm9foz4/4671cI1mkaYN2I34wUBTDUNVpl887O2Ss8 bEcVZnyVRr0ST7K84/FtplkgNkqceuDMlL/2mTwgfsCKzdmWBD2AZqxpEN9Ptpb2Y8dW nYGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698065867; x=1698670667; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=yCAXAY3fgSa7Kb0MKqcgqBNDm1vRIcXYYGEzF/D8jjI=; b=xM9usIiEYyWQQJw7Wq63KN4Biy0hiZ33hBrWMjbKANpzaFBj8rRQzgbfHBvCUOjdLT MJRNvjWEBauva+6nv07lbTNjTHhFFwF1aEdzOhnZesSaxg3GPQgu2wgbqhfukwNPv3Pz 3IiQd9ACjmvJt7QlLV/hWA37QSBDQAuqd42LAKvB9USZAhasy5/mTh90Xvm35Q3Hi7Q0 Vxqq5/spZcfS9V181SCvUXzshWVVtp+F2kx+LqHICJkWssjIWb1aqPqrtd3+YTMTbYhK GqxkFhwWSwgQNJNJ+QIvTC4VmpkMVHysm0EK+QVRYKcCeB7qCkV6ghevgsPdwC5x7Is/ J/uQ== X-Gm-Message-State: AOJu0Yyjinmb+p2IszHC7Rw4sc0FkW7pZxgJkCzoskGJ4/BdjPzASBDH jCUHdBYBue6xyeAifs6vMWGGcw== X-Received: by 2002:a05:6a20:6a06:b0:17e:87c1:7971 with SMTP id p6-20020a056a206a0600b0017e87c17971mr434493pzk.46.1698065866765; Mon, 23 Oct 2023 05:57:46 -0700 (PDT) Received: from C02FG34NMD6R.bytedance.net ([240e:6b1:c0:120::2:3]) by smtp.gmail.com with ESMTPSA id k28-20020aa79d1c000000b006bde2480806sm6056588pfp.47.2023.10.23.05.57.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Oct 2023 05:57:46 -0700 (PDT) From: Albert Huang To: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon Cc: Albert Huang , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , netdev@vger.kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 net-next] xsk: avoid starving the xsk further down the list Date: Mon, 23 Oct 2023 20:57:31 +0800 Message-Id: <20231023125732.82261-1-huangjie.albert@bytedance.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Mon, 23 Oct 2023 05:58:28 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780551160861687933 X-GMAIL-MSGID: 1780551160861687933 In the previous implementation, when multiple xsk sockets were associated with a single xsk_buff_pool, a situation could arise where the xsk_tx_list maintained data at the front for one xsk socket while starving the xsk sockets at the back of the list. This could result in issues such as the inability to transmit packets, increased latency, and jitter. To address this problem, we introduce a new variable called tx_budget_spent, which limits each xsk to transmit a maximum of MAX_PER_SOCKET_BUDGET tx descriptors. This allocation ensures equitable opportunities for subsequent xsk sockets to send tx descriptors. The value of MAX_PER_SOCKET_BUDGET is set to 32. Signed-off-by: Albert Huang Acked-by: Magnus Karlsson --- include/net/xdp_sock.h | 7 +++++++ net/xdp/xsk.c | 18 ++++++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/include/net/xdp_sock.h b/include/net/xdp_sock.h index 69b472604b86..de6819e50d54 100644 --- a/include/net/xdp_sock.h +++ b/include/net/xdp_sock.h @@ -63,6 +63,13 @@ struct xdp_sock { struct xsk_queue *tx ____cacheline_aligned_in_smp; struct list_head tx_list; + /* record the number of tx descriptors sent by this xsk and + * when it exceeds MAX_PER_SOCKET_BUDGET, an opportunity needs + * to be given to other xsks for sending tx descriptors, thereby + * preventing other XSKs from being starved. + */ + u32 tx_budget_spent; + /* Protects generic receive. */ spinlock_t rx_lock; diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index f5e96e0d6e01..65c32b85c326 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -33,6 +33,7 @@ #include "xsk.h" #define TX_BATCH_SIZE 32 +#define MAX_PER_SOCKET_BUDGET (TX_BATCH_SIZE) static DEFINE_PER_CPU(struct list_head, xskmap_flush_list); @@ -413,16 +414,25 @@ EXPORT_SYMBOL(xsk_tx_release); bool xsk_tx_peek_desc(struct xsk_buff_pool *pool, struct xdp_desc *desc) { + bool budget_exhausted = false; struct xdp_sock *xs; rcu_read_lock(); +again: list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list) { + if (xs->tx_budget_spent >= MAX_PER_SOCKET_BUDGET) { + budget_exhausted = true; + continue; + } + if (!xskq_cons_peek_desc(xs->tx, desc, pool)) { if (xskq_has_descs(xs->tx)) xskq_cons_release(xs->tx); continue; } + xs->tx_budget_spent++; + /* This is the backpressure mechanism for the Tx path. * Reserve space in the completion queue and only proceed * if there is space in it. This avoids having to implement @@ -436,6 +446,14 @@ bool xsk_tx_peek_desc(struct xsk_buff_pool *pool, struct xdp_desc *desc) return true; } + if (budget_exhausted) { + list_for_each_entry_rcu(xs, &pool->xsk_tx_list, tx_list) + xs->tx_budget_spent = 0; + + budget_exhausted = false; + goto again; + } + out: rcu_read_unlock(); return false;