From patchwork Wed Oct 19 08:25:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Greg KH X-Patchwork-Id: 4728 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp214092wrs; Wed, 19 Oct 2022 02:11:52 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4Tw2oJC6xTVHrg5JCb97B3ntX+iUwWJcQRgxSqAcDhJvcVFlBXBcRvJ0SWgRn8scYQIzVk X-Received: by 2002:a17:907:2c74:b0:78d:b3ae:8408 with SMTP id ib20-20020a1709072c7400b0078db3ae8408mr5952780ejc.172.1666170712337; Wed, 19 Oct 2022 02:11:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666170712; cv=none; d=google.com; s=arc-20160816; b=y/ZtuvBWYB9c1ErolfNVnMtY0nknJTXqfcyRj2wzoRC4W+mR6JANrGJLTnADfUWYvF bioktmsVPCYQ7xzH4H6bw89gvGk4V8JlZqDvG/92wc/TZXL82ao15b2OEz5lgpgkLCwx +pz75UqBPT8j4CQWS4n61Vf9qLN7zXkF3lCnmON4wi2wn+pZbZBuwFKgYEZOoj5hcei8 BmBheyRj9bW/g8R/XlxD0W+nYKVcrA+cxpVrLh0Byb3U/2teQvGoDnvYFhyOYx9gObY7 A6UHPcpqjmYZaoP9x+o5FU+0A9H9NZ+qF9TgqJ6iQkaHVHYWbmRZU5BTvPGV4cAy1cjM whJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=asRXrfQL5XwC+1q71+Lx0St7tP98D83GjlPLoMtzDls=; b=qwabFtrogh5VE+9NP+ecYc1oCkEIQUKvuYy43Vpo6xi/xamy267Sigp0/hibD+P23v KFZyMn0F0ZU7MKzB3HucDUPTglbk9BEQ4UOTC6LnovHZMEwTvk3J6/x5Dn3tnYK5+1Ek LyMkgGdBJkWL6R6jUJty+aOhjK37sxygMKSKbR/7K9IrnsYPSUoKxwiUYu5Q/kpMXN5M JIh9f3AzagHpMkHEq7aTda8ogVN2KTZBh4ovhXfzsKzxS9B8stWViTMpcARUaxY0rI7x vxJBDevadxRrV0dCMzrUnN5FZxWzCyx3vsDyrI+nQTx4kNY4p00AjOZboFa/19AcoN2F J4Kw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linuxfoundation.org header.s=korg header.b=Bxcai6RG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hr42-20020a1709073faa00b0078de51e658esi14782962ejc.208.2022.10.19.02.11.27; Wed, 19 Oct 2022 02:11:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linuxfoundation.org header.s=korg header.b=Bxcai6RG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232657AbiJSJIV (ORCPT + 99 others); Wed, 19 Oct 2022 05:08:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232328AbiJSJFf (ORCPT ); Wed, 19 Oct 2022 05:05:35 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4B752BFE; Wed, 19 Oct 2022 01:59:19 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A122A61843; Wed, 19 Oct 2022 08:49:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B308EC433D6; Wed, 19 Oct 2022 08:49:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1666169371; bh=naQsdsUwmv7ZYGgsCmUoqoGlQ8cz+SG8+yoMlmhiWOo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Bxcai6RGK+KkgttwADRHY7cL8r0BC68I0G64G9canfewR63lh6EnzqmfPFH1riQrx aXDMEIkH0Dt3Jx/MQChJi4rbVfl6UUBKyBADJohWUFhjO3lXEVoXObEPJKv+fEXWLH 0q7jpdSST45hErgrR2GVm3dWGW1pEE8jWyaNlmKc= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Magnus Karlsson , Maciej Fijalkowski , Daniel Borkmann , Sasha Levin Subject: [PATCH 6.0 251/862] xsk: Fix backpressure mechanism on Tx Date: Wed, 19 Oct 2022 10:25:38 +0200 Message-Id: <20221019083301.126980073@linuxfoundation.org> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221019083249.951566199@linuxfoundation.org> References: <20221019083249.951566199@linuxfoundation.org> User-Agent: quilt/0.67 MIME-Version: 1.0 X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747106620932809110?= X-GMAIL-MSGID: =?utf-8?q?1747106620932809110?= From: Maciej Fijalkowski [ Upstream commit c00c4461689e15ac2cc3b9a595a54e4d8afd3d77 ] Commit d678cbd2f867 ("xsk: Fix handling of invalid descriptors in XSK TX batching API") fixed batch API usage against set of descriptors with invalid ones but introduced a problem when AF_XDP SW rings are smaller than HW ones. Mismatch of reported Tx'ed frames between HW generator and user space app was observed. It turned out that backpressure mechanism became a bottleneck when the amount of produced descriptors to CQ is lower than what we grabbed from XSK Tx ring. Say that 512 entries had been taken from XSK Tx ring but we had only 490 free entries in CQ. Then callsite (ZC driver) will produce only 490 entries onto HW Tx ring but 512 entries will be released from Tx ring and this is what will be seen by the user space. In order to fix this case, mix XSK Tx/CQ ring interractions by moving around internal functions and changing call order: * pull out xskq_prod_nb_free() from xskq_prod_reserve_addr_batch() up to xsk_tx_peek_release_desc_batch(); ** move xskq_cons_release_n() into xskq_cons_read_desc_batch() After doing so, algorithm can be described as follows: 1. lookup Tx entries 2. use value from 1. to reserve space in CQ (*) 3. Read from Tx ring as much descriptors as value from 2 3a. release descriptors from XSK Tx ring (**) 4. Finally produce addresses to CQ Fixes: d678cbd2f867 ("xsk: Fix handling of invalid descriptors in XSK TX batching API") Signed-off-by: Magnus Karlsson Signed-off-by: Maciej Fijalkowski Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20220830121705.8618-1-maciej.fijalkowski@intel.com Signed-off-by: Sasha Levin --- net/xdp/xsk.c | 22 +++++++++++----------- net/xdp/xsk_queue.h | 22 ++++++++++------------ 2 files changed, 21 insertions(+), 23 deletions(-) diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index 7bada4e8460b..9f0561b67c12 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -355,16 +355,15 @@ static u32 xsk_tx_peek_release_fallback(struct xsk_buff_pool *pool, u32 max_entr return nb_pkts; } -u32 xsk_tx_peek_release_desc_batch(struct xsk_buff_pool *pool, u32 max_entries) +u32 xsk_tx_peek_release_desc_batch(struct xsk_buff_pool *pool, u32 nb_pkts) { struct xdp_sock *xs; - u32 nb_pkts; rcu_read_lock(); if (!list_is_singular(&pool->xsk_tx_list)) { /* Fallback to the non-batched version */ rcu_read_unlock(); - return xsk_tx_peek_release_fallback(pool, max_entries); + return xsk_tx_peek_release_fallback(pool, nb_pkts); } xs = list_first_or_null_rcu(&pool->xsk_tx_list, struct xdp_sock, tx_list); @@ -373,12 +372,7 @@ u32 xsk_tx_peek_release_desc_batch(struct xsk_buff_pool *pool, u32 max_entries) goto out; } - max_entries = xskq_cons_nb_entries(xs->tx, max_entries); - nb_pkts = xskq_cons_read_desc_batch(xs->tx, pool, max_entries); - if (!nb_pkts) { - xs->tx->queue_empty_descs++; - goto out; - } + nb_pkts = xskq_cons_nb_entries(xs->tx, nb_pkts); /* This is the backpressure mechanism for the Tx path. Try to * reserve space in the completion queue for all packets, but @@ -386,12 +380,18 @@ u32 xsk_tx_peek_release_desc_batch(struct xsk_buff_pool *pool, u32 max_entries) * packets. This avoids having to implement any buffering in * the Tx path. */ - nb_pkts = xskq_prod_reserve_addr_batch(pool->cq, pool->tx_descs, nb_pkts); + nb_pkts = xskq_prod_nb_free(pool->cq, nb_pkts); if (!nb_pkts) goto out; - xskq_cons_release_n(xs->tx, max_entries); + nb_pkts = xskq_cons_read_desc_batch(xs->tx, pool, nb_pkts); + if (!nb_pkts) { + xs->tx->queue_empty_descs++; + goto out; + } + __xskq_cons_release(xs->tx); + xskq_prod_write_addr_batch(pool->cq, pool->tx_descs, nb_pkts); xs->sk.sk_write_space(&xs->sk); out: diff --git a/net/xdp/xsk_queue.h b/net/xdp/xsk_queue.h index fb20bf7207cf..c6fb6b763658 100644 --- a/net/xdp/xsk_queue.h +++ b/net/xdp/xsk_queue.h @@ -205,6 +205,11 @@ static inline bool xskq_cons_read_desc(struct xsk_queue *q, return false; } +static inline void xskq_cons_release_n(struct xsk_queue *q, u32 cnt) +{ + q->cached_cons += cnt; +} + static inline u32 xskq_cons_read_desc_batch(struct xsk_queue *q, struct xsk_buff_pool *pool, u32 max) { @@ -226,6 +231,8 @@ static inline u32 xskq_cons_read_desc_batch(struct xsk_queue *q, struct xsk_buff cached_cons++; } + /* Release valid plus any invalid entries */ + xskq_cons_release_n(q, cached_cons - q->cached_cons); return nb_entries; } @@ -291,11 +298,6 @@ static inline void xskq_cons_release(struct xsk_queue *q) q->cached_cons++; } -static inline void xskq_cons_release_n(struct xsk_queue *q, u32 cnt) -{ - q->cached_cons += cnt; -} - static inline u32 xskq_cons_present_entries(struct xsk_queue *q) { /* No barriers needed since data is not accessed */ @@ -350,21 +352,17 @@ static inline int xskq_prod_reserve_addr(struct xsk_queue *q, u64 addr) return 0; } -static inline u32 xskq_prod_reserve_addr_batch(struct xsk_queue *q, struct xdp_desc *descs, - u32 max) +static inline void xskq_prod_write_addr_batch(struct xsk_queue *q, struct xdp_desc *descs, + u32 nb_entries) { struct xdp_umem_ring *ring = (struct xdp_umem_ring *)q->ring; - u32 nb_entries, i, cached_prod; - - nb_entries = xskq_prod_nb_free(q, max); + u32 i, cached_prod; /* A, matches D */ cached_prod = q->cached_prod; for (i = 0; i < nb_entries; i++) ring->desc[cached_prod++ & q->ring_mask] = descs[i].addr; q->cached_prod = cached_prod; - - return nb_entries; } static inline int xskq_prod_reserve_desc(struct xsk_queue *q,