From patchwork Tue Jan 31 05:25:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 50605 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2569649wrn; Mon, 30 Jan 2023 21:27:23 -0800 (PST) X-Google-Smtp-Source: AK7set8syOpnvGKUTwgn9qWJBivY8MAS7bGfZh7vE0qDN4icBK5qe8pR420RkhlxgVukfMufS6T3 X-Received: by 2002:a17:907:1c11:b0:889:5861:ad1e with SMTP id nc17-20020a1709071c1100b008895861ad1emr7259639ejc.72.1675142843397; Mon, 30 Jan 2023 21:27:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675142843; cv=none; d=google.com; s=arc-20160816; b=s59R8YPw7p7FdxMSS8jwuAtPtOmKHxESrfy6WogQTV2yIjCVZlhK5L8HnZc86HVk8v wodp4uyp567rvo4bS3Aoyv9B7KEcpzRYmCDJYr8Bd+VdhO7hJAMBosckBQ04/Cku8aEh Qxgxwoh1DuQ/M6sjHoz3oZN6iUTLgpFtkUdMWRAj254alrzubhg7u6O+Fe+h5HjB6vk8 ImUFrZTQoXClewk4Gb0umeJEtB1Y7iR5OoT2Ui4n+XAdyEsvn7uAp2FpSxJK99KZfjq8 yo/87zvg79s+91Xb5aNTLnxV9qCFmhpvX3vqmOPOF6hFvHmBjrNd4h7+KMbOovG5mEDM 8erA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=VmsFN1wOQa+/y6Ip6MIBv5QRIL6S+54KcKA38AyC6x8=; b=RSN90BP6/M0BKNMOytwV5cgnuNm4moRbBj21w1SpqgH/WowfRt2ZOsP4EqcCd9wK2H EW2Kb5BmW15MCMqaOD43KIzEWOzTAH2aHRnnShEkeVYyvh4CSAtOa05EbLzeO5OrOjuo fZddts4YMHURHEpBdzbLhVsINXmmRngz95F06qjrCFSp27XbtQY7Is3ZNcS4TOZgF4zD byajmTaXwsZo64EYEq5k9oO3X/dLKHY+E756XAnvEtQRThfOQEly35ixnqL+rS5vr5BU S6oE+EVWURHUssQQ1xEwA9HYxjUzxsDKTaZSRFiuPFAjbRNl/570IljEzJwYrgm/qmPM il9g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=QLAJxSsd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d4-20020a17090692c400b0087bdb28557esi13958647ejx.736.2023.01.30.21.26.59; Mon, 30 Jan 2023 21:27:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=QLAJxSsd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230511AbjAaF02 (ORCPT + 99 others); Tue, 31 Jan 2023 00:26:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35990 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230236AbjAaF0M (ORCPT ); Tue, 31 Jan 2023 00:26:12 -0500 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35D7936691 for ; Mon, 30 Jan 2023 21:26:09 -0800 (PST) Received: by mail-pj1-x102e.google.com with SMTP id e8-20020a17090a9a8800b0022c387f0f93so12824465pjp.3 for ; Mon, 30 Jan 2023 21:26:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=VmsFN1wOQa+/y6Ip6MIBv5QRIL6S+54KcKA38AyC6x8=; b=QLAJxSsdccmFrWi9QeEju0H9C+c4s3BVqgNKkzW9qCp7iMhJPMUpL+h6mH8rg1U6Ul Zu6d3tBG/aIGpOnIjliXFH22boj4RwxxbMYRIN91iEBYZJBNjTobSt8kbd1LmSDsSzV4 gbBzMzR6XR1KMEngoxw2WE2sO9QdY6Aym47Lk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VmsFN1wOQa+/y6Ip6MIBv5QRIL6S+54KcKA38AyC6x8=; b=El59P2Jvni7vyR7YG7jFIYa1vEZKET14RiN7boNc3Ssjfrkzj6vCRB8r2yJPuWOD6y UYM3fD+rpHgbKUu1frA66xejXfoLSXM3fxQd5/4cvLP+cWtvp4aRoehKDBjgIc8tMrSw 14HvIYfxtqVtJSZFNR1cwNP1l1Jnb51iUzgX08kftictMfs6k9Knmb5B7dlmQ1Qb3Ams nb8tGEVKIsdQTKmoY/NUT0glzlAm5Kazk105VXXIZevcuSIpsrZTHKyH81xCVqTLSjZZ OgvNEs5PetUdaYtZQxL08BE/9qNNXubAU+uYaVSrklLIvefuvstG6+PbHU3QCAtEUWr8 FK0Q== X-Gm-Message-State: AO0yUKWQcMfIhKSk6PHfJy9ElRFSQDaSNpFibA8rroNBKSCTEbrgUZh8 sywE9vdnuLHbwPmRlfBi41u/UA== X-Received: by 2002:a17:903:2351:b0:197:7e18:c3c3 with SMTP id c17-20020a170903235100b001977e18c3c3mr4222241plh.9.1675142768422; Mon, 30 Jan 2023 21:26:08 -0800 (PST) Received: from localhost.localdomain ([2605:a601:a780:1400:a879:b64b:d9bd:3c1]) by smtp.gmail.com with ESMTPSA id jk15-20020a170903330f00b001960cccc318sm2310106plb.121.2023.01.30.21.26.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Jan 2023 21:26:07 -0800 (PST) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, gregkh@linuxfoundation.org, Leon Romanovsky Subject: [PATCH net-next v9 1/8] bnxt_en: Add auxiliary driver support Date: Mon, 30 Jan 2023 21:25:50 -0800 Message-Id: <20230131052557.99119-2-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230131052557.99119-1-ajit.khaparde@broadcom.com> References: <20230131052557.99119-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756514581666597307?= X-GMAIL-MSGID: =?utf-8?q?1756514581666597307?= Add auxiliary driver support. An auxiliary device will be created if the hardware indicates support for RDMA. The bnxt_ulp_probe() function has been removed and a new bnxt_rdma_aux_device_add() function has been added. The bnxt_free_msix_vecs() and bnxt_req_msix_vecs() will now hold the RTNL lock when they call the bnxt_close_nic()and bnxt_open_nic() since the device close and open need to be protected under RTNL lock. The operations between the bnxt_en and bnxt_re will be protected using the en_ops_lock. This will be used by the bnxt_re driver in a follow-on patch to create ROCE interfaces. Signed-off-by: Ajit Khaparde Reviewed-by: Andy Gospodarek Reviewed-by: Selvin Xavier Reviewed-by: Leon Romanovsky --- drivers/net/ethernet/broadcom/Kconfig | 1 + drivers/net/ethernet/broadcom/bnxt/bnxt.c | 6 +- drivers/net/ethernet/broadcom/bnxt/bnxt.h | 8 + drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 138 +++++++++++++++--- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 3 +- 5 files changed, 130 insertions(+), 26 deletions(-) diff --git a/drivers/net/ethernet/broadcom/Kconfig b/drivers/net/ethernet/broadcom/Kconfig index f4ca0c6c0f51..948586bf1b5b 100644 --- a/drivers/net/ethernet/broadcom/Kconfig +++ b/drivers/net/ethernet/broadcom/Kconfig @@ -213,6 +213,7 @@ config BNXT select NET_DEVLINK select PAGE_POOL select DIMLIB + select AUXILIARY_BUS help This driver supports Broadcom NetXtreme-C/E 10/25/40/50 gigabit Ethernet cards. To compile this driver as a module, choose M here: diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 240a7e8a7652..a16f2b0b297f 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -13178,6 +13178,7 @@ static void bnxt_remove_one(struct pci_dev *pdev) struct net_device *dev = pci_get_drvdata(pdev); struct bnxt *bp = netdev_priv(dev); + bnxt_rdma_aux_device_uninit(bp); if (BNXT_PF(bp)) bnxt_sriov_disable(bp); @@ -13776,11 +13777,13 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) bnxt_dl_fw_reporters_create(bp); + bnxt_rdma_aux_device_init(bp); + bnxt_print_device_info(bp); pci_save_state(pdev); - return 0; + return 0; init_err_cleanup: bnxt_dl_unregister(bp); init_err_dl: @@ -13824,7 +13827,6 @@ static void bnxt_shutdown(struct pci_dev *pdev) if (netif_running(dev)) dev_close(dev); - bnxt_ulp_shutdown(bp); bnxt_clear_int_mode(bp); pci_disable_device(pdev); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethernet/broadcom/bnxt/bnxt.h index 5163ef4a49ea..dcb09fbe4007 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -1631,6 +1632,12 @@ struct bnxt_fw_health { #define BNXT_FW_IF_RETRY 10 #define BNXT_FW_SLOT_RESET_RETRY 4 +struct bnxt_aux_priv { + struct auxiliary_device aux_dev; + struct bnxt_en_dev *edev; + int id; +}; + enum board_idx { BCM57301, BCM57302, @@ -1852,6 +1859,7 @@ struct bnxt { #define BNXT_CHIP_P4_PLUS(bp) \ (BNXT_CHIP_P4(bp) || BNXT_CHIP_P5(bp)) + struct bnxt_aux_priv *aux_priv; struct bnxt_en_dev *edev; struct bnxt_napi **bnapi; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index 2e54bf4fc7a7..290c4c4d96d0 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -25,6 +25,8 @@ #include "bnxt_hwrm.h" #include "bnxt_ulp.h" +static DEFINE_IDA(bnxt_aux_dev_ids); + static int bnxt_register_dev(struct bnxt_en_dev *edev, unsigned int ulp_id, struct bnxt_ulp_ops *ulp_ops, void *handle) { @@ -32,7 +34,6 @@ static int bnxt_register_dev(struct bnxt_en_dev *edev, unsigned int ulp_id, struct bnxt *bp = netdev_priv(dev); struct bnxt_ulp *ulp; - ASSERT_RTNL(); if (ulp_id >= BNXT_MAX_ULP) return -EINVAL; @@ -69,10 +70,11 @@ static int bnxt_unregister_dev(struct bnxt_en_dev *edev, unsigned int ulp_id) struct bnxt_ulp *ulp; int i = 0; - ASSERT_RTNL(); if (ulp_id >= BNXT_MAX_ULP) return -EINVAL; + edev->flags |= BNXT_EN_FLAG_ULP_STOPPED; + ulp = &edev->ulp_tbl[ulp_id]; if (!rcu_access_pointer(ulp->ulp_ops)) { netdev_err(bp->dev, "ulp id %d not registered\n", ulp_id); @@ -126,7 +128,6 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, int total_vecs; int rc = 0; - ASSERT_RTNL(); if (ulp_id != BNXT_ROCE_ULP) return -EINVAL; @@ -153,6 +154,7 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, edev->ulp_tbl[ulp_id].msix_requested = avail_msix; hw_resc = &bp->hw_resc; total_vecs = idx + avail_msix; + rtnl_lock(); if (bp->total_irqs < total_vecs || (BNXT_NEW_RM(bp) && hw_resc->resv_irqs < total_vecs)) { if (netif_running(dev)) { @@ -162,6 +164,7 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, rc = bnxt_reserve_rings(bp, true); } } + rtnl_unlock(); if (rc) { edev->ulp_tbl[ulp_id].msix_requested = 0; return -EAGAIN; @@ -184,7 +187,6 @@ static int bnxt_free_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id) struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); - ASSERT_RTNL(); if (ulp_id != BNXT_ROCE_ULP) return -EINVAL; @@ -193,10 +195,13 @@ static int bnxt_free_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id) edev->ulp_tbl[ulp_id].msix_requested = 0; edev->flags &= ~BNXT_EN_FLAG_MSIX_REQUESTED; + rtnl_lock(); if (netif_running(dev) && !(edev->flags & BNXT_EN_FLAG_ULP_STOPPED)) { bnxt_close_nic(bp, true, false); bnxt_open_nic(bp, true, false); } + rtnl_unlock(); + return 0; } @@ -347,25 +352,6 @@ void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs) } } -void bnxt_ulp_shutdown(struct bnxt *bp) -{ - struct bnxt_en_dev *edev = bp->edev; - struct bnxt_ulp_ops *ops; - int i; - - if (!edev) - return; - - for (i = 0; i < BNXT_MAX_ULP; i++) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[i]; - - ops = rtnl_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_shutdown) - continue; - ops->ulp_shutdown(ulp->handle); - } -} - void bnxt_ulp_irq_stop(struct bnxt *bp) { struct bnxt_en_dev *edev = bp->edev; @@ -475,6 +461,112 @@ static const struct bnxt_en_ops bnxt_en_ops_tbl = { .bnxt_register_fw_async_events = bnxt_register_async_events, }; +void bnxt_rdma_aux_device_uninit(struct bnxt *bp) +{ + struct bnxt_aux_priv *aux_priv; + struct auxiliary_device *adev; + + /* Skip if no auxiliary device init was done. */ + if (!(bp->flags & BNXT_FLAG_ROCE_CAP)) + return; + + aux_priv = bp->aux_priv; + adev = &aux_priv->aux_dev; + auxiliary_device_delete(adev); + auxiliary_device_uninit(adev); +} + +static void bnxt_aux_dev_release(struct device *dev) +{ + struct bnxt_aux_priv *aux_priv = + container_of(dev, struct bnxt_aux_priv, aux_dev.dev); + struct bnxt *bp = netdev_priv(aux_priv->edev->net); + + ida_free(&bnxt_aux_dev_ids, aux_priv->id); + aux_priv->edev->en_ops = NULL; + kfree(aux_priv->edev); + aux_priv->edev = NULL; + bp->edev = NULL; + kfree(aux_priv); +} + +static void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp) +{ + edev->en_ops = &bnxt_en_ops_tbl; + edev->net = bp->dev; + edev->pdev = bp->pdev; + edev->l2_db_size = bp->db_size; + edev->l2_db_size_nc = bp->db_size; + + if (bp->flags & BNXT_FLAG_ROCEV1_CAP) + edev->flags |= BNXT_EN_FLAG_ROCEV1_CAP; + if (bp->flags & BNXT_FLAG_ROCEV2_CAP) + edev->flags |= BNXT_EN_FLAG_ROCEV2_CAP; +} + +void bnxt_rdma_aux_device_init(struct bnxt *bp) +{ + struct auxiliary_device *aux_dev; + struct bnxt_aux_priv *aux_priv; + struct bnxt_en_dev *edev; + int rc; + + if (!(bp->flags & BNXT_FLAG_ROCE_CAP)) + return; + + bp->aux_priv = kzalloc(sizeof(*bp->aux_priv), GFP_KERNEL); + if (!bp->aux_priv) + goto exit; + + bp->aux_priv->id = ida_alloc(&bnxt_aux_dev_ids, GFP_KERNEL); + if (bp->aux_priv->id < 0) { + netdev_warn(bp->dev, + "ida alloc failed for ROCE auxiliary device\n"); + kfree(bp->aux_priv); + goto exit; + } + + aux_priv = bp->aux_priv; + aux_dev = &aux_priv->aux_dev; + aux_dev->id = aux_priv->id; + aux_dev->name = "rdma"; + aux_dev->dev.parent = &bp->pdev->dev; + aux_dev->dev.release = bnxt_aux_dev_release; + + rc = auxiliary_device_init(aux_dev); + if (rc) { + ida_free(&bnxt_aux_dev_ids, bp->aux_priv->id); + kfree(bp->aux_priv); + goto exit; + } + + /* From this point, all cleanup will happen via the .release callback & + * any error unwinding will need to include a call to + * auxiliary_device_uninit. + */ + edev = kzalloc(sizeof(*edev), GFP_KERNEL); + if (!edev) + goto aux_dev_uninit; + + aux_priv->edev = edev; + bp->edev = edev; + bnxt_set_edev_info(edev, bp); + + rc = auxiliary_device_add(aux_dev); + if (rc) { + netdev_warn(bp->dev, + "Failed to add auxiliary device for ROCE\n"); + goto aux_dev_uninit; + } + + return; + +aux_dev_uninit: + auxiliary_device_uninit(aux_dev); +exit: + bp->flags &= ~BNXT_FLAG_ROCE_CAP; +} + struct bnxt_en_dev *bnxt_ulp_probe(struct net_device *dev) { struct bnxt *bp = netdev_priv(dev); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index 42b50abc3e91..740281e74781 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -102,10 +102,11 @@ int bnxt_get_ulp_stat_ctxs(struct bnxt *bp); void bnxt_ulp_stop(struct bnxt *bp); void bnxt_ulp_start(struct bnxt *bp, int err); void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs); -void bnxt_ulp_shutdown(struct bnxt *bp); void bnxt_ulp_irq_stop(struct bnxt *bp); void bnxt_ulp_irq_restart(struct bnxt *bp, int err); void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl); +void bnxt_rdma_aux_device_uninit(struct bnxt *bp); +void bnxt_rdma_aux_device_init(struct bnxt *bp); struct bnxt_en_dev *bnxt_ulp_probe(struct net_device *dev); #endif From patchwork Tue Jan 31 05:25:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 50606 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2569662wrn; Mon, 30 Jan 2023 21:27:26 -0800 (PST) X-Google-Smtp-Source: AK7set/Va40OewWh4ZVzT6Hb3/Ykogfnq24IzOE9/kIdAcFRm2dfqQ+vHSrEUQMLi3rv8OcSDQhe X-Received: by 2002:aa7:d585:0:b0:4a2:6660:8fbe with SMTP id r5-20020aa7d585000000b004a266608fbemr1294423edq.5.1675142845658; Mon, 30 Jan 2023 21:27:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675142845; cv=none; d=google.com; s=arc-20160816; b=JRXTU/rTGXmSWpsPp3Xzi/TsB59z1zmqgubos0ODsgn3/ay4Jj6FBsFoPF+AlndSou YTiXgWoh43SQOyG7y4aI1QbBQP9c4d/rzNC8xS+4McIvYIq8RPYQ61h7fke2XQjiuKw3 s1mSxznb7Z5UJIj54SHm4TgQvnTovvsfyLwkUldcj5xsDroOWrAPoY+HTGxw/adAns0k DxvfqaDqlB3xwxcKxTScIk7r5ZURh9Py8fbyP+mgPnP0TQAiRDdOGHV1axPkfv60HjYA 4zcJA/odUSIQlMhNv4J4AJN4QlrhCOXIjUMLlS6fhfU83T9n40M71QTL3ooHGqDO/X5W 53Zg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=m5NLEzOiD7DOYb12gzdY4gtdndKFQvUeCSRUpmlv0vA=; b=kOzOqYm2BvMv/mjXS7mn8V4t5XCJhvtLKuqklngXZ8mvxaN0McuwBb0mwT7QkEDRie mpr/q5ORS4h7yYO1ncemx3VmlMga7QXWd8hC92oD13CWkVhEEwtkpNUKXrt6O0Ow4GzR Ax1oTOoL6mlQiVuX9619NI8xFsGXsJDvfNyBb9C/1GaUKXLDB3rP9uM3NSMAw1bdLc7E /97OB+yUxtUp1YzCUebHk419/Pymny8BMUxAIhCOCsxJSnrwQOpiufE/Fu1SVayoPR0D Yq5ONGxYCEs2xFwy91hmUDbp04YmI5+u2aW44PxwLYrakSJKfO2GDzRs7t38dQd2illE Hv9g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=bgffVF04; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id wg15-20020a1709078f0f00b0087bcda69438si13281049ejc.465.2023.01.30.21.27.02; Mon, 30 Jan 2023 21:27:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=bgffVF04; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231126AbjAaF0b (ORCPT + 99 others); Tue, 31 Jan 2023 00:26:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36034 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230202AbjAaF0N (ORCPT ); Tue, 31 Jan 2023 00:26:13 -0500 Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A039E29155 for ; Mon, 30 Jan 2023 21:26:10 -0800 (PST) Received: by mail-pl1-x62e.google.com with SMTP id r8so7103724pls.2 for ; Mon, 30 Jan 2023 21:26:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=m5NLEzOiD7DOYb12gzdY4gtdndKFQvUeCSRUpmlv0vA=; b=bgffVF047fdrO5WlXPISeNvafapJ3/NCI45/gcTzw6HT380PQgFTDJmFEG1aed0BPY xaCxWimbaGY7wbt/qOiIwIfgkP8ZeshO/Um1Z6jNHj3gq6rMvX+k+T3BE8M7EjGsCLcs RoB06oQhTXWhJUX1suqFiMxY6MmqUf0ou9CC8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=m5NLEzOiD7DOYb12gzdY4gtdndKFQvUeCSRUpmlv0vA=; b=AIaUa7+Of1sQzZE7PNx1hhr+2g1TjDy7Gl48osfvbkfc9VNDys90T6p6Cl9ub3Yhvv cCNzkF4wWNZcysmDSwKxdld7+0vUXknXhzorxl5rzTcK5uj+F6Of+jY0+8QPETtYF9DZ vj0ZbLVPKkk1RIUbPZRfsTZk/hJIi06LgHgWS8NGlQLGUgZY5cy0v0ZExymYjHWiT+2M e9SV7RgOGHawRQzLZenPBJDUmS8ezddCiiGoT18IEX5rSrI+J6BQlFOgPWtx3Y969/c9 G7V0g7G9DcGsNsHWRoIpxX5pdKDOo6FfblmtcHEgcshWwjCw/T9C4gw6rrP0XFCapzon okZg== X-Gm-Message-State: AO0yUKXZJ8TN823TCcJCklPwd7MJ7OFV4t+YmHpwP+rjVO5ltApseogM /JlBLFQzG/NROLtpWNNElHhWHg== X-Received: by 2002:a05:6a20:3d1e:b0:be:9e43:82be with SMTP id y30-20020a056a203d1e00b000be9e4382bemr5049666pzi.16.1675142770098; Mon, 30 Jan 2023 21:26:10 -0800 (PST) Received: from localhost.localdomain ([2605:a601:a780:1400:a879:b64b:d9bd:3c1]) by smtp.gmail.com with ESMTPSA id jk15-20020a170903330f00b001960cccc318sm2310106plb.121.2023.01.30.21.26.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Jan 2023 21:26:09 -0800 (PST) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, gregkh@linuxfoundation.org, Leon Romanovsky Subject: [PATCH net-next v9 2/8] RDMA/bnxt_re: Use auxiliary driver interface Date: Mon, 30 Jan 2023 21:25:51 -0800 Message-Id: <20230131052557.99119-3-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230131052557.99119-1-ajit.khaparde@broadcom.com> References: <20230131052557.99119-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756514584413251829?= X-GMAIL-MSGID: =?utf-8?q?1756514584413251829?= Use auxiliary driver interface for driver load, unload ROCE driver. The driver does not need to register the interface using the netdev notifier anymore. Removed the bnxt_re_dev_list which is not needed. Currently probe, remove and shutdown ops have been implemented for the auxiliary device. Also remove exccessve validation checks for rdev. Signed-off-by: Ajit Khaparde Reviewed-by: Andy Gospodarek Reviewed-by: Selvin Xavier Reviewed-by: Leon Romanovsky --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 9 +- drivers/infiniband/hw/bnxt_re/main.c | 395 ++++++------------ drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 29 +- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 3 - 4 files changed, 132 insertions(+), 304 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index 785c37cae3c0..b0465c8d229a 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -89,13 +89,6 @@ struct bnxt_re_ring_attr { u8 mode; }; -struct bnxt_re_work { - struct work_struct work; - unsigned long event; - struct bnxt_re_dev *rdev; - struct net_device *vlan_dev; -}; - struct bnxt_re_sqp_entries { struct bnxt_qplib_sge sge; u64 wrid; @@ -132,6 +125,7 @@ struct bnxt_re_dev { #define BNXT_RE_FLAG_ERR_DEVICE_DETACHED 17 #define BNXT_RE_FLAG_ISSUE_ROCE_STATS 29 struct net_device *netdev; + struct notifier_block nb; unsigned int version, major, minor; struct bnxt_qplib_chip_ctx *chip_ctx; struct bnxt_en_dev *en_dev; @@ -194,5 +188,4 @@ static inline struct device *rdev_to_dev(struct bnxt_re_dev *rdev) return &rdev->ibdev.dev; return NULL; } - #endif diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 8c0c80a8d338..82c81ffea152 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -48,6 +48,7 @@ #include #include #include +#include #include #include @@ -74,14 +75,14 @@ MODULE_DESCRIPTION(BNXT_RE_DESC " Driver"); MODULE_LICENSE("Dual BSD/GPL"); /* globals */ -static struct list_head bnxt_re_dev_list = LIST_HEAD_INIT(bnxt_re_dev_list); -/* Mutex to protect the list of bnxt_re devices added */ -static DEFINE_MUTEX(bnxt_re_dev_lock); -static struct workqueue_struct *bnxt_re_wq; -static void bnxt_re_remove_device(struct bnxt_re_dev *rdev); -static void bnxt_re_dealloc_driver(struct ib_device *ib_dev); +static DEFINE_MUTEX(bnxt_re_mutex); + static void bnxt_re_stop_irq(void *handle); static void bnxt_re_dev_stop(struct bnxt_re_dev *rdev); +static int bnxt_re_netdev_event(struct notifier_block *notifier, + unsigned long event, void *ptr); +static struct bnxt_re_dev *bnxt_re_from_netdev(struct net_device *netdev); +static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev); static void bnxt_re_set_drv_mode(struct bnxt_re_dev *rdev, u8 mode) { @@ -233,7 +234,6 @@ static void bnxt_re_stop(void *p) if (!rdev) return; - ASSERT_RTNL(); /* L2 driver invokes this callback during device error/crash or device * reset. Current RoCE driver doesn't recover the device in case of @@ -269,9 +269,6 @@ static void bnxt_re_sriov_config(void *p, int num_vfs) { struct bnxt_re_dev *rdev = p; - if (!rdev) - return; - if (test_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags)) return; rdev->num_vfs = num_vfs; @@ -282,16 +279,14 @@ static void bnxt_re_sriov_config(void *p, int num_vfs) } } -static void bnxt_re_shutdown(void *p) +static void bnxt_re_shutdown(struct auxiliary_device *adev) { - struct bnxt_re_dev *rdev = p; + struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); if (!rdev) return; - ASSERT_RTNL(); - /* Release the MSIx vectors before queuing unregister */ - bnxt_re_stop_irq(rdev); - ib_unregister_device_queued(&rdev->ibdev); + ib_unregister_device(&rdev->ibdev); + bnxt_re_dev_uninit(rdev); } static void bnxt_re_stop_irq(void *handle) @@ -346,11 +341,9 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) } static struct bnxt_ulp_ops bnxt_re_ulp_ops = { - .ulp_async_notifier = NULL, .ulp_stop = bnxt_re_stop, .ulp_start = bnxt_re_start, .ulp_sriov_config = bnxt_re_sriov_config, - .ulp_shutdown = bnxt_re_shutdown, .ulp_irq_stop = bnxt_re_stop_irq, .ulp_irq_restart = bnxt_re_start_irq }; @@ -380,9 +373,6 @@ static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) struct bnxt_en_dev *en_dev; int rc = 0; - if (!rdev) - return -EINVAL; - en_dev = rdev->en_dev; rc = en_dev->en_ops->bnxt_register_device(en_dev, BNXT_ROCE_ULP, @@ -401,7 +391,6 @@ static int bnxt_re_free_msix(struct bnxt_re_dev *rdev) en_dev = rdev->en_dev; - rc = en_dev->en_ops->bnxt_free_msix(rdev->en_dev, BNXT_ROCE_ULP); return rc; @@ -412,9 +401,6 @@ static int bnxt_re_request_msix(struct bnxt_re_dev *rdev) int rc = 0, num_msix_want = BNXT_RE_MAX_MSIX, num_msix_got; struct bnxt_en_dev *en_dev; - if (!rdev) - return -EINVAL; - en_dev = rdev->en_dev; num_msix_want = min_t(u32, BNXT_RE_MAX_MSIX, num_online_cpus()); @@ -458,12 +444,17 @@ static void bnxt_re_fill_fw_msg(struct bnxt_fw_msg *fw_msg, void *msg, static int bnxt_re_net_ring_free(struct bnxt_re_dev *rdev, u16 fw_ring_id, int type) { - struct bnxt_en_dev *en_dev = rdev->en_dev; + struct bnxt_en_dev *en_dev; struct hwrm_ring_free_input req = {0}; struct hwrm_ring_free_output resp; struct bnxt_fw_msg fw_msg; int rc = -EINVAL; + if (!rdev) + return rc; + + en_dev = rdev->en_dev; + if (!en_dev) return rc; @@ -584,21 +575,6 @@ static int bnxt_re_net_stats_ctx_alloc(struct bnxt_re_dev *rdev, /* Device */ -static bool is_bnxt_re_dev(struct net_device *netdev) -{ - struct ethtool_drvinfo drvinfo; - - if (netdev->ethtool_ops && netdev->ethtool_ops->get_drvinfo) { - memset(&drvinfo, 0, sizeof(drvinfo)); - netdev->ethtool_ops->get_drvinfo(netdev, &drvinfo); - - if (strcmp(drvinfo.driver, "bnxt_en")) - return false; - return true; - } - return false; -} - static struct bnxt_re_dev *bnxt_re_from_netdev(struct net_device *netdev) { struct ib_device *ibdev = @@ -609,31 +585,6 @@ static struct bnxt_re_dev *bnxt_re_from_netdev(struct net_device *netdev) return container_of(ibdev, struct bnxt_re_dev, ibdev); } -static struct bnxt_en_dev *bnxt_re_dev_probe(struct net_device *netdev) -{ - struct bnxt_en_dev *en_dev; - struct pci_dev *pdev; - - en_dev = bnxt_ulp_probe(netdev); - if (IS_ERR(en_dev)) - return en_dev; - - pdev = en_dev->pdev; - if (!pdev) - return ERR_PTR(-EINVAL); - - if (!(en_dev->flags & BNXT_EN_FLAG_ROCE_CAP)) { - dev_info(&pdev->dev, - "%s: probe error: RoCE is not supported on this device", - ROCE_DRV_MODULE_NAME); - return ERR_PTR(-ENODEV); - } - - dev_hold(netdev); - - return en_dev; -} - static ssize_t hw_rev_show(struct device *device, struct device_attribute *attr, char *buf) { @@ -679,7 +630,6 @@ static const struct ib_device_ops bnxt_re_dev_ops = { .create_qp = bnxt_re_create_qp, .create_srq = bnxt_re_create_srq, .create_user_ah = bnxt_re_create_ah, - .dealloc_driver = bnxt_re_dealloc_driver, .dealloc_pd = bnxt_re_dealloc_pd, .dealloc_ucontext = bnxt_re_dealloc_ucontext, .del_gid = bnxt_re_del_gid, @@ -744,18 +694,7 @@ static int bnxt_re_register_ib(struct bnxt_re_dev *rdev) return ib_register_device(ibdev, "bnxt_re%d", &rdev->en_dev->pdev->dev); } -static void bnxt_re_dev_remove(struct bnxt_re_dev *rdev) -{ - dev_put(rdev->netdev); - rdev->netdev = NULL; - mutex_lock(&bnxt_re_dev_lock); - list_del_rcu(&rdev->list); - mutex_unlock(&bnxt_re_dev_lock); - - synchronize_rcu(); -} - -static struct bnxt_re_dev *bnxt_re_dev_add(struct net_device *netdev, +static struct bnxt_re_dev *bnxt_re_dev_add(struct bnxt_aux_priv *aux_priv, struct bnxt_en_dev *en_dev) { struct bnxt_re_dev *rdev; @@ -768,8 +707,8 @@ static struct bnxt_re_dev *bnxt_re_dev_add(struct net_device *netdev, return NULL; } /* Default values */ - rdev->netdev = netdev; - dev_hold(rdev->netdev); + rdev->nb.notifier_call = NULL; + rdev->netdev = en_dev->net; rdev->en_dev = en_dev; rdev->id = rdev->en_dev->pdev->devfn; INIT_LIST_HEAD(&rdev->qp_list); @@ -784,9 +723,6 @@ static struct bnxt_re_dev *bnxt_re_dev_add(struct net_device *netdev, rdev->cosq[0] = 0xFFFF; rdev->cosq[1] = 0xFFFF; - mutex_lock(&bnxt_re_dev_lock); - list_add_tail_rcu(&rdev->list, &bnxt_re_dev_list); - mutex_unlock(&bnxt_re_dev_lock); return rdev; } @@ -1323,7 +1259,7 @@ static int bnxt_re_ib_init(struct bnxt_re_dev *rdev) pr_err("Failed to register with IB: %#x\n", rc); return rc; } - dev_info(rdev_to_dev(rdev), "Device registered successfully"); + dev_info(rdev_to_dev(rdev), "Device registered with IB successfully"); ib_get_eth_speed(&rdev->ibdev, 1, &rdev->active_speed, &rdev->active_width); set_bit(BNXT_RE_FLAG_ISSUE_ROCE_STATS, &rdev->flags); @@ -1541,135 +1477,43 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 wqe_mode) return rc; } -static void bnxt_re_dev_unreg(struct bnxt_re_dev *rdev) -{ - struct net_device *netdev = rdev->netdev; - - bnxt_re_dev_remove(rdev); - - if (netdev) - dev_put(netdev); -} - -static int bnxt_re_dev_reg(struct bnxt_re_dev **rdev, struct net_device *netdev) +static int bnxt_re_add_device(struct auxiliary_device *adev, u8 wqe_mode) { + struct bnxt_aux_priv *aux_priv = + container_of(adev, struct bnxt_aux_priv, aux_dev); struct bnxt_en_dev *en_dev; + struct bnxt_re_dev *rdev; int rc = 0; - if (!is_bnxt_re_dev(netdev)) - return -ENODEV; + /* en_dev should never be NULL as long as adev and aux_dev are valid. */ + en_dev = aux_priv->edev; - en_dev = bnxt_re_dev_probe(netdev); - if (IS_ERR(en_dev)) { - if (en_dev != ERR_PTR(-ENODEV)) - ibdev_err(&(*rdev)->ibdev, "%s: Failed to probe\n", - ROCE_DRV_MODULE_NAME); - rc = PTR_ERR(en_dev); - goto exit; - } - *rdev = bnxt_re_dev_add(netdev, en_dev); - if (!*rdev) { + rdev = bnxt_re_dev_add(aux_priv, en_dev); + if (!rdev || !rdev_to_dev(rdev)) { rc = -ENOMEM; - dev_put(netdev); goto exit; } -exit: - return rc; -} - -static void bnxt_re_remove_device(struct bnxt_re_dev *rdev) -{ - bnxt_re_dev_uninit(rdev); - pci_dev_put(rdev->en_dev->pdev); - bnxt_re_dev_unreg(rdev); -} - -static int bnxt_re_add_device(struct bnxt_re_dev **rdev, - struct net_device *netdev, u8 wqe_mode) -{ - int rc; - rc = bnxt_re_dev_reg(rdev, netdev); - if (rc == -ENODEV) - return rc; - if (rc) { - pr_err("Failed to register with the device %s: %#x\n", - netdev->name, rc); - return rc; - } + rc = bnxt_re_dev_init(rdev, wqe_mode); + if (rc) + goto re_dev_dealloc; - pci_dev_get((*rdev)->en_dev->pdev); - rc = bnxt_re_dev_init(*rdev, wqe_mode); + rc = bnxt_re_ib_init(rdev); if (rc) { - pci_dev_put((*rdev)->en_dev->pdev); - bnxt_re_dev_unreg(*rdev); + pr_err("Failed to register with IB: %s", + aux_priv->aux_dev.name); + goto re_dev_uninit; } + auxiliary_set_drvdata(adev, rdev); - return rc; -} - -static void bnxt_re_dealloc_driver(struct ib_device *ib_dev) -{ - struct bnxt_re_dev *rdev = - container_of(ib_dev, struct bnxt_re_dev, ibdev); - - dev_info(rdev_to_dev(rdev), "Unregistering Device"); - - rtnl_lock(); - bnxt_re_remove_device(rdev); - rtnl_unlock(); -} - -/* Handle all deferred netevents tasks */ -static void bnxt_re_task(struct work_struct *work) -{ - struct bnxt_re_work *re_work; - struct bnxt_re_dev *rdev; - int rc = 0; - - re_work = container_of(work, struct bnxt_re_work, work); - rdev = re_work->rdev; - - if (re_work->event == NETDEV_REGISTER) { - rc = bnxt_re_ib_init(rdev); - if (rc) { - ibdev_err(&rdev->ibdev, - "Failed to register with IB: %#x", rc); - rtnl_lock(); - bnxt_re_remove_device(rdev); - rtnl_unlock(); - goto exit; - } - goto exit; - } - - if (!ib_device_try_get(&rdev->ibdev)) - goto exit; + return 0; - switch (re_work->event) { - case NETDEV_UP: - bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, - IB_EVENT_PORT_ACTIVE); - break; - case NETDEV_DOWN: - bnxt_re_dev_stop(rdev); - break; - case NETDEV_CHANGE: - if (!netif_carrier_ok(rdev->netdev)) - bnxt_re_dev_stop(rdev); - else if (netif_carrier_ok(rdev->netdev)) - bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, - IB_EVENT_PORT_ACTIVE); - ib_get_eth_speed(&rdev->ibdev, 1, &rdev->active_speed, - &rdev->active_width); - break; - default: - break; - } - ib_device_put(&rdev->ibdev); +re_dev_uninit: + bnxt_re_dev_uninit(rdev); +re_dev_dealloc: + ib_dealloc_device(&rdev->ibdev); exit: - put_device(&rdev->ibdev.dev); - kfree(re_work); + return rc; } /* @@ -1690,109 +1534,130 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, unsigned long event, void *ptr) { struct net_device *real_dev, *netdev = netdev_notifier_info_to_dev(ptr); - struct bnxt_re_work *re_work; struct bnxt_re_dev *rdev; - int rc = 0; - bool sch_work = false; - bool release = true; real_dev = rdma_vlan_dev_real_dev(netdev); if (!real_dev) real_dev = netdev; - rdev = bnxt_re_from_netdev(real_dev); - if (!rdev && event != NETDEV_REGISTER) - return NOTIFY_OK; - if (real_dev != netdev) goto exit; - switch (event) { - case NETDEV_REGISTER: - if (rdev) - break; - rc = bnxt_re_add_device(&rdev, real_dev, - BNXT_QPLIB_WQE_MODE_STATIC); - if (!rc) - sch_work = true; - release = false; - break; + rdev = bnxt_re_from_netdev(real_dev); + if (!rdev) + return NOTIFY_DONE; - case NETDEV_UNREGISTER: - ib_unregister_device_queued(&rdev->ibdev); - break; + switch (event) { + case NETDEV_UP: + case NETDEV_DOWN: + case NETDEV_CHANGE: + bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, + netif_carrier_ok(real_dev) ? + IB_EVENT_PORT_ACTIVE : + IB_EVENT_PORT_ERR); + break; default: - sch_work = true; break; } - if (sch_work) { - /* Allocate for the deferred task */ - re_work = kzalloc(sizeof(*re_work), GFP_KERNEL); - if (re_work) { - get_device(&rdev->ibdev.dev); - re_work->rdev = rdev; - re_work->event = event; - re_work->vlan_dev = (real_dev == netdev ? - NULL : netdev); - INIT_WORK(&re_work->work, bnxt_re_task); - queue_work(bnxt_re_wq, &re_work->work); - } - } - + ib_device_put(&rdev->ibdev); exit: - if (rdev && release) - ib_device_put(&rdev->ibdev); return NOTIFY_DONE; } -static struct notifier_block bnxt_re_netdev_notifier = { - .notifier_call = bnxt_re_netdev_event -}; +#define BNXT_ADEV_NAME "bnxt_en" -static int __init bnxt_re_mod_init(void) +static void bnxt_re_remove(struct auxiliary_device *adev) { - int rc = 0; + struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); - pr_info("%s: %s", ROCE_DRV_MODULE_NAME, version); + if (!rdev) + return; - bnxt_re_wq = create_singlethread_workqueue("bnxt_re"); - if (!bnxt_re_wq) - return -ENOMEM; + mutex_lock(&bnxt_re_mutex); + if (rdev->nb.notifier_call) { + unregister_netdevice_notifier(&rdev->nb); + rdev->nb.notifier_call = NULL; + } else { + /* If notifier is null, we should have already done a + * clean up before coming here. + */ + goto skip_remove; + } + + ib_unregister_device(&rdev->ibdev); + ib_dealloc_device(&rdev->ibdev); + bnxt_re_dev_uninit(rdev); +skip_remove: + mutex_unlock(&bnxt_re_mutex); +} + +static int bnxt_re_probe(struct auxiliary_device *adev, + const struct auxiliary_device_id *id) +{ + struct bnxt_re_dev *rdev; + int rc; + + mutex_lock(&bnxt_re_mutex); + rc = bnxt_re_add_device(adev, BNXT_QPLIB_WQE_MODE_STATIC); + if (rc) { + mutex_unlock(&bnxt_re_mutex); + return rc; + } - INIT_LIST_HEAD(&bnxt_re_dev_list); + rdev = auxiliary_get_drvdata(adev); - rc = register_netdevice_notifier(&bnxt_re_netdev_notifier); + rdev->nb.notifier_call = bnxt_re_netdev_event; + rc = register_netdevice_notifier(&rdev->nb); if (rc) { + rdev->nb.notifier_call = NULL; pr_err("%s: Cannot register to netdevice_notifier", ROCE_DRV_MODULE_NAME); - goto err_netdev; + goto err; } + + mutex_unlock(&bnxt_re_mutex); return 0; -err_netdev: - destroy_workqueue(bnxt_re_wq); +err: + mutex_unlock(&bnxt_re_mutex); + bnxt_re_remove(adev); return rc; } -static void __exit bnxt_re_mod_exit(void) +static const struct auxiliary_device_id bnxt_re_id_table[] = { + { .name = BNXT_ADEV_NAME ".rdma", }, + {}, +}; + +MODULE_DEVICE_TABLE(auxiliary, bnxt_re_id_table); + +static struct auxiliary_driver bnxt_re_driver = { + .name = "rdma", + .probe = bnxt_re_probe, + .remove = bnxt_re_remove, + .shutdown = bnxt_re_shutdown, + .id_table = bnxt_re_id_table, +}; + +static int __init bnxt_re_mod_init(void) { - struct bnxt_re_dev *rdev; + int rc = 0; - unregister_netdevice_notifier(&bnxt_re_netdev_notifier); - if (bnxt_re_wq) - destroy_workqueue(bnxt_re_wq); - list_for_each_entry(rdev, &bnxt_re_dev_list, list) { - /* VF device removal should be called before the removal - * of PF device. Queue VFs unregister first, so that VFs - * shall be removed before the PF during the call of - * ib_unregister_driver. - */ - if (rdev->is_virtfn) - ib_unregister_device(&rdev->ibdev); + pr_info("%s: %s", ROCE_DRV_MODULE_NAME, version); + rc = auxiliary_driver_register(&bnxt_re_driver); + if (rc) { + pr_err("%s: Failed to register auxiliary driver\n", + ROCE_DRV_MODULE_NAME); + return rc; } - ib_unregister_driver(RDMA_DRIVER_BNXT_RE); + return 0; +} + +static void __exit bnxt_re_mod_exit(void) +{ + auxiliary_driver_unregister(&bnxt_re_driver); } module_init(bnxt_re_mod_init); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index 290c4c4d96d0..40583357b9fc 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -19,6 +19,7 @@ #include #include #include +#include #include "bnxt_hsi.h" #include "bnxt.h" @@ -73,8 +74,6 @@ static int bnxt_unregister_dev(struct bnxt_en_dev *edev, unsigned int ulp_id) if (ulp_id >= BNXT_MAX_ULP) return -EINVAL; - edev->flags |= BNXT_EN_FLAG_ULP_STOPPED; - ulp = &edev->ulp_tbl[ulp_id]; if (!rcu_access_pointer(ulp->ulp_ops)) { netdev_err(bp->dev, "ulp id %d not registered\n", ulp_id); @@ -566,29 +565,3 @@ void bnxt_rdma_aux_device_init(struct bnxt *bp) exit: bp->flags &= ~BNXT_FLAG_ROCE_CAP; } - -struct bnxt_en_dev *bnxt_ulp_probe(struct net_device *dev) -{ - struct bnxt *bp = netdev_priv(dev); - struct bnxt_en_dev *edev; - - edev = bp->edev; - if (!edev) { - edev = kzalloc(sizeof(*edev), GFP_KERNEL); - if (!edev) - return ERR_PTR(-ENOMEM); - edev->en_ops = &bnxt_en_ops_tbl; - edev->net = dev; - edev->pdev = bp->pdev; - edev->l2_db_size = bp->db_size; - edev->l2_db_size_nc = bp->db_size; - bp->edev = edev; - } - edev->flags &= ~BNXT_EN_FLAG_ROCE_CAP; - if (bp->flags & BNXT_FLAG_ROCEV1_CAP) - edev->flags |= BNXT_EN_FLAG_ROCEV1_CAP; - if (bp->flags & BNXT_FLAG_ROCEV2_CAP) - edev->flags |= BNXT_EN_FLAG_ROCEV2_CAP; - return bp->edev; -} -EXPORT_SYMBOL(bnxt_ulp_probe); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index 740281e74781..67fc3be2447a 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -31,7 +31,6 @@ struct bnxt_ulp_ops { void (*ulp_stop)(void *); void (*ulp_start)(void *); void (*ulp_sriov_config)(void *, int); - void (*ulp_shutdown)(void *); void (*ulp_irq_stop)(void *); void (*ulp_irq_restart)(void *, struct bnxt_msix_entry *); }; @@ -107,6 +106,4 @@ void bnxt_ulp_irq_restart(struct bnxt *bp, int err); void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl); void bnxt_rdma_aux_device_uninit(struct bnxt *bp); void bnxt_rdma_aux_device_init(struct bnxt *bp); -struct bnxt_en_dev *bnxt_ulp_probe(struct net_device *dev); - #endif From patchwork Tue Jan 31 05:25:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 50607 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2569762wrn; Mon, 30 Jan 2023 21:27:42 -0800 (PST) X-Google-Smtp-Source: AK7set8TNpHJiuMqiBVx9tTyVmCHRMBhXbwovF2qVKo8NQlwDtyfl2kHS/mvolT06LKNH7c4fzNN X-Received: by 2002:a17:906:5957:b0:888:9693:fb18 with SMTP id g23-20020a170906595700b008889693fb18mr1819093ejr.37.1675142862435; Mon, 30 Jan 2023 21:27:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675142862; cv=none; d=google.com; s=arc-20160816; b=ASrHqxzKHqrVQX/27Pm4t9zMXufCmOp3MQ3mM0rPrXONDTDjJoOp2g9MmHIk4t4cYz z2QxpJzqZAd13yPnLjINfdcOWg4SgXk27pGIP9Iznly12aMnXGZCDnrFGykDY/tJ/qF5 YmI3e5qEp0dgJFpq9Yvz6ZEXSyMPveRVuKDWRahFfS1BRG/xn9bVp3rgQSoCZr1KDeZA 5ZLmBtZWjXHR/tVroAF4YahPrtawuMR6fTteQdi5XNe7gMtVFz/403ErlC2kAo1yGLBF h7Nukovx8Df2/6tAkhoGS7H0R1ewxJlRsT4xJVNsn/FGjK/AkLMz4NzB9NskbFNNnoIY zzpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=RAYF6P9k+QOwEb6NVniuPm70quCNgsnBIhsJoJ1ekHE=; b=hzyD0e5ScxKoIVI5tN/eEX9vLxsIkkz35aYgctUCq0q9oqRMjH8okxIgXMZFVIDfSW HMtORGjzwtbktt6cORjQI6Ii/kxGSydFSpxku9S39IP5NIO5AmEEntEa5NdA1/gB2czM nWYQm/WV4P9q3gOuHXmJN6i0Y8YyoEYwN9r/jR44ml9vnzjxrOob7aB+1hHpb2WXU5ou uLk1vXOD+ksDtKLoOh3ah7ehCn4IMAo2y6zJOHCcArOdJrUtsasmbgWBNkwCACVg21cK E14mUrgA2KUHd1boEWhuqy+Ax/mNvvbdxd8H6gFJcYHsuQYlnAZT8y0fpewZ1fKS/54v DTDg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=iAExKiVL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j3-20020a170906534300b0087bce26e655si12384192ejo.973.2023.01.30.21.27.18; Mon, 30 Jan 2023 21:27:41 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=iAExKiVL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231137AbjAaF0h (ORCPT + 99 others); Tue, 31 Jan 2023 00:26:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36268 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230315AbjAaF0P (ORCPT ); Tue, 31 Jan 2023 00:26:15 -0500 Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 52D473A5AF for ; Mon, 30 Jan 2023 21:26:12 -0800 (PST) Received: by mail-pj1-x1035.google.com with SMTP id c10-20020a17090a1d0a00b0022e63a94799so2950687pjd.2 for ; Mon, 30 Jan 2023 21:26:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=RAYF6P9k+QOwEb6NVniuPm70quCNgsnBIhsJoJ1ekHE=; b=iAExKiVLKMETvAdGZHtid5JCu+RkcFYnAYHUgLMsDDLV9v8Az4mFGs3QhVELqo1jrw EEFnYd/HCSUJREdL/k0JPUaz4Ku3RNIMUe2o1o/P38Lamd1pkeAGPg8hIdMWSI6K25TK AIThm0mnpDH037es5hXDpoLC44oO1TWyqePuQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RAYF6P9k+QOwEb6NVniuPm70quCNgsnBIhsJoJ1ekHE=; b=Ftvwwq/4+OR9N9W0tIk34FS9CQrmkFUhRk/xndhzQFa9SrQBcZsycoDm6hhxrWy2Js jK20fC09EJQFcbxCWwTaYsvPeYcMDh3wLinLvA2RULMmmuMIfWNGF4QilslsnlSz84Dz gN0eDsrvmcQkbnRDjTDxahdoQOmz49B68rsimsLBuRPk1bxBIyCGLMkZcW0KEFhSigNZ RgGnb/nYyAehFdgd1/Dv20HAeHlEvlekND4SnUCg1dVNt18ofMGTyLUzG6aJsZMD8qzl Bh492wB8DkQw5oqiJPj6cY4zb8N/mRLH7B0hTM9g+YqX/tGTuSmDMVFkM9LX7xGAsm1Q V95g== X-Gm-Message-State: AO0yUKVR/M7jC7EWEtZFv5XO003fAJf3zz0/VovqY0j230/owEl/+S7T Tf6gz7KMT76FFotFnDC3HaaIkg== X-Received: by 2002:a17:902:e5ca:b0:196:86c2:ee60 with SMTP id u10-20020a170902e5ca00b0019686c2ee60mr7655478plf.15.1675142771646; Mon, 30 Jan 2023 21:26:11 -0800 (PST) Received: from localhost.localdomain ([2605:a601:a780:1400:a879:b64b:d9bd:3c1]) by smtp.gmail.com with ESMTPSA id jk15-20020a170903330f00b001960cccc318sm2310106plb.121.2023.01.30.21.26.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Jan 2023 21:26:11 -0800 (PST) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, gregkh@linuxfoundation.org, Leon Romanovsky Subject: [PATCH net-next v9 3/8] bnxt_en: Remove usage of ulp_id Date: Mon, 30 Jan 2023 21:25:52 -0800 Message-Id: <20230131052557.99119-4-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230131052557.99119-1-ajit.khaparde@broadcom.com> References: <20230131052557.99119-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756514602062855339?= X-GMAIL-MSGID: =?utf-8?q?1756514602062855339?= Since the driver continues to use the single ULP model, the extra complexity and indirection is unnecessary. Remove the usage of ulp_id from the code. Suggested-by: Leon Romanovsky Signed-off-by: Ajit Khaparde Reviewed-by: Andy Gospodarek Reviewed-by: Selvin Xavier Reviewed-by: Leon Romanovsky --- drivers/infiniband/hw/bnxt_re/main.c | 24 +- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 2 +- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 208 ++++++++---------- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 26 +-- 4 files changed, 114 insertions(+), 146 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 82c81ffea152..5bf3dc067484 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -363,8 +363,7 @@ static int bnxt_re_unregister_netdev(struct bnxt_re_dev *rdev) en_dev = rdev->en_dev; - rc = en_dev->en_ops->bnxt_unregister_device(rdev->en_dev, - BNXT_ROCE_ULP); + rc = en_dev->en_ops->bnxt_unregister_device(rdev->en_dev); return rc; } @@ -375,7 +374,7 @@ static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) en_dev = rdev->en_dev; - rc = en_dev->en_ops->bnxt_register_device(en_dev, BNXT_ROCE_ULP, + rc = en_dev->en_ops->bnxt_register_device(en_dev, &bnxt_re_ulp_ops, rdev); rdev->qplib_res.pdev = rdev->en_dev->pdev; return rc; @@ -384,16 +383,15 @@ static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) static int bnxt_re_free_msix(struct bnxt_re_dev *rdev) { struct bnxt_en_dev *en_dev; - int rc; if (!rdev) return -EINVAL; en_dev = rdev->en_dev; - rc = en_dev->en_ops->bnxt_free_msix(rdev->en_dev, BNXT_ROCE_ULP); + en_dev->en_ops->bnxt_free_msix(rdev->en_dev); - return rc; + return 0; } static int bnxt_re_request_msix(struct bnxt_re_dev *rdev) @@ -405,7 +403,7 @@ static int bnxt_re_request_msix(struct bnxt_re_dev *rdev) num_msix_want = min_t(u32, BNXT_RE_MAX_MSIX, num_online_cpus()); - num_msix_got = en_dev->en_ops->bnxt_request_msix(en_dev, BNXT_ROCE_ULP, + num_msix_got = en_dev->en_ops->bnxt_request_msix(en_dev, rdev->msix_entries, num_msix_want); if (num_msix_got < BNXT_RE_MIN_MSIX) { @@ -468,7 +466,7 @@ static int bnxt_re_net_ring_free(struct bnxt_re_dev *rdev, req.ring_id = cpu_to_le16(fw_ring_id); bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (rc) ibdev_err(&rdev->ibdev, "Failed to free HW ring:%d :%#x", req.ring_id, rc); @@ -505,7 +503,7 @@ static int bnxt_re_net_ring_alloc(struct bnxt_re_dev *rdev, req.int_mode = ring_attr->mode; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (!rc) *fw_ring_id = le16_to_cpu(resp.ring_id); @@ -533,7 +531,7 @@ static int bnxt_re_net_stats_ctx_free(struct bnxt_re_dev *rdev, req.stat_ctx_id = cpu_to_le32(fw_stats_ctx_id); bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (rc) ibdev_err(&rdev->ibdev, "Failed to free HW stats context %#x", rc); @@ -566,7 +564,7 @@ static int bnxt_re_net_stats_ctx_alloc(struct bnxt_re_dev *rdev, req.stat_ctx_flags = STAT_CTX_ALLOC_REQ_STAT_CTX_FLAGS_ROCE; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (!rc) *fw_stats_ctx_id = le32_to_cpu(resp.stat_ctx_id); @@ -1052,7 +1050,7 @@ static int bnxt_re_query_hwrm_pri2cos(struct bnxt_re_dev *rdev, u8 dir, bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (rc) return rc; @@ -1235,7 +1233,7 @@ static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) req.hwrm_intf_upd = HWRM_VERSION_UPDATE; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (rc) { ibdev_err(&rdev->ibdev, "Failed to query HW version, rc = 0x%x", rc); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index a16f2b0b297f..76c1382fc09f 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -5538,7 +5538,7 @@ int bnxt_hwrm_vnic_cfg(struct bnxt *bp, u16 vnic_id) #endif if ((bp->flags & BNXT_FLAG_STRIP_VLAN) || def_vlan) req->flags |= cpu_to_le32(VNIC_CFG_REQ_FLAGS_VLAN_STRIP_MODE); - if (!vnic_id && bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) + if (!vnic_id && bnxt_ulp_registered(bp->edev)) req->flags |= cpu_to_le32(bnxt_get_roce_vnic_mode(bp)); return hwrm_req_send(bp, req); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index 40583357b9fc..5bf42fd296c1 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -28,59 +28,44 @@ static DEFINE_IDA(bnxt_aux_dev_ids); -static int bnxt_register_dev(struct bnxt_en_dev *edev, unsigned int ulp_id, - struct bnxt_ulp_ops *ulp_ops, void *handle) +static int bnxt_register_dev(struct bnxt_en_dev *edev, + struct bnxt_ulp_ops *ulp_ops, + void *handle) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); + unsigned int max_stat_ctxs; struct bnxt_ulp *ulp; - if (ulp_id >= BNXT_MAX_ULP) - return -EINVAL; - - ulp = &edev->ulp_tbl[ulp_id]; - if (rcu_access_pointer(ulp->ulp_ops)) { - netdev_err(bp->dev, "ulp id %d already registered\n", ulp_id); - return -EBUSY; - } - if (ulp_id == BNXT_ROCE_ULP) { - unsigned int max_stat_ctxs; + max_stat_ctxs = bnxt_get_max_func_stat_ctxs(bp); + if (max_stat_ctxs <= BNXT_MIN_ROCE_STAT_CTXS || + bp->cp_nr_rings == max_stat_ctxs) + return -ENOMEM; - max_stat_ctxs = bnxt_get_max_func_stat_ctxs(bp); - if (max_stat_ctxs <= BNXT_MIN_ROCE_STAT_CTXS || - bp->cp_nr_rings == max_stat_ctxs) - return -ENOMEM; - } + ulp = kzalloc(sizeof(*ulp), GFP_KERNEL); + if (!ulp) + return -ENOMEM; - atomic_set(&ulp->ref_count, 0); + edev->ulp_tbl = ulp; ulp->handle = handle; rcu_assign_pointer(ulp->ulp_ops, ulp_ops); - if (ulp_id == BNXT_ROCE_ULP) { - if (test_bit(BNXT_STATE_OPEN, &bp->state)) - bnxt_hwrm_vnic_cfg(bp, 0); - } + if (test_bit(BNXT_STATE_OPEN, &bp->state)) + bnxt_hwrm_vnic_cfg(bp, 0); return 0; } -static int bnxt_unregister_dev(struct bnxt_en_dev *edev, unsigned int ulp_id) +static int bnxt_unregister_dev(struct bnxt_en_dev *edev) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); struct bnxt_ulp *ulp; int i = 0; - if (ulp_id >= BNXT_MAX_ULP) - return -EINVAL; - - ulp = &edev->ulp_tbl[ulp_id]; - if (!rcu_access_pointer(ulp->ulp_ops)) { - netdev_err(bp->dev, "ulp id %d not registered\n", ulp_id); - return -EINVAL; - } - if (ulp_id == BNXT_ROCE_ULP && ulp->msix_requested) - edev->en_ops->bnxt_free_msix(edev, ulp_id); + ulp = edev->ulp_tbl; + if (ulp->msix_requested) + edev->en_ops->bnxt_free_msix(edev); if (ulp->max_async_event_id) bnxt_hwrm_func_drv_rgtr(bp, NULL, 0, true); @@ -93,6 +78,8 @@ static int bnxt_unregister_dev(struct bnxt_en_dev *edev, unsigned int ulp_id) msleep(100); i++; } + kfree(ulp); + edev->ulp_tbl = NULL; return 0; } @@ -101,8 +88,8 @@ static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) struct bnxt_en_dev *edev = bp->edev; int num_msix, idx, i; - num_msix = edev->ulp_tbl[BNXT_ROCE_ULP].msix_requested; - idx = edev->ulp_tbl[BNXT_ROCE_ULP].msix_base; + num_msix = edev->ulp_tbl->msix_requested; + idx = edev->ulp_tbl->msix_base; for (i = 0; i < num_msix; i++) { ent[i].vector = bp->irq_tbl[idx + i].vector; ent[i].ring_idx = idx + i; @@ -116,8 +103,9 @@ static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) } } -static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, - struct bnxt_msix_entry *ent, int num_msix) +static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, + struct bnxt_msix_entry *ent, + int num_msix) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); @@ -127,13 +115,10 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, int total_vecs; int rc = 0; - if (ulp_id != BNXT_ROCE_ULP) - return -EINVAL; - if (!(bp->flags & BNXT_FLAG_USING_MSIX)) return -ENODEV; - if (edev->ulp_tbl[ulp_id].msix_requested) + if (edev->ulp_tbl->msix_requested) return -EAGAIN; max_cp_rings = bnxt_get_max_func_cp_rings(bp); @@ -149,8 +134,8 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, max_idx = min_t(int, bp->total_irqs, max_cp_rings); idx = max_idx - avail_msix; } - edev->ulp_tbl[ulp_id].msix_base = idx; - edev->ulp_tbl[ulp_id].msix_requested = avail_msix; + edev->ulp_tbl->msix_base = idx; + edev->ulp_tbl->msix_requested = avail_msix; hw_resc = &bp->hw_resc; total_vecs = idx + avail_msix; rtnl_lock(); @@ -165,7 +150,7 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, } rtnl_unlock(); if (rc) { - edev->ulp_tbl[ulp_id].msix_requested = 0; + edev->ulp_tbl->msix_requested = 0; return -EAGAIN; } @@ -174,25 +159,22 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, resv_msix = hw_resc->resv_irqs - bp->cp_nr_rings; avail_msix = min_t(int, resv_msix, avail_msix); - edev->ulp_tbl[ulp_id].msix_requested = avail_msix; + edev->ulp_tbl->msix_requested = avail_msix; } bnxt_fill_msix_vecs(bp, ent); edev->flags |= BNXT_EN_FLAG_MSIX_REQUESTED; return avail_msix; } -static int bnxt_free_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id) +static void bnxt_free_msix_vecs(struct bnxt_en_dev *edev) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); - if (ulp_id != BNXT_ROCE_ULP) - return -EINVAL; - if (!(edev->flags & BNXT_EN_FLAG_MSIX_REQUESTED)) - return 0; + return; - edev->ulp_tbl[ulp_id].msix_requested = 0; + edev->ulp_tbl->msix_requested = 0; edev->flags &= ~BNXT_EN_FLAG_MSIX_REQUESTED; rtnl_lock(); if (netif_running(dev) && !(edev->flags & BNXT_EN_FLAG_ULP_STOPPED)) { @@ -201,43 +183,43 @@ static int bnxt_free_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id) } rtnl_unlock(); - return 0; + return; } int bnxt_get_ulp_msix_num(struct bnxt *bp) { - if (bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) { + if (bnxt_ulp_registered(bp->edev)) { struct bnxt_en_dev *edev = bp->edev; - return edev->ulp_tbl[BNXT_ROCE_ULP].msix_requested; + return edev->ulp_tbl->msix_requested; } return 0; } int bnxt_get_ulp_msix_base(struct bnxt *bp) { - if (bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) { + if (bnxt_ulp_registered(bp->edev)) { struct bnxt_en_dev *edev = bp->edev; - if (edev->ulp_tbl[BNXT_ROCE_ULP].msix_requested) - return edev->ulp_tbl[BNXT_ROCE_ULP].msix_base; + if (edev->ulp_tbl->msix_requested) + return edev->ulp_tbl->msix_base; } return 0; } int bnxt_get_ulp_stat_ctxs(struct bnxt *bp) { - if (bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) { + if (bnxt_ulp_registered(bp->edev)) { struct bnxt_en_dev *edev = bp->edev; - if (edev->ulp_tbl[BNXT_ROCE_ULP].msix_requested) + if (edev->ulp_tbl->msix_requested) return BNXT_MIN_ROCE_STAT_CTXS; } return 0; } -static int bnxt_send_msg(struct bnxt_en_dev *edev, unsigned int ulp_id, +static int bnxt_send_msg(struct bnxt_en_dev *edev, struct bnxt_fw_msg *fw_msg) { struct net_device *dev = edev->net; @@ -247,7 +229,7 @@ static int bnxt_send_msg(struct bnxt_en_dev *edev, unsigned int ulp_id, u32 resp_len; int rc; - if (ulp_id != BNXT_ROCE_ULP && bp->fw_reset_state) + if (bp->fw_reset_state) return -EBUSY; rc = hwrm_req_init(bp, req, 0 /* don't care */); @@ -286,27 +268,24 @@ void bnxt_ulp_stop(struct bnxt *bp) { struct bnxt_en_dev *edev = bp->edev; struct bnxt_ulp_ops *ops; - int i; + struct bnxt_ulp *ulp; if (!edev) return; edev->flags |= BNXT_EN_FLAG_ULP_STOPPED; - for (i = 0; i < BNXT_MAX_ULP; i++) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[i]; - - ops = rtnl_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_stop) - continue; - ops->ulp_stop(ulp->handle); - } + ulp = edev->ulp_tbl; + ops = rtnl_dereference(ulp->ulp_ops); + if (!ops || !ops->ulp_stop) + return; + ops->ulp_stop(ulp->handle); } void bnxt_ulp_start(struct bnxt *bp, int err) { struct bnxt_en_dev *edev = bp->edev; struct bnxt_ulp_ops *ops; - int i; + struct bnxt_ulp *ulp; if (!edev) return; @@ -316,39 +295,33 @@ void bnxt_ulp_start(struct bnxt *bp, int err) if (err) return; - for (i = 0; i < BNXT_MAX_ULP; i++) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[i]; - - ops = rtnl_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_start) - continue; - ops->ulp_start(ulp->handle); - } + ulp = edev->ulp_tbl; + ops = rtnl_dereference(ulp->ulp_ops); + if (!ops || !ops->ulp_start) + return; + ops->ulp_start(ulp->handle); } void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs) { struct bnxt_en_dev *edev = bp->edev; struct bnxt_ulp_ops *ops; - int i; + struct bnxt_ulp *ulp; if (!edev) return; + ulp = edev->ulp_tbl; - for (i = 0; i < BNXT_MAX_ULP; i++) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[i]; - - rcu_read_lock(); - ops = rcu_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_sriov_config) { - rcu_read_unlock(); - continue; - } - bnxt_ulp_get(ulp); + rcu_read_lock(); + ops = rcu_dereference(ulp->ulp_ops); + if (!ops || !ops->ulp_sriov_config) { rcu_read_unlock(); - ops->ulp_sriov_config(ulp->handle, num_vfs); - bnxt_ulp_put(ulp); + return; } + bnxt_ulp_get(ulp); + rcu_read_unlock(); + ops->ulp_sriov_config(ulp->handle, num_vfs); + bnxt_ulp_put(ulp); } void bnxt_ulp_irq_stop(struct bnxt *bp) @@ -359,8 +332,8 @@ void bnxt_ulp_irq_stop(struct bnxt *bp) if (!edev || !(edev->flags & BNXT_EN_FLAG_MSIX_REQUESTED)) return; - if (bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[BNXT_ROCE_ULP]; + if (bnxt_ulp_registered(bp->edev)) { + struct bnxt_ulp *ulp = edev->ulp_tbl; if (!ulp->msix_requested) return; @@ -380,8 +353,8 @@ void bnxt_ulp_irq_restart(struct bnxt *bp, int err) if (!edev || !(edev->flags & BNXT_EN_FLAG_MSIX_REQUESTED)) return; - if (bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[BNXT_ROCE_ULP]; + if (bnxt_ulp_registered(bp->edev)) { + struct bnxt_ulp *ulp = edev->ulp_tbl; struct bnxt_msix_entry *ent = NULL; if (!ulp->msix_requested) @@ -408,41 +381,38 @@ void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl) u16 event_id = le16_to_cpu(cmpl->event_id); struct bnxt_en_dev *edev = bp->edev; struct bnxt_ulp_ops *ops; - int i; + struct bnxt_ulp *ulp; - if (!edev) + if (!bnxt_ulp_registered(edev)) return; + ulp = edev->ulp_tbl; + rcu_read_lock(); - for (i = 0; i < BNXT_MAX_ULP; i++) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[i]; - - ops = rcu_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_async_notifier) - continue; - if (!ulp->async_events_bmap || - event_id > ulp->max_async_event_id) - continue; - - /* Read max_async_event_id first before testing the bitmap. */ - smp_rmb(); - if (test_bit(event_id, ulp->async_events_bmap)) - ops->ulp_async_notifier(ulp->handle, cmpl); - } + + ops = rcu_dereference(ulp->ulp_ops); + if (!ops || !ops->ulp_async_notifier) + goto exit; + if (!ulp->async_events_bmap || event_id > ulp->max_async_event_id) + goto exit; + + /* Read max_async_event_id first before testing the bitmap. */ + smp_rmb(); + if (test_bit(event_id, ulp->async_events_bmap)) + ops->ulp_async_notifier(ulp->handle, cmpl); +exit: rcu_read_unlock(); } -static int bnxt_register_async_events(struct bnxt_en_dev *edev, unsigned int ulp_id, - unsigned long *events_bmap, u16 max_id) +static int bnxt_register_async_events(struct bnxt_en_dev *edev, + unsigned long *events_bmap, + u16 max_id) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); struct bnxt_ulp *ulp; - if (ulp_id >= BNXT_MAX_ULP) - return -EINVAL; - - ulp = &edev->ulp_tbl[ulp_id]; + ulp = edev->ulp_tbl; ulp->async_events_bmap = events_bmap; /* Make sure bnxt_ulp_async_events() sees this order */ smp_wmb(); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index 67fc3be2447a..aa4e57f670d8 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -64,7 +64,7 @@ struct bnxt_en_dev { #define BNXT_EN_FLAG_MSIX_REQUESTED 0x4 #define BNXT_EN_FLAG_ULP_STOPPED 0x8 const struct bnxt_en_ops *en_ops; - struct bnxt_ulp ulp_tbl[BNXT_MAX_ULP]; + struct bnxt_ulp *ulp_tbl; int l2_db_size; /* Doorbell BAR size in * bytes mapped by L2 * driver. @@ -76,21 +76,21 @@ struct bnxt_en_dev { }; struct bnxt_en_ops { - int (*bnxt_register_device)(struct bnxt_en_dev *, unsigned int, - struct bnxt_ulp_ops *, void *); - int (*bnxt_unregister_device)(struct bnxt_en_dev *, unsigned int); - int (*bnxt_request_msix)(struct bnxt_en_dev *, unsigned int, - struct bnxt_msix_entry *, int); - int (*bnxt_free_msix)(struct bnxt_en_dev *, unsigned int); - int (*bnxt_send_fw_msg)(struct bnxt_en_dev *, unsigned int, - struct bnxt_fw_msg *); - int (*bnxt_register_fw_async_events)(struct bnxt_en_dev *, unsigned int, - unsigned long *, u16); + int (*bnxt_register_device)(struct bnxt_en_dev *edev, + struct bnxt_ulp_ops *ulp_ops, void *handle); + int (*bnxt_unregister_device)(struct bnxt_en_dev *edev); + int (*bnxt_request_msix)(struct bnxt_en_dev *edev, + struct bnxt_msix_entry *ent, int num_msix); + void (*bnxt_free_msix)(struct bnxt_en_dev *edev); + int (*bnxt_send_fw_msg)(struct bnxt_en_dev *edev, + struct bnxt_fw_msg *fw_msg); + int (*bnxt_register_fw_async_events)(struct bnxt_en_dev *edev, + unsigned long *events_bmap, u16 max_id); }; -static inline bool bnxt_ulp_registered(struct bnxt_en_dev *edev, int ulp_id) +static inline bool bnxt_ulp_registered(struct bnxt_en_dev *edev) { - if (edev && rcu_access_pointer(edev->ulp_tbl[ulp_id].ulp_ops)) + if (edev && edev->ulp_tbl) return true; return false; } From patchwork Tue Jan 31 05:25:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 50608 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2569852wrn; Mon, 30 Jan 2023 21:28:00 -0800 (PST) X-Google-Smtp-Source: AK7set/TlsTpFxsPygsay5RZkEYQ9DjJmEndS+a4U9pHD9NUCxWR7wU+LCQmksyxRZJeurM9OOAi X-Received: by 2002:a17:906:cc8a:b0:878:72d0:2817 with SMTP id oq10-20020a170906cc8a00b0087872d02817mr18854224ejb.29.1675142880196; Mon, 30 Jan 2023 21:28:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675142880; cv=none; d=google.com; s=arc-20160816; b=YcuCoQV6hNWX6dpd/sx8QZek814K6OHOPnDSp+LZ+mDSzcRmCEQQxOS73W/sVcXW+F Z1/OiM1tBtQBJH9HXzB9fcj4t9IJPOlbR9SnuxwlbBg/plzsJXLouS1jdXo+M0sbHOUt bA4jqHzTHkuHxpsqDK48zuCANM2lcE3KwOwiSwU95QDE8CXHaQZxUJ9Tlb+ncDRkFYrp djsqjKfSYvQpZG70+8mNiX4g64IToRH9LX+O0GhZwvNLdrngnYFXp1k7kp6hSI37m8WC oT7Ac63TFsB5Fobr0AtCVPdRFVfcTBGO77o0p9Gn5RokNPSdv9cicv1rGv6T1REXBofy X8UQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=zgr4/y6fssqjW9EMrA3FfZ546pFG9E+/71WXPVEW4xs=; b=z16rSOLMqZrWWclMmsamDw7fpx7pUG3H7kDglDaDeP8c5d67+c61cpdSxgDur0i81+ 17wkcQEeTvp/tfU8995y1kHeSYHRFBfqJNHTqDICSvTDAFu40AwFv4eId1yy8lQcP8gx QCEOmfUG03sugm/Cgn4qVga07AridPmog74khFs/gjY/1e/eapHMYaHprVsoSVR2LEUi /XYuvg7XX+kAWq+wm87WtjHx4CDa+6ZAHYZ2rOnZQrBYUz91mtR7kO0hdf88v2MA6bgb Ib5TDtaFAQsXt10MEAzda6sl/4STNnNuKn5eWilO0jFwdHsabRxbtHChZl9s8ttbn+8j 3A/A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=Dn2lu9PU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fu35-20020a170907b02300b00878809a0f92si14102405ejc.541.2023.01.30.21.27.34; Mon, 30 Jan 2023 21:27:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=Dn2lu9PU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230342AbjAaF05 (ORCPT + 99 others); Tue, 31 Jan 2023 00:26:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36376 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230338AbjAaF0R (ORCPT ); Tue, 31 Jan 2023 00:26:17 -0500 Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B1CF3B671 for ; Mon, 30 Jan 2023 21:26:14 -0800 (PST) Received: by mail-pj1-x1031.google.com with SMTP id cq16-20020a17090af99000b0022c9791ac39so5486821pjb.4 for ; Mon, 30 Jan 2023 21:26:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=zgr4/y6fssqjW9EMrA3FfZ546pFG9E+/71WXPVEW4xs=; b=Dn2lu9PUvD/aGLYAytpcyhATeMZ44b0kGkzoUvaoFM/xOgT6ZgyX7//jo8ggQT456F GjBhaEWY913rZLO6tMITuqgDdkUiHFG1NEWXtgr17c6Y0OaIyRUSwpSA3gR/qqam363O X9xuywn2okN6dZxziGWYp1nr8YENuG/YV+0og= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zgr4/y6fssqjW9EMrA3FfZ546pFG9E+/71WXPVEW4xs=; b=0pTO3sMjEkM6u9prkAi7sJ8h5c9edqsHxGDwUhoVbPQWgUNKMFGCiqtkdMsGRisREu GMY//t5BRzdGEc9ZKHgflpQAUF3jBEn0JL0POHlrVhKb06jG6MVVfSIyUUdJdXPRVsXa QyT2FNSZCb6NlgkPJAMnQAiJpCMvlANy+FoKE56teFxx5w91mLsVftTwCdl4wvbuFtrY Xmyg1OmcUTYWoYlvcW58alDIiqCVMzr3qfnQhWc1MlmQCJgflEXGngeo4+TuSniuKhZ+ LLEoXCyz+xNlXNoQ2cTd8hXQnEDN8E6AzpvFprnfwWKAAtvJLOzHghF0I6tknCBsTd0D Edgw== X-Gm-Message-State: AO0yUKXUatgHeo6TZrraO2NNhFcnasGppNmy5s826+lieOO2u8Hd0m3H hYNScHxWLZnB+t2mhKqRnazH6g== X-Received: by 2002:a17:902:cacc:b0:189:b4d0:aee with SMTP id y12-20020a170902cacc00b00189b4d00aeemr8556429pld.67.1675142773211; Mon, 30 Jan 2023 21:26:13 -0800 (PST) Received: from localhost.localdomain ([2605:a601:a780:1400:a879:b64b:d9bd:3c1]) by smtp.gmail.com with ESMTPSA id jk15-20020a170903330f00b001960cccc318sm2310106plb.121.2023.01.30.21.26.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Jan 2023 21:26:12 -0800 (PST) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, gregkh@linuxfoundation.org, Leon Romanovsky Subject: [PATCH net-next v9 4/8] bnxt_en: Use direct API instead of indirection Date: Mon, 30 Jan 2023 21:25:53 -0800 Message-Id: <20230131052557.99119-5-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230131052557.99119-1-ajit.khaparde@broadcom.com> References: <20230131052557.99119-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756514620511126470?= X-GMAIL-MSGID: =?utf-8?q?1756514620511126470?= For a single ULP user there is no need for complicating function indirection calls. Remove all this complexity in favour of direct function calls exported by the bnxt_en driver. This allows to simplify the code greatly. Also remove unused ulp_async_notifier. Suggested-by: Leon Romanovsky Signed-off-by: Ajit Khaparde Reviewed-by: Andy Gospodarek Reviewed-by: Selvin Xavier Reviewed-by: Leon Romanovsky --- drivers/infiniband/hw/bnxt_re/main.c | 71 ++++------------- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 1 - drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 76 +++++-------------- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 25 +++--- 4 files changed, 45 insertions(+), 128 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 5bf3dc067484..fa64fe9887bf 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -350,23 +350,6 @@ static struct bnxt_ulp_ops bnxt_re_ulp_ops = { /* RoCE -> Net driver */ -/* Driver registration routines used to let the networking driver (bnxt_en) - * to know that the RoCE driver is now installed - */ -static int bnxt_re_unregister_netdev(struct bnxt_re_dev *rdev) -{ - struct bnxt_en_dev *en_dev; - int rc; - - if (!rdev) - return -EINVAL; - - en_dev = rdev->en_dev; - - rc = en_dev->en_ops->bnxt_unregister_device(rdev->en_dev); - return rc; -} - static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) { struct bnxt_en_dev *en_dev; @@ -374,26 +357,12 @@ static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) en_dev = rdev->en_dev; - rc = en_dev->en_ops->bnxt_register_device(en_dev, - &bnxt_re_ulp_ops, rdev); - rdev->qplib_res.pdev = rdev->en_dev->pdev; + rc = bnxt_register_dev(en_dev, &bnxt_re_ulp_ops, rdev); + if (!rc) + rdev->qplib_res.pdev = rdev->en_dev->pdev; return rc; } -static int bnxt_re_free_msix(struct bnxt_re_dev *rdev) -{ - struct bnxt_en_dev *en_dev; - - if (!rdev) - return -EINVAL; - - en_dev = rdev->en_dev; - - en_dev->en_ops->bnxt_free_msix(rdev->en_dev); - - return 0; -} - static int bnxt_re_request_msix(struct bnxt_re_dev *rdev) { int rc = 0, num_msix_want = BNXT_RE_MAX_MSIX, num_msix_got; @@ -403,9 +372,9 @@ static int bnxt_re_request_msix(struct bnxt_re_dev *rdev) num_msix_want = min_t(u32, BNXT_RE_MAX_MSIX, num_online_cpus()); - num_msix_got = en_dev->en_ops->bnxt_request_msix(en_dev, - rdev->msix_entries, - num_msix_want); + num_msix_got = bnxt_req_msix_vecs(en_dev, + rdev->msix_entries, + num_msix_want); if (num_msix_got < BNXT_RE_MIN_MSIX) { rc = -EINVAL; goto done; @@ -466,7 +435,7 @@ static int bnxt_re_net_ring_free(struct bnxt_re_dev *rdev, req.ring_id = cpu_to_le16(fw_ring_id); bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (rc) ibdev_err(&rdev->ibdev, "Failed to free HW ring:%d :%#x", req.ring_id, rc); @@ -503,7 +472,7 @@ static int bnxt_re_net_ring_alloc(struct bnxt_re_dev *rdev, req.int_mode = ring_attr->mode; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (!rc) *fw_ring_id = le16_to_cpu(resp.ring_id); @@ -531,7 +500,7 @@ static int bnxt_re_net_stats_ctx_free(struct bnxt_re_dev *rdev, req.stat_ctx_id = cpu_to_le32(fw_stats_ctx_id); bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (rc) ibdev_err(&rdev->ibdev, "Failed to free HW stats context %#x", rc); @@ -564,7 +533,7 @@ static int bnxt_re_net_stats_ctx_alloc(struct bnxt_re_dev *rdev, req.stat_ctx_flags = STAT_CTX_ALLOC_REQ_STAT_CTX_FLAGS_ROCE; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (!rc) *fw_stats_ctx_id = le32_to_cpu(resp.stat_ctx_id); @@ -1050,7 +1019,7 @@ static int bnxt_re_query_hwrm_pri2cos(struct bnxt_re_dev *rdev, u8 dir, bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (rc) return rc; @@ -1233,7 +1202,7 @@ static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) req.hwrm_intf_upd = HWRM_VERSION_UPDATE; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (rc) { ibdev_err(&rdev->ibdev, "Failed to query HW version, rc = 0x%x", rc); @@ -1296,20 +1265,12 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev) bnxt_re_net_ring_free(rdev, rdev->rcfw.creq.ring_id, type); bnxt_qplib_free_rcfw_channel(&rdev->rcfw); } - if (test_and_clear_bit(BNXT_RE_FLAG_GOT_MSIX, &rdev->flags)) { - rc = bnxt_re_free_msix(rdev); - if (rc) - ibdev_warn(&rdev->ibdev, - "Failed to free MSI-X vectors: %#x", rc); - } + if (test_and_clear_bit(BNXT_RE_FLAG_GOT_MSIX, &rdev->flags)) + bnxt_free_msix_vecs(rdev->en_dev); bnxt_re_destroy_chip_ctx(rdev); - if (test_and_clear_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags)) { - rc = bnxt_re_unregister_netdev(rdev); - if (rc) - ibdev_warn(&rdev->ibdev, - "Failed to unregister with netdev: %#x", rc); - } + if (test_and_clear_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags)) + bnxt_unregister_dev(rdev->en_dev); } /* worker thread for polling periodic events. Now used for QoS programming*/ diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 76c1382fc09f..06e7da57c371 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -2414,7 +2414,6 @@ static int bnxt_async_event_process(struct bnxt *bp, } bnxt_queue_sp_work(bp); async_event_process_exit: - bnxt_ulp_async_events(bp, cmpl); return 0; } diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index 5bf42fd296c1..d603fdaf0b67 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -28,9 +28,9 @@ static DEFINE_IDA(bnxt_aux_dev_ids); -static int bnxt_register_dev(struct bnxt_en_dev *edev, - struct bnxt_ulp_ops *ulp_ops, - void *handle) +int bnxt_register_dev(struct bnxt_en_dev *edev, + struct bnxt_ulp_ops *ulp_ops, + void *handle) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); @@ -55,8 +55,9 @@ static int bnxt_register_dev(struct bnxt_en_dev *edev, return 0; } +EXPORT_SYMBOL(bnxt_register_dev); -static int bnxt_unregister_dev(struct bnxt_en_dev *edev) +void bnxt_unregister_dev(struct bnxt_en_dev *edev) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); @@ -65,7 +66,7 @@ static int bnxt_unregister_dev(struct bnxt_en_dev *edev) ulp = edev->ulp_tbl; if (ulp->msix_requested) - edev->en_ops->bnxt_free_msix(edev); + bnxt_free_msix_vecs(edev); if (ulp->max_async_event_id) bnxt_hwrm_func_drv_rgtr(bp, NULL, 0, true); @@ -80,8 +81,9 @@ static int bnxt_unregister_dev(struct bnxt_en_dev *edev) } kfree(ulp); edev->ulp_tbl = NULL; - return 0; + return; } +EXPORT_SYMBOL(bnxt_unregister_dev); static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) { @@ -103,7 +105,7 @@ static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) } } -static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, +int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, struct bnxt_msix_entry *ent, int num_msix) { @@ -165,8 +167,9 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, edev->flags |= BNXT_EN_FLAG_MSIX_REQUESTED; return avail_msix; } +EXPORT_SYMBOL(bnxt_req_msix_vecs); -static void bnxt_free_msix_vecs(struct bnxt_en_dev *edev) +void bnxt_free_msix_vecs(struct bnxt_en_dev *edev) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); @@ -185,6 +188,7 @@ static void bnxt_free_msix_vecs(struct bnxt_en_dev *edev) return; } +EXPORT_SYMBOL(bnxt_free_msix_vecs); int bnxt_get_ulp_msix_num(struct bnxt *bp) { @@ -219,7 +223,7 @@ int bnxt_get_ulp_stat_ctxs(struct bnxt *bp) return 0; } -static int bnxt_send_msg(struct bnxt_en_dev *edev, +int bnxt_send_msg(struct bnxt_en_dev *edev, struct bnxt_fw_msg *fw_msg) { struct net_device *dev = edev->net; @@ -253,6 +257,7 @@ static int bnxt_send_msg(struct bnxt_en_dev *edev, hwrm_req_drop(bp, req); return rc; } +EXPORT_SYMBOL(bnxt_send_msg); static void bnxt_ulp_get(struct bnxt_ulp *ulp) { @@ -312,14 +317,11 @@ void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs) return; ulp = edev->ulp_tbl; - rcu_read_lock(); ops = rcu_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_sriov_config) { - rcu_read_unlock(); + if (!ops || !ops->ulp_sriov_config) return; - } + bnxt_ulp_get(ulp); - rcu_read_unlock(); ops->ulp_sriov_config(ulp->handle, num_vfs); bnxt_ulp_put(ulp); } @@ -376,37 +378,9 @@ void bnxt_ulp_irq_restart(struct bnxt *bp, int err) } } -void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl) -{ - u16 event_id = le16_to_cpu(cmpl->event_id); - struct bnxt_en_dev *edev = bp->edev; - struct bnxt_ulp_ops *ops; - struct bnxt_ulp *ulp; - - if (!bnxt_ulp_registered(edev)) - return; - - ulp = edev->ulp_tbl; - - rcu_read_lock(); - - ops = rcu_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_async_notifier) - goto exit; - if (!ulp->async_events_bmap || event_id > ulp->max_async_event_id) - goto exit; - - /* Read max_async_event_id first before testing the bitmap. */ - smp_rmb(); - if (test_bit(event_id, ulp->async_events_bmap)) - ops->ulp_async_notifier(ulp->handle, cmpl); -exit: - rcu_read_unlock(); -} - -static int bnxt_register_async_events(struct bnxt_en_dev *edev, - unsigned long *events_bmap, - u16 max_id) +int bnxt_register_async_events(struct bnxt_en_dev *edev, + unsigned long *events_bmap, + u16 max_id) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); @@ -420,15 +394,7 @@ static int bnxt_register_async_events(struct bnxt_en_dev *edev, bnxt_hwrm_func_drv_rgtr(bp, events_bmap, max_id + 1, true); return 0; } - -static const struct bnxt_en_ops bnxt_en_ops_tbl = { - .bnxt_register_device = bnxt_register_dev, - .bnxt_unregister_device = bnxt_unregister_dev, - .bnxt_request_msix = bnxt_req_msix_vecs, - .bnxt_free_msix = bnxt_free_msix_vecs, - .bnxt_send_fw_msg = bnxt_send_msg, - .bnxt_register_fw_async_events = bnxt_register_async_events, -}; +EXPORT_SYMBOL(bnxt_register_async_events); void bnxt_rdma_aux_device_uninit(struct bnxt *bp) { @@ -452,7 +418,6 @@ static void bnxt_aux_dev_release(struct device *dev) struct bnxt *bp = netdev_priv(aux_priv->edev->net); ida_free(&bnxt_aux_dev_ids, aux_priv->id); - aux_priv->edev->en_ops = NULL; kfree(aux_priv->edev); aux_priv->edev = NULL; bp->edev = NULL; @@ -461,7 +426,6 @@ static void bnxt_aux_dev_release(struct device *dev) static void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp) { - edev->en_ops = &bnxt_en_ops_tbl; edev->net = bp->dev; edev->pdev = bp->pdev; edev->l2_db_size = bp->db_size; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index aa4e57f670d8..6ce2a1893bcc 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -26,8 +26,6 @@ struct bnxt_msix_entry { }; struct bnxt_ulp_ops { - /* async_notifier() cannot sleep (in BH context) */ - void (*ulp_async_notifier)(void *, struct hwrm_async_event_cmpl *); void (*ulp_stop)(void *); void (*ulp_start)(void *); void (*ulp_sriov_config)(void *, int); @@ -63,7 +61,6 @@ struct bnxt_en_dev { BNXT_EN_FLAG_ROCEV2_CAP) #define BNXT_EN_FLAG_MSIX_REQUESTED 0x4 #define BNXT_EN_FLAG_ULP_STOPPED 0x8 - const struct bnxt_en_ops *en_ops; struct bnxt_ulp *ulp_tbl; int l2_db_size; /* Doorbell BAR size in * bytes mapped by L2 @@ -75,19 +72,6 @@ struct bnxt_en_dev { */ }; -struct bnxt_en_ops { - int (*bnxt_register_device)(struct bnxt_en_dev *edev, - struct bnxt_ulp_ops *ulp_ops, void *handle); - int (*bnxt_unregister_device)(struct bnxt_en_dev *edev); - int (*bnxt_request_msix)(struct bnxt_en_dev *edev, - struct bnxt_msix_entry *ent, int num_msix); - void (*bnxt_free_msix)(struct bnxt_en_dev *edev); - int (*bnxt_send_fw_msg)(struct bnxt_en_dev *edev, - struct bnxt_fw_msg *fw_msg); - int (*bnxt_register_fw_async_events)(struct bnxt_en_dev *edev, - unsigned long *events_bmap, u16 max_id); -}; - static inline bool bnxt_ulp_registered(struct bnxt_en_dev *edev) { if (edev && edev->ulp_tbl) @@ -106,4 +90,13 @@ void bnxt_ulp_irq_restart(struct bnxt *bp, int err); void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl); void bnxt_rdma_aux_device_uninit(struct bnxt *bp); void bnxt_rdma_aux_device_init(struct bnxt *bp); +int bnxt_register_dev(struct bnxt_en_dev *edev, struct bnxt_ulp_ops *ulp_ops, + void *handle); +void bnxt_unregister_dev(struct bnxt_en_dev *edev); +int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, struct bnxt_msix_entry *ent, + int num_msix); +void bnxt_free_msix_vecs(struct bnxt_en_dev *edev); +int bnxt_send_msg(struct bnxt_en_dev *edev, struct bnxt_fw_msg *fw_msg); +int bnxt_register_async_events(struct bnxt_en_dev *edev, + unsigned long *events_bmap, u16 max_id); #endif From patchwork Tue Jan 31 05:25:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 50609 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2569990wrn; Mon, 30 Jan 2023 21:28:25 -0800 (PST) X-Google-Smtp-Source: AK7set9AAX80cI1qGhytys6Hvfn8PT3jrg1WiVl0LaWyXTl3irs6DtABIMuROisV1lKOn8JIEzji X-Received: by 2002:a05:6402:481:b0:4a0:b554:c26c with SMTP id k1-20020a056402048100b004a0b554c26cmr23336509edv.21.1675142905354; Mon, 30 Jan 2023 21:28:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675142905; cv=none; d=google.com; s=arc-20160816; b=Lcs/XvP11u+OLJkTocDPRFPkNH7Hjdvr8Xum4IHvePqhdCF23Hsjyh3KqPcimDrL0F 6Gvwgrv2gZXqU59BywzFh83HvETIISe3c7aoscxV7s7fqYvovWwiivJsNEhojP77iOyG ETMfo3JcYkAKGXZORpYvM6K/TDNFeAf3fujTGEu+wHfxJM/pVcj7KTQuwhiMzCjOrSAy tKbh+XoZrdAwes5TmC21hhYRDyNVq04if6d2N5FIAHPAk0EjlWuxGUdxHdDcM2Mn/tKU TYeAyYXfVLlyj4V3a0tMSkTMDDyEbCEhl5jtaH33mS45bXhvEuEjUwLWdt9005i1Rzl+ E+jA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=l2qhc26UpWeKqU5hsi6nPrKsAc2Z26BNiIPtNMD/IeU=; b=webpyHd+zxfFMzcWzy4yZFsJ7dD333lIscmxikOzCv5HQ5jJ3jDPshGJoeZg6/xEI9 1KP/copMoBA67oWnWWR4wgvkGvHONA7ETMte67QGM+nNTbuh9jfVrqh67c8I11NXMRY7 SO7YFoh1Lnn9GGsV4zPNQ/l6/dumr8lR0BbD37VnII6wvAT4GP3kkGxD2mRZ98C8J2AF lvczlLDqqhE95XKRlRfunYAfrkQkChlVaZtQfgpfasEqb2TV8+TFbrVc+HtqY2AApShW OOgLvd0BgPcFWoDl/TJ4tRc3j1fHD8ieIavUfhzlB6PGFgwvY9RbkBRxHn2xDwvcS4r0 zr9w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=fCjNI6oK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id jr24-20020a170906515800b008692b15cca1si14441963ejc.603.2023.01.30.21.28.01; Mon, 30 Jan 2023 21:28:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=fCjNI6oK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230371AbjAaF1A (ORCPT + 99 others); Tue, 31 Jan 2023 00:27:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36408 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230360AbjAaF0R (ORCPT ); Tue, 31 Jan 2023 00:26:17 -0500 Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6EEFC3CE22 for ; Mon, 30 Jan 2023 21:26:15 -0800 (PST) Received: by mail-pl1-x62d.google.com with SMTP id p24so13935763plw.11 for ; Mon, 30 Jan 2023 21:26:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=l2qhc26UpWeKqU5hsi6nPrKsAc2Z26BNiIPtNMD/IeU=; b=fCjNI6oKKtH6XKV6E6psi1000fVOiojLPBm/m/LxVHSgzmj7TICvBCCYxE8n/vHfcV /2/73hOoQQFDlOrfAV13y220XotR505x3d9DGdiCy+bTjuTjNaqadTb4vig5Oc7OyCKj uN2q3cFrc3dVbMl0Pub/FGR99Z3jxPQPgAjAs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=l2qhc26UpWeKqU5hsi6nPrKsAc2Z26BNiIPtNMD/IeU=; b=Gev60muSsJa+RveXSzEIW8ojsmQv8of294U27YlDsBa1vtBLa6fyduMMneNUgK5nSv oCYWKfVupiLJKtEeSX5AGSW1Gz4uz+8do/jsFXlmgee8vRr4+v7486o8HRRG/ViOHt47 Vm37XQGq72FrmGkGlbNjRDRUgt9CBi9EbFcV22R7/z/VjGKPogYmyFcpuE1vsuM74otO MdaFLs6kgP7CUtjCdJJjLkHPzyCQcgXQ1KqtAIuf7+cZBFdfwewTA+Jyb8mgPehRdbTT kzjPRkE7gM30tS8aKDw94utSkQK50OkNEIm50N19/Y04c5VEXylB3jsnVyFrmv6sGG+t 1G7A== X-Gm-Message-State: AO0yUKUlIHhm+N6e7pPyqvZcxcz6ida5YyX8xA2G2c4pIniPq93SSHAs m33xQlMxJEvJ/ZMR81r7iR8Yj3F5jraYLFxs X-Received: by 2002:a17:902:ce87:b0:196:341b:ed7 with SMTP id f7-20020a170902ce8700b00196341b0ed7mr25870256plg.15.1675142774690; Mon, 30 Jan 2023 21:26:14 -0800 (PST) Received: from localhost.localdomain ([2605:a601:a780:1400:a879:b64b:d9bd:3c1]) by smtp.gmail.com with ESMTPSA id jk15-20020a170903330f00b001960cccc318sm2310106plb.121.2023.01.30.21.26.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Jan 2023 21:26:14 -0800 (PST) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, gregkh@linuxfoundation.org, Leon Romanovsky Subject: [PATCH net-next v9 5/8] bnxt_en: Use auxiliary bus calls over proprietary calls Date: Mon, 30 Jan 2023 21:25:54 -0800 Message-Id: <20230131052557.99119-6-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230131052557.99119-1-ajit.khaparde@broadcom.com> References: <20230131052557.99119-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756514646760064499?= X-GMAIL-MSGID: =?utf-8?q?1756514646760064499?= Wherever possible use the function ops provided by auxiliary bus instead of using proprietary ops. Defined bnxt_re_suspend and bnxt_re_resume calls which can be invoked by the bnxt_en driver instead of the ULP stop/start calls. Signed-off-by: Ajit Khaparde Reviewed-by: Andy Gospodarek Reviewed-by: Selvin Xavier Reviewed-by: Leon Romanovsky --- drivers/infiniband/hw/bnxt_re/main.c | 102 +++++++++++------- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 40 ++++--- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 2 - 3 files changed, 87 insertions(+), 57 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index fa64fe9887bf..ffcd2a4d9f9a 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -226,45 +226,6 @@ static void bnxt_re_set_resource_limits(struct bnxt_re_dev *rdev) bnxt_re_limit_vf_res(&rdev->qplib_ctx, num_vfs); } -/* for handling bnxt_en callbacks later */ -static void bnxt_re_stop(void *p) -{ - struct bnxt_re_dev *rdev = p; - struct bnxt *bp; - - if (!rdev) - return; - - /* L2 driver invokes this callback during device error/crash or device - * reset. Current RoCE driver doesn't recover the device in case of - * error. Handle the error by dispatching fatal events to all qps - * ie. by calling bnxt_re_dev_stop and release the MSIx vectors as - * L2 driver want to modify the MSIx table. - */ - bp = netdev_priv(rdev->netdev); - - ibdev_info(&rdev->ibdev, "Handle device stop call from L2 driver"); - /* Check the current device state from L2 structure and move the - * device to detached state if FW_FATAL_COND is set. - * This prevents more commands to HW during clean-up, - * in case the device is already in error. - */ - if (test_bit(BNXT_STATE_FW_FATAL_COND, &bp->state)) - set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); - - bnxt_re_dev_stop(rdev); - bnxt_re_stop_irq(rdev); - /* Move the device states to detached and avoid sending any more - * commands to HW - */ - set_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags); - set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); -} - -static void bnxt_re_start(void *p) -{ -} - static void bnxt_re_sriov_config(void *p, int num_vfs) { struct bnxt_re_dev *rdev = p; @@ -341,8 +302,6 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) } static struct bnxt_ulp_ops bnxt_re_ulp_ops = { - .ulp_stop = bnxt_re_stop, - .ulp_start = bnxt_re_start, .ulp_sriov_config = bnxt_re_sriov_config, .ulp_irq_stop = bnxt_re_stop_irq, .ulp_irq_restart = bnxt_re_start_irq @@ -1585,6 +1544,65 @@ static int bnxt_re_probe(struct auxiliary_device *adev, return rc; } +static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state) +{ + struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); + struct bnxt *bp; + + if (!rdev) + return 0; + + mutex_lock(&bnxt_re_mutex); + /* L2 driver may invoke this callback during device error/crash or device + * reset. Current RoCE driver doesn't recover the device in case of + * error. Handle the error by dispatching fatal events to all qps + * ie. by calling bnxt_re_dev_stop and release the MSIx vectors as + * L2 driver want to modify the MSIx table. + */ + bp = netdev_priv(rdev->netdev); + + ibdev_info(&rdev->ibdev, "Handle device suspend call"); + /* Check the current device state from L2 structure and move the + * device to detached state if FW_FATAL_COND is set. + * This prevents more commands to HW during clean-up, + * in case the device is already in error. + */ + if (test_bit(BNXT_STATE_FW_FATAL_COND, &bp->state)) + set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); + + bnxt_re_dev_stop(rdev); + bnxt_re_stop_irq(rdev); + /* Move the device states to detached and avoid sending any more + * commands to HW + */ + set_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags); + set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); + mutex_unlock(&bnxt_re_mutex); + + return 0; +} + +static int bnxt_re_resume(struct auxiliary_device *adev) +{ + struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); + + if (!rdev) + return 0; + + mutex_lock(&bnxt_re_mutex); + /* L2 driver may invoke this callback during device recovery, resume. + * reset. Current RoCE driver doesn't recover the device in case of + * error. Handle the error by dispatching fatal events to all qps + * ie. by calling bnxt_re_dev_stop and release the MSIx vectors as + * L2 driver want to modify the MSIx table. + */ + + ibdev_info(&rdev->ibdev, "Handle device resume call"); + mutex_unlock(&bnxt_re_mutex); + + return 0; +} + static const struct auxiliary_device_id bnxt_re_id_table[] = { { .name = BNXT_ADEV_NAME ".rdma", }, {}, @@ -1597,6 +1615,8 @@ static struct auxiliary_driver bnxt_re_driver = { .probe = bnxt_re_probe, .remove = bnxt_re_remove, .shutdown = bnxt_re_shutdown, + .suspend = bnxt_re_suspend, + .resume = bnxt_re_resume, .id_table = bnxt_re_id_table, }; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index d603fdaf0b67..f6d348b36e13 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -271,26 +271,31 @@ static void bnxt_ulp_put(struct bnxt_ulp *ulp) void bnxt_ulp_stop(struct bnxt *bp) { + struct bnxt_aux_priv *aux_priv = bp->aux_priv; struct bnxt_en_dev *edev = bp->edev; - struct bnxt_ulp_ops *ops; - struct bnxt_ulp *ulp; if (!edev) return; edev->flags |= BNXT_EN_FLAG_ULP_STOPPED; - ulp = edev->ulp_tbl; - ops = rtnl_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_stop) - return; - ops->ulp_stop(ulp->handle); + if (aux_priv) { + struct auxiliary_device *adev; + + adev = &aux_priv->aux_dev; + if (adev->dev.driver) { + struct auxiliary_driver *adrv; + pm_message_t pm = {}; + + adrv = to_auxiliary_drv(adev->dev.driver); + adrv->suspend(adev, pm); + } + } } void bnxt_ulp_start(struct bnxt *bp, int err) { + struct bnxt_aux_priv *aux_priv = bp->aux_priv; struct bnxt_en_dev *edev = bp->edev; - struct bnxt_ulp_ops *ops; - struct bnxt_ulp *ulp; if (!edev) return; @@ -300,11 +305,18 @@ void bnxt_ulp_start(struct bnxt *bp, int err) if (err) return; - ulp = edev->ulp_tbl; - ops = rtnl_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_start) - return; - ops->ulp_start(ulp->handle); + if (aux_priv) { + struct auxiliary_device *adev; + + adev = &aux_priv->aux_dev; + if (adev->dev.driver) { + struct auxiliary_driver *adrv; + + adrv = to_auxiliary_drv(adev->dev.driver); + adrv->resume(adev); + } + } + } void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index 6ce2a1893bcc..adc5a9a235c2 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -26,8 +26,6 @@ struct bnxt_msix_entry { }; struct bnxt_ulp_ops { - void (*ulp_stop)(void *); - void (*ulp_start)(void *); void (*ulp_sriov_config)(void *, int); void (*ulp_irq_stop)(void *); void (*ulp_irq_restart)(void *, struct bnxt_msix_entry *); From patchwork Tue Jan 31 05:25:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 50610 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2570291wrn; Mon, 30 Jan 2023 21:29:36 -0800 (PST) X-Google-Smtp-Source: AK7set9iL5FmwG5Ezrg/egH3LkGyr15sIC8ugVVY+zm+uaO8Ei2npRdqz3PgaXXnFAzyJCnCJ+ab X-Received: by 2002:a17:907:7f25:b0:86c:13ca:9c90 with SMTP id qf37-20020a1709077f2500b0086c13ca9c90mr2590090ejc.59.1675142975827; Mon, 30 Jan 2023 21:29:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675142975; cv=none; d=google.com; s=arc-20160816; b=OLl3fcAlzcot1O45rinPfNIkBvlilaXwejx0iD856YI7nJJ+EUwMA93W8472BeRhz1 Faa5lTX27Lq1Ijpqv0+Pm6+hpK1mE/LCamvJRUUBm8n6AtDBdERYUEg/MjEFRYAVFMkL B7D+eUWZHZPtkd9xQYk3dRoRwNNFkax1ZDT7wDkW63hZWMfTvB+Q9f8Pqkkx73xM6wkU mBFFgefx2psi6zmAU07GJfK6wO9ZQsMbfVagNdhcROOxSUE+YGg/DVGAPdc//hyIGzjE m6XZIzXlAaGeMRn8UZnfH5pPSDvQ74UZgEaGXLGPuEJrAJxhb5fY3DARoPQ/PselO3y/ CO0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=RuQKO5YrxEySf3cSL8zFtjcpYNL+1zV367ctDrGihJU=; b=0BYtqagpLzzZONy3XW40ACKY0yoMEuAfktijyagkyGK3oAk6oaR28DR/QjI0EoyUH9 Rcjs7UA8Abjm14lnPpmKx8vAbtQV4t8DGo8q38BxvnrIwj+YO91RGy8/GFkvH4pet8Jn 6GNzXVJKsQ3fmA8POpl0w8tCisqb/RlZgjynPe+HJd6Yfr1MySw4F8JxTEv9ZgaOpcOT MIe3MzkqQSbNa+UyB+eItaEQio5GpXV88/F+F+5gKXzqQnVp90/d9Bcn+1giBXqJkIkT MVl9KF6bwjRoDSTLEH73J+6CQkW6k6veARWUQoisGdw+8E6IE5oQRLYU2CKI6j39H8BC HqnA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=OsJn0pdd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ep6-20020a1709069b4600b008788321c868si14178233ejc.917.2023.01.30.21.29.11; Mon, 30 Jan 2023 21:29:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=OsJn0pdd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231295AbjAaF1Q (ORCPT + 99 others); Tue, 31 Jan 2023 00:27:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230404AbjAaF0T (ORCPT ); Tue, 31 Jan 2023 00:26:19 -0500 Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C79F23D923 for ; Mon, 30 Jan 2023 21:26:16 -0800 (PST) Received: by mail-pl1-x62e.google.com with SMTP id r8so7103905pls.2 for ; Mon, 30 Jan 2023 21:26:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=RuQKO5YrxEySf3cSL8zFtjcpYNL+1zV367ctDrGihJU=; b=OsJn0pddQzjFJwGsVVpODkYMoVKnuwsgW5uzgICytkCUMGG0QcSn9y/5vuUD0Tv1BV mHeY46OP+XZZulg+gwj+7Ryy1u/Cn68B0DEJYnU9rT0dWjVJI0SCmDwx8xp6SJYvKRBX fKttDE8RdwhEuMvUQ03pozt3ohBV2NqGATgaA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RuQKO5YrxEySf3cSL8zFtjcpYNL+1zV367ctDrGihJU=; b=C5pV0nHwOXJqM3t9X7xk/eycSiTebDcHm6H0WVa+R+HIG/Cj2qzCgqu7S1Enda2c3X DKe+DDR/47QJF/Y3ne9DZEeZLF/kPn5oOoDoiXplYQcHmQRjowWw9/4C+H+FJO/apnEA X47D4Rvy1WQEereV1PuUXYQMx2bCbuXVRHvZK3VowLvXH/k8p9DnnUpyR+IVJCcwOpp/ jpZ5d9tFJkfX8lOt5wv2bLOUiZdcfeYFsnx0tfKK2wvx2MaahCnv5Wz1OqqbnwIVshh0 sIhA+k54KxrubJMYHs95ULomtN/4P7kqZkzPxYhQHaShoWcXj9XIhS8MQnq4kD8Y6a96 Vp7A== X-Gm-Message-State: AO0yUKVK2NvWvgKYYYQ8hezasyU3J9R9L8is/6Zt6m5KPCcVHuYgAP+2 DNk3hSf6RsNT3qd4r/R/cJEAIQ== X-Received: by 2002:a17:902:d48d:b0:196:4f0d:c31f with SMTP id c13-20020a170902d48d00b001964f0dc31fmr20226420plg.12.1675142776386; Mon, 30 Jan 2023 21:26:16 -0800 (PST) Received: from localhost.localdomain ([2605:a601:a780:1400:a879:b64b:d9bd:3c1]) by smtp.gmail.com with ESMTPSA id jk15-20020a170903330f00b001960cccc318sm2310106plb.121.2023.01.30.21.26.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Jan 2023 21:26:15 -0800 (PST) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, gregkh@linuxfoundation.org, Hongguang Gao , Leon Romanovsky Subject: [PATCH net-next v9 6/8] bnxt_en: Remove struct bnxt access from RoCE driver Date: Mon, 30 Jan 2023 21:25:55 -0800 Message-Id: <20230131052557.99119-7-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230131052557.99119-1-ajit.khaparde@broadcom.com> References: <20230131052557.99119-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756514720801673508?= X-GMAIL-MSGID: =?utf-8?q?1756514720801673508?= From: Hongguang Gao Decouple RoCE driver from directly accessing L2's private bnxt structure. Move the fields needed by RoCE driver into bnxt_en_dev. They'll be passed to RoCE driver by bnxt_rdma_aux_device_add() function. Signed-off-by: Hongguang Gao Signed-off-by: Ajit Khaparde Reviewed-by: Andy Gospodarek Reviewed-by: Selvin Xavier Reviewed-by: Leon Romanovsky --- drivers/infiniband/hw/bnxt_re/main.c | 22 ++++++------------- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 9 ++++++++ drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 11 ++++++++++ 3 files changed, 27 insertions(+), 15 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index ffcd2a4d9f9a..ed7ac6acaaff 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -112,16 +112,14 @@ static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev, u8 wqe_mode) { struct bnxt_qplib_chip_ctx *chip_ctx; struct bnxt_en_dev *en_dev; - struct bnxt *bp; en_dev = rdev->en_dev; - bp = netdev_priv(en_dev->net); chip_ctx = kzalloc(sizeof(*chip_ctx), GFP_KERNEL); if (!chip_ctx) return -ENOMEM; - chip_ctx->chip_num = bp->chip_num; - chip_ctx->hw_stats_size = bp->hw_ring_stats_size; + chip_ctx->chip_num = en_dev->chip_num; + chip_ctx->hw_stats_size = en_dev->hw_ring_stats_size; rdev->chip_ctx = chip_ctx; /* rest members to follow eventually */ @@ -129,7 +127,7 @@ static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev, u8 wqe_mode) rdev->qplib_res.cctx = rdev->chip_ctx; rdev->rcfw.res = &rdev->qplib_res; rdev->qplib_res.dattr = &rdev->dev_attr; - rdev->qplib_res.is_vf = BNXT_VF(bp); + rdev->qplib_res.is_vf = BNXT_EN_VF(en_dev); bnxt_re_set_drv_mode(rdev, wqe_mode); if (bnxt_qplib_determine_atomics(en_dev->pdev)) @@ -142,10 +140,7 @@ static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev, u8 wqe_mode) static void bnxt_re_get_sriov_func_type(struct bnxt_re_dev *rdev) { - struct bnxt *bp; - - bp = netdev_priv(rdev->en_dev->net); - if (BNXT_VF(bp)) + if (BNXT_EN_VF(rdev->en_dev)) rdev->is_virtfn = 1; } @@ -957,7 +952,6 @@ static int bnxt_re_query_hwrm_pri2cos(struct bnxt_re_dev *rdev, u8 dir, u64 *cid_map) { struct hwrm_queue_pri2cos_qcfg_input req = {0}; - struct bnxt *bp = netdev_priv(rdev->netdev); struct hwrm_queue_pri2cos_qcfg_output resp; struct bnxt_en_dev *en_dev = rdev->en_dev; struct bnxt_fw_msg fw_msg; @@ -974,7 +968,7 @@ static int bnxt_re_query_hwrm_pri2cos(struct bnxt_re_dev *rdev, u8 dir, flags |= (dir & 0x01); flags |= HWRM_QUEUE_PRI2COS_QCFG_INPUT_FLAGS_IVLAN; req.flags = cpu_to_le32(flags); - req.port_id = bp->pf.port_id; + req.port_id = en_dev->pf_port_id; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); @@ -1547,7 +1541,6 @@ static int bnxt_re_probe(struct auxiliary_device *adev, static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state) { struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); - struct bnxt *bp; if (!rdev) return 0; @@ -1559,15 +1552,14 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state) * ie. by calling bnxt_re_dev_stop and release the MSIx vectors as * L2 driver want to modify the MSIx table. */ - bp = netdev_priv(rdev->netdev); ibdev_info(&rdev->ibdev, "Handle device suspend call"); - /* Check the current device state from L2 structure and move the + /* Check the current device state from bnxt_en_dev and move the * device to detached state if FW_FATAL_COND is set. * This prevents more commands to HW during clean-up, * in case the device is already in error. */ - if (test_bit(BNXT_STATE_FW_FATAL_COND, &bp->state)) + if (test_bit(BNXT_STATE_FW_FATAL_COND, &rdev->en_dev->en_state)) set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); bnxt_re_dev_stop(rdev); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index f6d348b36e13..fc5ded5d0d09 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -287,6 +287,7 @@ void bnxt_ulp_stop(struct bnxt *bp) pm_message_t pm = {}; adrv = to_auxiliary_drv(adev->dev.driver); + edev->en_state = bp->state; adrv->suspend(adev, pm); } } @@ -313,6 +314,7 @@ void bnxt_ulp_start(struct bnxt *bp, int err) struct auxiliary_driver *adrv; adrv = to_auxiliary_drv(adev->dev.driver); + edev->en_state = bp->state; adrv->resume(adev); } } @@ -447,6 +449,13 @@ static void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp) edev->flags |= BNXT_EN_FLAG_ROCEV1_CAP; if (bp->flags & BNXT_FLAG_ROCEV2_CAP) edev->flags |= BNXT_EN_FLAG_ROCEV2_CAP; + if (bp->flags & BNXT_FLAG_VF) + edev->flags |= BNXT_EN_FLAG_VF; + + edev->chip_num = bp->chip_num; + edev->hw_ring_stats_size = bp->hw_ring_stats_size; + edev->pf_port_id = bp->pf.port_id; + edev->en_state = bp->state; } void bnxt_rdma_aux_device_init(struct bnxt *bp) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index adc5a9a235c2..c62986e4cc82 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -59,6 +59,9 @@ struct bnxt_en_dev { BNXT_EN_FLAG_ROCEV2_CAP) #define BNXT_EN_FLAG_MSIX_REQUESTED 0x4 #define BNXT_EN_FLAG_ULP_STOPPED 0x8 + #define BNXT_EN_FLAG_VF 0x10 +#define BNXT_EN_VF(edev) ((edev)->flags & BNXT_EN_FLAG_VF) + struct bnxt_ulp *ulp_tbl; int l2_db_size; /* Doorbell BAR size in * bytes mapped by L2 @@ -68,6 +71,14 @@ struct bnxt_en_dev { * bytes mapped as non- * cacheable. */ + u16 chip_num; + u16 hw_ring_stats_size; + u16 pf_port_id; + unsigned long en_state; /* Could be checked in + * RoCE driver suspend + * mode only. Will be + * updated in resume. + */ }; static inline bool bnxt_ulp_registered(struct bnxt_en_dev *edev) From patchwork Tue Jan 31 05:25:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 50611 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2570297wrn; Mon, 30 Jan 2023 21:29:37 -0800 (PST) X-Google-Smtp-Source: AK7set/MN03Xs/SM4RsLh8/L4wQQK+c6tppZEYwok7jC0cHjePuSzj/HR6rU4brhFz8jY7MulHE6 X-Received: by 2002:a17:906:6db:b0:87b:d597:1fd5 with SMTP id v27-20020a17090606db00b0087bd5971fd5mr17329009ejb.75.1675142977275; Mon, 30 Jan 2023 21:29:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675142977; cv=none; d=google.com; s=arc-20160816; b=wWGs19AiiqVpJcGiK0wxsb3EDqmYc+JgDwRhq9LeRnrM+zmVir9goSsrIQ4P1ARuTx wODXQxjeExMUgzGU1vvioyUWCivdZiwUELEKdh2jMO9qJnF0mNLoSLIUKwvPSv1vDF8C JYnjGH4jRkefgyqR3yYr5cP8JXtWXisUTQwmp+EuCkyUQagHJAi9sI0Xw/NtIKww3B9O sBe8iPboTwAEpwcIadQf4NChILE8XhTjoqBCV46ACbDT0v5+LZpHZm9JhnuI5a3WWyzS VAtSIbFXJnOtLE+qBqHCbSTN/b6IryM1BhRP/ivuY2FFpnGqQMyGHenyitFTdfA6u4S4 Kqvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=+O8KWCroo2161P9B9g6JCettFGAqWNsTAzgVxWSxUro=; b=OnIfdS7iKmhIFEnZxT1crzzaO7e5BROgTSijJ8ptNp+G5ojTTMCbdAv7vWADrqmqMx BXjHgKMoJUjsCDzL1MqM8KEcT28hWZcQLn3yxHn+Kft4J2gq5Gu0u0y2kv1iNmTCbx8y bNWvDt5VLF+VaG4YvNjOTpbBE4tmYyNFyzTAJ4OVt3/t8qVLL59CO8QRwV//sc6xoqNQ tkMSM8TvHlsb7xBbyncvMwsCoLK9K0FtvzMXD9yPf+29/PHEorrRdAR2tyOGmYkCaTWL OHFBo7UyUFXGYStdcuMUAQXlDb84K7lGK9fvVWdz4OenOsOh5emV1bBxVm5kIx3ZAPkV EjOg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b="QxhlS/jn"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id et16-20020a170907295000b008888f4120c1si5309412ejc.695.2023.01.30.21.29.12; Mon, 30 Jan 2023 21:29:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b="QxhlS/jn"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231302AbjAaF1U (ORCPT + 99 others); Tue, 31 Jan 2023 00:27:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36716 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230428AbjAaF0Z (ORCPT ); Tue, 31 Jan 2023 00:26:25 -0500 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 464DD3B0DE for ; Mon, 30 Jan 2023 21:26:18 -0800 (PST) Received: by mail-pj1-x102e.google.com with SMTP id e8-20020a17090a9a8800b0022c387f0f93so12824806pjp.3 for ; Mon, 30 Jan 2023 21:26:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=+O8KWCroo2161P9B9g6JCettFGAqWNsTAzgVxWSxUro=; b=QxhlS/jni1MnzIXLFI0rW2DoqXXXGXF/ckq4P5B1hBCgUrG050yp1GIUI7pTJqe75H 6DbXgGP5kTXpqOAbYAn5ewt+bU2GGJv0dsj84tBR8FX6UMypE1F6iPTDPkOHdiPzx1I3 1aY6+U4Z+CJ4BT+yeaih61EuvYRaGlFMAfm20= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+O8KWCroo2161P9B9g6JCettFGAqWNsTAzgVxWSxUro=; b=ot/bN8haBaQdxLs8wxvWdSoh8BgjjhR8LVkcX5Macx+OV//VFphPe+30LtCXI/DU5Z GO7ifqYbi+XcMnW0qPsbqekqoJZLv6EdoNYwKILVEJRbEvO29xMXQTJC47vidi6vxf26 XrpTuHttJXipXzb823YxZtumnBRBPeKuUgboNVTrxu37rJwsgrs9T24YDjAlay4tdyJf 6U6shiWd8Ws9iUVGHsWshuFTRjRDOveYNP22efOQwcpWbfY8HLqjbrp7oUAQXdo2REg5 mRM82zFJS3g/cG+djoNAzT7nyYCsM4e+ODMWy23akQd0B7hhaUXmBE4klTmrmddBlolY NyZw== X-Gm-Message-State: AFqh2koIZJKG0Q9ta8EPgWLtleykW2ohd0mrKVamhlEHcvK6zPPBMrmq Mmk7q+N4SxnxXH4Rfzk940LEUg== X-Received: by 2002:a17:903:124d:b0:195:e92e:c4d3 with SMTP id u13-20020a170903124d00b00195e92ec4d3mr43880033plh.46.1675142777861; Mon, 30 Jan 2023 21:26:17 -0800 (PST) Received: from localhost.localdomain ([2605:a601:a780:1400:a879:b64b:d9bd:3c1]) by smtp.gmail.com with ESMTPSA id jk15-20020a170903330f00b001960cccc318sm2310106plb.121.2023.01.30.21.26.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Jan 2023 21:26:17 -0800 (PST) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, gregkh@linuxfoundation.org, Leon Romanovsky Subject: [PATCH net-next v9 7/8] RDMA/bnxt_re: Remove the sriov config callback Date: Mon, 30 Jan 2023 21:25:56 -0800 Message-Id: <20230131052557.99119-8-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230131052557.99119-1-ajit.khaparde@broadcom.com> References: <20230131052557.99119-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756514722585885854?= X-GMAIL-MSGID: =?utf-8?q?1756514722585885854?= Remove the SRIOV config callback which the bnxt_en was calling to reconfigure the chip resources for a PF device when VFs are created. The code is now modified to provision the VF resources based on the total VF count instead of the actual VF count. This allows the SRIOV config callback to be removed from the list of ulp_ops. Suggested-by: Leon Romanovsky Signed-off-by: Ajit Khaparde Reviewed-by: Leon Romanovsky --- drivers/infiniband/hw/bnxt_re/main.c | 11 ++++--- .../net/ethernet/broadcom/bnxt/bnxt_sriov.c | 7 +---- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 29 ------------------- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 1 - 4 files changed, 8 insertions(+), 40 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index ed7ac6acaaff..60df6809bc60 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -221,13 +221,12 @@ static void bnxt_re_set_resource_limits(struct bnxt_re_dev *rdev) bnxt_re_limit_vf_res(&rdev->qplib_ctx, num_vfs); } -static void bnxt_re_sriov_config(void *p, int num_vfs) +static void bnxt_re_vf_res_config(struct bnxt_re_dev *rdev) { - struct bnxt_re_dev *rdev = p; if (test_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags)) return; - rdev->num_vfs = num_vfs; + rdev->num_vfs = pci_sriov_get_totalvfs(rdev->en_dev->pdev); if (!bnxt_qplib_is_chip_gen_p5(rdev->chip_ctx)) { bnxt_re_set_resource_limits(rdev); bnxt_qplib_set_func_resources(&rdev->qplib_res, &rdev->rcfw, @@ -297,7 +296,6 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) } static struct bnxt_ulp_ops bnxt_re_ulp_ops = { - .ulp_sriov_config = bnxt_re_sriov_config, .ulp_irq_stop = bnxt_re_stop_irq, .ulp_irq_restart = bnxt_re_start_irq }; @@ -1369,6 +1367,11 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 wqe_mode) INIT_DELAYED_WORK(&rdev->worker, bnxt_re_worker); set_bit(BNXT_RE_FLAG_QOS_WORK_REG, &rdev->flags); schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000)); + /* + * Use the total VF count since the actual VF count may not be + * available at this point. + */ + bnxt_re_vf_res_config(rdev); } return 0; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c index a4cba7cb2783..3ed3a2b3b3a9 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c @@ -749,7 +749,6 @@ int bnxt_cfg_hw_sriov(struct bnxt *bp, int *num_vfs, bool reset) *num_vfs = rc; } - bnxt_ulp_sriov_cfg(bp, *num_vfs); return 0; } @@ -823,10 +822,8 @@ static int bnxt_sriov_enable(struct bnxt *bp, int *num_vfs) goto err_out2; rc = pci_enable_sriov(bp->pdev, *num_vfs); - if (rc) { - bnxt_ulp_sriov_cfg(bp, 0); + if (rc) goto err_out2; - } return 0; @@ -872,8 +869,6 @@ void bnxt_sriov_disable(struct bnxt *bp) rtnl_lock(); bnxt_restore_pf_fw_resources(bp); rtnl_unlock(); - - bnxt_ulp_sriov_cfg(bp, 0); } int bnxt_sriov_configure(struct pci_dev *pdev, int num_vfs) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index fc5ded5d0d09..30d4a227fbf9 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -259,16 +259,6 @@ int bnxt_send_msg(struct bnxt_en_dev *edev, } EXPORT_SYMBOL(bnxt_send_msg); -static void bnxt_ulp_get(struct bnxt_ulp *ulp) -{ - atomic_inc(&ulp->ref_count); -} - -static void bnxt_ulp_put(struct bnxt_ulp *ulp) -{ - atomic_dec(&ulp->ref_count); -} - void bnxt_ulp_stop(struct bnxt *bp) { struct bnxt_aux_priv *aux_priv = bp->aux_priv; @@ -321,25 +311,6 @@ void bnxt_ulp_start(struct bnxt *bp, int err) } -void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs) -{ - struct bnxt_en_dev *edev = bp->edev; - struct bnxt_ulp_ops *ops; - struct bnxt_ulp *ulp; - - if (!edev) - return; - ulp = edev->ulp_tbl; - - ops = rcu_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_sriov_config) - return; - - bnxt_ulp_get(ulp); - ops->ulp_sriov_config(ulp->handle, num_vfs); - bnxt_ulp_put(ulp); -} - void bnxt_ulp_irq_stop(struct bnxt *bp) { struct bnxt_en_dev *edev = bp->edev; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index c62986e4cc82..ed2832975912 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -26,7 +26,6 @@ struct bnxt_msix_entry { }; struct bnxt_ulp_ops { - void (*ulp_sriov_config)(void *, int); void (*ulp_irq_stop)(void *); void (*ulp_irq_restart)(void *, struct bnxt_msix_entry *); }; From patchwork Tue Jan 31 05:25:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 50612 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2570817wrn; Mon, 30 Jan 2023 21:31:01 -0800 (PST) X-Google-Smtp-Source: AK7set/4t/QqX8pgn58lpNtQq9qguNl8S091gNuehC20Q48w/hbxPa5fqJSFCk03Jz/TesH7DTIw X-Received: by 2002:a17:907:d085:b0:879:ec1a:4ac with SMTP id vc5-20020a170907d08500b00879ec1a04acmr15699837ejc.76.1675143061196; Mon, 30 Jan 2023 21:31:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675143061; cv=none; d=google.com; s=arc-20160816; b=XlizuWr9SBm6O6/vGgH0cTSWEV2yI4kDFdVldWoJjdH3SeAeAJ404+48JDq19MXD6Q 66NfxcK4ZuZvg85xQn3dyNlKm823ek/yQjKs4CZYlL7JDKyBoFh4dj5VWIcnN00rdHwY hKv4u0Ani/m1ja25YNnm2uQTMFj0+OcbD4adW7xsP61Idwawy3dvBO62FtFwHJIec2Rs VmTxTYDQ0tjcgOCeIkjN3GNeDYR+rgec2NbOmfa8LTFfzf8v8byVZMPB/RGVi1djwGMV HdYiitC4tgNT/FfDGIhE50tKn1DQbVG/6M+9YBbNVagmDUDhBbRSgB+EVtQrHihAWxoC Wgyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=UwPjna4n7DbeH9h4mgB48iA3rJ0XUooQ+ITGPmM6Lqo=; b=Affo2SVpqHynqK15sO23VanV/qf/J9Fs1wij0duRSXKx7PEwtA0k7Lw36yRrp1Y3KH 1uff8wJTrQvhCsJjsOFH5cqu20CX0b4aaEre8vTxS1jyy3+B1zfCjT9tzx+8fpsLHvtt M6ZCu+hEu9+V861TUTgdnZf5T5Ep27Tb4NrEDl6xaVCoM9AjcCUtYp6QBobPFsKsfi7x EA7PwIRrycsxd6gVzctEeNG2rbBG4VekJWHfTGBwJxyFhVe2auKSeV0AZzCtK5hJrg0P jUfmoXv7zd3Y4y05hK9MBRMZ7XZ5Xk0xqLDlK+PanIKf2sj1NukGaFbzGXv6JG+r+NS0 IAYw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=d2+Vo8QX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id zu7-20020a17090708c700b008728cb7cdd2si14494135ejb.30.2023.01.30.21.30.37; Mon, 30 Jan 2023 21:31:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=d2+Vo8QX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230315AbjAaF1j (ORCPT + 99 others); Tue, 31 Jan 2023 00:27:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36822 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230303AbjAaF03 (ORCPT ); Tue, 31 Jan 2023 00:26:29 -0500 Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96F8A3CE0F for ; Mon, 30 Jan 2023 21:26:20 -0800 (PST) Received: by mail-pl1-x633.google.com with SMTP id m13so2058089plx.13 for ; Mon, 30 Jan 2023 21:26:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=UwPjna4n7DbeH9h4mgB48iA3rJ0XUooQ+ITGPmM6Lqo=; b=d2+Vo8QXozmu88E2OqI1+LyzT3MJVS8iAL2jHhIHQUO8b74S3FqVPy3eH+PLW8x/IN q2Z6Y8mdTaq/jRpxA0xY1JDGk0R58MdaGqY3Bbe7AIUvHBj/HGp4vJVA4jYToepP5gr8 4waJRt6FY92UEY/v2YX/cea2IIFT6xNPdVDjg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UwPjna4n7DbeH9h4mgB48iA3rJ0XUooQ+ITGPmM6Lqo=; b=aI3QCY99xARMjeHFQ5tZqAF99w9v4pqnfEl7Ck5jtluB3LtjIL4sVCdLYLHLxBX2ck 8BGpNgvw/25Qd2WutnnxFZvEJHN7PQ6m4LEgBTrexWA/06v1721BoGx3Z/X6Tcct90hQ JueMKivHct40W6z+GngU321BrVwehIlSKeqoyfABd3g9U3HCX6XmKVoxjO1uytQvxlou sFa2c17ugNZnu0LgQ0Lm6tGHDtJLRw0kEp9CisIA/Q2F+vHVzfVz1wDXCUpMvLMNPSAq fpWtGLX1Wzt8K8LK4WC8qnbDHtEYlY0SGVX/biixm17RQlVQhw8cYnYbMrBGnPmI5Dru S1Kw== X-Gm-Message-State: AO0yUKWkBgaHX2krOvEr/q/kH8SXSYQjVkDmyW7H1qXOErw6tWtXoH3R t3ysFjZ1H+kBNpWYPiGTCG63Jg== X-Received: by 2002:a17:902:d484:b0:196:8bd6:2398 with SMTP id c4-20020a170902d48400b001968bd62398mr6537977plg.18.1675142779477; Mon, 30 Jan 2023 21:26:19 -0800 (PST) Received: from localhost.localdomain ([2605:a601:a780:1400:a879:b64b:d9bd:3c1]) by smtp.gmail.com with ESMTPSA id jk15-20020a170903330f00b001960cccc318sm2310106plb.121.2023.01.30.21.26.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Jan 2023 21:26:18 -0800 (PST) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, gregkh@linuxfoundation.org, Leon Romanovsky Subject: [PATCH net-next v9 8/8] bnxt_en: Remove runtime interrupt vector allocation Date: Mon, 30 Jan 2023 21:25:57 -0800 Message-Id: <20230131052557.99119-9-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230131052557.99119-1-ajit.khaparde@broadcom.com> References: <20230131052557.99119-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756514810591030752?= X-GMAIL-MSGID: =?utf-8?q?1756514810591030752?= Modified the bnxt_en code to create and pre-configure RDMA devices with the right MSI-X vector count for the ROCE driver to use. This is to align the ROCE driver to the auxiliary device model which will simply bind the driver without getting into PCI-related handling. All PCI-related logic will now be in the bnxt_en driver. Suggested-by: Leon Romanovsky Signed-off-by: Ajit Khaparde Reviewed-by: Leon Romanovsky --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 1 - drivers/infiniband/hw/bnxt_re/main.c | 48 ++---- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 157 +++++------------- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 6 +- 4 files changed, 57 insertions(+), 155 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index b0465c8d229a..5a2baf49ecaa 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -129,7 +129,6 @@ struct bnxt_re_dev { unsigned int version, major, minor; struct bnxt_qplib_chip_ctx *chip_ctx; struct bnxt_en_dev *en_dev; - struct bnxt_msix_entry msix_entries[BNXT_RE_MAX_MSIX]; int num_msix; int id; diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 60df6809bc60..c5867e78f231 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -262,7 +262,7 @@ static void bnxt_re_stop_irq(void *handle) static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) { struct bnxt_re_dev *rdev = (struct bnxt_re_dev *)handle; - struct bnxt_msix_entry *msix_ent = rdev->msix_entries; + struct bnxt_msix_entry *msix_ent = rdev->en_dev->msix_entries; struct bnxt_qplib_rcfw *rcfw = &rdev->rcfw; struct bnxt_qplib_nq *nq; int indx, rc; @@ -281,7 +281,7 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) * in device sctructure. */ for (indx = 0; indx < rdev->num_msix; indx++) - rdev->msix_entries[indx].vector = ent[indx].vector; + rdev->en_dev->msix_entries[indx].vector = ent[indx].vector; bnxt_qplib_rcfw_start_irq(rcfw, msix_ent[BNXT_RE_AEQ_IDX].vector, false); @@ -315,32 +315,6 @@ static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) return rc; } -static int bnxt_re_request_msix(struct bnxt_re_dev *rdev) -{ - int rc = 0, num_msix_want = BNXT_RE_MAX_MSIX, num_msix_got; - struct bnxt_en_dev *en_dev; - - en_dev = rdev->en_dev; - - num_msix_want = min_t(u32, BNXT_RE_MAX_MSIX, num_online_cpus()); - - num_msix_got = bnxt_req_msix_vecs(en_dev, - rdev->msix_entries, - num_msix_want); - if (num_msix_got < BNXT_RE_MIN_MSIX) { - rc = -EINVAL; - goto done; - } - if (num_msix_got != num_msix_want) { - ibdev_warn(&rdev->ibdev, - "Requested %d MSI-X vectors, got %d\n", - num_msix_want, num_msix_got); - } - rdev->num_msix = num_msix_got; -done: - return rc; -} - static void bnxt_re_init_hwrm_hdr(struct bnxt_re_dev *rdev, struct input *hdr, u16 opcd, u16 crid, u16 trid) { @@ -785,7 +759,7 @@ static u32 bnxt_re_get_nqdb_offset(struct bnxt_re_dev *rdev, u16 indx) return bnxt_qplib_is_chip_gen_p5(rdev->chip_ctx) ? (rdev->is_virtfn ? BNXT_RE_GEN_P5_VF_NQ_DB : BNXT_RE_GEN_P5_PF_NQ_DB) : - rdev->msix_entries[indx].db_offset; + rdev->en_dev->msix_entries[indx].db_offset; } static void bnxt_re_cleanup_res(struct bnxt_re_dev *rdev) @@ -810,7 +784,7 @@ static int bnxt_re_init_res(struct bnxt_re_dev *rdev) for (i = 1; i < rdev->num_msix ; i++) { db_offt = bnxt_re_get_nqdb_offset(rdev, i); rc = bnxt_qplib_enable_nq(rdev->en_dev->pdev, &rdev->nq[i - 1], - i - 1, rdev->msix_entries[i].vector, + i - 1, rdev->en_dev->msix_entries[i].vector, db_offt, &bnxt_re_cqn_handler, &bnxt_re_srqn_handler); if (rc) { @@ -897,7 +871,7 @@ static int bnxt_re_alloc_res(struct bnxt_re_dev *rdev) rattr.type = type; rattr.mode = RING_ALLOC_REQ_INT_MODE_MSIX; rattr.depth = BNXT_QPLIB_NQE_MAX_CNT - 1; - rattr.lrid = rdev->msix_entries[i + 1].ring_idx; + rattr.lrid = rdev->en_dev->msix_entries[i + 1].ring_idx; rc = bnxt_re_net_ring_alloc(rdev, &rattr, &nq->ring_id); if (rc) { ibdev_err(&rdev->ibdev, @@ -1217,7 +1191,7 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev) bnxt_qplib_free_rcfw_channel(&rdev->rcfw); } if (test_and_clear_bit(BNXT_RE_FLAG_GOT_MSIX, &rdev->flags)) - bnxt_free_msix_vecs(rdev->en_dev); + rdev->num_msix = 0; bnxt_re_destroy_chip_ctx(rdev); if (test_and_clear_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags)) @@ -1262,13 +1236,15 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 wqe_mode) /* Check whether VF or PF */ bnxt_re_get_sriov_func_type(rdev); - rc = bnxt_re_request_msix(rdev); - if (rc) { + if (!rdev->en_dev->ulp_tbl->msix_requested) { ibdev_err(&rdev->ibdev, "Failed to get MSI-X vectors: %#x\n", rc); rc = -EINVAL; goto fail; } + ibdev_dbg(&rdev->ibdev, "Got %d MSI-X vectors\n", + rdev->en_dev->ulp_tbl->msix_requested); + rdev->num_msix = rdev->en_dev->ulp_tbl->msix_requested; set_bit(BNXT_RE_FLAG_GOT_MSIX, &rdev->flags); bnxt_re_query_hwrm_intf_version(rdev); @@ -1292,14 +1268,14 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 wqe_mode) rattr.type = type; rattr.mode = RING_ALLOC_REQ_INT_MODE_MSIX; rattr.depth = BNXT_QPLIB_CREQE_MAX_CNT - 1; - rattr.lrid = rdev->msix_entries[BNXT_RE_AEQ_IDX].ring_idx; + rattr.lrid = rdev->en_dev->msix_entries[BNXT_RE_AEQ_IDX].ring_idx; rc = bnxt_re_net_ring_alloc(rdev, &rattr, &creq->ring_id); if (rc) { ibdev_err(&rdev->ibdev, "Failed to allocate CREQ: %#x\n", rc); goto free_rcfw; } db_offt = bnxt_re_get_nqdb_offset(rdev, BNXT_RE_AEQ_IDX); - vid = rdev->msix_entries[BNXT_RE_AEQ_IDX].vector; + vid = rdev->en_dev->msix_entries[BNXT_RE_AEQ_IDX].vector; rc = bnxt_qplib_enable_rcfw_channel(&rdev->rcfw, vid, db_offt, rdev->is_virtfn, &bnxt_re_aeq_handler); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index 30d4a227fbf9..7147d9304761 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -28,6 +28,30 @@ static DEFINE_IDA(bnxt_aux_dev_ids); +static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) +{ + struct bnxt_en_dev *edev = bp->edev; + int num_msix, idx, i; + + if (!edev->ulp_tbl->msix_requested) { + netdev_warn(bp->dev, "Requested MSI-X vectors insufficient\n"); + return; + } + num_msix = edev->ulp_tbl->msix_requested; + idx = edev->ulp_tbl->msix_base; + for (i = 0; i < num_msix; i++) { + ent[i].vector = bp->irq_tbl[idx + i].vector; + ent[i].ring_idx = idx + i; + if (bp->flags & BNXT_FLAG_CHIP_P5) { + ent[i].db_offset = DB_PF_OFFSET_P5; + if (BNXT_VF(bp)) + ent[i].db_offset = DB_VF_OFFSET_P5; + } else { + ent[i].db_offset = (idx + i) * 0x80; + } + } +} + int bnxt_register_dev(struct bnxt_en_dev *edev, struct bnxt_ulp_ops *ulp_ops, void *handle) @@ -42,17 +66,18 @@ int bnxt_register_dev(struct bnxt_en_dev *edev, bp->cp_nr_rings == max_stat_ctxs) return -ENOMEM; - ulp = kzalloc(sizeof(*ulp), GFP_KERNEL); + ulp = edev->ulp_tbl; if (!ulp) return -ENOMEM; - edev->ulp_tbl = ulp; ulp->handle = handle; rcu_assign_pointer(ulp->ulp_ops, ulp_ops); if (test_bit(BNXT_STATE_OPEN, &bp->state)) bnxt_hwrm_vnic_cfg(bp, 0); + bnxt_fill_msix_vecs(bp, bp->edev->msix_entries); + edev->flags |= BNXT_EN_FLAG_MSIX_REQUESTED; return 0; } EXPORT_SYMBOL(bnxt_register_dev); @@ -66,7 +91,7 @@ void bnxt_unregister_dev(struct bnxt_en_dev *edev) ulp = edev->ulp_tbl; if (ulp->msix_requested) - bnxt_free_msix_vecs(edev); + edev->flags &= ~BNXT_EN_FLAG_MSIX_REQUESTED; if (ulp->max_async_event_id) bnxt_hwrm_func_drv_rgtr(bp, NULL, 0, true); @@ -79,125 +104,17 @@ void bnxt_unregister_dev(struct bnxt_en_dev *edev) msleep(100); i++; } - kfree(ulp); - edev->ulp_tbl = NULL; return; } EXPORT_SYMBOL(bnxt_unregister_dev); -static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) -{ - struct bnxt_en_dev *edev = bp->edev; - int num_msix, idx, i; - - num_msix = edev->ulp_tbl->msix_requested; - idx = edev->ulp_tbl->msix_base; - for (i = 0; i < num_msix; i++) { - ent[i].vector = bp->irq_tbl[idx + i].vector; - ent[i].ring_idx = idx + i; - if (bp->flags & BNXT_FLAG_CHIP_P5) { - ent[i].db_offset = DB_PF_OFFSET_P5; - if (BNXT_VF(bp)) - ent[i].db_offset = DB_VF_OFFSET_P5; - } else { - ent[i].db_offset = (idx + i) * 0x80; - } - } -} - -int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, - struct bnxt_msix_entry *ent, - int num_msix) -{ - struct net_device *dev = edev->net; - struct bnxt *bp = netdev_priv(dev); - struct bnxt_hw_resc *hw_resc; - int max_idx, max_cp_rings; - int avail_msix, idx; - int total_vecs; - int rc = 0; - - if (!(bp->flags & BNXT_FLAG_USING_MSIX)) - return -ENODEV; - - if (edev->ulp_tbl->msix_requested) - return -EAGAIN; - - max_cp_rings = bnxt_get_max_func_cp_rings(bp); - avail_msix = bnxt_get_avail_msix(bp, num_msix); - if (!avail_msix) - return -ENOMEM; - if (avail_msix > num_msix) - avail_msix = num_msix; - - if (BNXT_NEW_RM(bp)) { - idx = bp->cp_nr_rings; - } else { - max_idx = min_t(int, bp->total_irqs, max_cp_rings); - idx = max_idx - avail_msix; - } - edev->ulp_tbl->msix_base = idx; - edev->ulp_tbl->msix_requested = avail_msix; - hw_resc = &bp->hw_resc; - total_vecs = idx + avail_msix; - rtnl_lock(); - if (bp->total_irqs < total_vecs || - (BNXT_NEW_RM(bp) && hw_resc->resv_irqs < total_vecs)) { - if (netif_running(dev)) { - bnxt_close_nic(bp, true, false); - rc = bnxt_open_nic(bp, true, false); - } else { - rc = bnxt_reserve_rings(bp, true); - } - } - rtnl_unlock(); - if (rc) { - edev->ulp_tbl->msix_requested = 0; - return -EAGAIN; - } - - if (BNXT_NEW_RM(bp)) { - int resv_msix; - - resv_msix = hw_resc->resv_irqs - bp->cp_nr_rings; - avail_msix = min_t(int, resv_msix, avail_msix); - edev->ulp_tbl->msix_requested = avail_msix; - } - bnxt_fill_msix_vecs(bp, ent); - edev->flags |= BNXT_EN_FLAG_MSIX_REQUESTED; - return avail_msix; -} -EXPORT_SYMBOL(bnxt_req_msix_vecs); - -void bnxt_free_msix_vecs(struct bnxt_en_dev *edev) -{ - struct net_device *dev = edev->net; - struct bnxt *bp = netdev_priv(dev); - - if (!(edev->flags & BNXT_EN_FLAG_MSIX_REQUESTED)) - return; - - edev->ulp_tbl->msix_requested = 0; - edev->flags &= ~BNXT_EN_FLAG_MSIX_REQUESTED; - rtnl_lock(); - if (netif_running(dev) && !(edev->flags & BNXT_EN_FLAG_ULP_STOPPED)) { - bnxt_close_nic(bp, true, false); - bnxt_open_nic(bp, true, false); - } - rtnl_unlock(); - - return; -} -EXPORT_SYMBOL(bnxt_free_msix_vecs); - int bnxt_get_ulp_msix_num(struct bnxt *bp) { - if (bnxt_ulp_registered(bp->edev)) { - struct bnxt_en_dev *edev = bp->edev; + u32 roce_msix = BNXT_VF(bp) ? + BNXT_MAX_VF_ROCE_MSIX : BNXT_MAX_ROCE_MSIX; - return edev->ulp_tbl->msix_requested; - } - return 0; + return ((bp->flags & BNXT_FLAG_ROCE_CAP) ? + min_t(u32, roce_msix, num_online_cpus()) : 0); } int bnxt_get_ulp_msix_base(struct bnxt *bp) @@ -403,6 +320,8 @@ static void bnxt_aux_dev_release(struct device *dev) struct bnxt *bp = netdev_priv(aux_priv->edev->net); ida_free(&bnxt_aux_dev_ids, aux_priv->id); + kfree(aux_priv->edev->ulp_tbl); + aux_priv->edev->ulp_tbl = NULL; kfree(aux_priv->edev); aux_priv->edev = NULL; bp->edev = NULL; @@ -427,6 +346,8 @@ static void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp) edev->hw_ring_stats_size = bp->hw_ring_stats_size; edev->pf_port_id = bp->pf.port_id; edev->en_state = bp->state; + + edev->ulp_tbl->msix_requested = bnxt_get_ulp_msix_num(bp); } void bnxt_rdma_aux_device_init(struct bnxt *bp) @@ -434,6 +355,7 @@ void bnxt_rdma_aux_device_init(struct bnxt *bp) struct auxiliary_device *aux_dev; struct bnxt_aux_priv *aux_priv; struct bnxt_en_dev *edev; + struct bnxt_ulp *ulp; int rc; if (!(bp->flags & BNXT_FLAG_ROCE_CAP)) @@ -473,6 +395,11 @@ void bnxt_rdma_aux_device_init(struct bnxt *bp) if (!edev) goto aux_dev_uninit; + ulp = kzalloc(sizeof(*ulp), GFP_KERNEL); + if (!ulp) + goto aux_dev_uninit; + + edev->ulp_tbl = ulp; aux_priv->edev = edev; bp->edev = edev; bnxt_set_edev_info(edev, bp); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index ed2832975912..80cbc4b6130a 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -15,6 +15,8 @@ #define BNXT_MIN_ROCE_CP_RINGS 2 #define BNXT_MIN_ROCE_STAT_CTXS 1 +#define BNXT_MAX_ROCE_MSIX 9 +#define BNXT_MAX_VF_ROCE_MSIX 2 struct hwrm_async_event_cmpl; struct bnxt; @@ -51,6 +53,7 @@ struct bnxt_ulp { struct bnxt_en_dev { struct net_device *net; struct pci_dev *pdev; + struct bnxt_msix_entry msix_entries[BNXT_MAX_ROCE_MSIX]; u32 flags; #define BNXT_EN_FLAG_ROCEV1_CAP 0x1 #define BNXT_EN_FLAG_ROCEV2_CAP 0x2 @@ -101,9 +104,6 @@ void bnxt_rdma_aux_device_init(struct bnxt *bp); int bnxt_register_dev(struct bnxt_en_dev *edev, struct bnxt_ulp_ops *ulp_ops, void *handle); void bnxt_unregister_dev(struct bnxt_en_dev *edev); -int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, struct bnxt_msix_entry *ent, - int num_msix); -void bnxt_free_msix_vecs(struct bnxt_en_dev *edev); int bnxt_send_msg(struct bnxt_en_dev *edev, struct bnxt_fw_msg *fw_msg); int bnxt_register_async_events(struct bnxt_en_dev *edev, unsigned long *events_bmap, u16 max_id);