From patchwork Thu Jan 12 20:29:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 42796 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp4100862wrt; Thu, 12 Jan 2023 12:56:58 -0800 (PST) X-Google-Smtp-Source: AMrXdXtTOAyseCL0V2vFHv/nVc/Mu1AyXXd096yw5iH4vbiIEszKItcoaZvOKiH9VcXU6ESskwpS X-Received: by 2002:a17:906:eb09:b0:84d:34fa:f1a4 with SMTP id mb9-20020a170906eb0900b0084d34faf1a4mr17641421ejb.60.1673557017764; Thu, 12 Jan 2023 12:56:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673557017; cv=none; d=google.com; s=arc-20160816; b=wCKSL+LXpDexbNuwByznJF848WSd3AB/IkdS+Q9242voAsDkyOeGIA0DHX1J74LG2I 13gk9i+c804clItHNLKFMHD6S7gD94PQFdEoqY1ABE0muimtVXIdBxAa4OkrRhgFV+4o Q2PhFrwEwGjZXePYkqDhHhLygNRqMtCaMhaNQ9my+qgyA+6qjEWsRLflsPacgrETCyDM QQbKJzuhFiG2utZ/cs+yJW3ABz0G/m/mYMRqJNm53rv4Vt8is4RGUy1uQu+tuGVGqf+j JwIN5YGP7Nk0Ad57cABgndKD/DjemLdfLjHQZ/tlrJmLbPbwtyjR2coHaAvN9Moqyt0C TKsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=zxN71hJlYGaRWyMyj3R9KCuSks7g1CInS8/ECt3DiV8=; b=S5Wfe4kSLOn/H4EK9nTg6RfItsDQjifjHC/Ts9+fGS44F226JLnWHBoHvXbZiH2ocb 2ngF71u3uPt/gVP9GsN1zz9fOEzAn7OEZD37+64pZt3ueW1iVCaJIzNw5BAbOhdVDCzY 1HLuqbUazhMNdX1EVTjtYSdDn4MrZRfQWlVw0359I/CuruZrtbrj6rS9N2w/sTCqYADq oo9/K0IV+qxSfdpvQ3rEOK/n9n1uekyGbFoSgm+5c+Oins79fTaPZ3NXsVLSjMZqs2Oq 6eVGYAZPgF/q7S7TENSoB18pp0FupOZv3aDwDxV2STIdIsCH0OFyMOpWgD+FZ4oE72b9 1d7A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=YGY+IVHl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id di12-20020a170906730c00b007c0fa80794fsi20885715ejc.733.2023.01.12.12.56.31; Thu, 12 Jan 2023 12:56:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=YGY+IVHl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240570AbjALUyx (ORCPT + 99 others); Thu, 12 Jan 2023 15:54:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56054 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236952AbjALUxH (ORCPT ); Thu, 12 Jan 2023 15:53:07 -0500 Received: from mail-pl1-x631.google.com (mail-pl1-x631.google.com [IPv6:2607:f8b0:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6D961C92B for ; Thu, 12 Jan 2023 12:29:47 -0800 (PST) Received: by mail-pl1-x631.google.com with SMTP id w3so21378656ply.3 for ; Thu, 12 Jan 2023 12:29:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=zxN71hJlYGaRWyMyj3R9KCuSks7g1CInS8/ECt3DiV8=; b=YGY+IVHlpSMKucqgGXNQ9CqWH2WFpLbE7HzfEZxnrWkAkcXnUeaeEN69PeeQuse3gG A4lZmfKaadS3vaO/6CU4BGVvD3lOnIwi06tLiffjMCEFNkV65OdgaxtEE8waPancexT7 Rq0DfuC7LqXVTTlC4Ecg8+pZey6TcZWFnOmIw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zxN71hJlYGaRWyMyj3R9KCuSks7g1CInS8/ECt3DiV8=; b=x7LpHYAPRCfq3glX+Y270cJ07Cp3M8F3kOSEowsNSwTDftJTDuWw4bXN9hEEMZkr7Y 1qJENPFx/kpMTE7EXpn60TDlBzO6/ialBubpDbIwI9uHUH4NIrc7E+hnMbMowdawGKUr 4jmhiLU19l+bKM/V3IFPnMWhYpPa09zm9TvA98tqTk+VRx5waLeFqQETt2Fzq0DHky9T AG+XW2GjGwtVrr72MOxDBvIbOD+zgmbJ1Uf2RX1wcYLOssDBOCNLzCtkXvRrdDHchS1/ lP0AmwgDC8iMBP3uNSASm0qyjhgKjMzuoy87PjGMIw5yWo5UfhjavwFTUGmPfr/SDiLD 5LbA== X-Gm-Message-State: AFqh2kq51NGE6yjDpsixgC7m3HbAlYS1e4UbpqIyEmMOViL6eKj3bvmP 5lEkn7HC7sRqa5kTpLt1xMAB0g== X-Received: by 2002:a05:6a21:3a82:b0:b6:7e03:5723 with SMTP id zv2-20020a056a213a8200b000b67e035723mr3877821pzb.42.1673555386892; Thu, 12 Jan 2023 12:29:46 -0800 (PST) Received: from C02GC2QQMD6T.wifi.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id 129-20020a630887000000b004777c56747csm10283855pgi.11.2023.01.12.12.29.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 12:29:46 -0800 (PST) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, Leon Romanovsky Subject: [PATCH net-next v7 1/8] bnxt_en: Add auxiliary driver support Date: Thu, 12 Jan 2023 12:29:32 -0800 Message-Id: <20230112202939.19562-2-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230112202939.19562-1-ajit.khaparde@broadcom.com> References: <20230112202939.19562-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754851723691132527?= X-GMAIL-MSGID: =?utf-8?q?1754851723691132527?= Add auxiliary driver support. An auxiliary device will be created if the hardware indicates support for RDMA. The bnxt_ulp_probe() function has been removed and a new bnxt_rdma_aux_device_add() function has been added. The bnxt_free_msix_vecs() and bnxt_req_msix_vecs() will now hold the RTNL lock when they call the bnxt_close_nic()and bnxt_open_nic() since the device close and open need to be protected under RTNL lock. The operations between the bnxt_en and bnxt_re will be protected using the en_ops_lock. This will be used by the bnxt_re driver in a follow-on patch to create ROCE interfaces. Signed-off-by: Ajit Khaparde Reviewed-by: Andy Gospodarek Reviewed-by: Selvin Xavier Reviewed-by: Leon Romanovsky --- drivers/net/ethernet/broadcom/Kconfig | 1 + drivers/net/ethernet/broadcom/bnxt/bnxt.c | 8 +- drivers/net/ethernet/broadcom/bnxt/bnxt.h | 8 + drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 164 +++++++++++++++--- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 4 +- 5 files changed, 158 insertions(+), 27 deletions(-) diff --git a/drivers/net/ethernet/broadcom/Kconfig b/drivers/net/ethernet/broadcom/Kconfig index f4ca0c6c0f51..948586bf1b5b 100644 --- a/drivers/net/ethernet/broadcom/Kconfig +++ b/drivers/net/ethernet/broadcom/Kconfig @@ -213,6 +213,7 @@ config BNXT select NET_DEVLINK select PAGE_POOL select DIMLIB + select AUXILIARY_BUS help This driver supports Broadcom NetXtreme-C/E 10/25/40/50 gigabit Ethernet cards. To compile this driver as a module, choose M here: diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 16ce7a90610c..c5cbf2307e07 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -13178,6 +13178,9 @@ static void bnxt_remove_one(struct pci_dev *pdev) struct net_device *dev = pci_get_drvdata(pdev); struct bnxt *bp = netdev_priv(dev); + bnxt_rdma_aux_device_uninit(bp); + bnxt_aux_dev_free(bp); + if (BNXT_PF(bp)) bnxt_sriov_disable(bp); @@ -13776,11 +13779,13 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) bnxt_dl_fw_reporters_create(bp); + bnxt_rdma_aux_device_init(bp); + bnxt_print_device_info(bp); pci_save_state(pdev); - return 0; + return 0; init_err_cleanup: bnxt_dl_unregister(bp); init_err_dl: @@ -13824,7 +13829,6 @@ static void bnxt_shutdown(struct pci_dev *pdev) if (netif_running(dev)) dev_close(dev); - bnxt_ulp_shutdown(bp); bnxt_clear_int_mode(bp); pci_disable_device(pdev); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethernet/broadcom/bnxt/bnxt.h index 5163ef4a49ea..4df2da81708e 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -1631,6 +1632,12 @@ struct bnxt_fw_health { #define BNXT_FW_IF_RETRY 10 #define BNXT_FW_SLOT_RESET_RETRY 4 +struct bnxt_aux_dev { + struct auxiliary_device aux_dev; + struct bnxt_en_dev *edev; + int id; +}; + enum board_idx { BCM57301, BCM57302, @@ -1852,6 +1859,7 @@ struct bnxt { #define BNXT_CHIP_P4_PLUS(bp) \ (BNXT_CHIP_P4(bp) || BNXT_CHIP_P5(bp)) + struct bnxt_aux_dev *aux_dev; struct bnxt_en_dev *edev; struct bnxt_napi **bnapi; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index 2e54bf4fc7a7..899d9adfd35f 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -25,6 +25,8 @@ #include "bnxt_hwrm.h" #include "bnxt_ulp.h" +static DEFINE_IDA(bnxt_aux_dev_ids); + static int bnxt_register_dev(struct bnxt_en_dev *edev, unsigned int ulp_id, struct bnxt_ulp_ops *ulp_ops, void *handle) { @@ -32,7 +34,6 @@ static int bnxt_register_dev(struct bnxt_en_dev *edev, unsigned int ulp_id, struct bnxt *bp = netdev_priv(dev); struct bnxt_ulp *ulp; - ASSERT_RTNL(); if (ulp_id >= BNXT_MAX_ULP) return -EINVAL; @@ -50,7 +51,7 @@ static int bnxt_register_dev(struct bnxt_en_dev *edev, unsigned int ulp_id, return -ENOMEM; } - atomic_set(&ulp->ref_count, 0); + atomic_set(&ulp->ref_count, 1); ulp->handle = handle; rcu_assign_pointer(ulp->ulp_ops, ulp_ops); @@ -69,10 +70,11 @@ static int bnxt_unregister_dev(struct bnxt_en_dev *edev, unsigned int ulp_id) struct bnxt_ulp *ulp; int i = 0; - ASSERT_RTNL(); if (ulp_id >= BNXT_MAX_ULP) return -EINVAL; + edev->flags |= BNXT_EN_FLAG_ULP_STOPPED; + ulp = &edev->ulp_tbl[ulp_id]; if (!rcu_access_pointer(ulp->ulp_ops)) { netdev_err(bp->dev, "ulp id %d not registered\n", ulp_id); @@ -126,7 +128,6 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, int total_vecs; int rc = 0; - ASSERT_RTNL(); if (ulp_id != BNXT_ROCE_ULP) return -EINVAL; @@ -149,10 +150,12 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, max_idx = min_t(int, bp->total_irqs, max_cp_rings); idx = max_idx - avail_msix; } + edev->ulp_tbl[ulp_id].msix_base = idx; edev->ulp_tbl[ulp_id].msix_requested = avail_msix; hw_resc = &bp->hw_resc; total_vecs = idx + avail_msix; + rtnl_lock(); if (bp->total_irqs < total_vecs || (BNXT_NEW_RM(bp) && hw_resc->resv_irqs < total_vecs)) { if (netif_running(dev)) { @@ -162,6 +165,7 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, rc = bnxt_reserve_rings(bp, true); } } + rtnl_unlock(); if (rc) { edev->ulp_tbl[ulp_id].msix_requested = 0; return -EAGAIN; @@ -184,7 +188,6 @@ static int bnxt_free_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id) struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); - ASSERT_RTNL(); if (ulp_id != BNXT_ROCE_ULP) return -EINVAL; @@ -193,10 +196,13 @@ static int bnxt_free_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id) edev->ulp_tbl[ulp_id].msix_requested = 0; edev->flags &= ~BNXT_EN_FLAG_MSIX_REQUESTED; + rtnl_lock(); if (netif_running(dev) && !(edev->flags & BNXT_EN_FLAG_ULP_STOPPED)) { bnxt_close_nic(bp, true, false); bnxt_open_nic(bp, true, false); } + rtnl_unlock(); + return 0; } @@ -347,25 +353,6 @@ void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs) } } -void bnxt_ulp_shutdown(struct bnxt *bp) -{ - struct bnxt_en_dev *edev = bp->edev; - struct bnxt_ulp_ops *ops; - int i; - - if (!edev) - return; - - for (i = 0; i < BNXT_MAX_ULP; i++) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[i]; - - ops = rtnl_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_shutdown) - continue; - ops->ulp_shutdown(ulp->handle); - } -} - void bnxt_ulp_irq_stop(struct bnxt *bp) { struct bnxt_en_dev *edev = bp->edev; @@ -475,6 +462,135 @@ static const struct bnxt_en_ops bnxt_en_ops_tbl = { .bnxt_register_fw_async_events = bnxt_register_async_events, }; +void bnxt_aux_dev_free(struct bnxt *bp) +{ + kfree(bp->aux_dev); + bp->aux_dev = NULL; +} + +static struct bnxt_aux_dev *bnxt_aux_dev_alloc(struct bnxt *bp) +{ + return kzalloc(sizeof(struct bnxt_aux_dev), GFP_KERNEL); +} + +void bnxt_rdma_aux_device_uninit(struct bnxt *bp) +{ + struct bnxt_aux_dev *bnxt_adev; + struct auxiliary_device *adev; + + /* Skip if no auxiliary device init was done. */ + if (!(bp->flags & BNXT_FLAG_ROCE_CAP)) + return; + + bnxt_adev = bp->aux_dev; + adev = &bnxt_adev->aux_dev; + auxiliary_device_delete(adev); + auxiliary_device_uninit(adev); + if (bnxt_adev->id >= 0) + ida_free(&bnxt_aux_dev_ids, bnxt_adev->id); +} + +static void bnxt_aux_dev_release(struct device *dev) +{ + struct bnxt_aux_dev *bnxt_adev = + container_of(dev, struct bnxt_aux_dev, aux_dev.dev); + struct bnxt *bp = netdev_priv(bnxt_adev->edev->net); + + bnxt_adev->edev->en_ops = NULL; + kfree(bnxt_adev->edev); + bnxt_adev->edev = NULL; + bp->edev = NULL; +} + +static void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp) +{ + edev->en_ops = &bnxt_en_ops_tbl; + edev->net = bp->dev; + edev->pdev = bp->pdev; + edev->l2_db_size = bp->db_size; + edev->l2_db_size_nc = bp->db_size; + + if (bp->flags & BNXT_FLAG_ROCEV1_CAP) + edev->flags |= BNXT_EN_FLAG_ROCEV1_CAP; + if (bp->flags & BNXT_FLAG_ROCEV2_CAP) + edev->flags |= BNXT_EN_FLAG_ROCEV2_CAP; +} + +static int bnxt_rdma_aux_device_add(struct bnxt *bp) +{ + struct bnxt_aux_dev *bnxt_adev = bp->aux_dev; + struct bnxt_en_dev *edev = bnxt_adev->edev; + struct auxiliary_device *aux_dev; + int ret; + + edev = kzalloc(sizeof(*edev), GFP_KERNEL); + if (!edev) + return -ENOMEM; + + aux_dev = &bnxt_adev->aux_dev; + aux_dev->id = bnxt_adev->id; + aux_dev->name = "rdma"; + aux_dev->dev.parent = &bp->pdev->dev; + aux_dev->dev.release = bnxt_aux_dev_release; + + bnxt_adev->edev = edev; + bp->edev = edev; + bnxt_set_edev_info(edev, bp); + + ret = auxiliary_device_init(aux_dev); + if (ret) + goto free_edev; + + ret = auxiliary_device_add(aux_dev); + if (ret) + goto aux_dev_uninit; + + return 0; +aux_dev_uninit: + auxiliary_device_uninit(aux_dev); +free_edev: + kfree(edev); + bp->edev = NULL; + + return ret; +} + +void bnxt_rdma_aux_device_init(struct bnxt *bp) +{ + int rc; + + if (!(bp->flags & BNXT_FLAG_ROCE_CAP)) + return; + + bp->aux_dev = bnxt_aux_dev_alloc(bp); + if (!bp->aux_dev) + goto skip_ida_init; + + bp->aux_dev->id = ida_alloc(&bnxt_aux_dev_ids, GFP_KERNEL); + if (bp->aux_dev->id < 0) { + netdev_warn(bp->dev, + "ida alloc failed for ROCE auxiliary device\n"); + goto skip_aux_init; + } + + /* If aux bus init fails, continue with netdev init. */ + rc = bnxt_rdma_aux_device_add(bp); + if (rc) { + netdev_warn(bp->dev, + "Failed to add auxiliary device for ROCE\n"); + goto aux_add_failed; + } + return; + +aux_add_failed: + ida_free(&bnxt_aux_dev_ids, bp->aux_dev->id); + bp->aux_dev->id = -1; +skip_aux_init: + bnxt_aux_dev_free(bp); +skip_ida_init: + bp->flags &= ~BNXT_FLAG_ROCE_CAP; +} + struct bnxt_en_dev *bnxt_ulp_probe(struct net_device *dev) { struct bnxt *bp = netdev_priv(dev); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index 42b50abc3e91..47c7131e5549 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -102,10 +102,12 @@ int bnxt_get_ulp_stat_ctxs(struct bnxt *bp); void bnxt_ulp_stop(struct bnxt *bp); void bnxt_ulp_start(struct bnxt *bp, int err); void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs); -void bnxt_ulp_shutdown(struct bnxt *bp); void bnxt_ulp_irq_stop(struct bnxt *bp); void bnxt_ulp_irq_restart(struct bnxt *bp, int err); void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl); +void bnxt_rdma_aux_device_uninit(struct bnxt *bp); +void bnxt_rdma_aux_device_init(struct bnxt *bp); +void bnxt_aux_dev_free(struct bnxt *bp); struct bnxt_en_dev *bnxt_ulp_probe(struct net_device *dev); #endif From patchwork Thu Jan 12 20:29:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 42804 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp4101437wrt; Thu, 12 Jan 2023 12:58:37 -0800 (PST) X-Google-Smtp-Source: AMrXdXs7ZYB4ck4pO86xYUzDK41HL2BInvczDIr3o9MfL/5WlMSo6SS8cIE36RM+HI33D4UpVKwg X-Received: by 2002:a17:906:910:b0:7c1:9519:5cfa with SMTP id i16-20020a170906091000b007c195195cfamr62135910ejd.77.1673557116717; Thu, 12 Jan 2023 12:58:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673557116; cv=none; d=google.com; s=arc-20160816; b=akRO8bol78UcGlX6uBIlmQzBg+EQYI3BNRsJaTj+itr67KhzS4IwF3z7qf0js2x/+W BpC93gD4bFGOeIIT8C23Uo+gfsWj/kbyyxFe8TBL3v4rQRsMqVP7++e2lQyNMilBjnFF e9RmeQiltLYDjNEzZGzHlFo0Vmch0J3Ch3JYpbBgs0woOlhsUwtoV4Dqr+hUFW2HyA75 pvXCfAvER1qKThUhD6XRgzYkpaidsKTRq9shc6SEqLNH+zLRnbl6Fikexz3R3QMsXSNS CMRhLWXa6msRqaZTJMgIRweVcmt8GCZv0hMFVL2G/rHrsAMWaCbeEvk/zlfPbZmzzaXz +Mdg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=2pa7shTxfw6OT+mULimPTJ/3ubfS7Y3zaGceqvA9/gA=; b=ABu6e7MFdWOC+lwjc5aSWVAPPa6VjgtgLBJyVsEH3L8fl+2TlYCxiXh5LzVw1HRWsZ YF5DhkE6HgSSx0riqcdBln2iFAlHFxGxwHiXgpxJtF4yBv0nYeGB69Ydx5+y/jIILjKr +nxPgQ6nOvKHG+MBoS5RmDuEF88KC/QCciJn9+fmR+GAK02INaImrpx0YUeICJIgUx/U eeT/UUjRS02XPpvIK4dfWXC6fBa9LNfjkF/BvS7og7cQpZiRKLfuBN2F+Vjlrwds5RDL n+au+0IjrFHf/XgT0zl+8C4PQRH76g+rmOsGsHm+LfQH6wJlJ6NbaxP0dX+lWS74MPws SNBg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=R4VGCkQx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g25-20020a170906595900b007c10d4ceddesi14087088ejr.767.2023.01.12.12.58.11; Thu, 12 Jan 2023 12:58:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=R4VGCkQx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240986AbjALU43 (ORCPT + 99 others); Thu, 12 Jan 2023 15:56:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240370AbjALUxK (ORCPT ); Thu, 12 Jan 2023 15:53:10 -0500 Received: from mail-pf1-x435.google.com (mail-pf1-x435.google.com [IPv6:2607:f8b0:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 744F41CB11 for ; Thu, 12 Jan 2023 12:29:49 -0800 (PST) Received: by mail-pf1-x435.google.com with SMTP id g20so14661475pfb.3 for ; Thu, 12 Jan 2023 12:29:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=2pa7shTxfw6OT+mULimPTJ/3ubfS7Y3zaGceqvA9/gA=; b=R4VGCkQxBl4oxyj7PQYFDEgyJz2H3OUXCUAHCo+gEFf9VQpTO8RlWuNfLskY5CaEIl 7d/DLbPKfxOqeZFtRIJwhKVVnbNwPjXS4v7NyjY0qJgCCFo1/0RXGJt2Dj8J4X063fvB lON845Eu0IG4atBWrJY5++aK86NP5MMLrImSo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2pa7shTxfw6OT+mULimPTJ/3ubfS7Y3zaGceqvA9/gA=; b=EUMS66wrMLAXcOuVGb0E6BnotUMWSncy4pin3MVSrmDKnYmx++p4C4ezCAApg5dBPy A1i02OwtbXHtsYGMv9L01AXJKPR1PggGD1AOTtmifu9Bt3TVujbaZCUGqmbK4Vh8FiWK 4Za+1MpgYwMX0JE6iQU89IixWyEWDnNqhk65r8mwm5IiwHwy0IsN0JpgvHJYyVAFItBA rlpMojgaYCAcUdcZZiAmMgOqmz1yWpjHqyNiN41peVkZlaVurLvwq9pdd1U0UrCPNViE i08x/DETgYMMLVnQwbmauNasatOzSYMp7ZJuvrYqItTnb58TMBdBydlDZkLzqibHyMdB 5VCQ== X-Gm-Message-State: AFqh2kpOND10zBoT/LQdGLnTKYSXopItYgFgZww0V1FV4vfPSvPuz0Fa X22uTPHifmRU68cl3Z3S21MTnQ== X-Received: by 2002:a62:6487:0:b0:563:cc80:fb66 with SMTP id y129-20020a626487000000b00563cc80fb66mr59090112pfb.0.1673555388620; Thu, 12 Jan 2023 12:29:48 -0800 (PST) Received: from C02GC2QQMD6T.wifi.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id 129-20020a630887000000b004777c56747csm10283855pgi.11.2023.01.12.12.29.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 12:29:48 -0800 (PST) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, Leon Romanovsky Subject: [PATCH net-next v7 2/8] RDMA/bnxt_re: Use auxiliary driver interface Date: Thu, 12 Jan 2023 12:29:33 -0800 Message-Id: <20230112202939.19562-3-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230112202939.19562-1-ajit.khaparde@broadcom.com> References: <20230112202939.19562-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754851827273698177?= X-GMAIL-MSGID: =?utf-8?q?1754851827273698177?= Use auxiliary driver interface for driver load, unload ROCE driver. The driver does not need to register the interface using the netdev notifier anymore. Removed the bnxt_re_dev_list which is not needed. Currently probe, remove and shutdown ops have been implemented for the auxiliary device. Also remove exccessve validation checks for rdev. Signed-off-by: Ajit Khaparde Reviewed-by: Andy Gospodarek Reviewed-by: Selvin Xavier Reviewed-by: Leon Romanovsky --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 9 +- drivers/infiniband/hw/bnxt_re/main.c | 395 ++++++------------ drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 29 +- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 3 - 4 files changed, 132 insertions(+), 304 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index 785c37cae3c0..b0465c8d229a 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -89,13 +89,6 @@ struct bnxt_re_ring_attr { u8 mode; }; -struct bnxt_re_work { - struct work_struct work; - unsigned long event; - struct bnxt_re_dev *rdev; - struct net_device *vlan_dev; -}; - struct bnxt_re_sqp_entries { struct bnxt_qplib_sge sge; u64 wrid; @@ -132,6 +125,7 @@ struct bnxt_re_dev { #define BNXT_RE_FLAG_ERR_DEVICE_DETACHED 17 #define BNXT_RE_FLAG_ISSUE_ROCE_STATS 29 struct net_device *netdev; + struct notifier_block nb; unsigned int version, major, minor; struct bnxt_qplib_chip_ctx *chip_ctx; struct bnxt_en_dev *en_dev; @@ -194,5 +188,4 @@ static inline struct device *rdev_to_dev(struct bnxt_re_dev *rdev) return &rdev->ibdev.dev; return NULL; } - #endif diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 8c0c80a8d338..165ae11b927d 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -48,6 +48,7 @@ #include #include #include +#include #include #include @@ -74,14 +75,14 @@ MODULE_DESCRIPTION(BNXT_RE_DESC " Driver"); MODULE_LICENSE("Dual BSD/GPL"); /* globals */ -static struct list_head bnxt_re_dev_list = LIST_HEAD_INIT(bnxt_re_dev_list); -/* Mutex to protect the list of bnxt_re devices added */ -static DEFINE_MUTEX(bnxt_re_dev_lock); -static struct workqueue_struct *bnxt_re_wq; -static void bnxt_re_remove_device(struct bnxt_re_dev *rdev); -static void bnxt_re_dealloc_driver(struct ib_device *ib_dev); +static DEFINE_MUTEX(bnxt_re_mutex); + static void bnxt_re_stop_irq(void *handle); static void bnxt_re_dev_stop(struct bnxt_re_dev *rdev); +static int bnxt_re_netdev_event(struct notifier_block *notifier, + unsigned long event, void *ptr); +static struct bnxt_re_dev *bnxt_re_from_netdev(struct net_device *netdev); +static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev); static void bnxt_re_set_drv_mode(struct bnxt_re_dev *rdev, u8 mode) { @@ -233,7 +234,6 @@ static void bnxt_re_stop(void *p) if (!rdev) return; - ASSERT_RTNL(); /* L2 driver invokes this callback during device error/crash or device * reset. Current RoCE driver doesn't recover the device in case of @@ -269,9 +269,6 @@ static void bnxt_re_sriov_config(void *p, int num_vfs) { struct bnxt_re_dev *rdev = p; - if (!rdev) - return; - if (test_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags)) return; rdev->num_vfs = num_vfs; @@ -282,16 +279,14 @@ static void bnxt_re_sriov_config(void *p, int num_vfs) } } -static void bnxt_re_shutdown(void *p) +static void bnxt_re_shutdown(struct auxiliary_device *adev) { - struct bnxt_re_dev *rdev = p; + struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); if (!rdev) return; - ASSERT_RTNL(); - /* Release the MSIx vectors before queuing unregister */ - bnxt_re_stop_irq(rdev); - ib_unregister_device_queued(&rdev->ibdev); + ib_unregister_device(&rdev->ibdev); + bnxt_re_dev_uninit(rdev); } static void bnxt_re_stop_irq(void *handle) @@ -346,11 +341,9 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) } static struct bnxt_ulp_ops bnxt_re_ulp_ops = { - .ulp_async_notifier = NULL, .ulp_stop = bnxt_re_stop, .ulp_start = bnxt_re_start, .ulp_sriov_config = bnxt_re_sriov_config, - .ulp_shutdown = bnxt_re_shutdown, .ulp_irq_stop = bnxt_re_stop_irq, .ulp_irq_restart = bnxt_re_start_irq }; @@ -380,9 +373,6 @@ static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) struct bnxt_en_dev *en_dev; int rc = 0; - if (!rdev) - return -EINVAL; - en_dev = rdev->en_dev; rc = en_dev->en_ops->bnxt_register_device(en_dev, BNXT_ROCE_ULP, @@ -401,7 +391,6 @@ static int bnxt_re_free_msix(struct bnxt_re_dev *rdev) en_dev = rdev->en_dev; - rc = en_dev->en_ops->bnxt_free_msix(rdev->en_dev, BNXT_ROCE_ULP); return rc; @@ -412,9 +401,6 @@ static int bnxt_re_request_msix(struct bnxt_re_dev *rdev) int rc = 0, num_msix_want = BNXT_RE_MAX_MSIX, num_msix_got; struct bnxt_en_dev *en_dev; - if (!rdev) - return -EINVAL; - en_dev = rdev->en_dev; num_msix_want = min_t(u32, BNXT_RE_MAX_MSIX, num_online_cpus()); @@ -458,12 +444,17 @@ static void bnxt_re_fill_fw_msg(struct bnxt_fw_msg *fw_msg, void *msg, static int bnxt_re_net_ring_free(struct bnxt_re_dev *rdev, u16 fw_ring_id, int type) { - struct bnxt_en_dev *en_dev = rdev->en_dev; + struct bnxt_en_dev *en_dev; struct hwrm_ring_free_input req = {0}; struct hwrm_ring_free_output resp; struct bnxt_fw_msg fw_msg; int rc = -EINVAL; + if (!rdev) + return rc; + + en_dev = rdev->en_dev; + if (!en_dev) return rc; @@ -584,21 +575,6 @@ static int bnxt_re_net_stats_ctx_alloc(struct bnxt_re_dev *rdev, /* Device */ -static bool is_bnxt_re_dev(struct net_device *netdev) -{ - struct ethtool_drvinfo drvinfo; - - if (netdev->ethtool_ops && netdev->ethtool_ops->get_drvinfo) { - memset(&drvinfo, 0, sizeof(drvinfo)); - netdev->ethtool_ops->get_drvinfo(netdev, &drvinfo); - - if (strcmp(drvinfo.driver, "bnxt_en")) - return false; - return true; - } - return false; -} - static struct bnxt_re_dev *bnxt_re_from_netdev(struct net_device *netdev) { struct ib_device *ibdev = @@ -609,31 +585,6 @@ static struct bnxt_re_dev *bnxt_re_from_netdev(struct net_device *netdev) return container_of(ibdev, struct bnxt_re_dev, ibdev); } -static struct bnxt_en_dev *bnxt_re_dev_probe(struct net_device *netdev) -{ - struct bnxt_en_dev *en_dev; - struct pci_dev *pdev; - - en_dev = bnxt_ulp_probe(netdev); - if (IS_ERR(en_dev)) - return en_dev; - - pdev = en_dev->pdev; - if (!pdev) - return ERR_PTR(-EINVAL); - - if (!(en_dev->flags & BNXT_EN_FLAG_ROCE_CAP)) { - dev_info(&pdev->dev, - "%s: probe error: RoCE is not supported on this device", - ROCE_DRV_MODULE_NAME); - return ERR_PTR(-ENODEV); - } - - dev_hold(netdev); - - return en_dev; -} - static ssize_t hw_rev_show(struct device *device, struct device_attribute *attr, char *buf) { @@ -679,7 +630,6 @@ static const struct ib_device_ops bnxt_re_dev_ops = { .create_qp = bnxt_re_create_qp, .create_srq = bnxt_re_create_srq, .create_user_ah = bnxt_re_create_ah, - .dealloc_driver = bnxt_re_dealloc_driver, .dealloc_pd = bnxt_re_dealloc_pd, .dealloc_ucontext = bnxt_re_dealloc_ucontext, .del_gid = bnxt_re_del_gid, @@ -744,18 +694,7 @@ static int bnxt_re_register_ib(struct bnxt_re_dev *rdev) return ib_register_device(ibdev, "bnxt_re%d", &rdev->en_dev->pdev->dev); } -static void bnxt_re_dev_remove(struct bnxt_re_dev *rdev) -{ - dev_put(rdev->netdev); - rdev->netdev = NULL; - mutex_lock(&bnxt_re_dev_lock); - list_del_rcu(&rdev->list); - mutex_unlock(&bnxt_re_dev_lock); - - synchronize_rcu(); -} - -static struct bnxt_re_dev *bnxt_re_dev_add(struct net_device *netdev, +static struct bnxt_re_dev *bnxt_re_dev_add(struct bnxt_aux_dev *aux_dev, struct bnxt_en_dev *en_dev) { struct bnxt_re_dev *rdev; @@ -768,8 +707,8 @@ static struct bnxt_re_dev *bnxt_re_dev_add(struct net_device *netdev, return NULL; } /* Default values */ - rdev->netdev = netdev; - dev_hold(rdev->netdev); + rdev->nb.notifier_call = NULL; + rdev->netdev = en_dev->net; rdev->en_dev = en_dev; rdev->id = rdev->en_dev->pdev->devfn; INIT_LIST_HEAD(&rdev->qp_list); @@ -784,9 +723,6 @@ static struct bnxt_re_dev *bnxt_re_dev_add(struct net_device *netdev, rdev->cosq[0] = 0xFFFF; rdev->cosq[1] = 0xFFFF; - mutex_lock(&bnxt_re_dev_lock); - list_add_tail_rcu(&rdev->list, &bnxt_re_dev_list); - mutex_unlock(&bnxt_re_dev_lock); return rdev; } @@ -1323,7 +1259,7 @@ static int bnxt_re_ib_init(struct bnxt_re_dev *rdev) pr_err("Failed to register with IB: %#x\n", rc); return rc; } - dev_info(rdev_to_dev(rdev), "Device registered successfully"); + dev_info(rdev_to_dev(rdev), "Device registered with IB successfully"); ib_get_eth_speed(&rdev->ibdev, 1, &rdev->active_speed, &rdev->active_width); set_bit(BNXT_RE_FLAG_ISSUE_ROCE_STATS, &rdev->flags); @@ -1541,135 +1477,43 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 wqe_mode) return rc; } -static void bnxt_re_dev_unreg(struct bnxt_re_dev *rdev) -{ - struct net_device *netdev = rdev->netdev; - - bnxt_re_dev_remove(rdev); - - if (netdev) - dev_put(netdev); -} - -static int bnxt_re_dev_reg(struct bnxt_re_dev **rdev, struct net_device *netdev) +static int bnxt_re_add_device(struct auxiliary_device *adev, u8 wqe_mode) { + struct bnxt_aux_dev *aux_dev = + container_of(adev, struct bnxt_aux_dev, aux_dev); struct bnxt_en_dev *en_dev; + struct bnxt_re_dev *rdev; int rc = 0; - if (!is_bnxt_re_dev(netdev)) - return -ENODEV; + /* en_dev should never be NULL as long as adev and aux_dev are valid. */ + en_dev = aux_dev->edev; - en_dev = bnxt_re_dev_probe(netdev); - if (IS_ERR(en_dev)) { - if (en_dev != ERR_PTR(-ENODEV)) - ibdev_err(&(*rdev)->ibdev, "%s: Failed to probe\n", - ROCE_DRV_MODULE_NAME); - rc = PTR_ERR(en_dev); - goto exit; - } - *rdev = bnxt_re_dev_add(netdev, en_dev); - if (!*rdev) { + rdev = bnxt_re_dev_add(aux_dev, en_dev); + if (!rdev || !rdev_to_dev(rdev)) { rc = -ENOMEM; - dev_put(netdev); goto exit; } -exit: - return rc; -} - -static void bnxt_re_remove_device(struct bnxt_re_dev *rdev) -{ - bnxt_re_dev_uninit(rdev); - pci_dev_put(rdev->en_dev->pdev); - bnxt_re_dev_unreg(rdev); -} - -static int bnxt_re_add_device(struct bnxt_re_dev **rdev, - struct net_device *netdev, u8 wqe_mode) -{ - int rc; - rc = bnxt_re_dev_reg(rdev, netdev); - if (rc == -ENODEV) - return rc; - if (rc) { - pr_err("Failed to register with the device %s: %#x\n", - netdev->name, rc); - return rc; - } + rc = bnxt_re_dev_init(rdev, wqe_mode); + if (rc) + goto re_dev_dealloc; - pci_dev_get((*rdev)->en_dev->pdev); - rc = bnxt_re_dev_init(*rdev, wqe_mode); + rc = bnxt_re_ib_init(rdev); if (rc) { - pci_dev_put((*rdev)->en_dev->pdev); - bnxt_re_dev_unreg(*rdev); + pr_err("Failed to register with IB: %s", + aux_dev->aux_dev.name); + goto re_dev_uninit; } + auxiliary_set_drvdata(adev, rdev); - return rc; -} - -static void bnxt_re_dealloc_driver(struct ib_device *ib_dev) -{ - struct bnxt_re_dev *rdev = - container_of(ib_dev, struct bnxt_re_dev, ibdev); - - dev_info(rdev_to_dev(rdev), "Unregistering Device"); - - rtnl_lock(); - bnxt_re_remove_device(rdev); - rtnl_unlock(); -} - -/* Handle all deferred netevents tasks */ -static void bnxt_re_task(struct work_struct *work) -{ - struct bnxt_re_work *re_work; - struct bnxt_re_dev *rdev; - int rc = 0; - - re_work = container_of(work, struct bnxt_re_work, work); - rdev = re_work->rdev; - - if (re_work->event == NETDEV_REGISTER) { - rc = bnxt_re_ib_init(rdev); - if (rc) { - ibdev_err(&rdev->ibdev, - "Failed to register with IB: %#x", rc); - rtnl_lock(); - bnxt_re_remove_device(rdev); - rtnl_unlock(); - goto exit; - } - goto exit; - } - - if (!ib_device_try_get(&rdev->ibdev)) - goto exit; + return 0; - switch (re_work->event) { - case NETDEV_UP: - bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, - IB_EVENT_PORT_ACTIVE); - break; - case NETDEV_DOWN: - bnxt_re_dev_stop(rdev); - break; - case NETDEV_CHANGE: - if (!netif_carrier_ok(rdev->netdev)) - bnxt_re_dev_stop(rdev); - else if (netif_carrier_ok(rdev->netdev)) - bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, - IB_EVENT_PORT_ACTIVE); - ib_get_eth_speed(&rdev->ibdev, 1, &rdev->active_speed, - &rdev->active_width); - break; - default: - break; - } - ib_device_put(&rdev->ibdev); +re_dev_uninit: + bnxt_re_dev_uninit(rdev); +re_dev_dealloc: + ib_dealloc_device(&rdev->ibdev); exit: - put_device(&rdev->ibdev.dev); - kfree(re_work); + return rc; } /* @@ -1690,109 +1534,130 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, unsigned long event, void *ptr) { struct net_device *real_dev, *netdev = netdev_notifier_info_to_dev(ptr); - struct bnxt_re_work *re_work; struct bnxt_re_dev *rdev; - int rc = 0; - bool sch_work = false; - bool release = true; real_dev = rdma_vlan_dev_real_dev(netdev); if (!real_dev) real_dev = netdev; - rdev = bnxt_re_from_netdev(real_dev); - if (!rdev && event != NETDEV_REGISTER) - return NOTIFY_OK; - if (real_dev != netdev) goto exit; - switch (event) { - case NETDEV_REGISTER: - if (rdev) - break; - rc = bnxt_re_add_device(&rdev, real_dev, - BNXT_QPLIB_WQE_MODE_STATIC); - if (!rc) - sch_work = true; - release = false; - break; + rdev = bnxt_re_from_netdev(real_dev); + if (!rdev) + return NOTIFY_DONE; - case NETDEV_UNREGISTER: - ib_unregister_device_queued(&rdev->ibdev); - break; + switch (event) { + case NETDEV_UP: + case NETDEV_DOWN: + case NETDEV_CHANGE: + bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, + netif_carrier_ok(real_dev) ? + IB_EVENT_PORT_ACTIVE : + IB_EVENT_PORT_ERR); + break; default: - sch_work = true; break; } - if (sch_work) { - /* Allocate for the deferred task */ - re_work = kzalloc(sizeof(*re_work), GFP_KERNEL); - if (re_work) { - get_device(&rdev->ibdev.dev); - re_work->rdev = rdev; - re_work->event = event; - re_work->vlan_dev = (real_dev == netdev ? - NULL : netdev); - INIT_WORK(&re_work->work, bnxt_re_task); - queue_work(bnxt_re_wq, &re_work->work); - } - } - + ib_device_put(&rdev->ibdev); exit: - if (rdev && release) - ib_device_put(&rdev->ibdev); return NOTIFY_DONE; } -static struct notifier_block bnxt_re_netdev_notifier = { - .notifier_call = bnxt_re_netdev_event -}; +#define BNXT_ADEV_NAME "bnxt_en" -static int __init bnxt_re_mod_init(void) +static void bnxt_re_remove(struct auxiliary_device *adev) { - int rc = 0; + struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); - pr_info("%s: %s", ROCE_DRV_MODULE_NAME, version); + if (!rdev) + return; - bnxt_re_wq = create_singlethread_workqueue("bnxt_re"); - if (!bnxt_re_wq) - return -ENOMEM; + mutex_lock(&bnxt_re_mutex); + if (rdev->nb.notifier_call) { + unregister_netdevice_notifier(&rdev->nb); + rdev->nb.notifier_call = NULL; + } else { + /* If notifier is null, we should have already done a + * clean up before coming here. + */ + goto skip_remove; + } + + ib_unregister_device(&rdev->ibdev); + ib_dealloc_device(&rdev->ibdev); + bnxt_re_dev_uninit(rdev); +skip_remove: + mutex_unlock(&bnxt_re_mutex); +} + +static int bnxt_re_probe(struct auxiliary_device *adev, + const struct auxiliary_device_id *id) +{ + struct bnxt_re_dev *rdev; + int rc; + + mutex_lock(&bnxt_re_mutex); + rc = bnxt_re_add_device(adev, BNXT_QPLIB_WQE_MODE_STATIC); + if (rc) { + mutex_unlock(&bnxt_re_mutex); + return rc; + } - INIT_LIST_HEAD(&bnxt_re_dev_list); + rdev = auxiliary_get_drvdata(adev); - rc = register_netdevice_notifier(&bnxt_re_netdev_notifier); + rdev->nb.notifier_call = bnxt_re_netdev_event; + rc = register_netdevice_notifier(&rdev->nb); if (rc) { + rdev->nb.notifier_call = NULL; pr_err("%s: Cannot register to netdevice_notifier", ROCE_DRV_MODULE_NAME); - goto err_netdev; + goto err; } + + mutex_unlock(&bnxt_re_mutex); return 0; -err_netdev: - destroy_workqueue(bnxt_re_wq); +err: + mutex_unlock(&bnxt_re_mutex); + bnxt_re_remove(adev); return rc; } -static void __exit bnxt_re_mod_exit(void) +static const struct auxiliary_device_id bnxt_re_id_table[] = { + { .name = BNXT_ADEV_NAME ".rdma", }, + {}, +}; + +MODULE_DEVICE_TABLE(auxiliary, bnxt_re_id_table); + +static struct auxiliary_driver bnxt_re_driver = { + .name = "rdma", + .probe = bnxt_re_probe, + .remove = bnxt_re_remove, + .shutdown = bnxt_re_shutdown, + .id_table = bnxt_re_id_table, +}; + +static int __init bnxt_re_mod_init(void) { - struct bnxt_re_dev *rdev; + int rc = 0; - unregister_netdevice_notifier(&bnxt_re_netdev_notifier); - if (bnxt_re_wq) - destroy_workqueue(bnxt_re_wq); - list_for_each_entry(rdev, &bnxt_re_dev_list, list) { - /* VF device removal should be called before the removal - * of PF device. Queue VFs unregister first, so that VFs - * shall be removed before the PF during the call of - * ib_unregister_driver. - */ - if (rdev->is_virtfn) - ib_unregister_device(&rdev->ibdev); + pr_info("%s: %s", ROCE_DRV_MODULE_NAME, version); + rc = auxiliary_driver_register(&bnxt_re_driver); + if (rc) { + pr_err("%s: Failed to register auxiliary driver\n", + ROCE_DRV_MODULE_NAME); + return rc; } - ib_unregister_driver(RDMA_DRIVER_BNXT_RE); + return 0; +} + +static void __exit bnxt_re_mod_exit(void) +{ + auxiliary_driver_unregister(&bnxt_re_driver); } module_init(bnxt_re_mod_init); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index 899d9adfd35f..bc1f8c189251 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -19,6 +19,7 @@ #include #include #include +#include #include "bnxt_hsi.h" #include "bnxt.h" @@ -73,8 +74,6 @@ static int bnxt_unregister_dev(struct bnxt_en_dev *edev, unsigned int ulp_id) if (ulp_id >= BNXT_MAX_ULP) return -EINVAL; - edev->flags |= BNXT_EN_FLAG_ULP_STOPPED; - ulp = &edev->ulp_tbl[ulp_id]; if (!rcu_access_pointer(ulp->ulp_ops)) { netdev_err(bp->dev, "ulp id %d not registered\n", ulp_id); @@ -590,29 +589,3 @@ void bnxt_rdma_aux_device_init(struct bnxt *bp) skip_ida_init: bp->flags &= ~BNXT_FLAG_ROCE_CAP; } - -struct bnxt_en_dev *bnxt_ulp_probe(struct net_device *dev) -{ - struct bnxt *bp = netdev_priv(dev); - struct bnxt_en_dev *edev; - - edev = bp->edev; - if (!edev) { - edev = kzalloc(sizeof(*edev), GFP_KERNEL); - if (!edev) - return ERR_PTR(-ENOMEM); - edev->en_ops = &bnxt_en_ops_tbl; - edev->net = dev; - edev->pdev = bp->pdev; - edev->l2_db_size = bp->db_size; - edev->l2_db_size_nc = bp->db_size; - bp->edev = edev; - } - edev->flags &= ~BNXT_EN_FLAG_ROCE_CAP; - if (bp->flags & BNXT_FLAG_ROCEV1_CAP) - edev->flags |= BNXT_EN_FLAG_ROCEV1_CAP; - if (bp->flags & BNXT_FLAG_ROCEV2_CAP) - edev->flags |= BNXT_EN_FLAG_ROCEV2_CAP; - return bp->edev; -} -EXPORT_SYMBOL(bnxt_ulp_probe); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index 47c7131e5549..d90f8beb6d0e 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -31,7 +31,6 @@ struct bnxt_ulp_ops { void (*ulp_stop)(void *); void (*ulp_start)(void *); void (*ulp_sriov_config)(void *, int); - void (*ulp_shutdown)(void *); void (*ulp_irq_stop)(void *); void (*ulp_irq_restart)(void *, struct bnxt_msix_entry *); }; @@ -108,6 +107,4 @@ void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl); void bnxt_rdma_aux_device_uninit(struct bnxt *bp); void bnxt_rdma_aux_device_init(struct bnxt *bp); void bnxt_aux_dev_free(struct bnxt *bp); -struct bnxt_en_dev *bnxt_ulp_probe(struct net_device *dev); - #endif From patchwork Thu Jan 12 20:29:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 42802 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp4101382wrt; Thu, 12 Jan 2023 12:58:27 -0800 (PST) X-Google-Smtp-Source: AMrXdXvWmvwy0YpgxUYsE7vezNsuNuKEMOGUZNvPhWY+RkSjCHln/EeNEvLagQRs0qVE0utwDejH X-Received: by 2002:a17:907:9394:b0:84d:3adc:7e42 with SMTP id cm20-20020a170907939400b0084d3adc7e42mr15123448ejc.52.1673557107081; Thu, 12 Jan 2023 12:58:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673557107; cv=none; d=google.com; s=arc-20160816; b=I68WtIxZdEaRjhrZ8f0vsRIwTbs83JXk+2d37wofML4DyhcxKTnn4xCOJdoLWFAAz2 0rcRBuj8G8FfwufhYsAMAVaafNo6DgNwjAoiSpyjYZq54DI5Nx3x8fYhm+my00GBktrP KX3iRYSYk7Mq5b/64ZI81vPfxu43m9hQrK3IQgml6FhKRv1hA2+zUj98fkL1O+V1NWBs PefZpeJ6BbLPYCy2jxTAbX8SnysQ/GX77Tx38F+kOhIbQeVpbj/S9bY5P01lW8auL4dq N9ZbIuySxXI8oHPmhiO0uwaKl4JFlhRYSCII8VC4w3PfQsY6qVkqR6NSziNn3vrwmkZv IZcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=FDLYJDimZAA5ltDfClHePflWkOi03xo97EVmesJ4DXg=; b=wEOxsST57lMQTPAsC474o2+qStWWEq83B5unADYOdt5UAG5xqqg8Q6WyNHj7nuUF6p ilr6MBcycqVD5YHfOahTIzH3eJbXnWU+srkFxUWUSMDKbJgrtaxT0MBwa8eDRucjfmMD yinxn0DPjVGT4mNP2sVSVj7QHaW/c/mhkMVbuQ+68n/6eJQL2DOEP+j5lsSoY1jut6Zu h9nozqWsXEm7kdR9Kza5nURc12GluY+sHbfTBN787heo/qkwxn9CD7t93h2SenP5cM8W wpbw1yw0BjEzDusZYm1Z+bv3q/GZSvqisZj+jSt+OWI7W7yoh2NGluEqPZIN9WP0rbMl Rhvg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=TYDl0Gzn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id oz6-20020a1709077d8600b0085fea84a453si5443552ejc.402.2023.01.12.12.58.02; Thu, 12 Jan 2023 12:58:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=TYDl0Gzn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240974AbjALU4U (ORCPT + 99 others); Thu, 12 Jan 2023 15:56:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52952 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240372AbjALUxK (ORCPT ); Thu, 12 Jan 2023 15:53:10 -0500 Received: from mail-pf1-x42a.google.com (mail-pf1-x42a.google.com [IPv6:2607:f8b0:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 25A5A1CB14 for ; Thu, 12 Jan 2023 12:29:51 -0800 (PST) Received: by mail-pf1-x42a.google.com with SMTP id g20so14661529pfb.3 for ; Thu, 12 Jan 2023 12:29:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=FDLYJDimZAA5ltDfClHePflWkOi03xo97EVmesJ4DXg=; b=TYDl0GznJX5s9tcpqPUW+izJkLJ5hxhgSUqwzGMYITcAKpDYRwTUxmHLOAskOn6A+L cUNMARPfLIvekOzVnkTBPWhMVgVPxGSgbxb9CdBuNsHmvLXnoglqziFTHKU1RfJWW2F5 oUMdOE+inXUrYZztAKdqYuxeDYRQY2W2bmJ4M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FDLYJDimZAA5ltDfClHePflWkOi03xo97EVmesJ4DXg=; b=DB0eDba8zfMzrKOF3jtfpaEIwYJvhALwaQQzXiSzwkBCuE1PNbb/g8T7C9FOKeyUrm KUdetDxm42J0hiUlhBFARYpnf8P9JxnsBTGBFNe9+Dgx7J+TfuBtqCXGeZzPbGn/hxUY bXDP/gOuE3l3dpB/OyRXkg7pkV3KLwg/F7j2PjLm/Dms0knZSbpqM5n2cja8L5NeRmQC UUsaHYjUZjdoiGBViXldWo7LnWEJhrljfHJ0lrhYjH25v4QjuX51GE9Lf0C1olECjUDQ yI0wE2FGvSWK7ZIn/1+k9mtmuMSdQqs/Rh4ZNWcFoupxQxlw96w+oPTCWE00VwPFRkz5 IDog== X-Gm-Message-State: AFqh2kqhlXicsAnx5KYgyUnLsH3DAagtM6E7e1tYQKpe5B2vboaz1hKU 7OOaJYexch8molnqd2zSnICzrg== X-Received: by 2002:aa7:9154:0:b0:57b:30b6:9e85 with SMTP id 20-20020aa79154000000b0057b30b69e85mr74283508pfi.22.1673555390276; Thu, 12 Jan 2023 12:29:50 -0800 (PST) Received: from C02GC2QQMD6T.wifi.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id 129-20020a630887000000b004777c56747csm10283855pgi.11.2023.01.12.12.29.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 12:29:49 -0800 (PST) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, Leon Romanovsky Subject: [PATCH net-next v7 3/8] bnxt_en: Remove usage of ulp_id Date: Thu, 12 Jan 2023 12:29:34 -0800 Message-Id: <20230112202939.19562-4-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230112202939.19562-1-ajit.khaparde@broadcom.com> References: <20230112202939.19562-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754851817218582161?= X-GMAIL-MSGID: =?utf-8?q?1754851817218582161?= Since the driver continues to use the single ULP model, the extra complexity and indirection is unnecessary. Remove the usage of ulp_id from the code. Suggested-by: Leon Romanovsky Signed-off-by: Ajit Khaparde Reviewed-by: Andy Gospodarek Reviewed-by: Selvin Xavier Reviewed-by: Leon Romanovsky --- drivers/infiniband/hw/bnxt_re/main.c | 24 +- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 2 +- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 208 ++++++++---------- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 26 +-- 4 files changed, 114 insertions(+), 146 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 165ae11b927d..814f1d2e31b5 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -363,8 +363,7 @@ static int bnxt_re_unregister_netdev(struct bnxt_re_dev *rdev) en_dev = rdev->en_dev; - rc = en_dev->en_ops->bnxt_unregister_device(rdev->en_dev, - BNXT_ROCE_ULP); + rc = en_dev->en_ops->bnxt_unregister_device(rdev->en_dev); return rc; } @@ -375,7 +374,7 @@ static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) en_dev = rdev->en_dev; - rc = en_dev->en_ops->bnxt_register_device(en_dev, BNXT_ROCE_ULP, + rc = en_dev->en_ops->bnxt_register_device(en_dev, &bnxt_re_ulp_ops, rdev); rdev->qplib_res.pdev = rdev->en_dev->pdev; return rc; @@ -384,16 +383,15 @@ static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) static int bnxt_re_free_msix(struct bnxt_re_dev *rdev) { struct bnxt_en_dev *en_dev; - int rc; if (!rdev) return -EINVAL; en_dev = rdev->en_dev; - rc = en_dev->en_ops->bnxt_free_msix(rdev->en_dev, BNXT_ROCE_ULP); + en_dev->en_ops->bnxt_free_msix(rdev->en_dev); - return rc; + return 0; } static int bnxt_re_request_msix(struct bnxt_re_dev *rdev) @@ -405,7 +403,7 @@ static int bnxt_re_request_msix(struct bnxt_re_dev *rdev) num_msix_want = min_t(u32, BNXT_RE_MAX_MSIX, num_online_cpus()); - num_msix_got = en_dev->en_ops->bnxt_request_msix(en_dev, BNXT_ROCE_ULP, + num_msix_got = en_dev->en_ops->bnxt_request_msix(en_dev, rdev->msix_entries, num_msix_want); if (num_msix_got < BNXT_RE_MIN_MSIX) { @@ -468,7 +466,7 @@ static int bnxt_re_net_ring_free(struct bnxt_re_dev *rdev, req.ring_id = cpu_to_le16(fw_ring_id); bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (rc) ibdev_err(&rdev->ibdev, "Failed to free HW ring:%d :%#x", req.ring_id, rc); @@ -505,7 +503,7 @@ static int bnxt_re_net_ring_alloc(struct bnxt_re_dev *rdev, req.int_mode = ring_attr->mode; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (!rc) *fw_ring_id = le16_to_cpu(resp.ring_id); @@ -533,7 +531,7 @@ static int bnxt_re_net_stats_ctx_free(struct bnxt_re_dev *rdev, req.stat_ctx_id = cpu_to_le32(fw_stats_ctx_id); bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (rc) ibdev_err(&rdev->ibdev, "Failed to free HW stats context %#x", rc); @@ -566,7 +564,7 @@ static int bnxt_re_net_stats_ctx_alloc(struct bnxt_re_dev *rdev, req.stat_ctx_flags = STAT_CTX_ALLOC_REQ_STAT_CTX_FLAGS_ROCE; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (!rc) *fw_stats_ctx_id = le32_to_cpu(resp.stat_ctx_id); @@ -1052,7 +1050,7 @@ static int bnxt_re_query_hwrm_pri2cos(struct bnxt_re_dev *rdev, u8 dir, bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (rc) return rc; @@ -1235,7 +1233,7 @@ static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) req.hwrm_intf_upd = HWRM_VERSION_UPDATE; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (rc) { ibdev_err(&rdev->ibdev, "Failed to query HW version, rc = 0x%x", rc); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index c5cbf2307e07..fc0054c284c5 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -5538,7 +5538,7 @@ int bnxt_hwrm_vnic_cfg(struct bnxt *bp, u16 vnic_id) #endif if ((bp->flags & BNXT_FLAG_STRIP_VLAN) || def_vlan) req->flags |= cpu_to_le32(VNIC_CFG_REQ_FLAGS_VLAN_STRIP_MODE); - if (!vnic_id && bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) + if (!vnic_id && bnxt_ulp_registered(bp->edev)) req->flags |= cpu_to_le32(bnxt_get_roce_vnic_mode(bp)); return hwrm_req_send(bp, req); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index bc1f8c189251..988b09a9e99e 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -28,59 +28,44 @@ static DEFINE_IDA(bnxt_aux_dev_ids); -static int bnxt_register_dev(struct bnxt_en_dev *edev, unsigned int ulp_id, - struct bnxt_ulp_ops *ulp_ops, void *handle) +static int bnxt_register_dev(struct bnxt_en_dev *edev, + struct bnxt_ulp_ops *ulp_ops, + void *handle) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); + unsigned int max_stat_ctxs; struct bnxt_ulp *ulp; - if (ulp_id >= BNXT_MAX_ULP) - return -EINVAL; - - ulp = &edev->ulp_tbl[ulp_id]; - if (rcu_access_pointer(ulp->ulp_ops)) { - netdev_err(bp->dev, "ulp id %d already registered\n", ulp_id); - return -EBUSY; - } - if (ulp_id == BNXT_ROCE_ULP) { - unsigned int max_stat_ctxs; + max_stat_ctxs = bnxt_get_max_func_stat_ctxs(bp); + if (max_stat_ctxs <= BNXT_MIN_ROCE_STAT_CTXS || + bp->cp_nr_rings == max_stat_ctxs) + return -ENOMEM; - max_stat_ctxs = bnxt_get_max_func_stat_ctxs(bp); - if (max_stat_ctxs <= BNXT_MIN_ROCE_STAT_CTXS || - bp->cp_nr_rings == max_stat_ctxs) - return -ENOMEM; - } + ulp = kzalloc(sizeof(*ulp), GFP_KERNEL); + if (!ulp) + return -ENOMEM; - atomic_set(&ulp->ref_count, 1); + edev->ulp_tbl = ulp; ulp->handle = handle; rcu_assign_pointer(ulp->ulp_ops, ulp_ops); - if (ulp_id == BNXT_ROCE_ULP) { - if (test_bit(BNXT_STATE_OPEN, &bp->state)) - bnxt_hwrm_vnic_cfg(bp, 0); - } + if (test_bit(BNXT_STATE_OPEN, &bp->state)) + bnxt_hwrm_vnic_cfg(bp, 0); return 0; } -static int bnxt_unregister_dev(struct bnxt_en_dev *edev, unsigned int ulp_id) +static int bnxt_unregister_dev(struct bnxt_en_dev *edev) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); struct bnxt_ulp *ulp; int i = 0; - if (ulp_id >= BNXT_MAX_ULP) - return -EINVAL; - - ulp = &edev->ulp_tbl[ulp_id]; - if (!rcu_access_pointer(ulp->ulp_ops)) { - netdev_err(bp->dev, "ulp id %d not registered\n", ulp_id); - return -EINVAL; - } - if (ulp_id == BNXT_ROCE_ULP && ulp->msix_requested) - edev->en_ops->bnxt_free_msix(edev, ulp_id); + ulp = edev->ulp_tbl; + if (ulp->msix_requested) + edev->en_ops->bnxt_free_msix(edev); if (ulp->max_async_event_id) bnxt_hwrm_func_drv_rgtr(bp, NULL, 0, true); @@ -93,6 +78,8 @@ static int bnxt_unregister_dev(struct bnxt_en_dev *edev, unsigned int ulp_id) msleep(100); i++; } + kfree(ulp); + edev->ulp_tbl = NULL; return 0; } @@ -101,8 +88,8 @@ static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) struct bnxt_en_dev *edev = bp->edev; int num_msix, idx, i; - num_msix = edev->ulp_tbl[BNXT_ROCE_ULP].msix_requested; - idx = edev->ulp_tbl[BNXT_ROCE_ULP].msix_base; + num_msix = edev->ulp_tbl->msix_requested; + idx = edev->ulp_tbl->msix_base; for (i = 0; i < num_msix; i++) { ent[i].vector = bp->irq_tbl[idx + i].vector; ent[i].ring_idx = idx + i; @@ -116,8 +103,9 @@ static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) } } -static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, - struct bnxt_msix_entry *ent, int num_msix) +static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, + struct bnxt_msix_entry *ent, + int num_msix) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); @@ -127,13 +115,10 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, int total_vecs; int rc = 0; - if (ulp_id != BNXT_ROCE_ULP) - return -EINVAL; - if (!(bp->flags & BNXT_FLAG_USING_MSIX)) return -ENODEV; - if (edev->ulp_tbl[ulp_id].msix_requested) + if (edev->ulp_tbl->msix_requested) return -EAGAIN; max_cp_rings = bnxt_get_max_func_cp_rings(bp); @@ -150,8 +135,8 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, idx = max_idx - avail_msix; } - edev->ulp_tbl[ulp_id].msix_base = idx; - edev->ulp_tbl[ulp_id].msix_requested = avail_msix; + edev->ulp_tbl->msix_base = idx; + edev->ulp_tbl->msix_requested = avail_msix; hw_resc = &bp->hw_resc; total_vecs = idx + avail_msix; rtnl_lock(); @@ -166,7 +151,7 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, } rtnl_unlock(); if (rc) { - edev->ulp_tbl[ulp_id].msix_requested = 0; + edev->ulp_tbl->msix_requested = 0; return -EAGAIN; } @@ -175,25 +160,22 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, resv_msix = hw_resc->resv_irqs - bp->cp_nr_rings; avail_msix = min_t(int, resv_msix, avail_msix); - edev->ulp_tbl[ulp_id].msix_requested = avail_msix; + edev->ulp_tbl->msix_requested = avail_msix; } bnxt_fill_msix_vecs(bp, ent); edev->flags |= BNXT_EN_FLAG_MSIX_REQUESTED; return avail_msix; } -static int bnxt_free_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id) +static void bnxt_free_msix_vecs(struct bnxt_en_dev *edev) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); - if (ulp_id != BNXT_ROCE_ULP) - return -EINVAL; - if (!(edev->flags & BNXT_EN_FLAG_MSIX_REQUESTED)) - return 0; + return; - edev->ulp_tbl[ulp_id].msix_requested = 0; + edev->ulp_tbl->msix_requested = 0; edev->flags &= ~BNXT_EN_FLAG_MSIX_REQUESTED; rtnl_lock(); if (netif_running(dev) && !(edev->flags & BNXT_EN_FLAG_ULP_STOPPED)) { @@ -202,43 +184,43 @@ static int bnxt_free_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id) } rtnl_unlock(); - return 0; + return; } int bnxt_get_ulp_msix_num(struct bnxt *bp) { - if (bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) { + if (bnxt_ulp_registered(bp->edev)) { struct bnxt_en_dev *edev = bp->edev; - return edev->ulp_tbl[BNXT_ROCE_ULP].msix_requested; + return edev->ulp_tbl->msix_requested; } return 0; } int bnxt_get_ulp_msix_base(struct bnxt *bp) { - if (bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) { + if (bnxt_ulp_registered(bp->edev)) { struct bnxt_en_dev *edev = bp->edev; - if (edev->ulp_tbl[BNXT_ROCE_ULP].msix_requested) - return edev->ulp_tbl[BNXT_ROCE_ULP].msix_base; + if (edev->ulp_tbl->msix_requested) + return edev->ulp_tbl->msix_base; } return 0; } int bnxt_get_ulp_stat_ctxs(struct bnxt *bp) { - if (bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) { + if (bnxt_ulp_registered(bp->edev)) { struct bnxt_en_dev *edev = bp->edev; - if (edev->ulp_tbl[BNXT_ROCE_ULP].msix_requested) + if (edev->ulp_tbl->msix_requested) return BNXT_MIN_ROCE_STAT_CTXS; } return 0; } -static int bnxt_send_msg(struct bnxt_en_dev *edev, unsigned int ulp_id, +static int bnxt_send_msg(struct bnxt_en_dev *edev, struct bnxt_fw_msg *fw_msg) { struct net_device *dev = edev->net; @@ -248,7 +230,7 @@ static int bnxt_send_msg(struct bnxt_en_dev *edev, unsigned int ulp_id, u32 resp_len; int rc; - if (ulp_id != BNXT_ROCE_ULP && bp->fw_reset_state) + if (bp->fw_reset_state) return -EBUSY; rc = hwrm_req_init(bp, req, 0 /* don't care */); @@ -287,27 +269,24 @@ void bnxt_ulp_stop(struct bnxt *bp) { struct bnxt_en_dev *edev = bp->edev; struct bnxt_ulp_ops *ops; - int i; + struct bnxt_ulp *ulp; if (!edev) return; edev->flags |= BNXT_EN_FLAG_ULP_STOPPED; - for (i = 0; i < BNXT_MAX_ULP; i++) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[i]; - - ops = rtnl_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_stop) - continue; - ops->ulp_stop(ulp->handle); - } + ulp = edev->ulp_tbl; + ops = rtnl_dereference(ulp->ulp_ops); + if (!ops || !ops->ulp_stop) + return; + ops->ulp_stop(ulp->handle); } void bnxt_ulp_start(struct bnxt *bp, int err) { struct bnxt_en_dev *edev = bp->edev; struct bnxt_ulp_ops *ops; - int i; + struct bnxt_ulp *ulp; if (!edev) return; @@ -317,39 +296,33 @@ void bnxt_ulp_start(struct bnxt *bp, int err) if (err) return; - for (i = 0; i < BNXT_MAX_ULP; i++) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[i]; - - ops = rtnl_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_start) - continue; - ops->ulp_start(ulp->handle); - } + ulp = edev->ulp_tbl; + ops = rtnl_dereference(ulp->ulp_ops); + if (!ops || !ops->ulp_start) + return; + ops->ulp_start(ulp->handle); } void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs) { struct bnxt_en_dev *edev = bp->edev; struct bnxt_ulp_ops *ops; - int i; + struct bnxt_ulp *ulp; if (!edev) return; + ulp = edev->ulp_tbl; - for (i = 0; i < BNXT_MAX_ULP; i++) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[i]; - - rcu_read_lock(); - ops = rcu_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_sriov_config) { - rcu_read_unlock(); - continue; - } - bnxt_ulp_get(ulp); + rcu_read_lock(); + ops = rcu_dereference(ulp->ulp_ops); + if (!ops || !ops->ulp_sriov_config) { rcu_read_unlock(); - ops->ulp_sriov_config(ulp->handle, num_vfs); - bnxt_ulp_put(ulp); + return; } + bnxt_ulp_get(ulp); + rcu_read_unlock(); + ops->ulp_sriov_config(ulp->handle, num_vfs); + bnxt_ulp_put(ulp); } void bnxt_ulp_irq_stop(struct bnxt *bp) @@ -360,8 +333,8 @@ void bnxt_ulp_irq_stop(struct bnxt *bp) if (!edev || !(edev->flags & BNXT_EN_FLAG_MSIX_REQUESTED)) return; - if (bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[BNXT_ROCE_ULP]; + if (bnxt_ulp_registered(bp->edev)) { + struct bnxt_ulp *ulp = edev->ulp_tbl; if (!ulp->msix_requested) return; @@ -381,8 +354,8 @@ void bnxt_ulp_irq_restart(struct bnxt *bp, int err) if (!edev || !(edev->flags & BNXT_EN_FLAG_MSIX_REQUESTED)) return; - if (bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[BNXT_ROCE_ULP]; + if (bnxt_ulp_registered(bp->edev)) { + struct bnxt_ulp *ulp = edev->ulp_tbl; struct bnxt_msix_entry *ent = NULL; if (!ulp->msix_requested) @@ -409,41 +382,38 @@ void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl) u16 event_id = le16_to_cpu(cmpl->event_id); struct bnxt_en_dev *edev = bp->edev; struct bnxt_ulp_ops *ops; - int i; + struct bnxt_ulp *ulp; - if (!edev) + if (!bnxt_ulp_registered(edev)) return; + ulp = edev->ulp_tbl; + rcu_read_lock(); - for (i = 0; i < BNXT_MAX_ULP; i++) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[i]; - - ops = rcu_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_async_notifier) - continue; - if (!ulp->async_events_bmap || - event_id > ulp->max_async_event_id) - continue; - - /* Read max_async_event_id first before testing the bitmap. */ - smp_rmb(); - if (test_bit(event_id, ulp->async_events_bmap)) - ops->ulp_async_notifier(ulp->handle, cmpl); - } + + ops = rcu_dereference(ulp->ulp_ops); + if (!ops || !ops->ulp_async_notifier) + goto exit; + if (!ulp->async_events_bmap || event_id > ulp->max_async_event_id) + goto exit; + + /* Read max_async_event_id first before testing the bitmap. */ + smp_rmb(); + if (test_bit(event_id, ulp->async_events_bmap)) + ops->ulp_async_notifier(ulp->handle, cmpl); +exit: rcu_read_unlock(); } -static int bnxt_register_async_events(struct bnxt_en_dev *edev, unsigned int ulp_id, - unsigned long *events_bmap, u16 max_id) +static int bnxt_register_async_events(struct bnxt_en_dev *edev, + unsigned long *events_bmap, + u16 max_id) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); struct bnxt_ulp *ulp; - if (ulp_id >= BNXT_MAX_ULP) - return -EINVAL; - - ulp = &edev->ulp_tbl[ulp_id]; + ulp = edev->ulp_tbl; ulp->async_events_bmap = events_bmap; /* Make sure bnxt_ulp_async_events() sees this order */ smp_wmb(); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index d90f8beb6d0e..5433f3e17746 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -64,7 +64,7 @@ struct bnxt_en_dev { #define BNXT_EN_FLAG_MSIX_REQUESTED 0x4 #define BNXT_EN_FLAG_ULP_STOPPED 0x8 const struct bnxt_en_ops *en_ops; - struct bnxt_ulp ulp_tbl[BNXT_MAX_ULP]; + struct bnxt_ulp *ulp_tbl; int l2_db_size; /* Doorbell BAR size in * bytes mapped by L2 * driver. @@ -76,21 +76,21 @@ struct bnxt_en_dev { }; struct bnxt_en_ops { - int (*bnxt_register_device)(struct bnxt_en_dev *, unsigned int, - struct bnxt_ulp_ops *, void *); - int (*bnxt_unregister_device)(struct bnxt_en_dev *, unsigned int); - int (*bnxt_request_msix)(struct bnxt_en_dev *, unsigned int, - struct bnxt_msix_entry *, int); - int (*bnxt_free_msix)(struct bnxt_en_dev *, unsigned int); - int (*bnxt_send_fw_msg)(struct bnxt_en_dev *, unsigned int, - struct bnxt_fw_msg *); - int (*bnxt_register_fw_async_events)(struct bnxt_en_dev *, unsigned int, - unsigned long *, u16); + int (*bnxt_register_device)(struct bnxt_en_dev *edev, + struct bnxt_ulp_ops *ulp_ops, void *handle); + int (*bnxt_unregister_device)(struct bnxt_en_dev *edev); + int (*bnxt_request_msix)(struct bnxt_en_dev *edev, + struct bnxt_msix_entry *ent, int num_msix); + void (*bnxt_free_msix)(struct bnxt_en_dev *edev); + int (*bnxt_send_fw_msg)(struct bnxt_en_dev *edev, + struct bnxt_fw_msg *fw_msg); + int (*bnxt_register_fw_async_events)(struct bnxt_en_dev *edev, + unsigned long *events_bmap, u16 max_id); }; -static inline bool bnxt_ulp_registered(struct bnxt_en_dev *edev, int ulp_id) +static inline bool bnxt_ulp_registered(struct bnxt_en_dev *edev) { - if (edev && rcu_access_pointer(edev->ulp_tbl[ulp_id].ulp_ops)) + if (edev && edev->ulp_tbl) return true; return false; } From patchwork Thu Jan 12 20:29:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 42803 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp4101402wrt; Thu, 12 Jan 2023 12:58:31 -0800 (PST) X-Google-Smtp-Source: AMrXdXspE4o6VxJDlyPvhFkkuHhqwZyS/wMCvIiPytEdVdwIyx61gmOjKkdUqrD+rCktei5cXFI0 X-Received: by 2002:a05:6402:28cd:b0:46c:fabc:5897 with SMTP id ef13-20020a05640228cd00b0046cfabc5897mr65402362edb.7.1673557111208; Thu, 12 Jan 2023 12:58:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673557111; cv=none; d=google.com; s=arc-20160816; b=UROCfLUK/7j1EAS8XmxRjhBMnsMpH8DQzMBDIfrdl6styYRWSf8hn2v9gaj1SqUMAw Sq4AQVZKe5wRVALBXtxVKCexlAXvWKGf25fX/yXSf+suemI4qKFHyYxy5ZkK8cMtkIm7 t9XRErJ39NBGYAq89hGlpGVJ3i8zzXNc2SaWJepJvfC1Egn9V06c9oIP6uFxgR9FUJUM j+IO/ADXRMmjuC6OcRA+desBVZD8Eq/RbB4ZnwAVjI2V0qdPw9aLYU73O/owbFZFiFuu aMM7J9sL7WkdEXKSqkamY6S01inS2hchIDH9WCa2OmaHq9wRkAwJmLBhzAX+ucPDfnE0 SiFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=S+FtuMg8USUlPqCEri6xRXm/9B8/NpT/g14jRh8IrCY=; b=eEi0xm4uDxJa/InWQnJaGhx1x/TilYZBOnxEPuuyKn1QL3WzcY7HwGaDOQqAUfx5E4 4SEecipUW0BIhp60fU59pdn8V+Is75Kj2BhzJi52NBpYHoWB6AFJZz4r9VonkRFZMRxc nMOLsyvyb7QDbgVQt9cyVQH1r+XcF2silP88ZmW6xL5X3LGj3Y3nLc2CaFQpt/CjXoyZ HqG+szCxcEMszMxge+Fs+jtWI6QYTkh5Vl2B61rxXu/Lbf2EPwzHGfvrIboKQE4W393P 7m7brvt3KdaSTpGEnWQu4623vRXkTTDl3tFRFX+LwfYgh6wxzSHBQAXe9QOXhB8kJZn6 fYFQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=FVFC3Ibb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w20-20020a056402269400b0044f2fb68fe6si21583200edd.495.2023.01.12.12.58.06; Thu, 12 Jan 2023 12:58:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=FVFC3Ibb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240981AbjALU4Y (ORCPT + 99 others); Thu, 12 Jan 2023 15:56:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52918 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232730AbjALUxU (ORCPT ); Thu, 12 Jan 2023 15:53:20 -0500 Received: from mail-pl1-x636.google.com (mail-pl1-x636.google.com [IPv6:2607:f8b0:4864:20::636]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA3D01CFEC for ; Thu, 12 Jan 2023 12:29:52 -0800 (PST) Received: by mail-pl1-x636.google.com with SMTP id jl4so21359218plb.8 for ; Thu, 12 Jan 2023 12:29:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=S+FtuMg8USUlPqCEri6xRXm/9B8/NpT/g14jRh8IrCY=; b=FVFC3IbbX0gJ2NJdKF2O4v5rz2QFKfJ5t2RClAsH95coGRCwmY/ZBMdQhKwC1IkOVQ l6ofd5N06Kd5y6ty7RlhHF08l/e2OkiprNKxS44gq2HNv86Ll3pdKvEd2GeoMnPPHkMi Sj5DNeLQeOZxgjMEmhWUWwtVU/RHH+Z+lFvq0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=S+FtuMg8USUlPqCEri6xRXm/9B8/NpT/g14jRh8IrCY=; b=TVX/PuQ5AijvL4GUVLyyiHQnQG3Pci3MhoUkoWSbIqgn9B8ZRmXMgx5hum/74KY8+O xwJxeXPMIGcRjP4OR9Fu29NLSZVm7gbCOLofdeF6Jeu3CgkrnTjE7jLo/oDcQzb4VnrL BvpQPZRV7iptKjN90qflrKR5uDGb6su1UtS8A9ixSS85bk8Y44+/F0rGKcIAGMCsdqhU rAjcm5d+pCYRwhH/OsO/EO2WQr+2njyGOqIX0SVyrkSsjL954R6omoXkEEAEYZqHUvYJ ca0l1DSm7SMHr29tQQBureDhHVjuXMgnUPgDnyJzAGyTbAj+Q/2Zhh9/WCFqQgMlYkSo tTDQ== X-Gm-Message-State: AFqh2krHVGXf6o88HX4u0OTlNO77ND0aSuB4rNlPPJS1OktslwXfZVzb Xz1M/lcnLKAq/+Iob035ByCE9Q== X-Received: by 2002:a05:6a20:93a7:b0:b6:157d:bf48 with SMTP id x39-20020a056a2093a700b000b6157dbf48mr14422174pzh.43.1673555392090; Thu, 12 Jan 2023 12:29:52 -0800 (PST) Received: from C02GC2QQMD6T.wifi.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id 129-20020a630887000000b004777c56747csm10283855pgi.11.2023.01.12.12.29.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 12:29:51 -0800 (PST) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, Leon Romanovsky Subject: [PATCH net-next v7 4/8] bnxt_en: Use direct API instead of indirection Date: Thu, 12 Jan 2023 12:29:35 -0800 Message-Id: <20230112202939.19562-5-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230112202939.19562-1-ajit.khaparde@broadcom.com> References: <20230112202939.19562-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754851821137278060?= X-GMAIL-MSGID: =?utf-8?q?1754851821137278060?= For a single ULP user there is no need for complicating function indirection calls. Remove all this complexity in favour of direct function calls exported by the bnxt_en driver. This allows to simplify the code greatly. Also remove unused ulp_async_notifier. Suggested-by: Leon Romanovsky Signed-off-by: Ajit Khaparde Reviewed-by: Andy Gospodarek Reviewed-by: Selvin Xavier Reviewed-by: Leon Romanovsky --- drivers/infiniband/hw/bnxt_re/main.c | 71 ++++------------- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 1 - drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 76 +++++-------------- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 25 +++--- 4 files changed, 45 insertions(+), 128 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 814f1d2e31b5..e1777c8e6d5e 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -350,23 +350,6 @@ static struct bnxt_ulp_ops bnxt_re_ulp_ops = { /* RoCE -> Net driver */ -/* Driver registration routines used to let the networking driver (bnxt_en) - * to know that the RoCE driver is now installed - */ -static int bnxt_re_unregister_netdev(struct bnxt_re_dev *rdev) -{ - struct bnxt_en_dev *en_dev; - int rc; - - if (!rdev) - return -EINVAL; - - en_dev = rdev->en_dev; - - rc = en_dev->en_ops->bnxt_unregister_device(rdev->en_dev); - return rc; -} - static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) { struct bnxt_en_dev *en_dev; @@ -374,26 +357,12 @@ static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) en_dev = rdev->en_dev; - rc = en_dev->en_ops->bnxt_register_device(en_dev, - &bnxt_re_ulp_ops, rdev); - rdev->qplib_res.pdev = rdev->en_dev->pdev; + rc = bnxt_register_dev(en_dev, &bnxt_re_ulp_ops, rdev); + if (!rc) + rdev->qplib_res.pdev = rdev->en_dev->pdev; return rc; } -static int bnxt_re_free_msix(struct bnxt_re_dev *rdev) -{ - struct bnxt_en_dev *en_dev; - - if (!rdev) - return -EINVAL; - - en_dev = rdev->en_dev; - - en_dev->en_ops->bnxt_free_msix(rdev->en_dev); - - return 0; -} - static int bnxt_re_request_msix(struct bnxt_re_dev *rdev) { int rc = 0, num_msix_want = BNXT_RE_MAX_MSIX, num_msix_got; @@ -403,9 +372,9 @@ static int bnxt_re_request_msix(struct bnxt_re_dev *rdev) num_msix_want = min_t(u32, BNXT_RE_MAX_MSIX, num_online_cpus()); - num_msix_got = en_dev->en_ops->bnxt_request_msix(en_dev, - rdev->msix_entries, - num_msix_want); + num_msix_got = bnxt_req_msix_vecs(en_dev, + rdev->msix_entries, + num_msix_want); if (num_msix_got < BNXT_RE_MIN_MSIX) { rc = -EINVAL; goto done; @@ -466,7 +435,7 @@ static int bnxt_re_net_ring_free(struct bnxt_re_dev *rdev, req.ring_id = cpu_to_le16(fw_ring_id); bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (rc) ibdev_err(&rdev->ibdev, "Failed to free HW ring:%d :%#x", req.ring_id, rc); @@ -503,7 +472,7 @@ static int bnxt_re_net_ring_alloc(struct bnxt_re_dev *rdev, req.int_mode = ring_attr->mode; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (!rc) *fw_ring_id = le16_to_cpu(resp.ring_id); @@ -531,7 +500,7 @@ static int bnxt_re_net_stats_ctx_free(struct bnxt_re_dev *rdev, req.stat_ctx_id = cpu_to_le32(fw_stats_ctx_id); bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (rc) ibdev_err(&rdev->ibdev, "Failed to free HW stats context %#x", rc); @@ -564,7 +533,7 @@ static int bnxt_re_net_stats_ctx_alloc(struct bnxt_re_dev *rdev, req.stat_ctx_flags = STAT_CTX_ALLOC_REQ_STAT_CTX_FLAGS_ROCE; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (!rc) *fw_stats_ctx_id = le32_to_cpu(resp.stat_ctx_id); @@ -1050,7 +1019,7 @@ static int bnxt_re_query_hwrm_pri2cos(struct bnxt_re_dev *rdev, u8 dir, bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (rc) return rc; @@ -1233,7 +1202,7 @@ static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) req.hwrm_intf_upd = HWRM_VERSION_UPDATE; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (rc) { ibdev_err(&rdev->ibdev, "Failed to query HW version, rc = 0x%x", rc); @@ -1296,20 +1265,12 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev) bnxt_re_net_ring_free(rdev, rdev->rcfw.creq.ring_id, type); bnxt_qplib_free_rcfw_channel(&rdev->rcfw); } - if (test_and_clear_bit(BNXT_RE_FLAG_GOT_MSIX, &rdev->flags)) { - rc = bnxt_re_free_msix(rdev); - if (rc) - ibdev_warn(&rdev->ibdev, - "Failed to free MSI-X vectors: %#x", rc); - } + if (test_and_clear_bit(BNXT_RE_FLAG_GOT_MSIX, &rdev->flags)) + bnxt_free_msix_vecs(rdev->en_dev); bnxt_re_destroy_chip_ctx(rdev); - if (test_and_clear_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags)) { - rc = bnxt_re_unregister_netdev(rdev); - if (rc) - ibdev_warn(&rdev->ibdev, - "Failed to unregister with netdev: %#x", rc); - } + if (test_and_clear_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags)) + bnxt_unregister_dev(rdev->en_dev); } /* worker thread for polling periodic events. Now used for QoS programming*/ diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index fc0054c284c5..f24e08bbeaf4 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -2414,7 +2414,6 @@ static int bnxt_async_event_process(struct bnxt *bp, } bnxt_queue_sp_work(bp); async_event_process_exit: - bnxt_ulp_async_events(bp, cmpl); return 0; } diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index 988b09a9e99e..e7539946759d 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -28,9 +28,9 @@ static DEFINE_IDA(bnxt_aux_dev_ids); -static int bnxt_register_dev(struct bnxt_en_dev *edev, - struct bnxt_ulp_ops *ulp_ops, - void *handle) +int bnxt_register_dev(struct bnxt_en_dev *edev, + struct bnxt_ulp_ops *ulp_ops, + void *handle) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); @@ -55,8 +55,9 @@ static int bnxt_register_dev(struct bnxt_en_dev *edev, return 0; } +EXPORT_SYMBOL(bnxt_register_dev); -static int bnxt_unregister_dev(struct bnxt_en_dev *edev) +void bnxt_unregister_dev(struct bnxt_en_dev *edev) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); @@ -65,7 +66,7 @@ static int bnxt_unregister_dev(struct bnxt_en_dev *edev) ulp = edev->ulp_tbl; if (ulp->msix_requested) - edev->en_ops->bnxt_free_msix(edev); + bnxt_free_msix_vecs(edev); if (ulp->max_async_event_id) bnxt_hwrm_func_drv_rgtr(bp, NULL, 0, true); @@ -80,8 +81,9 @@ static int bnxt_unregister_dev(struct bnxt_en_dev *edev) } kfree(ulp); edev->ulp_tbl = NULL; - return 0; + return; } +EXPORT_SYMBOL(bnxt_unregister_dev); static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) { @@ -103,7 +105,7 @@ static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) } } -static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, +int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, struct bnxt_msix_entry *ent, int num_msix) { @@ -166,8 +168,9 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, edev->flags |= BNXT_EN_FLAG_MSIX_REQUESTED; return avail_msix; } +EXPORT_SYMBOL(bnxt_req_msix_vecs); -static void bnxt_free_msix_vecs(struct bnxt_en_dev *edev) +void bnxt_free_msix_vecs(struct bnxt_en_dev *edev) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); @@ -186,6 +189,7 @@ static void bnxt_free_msix_vecs(struct bnxt_en_dev *edev) return; } +EXPORT_SYMBOL(bnxt_free_msix_vecs); int bnxt_get_ulp_msix_num(struct bnxt *bp) { @@ -220,7 +224,7 @@ int bnxt_get_ulp_stat_ctxs(struct bnxt *bp) return 0; } -static int bnxt_send_msg(struct bnxt_en_dev *edev, +int bnxt_send_msg(struct bnxt_en_dev *edev, struct bnxt_fw_msg *fw_msg) { struct net_device *dev = edev->net; @@ -254,6 +258,7 @@ static int bnxt_send_msg(struct bnxt_en_dev *edev, hwrm_req_drop(bp, req); return rc; } +EXPORT_SYMBOL(bnxt_send_msg); static void bnxt_ulp_get(struct bnxt_ulp *ulp) { @@ -313,14 +318,11 @@ void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs) return; ulp = edev->ulp_tbl; - rcu_read_lock(); ops = rcu_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_sriov_config) { - rcu_read_unlock(); + if (!ops || !ops->ulp_sriov_config) return; - } + bnxt_ulp_get(ulp); - rcu_read_unlock(); ops->ulp_sriov_config(ulp->handle, num_vfs); bnxt_ulp_put(ulp); } @@ -377,37 +379,9 @@ void bnxt_ulp_irq_restart(struct bnxt *bp, int err) } } -void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl) -{ - u16 event_id = le16_to_cpu(cmpl->event_id); - struct bnxt_en_dev *edev = bp->edev; - struct bnxt_ulp_ops *ops; - struct bnxt_ulp *ulp; - - if (!bnxt_ulp_registered(edev)) - return; - - ulp = edev->ulp_tbl; - - rcu_read_lock(); - - ops = rcu_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_async_notifier) - goto exit; - if (!ulp->async_events_bmap || event_id > ulp->max_async_event_id) - goto exit; - - /* Read max_async_event_id first before testing the bitmap. */ - smp_rmb(); - if (test_bit(event_id, ulp->async_events_bmap)) - ops->ulp_async_notifier(ulp->handle, cmpl); -exit: - rcu_read_unlock(); -} - -static int bnxt_register_async_events(struct bnxt_en_dev *edev, - unsigned long *events_bmap, - u16 max_id) +int bnxt_register_async_events(struct bnxt_en_dev *edev, + unsigned long *events_bmap, + u16 max_id) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); @@ -421,15 +395,7 @@ static int bnxt_register_async_events(struct bnxt_en_dev *edev, bnxt_hwrm_func_drv_rgtr(bp, events_bmap, max_id + 1, true); return 0; } - -static const struct bnxt_en_ops bnxt_en_ops_tbl = { - .bnxt_register_device = bnxt_register_dev, - .bnxt_unregister_device = bnxt_unregister_dev, - .bnxt_request_msix = bnxt_req_msix_vecs, - .bnxt_free_msix = bnxt_free_msix_vecs, - .bnxt_send_fw_msg = bnxt_send_msg, - .bnxt_register_fw_async_events = bnxt_register_async_events, -}; +EXPORT_SYMBOL(bnxt_register_async_events); void bnxt_aux_dev_free(struct bnxt *bp) { @@ -465,7 +431,6 @@ static void bnxt_aux_dev_release(struct device *dev) container_of(dev, struct bnxt_aux_dev, aux_dev.dev); struct bnxt *bp = netdev_priv(bnxt_adev->edev->net); - bnxt_adev->edev->en_ops = NULL; kfree(bnxt_adev->edev); bnxt_adev->edev = NULL; bp->edev = NULL; @@ -473,7 +438,6 @@ static void bnxt_aux_dev_release(struct device *dev) static void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp) { - edev->en_ops = &bnxt_en_ops_tbl; edev->net = bp->dev; edev->pdev = bp->pdev; edev->l2_db_size = bp->db_size; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index 5433f3e17746..3bda864db1be 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -26,8 +26,6 @@ struct bnxt_msix_entry { }; struct bnxt_ulp_ops { - /* async_notifier() cannot sleep (in BH context) */ - void (*ulp_async_notifier)(void *, struct hwrm_async_event_cmpl *); void (*ulp_stop)(void *); void (*ulp_start)(void *); void (*ulp_sriov_config)(void *, int); @@ -63,7 +61,6 @@ struct bnxt_en_dev { BNXT_EN_FLAG_ROCEV2_CAP) #define BNXT_EN_FLAG_MSIX_REQUESTED 0x4 #define BNXT_EN_FLAG_ULP_STOPPED 0x8 - const struct bnxt_en_ops *en_ops; struct bnxt_ulp *ulp_tbl; int l2_db_size; /* Doorbell BAR size in * bytes mapped by L2 @@ -75,19 +72,6 @@ struct bnxt_en_dev { */ }; -struct bnxt_en_ops { - int (*bnxt_register_device)(struct bnxt_en_dev *edev, - struct bnxt_ulp_ops *ulp_ops, void *handle); - int (*bnxt_unregister_device)(struct bnxt_en_dev *edev); - int (*bnxt_request_msix)(struct bnxt_en_dev *edev, - struct bnxt_msix_entry *ent, int num_msix); - void (*bnxt_free_msix)(struct bnxt_en_dev *edev); - int (*bnxt_send_fw_msg)(struct bnxt_en_dev *edev, - struct bnxt_fw_msg *fw_msg); - int (*bnxt_register_fw_async_events)(struct bnxt_en_dev *edev, - unsigned long *events_bmap, u16 max_id); -}; - static inline bool bnxt_ulp_registered(struct bnxt_en_dev *edev) { if (edev && edev->ulp_tbl) @@ -107,4 +91,13 @@ void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl); void bnxt_rdma_aux_device_uninit(struct bnxt *bp); void bnxt_rdma_aux_device_init(struct bnxt *bp); void bnxt_aux_dev_free(struct bnxt *bp); +int bnxt_register_dev(struct bnxt_en_dev *edev, struct bnxt_ulp_ops *ulp_ops, + void *handle); +void bnxt_unregister_dev(struct bnxt_en_dev *edev); +int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, struct bnxt_msix_entry *ent, + int num_msix); +void bnxt_free_msix_vecs(struct bnxt_en_dev *edev); +int bnxt_send_msg(struct bnxt_en_dev *edev, struct bnxt_fw_msg *fw_msg); +int bnxt_register_async_events(struct bnxt_en_dev *edev, + unsigned long *events_bmap, u16 max_id); #endif From patchwork Thu Jan 12 20:29:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 42801 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp4101327wrt; Thu, 12 Jan 2023 12:58:17 -0800 (PST) X-Google-Smtp-Source: AMrXdXu/Us02atgQHIinUtWiUNJ6ufOICk2jopizwSmHG1W4u7TJFROrGiiJUE2KZbnIX78x/qF7 X-Received: by 2002:a17:907:9a0c:b0:84d:4e4f:1f85 with SMTP id kr12-20020a1709079a0c00b0084d4e4f1f85mr13934311ejc.59.1673557096774; Thu, 12 Jan 2023 12:58:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673557096; cv=none; d=google.com; s=arc-20160816; b=UwxCD/YZ7/n4rpiYswUBsT7G1Iu+OcBFES2WFFT85S42l+9qN5cFYoEoFhBhRH+5I+ YjWa4ouNxRn8Erjg5RJCJdaR8Du+Q8N0VVkgLo3wv6XhSgUW0IkhWIVIrWwb9QbBe25L pG/kJf0ptJKoZjBXOPaB6Iolrz9CNx/Qhe507o8/mE0dqoqrjju+sN+VADoi1eCXZMtp u+LOjNNVMHGEq+k2JXVI0TKgR8Q/dMiTaOfyxCb3U9pEMwGspvDNVPnkIh25xQu4YChJ eB1iFPHTsaNm70MAC1mJopb3KhdjJSZAH+4BzR+xaqFha/GaUpbEb9v1AYRjpUUpLFuW ahLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=D9R/QuaYL8+hSZP74LhIx+APTULZ9Bd2ouelS0e5v44=; b=eCdN+5+g/ABJ1m4CQvRmgGgLQXsxTubQbG+34gzfnBw6r216AWBWXS8gaNLqLfb4mT 4V/rdy8ueXF8Kh3YV7IWFtLGRz/6eqwVoNLWVjCh8AkOTLlk5dM7E0SRCTCgw/Q2y8yZ GrIvUkwITI6/bPnxUPpUVZZE2mzZOek9zUxGsDNxooqwOr9zil0482g+wvlU9SkjsT2A hRQIrNyFJ6PQ+WUMkqZey/cEhUIBrsCvc5VEBdUmePhjS7kNNaYZBtbT9yZwvlAuSXuk JEy7OpU1A8E7nu5p7GKg1dkbpsiq3ZaBhzXEazQGhqS7N4VpUYMJoPsoJAC8ew1/XT0+ a8rA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b="a5w4/eSi"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id nd21-20020a170907629500b0084d0ebd704bsi19468243ejc.71.2023.01.12.12.57.51; Thu, 12 Jan 2023 12:58:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b="a5w4/eSi"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240963AbjALU4Q (ORCPT + 99 others); Thu, 12 Jan 2023 15:56:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53334 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232570AbjALUxV (ORCPT ); Thu, 12 Jan 2023 15:53:21 -0500 Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 597681D0DE for ; Thu, 12 Jan 2023 12:29:54 -0800 (PST) Received: by mail-pl1-x633.google.com with SMTP id s8so13047339plk.5 for ; Thu, 12 Jan 2023 12:29:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=D9R/QuaYL8+hSZP74LhIx+APTULZ9Bd2ouelS0e5v44=; b=a5w4/eSiT0JX6aMskwu3MzeQGQdD2E4ySq5+v+vbGrXgUQURBtkx8SGwanVrzRLDN9 X5eYnIfInRx9Bg9DLdZFNP2fbnDjKh478Euoe049YVK07XpcnkNyIXC2t8OOFz2ouUsq xGiblEYWpskLmTZ16g5pJkLjH6I5AiXazMy4s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=D9R/QuaYL8+hSZP74LhIx+APTULZ9Bd2ouelS0e5v44=; b=RoyCH0gpWVp9PjFivJVDQEi/WjlLj4dq5cKSR77YMqQLDU1HzX9fX4lw2qfTC2xr31 xpavVTLb5U/hRhGikWWM91d+TQ6ssYy0tDhCGuaWA717zSsBbsB5DPBpVI2C/RLKRPj/ N/ZN8JBQZjpxZizRSF1ZLXbS3h7TZeSGweknaPDS2ktTMGyebEk+Q6Uns7xAxhTXqSyD 5qt6+cB7HQsVh8FsGZtclggX/AOt+tpe3JKEeP+4g7f5Izg2JKWvBjOEn5FsKLtlv0eS qMg6b65a6v7PzwV2P7uuZaiFsJEK6PZfUq/rwH5T+fgeDasro3RS+Pv0f4QHaJAGCUg8 ZA8Q== X-Gm-Message-State: AFqh2kr9rus/yQcH1iG31/YlzyZLUwRlSLVtEhoz9y53UrG4swTfY9uk KrKbL+QpbFgnM7nXdxfRp64zJw== X-Received: by 2002:a17:902:bb86:b0:193:1f24:a042 with SMTP id m6-20020a170902bb8600b001931f24a042mr18479651pls.29.1673555393800; Thu, 12 Jan 2023 12:29:53 -0800 (PST) Received: from C02GC2QQMD6T.wifi.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id 129-20020a630887000000b004777c56747csm10283855pgi.11.2023.01.12.12.29.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 12:29:53 -0800 (PST) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, Leon Romanovsky Subject: [PATCH net-next v7 5/8] bnxt_en: Use auxiliary bus calls over proprietary calls Date: Thu, 12 Jan 2023 12:29:36 -0800 Message-Id: <20230112202939.19562-6-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230112202939.19562-1-ajit.khaparde@broadcom.com> References: <20230112202939.19562-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754851806158836051?= X-GMAIL-MSGID: =?utf-8?q?1754851806158836051?= Wherever possible use the function ops provided by auxiliary bus instead of using proprietary ops. Defined bnxt_re_suspend and bnxt_re_resume calls which can be invoked by the bnxt_en driver instead of the ULP stop/start calls. Signed-off-by: Ajit Khaparde Reviewed-by: Andy Gospodarek Reviewed-by: Selvin Xavier Reviewed-by: Leon Romanovsky --- drivers/infiniband/hw/bnxt_re/main.c | 102 +++++++++++------- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 40 ++++--- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 2 - 3 files changed, 87 insertions(+), 57 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index e1777c8e6d5e..ae04198c33b7 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -226,45 +226,6 @@ static void bnxt_re_set_resource_limits(struct bnxt_re_dev *rdev) bnxt_re_limit_vf_res(&rdev->qplib_ctx, num_vfs); } -/* for handling bnxt_en callbacks later */ -static void bnxt_re_stop(void *p) -{ - struct bnxt_re_dev *rdev = p; - struct bnxt *bp; - - if (!rdev) - return; - - /* L2 driver invokes this callback during device error/crash or device - * reset. Current RoCE driver doesn't recover the device in case of - * error. Handle the error by dispatching fatal events to all qps - * ie. by calling bnxt_re_dev_stop and release the MSIx vectors as - * L2 driver want to modify the MSIx table. - */ - bp = netdev_priv(rdev->netdev); - - ibdev_info(&rdev->ibdev, "Handle device stop call from L2 driver"); - /* Check the current device state from L2 structure and move the - * device to detached state if FW_FATAL_COND is set. - * This prevents more commands to HW during clean-up, - * in case the device is already in error. - */ - if (test_bit(BNXT_STATE_FW_FATAL_COND, &bp->state)) - set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); - - bnxt_re_dev_stop(rdev); - bnxt_re_stop_irq(rdev); - /* Move the device states to detached and avoid sending any more - * commands to HW - */ - set_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags); - set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); -} - -static void bnxt_re_start(void *p) -{ -} - static void bnxt_re_sriov_config(void *p, int num_vfs) { struct bnxt_re_dev *rdev = p; @@ -341,8 +302,6 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) } static struct bnxt_ulp_ops bnxt_re_ulp_ops = { - .ulp_stop = bnxt_re_stop, - .ulp_start = bnxt_re_start, .ulp_sriov_config = bnxt_re_sriov_config, .ulp_irq_stop = bnxt_re_stop_irq, .ulp_irq_restart = bnxt_re_start_irq @@ -1585,6 +1544,65 @@ static int bnxt_re_probe(struct auxiliary_device *adev, return rc; } +static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state) +{ + struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); + struct bnxt *bp; + + if (!rdev) + return 0; + + mutex_lock(&bnxt_re_mutex); + /* L2 driver may invoke this callback during device error/crash or device + * reset. Current RoCE driver doesn't recover the device in case of + * error. Handle the error by dispatching fatal events to all qps + * ie. by calling bnxt_re_dev_stop and release the MSIx vectors as + * L2 driver want to modify the MSIx table. + */ + bp = netdev_priv(rdev->netdev); + + ibdev_info(&rdev->ibdev, "Handle device suspend call"); + /* Check the current device state from L2 structure and move the + * device to detached state if FW_FATAL_COND is set. + * This prevents more commands to HW during clean-up, + * in case the device is already in error. + */ + if (test_bit(BNXT_STATE_FW_FATAL_COND, &bp->state)) + set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); + + bnxt_re_dev_stop(rdev); + bnxt_re_stop_irq(rdev); + /* Move the device states to detached and avoid sending any more + * commands to HW + */ + set_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags); + set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); + mutex_unlock(&bnxt_re_mutex); + + return 0; +} + +static int bnxt_re_resume(struct auxiliary_device *adev) +{ + struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); + + if (!rdev) + return 0; + + mutex_lock(&bnxt_re_mutex); + /* L2 driver may invoke this callback during device recovery, resume. + * reset. Current RoCE driver doesn't recover the device in case of + * error. Handle the error by dispatching fatal events to all qps + * ie. by calling bnxt_re_dev_stop and release the MSIx vectors as + * L2 driver want to modify the MSIx table. + */ + + ibdev_info(&rdev->ibdev, "Handle device resume call"); + mutex_unlock(&bnxt_re_mutex); + + return 0; +} + static const struct auxiliary_device_id bnxt_re_id_table[] = { { .name = BNXT_ADEV_NAME ".rdma", }, {}, @@ -1597,6 +1615,8 @@ static struct auxiliary_driver bnxt_re_driver = { .probe = bnxt_re_probe, .remove = bnxt_re_remove, .shutdown = bnxt_re_shutdown, + .suspend = bnxt_re_suspend, + .resume = bnxt_re_resume, .id_table = bnxt_re_id_table, }; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index e7539946759d..e08b51ee8886 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -272,26 +272,31 @@ static void bnxt_ulp_put(struct bnxt_ulp *ulp) void bnxt_ulp_stop(struct bnxt *bp) { + struct bnxt_aux_dev *bnxt_aux = bp->aux_dev; struct bnxt_en_dev *edev = bp->edev; - struct bnxt_ulp_ops *ops; - struct bnxt_ulp *ulp; if (!edev) return; edev->flags |= BNXT_EN_FLAG_ULP_STOPPED; - ulp = edev->ulp_tbl; - ops = rtnl_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_stop) - return; - ops->ulp_stop(ulp->handle); + if (bnxt_aux) { + struct auxiliary_device *adev; + + adev = &bnxt_aux->aux_dev; + if (adev->dev.driver) { + struct auxiliary_driver *adrv; + pm_message_t pm = {}; + + adrv = to_auxiliary_drv(adev->dev.driver); + adrv->suspend(adev, pm); + } + } } void bnxt_ulp_start(struct bnxt *bp, int err) { + struct bnxt_aux_dev *bnxt_aux = bp->aux_dev; struct bnxt_en_dev *edev = bp->edev; - struct bnxt_ulp_ops *ops; - struct bnxt_ulp *ulp; if (!edev) return; @@ -301,11 +306,18 @@ void bnxt_ulp_start(struct bnxt *bp, int err) if (err) return; - ulp = edev->ulp_tbl; - ops = rtnl_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_start) - return; - ops->ulp_start(ulp->handle); + if (bnxt_aux) { + struct auxiliary_device *adev; + + adev = &bnxt_aux->aux_dev; + if (adev->dev.driver) { + struct auxiliary_driver *adrv; + + adrv = to_auxiliary_drv(adev->dev.driver); + adrv->resume(adev); + } + } + } void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index 3bda864db1be..6525aad3b642 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -26,8 +26,6 @@ struct bnxt_msix_entry { }; struct bnxt_ulp_ops { - void (*ulp_stop)(void *); - void (*ulp_start)(void *); void (*ulp_sriov_config)(void *, int); void (*ulp_irq_stop)(void *); void (*ulp_irq_restart)(void *, struct bnxt_msix_entry *); From patchwork Thu Jan 12 20:29:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 42794 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp4100798wrt; Thu, 12 Jan 2023 12:56:47 -0800 (PST) X-Google-Smtp-Source: AMrXdXtMfKEHKU1KxljuSKRS9bXU0+Syeon56HQl027fOiP/Q5z83nw494U2uVT6Piz4+frzUDsv X-Received: by 2002:a17:907:c007:b0:7ad:f165:70c2 with SMTP id ss7-20020a170907c00700b007adf16570c2mr91152896ejc.27.1673557006838; Thu, 12 Jan 2023 12:56:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673557006; cv=none; d=google.com; s=arc-20160816; b=tuoqGlQ5UHuHb1ItKkBnn/yOni/ICw0aa3QKgxXC1H/io5RsxxAAHk2JrDEFPRi+Wo kqHTw4ERJKqCeiRm6lcKVr+eKBDDQAp3kicaOq0PPVM7nDZUcyP2Lz5QTyym8Dg8Ob5u LKgv40vRgKOa6BpXGEEvOBJtaXVLTv+UZGtatIFyD2KFd8EPQt8AYKlKT2qzOX2LLRQu R3AMWUevHCxAi3F9I3Ak/GDkwWRNj9CEEzCBBXncPW4udkHOUQdoUXKndrcck1s9caaw 4AVhZmMpUcKV8k57t4WTs7wnCo0fCOOVmaV4kF8H6kc8Zdxtnr1K/frFd8EXGDDXnSUN GT9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=LOOTJxxRLa04/VImKPuqW8IXlQm6nQA+fiWCb0TiN28=; b=dewLkJxtGNzi2FoL3+aZEiPbyb7v6rkhhmEJEw0Njp1Ht4atKvEoSZqnjWWQx0bFX8 dKDjT+iAnl4TeB43oJeQev3yUwYUkFA+wAZOpRFda3ipp0ATDDN9clOwzuGmfBM/g09m D39P/BuxUQ4kraTBmZiM6R5pp+7K+eYlTE9S+X2CsDAzV+XqD5jBIMcUboxWsW3dRzsy 0G0oQWv8A/ePOrz/QJV4kUOobMzhNdrEefsIQphkyg7+siXLY/MPZyHQNvbIrDK2xu1u S0bytqWR64x/s7/B4FvxCJd16GSZCEKe6brzeahUdY5NpKwmWIhdIq7Cjz3ecO6frGKX a4yQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b="JW/NtFU2"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i36-20020a0564020f2400b0046360343debsi17680768eda.153.2023.01.12.12.56.21; Thu, 12 Jan 2023 12:56:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b="JW/NtFU2"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240589AbjALUzB (ORCPT + 99 others); Thu, 12 Jan 2023 15:55:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56250 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239958AbjALUxW (ORCPT ); Thu, 12 Jan 2023 15:53:22 -0500 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6BD091D0FF for ; Thu, 12 Jan 2023 12:29:56 -0800 (PST) Received: by mail-pj1-x1032.google.com with SMTP id bj3so17097635pjb.0 for ; Thu, 12 Jan 2023 12:29:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=LOOTJxxRLa04/VImKPuqW8IXlQm6nQA+fiWCb0TiN28=; b=JW/NtFU2fqJPgbmQqHwlxlywZfFYHyVQBPknHs1F8+BFl5b6JQZplIcvrzceW/2EGe lOizFSUSQC5n/YfhctfeRSPuL2WXnOBN+x95SvnVUmAeHEyBBSMS/DlBupBr0czprpOx TpnRDvUk4aQeomtUEFbFfkXyiqeBX8btzwKv8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LOOTJxxRLa04/VImKPuqW8IXlQm6nQA+fiWCb0TiN28=; b=bxHF9zVZRBiwahAJXYQXO6Dis0u3lRKDrlb7ApQvJN81Albu4vGdU1cThBTmkElsnc 3xLQyRV3AErRu7lO5lx6oBGzKRjJhjQb4IiX1sm20vxuHjUQpqSJlALiJgdi51WhQCFE g5wtuIvZV/SqmcVM6/njBEWpmUkyD170ctK5u1mpzO/ldnEY3wUowr7HbQpcjgqrfzZt JnwUbPiuwjpGxkAzqC78g6k8YsOkwSzibiuPIWXZZci96mhLSaOPATqrAqlq9oSQQFlX qpCmZ1y8d+3rBo4/p2jOL3e1PNNPt6VlOaeCaRP6Z6MYdpHzXjQwm8jEbmn7zqaq52EA 4PTQ== X-Gm-Message-State: AFqh2kogJInqh41nTxmPPE2pz6+zaNmOSusDxk6MwzduPDqmhZbtTrC+ uBceA9oLzPkhzqRcMvBwny/+8Q== X-Received: by 2002:a05:6a20:4655:b0:aa:23f9:7314 with SMTP id eb21-20020a056a20465500b000aa23f97314mr79399806pzb.46.1673555395690; Thu, 12 Jan 2023 12:29:55 -0800 (PST) Received: from C02GC2QQMD6T.wifi.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id 129-20020a630887000000b004777c56747csm10283855pgi.11.2023.01.12.12.29.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 12:29:55 -0800 (PST) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, Hongguang Gao , Leon Romanovsky Subject: [PATCH net-next v7 6/8] bnxt_en: Remove struct bnxt access from RoCE driver Date: Thu, 12 Jan 2023 12:29:37 -0800 Message-Id: <20230112202939.19562-7-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230112202939.19562-1-ajit.khaparde@broadcom.com> References: <20230112202939.19562-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754851711962725358?= X-GMAIL-MSGID: =?utf-8?q?1754851711962725358?= From: Hongguang Gao Decouple RoCE driver from directly accessing L2's private bnxt structure. Move the fields needed by RoCE driver into bnxt_en_dev. They'll be passed to RoCE driver by bnxt_rdma_aux_device_add() function. Signed-off-by: Hongguang Gao Signed-off-by: Ajit Khaparde Reviewed-by: Andy Gospodarek Reviewed-by: Selvin Xavier Reviewed-by: Leon Romanovsky --- drivers/infiniband/hw/bnxt_re/main.c | 22 ++++++------------- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 9 ++++++++ drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 11 ++++++++++ 3 files changed, 27 insertions(+), 15 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index ae04198c33b7..3251078e9fe3 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -112,16 +112,14 @@ static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev, u8 wqe_mode) { struct bnxt_qplib_chip_ctx *chip_ctx; struct bnxt_en_dev *en_dev; - struct bnxt *bp; en_dev = rdev->en_dev; - bp = netdev_priv(en_dev->net); chip_ctx = kzalloc(sizeof(*chip_ctx), GFP_KERNEL); if (!chip_ctx) return -ENOMEM; - chip_ctx->chip_num = bp->chip_num; - chip_ctx->hw_stats_size = bp->hw_ring_stats_size; + chip_ctx->chip_num = en_dev->chip_num; + chip_ctx->hw_stats_size = en_dev->hw_ring_stats_size; rdev->chip_ctx = chip_ctx; /* rest members to follow eventually */ @@ -129,7 +127,7 @@ static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev, u8 wqe_mode) rdev->qplib_res.cctx = rdev->chip_ctx; rdev->rcfw.res = &rdev->qplib_res; rdev->qplib_res.dattr = &rdev->dev_attr; - rdev->qplib_res.is_vf = BNXT_VF(bp); + rdev->qplib_res.is_vf = BNXT_EN_VF(en_dev); bnxt_re_set_drv_mode(rdev, wqe_mode); if (bnxt_qplib_determine_atomics(en_dev->pdev)) @@ -142,10 +140,7 @@ static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev, u8 wqe_mode) static void bnxt_re_get_sriov_func_type(struct bnxt_re_dev *rdev) { - struct bnxt *bp; - - bp = netdev_priv(rdev->en_dev->net); - if (BNXT_VF(bp)) + if (BNXT_EN_VF(rdev->en_dev)) rdev->is_virtfn = 1; } @@ -957,7 +952,6 @@ static int bnxt_re_query_hwrm_pri2cos(struct bnxt_re_dev *rdev, u8 dir, u64 *cid_map) { struct hwrm_queue_pri2cos_qcfg_input req = {0}; - struct bnxt *bp = netdev_priv(rdev->netdev); struct hwrm_queue_pri2cos_qcfg_output resp; struct bnxt_en_dev *en_dev = rdev->en_dev; struct bnxt_fw_msg fw_msg; @@ -974,7 +968,7 @@ static int bnxt_re_query_hwrm_pri2cos(struct bnxt_re_dev *rdev, u8 dir, flags |= (dir & 0x01); flags |= HWRM_QUEUE_PRI2COS_QCFG_INPUT_FLAGS_IVLAN; req.flags = cpu_to_le32(flags); - req.port_id = bp->pf.port_id; + req.port_id = en_dev->pf_port_id; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); @@ -1547,7 +1541,6 @@ static int bnxt_re_probe(struct auxiliary_device *adev, static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state) { struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); - struct bnxt *bp; if (!rdev) return 0; @@ -1559,15 +1552,14 @@ static int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state) * ie. by calling bnxt_re_dev_stop and release the MSIx vectors as * L2 driver want to modify the MSIx table. */ - bp = netdev_priv(rdev->netdev); ibdev_info(&rdev->ibdev, "Handle device suspend call"); - /* Check the current device state from L2 structure and move the + /* Check the current device state from bnxt_en_dev and move the * device to detached state if FW_FATAL_COND is set. * This prevents more commands to HW during clean-up, * in case the device is already in error. */ - if (test_bit(BNXT_STATE_FW_FATAL_COND, &bp->state)) + if (test_bit(BNXT_STATE_FW_FATAL_COND, &rdev->en_dev->en_state)) set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); bnxt_re_dev_stop(rdev); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index e08b51ee8886..8bc78d5525ad 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -288,6 +288,7 @@ void bnxt_ulp_stop(struct bnxt *bp) pm_message_t pm = {}; adrv = to_auxiliary_drv(adev->dev.driver); + edev->en_state = bp->state; adrv->suspend(adev, pm); } } @@ -314,6 +315,7 @@ void bnxt_ulp_start(struct bnxt *bp, int err) struct auxiliary_driver *adrv; adrv = to_auxiliary_drv(adev->dev.driver); + edev->en_state = bp->state; adrv->resume(adev); } } @@ -459,6 +461,13 @@ static void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp) edev->flags |= BNXT_EN_FLAG_ROCEV1_CAP; if (bp->flags & BNXT_FLAG_ROCEV2_CAP) edev->flags |= BNXT_EN_FLAG_ROCEV2_CAP; + if (bp->flags & BNXT_FLAG_VF) + edev->flags |= BNXT_EN_FLAG_VF; + + edev->chip_num = bp->chip_num; + edev->hw_ring_stats_size = bp->hw_ring_stats_size; + edev->pf_port_id = bp->pf.port_id; + edev->en_state = bp->state; } static int bnxt_rdma_aux_device_add(struct bnxt *bp) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index 6525aad3b642..ce939df24574 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -59,6 +59,9 @@ struct bnxt_en_dev { BNXT_EN_FLAG_ROCEV2_CAP) #define BNXT_EN_FLAG_MSIX_REQUESTED 0x4 #define BNXT_EN_FLAG_ULP_STOPPED 0x8 + #define BNXT_EN_FLAG_VF 0x10 +#define BNXT_EN_VF(edev) ((edev)->flags & BNXT_EN_FLAG_VF) + struct bnxt_ulp *ulp_tbl; int l2_db_size; /* Doorbell BAR size in * bytes mapped by L2 @@ -68,6 +71,14 @@ struct bnxt_en_dev { * bytes mapped as non- * cacheable. */ + u16 chip_num; + u16 hw_ring_stats_size; + u16 pf_port_id; + unsigned long en_state; /* Could be checked in + * RoCE driver suspend + * mode only. Will be + * updated in resume. + */ }; static inline bool bnxt_ulp_registered(struct bnxt_en_dev *edev) From patchwork Thu Jan 12 20:29:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 42795 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp4100822wrt; Thu, 12 Jan 2023 12:56:51 -0800 (PST) X-Google-Smtp-Source: AMrXdXuegX+7E5wjcfkv4eorJfppapkKX+g0g42I6VtLcXYrQwAg3dnLUObBndJMUH3A96LFA1t1 X-Received: by 2002:a17:907:7f12:b0:7c1:9eb:845b with SMTP id qf18-20020a1709077f1200b007c109eb845bmr1061346ejc.16.1673557011182; Thu, 12 Jan 2023 12:56:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673557011; cv=none; d=google.com; s=arc-20160816; b=CG7cmJ+5/cBX5vBm8xbDZKXEIJN7w70RXX5Mpt1RtzJUy12pJDKCGcODDIMDE1h7um tbvPorIOsd6RC2vpvcNqY6GHGhPvj2dv72abwCfh+fWIi4N13K/LCDYgP8rktXHFrzLm 8yG3iBQpMeScfKMtEd+GmwEwd4xgs3n4iDM3u+Vc97SiClySV8OPlf34wcBN27sxSHSh 5yyAewVpd+PpMWVilDI4Qv+Qgk0k0y54Gd2w5TOApM+884960GnnvBhqyCDMR6Yfptnn NeSjKRAimANV148zqHFYh6e2hfiFyAhBJjcR/6hqKQe3kBlFJziFBt8etGDMrJ9syoAz BFXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=h8C9WYRMHCIjP1yf/vn+urDTFS1WheXyJHwMT65cGWg=; b=YU7+CFO7G/l5lRHCt0ZonUi5wa1/6KUeGqZa/LiOynwBdXTzCTxwd4nVoQOfwjDZpg HZCuPTSsrcy33FpODCkOOcWMDgXdpWIF8L//Ooh8xwL9Pai6wTc0+dgiQQhVqdQzgI4o 9lsmebDI+Y6XaF9TkJxfKKguHk2AUz1kqba/qPlHE6YWZVW9ui45cJ02Vbe8iY27c68w aVmcsYLmtJn18uB/6Qt+1ND5NLYQ6ZTTnZWGckoCpFU8efZXSOTNeLNCfinqeqRiffM5 GqzE9Z4x9NC7lJmVUo9KbPNOFQHFeeQ1X7Q9PV0OPZXMqeYscaBF8o1pkjpSBr0Thn+a SAXA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=FrRFg8l4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id dd20-20020a1709069b9400b0085b4d436b51si8016697ejc.996.2023.01.12.12.56.26; Thu, 12 Jan 2023 12:56:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=FrRFg8l4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240602AbjALUzJ (ORCPT + 99 others); Thu, 12 Jan 2023 15:55:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53354 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240337AbjALUxX (ORCPT ); Thu, 12 Jan 2023 15:53:23 -0500 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3ADFA1DDD5 for ; Thu, 12 Jan 2023 12:29:58 -0800 (PST) Received: by mail-pj1-x1032.google.com with SMTP id q64so20304717pjq.4 for ; Thu, 12 Jan 2023 12:29:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=h8C9WYRMHCIjP1yf/vn+urDTFS1WheXyJHwMT65cGWg=; b=FrRFg8l4TYYessIIIX43rpVQiTlYnBkY9h92du5LekhF3j4DswuhL4/1YWBFeUUXO6 GWvoMgOu2v4fV4TJZzB3j/gpqCzw7Xcyf2ZjDr7oNogSSHXWdUrpJpdqvRD5SYb55g8b z00L7yyD8ZBxL3HBJnVTHGED2YDueU8iJPlkg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=h8C9WYRMHCIjP1yf/vn+urDTFS1WheXyJHwMT65cGWg=; b=aeKc3iLgk5DHgXkjIw0+3VXNPZscB1+aJysJWm/PRxD1xhSnrP1+BgMf73TNsrj6xo 95YEn6xK7KFBRwnGppAUNn5NmHgXidnWa3prlcWEaVUmV5PlC6ebAr9LDW/TtMOTS95m nv4nCtqwnv9Q1tI9pH4xMqF5+KRGbYR7h/H0e/yDbgnxoP8K8Dkq/8eo05o17wxECG2m UaBPSzforA/zGVn7M8liIwmPrzZ5VGIFE6rvB7krE+YBNadYg6tVz2tYJAvT4UQqvqc9 O/IrgoLwPbhMdv3KhQVktBiyKThnnFN98fxD4q6efsMCUWHlRaaWjYPXHzUxDpgK7/Je /zJg== X-Gm-Message-State: AFqh2koNhII/6iKO/h17ub/K3tqSzHtTu8NAxcgoehSTuG8L9tc3SwUw UdGjjB9RdndNk6p0E/uze9IBVg== X-Received: by 2002:a05:6a20:ba9d:b0:b6:1413:fff0 with SMTP id fb29-20020a056a20ba9d00b000b61413fff0mr12890962pzb.50.1673555397731; Thu, 12 Jan 2023 12:29:57 -0800 (PST) Received: from C02GC2QQMD6T.wifi.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id 129-20020a630887000000b004777c56747csm10283855pgi.11.2023.01.12.12.29.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 12:29:57 -0800 (PST) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, Leon Romanovsky Subject: [PATCH net-next v7 7/8] RDMA/bnxt_re: Remove the sriov config callback Date: Thu, 12 Jan 2023 12:29:38 -0800 Message-Id: <20230112202939.19562-8-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230112202939.19562-1-ajit.khaparde@broadcom.com> References: <20230112202939.19562-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754851716724568775?= X-GMAIL-MSGID: =?utf-8?q?1754851716724568775?= Remove the SRIOV config callback which the bnxt_en was calling to reconfigure the chip resources for a PF device when VFs are created. The code is now modified to provision the VF resources based on the total VF count instead of the actual VF count. This allows the SRIOV config callback to be removed from the list of ulp_ops. Suggested-by: Leon Romanovsky Signed-off-by: Ajit Khaparde Reviewed-by: Leon Romanovsky --- drivers/infiniband/hw/bnxt_re/main.c | 11 ++++--- .../net/ethernet/broadcom/bnxt/bnxt_sriov.c | 7 +---- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 29 ------------------- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 1 - 4 files changed, 8 insertions(+), 40 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 3251078e9fe3..b287c3ba9352 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -221,13 +221,12 @@ static void bnxt_re_set_resource_limits(struct bnxt_re_dev *rdev) bnxt_re_limit_vf_res(&rdev->qplib_ctx, num_vfs); } -static void bnxt_re_sriov_config(void *p, int num_vfs) +static void bnxt_re_vf_res_config(struct bnxt_re_dev *rdev) { - struct bnxt_re_dev *rdev = p; if (test_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags)) return; - rdev->num_vfs = num_vfs; + rdev->num_vfs = pci_sriov_get_totalvfs(rdev->en_dev->pdev); if (!bnxt_qplib_is_chip_gen_p5(rdev->chip_ctx)) { bnxt_re_set_resource_limits(rdev); bnxt_qplib_set_func_resources(&rdev->qplib_res, &rdev->rcfw, @@ -297,7 +296,6 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) } static struct bnxt_ulp_ops bnxt_re_ulp_ops = { - .ulp_sriov_config = bnxt_re_sriov_config, .ulp_irq_stop = bnxt_re_stop_irq, .ulp_irq_restart = bnxt_re_start_irq }; @@ -1369,6 +1367,11 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 wqe_mode) INIT_DELAYED_WORK(&rdev->worker, bnxt_re_worker); set_bit(BNXT_RE_FLAG_QOS_WORK_REG, &rdev->flags); schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000)); + /* + * Use the total VF count since the actual VF count may not be + * available at this point. + */ + bnxt_re_vf_res_config(rdev); } return 0; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c index a4cba7cb2783..3ed3a2b3b3a9 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_sriov.c @@ -749,7 +749,6 @@ int bnxt_cfg_hw_sriov(struct bnxt *bp, int *num_vfs, bool reset) *num_vfs = rc; } - bnxt_ulp_sriov_cfg(bp, *num_vfs); return 0; } @@ -823,10 +822,8 @@ static int bnxt_sriov_enable(struct bnxt *bp, int *num_vfs) goto err_out2; rc = pci_enable_sriov(bp->pdev, *num_vfs); - if (rc) { - bnxt_ulp_sriov_cfg(bp, 0); + if (rc) goto err_out2; - } return 0; @@ -872,8 +869,6 @@ void bnxt_sriov_disable(struct bnxt *bp) rtnl_lock(); bnxt_restore_pf_fw_resources(bp); rtnl_unlock(); - - bnxt_ulp_sriov_cfg(bp, 0); } int bnxt_sriov_configure(struct pci_dev *pdev, int num_vfs) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index 8bc78d5525ad..2164b4419263 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -260,16 +260,6 @@ int bnxt_send_msg(struct bnxt_en_dev *edev, } EXPORT_SYMBOL(bnxt_send_msg); -static void bnxt_ulp_get(struct bnxt_ulp *ulp) -{ - atomic_inc(&ulp->ref_count); -} - -static void bnxt_ulp_put(struct bnxt_ulp *ulp) -{ - atomic_dec(&ulp->ref_count); -} - void bnxt_ulp_stop(struct bnxt *bp) { struct bnxt_aux_dev *bnxt_aux = bp->aux_dev; @@ -322,25 +312,6 @@ void bnxt_ulp_start(struct bnxt *bp, int err) } -void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs) -{ - struct bnxt_en_dev *edev = bp->edev; - struct bnxt_ulp_ops *ops; - struct bnxt_ulp *ulp; - - if (!edev) - return; - ulp = edev->ulp_tbl; - - ops = rcu_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_sriov_config) - return; - - bnxt_ulp_get(ulp); - ops->ulp_sriov_config(ulp->handle, num_vfs); - bnxt_ulp_put(ulp); -} - void bnxt_ulp_irq_stop(struct bnxt *bp) { struct bnxt_en_dev *edev = bp->edev; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index ce939df24574..0e15b35c36dd 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -26,7 +26,6 @@ struct bnxt_msix_entry { }; struct bnxt_ulp_ops { - void (*ulp_sriov_config)(void *, int); void (*ulp_irq_stop)(void *); void (*ulp_irq_restart)(void *, struct bnxt_msix_entry *); }; From patchwork Thu Jan 12 20:29:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 42800 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp4101249wrt; Thu, 12 Jan 2023 12:58:02 -0800 (PST) X-Google-Smtp-Source: AMrXdXsKZOCKyPbJn1s5Lo+g2yqyHCjSTKS0zte9nwF3jvc/QzEM5V07OqeY8HE100W5ZNl7896i X-Received: by 2002:a17:906:b00d:b0:7c1:8ba6:6eb3 with SMTP id v13-20020a170906b00d00b007c18ba66eb3mr70329695ejy.35.1673557082566; Thu, 12 Jan 2023 12:58:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673557082; cv=none; d=google.com; s=arc-20160816; b=e02sounFOVqejeQShZOLRgTJuXvOqTDohkoc7gVx6hnt0OCg98V0Kv2iBKBnm5srJu bQO54IwANf2ZBcXZcRmNyZf27D1bH9npFhhYXhlSy8n0xiy2aUbABS9BH+F/fGoSf0jT mghcS5IXG/uzBC9yk/4NfnKmww8O3mtoNLpwZntaecQr8hfnfJeuhObI1NrlzQ463/wx ynwNXAkMNBL+Pe/WUghn9UyPb14GhWZ9QAOjl03pb2P7MeAiHd5JZCtKE/O6p31WmsPg o01YVdWZSGHrvl0LQxVs6D0ah+7BUPSW/prYrldSu22mjRnKB4Kp0v40MiAewAf2ML4n imLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=Cz/Kh/xlRNvhdQEEbOTclQ8WEFq2J3R5xfLDzuFtjL4=; b=i7C5LARaUYA7GxgqApft07yj0uxPcKNFWZD8hUCZogM4PH74DjKaluwZ5cKJBL6oW/ +++OytKGVWtzZ3aDmQDoftcvxT7Bf+KSoR/6Z1OlBZkNPKigYYBiA/Uv8Mt5Ypmnjg9P eART6rqNwLK9gytlY5kpW2ILUH+GuRCshGMbl6NTzPulOO+lBAEvB98fr/USXKWsEdAE QXLfavZgSL9pjlSwd0RbKHfoM9qPgVlmIZxzbVtDFRRTTyD5by69z9rWwo/DtN/7BbAh ToHZliWmCTEdJDfq9N+JPNicTbbrOF6Ox0pqo89LiUIQiwUok4pr8XJyzdbYa/zbTrvj Pc4Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b="iGGX4Zf/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id xb7-20020a170907070700b007877f3132d9si19255254ejb.438.2023.01.12.12.57.37; Thu, 12 Jan 2023 12:58:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b="iGGX4Zf/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240956AbjALU4N (ORCPT + 99 others); Thu, 12 Jan 2023 15:56:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55042 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240399AbjALUxZ (ORCPT ); Thu, 12 Jan 2023 15:53:25 -0500 Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 555F51DF18 for ; Thu, 12 Jan 2023 12:30:00 -0800 (PST) Received: by mail-pj1-x1030.google.com with SMTP id o13so16806830pjg.2 for ; Thu, 12 Jan 2023 12:30:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=Cz/Kh/xlRNvhdQEEbOTclQ8WEFq2J3R5xfLDzuFtjL4=; b=iGGX4Zf/E44O8cVQEnSVhO69IzGr1Sy9+mbU5l+SnpBAdiYe9Kz+LVJ+UBfpn8C2gg N6m2zdDyfe07Cn4/5fHFAP/IE5AvAppOUhYISxP0Fgi3Q+kPtmJAZ/ww9Gc6JbNDjydt Q3/eFUt/8MpPbyZVnUhCwDZB4r4XB6FeZiIOo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Cz/Kh/xlRNvhdQEEbOTclQ8WEFq2J3R5xfLDzuFtjL4=; b=Ki3zHTZCWGvJmJ2q5YA8EFd4DxMpdbEL2kys9ZtLycG9ZBTMHj52mANUXaljZjU2Gm uhxd6gfNcz24otaCPAYVT1dErmW/NWtFga/cB7I+dYuNyrw1a2mv4W4HVceIMClbCk+k RcSL7zQykcco9NzcdMl6dRdopOmvh31/t16WpaQtYvw8LNaSpLUI9gcC9AntpH0JRyo4 7dTtPx7sx1ng8Y+qF0q5NhiZORLtOUTZe5j6lsxBjQ9SFESwEqfx62hImt/MnrOw68tl TT8H9c7MoLUS+iOLfKTZFruPb+V+qZCrMTytQHUQ6dQ4tUXolyKs9kbl0xzQ+NY3zRPO bxdw== X-Gm-Message-State: AFqh2kqj33dNaWsLCM2Uvy4gBughd7RqAmXHn+N+VOvB0l7oJbUE8CXa pAzdQY1x1mz+2j+1GoxTDK8AKA== X-Received: by 2002:a17:903:40d1:b0:194:60c3:48bc with SMTP id t17-20020a17090340d100b0019460c348bcmr3309036pld.27.1673555399467; Thu, 12 Jan 2023 12:29:59 -0800 (PST) Received: from C02GC2QQMD6T.wifi.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id 129-20020a630887000000b004777c56747csm10283855pgi.11.2023.01.12.12.29.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 12:29:58 -0800 (PST) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, Leon Romanovsky Subject: [PATCH net-next v7 8/8] bnxt_en: Remove runtime interrupt vector allocation Date: Thu, 12 Jan 2023 12:29:39 -0800 Message-Id: <20230112202939.19562-9-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230112202939.19562-1-ajit.khaparde@broadcom.com> References: <20230112202939.19562-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754851791128106569?= X-GMAIL-MSGID: =?utf-8?q?1754851791128106569?= Modified the bnxt_en code to create and pre-configure RDMA devices with the right MSI-X vector count for the ROCE driver to use. This is to align the ROCE driver to the auxiliary device model which will simply bind the driver without getting into PCI-related handling. All PCI-related logic will now be in the bnxt_en driver. Suggested-by: Leon Romanovsky Signed-off-by: Ajit Khaparde Reviewed-by: Leon Romanovsky --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 1 - drivers/infiniband/hw/bnxt_re/main.c | 48 ++--- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 166 ++++++------------ drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 6 +- 4 files changed, 64 insertions(+), 157 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index b0465c8d229a..5a2baf49ecaa 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -129,7 +129,6 @@ struct bnxt_re_dev { unsigned int version, major, minor; struct bnxt_qplib_chip_ctx *chip_ctx; struct bnxt_en_dev *en_dev; - struct bnxt_msix_entry msix_entries[BNXT_RE_MAX_MSIX]; int num_msix; int id; diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index b287c3ba9352..8d991fa184ae 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -262,7 +262,7 @@ static void bnxt_re_stop_irq(void *handle) static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) { struct bnxt_re_dev *rdev = (struct bnxt_re_dev *)handle; - struct bnxt_msix_entry *msix_ent = rdev->msix_entries; + struct bnxt_msix_entry *msix_ent = rdev->en_dev->msix_entries; struct bnxt_qplib_rcfw *rcfw = &rdev->rcfw; struct bnxt_qplib_nq *nq; int indx, rc; @@ -281,7 +281,7 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) * in device sctructure. */ for (indx = 0; indx < rdev->num_msix; indx++) - rdev->msix_entries[indx].vector = ent[indx].vector; + rdev->en_dev->msix_entries[indx].vector = ent[indx].vector; bnxt_qplib_rcfw_start_irq(rcfw, msix_ent[BNXT_RE_AEQ_IDX].vector, false); @@ -315,32 +315,6 @@ static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) return rc; } -static int bnxt_re_request_msix(struct bnxt_re_dev *rdev) -{ - int rc = 0, num_msix_want = BNXT_RE_MAX_MSIX, num_msix_got; - struct bnxt_en_dev *en_dev; - - en_dev = rdev->en_dev; - - num_msix_want = min_t(u32, BNXT_RE_MAX_MSIX, num_online_cpus()); - - num_msix_got = bnxt_req_msix_vecs(en_dev, - rdev->msix_entries, - num_msix_want); - if (num_msix_got < BNXT_RE_MIN_MSIX) { - rc = -EINVAL; - goto done; - } - if (num_msix_got != num_msix_want) { - ibdev_warn(&rdev->ibdev, - "Requested %d MSI-X vectors, got %d\n", - num_msix_want, num_msix_got); - } - rdev->num_msix = num_msix_got; -done: - return rc; -} - static void bnxt_re_init_hwrm_hdr(struct bnxt_re_dev *rdev, struct input *hdr, u16 opcd, u16 crid, u16 trid) { @@ -785,7 +759,7 @@ static u32 bnxt_re_get_nqdb_offset(struct bnxt_re_dev *rdev, u16 indx) return bnxt_qplib_is_chip_gen_p5(rdev->chip_ctx) ? (rdev->is_virtfn ? BNXT_RE_GEN_P5_VF_NQ_DB : BNXT_RE_GEN_P5_PF_NQ_DB) : - rdev->msix_entries[indx].db_offset; + rdev->en_dev->msix_entries[indx].db_offset; } static void bnxt_re_cleanup_res(struct bnxt_re_dev *rdev) @@ -810,7 +784,7 @@ static int bnxt_re_init_res(struct bnxt_re_dev *rdev) for (i = 1; i < rdev->num_msix ; i++) { db_offt = bnxt_re_get_nqdb_offset(rdev, i); rc = bnxt_qplib_enable_nq(rdev->en_dev->pdev, &rdev->nq[i - 1], - i - 1, rdev->msix_entries[i].vector, + i - 1, rdev->en_dev->msix_entries[i].vector, db_offt, &bnxt_re_cqn_handler, &bnxt_re_srqn_handler); if (rc) { @@ -897,7 +871,7 @@ static int bnxt_re_alloc_res(struct bnxt_re_dev *rdev) rattr.type = type; rattr.mode = RING_ALLOC_REQ_INT_MODE_MSIX; rattr.depth = BNXT_QPLIB_NQE_MAX_CNT - 1; - rattr.lrid = rdev->msix_entries[i + 1].ring_idx; + rattr.lrid = rdev->en_dev->msix_entries[i + 1].ring_idx; rc = bnxt_re_net_ring_alloc(rdev, &rattr, &nq->ring_id); if (rc) { ibdev_err(&rdev->ibdev, @@ -1217,7 +1191,7 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev) bnxt_qplib_free_rcfw_channel(&rdev->rcfw); } if (test_and_clear_bit(BNXT_RE_FLAG_GOT_MSIX, &rdev->flags)) - bnxt_free_msix_vecs(rdev->en_dev); + rdev->num_msix = 0; bnxt_re_destroy_chip_ctx(rdev); if (test_and_clear_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags)) @@ -1262,13 +1236,15 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 wqe_mode) /* Check whether VF or PF */ bnxt_re_get_sriov_func_type(rdev); - rc = bnxt_re_request_msix(rdev); - if (rc) { + if (!rdev->en_dev->ulp_tbl->msix_requested) { ibdev_err(&rdev->ibdev, "Failed to get MSI-X vectors: %#x\n", rc); rc = -EINVAL; goto fail; } + ibdev_dbg(&rdev->ibdev, "Got %d MSI-X vectors\n", + rdev->en_dev->ulp_tbl->msix_requested); + rdev->num_msix = rdev->en_dev->ulp_tbl->msix_requested; set_bit(BNXT_RE_FLAG_GOT_MSIX, &rdev->flags); bnxt_re_query_hwrm_intf_version(rdev); @@ -1292,14 +1268,14 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 wqe_mode) rattr.type = type; rattr.mode = RING_ALLOC_REQ_INT_MODE_MSIX; rattr.depth = BNXT_QPLIB_CREQE_MAX_CNT - 1; - rattr.lrid = rdev->msix_entries[BNXT_RE_AEQ_IDX].ring_idx; + rattr.lrid = rdev->en_dev->msix_entries[BNXT_RE_AEQ_IDX].ring_idx; rc = bnxt_re_net_ring_alloc(rdev, &rattr, &creq->ring_id); if (rc) { ibdev_err(&rdev->ibdev, "Failed to allocate CREQ: %#x\n", rc); goto free_rcfw; } db_offt = bnxt_re_get_nqdb_offset(rdev, BNXT_RE_AEQ_IDX); - vid = rdev->msix_entries[BNXT_RE_AEQ_IDX].vector; + vid = rdev->en_dev->msix_entries[BNXT_RE_AEQ_IDX].vector; rc = bnxt_qplib_enable_rcfw_channel(&rdev->rcfw, vid, db_offt, rdev->is_virtfn, &bnxt_re_aeq_handler); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index 2164b4419263..75ae0f499793 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -28,6 +28,30 @@ static DEFINE_IDA(bnxt_aux_dev_ids); +static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) +{ + struct bnxt_en_dev *edev = bp->edev; + int num_msix, idx, i; + + if (!edev->ulp_tbl->msix_requested) { + netdev_warn(bp->dev, "Requested MSI-X vectors insufficient\n"); + return; + } + num_msix = edev->ulp_tbl->msix_requested; + idx = edev->ulp_tbl->msix_base; + for (i = 0; i < num_msix; i++) { + ent[i].vector = bp->irq_tbl[idx + i].vector; + ent[i].ring_idx = idx + i; + if (bp->flags & BNXT_FLAG_CHIP_P5) { + ent[i].db_offset = DB_PF_OFFSET_P5; + if (BNXT_VF(bp)) + ent[i].db_offset = DB_VF_OFFSET_P5; + } else { + ent[i].db_offset = (idx + i) * 0x80; + } + } +} + int bnxt_register_dev(struct bnxt_en_dev *edev, struct bnxt_ulp_ops *ulp_ops, void *handle) @@ -42,17 +66,18 @@ int bnxt_register_dev(struct bnxt_en_dev *edev, bp->cp_nr_rings == max_stat_ctxs) return -ENOMEM; - ulp = kzalloc(sizeof(*ulp), GFP_KERNEL); + ulp = edev->ulp_tbl; if (!ulp) return -ENOMEM; - edev->ulp_tbl = ulp; ulp->handle = handle; rcu_assign_pointer(ulp->ulp_ops, ulp_ops); if (test_bit(BNXT_STATE_OPEN, &bp->state)) bnxt_hwrm_vnic_cfg(bp, 0); + bnxt_fill_msix_vecs(bp, bp->edev->msix_entries); + edev->flags |= BNXT_EN_FLAG_MSIX_REQUESTED; return 0; } EXPORT_SYMBOL(bnxt_register_dev); @@ -66,7 +91,7 @@ void bnxt_unregister_dev(struct bnxt_en_dev *edev) ulp = edev->ulp_tbl; if (ulp->msix_requested) - bnxt_free_msix_vecs(edev); + edev->flags &= ~BNXT_EN_FLAG_MSIX_REQUESTED; if (ulp->max_async_event_id) bnxt_hwrm_func_drv_rgtr(bp, NULL, 0, true); @@ -79,126 +104,17 @@ void bnxt_unregister_dev(struct bnxt_en_dev *edev) msleep(100); i++; } - kfree(ulp); - edev->ulp_tbl = NULL; return; } EXPORT_SYMBOL(bnxt_unregister_dev); -static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) -{ - struct bnxt_en_dev *edev = bp->edev; - int num_msix, idx, i; - - num_msix = edev->ulp_tbl->msix_requested; - idx = edev->ulp_tbl->msix_base; - for (i = 0; i < num_msix; i++) { - ent[i].vector = bp->irq_tbl[idx + i].vector; - ent[i].ring_idx = idx + i; - if (bp->flags & BNXT_FLAG_CHIP_P5) { - ent[i].db_offset = DB_PF_OFFSET_P5; - if (BNXT_VF(bp)) - ent[i].db_offset = DB_VF_OFFSET_P5; - } else { - ent[i].db_offset = (idx + i) * 0x80; - } - } -} - -int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, - struct bnxt_msix_entry *ent, - int num_msix) -{ - struct net_device *dev = edev->net; - struct bnxt *bp = netdev_priv(dev); - struct bnxt_hw_resc *hw_resc; - int max_idx, max_cp_rings; - int avail_msix, idx; - int total_vecs; - int rc = 0; - - if (!(bp->flags & BNXT_FLAG_USING_MSIX)) - return -ENODEV; - - if (edev->ulp_tbl->msix_requested) - return -EAGAIN; - - max_cp_rings = bnxt_get_max_func_cp_rings(bp); - avail_msix = bnxt_get_avail_msix(bp, num_msix); - if (!avail_msix) - return -ENOMEM; - if (avail_msix > num_msix) - avail_msix = num_msix; - - if (BNXT_NEW_RM(bp)) { - idx = bp->cp_nr_rings; - } else { - max_idx = min_t(int, bp->total_irqs, max_cp_rings); - idx = max_idx - avail_msix; - } - - edev->ulp_tbl->msix_base = idx; - edev->ulp_tbl->msix_requested = avail_msix; - hw_resc = &bp->hw_resc; - total_vecs = idx + avail_msix; - rtnl_lock(); - if (bp->total_irqs < total_vecs || - (BNXT_NEW_RM(bp) && hw_resc->resv_irqs < total_vecs)) { - if (netif_running(dev)) { - bnxt_close_nic(bp, true, false); - rc = bnxt_open_nic(bp, true, false); - } else { - rc = bnxt_reserve_rings(bp, true); - } - } - rtnl_unlock(); - if (rc) { - edev->ulp_tbl->msix_requested = 0; - return -EAGAIN; - } - - if (BNXT_NEW_RM(bp)) { - int resv_msix; - - resv_msix = hw_resc->resv_irqs - bp->cp_nr_rings; - avail_msix = min_t(int, resv_msix, avail_msix); - edev->ulp_tbl->msix_requested = avail_msix; - } - bnxt_fill_msix_vecs(bp, ent); - edev->flags |= BNXT_EN_FLAG_MSIX_REQUESTED; - return avail_msix; -} -EXPORT_SYMBOL(bnxt_req_msix_vecs); - -void bnxt_free_msix_vecs(struct bnxt_en_dev *edev) -{ - struct net_device *dev = edev->net; - struct bnxt *bp = netdev_priv(dev); - - if (!(edev->flags & BNXT_EN_FLAG_MSIX_REQUESTED)) - return; - - edev->ulp_tbl->msix_requested = 0; - edev->flags &= ~BNXT_EN_FLAG_MSIX_REQUESTED; - rtnl_lock(); - if (netif_running(dev) && !(edev->flags & BNXT_EN_FLAG_ULP_STOPPED)) { - bnxt_close_nic(bp, true, false); - bnxt_open_nic(bp, true, false); - } - rtnl_unlock(); - - return; -} -EXPORT_SYMBOL(bnxt_free_msix_vecs); - int bnxt_get_ulp_msix_num(struct bnxt *bp) { - if (bnxt_ulp_registered(bp->edev)) { - struct bnxt_en_dev *edev = bp->edev; + u32 roce_msix = BNXT_VF(bp) ? + BNXT_MAX_VF_ROCE_MSIX : BNXT_MAX_ROCE_MSIX; - return edev->ulp_tbl->msix_requested; - } - return 0; + return ((bp->flags & BNXT_FLAG_ROCE_CAP) && bp->aux_dev ? + min_t(u32, roce_msix, num_online_cpus()) : 0); } int bnxt_get_ulp_msix_base(struct bnxt *bp) @@ -416,6 +332,8 @@ static void bnxt_aux_dev_release(struct device *dev) container_of(dev, struct bnxt_aux_dev, aux_dev.dev); struct bnxt *bp = netdev_priv(bnxt_adev->edev->net); + kfree(bnxt_adev->edev->ulp_tbl); + bnxt_adev->edev->ulp_tbl = NULL; kfree(bnxt_adev->edev); bnxt_adev->edev = NULL; bp->edev = NULL; @@ -439,6 +357,8 @@ static void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp) edev->hw_ring_stats_size = bp->hw_ring_stats_size; edev->pf_port_id = bp->pf.port_id; edev->en_state = bp->state; + + edev->ulp_tbl->msix_requested = bnxt_get_ulp_msix_num(bp); } static int bnxt_rdma_aux_device_add(struct bnxt *bp) @@ -446,12 +366,21 @@ static int bnxt_rdma_aux_device_add(struct bnxt *bp) struct bnxt_aux_dev *bnxt_adev = bp->aux_dev; struct bnxt_en_dev *edev = bnxt_adev->edev; struct auxiliary_device *aux_dev; + struct bnxt_ulp *ulp; int ret; edev = kzalloc(sizeof(*edev), GFP_KERNEL); if (!edev) return -ENOMEM; + ulp = kzalloc(sizeof(*ulp), GFP_KERNEL); + if (!ulp) { + ret = -ENOMEM; + goto free_edev; + } + + edev->ulp_tbl = ulp; + aux_dev = &bnxt_adev->aux_dev; aux_dev->id = bnxt_adev->id; aux_dev->name = "rdma"; @@ -464,7 +393,7 @@ static int bnxt_rdma_aux_device_add(struct bnxt *bp) ret = auxiliary_device_init(aux_dev); if (ret) - goto free_edev; + goto free_ulp; ret = auxiliary_device_add(aux_dev); if (ret) @@ -473,6 +402,9 @@ static int bnxt_rdma_aux_device_add(struct bnxt *bp) return 0; aux_dev_uninit: auxiliary_device_uninit(aux_dev); +free_ulp: + kfree(ulp); + bp->edev->ulp_tbl = NULL; free_edev: kfree(edev); bp->edev = NULL; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index 0e15b35c36dd..01a9dc336312 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -15,6 +15,8 @@ #define BNXT_MIN_ROCE_CP_RINGS 2 #define BNXT_MIN_ROCE_STAT_CTXS 1 +#define BNXT_MAX_ROCE_MSIX 9 +#define BNXT_MAX_VF_ROCE_MSIX 2 struct hwrm_async_event_cmpl; struct bnxt; @@ -51,6 +53,7 @@ struct bnxt_ulp { struct bnxt_en_dev { struct net_device *net; struct pci_dev *pdev; + struct bnxt_msix_entry msix_entries[BNXT_MAX_ROCE_MSIX]; u32 flags; #define BNXT_EN_FLAG_ROCEV1_CAP 0x1 #define BNXT_EN_FLAG_ROCEV2_CAP 0x2 @@ -102,9 +105,6 @@ void bnxt_aux_dev_free(struct bnxt *bp); int bnxt_register_dev(struct bnxt_en_dev *edev, struct bnxt_ulp_ops *ulp_ops, void *handle); void bnxt_unregister_dev(struct bnxt_en_dev *edev); -int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, struct bnxt_msix_entry *ent, - int num_msix); -void bnxt_free_msix_vecs(struct bnxt_en_dev *edev); int bnxt_send_msg(struct bnxt_en_dev *edev, struct bnxt_fw_msg *fw_msg); int bnxt_register_async_events(struct bnxt_en_dev *edev, unsigned long *events_bmap, u16 max_id);