From patchwork Tue Oct 25 17:31:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 10901 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1131451wru; Tue, 25 Oct 2022 10:36:32 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7d2NGFAsc7p2H5FMAKC9XJ98J1Oa3z6mV1z3gSucjpzEtEjOvYxzX2B/Cd1Hy8nTRKs3MD X-Received: by 2002:a05:6402:1454:b0:461:9b5c:2fbc with SMTP id d20-20020a056402145400b004619b5c2fbcmr14391190edx.276.1666719392217; Tue, 25 Oct 2022 10:36:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666719392; cv=none; d=google.com; s=arc-20160816; b=hvav7QB9q7TbBr/auXQJoDeeavWUy1YS9R21jHCdvB2GEALOefBa73eRBfwLsw4CJq VhKVOdWEw0xe0ddjG7PqcM5kh5J9tJyilGbFip1dfP0Kvw2palSbb1YuzImQ3muERDEK 0fbqvSp8fR2EvMolvoGGNn1eI8rLDscnLx+78RWmUz0Q7vLMTBHntkqIEReLu8XOTdFM tQeQfwcG6rRNlL0z3g/ZWk1WJfKlL82Ejas23viKb3f/QpnATi/Mtz3yUoX4ihNZyGGh rpN9Aq3ZpPROEdnCd2d4YxXtWSezrTw/ZLUypHp7MIgcBLS0l3/Ya8DUihGopQ/69bQ3 6snQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=JMaR0khpSdJbJKgixTso4QEgOG3pqxy7a97zJJUw22s=; b=SgpbfyE/W+e+/ZitMBK+3uuQhe918aj24sKKntY5RrNkEuRpiYKmrKDIG1CZspnHer uyFIhiTBvf804dGwIp/3kDYQfqmr/7UBYMQPaTFGaTdFK4hK/tB4i6LGUj3tANjAXZMY g9YzMOucmyoIm9BlnacsZ4sHc6xcjuiwT6fg00lC79RjsbiAwoqChv+ASnjbG8Ki93Zh yjNoofjTpiP2SxR/C0bElAa5ckbU2XA2yPvODs3a46GAoc+Sq1YPGUm8VaQdZP7qa3N/ DHpmDnRyBvhuiU8cuouHx16rDs3dWiGFmxpP2I3ED6C0pMnAwZqVFalk+8WU+iJJhn64 IKzA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=C+vcH71y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b20-20020a056402279400b00460e691561esi3651113ede.630.2022.10.25.10.36.05; Tue, 25 Oct 2022 10:36:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=C+vcH71y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231528AbiJYRbf (ORCPT + 99 others); Tue, 25 Oct 2022 13:31:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232537AbiJYRbd (ORCPT ); Tue, 25 Oct 2022 13:31:33 -0400 Received: from mail-qk1-x72c.google.com (mail-qk1-x72c.google.com [IPv6:2607:f8b0:4864:20::72c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DC9CA170B43 for ; Tue, 25 Oct 2022 10:31:23 -0700 (PDT) Received: by mail-qk1-x72c.google.com with SMTP id m6so8591225qkm.4 for ; Tue, 25 Oct 2022 10:31:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=JMaR0khpSdJbJKgixTso4QEgOG3pqxy7a97zJJUw22s=; b=C+vcH71yx/RCiYbOdCudYO4+WN3b20/PVjFL75HOLZ2JgSJQHC3vm4IM6Xkt1lwaj2 2li8pDaaWOoQ5ZXN8SvtbaQFKx58bmIHpmFw42SePVkhbLKaT7xrJXK76wqV110d3xvK u2Gb8lhXON2pOPt27myXa/HVOTzexrTzR3GEE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JMaR0khpSdJbJKgixTso4QEgOG3pqxy7a97zJJUw22s=; b=DkQtgnU2ENSmCdavWjBDRn+zsAnlPTTJ88NPfRrQydIn2PDc0S8r9syv3BdVj/Kteg WrMr3GVnMEKa6ul46oiPMdXfTtYJ4WlsXvlunUJYdJG6SrH0kq8xy9RvzY4nO253kMoo ks9syKCFqtBEPSfOu/kteQV6nxoufDp2ZeshgKNFxuApewL7hTKwXT8GKtRzVVNYYzm7 ijiDl5duPYYlewX1v1S/CEamOXUG9TgJKYQHMdhIo+iHEbVdB5NWIOyAtwkdAqqh1KYV O8uU9j0frVhJkz6TjhYRNvoUf3+ngQWHT//m52i74pO7cl4xpnHXvuYNNPLxqaW/2dzv zgrQ== X-Gm-Message-State: ACrzQf3+NJEo4ysKyjm09Ry3KwYDBDicX6xgR9BB6hfjuKge/BwYMx0u WXcFNZo5gtvkWPNVZL3UClYLoQ== X-Received: by 2002:a05:620a:2984:b0:6ee:cd87:26ae with SMTP id r4-20020a05620a298400b006eecd8726aemr27712042qkp.753.1666719082844; Tue, 25 Oct 2022 10:31:22 -0700 (PDT) Received: from C02GC2QQMD6T.wifi.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id u6-20020a37ab06000000b006eed094dcdasm2329034qke.70.2022.10.25.10.31.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Oct 2022 10:31:21 -0700 (PDT) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com Subject: [PATCH v2 1/6] bnxt_en: Add auxiliary driver support Date: Tue, 25 Oct 2022 10:31:05 -0700 Message-Id: <20221025173110.33192-2-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.0 (Apple Git-136) In-Reply-To: <20221025173110.33192-1-ajit.khaparde@broadcom.com> References: <20220724231458.93830-1-ajit.khaparde@broadcom.com> <20221025173110.33192-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SORTED_RECIPS,SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747681953191885811?= X-GMAIL-MSGID: =?utf-8?q?1747681953191885811?= Add auxiliary driver support. An auxiliary device will be created if the hardware indicates support for RDMA. The bnxt_ulp_probe() function has been removed and a new bnxt_rdma_aux_device_add() function has been added. The bnxt_free_msix_vecs() and bnxt_req_msix_vecs() will now hold the RTNL lock when they call the bnxt_close_nic()and bnxt_open_nic() since the device close and open need to be protected under RTNL lock. The operations between the bnxt_en and bnxt_re will be protected using the en_ops_lock. This will be used by the bnxt_re driver in a follow-on patch to create ROCE interfaces. Signed-off-by: Ajit Khaparde --- drivers/infiniband/hw/bnxt_re/main.c | 2 +- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 72 +++++++++- drivers/net/ethernet/broadcom/bnxt/bnxt.h | 8 ++ drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 123 +++++++++++------- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 5 +- 5 files changed, 159 insertions(+), 51 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 8c0c80a8d338..f868b285d2a1 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -614,7 +614,7 @@ static struct bnxt_en_dev *bnxt_re_dev_probe(struct net_device *netdev) struct bnxt_en_dev *en_dev; struct pci_dev *pdev; - en_dev = bnxt_ulp_probe(netdev); + en_dev = ((struct bnxt*)netdev_priv(netdev))->edev; if (IS_ERR(en_dev)) return en_dev; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 04cf7684f1b0..0c8786ea2c1c 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -79,6 +79,8 @@ MODULE_LICENSE("GPL"); MODULE_DESCRIPTION("Broadcom BCM573xx network driver"); +static DEFINE_IDA(bnxt_aux_dev_ids); + #define BNXT_RX_OFFSET (NET_SKB_PAD + NET_IP_ALIGN) #define BNXT_RX_DMA_OFFSET NET_SKB_PAD #define BNXT_RX_COPY_THRESH 256 @@ -211,6 +213,68 @@ static const struct pci_device_id bnxt_pci_tbl[] = { MODULE_DEVICE_TABLE(pci, bnxt_pci_tbl); +static void bnxt_aux_dev_free(struct bnxt *bp) +{ + kfree(bp->aux_dev); + bp->aux_dev = NULL; +} + +static struct bnxt_aux_dev *bnxt_aux_dev_alloc(struct bnxt *bp) +{ + struct bnxt_aux_dev *bnxt_adev; + + bnxt_adev = kzalloc(sizeof(*bnxt_adev), GFP_KERNEL); + if (!bnxt_adev) + return ERR_PTR(-ENOMEM); + + return bnxt_adev; +} + +static void bnxt_rdma_aux_device_uninit(struct bnxt_aux_dev *bnxt_adev) +{ + struct auxiliary_device *adev; + + if (IS_ERR_OR_NULL(bnxt_adev)) + return; + + adev = &bnxt_adev->aux_dev; + auxiliary_device_delete(adev); + auxiliary_device_uninit(adev); + if (bnxt_adev->id >= 0) + ida_free(&bnxt_aux_dev_ids, bnxt_adev->id); +} + +static void bnxt_rdma_aux_device_init(struct bnxt *bp) +{ + struct net_device *dev = bp->dev; + int rc; + + if (bp->flags & BNXT_FLAG_ROCE_CAP) { + bp->aux_dev = bnxt_aux_dev_alloc(bp); + if (IS_ERR_OR_NULL(bp->aux_dev)) { + netdev_warn(dev, "Failed to init auxiliary device for ROCE\n"); + goto skip_aux_init; + } + + bp->aux_dev->id = ida_alloc(&bnxt_aux_dev_ids, GFP_KERNEL); + if (bp->aux_dev->id < 0) { + netdev_warn(dev, "ida alloc failed for ROCE auxiliary device\n"); + bnxt_aux_dev_free(bp); + goto skip_aux_init; + } + + /* If aux bus init fails, continue with netdev init. */ + rc = bnxt_rdma_aux_device_add(bp); + if (rc) { + netdev_warn(dev, "Failed to add auxiliary device for ROCE\n"); + ida_free(&bnxt_aux_dev_ids, bp->aux_dev->id); + bnxt_aux_dev_free(bp); + } + } +skip_aux_init: + return; +} + static const u16 bnxt_vf_req_snif[] = { HWRM_FUNC_CFG, HWRM_FUNC_VF_CFG, @@ -13119,6 +13183,9 @@ static void bnxt_remove_one(struct pci_dev *pdev) struct net_device *dev = pci_get_drvdata(pdev); struct bnxt *bp = netdev_priv(dev); + bnxt_rdma_aux_device_uninit(bp->aux_dev); + bnxt_aux_dev_free(bp); + if (BNXT_PF(bp)) bnxt_sriov_disable(bp); @@ -13716,11 +13783,13 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent) devlink_port_type_eth_set(&bp->dl_port, bp->dev); bnxt_dl_fw_reporters_create(bp); + bnxt_rdma_aux_device_init(bp); + bnxt_print_device_info(bp); pci_save_state(pdev); - return 0; + return 0; init_err_cleanup: bnxt_dl_unregister(bp); init_err_dl: @@ -13764,7 +13833,6 @@ static void bnxt_shutdown(struct pci_dev *pdev) if (netif_running(dev)) dev_close(dev); - bnxt_ulp_shutdown(bp); bnxt_clear_int_mode(bp); pci_disable_device(pdev); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethernet/broadcom/bnxt/bnxt.h index 91a1ba0a914d..284e39d5ae49 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -1622,6 +1623,12 @@ struct bnxt_fw_health { #define BNXT_FW_RETRY 5 #define BNXT_FW_IF_RETRY 10 +struct bnxt_aux_dev { + struct auxiliary_device aux_dev; + struct bnxt_en_dev *edev; + int id; +}; + enum board_idx { BCM57301, BCM57302, @@ -1843,6 +1850,7 @@ struct bnxt { #define BNXT_CHIP_P4_PLUS(bp) \ (BNXT_CHIP_P4(bp) || BNXT_CHIP_P5(bp)) + struct bnxt_aux_dev *aux_dev; struct bnxt_en_dev *edev; struct bnxt_napi **bnapi; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index 2e54bf4fc7a7..44a0922663cf 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -31,26 +31,29 @@ static int bnxt_register_dev(struct bnxt_en_dev *edev, unsigned int ulp_id, struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); struct bnxt_ulp *ulp; + int rc = 0; - ASSERT_RTNL(); if (ulp_id >= BNXT_MAX_ULP) return -EINVAL; ulp = &edev->ulp_tbl[ulp_id]; if (rcu_access_pointer(ulp->ulp_ops)) { netdev_err(bp->dev, "ulp id %d already registered\n", ulp_id); - return -EBUSY; + rc = -EBUSY; + goto exit; } if (ulp_id == BNXT_ROCE_ULP) { unsigned int max_stat_ctxs; max_stat_ctxs = bnxt_get_max_func_stat_ctxs(bp); if (max_stat_ctxs <= BNXT_MIN_ROCE_STAT_CTXS || - bp->cp_nr_rings == max_stat_ctxs) - return -ENOMEM; + bp->cp_nr_rings == max_stat_ctxs) { + rc = -ENOMEM; + goto exit; + } } - atomic_set(&ulp->ref_count, 0); + atomic_set(&ulp->ref_count, 1); ulp->handle = handle; rcu_assign_pointer(ulp->ulp_ops, ulp_ops); @@ -59,7 +62,8 @@ static int bnxt_register_dev(struct bnxt_en_dev *edev, unsigned int ulp_id, bnxt_hwrm_vnic_cfg(bp, 0); } - return 0; +exit: + return rc; } static int bnxt_unregister_dev(struct bnxt_en_dev *edev, unsigned int ulp_id) @@ -69,10 +73,11 @@ static int bnxt_unregister_dev(struct bnxt_en_dev *edev, unsigned int ulp_id) struct bnxt_ulp *ulp; int i = 0; - ASSERT_RTNL(); if (ulp_id >= BNXT_MAX_ULP) return -EINVAL; + edev->flags |= BNXT_EN_FLAG_ULP_STOPPED; + ulp = &edev->ulp_tbl[ulp_id]; if (!rcu_access_pointer(ulp->ulp_ops)) { netdev_err(bp->dev, "ulp id %d not registered\n", ulp_id); @@ -126,7 +131,6 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, int total_vecs; int rc = 0; - ASSERT_RTNL(); if (ulp_id != BNXT_ROCE_ULP) return -EINVAL; @@ -149,6 +153,7 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, max_idx = min_t(int, bp->total_irqs, max_cp_rings); idx = max_idx - avail_msix; } + edev->ulp_tbl[ulp_id].msix_base = idx; edev->ulp_tbl[ulp_id].msix_requested = avail_msix; hw_resc = &bp->hw_resc; @@ -156,8 +161,10 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, if (bp->total_irqs < total_vecs || (BNXT_NEW_RM(bp) && hw_resc->resv_irqs < total_vecs)) { if (netif_running(dev)) { + rtnl_lock(); bnxt_close_nic(bp, true, false); rc = bnxt_open_nic(bp, true, false); + rtnl_unlock(); } else { rc = bnxt_reserve_rings(bp, true); } @@ -184,7 +191,6 @@ static int bnxt_free_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id) struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); - ASSERT_RTNL(); if (ulp_id != BNXT_ROCE_ULP) return -EINVAL; @@ -194,9 +200,12 @@ static int bnxt_free_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id) edev->ulp_tbl[ulp_id].msix_requested = 0; edev->flags &= ~BNXT_EN_FLAG_MSIX_REQUESTED; if (netif_running(dev) && !(edev->flags & BNXT_EN_FLAG_ULP_STOPPED)) { + rtnl_lock(); bnxt_close_nic(bp, true, false); bnxt_open_nic(bp, true, false); + rtnl_unlock(); } + return 0; } @@ -347,25 +356,6 @@ void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs) } } -void bnxt_ulp_shutdown(struct bnxt *bp) -{ - struct bnxt_en_dev *edev = bp->edev; - struct bnxt_ulp_ops *ops; - int i; - - if (!edev) - return; - - for (i = 0; i < BNXT_MAX_ULP; i++) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[i]; - - ops = rtnl_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_shutdown) - continue; - ops->ulp_shutdown(ulp->handle); - } -} - void bnxt_ulp_irq_stop(struct bnxt *bp) { struct bnxt_en_dev *edev = bp->edev; @@ -475,28 +465,69 @@ static const struct bnxt_en_ops bnxt_en_ops_tbl = { .bnxt_register_fw_async_events = bnxt_register_async_events, }; -struct bnxt_en_dev *bnxt_ulp_probe(struct net_device *dev) +void bnxt_aux_dev_release(struct device *dev) { - struct bnxt *bp = netdev_priv(dev); - struct bnxt_en_dev *edev; + struct bnxt_aux_dev *bnxt_adev = + container_of(dev, struct bnxt_aux_dev, aux_dev.dev); + struct bnxt *bp = netdev_priv(bnxt_adev->edev->net); + + bnxt_adev->edev->en_ops = NULL; + kfree(bnxt_adev->edev); + bnxt_adev->edev = NULL; + bp->edev = NULL; +} + +static inline void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp) +{ + edev->en_ops = &bnxt_en_ops_tbl; + edev->net = bp->dev; + edev->pdev = bp->pdev; + edev->l2_db_size = bp->db_size; + edev->l2_db_size_nc = bp->db_size; - edev = bp->edev; - if (!edev) { - edev = kzalloc(sizeof(*edev), GFP_KERNEL); - if (!edev) - return ERR_PTR(-ENOMEM); - edev->en_ops = &bnxt_en_ops_tbl; - edev->net = dev; - edev->pdev = bp->pdev; - edev->l2_db_size = bp->db_size; - edev->l2_db_size_nc = bp->db_size; - bp->edev = edev; - } - edev->flags &= ~BNXT_EN_FLAG_ROCE_CAP; if (bp->flags & BNXT_FLAG_ROCEV1_CAP) edev->flags |= BNXT_EN_FLAG_ROCEV1_CAP; if (bp->flags & BNXT_FLAG_ROCEV2_CAP) edev->flags |= BNXT_EN_FLAG_ROCEV2_CAP; - return bp->edev; } -EXPORT_SYMBOL(bnxt_ulp_probe); + +int bnxt_rdma_aux_device_add(struct bnxt *bp) +{ + struct bnxt_aux_dev *bnxt_adev = bp->aux_dev; + struct bnxt_en_dev *edev = bnxt_adev->edev; + struct auxiliary_device *aux_dev; + int ret; + + aux_dev = &bnxt_adev->aux_dev; + aux_dev->id = bnxt_adev->id; + aux_dev->name = "rdma"; + aux_dev->dev.parent = &bp->pdev->dev; + aux_dev->dev.release = bnxt_aux_dev_release; + + if (!edev) { + edev = kzalloc(sizeof(*edev), GFP_KERNEL); + if (!edev) + return -ENOMEM; + } + + bnxt_set_edev_info(edev, bp); + bnxt_adev->edev = edev; + bp->edev = edev; + + ret = auxiliary_device_init(aux_dev); + if (ret) { + kfree(edev); + kfree(bnxt_adev); + return ret; + } + + ret = auxiliary_device_add(aux_dev); + if (ret) { + kfree(edev); + kfree(bnxt_adev); + auxiliary_device_uninit(aux_dev); + return ret; + } + + return 0; +} diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index 42b50abc3e91..f26f1075bfee 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -17,6 +17,7 @@ #define BNXT_MIN_ROCE_STAT_CTXS 1 struct hwrm_async_event_cmpl; +struct bnxt_aux_dev; struct bnxt; struct bnxt_msix_entry { @@ -102,10 +103,10 @@ int bnxt_get_ulp_stat_ctxs(struct bnxt *bp); void bnxt_ulp_stop(struct bnxt *bp); void bnxt_ulp_start(struct bnxt *bp, int err); void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs); -void bnxt_ulp_shutdown(struct bnxt *bp); void bnxt_ulp_irq_stop(struct bnxt *bp); void bnxt_ulp_irq_restart(struct bnxt *bp, int err); void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl); -struct bnxt_en_dev *bnxt_ulp_probe(struct net_device *dev); +void bnxt_aux_dev_release(struct device *dev); +int bnxt_rdma_aux_device_add(struct bnxt *bp); #endif From patchwork Tue Oct 25 17:31:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 10903 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1131569wru; Tue, 25 Oct 2022 10:36:48 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7J4iHNYD4o8Dr3ALdXjTlw1EjUs3Fm//2OlfgPdRifxaHVTSYNotL/VYTZ3AvjKTdSRELb X-Received: by 2002:aa7:d1c3:0:b0:461:c627:2a7b with SMTP id g3-20020aa7d1c3000000b00461c6272a7bmr10804305edp.231.1666719408115; Tue, 25 Oct 2022 10:36:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666719408; cv=none; d=google.com; s=arc-20160816; b=J0+7hwu7BGMsPZXomnoIKXfaN9cUxPNBPLP2DHifdP7vnWrUSJ6I+efPWotn5Fbb32 Yr0NKVNuLKVnjnCNtT0fma+vIzjeqpBtoRr2dwhlZYIwC6zYfwh45k1ybvoboDRg1hJN 7+KPiqCAZeFlUPskEOSrjAoeelYBmS3eaTUNEXPSTxroa/KaYL16v2wGlGBuE9wUL7ls vSYnh1ENMfT7+KwhFQwcKh2P+lC0qwlBoUi42XkqbudPUFjq54ce07UDmQ0T5QWXaGL5 VWSmadSfp9YL3fyT/fIocbId074rJseXYarRNux9Qbz6gB7iRiHsMz3JSZuuA1qI/dGH NRTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=/5OMqcDYnuwe6s/vCboa0OZAUr4SygDgPRG5Ixzan4Y=; b=dE/lI2hgJRI9LkdcIoejEaeRpeYZibLjpizoklV+aWsqaikzSmi9P/s7UWjAT7XO4L AfujF35Gmyzz0eiVua0+bMKgU4rujQS0QyRyGabFq2O+kGbBA61IERTYzT/MY4KF4+f0 seAdghJJjzTfEms2zukaQXaE7jUVaf0q2Ewm7YUU1lq8MYpQ2lnHmsZNhvqRoOo3X1z+ 91glMLhDSNGtPSXq2uQjHgXCM+b1N8PqeUX2hVN9ZJr0bNn9EVSgdiGFc5fXxBn0+N5L 9UeOsx15b06wXH5nsGJ//0Y5FX/qvaw9sMKMLHb0Imfnta0bQfBL1O1lVBlC6GnzVxDK GhDg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=U69a19Y5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a15-20020a50e70f000000b0045fb7a04ce0si3194728edn.148.2022.10.25.10.36.17; Tue, 25 Oct 2022 10:36:47 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=U69a19Y5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232572AbiJYRbk (ORCPT + 99 others); Tue, 25 Oct 2022 13:31:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46200 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232553AbiJYRbd (ORCPT ); Tue, 25 Oct 2022 13:31:33 -0400 Received: from mail-qk1-x736.google.com (mail-qk1-x736.google.com [IPv6:2607:f8b0:4864:20::736]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A8FB516EA2A for ; Tue, 25 Oct 2022 10:31:26 -0700 (PDT) Received: by mail-qk1-x736.google.com with SMTP id f8so8600723qkg.3 for ; Tue, 25 Oct 2022 10:31:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=/5OMqcDYnuwe6s/vCboa0OZAUr4SygDgPRG5Ixzan4Y=; b=U69a19Y5O2qf+uWcln6XsNzbabq/i6RstqRbnknOpRUkR56obXyxeMedax9i77ocVR GI4a8m9d7+qg8U7VPdPSnbU2R7Gy30UPiaIChVg7NnLb2xksAn5WNjizXY70nrelx3DX esm1CmDeeN9HxoaXucc7IiE9/HrU48osT5pAY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/5OMqcDYnuwe6s/vCboa0OZAUr4SygDgPRG5Ixzan4Y=; b=VHDdEE0quSdVBGwtpep8ohSFsf8jxWpuseCxqrbWZNv4yMATliWJYP//LggHcYvsid E4JxgZG/qCaRUfhtFyQ3qKh1k4LX2gXVxTmwD82jnIvPsIf/qNq0vv06atWbMx8w9az9 dDIbK2prYyL+/3Lj2qMR2KcmLXXLrJQGk//WiayeW+K6jiyx/9uUZKDQuWJKeT5jHgVF fcjR65R6lHL1mZcuAU8d65xspp0+/wD8RB0MgQWIDZZ2mP2iiWKCmMIuWnyA9wmBdIzA 8ZNq5j/mvO9oURdU8ajD7TfiTjyxHo+UPE0FjAoMbeeMVECEWw9SVwHGCiIPWto/dtV4 GuaQ== X-Gm-Message-State: ACrzQf1Kvj1ryOiCRxJ6dhogQihm/N9F/KPRWQTVsTAq80UIe2rE0263 0ckaTXKPH5C9tFx1mVDqnu0MHcjsI9CY7A== X-Received: by 2002:a05:620a:2893:b0:6bc:6ecd:2c44 with SMTP id j19-20020a05620a289300b006bc6ecd2c44mr26857523qkp.593.1666719085592; Tue, 25 Oct 2022 10:31:25 -0700 (PDT) Received: from C02GC2QQMD6T.wifi.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id u6-20020a37ab06000000b006eed094dcdasm2329034qke.70.2022.10.25.10.31.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Oct 2022 10:31:24 -0700 (PDT) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com Subject: [PATCH v2 2/6] RDMA/bnxt_re: Use auxiliary driver interface Date: Tue, 25 Oct 2022 10:31:06 -0700 Message-Id: <20221025173110.33192-3-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.0 (Apple Git-136) In-Reply-To: <20221025173110.33192-1-ajit.khaparde@broadcom.com> References: <20220724231458.93830-1-ajit.khaparde@broadcom.com> <20221025173110.33192-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SORTED_RECIPS,SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747681969687797776?= X-GMAIL-MSGID: =?utf-8?q?1747681969687797776?= Use auxiliary driver interface for driver load, unload ROCE driver. The driver does not need to register the interface using the netdev notifier anymore. Removed the bnxt_re_dev_list which is not needed. Currently probe, remove and shutdown ops have been implemented for the auxiliary device. Signed-off-by: Ajit Khaparde Reviewed-by: Andy Gospodarek --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 9 +- drivers/infiniband/hw/bnxt_re/main.c | 387 +++++++----------- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 64 --- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 67 ++- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 3 + 5 files changed, 214 insertions(+), 316 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index 785c37cae3c0..b0465c8d229a 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -89,13 +89,6 @@ struct bnxt_re_ring_attr { u8 mode; }; -struct bnxt_re_work { - struct work_struct work; - unsigned long event; - struct bnxt_re_dev *rdev; - struct net_device *vlan_dev; -}; - struct bnxt_re_sqp_entries { struct bnxt_qplib_sge sge; u64 wrid; @@ -132,6 +125,7 @@ struct bnxt_re_dev { #define BNXT_RE_FLAG_ERR_DEVICE_DETACHED 17 #define BNXT_RE_FLAG_ISSUE_ROCE_STATS 29 struct net_device *netdev; + struct notifier_block nb; unsigned int version, major, minor; struct bnxt_qplib_chip_ctx *chip_ctx; struct bnxt_en_dev *en_dev; @@ -194,5 +188,4 @@ static inline struct device *rdev_to_dev(struct bnxt_re_dev *rdev) return &rdev->ibdev.dev; return NULL; } - #endif diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index f868b285d2a1..0bbd2b3095d6 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -48,6 +48,7 @@ #include #include #include +#include #include #include @@ -74,14 +75,14 @@ MODULE_DESCRIPTION(BNXT_RE_DESC " Driver"); MODULE_LICENSE("Dual BSD/GPL"); /* globals */ -static struct list_head bnxt_re_dev_list = LIST_HEAD_INIT(bnxt_re_dev_list); -/* Mutex to protect the list of bnxt_re devices added */ -static DEFINE_MUTEX(bnxt_re_dev_lock); -static struct workqueue_struct *bnxt_re_wq; -static void bnxt_re_remove_device(struct bnxt_re_dev *rdev); -static void bnxt_re_dealloc_driver(struct ib_device *ib_dev); +static DEFINE_MUTEX(bnxt_re_mutex); + static void bnxt_re_stop_irq(void *handle); static void bnxt_re_dev_stop(struct bnxt_re_dev *rdev); +static int bnxt_re_netdev_event(struct notifier_block *notifier, + unsigned long event, void *ptr); +static void bnxt_re_remove_device(struct bnxt_re_dev *rdev); +static struct bnxt_re_dev *bnxt_re_from_netdev(struct net_device *netdev); static void bnxt_re_set_drv_mode(struct bnxt_re_dev *rdev, u8 mode) { @@ -233,7 +234,6 @@ static void bnxt_re_stop(void *p) if (!rdev) return; - ASSERT_RTNL(); /* L2 driver invokes this callback during device error/crash or device * reset. Current RoCE driver doesn't recover the device in case of @@ -282,16 +282,14 @@ static void bnxt_re_sriov_config(void *p, int num_vfs) } } -static void bnxt_re_shutdown(void *p) +static void bnxt_re_shutdown(struct auxiliary_device *adev) { - struct bnxt_re_dev *rdev = p; + struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); if (!rdev) return; - ASSERT_RTNL(); - /* Release the MSIx vectors before queuing unregister */ - bnxt_re_stop_irq(rdev); - ib_unregister_device_queued(&rdev->ibdev); + ib_unregister_device(&rdev->ibdev); + bnxt_re_remove_device(rdev); } static void bnxt_re_stop_irq(void *handle) @@ -346,11 +344,9 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) } static struct bnxt_ulp_ops bnxt_re_ulp_ops = { - .ulp_async_notifier = NULL, .ulp_stop = bnxt_re_stop, .ulp_start = bnxt_re_start, .ulp_sriov_config = bnxt_re_sriov_config, - .ulp_shutdown = bnxt_re_shutdown, .ulp_irq_stop = bnxt_re_stop_irq, .ulp_irq_restart = bnxt_re_start_irq }; @@ -401,7 +397,6 @@ static int bnxt_re_free_msix(struct bnxt_re_dev *rdev) en_dev = rdev->en_dev; - rc = en_dev->en_ops->bnxt_free_msix(rdev->en_dev, BNXT_ROCE_ULP); return rc; @@ -458,12 +453,17 @@ static void bnxt_re_fill_fw_msg(struct bnxt_fw_msg *fw_msg, void *msg, static int bnxt_re_net_ring_free(struct bnxt_re_dev *rdev, u16 fw_ring_id, int type) { - struct bnxt_en_dev *en_dev = rdev->en_dev; + struct bnxt_en_dev *en_dev; struct hwrm_ring_free_input req = {0}; struct hwrm_ring_free_output resp; struct bnxt_fw_msg fw_msg; int rc = -EINVAL; + if (!rdev) + return rc; + + en_dev = rdev->en_dev; + if (!en_dev) return rc; @@ -584,21 +584,6 @@ static int bnxt_re_net_stats_ctx_alloc(struct bnxt_re_dev *rdev, /* Device */ -static bool is_bnxt_re_dev(struct net_device *netdev) -{ - struct ethtool_drvinfo drvinfo; - - if (netdev->ethtool_ops && netdev->ethtool_ops->get_drvinfo) { - memset(&drvinfo, 0, sizeof(drvinfo)); - netdev->ethtool_ops->get_drvinfo(netdev, &drvinfo); - - if (strcmp(drvinfo.driver, "bnxt_en")) - return false; - return true; - } - return false; -} - static struct bnxt_re_dev *bnxt_re_from_netdev(struct net_device *netdev) { struct ib_device *ibdev = @@ -609,27 +594,22 @@ static struct bnxt_re_dev *bnxt_re_from_netdev(struct net_device *netdev) return container_of(ibdev, struct bnxt_re_dev, ibdev); } -static struct bnxt_en_dev *bnxt_re_dev_probe(struct net_device *netdev) +static struct bnxt_en_dev *bnxt_re_dev_probe(struct auxiliary_device *adev) { - struct bnxt_en_dev *en_dev; + struct bnxt_aux_dev *aux_dev = + container_of(adev, struct bnxt_aux_dev, aux_dev); + struct bnxt_en_dev *en_dev = NULL; struct pci_dev *pdev; - en_dev = ((struct bnxt*)netdev_priv(netdev))->edev; - if (IS_ERR(en_dev)) - return en_dev; + if (aux_dev) + en_dev = aux_dev->edev; + + if (!en_dev) + return NULL; pdev = en_dev->pdev; if (!pdev) - return ERR_PTR(-EINVAL); - - if (!(en_dev->flags & BNXT_EN_FLAG_ROCE_CAP)) { - dev_info(&pdev->dev, - "%s: probe error: RoCE is not supported on this device", - ROCE_DRV_MODULE_NAME); - return ERR_PTR(-ENODEV); - } - - dev_hold(netdev); + return NULL; return en_dev; } @@ -679,7 +659,6 @@ static const struct ib_device_ops bnxt_re_dev_ops = { .create_qp = bnxt_re_create_qp, .create_srq = bnxt_re_create_srq, .create_user_ah = bnxt_re_create_ah, - .dealloc_driver = bnxt_re_dealloc_driver, .dealloc_pd = bnxt_re_dealloc_pd, .dealloc_ucontext = bnxt_re_dealloc_ucontext, .del_gid = bnxt_re_del_gid, @@ -744,18 +723,7 @@ static int bnxt_re_register_ib(struct bnxt_re_dev *rdev) return ib_register_device(ibdev, "bnxt_re%d", &rdev->en_dev->pdev->dev); } -static void bnxt_re_dev_remove(struct bnxt_re_dev *rdev) -{ - dev_put(rdev->netdev); - rdev->netdev = NULL; - mutex_lock(&bnxt_re_dev_lock); - list_del_rcu(&rdev->list); - mutex_unlock(&bnxt_re_dev_lock); - - synchronize_rcu(); -} - -static struct bnxt_re_dev *bnxt_re_dev_add(struct net_device *netdev, +static struct bnxt_re_dev *bnxt_re_dev_add(struct bnxt_aux_dev *aux_dev, struct bnxt_en_dev *en_dev) { struct bnxt_re_dev *rdev; @@ -768,8 +736,8 @@ static struct bnxt_re_dev *bnxt_re_dev_add(struct net_device *netdev, return NULL; } /* Default values */ - rdev->netdev = netdev; - dev_hold(rdev->netdev); + rdev->nb.notifier_call = NULL; + rdev->netdev = en_dev->net; rdev->en_dev = en_dev; rdev->id = rdev->en_dev->pdev->devfn; INIT_LIST_HEAD(&rdev->qp_list); @@ -784,9 +752,6 @@ static struct bnxt_re_dev *bnxt_re_dev_add(struct net_device *netdev, rdev->cosq[0] = 0xFFFF; rdev->cosq[1] = 0xFFFF; - mutex_lock(&bnxt_re_dev_lock); - list_add_tail_rcu(&rdev->list, &bnxt_re_dev_list); - mutex_unlock(&bnxt_re_dev_lock); return rdev; } @@ -1147,6 +1112,9 @@ static void bnxt_re_dev_stop(struct bnxt_re_dev *rdev) struct ib_qp_attr qp_attr; struct bnxt_re_qp *qp; + if (!rdev) + return; + qp_attr.qp_state = IB_QPS_ERR; mutex_lock(&rdev->qp_lock); list_for_each_entry(qp, &rdev->qp_list, list) { @@ -1323,7 +1291,7 @@ static int bnxt_re_ib_init(struct bnxt_re_dev *rdev) pr_err("Failed to register with IB: %#x\n", rc); return rc; } - dev_info(rdev_to_dev(rdev), "Device registered successfully"); + dev_info(rdev_to_dev(rdev), "Device registered with IB successfully"); ib_get_eth_speed(&rdev->ibdev, 1, &rdev->active_speed, &rdev->active_width); set_bit(BNXT_RE_FLAG_ISSUE_ROCE_STATS, &rdev->flags); @@ -1541,135 +1509,49 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 wqe_mode) return rc; } -static void bnxt_re_dev_unreg(struct bnxt_re_dev *rdev) +static void bnxt_re_remove_device(struct bnxt_re_dev *rdev) { - struct net_device *netdev = rdev->netdev; - - bnxt_re_dev_remove(rdev); - - if (netdev) - dev_put(netdev); + bnxt_re_dev_uninit(rdev); } -static int bnxt_re_dev_reg(struct bnxt_re_dev **rdev, struct net_device *netdev) +static int bnxt_re_add_device(struct auxiliary_device *adev, u8 wqe_mode) { + struct bnxt_aux_dev *aux_dev = + container_of(adev, struct bnxt_aux_dev, aux_dev); struct bnxt_en_dev *en_dev; + struct bnxt_re_dev *rdev; int rc = 0; - if (!is_bnxt_re_dev(netdev)) + en_dev = bnxt_re_dev_probe(adev); + if (!en_dev) return -ENODEV; - en_dev = bnxt_re_dev_probe(netdev); - if (IS_ERR(en_dev)) { - if (en_dev != ERR_PTR(-ENODEV)) - ibdev_err(&(*rdev)->ibdev, "%s: Failed to probe\n", - ROCE_DRV_MODULE_NAME); - rc = PTR_ERR(en_dev); - goto exit; - } - *rdev = bnxt_re_dev_add(netdev, en_dev); - if (!*rdev) { + rdev = bnxt_re_dev_add(aux_dev, en_dev); + if (!rdev || !rdev_to_dev(rdev)) { rc = -ENOMEM; - dev_put(netdev); goto exit; } -exit: - return rc; -} - -static void bnxt_re_remove_device(struct bnxt_re_dev *rdev) -{ - bnxt_re_dev_uninit(rdev); - pci_dev_put(rdev->en_dev->pdev); - bnxt_re_dev_unreg(rdev); -} - -static int bnxt_re_add_device(struct bnxt_re_dev **rdev, - struct net_device *netdev, u8 wqe_mode) -{ - int rc; - rc = bnxt_re_dev_reg(rdev, netdev); - if (rc == -ENODEV) - return rc; - if (rc) { - pr_err("Failed to register with the device %s: %#x\n", - netdev->name, rc); - return rc; - } + rc = bnxt_re_dev_init(rdev, wqe_mode); + if (rc) + goto re_dev_dealloc; - pci_dev_get((*rdev)->en_dev->pdev); - rc = bnxt_re_dev_init(*rdev, wqe_mode); + rc = bnxt_re_ib_init(rdev); if (rc) { - pci_dev_put((*rdev)->en_dev->pdev); - bnxt_re_dev_unreg(*rdev); + pr_err("Failed to register with IB: %s", + aux_dev->aux_dev.name); + goto re_dev_uninit; } + auxiliary_set_drvdata(adev, rdev); - return rc; -} - -static void bnxt_re_dealloc_driver(struct ib_device *ib_dev) -{ - struct bnxt_re_dev *rdev = - container_of(ib_dev, struct bnxt_re_dev, ibdev); - - dev_info(rdev_to_dev(rdev), "Unregistering Device"); - - rtnl_lock(); - bnxt_re_remove_device(rdev); - rtnl_unlock(); -} - -/* Handle all deferred netevents tasks */ -static void bnxt_re_task(struct work_struct *work) -{ - struct bnxt_re_work *re_work; - struct bnxt_re_dev *rdev; - int rc = 0; - - re_work = container_of(work, struct bnxt_re_work, work); - rdev = re_work->rdev; - - if (re_work->event == NETDEV_REGISTER) { - rc = bnxt_re_ib_init(rdev); - if (rc) { - ibdev_err(&rdev->ibdev, - "Failed to register with IB: %#x", rc); - rtnl_lock(); - bnxt_re_remove_device(rdev); - rtnl_unlock(); - goto exit; - } - goto exit; - } - - if (!ib_device_try_get(&rdev->ibdev)) - goto exit; + return 0; - switch (re_work->event) { - case NETDEV_UP: - bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, - IB_EVENT_PORT_ACTIVE); - break; - case NETDEV_DOWN: - bnxt_re_dev_stop(rdev); - break; - case NETDEV_CHANGE: - if (!netif_carrier_ok(rdev->netdev)) - bnxt_re_dev_stop(rdev); - else if (netif_carrier_ok(rdev->netdev)) - bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, - IB_EVENT_PORT_ACTIVE); - ib_get_eth_speed(&rdev->ibdev, 1, &rdev->active_speed, - &rdev->active_width); - break; - default: - break; - } - ib_device_put(&rdev->ibdev); +re_dev_uninit: + bnxt_re_dev_uninit(rdev); +re_dev_dealloc: + ib_dealloc_device(&rdev->ibdev); exit: - put_device(&rdev->ibdev.dev); - kfree(re_work); + return rc; } /* @@ -1690,109 +1572,130 @@ static int bnxt_re_netdev_event(struct notifier_block *notifier, unsigned long event, void *ptr) { struct net_device *real_dev, *netdev = netdev_notifier_info_to_dev(ptr); - struct bnxt_re_work *re_work; struct bnxt_re_dev *rdev; - int rc = 0; - bool sch_work = false; - bool release = true; real_dev = rdma_vlan_dev_real_dev(netdev); if (!real_dev) real_dev = netdev; - rdev = bnxt_re_from_netdev(real_dev); - if (!rdev && event != NETDEV_REGISTER) - return NOTIFY_OK; - if (real_dev != netdev) goto exit; - switch (event) { - case NETDEV_REGISTER: - if (rdev) - break; - rc = bnxt_re_add_device(&rdev, real_dev, - BNXT_QPLIB_WQE_MODE_STATIC); - if (!rc) - sch_work = true; - release = false; - break; + rdev = bnxt_re_from_netdev(real_dev); + if (!rdev) + return NOTIFY_DONE; - case NETDEV_UNREGISTER: - ib_unregister_device_queued(&rdev->ibdev); - break; + switch (event) { + case NETDEV_UP: + case NETDEV_DOWN: + case NETDEV_CHANGE: + bnxt_re_dispatch_event(&rdev->ibdev, NULL, 1, + netif_carrier_ok(real_dev) ? + IB_EVENT_PORT_ACTIVE : + IB_EVENT_PORT_ERR); + break; default: - sch_work = true; break; } - if (sch_work) { - /* Allocate for the deferred task */ - re_work = kzalloc(sizeof(*re_work), GFP_KERNEL); - if (re_work) { - get_device(&rdev->ibdev.dev); - re_work->rdev = rdev; - re_work->event = event; - re_work->vlan_dev = (real_dev == netdev ? - NULL : netdev); - INIT_WORK(&re_work->work, bnxt_re_task); - queue_work(bnxt_re_wq, &re_work->work); - } - } - + ib_device_put(&rdev->ibdev); exit: - if (rdev && release) - ib_device_put(&rdev->ibdev); return NOTIFY_DONE; } -static struct notifier_block bnxt_re_netdev_notifier = { - .notifier_call = bnxt_re_netdev_event -}; +#define BNXT_ADEV_NAME "bnxt_en" -static int __init bnxt_re_mod_init(void) +static void bnxt_re_remove(struct auxiliary_device *adev) { - int rc = 0; + struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); - pr_info("%s: %s", ROCE_DRV_MODULE_NAME, version); + if (!rdev) + return; - bnxt_re_wq = create_singlethread_workqueue("bnxt_re"); - if (!bnxt_re_wq) - return -ENOMEM; + mutex_lock(&bnxt_re_mutex); + if (rdev->nb.notifier_call) { + unregister_netdevice_notifier(&rdev->nb); + rdev->nb.notifier_call = NULL; + } else { + /* If notifier is null, we should have already done a + * clean up before coming here. + */ + goto skip_remove; + } + + ib_unregister_device(&rdev->ibdev); + ib_dealloc_device(&rdev->ibdev); + bnxt_re_remove_device(rdev); +skip_remove: + mutex_unlock(&bnxt_re_mutex); +} + +static int bnxt_re_probe(struct auxiliary_device *adev, + const struct auxiliary_device_id *id) +{ + struct bnxt_re_dev *rdev; + int rc; - INIT_LIST_HEAD(&bnxt_re_dev_list); + mutex_lock(&bnxt_re_mutex); + rc = bnxt_re_add_device(adev, BNXT_QPLIB_WQE_MODE_STATIC); + if (rc) { + mutex_unlock(&bnxt_re_mutex); + return rc; + } - rc = register_netdevice_notifier(&bnxt_re_netdev_notifier); + rdev = auxiliary_get_drvdata(adev); + + rdev->nb.notifier_call = bnxt_re_netdev_event; + rc = register_netdevice_notifier(&rdev->nb); if (rc) { + rdev->nb.notifier_call = NULL; pr_err("%s: Cannot register to netdevice_notifier", ROCE_DRV_MODULE_NAME); - goto err_netdev; + goto err; } + + mutex_unlock(&bnxt_re_mutex); return 0; -err_netdev: - destroy_workqueue(bnxt_re_wq); +err: + mutex_unlock(&bnxt_re_mutex); + bnxt_re_remove(adev); return rc; } -static void __exit bnxt_re_mod_exit(void) +static const struct auxiliary_device_id bnxt_re_id_table[] = { + { .name = BNXT_ADEV_NAME ".rdma", }, + {}, +}; + +MODULE_DEVICE_TABLE(auxiliary, bnxt_re_id_table); + +static struct auxiliary_driver bnxt_re_driver = { + .name = "rdma", + .probe = bnxt_re_probe, + .remove = bnxt_re_remove, + .shutdown = bnxt_re_shutdown, + .id_table = bnxt_re_id_table, +}; + +static int __init bnxt_re_mod_init(void) { - struct bnxt_re_dev *rdev; + int rc = 0; - unregister_netdevice_notifier(&bnxt_re_netdev_notifier); - if (bnxt_re_wq) - destroy_workqueue(bnxt_re_wq); - list_for_each_entry(rdev, &bnxt_re_dev_list, list) { - /* VF device removal should be called before the removal - * of PF device. Queue VFs unregister first, so that VFs - * shall be removed before the PF during the call of - * ib_unregister_driver. - */ - if (rdev->is_virtfn) - ib_unregister_device(&rdev->ibdev); + pr_info("%s: %s", ROCE_DRV_MODULE_NAME, version); + rc = auxiliary_driver_register(&bnxt_re_driver); + if (rc) { + pr_err("%s: Failed to register auxiliary driver\n", + ROCE_DRV_MODULE_NAME); + return rc; } - ib_unregister_driver(RDMA_DRIVER_BNXT_RE); + return 0; +} + +static void __exit bnxt_re_mod_exit(void) +{ + auxiliary_driver_unregister(&bnxt_re_driver); } module_init(bnxt_re_mod_init); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 0c8786ea2c1c..3cbedbecf3b3 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -79,8 +79,6 @@ MODULE_LICENSE("GPL"); MODULE_DESCRIPTION("Broadcom BCM573xx network driver"); -static DEFINE_IDA(bnxt_aux_dev_ids); - #define BNXT_RX_OFFSET (NET_SKB_PAD + NET_IP_ALIGN) #define BNXT_RX_DMA_OFFSET NET_SKB_PAD #define BNXT_RX_COPY_THRESH 256 @@ -213,68 +211,6 @@ static const struct pci_device_id bnxt_pci_tbl[] = { MODULE_DEVICE_TABLE(pci, bnxt_pci_tbl); -static void bnxt_aux_dev_free(struct bnxt *bp) -{ - kfree(bp->aux_dev); - bp->aux_dev = NULL; -} - -static struct bnxt_aux_dev *bnxt_aux_dev_alloc(struct bnxt *bp) -{ - struct bnxt_aux_dev *bnxt_adev; - - bnxt_adev = kzalloc(sizeof(*bnxt_adev), GFP_KERNEL); - if (!bnxt_adev) - return ERR_PTR(-ENOMEM); - - return bnxt_adev; -} - -static void bnxt_rdma_aux_device_uninit(struct bnxt_aux_dev *bnxt_adev) -{ - struct auxiliary_device *adev; - - if (IS_ERR_OR_NULL(bnxt_adev)) - return; - - adev = &bnxt_adev->aux_dev; - auxiliary_device_delete(adev); - auxiliary_device_uninit(adev); - if (bnxt_adev->id >= 0) - ida_free(&bnxt_aux_dev_ids, bnxt_adev->id); -} - -static void bnxt_rdma_aux_device_init(struct bnxt *bp) -{ - struct net_device *dev = bp->dev; - int rc; - - if (bp->flags & BNXT_FLAG_ROCE_CAP) { - bp->aux_dev = bnxt_aux_dev_alloc(bp); - if (IS_ERR_OR_NULL(bp->aux_dev)) { - netdev_warn(dev, "Failed to init auxiliary device for ROCE\n"); - goto skip_aux_init; - } - - bp->aux_dev->id = ida_alloc(&bnxt_aux_dev_ids, GFP_KERNEL); - if (bp->aux_dev->id < 0) { - netdev_warn(dev, "ida alloc failed for ROCE auxiliary device\n"); - bnxt_aux_dev_free(bp); - goto skip_aux_init; - } - - /* If aux bus init fails, continue with netdev init. */ - rc = bnxt_rdma_aux_device_add(bp); - if (rc) { - netdev_warn(dev, "Failed to add auxiliary device for ROCE\n"); - ida_free(&bnxt_aux_dev_ids, bp->aux_dev->id); - bnxt_aux_dev_free(bp); - } - } -skip_aux_init: - return; -} - static const u16 bnxt_vf_req_snif[] = { HWRM_FUNC_CFG, HWRM_FUNC_VF_CFG, diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index 44a0922663cf..1d2d30f97ed9 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -19,12 +19,15 @@ #include #include #include +#include #include "bnxt_hsi.h" #include "bnxt.h" #include "bnxt_hwrm.h" #include "bnxt_ulp.h" +static DEFINE_IDA(bnxt_aux_dev_ids); + static int bnxt_register_dev(struct bnxt_en_dev *edev, unsigned int ulp_id, struct bnxt_ulp_ops *ulp_ops, void *handle) { @@ -76,8 +79,6 @@ static int bnxt_unregister_dev(struct bnxt_en_dev *edev, unsigned int ulp_id) if (ulp_id >= BNXT_MAX_ULP) return -EINVAL; - edev->flags |= BNXT_EN_FLAG_ULP_STOPPED; - ulp = &edev->ulp_tbl[ulp_id]; if (!rcu_access_pointer(ulp->ulp_ops)) { netdev_err(bp->dev, "ulp id %d not registered\n", ulp_id); @@ -465,6 +466,68 @@ static const struct bnxt_en_ops bnxt_en_ops_tbl = { .bnxt_register_fw_async_events = bnxt_register_async_events, }; +void bnxt_aux_dev_free(struct bnxt *bp) +{ + kfree(bp->aux_dev); + bp->aux_dev = NULL; +} + +static struct bnxt_aux_dev *bnxt_aux_dev_alloc(struct bnxt *bp) +{ + struct bnxt_aux_dev *bnxt_adev; + + bnxt_adev = kzalloc(sizeof(*bnxt_adev), GFP_KERNEL); + if (!bnxt_adev) + return ERR_PTR(-ENOMEM); + + return bnxt_adev; +} + +void bnxt_rdma_aux_device_uninit(struct bnxt_aux_dev *bnxt_adev) +{ + struct auxiliary_device *adev; + + if (IS_ERR_OR_NULL(bnxt_adev)) + return; + + adev = &bnxt_adev->aux_dev; + auxiliary_device_delete(adev); + auxiliary_device_uninit(adev); + if (bnxt_adev->id >= 0) + ida_free(&bnxt_aux_dev_ids, bnxt_adev->id); +} + +void bnxt_rdma_aux_device_init(struct bnxt *bp) +{ + struct net_device *dev = bp->dev; + int rc; + + if (bp->flags & BNXT_FLAG_ROCE_CAP) { + bp->aux_dev = bnxt_aux_dev_alloc(bp); + if (IS_ERR_OR_NULL(bp->aux_dev)) { + netdev_warn(dev, "Failed to init auxiliary device for ROCE\n"); + goto skip_aux_init; + } + + bp->aux_dev->id = ida_alloc(&bnxt_aux_dev_ids, GFP_KERNEL); + if (bp->aux_dev->id < 0) { + netdev_warn(dev, "ida alloc failed for ROCE auxiliary device\n"); + bnxt_aux_dev_free(bp); + goto skip_aux_init; + } + + /* If aux bus init fails, continue with netdev init. */ + rc = bnxt_rdma_aux_device_add(bp); + if (rc) { + netdev_warn(dev, "Failed to add auxiliary device for ROCE\n"); + ida_free(&bnxt_aux_dev_ids, bp->aux_dev->id); + bnxt_aux_dev_free(bp); + } + } +skip_aux_init: + return; +} + void bnxt_aux_dev_release(struct device *dev) { struct bnxt_aux_dev *bnxt_adev = diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index f26f1075bfee..6cac4c0b708b 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -109,4 +109,7 @@ void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl); void bnxt_aux_dev_release(struct device *dev); int bnxt_rdma_aux_device_add(struct bnxt *bp); +void bnxt_rdma_aux_device_uninit(struct bnxt_aux_dev *bnxt_adev); +void bnxt_rdma_aux_device_init(struct bnxt *bp); +void bnxt_aux_dev_free(struct bnxt *bp); #endif From patchwork Tue Oct 25 17:31:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 10906 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1131865wru; Tue, 25 Oct 2022 10:37:29 -0700 (PDT) X-Google-Smtp-Source: AMsMyM55doUuPL88lsLzcbwqdcZ5KjRULvqAdpzlZmhWyKIlv24DDdTQtX36ER0/hwaQDiqiBHym X-Received: by 2002:a05:6402:1842:b0:461:59b6:3f1b with SMTP id v2-20020a056402184200b0046159b63f1bmr21047199edy.308.1666719448946; Tue, 25 Oct 2022 10:37:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666719448; cv=none; d=google.com; s=arc-20160816; b=xq21nmJG6lVlS3N8AHTkdynq04H4Adue5AA6RiIvi/MZEsK3jTxkQQO+qOEHfDWxso 2/yrpg9TYKbL67DL/1DMOJ41bGEIw+fy8M2r9OeEj8NGw9bya4ub9PGaJ/xxOxUJwR+m rwwX0KfxNQ+tiZGRp1lw5mmppdJA+xofqq3CaHtb9RHxBrn0FmegXuvib6GuugQmbIDW +WVz+H2FaKWJgrvlnz0BV3FzDNWtgjwUiFdcjbF6BRuQVY1hQHIao85AacEYI2LyGkOe 42ezzSF39odz/BCxWTD1T8FsZhgpuK5PIxZbBt6ph5g8dDWYqn0bCzktWvjWJxrkw8mn qifw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=/blX+tmze/DyHfd1ou5KLrkpEkQMQtjELdlhtuo/CJ4=; b=gihvKv1zqHBJfgobBj1Er1ewAJI4x/hq7H1ScdfRxEjkvhxwTnhH2qxkIi+yLGyETi 7J72ylri0Itq98VqSj5NMjT8JK1/GYv9/ATk6v5O/BZwy8FF2DWZMQbv/5hV61dRtSdt AALIkjdSP6I/3wKztLf8NWymouWZKonHOHVVpAnt7IS9YpecTS57m75IZOdTHIqjVtXq fxbTAPI0nVnU4q7Plt5vcNQtEqV/3WVuAWpkKOQK9p9W+zZtY1kF16LqBhDHXb1WQ/xR rJ20x1mRFStub0aA5TZTyQStcYhK12SDYoGzhl2A6v9KdHbMDEuanC45fEv7iIaX31rn Bs3Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=EEo3NoUN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id js5-20020a17090797c500b0078d27c1c8f4si3991032ejc.696.2022.10.25.10.37.04; Tue, 25 Oct 2022 10:37:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=EEo3NoUN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232086AbiJYRbz (ORCPT + 99 others); Tue, 25 Oct 2022 13:31:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46196 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232552AbiJYRbf (ORCPT ); Tue, 25 Oct 2022 13:31:35 -0400 Received: from mail-qt1-x832.google.com (mail-qt1-x832.google.com [IPv6:2607:f8b0:4864:20::832]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E0B5171CE9 for ; Tue, 25 Oct 2022 10:31:29 -0700 (PDT) Received: by mail-qt1-x832.google.com with SMTP id x3so1050489qtj.12 for ; Tue, 25 Oct 2022 10:31:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=/blX+tmze/DyHfd1ou5KLrkpEkQMQtjELdlhtuo/CJ4=; b=EEo3NoUNsuvV2Rw5R4xLmknOHKLx/rmQolYGTfdU/w1QUz6rSuPpaUVOEfpmenv7c7 81+xb/4plbQ1TDj6jaXVR5e25H3E4FxKtklh1qvvJi0E4Qz1LY13H/khCgj/FlCpdbIy 4N0dvtqsKDRq37AUd718C4v20nzeyXG8ySqz4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/blX+tmze/DyHfd1ou5KLrkpEkQMQtjELdlhtuo/CJ4=; b=fEHR4S3YSEfmQQji2StfFmdHGto6YqRC9nuncEGYu2iHCGlW8DclPxnIAxKB1mbPyn VIOyqT3I40YD5I0Aj+fdNuyDg+1yBNK00Ur50PI7+r9r9r81sO7PN+CNyxAglWOBmK7E p34y8H/YKjrjN2kJKqZbkEYS1tvrHiqJhCQB7kRHdhl9FxmUXMemcZ9DBDJaFgD25je+ koqv1hN2WKJHRBZJXL0RCwItJtGaNS2l+CciP4C//UThJ42rHoo+dYPyLO+sPPHK/ZAh dQK0rZvHt2Bg1HRtETui2b38NYxgVrEHNSZm9lgCNzo9EwbjGGuzF1aKowRt3Xe5fZ+y qepg== X-Gm-Message-State: ACrzQf1q9jeecrrDGlnipuoZhfxWwoDoZJMs6Yn291qI4W9aWp4iNYDX WEvjK4WR/kWJ3CH7hpcYODD7iQ== X-Received: by 2002:a05:622a:420d:b0:38d:961c:a57c with SMTP id cp13-20020a05622a420d00b0038d961ca57cmr32761589qtb.678.1666719088198; Tue, 25 Oct 2022 10:31:28 -0700 (PDT) Received: from C02GC2QQMD6T.wifi.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id u6-20020a37ab06000000b006eed094dcdasm2329034qke.70.2022.10.25.10.31.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Oct 2022 10:31:27 -0700 (PDT) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, Leon Romanovsky Subject: [PATCH v2 3/6] bnxt_en: Remove usage of ulp_id Date: Tue, 25 Oct 2022 10:31:07 -0700 Message-Id: <20221025173110.33192-4-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.0 (Apple Git-136) In-Reply-To: <20221025173110.33192-1-ajit.khaparde@broadcom.com> References: <20220724231458.93830-1-ajit.khaparde@broadcom.com> <20221025173110.33192-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747682012779921756?= X-GMAIL-MSGID: =?utf-8?q?1747682012779921756?= Since the driver continues to use the single ULP model, the extra complexity and indirection is unnecessary. Remove the usage of ulp_id from the code. Suggested-by: Leon Romanovsky Signed-off-by: Ajit Khaparde Reviewed-by: Andy Gospodarek --- drivers/infiniband/hw/bnxt_re/main.c | 24 +- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 2 +- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 212 +++++++----------- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 26 +-- 4 files changed, 111 insertions(+), 153 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 0bbd2b3095d6..755821c68020 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -366,8 +366,7 @@ static int bnxt_re_unregister_netdev(struct bnxt_re_dev *rdev) en_dev = rdev->en_dev; - rc = en_dev->en_ops->bnxt_unregister_device(rdev->en_dev, - BNXT_ROCE_ULP); + rc = en_dev->en_ops->bnxt_unregister_device(rdev->en_dev); return rc; } @@ -381,7 +380,7 @@ static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) en_dev = rdev->en_dev; - rc = en_dev->en_ops->bnxt_register_device(en_dev, BNXT_ROCE_ULP, + rc = en_dev->en_ops->bnxt_register_device(en_dev, &bnxt_re_ulp_ops, rdev); rdev->qplib_res.pdev = rdev->en_dev->pdev; return rc; @@ -390,16 +389,15 @@ static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) static int bnxt_re_free_msix(struct bnxt_re_dev *rdev) { struct bnxt_en_dev *en_dev; - int rc; if (!rdev) return -EINVAL; en_dev = rdev->en_dev; - rc = en_dev->en_ops->bnxt_free_msix(rdev->en_dev, BNXT_ROCE_ULP); + en_dev->en_ops->bnxt_free_msix(rdev->en_dev); - return rc; + return 0; } static int bnxt_re_request_msix(struct bnxt_re_dev *rdev) @@ -414,7 +412,7 @@ static int bnxt_re_request_msix(struct bnxt_re_dev *rdev) num_msix_want = min_t(u32, BNXT_RE_MAX_MSIX, num_online_cpus()); - num_msix_got = en_dev->en_ops->bnxt_request_msix(en_dev, BNXT_ROCE_ULP, + num_msix_got = en_dev->en_ops->bnxt_request_msix(en_dev, rdev->msix_entries, num_msix_want); if (num_msix_got < BNXT_RE_MIN_MSIX) { @@ -477,7 +475,7 @@ static int bnxt_re_net_ring_free(struct bnxt_re_dev *rdev, req.ring_id = cpu_to_le16(fw_ring_id); bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (rc) ibdev_err(&rdev->ibdev, "Failed to free HW ring:%d :%#x", req.ring_id, rc); @@ -514,7 +512,7 @@ static int bnxt_re_net_ring_alloc(struct bnxt_re_dev *rdev, req.int_mode = ring_attr->mode; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (!rc) *fw_ring_id = le16_to_cpu(resp.ring_id); @@ -542,7 +540,7 @@ static int bnxt_re_net_stats_ctx_free(struct bnxt_re_dev *rdev, req.stat_ctx_id = cpu_to_le32(fw_stats_ctx_id); bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (rc) ibdev_err(&rdev->ibdev, "Failed to free HW stats context %#x", rc); @@ -575,7 +573,7 @@ static int bnxt_re_net_stats_ctx_alloc(struct bnxt_re_dev *rdev, req.stat_ctx_flags = STAT_CTX_ALLOC_REQ_STAT_CTX_FLAGS_ROCE; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (!rc) *fw_stats_ctx_id = le32_to_cpu(resp.stat_ctx_id); @@ -1081,7 +1079,7 @@ static int bnxt_re_query_hwrm_pri2cos(struct bnxt_re_dev *rdev, u8 dir, bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (rc) return rc; @@ -1267,7 +1265,7 @@ static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) req.hwrm_intf_upd = HWRM_VERSION_UPDATE; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, BNXT_ROCE_ULP, &fw_msg); + rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); if (rc) { ibdev_err(&rdev->ibdev, "Failed to query HW version, rc = 0x%x", rc); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 3cbedbecf3b3..535f46dbe1f6 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -5508,7 +5508,7 @@ int bnxt_hwrm_vnic_cfg(struct bnxt *bp, u16 vnic_id) #endif if ((bp->flags & BNXT_FLAG_STRIP_VLAN) || def_vlan) req->flags |= cpu_to_le32(VNIC_CFG_REQ_FLAGS_VLAN_STRIP_MODE); - if (!vnic_id && bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) + if (!vnic_id && bnxt_ulp_registered(bp->edev)) req->flags |= cpu_to_le32(bnxt_get_roce_vnic_mode(bp)); return hwrm_req_send(bp, req); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index 1d2d30f97ed9..5fa653c2458a 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -28,76 +28,56 @@ static DEFINE_IDA(bnxt_aux_dev_ids); -static int bnxt_register_dev(struct bnxt_en_dev *edev, unsigned int ulp_id, - struct bnxt_ulp_ops *ulp_ops, void *handle) +static int bnxt_register_dev(struct bnxt_en_dev *edev, + struct bnxt_ulp_ops *ulp_ops, + void *handle) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); + unsigned int max_stat_ctxs; struct bnxt_ulp *ulp; - int rc = 0; - if (ulp_id >= BNXT_MAX_ULP) - return -EINVAL; + max_stat_ctxs = bnxt_get_max_func_stat_ctxs(bp); + if (max_stat_ctxs <= BNXT_MIN_ROCE_STAT_CTXS || + bp->cp_nr_rings == max_stat_ctxs) + return -ENOMEM; - ulp = &edev->ulp_tbl[ulp_id]; - if (rcu_access_pointer(ulp->ulp_ops)) { - netdev_err(bp->dev, "ulp id %d already registered\n", ulp_id); - rc = -EBUSY; - goto exit; - } - if (ulp_id == BNXT_ROCE_ULP) { - unsigned int max_stat_ctxs; - - max_stat_ctxs = bnxt_get_max_func_stat_ctxs(bp); - if (max_stat_ctxs <= BNXT_MIN_ROCE_STAT_CTXS || - bp->cp_nr_rings == max_stat_ctxs) { - rc = -ENOMEM; - goto exit; - } - } + ulp = kzalloc(sizeof(*ulp), GFP_KERNEL); + if (!ulp) + return -ENOMEM; - atomic_set(&ulp->ref_count, 1); + edev->ulp_tbl = ulp; ulp->handle = handle; rcu_assign_pointer(ulp->ulp_ops, ulp_ops); - if (ulp_id == BNXT_ROCE_ULP) { - if (test_bit(BNXT_STATE_OPEN, &bp->state)) - bnxt_hwrm_vnic_cfg(bp, 0); - } + if (test_bit(BNXT_STATE_OPEN, &bp->state)) + bnxt_hwrm_vnic_cfg(bp, 0); -exit: - return rc; + return 0; } -static int bnxt_unregister_dev(struct bnxt_en_dev *edev, unsigned int ulp_id) +static int bnxt_unregister_dev(struct bnxt_en_dev *edev) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); struct bnxt_ulp *ulp; int i = 0; - if (ulp_id >= BNXT_MAX_ULP) - return -EINVAL; - - ulp = &edev->ulp_tbl[ulp_id]; - if (!rcu_access_pointer(ulp->ulp_ops)) { - netdev_err(bp->dev, "ulp id %d not registered\n", ulp_id); - return -EINVAL; - } - if (ulp_id == BNXT_ROCE_ULP && ulp->msix_requested) - edev->en_ops->bnxt_free_msix(edev, ulp_id); + ulp = edev->ulp_tbl; + if (ulp->msix_requested) + edev->en_ops->bnxt_free_msix(edev); if (ulp->max_async_event_id) bnxt_hwrm_func_drv_rgtr(bp, NULL, 0, true); - RCU_INIT_POINTER(ulp->ulp_ops, NULL); - synchronize_rcu(); ulp->max_async_event_id = 0; ulp->async_events_bmap = NULL; while (atomic_read(&ulp->ref_count) != 0 && i < 10) { msleep(100); i++; } + kfree(ulp); + edev->ulp_tbl = NULL; return 0; } @@ -106,8 +86,8 @@ static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) struct bnxt_en_dev *edev = bp->edev; int num_msix, idx, i; - num_msix = edev->ulp_tbl[BNXT_ROCE_ULP].msix_requested; - idx = edev->ulp_tbl[BNXT_ROCE_ULP].msix_base; + num_msix = edev->ulp_tbl->msix_requested; + idx = edev->ulp_tbl->msix_base; for (i = 0; i < num_msix; i++) { ent[i].vector = bp->irq_tbl[idx + i].vector; ent[i].ring_idx = idx + i; @@ -121,8 +101,9 @@ static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) } } -static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, - struct bnxt_msix_entry *ent, int num_msix) +static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, + struct bnxt_msix_entry *ent, + int num_msix) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); @@ -132,13 +113,10 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, int total_vecs; int rc = 0; - if (ulp_id != BNXT_ROCE_ULP) - return -EINVAL; - if (!(bp->flags & BNXT_FLAG_USING_MSIX)) return -ENODEV; - if (edev->ulp_tbl[ulp_id].msix_requested) + if (edev->ulp_tbl->msix_requested) return -EAGAIN; max_cp_rings = bnxt_get_max_func_cp_rings(bp); @@ -155,8 +133,8 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, idx = max_idx - avail_msix; } - edev->ulp_tbl[ulp_id].msix_base = idx; - edev->ulp_tbl[ulp_id].msix_requested = avail_msix; + edev->ulp_tbl->msix_base = idx; + edev->ulp_tbl->msix_requested = avail_msix; hw_resc = &bp->hw_resc; total_vecs = idx + avail_msix; if (bp->total_irqs < total_vecs || @@ -171,7 +149,7 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, } } if (rc) { - edev->ulp_tbl[ulp_id].msix_requested = 0; + edev->ulp_tbl->msix_requested = 0; return -EAGAIN; } @@ -180,25 +158,22 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id, resv_msix = hw_resc->resv_irqs - bp->cp_nr_rings; avail_msix = min_t(int, resv_msix, avail_msix); - edev->ulp_tbl[ulp_id].msix_requested = avail_msix; + edev->ulp_tbl->msix_requested = avail_msix; } bnxt_fill_msix_vecs(bp, ent); edev->flags |= BNXT_EN_FLAG_MSIX_REQUESTED; return avail_msix; } -static int bnxt_free_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id) +static void bnxt_free_msix_vecs(struct bnxt_en_dev *edev) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); - if (ulp_id != BNXT_ROCE_ULP) - return -EINVAL; - if (!(edev->flags & BNXT_EN_FLAG_MSIX_REQUESTED)) - return 0; + return; - edev->ulp_tbl[ulp_id].msix_requested = 0; + edev->ulp_tbl->msix_requested = 0; edev->flags &= ~BNXT_EN_FLAG_MSIX_REQUESTED; if (netif_running(dev) && !(edev->flags & BNXT_EN_FLAG_ULP_STOPPED)) { rtnl_lock(); @@ -207,43 +182,43 @@ static int bnxt_free_msix_vecs(struct bnxt_en_dev *edev, unsigned int ulp_id) rtnl_unlock(); } - return 0; + return; } int bnxt_get_ulp_msix_num(struct bnxt *bp) { - if (bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) { + if (bnxt_ulp_registered(bp->edev)) { struct bnxt_en_dev *edev = bp->edev; - return edev->ulp_tbl[BNXT_ROCE_ULP].msix_requested; + return edev->ulp_tbl->msix_requested; } return 0; } int bnxt_get_ulp_msix_base(struct bnxt *bp) { - if (bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) { + if (bnxt_ulp_registered(bp->edev)) { struct bnxt_en_dev *edev = bp->edev; - if (edev->ulp_tbl[BNXT_ROCE_ULP].msix_requested) - return edev->ulp_tbl[BNXT_ROCE_ULP].msix_base; + if (edev->ulp_tbl->msix_requested) + return edev->ulp_tbl->msix_base; } return 0; } int bnxt_get_ulp_stat_ctxs(struct bnxt *bp) { - if (bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) { + if (bnxt_ulp_registered(bp->edev)) { struct bnxt_en_dev *edev = bp->edev; - if (edev->ulp_tbl[BNXT_ROCE_ULP].msix_requested) + if (edev->ulp_tbl->msix_requested) return BNXT_MIN_ROCE_STAT_CTXS; } return 0; } -static int bnxt_send_msg(struct bnxt_en_dev *edev, unsigned int ulp_id, +static int bnxt_send_msg(struct bnxt_en_dev *edev, struct bnxt_fw_msg *fw_msg) { struct net_device *dev = edev->net; @@ -253,7 +228,7 @@ static int bnxt_send_msg(struct bnxt_en_dev *edev, unsigned int ulp_id, u32 resp_len; int rc; - if (ulp_id != BNXT_ROCE_ULP && bp->fw_reset_state) + if (bp->fw_reset_state) return -EBUSY; rc = hwrm_req_init(bp, req, 0 /* don't care */); @@ -292,27 +267,24 @@ void bnxt_ulp_stop(struct bnxt *bp) { struct bnxt_en_dev *edev = bp->edev; struct bnxt_ulp_ops *ops; - int i; + struct bnxt_ulp *ulp; if (!edev) return; edev->flags |= BNXT_EN_FLAG_ULP_STOPPED; - for (i = 0; i < BNXT_MAX_ULP; i++) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[i]; - - ops = rtnl_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_stop) - continue; - ops->ulp_stop(ulp->handle); - } + ulp = edev->ulp_tbl; + ops = rtnl_dereference(ulp->ulp_ops); + if (!ops || !ops->ulp_stop) + return; + ops->ulp_stop(ulp->handle); } void bnxt_ulp_start(struct bnxt *bp, int err) { struct bnxt_en_dev *edev = bp->edev; struct bnxt_ulp_ops *ops; - int i; + struct bnxt_ulp *ulp; if (!edev) return; @@ -322,39 +294,32 @@ void bnxt_ulp_start(struct bnxt *bp, int err) if (err) return; - for (i = 0; i < BNXT_MAX_ULP; i++) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[i]; - - ops = rtnl_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_start) - continue; - ops->ulp_start(ulp->handle); - } + ulp = edev->ulp_tbl; + ops = rtnl_dereference(ulp->ulp_ops); + if (!ops || !ops->ulp_start) + return; + ops->ulp_start(ulp->handle); } void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs) { struct bnxt_en_dev *edev = bp->edev; struct bnxt_ulp_ops *ops; - int i; + struct bnxt_ulp *ulp; if (!edev) return; - for (i = 0; i < BNXT_MAX_ULP; i++) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[i]; - - rcu_read_lock(); - ops = rcu_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_sriov_config) { - rcu_read_unlock(); - continue; - } - bnxt_ulp_get(ulp); + rcu_read_lock(); + ops = rcu_dereference(ulp->ulp_ops); + if (!ops || !ops->ulp_sriov_config) { rcu_read_unlock(); - ops->ulp_sriov_config(ulp->handle, num_vfs); - bnxt_ulp_put(ulp); + return; } + bnxt_ulp_get(ulp); + rcu_read_unlock(); + ops->ulp_sriov_config(ulp->handle, num_vfs); + bnxt_ulp_put(ulp); } void bnxt_ulp_irq_stop(struct bnxt *bp) @@ -365,8 +330,8 @@ void bnxt_ulp_irq_stop(struct bnxt *bp) if (!edev || !(edev->flags & BNXT_EN_FLAG_MSIX_REQUESTED)) return; - if (bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[BNXT_ROCE_ULP]; + if (bnxt_ulp_registered(bp->edev)) { + struct bnxt_ulp *ulp = edev->ulp_tbl; if (!ulp->msix_requested) return; @@ -386,8 +351,8 @@ void bnxt_ulp_irq_restart(struct bnxt *bp, int err) if (!edev || !(edev->flags & BNXT_EN_FLAG_MSIX_REQUESTED)) return; - if (bnxt_ulp_registered(bp->edev, BNXT_ROCE_ULP)) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[BNXT_ROCE_ULP]; + if (bnxt_ulp_registered(bp->edev)) { + struct bnxt_ulp *ulp = edev->ulp_tbl; struct bnxt_msix_entry *ent = NULL; if (!ulp->msix_requested) @@ -414,41 +379,36 @@ void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl) u16 event_id = le16_to_cpu(cmpl->event_id); struct bnxt_en_dev *edev = bp->edev; struct bnxt_ulp_ops *ops; - int i; + struct bnxt_ulp *ulp; if (!edev) return; rcu_read_lock(); - for (i = 0; i < BNXT_MAX_ULP; i++) { - struct bnxt_ulp *ulp = &edev->ulp_tbl[i]; - - ops = rcu_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_async_notifier) - continue; - if (!ulp->async_events_bmap || - event_id > ulp->max_async_event_id) - continue; - - /* Read max_async_event_id first before testing the bitmap. */ - smp_rmb(); - if (test_bit(event_id, ulp->async_events_bmap)) - ops->ulp_async_notifier(ulp->handle, cmpl); - } + ulp = edev->ulp_tbl; + + ops = rcu_dereference(ulp->ulp_ops); + if (!ops || !ops->ulp_async_notifier) + return; + if (!ulp->async_events_bmap || event_id > ulp->max_async_event_id) + return; + + /* Read max_async_event_id first before testing the bitmap. */ + smp_rmb(); + if (test_bit(event_id, ulp->async_events_bmap)) + ops->ulp_async_notifier(ulp->handle, cmpl); rcu_read_unlock(); } -static int bnxt_register_async_events(struct bnxt_en_dev *edev, unsigned int ulp_id, - unsigned long *events_bmap, u16 max_id) +static int bnxt_register_async_events(struct bnxt_en_dev *edev, + unsigned long *events_bmap, + u16 max_id) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); struct bnxt_ulp *ulp; - if (ulp_id >= BNXT_MAX_ULP) - return -EINVAL; - - ulp = &edev->ulp_tbl[ulp_id]; + ulp = edev->ulp_tbl; ulp->async_events_bmap = events_bmap; /* Make sure bnxt_ulp_async_events() sees this order */ smp_wmb(); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index 6cac4c0b708b..357f8fde5d61 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -66,7 +66,7 @@ struct bnxt_en_dev { #define BNXT_EN_FLAG_MSIX_REQUESTED 0x4 #define BNXT_EN_FLAG_ULP_STOPPED 0x8 const struct bnxt_en_ops *en_ops; - struct bnxt_ulp ulp_tbl[BNXT_MAX_ULP]; + struct bnxt_ulp *ulp_tbl; int l2_db_size; /* Doorbell BAR size in * bytes mapped by L2 * driver. @@ -78,21 +78,21 @@ struct bnxt_en_dev { }; struct bnxt_en_ops { - int (*bnxt_register_device)(struct bnxt_en_dev *, unsigned int, - struct bnxt_ulp_ops *, void *); - int (*bnxt_unregister_device)(struct bnxt_en_dev *, unsigned int); - int (*bnxt_request_msix)(struct bnxt_en_dev *, unsigned int, - struct bnxt_msix_entry *, int); - int (*bnxt_free_msix)(struct bnxt_en_dev *, unsigned int); - int (*bnxt_send_fw_msg)(struct bnxt_en_dev *, unsigned int, - struct bnxt_fw_msg *); - int (*bnxt_register_fw_async_events)(struct bnxt_en_dev *, unsigned int, - unsigned long *, u16); + int (*bnxt_register_device)(struct bnxt_en_dev *edev, + struct bnxt_ulp_ops *ulp_ops, void *handle); + int (*bnxt_unregister_device)(struct bnxt_en_dev *edev); + int (*bnxt_request_msix)(struct bnxt_en_dev *edev, + struct bnxt_msix_entry *ent, int num_msix); + void (*bnxt_free_msix)(struct bnxt_en_dev *edev); + int (*bnxt_send_fw_msg)(struct bnxt_en_dev *edev, + struct bnxt_fw_msg *fw_msg); + int (*bnxt_register_fw_async_events)(struct bnxt_en_dev *edev, + unsigned long *events_bmap, u16 max_id); }; -static inline bool bnxt_ulp_registered(struct bnxt_en_dev *edev, int ulp_id) +static inline bool bnxt_ulp_registered(struct bnxt_en_dev *edev) { - if (edev && rcu_access_pointer(edev->ulp_tbl[ulp_id].ulp_ops)) + if (edev && edev->ulp_tbl) return true; return false; } From patchwork Tue Oct 25 17:31:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 10900 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1131197wru; Tue, 25 Oct 2022 10:35:57 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4TJsS+lVAIYWGZR5tnFNPzvOJMsHf/9825obg9Kh742jNPHVGDYND1NkIY4QDiJaKmoBbL X-Received: by 2002:a17:902:e28b:b0:186:ada2:8173 with SMTP id o11-20020a170902e28b00b00186ada28173mr10142331plc.138.1666719357043; Tue, 25 Oct 2022 10:35:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666719357; cv=none; d=google.com; s=arc-20160816; b=ylvhBz6/y5TuI8+1Cf523eTkD1qUyP01+DGMLYHdqFcrjAyfXaVfY0Q7y9z84JQ4KP GXog0aX7MfC18cuE/KRrVBvYsyb9NUckdd/WhFDvhsnwUeCBdy4QDX0Voz9dCGN06hC+ 4kSJKDt2zGbmZiLG0LNWTy8iq5EEx/4aEKUCf2l/rBAhs98+z43uJLq6VVQTCeFpiyZB DwMN0DHiPmPQeyOVlxQKebLhkvSV+Inl2uQMptuzzoIeyY/MGdJn24eVZS8TUvkGmzXh NREqbJ5htChF8cicKLChlH2IoMuUCWtB3mhT47xppXdgROEakNuLLV0FWAYPB97nZ2NC tlEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=pZOz4I5bzzTryB4gE5IARtRitb9b1Eip10FuQeNIaDE=; b=TAYtwqxyirE1Bb5rKeHtMq9dsYF39ds/tPom54liZD0pQ2zlqRmJbu7LYSRFfducFW FyiZnpJznyuIY4vUt8h/nEj5MRNenrpRPNdnYKF0PjkAGppV7AAnQrEibtKpUjHS6dre gUEJt1GNxwXBhgbaMCzKefZR6bRZYnoAbCS2+CpMaQ5ys0WkMXSVt+saqyeT0guMTuqV iuw9k8GGlh65omQ1w7/3K+RlvjvOab/9d9N1C9nZakP494tvKmzGyTZQESPo/RAME2XS zINHqHNs9DRvy9ioo/2/M3nJcL2WjIXjMZ5nsKBx+JahqVXVtUhYbPP9WPlTHcURJzLY gcmQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=f44TZPVH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d4-20020a170902b70400b0018552c4f4c3si3273420pls.615.2022.10.25.10.35.42; Tue, 25 Oct 2022 10:35:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=f44TZPVH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232568AbiJYRbo (ORCPT + 99 others); Tue, 25 Oct 2022 13:31:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46516 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232537AbiJYRbh (ORCPT ); Tue, 25 Oct 2022 13:31:37 -0400 Received: from mail-qk1-x72f.google.com (mail-qk1-x72f.google.com [IPv6:2607:f8b0:4864:20::72f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C4D5B17252C for ; Tue, 25 Oct 2022 10:31:31 -0700 (PDT) Received: by mail-qk1-x72f.google.com with SMTP id o2so8571816qkk.10 for ; Tue, 25 Oct 2022 10:31:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=pZOz4I5bzzTryB4gE5IARtRitb9b1Eip10FuQeNIaDE=; b=f44TZPVHQXOQn4AMX5aNUxks0VG2lNTCKhTHt6DLTM+P2QOnshQ68Gj3XDsYnSd7gY 7hE/Q7N6/4YjBOu1ZhCM2ae3ng2SyulxDHSrqiJYY9eryJKxaLKfwdAYSG73yezT/TgZ e9pvxLVhLISTN86q9ZZ4IPPt3JsZwvrRkaNko= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pZOz4I5bzzTryB4gE5IARtRitb9b1Eip10FuQeNIaDE=; b=6kImLF2/8xtyUpMmYA7VTF6aLLB5/GWXtv4tVejKmuaUviiOJHLehCzKwU9opM50Tg Qsh4o2mx7w+dSVIiuoDvAwIMHyBnp/EjrxRFK6aHTwm0gfosSsPbZ5rjXnvkbrLJsYIV HdOCvTUJsG4JEj9O1VKb41yMtNgkA0j5JsIySjGppGr22mHR1ayKhcYFwLnQzbGDmAya 054V2V2LqBVDWcEMG4pGejPxa+MfQuoVLmEdib+gTXwvslnhFZcYmfCzOwTPiWLT8Ocv rhO+WhBjGTXGZJXk4jOhNwfhP48SW7wNjmZgRVGGGV7HjjYnUUAHdxG82MSIXdUsqsom iB4A== X-Gm-Message-State: ACrzQf0G8teUq+LAJIJWY5iczrtlabG9FmgPh5+sbBQH/3j9+HHhn/2g f+wN9yCYIkoB4fCis92Y8PCAWQ== X-Received: by 2002:a05:620a:28ce:b0:6cf:933c:40d3 with SMTP id l14-20020a05620a28ce00b006cf933c40d3mr26816428qkp.258.1666719091295; Tue, 25 Oct 2022 10:31:31 -0700 (PDT) Received: from C02GC2QQMD6T.wifi.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id u6-20020a37ab06000000b006eed094dcdasm2329034qke.70.2022.10.25.10.31.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Oct 2022 10:31:30 -0700 (PDT) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, Leon Romanovsky Subject: [PATCH v2 4/6] bnxt_en: Use direct API instead of indirection Date: Tue, 25 Oct 2022 10:31:08 -0700 Message-Id: <20221025173110.33192-5-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.0 (Apple Git-136) In-Reply-To: <20221025173110.33192-1-ajit.khaparde@broadcom.com> References: <20220724231458.93830-1-ajit.khaparde@broadcom.com> <20221025173110.33192-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747681916619387175?= X-GMAIL-MSGID: =?utf-8?q?1747681916619387175?= For a single ULP user there is no need for complicating function indirection calls. Remove all this complexity in favour of direct function calls exported by the bnxt_en driver. This allows to simplify the code greatly. Suggested-by: Leon Romanovsky Signed-off-by: Ajit Khaparde --- drivers/infiniband/hw/bnxt_re/main.c | 71 +++++-------------- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 64 +++++------------ drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 24 +++---- 3 files changed, 43 insertions(+), 116 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 755821c68020..67872e9bfd63 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -353,23 +353,6 @@ static struct bnxt_ulp_ops bnxt_re_ulp_ops = { /* RoCE -> Net driver */ -/* Driver registration routines used to let the networking driver (bnxt_en) - * to know that the RoCE driver is now installed - */ -static int bnxt_re_unregister_netdev(struct bnxt_re_dev *rdev) -{ - struct bnxt_en_dev *en_dev; - int rc; - - if (!rdev) - return -EINVAL; - - en_dev = rdev->en_dev; - - rc = en_dev->en_ops->bnxt_unregister_device(rdev->en_dev); - return rc; -} - static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) { struct bnxt_en_dev *en_dev; @@ -380,26 +363,12 @@ static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) en_dev = rdev->en_dev; - rc = en_dev->en_ops->bnxt_register_device(en_dev, - &bnxt_re_ulp_ops, rdev); - rdev->qplib_res.pdev = rdev->en_dev->pdev; + rc = bnxt_register_dev(en_dev, &bnxt_re_ulp_ops, rdev); + if (!rc) + rdev->qplib_res.pdev = rdev->en_dev->pdev; return rc; } -static int bnxt_re_free_msix(struct bnxt_re_dev *rdev) -{ - struct bnxt_en_dev *en_dev; - - if (!rdev) - return -EINVAL; - - en_dev = rdev->en_dev; - - en_dev->en_ops->bnxt_free_msix(rdev->en_dev); - - return 0; -} - static int bnxt_re_request_msix(struct bnxt_re_dev *rdev) { int rc = 0, num_msix_want = BNXT_RE_MAX_MSIX, num_msix_got; @@ -412,9 +381,9 @@ static int bnxt_re_request_msix(struct bnxt_re_dev *rdev) num_msix_want = min_t(u32, BNXT_RE_MAX_MSIX, num_online_cpus()); - num_msix_got = en_dev->en_ops->bnxt_request_msix(en_dev, - rdev->msix_entries, - num_msix_want); + num_msix_got = bnxt_req_msix_vecs(en_dev, + rdev->msix_entries, + num_msix_want); if (num_msix_got < BNXT_RE_MIN_MSIX) { rc = -EINVAL; goto done; @@ -475,7 +444,7 @@ static int bnxt_re_net_ring_free(struct bnxt_re_dev *rdev, req.ring_id = cpu_to_le16(fw_ring_id); bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (rc) ibdev_err(&rdev->ibdev, "Failed to free HW ring:%d :%#x", req.ring_id, rc); @@ -512,7 +481,7 @@ static int bnxt_re_net_ring_alloc(struct bnxt_re_dev *rdev, req.int_mode = ring_attr->mode; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (!rc) *fw_ring_id = le16_to_cpu(resp.ring_id); @@ -540,7 +509,7 @@ static int bnxt_re_net_stats_ctx_free(struct bnxt_re_dev *rdev, req.stat_ctx_id = cpu_to_le32(fw_stats_ctx_id); bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (rc) ibdev_err(&rdev->ibdev, "Failed to free HW stats context %#x", rc); @@ -573,7 +542,7 @@ static int bnxt_re_net_stats_ctx_alloc(struct bnxt_re_dev *rdev, req.stat_ctx_flags = STAT_CTX_ALLOC_REQ_STAT_CTX_FLAGS_ROCE; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (!rc) *fw_stats_ctx_id = le32_to_cpu(resp.stat_ctx_id); @@ -1079,7 +1048,7 @@ static int bnxt_re_query_hwrm_pri2cos(struct bnxt_re_dev *rdev, u8 dir, bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (rc) return rc; @@ -1265,7 +1234,7 @@ static void bnxt_re_query_hwrm_intf_version(struct bnxt_re_dev *rdev) req.hwrm_intf_upd = HWRM_VERSION_UPDATE; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - rc = en_dev->en_ops->bnxt_send_fw_msg(en_dev, &fw_msg); + rc = bnxt_send_msg(en_dev, &fw_msg); if (rc) { ibdev_err(&rdev->ibdev, "Failed to query HW version, rc = 0x%x", rc); @@ -1328,20 +1297,12 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev) bnxt_re_net_ring_free(rdev, rdev->rcfw.creq.ring_id, type); bnxt_qplib_free_rcfw_channel(&rdev->rcfw); } - if (test_and_clear_bit(BNXT_RE_FLAG_GOT_MSIX, &rdev->flags)) { - rc = bnxt_re_free_msix(rdev); - if (rc) - ibdev_warn(&rdev->ibdev, - "Failed to free MSI-X vectors: %#x", rc); - } + if (test_and_clear_bit(BNXT_RE_FLAG_GOT_MSIX, &rdev->flags)) + bnxt_free_msix_vecs(rdev->en_dev); bnxt_re_destroy_chip_ctx(rdev); - if (test_and_clear_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags)) { - rc = bnxt_re_unregister_netdev(rdev); - if (rc) - ibdev_warn(&rdev->ibdev, - "Failed to unregister with netdev: %#x", rc); - } + if (test_and_clear_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags)) + bnxt_unregister_dev(rdev->en_dev); } /* worker thread for polling periodic events. Now used for QoS programming*/ diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index 5fa653c2458a..dbb1fa086013 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -28,9 +28,9 @@ static DEFINE_IDA(bnxt_aux_dev_ids); -static int bnxt_register_dev(struct bnxt_en_dev *edev, - struct bnxt_ulp_ops *ulp_ops, - void *handle) +int bnxt_register_dev(struct bnxt_en_dev *edev, + struct bnxt_ulp_ops *ulp_ops, + void *handle) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); @@ -55,8 +55,9 @@ static int bnxt_register_dev(struct bnxt_en_dev *edev, return 0; } +EXPORT_SYMBOL(bnxt_register_dev); -static int bnxt_unregister_dev(struct bnxt_en_dev *edev) +void bnxt_unregister_dev(struct bnxt_en_dev *edev) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); @@ -65,7 +66,7 @@ static int bnxt_unregister_dev(struct bnxt_en_dev *edev) ulp = edev->ulp_tbl; if (ulp->msix_requested) - edev->en_ops->bnxt_free_msix(edev); + bnxt_free_msix_vecs(edev); if (ulp->max_async_event_id) bnxt_hwrm_func_drv_rgtr(bp, NULL, 0, true); @@ -78,8 +79,9 @@ static int bnxt_unregister_dev(struct bnxt_en_dev *edev) } kfree(ulp); edev->ulp_tbl = NULL; - return 0; + return; } +EXPORT_SYMBOL(bnxt_unregister_dev); static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) { @@ -101,7 +103,7 @@ static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) } } -static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, +int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, struct bnxt_msix_entry *ent, int num_msix) { @@ -164,8 +166,9 @@ static int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, edev->flags |= BNXT_EN_FLAG_MSIX_REQUESTED; return avail_msix; } +EXPORT_SYMBOL(bnxt_req_msix_vecs); -static void bnxt_free_msix_vecs(struct bnxt_en_dev *edev) +void bnxt_free_msix_vecs(struct bnxt_en_dev *edev) { struct net_device *dev = edev->net; struct bnxt *bp = netdev_priv(dev); @@ -184,6 +187,7 @@ static void bnxt_free_msix_vecs(struct bnxt_en_dev *edev) return; } +EXPORT_SYMBOL(bnxt_free_msix_vecs); int bnxt_get_ulp_msix_num(struct bnxt *bp) { @@ -218,7 +222,7 @@ int bnxt_get_ulp_stat_ctxs(struct bnxt *bp) return 0; } -static int bnxt_send_msg(struct bnxt_en_dev *edev, +int bnxt_send_msg(struct bnxt_en_dev *edev, struct bnxt_fw_msg *fw_msg) { struct net_device *dev = edev->net; @@ -252,6 +256,7 @@ static int bnxt_send_msg(struct bnxt_en_dev *edev, hwrm_req_drop(bp, req); return rc; } +EXPORT_SYMBOL(bnxt_send_msg); static void bnxt_ulp_get(struct bnxt_ulp *ulp) { @@ -310,14 +315,11 @@ void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs) if (!edev) return; - rcu_read_lock(); - ops = rcu_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_sriov_config) { - rcu_read_unlock(); + ops = rtnl_dereference(ulp->ulp_ops); + if (!ops || !ops->ulp_sriov_config) return; - } + bnxt_ulp_get(ulp); - rcu_read_unlock(); ops->ulp_sriov_config(ulp->handle, num_vfs); bnxt_ulp_put(ulp); } @@ -384,10 +386,9 @@ void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl) if (!edev) return; - rcu_read_lock(); ulp = edev->ulp_tbl; - ops = rcu_dereference(ulp->ulp_ops); + ops = rtnl_dereference(ulp->ulp_ops); if (!ops || !ops->ulp_async_notifier) return; if (!ulp->async_events_bmap || event_id > ulp->max_async_event_id) @@ -397,35 +398,8 @@ void bnxt_ulp_async_events(struct bnxt *bp, struct hwrm_async_event_cmpl *cmpl) smp_rmb(); if (test_bit(event_id, ulp->async_events_bmap)) ops->ulp_async_notifier(ulp->handle, cmpl); - rcu_read_unlock(); -} - -static int bnxt_register_async_events(struct bnxt_en_dev *edev, - unsigned long *events_bmap, - u16 max_id) -{ - struct net_device *dev = edev->net; - struct bnxt *bp = netdev_priv(dev); - struct bnxt_ulp *ulp; - - ulp = edev->ulp_tbl; - ulp->async_events_bmap = events_bmap; - /* Make sure bnxt_ulp_async_events() sees this order */ - smp_wmb(); - ulp->max_async_event_id = max_id; - bnxt_hwrm_func_drv_rgtr(bp, events_bmap, max_id + 1, true); - return 0; } -static const struct bnxt_en_ops bnxt_en_ops_tbl = { - .bnxt_register_device = bnxt_register_dev, - .bnxt_unregister_device = bnxt_unregister_dev, - .bnxt_request_msix = bnxt_req_msix_vecs, - .bnxt_free_msix = bnxt_free_msix_vecs, - .bnxt_send_fw_msg = bnxt_send_msg, - .bnxt_register_fw_async_events = bnxt_register_async_events, -}; - void bnxt_aux_dev_free(struct bnxt *bp) { kfree(bp->aux_dev); @@ -494,7 +468,6 @@ void bnxt_aux_dev_release(struct device *dev) container_of(dev, struct bnxt_aux_dev, aux_dev.dev); struct bnxt *bp = netdev_priv(bnxt_adev->edev->net); - bnxt_adev->edev->en_ops = NULL; kfree(bnxt_adev->edev); bnxt_adev->edev = NULL; bp->edev = NULL; @@ -502,7 +475,6 @@ void bnxt_aux_dev_release(struct device *dev) static inline void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp) { - edev->en_ops = &bnxt_en_ops_tbl; edev->net = bp->dev; edev->pdev = bp->pdev; edev->l2_db_size = bp->db_size; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index 357f8fde5d61..42e324635e6b 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -47,7 +47,7 @@ struct bnxt_fw_msg { struct bnxt_ulp { void *handle; - struct bnxt_ulp_ops __rcu *ulp_ops; + struct bnxt_ulp_ops *ulp_ops; unsigned long *async_events_bmap; u16 max_async_event_id; u16 msix_requested; @@ -65,7 +65,6 @@ struct bnxt_en_dev { BNXT_EN_FLAG_ROCEV2_CAP) #define BNXT_EN_FLAG_MSIX_REQUESTED 0x4 #define BNXT_EN_FLAG_ULP_STOPPED 0x8 - const struct bnxt_en_ops *en_ops; struct bnxt_ulp *ulp_tbl; int l2_db_size; /* Doorbell BAR size in * bytes mapped by L2 @@ -77,19 +76,6 @@ struct bnxt_en_dev { */ }; -struct bnxt_en_ops { - int (*bnxt_register_device)(struct bnxt_en_dev *edev, - struct bnxt_ulp_ops *ulp_ops, void *handle); - int (*bnxt_unregister_device)(struct bnxt_en_dev *edev); - int (*bnxt_request_msix)(struct bnxt_en_dev *edev, - struct bnxt_msix_entry *ent, int num_msix); - void (*bnxt_free_msix)(struct bnxt_en_dev *edev); - int (*bnxt_send_fw_msg)(struct bnxt_en_dev *edev, - struct bnxt_fw_msg *fw_msg); - int (*bnxt_register_fw_async_events)(struct bnxt_en_dev *edev, - unsigned long *events_bmap, u16 max_id); -}; - static inline bool bnxt_ulp_registered(struct bnxt_en_dev *edev) { if (edev && edev->ulp_tbl) @@ -112,4 +98,12 @@ int bnxt_rdma_aux_device_add(struct bnxt *bp); void bnxt_rdma_aux_device_uninit(struct bnxt_aux_dev *bnxt_adev); void bnxt_rdma_aux_device_init(struct bnxt *bp); void bnxt_aux_dev_free(struct bnxt *bp); + +int bnxt_register_dev(struct bnxt_en_dev *edev, struct bnxt_ulp_ops *ulp_ops, + void *handle); +void bnxt_unregister_dev(struct bnxt_en_dev *edev); +int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, struct bnxt_msix_entry *ent, + int num_msix); +void bnxt_free_msix_vecs(struct bnxt_en_dev *edev); +int bnxt_send_msg(struct bnxt_en_dev *edev, struct bnxt_fw_msg *fw_msg); #endif From patchwork Tue Oct 25 17:31:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 10904 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1131637wru; Tue, 25 Oct 2022 10:36:56 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6piSqg9sOfFkRB+rqcCPPvbsCyV5Aj/Q6vJKLuY+Q1rGTSOtK9FggeA6vW5Jj1Ty/nf+LQ X-Received: by 2002:a17:906:dc93:b0:791:95dd:aaf6 with SMTP id cs19-20020a170906dc9300b0079195ddaaf6mr32979571ejc.456.1666719416453; Tue, 25 Oct 2022 10:36:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666719416; cv=none; d=google.com; s=arc-20160816; b=KGQG93xKx5vxEhXXt7Fh+EWE/Pl7fC21dvj1srovwHTkuDyYgNEcH/HDJeWMsnNdqs donH3DirIhhr2M3I77+e4z7lkKcNy81E0cHsj7Lfvza5S8ZLUb1DfxSGCSIooy770eCd rqqVR4XBZg8wVfox+L9RAXxl9PIyzoxsoJbuCTTTVIr3QDQwlftiMcZrEAaIbUyCbzgP aOX00B849LtAU2NKdBCEn71/Ac6cLcxDuiEPtbKLqcUKRQupcaBZ/QxB8aKXd+A/md/Q Jg3M/nAWMV4NlHazWrzP7wMQNpG1dXY8JoGI8YxrWDILq3rAhdbSAqfUeSakkW3r8zEl 3TXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=P86jBD1/iqvr2jOM/fkhFoJvrfoiFv8AV0hb+zowJpM=; b=nqKlvpzKqqsper1RNtYDgKpv06TWILGXuBpXad84kXnIh6PPIPZwzZfzwBJjaVk3YC f6oTepYI6ykRDkxHWCfon3BT1tjtUx0dYcZ0OeUvpkYuC9Che9FoByYZgGUgjScA40uQ TyjukNO/DaSDZ0zNkQth3UeFnGj86ASTCC+/XtYOVPCXM3ZkUsGLi/o0oNafl/Er/87V hAjBT6Bpu9r9iBYSyfuxrb30W+d9MNkoZhpvY/IEXQkRCbYlMVL/+P3I//s8XsHv7Ppo H3ZnvyCBNc8ENuf9MyO/lQwZsYpSzl7gP7KIoAhOmAVHOVrHN7xyFnzMz8Pj5lz6nNDY mc/g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=T4MRZ6ci; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 11-20020a170906310b00b0078dad42f75dsi2983550ejx.475.2022.10.25.10.36.31; Tue, 25 Oct 2022 10:36:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=T4MRZ6ci; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232539AbiJYRbv (ORCPT + 99 others); Tue, 25 Oct 2022 13:31:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46242 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232615AbiJYRbi (ORCPT ); Tue, 25 Oct 2022 13:31:38 -0400 Received: from mail-qk1-x72c.google.com (mail-qk1-x72c.google.com [IPv6:2607:f8b0:4864:20::72c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 477C9172B64 for ; Tue, 25 Oct 2022 10:31:33 -0700 (PDT) Received: by mail-qk1-x72c.google.com with SMTP id m6so8591667qkm.4 for ; Tue, 25 Oct 2022 10:31:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=P86jBD1/iqvr2jOM/fkhFoJvrfoiFv8AV0hb+zowJpM=; b=T4MRZ6ciW7+Y84RI8pB+bQX+PLT8c0UdDYx24x7ts7j280Q+T5dJ3QX4DELXOOhuPm V1MCBaaF/o8Hc0tdtzaE53QJ585jDdZplNVY1xdUXHhY83itGR2GcmrtEQMTWllikOUS by/l08ibTYCvrjA5NPyFNL39GYKRCE+gYO+jo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=P86jBD1/iqvr2jOM/fkhFoJvrfoiFv8AV0hb+zowJpM=; b=XoU1YRGtQaXthCUWQKxdK7li8aWbzHF/lc3WPZgvkNK+O5V0qELMc2Ild3sCqLcPs9 htvsSB8hX0gMTZhJcWnPcOcr5RTRZJF3eY0eSyDzyi2tDR35caHxZ0zlOiMI8niWBz5N LCh7LzWarcf1xlpaladtI6L+D5TvUF5bjeOTonp1Y4U7Qod1DSDDkNhTFI8zqX/mi9qA /0OZxAkVQkrne1YNlqP+/+IIcD+ibTRYvsMFsrlJ8Cz0GVCIhLJfrbKxJkOjTSF0ixkY EyZAYrQ1FiEzlCGZoktDOhWvyxCNvFyHRDFKXF/S5pKNzKcpZIzqzzfXhx3oQAIXXUws 2x+w== X-Gm-Message-State: ACrzQf02cexxBlBA3WIDk/G9rVMx9h2q5eRpAxqswJuDabvZ1px7iwq5 84IWzh74Zgf2NPZuqbvB5dJEbA== X-Received: by 2002:a05:620a:490d:b0:6e6:b1ad:7a81 with SMTP id ed13-20020a05620a490d00b006e6b1ad7a81mr25897585qkb.695.1666719093554; Tue, 25 Oct 2022 10:31:33 -0700 (PDT) Received: from C02GC2QQMD6T.wifi.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id u6-20020a37ab06000000b006eed094dcdasm2329034qke.70.2022.10.25.10.31.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Oct 2022 10:31:32 -0700 (PDT) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com Subject: [PATCH v2 5/6] bnxt_en: Use auxiliary bus calls over proprietary calls Date: Tue, 25 Oct 2022 10:31:09 -0700 Message-Id: <20221025173110.33192-6-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.0 (Apple Git-136) In-Reply-To: <20221025173110.33192-1-ajit.khaparde@broadcom.com> References: <20220724231458.93830-1-ajit.khaparde@broadcom.com> <20221025173110.33192-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SORTED_RECIPS,SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747681978934579878?= X-GMAIL-MSGID: =?utf-8?q?1747681978934579878?= Wherever possible use the function ops provided by auxiliary bus instead of using proprietary ops. Defined bnxt_re_suspend and bnxt_re_resume calls which can be invoked by the bnxt_en driver instead of the ULP stop/start calls. Signed-off-by: Ajit Khaparde --- drivers/infiniband/hw/bnxt_re/main.c | 102 +++++++++++------- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 40 ++++--- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 2 - 3 files changed, 87 insertions(+), 57 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 67872e9bfd63..1266cfe7ea88 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -226,45 +226,6 @@ static void bnxt_re_set_resource_limits(struct bnxt_re_dev *rdev) bnxt_re_limit_vf_res(&rdev->qplib_ctx, num_vfs); } -/* for handling bnxt_en callbacks later */ -static void bnxt_re_stop(void *p) -{ - struct bnxt_re_dev *rdev = p; - struct bnxt *bp; - - if (!rdev) - return; - - /* L2 driver invokes this callback during device error/crash or device - * reset. Current RoCE driver doesn't recover the device in case of - * error. Handle the error by dispatching fatal events to all qps - * ie. by calling bnxt_re_dev_stop and release the MSIx vectors as - * L2 driver want to modify the MSIx table. - */ - bp = netdev_priv(rdev->netdev); - - ibdev_info(&rdev->ibdev, "Handle device stop call from L2 driver"); - /* Check the current device state from L2 structure and move the - * device to detached state if FW_FATAL_COND is set. - * This prevents more commands to HW during clean-up, - * in case the device is already in error. - */ - if (test_bit(BNXT_STATE_FW_FATAL_COND, &bp->state)) - set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); - - bnxt_re_dev_stop(rdev); - bnxt_re_stop_irq(rdev); - /* Move the device states to detached and avoid sending any more - * commands to HW - */ - set_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags); - set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); -} - -static void bnxt_re_start(void *p) -{ -} - static void bnxt_re_sriov_config(void *p, int num_vfs) { struct bnxt_re_dev *rdev = p; @@ -344,8 +305,6 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) } static struct bnxt_ulp_ops bnxt_re_ulp_ops = { - .ulp_stop = bnxt_re_stop, - .ulp_start = bnxt_re_start, .ulp_sriov_config = bnxt_re_sriov_config, .ulp_irq_stop = bnxt_re_stop_irq, .ulp_irq_restart = bnxt_re_start_irq @@ -1623,6 +1582,65 @@ static int bnxt_re_probe(struct auxiliary_device *adev, return rc; } +int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state) +{ + struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); + struct bnxt *bp; + + if (!rdev) + return 0; + + mutex_lock(&bnxt_re_mutex); + /* L2 driver may invoke this callback during device error/crash or device + * reset. Current RoCE driver doesn't recover the device in case of + * error. Handle the error by dispatching fatal events to all qps + * ie. by calling bnxt_re_dev_stop and release the MSIx vectors as + * L2 driver want to modify the MSIx table. + */ + bp = netdev_priv(rdev->netdev); + + ibdev_info(&rdev->ibdev, "Handle device suspend call"); + /* Check the current device state from L2 structure and move the + * device to detached state if FW_FATAL_COND is set. + * This prevents more commands to HW during clean-up, + * in case the device is already in error. + */ + if (test_bit(BNXT_STATE_FW_FATAL_COND, &bp->state)) + set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); + + bnxt_re_dev_stop(rdev); + bnxt_re_stop_irq(rdev); + /* Move the device states to detached and avoid sending any more + * commands to HW + */ + set_bit(BNXT_RE_FLAG_ERR_DEVICE_DETACHED, &rdev->flags); + set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); + mutex_unlock(&bnxt_re_mutex); + + return 0; +} + +int bnxt_re_resume(struct auxiliary_device *adev) +{ + struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); + + if (!rdev) + return 0; + + mutex_lock(&bnxt_re_mutex); + /* L2 driver may invoke this callback during device recovery, resume. + * reset. Current RoCE driver doesn't recover the device in case of + * error. Handle the error by dispatching fatal events to all qps + * ie. by calling bnxt_re_dev_stop and release the MSIx vectors as + * L2 driver want to modify the MSIx table. + */ + + ibdev_info(&rdev->ibdev, "Handle device resume call"); + mutex_unlock(&bnxt_re_mutex); + + return 0; +} + static const struct auxiliary_device_id bnxt_re_id_table[] = { { .name = BNXT_ADEV_NAME ".rdma", }, {}, @@ -1635,6 +1653,8 @@ static struct auxiliary_driver bnxt_re_driver = { .probe = bnxt_re_probe, .remove = bnxt_re_remove, .shutdown = bnxt_re_shutdown, + .suspend = bnxt_re_suspend, + .resume = bnxt_re_resume, .id_table = bnxt_re_id_table, }; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index dbb1fa086013..ba0c281c74ba 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -270,26 +270,31 @@ static void bnxt_ulp_put(struct bnxt_ulp *ulp) void bnxt_ulp_stop(struct bnxt *bp) { + struct bnxt_aux_dev *bnxt_aux = bp->aux_dev; struct bnxt_en_dev *edev = bp->edev; - struct bnxt_ulp_ops *ops; - struct bnxt_ulp *ulp; if (!edev) return; edev->flags |= BNXT_EN_FLAG_ULP_STOPPED; - ulp = edev->ulp_tbl; - ops = rtnl_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_stop) - return; - ops->ulp_stop(ulp->handle); + if (bnxt_aux) { + struct auxiliary_device *adev; + + adev = &bnxt_aux->aux_dev; + if (adev->dev.driver) { + struct auxiliary_driver *adrv; + pm_message_t pm = {}; + + adrv = to_auxiliary_drv(adev->dev.driver); + adrv->suspend(adev, pm); + } + } } void bnxt_ulp_start(struct bnxt *bp, int err) { + struct bnxt_aux_dev *bnxt_aux = bp->aux_dev; struct bnxt_en_dev *edev = bp->edev; - struct bnxt_ulp_ops *ops; - struct bnxt_ulp *ulp; if (!edev) return; @@ -299,11 +304,18 @@ void bnxt_ulp_start(struct bnxt *bp, int err) if (err) return; - ulp = edev->ulp_tbl; - ops = rtnl_dereference(ulp->ulp_ops); - if (!ops || !ops->ulp_start) - return; - ops->ulp_start(ulp->handle); + if (bnxt_aux) { + struct auxiliary_device *adev; + + adev = &bnxt_aux->aux_dev; + if (adev->dev.driver) { + struct auxiliary_driver *adrv; + + adrv = to_auxiliary_drv(adev->dev.driver); + adrv->resume(adev); + } + } + } void bnxt_ulp_sriov_cfg(struct bnxt *bp, int num_vfs) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index 42e324635e6b..ba253ba5aba0 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -29,8 +29,6 @@ struct bnxt_msix_entry { struct bnxt_ulp_ops { /* async_notifier() cannot sleep (in BH context) */ void (*ulp_async_notifier)(void *, struct hwrm_async_event_cmpl *); - void (*ulp_stop)(void *); - void (*ulp_start)(void *); void (*ulp_sriov_config)(void *, int); void (*ulp_shutdown)(void *); void (*ulp_irq_stop)(void *); From patchwork Tue Oct 25 17:31:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 10902 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1131542wru; Tue, 25 Oct 2022 10:36:45 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7wUKYtVPjjOE/kdQ+ZFDZxGZ38OjXXw2x/8zc7eSPOqWFGp1c6sRHyf1ztvXKc9kmHj0b7 X-Received: by 2002:a63:6c84:0:b0:43c:700f:6218 with SMTP id h126-20020a636c84000000b0043c700f6218mr33316783pgc.420.1666719404630; Tue, 25 Oct 2022 10:36:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666719404; cv=none; d=google.com; s=arc-20160816; b=zy9TIqUtNZxBR/dq+mPE/3FLn+Va81YqsswK+y+KHmXG4KTE50BYT0QV0hzzi9G7nw 2N/vjHX7Bk92N53Yxv6Lb5wFwEBBCy4jVUuDowQd21tFAPwP0Z3gT6ysqapB+Dg+wLjp 0hJegWLC80Karv9pANn+9+0uL/qST4KFCPlxemcgDku8zNvKCEKVM9CEnjI/sWX7x/k9 LIChQy9ke47+clMuk4xWxFEQtAE5gVkUilGHtnIqHZOmqXoX8J92EC2q5BY7OCfSLZBs C7FS4VrkM274jvKkWed2lo8YQMpxfDcSasVydD94yUGi0CRCfAfJyQ4MPlihGrbJJaFt YdBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=e1Hk8Jxi8XJ4trQ43PckznaK08EZ6wlXi4GvXTp/srI=; b=s5qTvhVEjI+6L7yKP40S69knRRasqOnV+wF+9k9vSFTV35oMxyn6dqg8KtMas68NM+ ny8BXUBe8CYg3GCncaItvNyHhwm6qBw/N1hwdT/2tnnwaoWMQE375jfUp8yoOteqSTcS XmVI9dyp9+oPXSGV24tdmfiPBbB/f1oW1AljImAoFlcWssT+VRIEqS7YUaQQXg3hPFJi VRhEPANaLzYwEUp6xueK+Bt+TEq5fHTKDen5OlrOKgna+zHkp3D6JwWcX8jDknfDbn4l lGu6Q1kpoZJ+tnwxiRs6nDmkPAYY4aXBAuWVqHkuNdMYzTfBEd4TCxRDEMQMQYis3KDp X5FA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=b1F6ouYj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k3-20020a170902c40300b00186c3afbd25si2919814plk.349.2022.10.25.10.36.30; Tue, 25 Oct 2022 10:36:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b=b1F6ouYj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232630AbiJYRcm (ORCPT + 99 others); Tue, 25 Oct 2022 13:32:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47948 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232667AbiJYRcM (ORCPT ); Tue, 25 Oct 2022 13:32:12 -0400 Received: from mail-io1-xd33.google.com (mail-io1-xd33.google.com [IPv6:2607:f8b0:4864:20::d33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 009D3175782 for ; Tue, 25 Oct 2022 10:31:46 -0700 (PDT) Received: by mail-io1-xd33.google.com with SMTP id s27so1457462ioa.10 for ; Tue, 25 Oct 2022 10:31:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=e1Hk8Jxi8XJ4trQ43PckznaK08EZ6wlXi4GvXTp/srI=; b=b1F6ouYjcHL7zE2K/jNv3rVFf7HMR9IZ76hpedab+UY2hV1h3+seVgiW1JCgabuX8Y HTZGHNQe6OoQzrtWqBlLl4S8BqNuW2vlCzs2kI5Yy8vB0/jWRWUQ+6HVRyS8cvLLpdyl h03LCxexdLHbqz90jViwzFCmrM7S2gD/X3t/Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=e1Hk8Jxi8XJ4trQ43PckznaK08EZ6wlXi4GvXTp/srI=; b=lesPbJ/5Fulh12ylC1eNGxDAxzu21bTBrLvbbNKEd/mcaHYgiUrV5YEsUZRbecVTmO VGToFb/iHGeE2V5Pcxe65jZt56sY9Gzz76O5FMbKOZjkctUplXyTqNlBQ25SL1wnA5Xe ydG3MbGrKa0LQ502TKoLU3tMeCX+7e7+ELK+KhR8qz4T4owDMUQGbHoBjCPM3PhbUenB HMJSQt7bS7so17lSpHl/zmnYVVQ5UPhSkuJ2NQw2LRHCYpPnHGOfKarFHyb7MGgSzcla poTMhnu30vHSJNU+yL1KbLJMV3OWtkwyy11YieV8KA/VPN80sDykjKchaOqynT3QQ5Qw VTew== X-Gm-Message-State: ACrzQf3nexGownC9gBPy6TeCBuVhd2o9IraYNMVoku8EHgrUQdeKlioK mdAwgUaO54EPfpJWo1VUHecq6u155NTFcA== X-Received: by 2002:a05:622a:44b:b0:39c:f5bf:694d with SMTP id o11-20020a05622a044b00b0039cf5bf694dmr31862147qtx.531.1666719095804; Tue, 25 Oct 2022 10:31:35 -0700 (PDT) Received: from C02GC2QQMD6T.wifi.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id u6-20020a37ab06000000b006eed094dcdasm2329034qke.70.2022.10.25.10.31.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Oct 2022 10:31:35 -0700 (PDT) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, Hongguang Gao Subject: [PATCH v2 6/6] bnxt_en: Remove struct bnxt access from RoCE driver Date: Tue, 25 Oct 2022 10:31:10 -0700 Message-Id: <20221025173110.33192-7-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.0 (Apple Git-136) In-Reply-To: <20221025173110.33192-1-ajit.khaparde@broadcom.com> References: <20220724231458.93830-1-ajit.khaparde@broadcom.com> <20221025173110.33192-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747681966406254728?= X-GMAIL-MSGID: =?utf-8?q?1747681966406254728?= From: Hongguang Gao Decouple RoCE driver from directly accessing L2's private bnxt structure. Move the fields needed by RoCE driver into bnxt_en_dev. They'll be passed to RoCE driver by bnxt_rdma_aux_device_add() function. Signed-off-by: Hongguang Gao Signed-off-by: Ajit Khaparde Reviewed-by: Andy Gospodarek --- drivers/infiniband/hw/bnxt_re/main.c | 22 ++++++------------- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 9 ++++++++ drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 11 ++++++++++ 3 files changed, 27 insertions(+), 15 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 1266cfe7ea88..446025fa0bfb 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -112,16 +112,14 @@ static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev, u8 wqe_mode) { struct bnxt_qplib_chip_ctx *chip_ctx; struct bnxt_en_dev *en_dev; - struct bnxt *bp; en_dev = rdev->en_dev; - bp = netdev_priv(en_dev->net); chip_ctx = kzalloc(sizeof(*chip_ctx), GFP_KERNEL); if (!chip_ctx) return -ENOMEM; - chip_ctx->chip_num = bp->chip_num; - chip_ctx->hw_stats_size = bp->hw_ring_stats_size; + chip_ctx->chip_num = en_dev->chip_num; + chip_ctx->hw_stats_size = en_dev->hw_ring_stats_size; rdev->chip_ctx = chip_ctx; /* rest members to follow eventually */ @@ -129,7 +127,7 @@ static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev, u8 wqe_mode) rdev->qplib_res.cctx = rdev->chip_ctx; rdev->rcfw.res = &rdev->qplib_res; rdev->qplib_res.dattr = &rdev->dev_attr; - rdev->qplib_res.is_vf = BNXT_VF(bp); + rdev->qplib_res.is_vf = BNXT_EN_VF(en_dev); bnxt_re_set_drv_mode(rdev, wqe_mode); if (bnxt_qplib_determine_atomics(en_dev->pdev)) @@ -142,10 +140,7 @@ static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev, u8 wqe_mode) static void bnxt_re_get_sriov_func_type(struct bnxt_re_dev *rdev) { - struct bnxt *bp; - - bp = netdev_priv(rdev->en_dev->net); - if (BNXT_VF(bp)) + if (BNXT_EN_VF(rdev->en_dev)) rdev->is_virtfn = 1; } @@ -986,7 +981,6 @@ static int bnxt_re_query_hwrm_pri2cos(struct bnxt_re_dev *rdev, u8 dir, u64 *cid_map) { struct hwrm_queue_pri2cos_qcfg_input req = {0}; - struct bnxt *bp = netdev_priv(rdev->netdev); struct hwrm_queue_pri2cos_qcfg_output resp; struct bnxt_en_dev *en_dev = rdev->en_dev; struct bnxt_fw_msg fw_msg; @@ -1003,7 +997,7 @@ static int bnxt_re_query_hwrm_pri2cos(struct bnxt_re_dev *rdev, u8 dir, flags |= (dir & 0x01); flags |= HWRM_QUEUE_PRI2COS_QCFG_INPUT_FLAGS_IVLAN; req.flags = cpu_to_le32(flags); - req.port_id = bp->pf.port_id; + req.port_id = en_dev->pf_port_id; bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); @@ -1585,7 +1579,6 @@ static int bnxt_re_probe(struct auxiliary_device *adev, int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state) { struct bnxt_re_dev *rdev = auxiliary_get_drvdata(adev); - struct bnxt *bp; if (!rdev) return 0; @@ -1597,15 +1590,14 @@ int bnxt_re_suspend(struct auxiliary_device *adev, pm_message_t state) * ie. by calling bnxt_re_dev_stop and release the MSIx vectors as * L2 driver want to modify the MSIx table. */ - bp = netdev_priv(rdev->netdev); ibdev_info(&rdev->ibdev, "Handle device suspend call"); - /* Check the current device state from L2 structure and move the + /* Check the current device state from bnxt_en_dev and move the * device to detached state if FW_FATAL_COND is set. * This prevents more commands to HW during clean-up, * in case the device is already in error. */ - if (test_bit(BNXT_STATE_FW_FATAL_COND, &bp->state)) + if (test_bit(BNXT_STATE_FW_FATAL_COND, &rdev->en_dev->en_state)) set_bit(ERR_DEVICE_DETACHED, &rdev->rcfw.cmdq.flags); bnxt_re_dev_stop(rdev); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index ba0c281c74ba..1bbc76037910 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -286,6 +286,7 @@ void bnxt_ulp_stop(struct bnxt *bp) pm_message_t pm = {}; adrv = to_auxiliary_drv(adev->dev.driver); + edev->en_state = bp->state; adrv->suspend(adev, pm); } } @@ -312,6 +313,7 @@ void bnxt_ulp_start(struct bnxt *bp, int err) struct auxiliary_driver *adrv; adrv = to_auxiliary_drv(adev->dev.driver); + edev->en_state = bp->state; adrv->resume(adev); } } @@ -496,6 +498,13 @@ static inline void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp) edev->flags |= BNXT_EN_FLAG_ROCEV1_CAP; if (bp->flags & BNXT_FLAG_ROCEV2_CAP) edev->flags |= BNXT_EN_FLAG_ROCEV2_CAP; + if (bp->flags & BNXT_FLAG_VF) + edev->flags |= BNXT_EN_FLAG_VF; + + edev->chip_num = bp->chip_num; + edev->hw_ring_stats_size = bp->hw_ring_stats_size; + edev->pf_port_id = bp->pf.port_id; + edev->en_state = bp->state; } int bnxt_rdma_aux_device_add(struct bnxt *bp) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index ba253ba5aba0..9a30fd848aaa 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -63,6 +63,9 @@ struct bnxt_en_dev { BNXT_EN_FLAG_ROCEV2_CAP) #define BNXT_EN_FLAG_MSIX_REQUESTED 0x4 #define BNXT_EN_FLAG_ULP_STOPPED 0x8 + #define BNXT_EN_FLAG_VF 0x10 +#define BNXT_EN_VF(edev) ((edev)->flags & BNXT_EN_FLAG_VF) + struct bnxt_ulp *ulp_tbl; int l2_db_size; /* Doorbell BAR size in * bytes mapped by L2 @@ -72,6 +75,14 @@ struct bnxt_en_dev { * bytes mapped as non- * cacheable. */ + u16 chip_num; + u16 hw_ring_stats_size; + u16 pf_port_id; + unsigned long en_state; /* Could be checked in + * RoCE driver suspend + * mode only. Will be + * updated in resume. + */ }; static inline bool bnxt_ulp_registered(struct bnxt_en_dev *edev)