From patchwork Thu Jan 12 20:29:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ajit Khaparde X-Patchwork-Id: 42800 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp4101249wrt; Thu, 12 Jan 2023 12:58:02 -0800 (PST) X-Google-Smtp-Source: AMrXdXsKZOCKyPbJn1s5Lo+g2yqyHCjSTKS0zte9nwF3jvc/QzEM5V07OqeY8HE100W5ZNl7896i X-Received: by 2002:a17:906:b00d:b0:7c1:8ba6:6eb3 with SMTP id v13-20020a170906b00d00b007c18ba66eb3mr70329695ejy.35.1673557082566; Thu, 12 Jan 2023 12:58:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673557082; cv=none; d=google.com; s=arc-20160816; b=e02sounFOVqejeQShZOLRgTJuXvOqTDohkoc7gVx6hnt0OCg98V0Kv2iBKBnm5srJu bQO54IwANf2ZBcXZcRmNyZf27D1bH9npFhhYXhlSy8n0xiy2aUbABS9BH+F/fGoSf0jT mghcS5IXG/uzBC9yk/4NfnKmww8O3mtoNLpwZntaecQr8hfnfJeuhObI1NrlzQ463/wx ynwNXAkMNBL+Pe/WUghn9UyPb14GhWZ9QAOjl03pb2P7MeAiHd5JZCtKE/O6p31WmsPg o01YVdWZSGHrvl0LQxVs6D0ah+7BUPSW/prYrldSu22mjRnKB4Kp0v40MiAewAf2ML4n imLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=Cz/Kh/xlRNvhdQEEbOTclQ8WEFq2J3R5xfLDzuFtjL4=; b=i7C5LARaUYA7GxgqApft07yj0uxPcKNFWZD8hUCZogM4PH74DjKaluwZ5cKJBL6oW/ +++OytKGVWtzZ3aDmQDoftcvxT7Bf+KSoR/6Z1OlBZkNPKigYYBiA/Uv8Mt5Ypmnjg9P eART6rqNwLK9gytlY5kpW2ILUH+GuRCshGMbl6NTzPulOO+lBAEvB98fr/USXKWsEdAE QXLfavZgSL9pjlSwd0RbKHfoM9qPgVlmIZxzbVtDFRRTTyD5by69z9rWwo/DtN/7BbAh ToHZliWmCTEdJDfq9N+JPNicTbbrOF6Ox0pqo89LiUIQiwUok4pr8XJyzdbYa/zbTrvj Pc4Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b="iGGX4Zf/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id xb7-20020a170907070700b007877f3132d9si19255254ejb.438.2023.01.12.12.57.37; Thu, 12 Jan 2023 12:58:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@broadcom.com header.s=google header.b="iGGX4Zf/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240956AbjALU4N (ORCPT + 99 others); Thu, 12 Jan 2023 15:56:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55042 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240399AbjALUxZ (ORCPT ); Thu, 12 Jan 2023 15:53:25 -0500 Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 555F51DF18 for ; Thu, 12 Jan 2023 12:30:00 -0800 (PST) Received: by mail-pj1-x1030.google.com with SMTP id o13so16806830pjg.2 for ; Thu, 12 Jan 2023 12:30:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=Cz/Kh/xlRNvhdQEEbOTclQ8WEFq2J3R5xfLDzuFtjL4=; b=iGGX4Zf/E44O8cVQEnSVhO69IzGr1Sy9+mbU5l+SnpBAdiYe9Kz+LVJ+UBfpn8C2gg N6m2zdDyfe07Cn4/5fHFAP/IE5AvAppOUhYISxP0Fgi3Q+kPtmJAZ/ww9Gc6JbNDjydt Q3/eFUt/8MpPbyZVnUhCwDZB4r4XB6FeZiIOo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Cz/Kh/xlRNvhdQEEbOTclQ8WEFq2J3R5xfLDzuFtjL4=; b=Ki3zHTZCWGvJmJ2q5YA8EFd4DxMpdbEL2kys9ZtLycG9ZBTMHj52mANUXaljZjU2Gm uhxd6gfNcz24otaCPAYVT1dErmW/NWtFga/cB7I+dYuNyrw1a2mv4W4HVceIMClbCk+k RcSL7zQykcco9NzcdMl6dRdopOmvh31/t16WpaQtYvw8LNaSpLUI9gcC9AntpH0JRyo4 7dTtPx7sx1ng8Y+qF0q5NhiZORLtOUTZe5j6lsxBjQ9SFESwEqfx62hImt/MnrOw68tl TT8H9c7MoLUS+iOLfKTZFruPb+V+qZCrMTytQHUQ6dQ4tUXolyKs9kbl0xzQ+NY3zRPO bxdw== X-Gm-Message-State: AFqh2kqj33dNaWsLCM2Uvy4gBughd7RqAmXHn+N+VOvB0l7oJbUE8CXa pAzdQY1x1mz+2j+1GoxTDK8AKA== X-Received: by 2002:a17:903:40d1:b0:194:60c3:48bc with SMTP id t17-20020a17090340d100b0019460c348bcmr3309036pld.27.1673555399467; Thu, 12 Jan 2023 12:29:59 -0800 (PST) Received: from C02GC2QQMD6T.wifi.broadcom.net ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id 129-20020a630887000000b004777c56747csm10283855pgi.11.2023.01.12.12.29.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 12:29:58 -0800 (PST) From: Ajit Khaparde To: ajit.khaparde@broadcom.com Cc: andrew.gospodarek@broadcom.com, davem@davemloft.net, edumazet@google.com, jgg@ziepe.ca, kuba@kernel.org, leon@kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, michael.chan@broadcom.com, netdev@vger.kernel.org, pabeni@redhat.com, selvin.xavier@broadcom.com, Leon Romanovsky Subject: [PATCH net-next v7 8/8] bnxt_en: Remove runtime interrupt vector allocation Date: Thu, 12 Jan 2023 12:29:39 -0800 Message-Id: <20230112202939.19562-9-ajit.khaparde@broadcom.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20230112202939.19562-1-ajit.khaparde@broadcom.com> References: <20230112202939.19562-1-ajit.khaparde@broadcom.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754851791128106569?= X-GMAIL-MSGID: =?utf-8?q?1754851791128106569?= Modified the bnxt_en code to create and pre-configure RDMA devices with the right MSI-X vector count for the ROCE driver to use. This is to align the ROCE driver to the auxiliary device model which will simply bind the driver without getting into PCI-related handling. All PCI-related logic will now be in the bnxt_en driver. Suggested-by: Leon Romanovsky Signed-off-by: Ajit Khaparde Reviewed-by: Leon Romanovsky --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 1 - drivers/infiniband/hw/bnxt_re/main.c | 48 ++--- drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c | 166 ++++++------------ drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h | 6 +- 4 files changed, 64 insertions(+), 157 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index b0465c8d229a..5a2baf49ecaa 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -129,7 +129,6 @@ struct bnxt_re_dev { unsigned int version, major, minor; struct bnxt_qplib_chip_ctx *chip_ctx; struct bnxt_en_dev *en_dev; - struct bnxt_msix_entry msix_entries[BNXT_RE_MAX_MSIX]; int num_msix; int id; diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index b287c3ba9352..8d991fa184ae 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -262,7 +262,7 @@ static void bnxt_re_stop_irq(void *handle) static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) { struct bnxt_re_dev *rdev = (struct bnxt_re_dev *)handle; - struct bnxt_msix_entry *msix_ent = rdev->msix_entries; + struct bnxt_msix_entry *msix_ent = rdev->en_dev->msix_entries; struct bnxt_qplib_rcfw *rcfw = &rdev->rcfw; struct bnxt_qplib_nq *nq; int indx, rc; @@ -281,7 +281,7 @@ static void bnxt_re_start_irq(void *handle, struct bnxt_msix_entry *ent) * in device sctructure. */ for (indx = 0; indx < rdev->num_msix; indx++) - rdev->msix_entries[indx].vector = ent[indx].vector; + rdev->en_dev->msix_entries[indx].vector = ent[indx].vector; bnxt_qplib_rcfw_start_irq(rcfw, msix_ent[BNXT_RE_AEQ_IDX].vector, false); @@ -315,32 +315,6 @@ static int bnxt_re_register_netdev(struct bnxt_re_dev *rdev) return rc; } -static int bnxt_re_request_msix(struct bnxt_re_dev *rdev) -{ - int rc = 0, num_msix_want = BNXT_RE_MAX_MSIX, num_msix_got; - struct bnxt_en_dev *en_dev; - - en_dev = rdev->en_dev; - - num_msix_want = min_t(u32, BNXT_RE_MAX_MSIX, num_online_cpus()); - - num_msix_got = bnxt_req_msix_vecs(en_dev, - rdev->msix_entries, - num_msix_want); - if (num_msix_got < BNXT_RE_MIN_MSIX) { - rc = -EINVAL; - goto done; - } - if (num_msix_got != num_msix_want) { - ibdev_warn(&rdev->ibdev, - "Requested %d MSI-X vectors, got %d\n", - num_msix_want, num_msix_got); - } - rdev->num_msix = num_msix_got; -done: - return rc; -} - static void bnxt_re_init_hwrm_hdr(struct bnxt_re_dev *rdev, struct input *hdr, u16 opcd, u16 crid, u16 trid) { @@ -785,7 +759,7 @@ static u32 bnxt_re_get_nqdb_offset(struct bnxt_re_dev *rdev, u16 indx) return bnxt_qplib_is_chip_gen_p5(rdev->chip_ctx) ? (rdev->is_virtfn ? BNXT_RE_GEN_P5_VF_NQ_DB : BNXT_RE_GEN_P5_PF_NQ_DB) : - rdev->msix_entries[indx].db_offset; + rdev->en_dev->msix_entries[indx].db_offset; } static void bnxt_re_cleanup_res(struct bnxt_re_dev *rdev) @@ -810,7 +784,7 @@ static int bnxt_re_init_res(struct bnxt_re_dev *rdev) for (i = 1; i < rdev->num_msix ; i++) { db_offt = bnxt_re_get_nqdb_offset(rdev, i); rc = bnxt_qplib_enable_nq(rdev->en_dev->pdev, &rdev->nq[i - 1], - i - 1, rdev->msix_entries[i].vector, + i - 1, rdev->en_dev->msix_entries[i].vector, db_offt, &bnxt_re_cqn_handler, &bnxt_re_srqn_handler); if (rc) { @@ -897,7 +871,7 @@ static int bnxt_re_alloc_res(struct bnxt_re_dev *rdev) rattr.type = type; rattr.mode = RING_ALLOC_REQ_INT_MODE_MSIX; rattr.depth = BNXT_QPLIB_NQE_MAX_CNT - 1; - rattr.lrid = rdev->msix_entries[i + 1].ring_idx; + rattr.lrid = rdev->en_dev->msix_entries[i + 1].ring_idx; rc = bnxt_re_net_ring_alloc(rdev, &rattr, &nq->ring_id); if (rc) { ibdev_err(&rdev->ibdev, @@ -1217,7 +1191,7 @@ static void bnxt_re_dev_uninit(struct bnxt_re_dev *rdev) bnxt_qplib_free_rcfw_channel(&rdev->rcfw); } if (test_and_clear_bit(BNXT_RE_FLAG_GOT_MSIX, &rdev->flags)) - bnxt_free_msix_vecs(rdev->en_dev); + rdev->num_msix = 0; bnxt_re_destroy_chip_ctx(rdev); if (test_and_clear_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags)) @@ -1262,13 +1236,15 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 wqe_mode) /* Check whether VF or PF */ bnxt_re_get_sriov_func_type(rdev); - rc = bnxt_re_request_msix(rdev); - if (rc) { + if (!rdev->en_dev->ulp_tbl->msix_requested) { ibdev_err(&rdev->ibdev, "Failed to get MSI-X vectors: %#x\n", rc); rc = -EINVAL; goto fail; } + ibdev_dbg(&rdev->ibdev, "Got %d MSI-X vectors\n", + rdev->en_dev->ulp_tbl->msix_requested); + rdev->num_msix = rdev->en_dev->ulp_tbl->msix_requested; set_bit(BNXT_RE_FLAG_GOT_MSIX, &rdev->flags); bnxt_re_query_hwrm_intf_version(rdev); @@ -1292,14 +1268,14 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 wqe_mode) rattr.type = type; rattr.mode = RING_ALLOC_REQ_INT_MODE_MSIX; rattr.depth = BNXT_QPLIB_CREQE_MAX_CNT - 1; - rattr.lrid = rdev->msix_entries[BNXT_RE_AEQ_IDX].ring_idx; + rattr.lrid = rdev->en_dev->msix_entries[BNXT_RE_AEQ_IDX].ring_idx; rc = bnxt_re_net_ring_alloc(rdev, &rattr, &creq->ring_id); if (rc) { ibdev_err(&rdev->ibdev, "Failed to allocate CREQ: %#x\n", rc); goto free_rcfw; } db_offt = bnxt_re_get_nqdb_offset(rdev, BNXT_RE_AEQ_IDX); - vid = rdev->msix_entries[BNXT_RE_AEQ_IDX].vector; + vid = rdev->en_dev->msix_entries[BNXT_RE_AEQ_IDX].vector; rc = bnxt_qplib_enable_rcfw_channel(&rdev->rcfw, vid, db_offt, rdev->is_virtfn, &bnxt_re_aeq_handler); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c index 2164b4419263..75ae0f499793 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.c @@ -28,6 +28,30 @@ static DEFINE_IDA(bnxt_aux_dev_ids); +static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) +{ + struct bnxt_en_dev *edev = bp->edev; + int num_msix, idx, i; + + if (!edev->ulp_tbl->msix_requested) { + netdev_warn(bp->dev, "Requested MSI-X vectors insufficient\n"); + return; + } + num_msix = edev->ulp_tbl->msix_requested; + idx = edev->ulp_tbl->msix_base; + for (i = 0; i < num_msix; i++) { + ent[i].vector = bp->irq_tbl[idx + i].vector; + ent[i].ring_idx = idx + i; + if (bp->flags & BNXT_FLAG_CHIP_P5) { + ent[i].db_offset = DB_PF_OFFSET_P5; + if (BNXT_VF(bp)) + ent[i].db_offset = DB_VF_OFFSET_P5; + } else { + ent[i].db_offset = (idx + i) * 0x80; + } + } +} + int bnxt_register_dev(struct bnxt_en_dev *edev, struct bnxt_ulp_ops *ulp_ops, void *handle) @@ -42,17 +66,18 @@ int bnxt_register_dev(struct bnxt_en_dev *edev, bp->cp_nr_rings == max_stat_ctxs) return -ENOMEM; - ulp = kzalloc(sizeof(*ulp), GFP_KERNEL); + ulp = edev->ulp_tbl; if (!ulp) return -ENOMEM; - edev->ulp_tbl = ulp; ulp->handle = handle; rcu_assign_pointer(ulp->ulp_ops, ulp_ops); if (test_bit(BNXT_STATE_OPEN, &bp->state)) bnxt_hwrm_vnic_cfg(bp, 0); + bnxt_fill_msix_vecs(bp, bp->edev->msix_entries); + edev->flags |= BNXT_EN_FLAG_MSIX_REQUESTED; return 0; } EXPORT_SYMBOL(bnxt_register_dev); @@ -66,7 +91,7 @@ void bnxt_unregister_dev(struct bnxt_en_dev *edev) ulp = edev->ulp_tbl; if (ulp->msix_requested) - bnxt_free_msix_vecs(edev); + edev->flags &= ~BNXT_EN_FLAG_MSIX_REQUESTED; if (ulp->max_async_event_id) bnxt_hwrm_func_drv_rgtr(bp, NULL, 0, true); @@ -79,126 +104,17 @@ void bnxt_unregister_dev(struct bnxt_en_dev *edev) msleep(100); i++; } - kfree(ulp); - edev->ulp_tbl = NULL; return; } EXPORT_SYMBOL(bnxt_unregister_dev); -static void bnxt_fill_msix_vecs(struct bnxt *bp, struct bnxt_msix_entry *ent) -{ - struct bnxt_en_dev *edev = bp->edev; - int num_msix, idx, i; - - num_msix = edev->ulp_tbl->msix_requested; - idx = edev->ulp_tbl->msix_base; - for (i = 0; i < num_msix; i++) { - ent[i].vector = bp->irq_tbl[idx + i].vector; - ent[i].ring_idx = idx + i; - if (bp->flags & BNXT_FLAG_CHIP_P5) { - ent[i].db_offset = DB_PF_OFFSET_P5; - if (BNXT_VF(bp)) - ent[i].db_offset = DB_VF_OFFSET_P5; - } else { - ent[i].db_offset = (idx + i) * 0x80; - } - } -} - -int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, - struct bnxt_msix_entry *ent, - int num_msix) -{ - struct net_device *dev = edev->net; - struct bnxt *bp = netdev_priv(dev); - struct bnxt_hw_resc *hw_resc; - int max_idx, max_cp_rings; - int avail_msix, idx; - int total_vecs; - int rc = 0; - - if (!(bp->flags & BNXT_FLAG_USING_MSIX)) - return -ENODEV; - - if (edev->ulp_tbl->msix_requested) - return -EAGAIN; - - max_cp_rings = bnxt_get_max_func_cp_rings(bp); - avail_msix = bnxt_get_avail_msix(bp, num_msix); - if (!avail_msix) - return -ENOMEM; - if (avail_msix > num_msix) - avail_msix = num_msix; - - if (BNXT_NEW_RM(bp)) { - idx = bp->cp_nr_rings; - } else { - max_idx = min_t(int, bp->total_irqs, max_cp_rings); - idx = max_idx - avail_msix; - } - - edev->ulp_tbl->msix_base = idx; - edev->ulp_tbl->msix_requested = avail_msix; - hw_resc = &bp->hw_resc; - total_vecs = idx + avail_msix; - rtnl_lock(); - if (bp->total_irqs < total_vecs || - (BNXT_NEW_RM(bp) && hw_resc->resv_irqs < total_vecs)) { - if (netif_running(dev)) { - bnxt_close_nic(bp, true, false); - rc = bnxt_open_nic(bp, true, false); - } else { - rc = bnxt_reserve_rings(bp, true); - } - } - rtnl_unlock(); - if (rc) { - edev->ulp_tbl->msix_requested = 0; - return -EAGAIN; - } - - if (BNXT_NEW_RM(bp)) { - int resv_msix; - - resv_msix = hw_resc->resv_irqs - bp->cp_nr_rings; - avail_msix = min_t(int, resv_msix, avail_msix); - edev->ulp_tbl->msix_requested = avail_msix; - } - bnxt_fill_msix_vecs(bp, ent); - edev->flags |= BNXT_EN_FLAG_MSIX_REQUESTED; - return avail_msix; -} -EXPORT_SYMBOL(bnxt_req_msix_vecs); - -void bnxt_free_msix_vecs(struct bnxt_en_dev *edev) -{ - struct net_device *dev = edev->net; - struct bnxt *bp = netdev_priv(dev); - - if (!(edev->flags & BNXT_EN_FLAG_MSIX_REQUESTED)) - return; - - edev->ulp_tbl->msix_requested = 0; - edev->flags &= ~BNXT_EN_FLAG_MSIX_REQUESTED; - rtnl_lock(); - if (netif_running(dev) && !(edev->flags & BNXT_EN_FLAG_ULP_STOPPED)) { - bnxt_close_nic(bp, true, false); - bnxt_open_nic(bp, true, false); - } - rtnl_unlock(); - - return; -} -EXPORT_SYMBOL(bnxt_free_msix_vecs); - int bnxt_get_ulp_msix_num(struct bnxt *bp) { - if (bnxt_ulp_registered(bp->edev)) { - struct bnxt_en_dev *edev = bp->edev; + u32 roce_msix = BNXT_VF(bp) ? + BNXT_MAX_VF_ROCE_MSIX : BNXT_MAX_ROCE_MSIX; - return edev->ulp_tbl->msix_requested; - } - return 0; + return ((bp->flags & BNXT_FLAG_ROCE_CAP) && bp->aux_dev ? + min_t(u32, roce_msix, num_online_cpus()) : 0); } int bnxt_get_ulp_msix_base(struct bnxt *bp) @@ -416,6 +332,8 @@ static void bnxt_aux_dev_release(struct device *dev) container_of(dev, struct bnxt_aux_dev, aux_dev.dev); struct bnxt *bp = netdev_priv(bnxt_adev->edev->net); + kfree(bnxt_adev->edev->ulp_tbl); + bnxt_adev->edev->ulp_tbl = NULL; kfree(bnxt_adev->edev); bnxt_adev->edev = NULL; bp->edev = NULL; @@ -439,6 +357,8 @@ static void bnxt_set_edev_info(struct bnxt_en_dev *edev, struct bnxt *bp) edev->hw_ring_stats_size = bp->hw_ring_stats_size; edev->pf_port_id = bp->pf.port_id; edev->en_state = bp->state; + + edev->ulp_tbl->msix_requested = bnxt_get_ulp_msix_num(bp); } static int bnxt_rdma_aux_device_add(struct bnxt *bp) @@ -446,12 +366,21 @@ static int bnxt_rdma_aux_device_add(struct bnxt *bp) struct bnxt_aux_dev *bnxt_adev = bp->aux_dev; struct bnxt_en_dev *edev = bnxt_adev->edev; struct auxiliary_device *aux_dev; + struct bnxt_ulp *ulp; int ret; edev = kzalloc(sizeof(*edev), GFP_KERNEL); if (!edev) return -ENOMEM; + ulp = kzalloc(sizeof(*ulp), GFP_KERNEL); + if (!ulp) { + ret = -ENOMEM; + goto free_edev; + } + + edev->ulp_tbl = ulp; + aux_dev = &bnxt_adev->aux_dev; aux_dev->id = bnxt_adev->id; aux_dev->name = "rdma"; @@ -464,7 +393,7 @@ static int bnxt_rdma_aux_device_add(struct bnxt *bp) ret = auxiliary_device_init(aux_dev); if (ret) - goto free_edev; + goto free_ulp; ret = auxiliary_device_add(aux_dev); if (ret) @@ -473,6 +402,9 @@ static int bnxt_rdma_aux_device_add(struct bnxt *bp) return 0; aux_dev_uninit: auxiliary_device_uninit(aux_dev); +free_ulp: + kfree(ulp); + bp->edev->ulp_tbl = NULL; free_edev: kfree(edev); bp->edev = NULL; diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h index 0e15b35c36dd..01a9dc336312 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ulp.h @@ -15,6 +15,8 @@ #define BNXT_MIN_ROCE_CP_RINGS 2 #define BNXT_MIN_ROCE_STAT_CTXS 1 +#define BNXT_MAX_ROCE_MSIX 9 +#define BNXT_MAX_VF_ROCE_MSIX 2 struct hwrm_async_event_cmpl; struct bnxt; @@ -51,6 +53,7 @@ struct bnxt_ulp { struct bnxt_en_dev { struct net_device *net; struct pci_dev *pdev; + struct bnxt_msix_entry msix_entries[BNXT_MAX_ROCE_MSIX]; u32 flags; #define BNXT_EN_FLAG_ROCEV1_CAP 0x1 #define BNXT_EN_FLAG_ROCEV2_CAP 0x2 @@ -102,9 +105,6 @@ void bnxt_aux_dev_free(struct bnxt *bp); int bnxt_register_dev(struct bnxt_en_dev *edev, struct bnxt_ulp_ops *ulp_ops, void *handle); void bnxt_unregister_dev(struct bnxt_en_dev *edev); -int bnxt_req_msix_vecs(struct bnxt_en_dev *edev, struct bnxt_msix_entry *ent, - int num_msix); -void bnxt_free_msix_vecs(struct bnxt_en_dev *edev); int bnxt_send_msg(struct bnxt_en_dev *edev, struct bnxt_fw_msg *fw_msg); int bnxt_register_async_events(struct bnxt_en_dev *edev, unsigned long *events_bmap, u16 max_id);