From patchwork Sun Aug 13 14:51:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 135045 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp2249419vqi; Sun, 13 Aug 2023 08:15:01 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEvZosu0MoNFOvFm+A9gHeEakG3wU9pZtDUgPDMdlf1dtOg61fLX5DtzN8YlGmWG4sSYUq8 X-Received: by 2002:a05:6a20:5487:b0:136:e26b:6401 with SMTP id i7-20020a056a20548700b00136e26b6401mr11297479pzk.16.1691939701269; Sun, 13 Aug 2023 08:15:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691939701; cv=none; d=google.com; s=arc-20160816; b=t3B8ORaHYz+BH8N5B8hKOTexfWRHkxFgRyPzbN7h16t+wnSuYQ0IYu9OaLFZJOBi03 KkCJq58ru6CRiQifVQPjwNqWh27a6UM85x+PjNbyhCB89k3DolIUyNln7vcLIGp3v7+G Wq3QWCRMyQOLdcrerbRG0iG0kSE0bCukmFiwUpSOkiaYIUbaiiJfuAhl4n+BKWxQ9nM8 I+gWAvLl72jl84W3ZbGkttwIxwkIga7zV/FfyZ77EZAW2gf7robCBGaoNCD2OAdmABAm SeNbi+koBa6KkSNc1ViUXtjA8uZlA3/4Zpwt8XP4W58QpFhqmp7v6NPQWO5D+Q9va9sR OGdA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=mg+9Z7qbAnnpbMbp7NGUXXtkU2DN+Q6EW1LYSqeD6gg=; fh=thUxDq4m5D3FQct5425V7h171/dEmaHUwwAlFhAMPtU=; b=KrQutcDzyNsppbNcfLV0lvudPaafGjUA5JU+pdkLiE3nGna7+0+MjsZDr2MotM2KH8 sAFKA0h6jgyOvhr4+OxDef6Ljp3JSvhl9p1UHQYV2OnL0ZlBY38MfEeXsm3E9jDUJF5s 5l4oJHvJ13fq39AC1ptjiAJvIxvH151WSiyVpBCsYaAZHNCBc5geg4tlfgt7T0Ldvcdj +561Zo5Lktf1y335IsAa9P091zPLaqVb/EXIh8ELj6ld2NlDkWKjF/yPoJ/geCqWqWsS yxrFTAtv/WMUCiYrjEbr6oJQ79GpGeCzfq0lNl9uKvGs1t7TEnf79CTrfkikor0KYW5t 9xLA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=C11BhugW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 195-20020a6301cc000000b0055384255b9bsi6513477pgb.24.2023.08.13.08.14.35; Sun, 13 Aug 2023 08:15:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=C11BhugW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229578AbjHMOw1 (ORCPT + 99 others); Sun, 13 Aug 2023 10:52:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59954 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231313AbjHMOwH (ORCPT ); Sun, 13 Aug 2023 10:52:07 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A2A651703; Sun, 13 Aug 2023 07:52:05 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 3AD0E21954; Sun, 13 Aug 2023 14:52:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1691938324; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mg+9Z7qbAnnpbMbp7NGUXXtkU2DN+Q6EW1LYSqeD6gg=; b=C11BhugWoMBvBzRqLdxN5sTqKs1Lj0+Aco5E3izg6NNRUKujW1iDWpL6m/z+HLlL20vhg6 h6WhbBRbosg6jyJLyQc+tbIqLF3F0sjVCJTqogCsurfmoba380Vs6HeihFezF0KvMaiHXh nBYZoWj2OdhLeiWpTuUKYVhdGpJIDks= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 019A313593; Sun, 13 Aug 2023 14:52:03 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id gNc6OxPu2GSDFAAAMHmgww (envelope-from ); Sun, 13 Aug 2023 14:52:03 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v2 01/10] mlx4: Get rid of the mlx4_interface.get_dev callback Date: Sun, 13 Aug 2023 16:51:18 +0200 Message-Id: <20230813145127.10653-2-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230813145127.10653-1-petr.pavlu@suse.com> References: <20230813145127.10653-1-petr.pavlu@suse.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED,SPF_PASS, T_SPF_HELO_TEMPERROR autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774127364206114199 X-GMAIL-MSGID: 1774127364206114199 Simplify the mlx4 driver interface by removing mlx4_get_protocol_dev() and the associated mlx4_interface.get_dev callbacks. This is done in preparation to use an auxiliary bus to model the mlx4 driver structure. The change is motivated by the following situation: * The mlx4_en interface is being initialized by mlx4_en_add() and mlx4_en_activate(). * The latter activate function calls mlx4_en_init_netdev() -> register_netdev() to register a new net_device. * A netdev event NETDEV_REGISTER is raised for the device. * The netdev notififier mlx4_ib_netdev_event() is called and it invokes mlx4_ib_scan_netdevs() -> mlx4_get_protocol_dev() -> mlx4_en_get_netdev() [via mlx4_interface.get_dev]. This chain creates a problem when mlx4_en gets switched to be an auxiliary driver. It contains two device calls which would both need to take a respective device lock. Avoid this situation by updating mlx4_ib_scan_netdevs() to no longer call mlx4_get_protocol_dev() but instead to utilize the information passed in net_device.parent and net_device.dev_port. This data is sufficient to determine that an updated port is one that the mlx4_ib driver should take care of and to keep mlx4_ib_dev.iboe.netdevs up to date. Following that, update mlx4_ib_get_netdev() to also not call mlx4_get_protocol_dev() and instead scan all current netdevs to find find a matching one. Note that mlx4_ib_get_netdev() is called early from ib_register_device() and cannot use data tracked in mlx4_ib_dev.iboe.netdevs which is not at that point yet set. Finally, remove function mlx4_get_protocol_dev() and the mlx4_interface.get_dev callbacks (only mlx4_en_get_netdev()) as they became unused. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- drivers/infiniband/hw/mlx4/main.c | 89 ++++++++++---------- drivers/net/ethernet/mellanox/mlx4/en_main.c | 8 -- drivers/net/ethernet/mellanox/mlx4/intf.c | 21 ----- include/linux/mlx4/driver.h | 3 - 4 files changed, 43 insertions(+), 78 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c index b18e9f2adc82..7dd70d778b6b 100644 --- a/drivers/infiniband/hw/mlx4/main.c +++ b/drivers/infiniband/hw/mlx4/main.c @@ -125,12 +125,14 @@ static struct net_device *mlx4_ib_get_netdev(struct ib_device *device, u32 port_num) { struct mlx4_ib_dev *ibdev = to_mdev(device); - struct net_device *dev; + struct net_device *dev, *ret = NULL; rcu_read_lock(); - dev = mlx4_get_protocol_dev(ibdev->dev, MLX4_PROT_ETH, port_num); + for_each_netdev_rcu(&init_net, dev) { + if (dev->dev.parent != ibdev->ib_dev.dev.parent || + dev->dev_port + 1 != port_num) + continue; - if (dev) { if (mlx4_is_bonded(ibdev->dev)) { struct net_device *upper = NULL; @@ -143,11 +145,14 @@ static struct net_device *mlx4_ib_get_netdev(struct ib_device *device, dev = active; } } + + dev_hold(dev); + ret = dev; + break; } - dev_hold(dev); rcu_read_unlock(); - return dev; + return ret; } static int mlx4_ib_update_gids_v1(struct gid_entry *gids, @@ -2319,61 +2324,53 @@ static void mlx4_ib_update_qps(struct mlx4_ib_dev *ibdev, mutex_unlock(&ibdev->qp1_proxy_lock[port - 1]); } -static void mlx4_ib_scan_netdevs(struct mlx4_ib_dev *ibdev, - struct net_device *dev, - unsigned long event) +static void mlx4_ib_scan_netdev(struct mlx4_ib_dev *ibdev, + struct net_device *dev, + unsigned long event) { - struct mlx4_ib_iboe *iboe; - int update_qps_port = -1; - int port; + struct mlx4_ib_iboe *iboe = &ibdev->iboe; ASSERT_RTNL(); - iboe = &ibdev->iboe; + if (dev->dev.parent != ibdev->ib_dev.dev.parent) + return; spin_lock_bh(&iboe->lock); - mlx4_foreach_ib_transport_port(port, ibdev->dev) { - - iboe->netdevs[port - 1] = - mlx4_get_protocol_dev(ibdev->dev, MLX4_PROT_ETH, port); - if (dev == iboe->netdevs[port - 1] && - (event == NETDEV_CHANGEADDR || event == NETDEV_REGISTER || - event == NETDEV_UP || event == NETDEV_CHANGE)) - update_qps_port = port; + iboe->netdevs[dev->dev_port] = event != NETDEV_UNREGISTER ? dev : NULL; - if (dev == iboe->netdevs[port - 1] && - (event == NETDEV_UP || event == NETDEV_DOWN)) { - enum ib_port_state port_state; - struct ib_event ibev = { }; - - if (ib_get_cached_port_state(&ibdev->ib_dev, port, - &port_state)) - continue; + if (event == NETDEV_UP || event == NETDEV_DOWN) { + enum ib_port_state port_state; + struct ib_event ibev = { }; - if (event == NETDEV_UP && - (port_state != IB_PORT_ACTIVE || - iboe->last_port_state[port - 1] != IB_PORT_DOWN)) - continue; - if (event == NETDEV_DOWN && - (port_state != IB_PORT_DOWN || - iboe->last_port_state[port - 1] != IB_PORT_ACTIVE)) - continue; - iboe->last_port_state[port - 1] = port_state; + if (ib_get_cached_port_state(&ibdev->ib_dev, dev->dev_port + 1, + &port_state)) + goto iboe_out; - ibev.device = &ibdev->ib_dev; - ibev.element.port_num = port; - ibev.event = event == NETDEV_UP ? IB_EVENT_PORT_ACTIVE : - IB_EVENT_PORT_ERR; - ib_dispatch_event(&ibev); - } + if (event == NETDEV_UP && + (port_state != IB_PORT_ACTIVE || + iboe->last_port_state[dev->dev_port] != IB_PORT_DOWN)) + goto iboe_out; + if (event == NETDEV_DOWN && + (port_state != IB_PORT_DOWN || + iboe->last_port_state[dev->dev_port] != IB_PORT_ACTIVE)) + goto iboe_out; + iboe->last_port_state[dev->dev_port] = port_state; + ibev.device = &ibdev->ib_dev; + ibev.element.port_num = dev->dev_port + 1; + ibev.event = event == NETDEV_UP ? IB_EVENT_PORT_ACTIVE : + IB_EVENT_PORT_ERR; + ib_dispatch_event(&ibev); } + +iboe_out: spin_unlock_bh(&iboe->lock); - if (update_qps_port > 0) - mlx4_ib_update_qps(ibdev, dev, update_qps_port); + if (event == NETDEV_CHANGEADDR || event == NETDEV_REGISTER || + event == NETDEV_UP || event == NETDEV_CHANGE) + mlx4_ib_update_qps(ibdev, dev, dev->dev_port + 1); } static int mlx4_ib_netdev_event(struct notifier_block *this, @@ -2386,7 +2383,7 @@ static int mlx4_ib_netdev_event(struct notifier_block *this, return NOTIFY_DONE; ibdev = container_of(this, struct mlx4_ib_dev, iboe.nb); - mlx4_ib_scan_netdevs(ibdev, dev, event); + mlx4_ib_scan_netdev(ibdev, dev, event); return NOTIFY_DONE; } diff --git a/drivers/net/ethernet/mellanox/mlx4/en_main.c b/drivers/net/ethernet/mellanox/mlx4/en_main.c index f1259bdb1a29..6a42bec6bd85 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_main.c @@ -183,13 +183,6 @@ static void mlx4_en_get_profile(struct mlx4_en_dev *mdev) } } -static void *mlx4_en_get_netdev(struct mlx4_dev *dev, void *ctx, u8 port) -{ - struct mlx4_en_dev *endev = ctx; - - return endev->pndev[port]; -} - static void mlx4_en_event(struct mlx4_dev *dev, void *endev_ptr, enum mlx4_dev_event event, unsigned long port) { @@ -354,7 +347,6 @@ static struct mlx4_interface mlx4_en_interface = { .add = mlx4_en_add, .remove = mlx4_en_remove, .event = mlx4_en_event, - .get_dev = mlx4_en_get_netdev, .protocol = MLX4_PROT_ETH, .activate = mlx4_en_activate, }; diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c index 65482f004e50..28d7da925d36 100644 --- a/drivers/net/ethernet/mellanox/mlx4/intf.c +++ b/drivers/net/ethernet/mellanox/mlx4/intf.c @@ -245,27 +245,6 @@ void mlx4_unregister_device(struct mlx4_dev *dev) mutex_unlock(&intf_mutex); } -void *mlx4_get_protocol_dev(struct mlx4_dev *dev, enum mlx4_protocol proto, int port) -{ - struct mlx4_priv *priv = mlx4_priv(dev); - struct mlx4_device_context *dev_ctx; - unsigned long flags; - void *result = NULL; - - spin_lock_irqsave(&priv->ctx_lock, flags); - - list_for_each_entry(dev_ctx, &priv->ctx_list, list) - if (dev_ctx->intf->protocol == proto && dev_ctx->intf->get_dev) { - result = dev_ctx->intf->get_dev(dev, dev_ctx->context, port); - break; - } - - spin_unlock_irqrestore(&priv->ctx_lock, flags); - - return result; -} -EXPORT_SYMBOL_GPL(mlx4_get_protocol_dev); - struct devlink_port *mlx4_get_devlink_port(struct mlx4_dev *dev, int port) { struct mlx4_port_info *info = &mlx4_priv(dev)->port[port]; diff --git a/include/linux/mlx4/driver.h b/include/linux/mlx4/driver.h index 1834c8fad12e..923951e19300 100644 --- a/include/linux/mlx4/driver.h +++ b/include/linux/mlx4/driver.h @@ -59,7 +59,6 @@ struct mlx4_interface { void (*remove)(struct mlx4_dev *dev, void *context); void (*event) (struct mlx4_dev *dev, void *context, enum mlx4_dev_event event, unsigned long param); - void * (*get_dev)(struct mlx4_dev *dev, void *context, u8 port); void (*activate)(struct mlx4_dev *dev, void *context); struct list_head list; enum mlx4_protocol protocol; @@ -88,8 +87,6 @@ struct mlx4_port_map { int mlx4_port_map_set(struct mlx4_dev *dev, struct mlx4_port_map *v2p); -void *mlx4_get_protocol_dev(struct mlx4_dev *dev, enum mlx4_protocol proto, int port); - struct devlink_port *mlx4_get_devlink_port(struct mlx4_dev *dev, int port); #endif /* MLX4_DRIVER_H */ From patchwork Sun Aug 13 14:51:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 135059 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp2277998vqi; Sun, 13 Aug 2023 09:25:29 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFyqoxy4N/PtGXOBsLuaDGlLSpDKZgOWzXXWXbxq7FxHzhQnvYGg0w7F14NvFfrCNoR51RU X-Received: by 2002:a17:907:d8f:b0:98e:738c:6d39 with SMTP id go15-20020a1709070d8f00b0098e738c6d39mr10588810ejc.36.1691943929234; Sun, 13 Aug 2023 09:25:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691943929; cv=none; d=google.com; s=arc-20160816; b=OimgsNXbTQuRp4JVSqHBUgWG0W/r6Kg0SaLVupyiefE2u5xJqGp9AmS0dI3x4lDg0m YZ/QGQUUzxQNMyQDjbpp3XyNX8/AmkQ5fHM1ESxi0a+o0tZVfnV5octu7UXbMb0hZK2K VhFVPnkzplv5l8C4wG+0hN89TIJanfI6Nj97/70FZ6Y55DROxpzqmZxnU4MK9BXV7y0C zMjt3q8YLeaqdzWKN6GiOwJ0k9WjJzaLpG91yN5H3iX/zvtzxW4lLXfsCprNYQqySCMR NefH2LtEKFlG+OmiY0E0AvrvFVkHyQNp9OSrDTA3XpcMxbux+anBRXzl4dvM3oHDzkPR oQ2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=MlkLwPoGSJ/eBik/CH7trGhSdIICH1/nZbIFJG2ENWw=; fh=thUxDq4m5D3FQct5425V7h171/dEmaHUwwAlFhAMPtU=; b=xlyil/787qWcRWKZhEzdNYNOZ2n7ts/k9RMpld7V5/h2F9DexH+2KS5OMOtObCrzCT TbMRldBeukEHO0tkL46QmPe+J2LX8Wv6IDL3Rx3izvA3trPlyqSJxEXQCE2NK4jM8EOG FNRsYSAdQiLghpuMIYVpnt/BJ+Rb1KsM+CKduuZiiOGu+qAPIWNgGeLdWlmHjkAZuT5X hYxiAQRRLiQFHZrhFlEA5/VB4fjSef8Qsq+dZiri3RCXbaOSwtsS8CrgMiJlVrjsOJKF Dq40hjaUIVSXLKno4PtBoQ43lLVnEJq2i2ggWxFl+nwA+ExC4ZZzcRlJnptaoLi047qV RMSQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=JgNft9kT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n1-20020a170906088100b00996e0acf032si6545150eje.630.2023.08.13.09.25.04; Sun, 13 Aug 2023 09:25:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=JgNft9kT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231320AbjHMOwH (ORCPT + 99 others); Sun, 13 Aug 2023 10:52:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42470 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230005AbjHMOwE (ORCPT ); Sun, 13 Aug 2023 10:52:04 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D39AD1704; Sun, 13 Aug 2023 07:52:05 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 715651F8C8; Sun, 13 Aug 2023 14:52:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1691938324; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MlkLwPoGSJ/eBik/CH7trGhSdIICH1/nZbIFJG2ENWw=; b=JgNft9kTY/IZrc8gQqxW8iTayMD6FYnl8qngHUyIAmn21ME/+tKydRFpmt3BUDTgNknySO g/IiLUPw6ac3DkyibGC1dvm8RkAR9XKgLqCAtZUia7fqj60pipY3RJ2yujDIor+FDlQddw cXmFsZ4r78G684IW1Q8YMyxSP1JHWkY= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 3CDE31322C; Sun, 13 Aug 2023 14:52:04 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id GNkZDhTu2GSDFAAAMHmgww (envelope-from ); Sun, 13 Aug 2023 14:52:04 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v2 02/10] mlx4: Rename member mlx4_en_dev.nb to netdev_nb Date: Sun, 13 Aug 2023 16:51:19 +0200 Message-Id: <20230813145127.10653-3-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230813145127.10653-1-petr.pavlu@suse.com> References: <20230813145127.10653-1-petr.pavlu@suse.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774131797559875247 X-GMAIL-MSGID: 1774131797559875247 Rename the mlx4_en_dev.nb notifier_block member to netdev_nb in preparation to add a mlx4 core notifier_block. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx4/en_main.c | 14 +++++++------- drivers/net/ethernet/mellanox/mlx4/en_netdev.c | 2 +- drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 2 +- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/en_main.c b/drivers/net/ethernet/mellanox/mlx4/en_main.c index 6a42bec6bd85..be8ba34c9025 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_main.c @@ -235,8 +235,8 @@ static void mlx4_en_remove(struct mlx4_dev *dev, void *endev_ptr) iounmap(mdev->uar_map); mlx4_uar_free(dev, &mdev->priv_uar); mlx4_pd_free(dev, mdev->priv_pdn); - if (mdev->nb.notifier_call) - unregister_netdevice_notifier(&mdev->nb); + if (mdev->netdev_nb.notifier_call) + unregister_netdevice_notifier(&mdev->netdev_nb); kfree(mdev); } @@ -252,11 +252,11 @@ static void mlx4_en_activate(struct mlx4_dev *dev, void *ctx) mdev->pndev[i] = NULL; } - /* register notifier */ - mdev->nb.notifier_call = mlx4_en_netdev_event; - if (register_netdevice_notifier(&mdev->nb)) { - mdev->nb.notifier_call = NULL; - mlx4_err(mdev, "Failed to create notifier\n"); + /* register netdev notifier */ + mdev->netdev_nb.notifier_call = mlx4_en_netdev_event; + if (register_netdevice_notifier(&mdev->netdev_nb)) { + mdev->netdev_nb.notifier_call = NULL; + mlx4_err(mdev, "Failed to create netdev notifier\n"); } } diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c index 403604ceebc8..7066c426b95c 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c @@ -2967,7 +2967,7 @@ int mlx4_en_netdev_event(struct notifier_block *this, if (!net_eq(dev_net(ndev), &init_net)) return NOTIFY_DONE; - mdev = container_of(this, struct mlx4_en_dev, nb); + mdev = container_of(this, struct mlx4_en_dev, netdev_nb); dev = mdev->dev; /* Go into this mode only when two network devices set on two ports diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h index 321f801c1d7c..72a3fea36702 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h @@ -432,7 +432,7 @@ struct mlx4_en_dev { unsigned long last_overflow_check; struct ptp_clock *ptp_clock; struct ptp_clock_info ptp_clock_info; - struct notifier_block nb; + struct notifier_block netdev_nb; }; From patchwork Sun Aug 13 14:51:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 135047 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp2253061vqi; Sun, 13 Aug 2023 08:24:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHUu9fbt+lZXolAvnQ2JpU6wzbY1LQvCbIYur2zleAuy6vOM0LZBRoBWHbV4BjX9AQZHk9h X-Received: by 2002:a05:6a21:47cb:b0:133:6b64:c92 with SMTP id as11-20020a056a2147cb00b001336b640c92mr6892059pzc.44.1691940255773; Sun, 13 Aug 2023 08:24:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691940255; cv=none; d=google.com; s=arc-20160816; b=w8o8xv+j0f3p5M0lV5GdJH2yZFo0+WaacNkQYJCKqoFmFAyv24Rsi0rUGTlTIdVC2g kssa0L4Kitc2EiLvbJ07BFwm5qlRurGNCR9265r7foqAjEG/+9YJRwJ6MxdhWCiPSdPn iH6Apu8hntP9wOkreiVhzTNl2fqER+TO/OYuOur1uH6CKgR5QGikX1JsZ2gSRi7pW88Y vNyJOz5EL8QercCaRUZVGZfTIg8GsnlIPHlqp6s1hj+HrtipbZQuYnYFCFOWZHsibzZl 0YC6YIn1NsrLO5UtLvBimedv/qn58BcF8ealhriEVd59ZVU4brL+DfDBj48YdyUUctNL ORzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=iOtYDSnUsEp4f4vvbB+T48cs7NPkWMFwyATia++Khlg=; fh=thUxDq4m5D3FQct5425V7h171/dEmaHUwwAlFhAMPtU=; b=rWZ96EWQrJ87Ge5Ncpfh3Ph/YYQcQO5GLZXYHoYrljxwJqT64HTWZoNLOERiM97/Ew GhT00pHlRqJF8t+2lgMPbrGziA7xh3E89ZTI21OXzNg1O+vwNBXW2K5oaW6JUMzrrAiq MPvWAjv2Mn9pPiCyGXrfeY0pi7681FaznDazskzbx4hnboMrqxVhV+RvYPHiDK0ptrbd 4iZa0FFYNf1hfgVbChLWLcGtzZlmFIsDjdIjw+bNQiuyZec6GkJZqNa40Hgtkafn/bSp uxXrX1AX/ZsbGr7byJIvJgh8id4mgTsCcHgrMEG9jsUwi5II5LTpr4x5Kp+hqG/8Fk9m /ORA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=nEpfbf8B; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bw9-20020a056a00408900b0064f6a531c88si6484907pfb.323.2023.08.13.08.23.57; Sun, 13 Aug 2023 08:24:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=nEpfbf8B; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231396AbjHMOwP (ORCPT + 99 others); Sun, 13 Aug 2023 10:52:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42498 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229441AbjHMOwG (ORCPT ); Sun, 13 Aug 2023 10:52:06 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 158F7170C; Sun, 13 Aug 2023 07:52:06 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id AD92F1F8CD; Sun, 13 Aug 2023 14:52:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1691938324; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iOtYDSnUsEp4f4vvbB+T48cs7NPkWMFwyATia++Khlg=; b=nEpfbf8B2msKulR8DkVS03C3UyjWow/gOdtnwUXg17ykeE2K7a/bdytpO2QRb0oSVpspjv Wuf27aP5zcDbUKpYTJlaLLMzstECxjAeq7xa3yXZcpHWoY6wUMFQe2SHASGrJYnUgAfYoh MzWWQ+Pz15sQvibKhZLXhw2BNYFmx+4= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 761A813593; Sun, 13 Aug 2023 14:52:04 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id QIEYHBTu2GSDFAAAMHmgww (envelope-from ); Sun, 13 Aug 2023 14:52:04 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v2 03/10] mlx4: Replace the mlx4_interface.event callback with a notifier Date: Sun, 13 Aug 2023 16:51:20 +0200 Message-Id: <20230813145127.10653-4-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230813145127.10653-1-petr.pavlu@suse.com> References: <20230813145127.10653-1-petr.pavlu@suse.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774127945863729598 X-GMAIL-MSGID: 1774127945863729598 Use a notifier to implement mlx4_dispatch_event() in preparation to switch mlx4_en and mlx4_ib to be an auxiliary device. A problem is that if the mlx4_interface.event callback was replaced with something as mlx4_adrv.event then the implementation of mlx4_dispatch_event() would need to acquire a lock on a given device before executing this callback. That is necessary because otherwise there is no guarantee that the associated driver cannot get unbound when the callback is running. However, taking this lock is not possible because mlx4_dispatch_event() can be invoked from the hardirq context. Using an atomic notifier allows the driver to accurately record when it wants to receive these events and solves this problem. A handler registration is done by both mlx4_en and mlx4_ib at the end of their mlx4_interface.add callback. This matches the current situation when mlx4_add_device() would enable events for a given device immediately after this callback, by adding the device on the mlx4_priv.list. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Acked-by: Tariq Toukan --- drivers/infiniband/hw/mlx4/main.c | 41 +++++++++++++------- drivers/infiniband/hw/mlx4/mlx4_ib.h | 2 + drivers/net/ethernet/mellanox/mlx4/en_main.c | 27 +++++++++---- drivers/net/ethernet/mellanox/mlx4/intf.c | 24 ++++++++---- drivers/net/ethernet/mellanox/mlx4/main.c | 2 + drivers/net/ethernet/mellanox/mlx4/mlx4.h | 2 + drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 2 + include/linux/mlx4/driver.h | 8 +++- 8 files changed, 77 insertions(+), 31 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c index 7dd70d778b6b..0761c465120b 100644 --- a/drivers/infiniband/hw/mlx4/main.c +++ b/drivers/infiniband/hw/mlx4/main.c @@ -82,6 +82,8 @@ static const char mlx4_ib_version[] = static void do_slave_init(struct mlx4_ib_dev *ibdev, int slave, int do_init); static enum rdma_link_layer mlx4_ib_port_link_layer(struct ib_device *device, u32 port_num); +static int mlx4_ib_event(struct notifier_block *this, unsigned long event, + void *ptr); static struct workqueue_struct *wq; @@ -2836,6 +2838,12 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) do_slave_init(ibdev, j, 1); } } + + /* register mlx4 core notifier */ + ibdev->mlx_nb.notifier_call = mlx4_ib_event; + err = mlx4_register_event_notifier(dev, &ibdev->mlx_nb); + WARN(err, "failed to register mlx4 event notifier (%d)", err); + return ibdev; err_notif: @@ -2953,6 +2961,8 @@ static void mlx4_ib_remove(struct mlx4_dev *dev, void *ibdev_ptr) int p; int i; + mlx4_unregister_event_notifier(dev, &ibdev->mlx_nb); + mlx4_foreach_port(i, dev, MLX4_PORT_TYPE_IB) devlink_port_type_clear(mlx4_get_devlink_port(dev, i)); ibdev->ib_active = false; @@ -3173,11 +3183,14 @@ void mlx4_sched_ib_sl2vl_update_work(struct mlx4_ib_dev *ibdev, } } -static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, - enum mlx4_dev_event event, unsigned long param) +static int mlx4_ib_event(struct notifier_block *this, unsigned long event, + void *ptr) { + struct mlx4_ib_dev *ibdev = + container_of(this, struct mlx4_ib_dev, mlx_nb); + struct mlx4_dev *dev = ibdev->dev; + unsigned long param = *(unsigned long *)ptr; struct ib_event ibev; - struct mlx4_ib_dev *ibdev = to_mdev((struct ib_device *) ibdev_ptr); struct mlx4_eqe *eqe = NULL; struct ib_event_work *ew; int p = 0; @@ -3187,11 +3200,11 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, (event == MLX4_DEV_EVENT_PORT_DOWN))) { ew = kmalloc(sizeof(*ew), GFP_ATOMIC); if (!ew) - return; + return NOTIFY_DONE; INIT_WORK(&ew->work, handle_bonded_port_state_event); ew->ib_dev = ibdev; queue_work(wq, &ew->work); - return; + return NOTIFY_DONE; } if (event == MLX4_DEV_EVENT_PORT_MGMT_CHANGE) @@ -3202,7 +3215,7 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, switch (event) { case MLX4_DEV_EVENT_PORT_UP: if (p > ibdev->num_ports) - return; + return NOTIFY_DONE; if (!mlx4_is_slave(dev) && rdma_port_get_link_layer(&ibdev->ib_dev, p) == IB_LINK_LAYER_INFINIBAND) { @@ -3217,7 +3230,7 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, case MLX4_DEV_EVENT_PORT_DOWN: if (p > ibdev->num_ports) - return; + return NOTIFY_DONE; ibev.event = IB_EVENT_PORT_ERR; break; @@ -3230,7 +3243,7 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, case MLX4_DEV_EVENT_PORT_MGMT_CHANGE: ew = kmalloc(sizeof *ew, GFP_ATOMIC); if (!ew) - return; + return NOTIFY_DONE; INIT_WORK(&ew->work, handle_port_mgmt_change_event); memcpy(&ew->ib_eqe, eqe, sizeof *eqe); @@ -3240,7 +3253,7 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, queue_work(wq, &ew->work); else handle_port_mgmt_change_event(&ew->work); - return; + return NOTIFY_DONE; case MLX4_DEV_EVENT_SLAVE_INIT: /* here, p is the slave id */ @@ -3256,7 +3269,7 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, 1); } } - return; + return NOTIFY_DONE; case MLX4_DEV_EVENT_SLAVE_SHUTDOWN: if (mlx4_is_master(dev)) { @@ -3272,22 +3285,22 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, } /* here, p is the slave id */ do_slave_init(ibdev, p, 0); - return; + return NOTIFY_DONE; default: - return; + return NOTIFY_DONE; } - ibev.device = ibdev_ptr; + ibev.device = &ibdev->ib_dev; ibev.element.port_num = mlx4_is_bonded(ibdev->dev) ? 1 : (u8)p; ib_dispatch_event(&ibev); + return NOTIFY_DONE; } static struct mlx4_interface mlx4_ib_interface = { .add = mlx4_ib_add, .remove = mlx4_ib_remove, - .event = mlx4_ib_event, .protocol = MLX4_PROT_IB_IPV6, .flags = MLX4_INTFF_BONDING }; diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h index 17fee1e73a45..41ca1114a995 100644 --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h @@ -38,6 +38,7 @@ #include #include #include +#include #include #include @@ -644,6 +645,7 @@ struct mlx4_ib_dev { spinlock_t reset_flow_resource_lock; struct list_head qp_list; struct mlx4_ib_diag_counters diag_counters[MLX4_DIAG_COUNTERS_TYPES]; + struct notifier_block mlx_nb; }; struct ib_event_work { diff --git a/drivers/net/ethernet/mellanox/mlx4/en_main.c b/drivers/net/ethernet/mellanox/mlx4/en_main.c index be8ba34c9025..77e771e97da0 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_main.c @@ -183,17 +183,20 @@ static void mlx4_en_get_profile(struct mlx4_en_dev *mdev) } } -static void mlx4_en_event(struct mlx4_dev *dev, void *endev_ptr, - enum mlx4_dev_event event, unsigned long port) +static int mlx4_en_event(struct notifier_block *this, unsigned long event, + void *ptr) { - struct mlx4_en_dev *mdev = (struct mlx4_en_dev *) endev_ptr; + struct mlx4_en_dev *mdev = + container_of(this, struct mlx4_en_dev, mlx_nb); + struct mlx4_dev *dev = mdev->dev; + unsigned long port = *(unsigned long *)ptr; struct mlx4_en_priv *priv; switch (event) { case MLX4_DEV_EVENT_PORT_UP: case MLX4_DEV_EVENT_PORT_DOWN: if (!mdev->pndev[port]) - return; + return NOTIFY_DONE; priv = netdev_priv(mdev->pndev[port]); /* To prevent races, we poll the link state in a separate task rather than changing it here */ @@ -211,10 +214,12 @@ static void mlx4_en_event(struct mlx4_dev *dev, void *endev_ptr, default: if (port < 1 || port > dev->caps.num_ports || !mdev->pndev[port]) - return; - mlx4_warn(mdev, "Unhandled event %d for port %d\n", event, + return NOTIFY_DONE; + mlx4_warn(mdev, "Unhandled event %d for port %d\n", (int) event, (int) port); } + + return NOTIFY_DONE; } static void mlx4_en_remove(struct mlx4_dev *dev, void *endev_ptr) @@ -222,6 +227,8 @@ static void mlx4_en_remove(struct mlx4_dev *dev, void *endev_ptr) struct mlx4_en_dev *mdev = endev_ptr; int i; + mlx4_unregister_event_notifier(dev, &mdev->mlx_nb); + mutex_lock(&mdev->state_lock); mdev->device_up = false; mutex_unlock(&mdev->state_lock); @@ -263,7 +270,7 @@ static void mlx4_en_activate(struct mlx4_dev *dev, void *ctx) static void *mlx4_en_add(struct mlx4_dev *dev) { struct mlx4_en_dev *mdev; - int i; + int err, i; printk_once(KERN_INFO "%s", mlx4_en_version); @@ -326,6 +333,11 @@ static void *mlx4_en_add(struct mlx4_dev *dev) mutex_init(&mdev->state_lock); mdev->device_up = true; + /* register mlx4 core notifier */ + mdev->mlx_nb.notifier_call = mlx4_en_event; + err = mlx4_register_event_notifier(dev, &mdev->mlx_nb); + WARN(err, "failed to register mlx4 event notifier (%d)", err); + return mdev; err_mr: @@ -346,7 +358,6 @@ static void *mlx4_en_add(struct mlx4_dev *dev) static struct mlx4_interface mlx4_en_interface = { .add = mlx4_en_add, .remove = mlx4_en_remove, - .event = mlx4_en_event, .protocol = MLX4_PROT_ETH, .activate = mlx4_en_activate, }; diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c index 28d7da925d36..a7c3e2efa464 100644 --- a/drivers/net/ethernet/mellanox/mlx4/intf.c +++ b/drivers/net/ethernet/mellanox/mlx4/intf.c @@ -183,17 +183,27 @@ void mlx4_dispatch_event(struct mlx4_dev *dev, enum mlx4_dev_event type, unsigned long param) { struct mlx4_priv *priv = mlx4_priv(dev); - struct mlx4_device_context *dev_ctx; - unsigned long flags; - spin_lock_irqsave(&priv->ctx_lock, flags); + atomic_notifier_call_chain(&priv->event_nh, type, ¶m); +} - list_for_each_entry(dev_ctx, &priv->ctx_list, list) - if (dev_ctx->intf->event) - dev_ctx->intf->event(dev, dev_ctx->context, type, param); +int mlx4_register_event_notifier(struct mlx4_dev *dev, + struct notifier_block *nb) +{ + struct mlx4_priv *priv = mlx4_priv(dev); - spin_unlock_irqrestore(&priv->ctx_lock, flags); + return atomic_notifier_chain_register(&priv->event_nh, nb); +} +EXPORT_SYMBOL(mlx4_register_event_notifier); + +int mlx4_unregister_event_notifier(struct mlx4_dev *dev, + struct notifier_block *nb) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + + return atomic_notifier_chain_unregister(&priv->event_nh, nb); } +EXPORT_SYMBOL(mlx4_unregister_event_notifier); int mlx4_register_device(struct mlx4_dev *dev) { diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c index 8a5409b00530..5f3ba8385e23 100644 --- a/drivers/net/ethernet/mellanox/mlx4/main.c +++ b/drivers/net/ethernet/mellanox/mlx4/main.c @@ -3378,6 +3378,8 @@ static int mlx4_load_one(struct pci_dev *pdev, int pci_dev_data, INIT_LIST_HEAD(&priv->ctx_list); spin_lock_init(&priv->ctx_lock); + ATOMIC_INIT_NOTIFIER_HEAD(&priv->event_nh); + mutex_init(&priv->port_mutex); mutex_init(&priv->bond_mutex); diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4.h b/drivers/net/ethernet/mellanox/mlx4/mlx4.h index 6ccf340660d9..10f12e4992f1 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4.h @@ -47,6 +47,7 @@ #include #include #include +#include #include #include @@ -878,6 +879,7 @@ struct mlx4_priv { struct list_head dev_list; struct list_head ctx_list; spinlock_t ctx_lock; + struct atomic_notifier_head event_nh; int pci_dev_data; int removed; diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h index 72a3fea36702..efe3f97b874f 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h @@ -49,6 +49,7 @@ #include #include #include +#include #include #include @@ -433,6 +434,7 @@ struct mlx4_en_dev { struct ptp_clock *ptp_clock; struct ptp_clock_info ptp_clock_info; struct notifier_block netdev_nb; + struct notifier_block mlx_nb; }; diff --git a/include/linux/mlx4/driver.h b/include/linux/mlx4/driver.h index 923951e19300..228da8ed7e75 100644 --- a/include/linux/mlx4/driver.h +++ b/include/linux/mlx4/driver.h @@ -34,6 +34,7 @@ #define MLX4_DRIVER_H #include +#include #include struct mlx4_dev; @@ -57,8 +58,6 @@ enum { struct mlx4_interface { void * (*add) (struct mlx4_dev *dev); void (*remove)(struct mlx4_dev *dev, void *context); - void (*event) (struct mlx4_dev *dev, void *context, - enum mlx4_dev_event event, unsigned long param); void (*activate)(struct mlx4_dev *dev, void *context); struct list_head list; enum mlx4_protocol protocol; @@ -87,6 +86,11 @@ struct mlx4_port_map { int mlx4_port_map_set(struct mlx4_dev *dev, struct mlx4_port_map *v2p); +int mlx4_register_event_notifier(struct mlx4_dev *dev, + struct notifier_block *nb); +int mlx4_unregister_event_notifier(struct mlx4_dev *dev, + struct notifier_block *nb); + struct devlink_port *mlx4_get_devlink_port(struct mlx4_dev *dev, int port); #endif /* MLX4_DRIVER_H */ From patchwork Sun Aug 13 14:51:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 135044 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp2249315vqi; Sun, 13 Aug 2023 08:14:46 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGRW/pnvWvo9s1P2cfyw7VQEoLMU7BNs+ptjMjoa7dUwUiRWZn/gls16OyLT4UkKBCjnCp4 X-Received: by 2002:a17:903:1cb:b0:1bb:bbda:70d9 with SMTP id e11-20020a17090301cb00b001bbbbda70d9mr6085333plh.63.1691939686308; Sun, 13 Aug 2023 08:14:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691939686; cv=none; d=google.com; s=arc-20160816; b=S/RM3SzCFqeutmERFTwEYbSlRnIFXppfSe/XSWwDQpurgLH+Lh7vkuzeIqiY/xQwPO DdqKCNtzO1zqnL7qo2wG+r4INjMwAXcuiQbZ0RQurQYN0aumqYZHItsKcKMlnkXbKATV 2xR0wYma+N4c/LX0adfmhlJTa2rGh4zZ51CfAmrbA4KMfQhPaYMwl2UkliPuSyG205wa su7/eZVXz6FgltB1LR/XDLa477sNrZn/WlPIKkg8qdm6MOkX02kF6aduKdLHjOYIYRnC w6Rn5F4jA5bnpD/733Iohn3w+Nr5C41rOQSjYKe6gvRJnS41i4FGBwBlyIc8LtpbpxwB 8qlA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=fI8AvByIo5bSABW0DcgMCL+LN/SUaCF+2Hx1xyjMwgA=; fh=thUxDq4m5D3FQct5425V7h171/dEmaHUwwAlFhAMPtU=; b=VgNfuBe1+T1oxALyT53/gTvE2vatZU4wayLJpxc0CM83AFqqnAXinav7oxItyBN9/l 8tY2NGrIN9X9tW+BWLVSOSzQw8nQbSuLLDwAJ8NClaljyZ68wgIn+2SzU6B6m1JiV2Xd FZq77XCQK6zRzu9Z66uUmQVpP8vMXbdqVdIch9htaQikD1XshoieStR9k3JlyCvh65du cTW1sWcuJv9tQvN54CPaUfDnr501zFTZ6mgpNaMOufLoHqyCa1SUWL1Ei36T1S8iWlDg cC3r15MYMym6XBf3u97nfR3JYJ421W422tRVHW1V1ENc86Gz8UIkYPdagoirAb45EbOf Uvgw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=ZZafM9K7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b7-20020a170902650700b001b87ecb0e5csi6427871plk.126.2023.08.13.08.14.33; Sun, 13 Aug 2023 08:14:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=ZZafM9K7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231335AbjHMOwI (ORCPT + 99 others); Sun, 13 Aug 2023 10:52:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231299AbjHMOwF (ORCPT ); Sun, 13 Aug 2023 10:52:05 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4983D170D; Sun, 13 Aug 2023 07:52:06 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id E47B61F8C4; Sun, 13 Aug 2023 14:52:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1691938324; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fI8AvByIo5bSABW0DcgMCL+LN/SUaCF+2Hx1xyjMwgA=; b=ZZafM9K7dk//twf52fN2gzsI9DNM8SK8i6xgQrB5k+AwlRpQ6vILuMmikLw59Vvti4/Ak3 mMmhPSaeKMgGHFPz7IgJG0qXTO7Mrn7uNqh55NFIDZbS0R1DEdEFx2N4JKQyXg61LJQK2h wb4/MWcHUbxqQuGrNkQYo7B/WxOQV8o= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id B15301322C; Sun, 13 Aug 2023 14:52:04 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id wGKPKhTu2GSDFAAAMHmgww (envelope-from ); Sun, 13 Aug 2023 14:52:04 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v2 04/10] mlx4: Get rid of the mlx4_interface.activate callback Date: Sun, 13 Aug 2023 16:51:21 +0200 Message-Id: <20230813145127.10653-5-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230813145127.10653-1-petr.pavlu@suse.com> References: <20230813145127.10653-1-petr.pavlu@suse.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774127348361945256 X-GMAIL-MSGID: 1774127348361945256 The mlx4_interface.activate callback was introduced in commit 79857cd31fe7 ("net/mlx4: Postpone the registration of net_device"). It dealt with a situation when a netdev notifier received a NETDEV_REGISTER event for a new net_device created by mlx4_en but the same device was not yet visible to mlx4_get_protocol_dev(). The callback can be removed now that mlx4_get_protocol_dev() is gone. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx4/en_main.c | 37 +++++++++----------- drivers/net/ethernet/mellanox/mlx4/intf.c | 2 -- include/linux/mlx4/driver.h | 1 - 3 files changed, 16 insertions(+), 24 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/en_main.c b/drivers/net/ethernet/mellanox/mlx4/en_main.c index 77e771e97da0..263f9274927b 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_main.c @@ -247,26 +247,6 @@ static void mlx4_en_remove(struct mlx4_dev *dev, void *endev_ptr) kfree(mdev); } -static void mlx4_en_activate(struct mlx4_dev *dev, void *ctx) -{ - int i; - struct mlx4_en_dev *mdev = ctx; - - /* Create a netdev for each port */ - mlx4_foreach_port(i, dev, MLX4_PORT_TYPE_ETH) { - mlx4_info(mdev, "Activating port:%d\n", i); - if (mlx4_en_init_netdev(mdev, i, &mdev->profile.prof[i])) - mdev->pndev[i] = NULL; - } - - /* register netdev notifier */ - mdev->netdev_nb.notifier_call = mlx4_en_netdev_event; - if (register_netdevice_notifier(&mdev->netdev_nb)) { - mdev->netdev_nb.notifier_call = NULL; - mlx4_err(mdev, "Failed to create netdev notifier\n"); - } -} - static void *mlx4_en_add(struct mlx4_dev *dev) { struct mlx4_en_dev *mdev; @@ -338,6 +318,22 @@ static void *mlx4_en_add(struct mlx4_dev *dev) err = mlx4_register_event_notifier(dev, &mdev->mlx_nb); WARN(err, "failed to register mlx4 event notifier (%d)", err); + /* Setup ports */ + + /* Create a netdev for each port */ + mlx4_foreach_port(i, dev, MLX4_PORT_TYPE_ETH) { + mlx4_info(mdev, "Activating port:%d\n", i); + if (mlx4_en_init_netdev(mdev, i, &mdev->profile.prof[i])) + mdev->pndev[i] = NULL; + } + + /* register netdev notifier */ + mdev->netdev_nb.notifier_call = mlx4_en_netdev_event; + if (register_netdevice_notifier(&mdev->netdev_nb)) { + mdev->netdev_nb.notifier_call = NULL; + mlx4_err(mdev, "Failed to create netdev notifier\n"); + } + return mdev; err_mr: @@ -359,7 +355,6 @@ static struct mlx4_interface mlx4_en_interface = { .add = mlx4_en_add, .remove = mlx4_en_remove, .protocol = MLX4_PROT_ETH, - .activate = mlx4_en_activate, }; static void mlx4_en_verify_params(void) diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c index a7c3e2efa464..8b2c1404cb66 100644 --- a/drivers/net/ethernet/mellanox/mlx4/intf.c +++ b/drivers/net/ethernet/mellanox/mlx4/intf.c @@ -64,8 +64,6 @@ static void mlx4_add_device(struct mlx4_interface *intf, struct mlx4_priv *priv) spin_lock_irq(&priv->ctx_lock); list_add_tail(&dev_ctx->list, &priv->ctx_list); spin_unlock_irq(&priv->ctx_lock); - if (intf->activate) - intf->activate(&priv->dev, dev_ctx->context); } else kfree(dev_ctx); diff --git a/include/linux/mlx4/driver.h b/include/linux/mlx4/driver.h index 228da8ed7e75..0f8c9ba4c574 100644 --- a/include/linux/mlx4/driver.h +++ b/include/linux/mlx4/driver.h @@ -58,7 +58,6 @@ enum { struct mlx4_interface { void * (*add) (struct mlx4_dev *dev); void (*remove)(struct mlx4_dev *dev, void *context); - void (*activate)(struct mlx4_dev *dev, void *context); struct list_head list; enum mlx4_protocol protocol; int flags; From patchwork Sun Aug 13 14:51:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 135046 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp2249447vqi; Sun, 13 Aug 2023 08:15:06 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH9GfeYYqdiWZd5JrQRwBKe8GCOYRrNasdpHU32JVESWLTGXjWNXTityuarPox1BqQj1yXc X-Received: by 2002:a05:6a00:1907:b0:686:6e90:a99b with SMTP id y7-20020a056a00190700b006866e90a99bmr6129837pfi.25.1691939706090; Sun, 13 Aug 2023 08:15:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691939706; cv=none; d=google.com; s=arc-20160816; b=m57SAa7itoBFkE/21fGWvpVVT+WhrRjS1EEz1JhpOsFJl6Mqpad8OCF1GMBEdVDA4U F4ph+LxEJdMLHdKRS2r95sWRqHtldNXRnUOwUUPPlsxV8GYnxoLcgTAkZOqO2V23w2Er NEwjtIYVoF+32GZsexO34CaBqLMLtJxSTCJQDZOw7QPLJprdNRGuoh4jv1jXW0Oi2yKV ZWDEN8Mq5/AH1TWSuWy/ZkgACQ3F5ONXtstyl3y7v+K5tgvEv8wjuWGAWPgcBRSMzs7y SWOiwQnWv3lqakinQGrL9+T87EyGX5tzd4pLghWdPzXofGpePvsEdXLOQ/FVzjZOZMpz j3kQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=sGgjaLs4VvXuRgt+3H0LAGi5Jo3X5YW0I5eRhGrfil0=; fh=thUxDq4m5D3FQct5425V7h171/dEmaHUwwAlFhAMPtU=; b=lETTZtMP0P9rdLv3dPH02mfN4vqyBo5fS8+dgphuN/KxUzDAghovnYY5TakgA5JU0c /Q1yjsZ7QmtsaXewuM2QPkoacr6RtzcqKcuuP5KIR3GmDbtbrjbtUIaF65DzbHfKSRNF 2mibrZc0PYAVBM8Y4H/KycpNBuDdpc4exWxCT0kPOJ40xanYSdd+GC/PnUnel8SSkPLU XW6QogzzUyz9o0hWkLFoilRIJX+w2lbCaA2L0Ggh3RCG8fbPwLNADIGT8dgJ8eauxzSK IiTaJv7rpSkhfvt9a4pAvBPFVaeyZ115vvJGso3FWBGbDYr++8n1U/C0a2MlY3Lbogf5 T6sg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=ASIsQsts; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p13-20020a056a000a0d00b00679a6ce03f6si6678564pfh.59.2023.08.13.08.14.51; Sun, 13 Aug 2023 08:15:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=ASIsQsts; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231381AbjHMOwM (ORCPT + 99 others); Sun, 13 Aug 2023 10:52:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231302AbjHMOwG (ORCPT ); Sun, 13 Aug 2023 10:52:06 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 917641710; Sun, 13 Aug 2023 07:52:06 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 2F0F61F8D4; Sun, 13 Aug 2023 14:52:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1691938325; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sGgjaLs4VvXuRgt+3H0LAGi5Jo3X5YW0I5eRhGrfil0=; b=ASIsQstsRPkd9zbH/HrDDsPK2IJ+AhPqvrkeyCQXNiMDf5qozopSLMKCs0f5W2MYHEE2kF jH8DmbcsWew07ze+PJkE6w/JhlpFzqh0leDzYjzVenb5nRyRyFNczI0PS+u7q3P+VeJNFh NTSJ17QlKktQiYRRCJP3N5wQGwt/vOo= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id E876C13593; Sun, 13 Aug 2023 14:52:04 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id GEbxNxTu2GSDFAAAMHmgww (envelope-from ); Sun, 13 Aug 2023 14:52:04 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v2 05/10] mlx4: Move the bond work to the core driver Date: Sun, 13 Aug 2023 16:51:22 +0200 Message-Id: <20230813145127.10653-6-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230813145127.10653-1-petr.pavlu@suse.com> References: <20230813145127.10653-1-petr.pavlu@suse.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774127369480665235 X-GMAIL-MSGID: 1774127369480665235 Function mlx4_en_queue_bond_work() is used in mlx4_en to start a bond reconfiguration. It gathers data about a new port map setting, takes a reference on the netdev that triggered the change and queues a work object on mlx4_en_priv.mdev.workqueue to perform the operation. The scheduled work is mlx4_en_bond_work() which calls mlx4_bond()/mlx4_unbond() and consequently mlx4_do_bond(). At the same time, function mlx4_change_port_types() in mlx4_core might be invoked to change the port type configuration. As part of its logic, it re-registers the whole device by calling mlx4_unregister_device(), followed by mlx4_register_device(). The two operations can result in concurrent access to the data about currently active interfaces on the device. Functions mlx4_register_device() and mlx4_unregister_device() lock the intf_mutex to gain exclusive access to this data. The current implementation of mlx4_do_bond() doesn't do that which could result in an unexpected behavior. An updated version of mlx4_do_bond() for use with an auxiliary bus goes and locks the intf_mutex when accessing a new auxiliary device array. However, doing so can then result in the following deadlock: * A two-port mlx4 device is configured as an Ethernet bond. * One of the ports is changed from eth to ib, for instance, by writing into a mlx4_port sysfs attribute file. * mlx4_change_port_types() is called to update port types. It invokes mlx4_unregister_device() to unregister the device which locks the intf_mutex and starts removing all associated interfaces. * Function mlx4_en_remove() gets invoked and starts destroying its first netdev. This triggers mlx4_en_netdev_event() which recognizes that the configured bond is broken. It runs mlx4_en_queue_bond_work() which takes a reference on the netdev. Removing the netdev now cannot proceed until the work is completed. * Work function mlx4_en_bond_work() gets scheduled. It calls mlx4_unbond() -> mlx4_do_bond(). The latter function tries to lock the intf_mutex but that is not possible because it is held already by mlx4_unregister_device(). This particular case could be possibly solved by unregistering the mlx4_en_netdev_event() notifier in mlx4_en_remove() earlier, but it seems better to decouple mlx4_en more and break this reference order. Avoid then this scenario by recognizing that the bond reconfiguration operates only on a mlx4_dev. The logic to queue and execute the bond work can be moved into the mlx4_core driver. Only a reference on the respective mlx4_dev object is needed to be taken during the work's lifetime. This removes a call from mlx4_en that can directly result in needing to lock the intf_mutex, it remains a privilege of the core driver. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- .../net/ethernet/mellanox/mlx4/en_netdev.c | 62 +----------------- drivers/net/ethernet/mellanox/mlx4/main.c | 65 +++++++++++++++++-- drivers/net/ethernet/mellanox/mlx4/mlx4.h | 5 ++ include/linux/mlx4/device.h | 13 ++++ include/linux/mlx4/driver.h | 19 ------ 5 files changed, 77 insertions(+), 87 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c index 7066c426b95c..33bbcced8105 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c @@ -2894,63 +2894,6 @@ static const struct xdp_metadata_ops mlx4_xdp_metadata_ops = { .xmo_rx_hash = mlx4_en_xdp_rx_hash, }; -struct mlx4_en_bond { - struct work_struct work; - struct mlx4_en_priv *priv; - int is_bonded; - struct mlx4_port_map port_map; -}; - -static void mlx4_en_bond_work(struct work_struct *work) -{ - struct mlx4_en_bond *bond = container_of(work, - struct mlx4_en_bond, - work); - int err = 0; - struct mlx4_dev *dev = bond->priv->mdev->dev; - - if (bond->is_bonded) { - if (!mlx4_is_bonded(dev)) { - err = mlx4_bond(dev); - if (err) - en_err(bond->priv, "Fail to bond device\n"); - } - if (!err) { - err = mlx4_port_map_set(dev, &bond->port_map); - if (err) - en_err(bond->priv, "Fail to set port map [%d][%d]: %d\n", - bond->port_map.port1, - bond->port_map.port2, - err); - } - } else if (mlx4_is_bonded(dev)) { - err = mlx4_unbond(dev); - if (err) - en_err(bond->priv, "Fail to unbond device\n"); - } - dev_put(bond->priv->dev); - kfree(bond); -} - -static int mlx4_en_queue_bond_work(struct mlx4_en_priv *priv, int is_bonded, - u8 v2p_p1, u8 v2p_p2) -{ - struct mlx4_en_bond *bond; - - bond = kzalloc(sizeof(*bond), GFP_ATOMIC); - if (!bond) - return -ENOMEM; - - INIT_WORK(&bond->work, mlx4_en_bond_work); - bond->priv = priv; - bond->is_bonded = is_bonded; - bond->port_map.port1 = v2p_p1; - bond->port_map.port2 = v2p_p2; - dev_hold(priv->dev); - queue_work(priv->mdev->workqueue, &bond->work); - return 0; -} - int mlx4_en_netdev_event(struct notifier_block *this, unsigned long event, void *ptr) { @@ -2960,7 +2903,6 @@ int mlx4_en_netdev_event(struct notifier_block *this, struct mlx4_dev *dev; int i, num_eth_ports = 0; bool do_bond = true; - struct mlx4_en_priv *priv; u8 v2p_port1 = 0; u8 v2p_port2 = 0; @@ -2995,7 +2937,6 @@ int mlx4_en_netdev_event(struct notifier_block *this, if ((do_bond && (event != NETDEV_BONDING_INFO)) || !port) return NOTIFY_DONE; - priv = netdev_priv(ndev); if (do_bond) { struct netdev_notifier_bonding_info *notifier_info = ptr; struct netdev_bonding_info *bonding_info = @@ -3062,8 +3003,7 @@ int mlx4_en_netdev_event(struct notifier_block *this, } } - mlx4_en_queue_bond_work(priv, do_bond, - v2p_port1, v2p_port2); + mlx4_queue_bond_work(dev, do_bond, v2p_port1, v2p_port2); return NOTIFY_DONE; } diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c index 5f3ba8385e23..0ed490b99163 100644 --- a/drivers/net/ethernet/mellanox/mlx4/main.c +++ b/drivers/net/ethernet/mellanox/mlx4/main.c @@ -1441,7 +1441,7 @@ static int mlx4_mf_unbond(struct mlx4_dev *dev) return ret; } -int mlx4_bond(struct mlx4_dev *dev) +static int mlx4_bond(struct mlx4_dev *dev) { int ret = 0; struct mlx4_priv *priv = mlx4_priv(dev); @@ -1467,9 +1467,8 @@ int mlx4_bond(struct mlx4_dev *dev) return ret; } -EXPORT_SYMBOL_GPL(mlx4_bond); -int mlx4_unbond(struct mlx4_dev *dev) +static int mlx4_unbond(struct mlx4_dev *dev) { int ret = 0; struct mlx4_priv *priv = mlx4_priv(dev); @@ -1496,10 +1495,8 @@ int mlx4_unbond(struct mlx4_dev *dev) return ret; } -EXPORT_SYMBOL_GPL(mlx4_unbond); - -int mlx4_port_map_set(struct mlx4_dev *dev, struct mlx4_port_map *v2p) +static int mlx4_port_map_set(struct mlx4_dev *dev, struct mlx4_port_map *v2p) { u8 port1 = v2p->port1; u8 port2 = v2p->port2; @@ -1541,7 +1538,61 @@ int mlx4_port_map_set(struct mlx4_dev *dev, struct mlx4_port_map *v2p) mutex_unlock(&priv->bond_mutex); return err; } -EXPORT_SYMBOL_GPL(mlx4_port_map_set); + +struct mlx4_bond { + struct work_struct work; + struct mlx4_dev *dev; + int is_bonded; + struct mlx4_port_map port_map; +}; + +static void mlx4_bond_work(struct work_struct *work) +{ + struct mlx4_bond *bond = container_of(work, struct mlx4_bond, work); + int err = 0; + + if (bond->is_bonded) { + if (!mlx4_is_bonded(bond->dev)) { + err = mlx4_bond(bond->dev); + if (err) + mlx4_err(bond->dev, "Fail to bond device\n"); + } + if (!err) { + err = mlx4_port_map_set(bond->dev, &bond->port_map); + if (err) + mlx4_err(bond->dev, + "Fail to set port map [%d][%d]: %d\n", + bond->port_map.port1, + bond->port_map.port2, err); + } + } else if (mlx4_is_bonded(bond->dev)) { + err = mlx4_unbond(bond->dev); + if (err) + mlx4_err(bond->dev, "Fail to unbond device\n"); + } + put_device(&bond->dev->persist->pdev->dev); + kfree(bond); +} + +int mlx4_queue_bond_work(struct mlx4_dev *dev, int is_bonded, u8 v2p_p1, + u8 v2p_p2) +{ + struct mlx4_bond *bond; + + bond = kzalloc(sizeof(*bond), GFP_ATOMIC); + if (!bond) + return -ENOMEM; + + INIT_WORK(&bond->work, mlx4_bond_work); + get_device(&dev->persist->pdev->dev); + bond->dev = dev; + bond->is_bonded = is_bonded; + bond->port_map.port1 = v2p_p1; + bond->port_map.port2 = v2p_p2; + queue_work(mlx4_wq, &bond->work); + return 0; +} +EXPORT_SYMBOL(mlx4_queue_bond_work); static int mlx4_load_fw(struct mlx4_dev *dev) { diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4.h b/drivers/net/ethernet/mellanox/mlx4/mlx4.h index 10f12e4992f1..ece9acb6a869 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4.h @@ -863,6 +863,11 @@ struct mlx4_steer { struct list_head steer_entries[MLX4_NUM_STEERS]; }; +struct mlx4_port_map { + u8 port1; + u8 port2; +}; + enum { MLX4_PCI_DEV_IS_VF = 1 << 0, MLX4_PCI_DEV_FORCE_SENSE_PORT = 1 << 1, diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h index 6646634a0b9d..049d8a4b044d 100644 --- a/include/linux/mlx4/device.h +++ b/include/linux/mlx4/device.h @@ -1087,6 +1087,19 @@ static inline void *mlx4_buf_offset(struct mlx4_buf *buf, int offset) (offset & (PAGE_SIZE - 1)); } +static inline int mlx4_is_bonded(struct mlx4_dev *dev) +{ + return !!(dev->flags & MLX4_FLAG_BONDED); +} + +static inline int mlx4_is_mf_bonded(struct mlx4_dev *dev) +{ + return (mlx4_is_bonded(dev) && mlx4_is_mfunc(dev)); +} + +int mlx4_queue_bond_work(struct mlx4_dev *dev, int is_bonded, u8 v2p_p1, + u8 v2p_p2); + int mlx4_pd_alloc(struct mlx4_dev *dev, u32 *pdn); void mlx4_pd_free(struct mlx4_dev *dev, u32 pdn); int mlx4_xrcd_alloc(struct mlx4_dev *dev, u32 *xrcdn); diff --git a/include/linux/mlx4/driver.h b/include/linux/mlx4/driver.h index 0f8c9ba4c574..781d5a0c2faa 100644 --- a/include/linux/mlx4/driver.h +++ b/include/linux/mlx4/driver.h @@ -66,25 +66,6 @@ struct mlx4_interface { int mlx4_register_interface(struct mlx4_interface *intf); void mlx4_unregister_interface(struct mlx4_interface *intf); -int mlx4_bond(struct mlx4_dev *dev); -int mlx4_unbond(struct mlx4_dev *dev); -static inline int mlx4_is_bonded(struct mlx4_dev *dev) -{ - return !!(dev->flags & MLX4_FLAG_BONDED); -} - -static inline int mlx4_is_mf_bonded(struct mlx4_dev *dev) -{ - return (mlx4_is_bonded(dev) && mlx4_is_mfunc(dev)); -} - -struct mlx4_port_map { - u8 port1; - u8 port2; -}; - -int mlx4_port_map_set(struct mlx4_dev *dev, struct mlx4_port_map *v2p); - int mlx4_register_event_notifier(struct mlx4_dev *dev, struct notifier_block *nb); int mlx4_unregister_event_notifier(struct mlx4_dev *dev, From patchwork Sun Aug 13 14:51:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 135058 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp2269170vqi; Sun, 13 Aug 2023 09:05:06 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHAnVPaMM7UIKO5uYt/3dn8GrEW63c0UPPW0bzv0QlCThEenJymgvPC+gwHJAc9CKDkTKom X-Received: by 2002:a05:6a21:7189:b0:140:7b6a:6910 with SMTP id wq9-20020a056a21718900b001407b6a6910mr8360275pzb.34.1691942705962; Sun, 13 Aug 2023 09:05:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691942705; cv=none; d=google.com; s=arc-20160816; b=KgWgiRdH7fn9Mwzg7UvTkvoTrP9K9laznjQwGtWsqfrUAzAvjreRkvuYbQ0diSBn/9 hlqz+9eddXizKJLFFNR5U6IEPrrfZFj+0R7C81K49b7gMAO3JHNGcxloCjgEX/GbU87C vPr26KyOy2FBM5K+IycadZg30AoHnDuGkl76qrM5jkBj+/wwaDgACCzrAJKQwXV0KDsV uFzU6BcHnqRAyyOdSNUsVj/7926Ay/owb/WqHTOzgJVB+XvhLdEzma6tD8jtvO8Wun6H pS1Xdi6u5rj7OGfJZbeOTSqPtcxzPn2lKIHGIUjkC0KB9olsYtNwBhAmtj0X0IuhjJR5 czTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=O9Q66lEIFKkRcUzjXIAp7yKb9m2EclBPlOJJKm5nUB4=; fh=thUxDq4m5D3FQct5425V7h171/dEmaHUwwAlFhAMPtU=; b=zgNLgsPoEYgxRL7xEPmrBEfnSyr2cGnRYTpJRnr+ujBHAkEnqOwAj0REcqgwvCPeEr /4q8+9ebNpInAl4QejAfEN+P0SRoaB17lz3N56tGpezdBS09PnZuStedYB+xlnBg5Fho m9ST8bn8dpcz6y1Om9Uf6Ue1R0sNpRskZmMB6x2ZToUu1eMs7KB9+pdwyJMq2bAKFWDk M39C0gxV1rfFSC3uFO6yB/P2nigw3bCnjo4auwsE7SUaAgJ4nUa6Fv7Gy0MWIRztf83b GpSNv57wz3yEuX4mMXphjalchYs8jv4Xcch55pVWqN4m4ERliswg0pxRa6F1v9oHWaOC rCXw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=vSobd0sP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a7-20020a634d07000000b00563e0e86f17si6814151pgb.573.2023.08.13.09.04.52; Sun, 13 Aug 2023 09:05:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=vSobd0sP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231443AbjHMOwi (ORCPT + 99 others); Sun, 13 Aug 2023 10:52:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59962 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231312AbjHMOwH (ORCPT ); Sun, 13 Aug 2023 10:52:07 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC15F170D; Sun, 13 Aug 2023 07:52:08 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 691691F8D5; Sun, 13 Aug 2023 14:52:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1691938325; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O9Q66lEIFKkRcUzjXIAp7yKb9m2EclBPlOJJKm5nUB4=; b=vSobd0sP/xXfZQBlkq7rVosb8E8cggVOWqNUGH+NY+5Y4IwjB5kFb1VWTaGQ6hGPhlmS4+ TWivho1Kiu/eolnKcgwfnik2cYET8NQkPIJq6ljgbsvgqHpPX3YwGsV2cX96DdohGVUMEr 70+v8KuYVCTIva1xGcJxeZ1ghk8V0/E= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 325F71322C; Sun, 13 Aug 2023 14:52:05 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id EEeJCxXu2GSDFAAAMHmgww (envelope-from ); Sun, 13 Aug 2023 14:52:05 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v2 06/10] mlx4: Avoid resetting MLX4_INTFF_BONDING per driver Date: Sun, 13 Aug 2023 16:51:23 +0200 Message-Id: <20230813145127.10653-7-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230813145127.10653-1-petr.pavlu@suse.com> References: <20230813145127.10653-1-petr.pavlu@suse.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774130514783463714 X-GMAIL-MSGID: 1774130514783463714 The mlx4_core driver has a logic that allows a sub-driver to set the MLX4_INTFF_BONDING flag which then causes that function mlx4_do_bond() asks the sub-driver to fully re-probe a device when its bonding configuration changes. Performing this operation is disallowed in mlx4_register_interface() when it is detected that any mlx4 device is multifunction (SRIOV). The code then resets MLX4_INTFF_BONDING in the driver flags. Move this check directly into mlx4_do_bond(). It provides a better separation as mlx4_core no longer directly modifies the sub-driver flags and it will allow to get rid of explicitly keeping track of all mlx4 devices by the intf.c code when it is switched to an auxiliary bus. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx4/intf.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c index 8b2c1404cb66..30aead34ce08 100644 --- a/drivers/net/ethernet/mellanox/mlx4/intf.c +++ b/drivers/net/ethernet/mellanox/mlx4/intf.c @@ -96,11 +96,6 @@ int mlx4_register_interface(struct mlx4_interface *intf) list_add_tail(&intf->list, &intf_list); list_for_each_entry(priv, &dev_list, dev_list) { - if (mlx4_is_mfunc(&priv->dev) && (intf->flags & MLX4_INTFF_BONDING)) { - mlx4_dbg(&priv->dev, - "SRIOV, disabling HA mode for intf proto %d\n", intf->protocol); - intf->flags &= ~MLX4_INTFF_BONDING; - } mlx4_add_device(intf, priv); } @@ -155,10 +150,18 @@ int mlx4_do_bond(struct mlx4_dev *dev, bool enable) spin_lock_irqsave(&priv->ctx_lock, flags); list_for_each_entry_safe(dev_ctx, temp_dev_ctx, &priv->ctx_list, list) { - if (dev_ctx->intf->flags & MLX4_INTFF_BONDING) { - list_add_tail(&dev_ctx->bond_list, &bond_list); - list_del(&dev_ctx->list); + if (!(dev_ctx->intf->flags & MLX4_INTFF_BONDING)) + continue; + + if (mlx4_is_mfunc(dev)) { + mlx4_dbg(dev, + "SRIOV, disabled HA mode for intf proto %d\n", + dev_ctx->intf->protocol); + continue; } + + list_add_tail(&dev_ctx->bond_list, &bond_list); + list_del(&dev_ctx->list); } spin_unlock_irqrestore(&priv->ctx_lock, flags); From patchwork Sun Aug 13 14:51:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 135039 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp2239664vqi; Sun, 13 Aug 2023 07:55:38 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHaj97vdb8JB5h11y0gprSBOnhtF/pBYEZ+CtyPJ25/9yHfjCZXa0R8hUEIDOK8JP9bRIUO X-Received: by 2002:aa7:d054:0:b0:522:ca6b:ad7d with SMTP id n20-20020aa7d054000000b00522ca6bad7dmr5488144edo.9.1691938538161; Sun, 13 Aug 2023 07:55:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691938538; cv=none; d=google.com; s=arc-20160816; b=pEmIe/Dq0vYieqb5KKiqm8lwJg1FdeKYr0lhhpQr76UN7gF6Lohu1P9DLaceayE+l5 bOFcSdtekElJRA4MEmMlx16V+tzF4+nqCT5lxoBdT58b5cpLPJIsDChiJ0en19E1Zbqx yZcw/c+VLZPIZM68UR8a0M4nW7NpfKdbaJVkHHZe9seQIFn4btoyg4b1/V72crYEdKcl W6Ils2NkmKPXzwksQHKNY8I3HNPRiGElLBI0lyEPWfbQ3XcNuYChBB7x6IOYeaSwg+aX lLEM3voaKjy3hdba7VeeqgmdZ7J+GzDFrJVZjHXE5LoLqbvAjhdXwXE5aDUQYFog2al/ pN5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=s1HayEWlygkespJ1nUcUnEStXGw5l8zaePkgciVxiOg=; fh=thUxDq4m5D3FQct5425V7h171/dEmaHUwwAlFhAMPtU=; b=VUwO5YGRhdBDWJLGGqagw7q5daqPV5Iad8hb6tbKSADUvClClX5BQ26FFBSI6F8rTf l+PZHL0TyOOMXz7LfUnNZFvQMLEdnRwWqi4jX1R4/MvxzcSRdVYGNtvwAeR175IN70vM Cbn/VqX1zEsxtZSX//gL49m0pqccv46zJgQBl7Ym0SpV8Ou+I0pH0ggnyOGNTypkJC2K 9tzzV2My13RLruOvZP90WuHa6J80tohi1NKu3FIyVbxPHVHpiJj3RyygyExYkMHM0Y6y awVByIIkaR+1C+5DVmAe/VnD3ek+dnSBjbiZwyLfPFnjaUPlZ5XUsU4vnJ8B9Da8NKbL vBkA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=kMoB+yLV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ay9-20020a056402202900b005231db540d4si6699547edb.230.2023.08.13.07.55.14; Sun, 13 Aug 2023 07:55:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=kMoB+yLV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231360AbjHMOwf (ORCPT + 99 others); Sun, 13 Aug 2023 10:52:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60020 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231409AbjHMOwT (ORCPT ); Sun, 13 Aug 2023 10:52:19 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC368171D; Sun, 13 Aug 2023 07:52:08 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id A6AB221976; Sun, 13 Aug 2023 14:52:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1691938325; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=s1HayEWlygkespJ1nUcUnEStXGw5l8zaePkgciVxiOg=; b=kMoB+yLVAiNwQAabZsHRHJqMbQ19XkLCToM61wJWb+ka0YDTubadbGK4IlEl4I7jxmMlbf SJM48izMEHPiHAl4XHo8LJgb6w3/Ev1txWzmsiGRnh74dZWIY+77AUFP2Tupd7HC82pOri tLE+tMGQ/oZx/3ltaWK+Q53XYKvPhic= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 6CD0613593; Sun, 13 Aug 2023 14:52:05 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 0D3YGRXu2GSDFAAAMHmgww (envelope-from ); Sun, 13 Aug 2023 14:52:05 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v2 07/10] mlx4: Register mlx4 devices to an auxiliary virtual bus Date: Sun, 13 Aug 2023 16:51:24 +0200 Message-Id: <20230813145127.10653-8-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230813145127.10653-1-petr.pavlu@suse.com> References: <20230813145127.10653-1-petr.pavlu@suse.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED,SPF_PASS, T_SPF_HELO_TEMPERROR autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774126144571565688 X-GMAIL-MSGID: 1774126144571565688 Add an auxiliary virtual bus to model the mlx4 driver structure. The code is added along the current custom device management logic. Subsequent patches switch mlx4_en and mlx4_ib to the auxiliary bus and the old interface is then removed. Structure mlx4_priv gains a new adev dynamic array to keep track of its auxiliary devices. Access to the array is protected by the global mlx4_intf mutex. Functions mlx4_register_device() and mlx4_unregister_device() are updated to expose auxiliary devices on the bus in order to load mlx4_en and/or mlx4_ib. Functions mlx4_register_auxiliary_driver() and mlx4_unregister_auxiliary_driver() are added to substitute mlx4_register_interface() and mlx4_unregister_interface(), respectively. Function mlx4_do_bond() is adjusted to walk over the adev array and re-adds a specific auxiliary device if its driver sets the MLX4_INTFF_BONDING flag. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx4/Kconfig | 1 + drivers/net/ethernet/mellanox/mlx4/intf.c | 230 ++++++++++++++++++++- drivers/net/ethernet/mellanox/mlx4/main.c | 17 +- drivers/net/ethernet/mellanox/mlx4/mlx4.h | 6 + include/linux/mlx4/device.h | 7 + include/linux/mlx4/driver.h | 11 + 6 files changed, 268 insertions(+), 4 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/Kconfig b/drivers/net/ethernet/mellanox/mlx4/Kconfig index 1b4b1f642317..825e05fb8607 100644 --- a/drivers/net/ethernet/mellanox/mlx4/Kconfig +++ b/drivers/net/ethernet/mellanox/mlx4/Kconfig @@ -27,6 +27,7 @@ config MLX4_EN_DCB config MLX4_CORE tristate depends on PCI + select AUXILIARY_BUS select NET_DEVLINK default n diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c index 30aead34ce08..4b1e18e4a682 100644 --- a/drivers/net/ethernet/mellanox/mlx4/intf.c +++ b/drivers/net/ethernet/mellanox/mlx4/intf.c @@ -48,6 +48,89 @@ struct mlx4_device_context { static LIST_HEAD(intf_list); static LIST_HEAD(dev_list); static DEFINE_MUTEX(intf_mutex); +static DEFINE_IDA(mlx4_adev_ida); + +static const struct mlx4_adev_device { + const char *suffix; + bool (*is_supported)(struct mlx4_dev *dev); +} mlx4_adev_devices[1] = {}; + +int mlx4_adev_init(struct mlx4_dev *dev) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + + priv->adev_idx = ida_alloc(&mlx4_adev_ida, GFP_KERNEL); + if (priv->adev_idx < 0) + return priv->adev_idx; + + priv->adev = kcalloc(ARRAY_SIZE(mlx4_adev_devices), + sizeof(struct mlx4_adev *), GFP_KERNEL); + if (!priv->adev) { + ida_free(&mlx4_adev_ida, priv->adev_idx); + return -ENOMEM; + } + + return 0; +} + +void mlx4_adev_cleanup(struct mlx4_dev *dev) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + + kfree(priv->adev); + ida_free(&mlx4_adev_ida, priv->adev_idx); +} + +static void adev_release(struct device *dev) +{ + struct mlx4_adev *mlx4_adev = + container_of(dev, struct mlx4_adev, adev.dev); + struct mlx4_priv *priv = mlx4_priv(mlx4_adev->mdev); + int idx = mlx4_adev->idx; + + kfree(mlx4_adev); + priv->adev[idx] = NULL; +} + +static struct mlx4_adev *add_adev(struct mlx4_dev *dev, int idx) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + const char *suffix = mlx4_adev_devices[idx].suffix; + struct auxiliary_device *adev; + struct mlx4_adev *madev; + int ret; + + madev = kzalloc(sizeof(*madev), GFP_KERNEL); + if (!madev) + return ERR_PTR(-ENOMEM); + + adev = &madev->adev; + adev->id = priv->adev_idx; + adev->name = suffix; + adev->dev.parent = &dev->persist->pdev->dev; + adev->dev.release = adev_release; + madev->mdev = dev; + madev->idx = idx; + + ret = auxiliary_device_init(adev); + if (ret) { + kfree(madev); + return ERR_PTR(ret); + } + + ret = auxiliary_device_add(adev); + if (ret) { + auxiliary_device_uninit(adev); + return ERR_PTR(ret); + } + return madev; +} + +static void del_adev(struct auxiliary_device *adev) +{ + auxiliary_device_delete(adev); + auxiliary_device_uninit(adev); +} static void mlx4_add_device(struct mlx4_interface *intf, struct mlx4_priv *priv) { @@ -120,12 +203,24 @@ void mlx4_unregister_interface(struct mlx4_interface *intf) } EXPORT_SYMBOL_GPL(mlx4_unregister_interface); +int mlx4_register_auxiliary_driver(struct mlx4_adrv *madrv) +{ + return auxiliary_driver_register(&madrv->adrv); +} +EXPORT_SYMBOL_GPL(mlx4_register_auxiliary_driver); + +void mlx4_unregister_auxiliary_driver(struct mlx4_adrv *madrv) +{ + auxiliary_driver_unregister(&madrv->adrv); +} +EXPORT_SYMBOL_GPL(mlx4_unregister_auxiliary_driver); + int mlx4_do_bond(struct mlx4_dev *dev, bool enable) { struct mlx4_priv *priv = mlx4_priv(dev); struct mlx4_device_context *dev_ctx = NULL, *temp_dev_ctx; unsigned long flags; - int ret; + int i, ret; LIST_HEAD(bond_list); if (!(dev->caps.flags2 & MLX4_DEV_CAP_FLAG2_PORT_REMAP)) @@ -177,6 +272,57 @@ int mlx4_do_bond(struct mlx4_dev *dev, bool enable) dev_ctx->intf->protocol, enable ? "enabled" : "disabled"); } + + mutex_lock(&intf_mutex); + + for (i = 0; i < ARRAY_SIZE(mlx4_adev_devices); i++) { + struct mlx4_adev *madev = priv->adev[i]; + struct mlx4_adrv *madrv; + enum mlx4_protocol protocol; + + if (!madev) + continue; + + device_lock(&madev->adev.dev); + if (!madev->adev.dev.driver) { + device_unlock(&madev->adev.dev); + continue; + } + + madrv = container_of(madev->adev.dev.driver, struct mlx4_adrv, + adrv.driver); + if (!(madrv->flags & MLX4_INTFF_BONDING)) { + device_unlock(&madev->adev.dev); + continue; + } + + if (mlx4_is_mfunc(dev)) { + mlx4_dbg(dev, + "SRIOV, disabled HA mode for intf proto %d\n", + madrv->protocol); + device_unlock(&madev->adev.dev); + continue; + } + + protocol = madrv->protocol; + device_unlock(&madev->adev.dev); + + del_adev(&madev->adev); + priv->adev[i] = add_adev(dev, i); + if (IS_ERR(priv->adev[i])) { + mlx4_warn(dev, "Device[%d] (%s) failed to load\n", i, + mlx4_adev_devices[i].suffix); + priv->adev[i] = NULL; + continue; + } + + mlx4_dbg(dev, + "Interface for protocol %d restarted with bonded mode %s\n", + protocol, enable ? "enabled" : "disabled"); + } + + mutex_unlock(&intf_mutex); + return 0; } @@ -206,10 +352,80 @@ int mlx4_unregister_event_notifier(struct mlx4_dev *dev, } EXPORT_SYMBOL(mlx4_unregister_event_notifier); +static int add_drivers(struct mlx4_dev *dev) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + int i, ret = 0; + + for (i = 0; i < ARRAY_SIZE(mlx4_adev_devices); i++) { + bool is_supported = false; + + if (priv->adev[i]) + continue; + + if (mlx4_adev_devices[i].is_supported) + is_supported = mlx4_adev_devices[i].is_supported(dev); + + if (!is_supported) + continue; + + priv->adev[i] = add_adev(dev, i); + if (IS_ERR(priv->adev[i])) { + mlx4_warn(dev, "Device[%d] (%s) failed to load\n", i, + mlx4_adev_devices[i].suffix); + /* We continue to rescan drivers and leave to the caller + * to make decision if to release everything or + * continue. */ + ret = PTR_ERR(priv->adev[i]); + priv->adev[i] = NULL; + } + } + return ret; +} + +static void delete_drivers(struct mlx4_dev *dev) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + bool delete_all; + int i; + + delete_all = !(dev->persist->interface_state & MLX4_INTERFACE_STATE_UP); + + for (i = ARRAY_SIZE(mlx4_adev_devices) - 1; i >= 0; i--) { + bool is_supported = false; + + if (!priv->adev[i]) + continue; + + if (mlx4_adev_devices[i].is_supported && !delete_all) + is_supported = mlx4_adev_devices[i].is_supported(dev); + + if (is_supported) + continue; + + del_adev(&priv->adev[i]->adev); + priv->adev[i] = NULL; + } +} + +/* This function is used after mlx4_dev is reconfigured. + */ +static int rescan_drivers_locked(struct mlx4_dev *dev) +{ + lockdep_assert_held(&intf_mutex); + + delete_drivers(dev); + if (!(dev->persist->interface_state & MLX4_INTERFACE_STATE_UP)) + return 0; + + return add_drivers(dev); +} + int mlx4_register_device(struct mlx4_dev *dev) { struct mlx4_priv *priv = mlx4_priv(dev); struct mlx4_interface *intf; + int ret; mutex_lock(&intf_mutex); @@ -218,10 +434,18 @@ int mlx4_register_device(struct mlx4_dev *dev) list_for_each_entry(intf, &intf_list, list) mlx4_add_device(intf, priv); + ret = rescan_drivers_locked(dev); + mutex_unlock(&intf_mutex); + + if (ret) { + mlx4_unregister_device(dev); + return ret; + } + mlx4_start_catas_poll(dev); - return 0; + return ret; } void mlx4_unregister_device(struct mlx4_dev *dev) @@ -253,6 +477,8 @@ void mlx4_unregister_device(struct mlx4_dev *dev) list_del(&priv->dev_list); dev->persist->interface_state &= ~MLX4_INTERFACE_STATE_UP; + rescan_drivers_locked(dev); + mutex_unlock(&intf_mutex); } diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c index 0ed490b99163..c4ec7377aa71 100644 --- a/drivers/net/ethernet/mellanox/mlx4/main.c +++ b/drivers/net/ethernet/mellanox/mlx4/main.c @@ -3429,6 +3429,10 @@ static int mlx4_load_one(struct pci_dev *pdev, int pci_dev_data, INIT_LIST_HEAD(&priv->ctx_list); spin_lock_init(&priv->ctx_lock); + err = mlx4_adev_init(dev); + if (err) + return err; + ATOMIC_INIT_NOTIFIER_HEAD(&priv->event_nh); mutex_init(&priv->port_mutex); @@ -3455,10 +3459,11 @@ static int mlx4_load_one(struct pci_dev *pdev, int pci_dev_data, err = mlx4_get_ownership(dev); if (err) { if (err < 0) - return err; + goto err_adev; else { mlx4_warn(dev, "Multiple PFs not yet supported - Skipping PF\n"); - return -EINVAL; + err = -EINVAL; + goto err_adev; } } @@ -3806,6 +3811,9 @@ static int mlx4_load_one(struct pci_dev *pdev, int pci_dev_data, mlx4_free_ownership(dev); kfree(dev_cap); + +err_adev: + mlx4_adev_cleanup(dev); return err; } @@ -4186,6 +4194,8 @@ static void mlx4_unload_one(struct pci_dev *pdev) mlx4_slave_destroy_special_qp_cap(dev); kfree(dev->dev_vfs); + mlx4_adev_cleanup(dev); + mlx4_clean_dev(dev); priv->pci_dev_data = pci_dev_data; priv->removed = 1; @@ -4573,6 +4583,9 @@ static int __init mlx4_init(void) { int ret; + WARN_ONCE(strcmp(MLX4_ADEV_NAME, KBUILD_MODNAME), + "mlx4_core name not in sync with kernel module name"); + if (mlx4_verify_params()) return -EINVAL; diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4.h b/drivers/net/ethernet/mellanox/mlx4/mlx4.h index ece9acb6a869..d5050bfb342f 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4.h @@ -47,6 +47,7 @@ #include #include #include +#include #include #include @@ -884,6 +885,8 @@ struct mlx4_priv { struct list_head dev_list; struct list_head ctx_list; spinlock_t ctx_lock; + struct mlx4_adev **adev; + int adev_idx; struct atomic_notifier_head event_nh; int pci_dev_data; @@ -1052,6 +1055,9 @@ void mlx4_catas_end(struct mlx4_dev *dev); int mlx4_crdump_init(struct mlx4_dev *dev); void mlx4_crdump_end(struct mlx4_dev *dev); int mlx4_restart_one(struct pci_dev *pdev); + +int mlx4_adev_init(struct mlx4_dev *dev); +void mlx4_adev_cleanup(struct mlx4_dev *dev); int mlx4_register_device(struct mlx4_dev *dev); void mlx4_unregister_device(struct mlx4_dev *dev); void mlx4_dispatch_event(struct mlx4_dev *dev, enum mlx4_dev_event type, diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h index 049d8a4b044d..27f42f713c89 100644 --- a/include/linux/mlx4/device.h +++ b/include/linux/mlx4/device.h @@ -33,6 +33,7 @@ #ifndef MLX4_DEVICE_H #define MLX4_DEVICE_H +#include #include #include #include @@ -889,6 +890,12 @@ struct mlx4_dev { u8 uar_page_shift; }; +struct mlx4_adev { + struct auxiliary_device adev; + struct mlx4_dev *mdev; + int idx; +}; + struct mlx4_clock_params { u64 offset; u8 bar; diff --git a/include/linux/mlx4/driver.h b/include/linux/mlx4/driver.h index 781d5a0c2faa..9cf157d381c6 100644 --- a/include/linux/mlx4/driver.h +++ b/include/linux/mlx4/driver.h @@ -34,9 +34,12 @@ #define MLX4_DRIVER_H #include +#include #include #include +#define MLX4_ADEV_NAME "mlx4_core" + struct mlx4_dev; #define MLX4_MAC_MASK 0xffffffffffffULL @@ -63,8 +66,16 @@ struct mlx4_interface { int flags; }; +struct mlx4_adrv { + struct auxiliary_driver adrv; + enum mlx4_protocol protocol; + int flags; +}; + int mlx4_register_interface(struct mlx4_interface *intf); void mlx4_unregister_interface(struct mlx4_interface *intf); +int mlx4_register_auxiliary_driver(struct mlx4_adrv *madrv); +void mlx4_unregister_auxiliary_driver(struct mlx4_adrv *madrv); int mlx4_register_event_notifier(struct mlx4_dev *dev, struct notifier_block *nb); From patchwork Sun Aug 13 14:51:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 135043 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp2249081vqi; Sun, 13 Aug 2023 08:14:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH9GMCOinWlzAYMVysi9EfASvvYXFfiB0eesWZtT86MH48HmZ7dO0C3xBaBu/g7enX3GKy5 X-Received: by 2002:a05:6a20:42a0:b0:133:e3e3:dc07 with SMTP id o32-20020a056a2042a000b00133e3e3dc07mr6799689pzj.49.1691939655853; Sun, 13 Aug 2023 08:14:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691939655; cv=none; d=google.com; s=arc-20160816; b=PisVBNiL3C3ZmDm0pSWop6i2fLCqoxPAPAWHMaSyw+g4SwHOmPDZeQ8rLSARqDze3l AZW2GJT78lVZOBMx9ok5hwLQQux0yrNtzETX6fD2s/pmHJuh9ErLcQL7KaUes1XgOH8T Juo2NdBHjLzZ8JZrKCkAROo8c3suJFXRlnIS4OkUBLtiWriDP9AKFD2SZsdAgASRbsDQ /Zp/WJaFkrVkvSUvehsh48cNee3b+U182htfPpYClSmcss1QwtPFNYTK/LSBYxStiDY6 bMXnylmRP/9fR3BIGhTpBJIopcXyqN4+cPPI1kCJFKrneDm2ZveQzpF4tjxZ0uc0XANo vBUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=bVYQLZadpbIHkiHRlc0Nojw7TxPVG8nKYGsIyAJfqiI=; fh=thUxDq4m5D3FQct5425V7h171/dEmaHUwwAlFhAMPtU=; b=IQqFNTOp1tzwOas5Mxnn17cz3j7WkOPttj8NwiVllg7VAkB+OZuftx2HS9vg+BHm2/ pwfUmQBBJTVEQvX+1M/mJBU/4Asv+Y5w6NUiUfcvGWt7DU4R2uM57pc+KeC0jx56ICM8 rc6lj/oU72QbvjesWLa8RDKf1+CSuyyOich3+nkqh8bz/EfC63KUJj0oXXogr0HJgJGN ClleXxTznnWMBWwMd3SXX2obilmIeluXpmhc0VtzIsaKBt3mVNg1N+8Kz/fNoPoJBc7L N3KQZcFbObNzx6dGWXzFg349tBNNp80wN5D05zubCO8rYGCjzGbOvjrSLnhUEpO8o9l0 WkjQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=MfrE+Cq3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j63-20020a638b42000000b00543a6ce8c33si6457120pge.78.2023.08.13.08.14.02; Sun, 13 Aug 2023 08:14:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=MfrE+Cq3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231429AbjHMOwc (ORCPT + 99 others); Sun, 13 Aug 2023 10:52:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231408AbjHMOwT (ORCPT ); Sun, 13 Aug 2023 10:52:19 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC2B8171A; Sun, 13 Aug 2023 07:52:08 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id DC99321972; Sun, 13 Aug 2023 14:52:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1691938325; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bVYQLZadpbIHkiHRlc0Nojw7TxPVG8nKYGsIyAJfqiI=; b=MfrE+Cq3EKlLS6SO46s95zHEf51oaOR4LdpjZ8Oeizqqy9aBZ9NcGEjb7jkVuov+d0/4J8 aqe22Im6Nu0zYE4ajNazfOyLQv1bI93jbYOsDgD5SU0fjVvXZOuxyYkC11bEPjUqBUycAD 8i0nRjnU/PxSPH3psS8gXkv6WyrggCI= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id A9AB71322C; Sun, 13 Aug 2023 14:52:05 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 8JulKBXu2GSDFAAAMHmgww (envelope-from ); Sun, 13 Aug 2023 14:52:05 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v2 08/10] mlx4: Connect the ethernet part to the auxiliary bus Date: Sun, 13 Aug 2023 16:51:25 +0200 Message-Id: <20230813145127.10653-9-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230813145127.10653-1-petr.pavlu@suse.com> References: <20230813145127.10653-1-petr.pavlu@suse.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED,SPF_PASS, T_SPF_HELO_TEMPERROR autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774127316706983236 X-GMAIL-MSGID: 1774127316706983236 Use the auxiliary bus to perform device management of the ethernet part of the mlx4 driver. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx4/en_main.c | 65 ++++++++++++++------ drivers/net/ethernet/mellanox/mlx4/intf.c | 13 +++- 2 files changed, 58 insertions(+), 20 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/en_main.c b/drivers/net/ethernet/mellanox/mlx4/en_main.c index 263f9274927b..8a7e3aabbcfd 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_main.c @@ -222,9 +222,11 @@ static int mlx4_en_event(struct notifier_block *this, unsigned long event, return NOTIFY_DONE; } -static void mlx4_en_remove(struct mlx4_dev *dev, void *endev_ptr) +static void mlx4_en_remove(struct auxiliary_device *adev) { - struct mlx4_en_dev *mdev = endev_ptr; + struct mlx4_adev *madev = container_of(adev, struct mlx4_adev, adev); + struct mlx4_dev *dev = madev->mdev; + struct mlx4_en_dev *mdev = auxiliary_get_drvdata(adev); int i; mlx4_unregister_event_notifier(dev, &mdev->mlx_nb); @@ -247,27 +249,36 @@ static void mlx4_en_remove(struct mlx4_dev *dev, void *endev_ptr) kfree(mdev); } -static void *mlx4_en_add(struct mlx4_dev *dev) +static int mlx4_en_probe(struct auxiliary_device *adev, + const struct auxiliary_device_id *id) { + struct mlx4_adev *madev = container_of(adev, struct mlx4_adev, adev); + struct mlx4_dev *dev = madev->mdev; struct mlx4_en_dev *mdev; int err, i; printk_once(KERN_INFO "%s", mlx4_en_version); mdev = kzalloc(sizeof(*mdev), GFP_KERNEL); - if (!mdev) + if (!mdev) { + err = -ENOMEM; goto err_free_res; + } - if (mlx4_pd_alloc(dev, &mdev->priv_pdn)) + err = mlx4_pd_alloc(dev, &mdev->priv_pdn); + if (err) goto err_free_dev; - if (mlx4_uar_alloc(dev, &mdev->priv_uar)) + err = mlx4_uar_alloc(dev, &mdev->priv_uar); + if (err) goto err_pd; mdev->uar_map = ioremap((phys_addr_t) mdev->priv_uar.pfn << PAGE_SHIFT, PAGE_SIZE); - if (!mdev->uar_map) + if (!mdev->uar_map) { + err = -ENOMEM; goto err_uar; + } spin_lock_init(&mdev->uar_lock); mdev->dev = dev; @@ -279,13 +290,15 @@ static void *mlx4_en_add(struct mlx4_dev *dev) if (!mdev->LSO_support) mlx4_warn(mdev, "LSO not supported, please upgrade to later FW version to enable LSO\n"); - if (mlx4_mr_alloc(mdev->dev, mdev->priv_pdn, 0, ~0ull, - MLX4_PERM_LOCAL_WRITE | MLX4_PERM_LOCAL_READ, - 0, 0, &mdev->mr)) { + err = mlx4_mr_alloc(mdev->dev, mdev->priv_pdn, 0, ~0ull, + MLX4_PERM_LOCAL_WRITE | MLX4_PERM_LOCAL_READ, 0, 0, + &mdev->mr); + if (err) { mlx4_err(mdev, "Failed allocating memory region\n"); goto err_map; } - if (mlx4_mr_enable(mdev->dev, &mdev->mr)) { + err = mlx4_mr_enable(mdev->dev, &mdev->mr); + if (err) { mlx4_err(mdev, "Failed enabling memory region\n"); goto err_mr; } @@ -305,8 +318,10 @@ static void *mlx4_en_add(struct mlx4_dev *dev) * Note: we cannot use the shared workqueue because of deadlocks caused * by the rtnl lock */ mdev->workqueue = create_singlethread_workqueue("mlx4_en"); - if (!mdev->workqueue) + if (!mdev->workqueue) { + err = -ENOMEM; goto err_mr; + } /* At this stage all non-port specific tasks are complete: * mark the card state as up */ @@ -334,7 +349,8 @@ static void *mlx4_en_add(struct mlx4_dev *dev) mlx4_err(mdev, "Failed to create netdev notifier\n"); } - return mdev; + auxiliary_set_drvdata(adev, mdev); + return 0; err_mr: (void) mlx4_mr_free(dev, &mdev->mr); @@ -348,12 +364,23 @@ static void *mlx4_en_add(struct mlx4_dev *dev) err_free_dev: kfree(mdev); err_free_res: - return NULL; + return err; } -static struct mlx4_interface mlx4_en_interface = { - .add = mlx4_en_add, - .remove = mlx4_en_remove, +static const struct auxiliary_device_id mlx4_en_id_table[] = { + { .name = MLX4_ADEV_NAME ".eth" }, + {}, +}; + +MODULE_DEVICE_TABLE(auxiliary, mlx4_en_id_table); + +static struct mlx4_adrv mlx4_en_adrv = { + .adrv = { + .name = "eth", + .probe = mlx4_en_probe, + .remove = mlx4_en_remove, + .id_table = mlx4_en_id_table, + }, .protocol = MLX4_PROT_ETH, }; @@ -383,12 +410,12 @@ static int __init mlx4_en_init(void) mlx4_en_verify_params(); mlx4_en_init_ptys2ethtool_map(); - return mlx4_register_interface(&mlx4_en_interface); + return mlx4_register_auxiliary_driver(&mlx4_en_adrv); } static void __exit mlx4_en_cleanup(void) { - mlx4_unregister_interface(&mlx4_en_interface); + mlx4_unregister_auxiliary_driver(&mlx4_en_adrv); } module_init(mlx4_en_init); diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c index 4b1e18e4a682..0a27820ece2e 100644 --- a/drivers/net/ethernet/mellanox/mlx4/intf.c +++ b/drivers/net/ethernet/mellanox/mlx4/intf.c @@ -50,10 +50,21 @@ static LIST_HEAD(dev_list); static DEFINE_MUTEX(intf_mutex); static DEFINE_IDA(mlx4_adev_ida); +static bool is_eth_supported(struct mlx4_dev *dev) +{ + for (int port = 1; port <= dev->caps.num_ports; port++) + if (dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH) + return true; + + return false; +} + static const struct mlx4_adev_device { const char *suffix; bool (*is_supported)(struct mlx4_dev *dev); -} mlx4_adev_devices[1] = {}; +} mlx4_adev_devices[] = { + { "eth", is_eth_supported }, +}; int mlx4_adev_init(struct mlx4_dev *dev) { From patchwork Sun Aug 13 14:51:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 135040 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp2243184vqi; Sun, 13 Aug 2023 08:02:44 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHi78PBJp65W3eLKZcX4zb9q4fdQ3HM2ODUzMKZTBK95/+X/jJi+734VKe4tnPPwrVVDZbx X-Received: by 2002:a05:6402:182:b0:514:9934:de96 with SMTP id r2-20020a056402018200b005149934de96mr5232454edv.26.1691938964588; Sun, 13 Aug 2023 08:02:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691938964; cv=none; d=google.com; s=arc-20160816; b=UdjtsrZEsC4BrZJnMwI1LzpJebLtJ0NCwNzjGrQ6Y9Kk/iicZ5wfeMx2vCHaxUf8FO EM2bVCM1sLIKabsbZu0dmL8Jedb7wCw3e7CTSvZeR+HdulzF9Jymn7NRvZPUADXhGnmE lGUZKN/5neGthpciYde2FabekI6oF12Q9kQHls8EG0bjeZmHPPA3Zh+dJHcfx/XrO2YV pHeWY1z2qKmwkkSFQqufG2zU1/ynblu61V0Wn8Mvko/LPNazZEk5cLG1q160Fh89llI3 X5o0pF4kpcof35AhQ98UQrV9pmSG9xriiDiH+ARWKgsKY6RDP7LtqWfGcSywozz3w8Sc XT3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Vo3VG09P6nBkj8Z7HgwZLpS7JWI7W0+dsbUV78jVfnE=; fh=thUxDq4m5D3FQct5425V7h171/dEmaHUwwAlFhAMPtU=; b=GQIv3rGOoN7nGBmjL8iC6L9R7hWDkONksWkG0m4TqH46Fkgws10nxTUFJFKDvgZeq/ ckfGjvrHUVfx/ZUeZskD5vX7/ZE+Y/CdPuLQGR4FNk6fH6I3gZQxY/i9BWprJ51TpY5H KC9iPhUXEv1HPclUaw5fOBGw5go3SqIvgZJjcaaKRIPcjTmFZillXvjHqpVWZmMQ745b ZQDQC2OUrRaXHpDDm5NdciPj7PE81srrXG6pf4RCHNEeqk+XvclW7Y8B4L7wMPM04I8C 12fe5sAEwzpcTdULP/1A2qdWoKOkgzUEwDydaPwcxfeS2mG4iy988SQjsK9EBZkZfLL/ heVQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=dCUqbwY2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n27-20020a056402515b00b005222b60fc67si4780952edd.303.2023.08.13.08.02.16; Sun, 13 Aug 2023 08:02:44 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=dCUqbwY2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231459AbjHMOwY (ORCPT + 99 others); Sun, 13 Aug 2023 10:52:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231349AbjHMOwJ (ORCPT ); Sun, 13 Aug 2023 10:52:09 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 87FC41702; Sun, 13 Aug 2023 07:52:07 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 2137421977; Sun, 13 Aug 2023 14:52:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1691938326; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Vo3VG09P6nBkj8Z7HgwZLpS7JWI7W0+dsbUV78jVfnE=; b=dCUqbwY2wcddKkDz0HYY9fVTZGQNSoj6Xq8ag4fVgGKPbO/8fnEvkbtdvuRP/up8CTC3SV sLFwcWz6moG4mtF7S386vq6rKFrhhfzfw2AaD093SMaHQdTPVlV3WufI+rK623yQ01884F JWp66ttL8IRqQaYLwAwmdRJaP/LiwUo= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id E037513593; Sun, 13 Aug 2023 14:52:05 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 2MwLNhXu2GSDFAAAMHmgww (envelope-from ); Sun, 13 Aug 2023 14:52:05 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v2 09/10] mlx4: Connect the infiniband part to the auxiliary bus Date: Sun, 13 Aug 2023 16:51:26 +0200 Message-Id: <20230813145127.10653-10-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230813145127.10653-1-petr.pavlu@suse.com> References: <20230813145127.10653-1-petr.pavlu@suse.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED,SPF_PASS, T_SPF_HELO_TEMPERROR autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774126592114893407 X-GMAIL-MSGID: 1774126592114893407 Use the auxiliary bus to perform device management of the infiniband part of the mlx4 driver. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- drivers/infiniband/hw/mlx4/main.c | 77 ++++++++++++++++------- drivers/net/ethernet/mellanox/mlx4/intf.c | 13 ++++ 2 files changed, 67 insertions(+), 23 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c index 0761c465120b..d92069da2b3f 100644 --- a/drivers/infiniband/hw/mlx4/main.c +++ b/drivers/infiniband/hw/mlx4/main.c @@ -2609,8 +2609,11 @@ static const struct ib_device_ops mlx4_ib_dev_fs_ops = { .destroy_flow = mlx4_ib_destroy_flow, }; -static void *mlx4_ib_add(struct mlx4_dev *dev) +static int mlx4_ib_probe(struct auxiliary_device *adev, + const struct auxiliary_device_id *id) { + struct mlx4_adev *madev = container_of(adev, struct mlx4_adev, adev); + struct mlx4_dev *dev = madev->mdev; struct mlx4_ib_dev *ibdev; int num_ports = 0; int i, j; @@ -2630,27 +2633,31 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) /* No point in registering a device with no ports... */ if (num_ports == 0) - return NULL; + return -ENODEV; ibdev = ib_alloc_device(mlx4_ib_dev, ib_dev); if (!ibdev) { dev_err(&dev->persist->pdev->dev, "Device struct alloc failed\n"); - return NULL; + return -ENOMEM; } iboe = &ibdev->iboe; - if (mlx4_pd_alloc(dev, &ibdev->priv_pdn)) + err = mlx4_pd_alloc(dev, &ibdev->priv_pdn); + if (err) goto err_dealloc; - if (mlx4_uar_alloc(dev, &ibdev->priv_uar)) + err = mlx4_uar_alloc(dev, &ibdev->priv_uar); + if (err) goto err_pd; ibdev->uar_map = ioremap((phys_addr_t) ibdev->priv_uar.pfn << PAGE_SHIFT, PAGE_SIZE); - if (!ibdev->uar_map) + if (!ibdev->uar_map) { + err = -ENOMEM; goto err_uar; + } MLX4_INIT_DOORBELL_LOCK(&ibdev->uar_lock); ibdev->dev = dev; @@ -2694,7 +2701,8 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) spin_lock_init(&iboe->lock); - if (init_node_data(ibdev)) + err = init_node_data(ibdev); + if (err) goto err_map; mlx4_init_sl2vl_tbl(ibdev); @@ -2726,6 +2734,7 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) new_counter_index = kmalloc(sizeof(*new_counter_index), GFP_KERNEL); if (!new_counter_index) { + err = -ENOMEM; if (allocated) mlx4_counter_free(ibdev->dev, counter_index); goto err_counter; @@ -2743,8 +2752,10 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) new_counter_index = kmalloc(sizeof(struct counter_index), GFP_KERNEL); - if (!new_counter_index) + if (!new_counter_index) { + err = -ENOMEM; goto err_counter; + } new_counter_index->index = counter_index; new_counter_index->allocated = 0; list_add_tail(&new_counter_index->list, @@ -2773,8 +2784,10 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) ibdev->ib_uc_qpns_bitmap = bitmap_alloc(ibdev->steer_qpn_count, GFP_KERNEL); - if (!ibdev->ib_uc_qpns_bitmap) + if (!ibdev->ib_uc_qpns_bitmap) { + err = -ENOMEM; goto err_steer_qp_release; + } if (dev->caps.flags2 & MLX4_DEV_CAP_FLAG2_DMFS_IPOIB) { bitmap_zero(ibdev->ib_uc_qpns_bitmap, @@ -2794,17 +2807,21 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) for (j = 1; j <= ibdev->dev->caps.num_ports; j++) atomic64_set(&iboe->mac[j - 1], ibdev->dev->caps.def_mac[j]); - if (mlx4_ib_alloc_diag_counters(ibdev)) + err = mlx4_ib_alloc_diag_counters(ibdev); + if (err) goto err_steer_free_bitmap; - if (ib_register_device(&ibdev->ib_dev, "mlx4_%d", - &dev->persist->pdev->dev)) + err = ib_register_device(&ibdev->ib_dev, "mlx4_%d", + &dev->persist->pdev->dev); + if (err) goto err_diag_counters; - if (mlx4_ib_mad_init(ibdev)) + err = mlx4_ib_mad_init(ibdev); + if (err) goto err_reg; - if (mlx4_ib_init_sriov(ibdev)) + err = mlx4_ib_init_sriov(ibdev); + if (err) goto err_mad; if (!iboe->nb.notifier_call) { @@ -2844,7 +2861,8 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) err = mlx4_register_event_notifier(dev, &ibdev->mlx_nb); WARN(err, "failed to register mlx4 event notifier (%d)", err); - return ibdev; + auxiliary_set_drvdata(adev, ibdev); + return 0; err_notif: if (ibdev->iboe.nb.notifier_call) { @@ -2888,7 +2906,7 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) err_dealloc: ib_dealloc_device(&ibdev->ib_dev); - return NULL; + return err; } int mlx4_ib_steer_qp_alloc(struct mlx4_ib_dev *dev, int count, int *qpn) @@ -2955,9 +2973,11 @@ int mlx4_ib_steer_qp_reg(struct mlx4_ib_dev *mdev, struct mlx4_ib_qp *mqp, return err; } -static void mlx4_ib_remove(struct mlx4_dev *dev, void *ibdev_ptr) +static void mlx4_ib_remove(struct auxiliary_device *adev) { - struct mlx4_ib_dev *ibdev = ibdev_ptr; + struct mlx4_adev *madev = container_of(adev, struct mlx4_adev, adev); + struct mlx4_dev *dev = madev->mdev; + struct mlx4_ib_dev *ibdev = auxiliary_get_drvdata(adev); int p; int i; @@ -3298,9 +3318,20 @@ static int mlx4_ib_event(struct notifier_block *this, unsigned long event, return NOTIFY_DONE; } -static struct mlx4_interface mlx4_ib_interface = { - .add = mlx4_ib_add, - .remove = mlx4_ib_remove, +static const struct auxiliary_device_id mlx4_ib_id_table[] = { + { .name = MLX4_ADEV_NAME ".ib" }, + {}, +}; + +MODULE_DEVICE_TABLE(auxiliary, mlx4_ib_id_table); + +static struct mlx4_adrv mlx4_ib_adrv = { + .adrv = { + .name = "ib", + .probe = mlx4_ib_probe, + .remove = mlx4_ib_remove, + .id_table = mlx4_ib_id_table, + }, .protocol = MLX4_PROT_IB_IPV6, .flags = MLX4_INTFF_BONDING }; @@ -3325,7 +3356,7 @@ static int __init mlx4_ib_init(void) if (err) goto clean_cm; - err = mlx4_register_interface(&mlx4_ib_interface); + err = mlx4_register_auxiliary_driver(&mlx4_ib_adrv); if (err) goto clean_mcg; @@ -3347,7 +3378,7 @@ static int __init mlx4_ib_init(void) static void __exit mlx4_ib_cleanup(void) { - mlx4_unregister_interface(&mlx4_ib_interface); + mlx4_unregister_auxiliary_driver(&mlx4_ib_adrv); mlx4_ib_mcg_destroy(); mlx4_ib_cm_destroy(); mlx4_ib_qp_event_cleanup(); diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c index 0a27820ece2e..16b2c99ff737 100644 --- a/drivers/net/ethernet/mellanox/mlx4/intf.c +++ b/drivers/net/ethernet/mellanox/mlx4/intf.c @@ -59,11 +59,24 @@ static bool is_eth_supported(struct mlx4_dev *dev) return false; } +static bool is_ib_supported(struct mlx4_dev *dev) +{ + for (int port = 1; port <= dev->caps.num_ports; port++) + if (dev->caps.port_type[port] == MLX4_PORT_TYPE_IB) + return true; + + if (dev->caps.flags & MLX4_DEV_CAP_FLAG_IBOE) + return true; + + return false; +} + static const struct mlx4_adev_device { const char *suffix; bool (*is_supported)(struct mlx4_dev *dev); } mlx4_adev_devices[] = { { "eth", is_eth_supported }, + { "ib", is_ib_supported }, }; int mlx4_adev_init(struct mlx4_dev *dev) From patchwork Sun Aug 13 14:51:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 135048 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b824:0:b0:3f2:4152:657d with SMTP id z4csp2254113vqi; Sun, 13 Aug 2023 08:27:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGrSL1VZNE8nqraYakOE2p+9Xfz6hf73oDTxZYWbEZUneqfSmow9gEkCQXfD71lQ4xled98 X-Received: by 2002:a05:6a20:f398:b0:13f:5a05:6038 with SMTP id qr24-20020a056a20f39800b0013f5a056038mr5634136pzb.58.1691940422553; Sun, 13 Aug 2023 08:27:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1691940422; cv=none; d=google.com; s=arc-20160816; b=RpSGl295CavDycAlq41jPb0j2gNikflmZR/qx/ckl2hJpcfemQ16NATsLgCkwIOB2E RzO6MPepyr96wF8l8lhJ9pVyZjBevs5WBb1h3SJjagt3lqytb+IEaqD2f04A4Gf4QsRf S3FujzGp1ZeYpQK7SIUGSn9MjUgYs/Ot90qzZML45P7zvDoEkEH9x2e8P6Izo+Z58/Aq 7BtEUt+qFpBT+d/nn7K5k7F/fUlTX4UZ0Zf/w0MZWXnvbRf3kegLVvCxkyrVr0VwqjHi cU4IXtPGB4M7uMoAZiurgXyFoo2FcueJB2BTNY3U0lrBfl6IhGrBocihHHGCfhzs3N6e H3sQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=SOXs92XpDZ2m8VCK5Qf/JTvBeaJd0cZxFhhssMI3N5Y=; fh=thUxDq4m5D3FQct5425V7h171/dEmaHUwwAlFhAMPtU=; b=kq/WsXFQ0sfeXRu3Bau5xo1RxkCkfgqYfZ4itRyVQ1Yse8KhFeqinr85V5lYWEgPtr n141ECcewUtB3RyWdLJS3b3+1fUyyZcenH4WThhutAcCvKUOgXZ08qWqJSJFd0yo1vUD QPXsc0CkUeSaW/uwVq62paDrfN2WxKPZMlBgSLJwdEdL7Awd5YZWeQZee0URsrbpQSto ujTy+PYMo0WLZIXri5AF7EcE0XnY93cc1uHpOIqvT/mn2+DV9wTyYbwRL89RADOQlHyB CkkqHyU8HBUJe2MSTSnmIQKiVHm3d6roMGOdVnZKVgyKQAgoEHbBzssA6XSaDWTuqBRV TB1Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=Qq8EvmW2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cj10-20020a056a00298a00b006871cbe9bb9si6581904pfb.339.2023.08.13.08.26.48; Sun, 13 Aug 2023 08:27:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=Qq8EvmW2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231425AbjHMOwT (ORCPT + 99 others); Sun, 13 Aug 2023 10:52:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59974 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231316AbjHMOwH (ORCPT ); Sun, 13 Aug 2023 10:52:07 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC08A1704; Sun, 13 Aug 2023 07:52:08 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 5C5FB1F8CC; Sun, 13 Aug 2023 14:52:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1691938326; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SOXs92XpDZ2m8VCK5Qf/JTvBeaJd0cZxFhhssMI3N5Y=; b=Qq8EvmW2arMIp3J0zjij8yFcTZJbSFYWK8Ip6xMkr5/A+t8T57epyqGqZjcQqV4Cn6hXzI qd80Wc4FGQ2D8S3bwpzhVSXU4kX6mNspizZ8WnnmyKwTSoyvkJGIHd2AfWRMjciMeLLQWc JYlV9kHtmhShmGUJ/6/Sve6M2GXOGiE= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 263DD1322C; Sun, 13 Aug 2023 14:52:06 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id OO2aCBbu2GSDFAAAMHmgww (envelope-from ); Sun, 13 Aug 2023 14:52:06 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v2 10/10] mlx4: Delete custom device management logic Date: Sun, 13 Aug 2023 16:51:27 +0200 Message-Id: <20230813145127.10653-11-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230813145127.10653-1-petr.pavlu@suse.com> References: <20230813145127.10653-1-petr.pavlu@suse.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774128120180570208 X-GMAIL-MSGID: 1774128120180570208 After the conversion to use the auxiliary bus, the custom device management is not needed anymore and can be deleted. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx4/intf.c | 125 ---------------------- drivers/net/ethernet/mellanox/mlx4/main.c | 28 ----- drivers/net/ethernet/mellanox/mlx4/mlx4.h | 3 - include/linux/mlx4/driver.h | 10 -- 4 files changed, 166 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c index 16b2c99ff737..c7697ee0dd05 100644 --- a/drivers/net/ethernet/mellanox/mlx4/intf.c +++ b/drivers/net/ethernet/mellanox/mlx4/intf.c @@ -38,15 +38,6 @@ #include "mlx4.h" -struct mlx4_device_context { - struct list_head list; - struct list_head bond_list; - struct mlx4_interface *intf; - void *context; -}; - -static LIST_HEAD(intf_list); -static LIST_HEAD(dev_list); static DEFINE_MUTEX(intf_mutex); static DEFINE_IDA(mlx4_adev_ida); @@ -156,77 +147,6 @@ static void del_adev(struct auxiliary_device *adev) auxiliary_device_uninit(adev); } -static void mlx4_add_device(struct mlx4_interface *intf, struct mlx4_priv *priv) -{ - struct mlx4_device_context *dev_ctx; - - dev_ctx = kmalloc(sizeof(*dev_ctx), GFP_KERNEL); - if (!dev_ctx) - return; - - dev_ctx->intf = intf; - dev_ctx->context = intf->add(&priv->dev); - - if (dev_ctx->context) { - spin_lock_irq(&priv->ctx_lock); - list_add_tail(&dev_ctx->list, &priv->ctx_list); - spin_unlock_irq(&priv->ctx_lock); - } else - kfree(dev_ctx); - -} - -static void mlx4_remove_device(struct mlx4_interface *intf, struct mlx4_priv *priv) -{ - struct mlx4_device_context *dev_ctx; - - list_for_each_entry(dev_ctx, &priv->ctx_list, list) - if (dev_ctx->intf == intf) { - spin_lock_irq(&priv->ctx_lock); - list_del(&dev_ctx->list); - spin_unlock_irq(&priv->ctx_lock); - - intf->remove(&priv->dev, dev_ctx->context); - kfree(dev_ctx); - return; - } -} - -int mlx4_register_interface(struct mlx4_interface *intf) -{ - struct mlx4_priv *priv; - - if (!intf->add || !intf->remove) - return -EINVAL; - - mutex_lock(&intf_mutex); - - list_add_tail(&intf->list, &intf_list); - list_for_each_entry(priv, &dev_list, dev_list) { - mlx4_add_device(intf, priv); - } - - mutex_unlock(&intf_mutex); - - return 0; -} -EXPORT_SYMBOL_GPL(mlx4_register_interface); - -void mlx4_unregister_interface(struct mlx4_interface *intf) -{ - struct mlx4_priv *priv; - - mutex_lock(&intf_mutex); - - list_for_each_entry(priv, &dev_list, dev_list) - mlx4_remove_device(intf, priv); - - list_del(&intf->list); - - mutex_unlock(&intf_mutex); -} -EXPORT_SYMBOL_GPL(mlx4_unregister_interface); - int mlx4_register_auxiliary_driver(struct mlx4_adrv *madrv) { return auxiliary_driver_register(&madrv->adrv); @@ -242,10 +162,7 @@ EXPORT_SYMBOL_GPL(mlx4_unregister_auxiliary_driver); int mlx4_do_bond(struct mlx4_dev *dev, bool enable) { struct mlx4_priv *priv = mlx4_priv(dev); - struct mlx4_device_context *dev_ctx = NULL, *temp_dev_ctx; - unsigned long flags; int i, ret; - LIST_HEAD(bond_list); if (!(dev->caps.flags2 & MLX4_DEV_CAP_FLAG2_PORT_REMAP)) return -EOPNOTSUPP; @@ -267,36 +184,6 @@ int mlx4_do_bond(struct mlx4_dev *dev, bool enable) dev->flags &= ~MLX4_FLAG_BONDED; } - spin_lock_irqsave(&priv->ctx_lock, flags); - list_for_each_entry_safe(dev_ctx, temp_dev_ctx, &priv->ctx_list, list) { - if (!(dev_ctx->intf->flags & MLX4_INTFF_BONDING)) - continue; - - if (mlx4_is_mfunc(dev)) { - mlx4_dbg(dev, - "SRIOV, disabled HA mode for intf proto %d\n", - dev_ctx->intf->protocol); - continue; - } - - list_add_tail(&dev_ctx->bond_list, &bond_list); - list_del(&dev_ctx->list); - } - spin_unlock_irqrestore(&priv->ctx_lock, flags); - - list_for_each_entry(dev_ctx, &bond_list, bond_list) { - dev_ctx->intf->remove(dev, dev_ctx->context); - dev_ctx->context = dev_ctx->intf->add(dev); - - spin_lock_irqsave(&priv->ctx_lock, flags); - list_add_tail(&dev_ctx->list, &priv->ctx_list); - spin_unlock_irqrestore(&priv->ctx_lock, flags); - - mlx4_dbg(dev, "Interface for protocol %d restarted with bonded mode %s\n", - dev_ctx->intf->protocol, enable ? - "enabled" : "disabled"); - } - mutex_lock(&intf_mutex); for (i = 0; i < ARRAY_SIZE(mlx4_adev_devices); i++) { @@ -447,16 +334,11 @@ static int rescan_drivers_locked(struct mlx4_dev *dev) int mlx4_register_device(struct mlx4_dev *dev) { - struct mlx4_priv *priv = mlx4_priv(dev); - struct mlx4_interface *intf; int ret; mutex_lock(&intf_mutex); dev->persist->interface_state |= MLX4_INTERFACE_STATE_UP; - list_add_tail(&priv->dev_list, &dev_list); - list_for_each_entry(intf, &intf_list, list) - mlx4_add_device(intf, priv); ret = rescan_drivers_locked(dev); @@ -474,9 +356,6 @@ int mlx4_register_device(struct mlx4_dev *dev) void mlx4_unregister_device(struct mlx4_dev *dev) { - struct mlx4_priv *priv = mlx4_priv(dev); - struct mlx4_interface *intf; - if (!(dev->persist->interface_state & MLX4_INTERFACE_STATE_UP)) return; @@ -495,10 +374,6 @@ void mlx4_unregister_device(struct mlx4_dev *dev) } mutex_lock(&intf_mutex); - list_for_each_entry(intf, &intf_list, list) - mlx4_remove_device(intf, priv); - - list_del(&priv->dev_list); dev->persist->interface_state &= ~MLX4_INTERFACE_STATE_UP; rescan_drivers_locked(dev); diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c index c4ec7377aa71..2581226836b5 100644 --- a/drivers/net/ethernet/mellanox/mlx4/main.c +++ b/drivers/net/ethernet/mellanox/mlx4/main.c @@ -42,7 +42,6 @@ #include #include #include -#include #include #include @@ -1091,27 +1090,6 @@ static int mlx4_slave_cap(struct mlx4_dev *dev) return err; } -static void mlx4_request_modules(struct mlx4_dev *dev) -{ - int port; - int has_ib_port = false; - int has_eth_port = false; -#define EN_DRV_NAME "mlx4_en" -#define IB_DRV_NAME "mlx4_ib" - - for (port = 1; port <= dev->caps.num_ports; port++) { - if (dev->caps.port_type[port] == MLX4_PORT_TYPE_IB) - has_ib_port = true; - else if (dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH) - has_eth_port = true; - } - - if (has_eth_port) - request_module_nowait(EN_DRV_NAME); - if (has_ib_port || (dev->caps.flags & MLX4_DEV_CAP_FLAG_IBOE)) - request_module_nowait(IB_DRV_NAME); -} - /* * Change the port configuration of the device. * Every user of this function must hold the port mutex. @@ -1147,7 +1125,6 @@ int mlx4_change_port_types(struct mlx4_dev *dev, mlx4_err(dev, "Failed to register device\n"); goto out; } - mlx4_request_modules(dev); } out: @@ -3426,9 +3403,6 @@ static int mlx4_load_one(struct pci_dev *pdev, int pci_dev_data, devl_assert_locked(devlink); dev = &priv->dev; - INIT_LIST_HEAD(&priv->ctx_list); - spin_lock_init(&priv->ctx_lock); - err = mlx4_adev_init(dev); if (err) return err; @@ -3732,8 +3706,6 @@ static int mlx4_load_one(struct pci_dev *pdev, int pci_dev_data, if (err) goto err_port; - mlx4_request_modules(dev); - mlx4_sense_init(dev); mlx4_start_sense(dev); diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4.h b/drivers/net/ethernet/mellanox/mlx4/mlx4.h index d5050bfb342f..d707b790536f 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4.h @@ -882,9 +882,6 @@ enum { struct mlx4_priv { struct mlx4_dev dev; - struct list_head dev_list; - struct list_head ctx_list; - spinlock_t ctx_lock; struct mlx4_adev **adev; int adev_idx; struct atomic_notifier_head event_nh; diff --git a/include/linux/mlx4/driver.h b/include/linux/mlx4/driver.h index 9cf157d381c6..69825223081f 100644 --- a/include/linux/mlx4/driver.h +++ b/include/linux/mlx4/driver.h @@ -58,22 +58,12 @@ enum { MLX4_INTFF_BONDING = 1 << 0 }; -struct mlx4_interface { - void * (*add) (struct mlx4_dev *dev); - void (*remove)(struct mlx4_dev *dev, void *context); - struct list_head list; - enum mlx4_protocol protocol; - int flags; -}; - struct mlx4_adrv { struct auxiliary_driver adrv; enum mlx4_protocol protocol; int flags; }; -int mlx4_register_interface(struct mlx4_interface *intf); -void mlx4_unregister_interface(struct mlx4_interface *intf); int mlx4_register_auxiliary_driver(struct mlx4_adrv *madrv); void mlx4_unregister_auxiliary_driver(struct mlx4_adrv *madrv);