From patchwork Wed Feb 7 07:24:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 197782 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp2059627dyb; Tue, 6 Feb 2024 23:28:56 -0800 (PST) X-Google-Smtp-Source: AGHT+IERmz4qo1cXtDm9YskzJ83n7nzKs909TsdEJ188dZBQT91aMtDV9t0erca9cDOeMaKEkuOu X-Received: by 2002:a2e:9b95:0:b0:2d0:a0a7:e3ef with SMTP id z21-20020a2e9b95000000b002d0a0a7e3efmr3612866lji.4.1707290936112; Tue, 06 Feb 2024 23:28:56 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707290936; cv=pass; d=google.com; s=arc-20160816; b=0cq8nMyy9JozJlzVn2kwI4Gg7i4S1kXMkESzl9p/F18AyWr6bnSqhB6GWUIBF/Bfgj p9TLdIpbr8p/bAA35kyJIfRbON+1ZkWiSNmeuS0RpBQ8Onnw0M1qHxpN6QTIOLI0FC2r iGL19Pwn5pMT0/otrbBem3E6ScmDIob2lnbkl6mCQU+XdXRHyDRKBdlKLbwbVt/0izTo Qq6F2TkzvYE+DYU4trmTq5S2fgpIee2asG76VKGGXUIx3pMWpXIDQGr+eJRFmJE8VVQt zTBTY8S0TKEy7TrOpCPFwiWiYD58KlumEYsDH/mfz/LDaLZmMKxkQvtUwrnNlYGQ4OM2 zinw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=XLot7mTyGhFw+GmvCLq7oj9AS61JB1sIxUgjAWaai9s=; fh=iLvlVTEDZawTuzUoS/uHvWrb8JNF77Vpbk5LZQ4sldE=; b=EGJcXlMb0jFnWNdl+XJJYAAOwDwXBroQIvdNkodAENiznhppcOfe/h/NHv5Q7IHRZ2 NAtlHwkxzTycrd/EnKdQjnUf70IV9q1ikpZzf8isWxr/kSOouDY8a2Wr1cTYOlg5Kuh+ X546R9pHpts5rR6YAqtJZrVAP3NVMag4fsfQRosudirnVGoDEUG3rXDsjzjGits+zCw7 CKn72QnO4KmW/KVkAIHOvX3VLSzXLl6eYAPfe/+y9qemVGG282Y9TP1e6a+EtF5TJenq rWjuxLx6UIMzHWy05BQtdMUAB6JxS4GilPGsMNQUr67IHefRpYUw4ilNexQ7EBPHnecT I5Lw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=JtZEUOh0; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-56073-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-56073-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=2; AJvYcCXxf1tjfz8PVVuF3iQZDhoNf9uy4ElfGqiEXGn/gfR4Vj89Qzcb1M5++IsN/+4PA/jDRkXtg+pLZ50ceHNnHyLC0GGhEA== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id l19-20020aa7c3d3000000b0055fa07cd8d5si526740edr.563.2024.02.06.23.28.55 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 23:28:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-56073-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=JtZEUOh0; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-56073-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-56073-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id AFA751F256A6 for ; Wed, 7 Feb 2024 07:28:55 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 57ADE20320; Wed, 7 Feb 2024 07:28:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="JtZEUOh0" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8DFA21F953 for ; Wed, 7 Feb 2024 07:28:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707290905; cv=none; b=ITK3nmmicoPBL86jVBP+ZUOZ+v/xdklDsNCSrISWk3j21Yw8XzKIaJopxIQd01TAg1yp+cl6ls6ZOARg/0a4Y14UVYuutyx59byHcgR8wukc/VnHb+HEnKxri3+3Mv+70T9PZCoPOyQOfWqYc9gf9iEnqLKJ3RodZF0brxmlXMg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707290905; c=relaxed/simple; bh=BaBHHsLR+o2kRrQL4hy2Xypz7kcRhGFamMXsbhKvWuI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kmW5w3TxQbrEvLZGtIIDWd5E3+7QsWvEnjMc0ahJWHfQl2COW7LTI+FKxSoJBGEkYNuqKABdHEqbwQOvlijJPONa5v6WPp26sYf8UVtK4jKP5pMwnGcPv1XRgNWYTueVGVKrqFGQWEDcraAJA01YY93U+Ii6D/NYgC7de9qdrW8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=JtZEUOh0; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id E5C81C433F1; Wed, 7 Feb 2024 07:28:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707290905; bh=BaBHHsLR+o2kRrQL4hy2Xypz7kcRhGFamMXsbhKvWuI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JtZEUOh0zRyxwwk34XIHx6A3v4f86vUzDdohVwqDarRmZ/nvXulBxuTQvTLAZ1vaG VjAHHJ6Cokwkisng7pRIgUVIwIVE8H+8mcUrdOSAe6MQtzwbhHEo2cqMEmZ/Ji5Nlp yyCsglY+/e1OChNvvsDKK2/1PBXOncxfW1FuMd7uFgqgQ4/UimYP9qNAVXko+j4g0p MwPK1ySscNrlm0BKysPlOnOcJ9j7doblgAsrYoUNu1XBM90JFTW9Q9jXx6QakIqUMU yB0bfZwmt/vRgktKMNCBwr3sHV02XK6Fa4UU2gPZ5Pav2MvtIY4SEvQyVHxYDgMSwH RkDBRtPh2GgGg== From: Saeed Mahameed To: Arnd Bergmann , Greg Kroah-Hartman Cc: Leon Romanovsky , Jason Gunthorpe , Jiri Pirko , Leonid Bloch , Itay Avraham , Jakub Kicinski , Saeed Mahameed , David Ahern , Aron Silverton , Christoph Hellwig , andrew.gospodarek@broadcom.com, linux-kernel@vger.kernel.org Subject: [PATCH V4 1/5] mlx5: Add aux dev for ctl interface Date: Tue, 6 Feb 2024 23:24:31 -0800 Message-ID: <20240207072435.14182-2-saeed@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240207072435.14182-1-saeed@kernel.org> References: <20240207072435.14182-1-saeed@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790224300774134106 X-GMAIL-MSGID: 1790224300774134106 From: Saeed Mahameed Allow ctl protocol interface auxiliary driver in mlx5. Signed-off-by: Saeed Mahameed --- drivers/net/ethernet/mellanox/mlx5/core/dev.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/dev.c b/drivers/net/ethernet/mellanox/mlx5/core/dev.c index cf0477f53dc4..753c11f7050d 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/dev.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/dev.c @@ -228,8 +228,14 @@ enum { MLX5_INTERFACE_PROTOCOL_VNET, MLX5_INTERFACE_PROTOCOL_DPLL, + MLX5_INTERFACE_PROTOCOL_CTL, }; +static bool is_ctl_supported(struct mlx5_core_dev *dev) +{ + return MLX5_CAP_GEN(dev, uctx_cap); +} + static const struct mlx5_adev_device { const char *suffix; bool (*is_supported)(struct mlx5_core_dev *dev); @@ -252,6 +258,8 @@ static const struct mlx5_adev_device { .is_supported = &is_mp_supported }, [MLX5_INTERFACE_PROTOCOL_DPLL] = { .suffix = "dpll", .is_supported = &is_dpll_supported }, + [MLX5_INTERFACE_PROTOCOL_CTL] = { .suffix = "ctl", + .is_supported = &is_ctl_supported }, }; int mlx5_adev_idx_alloc(void) From patchwork Wed Feb 7 07:24:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 197784 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp2059905dyb; Tue, 6 Feb 2024 23:29:39 -0800 (PST) X-Google-Smtp-Source: AGHT+IEAfGyWy0HYWr+evt9HVEBqoz996oly4jWMZJPSCymFO7BkCH8RpXmqs9HTYt0Qqn9bdsEN X-Received: by 2002:a05:6214:f02:b0:685:4c80:c044 with SMTP id gw2-20020a0562140f0200b006854c80c044mr5825596qvb.7.1707290979034; Tue, 06 Feb 2024 23:29:39 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707290979; cv=pass; d=google.com; s=arc-20160816; b=n39AmtxLdcajdfRTN3F5bnZS0Qgsq/IKppxNPoYvznQccUiTUDa7A/fvAAzhCp2BAa NReB7tduVIRQIU6tra3kROHpxZO24L+Pz/sydX0PcPbSJYZcdy9tHAcA6wDueVAuKbQA bY1KVGfNtJTmdS8lu6yYsyuwATHobl3qF/bEnGtij/4sM6sAQoAqFBRam2448T+0t7UU qtXViokwgi2+571qP8eJ/F79/+XCgqZtS3+T6eVK6Q59fVtszAoClXf15YYPVaYVG+7K Xb4H0zasXTQstEUOaUAqmxx1OFni0Zi8kLA3J9j/17j5m1efz2B8PETYW1sHV9lOKVTI ghGw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=pHudq9CosK2UK8D9MLM4M9bFLkmGBN9X0lm5CFHDeuQ=; fh=vRadAjI6Pbh9X1tufEn5uMdj/usdqK5WZj5g5mznMSQ=; b=LVZS+4XK8X/ucxdYuNeIOtTcoUbOUY4R8+7Gq/xBahsqKAzNAwp671zJLPasSCLJiM TppudyP1LHYt/IfcAtu0+5vG0SlZXoNK4VuvyCMiGdVvyFoNv2g/h2MZK9ZepfdlBpI5 a5LZnywT0GApz4u4UmvsFMd4rA90+WDaoit123CBluBQXsopTSQbaf3v5bYvIQensgS1 TQdCPHFNqHtYCFYx3zNE5PLc+HBw756wmbxCUGSfz1WCdzw8DW9W7flUXD4FHsAoBKyb bRYqb5tPbgT+fVd2gxIMwRQwRDFnPCJFH7QGpXPKy8HyGF/PsvBLv29lC0Cgt2mFry0u cLuQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=N9K1oqiP; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-56074-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-56074-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=2; AJvYcCUlB5Yr+cGVVcl694HhQIDYeQaTBdYJBK4IYb4TpL9KiGEfiwhD+KcDgq4RoxzZX2ENa0G6/JTrJV/ti90sS7p/b5LtUg== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id m11-20020ad44b6b000000b0068c72820ce2si590804qvx.187.2024.02.06.23.29.38 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 23:29:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-56074-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=N9K1oqiP; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-56074-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-56074-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id C120A1C23FB7 for ; Wed, 7 Feb 2024 07:29:38 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id DBD6A20DFA; Wed, 7 Feb 2024 07:28:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="N9K1oqiP" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9220B200A5 for ; Wed, 7 Feb 2024 07:28:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707290906; cv=none; b=X5IVCNy0TaIjgZP2yQj+KkMGl2mI7tEeMe+0JNOZXaek06EYBQL9fmFvD1OaETrwHo0u0vL81AhAnd2kYrWU5Od+9BPCUVoTbH2blYhfS0badZLTjUuZV6M0Pis7FGkdnQqse8TeTWILOaTkhf/e9ewgI6ZzgZXFXjEZ7KSIx1I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707290906; c=relaxed/simple; bh=kuF2c9QD0mQvG5AXOSXT36rGaI17zhRjFvDMYHsbT3M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hNTkV6QN5BUFsB7RTZr1CqEEA0mYW58Dha9exCIlZx13ynQjuibWgIWIrSYX+vgUowzWgWjSlbCVyj6xsg53fHpmNX1Y3WNgr9pY/WqF1pFMJPD8ZU287cRc3gWFH8/KdrJT5DWgkTURmjNPua6JOZU+qERVoKtYxgxmNHGwp8c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=N9K1oqiP; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id EE1B1C433C7; Wed, 7 Feb 2024 07:28:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707290906; bh=kuF2c9QD0mQvG5AXOSXT36rGaI17zhRjFvDMYHsbT3M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=N9K1oqiPQ1e6W/62VCX81MwrdFQ+xfSo6rA8BQ3mJCqbyg8E4Yrh93Ti+ZOfI39SH Hc/DMcRRbzc5SrHUJqOs5NW8WwSHtlKhFHYirHgbsFhgeXLHVXCTqSp16P7+y+Z2Fz GEjWlF/ODLwhrmsSZkcHIf6iMmVY0fyHJKIOUV1Nteh6l6IJynTWXzELCdw+r18b59 zRjS19IzOOYKpcbpJBE0AVsIITiWee31U8QkmqYCkX+NGTblUzpnnxu0X0spCdHC4h IVdTdvKu0olOpbn7Phc8Xx2s2tD2kQxKjPJpL6rKpv/LV3C0X/zyI6P+K35v5U8nJd GoLw247/ZvZeA== From: Saeed Mahameed To: Arnd Bergmann , Greg Kroah-Hartman Cc: Leon Romanovsky , Jason Gunthorpe , Jiri Pirko , Leonid Bloch , Itay Avraham , Jakub Kicinski , Saeed Mahameed , David Ahern , Aron Silverton , Christoph Hellwig , andrew.gospodarek@broadcom.com, linux-kernel@vger.kernel.org Subject: [PATCH V4 2/5] misc: mlx5ctl: Add mlx5ctl misc driver Date: Tue, 6 Feb 2024 23:24:32 -0800 Message-ID: <20240207072435.14182-3-saeed@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240207072435.14182-1-saeed@kernel.org> References: <20240207072435.14182-1-saeed@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790224345587178148 X-GMAIL-MSGID: 1790224345587178148 From: Saeed Mahameed The ConnectX HW family supported by the mlx5 drivers uses an architecture where a FW component executes "mailbox RPCs" issued by the driver to make changes to the device. This results in a complex debugging environment where the FW component has information and low level configuration that needs to be accessed to userspace for debugging purposes. Historically a userspace program was used that accessed the PCI register and config space directly through /sys/bus/pci/.../XXX and could operate these debugging interfaces in parallel with the running driver. This approach is incompatible with secure boot and kernel lockdown so this driver provides a secure and restricted interface to that same data. On open the driver would allocate a special FW UID (user context ID) restrected to debug RPCs only, later in this series all user RPCs will be stamped with this UID. Reviewed-by: Jiri Pirko Reviewed-by: Leon Romanovsky Reviewed-by: Jason Gunthorpe Signed-off-by: Saeed Mahameed --- MAINTAINERS | 8 + drivers/misc/Kconfig | 1 + drivers/misc/Makefile | 1 + drivers/misc/mlx5ctl/Kconfig | 14 ++ drivers/misc/mlx5ctl/Makefile | 4 + drivers/misc/mlx5ctl/main.c | 332 ++++++++++++++++++++++++++++++++++ 6 files changed, 360 insertions(+) create mode 100644 drivers/misc/mlx5ctl/Kconfig create mode 100644 drivers/misc/mlx5ctl/Makefile create mode 100644 drivers/misc/mlx5ctl/main.c diff --git a/MAINTAINERS b/MAINTAINERS index 8d1052fa6a69..a41dc2056ae1 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -14011,6 +14011,14 @@ L: virtualization@lists.linux-foundation.org S: Supported F: drivers/vdpa/mlx5/ +MELLANOX MLX5 ConnectX Diag DRIVER +M: Saeed Mahameed +R: Itay Avraham +L: linux-kernel@vger.kernel.org +S: Supported +F: drivers/misc/mlx5ctl/ +F: include/uapi/misc/mlx5ctl.h + MELLANOX MLXCPLD I2C AND MUX DRIVER M: Vadim Pasternak M: Michael Shych diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig index 4fb291f0bf7c..4ab825acfd54 100644 --- a/drivers/misc/Kconfig +++ b/drivers/misc/Kconfig @@ -591,4 +591,5 @@ source "drivers/misc/cardreader/Kconfig" source "drivers/misc/uacce/Kconfig" source "drivers/misc/pvpanic/Kconfig" source "drivers/misc/mchp_pci1xxxx/Kconfig" +source "drivers/misc/mlx5ctl/Kconfig" endmenu diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile index ea6ea5bbbc9c..c491c2b8ac88 100644 --- a/drivers/misc/Makefile +++ b/drivers/misc/Makefile @@ -68,3 +68,4 @@ obj-$(CONFIG_TMR_INJECT) += xilinx_tmr_inject.o obj-$(CONFIG_TPS6594_ESM) += tps6594-esm.o obj-$(CONFIG_TPS6594_PFSM) += tps6594-pfsm.o obj-$(CONFIG_NSM) += nsm.o +obj-$(CONFIG_MLX5CTL) += mlx5ctl/ diff --git a/drivers/misc/mlx5ctl/Kconfig b/drivers/misc/mlx5ctl/Kconfig new file mode 100644 index 000000000000..faaa1dba2cc2 --- /dev/null +++ b/drivers/misc/mlx5ctl/Kconfig @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: GPL-2.0 +# + +config MLX5CTL + tristate "mlx5 ConnectX control misc driver" + depends on MLX5_CORE + help + MLX5CTL provides interface for the user process to access the debug and + configuration registers of the ConnectX hardware family + (NICs, PCI switches and SmartNIC SoCs). + This will allow configuration and debug tools to work out of the box on + mainstream kernel. + + If you don't know what to do here, say N. diff --git a/drivers/misc/mlx5ctl/Makefile b/drivers/misc/mlx5ctl/Makefile new file mode 100644 index 000000000000..b5c7f99e0ab6 --- /dev/null +++ b/drivers/misc/mlx5ctl/Makefile @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: GPL-2.0 + +obj-$(CONFIG_MLX5CTL) += mlx5ctl.o +mlx5ctl-y := main.o diff --git a/drivers/misc/mlx5ctl/main.c b/drivers/misc/mlx5ctl/main.c new file mode 100644 index 000000000000..314aeb45bb8e --- /dev/null +++ b/drivers/misc/mlx5ctl/main.c @@ -0,0 +1,332 @@ +// SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 +/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +MODULE_DESCRIPTION("mlx5 ConnectX control misc driver"); +MODULE_AUTHOR("Saeed Mahameed "); +MODULE_LICENSE("Dual BSD/GPL"); + +/* mlx5ctl_dev lifecycle locking scheme: + * + * 1. mcdev->rw_lock: protects mdev from removal on mlx5ctl_remove() + * + * - Write lock is taken by auxiliary_driver.remove + * - Set mdev to NULL + * - Read lock is taken by open/release and ioctls + * - To prevent mdev from being removed + * - Check mdev is not NULL, abort otherwise + * + * 2. mcdev->refcount: protects mcdev from removal after miscdevice is unregistered + * - miscdevice does not have a refcount, so we use kref + * - miscdevice is unregistered on mlx5ctl_remove() + * - already open fds will still have a reference to mcdev + * - mcdev is freed when refcount reaches 0 on last fd release + */ +struct mlx5ctl_dev { + struct mlx5_core_dev *mdev; + struct miscdevice miscdev; + struct auxiliary_device *adev; + struct list_head fd_list; + spinlock_t fd_list_lock; /* protect list add/del */ + struct rw_semaphore rw_lock; /* protect mdev from device removal */ + struct kref refcount; +}; + +struct mlx5ctl_fd { + u16 uctx_uid; + u32 uctx_cap; + u32 ucap; /* user cap */ + struct mlx5ctl_dev *mcdev; + struct list_head list; +}; + +#define mlx5ctl_err(mcdev, format, ...) \ + dev_err(mcdev->miscdev.parent, format, ##__VA_ARGS__) + +#define mlx5ctl_dbg(mcdev, format, ...) \ + dev_dbg(mcdev->miscdev.parent, "PID %d: " format, \ + current->pid, ##__VA_ARGS__) + +enum { + MLX5_UCTX_OBJECT_CAP_RAW_TX = 0x1, + MLX5_UCTX_OBJECT_CAP_INTERNAL_DEVICE_RESOURCES = 0x2, + MLX5_UCTX_OBJECT_CAP_TOOLS_RESOURCES = 0x4, +}; + +static int mlx5ctl_alloc_uid(struct mlx5ctl_dev *mcdev, u32 cap) +{ + u32 out[MLX5_ST_SZ_DW(create_uctx_out)] = {}; + u32 in[MLX5_ST_SZ_DW(create_uctx_in)] = {}; + void *uctx; + int err; + u16 uid; + + uctx = MLX5_ADDR_OF(create_uctx_in, in, uctx); + + mlx5ctl_dbg(mcdev, "MLX5_CMD_OP_CREATE_UCTX: caps 0x%x\n", cap); + MLX5_SET(create_uctx_in, in, opcode, MLX5_CMD_OP_CREATE_UCTX); + MLX5_SET(uctx, uctx, cap, cap); + + err = mlx5_cmd_exec(mcdev->mdev, in, sizeof(in), out, sizeof(out)); + if (err) + return err; + + uid = MLX5_GET(create_uctx_out, out, uid); + mlx5ctl_dbg(mcdev, "allocated uid %d with caps 0x%x\n", uid, cap); + return uid; +} + +static void mlx5ctl_release_uid(struct mlx5ctl_dev *mcdev, u16 uid) +{ + u32 in[MLX5_ST_SZ_DW(destroy_uctx_in)] = {}; + struct mlx5_core_dev *mdev = mcdev->mdev; + int err; + + MLX5_SET(destroy_uctx_in, in, opcode, MLX5_CMD_OP_DESTROY_UCTX); + MLX5_SET(destroy_uctx_in, in, uid, uid); + + err = mlx5_cmd_exec_in(mdev, destroy_uctx, in); + mlx5ctl_dbg(mcdev, "released uid %d err(%d)\n", uid, err); +} + +static void mcdev_get(struct mlx5ctl_dev *mcdev); +static void mcdev_put(struct mlx5ctl_dev *mcdev); + +static int mlx5ctl_open_mfd(struct mlx5ctl_fd *mfd) +{ + struct mlx5_core_dev *mdev = mfd->mcdev->mdev; + struct mlx5ctl_dev *mcdev = mfd->mcdev; + u32 ucap = 0, cap = 0; + int uid; + +#define MLX5_UCTX_CAP(mdev, cap) \ + (MLX5_CAP_GEN(mdev, uctx_cap) & MLX5_UCTX_OBJECT_CAP_##cap) + + if (capable(CAP_NET_RAW) && MLX5_UCTX_CAP(mdev, RAW_TX)) { + ucap |= CAP_NET_RAW; + cap |= MLX5_UCTX_OBJECT_CAP_RAW_TX; + } + + if (capable(CAP_SYS_RAWIO) && MLX5_UCTX_CAP(mdev, INTERNAL_DEVICE_RESOURCES)) { + ucap |= CAP_SYS_RAWIO; + cap |= MLX5_UCTX_OBJECT_CAP_INTERNAL_DEVICE_RESOURCES; + } + + if (capable(CAP_SYS_ADMIN) && MLX5_UCTX_CAP(mdev, TOOLS_RESOURCES)) { + ucap |= CAP_SYS_ADMIN; + cap |= MLX5_UCTX_OBJECT_CAP_TOOLS_RESOURCES; + } + + uid = mlx5ctl_alloc_uid(mcdev, cap); + if (uid < 0) + return uid; + + mfd->uctx_uid = uid; + mfd->uctx_cap = cap; + mfd->ucap = ucap; + mfd->mcdev = mcdev; + + mlx5ctl_dbg(mcdev, "allocated uid %d with uctx caps 0x%x, user cap 0x%x\n", + uid, cap, ucap); + return 0; +} + +static void mlx5ctl_release_mfd(struct mlx5ctl_fd *mfd) +{ + struct mlx5ctl_dev *mcdev = mfd->mcdev; + + mlx5ctl_release_uid(mcdev, mfd->uctx_uid); +} + +static int mlx5ctl_open(struct inode *inode, struct file *file) +{ + struct mlx5_core_dev *mdev; + struct mlx5ctl_dev *mcdev; + struct mlx5ctl_fd *mfd; + int err = 0; + + mcdev = container_of(file->private_data, struct mlx5ctl_dev, miscdev); + mcdev_get(mcdev); + down_read(&mcdev->rw_lock); + mdev = mcdev->mdev; + if (!mdev) { + err = -ENODEV; + goto unlock; + } + + mfd = kzalloc(sizeof(*mfd), GFP_KERNEL_ACCOUNT); + if (!mfd) + return -ENOMEM; + + mfd->mcdev = mcdev; + err = mlx5ctl_open_mfd(mfd); + if (err) + goto unlock; + + spin_lock(&mcdev->fd_list_lock); + list_add_tail(&mfd->list, &mcdev->fd_list); + spin_unlock(&mcdev->fd_list_lock); + + file->private_data = mfd; + +unlock: + up_read(&mcdev->rw_lock); + if (err) { + mcdev_put(mcdev); + kfree(mfd); + } + return err; +} + +static int mlx5ctl_release(struct inode *inode, struct file *file) +{ + struct mlx5ctl_fd *mfd = file->private_data; + struct mlx5ctl_dev *mcdev = mfd->mcdev; + + down_read(&mcdev->rw_lock); + if (!mcdev->mdev) { + pr_debug("[%d] UID %d mlx5ctl: mdev is already released\n", + current->pid, mfd->uctx_uid); + /* All mfds are already released, skip ... */ + goto unlock; + } + + spin_lock(&mcdev->fd_list_lock); + list_del(&mfd->list); + spin_unlock(&mcdev->fd_list_lock); + + mlx5ctl_release_mfd(mfd); + +unlock: + kfree(mfd); + up_read(&mcdev->rw_lock); + mcdev_put(mcdev); + file->private_data = NULL; + return 0; +} + +static const struct file_operations mlx5ctl_fops = { + .owner = THIS_MODULE, + .open = mlx5ctl_open, + .release = mlx5ctl_release, +}; + +static int mlx5ctl_probe(struct auxiliary_device *adev, + const struct auxiliary_device_id *id) + +{ + struct mlx5_adev *madev = container_of(adev, struct mlx5_adev, adev); + struct mlx5_core_dev *mdev = madev->mdev; + struct mlx5ctl_dev *mcdev; + char *devname = NULL; + int err; + + mcdev = kzalloc(sizeof(*mcdev), GFP_KERNEL_ACCOUNT); + if (!mcdev) + return -ENOMEM; + + kref_init(&mcdev->refcount); + INIT_LIST_HEAD(&mcdev->fd_list); + spin_lock_init(&mcdev->fd_list_lock); + init_rwsem(&mcdev->rw_lock); + mcdev->mdev = mdev; + mcdev->adev = adev; + devname = kasprintf(GFP_KERNEL_ACCOUNT, "mlx5ctl-%s", + dev_name(&adev->dev)); + if (!devname) { + err = -ENOMEM; + goto abort; + } + + mcdev->miscdev = (struct miscdevice) { + .minor = MISC_DYNAMIC_MINOR, + .name = devname, + .fops = &mlx5ctl_fops, + .parent = &adev->dev, + }; + + err = misc_register(&mcdev->miscdev); + if (err) { + mlx5ctl_err(mcdev, "mlx5ctl: failed to register misc device err %d\n", err); + goto abort; + } + + mlx5ctl_dbg(mcdev, "probe mdev@%s %s\n", + dev_driver_string(mdev->device), + dev_name(mdev->device)); + + auxiliary_set_drvdata(adev, mcdev); + + return 0; + +abort: + kfree(devname); + kfree(mcdev); + return err; +} + +static void mlx5ctl_remove(struct auxiliary_device *adev) +{ + struct mlx5ctl_dev *mcdev = auxiliary_get_drvdata(adev); + struct mlx5_core_dev *mdev = mcdev->mdev; + struct mlx5ctl_fd *mfd, *n; + + misc_deregister(&mcdev->miscdev); + down_write(&mcdev->rw_lock); + + list_for_each_entry_safe(mfd, n, &mcdev->fd_list, list) { + mlx5ctl_dbg(mcdev, "UID %d still has open FDs\n", mfd->uctx_uid); + list_del(&mfd->list); + mlx5ctl_release_mfd(mfd); + } + + mlx5ctl_dbg(mcdev, "removed mdev %s %s\n", + dev_driver_string(mdev->device), dev_name(mdev->device)); + + mcdev->mdev = NULL; /* prevent already open fds from accessing the device */ + up_write(&mcdev->rw_lock); + mcdev_put(mcdev); +} + +static void mcdev_free(struct kref *ref) +{ + struct mlx5ctl_dev *mcdev = container_of(ref, struct mlx5ctl_dev, refcount); + + kfree(mcdev->miscdev.name); + kfree(mcdev); +} + +static void mcdev_get(struct mlx5ctl_dev *mcdev) +{ + kref_get(&mcdev->refcount); +} + +static void mcdev_put(struct mlx5ctl_dev *mcdev) +{ + kref_put(&mcdev->refcount, mcdev_free); +} + +static const struct auxiliary_device_id mlx5ctl_id_table[] = { + { .name = MLX5_ADEV_NAME ".ctl", }, + {}, +}; + +MODULE_DEVICE_TABLE(auxiliary, mlx5ctl_id_table); + +static struct auxiliary_driver mlx5ctl_driver = { + .name = "ctl", + .probe = mlx5ctl_probe, + .remove = mlx5ctl_remove, + .id_table = mlx5ctl_id_table, +}; + +module_auxiliary_driver(mlx5ctl_driver); From patchwork Wed Feb 7 07:24:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 197783 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp2059826dyb; Tue, 6 Feb 2024 23:29:27 -0800 (PST) X-Google-Smtp-Source: AGHT+IF21S2k1knd5TrAhj9oamypZk3dWfN0pDIOatByA+6FD6lYKUOu+EClfmnGMaBypSs40FFE X-Received: by 2002:ac8:5745:0:b0:42b:ebf7:d207 with SMTP id 5-20020ac85745000000b0042bebf7d207mr9763849qtx.29.1707290967632; Tue, 06 Feb 2024 23:29:27 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707290967; cv=pass; d=google.com; s=arc-20160816; b=QxSQ4Eb9HHZyNiLQhPI/KmMJaTbYDoPAyaZRylswkiJRwqWtG1UyWpgRZOIY3ZilTt 8dMzouwGN8b3Tc50ZvGxqTSLWt9/gRvDaNLrvQsQvo8fwmLIafWXSvXQt7WS8avREniF WtwfQmrWDTl/mMHaEiBK6kIc/94aRWyVSg00nxL2pJ80aPit2ofu8M9mBP9d8SlAqYrC Qhkoqv7QY9hN+2r1/kwFJyWUULxv5ntUYI/yajJQ67LYFdGCzwdDDrrZW9y+8OWGWclY CrlhvdqIVvZxxxY0imG3CeTHwOOZk/nlrXcznJET13NRK5TtjcaDV088HitcGTccRGSA aXXQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=ZQO2/3z5y2fdy5JDm+n43uR7LpbQuGI/cFanJohlUMw=; fh=hIG0VOPpFLB700nArIjxZFXmIkzP0XnKSTKd8llrV7s=; b=FSMlsMNXUgvzQC7YYYSvWhoi6eQDHSwVH+Ye/w4riH0E+YN3hDkQHvt8HuwtPuSITW LTIvppec7msgfX+TY7VYbr6iGx3AvUnCm/O9y3FgA/DtmybTmNg3wwueF6I69i13+Wco t3nzL+tkFXxX6OvGFgFY/m+W5cLlUOZwijvCZKaRWF+aBZKRWaqB1dtC2471IF3ikYcV 6cHDsM2GzKqf49AnKz3Y8FSA0HM8ylHPuj02KaKznza4h3T29Tn+RNvXrpGWRzMIMcQw Ixka7tbj6xCjhXjBL+UEQxMPOgRcsRSX05UwuyvfZ0tsnPE9wXDCCaJFi/Tef28yWE4o pQGA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=g3bolgMW; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-56075-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-56075-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=2; AJvYcCXarEmaYK5xN3exPs6dsl/7t4l+sZ6AjImsDo3iZUiWdOH+7sN7hI4EbKgvlGO1B3g+GzXZZAucn4UZLyvtlUI58SciSQ== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id d22-20020ac85ad6000000b0042c26e5e347si586010qtd.253.2024.02.06.23.29.27 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 23:29:27 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-56075-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=g3bolgMW; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-56075-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-56075-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 5F0CB1C235E4 for ; Wed, 7 Feb 2024 07:29:27 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id DEABF20B17; Wed, 7 Feb 2024 07:28:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="g3bolgMW" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C1A1200D8 for ; Wed, 7 Feb 2024 07:28:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707290907; cv=none; b=EHYZTLJIoD8HTxkQD1GMn0JVfDXC6EpdACujMVBOCnogPGJ226Q18rdiRzGks7JjVhzFDcNHdr2fUupsq9VftZRb1ccPgmLSM9wEeWvUBqdvKaUn6uTiOpUX6+hdSpL6d69/1QaIh1QhW1cDsCZbNago1qbZ+SYbv9Sgm+TjL6o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707290907; c=relaxed/simple; bh=HJ8IhDMnRtXRcH7QQWNi/3y6XYnFHwSd1H1sIxy1Luo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uJ4FTg4lFYiDuCvaCAFSW7uNaMBSRUEwT5ZSieFuLz4FBA69l4La8O/gy2nHeWGVNOcKP0O2qiWvTuqzZJTkTYw9XKIMsAp0DnkknJOwfBK9rmxoWIEIhckxeZmtVCUblNNe2PrEQasuqdIiKKyMfIhadQ7lWm38HlDeKS4zcCo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=g3bolgMW; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id E6BE8C433F1; Wed, 7 Feb 2024 07:28:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707290907; bh=HJ8IhDMnRtXRcH7QQWNi/3y6XYnFHwSd1H1sIxy1Luo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=g3bolgMWjULeKQeo+0Oa9LrdecwhDRU38coASeGAR3jLmh6ZBKfnbaIj1VQebVm0C W0Dw5FE4WeUfIJM28Lx0LFaw7OhhcEFLWfU3pjTA3XWt9jlQ6HRwoPsVoApPDI4eHt MtZWzItScs3bZsCm5PfckL3/XkiyapCnCVMWZnIM7hIyLSStwyXdXbVVWd99yfCcOK mGjicbOdj5bPpDYp5wX0SVPn3MRNC7QaH/PmlBP3naTXwmKDyankEMQ3SI+aYWFYB8 tzj5F81DEJ4XS9fSZIoHZjKXhM50eS+olG9L/oAntJygN3YLG3XA7WuSP7rYErR41e G3s0BVw8o31sw== From: Saeed Mahameed To: Arnd Bergmann , Greg Kroah-Hartman Cc: Leon Romanovsky , Jason Gunthorpe , Jiri Pirko , Leonid Bloch , Itay Avraham , Jakub Kicinski , Saeed Mahameed , David Ahern , Aron Silverton , Christoph Hellwig , andrew.gospodarek@broadcom.com, linux-kernel@vger.kernel.org Subject: [PATCH V4 3/5] misc: mlx5ctl: Add info ioctl Date: Tue, 6 Feb 2024 23:24:33 -0800 Message-ID: <20240207072435.14182-4-saeed@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240207072435.14182-1-saeed@kernel.org> References: <20240207072435.14182-1-saeed@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790224333704170724 X-GMAIL-MSGID: 1790224333704170724 From: Saeed Mahameed Implement INFO ioctl to return the allocated UID and the capability flags and some other useful device information. Example: $ sudo ./mlx5ctl mlx5_core.ctl.0 UCTX UID: 1 UCTX CAP: 0x3 DEV UCTX CAP: 0x3 USER CAP: 0x1d Reviewed-by: Jiri Pirko Reviewed-by: Leon Romanovsky Reviewed-by: Jason Gunthorpe Signed-off-by: Saeed Mahameed --- .../userspace-api/ioctl/ioctl-number.rst | 1 + drivers/misc/mlx5ctl/main.c | 68 +++++++++++++++++++ include/uapi/misc/mlx5ctl.h | 20 ++++++ 3 files changed, 89 insertions(+) create mode 100644 include/uapi/misc/mlx5ctl.h diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst index 457e16f06e04..17e6ab6d0d9f 100644 --- a/Documentation/userspace-api/ioctl/ioctl-number.rst +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst @@ -89,6 +89,7 @@ Code Seq# Include File Comments 0x20 all drivers/cdrom/cm206.h 0x22 all scsi/sg.h 0x3E 00-0F linux/counter.h +0x5c all uapi/misc/mlx5ctl.h Nvidia ConnectX control '!' 00-1F uapi/linux/seccomp.h '#' 00-3F IEEE 1394 Subsystem Block for the entire subsystem diff --git a/drivers/misc/mlx5ctl/main.c b/drivers/misc/mlx5ctl/main.c index 314aeb45bb8e..e4e70359dbe8 100644 --- a/drivers/misc/mlx5ctl/main.c +++ b/drivers/misc/mlx5ctl/main.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include @@ -214,10 +215,77 @@ static int mlx5ctl_release(struct inode *inode, struct file *file) return 0; } +static int mlx5ctl_info_ioctl(struct file *file, + struct mlx5ctl_info __user *arg, + size_t usize) +{ + struct mlx5ctl_fd *mfd = file->private_data; + size_t ksize = sizeof(struct mlx5ctl_info); + struct mlx5ctl_dev *mcdev = mfd->mcdev; + struct mlx5_core_dev *mdev = mcdev->mdev; + struct mlx5ctl_info *info; + int err = 0; + + if (usize < ksize) + return -EINVAL; + + info = kzalloc(ksize, GFP_KERNEL); + if (!info) + return -ENOMEM; + + info->dev_uctx_cap = MLX5_CAP_GEN(mdev, uctx_cap); + info->uctx_cap = mfd->uctx_cap; + info->uctx_uid = mfd->uctx_uid; + info->ucap = mfd->ucap; + + if (copy_to_user(arg, info, ksize)) + err = -EFAULT; + + kfree(info); + return err; +} + +static long mlx5ctl_ioctl(struct file *file, unsigned int cmd, unsigned long arg) +{ + struct mlx5ctl_fd *mfd = file->private_data; + struct mlx5ctl_dev *mcdev = mfd->mcdev; + void __user *argp = (void __user *)arg; + size_t size = _IOC_SIZE(cmd); + int err = 0; + + if (!capable(CAP_SYS_ADMIN)) + return -EPERM; + + mlx5ctl_dbg(mcdev, "ioctl 0x%x type/nr: %d/%d size: %d DIR:%d\n", cmd, + _IOC_TYPE(cmd), _IOC_NR(cmd), _IOC_SIZE(cmd), _IOC_DIR(cmd)); + + down_read(&mcdev->rw_lock); + if (!mcdev->mdev) { + err = -ENODEV; + goto unlock; + } + + switch (cmd) { + case MLX5CTL_IOCTL_INFO: + err = mlx5ctl_info_ioctl(file, argp, size); + break; + + default: + mlx5ctl_dbg(mcdev, "Unknown ioctl %x\n", cmd); + err = -ENOTTY; + break; + } +unlock: + up_read(&mcdev->rw_lock); + return err; +} + static const struct file_operations mlx5ctl_fops = { .owner = THIS_MODULE, .open = mlx5ctl_open, .release = mlx5ctl_release, + .unlocked_ioctl = mlx5ctl_ioctl, + .compat_ioctl = compat_ptr_ioctl, }; static int mlx5ctl_probe(struct auxiliary_device *adev, diff --git a/include/uapi/misc/mlx5ctl.h b/include/uapi/misc/mlx5ctl.h new file mode 100644 index 000000000000..9be944128025 --- /dev/null +++ b/include/uapi/misc/mlx5ctl.h @@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 WITH Linux-syscall-note */ +/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */ + +#ifndef __MLX5CTL_IOCTL_H__ +#define __MLX5CTL_IOCTL_H__ + +struct mlx5ctl_info { + __u16 uctx_uid; /* current process allocated UCTX UID */ + __u16 reserved1; /* explicit padding must be zero */ + __u32 uctx_cap; /* current process effective UCTX cap */ + __u32 dev_uctx_cap; /* device's UCTX capabilities */ + __u32 ucap; /* process user capability */ +}; + +#define MLX5CTL_IOCTL_MAGIC 0x5c + +#define MLX5CTL_IOCTL_INFO \ + _IOR(MLX5CTL_IOCTL_MAGIC, 0x0, struct mlx5ctl_info) + +#endif /* __MLX5CTL_IOCTL_H__ */ From patchwork Wed Feb 7 07:24:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 197786 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp2061629dyb; Tue, 6 Feb 2024 23:33:30 -0800 (PST) X-Google-Smtp-Source: AGHT+IG9vKMDyHR7FayPEQ5AqfgaAgnbF628Qs5VDm+XItnGc5XXEBFMANO1SjD3F9hQ+XbhbTJs X-Received: by 2002:a62:ce82:0:b0:6dd:a072:867 with SMTP id y124-20020a62ce82000000b006dda0720867mr1798290pfg.15.1707291210155; Tue, 06 Feb 2024 23:33:30 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707291210; cv=pass; d=google.com; s=arc-20160816; b=Lce+HsFC5+0XYk3l8UXwpbw1rs8hZbz8u9qD4DO8deAlQ6zEkF87v1k7iiA8cdUDLd /sjpu+EhCzof97FZCKPRVBo4GET9Dsygbyof4y/Et1W2MRrYOn2ylR2Epg/rzzSXjozi LyfJZUP3qLIKLfEKcq+J7cxrFA2IZUoW2GOLkrMZzKLFt0SdSzVn+3PW2/s2GLhHqhKB tKFOvle4kyNZaZ+sp+zrUYwXk5pphf/E1GVF/XG7pJM4PHwLagS9AIz+PQYIjQK7yxB1 w7hXA9Avp6OUQzqU8aZzITQorrTtEKL5vSezBC1+U6eexdL84lODVYUzJ3GI9BhEVffy uOqQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=QNN760rSp2aLl/aqcBN1X/32QXU4X/5iWMdhFZ7Jr6Y=; fh=eI1+GQbDW7jYL1h2XL8YYDYat6Yb0cYpF9/tf/4/PS8=; b=AfMnIItcuh0hoOAOkyQPzkRaMg6Klomt6CvokpOxgMxs49HPL+tqexiAvxVuKs7/ex vUPsrEqgt3RZrGtVIZhzT5rIkqnw1zPd1X0bvEPk8LCXIMF3R4zM6O37CeGKTiEbqB3P ZwTb4LdewhBCgrk8DH40ivAIz0gN/J9gGEklGUi51cQxFe5eeBPrQdOXp5CWeKLFb3aR W74faF9L1DkxvCG4MZl9AyTB0wPch0bJOp8mw0XL8C5OzRR9N68ODG336M76FFmV9d9h wy15jrO6nBATnU91HAIiUiGj6B6IheNFLFCKirZpHnqEOxutHP3DPB43Y5Kc8gY/kbik FH7w==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=g0dpA67q; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-56076-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-56076-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org X-Forwarded-Encrypted: i=2; AJvYcCVOuM4RCZPBUq6piCD+SmJai0/MsYOhXUJRO1pkttMCPV7/6JccZEA2dk4QsnvUM0ekOrV4aHoMT1bYaYP+5V2hEQ+rZg== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id o26-20020a63921a000000b005d8ad5aa264si837903pgd.798.2024.02.06.23.33.29 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 23:33:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-56076-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=g0dpA67q; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-56076-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-56076-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 1C4E2B24B2F for ; Wed, 7 Feb 2024 07:29:39 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D637820DE5; Wed, 7 Feb 2024 07:28:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="g0dpA67q" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4F56C2031E for ; Wed, 7 Feb 2024 07:28:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707290908; cv=none; b=qHlgnq+1fnnDWrBF3TvS5w3LWSbAimeKPjLT1NndaXW6jm12hvYqgwURa/ST/M+Np1hY6A1agMz0piaAoND952UKK2ia70QhtHuI+GgEu7zqSexqmf0rIbscZk5j+ZvtBjN3tWdSuP4RWOHVx0U+SNIDAfgTpdZFXJ827DZInlE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707290908; c=relaxed/simple; bh=bWaLjtidBr05TM3D/n6lluQMqun8vTEheO+oRY9qZFc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZB34D5fI4IlhBkFuyVnp/+CJb1fHRIsffPoNENGG0L7YGiyrQqRdT7ykQ18P4bapiRYZ47jybWUOdmKPpz9cTdIEmtYEd73MOWieRqo+0WAp/ACHE/hishgnXNuO7XUm6AFWS+jFRMWs+U/SVyQYL9b3gOqddzjEHvpqM2zi4iI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=g0dpA67q; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id C9A13C43394; Wed, 7 Feb 2024 07:28:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707290907; bh=bWaLjtidBr05TM3D/n6lluQMqun8vTEheO+oRY9qZFc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=g0dpA67qPnQmmuH5cjhc5BndXI09AW43QOWyyduSQG5elWe3RG0Wmq2ycUfQh+qmE 8QlJsjkZSrNDFyszol3FV/sSxkwhOdQ1+zTbBYy/TLep+hgGpecQlYPorbKhcbez5M RCbbnpTNpj9pmMHtS1oU91nFgpGSKv0eNqyUnufn/MOGbCDJnhOphOy6jOva/bkOJ+ jFzzP5ZHmhQqklhc+E0r28jFbopXqyK7CmJYfnGTkSlEvl2l8YO42TXJ77ovR/s4au swyvN7C4avpqkiCBHd4dvAqPo8bnOBmj/+SctTwEbNtIhnl0VWK3QWXwGwZB8pAMp7 Wu7AcsINtBzTg== From: Saeed Mahameed To: Arnd Bergmann , Greg Kroah-Hartman Cc: Leon Romanovsky , Jason Gunthorpe , Jiri Pirko , Leonid Bloch , Itay Avraham , Jakub Kicinski , Saeed Mahameed , David Ahern , Aron Silverton , Christoph Hellwig , andrew.gospodarek@broadcom.com, linux-kernel@vger.kernel.org Subject: [PATCH V4 4/5] misc: mlx5ctl: Add command rpc ioctl Date: Tue, 6 Feb 2024 23:24:34 -0800 Message-ID: <20240207072435.14182-5-saeed@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240207072435.14182-1-saeed@kernel.org> References: <20240207072435.14182-1-saeed@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790224587633248940 X-GMAIL-MSGID: 1790224587633248940 From: Saeed Mahameed Add new IOCTL to allow user space to send device debug rpcs and attach the user's uctx UID to each rpc. In the mlx5 architecture the FW RPC commands are of the format of inbox and outbox buffers. The inbox buffer contains the command rpc layout as described in the ConnectX Programmers Reference Manual (PRM) document and as defined in include/linux/mlx5/mlx5_ifc.h. On success the user outbox buffer will be filled with the device's rpc response. For example to query device capabilities: a user fills out an inbox buffer with the inbox layout: struct mlx5_ifc_query_hca_cap_in_bits and expects an outbox buffer with the layout: struct mlx5_ifc_cmd_hca_cap_bits Reviewed-by: Jiri Pirko Reviewed-by: Leon Romanovsky Reviewed-by: Jason Gunthorpe Signed-off-by: Saeed Mahameed --- drivers/misc/mlx5ctl/main.c | 98 +++++++++++++++++++++++++++++++++++++ include/uapi/misc/mlx5ctl.h | 12 +++++ 2 files changed, 110 insertions(+) diff --git a/drivers/misc/mlx5ctl/main.c b/drivers/misc/mlx5ctl/main.c index e4e70359dbe8..c02b80efffc1 100644 --- a/drivers/misc/mlx5ctl/main.c +++ b/drivers/misc/mlx5ctl/main.c @@ -245,6 +245,94 @@ static int mlx5ctl_info_ioctl(struct file *file, return err; } +struct mlx5_ifc_mbox_in_hdr_bits { + u8 opcode[0x10]; + u8 uid[0x10]; + + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + + u8 reserved_at_40[0x40]; +}; + +struct mlx5_ifc_mbox_out_hdr_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + + u8 syndrome[0x20]; + + u8 reserved_at_40[0x40]; +}; + +static int mlx5ctl_cmdrpc_ioctl(struct file *file, + struct mlx5ctl_cmdrpc __user *arg, + size_t usize) +{ + size_t ksize = sizeof(struct mlx5ctl_cmdrpc); + struct mlx5ctl_fd *mfd = file->private_data; + struct mlx5ctl_dev *mcdev = mfd->mcdev; + struct mlx5ctl_cmdrpc *rpc = NULL; + void *in = NULL, *out = NULL; + int err; + + if (usize < ksize) + return -EINVAL; + + rpc = kzalloc(ksize, GFP_KERNEL); + if (!rpc) + return -ENOMEM; + + err = copy_from_user(rpc, arg, usize); + if (err) + goto out; + + mlx5ctl_dbg(mcdev, "[UID %d] cmdrpc: inlen %d outlen %d\n", + mfd->uctx_uid, rpc->inlen, rpc->outlen); + + if (rpc->inlen < MLX5_ST_SZ_BYTES(mbox_in_hdr) || + rpc->outlen < MLX5_ST_SZ_BYTES(mbox_out_hdr) || + rpc->inlen > MLX5CTL_MAX_RPC_SIZE || + rpc->outlen > MLX5CTL_MAX_RPC_SIZE) { + err = -EINVAL; + goto out; + } + + in = memdup_user(u64_to_user_ptr(rpc->in), rpc->inlen); + if (IS_ERR(in)) { + err = PTR_ERR(in); + goto out; + } + + out = kvzalloc(rpc->outlen, GFP_KERNEL); + if (!out) { + err = -ENOMEM; + goto out; + } + + mlx5ctl_dbg(mcdev, "[UID %d] cmdif: opcode 0x%x inlen %d outlen %d\n", + mfd->uctx_uid, + MLX5_GET(mbox_in_hdr, in, opcode), rpc->inlen, rpc->outlen); + + MLX5_SET(mbox_in_hdr, in, uid, mfd->uctx_uid); + err = mlx5_cmd_do(mcdev->mdev, in, rpc->inlen, out, rpc->outlen); + mlx5ctl_dbg(mcdev, "[UID %d] cmdif: opcode 0x%x retval %d\n", + mfd->uctx_uid, + MLX5_GET(mbox_in_hdr, in, opcode), err); + + /* -EREMOTEIO means outbox is valid, but out.status is not */ + if (!err || err == -EREMOTEIO) { + err = 0; + if (copy_to_user(u64_to_user_ptr(rpc->out), out, rpc->outlen)) + err = -EFAULT; + } + +out: + kvfree(out); + kfree(in); + kfree(rpc); + return err; +} + static long mlx5ctl_ioctl(struct file *file, unsigned int cmd, unsigned long arg) { struct mlx5ctl_fd *mfd = file->private_data; @@ -270,6 +358,10 @@ static long mlx5ctl_ioctl(struct file *file, unsigned int cmd, unsigned long arg err = mlx5ctl_info_ioctl(file, argp, size); break; + case MLX5CTL_IOCTL_CMDRPC: + err = mlx5ctl_cmdrpc_ioctl(file, argp, size); + break; + default: mlx5ctl_dbg(mcdev, "Unknown ioctl %x\n", cmd); err = -ENOTTY; @@ -328,6 +420,11 @@ static int mlx5ctl_probe(struct auxiliary_device *adev, goto abort; } + err = sysfs_create_link_nowarn(&mcdev->miscdev.this_device->kobj, + &mdev->device->kobj, "mdev"); + if (err) + mlx5ctl_dbg(mcdev, "mlx5ctl: failed to create sysfs link err %d\n", err); + mlx5ctl_dbg(mcdev, "probe mdev@%s %s\n", dev_driver_string(mdev->device), dev_name(mdev->device)); @@ -348,6 +445,7 @@ static void mlx5ctl_remove(struct auxiliary_device *adev) struct mlx5_core_dev *mdev = mcdev->mdev; struct mlx5ctl_fd *mfd, *n; + sysfs_remove_link(&mcdev->miscdev.this_device->kobj, "mdev"); misc_deregister(&mcdev->miscdev); down_write(&mcdev->rw_lock); diff --git a/include/uapi/misc/mlx5ctl.h b/include/uapi/misc/mlx5ctl.h index 9be944128025..1e4622c5979f 100644 --- a/include/uapi/misc/mlx5ctl.h +++ b/include/uapi/misc/mlx5ctl.h @@ -12,9 +12,21 @@ struct mlx5ctl_info { __u32 ucap; /* process user capability */ }; +struct mlx5ctl_cmdrpc { + __aligned_u64 in; /* RPC inbox buffer user address */ + __aligned_u64 out; /* RPC outbox buffer user address */ + __u32 inlen; /* inbox buffer length */ + __u32 outlen; /* outbox buffer length */ +}; + +#define MLX5CTL_MAX_RPC_SIZE (512 * 512) /* max FW RPC buffer size 512 blocks of 512 bytes */ + #define MLX5CTL_IOCTL_MAGIC 0x5c #define MLX5CTL_IOCTL_INFO \ _IOR(MLX5CTL_IOCTL_MAGIC, 0x0, struct mlx5ctl_info) +#define MLX5CTL_IOCTL_CMDRPC \ + _IOWR(MLX5CTL_IOCTL_MAGIC, 0x1, struct mlx5ctl_cmdrpc) + #endif /* __MLX5CTL_IOCTL_H__ */ From patchwork Wed Feb 7 07:24:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 197785 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp2059981dyb; Tue, 6 Feb 2024 23:29:50 -0800 (PST) X-Forwarded-Encrypted: i=2; AJvYcCXf5PXFtdHmNwgpku3KicY20BTsSwG9ubCZIGHyRYgVoLNVU7fontWjwK/86VTjaprYgiyeWQ9rtt4Zc1ijfEYEmVN5iA== X-Google-Smtp-Source: AGHT+IEWS/gI9JjPXBRd3KT7rLRb+k3TwYy7vK2CVm4yQGpC8sHgG02yn0Wz6qb71ZuRwKsY9bQO X-Received: by 2002:a05:620a:11b9:b0:785:4e09:810d with SMTP id c25-20020a05620a11b900b007854e09810dmr7911143qkk.21.1707290990403; Tue, 06 Feb 2024 23:29:50 -0800 (PST) X-Forwarded-Encrypted: i=2; AJvYcCXtYZbttvJR1WIPmXGmnPGACf0ouR1NoglvwVXjvxHLbo9HtAR8OqndBqWHr+zlVmbSXikNpHDb6RJxWRDdIvuMLE8Ctw== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id bs12-20020a05620a470c00b0078596900933si639199qkb.485.2024.02.06.23.29.50 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 23:29:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-56077-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@kernel.org header.s=k20201202 header.b=GKzEAhRr; arc=fail (body hash mismatch); spf=pass (google.com: domain of linux-kernel+bounces-56077-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-56077-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 28C8E1C22F5C for ; Wed, 7 Feb 2024 07:29:50 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D3306224C9; Wed, 7 Feb 2024 07:28:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="GKzEAhRr" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F4110208A5 for ; Wed, 7 Feb 2024 07:28:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707290909; cv=none; b=toJIwvIWGtHWLV+qPghFI5wWYYZeLK4CCT8ggyaI4yu9hr1nIllM2NWP89VrqlPpoB9d+LqR9ITMA1ZcHZvL8vj9jt6dknW5IcgLnnu4si3l8+YzPIiZoOjCezNSqcth7xW6ZSF1VL+A01VfFAlQ4vAZfgKaty33Ivq9cz40UqM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707290909; c=relaxed/simple; bh=Qkue2Af7Q7rcEc2gVZiT60RzYZW+XViGUWZEeJVPAmg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OSpoaWCj8qiWlb8GcpblkovM163ti61mqcEcSsj66SthjHMdq9+QbehvipgXWvjL2JZ2EUltUSkw1evEKx/4xXAYF2GbXWJEzjw9qNrKQoNwfiuWZJeyzlKs4nxttQqjsDt35+D8arcj9qekCgGm6ivVAjS5t63H4NMvGVdDMFs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=GKzEAhRr; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id B6CB0C43390; Wed, 7 Feb 2024 07:28:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1707290908; bh=Qkue2Af7Q7rcEc2gVZiT60RzYZW+XViGUWZEeJVPAmg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GKzEAhRr6QzTELX0Fmihe8kYsy8W1PDMuLCFNlPUXFX8JbyPZtW1OPjpSvFxEmRuX gLyUH4Hi57q+fOW9QcADODL/QbrJ/fXhlpeVhOCyfRZ9pHNdCpEqI7kk9AogVUjWHI 6IqBBwaFDP1Cuvv/huI38TECtSbQYDNen9KRHf2f4FkkL9VockzQHNVe+C2z1K/IJ6 nsKSTwfpQK9cFifuZjMF56emgsGKkH4aPcnrAuHaSVXYuY5xaegdJQaRpgf5JeO5d5 RrsU+FletSHWLoiiAFngJQOHnTKqJpIMdZt/IVT0AoPD1qv1YHztQE/JLjr9gCPDE5 PhYMI+QdVLweg== From: Saeed Mahameed To: Arnd Bergmann , Greg Kroah-Hartman Cc: Leon Romanovsky , Jason Gunthorpe , Jiri Pirko , Leonid Bloch , Itay Avraham , Jakub Kicinski , Saeed Mahameed , David Ahern , Aron Silverton , Christoph Hellwig , andrew.gospodarek@broadcom.com, linux-kernel@vger.kernel.org Subject: [PATCH V4 5/5] misc: mlx5ctl: Add umem reg/unreg ioctl Date: Tue, 6 Feb 2024 23:24:35 -0800 Message-ID: <20240207072435.14182-6-saeed@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240207072435.14182-1-saeed@kernel.org> References: <20240207072435.14182-1-saeed@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790224357433141646 X-GMAIL-MSGID: 1790224357433141646 From: Saeed Mahameed Command rpc outbox buffer is limited in size, which can be very annoying when trying to pull large traces out of the device. Many device rpcs offer the ability to scatter output traces, contexts and logs directly into user space buffers in a single shot. Allow user to register user memory space, so the device may dump information directly into user memory space. The registered memory will be described by a device UMEM object which has a unique umem_id, this umem_id can be later used in the rpc inbox to tell the device where to populate the response output, e.g HW traces and other debug object queries. To do so this patch introduces two ioctls: MLX5CTL_IOCTL_UMEM_REG(va_address, size): - calculate page fragments from the user provided virtual address - pin the pages, and allocate a sg list - dma map the sg list - create a UMEM device object that points to the dma addresses - add a driver umem object to an xarray data base for bookkeeping - return UMEM ID to user so it can be used in subsequent rpcs MLX5CTL_IOCTL_UMEM_UNREG(umem_id): - user provides a pre allocated umem ID - unwinds the above Example usecase, ConnectX device coredump can be as large as 2MB. Using inline rpcs will take thousands of rpcs to get the full coredump which can take multiple seconds. With UMEM, it can be done in a single rpc, using 2MB of umem user buffer. $ ./mlx5ctlu mlx5_core.ctl.0 coredump --umem_size=$(( 2 ** 20 )) 00 00 00 00 01 00 20 00 00 00 00 04 00 00 48 ec 00 00 00 08 00 00 00 00 00 00 00 0c 00 00 00 03 00 00 00 10 00 00 00 00 00 00 00 14 00 00 00 00 ... 00 50 0b 3c 00 00 00 00 00 50 0b 40 00 00 00 00 00 50 0b 44 00 00 00 00 00 50 0b 48 00 00 00 00 00 50 0c 00 00 00 00 00 INFO : Core dump done INFO : Core dump size 831304 INFO : Core dump address 0x0 INFO : Core dump cookie 0x500c04 INFO : More Dump 0 Other usecases are: dynamic HW and FW trace monitoring, high frequency diagnostic counters sampling and batched objects and resource dumps. Reviewed-by: Jiri Pirko Reviewed-by: Leon Romanovsky Reviewed-by: Jason Gunthorpe Signed-off-by: Saeed Mahameed --- drivers/misc/mlx5ctl/Makefile | 1 + drivers/misc/mlx5ctl/main.c | 99 +++++++++++ drivers/misc/mlx5ctl/umem.c | 322 ++++++++++++++++++++++++++++++++++ drivers/misc/mlx5ctl/umem.h | 17 ++ include/uapi/misc/mlx5ctl.h | 18 ++ 5 files changed, 457 insertions(+) create mode 100644 drivers/misc/mlx5ctl/umem.c create mode 100644 drivers/misc/mlx5ctl/umem.h diff --git a/drivers/misc/mlx5ctl/Makefile b/drivers/misc/mlx5ctl/Makefile index b5c7f99e0ab6..f35234e931a8 100644 --- a/drivers/misc/mlx5ctl/Makefile +++ b/drivers/misc/mlx5ctl/Makefile @@ -2,3 +2,4 @@ obj-$(CONFIG_MLX5CTL) += mlx5ctl.o mlx5ctl-y := main.o +mlx5ctl-y += umem.o diff --git a/drivers/misc/mlx5ctl/main.c b/drivers/misc/mlx5ctl/main.c index c02b80efffc1..f79e1aa62b8f 100644 --- a/drivers/misc/mlx5ctl/main.c +++ b/drivers/misc/mlx5ctl/main.c @@ -12,6 +12,8 @@ #include #include +#include "umem.h" + MODULE_DESCRIPTION("mlx5 ConnectX control misc driver"); MODULE_AUTHOR("Saeed Mahameed "); MODULE_LICENSE("Dual BSD/GPL"); @@ -46,6 +48,8 @@ struct mlx5ctl_fd { u16 uctx_uid; u32 uctx_cap; u32 ucap; /* user cap */ + + struct mlx5ctl_umem_db *umem_db; struct mlx5ctl_dev *mcdev; struct list_head list; }; @@ -131,6 +135,12 @@ static int mlx5ctl_open_mfd(struct mlx5ctl_fd *mfd) if (uid < 0) return uid; + mfd->umem_db = mlx5ctl_umem_db_create(mdev, uid); + if (IS_ERR(mfd->umem_db)) { + mlx5ctl_release_uid(mcdev, uid); + return PTR_ERR(mfd->umem_db); + } + mfd->uctx_uid = uid; mfd->uctx_cap = cap; mfd->ucap = ucap; @@ -145,6 +155,7 @@ static void mlx5ctl_release_mfd(struct mlx5ctl_fd *mfd) { struct mlx5ctl_dev *mcdev = mfd->mcdev; + mlx5ctl_umem_db_destroy(mfd->umem_db); mlx5ctl_release_uid(mcdev, mfd->uctx_uid); } @@ -333,6 +344,86 @@ static int mlx5ctl_cmdrpc_ioctl(struct file *file, return err; } +static int mlx5ctl_ioctl_umem_reg(struct file *file, + struct mlx5ctl_umem_reg __user *arg, + size_t usize) +{ + size_t ksize = sizeof(struct mlx5ctl_umem_reg); + struct mlx5ctl_fd *mfd = file->private_data; + struct mlx5ctl_umem_reg *umem_reg; + int umem_id, err = 0; + + if (usize < ksize) + return -EINVAL; + + umem_reg = kzalloc(ksize, GFP_KERNEL); + if (!umem_reg) + return -ENOMEM; + + if (copy_from_user(umem_reg, arg, ksize)) { + err = -EFAULT; + goto out; + } + + if (umem_reg->reserved) { + err = -EINVAL; + goto out; + } + + umem_id = mlx5ctl_umem_reg(mfd->umem_db, + (unsigned long)umem_reg->addr, + umem_reg->len); + if (umem_id < 0) { + err = umem_id; + goto out; + } + + umem_reg->umem_id = umem_id; + + if (copy_to_user(arg, umem_reg, ksize)) { + mlx5ctl_umem_unreg(mfd->umem_db, umem_id); + err = -EFAULT; + } +out: + kfree(umem_reg); + return err; +} + +static int mlx5ctl_ioctl_umem_unreg(struct file *file, + struct mlx5ctl_umem_unreg __user *arg, + size_t usize) +{ + size_t ksize = sizeof(struct mlx5ctl_umem_unreg); + struct mlx5ctl_fd *mfd = file->private_data; + struct mlx5ctl_umem_unreg *umem_unreg; + int err = 0; + + if (usize < ksize) + return -EINVAL; + + umem_unreg = kzalloc(ksize, GFP_KERNEL); + if (!umem_unreg) + return -ENOMEM; + + if (copy_from_user(umem_unreg, arg, ksize)) { + err = -EFAULT; + goto out; + } + + if (umem_unreg->reserved) { + err = -EOPNOTSUPP; + goto out; + } + + err = mlx5ctl_umem_unreg(mfd->umem_db, umem_unreg->umem_id); + + if (!err && copy_to_user(arg, umem_unreg, ksize)) + err = -EFAULT; +out: + kfree(umem_unreg); + return err; +} + static long mlx5ctl_ioctl(struct file *file, unsigned int cmd, unsigned long arg) { struct mlx5ctl_fd *mfd = file->private_data; @@ -362,6 +453,14 @@ static long mlx5ctl_ioctl(struct file *file, unsigned int cmd, unsigned long arg err = mlx5ctl_cmdrpc_ioctl(file, argp, size); break; + case MLX5CTL_IOCTL_UMEM_REG: + err = mlx5ctl_ioctl_umem_reg(file, argp, size); + break; + + case MLX5CTL_IOCTL_UMEM_UNREG: + err = mlx5ctl_ioctl_umem_unreg(file, argp, size); + break; + default: mlx5ctl_dbg(mcdev, "Unknown ioctl %x\n", cmd); err = -ENOTTY; diff --git a/drivers/misc/mlx5ctl/umem.c b/drivers/misc/mlx5ctl/umem.c new file mode 100644 index 000000000000..29091a19305b --- /dev/null +++ b/drivers/misc/mlx5ctl/umem.c @@ -0,0 +1,322 @@ +// SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 +/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */ + +#include +#include +#include + +#include "umem.h" + +#define MLX5CTL_UMEM_MAX_MB 64 + +static unsigned long umem_num_pages(u64 addr, size_t len) +{ + return DIV_ROUND_UP(addr + len - PAGE_ALIGN_DOWN(addr), PAGE_SIZE); +} + +struct mlx5ctl_umem { + struct sg_table sgt; + unsigned long addr; + size_t size; + size_t offset; + size_t npages; + struct task_struct *source_task; + struct mm_struct *source_mm; + struct user_struct *source_user; + u32 umem_id; + struct page **page_list; +}; + +struct mlx5ctl_umem_db { + struct xarray xarray; + struct mlx5_core_dev *mdev; + u32 uctx_uid; +}; + +static int inc_user_locked_vm(struct mlx5ctl_umem *umem, unsigned long npages) +{ + unsigned long lock_limit; + unsigned long cur_pages; + unsigned long new_pages; + + lock_limit = task_rlimit(umem->source_task, RLIMIT_MEMLOCK) >> + PAGE_SHIFT; + do { + cur_pages = atomic_long_read(&umem->source_user->locked_vm); + new_pages = cur_pages + npages; + if (new_pages > lock_limit) + return -ENOMEM; + } while (atomic_long_cmpxchg(&umem->source_user->locked_vm, cur_pages, + new_pages) != cur_pages); + return 0; +} + +static void dec_user_locked_vm(struct mlx5ctl_umem *umem, unsigned long npages) +{ + if (WARN_ON(atomic_long_read(&umem->source_user->locked_vm) < npages)) + return; + atomic_long_sub(npages, &umem->source_user->locked_vm); +} + +#define PAGES_2_MB(pages) ((pages) >> (20 - PAGE_SHIFT)) + +static struct mlx5ctl_umem *mlx5ctl_umem_pin(struct mlx5ctl_umem_db *umem_db, + unsigned long addr, size_t size) +{ + size_t npages = umem_num_pages(addr, size); + struct mlx5_core_dev *mdev = umem_db->mdev; + unsigned long endaddr = addr + size; + struct mlx5ctl_umem *umem; + struct page **page_list; + int err = -EINVAL; + int pinned = 0; + + dev_dbg(mdev->device, "%s: addr %p size %zu npages %zu\n", + __func__, (void __user *)addr, size, npages); + + /* Avoid integer overflow */ + if (endaddr < addr || PAGE_ALIGN(endaddr) < endaddr) + return ERR_PTR(-EINVAL); + + if (npages == 0 || PAGES_2_MB(npages) > MLX5CTL_UMEM_MAX_MB) + return ERR_PTR(-EINVAL); + + page_list = kvmalloc_array(npages, sizeof(struct page *), GFP_KERNEL_ACCOUNT); + if (!page_list) + return ERR_PTR(-ENOMEM); + + umem = kzalloc(sizeof(*umem), GFP_KERNEL_ACCOUNT); + if (!umem) { + kvfree(page_list); + return ERR_PTR(-ENOMEM); + } + + umem->addr = addr; + umem->size = size; + umem->offset = addr & ~PAGE_MASK; + umem->npages = npages; + + umem->page_list = page_list; + umem->source_mm = current->mm; + umem->source_task = current->group_leader; + get_task_struct(current->group_leader); + umem->source_user = get_uid(current_user()); + + /* mm and RLIMIT_MEMLOCK user task accounting similar to what is + * being done in iopt_alloc_pages() and do_update_pinned() + * for IOPT_PAGES_ACCOUNT_USER @drivers/iommu/iommufd/pages.c + */ + mmgrab(umem->source_mm); + + pinned = pin_user_pages_fast(addr, npages, FOLL_WRITE, page_list); + if (pinned != npages) { + dev_dbg(mdev->device, "pin_user_pages_fast failed %d\n", pinned); + err = pinned < 0 ? pinned : -ENOMEM; + goto pin_failed; + } + + err = inc_user_locked_vm(umem, npages); + if (err) + goto pin_failed; + + atomic64_add(npages, &umem->source_mm->pinned_vm); + + err = sg_alloc_table_from_pages(&umem->sgt, page_list, npages, 0, + npages << PAGE_SHIFT, GFP_KERNEL_ACCOUNT); + if (err) { + dev_dbg(mdev->device, "sg_alloc_table failed: %d\n", err); + goto sgt_failed; + } + + dev_dbg(mdev->device, "\tsgt: size %zu npages %zu sgt.nents (%d)\n", + size, npages, umem->sgt.nents); + + err = dma_map_sgtable(mdev->device, &umem->sgt, DMA_BIDIRECTIONAL, 0); + if (err) { + dev_dbg(mdev->device, "dma_map_sgtable failed: %d\n", err); + goto dma_failed; + } + + dev_dbg(mdev->device, "\tsgt: dma_nents %d\n", umem->sgt.nents); + return umem; + +dma_failed: +sgt_failed: + sg_free_table(&umem->sgt); + atomic64_sub(npages, &umem->source_mm->pinned_vm); + dec_user_locked_vm(umem, npages); +pin_failed: + if (pinned > 0) + unpin_user_pages(page_list, pinned); + mmdrop(umem->source_mm); + free_uid(umem->source_user); + put_task_struct(umem->source_task); + + kfree(umem); + kvfree(page_list); + return ERR_PTR(err); +} + +static void mlx5ctl_umem_unpin(struct mlx5ctl_umem_db *umem_db, + struct mlx5ctl_umem *umem) +{ + struct mlx5_core_dev *mdev = umem_db->mdev; + + dev_dbg(mdev->device, "%s: addr %p size %zu npages %zu dma_nents %d\n", + __func__, (void *)umem->addr, umem->size, umem->npages, + umem->sgt.nents); + + dma_unmap_sgtable(mdev->device, &umem->sgt, DMA_BIDIRECTIONAL, 0); + sg_free_table(&umem->sgt); + + atomic64_sub(umem->npages, &umem->source_mm->pinned_vm); + dec_user_locked_vm(umem, umem->npages); + unpin_user_pages(umem->page_list, umem->npages); + mmdrop(umem->source_mm); + free_uid(umem->source_user); + put_task_struct(umem->source_task); + + kvfree(umem->page_list); + kfree(umem); +} + +static int mlx5ctl_umem_create(struct mlx5_core_dev *mdev, + struct mlx5ctl_umem *umem, u32 uid) +{ + u32 out[MLX5_ST_SZ_DW(create_umem_out)] = {}; + int err, inlen, i, n = 0; + struct scatterlist *sg; + void *in, *umemptr; + __be64 *mtt; + + inlen = MLX5_ST_SZ_BYTES(create_umem_in) + + umem->npages * MLX5_ST_SZ_BYTES(mtt); + + in = kzalloc(inlen, GFP_KERNEL); + if (!in) + return -ENOMEM; + + MLX5_SET(create_umem_in, in, opcode, MLX5_CMD_OP_CREATE_UMEM); + MLX5_SET(create_umem_in, in, uid, uid); + + umemptr = MLX5_ADDR_OF(create_umem_in, in, umem); + + MLX5_SET(umem, umemptr, log_page_size, + PAGE_SHIFT - MLX5_ADAPTER_PAGE_SHIFT); + MLX5_SET64(umem, umemptr, num_of_mtt, umem->npages); + MLX5_SET(umem, umemptr, page_offset, umem->offset); + + dev_dbg(mdev->device, + "UMEM CREATE: log_page_size %d num_of_mtt %lld page_offset %d\n", + MLX5_GET(umem, umemptr, log_page_size), + MLX5_GET64(umem, umemptr, num_of_mtt), + MLX5_GET(umem, umemptr, page_offset)); + + mtt = MLX5_ADDR_OF(create_umem_in, in, umem.mtt); + for_each_sgtable_dma_sg(&umem->sgt, sg, i) { + u64 dma_addr = sg_dma_address(sg); + ssize_t len = sg_dma_len(sg); + + for (; n < umem->npages && len > 0; n++, mtt++) { + *mtt = cpu_to_be64(dma_addr); + MLX5_SET(mtt, mtt, wr_en, 1); + MLX5_SET(mtt, mtt, rd_en, 1); + dma_addr += PAGE_SIZE; + len -= PAGE_SIZE; + } + WARN_ON_ONCE(n == umem->npages && len > 0); + } + + err = mlx5_cmd_exec(mdev, in, inlen, out, sizeof(out)); + if (err) + goto out; + + umem->umem_id = MLX5_GET(create_umem_out, out, umem_id); + dev_dbg(mdev->device, "\tUMEM CREATED: umem_id %d\n", umem->umem_id); +out: + kfree(in); + return err; +} + +static void mlx5ctl_umem_destroy(struct mlx5_core_dev *mdev, + struct mlx5ctl_umem *umem) +{ + u32 in[MLX5_ST_SZ_DW(destroy_umem_in)] = {}; + + MLX5_SET(destroy_umem_in, in, opcode, MLX5_CMD_OP_DESTROY_UMEM); + MLX5_SET(destroy_umem_in, in, umem_id, umem->umem_id); + + dev_dbg(mdev->device, "UMEM DESTROY: umem_id %d\n", umem->umem_id); + mlx5_cmd_exec_in(mdev, destroy_umem, in); +} + +int mlx5ctl_umem_reg(struct mlx5ctl_umem_db *umem_db, unsigned long addr, + size_t size) +{ + struct mlx5ctl_umem *umem; + void *ret; + int err; + + umem = mlx5ctl_umem_pin(umem_db, addr, size); + if (IS_ERR(umem)) + return PTR_ERR(umem); + + err = mlx5ctl_umem_create(umem_db->mdev, umem, umem_db->uctx_uid); + if (err) + goto umem_create_err; + + ret = xa_store(&umem_db->xarray, umem->umem_id, umem, GFP_KERNEL_ACCOUNT); + if (WARN(xa_is_err(ret), "Failed to store UMEM")) { + err = xa_err(ret); + goto xa_store_err; + } + + return umem->umem_id; + +xa_store_err: + mlx5ctl_umem_destroy(umem_db->mdev, umem); +umem_create_err: + mlx5ctl_umem_unpin(umem_db, umem); + return err; +} + +int mlx5ctl_umem_unreg(struct mlx5ctl_umem_db *umem_db, u32 umem_id) +{ + struct mlx5ctl_umem *umem; + + umem = xa_erase(&umem_db->xarray, umem_id); + if (!umem) + return -ENOENT; + + mlx5ctl_umem_destroy(umem_db->mdev, umem); + mlx5ctl_umem_unpin(umem_db, umem); + return 0; +} + +struct mlx5ctl_umem_db *mlx5ctl_umem_db_create(struct mlx5_core_dev *mdev, + u32 uctx_uid) +{ + struct mlx5ctl_umem_db *umem_db; + + umem_db = kzalloc(sizeof(*umem_db), GFP_KERNEL_ACCOUNT); + if (!umem_db) + return ERR_PTR(-ENOMEM); + + xa_init(&umem_db->xarray); + umem_db->mdev = mdev; + umem_db->uctx_uid = uctx_uid; + + return umem_db; +} + +void mlx5ctl_umem_db_destroy(struct mlx5ctl_umem_db *umem_db) +{ + struct mlx5ctl_umem *umem; + unsigned long index; + + xa_for_each(&umem_db->xarray, index, umem) + mlx5ctl_umem_unreg(umem_db, umem->umem_id); + + xa_destroy(&umem_db->xarray); + kfree(umem_db); +} diff --git a/drivers/misc/mlx5ctl/umem.h b/drivers/misc/mlx5ctl/umem.h new file mode 100644 index 000000000000..9cf62e5e775e --- /dev/null +++ b/drivers/misc/mlx5ctl/umem.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: BSD-3-Clause OR GPL-2.0 */ +/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */ + +#ifndef __MLX5CTL_UMEM_H__ +#define __MLX5CTL_UMEM_H__ + +#include +#include + +struct mlx5ctl_umem_db; + +struct mlx5ctl_umem_db *mlx5ctl_umem_db_create(struct mlx5_core_dev *mdev, u32 uctx_uid); +void mlx5ctl_umem_db_destroy(struct mlx5ctl_umem_db *umem_db); +int mlx5ctl_umem_reg(struct mlx5ctl_umem_db *umem_db, unsigned long addr, size_t size); +int mlx5ctl_umem_unreg(struct mlx5ctl_umem_db *umem_db, u32 umem_id); + +#endif /* __MLX5CTL_UMEM_H__ */ diff --git a/include/uapi/misc/mlx5ctl.h b/include/uapi/misc/mlx5ctl.h index 1e4622c5979f..bb9ca8581112 100644 --- a/include/uapi/misc/mlx5ctl.h +++ b/include/uapi/misc/mlx5ctl.h @@ -19,6 +19,18 @@ struct mlx5ctl_cmdrpc { __u32 outlen; /* outbox buffer length */ }; +struct mlx5ctl_umem_reg { + __aligned_u64 addr; /* user address */ + __aligned_u64 len; /* user buffer length */ + __u32 umem_id; /* returned device's umem ID */ + __u32 reserved; /* explicit padding must be zero */ +}; + +struct mlx5ctl_umem_unreg { + __u32 umem_id; + __u32 reserved; /* explicit padding must be zero */ +}; + #define MLX5CTL_MAX_RPC_SIZE (512 * 512) /* max FW RPC buffer size 512 blocks of 512 bytes */ #define MLX5CTL_IOCTL_MAGIC 0x5c @@ -29,4 +41,10 @@ struct mlx5ctl_cmdrpc { #define MLX5CTL_IOCTL_CMDRPC \ _IOWR(MLX5CTL_IOCTL_MAGIC, 0x1, struct mlx5ctl_cmdrpc) +#define MLX5CTL_IOCTL_UMEM_REG \ + _IOWR(MLX5CTL_IOCTL_MAGIC, 0x2, struct mlx5ctl_umem_reg) + +#define MLX5CTL_IOCTL_UMEM_UNREG \ + _IOWR(MLX5CTL_IOCTL_MAGIC, 0x3, struct mlx5ctl_umem_unreg) + #endif /* __MLX5CTL_IOCTL_H__ */