From patchwork Mon Oct 9 11:23:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 149953 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp1797839vqo; Mon, 9 Oct 2023 04:24:50 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF10oEiX67i1jPnLWWOaE/FTfRBbEAvC2J8dtlCRMSwPfu+kT2PXgC3StZd9IOE7ciME8Ic X-Received: by 2002:a17:902:c084:b0:1c3:aa8d:4daa with SMTP id j4-20020a170902c08400b001c3aa8d4daamr11448347pld.44.1696850690627; Mon, 09 Oct 2023 04:24:50 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696850690; cv=pass; d=google.com; s=arc-20160816; b=fEGRG6KChEzxq11S4ibxxLL6aU3/1N3r5HJ8iCWkuXMkqZL2LfOsGe3+/o/z1t7kAg twq5DhsPEUDJw9d8mFB7ABF0sir60HI2rscUxrR8ecOr5MWF5t8dJBce3+jz8kSggIDV uxi0w3FJD88sgIjV1fc6wHxW7w37MJsvxXWhhfh+JnzekYqMaYK/qFbjw8LS4PBUI1dT jIM5JDFqSOAsL+CAKR71zQ1E1qcJ3QTULkJn2PqFJvuMwinf5iLmYAFFWf9TkW4Ghz48 IDZq0M8tNRO295llfThKURHb0SaYPf/pJk/7LTiV0xMwFhtCMqY1jJRd2rIgcv7kFAcq ARvA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Ks0yqKFa9s2l5F9bXQdb3gSn8cxS7t2k+S80slwQQFY=; fh=iwiW798sTWZr0bClUZYdSITt06HHhXlZHI8Adgm6+CE=; b=zLqggPjSQomWWvTcI2/JAy794vu0sqej6yibShFo7TxBqe5AgdNsUlouZh9WTGO+wf HcOe5f19inmaiAV78Ao5Jr+8AMbPDkpNeMBLtZlFeVqdELRShslMVmMZbVR9BP0cGjoW oBB5l7+mZXXhq74/C8U+lIodbyJ8kK0nqzxsU9/skUeOB8hhvKozRYGtGZ3qJtZwGlXz PZD9MDXyTa7wtF3Le7Pt73Vidy5W7k3DwgIVquTE/0ivNs879kLErWsDBm4xlBjUOLhN /3cPUaq2o/B93lTljd15Qc0s+L6xEmHY21/pV3QkLNFkj6DdKE9DYskQ0Ble7QV1B/s5 zddA== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=E5Fdl+S5; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id d4-20020a170902cec400b001c5e0672f53si10181371plg.466.2023.10.09.04.24.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 04:24:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=E5Fdl+S5; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 8DBD8807E9AA; Mon, 9 Oct 2023 04:24:48 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346242AbjJILYk (ORCPT + 18 others); Mon, 9 Oct 2023 07:24:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59052 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346216AbjJILYj (ORCPT ); Mon, 9 Oct 2023 07:24:39 -0400 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2075.outbound.protection.outlook.com [40.107.244.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A1542C6; Mon, 9 Oct 2023 04:24:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Dl/oZ7/fLgTecJO74VEcKKb9DoY2t/tEzoC1/F8J/OqdrKuRYBOx5RPdJq2+Uucy8DW/EFN22xttmmEkca6/v256vdcVZ6kHocgVZ22oyoqxzkMf8Mq5eMopmcR1R1Z/sXhP1cuhDrDOD1yERCsGzGc7qI+D8P0tfqOm15H1Nt+IGkdjxnOUu3j8zHgW6GquzF+Vl2yFSywjXff9RXsylDi6aaD1+iHogO7naoU2QVNIlI9LRd+BTKPzL/MOULey7pRyM91B+EX4UXMay+ktxtdT2Iyoa69VT49fxPujrf64QVw88H446Tys3MgZRj48fDLqsQeN6EDAZvORE211dQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ks0yqKFa9s2l5F9bXQdb3gSn8cxS7t2k+S80slwQQFY=; b=mLXjL5ApuVmO7pBuA0qZmyockV9qT70VRNLBiSFi+DSl1813Xpc4HEAIkM727u2QOyeM2F+tR//5tZO2FNp0jYUD5OnTYKNwmD2KGXotlWGNs5agivLd4kx2wpLOYqGpfv56u9UHBSxO7skFcrlBSsbnQJfp/TL6O6FsfO2ZLRoCnPQpjysNxyv2xZa78qqDqr5kzfhlHLuetfrRNJBbX0bkzPfRuqZVcG2fDT0Ii4syKn3rfKUIlj47mcKfN4tH2e2fheCad2DI6sIsj7SWrX1BC4NVoQiHzFUTN1mhORz6XyfHQVh8p0k8VZGKE7r8DBZ89D1omuDvXjva7KR0/g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ks0yqKFa9s2l5F9bXQdb3gSn8cxS7t2k+S80slwQQFY=; b=E5Fdl+S54A5JB3fXMBXrNVsX5kI/FW4stDAsa4q1IOF0eLpuotXWViAv0TS/oRHTAJV2UOlo6exLeLrVdyH/PPSexgOaFOJrPx0C7Tx98umcIr8C6U/oFdqJej5Lu0zgY2QT7glOit3wUmumrKXnO7OshXomOnRGXCDZ7+v7E3LSuDbn0cX9AlCTieQzDRhSOpPusvEBfojRZov7Q0914UOYwJwBLCAuXHed8WOIKPE/JbgHFoHX9ORv1YRU3EhmxcO3x+OJYsidkscHpDzeVw1fCK5z+wZ03c08Ms+q4qLGJhzj2s3sc8sVkupMfO9TOJcaY9LA7hCVOfnSyE3a+w== Received: from BL1PR13CA0100.namprd13.prod.outlook.com (2603:10b6:208:2b9::15) by SJ2PR12MB8806.namprd12.prod.outlook.com (2603:10b6:a03:4d0::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.37; Mon, 9 Oct 2023 11:24:35 +0000 Received: from MN1PEPF0000ECD9.namprd02.prod.outlook.com (2603:10b6:208:2b9:cafe::5d) by BL1PR13CA0100.outlook.office365.com (2603:10b6:208:2b9::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.21 via Frontend Transport; Mon, 9 Oct 2023 11:24:34 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MN1PEPF0000ECD9.mail.protection.outlook.com (10.167.242.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.14 via Frontend Transport; Mon, 9 Oct 2023 11:24:34 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:19 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:19 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Mon, 9 Oct 2023 04:24:16 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo , Gal Pressman Subject: [PATCH mlx5-vhost v3 01/16] vdpa/mlx5: Expose descriptor group mkey hw capability Date: Mon, 9 Oct 2023 14:23:46 +0300 Message-ID: <20231009112401.1060447-2-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009112401.1060447-1-dtatulea@nvidia.com> References: <20231009112401.1060447-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000ECD9:EE_|SJ2PR12MB8806:EE_ X-MS-Office365-Filtering-Correlation-Id: 0ac7d9d3-7c63-4cb1-ae2a-08dbc8ba50d6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: wyeIsLVFGFhg1JtmwvSbzR+KaaobtyhrXYfrQAIMxy2FsYcVEsx6ZWcAbX65FEVJbTO1FqGaOpnuKGmAT/tAI/DEEcwca0y5qSyCx0Qib84EkXme7k3qMGMmmqPIIUHh+c7V5MObecCAbPWpVKFf2SE/wpbgFiP12akqaWQhGGAA88eEIwDTEuGex8WXno6xj2MMjzHFkwR74Vh4dQtFZO/FuhEW6ETUI3Et5jasr6yZ6J+mAQhdkP7sDKsukiCtFPb6IfXNKGYJ3Rv9TS0fhIpR98Hj5ygSzY2CSSIx1McbT0cQXMyVm2aai3qi5lV+6qCYcFyTMGU9JM+6LVxSWfWp90AEhabeivdU/c8+WIWjPjJKH4m1hcnQBmipXWyXDF+QtyXkRalWsc0HEP6U34cMi0Tc5UK5mpu1b9m+Bj7vbPbxOEyKkljGVUxu3b5t18C+VCaMHzSTO2OBqb7ubVC3QXmZczShPNS3VAkm93i/I267KGm6xof5+YxGZbYGQUqd6j1tPdWhqJH0yXv+p+7YFUVPmJdrFV+XtsGDgxxiNU2AvzdHJJMacFLOW5Iq7YmrCpPWlwrSfQbOrYSA40GY29yk7S2uHgoLvjHe6w81OyGsANfkoCa+hU9dY6c00bxZkck/d4f2URzJdsogKWEGLxQ9SZDrPzqJGtSfj4liAVhtQEU1QFS7CZBK77HCdBkC5p4OphF3EU0M6pXFjXZ66Grg1zoUJT6f5wfHsOCtM5ESum/0eC10QDomVUxj X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(346002)(136003)(376002)(39860400002)(396003)(230922051799003)(186009)(64100799003)(82310400011)(451199024)(1800799009)(40470700004)(36840700001)(46966006)(40480700001)(40460700003)(83380400001)(2616005)(107886003)(1076003)(336012)(426003)(26005)(47076005)(36860700001)(54906003)(110136005)(70586007)(316002)(70206006)(8936002)(8676002)(4326008)(41300700001)(5660300002)(6666004)(2906002)(4744005)(478600001)(82740400003)(36756003)(356005)(7636003)(86362001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2023 11:24:34.6901 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0ac7d9d3-7c63-4cb1-ae2a-08dbc8ba50d6 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000ECD9.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8806 X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 09 Oct 2023 04:24:48 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779276910160025748 X-GMAIL-MSGID: 1779276910160025748 Necessary for improved live migration flow. Actual support will be added in a downstream patch. Reviewed-by: Gal Pressman Signed-off-by: Dragos Tatulea --- include/linux/mlx5/mlx5_ifc.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index dd8421d021cf..ec15330b970d 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -1231,7 +1231,13 @@ struct mlx5_ifc_virtio_emulation_cap_bits { u8 max_emulated_devices[0x8]; u8 max_num_virtio_queues[0x18]; - u8 reserved_at_a0[0x60]; + u8 reserved_at_a0[0x20]; + + u8 reserved_at_c0[0x13]; + u8 desc_group_mkey_supported[0x1]; + u8 reserved_at_d4[0xc]; + + u8 reserved_at_e0[0x20]; u8 umem_1_buffer_param_a[0x20]; From patchwork Mon Oct 9 11:23:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 149955 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp1797951vqo; Mon, 9 Oct 2023 04:25:07 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFffYbI0fMoLlymc9IWNHiWK8YUk/SpKLmJHSKpb/GeHE8vB89u9twgNG818F/jrYLZPhlf X-Received: by 2002:a05:6a20:6a25:b0:163:f945:42da with SMTP id p37-20020a056a206a2500b00163f94542damr18221535pzk.48.1696850706902; Mon, 09 Oct 2023 04:25:06 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696850706; cv=pass; d=google.com; s=arc-20160816; b=ZndlbxM1o/mLtRd4SFFQvURZAZatuuYpbzW26FmHXN4MQekLUPY1fnGjj7plLGkAGu bOma1dNvlDrZ5KK4ZugQOAC1Axm2Qwa2rB2jgqrH07Gohrmjxf3er+b1VtJd1tbFYy/v CCPGCkrhwNUWKGEcR/kBZQyEXm3BZErS6S0o0Dd09B2eF/2BE/DoRqTfNg2AGcbjO27j gZCQd8VnfQGBHJ5X8yCbs2V3SKQENVS3YdZMXr+doKMZ2ARONg5Owp687fEdObpXT+k/ DNX2BcVsaKTDc0ZnMF/JRnZ8cpjdnSLuhHzayKg80XM5xMqkozRER8CAM30HzoFUuQGZ lYig== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Zs2p5Im8kifm/aJ1lEAfNpIB6nh0RoaB6CA/cNIHKDA=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=Yrw/t80vGGZ2Yhm7WiYq6ZdZPvrtBChNblankbihltEdqm1IyWJUt/wPAdRJCmOd/v mNgkcyTi0LSqWGT5aYAJRsoShrozH0tpEasugLQ5E1d9JwmdLBy+3MTM4Vsip8JUTae3 RmeOtNghNSgNlCfANU4QRM+ICcA5NTNGpb6Sdp7Q74vUJ1S19XYa4nAomAfDUwvt1UzG AlG2YoTMuhZbmkQmBdCp3LQ22NK3SS2SPkHDRi9TTWIJNHaxMZOBdY14QpzWuQ+FMdRn scUvcrI8jZHN4nQucW1NAMxoh41xHuZIKiPziglSFmEPKgtdM+56HNXE5nWqIgZKBHMu TPWA== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=ruk9IJ3X; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id p23-20020a056a000a1700b0068fe12b361dsi7535043pfh.249.2023.10.09.04.25.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 04:25:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=ruk9IJ3X; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id A6C3F802A825; Mon, 9 Oct 2023 04:25:05 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346256AbjJILYz (ORCPT + 18 others); Mon, 9 Oct 2023 07:24:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346221AbjJILYv (ORCPT ); Mon, 9 Oct 2023 07:24:51 -0400 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2057.outbound.protection.outlook.com [40.107.223.57]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BEFCAC6; Mon, 9 Oct 2023 04:24:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gmqBsOoDIBN9/lWK3UIKJ9G3hU1UlrErXZ6Ollo6Z4vWRsUHEq/NwBlPEL6dJ9lZAVZpR9HwM906XqEgWviyA7MEFEHo9hPM7J7QvTjcdbJLg/RP3zYXXIygc033us8HU0j14tHJgJ7YNvkRrIysd3bqThW5jQvKbjW3pquu+Kar61HzbjHj8xyF8ouSC9ozLFfn7b0+A8oSGh4+NMakfZKdWR3csyQg1g+c+MAIER0QEMHx7C5+Q74pOSIIqKb38hsUhAa/1LVGAky3VxQbA0yGk7RScgoPyXgdrgkYikOt/dFcAEyDSPSL3oXppu9z6X1kFDa2LOMY0M/1t0A8Eg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Zs2p5Im8kifm/aJ1lEAfNpIB6nh0RoaB6CA/cNIHKDA=; b=asC15CuIXX2TeKTj2uWytYbbmz+lsej6MQRE+zA6aF0fsItboxs6ljNcxdJPBKnxTJAPJxHAvwdGhweePm5BAd73IpKbpFEjhafi2uj02v8WVI/fkhEMPfKVAMNIm6wiKPqSkZKRmpbIZ3pPgil2BTt93uGpa3rYlPosp7UeNAqCNpAsOIvau3c+iQtVYEXxpcwBmuDqt95VfY0rht/2xTAS19VaWmG/IPN+pMTnEOFB/HOtxDyS+VqAf/jCSnnIVgAOpEhPiDWaqgS7ygAsm+o7VFxPw6CcfopmCaAbCIvQDS+8zIxDigwPdmzW2dxdU7j9dk+yDPgFOoyT6KJoTg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Zs2p5Im8kifm/aJ1lEAfNpIB6nh0RoaB6CA/cNIHKDA=; b=ruk9IJ3Xl+QhzUEeutHga6vrxIq8WHa6nZIOW7/zF1B9xiABiMmp1IxHmxRVug7YTKFC/gBSM0cIYjSaopJ0RuXHS9a/8jvr4NIjuive+lRpKQBgnfIG9F5Te4FgwaLp92MVxvyviu+fH48RzJwjSWNIUWxWVR06AWeRevG/mwxIxU/xoO3h6CdU4fhgR0/z6zq2/nPHzs/qS7UJzIURoy1dJ5X4R0Azlx+iXBMEqk9dfJzvY7hR1WzAPi+tQXKqCrkOKGQapLCrt9JBBUr2Qfy++Gl2kEQ3f5Fpwd72bAcsu++FpXENMHOedeSVbiSRXjJhv7Ax3aX3gESf8TSFiw== Received: from PR0P264CA0241.FRAP264.PROD.OUTLOOK.COM (2603:10a6:100::13) by CH2PR12MB5002.namprd12.prod.outlook.com (2603:10b6:610:6d::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.38; Mon, 9 Oct 2023 11:24:47 +0000 Received: from SN1PEPF0002BA4C.namprd03.prod.outlook.com (2603:10a6:100:0:cafe::9f) by PR0P264CA0241.outlook.office365.com (2603:10a6:100::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.36 via Frontend Transport; Mon, 9 Oct 2023 11:24:46 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SN1PEPF0002BA4C.mail.protection.outlook.com (10.167.242.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.14 via Frontend Transport; Mon, 9 Oct 2023 11:24:46 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:25 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:25 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Mon, 9 Oct 2023 04:24:22 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v3 02/16] vdpa: introduce dedicated descriptor group for virtqueue Date: Mon, 9 Oct 2023 14:23:47 +0300 Message-ID: <20231009112401.1060447-3-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009112401.1060447-1-dtatulea@nvidia.com> References: <20231009112401.1060447-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002BA4C:EE_|CH2PR12MB5002:EE_ X-MS-Office365-Filtering-Correlation-Id: cc6402e1-76f9-4edf-0852-08dbc8ba57a5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Bahy9pqa1braQgeWptccs3LXAFRXcM17+sab8gAv06vD4tF29BiKAGy23hQtjeidD9K7KCnlSGEk8234wfoCYFLggT5UA0lPpvkPXHK3LkzvOOcDzeWO59ovjWalxnmH6IfkcPPegKtZeq9thIozZ8OAd6JpSvFgfkq4sDMf7yQ2ebpczNWGkyK8iYG4HMqPvscoXspX5u4vT9bRP78i6o+HWZnN5AMhE04zpxg+B9hUTeYYqeiWfYlb9lkFpHKBuUPc6CbTkA43/aMzDCFlBGFzX+J0K+RNzIcCLufdmd8UFW7OguIlpWBXtW0LMAkQ0HXDITSKZZKw38OM02YpzdvK9+CLHISOoUa4pwyUUVSl0hcF1CXK+qCngtotypYRpck0NslzRRKIJA+qiufjWqoqGlCtnXgonp2jsSjYlyFutVq6ql0aX1j5uQ4YgXlX3pvP4c3/oOS1R0jL19lAP/nG3l3lZOS1gX2Vyv0XMqzl1yhO51Kz27pYLNoFomoal0dpn66dfMZEb7jWrLhhVSA3sxJl3BPuliv9Fr5HZ+6LkQ3CJa6uC6TD8dSy6Uru9SNe5ic0qKW0PII5E3Cbx49Bu8tqlYCRc+jBmDU6xuDJHwj8vCat5LP/prYAPATYvArF/vHgSPhwwSuD4bsGD+73qAbszAzcsIK8Tg1EaH/bo9m/O+rBwP6o6EdKyegOpEYGCN26CJr3ofocBpMPmhOF6Oroz36NnXVoukyyrG9Yi3pFgWY6HNzgJvkaEzdH X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(376002)(396003)(136003)(39860400002)(346002)(230922051799003)(186009)(64100799003)(451199024)(1800799009)(82310400011)(46966006)(36840700001)(40470700004)(2616005)(1076003)(426003)(26005)(336012)(40460700003)(40480700001)(36756003)(86362001)(82740400003)(7636003)(356005)(83380400001)(8676002)(4326008)(2906002)(8936002)(6666004)(478600001)(47076005)(36860700001)(66574015)(41300700001)(5660300002)(316002)(54906003)(70586007)(70206006)(110136005);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2023 11:24:46.1712 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cc6402e1-76f9-4edf-0852-08dbc8ba57a5 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002BA4C.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB5002 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 09 Oct 2023 04:25:05 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779276926553000324 X-GMAIL-MSGID: 1779276926553000324 From: Si-Wei Liu In some cases, the access to the virtqueue's descriptor area, device and driver areas (precluding indirect descriptor table in guest memory) may have to be confined to a different address space than where its buffers reside. Without loss of simplicity and generality with already established terminology, let's fold up these 3 areas and call them as a whole as descriptor table group, or descriptor group for short. Specifically, in case of split virtqueues, descriptor group consists of regions for Descriptor Table, Available Ring and Used Ring; for packed virtqueues layout, descriptor group contains Descriptor Ring, Driver and Device Event Suppression structures. The group ID for a dedicated descriptor group can be obtained through a new .get_vq_desc_group() op. If driver implements this op, it means that the descriptor, device and driver areas of the virtqueue may reside in a dedicated group than where its buffers reside, a.k.a the default virtqueue group through the .get_vq_group() op. In principle, the descriptor group may or may not have same group ID as the default group. Even if the descriptor group has a different ID, meaning the vq's descriptor group areas can optionally move to a separate address space than where guest memory resides, the descriptor group may still start from a default address space, same as where its buffers reside. To move the descriptor group to a different address space, .set_group_asid() has to be called to change the ASID binding for the group, which is no different than what needs to be done on any other virtqueue group. On the other hand, the .reset() semantics also applies on descriptor table group, meaning the device reset will clear all ASID bindings and move all virtqueue groups including descriptor group back to the default address space, i.e. in ASID 0. QEMU's shadow virtqueue is going to utilize dedicated descriptor group to speed up map and unmap operations, yielding tremendous downtime reduction by avoiding the full and slow remap cycle in SVQ switching. Signed-off-by: Si-Wei Liu Acked-by: Eugenio Pérez Acked-by: Jason Wang --- include/linux/vdpa.h | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/include/linux/vdpa.h b/include/linux/vdpa.h index 0e652026b776..d376309b99cf 100644 --- a/include/linux/vdpa.h +++ b/include/linux/vdpa.h @@ -204,6 +204,16 @@ struct vdpa_map_file { * @vdev: vdpa device * @idx: virtqueue index * Returns u32: group id for this virtqueue + * @get_vq_desc_group: Get the group id for the descriptor table of + * a specific virtqueue (optional) + * @vdev: vdpa device + * @idx: virtqueue index + * Returns u32: group id for the descriptor table + * portion of this virtqueue. Could be different + * than the one from @get_vq_group, in which case + * the access to the descriptor table can be + * confined to a separate asid, isolating from + * the virtqueue's buffer address access. * @get_device_features: Get virtio features supported by the device * @vdev: vdpa device * Returns the virtio features support by the @@ -360,6 +370,7 @@ struct vdpa_config_ops { /* Device ops */ u32 (*get_vq_align)(struct vdpa_device *vdev); u32 (*get_vq_group)(struct vdpa_device *vdev, u16 idx); + u32 (*get_vq_desc_group)(struct vdpa_device *vdev, u16 idx); u64 (*get_device_features)(struct vdpa_device *vdev); u64 (*get_backend_features)(const struct vdpa_device *vdev); int (*set_driver_features)(struct vdpa_device *vdev, u64 features); From patchwork Mon Oct 9 11:23:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 149966 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp1798924vqo; Mon, 9 Oct 2023 04:27:20 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFDAIBMXWJhKRSr9mmxHLiX/xwSaI6mkM4vTKA+8sIvJ0XsWX1q0LhTKr5SNGP0kblrTWUf X-Received: by 2002:a92:cdab:0:b0:34f:d822:baab with SMTP id g11-20020a92cdab000000b0034fd822baabmr19562241ild.12.1696850839943; Mon, 09 Oct 2023 04:27:19 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696850839; cv=pass; d=google.com; s=arc-20160816; b=BVTuLLlqOLLMg3AWVNbWTeuMtgnTOXkppU3ZaOm1XJOAw4GcwxltSSwn41KbtoulQn F0h7pEKxPY6/WBgveqED1ESO4hIDDWDqfCXEGPMWCgXLhszBlhiGEInC37q+GTWirCK7 QBOpTftt/Yn3MVre7crbW0gbM8+LTy8f+jqhus584uUxdFeO9RUahI64gCYI/qYE4CUt sOuej7ko7m3Sqy5+PHGnSBCsEp6HGO9VzWP5l2/G9HNJQStrVoPDYsJRyERwACyLYXQS NU85XCP7UcK92HzQs9E4cvE4+2pmKxb22O5srGlNOaVWZr1m1sjYmc+Gh2JQGZQTh+S2 J+rw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ILKog4btT16IMu9HXAHH95fAm+8LcOyK/4mzDrViAvI=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=dAJ48DaKM/qmtFns8JUOtR8rsccp2IMX7Jbv7OAFgCmAuuk87m9FnVAMsUczzW7qtG Q+Uk1b8UWoqAdW6tk0g6TkVPROcXO32aZ/5b0qc4SyPXdBK3+uVcWZ3LdBr/vzXilXe8 OEGyl41r6NySt8RTayHOnMQKJI/OqhlIgMkiglcosXJpcEIqOa0iqxxYSNvOMC/foAej 2onkFOrJrzybkN3EpBjYjRKCq6pukwsTa/Aj6Jzgbgc51jHL4KRqLlu5zIH/7DIRBnmg 9hIcoPFe9REVpWtiKaermzwv36FAi/zabPsPKvpr32O0/0MYh7Mwf+90LYix3wREqCKp PBtA== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="DhvWI/tQ"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id a70-20020a639049000000b00585a16c1077si9316795pge.367.2023.10.09.04.27.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 04:27:19 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="DhvWI/tQ"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id E61B6806B042; Mon, 9 Oct 2023 04:26:41 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346244AbjJILYw (ORCPT + 18 others); Mon, 9 Oct 2023 07:24:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42512 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346246AbjJILYt (ORCPT ); Mon, 9 Oct 2023 07:24:49 -0400 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2080.outbound.protection.outlook.com [40.107.243.80]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2DC1AAB; Mon, 9 Oct 2023 04:24:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=l4yeGg4qYqD8O/AKU+sGm7x1dyuipCFNJ49wARZLSPjtXX9OiQt0qqwvZc6Vpjcyoq9TdQUUVVUafZ8YgZxqDyG9uD1AAs3dw7F6P2NRFOa/OlAfufNeRdrzTLaYlsXEQEEpWuDDaMNrB/1IKaP5nDO69SMR74+hdbsvVqXfwasG/bJTep7tCigjrON2sbUk459eT6lKNPJA2f2YkYl5t8NZiIzmUS5YrngIqcCRfury7ILd8ZskX7Xhpjk1y7Zkp8ggkD1hknNOgL4O5PMkriailm/AWBLa7zf2aQSKaBHqKGCzKK/7w6gZ26iXWR+28ZVnR0q5ofARJP/shDWnxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ILKog4btT16IMu9HXAHH95fAm+8LcOyK/4mzDrViAvI=; b=OQhHJ9NMa6c/EoNeWnfmWH8rPp+/eDyCXqwxWvvREgiZvydui0H+CjuM0jBN1l5aF2mUtbrBb+4Sq+u5Xnk6iql1DQuoOJj1pw4DK7xP8eaEAJKV8y1+1zqIS/6DYxYXaFC6y3XFgvkKqO5rC1vDjAXv9UuxKaszCjpXOJH+RlgIDkXXLOeA57jL0KF/CUZaIqZ+RaJMdiULw0SzGv1BfxUZ+9w2/j8yIVN4v9m9gbYnaDKBLKmsOSJdDcWl2TvASCHOMnwUHanvvmiZG5F/erZpYFPZwJ3wYrPww7Eu5q6ReCqkKPCpiWUgu0p78BK+uGFBIegtVYjJHRIcR24MZQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ILKog4btT16IMu9HXAHH95fAm+8LcOyK/4mzDrViAvI=; b=DhvWI/tQWbkP3fImzS6bOL36R4Sjx2D8iib+JYroxzKMhFGQ8tOnN4mGc3OCj5jEFkxNnYw6lZO0p+akAghgiXA/oPNK4Bc0+7OV3NsMpo9jFSwOz2EPD0T9gssFQqSUJZH5By0CKpfMygn1V4NwIFBLlQNmiRTo8yyfAJUuaCmTAJd4aF5tNuDaDR4+15w5df3SnFJbPuodcuGCIMF3GosrTTpwdhejva883gaJ343QEtJ6Ld6I3jLSIgJS4d0DW6sPmjyQwT/h1stR/4OkX/e97HaCmmQeAPyR11V8XU5/f3dg7QACBlR+5HKn87zXmCaHqrfbaPk0g9obYhqLyA== Received: from BLAPR03CA0090.namprd03.prod.outlook.com (2603:10b6:208:329::35) by CY8PR12MB7682.namprd12.prod.outlook.com (2603:10b6:930:85::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.37; Mon, 9 Oct 2023 11:24:45 +0000 Received: from MN1PEPF0000ECDB.namprd02.prod.outlook.com (2603:10b6:208:329:cafe::1) by BLAPR03CA0090.outlook.office365.com (2603:10b6:208:329::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.37 via Frontend Transport; Mon, 9 Oct 2023 11:24:45 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MN1PEPF0000ECDB.mail.protection.outlook.com (10.167.242.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.18 via Frontend Transport; Mon, 9 Oct 2023 11:24:45 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:29 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:28 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Mon, 9 Oct 2023 04:24:25 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v3 03/16] vhost-vdpa: introduce descriptor group backend feature Date: Mon, 9 Oct 2023 14:23:48 +0300 Message-ID: <20231009112401.1060447-4-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009112401.1060447-1-dtatulea@nvidia.com> References: <20231009112401.1060447-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000ECDB:EE_|CY8PR12MB7682:EE_ X-MS-Office365-Filtering-Correlation-Id: d719a82a-0216-4627-de84-08dbc8ba571e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: merPLwWS2lIyOWsQFsjoG0hzTrqwl6RsyKGioZa9EJLgrRYcxG2wF3NcmbB1AMIKV8NU6PJYlN1lVsO1SgU0+vMNrJfuYodnZcIUFCEyw7/Ne6piprvC/6BSAnpWGKAXzt6z5igKYOTfeIie2/L+V57B6WWz5c3Tvia/4s//9Gc8DHJYgaFOdEYKRfLmK8ZCzWnYwYx0A9PGvHc1ugCGp3y/LT2vJlpKhV6B2QNrBNUHNUblMjktCStZKCphqlY96QseKWZkj3B7Sw/KL1NcfOygZMGhxNvGb8BuMuINQujS4AsGLG5yIKrhpIXZSivNlvvBhlOJgEK/GXivyFpHcyYIAT2MZgDNGuTxqejuep5+uWndERCtLRyLjhhHk+pAtglCScfNTkGL2gQfVnEPQuaDQVsclriZ38aKqef2DwxbHhL05Fr0uQ0j4SsEldaK+gkVLLs/1oZWldtbGPKTdzna6ok+aRtwNY4/hxjxQIZF4KHPsJsvTy4100Bi7NnC8eLw6f87MBZRG04yia6aq/QK8OMHQUqFVUdH7zxRVqSEEm/2wNTzyDaBAYK1ctPOv4Outi61wbZsibEGSeG4pLRzx7EBImGTH4r+x7CRGv028Xa6wAwZTOLyivWP4/aluUtZBlUlHyLE5EBsBhrEsHP4sBX3bp4AS1Y8qH9DHOp6h9BUr1Mq4oe7NsnlRmPet0b/BYvxKJd0vVGXX8MI/NJUf4w8P2vK6RfYP9Yi2IQzsUC3iqEbB355wgr9czC9 X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(39860400002)(346002)(376002)(396003)(230922051799003)(451199024)(82310400011)(64100799003)(1800799009)(186009)(36840700001)(46966006)(40470700004)(40480700001)(40460700003)(1076003)(83380400001)(2616005)(336012)(426003)(26005)(66574015)(47076005)(316002)(36860700001)(110136005)(54906003)(70206006)(70586007)(8936002)(8676002)(4326008)(5660300002)(41300700001)(6666004)(2906002)(478600001)(82740400003)(36756003)(356005)(86362001)(7636003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2023 11:24:45.2139 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d719a82a-0216-4627-de84-08dbc8ba571e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000ECDB.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7682 X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 09 Oct 2023 04:26:42 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779277066236713458 X-GMAIL-MSGID: 1779277066236713458 From: Si-Wei Liu Userspace knows if the device has dedicated descriptor group or not by checking this feature bit. It's only exposed if the vdpa driver backend implements the .get_vq_desc_group() operation callback. Userspace trying to negotiate this feature when it or the dependent _F_IOTLB_ASID feature hasn't been exposed will result in an error. Signed-off-by: Si-Wei Liu Acked-by: Eugenio Pérez Acked-by: Jason Wang --- drivers/vhost/vdpa.c | 17 +++++++++++++++++ include/uapi/linux/vhost_types.h | 5 +++++ 2 files changed, 22 insertions(+) diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index 78379ffd2336..2f21798a37ee 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -389,6 +389,14 @@ static bool vhost_vdpa_can_resume(const struct vhost_vdpa *v) return ops->resume; } +static bool vhost_vdpa_has_desc_group(const struct vhost_vdpa *v) +{ + struct vdpa_device *vdpa = v->vdpa; + const struct vdpa_config_ops *ops = vdpa->config; + + return ops->get_vq_desc_group; +} + static long vhost_vdpa_get_features(struct vhost_vdpa *v, u64 __user *featurep) { struct vdpa_device *vdpa = v->vdpa; @@ -690,6 +698,7 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, if (copy_from_user(&features, featurep, sizeof(features))) return -EFAULT; if (features & ~(VHOST_VDPA_BACKEND_FEATURES | + BIT_ULL(VHOST_BACKEND_F_DESC_ASID) | BIT_ULL(VHOST_BACKEND_F_SUSPEND) | BIT_ULL(VHOST_BACKEND_F_RESUME) | BIT_ULL(VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK))) @@ -700,6 +709,12 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, if ((features & BIT_ULL(VHOST_BACKEND_F_RESUME)) && !vhost_vdpa_can_resume(v)) return -EOPNOTSUPP; + if ((features & BIT_ULL(VHOST_BACKEND_F_DESC_ASID)) && + !(features & BIT_ULL(VHOST_BACKEND_F_IOTLB_ASID))) + return -EINVAL; + if ((features & BIT_ULL(VHOST_BACKEND_F_DESC_ASID)) && + !vhost_vdpa_has_desc_group(v)) + return -EOPNOTSUPP; vhost_set_backend_features(&v->vdev, features); return 0; } @@ -753,6 +768,8 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, features |= BIT_ULL(VHOST_BACKEND_F_SUSPEND); if (vhost_vdpa_can_resume(v)) features |= BIT_ULL(VHOST_BACKEND_F_RESUME); + if (vhost_vdpa_has_desc_group(v)) + features |= BIT_ULL(VHOST_BACKEND_F_DESC_ASID); features |= vhost_vdpa_get_backend_features(v); if (copy_to_user(featurep, &features, sizeof(features))) r = -EFAULT; diff --git a/include/uapi/linux/vhost_types.h b/include/uapi/linux/vhost_types.h index 2d827d22cd99..18ad6ae7ab5c 100644 --- a/include/uapi/linux/vhost_types.h +++ b/include/uapi/linux/vhost_types.h @@ -185,5 +185,10 @@ struct vhost_vdpa_iova_range { * DRIVER_OK */ #define VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK 0x6 +/* Device may expose the virtqueue's descriptor area, driver area and + * device area to a different group for ASID binding than where its + * buffers may reside. Requires VHOST_BACKEND_F_IOTLB_ASID. + */ +#define VHOST_BACKEND_F_DESC_ASID 0x7 #endif From patchwork Mon Oct 9 11:23:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 149954 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp1797957vqo; Mon, 9 Oct 2023 04:25:07 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHQ4qDT9gfS4nGg0qY5j5cY4FqJbnrnADvMdT1+i5UuEaIPPLkryThPQOJYPHtYK8KLgvtk X-Received: by 2002:a05:6358:71c:b0:14c:e2d3:fb2d with SMTP id e28-20020a056358071c00b0014ce2d3fb2dmr8840616rwj.5.1696850707422; Mon, 09 Oct 2023 04:25:07 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696850707; cv=pass; d=google.com; s=arc-20160816; b=J6KII/ZoDU+I779PF+UxfV3Sj5iXnt39R4AGF06OCB1hNg8J24SNwFbnd+pczynX9Q 1oRSuQTi7V26lSvFtatTb7z17HCkVy8d9fZ6qxqRYwhkt61PN7ewsLA5DfHp8GF959IA cgSq4b58Ks4Rx1loK1zfCWKSbbLZ1JFfoPPzECLqBC19zQxtlLIK5W9FkE5kpjfTeGNU Ojn1Dp0WTgM3bU7iGphCmjr0oVMNiTg+9FLeIBm2MLyVGRq4r+zDvzLNf255Dcm0IzDV +2Oma7nXKTfrp2YAlNQwKlGhM2Lvumib2yrCFYrkpZPovhOdSTh4vdG2N74Mz5FXgtno 33SA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=u64U1vHaUjCMSpg15sklK0flAB7mos1z6H94Kmb4gsM=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=oRSNOIOmtTlfbuz+OkRLFKLvIJs0k8goPVv7IU7sqf+mmhYBAieOEoaHhiVpzg2l4Y rfAhnX4YC5V/rHzuGPN4glzNjJoTEwvECIX1SFpDYi1F5sJTQdmOPuNSRdczJOF3fAbT KE1ttoWQQQK6VYa57hAgA8TvtFU/4x/+nmi4CgVwETLQmnkAuSyzxByp1hWEsNYPTmUG u+poE721LGx+fuyw9y7BKOB8S8jSbl3Bxhsxezt/ru28mbOva5854aD+kHylcyV2xzYb 8o9vFAqZT91GtdYSZmjxfOqilYztVtkmCtf9NLSjW/Fa1B27f+e0lbIgYm1g93Gv94UA WyhA== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=VRaxj0VC; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id u5-20020a17090a890500b00279202f4151si11479990pjn.45.2023.10.09.04.25.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 04:25:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=VRaxj0VC; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 7CDEF802A81F; Mon, 9 Oct 2023 04:25:06 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346249AbjJILZB (ORCPT + 18 others); Mon, 9 Oct 2023 07:25:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42650 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346255AbjJILYz (ORCPT ); Mon, 9 Oct 2023 07:24:55 -0400 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2050.outbound.protection.outlook.com [40.107.100.50]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A503EAF; Mon, 9 Oct 2023 04:24:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CQ3d9818wWh6FnD+K5dPjN6pT7FxjlBYRV7dg5tIdMOTPUfrrZ9J+6FEBPuO5wFmber687SlkujE2Iy6Rhc1Erk2FQlLR9aQNgkAzup6+E2jmFF1UFTfNAFaqjvu8c+Qg/MjH15qsZn5YWulEshvBJ+esuhoWDIBjsibNOU95W5/hNsx7dReOJMQnIz8JB9DGdaIz7UNUDBt/uaT3A8XuX2sJHwmq1TLWcxSzrMEniIDcBE4HW4V2l/05NvBwnphxEkG2Lh8A3gXkNS4TKyCH1UvEjAErzC8NMZ4lsfIAbdyyslHfoVfopCQqhBghcnblw9htMhnvH7yMR4XgA4nrQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=u64U1vHaUjCMSpg15sklK0flAB7mos1z6H94Kmb4gsM=; b=DUWc/92fc1tts21LHN663DMS5xT91IzZxqqlDhWinxOmIVf7ZtTsO8QfgJJYM0DIZLgdD8I25F1sYSS4VKGgDk2LuBS8oS8Wxxuo3LJKjUr4rccigA4n3MBYUQHrTWn+cDQ0s2Q5tv0Jkfzmqah1r7ieyToBC+kE1nxfflHvDwL+h2XP+YKh6xtJXXAo56glroRF07seeCKfxuauyBT4gLooTATyZHrys2s5xiv7Bc2xGB6Oh42C8aV7jRSi/g8fo1dhD2XmQVSZvgxZiLJ1uokTIpZu7hZTrDb7aYJsZYUmgJhc48VsXJfSx7kWh62VxvK4Bmk/rYdoG45bz7Ur9Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=u64U1vHaUjCMSpg15sklK0flAB7mos1z6H94Kmb4gsM=; b=VRaxj0VCg55fHht/uxDVi4YZdXiz8W82DybnF3jR3brasAnsy7JhVoxbcHM6pZ+Fd/aOjS0FT+n8ySv17fR35VXm4YZj9VYhdGnT7F6P8f7JXdMnqOa57HBV8UmGTI5ET/qrgL9PTJ2zxtnI/mvLhY4tLY5bMdgEXxBD7R1ri3HdLGjJ01I/vaJfmmo/B88BCRMFhq3s5OuMNKqfKnCIIpjTQHVtjbRabMK9wuQZncPybU7Euj7cEYU6MSP5WuiNb3r1kFQJm4TvJkOAcv3w3ZS60MvXLJ6YLLimi7hcLor9x0vDzo7m49BZIBKbPwbTR2UFZOGVSOI0RyucBiyTwA== Received: from BLAPR03CA0085.namprd03.prod.outlook.com (2603:10b6:208:329::30) by PH7PR12MB8013.namprd12.prod.outlook.com (2603:10b6:510:27c::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.30; Mon, 9 Oct 2023 11:24:50 +0000 Received: from MN1PEPF0000ECDB.namprd02.prod.outlook.com (2603:10b6:208:329:cafe::f9) by BLAPR03CA0085.outlook.office365.com (2603:10b6:208:329::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.36 via Frontend Transport; Mon, 9 Oct 2023 11:24:50 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MN1PEPF0000ECDB.mail.protection.outlook.com (10.167.242.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.18 via Frontend Transport; Mon, 9 Oct 2023 11:24:49 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:33 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:32 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Mon, 9 Oct 2023 04:24:29 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v3 04/16] vhost-vdpa: uAPI to get dedicated descriptor group id Date: Mon, 9 Oct 2023 14:23:49 +0300 Message-ID: <20231009112401.1060447-5-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009112401.1060447-1-dtatulea@nvidia.com> References: <20231009112401.1060447-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000ECDB:EE_|PH7PR12MB8013:EE_ X-MS-Office365-Filtering-Correlation-Id: 4e323531-6119-4d64-0a55-08dbc8ba59f6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: O40x9nFbR6U5s4J/IDnKsbBDIwl2RoVi0ikfU+HWPLZ0E2RoRP0tlNeKvSLfurqaGJVu9WTKPXeUyF6B8PljFqldoCeLkbCi4nW8/Ng3F3PnnyCP6pqwci08mMLtNUi7XmV7InZPni9Lprz4cVLYooQVzuWjs8s+EAki+x/KFfKkodxqMZUJETKKQvLWuQKpO6ZzrN53Nxrll0z75MBGAkG0JOYaJFdsIQNp/DgzYnX1FrDJmZlshe2iNq1ELMUFWQMKrmz7MOixMVyF7bk7lJEPJApyvipJCMv0+PXYLu8wAp+JS+/x1ONDOR9ZuuKcg4dIMda25bm2ge6UDhi0oEolxelVsLzJzQzWTOpWK93fRHMonUNECLAbrVUCt7eV1kZicocR3bUrJvxX3hzskUBkhE/bzFzGxmZp9ULBzmKfl2G9XL+rvtRF5nAaBcav7bdjhrVCyI6qg0nssDM7idCFFEKLduxt3KLeia77c1U8con1WG08jyK0IYil2Z0ynoYJO5zTeS0+M9WQpKa3H9sNCcnSFwLUy1bj1FLRsQ5hJC9G8uSrvoQYtsFCHs+vQ1NBGKHBtMP1ohvJQm7rjHMj+cHjkG1cf05kZIpUhVQk7g2ZqMEy2NDtrRcJ8wN/alI4cR9TRv65nLdf8IcA/3AfI48h/5vKFXpli6zkgwHfFT1wSKP2hQ3Hs8RvbnL7/46EWR3WGKJRnt/WWySW99BuwOJXQZjvBM/H3TCI+SgRMQKaBqDAUDDIzDj1BFkn X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(376002)(396003)(136003)(346002)(39860400002)(230922051799003)(186009)(82310400011)(1800799009)(451199024)(64100799003)(36840700001)(40470700004)(46966006)(40480700001)(40460700003)(36860700001)(478600001)(47076005)(6666004)(110136005)(8936002)(5660300002)(8676002)(4326008)(41300700001)(70586007)(70206006)(2616005)(54906003)(83380400001)(336012)(426003)(26005)(1076003)(86362001)(316002)(2906002)(7636003)(36756003)(356005)(82740400003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2023 11:24:49.9794 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4e323531-6119-4d64-0a55-08dbc8ba59f6 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000ECDB.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB8013 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 09 Oct 2023 04:25:06 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779276926891031463 X-GMAIL-MSGID: 1779276926891031463 From: Si-Wei Liu With _F_DESC_ASID backend feature, the device can now support the VHOST_VDPA_GET_VRING_DESC_GROUP ioctl, and it may expose the descriptor table (including avail and used ring) in a different group than the buffers it contains. This new uAPI will fetch the group ID of the descriptor table. Signed-off-by: Si-Wei Liu Acked-by: Eugenio Pérez Acked-by: Jason Wang --- drivers/vhost/vdpa.c | 10 ++++++++++ include/uapi/linux/vhost.h | 8 ++++++++ 2 files changed, 18 insertions(+) diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index 2f21798a37ee..851535f57b95 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -613,6 +613,16 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd, else if (copy_to_user(argp, &s, sizeof(s))) return -EFAULT; return 0; + case VHOST_VDPA_GET_VRING_DESC_GROUP: + if (!vhost_vdpa_has_desc_group(v)) + return -EOPNOTSUPP; + s.index = idx; + s.num = ops->get_vq_desc_group(vdpa, idx); + if (s.num >= vdpa->ngroups) + return -EIO; + else if (copy_to_user(argp, &s, sizeof(s))) + return -EFAULT; + return 0; case VHOST_VDPA_SET_GROUP_ASID: if (copy_from_user(&s, argp, sizeof(s))) return -EFAULT; diff --git a/include/uapi/linux/vhost.h b/include/uapi/linux/vhost.h index f5c48b61ab62..649560c685f1 100644 --- a/include/uapi/linux/vhost.h +++ b/include/uapi/linux/vhost.h @@ -219,4 +219,12 @@ */ #define VHOST_VDPA_RESUME _IO(VHOST_VIRTIO, 0x7E) +/* Get the group for the descriptor table including driver & device areas + * of a virtqueue: read index, write group in num. + * The virtqueue index is stored in the index field of vhost_vring_state. + * The group ID of the descriptor table for this specific virtqueue + * is returned via num field of vhost_vring_state. + */ +#define VHOST_VDPA_GET_VRING_DESC_GROUP _IOWR(VHOST_VIRTIO, 0x7F, \ + struct vhost_vring_state) #endif From patchwork Mon Oct 9 11:23:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 149956 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp1798109vqo; Mon, 9 Oct 2023 04:25:28 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGlhnUk3SDwAKaJd0mi25+NrrAB4oSK+7J1byg/Z5TizIU8OznliuntcVB3wOMzHrQNu+e5 X-Received: by 2002:a05:6a20:4292:b0:14c:a53c:498c with SMTP id o18-20020a056a20429200b0014ca53c498cmr15391901pzj.10.1696850728363; Mon, 09 Oct 2023 04:25:28 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696850728; cv=pass; d=google.com; s=arc-20160816; b=vXFX8AAVzyY964RuMX6t2q32vuZSHPoAjCJWcnkry7/6xkRd4fX9V9CKU6GVkCM+fP CKp6WLSGSBGQKgy5L7HQflD5+naWSpiefbV3Pqje4ofDPM+9C766Ai/CH5BiIuQlfkf9 34jiiOFDrpsRI0QVAaJcGqvB2XWS2656W+sBlx5SkItrw/Q+0iDn3G2n1HNhVOjboPhN zREbT52CN9Hanlei4tfUX1jfcF2DkcuWa15twTdyN1jlcS1XeMthoLUznhcPM0k/1InJ eSUtWusdhI/VqLn65ybvAu/2/mMmhzbUSBgW39l6kT1PWC0j2t99miyXie7R3PNVG/eM /KcA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0k7vZDkq8HIAIRoAeeLZ+cqnJ3LV329QdGvZjtWRFo4=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=TeWWGfypycNaWOjpIkdnDiNF2ZiCCclAnyyvL7VyYH1Grfm/cTBmwKbOb8XZlQlPnC CXORmOcDzntM12FFp0N2OkM2cjw+AyuGcuzvlwAtpZxnUms5RKXyMMGkFya3iChaO/2Q tF8ALGD6VR1hBah+UXAEOizTmhwJ/5EtI2b/cgPsa1wlIyf4oxmktqd1EUbF20iWSt8n cgRDHevryatw/xCVWGmXCKoOrZkeA/iFUYJ+BcOZdmRiSWnTn+nomcn7u8damatHkvqA UCvLSrZZ0BGBxsP90UcP3PcUIHy36z4gCT4Ag9Jk4EBNiCE/0VEwqdZ+6GzgkJPmFoRW Vyfw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=aZd72BvB; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id f10-20020a056a00228a00b006933b29f560si691187pfe.46.2023.10.09.04.25.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 04:25:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=aZd72BvB; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 089DF8020A35; Mon, 9 Oct 2023 04:25:26 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346296AbjJILZP (ORCPT + 18 others); Mon, 9 Oct 2023 07:25:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51970 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346293AbjJILZK (ORCPT ); Mon, 9 Oct 2023 07:25:10 -0400 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2060.outbound.protection.outlook.com [40.107.212.60]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0FB7115; Mon, 9 Oct 2023 04:24:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cOZ6osvtSnIv2VPJQ/dmmWImZIZmjT3b47fURvh0v1IfGP3me2eHvot3duqq9t135d0erDnNPB757UTr0insOCaoMHRMhMsqW9h8Qo4IZHV4mlWM6R6CqFn3m2RsdgfvO/bhhu53LBCpe3Lcp4rA9+fKyPY/KPRGLUQ49VNlahoDRabEyusKAUOVm49I9HTjA40CgKIerLnJhATX1x1pxGOaXaqic+N3r45poTfRWraTOUChj0yEEreBNIzs556dn86V9cJ2Y++v2aZ9b+eJed6r/zNzyGbearyjejqGH3+IRNdxdjEGAoI/0/WYYydhHn5P6DuC/gXiNQEAtPdRLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0k7vZDkq8HIAIRoAeeLZ+cqnJ3LV329QdGvZjtWRFo4=; b=SlM1MX3If+1oa0yumFvsmlNmtgi3DScGrsG+rBvkbbMwrokGgWX/CfcFtq0tYxISEAtRSjO1dAn+lMP260dMSjg4X0ozEbFGydJU7d2sHIHB+oBYw6hSGgCUUNWW1fipwBFQT+8NFYCHjEdwQWhA+8TTgkEYoY+OHxgwqyEGhNieWtWBSlUt4v/nSRvxhEZITnrnZoafwSNvnFSUKJ5sPcUBdpEJRIFSXS/VifOYO4UWFw1Q2vb5DAYOI2EzaBnO2go2mfPTK0z9+Z2lR6A9P0+yWAhTOaJjCkuuYoyThCb1IVOok3TrrY6SEtHzLucYQLPd/0SjK61fQOAue7Tdeg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0k7vZDkq8HIAIRoAeeLZ+cqnJ3LV329QdGvZjtWRFo4=; b=aZd72BvBri3ocRkEetFBtOAJX4xNj3svGZN1DK/VRBqwQOvgfdol/W85sLUfFsm+jsFB5wi1C/IZzCQ9AFZ/bmXo7Guyab9khCo/7bg6eINLrvroGxpL5Mk8M7Eu/5Of3cVVufvHw1O2Zm8HgybwUtZFnuiGCRDn36tooWRwC7BczOI0d+N3GJpRJ0XKZtnkus/2Te7FfNI3I+obGXLCdzpNEEw5pkJVFkPSFzw2qAWqCxqc+Vam0B2yELTAeunR6deu2YUw70SquEcsK4ySyjge7COd+NhlTrmwgSdMJJOuNSGqvAEkQ+Zd1CGPzPoKxCWsFc0Ny+3H9KSPxdR5AQ== Received: from PH8PR20CA0002.namprd20.prod.outlook.com (2603:10b6:510:23c::6) by SA0PR12MB7073.namprd12.prod.outlook.com (2603:10b6:806:2d5::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.36; Mon, 9 Oct 2023 11:24:57 +0000 Received: from SN1PEPF0002BA4B.namprd03.prod.outlook.com (2603:10b6:510:23c:cafe::65) by PH8PR20CA0002.outlook.office365.com (2603:10b6:510:23c::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.35 via Frontend Transport; Mon, 9 Oct 2023 11:24:57 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SN1PEPF0002BA4B.mail.protection.outlook.com (10.167.242.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.14 via Frontend Transport; Mon, 9 Oct 2023 11:24:57 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:36 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:36 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Mon, 9 Oct 2023 04:24:33 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v3 05/16] vdpa/mlx5: Create helper function for dma mappings Date: Mon, 9 Oct 2023 14:23:50 +0300 Message-ID: <20231009112401.1060447-6-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009112401.1060447-1-dtatulea@nvidia.com> References: <20231009112401.1060447-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002BA4B:EE_|SA0PR12MB7073:EE_ X-MS-Office365-Filtering-Correlation-Id: 844f0bcb-dcc0-4259-752e-08dbc8ba5e2e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: X4+BdXqnK4QSivMcvtklbKRxX8UmnNcsqbOpBupHYuPSWtlfIkJXLSsYXAv5/e4UtQy++y4LcS1SzIe+sxsKr3cBWPseeCououh1chSpTfVZq6Zp1Ap3Y4Hqq3aTTRah5WMj9CFgbN2MahUVytVsow6IaB5wecGTT73VdytO5dOnFxSedA2uvrzShoYqYpbm5/oyweOBWVTbtqL8aono82dGMX8s+7R3xGafm1Yd1pTQtFwTJhByJjKobxcXt2rLFTybeRybyVXH8BqOrlfA21zz1lG17Kx1sfd+2/On/UX02c8H+qJv9/7ZlWoHHpD2VM/svw0L93vNiy7hEjidKaURYtoDZdkkOrYPyKEPbkf+OFmBtM3MQJWHq96DpUT9J+MjYG08LDA20wwjzKPkID++9GEQLipsPJi4CZD1qurYa7tNqnXi1otiQjd0oN7CMhEjhYiXcvZ8D+ndP35rnV61kGW68tTiPyJ3Hokw/GfPu6wa6goQAnK/h7tr0Nm4IrWA3TSFUvzE6JcxyltgryTQzbohUFdxoccagEvssJaXONtxD1SCrxK56YEqRdX5JKdICu8e++MeHv9+xYyJj0uF/wy7ih6kOCKYl2oaWW8TsdWEW5gblLLn9A9n7nJfHi+NQgKicCK9X1/uF4yVcHk277geaYcjtO2dmenUtOwJNVvesC8ZD1jNIt5sIgrMknamjq/w0612SWpInkS/y5Qqja3ymyB3DDYmFG40Y5o= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(39860400002)(346002)(376002)(136003)(230922051799003)(82310400011)(451199024)(1800799009)(64100799003)(186009)(36840700001)(46966006)(40470700004)(1076003)(2616005)(426003)(336012)(26005)(40460700003)(478600001)(6666004)(8676002)(47076005)(36860700001)(83380400001)(2906002)(4326008)(54906003)(70586007)(70206006)(110136005)(8936002)(41300700001)(316002)(5660300002)(7636003)(356005)(36756003)(82740400003)(40480700001)(86362001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2023 11:24:57.1226 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 844f0bcb-dcc0-4259-752e-08dbc8ba5e2e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002BA4B.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB7073 X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 09 Oct 2023 04:25:26 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779276948915970896 X-GMAIL-MSGID: 1779276948915970896 Necessary for upcoming cvq separation from mr allocation. Acked-by: Jason Wang Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 1 + drivers/vdpa/mlx5/core/mr.c | 5 +++++ drivers/vdpa/mlx5/net/mlx5_vnet.c | 4 ++-- 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index ca56242972b3..3748f027cfe9 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -120,6 +120,7 @@ int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid); void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev); void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid); +int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvdev); #define mlx5_vdpa_warn(__dev, format, ...) \ dev_warn((__dev)->mdev->device, "%s:%d:(pid %d) warning: " format, __func__, __LINE__, \ diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 5a1971fcd87b..7bd0883b8b25 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -619,3 +619,8 @@ int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *io return err; } + +int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvdev) +{ + return mlx5_vdpa_create_mr(mvdev, NULL, 0); +} diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 40a03b08d7cf..65b6a54ad344 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -2836,7 +2836,7 @@ static int mlx5_vdpa_reset(struct vdpa_device *vdev) ++mvdev->generation; if (MLX5_CAP_GEN(mvdev->mdev, umem_uid_0)) { - if (mlx5_vdpa_create_mr(mvdev, NULL, 0)) + if (mlx5_vdpa_create_dma_mr(mvdev)) mlx5_vdpa_warn(mvdev, "create MR failed\n"); } up_write(&ndev->reslock); @@ -3441,7 +3441,7 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name, goto err_mpfs; if (MLX5_CAP_GEN(mvdev->mdev, umem_uid_0)) { - err = mlx5_vdpa_create_mr(mvdev, NULL, 0); + err = mlx5_vdpa_create_dma_mr(mvdev); if (err) goto err_res; } From patchwork Mon Oct 9 11:23:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 149967 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp1799003vqo; Mon, 9 Oct 2023 04:27:36 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGhrxTe7cmvI2KL6sGVfQz5jV+QDVjbV3EW49aHL2wFvG0DGcDeJ5pvctViNqKYRHk3yR27 X-Received: by 2002:a17:902:d50c:b0:1c4:fae:bf28 with SMTP id b12-20020a170902d50c00b001c40faebf28mr13929682plg.32.1696850856213; Mon, 09 Oct 2023 04:27:36 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696850856; cv=pass; d=google.com; s=arc-20160816; b=zyMlXHyATvc4b4DQkC3/eTKIUIy62A5VjXEFSh2uW2qWXdi3vyhnStRtOAE3w/ZNtk aadZoiQUMhak2ZMoZu25xoEAPMTrIVlCUyJBoq48jzhBBRa5z8s7gjcqeoCi8hNf6XEn jM5EnQLooIuoWJQSRuWbldbBUF7rF5ySKoKIBQnQSn2/mYK4ypPDBVz9gb2tOyqtfPDK R28Ik1xQVqXSNRnq+6hLxiq+zMd5bRtyzdRnzLJFEGDbUHoKdfvgUGTBjGf8g+MTHWZt t9FAmdQo3fMwVKpfozP/soYJq/Z+FZ9dS0sjL7vQep9i89rJ8w+0eoDI408y+vVnkJLS a+nA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3YoDCHkpinOQBJC5OCWVs0dAVcWKDrjzxMHgzD+IBkE=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=xEBRDTMfZwxJEb0dqMdlEoW3Rt2DHGH2U21dYr9qP1hF5AeKvRUv9FR/531g4Oz9vc x4K7hFFLzhOrHMbUZFZIexl11HoZPj4uNVKktMpdzXrKBVXQqqoxtgaJF4AkRwLRFW6k 1ouxhgeJFmSB4MsvRZNSm7Y3xaKLZanI03mImqq2NFSu1jLBrS8SVTuiUKrw5IOXTnaP 8cz/CYosvkoZL3yXUgSOBlSy3203aEO3YLshLEIxTOOM/Si2/lgfL3tlGCy/TaABkIWt JNHODUgZv4aq2PACaqIaJynODgDG1rN4KknUiegM62yqPaNajJzgUpT6DdJ0bL/fHay1 YgFw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=SdljBuoG; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id kw14-20020a170902f90e00b001c72c893ad0si9116420plb.48.2023.10.09.04.27.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 04:27:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=SdljBuoG; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 51B2A8076E43; Mon, 9 Oct 2023 04:27:16 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346318AbjJILZU (ORCPT + 18 others); Mon, 9 Oct 2023 07:25:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52072 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346306AbjJILZL (ORCPT ); Mon, 9 Oct 2023 07:25:11 -0400 Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2089.outbound.protection.outlook.com [40.107.101.89]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D710112F; Mon, 9 Oct 2023 04:25:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cBdumJDOlDNfUwpoY7KBpiLJWduF06nxrIIu71b9wo+D6rNzt13Hn6bkdlr6ZVG2KrLdWVaWuDiGYJ6hs6cNnCvKQepjY2vXMs1P5/w19iTcYN+SA3WpZaaULbAIrRAGqUezbiWOsO2ueTVJ+DKVgxtQ8uRwg6rKSkyzC0OJgTiFFoanP6YpNdiw/BuNJdks0ObQDUxG/CpzqC2hVLU/U1FXbEqUrTEy3wrPcqzqPH9vXg6moGqdyqgpTQS4sn2ppkXqTCoN7WuBzugfPYwja/FeV2lZXfLVBjrCYvluPJYpJiY8+SwlDB04dLeULjc796cxLCuiPfgKi2iGZ8VLGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3YoDCHkpinOQBJC5OCWVs0dAVcWKDrjzxMHgzD+IBkE=; b=bJUIVHvjxy0HE/Lv/K+Ae8LIEhCEfcxRtbzOBqiOf0r6opqvlRXG1U1Lkppcy8xWXt42zROsESsNISDBqVKcfhgPntLz4KQN3OZllgog6Z2mvGKOxlQ5dNo4TnHfwmNSCiZZWtjzcaVgWCGO1HZ76rvolNibwXwqP0SP/7L/kk/TK4ne6SDMfngezuCtzUYzoyltagWmGATF7KC0aJqPlp7+vksoUGloCYUqHf7GN9rVnoSbmmReoRvk0HJJ3r8PA9IYcstsFP/vDnx0MXOpnZGSmZ24SE2SUiMVgfPDkjjJNYVgMrY3lEhuOUlSKR3eWomZTxxFvJG+VdhqRmMEhw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3YoDCHkpinOQBJC5OCWVs0dAVcWKDrjzxMHgzD+IBkE=; b=SdljBuoGdSS5da2eAYp0JmpjDPUIvjbo9nXosvgwPbnEDp3hYQ+hbRpF6vxO28cuim1u7w/mEaOeSbUIatt40oC22T8wSOyL0w+jngHDondBU0wFFmFZ7G4RMapgL9y9kP9tOxHli9tl0spUjYOjGBYxBgWDLzxSdDWW/QBJtVMZ3r9VoV+bz0mGIsgLqfwP6MxYeOBFHR0kGM+TWc7yJpi5jeSFhmimocgkPWZV6KK8PYSGRCyQ761mwBjAiQxrRk5dRjoYc1sYvzj6LgOJ6ecw1tl4A5ek/Q0H2tN/1jr+herWT43+xWwDuRtPpjJ5VN6BZc2ceL28hFQ8YZTwag== Received: from PH8PR20CA0017.namprd20.prod.outlook.com (2603:10b6:510:23c::17) by PH7PR12MB7844.namprd12.prod.outlook.com (2603:10b6:510:27b::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.36; Mon, 9 Oct 2023 11:24:59 +0000 Received: from SN1PEPF0002BA4B.namprd03.prod.outlook.com (2603:10b6:510:23c:cafe::42) by PH8PR20CA0017.outlook.office365.com (2603:10b6:510:23c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.36 via Frontend Transport; Mon, 9 Oct 2023 11:24:59 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SN1PEPF0002BA4B.mail.protection.outlook.com (10.167.242.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.14 via Frontend Transport; Mon, 9 Oct 2023 11:24:59 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:40 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:39 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Mon, 9 Oct 2023 04:24:36 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v3 06/16] vdpa/mlx5: Decouple cvq iotlb handling from hw mapping code Date: Mon, 9 Oct 2023 14:23:51 +0300 Message-ID: <20231009112401.1060447-7-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009112401.1060447-1-dtatulea@nvidia.com> References: <20231009112401.1060447-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002BA4B:EE_|PH7PR12MB7844:EE_ X-MS-Office365-Filtering-Correlation-Id: cc8d5458-f813-400b-f140-08dbc8ba5f4e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Ck4vLrhzJNjydA9vZJZxurot9e+CnEh79o6FYA4OpalJPVsF2/up/d9Zv3bpzfQTXVJeAzzR2jYmoguyiX91nhsBA3V3OwssH52T+4xCwQHaKCdlVT24yA6LwhezqJp8M3sQgXzw9V9Z4L9ENF2ySPS6si/ec3TpNcl3eIbgy04ZtXr6wHoL3JpvOjcBsHFKw/F7SQionHw2+RCUlcYadOPE1p50sBrkq9NjcdRm1AGZXot1L3vdlzg0VFZAgxzWgSK4OtVUVwBlDHjKFXyQGwYuYaowimV76/QxNbMiboa2XL7ieW5F901CS20AqRiDkcl6SgX1J+R3Sxvj5uCIyCahIsapIMP2ZUCL/9ZcrQbmzGbLok7zHMyFynv8w3aNhQkS4AOwru2ILffxj3jFXlURemqAIkewtllUTM04eEd2HqYQ9vvbu/N85ARmVQvOkktUuvBsrAjqWijmHWtBhIp3+aawJAP8jfSE4CVryUAK1ZIWxYubCgQ/tDY3NYQChM8N9a5owiFtXIiUfQUMRXP/Uknky19MGLX1MJRnb+fLtezoa0k8I+d0SsYoyrNDfBPFFhRVKiYABkH1OxCaaWW7OpxZgTgJ6OT/u93rcAFYYqt7cObOFgNT/U+GgoUvfwNdGkMQcw6vcz6UXuThAb7bQeOghHbNwhZwx+oBttPLdTEbvHkPF6WdwoL2mca38sOSUCZpnRCHHfexQkO50gKPj4EwI5MmjAx7cpCf+AU= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(376002)(346002)(39860400002)(136003)(230922051799003)(64100799003)(451199024)(1800799009)(82310400011)(186009)(40470700004)(46966006)(36840700001)(7636003)(356005)(86362001)(36756003)(40480700001)(2906002)(82740400003)(478600001)(41300700001)(8936002)(5660300002)(4326008)(8676002)(6666004)(83380400001)(1076003)(426003)(336012)(2616005)(40460700003)(316002)(36860700001)(70586007)(70206006)(54906003)(110136005)(26005)(47076005)(66574015);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2023 11:24:59.0132 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cc8d5458-f813-400b-f140-08dbc8ba5f4e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002BA4B.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7844 X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Mon, 09 Oct 2023 04:27:16 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779277083502486962 X-GMAIL-MSGID: 1779277083502486962 The handling of the cvq iotlb is currently coupled with the creation and destruction of the hardware mkeys (mr). This patch moves cvq iotlb handling into its own function and shifts it to a scope that is not related to mr handling. As cvq handling is just a prune_iotlb + dup_iotlb cycle, put it all in the same "update" function. Finally, the destruction path is handled by directly pruning the iotlb. After this move is done the ASID mr code can be collapsed into a single function. Acked-by: Jason Wang Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 3 ++ drivers/vdpa/mlx5/core/mr.c | 57 +++++++++++------------------- drivers/vdpa/mlx5/net/mlx5_vnet.c | 7 ++-- 3 files changed, 28 insertions(+), 39 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 3748f027cfe9..554899a80241 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -120,6 +120,9 @@ int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid); void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev); void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid); +int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, + struct vhost_iotlb *iotlb, + unsigned int asid); int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvdev); #define mlx5_vdpa_warn(__dev, format, ...) \ diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 7bd0883b8b25..fcb6ae32e9ed 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -489,14 +489,6 @@ static void destroy_user_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr } } -static void _mlx5_vdpa_destroy_cvq_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) -{ - if (mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] != asid) - return; - - prune_iotlb(mvdev); -} - static void _mlx5_vdpa_destroy_dvq_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) { struct mlx5_vdpa_mr *mr = &mvdev->mr; @@ -522,25 +514,14 @@ void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid) mutex_lock(&mr->mkey_mtx); _mlx5_vdpa_destroy_dvq_mr(mvdev, asid); - _mlx5_vdpa_destroy_cvq_mr(mvdev, asid); mutex_unlock(&mr->mkey_mtx); } void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev) { - mlx5_vdpa_destroy_mr_asid(mvdev, mvdev->group2asid[MLX5_VDPA_CVQ_GROUP]); mlx5_vdpa_destroy_mr_asid(mvdev, mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]); -} - -static int _mlx5_vdpa_create_cvq_mr(struct mlx5_vdpa_dev *mvdev, - struct vhost_iotlb *iotlb, - unsigned int asid) -{ - if (mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] != asid) - return 0; - - return dup_iotlb(mvdev, iotlb); + prune_iotlb(mvdev); } static int _mlx5_vdpa_create_dvq_mr(struct mlx5_vdpa_dev *mvdev, @@ -572,22 +553,7 @@ static int _mlx5_vdpa_create_dvq_mr(struct mlx5_vdpa_dev *mvdev, static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid) { - int err; - - err = _mlx5_vdpa_create_dvq_mr(mvdev, iotlb, asid); - if (err) - return err; - - err = _mlx5_vdpa_create_cvq_mr(mvdev, iotlb, asid); - if (err) - goto out_err; - - return 0; - -out_err: - _mlx5_vdpa_destroy_dvq_mr(mvdev, asid); - - return err; + return _mlx5_vdpa_create_dvq_mr(mvdev, iotlb, asid); } int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, @@ -620,7 +586,24 @@ int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *io return err; } +int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, + struct vhost_iotlb *iotlb, + unsigned int asid) +{ + if (mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] != asid) + return 0; + + prune_iotlb(mvdev); + return dup_iotlb(mvdev, iotlb); +} + int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvdev) { - return mlx5_vdpa_create_mr(mvdev, NULL, 0); + int err; + + err = mlx5_vdpa_create_mr(mvdev, NULL, 0); + if (err) + return err; + + return mlx5_vdpa_update_cvq_iotlb(mvdev, NULL, 0); } diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 65b6a54ad344..aa4896662699 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -2884,10 +2884,13 @@ static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, return err; } - if (change_map) + if (change_map) { err = mlx5_vdpa_change_map(mvdev, iotlb, asid); + if (err) + return err; + } - return err; + return mlx5_vdpa_update_cvq_iotlb(mvdev, iotlb, asid); } static int mlx5_vdpa_set_map(struct vdpa_device *vdev, unsigned int asid, From patchwork Mon Oct 9 11:23:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 149959 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp1798694vqo; Mon, 9 Oct 2023 04:26:46 -0700 (PDT) X-Google-Smtp-Source: AGHT+IENBnJjRfWQLKtkoklVn1MCG96y8vLDBxvDbhHznOQDolCvBdzKs3QVaFfT5909cI4+aSNx X-Received: by 2002:a05:6808:1786:b0:3ad:fcd5:3dd6 with SMTP id bg6-20020a056808178600b003adfcd53dd6mr21411710oib.13.1696850806106; Mon, 09 Oct 2023 04:26:46 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696850806; cv=pass; d=google.com; s=arc-20160816; b=YdeRfQ9H93bFlN4PHSLyuOM7t8upkxdo2yZN2Ii0sNtqiDyk9va6cPoIcAkyL+ILEd 9gEHThJ5x8jbpi1KCZDJ4e30kt7R5P8rRB9qhLjYDf4y1du0A/v879GY2yNXSCb8Nf/G z2pUV/nBlBwxGafZlmHUX/5lfCaaN+nUuvEiXWsd1NBxuDe8ChCyAxbdYPDAMyH+tc36 HBSu9OVg3mbtovCaAolI/cmw1qsYVN/DREUBX/pmSWeZBjuieGuo4BZln6RgS4/XXv2f vvSWn+M3Q7+wL6ZQ3udHjsHH+4pbkY32MUg5dc8SCFuPWYTsdZG8C0+/qMK9Y30LxZW8 eGgA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=sqfty/jYzKnLc/JEwSMOvq4UGNN3rkOgV9qEyvqXUKk=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=Hqt6mTC2vT5c0/B/WwAfihl17KtqPN8AfI8nTYnCKox6Qn8oJGrLqASHmZcnV6VOMM nVYKYutpPgecmfXkbZA5CmAy0tx29eK8Dg0JgPaMhC5gJIwHdyEQ8y4zGqT2Nv57OaNW cwo5lLZCbWg0HdatKvQd6AvMn04psCBGecW76KRrhE/2oMrzxDH5QiH9jVmyv1K+TtA2 +YnVqo5UIfTgBZgIiYQW3rKuyQQQ9sZf8NpBt+y03HMeIrXPmQZEhBIzXqtPqrBkEwe+ QxK3BSrmXKtWRvQbn6qSV4dhoRE++yilOeSLw28Fo53yVR3FPTunoAGPhjoj2INMVgOe hjjQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=UyTqBMsM; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id a21-20020a63e855000000b005784ab1c4afsi9739144pgk.736.2023.10.09.04.26.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 04:26:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=UyTqBMsM; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 6823680BB5C5; Mon, 9 Oct 2023 04:26:09 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346364AbjJILZ3 (ORCPT + 18 others); Mon, 9 Oct 2023 07:25:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51902 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346302AbjJILZL (ORCPT ); Mon, 9 Oct 2023 07:25:11 -0400 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2081.outbound.protection.outlook.com [40.107.94.81]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 599EC129; Mon, 9 Oct 2023 04:25:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nR54YzSwT/cxGJ9R0frM+IRMn6GtsZX4yDgGb1NMhfervUARASdO3NAQEL4jcKVQ3JKdNIDeMgwhrU1LnL8Q8RvA4mefiOcz8ZilLhv4fU1NTkmV7VpWn3lRTCj1RShdn840kXmqlD2ihbYWOnEEraxhTO/jNWCodwQbWPeRqkOgVTI6mr5kHUMoVbd6Qr/eKdPT1c00GiLEUUUpKrXb7r39dEF8JvGNrGocs0HtP78TjzBJPqsFxlW63qF009B/tn/L0RDTCbTe4LGlx4BwQrBLKgHE1o4Fjw/ER4uiWhq8SymqD6zZFXh2dmhC+pIY7RkTwUKpvdxp7U3vtrbjDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sqfty/jYzKnLc/JEwSMOvq4UGNN3rkOgV9qEyvqXUKk=; b=KAQlrfx9qDTnGtkQ8+psa0wgciQ68xC2ePwm8FzVzYkpqpbeYq0f3VaB+4DqG3MAMbAfnFMXkidH7pbq0n/6FyFfnMrtwWdLQnjnCI3WI3nZV5XfvZu/1XjVUC/nqEupllwmr5r3j/qwkJt62kTaNfe7GQN9qUmPE/hI/GW22YkPm7EhzOma4WIDCa6a900JUJNmTxNphZdMKz1dIisn36dKzpn788VzXomiH7jf5rYNu7kFqChNeS9rfTB/Fsiq8nL/s6Oa0NRPrKOa3+/1iH3Ul2+s7Fmt7bWH58YiIO2SeKRMgB7urOUVWLiP3M7x/GhTPPGZM+S1RTbLRUCnzg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sqfty/jYzKnLc/JEwSMOvq4UGNN3rkOgV9qEyvqXUKk=; b=UyTqBMsMAAhpe0S2VLxY8lziOC1IqsF7U1hYbAC8IpOKLIhJbzxvB0Wbyjx+KsQastCYWL1UkyiufiYyj0FFy8hzNB/bitKR9s/rlvg821ad76UnnvieP48DEDcb9pkSZ6n2SR6sIwPgdGpMRI1+dflCfxW42J9YooS1+Qp662sZ4MlHLsQr+MrumLzeQHvOPB9w0jHbK+2a+XcJeQXYw7TtTB9xy2uNi8YexDcNtGKUWYDVrp65XQqrNbYbTWIeOqP3I472IM2qz74tAPG5zgimc52FCtip8GubxDMsN69V81aqdRu5TU0X1dUYA7Y0NVaCJ1ZpD5L3ik77e5XtbQ== Received: from BLAPR03CA0032.namprd03.prod.outlook.com (2603:10b6:208:32d::7) by PH0PR12MB7887.namprd12.prod.outlook.com (2603:10b6:510:26d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.43; Mon, 9 Oct 2023 11:24:59 +0000 Received: from MN1PEPF0000ECD5.namprd02.prod.outlook.com (2603:10b6:208:32d:cafe::5) by BLAPR03CA0032.outlook.office365.com (2603:10b6:208:32d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.37 via Frontend Transport; Mon, 9 Oct 2023 11:24:59 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MN1PEPF0000ECD5.mail.protection.outlook.com (10.167.242.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.22 via Frontend Transport; Mon, 9 Oct 2023 11:24:59 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:43 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:43 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Mon, 9 Oct 2023 04:24:40 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v3 07/16] vdpa/mlx5: Take cvq iotlb lock during refresh Date: Mon, 9 Oct 2023 14:23:52 +0300 Message-ID: <20231009112401.1060447-8-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009112401.1060447-1-dtatulea@nvidia.com> References: <20231009112401.1060447-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000ECD5:EE_|PH0PR12MB7887:EE_ X-MS-Office365-Filtering-Correlation-Id: df07ddcd-29a2-4475-2fd4-08dbc8ba5fa6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: E6pNu0ufCgR5HLXjXutWH4pe+K8PpVLKlbzyh54pM8878HddeSxfdOTqzNV+nefhz5CtLrhzxUx0PEVIsvtWW2dxHaxwt4r5+hGpT9bHrbDipx0kxZwoMF7oTSZ2ruScJQ9mefReN+sOE4ZZKYDGZzrkJKMstKlJjBGwnv7Ao+OLF8fHXsd0pJpY2lAH1/3p5oRbH6TISgFhAxG9gJz5z7THUNjvuIBkqu2PG+KDtLCtm789BquLH9fkped+rwn/tbC57XkXiZAnuUPbR1OGoakCuF70AuNFiacCblskNJIj1Bi/hwU/edzEVhTCzkyIm1JVHJvMd5iCdO3OGXs6RLwlDqmzUajFlzth4zsvJEDXP8S0OY3/uD3LjsN4BdgDXNAq9mSoZqbFR7P7FNBxOpF9xT8HOdc4Ox0rvkwuUPDfBFydEBxIykhGQFFvKFNBNDQHZE03zqjlYajYPmyGIWQelJbv9nUWvEAhvYJREUFHBdRPkW/ULUPq4r8TBOgSVg45x75Q3G++ymH+xgCWKk4Ew4lbir6V5fGDhsFNsdVIIzk7xiP21mswRCB/TaQpmJ4CKJyeb1o1bLBq+TFuLbbK3TlJVYdQ+jtRJPPpXKoY3hRrLF89aXRaOWfasl26A+knSDfjeExepZ+la/O/A6b3PMG4JXbdO6AUr/EwLR4Id4au7gOkNxyUXxGPc77YmF4vTUCLsEWSRIq9HlpfLlAjOn52CuX7d7zEj5Zy8pTc/CFJ1iTV61BRu7FWkpJ7n+AIOK6fpaBjwyyU/gp4u+YBI8m3Gow64odmZHc5XWo= X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(136003)(396003)(376002)(346002)(230922051799003)(451199024)(64100799003)(1800799009)(186009)(82310400011)(40470700004)(36840700001)(46966006)(1076003)(478600001)(966005)(2616005)(66574015)(2906002)(47076005)(336012)(426003)(26005)(83380400001)(5660300002)(110136005)(54906003)(70206006)(8936002)(4326008)(8676002)(41300700001)(70586007)(316002)(40480700001)(82740400003)(356005)(7636003)(36860700001)(36756003)(40460700003)(86362001)(6666004);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2023 11:24:59.5224 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: df07ddcd-29a2-4475-2fd4-08dbc8ba5fa6 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000ECD5.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB7887 X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 09 Oct 2023 04:26:09 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779277030559356846 X-GMAIL-MSGID: 1779277030559356846 The reslock is taken while refresh is called but iommu_lock is more specific to this resource. So take the iommu_lock during cvq iotlb refresh. Based on Eugenio's patch [0]. [0] https://lore.kernel.org/lkml/20230112142218.725622-4-eperezma@redhat.com/ Acked-by: Jason Wang Suggested-by: Eugenio Pérez Reviewed-by: Eugenio Pérez Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/core/mr.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index fcb6ae32e9ed..587300e7c18e 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -590,11 +590,19 @@ int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid) { + int err; + if (mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] != asid) return 0; + spin_lock(&mvdev->cvq.iommu_lock); + prune_iotlb(mvdev); - return dup_iotlb(mvdev, iotlb); + err = dup_iotlb(mvdev, iotlb); + + spin_unlock(&mvdev->cvq.iommu_lock); + + return err; } int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvdev) From patchwork Mon Oct 9 11:23:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 149968 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp1799123vqo; Mon, 9 Oct 2023 04:27:51 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF/5JY5rZSKglI9L+UT3RV3s2uwxPK9pRA/+9ku01NyBLGBMsLLgsM6d81mgZSVLcXi4T4A X-Received: by 2002:a17:90b:1e51:b0:26b:49de:13bd with SMTP id pi17-20020a17090b1e5100b0026b49de13bdmr11802582pjb.36.1696850871217; Mon, 09 Oct 2023 04:27:51 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696850871; cv=pass; d=google.com; s=arc-20160816; b=hYUAA4HEYn5kELkOc2t+1sde7FVsyT35qEPPGJS+wvLvRdqXFEK2Adq23K8jhd8ljx YbHQN3jMUnzv2QqDnW7AJOhoDWkpLtUu3+JK3c2mrzrhZpNgYS1hXJkCTpWOdY1X2q9y h+Uil2SBSax1zgKwk4ZQMk+4+GGVGdoIGezCRyalm6qI7mbloK8YfB5Ve8qC92vo47fA 3fUOudAwrFBIhPwn2Jxo9SSHE/+QQ1RDgXmyunRpxYLlMp8I9ORm8CGrRvPUh7j7KO6z NYrG3VaseQ/8+zkh+6gpKFtOS5JTk0Vmr1hujDYtu8Y7CggatydCqKHgpVQ1lWJi67mH UVcg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=elzKdMTRh3uezwMcgzMejLd/DLpkMeh3g8HOvOAJXm0=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=GjI0ruzgwTcTaRkwMyPmbrpqM9ws6PZqdkE+8hDh6DW8DfGBpJ+D6/7kzcKFU3L4LV zwbWYCT30rrbUQKJqUkaitJVSHT/2vs76T/T+bCRjfKiAlupxfXqFTF2ImY9SfY1C/Y0 tHIhOEnt58tsAHS5bvifodjMl9AlPx8doGLQ2MWE5nVQIFsHJAhu2Gqau2oAh2ZGG5cO AYEq6twa+s/g05ZkErRWXi2iOX3PYa/wggsPSehq0lVSDRj9hxheT6zPzvAzmIeR6dFA 4tSdnqlw0EAozIwW4ntmojPiwJZH2Y/LMG6bI8NHoG7juhQMK1/YatAS67vG7gsZOWFC 3xmw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="SKc/HR0s"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id d4-20020a17090a2a4400b002635d3815adsi11118005pjg.74.2023.10.09.04.27.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 04:27:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="SKc/HR0s"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 8499F80D789D; Mon, 9 Oct 2023 04:27:37 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346281AbjJILZ0 (ORCPT + 18 others); Mon, 9 Oct 2023 07:25:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52024 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346298AbjJILZP (ORCPT ); Mon, 9 Oct 2023 07:25:15 -0400 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2042.outbound.protection.outlook.com [40.107.243.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E46D0D6; Mon, 9 Oct 2023 04:25:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CwRzzl9e3x/DsEkNLZsnsg0ch8BoX/Xkq5LZ+2Yf9A6IRaiT7MazD1WyRLPD4fEITHvl5NwBRF1LzdeDlQhCV8OWYhq24Sff4E0gjq5wu8GIFnmZE9rYMwTVM2p0xhc4mXhi4Gzb1CcY4KxOLxJMGBWBoqr7b06PHhVC7EAUVa86J0SfHmG5T92rR5c9iyO6D/ah0hQCYfz939zoUrM5Gwf7FH0aZbEEo0WhHVQuLmpUr/oxUBga2EkBd/PFN+kBiLyrEjOczl0xBQzGAhAlahQAzgn61k+DgTMU81y+liugAgsBmgSr0SbMWFJo4/uOhRr3En9CvMKj4elkiYBPKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=elzKdMTRh3uezwMcgzMejLd/DLpkMeh3g8HOvOAJXm0=; b=CADQCMUvdAHA6k4i+WBpyBb3HNUjtoUUH+VcgT9nrSUkzmh6bRsbrxZg9+K765nwrVekj2J5Gy0nlr8PHSyo1WFm8JwWGlZHi1QnjCvktgvxTg2oMhMTsbjfLkNc08sadZmZ6f+YTvOoI08E/6U/MzbwhGr1XeL8ucu8RI6/C+QSyyY503+hcMdM+iGGkgT7awUQ0GOZJz/zUJbOMkvVc7kdVvpVrAwCgS7TblfdkI7O2Aapq1tQ1KcuM4tm4k9vX1zLlllVA8L9apDScW5A2Qjj7rXOUg45HS+NpecDOsWMsDIQN6tDWqfQS09CPiXUcwSwUu0p/ivCH5EwzUjbIA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=elzKdMTRh3uezwMcgzMejLd/DLpkMeh3g8HOvOAJXm0=; b=SKc/HR0sXMjPZysXJmI1igcxXatIknchiKwM1tdoApypoZDT+uKDvjw9hTdPsVfZRXo7OTqlDfHatmEspB6iU8cc4xI+Yk4hnF2YcYaq6sfvEVS1baN40zV78Gks2iDL4jZGxc8nX0fLd16Vxt4dEYm2CCA/rg6ctiArzOElWXQaIjWhuVCp4QcbwMFxspRjnaX5Af3xVjdH+xe/31R4h+8WfPoVvpWBsTjE0AKI/Zjy7nzDnMmWY4pRA/LvNPRc87/wf6Ix9fz3+gTb7scoxCdZPmqRPi6Q4PON0VQTjsQCg/pkzciaKOJ4fTXbtoTdfyK82dfxpaOxK7wD/SI3Mw== Received: from PH8PR20CA0012.namprd20.prod.outlook.com (2603:10b6:510:23c::10) by DM6PR12MB4975.namprd12.prod.outlook.com (2603:10b6:5:1bd::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.38; Mon, 9 Oct 2023 11:25:05 +0000 Received: from SN1PEPF0002BA4B.namprd03.prod.outlook.com (2603:10b6:510:23c:cafe::29) by PH8PR20CA0012.outlook.office365.com (2603:10b6:510:23c::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.33 via Frontend Transport; Mon, 9 Oct 2023 11:25:03 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SN1PEPF0002BA4B.mail.protection.outlook.com (10.167.242.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.14 via Frontend Transport; Mon, 9 Oct 2023 11:25:02 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:47 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:47 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Mon, 9 Oct 2023 04:24:44 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v3 08/16] vdpa/mlx5: Collapse "dvq" mr add/delete functions Date: Mon, 9 Oct 2023 14:23:53 +0300 Message-ID: <20231009112401.1060447-9-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009112401.1060447-1-dtatulea@nvidia.com> References: <20231009112401.1060447-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002BA4B:EE_|DM6PR12MB4975:EE_ X-MS-Office365-Filtering-Correlation-Id: d6397982-5a08-453b-e820-08dbc8ba61ae X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: V2OOseJi8bJ46Ttdkw32Pd8G39naxrrlMCs3lb+5AOnx3J+fVVtyov1gP1lajmWz0HaEYKp6qMylIlLa8RK4uOOayoAqUiOIgG9SX1VrjP8yrPtRI8mMOtxuhxsLrYn6CEvp+Ar4vzFOo8kuKvee8b+dy4vKdoExaZ07XMT6trm2yPr4Y9XCjMi9m9DkRXRT+lJCPjT3VyGhFsvCdGKFq56E3HFLAc37RScBR5kkSUeTdFeHIltZXQUEi14SrtEW15e3TkNY9XFIVG8q6ptaeYP04Yznnu5+plTukGUtlkqt3dmJ0pvCRbIbxuc/ZH8CGLiSSVhpl6J7suPjCX448p+DV1eB5ya9Fzeq6ldblEzSDnvWdLwzLZ11b1ODZEQSg0C4foOaalAa9w2MbjC3Ay4fLZMtUFafsDP3FTc2cVcps70FG8RuKpjEceDYD0ePZauJiGL/ynfS6KraOaCOtSeVRBQADudk+vVwC/IzikQi6f9Bi3+agBMQ1uM2b2RDk+ZPgNDBd/MEV2VWGBbSJ3LZzvkk4plE//IN8lJ6L4XRnVLbBEWMA9QQHMO7ABpa7BbLa/VObm+tVbTVthGblfkUx9qXOzjf1ndK5QNq4Poa6JF/L+T6Ar1aXtStVmCtE+GCm5xsokgpoQaAPJTiIF90S0TjZGqQtQQPn2fe5EuseR6HnG6N4NbOJb1WDfWN9WLO8k1hCcTdwa7bxA0+S4J1c/En/77lJlZy8npS0j3DPpYQI6P/UL1rLtuRU/2x X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(39860400002)(346002)(396003)(376002)(230922051799003)(451199024)(64100799003)(82310400011)(186009)(1800799009)(46966006)(40470700004)(36840700001)(40480700001)(40460700003)(83380400001)(1076003)(2616005)(336012)(426003)(26005)(47076005)(66574015)(36860700001)(54906003)(70206006)(70586007)(110136005)(316002)(8936002)(8676002)(4326008)(5660300002)(41300700001)(6666004)(2906002)(478600001)(82740400003)(36756003)(356005)(7636003)(86362001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2023 11:25:02.9975 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d6397982-5a08-453b-e820-08dbc8ba61ae X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002BA4B.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4975 X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Mon, 09 Oct 2023 04:27:37 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779277098919238663 X-GMAIL-MSGID: 1779277098919238663 Now that the cvq code is out of mlx5_vdpa_create/destroy_mr, the "dvq" functions can be folded into their callers. Having "dvq" in the naming will no longer be accurate in the downstream patches. Acked-by: Jason Wang Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/core/mr.c | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 587300e7c18e..fde00497f4ad 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -489,7 +489,7 @@ static void destroy_user_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr } } -static void _mlx5_vdpa_destroy_dvq_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) +static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) { struct mlx5_vdpa_mr *mr = &mvdev->mr; @@ -513,7 +513,7 @@ void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid) mutex_lock(&mr->mkey_mtx); - _mlx5_vdpa_destroy_dvq_mr(mvdev, asid); + _mlx5_vdpa_destroy_mr(mvdev, asid); mutex_unlock(&mr->mkey_mtx); } @@ -524,9 +524,9 @@ void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev) prune_iotlb(mvdev); } -static int _mlx5_vdpa_create_dvq_mr(struct mlx5_vdpa_dev *mvdev, - struct vhost_iotlb *iotlb, - unsigned int asid) +static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, + struct vhost_iotlb *iotlb, + unsigned int asid) { struct mlx5_vdpa_mr *mr = &mvdev->mr; int err; @@ -550,12 +550,6 @@ static int _mlx5_vdpa_create_dvq_mr(struct mlx5_vdpa_dev *mvdev, return 0; } -static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, - struct vhost_iotlb *iotlb, unsigned int asid) -{ - return _mlx5_vdpa_create_dvq_mr(mvdev, iotlb, asid); -} - int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid) { From patchwork Mon Oct 9 11:23:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 149957 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp1798437vqo; Mon, 9 Oct 2023 04:26:06 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEu6VFQBIyW+LZPmaQAHx3pM6R3DaBzRcPiopgu5Qmu471zNAXy2/aTJdeOBJBrlELuD9uB X-Received: by 2002:a17:902:680e:b0:1bb:fcb9:f85 with SMTP id h14-20020a170902680e00b001bbfcb90f85mr11991058plk.32.1696850766370; Mon, 09 Oct 2023 04:26:06 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696850766; cv=pass; d=google.com; s=arc-20160816; b=kFSPCgGxyKrDWLxcwSBvoJCA32l5YYNadiuaUS4OCpcjdn9ovD2ebYxJfirliGYBIM rK70J9T45TnnJ9TaYI+VHC5qzJnLspiPD93VY/gF+wb1NPIKa1sx5ZlIjacoCF+FYcGd mcyam51GvjeS2rbmZ3wfY3E5Kc1spX5TnhOdWFbd9EWvKkqtFEIl8QMSHID0zng9so4C 5BAa6zI5PHn3VS/N1OCor/AXJsARPB6Ot+ktdI6zOeCZu63JFK6GLMTbyQfNK7Omb06l SnyMhLhldXwcbhLnhvw+QI/Xgg1YWrUbshFmP8HqNsLD38rum6soLcPUU66NmBc7RQIE Ezcg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XmILL87kEXieQsFyA46Z/A2SnEkDR6phjuffw0L8hYk=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=MH9NuPvHZtYUfvZIdqz374kKMNp7W0+2LxXPaxeWNV+Wsjc12yFSQEYuBK9wRqBz7k Pf/eEL1L7u7wzgUrtQ1xZ3+zNbp/C5VJJhK1nXWtk3csANDEAxYRZe65ptHgwluDCR2X fjVaA7MmrJEwL/YU1Z1lU/brWnkKrmz1CPsTXewSsIWJADUDn9lovIICWrnZjqY+21lK Q5Zdy0LcI8KLogb0nIiY0KTpAwMa22k3T1M7t9mLZ42puj0RUjtt3BiHF/ksMkyDYYjO 6dzz06M91j4CPegf06Enh6mgNUKiCBaZyYT6XscxDDOP4ZAUjI0ibHoLx9NWvcfLAS87 Jy7A== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=nPe5FB2u; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id v1-20020a170902b7c100b001c747cafce9si9239447plz.231.2023.10.09.04.26.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 04:26:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=nPe5FB2u; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id E04CA8093425; Mon, 9 Oct 2023 04:26:03 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346384AbjJILZi (ORCPT + 18 others); Mon, 9 Oct 2023 07:25:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346343AbjJILZQ (ORCPT ); Mon, 9 Oct 2023 07:25:16 -0400 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2056.outbound.protection.outlook.com [40.107.100.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BB6B4D41; Mon, 9 Oct 2023 04:25:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=km/uYejK0JdKLY0ZOuEP1RYuptg+IJBkcpwURBIloic9xv8tD+sYw8TfXltkMCqIacPBv2l8P050caeDx3ssEjT8264AXEeCWMUAP6KNOW8LvQ/2xngjGBOf2Jay20YozAR50YVFdgP848phdqITDJqNpewiiqitWFSvcMsCTbGWoA5+VoQVI7KQnq5re2lbuFrjjbOK9+e6hJXqvEantg5t/4CsEpSORQLd9tKsT+BsaYceZtIeWaofqcCMMpth8tHSwRjPanYPJGYlztc/EVxJcJHIncsu1Fe2yxQ7aat0FkIysSc7zmDjP5yHK6uiFCczoRVVsDZdgq1lQe4moA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XmILL87kEXieQsFyA46Z/A2SnEkDR6phjuffw0L8hYk=; b=PPoEK96kq5EmT0rXh/JqojfaUFGdQ+qBkACjr6oycAFamAwoxAOuEajVNQUs6deElb1GcmIC0nAta3DUQauowbBID72AQnrCUUHhRjqWJawGQFumf7IF3z0QS3FR+gIdQTEwVZTH5dVxnNSr7kGMtM5QkJbd4/JHGvSolpH6ekmiISDNoiyo1moX+Xv5qDyuBTQQqTGefbiKLPoHybApUB6KGhIQs8TNzlKUHM/n/LVyNb8k3OzDwCFOYrbp0FezJvgKuvfVCqlTCPxA8Ln6TFz0XT6biqlNX2XdtlnT9zXep3WEKIMKXRvuGpNpboy1b/3zClyGug4DUheYVN3tIQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XmILL87kEXieQsFyA46Z/A2SnEkDR6phjuffw0L8hYk=; b=nPe5FB2uMbRG/Y45h2BHZgVn7z3Z8COr0xlo2GKNw8VXXYwgMEg47iTq2Ja1dsBvG7dSoZpKCvPmYpOBSJluq0gBBBJaSRgSXz24RhNq32w83jhFDGT9B78NpqanWDNSFGAsF5iZjGa8bqglwzhDL3wJvsVNTfGapB7jAWqlLP3NvJYDpx2TsobvTikRhZlem+sqkMVJaITBQAz0EPPylLvITd3NQGN1u/XZ0/5sOIsmtAH4dlBiSd1OAu70dIq6PPveRUu43G5E68WCcA2/jQwGq0KIYZ/hC4n+5QeBVHjk6dDpGWcvJw8d9Dsz4UusRYD+Tk6EeZWgE/wnm/v67w== Received: from BL1PR13CA0100.namprd13.prod.outlook.com (2603:10b6:208:2b9::15) by PH8PR12MB8607.namprd12.prod.outlook.com (2603:10b6:510:1cf::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.36; Mon, 9 Oct 2023 11:25:04 +0000 Received: from MN1PEPF0000ECD9.namprd02.prod.outlook.com (2603:10b6:208:2b9:cafe::6e) by BL1PR13CA0100.outlook.office365.com (2603:10b6:208:2b9::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.21 via Frontend Transport; Mon, 9 Oct 2023 11:25:04 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MN1PEPF0000ECD9.mail.protection.outlook.com (10.167.242.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.14 via Frontend Transport; Mon, 9 Oct 2023 11:25:04 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:51 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:50 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Mon, 9 Oct 2023 04:24:47 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v3 09/16] vdpa/mlx5: Rename mr destroy functions Date: Mon, 9 Oct 2023 14:23:54 +0300 Message-ID: <20231009112401.1060447-10-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009112401.1060447-1-dtatulea@nvidia.com> References: <20231009112401.1060447-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000ECD9:EE_|PH8PR12MB8607:EE_ X-MS-Office365-Filtering-Correlation-Id: 2300e617-f3c9-4e6c-c03c-08dbc8ba627c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: C/9jXqU6tSJpGrYmBXdPOuQ4qTbWe6702nXYaU2k7+Lbgk1kmSVwXWH+jI2nJ+QdaFf+iR7bDaTn5nsmz+a3PQoy5h+gxgZLvGJdk/5HpYDmWFUSQvn8xe1sNUIN6ySl86id6vd7HS5b/LKbD4RUc1vu7EXliAYouWKXzjQQvReozF1G2+tHfRvO6ffi+iFMOsXiY4ptbDsqufqAb4M7P83syUeGHr989s2rnkj3h8jKuQceFItSUWfXNRqOeuMD6diakN3TYmLHqZwMtlGB84vdtJPIGIXkLUUb1p0UKczm4D8+k2wEAABh+epVHTc3onEdyRtLqSDcJKYGHEKDthmnfN0tN9nqmC9n6a9+ySMH/9w6iFWxupPOzJy9/NymrYxwG0CAgnZ5LPOUqwXQzx9xguhBJ2q3E/B2KT8r8a7JvXxwhkUJc0CMsneWEBOaEsm1z+ZooANvtgwb6mx4Jhh7WHXXuJsh8z3rxTiNcIGOIYrc4qwol4/HWrf8kVbHWgtGC4fawQ75qHBn4Wh2XVl0E5W6mlWHf0HvFImqlB5Gcu/vawE8jJEAErTTKcdzSRFLluQ8fUCKA2LP5Qs9vkK8DRTzAHQ9biV5ZncrLggnsfF93ZqVouRLWtgMXmygPff4SrMupAH9RA3c2BnmjMZCtvOeWoCLNOPQ7KgIHPqRNETgtaJf0yMop2hn+HRsyIRQkZ2aNuoqM2kyKv+B1ZY4XVdAhXVeMrYrFpVXZ+q+cSZdobd0AqgNINgQx9dO X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(376002)(136003)(346002)(396003)(230922051799003)(82310400011)(451199024)(186009)(1800799009)(64100799003)(36840700001)(46966006)(40470700004)(2616005)(1076003)(478600001)(41300700001)(66574015)(336012)(426003)(47076005)(26005)(83380400001)(2906002)(5660300002)(110136005)(70586007)(70206006)(54906003)(316002)(4326008)(8676002)(8936002)(40460700003)(82740400003)(36860700001)(356005)(36756003)(40480700001)(7636003)(86362001)(6666004);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2023 11:25:04.2997 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2300e617-f3c9-4e6c-c03c-08dbc8ba627c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000ECD9.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB8607 X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 09 Oct 2023 04:26:03 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779276989375407721 X-GMAIL-MSGID: 1779276989375407721 Make mlx5_destroy_mr symmetric to mlx5_create_mr. Acked-by: Jason Wang Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 4 ++-- drivers/vdpa/mlx5/core/mr.c | 6 +++--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 12 ++++++------ 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 554899a80241..e1e6e7aba50e 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -118,8 +118,8 @@ int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *io bool *change_map, unsigned int asid); int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid); -void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev); -void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid); +void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev); +void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid); int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid); diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index fde00497f4ad..00dcce190a1f 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -507,7 +507,7 @@ static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid mr->initialized = false; } -void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid) +void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) { struct mlx5_vdpa_mr *mr = &mvdev->mr; @@ -518,9 +518,9 @@ void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid) mutex_unlock(&mr->mkey_mtx); } -void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev) +void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) { - mlx5_vdpa_destroy_mr_asid(mvdev, mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]); + mlx5_vdpa_destroy_mr(mvdev, mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]); prune_iotlb(mvdev); } diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index aa4896662699..ab196c43694c 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -2644,7 +2644,7 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, goto err_mr; teardown_driver(ndev); - mlx5_vdpa_destroy_mr_asid(mvdev, asid); + mlx5_vdpa_destroy_mr(mvdev, asid); err = mlx5_vdpa_create_mr(mvdev, iotlb, asid); if (err) goto err_mr; @@ -2660,7 +2660,7 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, return 0; err_setup: - mlx5_vdpa_destroy_mr_asid(mvdev, asid); + mlx5_vdpa_destroy_mr(mvdev, asid); err_mr: return err; } @@ -2797,7 +2797,7 @@ static void mlx5_vdpa_set_status(struct vdpa_device *vdev, u8 status) err_driver: unregister_link_notifier(ndev); err_setup: - mlx5_vdpa_destroy_mr(&ndev->mvdev); + mlx5_vdpa_destroy_mr_resources(&ndev->mvdev); ndev->mvdev.status |= VIRTIO_CONFIG_S_FAILED; err_clear: up_write(&ndev->reslock); @@ -2824,7 +2824,7 @@ static int mlx5_vdpa_reset(struct vdpa_device *vdev) unregister_link_notifier(ndev); teardown_driver(ndev); clear_vqs_ready(ndev); - mlx5_vdpa_destroy_mr(&ndev->mvdev); + mlx5_vdpa_destroy_mr_resources(&ndev->mvdev); ndev->mvdev.status = 0; ndev->mvdev.suspended = false; ndev->cur_num_vqs = 0; @@ -2944,7 +2944,7 @@ static void mlx5_vdpa_free(struct vdpa_device *vdev) ndev = to_mlx5_vdpa_ndev(mvdev); free_resources(ndev); - mlx5_vdpa_destroy_mr(mvdev); + mlx5_vdpa_destroy_mr_resources(mvdev); if (!is_zero_ether_addr(ndev->config.mac)) { pfmdev = pci_get_drvdata(pci_physfn(mvdev->mdev->pdev)); mlx5_mpfs_del_mac(pfmdev, ndev->config.mac); @@ -3474,7 +3474,7 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name, err_res2: free_resources(ndev); err_mr: - mlx5_vdpa_destroy_mr(mvdev); + mlx5_vdpa_destroy_mr_resources(mvdev); err_res: mlx5_vdpa_free_resources(&ndev->mvdev); err_mpfs: From patchwork Mon Oct 9 11:23:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 149965 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp1798830vqo; Mon, 9 Oct 2023 04:27:06 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHxMHJrUhshWa5bDshDmV5ccgM7sCIMLKQNtnV7xn0N1Ks1Z3Ea2OCW1w4YYkTf/MhQqQoS X-Received: by 2002:a05:6a20:9188:b0:140:f6c4:aa71 with SMTP id v8-20020a056a20918800b00140f6c4aa71mr15285892pzd.8.1696850826187; Mon, 09 Oct 2023 04:27:06 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696850826; cv=pass; d=google.com; s=arc-20160816; b=Fk4pfNh19wJFjRvHaO8P3PNlpcCQH4KSCy5CrPTrleYVj1vTKldRXQC/9a1hHczEbO wrETOcCM76J5CnWztOBtQRgYC30DWCKMMekHvqgLHXoECj2EvWu4Ao58fMP0vr81eT1B 4LYjsHjfZaJuIUl7g4N7O5mE7G6Sha2esWrrJS/eQSB4fFd1t8u7QXjpiiEgnuCH1qA8 ufYHWh7d3EVQ+yp+nVJEaATh/Ut3VgF4EornhAlqQxT0MwvET4jTRJrPmHEpCAZTqbic GL2rFX4VpX5zUVVfnF4PuZ5RcitDU4kKm7L34LQMRTTwtLiK1G7TXGR8+GTqThfL8J+K 5BHA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9ARPpe47gz+MA73Ix9lUbLmF/6h5J92rUxRFMm3wXHY=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=UhItCKkPB2NIQTglEZuKh1T5b/97gjcjkfpQrqZQOx9S3vYpFhN9RqRQRFNX9bWg3m nsabNRiywNXYAPQH53eNFZEX1v3wmIsuXt8Ytrwomm7rjXS8igdCX3d9uNfl0LuOxMho kyh6O5CU8jf8rR2NimRde97kpRDxj9+F06XAzMxtT5wpf66evCErRd83SOV7LpYt02/l nLFyYs27ajoIkG9vWg8JZuQTIPNCutMlOEHLO8TFdLT41G3VL7NsQzfhc1BGLu9KJIuz Rt9ireYSoKnE+lcAEaDwUL+I0rpWdz19iivU1nU3hP3+mA97xWyk/zG6CKcxk7tC3Szg GeVg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=tg1Mf7Ln; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id h7-20020a636c07000000b00589878c0a65si7704643pgc.735.2023.10.09.04.27.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 04:27:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=tg1Mf7Ln; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 7F6DA8088693; Mon, 9 Oct 2023 04:26:08 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346320AbjJILZn (ORCPT + 18 others); Mon, 9 Oct 2023 07:25:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346311AbjJILZT (ORCPT ); Mon, 9 Oct 2023 07:25:19 -0400 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2082.outbound.protection.outlook.com [40.107.244.82]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 614009D; Mon, 9 Oct 2023 04:25:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gyp5TFienEIZqvGWFen12OZuinkxfhojYBRmaa8/0DPe//9GCyHHOjb36qGAZBN9Zo/7yFm6wFzP3M6TPYaQeXWhz63pKzw1iCTzuCL5UZIt04ToKp1AWTzNKBfKpvQoeYY7Ag6PsZGoHPfdqqrb9sKItt0qy0G30vKWbyrv0VQWMaZ0Hs5mdpEwGarFEqvFF+c4I6KH2UMGm9xASati7HMgyutTlBvLFfsEQvZ/H6WumjYDmmDAQnW/AbxA2URH/xLGS9LImOBcknHx7mhdEaIXyts7rAS0M61ENSShBu9yQqM1pAQpLfphyIjQuXsCQKaIhZyqTsFMk48tcJ7+tg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9ARPpe47gz+MA73Ix9lUbLmF/6h5J92rUxRFMm3wXHY=; b=UZRJrijkEOmYcqDuyHVWkDCm/riTTFu6tvESkCYbef5kw9Drdi0KkLl0Miiq0VOVAIKMBPmyWvw4KPsP8QvD91Rw5bHt/DxF6ePAwzYpYCZ1oRuV9BFxxrg6AOKsz7S+Tho2TqpSfr2P4ZswBxUwl2OEAWG3KhzhfvE73PUJN3GjCIYNZd8rZHOglwLAb7dBn9xzjyFiy6vmT/NmsSUqeTQlNhu0CRXuYX8n1jYW7cydtLsxGNBXnHEezSktrrTA0RgD3b5vMjRmoGM2LMqYz8aXaPNoBmEpyCDYBsHde9JpnBg5Q/IumcNTblAmSyLCsre/vG1VwKXrxfcOE7RiIg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9ARPpe47gz+MA73Ix9lUbLmF/6h5J92rUxRFMm3wXHY=; b=tg1Mf7LnrCcUXBAu3lg51wxvMZckyUe5eF/GemL/r++l3g9eu7QnWD7WU1j8GpBNj5g4LVaJ4Eq8f+wGlnPMsAqslsOPF0os7AUKoqLBOje9LlEIlGY0/mKorbsb5PsX8wXe54gdEBFW4jMYEC4kG2hSo4MmLshGaZTyu+yQvmeYiOU8NVtdHT4VBmzqdpQDtbrmcK5esT3CptlWTbOXXcfYe5Ek5sqQXGKc3oNDVU/+ZCBNZ91yQnVwjs5oLjI0by6vfmeWlfapLWLtLk8YSzsSxyLLQjzhs8+VFHZpEPZiXRF5DbxAEDff8+bP/Q+3AyU9wGgczyEyNr7aDYJTlA== Received: from SA9P223CA0011.NAMP223.PROD.OUTLOOK.COM (2603:10b6:806:26::16) by CH3PR12MB9250.namprd12.prod.outlook.com (2603:10b6:610:1ae::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.36; Mon, 9 Oct 2023 11:25:08 +0000 Received: from SN1PEPF0002BA4D.namprd03.prod.outlook.com (2603:10b6:806:26:cafe::69) by SA9P223CA0011.outlook.office365.com (2603:10b6:806:26::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.37 via Frontend Transport; Mon, 9 Oct 2023 11:25:07 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SN1PEPF0002BA4D.mail.protection.outlook.com (10.167.242.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.14 via Frontend Transport; Mon, 9 Oct 2023 11:25:07 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:54 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:54 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Mon, 9 Oct 2023 04:24:51 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v3 10/16] vdpa/mlx5: Allow creation/deletion of any given mr struct Date: Mon, 9 Oct 2023 14:23:55 +0300 Message-ID: <20231009112401.1060447-11-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009112401.1060447-1-dtatulea@nvidia.com> References: <20231009112401.1060447-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002BA4D:EE_|CH3PR12MB9250:EE_ X-MS-Office365-Filtering-Correlation-Id: 8c70b86e-1fbb-4796-78c2-08dbc8ba6485 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Cao5gRHOxoh98YTvA4BlZno6sG9qTIB+Ue6j09rpNtSgXfXfaeckcuEGIkhjdfcfKuDJxRFg8I7tTVx1CadBvbPRkBUglqfCicXT9hXT75k0PZXheWns506Jqqj4+NATT5K8Hp3WylQhUasRMkB809RnBby6IYckGtwzaZAc/j7v760qLXW7iHGnjsokQetcp/csT6yBV2oAKAldvrzHIzZrOqZLFLuUdRRn6KEx2OsT8W+giJ29R0HYPnn389G9VaiPCUB2oHpPrNILYD0JCkh6NBol89PFBjSowcr8paAdIKr+JFra4aXmEwX8WlBM5a6shZ617djfi3x2bg4eaJMs9YEvJXJoMnIN3NKq1/qz25revD97Rum3/8Sd6Jym3gilB1YlZz2mFgPL5WoYEuEiKJ8MHANzuAPUm5RjSr+w5/p18srprUjSDgx5mI9QYY8pKhPYpFnRzKSlWb8YZcy0v5i1REe6TZVLh4TZw2jvimFHnpA45YpkGLYscJLwcfeiQPzFtkwJwmJ3RUrDvuw0kgABib0+vGdY5kJqTVyCqTPmusRekc5rh2p7xjO01vGMeZhx6JBJFhLIp2dsMMzcPrR0rPuAHYVhZp+RwHo2stdJcq4rWPsNXl9QCro7mEw3JVWR9STt/MN5iBd564kKc4AtNvse0n2/vMVoqod6nfSNyXy301jM87MRzQVeq4Sg4TYAgO+2rm609enub1b1X1nmD+ryTg/SqMsDlssIyc/ZzLXjeZ70+6vONFwC X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(346002)(376002)(136003)(39860400002)(230922051799003)(82310400011)(64100799003)(186009)(1800799009)(451199024)(46966006)(36840700001)(40470700004)(1076003)(2616005)(426003)(336012)(26005)(36860700001)(8676002)(478600001)(66574015)(47076005)(8936002)(83380400001)(2906002)(4326008)(110136005)(54906003)(70206006)(70586007)(41300700001)(316002)(40460700003)(7636003)(356005)(36756003)(82740400003)(40480700001)(86362001)(5660300002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2023 11:25:07.7591 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8c70b86e-1fbb-4796-78c2-08dbc8ba6485 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002BA4D.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB9250 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 09 Oct 2023 04:26:08 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779277051707842551 X-GMAIL-MSGID: 1779277051707842551 This patch adapts the mr creation/deletion code to be able to work with any given mr struct pointer. All the APIs are adapted to take an extra parameter for the mr. mlx5_vdpa_create/delete_mr doesn't need a ASID parameter anymore. The check is done in the caller instead (mlx5_set_map). This change is needed for a followup patch which will introduce an additional mr for the vq descriptor data. Signed-off-by: Dragos Tatulea Acked-by: Eugenio Pérez Acked-by: Jason Wang --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 8 +++-- drivers/vdpa/mlx5/core/mr.c | 53 ++++++++++++++---------------- drivers/vdpa/mlx5/net/mlx5_vnet.c | 10 ++++-- 3 files changed, 36 insertions(+), 35 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index e1e6e7aba50e..01d4ee58ccb1 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -116,10 +116,12 @@ int mlx5_vdpa_create_mkey(struct mlx5_vdpa_dev *mvdev, u32 *mkey, u32 *in, int mlx5_vdpa_destroy_mkey(struct mlx5_vdpa_dev *mvdev, u32 mkey); int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, bool *change_map, unsigned int asid); -int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, - unsigned int asid); +int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr, + struct vhost_iotlb *iotlb); void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev); -void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid); +void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr); int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid); diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 00dcce190a1f..6f29e8eaabb1 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -301,10 +301,13 @@ static void unmap_direct_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_direct sg_free_table(&mr->sg_head); } -static int add_direct_chain(struct mlx5_vdpa_dev *mvdev, u64 start, u64 size, u8 perm, +static int add_direct_chain(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr, + u64 start, + u64 size, + u8 perm, struct vhost_iotlb *iotlb) { - struct mlx5_vdpa_mr *mr = &mvdev->mr; struct mlx5_vdpa_direct_mr *dmr; struct mlx5_vdpa_direct_mr *n; LIST_HEAD(tmp); @@ -354,9 +357,10 @@ static int add_direct_chain(struct mlx5_vdpa_dev *mvdev, u64 start, u64 size, u8 * indirect memory key that provides access to the enitre address space given * by iotlb. */ -static int create_user_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb) +static int create_user_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr, + struct vhost_iotlb *iotlb) { - struct mlx5_vdpa_mr *mr = &mvdev->mr; struct mlx5_vdpa_direct_mr *dmr; struct mlx5_vdpa_direct_mr *n; struct vhost_iotlb_map *map; @@ -384,7 +388,7 @@ static int create_user_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb LOG_MAX_KLM_SIZE); mr->num_klms += nnuls; } - err = add_direct_chain(mvdev, ps, pe - ps, pperm, iotlb); + err = add_direct_chain(mvdev, mr, ps, pe - ps, pperm, iotlb); if (err) goto err_chain; } @@ -393,7 +397,7 @@ static int create_user_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb pperm = map->perm; } } - err = add_direct_chain(mvdev, ps, pe - ps, pperm, iotlb); + err = add_direct_chain(mvdev, mr, ps, pe - ps, pperm, iotlb); if (err) goto err_chain; @@ -489,13 +493,8 @@ static void destroy_user_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr } } -static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) +static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { - struct mlx5_vdpa_mr *mr = &mvdev->mr; - - if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] != asid) - return; - if (!mr->initialized) return; @@ -507,38 +506,33 @@ static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid mr->initialized = false; } -void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) +void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr) { - struct mlx5_vdpa_mr *mr = &mvdev->mr; - mutex_lock(&mr->mkey_mtx); - _mlx5_vdpa_destroy_mr(mvdev, asid); + _mlx5_vdpa_destroy_mr(mvdev, mr); mutex_unlock(&mr->mkey_mtx); } void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) { - mlx5_vdpa_destroy_mr(mvdev, mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]); + mlx5_vdpa_destroy_mr(mvdev, &mvdev->mr); prune_iotlb(mvdev); } static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, - struct vhost_iotlb *iotlb, - unsigned int asid) + struct mlx5_vdpa_mr *mr, + struct vhost_iotlb *iotlb) { - struct mlx5_vdpa_mr *mr = &mvdev->mr; int err; - if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] != asid) - return 0; - if (mr->initialized) return 0; if (iotlb) - err = create_user_mr(mvdev, iotlb); + err = create_user_mr(mvdev, mr, iotlb); else err = create_dma_mr(mvdev, mr); @@ -550,13 +544,14 @@ static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, return 0; } -int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, - unsigned int asid) +int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr, + struct vhost_iotlb *iotlb) { int err; mutex_lock(&mvdev->mr.mkey_mtx); - err = _mlx5_vdpa_create_mr(mvdev, iotlb, asid); + err = _mlx5_vdpa_create_mr(mvdev, mr, iotlb); mutex_unlock(&mvdev->mr.mkey_mtx); return err; } @@ -574,7 +569,7 @@ int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *io *change_map = true; } if (!*change_map) - err = _mlx5_vdpa_create_mr(mvdev, iotlb, asid); + err = _mlx5_vdpa_create_mr(mvdev, mr, iotlb); mutex_unlock(&mr->mkey_mtx); return err; @@ -603,7 +598,7 @@ int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvdev) { int err; - err = mlx5_vdpa_create_mr(mvdev, NULL, 0); + err = mlx5_vdpa_create_mr(mvdev, &mvdev->mr, NULL); if (err) return err; diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index ab196c43694c..256fdd80c321 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -2644,8 +2644,8 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, goto err_mr; teardown_driver(ndev); - mlx5_vdpa_destroy_mr(mvdev, asid); - err = mlx5_vdpa_create_mr(mvdev, iotlb, asid); + mlx5_vdpa_destroy_mr(mvdev, &mvdev->mr); + err = mlx5_vdpa_create_mr(mvdev, &mvdev->mr, iotlb); if (err) goto err_mr; @@ -2660,7 +2660,7 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, return 0; err_setup: - mlx5_vdpa_destroy_mr(mvdev, asid); + mlx5_vdpa_destroy_mr(mvdev, &mvdev->mr); err_mr: return err; } @@ -2878,6 +2878,9 @@ static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, bool change_map; int err; + if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] != asid) + goto end; + err = mlx5_vdpa_handle_set_map(mvdev, iotlb, &change_map, asid); if (err) { mlx5_vdpa_warn(mvdev, "set map failed(%d)\n", err); @@ -2890,6 +2893,7 @@ static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, return err; } +end: return mlx5_vdpa_update_cvq_iotlb(mvdev, iotlb, asid); } From patchwork Mon Oct 9 11:23:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 149958 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp1798595vqo; Mon, 9 Oct 2023 04:26:31 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHo/Bgs7303Ea2a5qFvG6JuJ+BPqQy14dVnmAYWj4/gbnlkDhZfYF0PVk+Djl8ZU47w0BEF X-Received: by 2002:a17:902:c952:b0:1c0:d17a:bfef with SMTP id i18-20020a170902c95200b001c0d17abfefmr14609269pla.30.1696850791213; Mon, 09 Oct 2023 04:26:31 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696850791; cv=pass; d=google.com; s=arc-20160816; b=O9CmmxhgxoIf1pEAz+cXxwcMknwdd/jg1wZM/h4FWdRMYYq14SGZs4YpVfXGtcs2NX yQfWSoyHv/SIXFrZOdgpaVSmkoFQiMrhMON0waUK2wkj5jjhf+KRHsO+Ull1c0KTsjCo y2sxbhk52DLDsRrfUgHQX94ZX+igl+fFd9ej5LSrDcOrOOoNc1Tsf56ngBoIOR5fXmf7 woQU/D7ouqJu3p+b2Twpel748lgsnn5NDfc+3S3CshyN9rlOOOdvSMxD1zxYqQyhaoRi uBaXoHo/JfO+ol6/Y5lSYjyjW8ubEYxguBg0N0c4a8FXJyamfUDz7Qd6BLexsSv1R2d7 m8nQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=wy1kaZQzU+m3/SX/tOV5uK0InINX2soflKNDLMzJf7k=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=SSlTS8Wol62cMS1VqevDnBY6DiQKwzJBRRkb53DT3/Brgl2auxA7+IQT/RmFJVIag1 1U3fGJ7JsFJ3MonLgklVmtPPT7JeTVcO0CeioiAIjw4Wy4QJRRoADTv6Be0dOXWcXOnN slXzyiM38xYByAqyfaN25N12RX2NxYGq9vBx8hTUG5uEy4IexSdyYYUbmBKTZHGy81JL Ss+GHylf5NyI6XMY58SqRnfm5Junw18cMlxwHqFxnjEiU1qAlLIbwXfwZ3V98VEu14Md 4WjI5blZmu1RmGmLJ8nO7YQ0uPx7Yii9VBUoT5QvPuF898uvjoFWpteBIMSvgOkIGu/p NzeQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="Eg//R/Hm"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id g19-20020a170902e39300b001c5eb34763esi8934785ple.199.2023.10.09.04.26.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 04:26:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="Eg//R/Hm"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 12B6880A7CE6; Mon, 9 Oct 2023 04:26:29 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346230AbjJILZv (ORCPT + 18 others); Mon, 9 Oct 2023 07:25:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60252 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346304AbjJILZZ (ORCPT ); Mon, 9 Oct 2023 07:25:25 -0400 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2073.outbound.protection.outlook.com [40.107.237.73]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8683410B; Mon, 9 Oct 2023 04:25:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=S+A2PJFcPjRoJEqL8KdMONbWa9KzJj5sDW5A75269KwKIySoNq1Vi9bnVezRb5JgSIzZ/x9RyGflZKKMONsFXEQjtXIdRGEhBGqsayKj2oLhOafq5wVpexFxr9VdO99A+oCwSaXgGJ4YFCKxYHKy0eE8wXsEDF0YvqSeqXtkxraagOEVH98/yewh/Pf6+Rf+gQpo/S9xXa7AEewBF3okGLsT6W44b9Sz7ZTDWnd0oQdEumvIswkIcnzsiG01SOOZpxpbXrGzCx2z9e0gDp778aV0Z8c/cjRoyrU8iBXOuxHC74WFvLW8iEHRyxS57xN4OqpF05KWHkbWpobvq7NaFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wy1kaZQzU+m3/SX/tOV5uK0InINX2soflKNDLMzJf7k=; b=JWJIPnVbhm/F59/++MNrhM2qj5dwXEZuVim3DiAsFbKspVJuOJQFGapDz7Bb7yAY8GMEBXCa9FkM3ovhE1KjtGf6bHolHgu7ZcDkFAIeegJWvGu5Um1zmJtoP6oSa+8KA2NiaQ2kgyqd7PigW7qqpCMk+t5pTJEoyhpKD9orj+Dr2joSN9Q4DtVvWIpRMfVM/d88ZfyLiP9cAyUnrJzmtfG0lPoU04E2UF4+a48yk8w6l3GTx08liFj7HektxpmB96GSOFmeXUpPMKcmGDLBqXZrZbefkJix0rxM4u8n8t3utVqB3MbqHRPlrr52T2ASGbs5s3LVTGYY7r+C3cZ9yg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wy1kaZQzU+m3/SX/tOV5uK0InINX2soflKNDLMzJf7k=; b=Eg//R/Hm6vAtS62YUqU1L8bsrGGSMqw/k8I4M8UfE/MxKcw6T5KVqvVv8LAj6sD6HL4tvBnBIXhOgixZL54S7cjY5YuA9kEuKLjdzyQdYzhkt/FQ5n4BeUYBCUKT4M4IdNJgxcHJV7BDe7l0SpEzb/ojyYsI2Sy0n5t7ODc2YdVAvIkwzq2Mux+FftNqnEsJLJl4k/CktWKe5HlQMspTCO0/mHPN+g1GKzZat2xVTNQ9sCMbfyWR7czagBF4CM5RWRx67EZxC8lzRWjk9/8CJa8b7YZ+SfEoqiqWElxfIOy+vlLfPFgmuVqdd7of+w/oVWfphxLDxYK8BYicSsnSxQ== Received: from BLAPR03CA0050.namprd03.prod.outlook.com (2603:10b6:208:32d::25) by IA0PR12MB7750.namprd12.prod.outlook.com (2603:10b6:208:431::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.37; Mon, 9 Oct 2023 11:25:14 +0000 Received: from MN1PEPF0000ECD5.namprd02.prod.outlook.com (2603:10b6:208:32d:cafe::fa) by BLAPR03CA0050.outlook.office365.com (2603:10b6:208:32d::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.37 via Frontend Transport; Mon, 9 Oct 2023 11:25:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MN1PEPF0000ECD5.mail.protection.outlook.com (10.167.242.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.22 via Frontend Transport; Mon, 9 Oct 2023 11:25:14 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:58 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:24:57 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Mon, 9 Oct 2023 04:24:54 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v3 11/16] vdpa/mlx5: Move mr mutex out of mr struct Date: Mon, 9 Oct 2023 14:23:56 +0300 Message-ID: <20231009112401.1060447-12-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009112401.1060447-1-dtatulea@nvidia.com> References: <20231009112401.1060447-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000ECD5:EE_|IA0PR12MB7750:EE_ X-MS-Office365-Filtering-Correlation-Id: 1037588c-8da1-43b0-564d-08dbc8ba6886 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: WiMISNbywyjp7ySCHohJd641a1v0sXTZ7inHtLvE+SSsIkyVaNtoAp3zS5V6HIvwN1GHA7FPQGhr5a1AOzpUkAD7DimohuDeRcr/IH0+5GqMA6pnWvu33cf9zARYYRjNlFKvtC+Wh5KducIdTmy8P2JDA5MtxL3l8gsl1S7itkh96QyeC+NiATPhkBxYqGgfVRiaD6ZAOw40p2DIo4F64pmwrq0+gVrJGVxBqTif9YFfyC4BFntEkusyjn2NQ9qes1NzMzXCt2yDyA1smkGWC2GRrPl7jXU3guE5BiyDllOGfVzonNi2QoBvnC9u5c7RIssL9FV4OKbiWcWYGFcud4Vdlx1sO58JsyASae7F1tnvKxwkIdF/cHnJlFjGZylk4iNsoltknRPQG1ljlZM4JlJ9LbBdMzzSzoPBpmBiLEt6+/D8QULdcTCK/4iQgSECpf6exjWznso1V3LQtRB2eEcDAsuDKeJKxeerlAY0pXs4CO/hMkfPmNTZOwYF6FnHo4PrjLs9FAQEJ7YrdC4Asm16ePWnzIDnZgiJuTmGs9pY8wKQoP08kYEftWfhYnNl88Jp3uSF2hzO53Y4b5eYngCsm9PxqpXGlODzY8DcAvGR9f7iSbTWWQ0ZMPWi2BzW3BYcqSmzgTUYCvLWYcWucXKqmu36rTMfZHojJC0zzLqmY19oIQGL6+SInn2pilqH/xMs4AwRzJI8h3Xdd178n/fADuRBEWGnDZYVBruhIzupfA7ehkhjKNjvReygxf9Y X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(346002)(136003)(396003)(39860400002)(376002)(230922051799003)(186009)(64100799003)(451199024)(1800799009)(82310400011)(46966006)(36840700001)(40470700004)(86362001)(7636003)(356005)(36756003)(66899024)(40480700001)(82740400003)(2906002)(478600001)(41300700001)(8936002)(5660300002)(4326008)(8676002)(1076003)(83380400001)(426003)(336012)(2616005)(40460700003)(316002)(36860700001)(70586007)(70206006)(54906003)(110136005)(26005)(47076005)(66574015);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2023 11:25:14.4132 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1037588c-8da1-43b0-564d-08dbc8ba6886 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000ECD5.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB7750 X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 09 Oct 2023 04:26:29 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779277015179368954 X-GMAIL-MSGID: 1779277015179368954 The mutex is named like it is supposed to protect only the mkey but in reality it is a global lock for all mr resources. Shift the mutex to it's rightful location (struct mlx5_vdpa_dev) and give it a more appropriate name. Signed-off-by: Dragos Tatulea Acked-by: Eugenio Pérez Acked-by: Jason Wang --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 4 ++-- drivers/vdpa/mlx5/core/mr.c | 13 +++++++------ drivers/vdpa/mlx5/core/resources.c | 6 +++--- 3 files changed, 12 insertions(+), 11 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 01d4ee58ccb1..9c6ac42c21e1 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -34,8 +34,6 @@ struct mlx5_vdpa_mr { /* state of dvq mr */ bool initialized; - /* serialize mkey creation and destruction */ - struct mutex mkey_mtx; bool user_mr; }; @@ -94,6 +92,8 @@ struct mlx5_vdpa_dev { u32 generation; struct mlx5_vdpa_mr mr; + /* serialize mr access */ + struct mutex mr_mtx; struct mlx5_control_vq cvq; struct workqueue_struct *wq; unsigned int group2asid[MLX5_VDPA_NUMVQ_GROUPS]; diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 6f29e8eaabb1..abd6a6fb122f 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -509,11 +509,11 @@ static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_ void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { - mutex_lock(&mr->mkey_mtx); + mutex_lock(&mvdev->mr_mtx); _mlx5_vdpa_destroy_mr(mvdev, mr); - mutex_unlock(&mr->mkey_mtx); + mutex_unlock(&mvdev->mr_mtx); } void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) @@ -550,9 +550,10 @@ int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, { int err; - mutex_lock(&mvdev->mr.mkey_mtx); + mutex_lock(&mvdev->mr_mtx); err = _mlx5_vdpa_create_mr(mvdev, mr, iotlb); - mutex_unlock(&mvdev->mr.mkey_mtx); + mutex_unlock(&mvdev->mr_mtx); + return err; } @@ -563,14 +564,14 @@ int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *io int err = 0; *change_map = false; - mutex_lock(&mr->mkey_mtx); + mutex_lock(&mvdev->mr_mtx); if (mr->initialized) { mlx5_vdpa_info(mvdev, "memory map update\n"); *change_map = true; } if (!*change_map) err = _mlx5_vdpa_create_mr(mvdev, mr, iotlb); - mutex_unlock(&mr->mkey_mtx); + mutex_unlock(&mvdev->mr_mtx); return err; } diff --git a/drivers/vdpa/mlx5/core/resources.c b/drivers/vdpa/mlx5/core/resources.c index d5a59c9035fb..5c5a41b64bfc 100644 --- a/drivers/vdpa/mlx5/core/resources.c +++ b/drivers/vdpa/mlx5/core/resources.c @@ -256,7 +256,7 @@ int mlx5_vdpa_alloc_resources(struct mlx5_vdpa_dev *mvdev) mlx5_vdpa_warn(mvdev, "resources already allocated\n"); return -EINVAL; } - mutex_init(&mvdev->mr.mkey_mtx); + mutex_init(&mvdev->mr_mtx); res->uar = mlx5_get_uars_page(mdev); if (IS_ERR(res->uar)) { err = PTR_ERR(res->uar); @@ -301,7 +301,7 @@ int mlx5_vdpa_alloc_resources(struct mlx5_vdpa_dev *mvdev) err_uctx: mlx5_put_uars_page(mdev, res->uar); err_uars: - mutex_destroy(&mvdev->mr.mkey_mtx); + mutex_destroy(&mvdev->mr_mtx); return err; } @@ -318,6 +318,6 @@ void mlx5_vdpa_free_resources(struct mlx5_vdpa_dev *mvdev) dealloc_pd(mvdev, res->pdn, res->uid); destroy_uctx(mvdev, res->uid); mlx5_put_uars_page(mvdev->mdev, res->uar); - mutex_destroy(&mvdev->mr.mkey_mtx); + mutex_destroy(&mvdev->mr_mtx); res->valid = false; } From patchwork Mon Oct 9 11:23:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 149964 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp1798809vqo; Mon, 9 Oct 2023 04:27:03 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG++wmL4FWQxwPAJp4tY5BVWUiDtswmifpCM6VcLrNLqEEYtmytCfN/JeiG5zC9JO2KrR6N X-Received: by 2002:a05:6a00:234a:b0:68f:b5a1:12bf with SMTP id j10-20020a056a00234a00b0068fb5a112bfmr17632800pfj.29.1696850823641; Mon, 09 Oct 2023 04:27:03 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696850823; cv=pass; d=google.com; s=arc-20160816; b=A5q5mHT7l2aC+UxO5cHKx8G401nw/utBEqILG4vcIl7aLmJ1AfRHzGsnHxGUOWQVtB Sk7T/enm3dWZ1KVpTT7yxMpefqCO8ETfqHawzlUM4GO+HfI7anPqK+FuH5Kv2qJ4U10w wBnKsgNXQsIPjexbZ+itqnGv+ydgcumA9pgTLfgmrwgVR/DGm7VTEKRMKsqMZaeCQecl SPpYT/2Ni1qsLJ20O4UF9dRZrmjzkI6FvF/dJ54Nx7KeeWiE4zx0p8a7kLMrS92JRNBu 8HK5X+sC8vs6IP1CAYhUaT7M1ggBXW88qcqKP3ll2/OmUIUYlwHxlZ3Z4VmQc59CATFy OxvA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=7jjiNTy8al6/zLsHEFtY10NWCOwo4c2oncU7C5mb/4Q=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=VgqM5fxsK4VAU4b/IbPMM0kl/57D+oPHal2s7SPs1MtEy7yFdPhF2vyRTkFLmejCcO aBn+wX57aPHh3PDDmrHOG2nR9xmjK/+2lsNiwPiPVr6RDDD/tfhM47nX2qANx0Zsd5C/ TJaAyukKN/C5fLiHns8wE9z39l5ncU313AIj3plJKorab7sqCQgp4fd9gemHn/7mRT03 E/v/2EXzI0aAHQ/ji5tOPEQmJbMTFzeluWO7nxucXHVHosUYPHGRiif69hxW9AafIZOy O98kEeG3545in3eP98E/h43VE06FlBBAYnooocyOnGn4eRtJzz7FDxlcqiZVExfOifyr gleQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=OiP1unnF; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id cq24-20020a056a00331800b0068fba70d25dsi7064471pfb.33.2023.10.09.04.27.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 04:27:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=OiP1unnF; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id DE8928082A4B; Mon, 9 Oct 2023 04:27:00 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346379AbjJIL0B (ORCPT + 18 others); Mon, 9 Oct 2023 07:26:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346357AbjJILZ2 (ORCPT ); Mon, 9 Oct 2023 07:25:28 -0400 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2074.outbound.protection.outlook.com [40.107.220.74]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4933B11C; Mon, 9 Oct 2023 04:25:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iYb0bmiZCZWRv3iAcPVEflNhhGrwZ1OjGEbafYUYZGe1r5QPmUl7s1ieSejjlEx9A21iTTZ0FW4vvs32p+pX0pxdZ3VQHWa/y4+3+v16zEbbalWnLHyXDkzeIqd7FEBcrFLNGyiyt9LYMNms54bf977SXArkZeWxDzLDwbUa68Uj6esNrL0+dGcazZfjMZC9J9dHnf1br7GHlxqJCY9UWQWjdTfOfFuqC9+aKG53lQYchGEcFSsAUnZITON/M9Kq6TG9KYCuaB5zjSsThUkDorGegXx9EgjInLl6s00NRtzB6brcrLuytQwjb/sxYhyRzEVvhkaVnoakfmJA6XMzOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7jjiNTy8al6/zLsHEFtY10NWCOwo4c2oncU7C5mb/4Q=; b=Yvjvla3Jb8OsDg3bLGAklqiXxW9k0eK/Q37CRRJyiTwGk2ArSDBHq2IAvWpOrpxtVmmPFD4eEvlPNs2nSA+FSwK182sqKUilKKn2/QAgNe11Dpq6hp09R4E7BUY5nCbv6uoceGYg5kLYtyrrGx3Li4Cp4RACP7+zDbXxvyyHbEFtN9VV//5rPlnZ2dPfRyMuFfwlFuXbnGxIHVQvCYPLRRdZ3Cowf6RNmWTfl6CkUwirdVPHze/fwjVHTSrpCU+8wBq8Gn3yh1BwnUWUpnWUMFijEKXF0wI7gUlG/rGTA4g3cN7T4YDA/fZm8GZ9sLo/SW7mXqpvkK//RFCAit3z6g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7jjiNTy8al6/zLsHEFtY10NWCOwo4c2oncU7C5mb/4Q=; b=OiP1unnFlgWkvBiyDlDeLDTLgXoejKcu+yxEyuKg5jjpPdqAQsQxsAguNE8UxUTkXqqEb5acBC+pdORadsT5+aqm0L4/bulka+dfcanqPy8N0HkgQWebPl3GLRswvoStAcd3b6DLn4EOsN1B9o14X6nkY2L+XXoW9K2x2+wR7KTqYRjOoN89kRmWjMYk45WzkWMRDHFSWF7CTQrjT4IA1pSOPjxAYOIkn7u0vFfERQUsYUKbeBy8vOBizdrMVB+hPN90oBVRXzJD9qWkD3kZykDgqx7yMLIAvJybqvRnrdsral8SZo7WxX76dstMtAuYb7iz6ctOV3DvcCnkfqFZOw== Received: from MN2PR13CA0024.namprd13.prod.outlook.com (2603:10b6:208:160::37) by LV2PR12MB5920.namprd12.prod.outlook.com (2603:10b6:408:172::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.41; Mon, 9 Oct 2023 11:25:15 +0000 Received: from MN1PEPF0000ECD6.namprd02.prod.outlook.com (2603:10b6:208:160:cafe::7f) by MN2PR13CA0024.outlook.office365.com (2603:10b6:208:160::37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.21 via Frontend Transport; Mon, 9 Oct 2023 11:25:15 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MN1PEPF0000ECD6.mail.protection.outlook.com (10.167.242.135) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.14 via Frontend Transport; Mon, 9 Oct 2023 11:25:15 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:25:02 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:25:01 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Mon, 9 Oct 2023 04:24:58 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v3 12/16] vdpa/mlx5: Improve mr update flow Date: Mon, 9 Oct 2023 14:23:57 +0300 Message-ID: <20231009112401.1060447-13-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009112401.1060447-1-dtatulea@nvidia.com> References: <20231009112401.1060447-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000ECD6:EE_|LV2PR12MB5920:EE_ X-MS-Office365-Filtering-Correlation-Id: 9d5f6010-9dfb-4f39-beb7-08dbc8ba693c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: l/ypnURB6iT7rw4XEPZ6fg+HgjfkVsC/D/Fcej266zdjTReOOtIWCN5MrlxzOzuwiORmzt4xubKTeGkNB5eC5yPO7Q86lNdEr9Kjvja+P6rFIFX3LTEbuqz4xs+OlU7rJ09vpQnF3qXUDv6BZPtSCRKXKTXJsurWkrAiHYWSWjudrJhbLNErIYVgxztatnfkNvrQ5jWpIgJgarDEPIx7MzzFzmidHXkqqBQ+Dg8cc5UpaHRmmNvz3pZmkaLyOMpn3psMxIKvWI6uD/WiCITOWkwnQKr/RDxwOv5osvlj2UZJSZKwtpPH4Z/QQL7nqHcZ4GvMa+wbQjc7wV5uB78Qc5c5QwxvwPqTG25baZALbhPlhN/VpQ315dUwu/WSF5eOo5k7WnoJkHGt7M56pwtmBl09na7bxobyorvJOXyWtTEmR0GZxmc5iA+0vSRZtSwmt3s5C0MR8U1ZkOR5dSRd3KP5oWnetH2ywIIHcvTMowwK0ZO9d+jXBPN7fI3X4385x9iE4y2tC3OsSkUFIIiCVOce4t4QyzTaVsRrk/GdPvEDKZoY22BYp/BVZ32WqVU8mryXQ11yrb6Lkrv2+7PebE43xFJUHBFFBNzvEjze7VjdW6/gkBlWO/Qnuo5y4e5eFULU1bA7Wv3197xwbrs1Hj6JOyEAgPZlhZJsUs3fPqSSU49vko1Pb9vC3kfJ7VxN/kMRpletiWeVwm8ZZwJqNcaMLrG4QZi9QvAceiMmY0Y= X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(396003)(346002)(39860400002)(376002)(230922051799003)(186009)(451199024)(64100799003)(82310400011)(1800799009)(46966006)(36840700001)(356005)(86362001)(7636003)(36756003)(2906002)(41300700001)(15650500001)(8676002)(5660300002)(8936002)(40480700001)(478600001)(4326008)(1076003)(2616005)(83380400001)(70586007)(70206006)(316002)(82740400003)(110136005)(336012)(426003)(36860700001)(26005)(66574015)(54906003)(47076005)(334744004);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2023 11:25:15.6086 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9d5f6010-9dfb-4f39-beb7-08dbc8ba693c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000ECD6.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5920 X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Mon, 09 Oct 2023 04:27:00 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779277049018746635 X-GMAIL-MSGID: 1779277049018746635 The current flow for updating an mr works directly on mvdev->mr which makes it cumbersome to handle multiple new mr structs. This patch makes the flow more straightforward by having mlx5_vdpa_create_mr return a new mr which will update the old mr (if any). The old mr will be deleted and unlinked from mvdev. This change paves the way for adding mrs for different ASIDs. The initialized bool is no longer needed as mr is now a pointer in the mlx5_vdpa_dev struct which will be NULL when not initialized. Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea Acked-by: Jason Wang --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 14 +++-- drivers/vdpa/mlx5/core/mr.c | 87 ++++++++++++++++-------------- drivers/vdpa/mlx5/net/mlx5_vnet.c | 46 ++++++++-------- 3 files changed, 76 insertions(+), 71 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 9c6ac42c21e1..bbe4335106bd 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -31,8 +31,6 @@ struct mlx5_vdpa_mr { struct list_head head; unsigned long num_directs; unsigned long num_klms; - /* state of dvq mr */ - bool initialized; bool user_mr; }; @@ -91,7 +89,7 @@ struct mlx5_vdpa_dev { u16 max_idx; u32 generation; - struct mlx5_vdpa_mr mr; + struct mlx5_vdpa_mr *mr; /* serialize mr access */ struct mutex mr_mtx; struct mlx5_control_vq cvq; @@ -114,14 +112,14 @@ void mlx5_vdpa_free_resources(struct mlx5_vdpa_dev *mvdev); int mlx5_vdpa_create_mkey(struct mlx5_vdpa_dev *mvdev, u32 *mkey, u32 *in, int inlen); int mlx5_vdpa_destroy_mkey(struct mlx5_vdpa_dev *mvdev, u32 mkey); -int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, - bool *change_map, unsigned int asid); -int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, - struct mlx5_vdpa_mr *mr, - struct vhost_iotlb *iotlb); +struct mlx5_vdpa_mr *mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, + struct vhost_iotlb *iotlb); void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev); void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr); +void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr, + unsigned int asid); int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid); diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index abd6a6fb122f..00eff5a07152 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -495,30 +495,51 @@ static void destroy_user_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { - if (!mr->initialized) - return; - if (mr->user_mr) destroy_user_mr(mvdev, mr); else destroy_dma_mr(mvdev, mr); - - mr->initialized = false; } void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { + if (!mr) + return; + mutex_lock(&mvdev->mr_mtx); _mlx5_vdpa_destroy_mr(mvdev, mr); + if (mvdev->mr == mr) + mvdev->mr = NULL; + + mutex_unlock(&mvdev->mr_mtx); + + kfree(mr); +} + +void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *new_mr, + unsigned int asid) +{ + struct mlx5_vdpa_mr *old_mr = mvdev->mr; + + mutex_lock(&mvdev->mr_mtx); + + mvdev->mr = new_mr; + if (old_mr) { + _mlx5_vdpa_destroy_mr(mvdev, old_mr); + kfree(old_mr); + } + mutex_unlock(&mvdev->mr_mtx); + } void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) { - mlx5_vdpa_destroy_mr(mvdev, &mvdev->mr); + mlx5_vdpa_destroy_mr(mvdev, mvdev->mr); prune_iotlb(mvdev); } @@ -528,52 +549,36 @@ static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, { int err; - if (mr->initialized) - return 0; - if (iotlb) err = create_user_mr(mvdev, mr, iotlb); else err = create_dma_mr(mvdev, mr); - if (err) - return err; - - mr->initialized = true; - - return 0; + return err; } -int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, - struct mlx5_vdpa_mr *mr, - struct vhost_iotlb *iotlb) +struct mlx5_vdpa_mr *mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, + struct vhost_iotlb *iotlb) { + struct mlx5_vdpa_mr *mr; int err; + mr = kzalloc(sizeof(*mr), GFP_KERNEL); + if (!mr) + return ERR_PTR(-ENOMEM); + mutex_lock(&mvdev->mr_mtx); err = _mlx5_vdpa_create_mr(mvdev, mr, iotlb); mutex_unlock(&mvdev->mr_mtx); - return err; -} - -int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, - bool *change_map, unsigned int asid) -{ - struct mlx5_vdpa_mr *mr = &mvdev->mr; - int err = 0; + if (err) + goto out_err; - *change_map = false; - mutex_lock(&mvdev->mr_mtx); - if (mr->initialized) { - mlx5_vdpa_info(mvdev, "memory map update\n"); - *change_map = true; - } - if (!*change_map) - err = _mlx5_vdpa_create_mr(mvdev, mr, iotlb); - mutex_unlock(&mvdev->mr_mtx); + return mr; - return err; +out_err: + kfree(mr); + return ERR_PTR(err); } int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, @@ -597,11 +602,13 @@ int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvdev) { - int err; + struct mlx5_vdpa_mr *mr; - err = mlx5_vdpa_create_mr(mvdev, &mvdev->mr, NULL); - if (err) - return err; + mr = mlx5_vdpa_create_mr(mvdev, NULL); + if (IS_ERR(mr)) + return PTR_ERR(mr); + + mlx5_vdpa_update_mr(mvdev, mr, 0); return mlx5_vdpa_update_cvq_iotlb(mvdev, NULL, 0); } diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 256fdd80c321..4a87f9119fca 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -873,7 +873,7 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque MLX5_SET64(virtio_q, vq_ctx, desc_addr, mvq->desc_addr); MLX5_SET64(virtio_q, vq_ctx, used_addr, mvq->device_addr); MLX5_SET64(virtio_q, vq_ctx, available_addr, mvq->driver_addr); - MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, ndev->mvdev.mr.mkey); + MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, ndev->mvdev.mr->mkey); MLX5_SET(virtio_q, vq_ctx, umem_1_id, mvq->umem1.id); MLX5_SET(virtio_q, vq_ctx, umem_1_size, mvq->umem1.size); MLX5_SET(virtio_q, vq_ctx, umem_2_id, mvq->umem2.id); @@ -2633,7 +2633,7 @@ static void restore_channels_info(struct mlx5_vdpa_net *ndev) } static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, - struct vhost_iotlb *iotlb, unsigned int asid) + struct mlx5_vdpa_mr *new_mr, unsigned int asid) { struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev); int err; @@ -2641,27 +2641,18 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, suspend_vqs(ndev); err = save_channels_info(ndev); if (err) - goto err_mr; + return err; teardown_driver(ndev); - mlx5_vdpa_destroy_mr(mvdev, &mvdev->mr); - err = mlx5_vdpa_create_mr(mvdev, &mvdev->mr, iotlb); - if (err) - goto err_mr; + + mlx5_vdpa_update_mr(mvdev, new_mr, asid); if (!(mvdev->status & VIRTIO_CONFIG_S_DRIVER_OK) || mvdev->suspended) - goto err_mr; + return 0; restore_channels_info(ndev); err = setup_driver(mvdev); - if (err) - goto err_setup; - - return 0; -err_setup: - mlx5_vdpa_destroy_mr(mvdev, &mvdev->mr); -err_mr: return err; } @@ -2875,26 +2866,35 @@ static u32 mlx5_vdpa_get_generation(struct vdpa_device *vdev) static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid) { - bool change_map; + struct mlx5_vdpa_mr *new_mr; int err; if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] != asid) goto end; - err = mlx5_vdpa_handle_set_map(mvdev, iotlb, &change_map, asid); - if (err) { - mlx5_vdpa_warn(mvdev, "set map failed(%d)\n", err); + new_mr = mlx5_vdpa_create_mr(mvdev, iotlb); + if (IS_ERR(new_mr)) { + err = PTR_ERR(new_mr); + mlx5_vdpa_warn(mvdev, "create map failed(%d)\n", err); return err; } - if (change_map) { - err = mlx5_vdpa_change_map(mvdev, iotlb, asid); - if (err) - return err; + if (!mvdev->mr) { + mlx5_vdpa_update_mr(mvdev, new_mr, asid); + } else { + err = mlx5_vdpa_change_map(mvdev, new_mr, asid); + if (err) { + mlx5_vdpa_warn(mvdev, "change map failed(%d)\n", err); + goto out_err; + } } end: return mlx5_vdpa_update_cvq_iotlb(mvdev, iotlb, asid); + +out_err: + mlx5_vdpa_destroy_mr(mvdev, new_mr); + return err; } static int mlx5_vdpa_set_map(struct vdpa_device *vdev, unsigned int asid, From patchwork Mon Oct 9 11:23:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 149961 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp1798749vqo; Mon, 9 Oct 2023 04:26:54 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE8G7ikIwpAd7cb9HteGXE3nhOZ920UNPeKgrpDy+vLlah8ENwYnvop+mgwrXfDyAZq7Hh0 X-Received: by 2002:a9d:6a05:0:b0:6c4:fc45:8681 with SMTP id g5-20020a9d6a05000000b006c4fc458681mr15675254otn.25.1696850814612; Mon, 09 Oct 2023 04:26:54 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696850814; cv=pass; d=google.com; s=arc-20160816; b=yRa7WufJaHEPpNaZV2j9/Sq9Dj8Hnwza2oqLBiYTKOEn8dJ75gpJGrgL0Ipf/VT/yt CYVgiRRnrZTQFXO+2a6qVf1WO8dpY2yvBDn0c1atglWtsNv8i7Fx4+8TOMc7AI0nPw9i JaVJTgmXgWOCyD9+Pl8R7jjpBf+j/Yvek7wfYbbYNBEKaYYTV1GGoJTAucr7d60jpKAG 6Xydrxatlo0VePjUQxAL50Lu0ZzJP1LtfvUJcbB/7CWJKBgjuQKwDVaN8hZMPeg3nuNT qVPMlqK1NQ3rvXQUnHxbhx6KBUZfNiOS0xAqToeZH8jZujWFKfN11QT/mwV1Blc02nyF fZqw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=z+AGp+BRV0QlNoxWRv0zqkqDVrcRiRrwcHIbWi2U8fQ=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=rMXnBZ3GFtR2Qm1Z6pHxsfRvVii4VEV44Hf9PjH6S8yKNS5OjneOf7VDi/rCqtNMdW iEWBn6QbjVA3rIxZccevj/DiKDIAbWA5f+Sya8e5dvfD4slVysLL6W+oNWKmTpaVD7NE HcbBrLf/n4vV10UdMPt7gQSxIKJ2MTCG4qTI62L31WnJYz4f07ZW34ixinlYKklhNczv HreO8qTn87Ix/pcxEymd0O9Xr37oODXlOQByXY6h97WXgQ7tJluVk07r5+UH+ErhfqCs R1B3JLQ4EwREIyCSx+3riQgu1/9LUMoefzbEZVswYQ/YfSaod1+aRpnHrHXlYyPMRBgx KfYg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="Y9/EU3kS"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id az1-20020a056a02004100b00578d71ef3d7si7451193pgb.733.2023.10.09.04.26.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 04:26:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="Y9/EU3kS"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 6508280BB5C5; Mon, 9 Oct 2023 04:26:50 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346414AbjJIL0Q (ORCPT + 18 others); Mon, 9 Oct 2023 07:26:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52944 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346349AbjJILZl (ORCPT ); Mon, 9 Oct 2023 07:25:41 -0400 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2078.outbound.protection.outlook.com [40.107.223.78]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 220D5193; Mon, 9 Oct 2023 04:25:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=naH19dQcnCle6ulZ4d2QyNUs+cNBX9gqqt+bAkV8WQaqHol8mJKRryGEkIKJe3TK9MpZJ64p5UvE7lKbpVev04ZevOMWv4V2tLDPeaSJD8udXbKuPGJ6sSxaasAToDAEllXZV+Xza9qeugL8/owp/kv0cq81qtS/5ZnPu4UPYbdlrTqurpLSj0u5CrmIKjN7dd1yPRxBcbUkTjW3vzV57FNvUjaZo0OjczGHU0XwE81pGRqUpElsx9m5hLsfTLu+bQn2z2Ps/13qh+f7ZQhMuXSQHokxS7JDzHutmEfbRiZtytZz5dicuSx5TMl/rMj9UrchkS77Y9IK+ILkaf7KcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=z+AGp+BRV0QlNoxWRv0zqkqDVrcRiRrwcHIbWi2U8fQ=; b=F3PH850YGaMja6A/jcLaYTxHzYmmMRXCC6ptnmhGzFYazt2S2th8JiwwM8gHQFl5Oj1lZNMah1cwldQEG6m+aog7h/x8kz0o5LHGDbjoDkfexvrhQ3OR/LsJNXGpJUqQwjKIdtzzu8qvavMCCVg/hNtHEO4xuzZam28m7GJG9QYYpkPfmqL6UhAI6P9tBKRPad2lVP7KVxzwCVs0hWaQPO1ZNTFb3Mxq6+W0bVs1ON43O8s9v5KEvUrYrQVMV7r3eo4xedCwA8wPpxYyG9joDWUvu8HeTfRgRRGYEuEEEM2GEwz/ywd4qjOJyeO8LSJKTnXIb47v7hkavIkOA5ZSzw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=z+AGp+BRV0QlNoxWRv0zqkqDVrcRiRrwcHIbWi2U8fQ=; b=Y9/EU3kSvr6sy2hdJk/w1tSAZzeuUBCDb7997Ak2c+myC/cd9EKQ20kB6LHjYQHrIF/oHo7WJ6R/rokjqcdbQeN6Rn0wGlgQIHnacZlXSzD1mCewqwB7aZsJtZFv4mlNXayKEk9/zkjtOZgclnGOlCajcATQI8LgX8947h/guwv9kx24R3a3gbsBAtk5uHhpbWzYB8yIYWEyQT4m1X/iTEaLu9lG+BVGI1k38/BRX1VMYOPwUtdapBRnrgMxgc9Ze6XL7skqneuqWdgR6ii2wtsc5PbF9bZGevNvvbZ5+M7JyuGdAa5YGxfhl0omjf36r9G2+s3TvD2o8T0/obJTrw== Received: from MN2PR08CA0023.namprd08.prod.outlook.com (2603:10b6:208:239::28) by CY8PR12MB7268.namprd12.prod.outlook.com (2603:10b6:930:54::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.30; Mon, 9 Oct 2023 11:25:22 +0000 Received: from MN1PEPF0000ECD4.namprd02.prod.outlook.com (2603:10b6:208:239:cafe::de) by MN2PR08CA0023.outlook.office365.com (2603:10b6:208:239::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.36 via Frontend Transport; Mon, 9 Oct 2023 11:25:22 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MN1PEPF0000ECD4.mail.protection.outlook.com (10.167.242.132) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.14 via Frontend Transport; Mon, 9 Oct 2023 11:25:22 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:25:05 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:25:05 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Mon, 9 Oct 2023 04:25:02 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v3 13/16] vdpa/mlx5: Introduce mr for vq descriptor Date: Mon, 9 Oct 2023 14:23:58 +0300 Message-ID: <20231009112401.1060447-14-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009112401.1060447-1-dtatulea@nvidia.com> References: <20231009112401.1060447-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000ECD4:EE_|CY8PR12MB7268:EE_ X-MS-Office365-Filtering-Correlation-Id: 6e40eb4e-7e37-40a0-9f6b-08dbc8ba6d5c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8XC0a6D6orSTMY5p3vwAK/IhqDoAlmPn58AIaGTuHwEp1PynjFWAtvgpFyTxffI3cJzj9X3uJG60uCj67HTczzuCOSoaUxsdlxT/4Ir3WFLEsngYfh7+ZDGMXmu7RLXfXP/XuRf8gVr5znu3FGkd19+r4OlXE7pGmy1btS7AvKtsmw1fS3anOu0k5wVIofcUcccMq2Q1HC+idXLT6CucJuPOKJb4TrEN4/3otcEIJfchJeAiihCvpp2LF4jnKraols12LczlLHomi7T8cMfXjy2ZFUaFcSd00nA0wirDgppcMOXWxtzLjtDulybbkqY/85EsD2RQcLSjUISPEoArULg6asNsMLSCkDtwnLy+F1bw0yUp648bifWbGvxTFIx658Jpjy9KLHHWwXfeH7jEpe+LLyK53JdWrL35HKia843xT439rZpy/4RiDCzdpAPIq4RzLYPpHsrz0m6FliATJXK+ece6Cgx0Y1WdzxHR9Z3xl2l3biZJobpQyFEGXqLhBR45muolx/Q4DlIvO536OdooZSgwtrVJLKaRdn1sLkCmCaa4OYiMj2uQIzaaOp8fEdnvA+FE4xyDNrHwlt8j6UDj4EWCGS0jL2v8/LO5scwqFJ1OKxgbhEij/trcTi1UqdRRjuG5mqo9xA6Y94zSsfkZFTe6qm83Ys7t7ZrleuACoptZe4ni98F/AY8q7TiDZlQ8oixKPh+D2c0zoXc7ePaYXBoN2srpE818F0kwnVE= X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(376002)(396003)(346002)(136003)(230922051799003)(451199024)(82310400011)(1800799009)(186009)(64100799003)(46966006)(40470700004)(36840700001)(40460700003)(40480700001)(82740400003)(2616005)(7636003)(356005)(26005)(1076003)(336012)(426003)(66574015)(110136005)(54906003)(70206006)(70586007)(6666004)(478600001)(47076005)(36860700001)(83380400001)(86362001)(36756003)(8936002)(8676002)(2906002)(4326008)(5660300002)(316002)(41300700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2023 11:25:22.5284 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6e40eb4e-7e37-40a0-9f6b-08dbc8ba6d5c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000ECD4.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7268 X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 09 Oct 2023 04:26:51 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779277040003524781 X-GMAIL-MSGID: 1779277040003524781 Introduce the vq descriptor group and mr per ASID. Until now .set_map on ASID 1 was only updating the cvq iotlb. From now on it also creates a mkey for it. The current patch doesn't use it but follow-up patches will add hardware support for mapping the vq descriptors. Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea Acked-by: Jason Wang --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 5 +++-- drivers/vdpa/mlx5/core/mr.c | 14 +++++++++----- drivers/vdpa/mlx5/net/mlx5_vnet.c | 20 +++++++++++++------- 3 files changed, 25 insertions(+), 14 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index bbe4335106bd..ae09296f4270 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -70,11 +70,12 @@ struct mlx5_vdpa_wq_ent { enum { MLX5_VDPA_DATAVQ_GROUP, MLX5_VDPA_CVQ_GROUP, + MLX5_VDPA_DATAVQ_DESC_GROUP, MLX5_VDPA_NUMVQ_GROUPS }; enum { - MLX5_VDPA_NUM_AS = MLX5_VDPA_NUMVQ_GROUPS + MLX5_VDPA_NUM_AS = 2 }; struct mlx5_vdpa_dev { @@ -89,7 +90,7 @@ struct mlx5_vdpa_dev { u16 max_idx; u32 generation; - struct mlx5_vdpa_mr *mr; + struct mlx5_vdpa_mr *mr[MLX5_VDPA_NUM_AS]; /* serialize mr access */ struct mutex mr_mtx; struct mlx5_control_vq cvq; diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 00eff5a07152..3dee6d9bed6b 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -511,8 +511,10 @@ void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, _mlx5_vdpa_destroy_mr(mvdev, mr); - if (mvdev->mr == mr) - mvdev->mr = NULL; + for (int i = 0; i < MLX5_VDPA_NUM_AS; i++) { + if (mvdev->mr[i] == mr) + mvdev->mr[i] = NULL; + } mutex_unlock(&mvdev->mr_mtx); @@ -523,11 +525,11 @@ void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *new_mr, unsigned int asid) { - struct mlx5_vdpa_mr *old_mr = mvdev->mr; + struct mlx5_vdpa_mr *old_mr = mvdev->mr[asid]; mutex_lock(&mvdev->mr_mtx); - mvdev->mr = new_mr; + mvdev->mr[asid] = new_mr; if (old_mr) { _mlx5_vdpa_destroy_mr(mvdev, old_mr); kfree(old_mr); @@ -539,7 +541,9 @@ void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) { - mlx5_vdpa_destroy_mr(mvdev, mvdev->mr); + for (int i = 0; i < MLX5_VDPA_NUM_AS; i++) + mlx5_vdpa_destroy_mr(mvdev, mvdev->mr[i]); + prune_iotlb(mvdev); } diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 4a87f9119fca..25bd2c324f5b 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -821,6 +821,8 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque { int inlen = MLX5_ST_SZ_BYTES(create_virtio_net_q_in); u32 out[MLX5_ST_SZ_DW(create_virtio_net_q_out)] = {}; + struct mlx5_vdpa_dev *mvdev = &ndev->mvdev; + struct mlx5_vdpa_mr *vq_mr; void *obj_context; u16 mlx_features; void *cmd_hdr; @@ -873,7 +875,9 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque MLX5_SET64(virtio_q, vq_ctx, desc_addr, mvq->desc_addr); MLX5_SET64(virtio_q, vq_ctx, used_addr, mvq->device_addr); MLX5_SET64(virtio_q, vq_ctx, available_addr, mvq->driver_addr); - MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, ndev->mvdev.mr->mkey); + vq_mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]]; + if (vq_mr) + MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, vq_mr->mkey); MLX5_SET(virtio_q, vq_ctx, umem_1_id, mvq->umem1.id); MLX5_SET(virtio_q, vq_ctx, umem_1_size, mvq->umem1.size); MLX5_SET(virtio_q, vq_ctx, umem_2_id, mvq->umem2.id); @@ -2633,7 +2637,8 @@ static void restore_channels_info(struct mlx5_vdpa_net *ndev) } static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, - struct mlx5_vdpa_mr *new_mr, unsigned int asid) + struct mlx5_vdpa_mr *new_mr, + unsigned int asid) { struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev); int err; @@ -2652,8 +2657,10 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, restore_channels_info(ndev); err = setup_driver(mvdev); + if (err) + return err; - return err; + return 0; } /* reslock must be held for this function */ @@ -2869,8 +2876,8 @@ static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, struct mlx5_vdpa_mr *new_mr; int err; - if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] != asid) - goto end; + if (asid >= MLX5_VDPA_NUM_AS) + return -EINVAL; new_mr = mlx5_vdpa_create_mr(mvdev, iotlb); if (IS_ERR(new_mr)) { @@ -2879,7 +2886,7 @@ static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, return err; } - if (!mvdev->mr) { + if (!mvdev->mr[asid]) { mlx5_vdpa_update_mr(mvdev, new_mr, asid); } else { err = mlx5_vdpa_change_map(mvdev, new_mr, asid); @@ -2889,7 +2896,6 @@ static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, } } -end: return mlx5_vdpa_update_cvq_iotlb(mvdev, iotlb, asid); out_err: From patchwork Mon Oct 9 11:23:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 149960 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp1798713vqo; Mon, 9 Oct 2023 04:26:49 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEY+AeLxcm0bJdBwbXBuoQ+bUCGjsetJdLTcmkeDlWhl41k/panomEMvVcXp+572XBVVm8Q X-Received: by 2002:a05:6a00:2316:b0:690:c5cf:91f5 with SMTP id h22-20020a056a00231600b00690c5cf91f5mr15342367pfh.18.1696850809545; Mon, 09 Oct 2023 04:26:49 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696850809; cv=pass; d=google.com; s=arc-20160816; b=b9Q0dr2UvVfwP0fFYWV1iMu7E5rjVIbwzdCHLXkAgX+US+iEpvX6Dg/tKQZwuxda82 +4Yqw0JNKS2uwHlCmXcEUH+v7KeTXMgaBn872yfvCZ86nO2X0M/CmoBG6IMmXdrzJXsl 6znTUDFNXDRq0muXZoRoLcBwcaIInZSpQkyb45GMewLNE3uMYDYJFZncFMtghlU49dMs 80TXoHzTG3FeVNU6VO11k+fGrJ5Lo8M4JZxhY1+S3xo8LB74AOUws32M1VTk7V//62UH cB+ZxCd9LmxIQV00SOwM+MsPRfL79vmgDclC1Mb+yQ8QYO0kAW9PESjt8JPh9O93K8QS NIyA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QovrclrVp/hpPM4yCvipuO0xAMnuUXtRKEW8YwyUiFM=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=E7jTvkKWom/id/O1fbgzbMAKDN61y4TJqzBx/16FR+t3VZZ3953W8cJwuTeYJFU7qO H/jS/s73AfAI8/GgNxsU0jkVb6FQjaFwowqSYV5c+nplVngQJfc4QaI/Q1ueckQW4ukS R7S+dhJVTPLfmZ0Jq2ewbljyj5tkSmEOlSM1GI8IbLE2crhJ8xrbANm2P/H+fnavsJ1p hZBNEfsCkdB813PANDxSGey4vyPb6z1NaRuP9/og2DpHZDdRsw5BOvWpA06qorFHM8UT L488PWubhQ8dqvRZGuok/gGFf+x/zWRwZZ641/k/857IVY00pv3Yx9RzXgw1G9AeOCJL hAjg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=OgI1um3m; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id m1-20020a656a01000000b0056fa1bc208esi10360508pgu.722.2023.10.09.04.26.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 04:26:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=OgI1um3m; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id ACE398020A32; Mon, 9 Oct 2023 04:26:38 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346331AbjJIL0V (ORCPT + 18 others); Mon, 9 Oct 2023 07:26:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52236 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346416AbjJILZr (ORCPT ); Mon, 9 Oct 2023 07:25:47 -0400 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2071.outbound.protection.outlook.com [40.107.93.71]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C39CB1AE; Mon, 9 Oct 2023 04:25:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XD0EeCwCGvFWVhY4sSZ/tAJapwKaN9oRiG7jPQ24Dm2ryhzrBZcgg7+KB44ldeUFk501T2TciyH6csMk6Jpm5dMe3wdOUnyQeMjCg9fKqnuRv/d4m/Prwav5UHN28Ez8p0COXOlEu6PYNq3za79sCW1zxggE8fm64jxEwwy9cxNnkhL2zjZeK0B0S0XRpGvSUgRwz1GTq5ORGbLR1iP/VRCXAQw7cdsjJnHbEwab89Le76637DLdhtJlPiHle5PCr4rSgVYpwnoftUbgrVR25hgDOdeGQ5+XC9istwkvyqIlZ1/voHgokS98RuQIs/NF2WoTXc8H0qPyhrNg+BJ+mw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QovrclrVp/hpPM4yCvipuO0xAMnuUXtRKEW8YwyUiFM=; b=W8BNjdBSEx9LlIXX64xWIeHJybyueUD/WiB1ov1EvwBC2KtRGlfZ8yRn5W1Dt5Hsdgu/WKOrrbS2UD6IGnZoJsJWZx68xmyDexmWIg4r2V3WWUxcuRGVlStAZKXDi4ucEy0uJBjBAR81HvNwGXksMElZ3XNr5yq5Q2oG6doAhC9cuEgoQ8dIlpVfsBsyc/S04FYKagCpJ/18BziYPQ4ucDGrRDYstIXBwB8jTSzYHKqDIqHUUX1zHZ82sqICvpr7vQ3J5FvhfkDYh2Xl8X9RXMXmjsHp0KQItiI6RvQ4jVX+UCpxTZm86nVYMJrF7h6j1EHMUCDMvwS3sd9Pr7vgPQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QovrclrVp/hpPM4yCvipuO0xAMnuUXtRKEW8YwyUiFM=; b=OgI1um3mZyGSrL9ztrdTPVGUtvAaoWl8FFIcx1bgsxQ6nJT82YlAhLj8uurbwzxVYnW2AXG2xLdU16yKS+55NO70AGdYh/5+/s2yeeukMP6i+zS+PEoiQj1s9F4uKtZIh0T6LzSOBPtHNv+FyUG66EAhqadgPReBxdu94Tjg170lx401P6YGlgHqZXFfyc55Rx7J9gHnIDy+Q+fTvFs3BNDmtMMBeai4fBLbWZRwSDVgU7I96O/MrKTsX4YLMqIEpRWgTfYNEDGc4SGrPRh9xkIL0XW4/GBHqsGCXqPJRH6jbRMnpu2Co3hHDsg8/fYmvPHbVNrC/DAFb9vL8phm5w== Received: from SA1P222CA0097.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:35e::24) by SJ0PR12MB5635.namprd12.prod.outlook.com (2603:10b6:a03:42a::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.36; Mon, 9 Oct 2023 11:25:24 +0000 Received: from SN1PEPF0002BA50.namprd03.prod.outlook.com (2603:10b6:806:35e:cafe::5f) by SA1P222CA0097.outlook.office365.com (2603:10b6:806:35e::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.36 via Frontend Transport; Mon, 9 Oct 2023 11:25:24 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SN1PEPF0002BA50.mail.protection.outlook.com (10.167.242.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.14 via Frontend Transport; Mon, 9 Oct 2023 11:25:24 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:25:09 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:25:08 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Mon, 9 Oct 2023 04:25:05 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v3 14/16] vdpa/mlx5: Enable hw support for vq descriptor mapping Date: Mon, 9 Oct 2023 14:23:59 +0300 Message-ID: <20231009112401.1060447-15-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009112401.1060447-1-dtatulea@nvidia.com> References: <20231009112401.1060447-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002BA50:EE_|SJ0PR12MB5635:EE_ X-MS-Office365-Filtering-Correlation-Id: d01de39d-8987-474f-330e-08dbc8ba6e56 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: wfwS0H83RW+IgOwrIuMsijPquvm7BNM0MTqv8SNOyQk+rtrDCbdaB6xFosY48C7cFZ8FdDibMEICBKHCbP9DHXw1g5M6/W8Ir5m0guZrrnqnBKdb5Tp2OISRIPEXJ7fIWBFzwJjzZyXcn2O9VCD6fWtOd1AIp1cwayils8jtAEcZuO8UaWjOspwAnJW1o/we4EmgOv+K1T+eF6kLJKlHWSub4cRUwQuL6S/HYIr7SVRQjDHoGUfnTk7mF53V6vQRWM6SfVLJ0brvSLJFhG+MW3DR93Z35bXnxhZUJRcvsGelaeijgzxnKJUJ71g7FSbHQR7MZH2XcLQ1/9+IZolgz0AHy84x0HxVOYUBTYQ3zbTfVjUbUhT+k12Umj1xjSi/dhchNySYTzCAICZMoOfe9liE4ShCRbdU/9o8iiVe4x84sErT0zy48lMNYQZV4rZkpxdViyd6/1qlAnO4VyNJzfnCkzjVTvxB143C+TikRzxxAgc4ovacoXqp8yHn+dE99iZMdt7/XyBkGkwpR4Ah918SRfl4ntUV6hZilxsSU0DJj2SSMMU/9nbQra9y2t/cl1ZEkuadbDd66OoQ349cXN2RKfrHuxa12PDbY8TgkFCqKkNSrlneHlEX9HK9Znu41sztMKzyCZIXtzV0zdQnPpCBuktNQYlEymzzKDj8KkKW+VbZK+z2B/iqBYp3iCOdHYGmo1gTU0f4cpOIJjDWufzRBWLoaGjxIUVkosSUts0gCCnaAn8S+ShkDHzHXrTtJaB3cwuDsWvWus7WMiIq3Eo5JZgt8sop1ViS7aL8CI0= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(376002)(39860400002)(396003)(346002)(136003)(230922051799003)(186009)(82310400011)(451199024)(1800799009)(64100799003)(40470700004)(36840700001)(46966006)(2906002)(40460700003)(426003)(336012)(26005)(2616005)(86362001)(1076003)(6666004)(36756003)(478600001)(66574015)(36860700001)(47076005)(83380400001)(7636003)(82740400003)(356005)(40480700001)(54906003)(70586007)(70206006)(316002)(110136005)(4326008)(41300700001)(5660300002)(8936002)(8676002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2023 11:25:24.2266 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d01de39d-8987-474f-330e-08dbc8ba6e56 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002BA50.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5635 X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 09 Oct 2023 04:26:38 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779277034751673165 X-GMAIL-MSGID: 1779277034751673165 Vq descriptor mappings are supported in hardware by filling in an additional mkey which contains the descriptor mappings to the hw vq. A previous patch in this series added support for hw mkey (mr) creation for ASID 1. This patch fills in both the vq data and vq descriptor mkeys based on group ASID mapping. The feature is signaled to the vdpa core through the presence of the .get_vq_desc_group op. Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea Acked-by: Jason Wang --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 24 +++++++++++++++++++++++- include/linux/mlx5/mlx5_ifc_vdpa.h | 7 ++++++- 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 25bd2c324f5b..2e0a3ce1c0cf 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -823,6 +823,7 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque u32 out[MLX5_ST_SZ_DW(create_virtio_net_q_out)] = {}; struct mlx5_vdpa_dev *mvdev = &ndev->mvdev; struct mlx5_vdpa_mr *vq_mr; + struct mlx5_vdpa_mr *vq_desc_mr; void *obj_context; u16 mlx_features; void *cmd_hdr; @@ -878,6 +879,11 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque vq_mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]]; if (vq_mr) MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, vq_mr->mkey); + + vq_desc_mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP]]; + if (vq_desc_mr) + MLX5_SET(virtio_q, vq_ctx, desc_group_mkey, vq_desc_mr->mkey); + MLX5_SET(virtio_q, vq_ctx, umem_1_id, mvq->umem1.id); MLX5_SET(virtio_q, vq_ctx, umem_1_size, mvq->umem1.size); MLX5_SET(virtio_q, vq_ctx, umem_2_id, mvq->umem2.id); @@ -2265,6 +2271,16 @@ static u32 mlx5_vdpa_get_vq_group(struct vdpa_device *vdev, u16 idx) return MLX5_VDPA_DATAVQ_GROUP; } +static u32 mlx5_vdpa_get_vq_desc_group(struct vdpa_device *vdev, u16 idx) +{ + struct mlx5_vdpa_dev *mvdev = to_mvdev(vdev); + + if (is_ctrl_vq_idx(mvdev, idx)) + return MLX5_VDPA_CVQ_GROUP; + + return MLX5_VDPA_DATAVQ_DESC_GROUP; +} + static u64 mlx_to_vritio_features(u16 dev_features) { u64 result = 0; @@ -3160,6 +3176,7 @@ static const struct vdpa_config_ops mlx5_vdpa_ops = { .get_vq_irq = mlx5_get_vq_irq, .get_vq_align = mlx5_vdpa_get_vq_align, .get_vq_group = mlx5_vdpa_get_vq_group, + .get_vq_desc_group = mlx5_vdpa_get_vq_desc_group, /* Op disabled if not supported. */ .get_device_features = mlx5_vdpa_get_device_features, .set_driver_features = mlx5_vdpa_set_driver_features, .get_driver_features = mlx5_vdpa_get_driver_features, @@ -3258,6 +3275,7 @@ struct mlx5_vdpa_mgmtdev { struct vdpa_mgmt_dev mgtdev; struct mlx5_adev *madev; struct mlx5_vdpa_net *ndev; + struct vdpa_config_ops vdpa_ops; }; static int config_func_mtu(struct mlx5_core_dev *mdev, u16 mtu) @@ -3371,7 +3389,7 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name, max_vqs = 2; } - ndev = vdpa_alloc_device(struct mlx5_vdpa_net, mvdev.vdev, mdev->device, &mlx5_vdpa_ops, + ndev = vdpa_alloc_device(struct mlx5_vdpa_net, mvdev.vdev, mdev->device, &mgtdev->vdpa_ops, MLX5_VDPA_NUMVQ_GROUPS, MLX5_VDPA_NUM_AS, name, false); if (IS_ERR(ndev)) return PTR_ERR(ndev); @@ -3546,6 +3564,10 @@ static int mlx5v_probe(struct auxiliary_device *adev, MLX5_CAP_DEV_VDPA_EMULATION(mdev, max_num_virtio_queues) + 1; mgtdev->mgtdev.supported_features = get_supported_features(mdev); mgtdev->madev = madev; + mgtdev->vdpa_ops = mlx5_vdpa_ops; + + if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, desc_group_mkey_supported)) + mgtdev->vdpa_ops.get_vq_desc_group = NULL; err = vdpa_mgmtdev_register(&mgtdev->mgtdev); if (err) diff --git a/include/linux/mlx5/mlx5_ifc_vdpa.h b/include/linux/mlx5/mlx5_ifc_vdpa.h index 9becdc3fa503..b86d51a855f6 100644 --- a/include/linux/mlx5/mlx5_ifc_vdpa.h +++ b/include/linux/mlx5/mlx5_ifc_vdpa.h @@ -74,7 +74,11 @@ struct mlx5_ifc_virtio_q_bits { u8 reserved_at_320[0x8]; u8 pd[0x18]; - u8 reserved_at_340[0xc0]; + u8 reserved_at_340[0x20]; + + u8 desc_group_mkey[0x20]; + + u8 reserved_at_380[0x80]; }; struct mlx5_ifc_virtio_net_q_object_bits { @@ -141,6 +145,7 @@ enum { MLX5_VIRTQ_MODIFY_MASK_STATE = (u64)1 << 0, MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_PARAMS = (u64)1 << 3, MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_DUMP_ENABLE = (u64)1 << 4, + MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY = (u64)1 << 14, }; enum { From patchwork Mon Oct 9 11:24:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 149962 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp1798777vqo; Mon, 9 Oct 2023 04:26:59 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHHVH68J3KAazatxf70NaXqGHD5v8lJR79KUe1BXa6V887PT0bQetFzh4v0vQxipYoKQZTw X-Received: by 2002:a05:6358:42a6:b0:145:6e16:fa8f with SMTP id s38-20020a05635842a600b001456e16fa8fmr14285653rwc.13.1696850819441; Mon, 09 Oct 2023 04:26:59 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696850819; cv=pass; d=google.com; s=arc-20160816; b=a4l/F3+jE1j4qLBR3Wa79RktnpLGxG6LmLTAvybKFeFSh+YAh842y57H99JmCy+GxW RTog+J1uxdKn/WyHi4Nkap8yZBFQCy42g/78nkFPLKAT7/9sGxFElwSqOURwdI4tNTaR ll51lpziE77JmrjLr9e1lOVbNfpB71CCJnDn99LcCvMOqaaS2DTkpu9q3r2VeIPaxWt7 V+2P9jF4DhOU87oh6KaSmu3lUF5AHyODKCd8DWtISX5X3bFVmOBFOcYu0pJp3c1lekUh Z1JK9t09cp3NGfQ1VCG1x9DqkpnIIHIo4dM0/J0+JkE/1R8epLsq9NM5PqoBXqUm0/Fk Edmg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=n+I3U15Hi59woTFRPbH5j837sff+uPVTo/I0JiXr6qk=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=v5Z9dFYioqM89itVjvIufFIlsLV3L4sQpobR7UgelMrobitkrwBVTfAHjCKR9ZDuP0 ytoG3HXRAb1kl0EQxjOiajnLuUXIqknaJmT0FmX8PsPD0Yu7rDwLp4dN4DfGSes9OSNB LCuvkL1axMQ/l6pdGPuQ9blqTpAxrm/47Bn9jgoQUKAiFrSbDP/WaEucEMNGZigAY3QY 4x42YTGY5xWBXFNBeXnC7RZL1tUdKLNh7fAawgimYlY+nExPWKp+pa7Q7NTBcKpv9D+H sBzClc7Wsp++axXqVeBti/h3qdBvnRQr7YNgnupx7r9THdYHjA+abZCw6liJq5VcEjf4 d11g== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=Eg+YMQuD; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id 26-20020a630f5a000000b005859e8c7c2dsi9105537pgp.639.2023.10.09.04.26.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 04:26:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=Eg+YMQuD; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 5869280BB5F2; Mon, 9 Oct 2023 04:26:57 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346288AbjJIL0p (ORCPT + 18 others); Mon, 9 Oct 2023 07:26:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51954 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346390AbjJIL0L (ORCPT ); Mon, 9 Oct 2023 07:26:11 -0400 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2041.outbound.protection.outlook.com [40.107.236.41]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9DBE01B5; Mon, 9 Oct 2023 04:25:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bowwe30t+StjCjy5iQfi4XfLyXhV7dVudffAvh57eHsmxaraCxOB3SsPOSyaBqNyaXMs2Ng7BlSRfnClgT8e9glXOL6X2o4+8MaGOBRS/upFkUGt2GmJ4BVU8OvklErup+9RIwTBmfrqiixPzXSfN+3hSfwSJf6demJsiKSLMwzel9xP0o8M23E1wTbDo9+rm0fh1Dwr+xmoomHVwEn7ixamQGHMnkVRqc4FcyZNeKt3d+ljoN8m5Qaz7jN/nr5kp8DJSuIFQpTFX9c0B+tFK6MHE6yJzyid+s7lGIMPxFzotiBCOtwzWHfndD6pvvyH66W6VEG+yTyoXGUUqj/ArA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=n+I3U15Hi59woTFRPbH5j837sff+uPVTo/I0JiXr6qk=; b=kunyezStVwuYQHX8GEXZFRmh3bgpDBHVjzU19X35Wg5TrJdBN/FkN7gKERs+rJcIFvvn2GF/kZWPAr1kAoCs06vDoT8rtgBYWS98Kn6NkEbTgudYEq/iMXjoJ+zyZYbWv3SacrRAZhGeJGxqtQGsQa8G/4j6O7VzAunM/Q8EOgNVVfQYYu6tQg5TCR/L37Tekci96IcePIZpUBF/C29LSupj/8vkdkxGzL//9k7ZLtNuxVoC1aWr1LszExCxF0p+IiPXqrORc/m61QF6DSeg13pUlcJ47kPEiZGI69bSEwAqvlYdUgwFCZdQ2IqRrgNWQSxjEaFtLwo4U8VwcDxiiA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=n+I3U15Hi59woTFRPbH5j837sff+uPVTo/I0JiXr6qk=; b=Eg+YMQuD97q+R3+Aia9ooJavyDI4h66sZc/nWTPgl6WUNl75srEaA8tJlfdFGKc5wdITrVRgV5UAyms8Xp4LkdnQev4UqG6iw7Q0PO4v5SgcNUZdMrfuf/VeY5mGfqiOVeHyFJz60du9e5rFXKr7TdqdGWGu46ioVbRscBcYxQN24c7yYcf7bskF/OnOGB98Z63sCBv9YHmxTabm1DWj0X+bTizAQ7ZWL2KPpgJz5bHRTkdlx4S62egZ+MkAFckY3LVo88Z7tXRp+8h7xo/ZpLaTBNYUkpb2GSpU9wIpW6BpEhVWoGAGFj/vrdrURIJrcvtzkqogOLO02UAsdquNlg== Received: from BL1PR13CA0375.namprd13.prod.outlook.com (2603:10b6:208:2c0::20) by BL1PR12MB5175.namprd12.prod.outlook.com (2603:10b6:208:318::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.38; Mon, 9 Oct 2023 11:25:31 +0000 Received: from MN1PEPF0000ECDB.namprd02.prod.outlook.com (2603:10b6:208:2c0:cafe::19) by BL1PR13CA0375.outlook.office365.com (2603:10b6:208:2c0::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.20 via Frontend Transport; Mon, 9 Oct 2023 11:25:31 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MN1PEPF0000ECDB.mail.protection.outlook.com (10.167.242.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.18 via Frontend Transport; Mon, 9 Oct 2023 11:25:30 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:25:12 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:25:12 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Mon, 9 Oct 2023 04:25:09 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v3 15/16] vdpa/mlx5: Make iotlb helper functions more generic Date: Mon, 9 Oct 2023 14:24:00 +0300 Message-ID: <20231009112401.1060447-16-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009112401.1060447-1-dtatulea@nvidia.com> References: <20231009112401.1060447-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000ECDB:EE_|BL1PR12MB5175:EE_ X-MS-Office365-Filtering-Correlation-Id: 7bf27568-5830-40c9-de08-08dbc8ba724b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: SLn3o87QY/dkPzvhaJhBJJ+BOIi5zul4S3GDgvyqeI7oUsTOqJ1RWELdfyUcvf30iP/DHeHjBG9IYvHv/5Jf8MO5blBE/VfFlrZTXwkQNhRuH0enhwyqsADxK1otLRDjsBJ5ZtNwoByi0KDN4q4lXZev1cvnccfJ6NmBmm276JBjDyI5mZMzdZ9cD3fNQ/NpNGswgq0ViJqYenFthbPFoV17G55RZEqQjESjQLhEir7ok0T/Dz8kZ+Y5AZsqrefETEAvGodv8zMtqvT5KwcgZsXtJBm3+kHGZ94TJdHkYn0mb28RLS0wi0GzLnCZA8Psnw/FeuMcl+u3Q24LrsiQnMfnHhEKZZsLh+kzMB50vNoxMq4vhb+vz6ihF60dcxOhlkUPQBCSOF6Uw73N0+Yo/m1NjuYMtefz3mxICX7EaXoM1fYunUdhgOm9N6ak7430/4BIsUbD0L/+zCs7MvJJO5rR+BwH0CZAty9ygTauVMCXNVdCBY3D5hGd3IYQ5qYgmmaeiuZ0AyIlaHdwRU26gd/cnpIkmk+yIExcJIabY5PiaxvjvtCs/SQ3livk/sA2qkmjjXQyw9pS2IOABispJsruyH0A+4cV0kOuACq1NQTwtlM+7YmGNMFizWiJ/WyLtqTcnKpxZCv0WymfMjrtRuk4NE0Ses9h9s4PRo2j8LCXCP2A3/t1CjqT6XwOFROBmpbXA25PGHjdraPbgvioKHHRWw/NVU7uh06IDGPUsbw4tmkwIrxWKht+26KFDtk4WEkJDApImWG2gUhz3CBLUw== X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(346002)(396003)(376002)(39860400002)(230922051799003)(1800799009)(82310400011)(64100799003)(451199024)(186009)(40470700004)(46966006)(36840700001)(7636003)(356005)(86362001)(36756003)(40460700003)(40480700001)(2906002)(478600001)(82740400003)(5660300002)(4326008)(8936002)(8676002)(41300700001)(6666004)(83380400001)(426003)(336012)(2616005)(1076003)(36860700001)(110136005)(70586007)(70206006)(54906003)(316002)(26005)(47076005)(66574015)(41533002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2023 11:25:30.7916 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7bf27568-5830-40c9-de08-08dbc8ba724b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000ECDB.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5175 X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 09 Oct 2023 04:26:57 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779277044482520687 X-GMAIL-MSGID: 1779277044482520687 They will be used in a follow-up patch. For dup_iotlb, avoid the src == dst case. This is an error. Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea Acked-by: Jason Wang --- drivers/vdpa/mlx5/core/mr.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 3dee6d9bed6b..4a3df865df40 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -454,20 +454,23 @@ static void destroy_dma_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) mlx5_vdpa_destroy_mkey(mvdev, mr->mkey); } -static int dup_iotlb(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *src) +static int dup_iotlb(struct vhost_iotlb *dst, struct vhost_iotlb *src) { struct vhost_iotlb_map *map; u64 start = 0, last = ULLONG_MAX; int err; + if (dst == src) + return -EINVAL; + if (!src) { - err = vhost_iotlb_add_range(mvdev->cvq.iotlb, start, last, start, VHOST_ACCESS_RW); + err = vhost_iotlb_add_range(dst, start, last, start, VHOST_ACCESS_RW); return err; } for (map = vhost_iotlb_itree_first(src, start, last); map; map = vhost_iotlb_itree_next(map, start, last)) { - err = vhost_iotlb_add_range(mvdev->cvq.iotlb, map->start, map->last, + err = vhost_iotlb_add_range(dst, map->start, map->last, map->addr, map->perm); if (err) return err; @@ -475,9 +478,9 @@ static int dup_iotlb(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *src) return 0; } -static void prune_iotlb(struct mlx5_vdpa_dev *mvdev) +static void prune_iotlb(struct vhost_iotlb *iotlb) { - vhost_iotlb_del_range(mvdev->cvq.iotlb, 0, ULLONG_MAX); + vhost_iotlb_del_range(iotlb, 0, ULLONG_MAX); } static void destroy_user_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) @@ -544,7 +547,7 @@ void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) for (int i = 0; i < MLX5_VDPA_NUM_AS; i++) mlx5_vdpa_destroy_mr(mvdev, mvdev->mr[i]); - prune_iotlb(mvdev); + prune_iotlb(mvdev->cvq.iotlb); } static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, @@ -596,8 +599,8 @@ int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, spin_lock(&mvdev->cvq.iommu_lock); - prune_iotlb(mvdev); - err = dup_iotlb(mvdev, iotlb); + prune_iotlb(mvdev->cvq.iotlb); + err = dup_iotlb(mvdev->cvq.iotlb, iotlb); spin_unlock(&mvdev->cvq.iommu_lock); From patchwork Mon Oct 9 11:24:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 149963 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp1798786vqo; Mon, 9 Oct 2023 04:27:01 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFFO/KlPzrGh5Kg0G269KRFVzWoONsyLemtSqXkmGz0Lg2671EICigPw5CYfYsSGvGT6Hcz X-Received: by 2002:a17:90a:c20d:b0:268:2af6:e48c with SMTP id e13-20020a17090ac20d00b002682af6e48cmr19342455pjt.4.1696850821066; Mon, 09 Oct 2023 04:27:01 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696850821; cv=pass; d=google.com; s=arc-20160816; b=r0Ik2EOllCwTSJ4/Y23jb2p+UAjPyihheRB1uyTRbZ8QOyegCJShFXwH1JyDico7rV QjphTsd1TDgqb4Lstrz86jNuhEHQJ2ExSS6GYnwv1tDULskVtfV7EECoEkGIV7iWsITt 8H75U+UhFD1ODVMY+R4DrnCuItk9EDrqXYP4RYFYHe1ZCcqVLoAiFYx0jHCpiEe9HJG6 0XPIcqBviiLO9Voaf9wUI+srUEEHLT9Ap25Y9ibFCnEFxE7qFyJgSIGszKzV2TvBDbiz X8jWgfHUfbAsRk9+ota7MA4Y/LGw7b9YE9w6Bkzdtx0XWD+Xu4OL41pYeBjxrD27iRt8 B9FA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=WUyxh+q8JiTvEHDFg9Oqu+9F9CPr0I46v6fSiFkbajM=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=qhani4z564Lg0+aM3XWE4OZ8VecKJU443GDyPMhlOSSMcQujQCuVJuDNKffIApXfGu k2fG2hxrRwR4DJFjNvKXYVY/rTFnkDIMJguYoTf1ubKqE1FWfZ29cbrr9gQnhnS4esHV wzW4Zidl4BMsEtsfLf6t6prNqSwJK6syv7sMcDluieBMp4KRg6Mz6kqeP6sSmzNVQ7B/ HMOlBQm9gDO3PNGA6hY7K195zWzcGZYQq694vA82GsQh/8ZIgOMrU3OzagwN9bnf+rHI pYZxXMuoS8bTdPOT6qFXrVIfPb1ZpzI6KEbEvnccrBfKR0hqJuYLYEj8v9n5Y7nJF4Tb swXw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=aHgLa5R+; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id mz17-20020a17090b379100b00274e2ba0ca6si11385395pjb.134.2023.10.09.04.27.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 Oct 2023 04:27:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=aHgLa5R+; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 19C31803FC22; Mon, 9 Oct 2023 04:26:58 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346276AbjJIL0t (ORCPT + 18 others); Mon, 9 Oct 2023 07:26:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60140 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346395AbjJIL0L (ORCPT ); Mon, 9 Oct 2023 07:26:11 -0400 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2069.outbound.protection.outlook.com [40.107.237.69]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E591710D5; Mon, 9 Oct 2023 04:25:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SVjEKc8B8gPjJPHPgvMjRlmMm1JnVZYepX+pm+U0nowxVkDVTRcK/azGCrPRDsTmsto6Hgu5pdSprTJVtykcFrZzNw4F36aK6c5rd88DO7Jg8/vHgMQAx4RGe/m55QVrZuNYdgpHivcLcRo+p6a3w2ibvgjzQCc23ikFyHEPu9PqoKTpUfFm+MRIVQ3YzUoLQTC2n/rS6LL6e8pKefS6/aJj8s6tVDvNCiFmPq7Y/+Zof8jLYXhNDrNA+k/pyyH8Zi2OTmslpzovwmbyHn4rMh3ZsU4grXcYv2bf7QKNc4y7zfqB1ULC0fw/cp72cbN/r9n0FS8AjzVillW7QGGyeA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WUyxh+q8JiTvEHDFg9Oqu+9F9CPr0I46v6fSiFkbajM=; b=dJx04D53+G2AoxAMIkrl2MwSxtWjd1SelerBnOqrCtn2EQa+gfIl4rrLtEyMoCQdWOcsvqnZoIVggA3n+lqVfmRkcchgyLN9VfWKOel0HoDb4AXjnLXOIz8H2bwkpI8hVQdwtsHwXIeN0UiaA8+xNVUfpnJe3d22vcpGshVkG3N3aPfdr4lB/hhbdn1mPNr59GtEg/mW8+bXJKAyHMDqdTrr2Vas8iMQhze3AzvJYhV/JF2bdHvDrvXVem6qQC/EDEqjRWo+bjqaST7cRc6r5H2TXLsdEzXZv9IeFVoN/tq0qYMs5GJGYkJAhyCATWa2HR7rXQKEqazNh0kHdPV3gw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WUyxh+q8JiTvEHDFg9Oqu+9F9CPr0I46v6fSiFkbajM=; b=aHgLa5R+W8/rSGqWwoqw+HFb1FQaUziOsF6OYJeIO0nqrM3apQUBtblmpVdjA+4+ZLCkEgq4726oe/n8NETcadqFQiV37LjK30QuXFmbWe/e6QPw/nO9r8k30cdXCrE+p2ixBZMot20VD92+zUZ9krU0si8DzkfKCkbJ1pqjdM3cbhczHNam1ZX8kFg9eILDS+hwg7U9wKEdfXINC8g1RzGcW3nHsCeYqVIW65Swy+O64d3rYh2PNNKXbcRSXu3oY0xJ8V+nyb6P+8F/JyNaP8M0JvZz5GGvU7iMDhn/wesMno7WFOgDLaaPSxvdzTWqbhKS2k9pVvIX8g6Ytdik3A== Received: from BLAPR03CA0045.namprd03.prod.outlook.com (2603:10b6:208:32d::20) by CH2PR12MB4889.namprd12.prod.outlook.com (2603:10b6:610:68::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.37; Mon, 9 Oct 2023 11:25:33 +0000 Received: from MN1PEPF0000ECD5.namprd02.prod.outlook.com (2603:10b6:208:32d:cafe::a6) by BLAPR03CA0045.outlook.office365.com (2603:10b6:208:32d::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.36 via Frontend Transport; Mon, 9 Oct 2023 11:25:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MN1PEPF0000ECD5.mail.protection.outlook.com (10.167.242.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.22 via Frontend Transport; Mon, 9 Oct 2023 11:25:33 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:25:16 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Mon, 9 Oct 2023 04:25:16 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Mon, 9 Oct 2023 04:25:13 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v3 16/16] vdpa/mlx5: Update cvq iotlb mapping on ASID change Date: Mon, 9 Oct 2023 14:24:01 +0300 Message-ID: <20231009112401.1060447-17-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231009112401.1060447-1-dtatulea@nvidia.com> References: <20231009112401.1060447-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN1PEPF0000ECD5:EE_|CH2PR12MB4889:EE_ X-MS-Office365-Filtering-Correlation-Id: 6b1e4f64-b605-486f-d985-08dbc8ba73a9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: r8pNfuNGDy5RWcA4mthcG8kkb1yHSIvsf4NTVpREDloZWvmkSUqmQacj+DtEKaXYR7iNYYZCxrlxECdyODzMdBpC6E2XQnFBR9jOelDo8f2GifQnMOMSlvWBKONQXczb26sto76hkN2D3a0a36/5QAO4v+PL7uEBG2hKbqSQ2UwBS4TeM1PgdFh9LUkVvrsZTx18u2gLiaPbqIkszan+mVe3qTfXynRf3SRCGiEAr4rrWFxE9IKZf1DXEj1VyV04CPh3OQjYmGyKl18E2b593oIRXxMZSPlDE0gQMz8qYMnIsPEA3dcqz/EEEjW0TZdD4XCZoBVWpOP5bTaOkagg+gr9KZpV6pYpIJKmsiJXy1zcrIafshtoTA/pveGyOQsvgbmL9xZyMEA0N2p6IA9z2tDDYyHEK7A1aq6iU4NX9QMimgagbQwABOx54tr3ggVZPiWCUNF1JBdyABl2MlACF3/ji1Q/+X753pa7lV7PYcz8pGfKiRAdl+oM9DuaMVlnTTVCDjbgeDZvejXF2ktxbk64TkYFWe/c1McALAWioh1cb1eGe/Y17aTcUuPWIhHGO4iD7CMNGIEYxcPdlv+G3T3p6FNUhydnl2HJNWlxen8NZ6LxPbtdD+3+q6nLEPDVy8s6NvAZ857/lIuZY4IVjNtOF8Hd/rrDd2OopbTybsE0CGQejKu4KjJ+UqijCxBojdBtBIdsXtacSU5eqGqEZC96VvJ8LxDkZdfENJw+5DLTHujUUP6xO0fe4+l6C0v2 X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(346002)(376002)(39860400002)(136003)(230922051799003)(451199024)(186009)(64100799003)(1800799009)(82310400011)(40470700004)(36840700001)(46966006)(6666004)(1076003)(2616005)(40460700003)(36756003)(86362001)(7636003)(40480700001)(82740400003)(356005)(36860700001)(26005)(83380400001)(47076005)(426003)(336012)(66574015)(2906002)(478600001)(41300700001)(8676002)(4326008)(316002)(8936002)(5660300002)(70586007)(110136005)(54906003)(70206006);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2023 11:25:33.1009 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6b1e4f64-b605-486f-d985-08dbc8ba73a9 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MN1PEPF0000ECD5.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4889 X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 09 Oct 2023 04:26:58 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779277046075807674 X-GMAIL-MSGID: 1779277046075807674 For the following sequence: - cvq group is in ASID 0 - .set_map(1, cvq_iotlb) - .set_group_asid(cvq_group, 1) ... the cvq mapping from ASID 0 will be used. This is not always correct behaviour. This patch adds support for the above mentioned flow by saving the iotlb on each .set_map and updating the cvq iotlb with it on a cvq group change. Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea Acked-by: Jason Wang --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 2 ++ drivers/vdpa/mlx5/core/mr.c | 26 ++++++++++++++++++++++++++ drivers/vdpa/mlx5/net/mlx5_vnet.c | 9 ++++++++- 3 files changed, 36 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index ae09296f4270..db988ced5a5d 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -32,6 +32,8 @@ struct mlx5_vdpa_mr { unsigned long num_directs; unsigned long num_klms; + struct vhost_iotlb *iotlb; + bool user_mr; }; diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 4a3df865df40..66530e28f327 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -502,6 +502,8 @@ static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_ destroy_user_mr(mvdev, mr); else destroy_dma_mr(mvdev, mr); + + vhost_iotlb_free(mr->iotlb); } void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, @@ -561,6 +563,30 @@ static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, else err = create_dma_mr(mvdev, mr); + if (err) + return err; + + mr->iotlb = vhost_iotlb_alloc(0, 0); + if (!mr->iotlb) { + err = -ENOMEM; + goto err_mr; + } + + err = dup_iotlb(mr->iotlb, iotlb); + if (err) + goto err_iotlb; + + return 0; + +err_iotlb: + vhost_iotlb_free(mr->iotlb); + +err_mr: + if (iotlb) + destroy_user_mr(mvdev, mr); + else + destroy_dma_mr(mvdev, mr); + return err; } diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 2e0a3ce1c0cf..6abe02310f2b 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -3154,12 +3154,19 @@ static int mlx5_set_group_asid(struct vdpa_device *vdev, u32 group, unsigned int asid) { struct mlx5_vdpa_dev *mvdev = to_mvdev(vdev); + int err = 0; if (group >= MLX5_VDPA_NUMVQ_GROUPS) return -EINVAL; mvdev->group2asid[group] = asid; - return 0; + + mutex_lock(&mvdev->mr_mtx); + if (group == MLX5_VDPA_CVQ_GROUP && mvdev->mr[asid]) + err = mlx5_vdpa_update_cvq_iotlb(mvdev, mvdev->mr[asid]->iotlb, asid); + mutex_unlock(&mvdev->mr_mtx); + + return err; } static const struct vdpa_config_ops mlx5_vdpa_ops = {