From patchwork Wed Oct 18 17:14:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 155007 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4945142vqb; Wed, 18 Oct 2023 10:16:31 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHk2TywJJC2RKcPx1vPGSllZm2pBXf3zrHdDzw2tq01DFMqC/Mgv3DkRyKnGPRXkbHqKq77 X-Received: by 2002:a17:903:78f:b0:1ca:3e40:beae with SMTP id kn15-20020a170903078f00b001ca3e40beaemr5342931plb.38.1697649391095; Wed, 18 Oct 2023 10:16:31 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697649391; cv=pass; d=google.com; s=arc-20160816; b=UkqUgp0gm5JmAkiUysdKJ8hlAZrKeehpjMigi/+/d7cTN6kPRAyBWyll2qOQz30Pat 409i7KnOIcs6p5JvneNwRg2ehzl4464jRuox4qo0vyWl1wYi2oUUxpAbSoSH/Uhut7u3 a2H348BgAoW/i3m7WOk1KSZhAnP5J93ALMt2thPjR1XliDhOnLqXbw3oKKrZEnjfPD6Q QTXyicI+r6P3N2CGXudMcLuVS2TYf0PUMvnrWP5CJwqhweXU2P9m5fRbBmezIt2JK4sZ 8eVXD3uyNEnxjGQHBljX/i27W74K0n9kdF2DujP3QhmZjgRQaPCUerJ5uf4fXIXCsU9v zhMg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3STa7lEVTN260CF+7a2Ikvh2pPY1BfsD0f/6MUcbh0M=; fh=iwiW798sTWZr0bClUZYdSITt06HHhXlZHI8Adgm6+CE=; b=YA6wegIRfcOZxaQgYKjes623jsiMM5PILFTCumQ1FYB6WZnL4lEPq0TbPiUIO5Mt7d d5RGF18jpfmUs9b6Q4Rms5x7/LDzKefbdlZIFAsecAJjmdzrIjCpaGLuK4krK3mfWrQZ FyAXCFJG3gla3AIH878Q06iBEiLfHWoPfQDSFIm2/9uHS3F8Yj/fvr/cKq2UEC8gZ2pE dqybZL66WjKv68elvIInzIw9NVLJxVujB+h1xQNvCm8j1lgZIY2xQtqC0qB5Q6BuB1hw yiGOaBOhAEi4uIE4xWUu2DUCuFvC9IUCaVH2VQvqY5Zdxc7xUzaL9CaOTkPN/KXPOl3u Tipg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=ACLcwwiY; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id 10-20020a170902c14a00b001c9eed814dbsi247624plj.562.2023.10.18.10.16.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 10:16:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=ACLcwwiY; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 7105C8028530; Wed, 18 Oct 2023 10:16:11 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230233AbjJRRP6 (ORCPT + 24 others); Wed, 18 Oct 2023 13:15:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59524 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230316AbjJRRPw (ORCPT ); Wed, 18 Oct 2023 13:15:52 -0400 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2084.outbound.protection.outlook.com [40.107.92.84]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 82BA8122; Wed, 18 Oct 2023 10:15:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mIW9R9BBl18osvuFk+n1VHj2eljqG0k7GLTWfGTNtmM14TIs6iS430Z4pDLYp4Iry8vgKIuiFi0XBrHAv0wuaRGxtiotbtBO51aG87FA9JioVbLOCT+/VnccQu//LgLfNkn/IdNKTxuN9gAdB0pY4hSc1mKLKPoTNkqhbnzoINotFjFCkoAwyLM+e0RG+3EVGjqXoxu3oZBKLCfQZC/Lw24L+wJFbjUQYjwqwpGN3q2RUlSKYx3Y5l2w1LmA6CgtnJpynzyqw8l1/b/fIR60UCS53HxXkWLJLSxBOoAmKBlLIfU/m9JCf1oPx2dhAZh0Q0QMnNeGUvYaadjZ15JGyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3STa7lEVTN260CF+7a2Ikvh2pPY1BfsD0f/6MUcbh0M=; b=MqDDn+gFn3vK8G4/dHRvfpHK+IkNEgpES3r12ERHaG8B0sulcrkmr0mQEp6qnCGokchKf0PhVcVuQhjNDUsCSDIH+NXp7qGQ6kEv9QrEolewowSyWMVU/6Uf/IMzVEFVlEz6fLfLdaT9FxkkeD4lL/84EuYFKW1xg1igv6y4PA2wv1a8Ry4zMVlqk76sz2f73624LNcdBKT9T69aYGAzsYgY+PEL2k033SmLAUoGHFfNlIjdroAEbCMamfwBuFpwOLq7s6V+bKiOndrLmbjKNnvrFuTL9yk9+wrLPt3hMw+rOsco/oHx/BRkZgKo3ktcYUbYXE/l9cNuT9Dxn5niyQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3STa7lEVTN260CF+7a2Ikvh2pPY1BfsD0f/6MUcbh0M=; b=ACLcwwiYLdR228ZCNNxR+3D1vsKViBIq9Zeak19PEkO6dAeWZBseBXX27S/CjKKU2klwYVYhzkgK2UJHcHXF3ZCL2PF3r6/WDQxnc0C8OXspikeYgKz61RIzOPBrVWj73IhoW7eifKnEsbpUYgz8S232E3OM7gpOSQ9tDUXHLrSuqJ9AKfprkHWW8YMdVah3gRxp5egvxgkRqfJ86DkGEky2JhSrDoo37iXt/WJREgK1O9GgBr34wydTyqBxLgmeD+QO9PX7NaBTs+xQkfX5gUqARiffbNp0O70uuNYRad7bvbszmIXT+EWOlYvVyTfFm8rAUmJ2dZe2egJ0Kxzigg== Received: from CY5P221CA0158.NAMP221.PROD.OUTLOOK.COM (2603:10b6:930:6a::11) by SJ2PR12MB9085.namprd12.prod.outlook.com (2603:10b6:a03:564::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.36; Wed, 18 Oct 2023 17:15:48 +0000 Received: from CY4PEPF0000FCBF.namprd03.prod.outlook.com (2603:10b6:930:6a:cafe::d4) by CY5P221CA0158.outlook.office365.com (2603:10b6:930:6a::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.34 via Frontend Transport; Wed, 18 Oct 2023 17:15:48 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000FCBF.mail.protection.outlook.com (10.167.242.101) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.23 via Frontend Transport; Wed, 18 Oct 2023 17:15:48 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:31 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:30 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Wed, 18 Oct 2023 10:15:27 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo , Gal Pressman Subject: [PATCH vhost v4 01/16] vdpa/mlx5: Expose descriptor group mkey hw capability Date: Wed, 18 Oct 2023 20:14:40 +0300 Message-ID: <20231018171456.1624030-3-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231018171456.1624030-2-dtatulea@nvidia.com> References: <20231018171456.1624030-2-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000FCBF:EE_|SJ2PR12MB9085:EE_ X-MS-Office365-Filtering-Correlation-Id: ac40035a-faac-4543-270b-08dbcffddf5f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: QOeiaaqTThJ/Q7gIK6eUCE2G7m9uBODGxXLOJPtQn+SIL9lG/SBhEtpSFpSPZazyteqFavMlte/7Ft4l7kAbppt2ic5Y10s2bw7EcMHyyyo20YaRBU49noeRQySYMHRBPEGQ42tbYzi6oWdOH0DFXUSzwerUL/yOrr7h0tf4er5SNND3+CgwWgmeyqu1u+QUa+W16wmmkADoCCwzWFHaRKXQCtMvNFRIaxIQx8bKohigeMHgqv5F5BHyfnqK1PV7bkwZsXCLIpl+8lNOJGs8JQHKx/hK/1pGTGMYl4VxBPeV7pM/I+Vb7IyP1/wLtuwy6V6Fsf3/BCa/ICk01VJ0xScfXTO5YixOCg95Xv6sJDv4qiDrV+na8ARmN6JxwqlervqnYy+jWlG/4iDB/2ZIg/EbL8CvFhfeqTL7On+kJTYPG2usIObW4FbVZk5XA9XFsYb5BF9Ykt8fzOH1Er2zgOf0siuafIbOSZvRrre6X4jXD7d4oWuahApwCO72/AWaaip6Itc+PAAfUkYnIJuMSntWbXGd6zfuupr97MwPU7Oh6FagBKR3iUs4Jc/kCt7T25wRN8kMEeCKt9/KSEQ1dhqQiMiBUWJDTO9y5q0uGqEHwt6meRULX9X+0k48I0EnxiAldpSWqmljMhMiq/Y86qjh38Ga+kBg78cWxKH9BjxvlbTmi/Saq5nb0qNrCt5gtNiOZJ71AmVA9RpexEMukzCYAgIgyDL5eJymNI7wn2Qf7ezW/RviBKKVESy+kKJ7 X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(376002)(136003)(346002)(396003)(39860400002)(230922051799003)(82310400011)(451199024)(186009)(64100799003)(1800799009)(40470700004)(46966006)(36840700001)(1076003)(478600001)(107886003)(2616005)(7636003)(83380400001)(36860700001)(26005)(82740400003)(356005)(47076005)(6666004)(336012)(426003)(41300700001)(8676002)(40460700003)(5660300002)(4326008)(8936002)(40480700001)(316002)(54906003)(4744005)(70206006)(70586007)(110136005)(86362001)(2906002)(36756003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 17:15:48.2881 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ac40035a-faac-4543-270b-08dbcffddf5f X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000FCBF.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB9085 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 18 Oct 2023 10:16:12 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780114407691435956 X-GMAIL-MSGID: 1780114407691435956 Necessary for improved live migration flow. Actual support will be added in a downstream patch. Signed-off-by: Dragos Tatulea Reviewed-by: Gal Pressman --- include/linux/mlx5/mlx5_ifc.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index b23d8ff286a1..6b6be1a0d575 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -1231,7 +1231,13 @@ struct mlx5_ifc_virtio_emulation_cap_bits { u8 max_emulated_devices[0x8]; u8 max_num_virtio_queues[0x18]; - u8 reserved_at_a0[0x60]; + u8 reserved_at_a0[0x20]; + + u8 reserved_at_c0[0x13]; + u8 desc_group_mkey_supported[0x1]; + u8 reserved_at_d4[0xc]; + + u8 reserved_at_e0[0x20]; u8 umem_1_buffer_param_a[0x20]; From patchwork Wed Oct 18 17:14:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 155008 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4945155vqb; Wed, 18 Oct 2023 10:16:33 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGn+0IVa5ayVLZ8Tw01aly08SjBTfTnT4t1nt9prBinhKfiu0Lc9/MAjByqi55Z9cge610/ X-Received: by 2002:a05:6a00:2395:b0:6b3:55fd:d851 with SMTP id f21-20020a056a00239500b006b355fdd851mr5348063pfc.10.1697649392929; Wed, 18 Oct 2023 10:16:32 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697649392; cv=pass; d=google.com; s=arc-20160816; b=pwOfgcezOhZ3/MvPqSTDVUAq6gbxCB6JYhtejB3KDiZMtBBr99MKHzTKuw5Z9CQsZs wGr+5Q4PIRwJl1Y8jZh+awikFO+DJMUyfi+SdykIOx5CYXo07cyVNGch4jI3ML1lDhHc rub4Xacoy0QIGCLarjBLxoAHppmfojtme3JGDZ32qVbWh25UcLyrxpqEfy7DW63c4LRf YdQYSBctEmk/bXP4iLomVf0nAFtZsf4Hkx1JPceIo1uCpSEyZPUcDQkI1Iur4Rth7+uj rh7RC/3GJVn2nWP+efayLRqKt7VXLkhUT098JUrLYVakwH53Z3bPNZ6h000lK2xosuQI aFjQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Zs2p5Im8kifm/aJ1lEAfNpIB6nh0RoaB6CA/cNIHKDA=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=g/Pr/sK9PE5Hq4ZDiDv6EC4WFkHUh9jmgprFrolYUo/2BufgEQec7vLu8bG3m8mHav Z4Ohtcl3OtBT7lDJ1MnYwF/jB0eX5Gl1BCcu50UDbyNMiMiGr7CI2Qf3fv1s2EAVlSib Evcd+jhR4eEAoavgWLdPu78/e46rXAwVWiS6U2R2Og5qaFdZ92vAKxK+wMk2Kgw9w6uc fF67pjEcIGqvOeEVzJHWN5H7lTtyw1pjgNMC4ZJs8HTFvYpluE+jCWW3GjIjU/6xVUWm 22lA/1w/rZLukVAsbA41JfR0dBKKORC2MsbnDa+kbZ4RLGrGttz7CFCWFtRVR1WwBM5+ +Ufw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=lZtmCB4T; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id w8-20020a63f508000000b005ac2d06099esi1608970pgh.478.2023.10.18.10.16.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 10:16:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=lZtmCB4T; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 5F6BF81F8437; Wed, 18 Oct 2023 10:16:30 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231542AbjJRRQC (ORCPT + 24 others); Wed, 18 Oct 2023 13:16:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230018AbjJRRP4 (ORCPT ); Wed, 18 Oct 2023 13:15:56 -0400 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2055.outbound.protection.outlook.com [40.107.243.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 141AAB0; Wed, 18 Oct 2023 10:15:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HmJYR5A8aNs1/qlTyBiaF8H4uFSbrnjmuVtRv82VGjOX2Cwwz/k29fdJoVxpm0bzkN7Wc2BjBdA5D4NcXm2tZJKgvnggRK8/slYp0mKA2rj6eQoVgadwEEHPch/N6WrOfG2icEaFKCDI22nXKkNzW4NbCv6NqnXFQhU8dfhlCCJJQCDzxTfVUnZdRpH1Zsv+Mhtb3l05SIRWr9MxJmh/Ay/4if4rSmoK5qSQALk1u0LdmtslaeAv3R5Re8jFGHfhoQ0FirME+QJGPjJPdOHDungUGqUf9gyc+iixkqojx1CeWaCPT3Y/yCWk62Uyidtumjfu1HRPPbDV2rA3GuddMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Zs2p5Im8kifm/aJ1lEAfNpIB6nh0RoaB6CA/cNIHKDA=; b=YJEZzKdlqNSSVBxtOeftWp6F+IYm1anjd92kkxaOW/AXMjMJwMt9TqlYWudhGkZA7niWOYCAAIWDMnV1UB1tnc0O4erb+dv63Lx7RvFSG0X757Y95aKOm/a6Lfz4Ael8pyXY5a+YiNx/Qnu96LCFGw/a/rW0dNsMqxWUgdVoXIt1NlcAZgDEA60LECVVRajhJK6YO30VxLaFsLR/xTsATwzMy6Li9RtY7PZvt5LgMG7MJiLMWk1ZfM95t/mUtTJhHpagLOAe9l7fQdPkeEYE75bcNb2pGC5i27dv2EQZnVUe+daOcVb7wVs1qUNh0VupBvcXkuHQ3IV7XbyKz8K8eQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Zs2p5Im8kifm/aJ1lEAfNpIB6nh0RoaB6CA/cNIHKDA=; b=lZtmCB4T5+OptN5sueNbgxReNPY/IQnUGw0wFPIs5WOBBWDpZjE7X23CDZ85Skzl28oXsIHOprOjB4Tmsehi7aVz0HKBcdm1JeTMtuHfu/LdKLkAbJNsVQnoG/4D4RGpdZUQvO7QnzWpy1FL9rCvSp6MoOSCwtQtR2DTl3e85/HdPWlvuqMzr55qWAxnbF7rDBbpIffNzmwpH5wjQM/9e199i2+P0BHDHa5ljC2spNSvL7EHVuCBiA79s/d3b5wG9DnVI5z9/0WV2VEFyhurZHcu51sW632Oh63nB8oovDoIR5Bv5AW1ihmJ6mKXNZQrt++TyXPFyrWpOzSeYNf6wA== Received: from DM6PR08CA0025.namprd08.prod.outlook.com (2603:10b6:5:80::38) by SA3PR12MB7857.namprd12.prod.outlook.com (2603:10b6:806:31e::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.36; Wed, 18 Oct 2023 17:15:53 +0000 Received: from CY4PEPF0000FCC4.namprd03.prod.outlook.com (2603:10b6:5:80:cafe::79) by DM6PR08CA0025.outlook.office365.com (2603:10b6:5:80::38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.23 via Frontend Transport; Wed, 18 Oct 2023 17:15:52 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000FCC4.mail.protection.outlook.com (10.167.242.106) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.22 via Frontend Transport; Wed, 18 Oct 2023 17:15:52 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:34 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:34 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Wed, 18 Oct 2023 10:15:31 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 02/16] vdpa: introduce dedicated descriptor group for virtqueue Date: Wed, 18 Oct 2023 20:14:41 +0300 Message-ID: <20231018171456.1624030-4-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231018171456.1624030-2-dtatulea@nvidia.com> References: <20231018171456.1624030-2-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000FCC4:EE_|SA3PR12MB7857:EE_ X-MS-Office365-Filtering-Correlation-Id: 6355522f-8b05-4ef0-9aac-08dbcffde20b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7XFUZzhXzDbegIFEx6hdNFO0fLI2aLS3CYZ4xCZvBC5vHfK3xXIa96JBSfvpD5nBdL8A8s/L57XYbkDtf7s/qTR9/bFIDOinS7CnoI1nQl0cnIqqGrR6IccEBtLtoQ8JNn7r9GBIfyA3qm+J7gpUF2NW/I2ohMivzAv9v3smwNlAw/f+Py7M3rxW1uydQmUYuFK+sw/qomkiEi6u+8OI8leBNXU3DbBDlN7DTZcuprCjemyOe+hQv6p4Bzm9IpOD+meIz0DPsKLBvXpi+/TABKol61P9gjKODdFOtf4zu6QB+gQA4aQYQKD6w1kdY/zUFXejcaU7Kg0jMUspb3pyCBvPMe15omkgOqX+DQGF9zm/dCwQqWKZ166PIMXoJNigzApHYiMg8SfbnqBp7pVnZyK8uKSFOLydffeT7+T/elemQdLw3gqpRS+EzUsOX3zzpCUF2MsYLfgK7ajqAoM34DwBfUfwdlo8+Qt9zK1+hM0tl0fzWoNupc1On6F2jKflgPBxcqAvCT9V8GRx4lT8YN4kjXKz5vMntVWHCR0k/h8RY5ck0u9ZVRDlHm/zxMZobRZlOgSoH3lK9ERFdxUez5bKdFH1Ymn8zCifTnvG58+VHN/o2e0q2hIhNWjzxsR/WKIbRZPpvMnnEtpfZ/vhlYV8sX55bzZFgJ6k2UGTzKRprzPybCjAlrNc/2L0kyrhXTmTiHWhUlEr71Hs5eLIE7nI867xGV7Q5zPTG0nKhNpYVkpL5/6l9TXdB7np0J8n X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(376002)(346002)(136003)(396003)(39860400002)(230922051799003)(1800799009)(82310400011)(186009)(451199024)(64100799003)(46966006)(40470700004)(36840700001)(40480700001)(40460700003)(36756003)(426003)(110136005)(316002)(70206006)(7636003)(86362001)(70586007)(54906003)(82740400003)(356005)(1076003)(47076005)(83380400001)(478600001)(36860700001)(8936002)(336012)(26005)(6666004)(66574015)(2906002)(5660300002)(41300700001)(2616005)(4326008)(8676002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 17:15:52.7737 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6355522f-8b05-4ef0-9aac-08dbcffde20b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000FCC4.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR12MB7857 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 18 Oct 2023 10:16:30 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780114410048568712 X-GMAIL-MSGID: 1780114410048568712 From: Si-Wei Liu In some cases, the access to the virtqueue's descriptor area, device and driver areas (precluding indirect descriptor table in guest memory) may have to be confined to a different address space than where its buffers reside. Without loss of simplicity and generality with already established terminology, let's fold up these 3 areas and call them as a whole as descriptor table group, or descriptor group for short. Specifically, in case of split virtqueues, descriptor group consists of regions for Descriptor Table, Available Ring and Used Ring; for packed virtqueues layout, descriptor group contains Descriptor Ring, Driver and Device Event Suppression structures. The group ID for a dedicated descriptor group can be obtained through a new .get_vq_desc_group() op. If driver implements this op, it means that the descriptor, device and driver areas of the virtqueue may reside in a dedicated group than where its buffers reside, a.k.a the default virtqueue group through the .get_vq_group() op. In principle, the descriptor group may or may not have same group ID as the default group. Even if the descriptor group has a different ID, meaning the vq's descriptor group areas can optionally move to a separate address space than where guest memory resides, the descriptor group may still start from a default address space, same as where its buffers reside. To move the descriptor group to a different address space, .set_group_asid() has to be called to change the ASID binding for the group, which is no different than what needs to be done on any other virtqueue group. On the other hand, the .reset() semantics also applies on descriptor table group, meaning the device reset will clear all ASID bindings and move all virtqueue groups including descriptor group back to the default address space, i.e. in ASID 0. QEMU's shadow virtqueue is going to utilize dedicated descriptor group to speed up map and unmap operations, yielding tremendous downtime reduction by avoiding the full and slow remap cycle in SVQ switching. Signed-off-by: Si-Wei Liu Acked-by: Eugenio Pérez Acked-by: Jason Wang --- include/linux/vdpa.h | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/include/linux/vdpa.h b/include/linux/vdpa.h index 0e652026b776..d376309b99cf 100644 --- a/include/linux/vdpa.h +++ b/include/linux/vdpa.h @@ -204,6 +204,16 @@ struct vdpa_map_file { * @vdev: vdpa device * @idx: virtqueue index * Returns u32: group id for this virtqueue + * @get_vq_desc_group: Get the group id for the descriptor table of + * a specific virtqueue (optional) + * @vdev: vdpa device + * @idx: virtqueue index + * Returns u32: group id for the descriptor table + * portion of this virtqueue. Could be different + * than the one from @get_vq_group, in which case + * the access to the descriptor table can be + * confined to a separate asid, isolating from + * the virtqueue's buffer address access. * @get_device_features: Get virtio features supported by the device * @vdev: vdpa device * Returns the virtio features support by the @@ -360,6 +370,7 @@ struct vdpa_config_ops { /* Device ops */ u32 (*get_vq_align)(struct vdpa_device *vdev); u32 (*get_vq_group)(struct vdpa_device *vdev, u16 idx); + u32 (*get_vq_desc_group)(struct vdpa_device *vdev, u16 idx); u64 (*get_device_features)(struct vdpa_device *vdev); u64 (*get_backend_features)(const struct vdpa_device *vdev); int (*set_driver_features)(struct vdpa_device *vdev, u64 features); From patchwork Wed Oct 18 17:14:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 155009 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4945178vqb; Wed, 18 Oct 2023 10:16:35 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGkSufQ4Tlj25hkyzU/bbNQDD4gvRtPfsJ+dkpWr2GHRKfgW4VnjR2iQZFukTQs8ysU4Zo0 X-Received: by 2002:a05:6a00:22cb:b0:690:c75e:25d7 with SMTP id f11-20020a056a0022cb00b00690c75e25d7mr6679232pfj.18.1697649395046; Wed, 18 Oct 2023 10:16:35 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697649395; cv=pass; d=google.com; s=arc-20160816; b=P+O79QUGa3EX8LmG1BcDaLd4g0DTT9aoH6yFToneNv2lfwaigk2wSgj+57F9nFczax V/lukgzLOAfqsWbdzcTnvShcXclS+S5FVXOgnwP2Tma3qd3XR+huk0RYx30hb/LLK++w Z8jaLqAeEK2f8A8x41cKhvleKh8hpCTM6I8FWmS3/oq/sh51O5CTvtfSZNgFKS4YgftB HAk5/kJRNxRYgDEx9QHESXEueS/Ta0U9BJj6hBgVjvgBOC/8uypr8iyZUvQQ1P7hicGe ev5Xf3nMkgIh+u3PqyuaMaKcwpXe28J/B9MyJYxGTwlR6jracXK0tZxK/Q29xLc+Y/uj Lo2w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ILKog4btT16IMu9HXAHH95fAm+8LcOyK/4mzDrViAvI=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=uVGT4F0UfDns5Yv/bWrtFTyZhjS2rM2mE4lGlOcL/G1PPAdgOr7q1htsrjD5kcxmTg 6f1L4orhwbO1xnWGHY3nvwntf8hGYwGEht2Jr9CDgZQZ3xNDn9nnU4z/Zuqel1eUpyXs KR5dE/UeaUZVM4cLgXQWvzSnwpz8XTX0rqPnkyF70R8RjlpwuMH2Koc7N+Nut6+NaFxp mIMj7IRp4vfH+2TQl+sM8H5PtGr38WLRE6T6J6sVccrofCQhJgRPROhMLfrhyCcE0l7e QjbAn/Lwz2L6+9J/hudP+b0X+M+4k0sn65r/RZVC3XzSrDaddEwBWaaPMbaBycgrSJwe JVNA== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=InGIHWcY; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id k11-20020a637b4b000000b005b7180bc067si2714161pgn.17.2023.10.18.10.16.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 10:16:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=InGIHWcY; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id AE04680BB6EF; Wed, 18 Oct 2023 10:16:32 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231293AbjJRRQM (ORCPT + 24 others); Wed, 18 Oct 2023 13:16:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40004 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231417AbjJRRQJ (ORCPT ); Wed, 18 Oct 2023 13:16:09 -0400 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2060.outbound.protection.outlook.com [40.107.220.60]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B6D0111D; Wed, 18 Oct 2023 10:15:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Mj+rARdtwp+7+AS7q0gsnEM7x6jV28yYYxzfxgOqv+3BLduCa3u+1lHjyhG3//I77pp3PF1zEwuKrJaH5i3RVnSk4yzUcC+tyRAfnkr+gDDgKluPEWmhgAQn4z+wRrKiHxBJ98/6RR9XvCaHaqi+o8M7m9SRadoRaB0AO8ZfncMwXrWURxFjRRO9TvlX7JE3dicME6d2AVILB2LvVAXMMkE15BTJ5zOJfAI/qs7MeOGlK+0iNiUNf6pwQ9gqJXyVwkuYqshGljhmhLoE526D08aEIZV6uRvFu1MWRR6CSjTSX8Lbpopvt1m5ov41oeCRfpQaedWf92n9k6kMbBj+0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ILKog4btT16IMu9HXAHH95fAm+8LcOyK/4mzDrViAvI=; b=KawsJdcxemMZiWwqT8Qd5a6nUVhd60FhQiJuuNG0bpmmUeJPA5pvEiS1/ZCSNsJ7EpfdJoxAyCWLP+AG9ATLjwF+Xf+AQr8LtUyjmLf+Uu0BCpSHXY09jS5MMLfVWotN7oJX9hoI2Eu4gfoG/25p7x4+4wl1I+26KX5P7iqI6JoKo0mktb1qcp5YlO76zs8KCl8IBsdtr/fXTlRHief6xmQ7xi95uag9wpm2JmHrMYH4eQlCkKJWFUfgUGQS/MxMJ+vUV6d3TpBSY6Lt9H/mm/jh71olwjpHn2/Glypn+MuiRTO0tC2vFuNQdhXG3PqaOiBcm8ql8+H2BF7GxUaHlA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ILKog4btT16IMu9HXAHH95fAm+8LcOyK/4mzDrViAvI=; b=InGIHWcYhkZD3T9eUB9/N5njheGFShSXpwrFfviZzFtfOzbYH1ReyM1U2Uvk0CUunTOdpTi7/+B0+hLthT2TEBVfST6+ABJuJ4ijHVqhvLbJqFnyDKd6cIh4oLlEDXlMiey4OuVM6sVTAsjzT+AsPpRWfOyY+0FU1+gb2yQStNZDvS79y0PJkm/nAgLHQKbuIThQaiilBj3uZqj/JadXBi74lybKEiQAH/BT9YOQun+YPlKSG4u257FKlaC6Pl143bVLXlsq7taFyjgOYFn5ETxc22vUl2+t8eWdx6/po8ygjYBID4zEt3OqkVEInpuIPFEG3G1LY1L4OZ7/DbkzPQ== Received: from SJ2PR07CA0002.namprd07.prod.outlook.com (2603:10b6:a03:505::25) by DS0PR12MB6413.namprd12.prod.outlook.com (2603:10b6:8:ce::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21; Wed, 18 Oct 2023 17:15:57 +0000 Received: from CY4PEPF0000FCC5.namprd03.prod.outlook.com (2603:10b6:a03:505:cafe::ac) by SJ2PR07CA0002.outlook.office365.com (2603:10b6:a03:505::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21 via Frontend Transport; Wed, 18 Oct 2023 17:15:57 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000FCC5.mail.protection.outlook.com (10.167.242.107) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.22 via Frontend Transport; Wed, 18 Oct 2023 17:15:57 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:38 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:37 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Wed, 18 Oct 2023 10:15:34 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 03/16] vhost-vdpa: introduce descriptor group backend feature Date: Wed, 18 Oct 2023 20:14:42 +0300 Message-ID: <20231018171456.1624030-5-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231018171456.1624030-2-dtatulea@nvidia.com> References: <20231018171456.1624030-2-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000FCC5:EE_|DS0PR12MB6413:EE_ X-MS-Office365-Filtering-Correlation-Id: 9baee8f1-9f7e-4277-8785-08dbcffde4c1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: fWm1V7o39YwSWK+uHNaqLyxEh4EbQaw1XmxinJNGTLo6+LeXt6NiT2tWbysJj7CXJEai3glQLlZVF9p+5eetmgoFaAuYm1Adyaowpjs6yta077Pxe6VWJHJgoYQhvcXrF/lfRT6pU/VrxcZtwhebRTQeHGvTuzfLw1aZiAFDQx1SIb99swzCSVrJq+9SFcHJL+KawtqrRFvPd6qvGI27b8ISPcN3XknagZM28Xk3fZH8jQas7opMVCuuAJNFiKDCGr7kqfreKT/G8NV4s0nkt7ik1on0tUZQv1JeHMnvw6+mF29nr7286Ee9HP0bgY6OjovQTbqSuyq54DjP/iyoPtcerqtjjIw6GUEM2aK7pdFWri51YVtYXXvfyq7iFQqboauIY1Y//DMNhGBzofOU3XHaL6ngoEIno0UXqNRXEJ4Bqd3eB07kxL3ZWLnDF26MO9LlZQAidrLc/pnlUWod3A/nE8h4QmhGJOc44qdgznmjSUV2J8wEIAwgrmIDNGGPHn6FWO9oNMVeGifIgdOvfy71gqNj49krBsz8uWhf2IY9R3FBNq9aFiTkni5fh/PniGWfnVlgFm8cQqR4q7Gf8T6gYiEoLSL/ezjqmmwZuipmUaC1j1f8RvNFwMvtk2NSXiZMbs9kTbOGB/XdtfBVjMLlL9fIC99JjvLQtuj6lfBr3kkiKnPJgEel70LcYzYZGQ4+z9Wa3T6BXwOXy8l/iI9EMAyS45KzvaiSKi3Iczq/JWGQqINzJm35xeCku/g+ X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(376002)(346002)(39860400002)(396003)(230922051799003)(1800799009)(82310400011)(64100799003)(186009)(451199024)(46966006)(40470700004)(36840700001)(5660300002)(66574015)(336012)(426003)(83380400001)(82740400003)(6666004)(356005)(7636003)(1076003)(26005)(2616005)(478600001)(36860700001)(47076005)(54906003)(70586007)(70206006)(110136005)(4326008)(41300700001)(8676002)(40460700003)(316002)(40480700001)(86362001)(2906002)(8936002)(36756003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 17:15:57.3214 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9baee8f1-9f7e-4277-8785-08dbcffde4c1 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000FCC5.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB6413 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 18 Oct 2023 10:16:32 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780114412110530901 X-GMAIL-MSGID: 1780114412110530901 From: Si-Wei Liu Userspace knows if the device has dedicated descriptor group or not by checking this feature bit. It's only exposed if the vdpa driver backend implements the .get_vq_desc_group() operation callback. Userspace trying to negotiate this feature when it or the dependent _F_IOTLB_ASID feature hasn't been exposed will result in an error. Signed-off-by: Si-Wei Liu Acked-by: Eugenio Pérez Acked-by: Jason Wang --- drivers/vhost/vdpa.c | 17 +++++++++++++++++ include/uapi/linux/vhost_types.h | 5 +++++ 2 files changed, 22 insertions(+) diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index 78379ffd2336..2f21798a37ee 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -389,6 +389,14 @@ static bool vhost_vdpa_can_resume(const struct vhost_vdpa *v) return ops->resume; } +static bool vhost_vdpa_has_desc_group(const struct vhost_vdpa *v) +{ + struct vdpa_device *vdpa = v->vdpa; + const struct vdpa_config_ops *ops = vdpa->config; + + return ops->get_vq_desc_group; +} + static long vhost_vdpa_get_features(struct vhost_vdpa *v, u64 __user *featurep) { struct vdpa_device *vdpa = v->vdpa; @@ -690,6 +698,7 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, if (copy_from_user(&features, featurep, sizeof(features))) return -EFAULT; if (features & ~(VHOST_VDPA_BACKEND_FEATURES | + BIT_ULL(VHOST_BACKEND_F_DESC_ASID) | BIT_ULL(VHOST_BACKEND_F_SUSPEND) | BIT_ULL(VHOST_BACKEND_F_RESUME) | BIT_ULL(VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK))) @@ -700,6 +709,12 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, if ((features & BIT_ULL(VHOST_BACKEND_F_RESUME)) && !vhost_vdpa_can_resume(v)) return -EOPNOTSUPP; + if ((features & BIT_ULL(VHOST_BACKEND_F_DESC_ASID)) && + !(features & BIT_ULL(VHOST_BACKEND_F_IOTLB_ASID))) + return -EINVAL; + if ((features & BIT_ULL(VHOST_BACKEND_F_DESC_ASID)) && + !vhost_vdpa_has_desc_group(v)) + return -EOPNOTSUPP; vhost_set_backend_features(&v->vdev, features); return 0; } @@ -753,6 +768,8 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, features |= BIT_ULL(VHOST_BACKEND_F_SUSPEND); if (vhost_vdpa_can_resume(v)) features |= BIT_ULL(VHOST_BACKEND_F_RESUME); + if (vhost_vdpa_has_desc_group(v)) + features |= BIT_ULL(VHOST_BACKEND_F_DESC_ASID); features |= vhost_vdpa_get_backend_features(v); if (copy_to_user(featurep, &features, sizeof(features))) r = -EFAULT; diff --git a/include/uapi/linux/vhost_types.h b/include/uapi/linux/vhost_types.h index 2d827d22cd99..18ad6ae7ab5c 100644 --- a/include/uapi/linux/vhost_types.h +++ b/include/uapi/linux/vhost_types.h @@ -185,5 +185,10 @@ struct vhost_vdpa_iova_range { * DRIVER_OK */ #define VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK 0x6 +/* Device may expose the virtqueue's descriptor area, driver area and + * device area to a different group for ASID binding than where its + * buffers may reside. Requires VHOST_BACKEND_F_IOTLB_ASID. + */ +#define VHOST_BACKEND_F_DESC_ASID 0x7 #endif From patchwork Wed Oct 18 17:14:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 155013 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4945539vqb; Wed, 18 Oct 2023 10:17:07 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGTBKXqz0FIA3hozXvM+uTew3oacN4M2L2sfktEdIBTs/ejV5zRCV0Uv0SADu/gik/B4W1x X-Received: by 2002:a9d:6657:0:b0:6b9:8ea6:fbc2 with SMTP id q23-20020a9d6657000000b006b98ea6fbc2mr5621217otm.27.1697649427090; Wed, 18 Oct 2023 10:17:07 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697649427; cv=pass; d=google.com; s=arc-20160816; b=RSF27X/biiup8Irx0slIVbTEc5ujxMCkUBOvMUkxXzMZVPpBUUp6eVMnDg6Wnh0cLN T1cZAsMFNWLJyL+4GnwwmEYfCAFhFRrK5ac5lanwUQyTPIfGLiXWiqRuSoLFvWUjJRqL kp5Vpg5lQCrkMAgufjjiUqrFhwmeUSj9ng5rvSlarYdsVHGt3NSynd8sowtcGAwAZC8v 3v90pFX3YORj9N8SWJCgC8IpPw62t3krqgyme8zGV3Xnn6TWG74ox2O6+7ylOIInVd1Q 5Flw0WWsHYWcr1EmsP7xzt1SzH3SbMcuurmQiE2P8zJz76zxSahfjfmx4LKmBp6B7Y9R ohBw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=u64U1vHaUjCMSpg15sklK0flAB7mos1z6H94Kmb4gsM=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=Cg7fXsqcTD7x31tWLvI3mpJQrRrFxxneav1klFJ87ySHMwAMR2hI9YJl25mWvxEEXc hwgjTKN2CvdOkbobRSUOISeTfZpolcPRMqK6kSsUUDz90hFGDg+pt92JXGpLISz0ONPc bNFLuCUAKBB9zzmsFj5L/sEJeb8xoGHl1u+NQ98zm4Bj8ol9QAOO5WI+dHFDiXD3E8cf gf5f1F+5Pbg8va7OGymqg/mmtdOrJb8joCFwpT5m+uL7Pqno6lXeVRcTsUudF7Cc+Zo/ 8dz7XPOQXJmiR/s/4hN/FaHFXDO1ISXZPdwV2PS0GEa2xfhmHZ4E8fGBINZZyRB/+BEt OE8w== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="Y6hg2/N5"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id u3-20020a656703000000b005ac8d44bad4si2523410pgf.592.2023.10.18.10.17.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 10:17:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="Y6hg2/N5"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id F1B338201B0A; Wed, 18 Oct 2023 10:17:03 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232373AbjJRRQd (ORCPT + 24 others); Wed, 18 Oct 2023 13:16:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40208 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232226AbjJRRQN (ORCPT ); Wed, 18 Oct 2023 13:16:13 -0400 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2080.outbound.protection.outlook.com [40.107.223.80]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB52C18D; Wed, 18 Oct 2023 10:16:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iEj/bxhAwEkg7fQMmRVPgdsoECTbq4OB1JRCRG3nCVeHuTvvjA5eJ4K3woABvaQw/l265MLrxnwDDDyJfu2HM+JKbWQ+z44GDPAKq5epWJG/9dpj+qSSz6dF8hXx7KriTSoGdKtUsVtAbmhRrhYesFDH6fDGoVhCw4WIQd3rjBrzBpxpeKZpwL55+ZSIlrauqqSAJCHm9A6EvHEQRaCVuB04U/MBueeirjDdhSZ/wx/nWYnEaDgpR5WlkS7KejP/Ybib/1+UYGwdTx+uGfOn3vS3YI7/G2GVp7+rLopvOOh6HjGeh7Nnbq8iDILmtRRl2FxoZV96sAr3FhU2cd/Dew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=u64U1vHaUjCMSpg15sklK0flAB7mos1z6H94Kmb4gsM=; b=ObpmFBFUcASwundJf17ouX5HXIq1+hD4eWBaFW1esQ68A9va2SltWqoJ3NfBeaCXIvOWLLMw4QKqxVh4dcpo8RdKgvrngAcRvZ8JTubWlhrli5LLZF6LNI8vFDc70NC1+iB4B325QOOrOET379xvdOnOddXYPyktm1QuHAJ0Ct+yK4buqJ1R4RtZq1eV01y+iFANdsgzTDla6WTeA+KNZdIypKmfGEILE+KvOkNWoM0fgTeHjmEpz1fFyTcQL/qOTwyjnRTpnLaMJ4ULGEWm1CkMDhy936J1e+/K4XzSNRd/oQCEjIf4IiuDDp2SKO92rsw2eKCynmFA12mHJPD06w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=u64U1vHaUjCMSpg15sklK0flAB7mos1z6H94Kmb4gsM=; b=Y6hg2/N54tfKr/Rb30K/PwbSMwKrF1WkPOLVRLcbggk6T+khh/rsG7qE5bnVmI3AUmHY3hX3c118BZ4fwrbi4cVqPZgLrEqe131tbeO1AzSUtT1J0UKZNMM56ojI5rJtb+rrkNrQXxWOFhJamUny+55O3baP/DJy8n471G52U5Ldf6AzwYhBsEDALmLrh07p6pVXrOjIND02YfktKPTZCc8bFX0lAwX8ohjY4JwThWLPgOaELbofTfPJ+PrgvQ7We7S63oCn8QHe80tV6l28DbMooFgiiXmpltyhDuVqsRMvpNpb7pCzFCOGgOq0AgIMLCaVBmabsOq1vo+F7B9vCg== Received: from BN0PR02CA0028.namprd02.prod.outlook.com (2603:10b6:408:e4::33) by DM4PR12MB5373.namprd12.prod.outlook.com (2603:10b6:5:39d::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.23; Wed, 18 Oct 2023 17:16:01 +0000 Received: from CY4PEPF0000FCC0.namprd03.prod.outlook.com (2603:10b6:408:e4:cafe::cd) by BN0PR02CA0028.outlook.office365.com (2603:10b6:408:e4::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.36 via Frontend Transport; Wed, 18 Oct 2023 17:16:01 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000FCC0.mail.protection.outlook.com (10.167.242.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.23 via Frontend Transport; Wed, 18 Oct 2023 17:16:01 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:41 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:41 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Wed, 18 Oct 2023 10:15:38 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 04/16] vhost-vdpa: uAPI to get dedicated descriptor group id Date: Wed, 18 Oct 2023 20:14:43 +0300 Message-ID: <20231018171456.1624030-6-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231018171456.1624030-2-dtatulea@nvidia.com> References: <20231018171456.1624030-2-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000FCC0:EE_|DM4PR12MB5373:EE_ X-MS-Office365-Filtering-Correlation-Id: 87625a0a-753f-4aee-b47c-08dbcffde708 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: e5h0gy9lojRRmXdsIs2ENcNTm3YEt0wrdqIZWse4NkvCh000i12j/5Vkicf4UDSmQFEN++3lXJXpyI+WxUj2tnfpViuA6dvJPxA3kdFvw74S47TNH/e0QI/qBwRA6qasmTCZlSJYYF9GZUmyA+yO6E8zb+73hc1ecJwT36SaKyaMfmS00wlj15vERCNUePwxRMHRcdAORsWVu4TQ8EaRHWdh/utRlVbZ/4tBTruPcyWW24SOK1IFCeQCXiUn9R4+6bIeuA+AalG8UHiZ1DnyZfCUSlVFKeUZ5/sADdH0wZOuJPWQmyB3R3nx6SmPa10444WCR/Tf+dHva4fkyaJGjf+C5frduPc3uLJaSxYJDREbqoWRpxo7oQWF01ueHEiQjgu9DsTnPMmzEPn5le2X7r2XwmhGLbn6yz4urMePczIfQ0bevZhdAzIFkpYalNfxcWEuxMZlm5bXdv76UahnFSM58bu2HXxzBbhuGaiRMg1ohk/66GQbuXhAc7sZrng+OAJL/6jfV70G5uFc4tGLXOnZ4c/Yuv8FGf07JXJsmedBg8PP8d5HbCud3O/ZQ+xwQNNTUHJTr015FcVCBrx/R07uVJ/mM5mcZ2Yw2mOXdljOFPB++9PjkvgpceDR3jxtwJ/aNaLgb3u3sGRPIwTnDER8aTdCqKNDptNeiOMuDjrzsw+vxy4upAWbtk4dg5j0R5GvdXH5ykmOFsHV90w6vAdYVHzUpQLDWBq/DfDV5g36ihz8GknNmNGRRz3u45Ty X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(376002)(39860400002)(346002)(136003)(230922051799003)(82310400011)(186009)(64100799003)(1800799009)(451199024)(36840700001)(46966006)(40470700004)(40460700003)(7636003)(356005)(26005)(2616005)(83380400001)(1076003)(41300700001)(36860700001)(336012)(47076005)(426003)(82740400003)(478600001)(6666004)(54906003)(40480700001)(70586007)(110136005)(70206006)(316002)(2906002)(36756003)(86362001)(8936002)(8676002)(5660300002)(4326008);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 17:16:01.1433 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 87625a0a-753f-4aee-b47c-08dbcffde708 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000FCC0.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5373 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 18 Oct 2023 10:17:04 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780114445676147173 X-GMAIL-MSGID: 1780114445676147173 From: Si-Wei Liu With _F_DESC_ASID backend feature, the device can now support the VHOST_VDPA_GET_VRING_DESC_GROUP ioctl, and it may expose the descriptor table (including avail and used ring) in a different group than the buffers it contains. This new uAPI will fetch the group ID of the descriptor table. Signed-off-by: Si-Wei Liu Acked-by: Eugenio Pérez Acked-by: Jason Wang --- drivers/vhost/vdpa.c | 10 ++++++++++ include/uapi/linux/vhost.h | 8 ++++++++ 2 files changed, 18 insertions(+) diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index 2f21798a37ee..851535f57b95 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -613,6 +613,16 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd, else if (copy_to_user(argp, &s, sizeof(s))) return -EFAULT; return 0; + case VHOST_VDPA_GET_VRING_DESC_GROUP: + if (!vhost_vdpa_has_desc_group(v)) + return -EOPNOTSUPP; + s.index = idx; + s.num = ops->get_vq_desc_group(vdpa, idx); + if (s.num >= vdpa->ngroups) + return -EIO; + else if (copy_to_user(argp, &s, sizeof(s))) + return -EFAULT; + return 0; case VHOST_VDPA_SET_GROUP_ASID: if (copy_from_user(&s, argp, sizeof(s))) return -EFAULT; diff --git a/include/uapi/linux/vhost.h b/include/uapi/linux/vhost.h index f5c48b61ab62..649560c685f1 100644 --- a/include/uapi/linux/vhost.h +++ b/include/uapi/linux/vhost.h @@ -219,4 +219,12 @@ */ #define VHOST_VDPA_RESUME _IO(VHOST_VIRTIO, 0x7E) +/* Get the group for the descriptor table including driver & device areas + * of a virtqueue: read index, write group in num. + * The virtqueue index is stored in the index field of vhost_vring_state. + * The group ID of the descriptor table for this specific virtqueue + * is returned via num field of vhost_vring_state. + */ +#define VHOST_VDPA_GET_VRING_DESC_GROUP _IOWR(VHOST_VIRTIO, 0x7F, \ + struct vhost_vring_state) #endif From patchwork Wed Oct 18 17:14:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 155011 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4945452vqb; Wed, 18 Oct 2023 10:17:00 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHMpR3Q3MhfH13eOS0ElDH7DoKKuuyLxRaBbLVTm6zPrpgUIN0n4b53a3kuklepM49h1vf0 X-Received: by 2002:a17:90b:a0c:b0:27d:5693:7340 with SMTP id gg12-20020a17090b0a0c00b0027d56937340mr5412352pjb.24.1697649420357; Wed, 18 Oct 2023 10:17:00 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697649420; cv=pass; d=google.com; s=arc-20160816; b=A6lo0lEPK3xnbdfyQEbFnMbIS0VpPQdiMP8ut02HLui7AM7Fp6TVuzsgYtShSl8ucQ UAR5/qFbGVS2zvmXeFJhA3D62huGXLvOm51/CTJ897Q8sex2x2V/irL5JZ2x/+UwremG +V+sVaZNK3huM+qya+8rB+NvQ+B2yRL5Dt7ciSkcr0cSvp3S3YRvjoEPRnaf42hvYHr9 5cLyYLh/7Nq/xKZdo8ky6VUj/IGaM4HzEVUaK4AUQ3k48JxvlrXLPml30WgY058DBXP9 hxxXcuVWvn1F8AfXcb7ARsc3GmIt8UVoIuA6xPeSwBM9q7V5O5dtc1uLgwnA4PgEvola mktA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0k7vZDkq8HIAIRoAeeLZ+cqnJ3LV329QdGvZjtWRFo4=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=ZTKFuadjLsgMmPbcUld1t8iTnqWvzcM9lLRZPIdR4h4Wad0/d/xZclWO2R8v2wwrxq bmRV1j08d34fKRh9kQwcbu+AxN3jROr6izqeVY7QBG5SzlvNfSRgenmvCxVwDKmCgIIE ArOJ2jiOY+Va0xSBMu2L2Li2zb0WMCc9o0paWyc+xikuhItY1ZlnBiUA/Jy8C+/GtkES jq4rYtOeDt6DOO1cVEXsKqDj3deTD7434IAKeDeNG6+lsDc8d5r/KALJ1gmSOEhe1Sbu SF0FBd2uAGLoYeQvgW6+5Jzh3lp9I91oRJk4FNynSmBSnOGWxsCyKLZjFP55nehUinW2 +zkw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=FzRWewuX; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id i7-20020a17090acf8700b0026d6ad52473si316602pju.34.2023.10.18.10.16.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 10:17:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=FzRWewuX; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id B205B809E709; Wed, 18 Oct 2023 10:16:25 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232057AbjJRRQT (ORCPT + 24 others); Wed, 18 Oct 2023 13:16:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231963AbjJRRQM (ORCPT ); Wed, 18 Oct 2023 13:16:12 -0400 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2085.outbound.protection.outlook.com [40.107.244.85]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A025713D; Wed, 18 Oct 2023 10:16:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hejk+cd/VCOpaqrh5zb3UhnRhn+2M3nEBAqw6+j9h1bsZ648YACnyHZWM5MeHiR3V3iziFe8TW+eFXO9yNa56ufqLvLlMfGqNefKelXgkMaT1oJTzoZxdIqO4pdGsoVcmL+HYihgozZTP2Ga3xQWmUUBdEjqr1cG4HQGtqNogQaym8Wuq9fkirSaUwusnfEGgzWxjK3uP0bkP19bQu5GPucKYbT3plEdy6nH4rFs9KxYDL+nUFdfH62R2VS1mD47Mehp6rTyfn23UP2Q41wy/D4bD1tXaoXt8hnV11LuDBZ11bdNY5m9Whrz8oX+lLPCghOBe8EXu1oaA2LkBSyB4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0k7vZDkq8HIAIRoAeeLZ+cqnJ3LV329QdGvZjtWRFo4=; b=j1uPbKCiwfImYhOSjvvFgrpHuJJNiUo4KQpbjm7GcLrojsNTLCk2HJPy+AZH5DnEH+G45F75Kca1MM5iEoqRD1U7q8m+zdQRI8h5qVCQOqroUaLidKvsHEzlLL4DjdrtiwufAJvaoN0NOqo8m5VDPH8vExAJ71Eko2KWwq7Aqb3Z1HM7P4/bohjof/5QLQkGHICxJnLBwCe1216/9n76hJKgNxEPIaazidMhtXdJtaXug75F4/gBk0PT7bSPDrg8+UcmSi0GD7pKqxtQDlBzFeRaDZh8VzELgZx21xcbtcPwzfoL6/sPmD/cFcYguS/Hg7oGQpKiWXcYmesDOQ4/yg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0k7vZDkq8HIAIRoAeeLZ+cqnJ3LV329QdGvZjtWRFo4=; b=FzRWewuXPIGy36nHV8vbHSPZ9td+1WYwWUX5J1OySLuRmlLr0pJclBYCRbr4rLdyCBSobIeEIePedrQf+NllekfC6T7+KgjqVpLqBeT7Z8cehtvmQvyO8Dk7XaoN+KO+HNgZyMmy6aY0XniwFQt5UyAjZEdzR2rRSfa1VLoRjHNum97fMngtqmSmfpp3lfAOOaB7UI8EAzDjJqGuAqpMcNfE2CAyWUPa5IUXh+kfwgQ/gXj/s9hjFX64vhnoZ2bcTLdgxvT+wOXS/nfYV2IGO3QnjscZCyVF0HGv1XksBsxrwojwEncsHrAzU4wsOqC2rcLlBiyLxZ45jWG+hTKSrA== Received: from CH0PR07CA0017.namprd07.prod.outlook.com (2603:10b6:610:32::22) by PH8PR12MB6771.namprd12.prod.outlook.com (2603:10b6:510:1c6::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.23; Wed, 18 Oct 2023 17:16:01 +0000 Received: from DS2PEPF00003443.namprd04.prod.outlook.com (2603:10b6:610:32:cafe::1d) by CH0PR07CA0017.outlook.office365.com (2603:10b6:610:32::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.35 via Frontend Transport; Wed, 18 Oct 2023 17:16:00 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by DS2PEPF00003443.mail.protection.outlook.com (10.167.17.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.20 via Frontend Transport; Wed, 18 Oct 2023 17:16:00 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:44 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:44 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Wed, 18 Oct 2023 10:15:41 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 05/16] vdpa/mlx5: Create helper function for dma mappings Date: Wed, 18 Oct 2023 20:14:44 +0300 Message-ID: <20231018171456.1624030-7-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231018171456.1624030-2-dtatulea@nvidia.com> References: <20231018171456.1624030-2-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PEPF00003443:EE_|PH8PR12MB6771:EE_ X-MS-Office365-Filtering-Correlation-Id: 91f6820d-03c1-4951-adf2-08dbcffde6d1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /op12pZFBzBslVm2Kacjtwul5qfXbh6qy+yXfPkdP+hG8+w0Dgz72vAmz/Risq4OtSpMAM/8rSg5RpFfwgierH8DByqQgUfZWBTS6Eevb+YSkIggGRvc6BgbQrouGfWJx3ZsG8hnd1DBMmg1OaWf9SjG0YRsOg+KPgWvzELV5ycXXQBWvZFyLgim5+NW3s4/JZCJ+kaBRRHU2nluq4yRxKRCrBhkbCe4DQUkzc+6WC4o7VVzdj2c9V/IKnktWsZthl9tHRK9RHWWO9U0f0ntH+6igbAyBZjW2wda8YAhWTj++HOmfPJixvo6M+8V3hgIpv3e6wBx/iRbEyVa/PyD+Q6UoSc5Py5q70/xIWE2LbU8W2t14pGR8IZ6OpCXUMRNm46ZmFmO8laDScdKFRDzmC+/YO3hG4bMhNA7GBJFMtKVKT0X8gJZn542AO/wTh9xDWonHbHIPypaVLIjCiEsnUMYUbHL52hsF6eA6uXQa3mSE23WrLhs7Gi3x2rnSK0f1yANOSKiWBzDFFyyLwCs43jniNq6xC6EYDU3xY34KI1n1I6Xyx5uxLh8l5AwUcVGu+2fJ525q8lg8oRPDi3TkRLCKbXbK6lf+u30Ghj//1Q08EW3IjM/WMsKvPKm7+Wws52JN8Y6xujb8st2wNaXnviHutHF8PTGKeyvRM3vNATHPo8f4ufbRx4PXMRQh+LCiVMZtcuNl4stdZe+RloZUQzLxKjPcRoYVfY6zq7qyAE= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(376002)(396003)(136003)(39860400002)(346002)(230922051799003)(186009)(1800799009)(451199024)(64100799003)(82310400011)(46966006)(36840700001)(40470700004)(83380400001)(40460700003)(40480700001)(316002)(36756003)(2616005)(86362001)(2906002)(110136005)(1076003)(41300700001)(5660300002)(8936002)(70586007)(4326008)(8676002)(54906003)(70206006)(478600001)(426003)(336012)(356005)(7636003)(26005)(82740400003)(36860700001)(47076005);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 17:16:00.7671 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 91f6820d-03c1-4951-adf2-08dbcffde6d1 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS2PEPF00003443.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB6771 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 18 Oct 2023 10:16:25 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780114438743851745 X-GMAIL-MSGID: 1780114438743851745 Necessary for upcoming cvq separation from mr allocation. Acked-by: Jason Wang Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 1 + drivers/vdpa/mlx5/core/mr.c | 5 +++++ drivers/vdpa/mlx5/net/mlx5_vnet.c | 4 ++-- 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index ca56242972b3..3748f027cfe9 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -120,6 +120,7 @@ int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid); void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev); void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid); +int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvdev); #define mlx5_vdpa_warn(__dev, format, ...) \ dev_warn((__dev)->mdev->device, "%s:%d:(pid %d) warning: " format, __func__, __LINE__, \ diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 5a1971fcd87b..7bd0883b8b25 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -619,3 +619,8 @@ int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *io return err; } + +int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvdev) +{ + return mlx5_vdpa_create_mr(mvdev, NULL, 0); +} diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 40a03b08d7cf..65b6a54ad344 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -2836,7 +2836,7 @@ static int mlx5_vdpa_reset(struct vdpa_device *vdev) ++mvdev->generation; if (MLX5_CAP_GEN(mvdev->mdev, umem_uid_0)) { - if (mlx5_vdpa_create_mr(mvdev, NULL, 0)) + if (mlx5_vdpa_create_dma_mr(mvdev)) mlx5_vdpa_warn(mvdev, "create MR failed\n"); } up_write(&ndev->reslock); @@ -3441,7 +3441,7 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name, goto err_mpfs; if (MLX5_CAP_GEN(mvdev->mdev, umem_uid_0)) { - err = mlx5_vdpa_create_mr(mvdev, NULL, 0); + err = mlx5_vdpa_create_dma_mr(mvdev); if (err) goto err_res; } From patchwork Wed Oct 18 17:14:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 155015 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4945636vqb; Wed, 18 Oct 2023 10:17:14 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEY0HPaGCDI6Wi2qOqYUV5o1fuiGP4XmkfnnBIppN7tRHyVhulwNpiBr4AyoYk+Wwnxd2cK X-Received: by 2002:a05:6a20:8e28:b0:171:e3b2:7d52 with SMTP id y40-20020a056a208e2800b00171e3b27d52mr6474537pzj.59.1697649434074; Wed, 18 Oct 2023 10:17:14 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697649434; cv=pass; d=google.com; s=arc-20160816; b=yQH0S7FtJKXOq4kHpDXf5F6lPAiEmyKURTSvqK9q39N0tw7c8CRxf1S1pFoXkOIYdX KDdaWYAUraoXjKJIEHyKe6bRdKCma3H99dB3GnJ+kfBL6DNQkSbcNdAXaSCf3pMPjsJe G+0MMBXkmNxgjQLTOoZYE1wY5Ksx18NHNAlBcaKWxWuDQYOU1R1XJWPJffdbrcnIvQtR XIXBotXr13LmqV0ggnpgt531IVA7KKRv0/rJJjOST5ShLWW5oKyrAoL6qOtKtDR1y9Tl ymKCdy6Yc4aXy7Tb/bwkIe4YEQLsAz0sYFR9HxZSo9fuOXGH3ClmSbVHVXZ4uCHY8urN NxTw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3YoDCHkpinOQBJC5OCWVs0dAVcWKDrjzxMHgzD+IBkE=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=ILzRK6NgA5zWgLus/SnkmnjaFWYJCzN/VmRDjQAiov72Mr5ziSHNRE2s2NYZUFVwAo SwJ7fB2Xhvorrk5Yc6pJe58/E9YjJxvcCNfdLw0urW/sBfylXWpkxgRc+ALbJZEjvuQT F1qqlhr7wOTGnjtJJivzWZhM9yOeJOsKmR9PJDFNPwWCdRLh0Lu3wJWH4vRP4tnd9+ly rGFCylnRt207Ayhd/ssO2enba8GVfXGLMaHnaPvn9hndUh49Zbl0ls11ggfgC/mzECa3 Qnw3VmFg5t/UiMZiIg4guNbHOijI4vlh/QN+Vpis4rKLVuaP7bH7qD5qicrdrFe3rCCz 1t6Q== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=UH6KZI1l; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id b5-20020a170902d30500b001bbcddc33dasi291665plc.180.2023.10.18.10.17.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 10:17:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=UH6KZI1l; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 8A64B81F841D; Wed, 18 Oct 2023 10:17:11 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235224AbjJRRQz (ORCPT + 24 others); Wed, 18 Oct 2023 13:16:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231837AbjJRRQO (ORCPT ); Wed, 18 Oct 2023 13:16:14 -0400 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2053.outbound.protection.outlook.com [40.107.95.53]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DA085196; Wed, 18 Oct 2023 10:16:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FzURwtcjCtj1tUS6jQQu2UZP/rQ/wgpFf1LES2ttIrN9GmB2/GDYFPq9MIrhIVZdCAEdSptw/Xy4Y6Rq6fanGcuhGNL8ZI6yHyZdU4/u0/8NUrATJ0FoEtNENbgQ56Ujkm7IlYD93PsMnEOVUExhsu60bqy1TpLmm+5E2hCysGg1DH9eCJChNZUZoj7Daj7rPkf6qcl+QEK1s6s8EUjOemvEyNxnO+TQZbU50qa9P6otVZhHUCOXF6nqvTsJAPE43vi8mSE8Dxq9kv5GP4FZ/BPqF4FZoPeKzReFvNV4FjjR9WWyr/WBIq7thZKgm8pnU4g4kWjKZwOTiwiq9HRInA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3YoDCHkpinOQBJC5OCWVs0dAVcWKDrjzxMHgzD+IBkE=; b=Re14IwBhCgykmzyR/e15rc46LVMzxobHssDfxI19/UAd0LMN0sm9TacX6KFxPH62OW01VwA0qTdnR+sKjl6sYbe27ydWvbApIe2i6Cdzt57/8sdZ3WTYAIPEgt2rkqcG6kG94qnIK2fjzsPmQcDcevHRtwCBvl86NYKFqIy9S9kVpjb20cmBsP+4e2OwX03IQgsmQ1VYqnU4t/vbl0spxE5agQiwEufcDsBy9SlnTxEZG5gjxGohNgrqf6Ei6hcgPD5mv4D6AKE6XJpRXcxngk8I6Pv5MmgeZUF/boBIjcc3IWABq77M31UXCl+awLri73vCro+TBkN9OavEpLl9sQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3YoDCHkpinOQBJC5OCWVs0dAVcWKDrjzxMHgzD+IBkE=; b=UH6KZI1lHQerFcNa3JFP4358BSC5d6yAE+g47gVmFRkrn/pP1iUfONZY3MKd1rUTt+vdqKUgfWPPhnPM538bgQDHRcjCU5mG3MUCYMWiWvFBFlLzTnpp3PjADPwZqIctqR2OcNLQt1enUwtU0b4+bI9U825ncyTPHEZ169NhHzI/iQVAYuMKLw9np9SuOQLmeP5HCWr6bMiWO5E9aTj90WwEoByNqsDe9bcfPg3ARQ30LHwoPpnIS54cwI7Cdi6KZL2jhx0G6sscfbpSdqgCmDmqknJv1LknVeV9msDONPooOo9nme3sgYX2op2n5tOLEPKW8BbMb23eCSVnlonqRw== Received: from CH2PR18CA0027.namprd18.prod.outlook.com (2603:10b6:610:4f::37) by CY5PR12MB6155.namprd12.prod.outlook.com (2603:10b6:930:25::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.23; Wed, 18 Oct 2023 17:16:03 +0000 Received: from DS2PEPF00003447.namprd04.prod.outlook.com (2603:10b6:610:4f:cafe::25) by CH2PR18CA0027.outlook.office365.com (2603:10b6:610:4f::37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.36 via Frontend Transport; Wed, 18 Oct 2023 17:16:03 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by DS2PEPF00003447.mail.protection.outlook.com (10.167.17.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.20 via Frontend Transport; Wed, 18 Oct 2023 17:16:02 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:48 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:47 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Wed, 18 Oct 2023 10:15:45 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 06/16] vdpa/mlx5: Decouple cvq iotlb handling from hw mapping code Date: Wed, 18 Oct 2023 20:14:45 +0300 Message-ID: <20231018171456.1624030-8-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231018171456.1624030-2-dtatulea@nvidia.com> References: <20231018171456.1624030-2-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PEPF00003447:EE_|CY5PR12MB6155:EE_ X-MS-Office365-Filtering-Correlation-Id: 3a9bba6b-8a41-4a77-cc05-08dbcffde81e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ZcV/TNpo/gH9Z+tYGkZqWWveR137wA09QBnUA7PxMbr7/XaD5EHfWUqyWfqRFSAnqZH3O+aL/M75MyQem0Y4IEh1ZTXgfKyvuK0hxZooFrBdS2IHI37ebr6xhbyDEbyMrsoqPFn+XpF/WMA0tPvGGRY8dECS1o9j8xiibT3L3tLr85QnSaaIttU8OfCjehhFQO2itmHqD3G7ktblIG6p5B+BNmIwKyEPpbvTbuR+LbtZ9nVW6GXToOEBsmiEU2jDTjUt1kmk10xnUg8WrgJiIY2zxj5FrvW9ypLQreZtHzzmlMWTAbY8l3cd/cMYGIUIBHIsKiHcutqdOowSMsxIjEPFHUtEDaRNja5TODGfBoQh7HH3DwIo55w8LDzJM6ocapcMxNXnjPZF32NHUWd0dwI+gEV1Br0HaycKYgHwVg14J5Tz0s1l7nzJJAuQOJdn5xILAnkPUARHqZzBDaScM0KMPao26t87SgpIEq5V6flpSsk8kCzB+rCqP61LefjpOfJgC/V/BOu5B/I2UoFV3Qez+gCnFW6mUcw8WRWm8NM6k5YiSuXzBKJufiktDFVamVN4jWNtolvIF02Itir9Ssv2ZBopUh762569ETE/E3rLXESCktIY80uwQ5mCRYt77TxU0zx0TwkB4f2K2MZ85SRNLmnQMoUf7FlXlwTeesPQId1FQ5MBD5zuvizRt01MtzRVpETek7gRF3aJISReaZLumxzoyR3q9eMvgXQPryk= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(346002)(39860400002)(376002)(396003)(136003)(230922051799003)(82310400011)(64100799003)(186009)(451199024)(1800799009)(36840700001)(46966006)(40470700004)(54906003)(478600001)(70586007)(70206006)(110136005)(82740400003)(47076005)(86362001)(316002)(5660300002)(8676002)(1076003)(8936002)(66574015)(426003)(26005)(336012)(2616005)(4326008)(40460700003)(40480700001)(2906002)(83380400001)(36756003)(36860700001)(7636003)(356005)(41300700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 17:16:02.9189 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3a9bba6b-8a41-4a77-cc05-08dbcffde81e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS2PEPF00003447.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6155 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 18 Oct 2023 10:17:11 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780114452894002528 X-GMAIL-MSGID: 1780114452894002528 The handling of the cvq iotlb is currently coupled with the creation and destruction of the hardware mkeys (mr). This patch moves cvq iotlb handling into its own function and shifts it to a scope that is not related to mr handling. As cvq handling is just a prune_iotlb + dup_iotlb cycle, put it all in the same "update" function. Finally, the destruction path is handled by directly pruning the iotlb. After this move is done the ASID mr code can be collapsed into a single function. Acked-by: Jason Wang Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 3 ++ drivers/vdpa/mlx5/core/mr.c | 57 +++++++++++------------------- drivers/vdpa/mlx5/net/mlx5_vnet.c | 7 ++-- 3 files changed, 28 insertions(+), 39 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 3748f027cfe9..554899a80241 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -120,6 +120,9 @@ int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid); void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev); void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid); +int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, + struct vhost_iotlb *iotlb, + unsigned int asid); int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvdev); #define mlx5_vdpa_warn(__dev, format, ...) \ diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 7bd0883b8b25..fcb6ae32e9ed 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -489,14 +489,6 @@ static void destroy_user_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr } } -static void _mlx5_vdpa_destroy_cvq_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) -{ - if (mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] != asid) - return; - - prune_iotlb(mvdev); -} - static void _mlx5_vdpa_destroy_dvq_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) { struct mlx5_vdpa_mr *mr = &mvdev->mr; @@ -522,25 +514,14 @@ void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid) mutex_lock(&mr->mkey_mtx); _mlx5_vdpa_destroy_dvq_mr(mvdev, asid); - _mlx5_vdpa_destroy_cvq_mr(mvdev, asid); mutex_unlock(&mr->mkey_mtx); } void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev) { - mlx5_vdpa_destroy_mr_asid(mvdev, mvdev->group2asid[MLX5_VDPA_CVQ_GROUP]); mlx5_vdpa_destroy_mr_asid(mvdev, mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]); -} - -static int _mlx5_vdpa_create_cvq_mr(struct mlx5_vdpa_dev *mvdev, - struct vhost_iotlb *iotlb, - unsigned int asid) -{ - if (mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] != asid) - return 0; - - return dup_iotlb(mvdev, iotlb); + prune_iotlb(mvdev); } static int _mlx5_vdpa_create_dvq_mr(struct mlx5_vdpa_dev *mvdev, @@ -572,22 +553,7 @@ static int _mlx5_vdpa_create_dvq_mr(struct mlx5_vdpa_dev *mvdev, static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid) { - int err; - - err = _mlx5_vdpa_create_dvq_mr(mvdev, iotlb, asid); - if (err) - return err; - - err = _mlx5_vdpa_create_cvq_mr(mvdev, iotlb, asid); - if (err) - goto out_err; - - return 0; - -out_err: - _mlx5_vdpa_destroy_dvq_mr(mvdev, asid); - - return err; + return _mlx5_vdpa_create_dvq_mr(mvdev, iotlb, asid); } int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, @@ -620,7 +586,24 @@ int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *io return err; } +int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, + struct vhost_iotlb *iotlb, + unsigned int asid) +{ + if (mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] != asid) + return 0; + + prune_iotlb(mvdev); + return dup_iotlb(mvdev, iotlb); +} + int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvdev) { - return mlx5_vdpa_create_mr(mvdev, NULL, 0); + int err; + + err = mlx5_vdpa_create_mr(mvdev, NULL, 0); + if (err) + return err; + + return mlx5_vdpa_update_cvq_iotlb(mvdev, NULL, 0); } diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 65b6a54ad344..aa4896662699 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -2884,10 +2884,13 @@ static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, return err; } - if (change_map) + if (change_map) { err = mlx5_vdpa_change_map(mvdev, iotlb, asid); + if (err) + return err; + } - return err; + return mlx5_vdpa_update_cvq_iotlb(mvdev, iotlb, asid); } static int mlx5_vdpa_set_map(struct vdpa_device *vdev, unsigned int asid, From patchwork Wed Oct 18 17:14:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 155012 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4945490vqb; Wed, 18 Oct 2023 10:17:03 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGZ51bzIicClIe2qDhFLJMRxp88e4lRYEjg1Nqslyl33zzCOuzTF4B5q8FbEwPm2834CqgL X-Received: by 2002:a17:90a:5d:b0:27d:661f:59ac with SMTP id 29-20020a17090a005d00b0027d661f59acmr6211322pjb.38.1697649422867; Wed, 18 Oct 2023 10:17:02 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697649422; cv=pass; d=google.com; s=arc-20160816; b=Rcos4VTqrfWUslLHxCOO4wCXVrXeSjQzE6/IMmYEg+momdehlnKu9/TC+y6rznjDU5 biMgpbAWouFTvqKRHpUdKFFfk+KH3x9ovcnTXxl2rAAia8N+8uMwGAWtt3SutrBinzrt 0M+pgwgOOEpktM8ydZLHP33yK1MWSc4GKoPRuoZEWFv2lOC491WvPzuczZzXQYlmWpPq Gb6qrn76hfxvoryB0fog/9w1A+79gf0H/4g7fp/bMae9Z8w4WFUglPbasLgSLgc2/Ruf vWio92yAGrZYIGMGKK/EA1IsFxqFnFP3+McuCJoTjCkoMtbpLQGlTzRLpUQSPrtAOeBu b5gA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=sqfty/jYzKnLc/JEwSMOvq4UGNN3rkOgV9qEyvqXUKk=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=iaWKUxaNJLmFnDRi/73UM/jpKp9ImPz4Ij+q5ybst8TrhkP/6G5qdbzjl5MTTjoFkN ZxQ/4063N7r2xfm2+wN9AVqdnE2wyJn3ZNo4VbmamfjZwfxddpPiRwcv9bHy9DMu3Vhb /LAvBJS+iYSbQSqeDkvhmUfTzKbmQFW6hCqhLonleb0TRoHtWmb9PP0ykr7vVXyIRywP u74vOQaVIuFCnrJ7W8B5xqP+bsbHbAWZv+sEA6yAVk9ZgLpWWI/+fhpaMUxKJXCYwH2S iO4BG2SUKrqoYfOVBc5dsXkieOcPAtbUgeV14QsVrTg3tXQElTB2+LvQrrf5AQIYz6t4 PkVA== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=Oez1GLAL; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id z11-20020a17090abd8b00b00273dfe0741csi275853pjr.90.2023.10.18.10.17.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 10:17:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=Oez1GLAL; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 1DBE781B8971; Wed, 18 Oct 2023 10:17:02 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235208AbjJRRQs (ORCPT + 24 others); Wed, 18 Oct 2023 13:16:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40216 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235060AbjJRRQ2 (ORCPT ); Wed, 18 Oct 2023 13:16:28 -0400 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2068.outbound.protection.outlook.com [40.107.220.68]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 57127115; Wed, 18 Oct 2023 10:16:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=btC+LOXmSbno2OsIgFJWSs072DVNH9TTLZT57ONCYiblp2G45lcNrFCzYVmynfT8WIYX7SZDdfCCMVeqdz6Vzzb33jpkzCXUSV9fxrt17LFhmtWtzmT6AGdLCJD7wFqAUoO66cluaOFA9g3LqX6GCzJDXGp6Ups+voayQb8PdQ9yu9ToKv43CkWGglz6ylrC7XMKLLwuP3aOSI5tEe6VWTXcmiDyhVNZlAaBhwh6pJCidYM9GVxDsi+RtZdVyGXTZhgDUYpTj/yL2h5l66StsuUvLMtCbd7JjQ33CSzb/leLxJRU+qg5cKviOUMdx9gMuxorhpP2bWDRSZ8ATnOC7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sqfty/jYzKnLc/JEwSMOvq4UGNN3rkOgV9qEyvqXUKk=; b=n64CdEgPJ1UDXeKA3Y/TzoDiU3THuBa5cS5t71W1AzjQZEg8xMdm8ACRoGk6yul1+MzeLeqXV2FN7rjYiqaTaZAXx0ljtKxmHcNLfoU4zsfxNZT3s/smP2SDAiQAhl3b92YVOAWDuffmaWIaViBwuk3xpeAEXXZAGPx4LlYLW/ahL+5V7oO4GnFjnyzdCBsRwiM4uRTqmLOjqoZd9+ilS1rRxUi8ksXSjjQzzX/cKZyXLzBu06TsM2T0k6q1wlrKeS1J8ofrqefsOKus9nuPyPVzkQt1cviwNez2Nv8rE0t/UFtAy4FgW3E2BUjLYMw2xhbbek/xLiOpka2JvKk7KQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sqfty/jYzKnLc/JEwSMOvq4UGNN3rkOgV9qEyvqXUKk=; b=Oez1GLALNnZ7ZOHXzkFw5O4oWPXVqDTzBqlvolwaBEO/3htdDPZd2tcMtzaYBFZWV1rpfBk/D9FHNmATP4ivqTZSlp/uCcf53r9ZfuaCf1e8Z+EkUsDx3eZpsEitmTcuYRT/0QgIrac1W7E9z+dzxOEEVy+Ffvg6pgNVzOUOd6HaPZycGcNSqa6oOqAy05uGb9u5PNqfJAk015DNBwadG1/9SkNegCe5d0U6tc9AX6WzgjqnyFdwTsmthSE4bgtpx82crHK6si5Eq3CTBaoIsmHBhyth5iDLxnBns0WuQrg7lvQ56n8/i0AYGhsVDxi7zj85wj06hA4A80nNh8neow== Received: from CY5P221CA0147.NAMP221.PROD.OUTLOOK.COM (2603:10b6:930:6a::20) by DS7PR12MB9041.namprd12.prod.outlook.com (2603:10b6:8:ea::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.36; Wed, 18 Oct 2023 17:16:07 +0000 Received: from CY4PEPF0000FCBF.namprd03.prod.outlook.com (2603:10b6:930:6a:cafe::e5) by CY5P221CA0147.outlook.office365.com (2603:10b6:930:6a::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.23 via Frontend Transport; Wed, 18 Oct 2023 17:16:07 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000FCBF.mail.protection.outlook.com (10.167.242.101) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.23 via Frontend Transport; Wed, 18 Oct 2023 17:16:07 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:52 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:51 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Wed, 18 Oct 2023 10:15:48 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 07/16] vdpa/mlx5: Take cvq iotlb lock during refresh Date: Wed, 18 Oct 2023 20:14:46 +0300 Message-ID: <20231018171456.1624030-9-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231018171456.1624030-2-dtatulea@nvidia.com> References: <20231018171456.1624030-2-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000FCBF:EE_|DS7PR12MB9041:EE_ X-MS-Office365-Filtering-Correlation-Id: 81ee4425-fdf7-4ec1-8ce4-08dbcffdeacc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: rodFcS3UTwp91jSJLaHsfhaynSqlJZAsI2PldA7T0Z6t6cqMJT+tECEUGf/RwdhYVQTavQ1AvZVNF9YKPe/y6x59ixMHwQfX+0/QeVK6NPRVB0HQ+lYstGwavufnb6PxiJjQaovBe+Vd7FDYmqC9MnJ+qzNi0vZW6m1jGdVVrtImYoJowtd105/YCijlsn66yyycM65UEPhwkgbOHbiZHSqScso3KQS0DEi055mCYG+nVTgF36QT9I4ORQJ7V0+9YpQCIhx7a/pC5wV4VZf5kc2Uck8BFYFCsF1xOMpeOZGAhHS5McZVMJsD+fQFWA/SLwoYml+xZ6yf0VRmoUhhJVE6W4SFJo/crpV36ZLb0d7XOf2uVe6atzNlZv6qEZPelQyAGixzm0Tajrn8ru9pkF9zIfUaiGLfy7Ep81lmuocxz4aM+hf6VRkPDh+zRiLbQgRtMPnzLLmA9JNCEL0liCQEDZ/fZxLy0KWHWJ9WQtwT+x60E42h8Uyko5izrvtabVJspLNcCQy5Em/BC7jskWMMOZQcRUdJJDN+1K4lLS3aw831tC5jqgM2H1HaAigLFgJYX2FLgwWKg6emuGN2R5m5kyJ0XWhz6tNE4l/1vJBjkqEiuT9x2cWVEHEfuSvVgGOygIu0jndOuSefPMP1YJT+WE2C5V/IPinuelS9qom5yM1PyH518Osomb+JRsrpWJaLjjoyNcC8axkSIoLnjbdKBfcGF2E+QDl9DGxybW8xkvEbuIca4owcdFbT2VNRkRRWicvqiv/rcTwOaBWXcdWu1JD2Ul4/AEl+648hzZc= X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(346002)(39860400002)(396003)(376002)(136003)(230922051799003)(64100799003)(451199024)(186009)(82310400011)(1800799009)(36840700001)(46966006)(40470700004)(70206006)(54906003)(110136005)(36756003)(83380400001)(336012)(426003)(70586007)(47076005)(2906002)(66574015)(316002)(966005)(478600001)(82740400003)(8676002)(8936002)(36860700001)(41300700001)(40460700003)(26005)(356005)(2616005)(4326008)(40480700001)(7636003)(86362001)(1076003)(5660300002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 17:16:07.4444 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 81ee4425-fdf7-4ec1-8ce4-08dbcffdeacc X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000FCBF.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB9041 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 18 Oct 2023 10:17:02 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780114441233926754 X-GMAIL-MSGID: 1780114441233926754 The reslock is taken while refresh is called but iommu_lock is more specific to this resource. So take the iommu_lock during cvq iotlb refresh. Based on Eugenio's patch [0]. [0] https://lore.kernel.org/lkml/20230112142218.725622-4-eperezma@redhat.com/ Acked-by: Jason Wang Suggested-by: Eugenio Pérez Reviewed-by: Eugenio Pérez Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/core/mr.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index fcb6ae32e9ed..587300e7c18e 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -590,11 +590,19 @@ int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid) { + int err; + if (mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] != asid) return 0; + spin_lock(&mvdev->cvq.iommu_lock); + prune_iotlb(mvdev); - return dup_iotlb(mvdev, iotlb); + err = dup_iotlb(mvdev, iotlb); + + spin_unlock(&mvdev->cvq.iommu_lock); + + return err; } int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvdev) From patchwork Wed Oct 18 17:14:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 155010 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4945420vqb; Wed, 18 Oct 2023 10:16:58 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHk9AN3UCiYojDkITCul07aptqCPwt/WIaJzD6lDZfyWr3Tps/nKPXY2wGersQDTSkF/Ism X-Received: by 2002:a05:6a21:7906:b0:16b:c20d:fcd1 with SMTP id bg6-20020a056a21790600b0016bc20dfcd1mr5676042pzc.21.1697649418370; Wed, 18 Oct 2023 10:16:58 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697649418; cv=pass; d=google.com; s=arc-20160816; b=yyNrL/+z5QG8rspPhlemChqHuC9g+u8LqAc1T/ZBYp55D5i7rHtBpyQRlGWW7acq9E MkfHp3VFvfSFC0YXYUW+lKtHTwRlLAPD/tEx/YFx3OGuITV1MtAy5UKWP4IaSJCFqcys 6Y3Muww3/wtLavJniNnFgEA+hV8JfpQvDwhnnDwnFl0p7C7ppd2R1VDqdbNLMBnWTNLx ixyYJqWoLw6Y1XIb6jGuiFr63YlnszUCmJa7jSAeSTN4jAFNCNhuijLKDuF2DX4/SZC4 B6P0wkFw44RTPXlCSz2wQjMVS2K5XtcB93k6zIY4ArfiAY3sN/AZNnGihgTaot9GlOM6 wNxg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=elzKdMTRh3uezwMcgzMejLd/DLpkMeh3g8HOvOAJXm0=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=IgivZoITzLG62eyePZmRSsVRCMBY4gcl54jE1TDMDlfcAnx2dD3sSWjvLL2pzaxLRz CKREpYq03oDbX6poYO21h6XybEHCxJFacAH3tr8BuryYBiug9F3DkKPx7abp1GDmMK2z Hsgx6DQZJ6V2utGvlRpMb1LLCcFU1dxLvkYsznUmSTXQpt55x2BjCPspSHERyA0PlSw5 rscOLKY4m9ulhkDytFvOnYw2zSVA2zTcicvNNxSE7vhSXItS4qSuEPUNALm/YxzlAbbE dAq1bUPAZZ4USXhfeeiVb4FNFlSj0GKgIuiev3JnfVunbzxqrvMFImPIAASd9zfOq3VR FG9w== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="KUTXn69/"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id q11-20020a170902dacb00b001c3a05b0b67si290733plx.566.2023.10.18.10.16.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 10:16:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="KUTXn69/"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id EEE958201B19; Wed, 18 Oct 2023 10:16:54 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232156AbjJRRQm (ORCPT + 24 others); Wed, 18 Oct 2023 13:16:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235146AbjJRRQ2 (ORCPT ); Wed, 18 Oct 2023 13:16:28 -0400 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2077.outbound.protection.outlook.com [40.107.243.77]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B3BED4F; Wed, 18 Oct 2023 10:16:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RODKhLcytcyJmaRu1MEW/3wFEeBr+qRc1ykeaFRtAyiampr2eQARjgrD9pO2G9NZQ8lQC7lL9e3y0FW1YQMjeVUHj6qJZe0CeDLEgnRAGuqCeR6JPAC2npLdHQIyyqKyf/xnZnLEcc2tO3VBcq/ViSQTxRqDhAEuVwBXL0K4J7rXO6P9PcQW8epXbB+kX5GY58+h2t0fzX4cEdZDi2LwnqVrB8SiltHF6HmVdshV2sl3A38Bco9QHIzqMnR6EyYLLvw5q41F38Sk86a9/v64IJ8z+lPXLV25qzjY+/ezHz9Vw/Nt+xRB+54J8HGvoxR3wXy35YEGrzbbOrJZiN+/Ww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=elzKdMTRh3uezwMcgzMejLd/DLpkMeh3g8HOvOAJXm0=; b=PTfAczDB5IutLYWhcuL1J25MtESt5vBfnjRezGDhrQDisu7R8E1w9jMiuRlqs+dk/bWlSl87wBdnEer8x/2npWQVOFS9dbW8ReNw4p5o8DhgyAwEWLNHLaUhhIvcTnzlLBHCLpyrhHZ/GqQeQesbeKjsTeuclk8wpHlu6x8ln54pjwryXcJI4de7C2DlHLmvPQZWrL6f4QDS/RWMP8dddFR8nbpX+3R0zqfQtm9ZNWJyf8/lewl8+yEEOiAD+D9HrV0b0cryhHvTjGZZpYP/8BKdlONewbaZuBlJmmbObeP6fl59jbVN8nDol047IkNn4xQnVJ4Xj/vrqFTrxAc0ag== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=elzKdMTRh3uezwMcgzMejLd/DLpkMeh3g8HOvOAJXm0=; b=KUTXn69/fiQrTcGUbeWTKJUCpB+Xjs3LTyFzwlli6YGtU0epVfL2J41TFVQHGhH07SB151l+v35q85aNm5OXcmgONi+RrtvAPF+0eHzybZ85yW2HIPKrkL72AXsyQ+YCn6cAhtx0VQQDc7bSiBryLyu7DtYvVCy4vKQppNKn0+pApCuRr4Y1oTOtbgtwPJvqtygQTSAV06klydaWm9Z9IBq1wpr8J9gVC3H1JjY3JV85mlSaH/P+dQOS85yk6ff9INO8Qj38yjg8Hgc+EnNvLG8R4KUZ6WrKTYnJlO2GFxu4vZ9KQKNhAm2fMfHhY+ga/Wua+YyLk7vbg4ClZTjKfQ== Received: from CH2PR17CA0015.namprd17.prod.outlook.com (2603:10b6:610:53::25) by SA0PR12MB4573.namprd12.prod.outlook.com (2603:10b6:806:9c::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.24; Wed, 18 Oct 2023 17:16:09 +0000 Received: from DS2PEPF00003442.namprd04.prod.outlook.com (2603:10b6:610:53:cafe::6c) by CH2PR17CA0015.outlook.office365.com (2603:10b6:610:53::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21 via Frontend Transport; Wed, 18 Oct 2023 17:16:08 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by DS2PEPF00003442.mail.protection.outlook.com (10.167.17.69) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21 via Frontend Transport; Wed, 18 Oct 2023 17:16:08 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:55 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:54 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Wed, 18 Oct 2023 10:15:52 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 08/16] vdpa/mlx5: Collapse "dvq" mr add/delete functions Date: Wed, 18 Oct 2023 20:14:47 +0300 Message-ID: <20231018171456.1624030-10-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231018171456.1624030-2-dtatulea@nvidia.com> References: <20231018171456.1624030-2-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PEPF00003442:EE_|SA0PR12MB4573:EE_ X-MS-Office365-Filtering-Correlation-Id: a8802d4e-9112-4cfe-0908-08dbcffdeb9e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: +LwgzQrvSPtrOmmiOo4wqw5eAgP1JlQGuBRoI092gF3A2sGkqiKj9+GOXD/4Qwq7FiXc9+vXGnH2sMI4fMCImifaLCUKlpcRws9HbfpAtyI9nOjvvZmqMS33AhoI1zCsl2sL9vsoOfYiz17mDLc8A2JnjXze28BH3JRbKqoZ6mHlYGSX7FAP76FC01sJUkbpqXN0+OGpWe7K7A13B0F/LiSwHP8AFXE5TMURoZ/2OAw1878//0+TjHn84BfAEAJS3qB5+KepmyerERt6wU5DUq02k05zARPlZQyhF2c1cSqKqoIllqkeGuu3UYx6WxWmsY/d9uEdFWr5vXV3tDEt4noA4j05kxQ29/3t5fioNdGwjR+eBEi88IxuBuUNSGRrRIGX2EByoxHfUFS19qUwD8sj0D3WG4AkAnjy7MdbOPuRyzGSdaHSkpe38Bc3U28SybYVnd17if6q40/uzWGJZmbKVNJi5JQXaDatp8wX0GwqQmn98HEA+SwNxl4zTJKOv8fpah0BKlHp5RbGlx29MZfS7Fe69IZ7hXft+roFRwxNivQ2hm3ZgzABtvGo/f7xiYblr8yRLDtqjhGtr+vxVwB6TQWeFxoVoO7hIXKSCLxHwIuZ0G0MOzm0J460H/Ljxu66IoVabK3p7BLvgFE5W7N2efVNWV2b+EgXG+lvMKpGwv2997OmVoAZArJG++5BpvXyLHqt5aI1IZoT5q4UFZ0ka9+0HCR3FKchhuXiuYilo5cb8wsFLR+KunpKlwwC X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(396003)(136003)(346002)(376002)(230922051799003)(186009)(1800799009)(451199024)(82310400011)(64100799003)(46966006)(36840700001)(40470700004)(83380400001)(40460700003)(40480700001)(36756003)(2616005)(316002)(1076003)(2906002)(86362001)(110136005)(70206006)(41300700001)(54906003)(5660300002)(4326008)(8676002)(8936002)(70586007)(478600001)(6666004)(336012)(426003)(66574015)(7636003)(356005)(26005)(82740400003)(36860700001)(47076005);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 17:16:08.8172 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a8802d4e-9112-4cfe-0908-08dbcffdeb9e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS2PEPF00003442.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4573 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 18 Oct 2023 10:16:55 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780114436847991954 X-GMAIL-MSGID: 1780114436847991954 Now that the cvq code is out of mlx5_vdpa_create/destroy_mr, the "dvq" functions can be folded into their callers. Having "dvq" in the naming will no longer be accurate in the downstream patches. Acked-by: Jason Wang Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/core/mr.c | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 587300e7c18e..fde00497f4ad 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -489,7 +489,7 @@ static void destroy_user_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr } } -static void _mlx5_vdpa_destroy_dvq_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) +static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) { struct mlx5_vdpa_mr *mr = &mvdev->mr; @@ -513,7 +513,7 @@ void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid) mutex_lock(&mr->mkey_mtx); - _mlx5_vdpa_destroy_dvq_mr(mvdev, asid); + _mlx5_vdpa_destroy_mr(mvdev, asid); mutex_unlock(&mr->mkey_mtx); } @@ -524,9 +524,9 @@ void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev) prune_iotlb(mvdev); } -static int _mlx5_vdpa_create_dvq_mr(struct mlx5_vdpa_dev *mvdev, - struct vhost_iotlb *iotlb, - unsigned int asid) +static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, + struct vhost_iotlb *iotlb, + unsigned int asid) { struct mlx5_vdpa_mr *mr = &mvdev->mr; int err; @@ -550,12 +550,6 @@ static int _mlx5_vdpa_create_dvq_mr(struct mlx5_vdpa_dev *mvdev, return 0; } -static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, - struct vhost_iotlb *iotlb, unsigned int asid) -{ - return _mlx5_vdpa_create_dvq_mr(mvdev, iotlb, asid); -} - int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid) { From patchwork Wed Oct 18 17:14:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 155014 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4945610vqb; Wed, 18 Oct 2023 10:17:12 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEbXcBU3MAPR6KVGlNu6j8Vbja97VqILCsPQupOqiL759UYZl3iRlNtg/kJKc3hC1dclN1t X-Received: by 2002:a9d:7418:0:b0:6b9:1af3:3307 with SMTP id n24-20020a9d7418000000b006b91af33307mr5877028otk.17.1697649432703; Wed, 18 Oct 2023 10:17:12 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697649432; cv=pass; d=google.com; s=arc-20160816; b=0kO6J6yE83n4iPR4L60Ygu5sxvehuMxkXVHb9S6103XZFnGqtCIa7WJLG4/4X7S7Zc PtgwepSGVhNho7BrpdT8fJ/rOaZoeXgAt9uiZbluRr6X+fbNvoukvVK/eEqct988ocFo e3jMXUxrExpBdevmC/24siXzAaSNot4QOmwFDSNfFASDYUyFE7gGvoCSp+c79GcS5IXP y2ZStXAPrgzyAWLHJaJeb5EN5cv5lBzPNmwIPJk+Qzbds47B0UtgZxWw9HJyjV8DNHl6 /yP0YOsGQNPu0+gLuDGhZhPlw/ZLrgw+cdraTMYWEOHJKsnvfDwEomw36C97EZyTC94T wGjw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XmILL87kEXieQsFyA46Z/A2SnEkDR6phjuffw0L8hYk=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=OAFQ/XA/JnGm9/cV6llAGjex84A6yK1oK9lL+sS300NHbKVB6Owz63RP1Eb+c1Srpn MfE83/RW0IpoPHIgh5vQkPvmpwG9PbdeDlq8Y5pgc8HHQH8jDmr3DSwyzzc10uxypoLM YHbWzsBqFcUHq6Xm5uMBjO3/WgdwncoVoyDwdEd4/paR82f0sNW4VqlgYYYsnidDnGZQ lFcSjSMMvTbDBTqtjNGmW6A2/MHr+OyqqbJGKWZns+HWCq0Z1y0r/J6pE8x3UlUYpm1G K/BuUY/5+2FpkpVUYtiA1UXSAaZxOBUWM3reA2mIKz/UKNEwjMl8xpk/f4B/zgvuXVUg ZSEQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=MKly6PPV; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id cm23-20020a056a020a1700b005824bad8f83si421394pgb.846.2023.10.18.10.17.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 10:17:12 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=MKly6PPV; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id B45B581B93CA; Wed, 18 Oct 2023 10:17:10 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235420AbjJRRRF (ORCPT + 24 others); Wed, 18 Oct 2023 13:17:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232691AbjJRRQa (ORCPT ); Wed, 18 Oct 2023 13:16:30 -0400 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2043.outbound.protection.outlook.com [40.107.243.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A32D413D; Wed, 18 Oct 2023 10:16:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=khJOpENTdd3MTcuP4EGZz3oxsfIotYaX3M3SlSxmPlRs2bd6IHNVezF+SY5z56blb0pO7h6pOeclUnSdRdWBMtOxWdtgtdyvX9fRvGVOqCCPSp//ljEGAkuZcnd4z0uecy9ty4d5lreeBXzeQSzENDx3EfSdQj5pfbkg5RvKTOefkeXd4X8d1ZzGwzFYri+u+i7k/Vc25wqM+SMujFAW50KIcRV+s0K5LM8r84K13BUxGhYHdLTDGsigLiS32I3OSiQmYC7SGjaXejsQz2FCyrP/hjQV2DYGaTRGUW9EMPgd5qvtQuiW3qCeFZzriXYzyvrmCoRckk9XkXQN46POPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XmILL87kEXieQsFyA46Z/A2SnEkDR6phjuffw0L8hYk=; b=MF+j7+RKZs8aGoU0ScZPV9hzh52FnwzcJdJRiZ+EkfHB9/vV8XSTdEwUV58pndJK+TJ1f8vJLR4wGOm+VQ6ymmBQKORrcEJv8U77xo3jYt5wHKnCHZcXmn4+s+nfHo3irmoAQxXzwiRm4Zecb6S2dh+aXb4pceBoSFbRHUZ71yZwxk2QsfdsflENdMJgc141eb7NY5+gofD6LA4DCf6RzR0PyA7DL0IRJmj8b3r9UrRNgChxo0ju1tvi8A6bxdGeRcnz+EcuO+33Ibyktaff/Xktzoh4eEuZ3u2Vn+4UN9mBO3Ftsvb7VljI/y83TcZvQP2/t8mP3QxRkABk28klKQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XmILL87kEXieQsFyA46Z/A2SnEkDR6phjuffw0L8hYk=; b=MKly6PPV9EtsnmXliLkOQxWB9Ns59Jq09zFbCkbnm/y48+cxmzonufoWzA6CT/tkcI46lInqpyVMQFfDBjI59eCGnu5BsjcMXUU9s8Fi72I1uF807z6ZgJRDTO7HG7I5vbiD3M6qa9ne4pnnGEsHFJrgDwG1ykcl9N3pMlQN6Ks869W2lHgH94bvBtcZNbqU2nausRctqmocL979BqHKfIfJdYO4oe97MZGOeeCu9EQ69veqQVedhXOivwyoQ/vHr7VDnpCRVCmx39fYLUfcxFbVhivIXZeigwtt2OU0Ae8K7kbbbUeUqy4A0a/3Jdh0WGFmPFgQewhsZ4GxMtFiNA== Received: from CY5PR22CA0075.namprd22.prod.outlook.com (2603:10b6:930:80::18) by MW6PR12MB7085.namprd12.prod.outlook.com (2603:10b6:303:238::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21; Wed, 18 Oct 2023 17:16:11 +0000 Received: from CY4PEPF0000FCBE.namprd03.prod.outlook.com (2603:10b6:930:80:cafe::f7) by CY5PR22CA0075.outlook.office365.com (2603:10b6:930:80::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.24 via Frontend Transport; Wed, 18 Oct 2023 17:16:11 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000FCBE.mail.protection.outlook.com (10.167.242.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.23 via Frontend Transport; Wed, 18 Oct 2023 17:16:11 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:58 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:15:58 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Wed, 18 Oct 2023 10:15:55 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 09/16] vdpa/mlx5: Rename mr destroy functions Date: Wed, 18 Oct 2023 20:14:48 +0300 Message-ID: <20231018171456.1624030-11-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231018171456.1624030-2-dtatulea@nvidia.com> References: <20231018171456.1624030-2-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000FCBE:EE_|MW6PR12MB7085:EE_ X-MS-Office365-Filtering-Correlation-Id: 59c047f1-d340-4bfb-1f95-08dbcffded36 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oosKnh16ICDOtiiBOlH1EWiZ1cPFqU7msq/3X4G+GI7q/buNgHYspwp5ZpPIZFhY2nJvwRrVU9Lm98s3H98fsLjFhxYOS/7IzxGw7bLBzZgRh0H7/Vl+nrcRM3vPuPfqkYgE/2qRzKQHC0UYK4kZZYyW2dggJbtUo8gsQngRQTrzHesQcHEjqY2P+xQEvAMwdvaculY8ybIRRZXS7w28SaXb9j9fRUrKog9xUW+x+dE/d33rZC4Mhs93S8nwqSb2HKfVUVhOywG7V13jhlK+4O5vy4VwQ06pt1nSK1ihaCvc6S9AyXOW1b8fAhfUouupBM6+5LYpIsjnOJh4XEhfWOSBfrdB8dAmfuU1PiZNyD2fcmx8OLapTs5KUOwnAtRAOH0nk9LKAWKUXjcn06b7mqRYVphLIfnQlI4mn1n/TOPrBi+FsGKTzY+NvfU64wYvl++cxKRWStDN4dMCr6mkLOq31Zh7kasPuq2vxXPq1j1YclKImXtgkcYNfBF9H9rO6tJDEoCgXTZZdYx7qaksMIKfqyoCl2A+17w7dBWmJnVTZiX80gqrTfgEdH4KMwP9IC7uybjlpkVbXuKOODKLAV+pD2NvnglkY3BwqE0kimaxc62xWacSV5/5Kf0PjRq4LAOgjKCXHz6WC4CSYDSV1MwckjgVg70/V0FTnFphqFx9LuQufl5E1Ug/jp1UavMzvRSGoDPLVjIc8MYYs3azIm8sVcsghuNu5k46b1zXD+nwUQVVW4Lv9OeXzzNoJ/Al X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(396003)(136003)(346002)(376002)(230922051799003)(82310400011)(186009)(1800799009)(64100799003)(451199024)(46966006)(40470700004)(36840700001)(40480700001)(36756003)(40460700003)(110136005)(316002)(54906003)(7636003)(86362001)(82740400003)(356005)(83380400001)(336012)(426003)(66574015)(478600001)(8936002)(1076003)(26005)(2616005)(6666004)(2906002)(36860700001)(70586007)(70206006)(41300700001)(47076005)(5660300002)(4326008)(8676002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 17:16:11.5073 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 59c047f1-d340-4bfb-1f95-08dbcffded36 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000FCBE.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW6PR12MB7085 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 18 Oct 2023 10:17:10 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780114451620008250 X-GMAIL-MSGID: 1780114451620008250 Make mlx5_destroy_mr symmetric to mlx5_create_mr. Acked-by: Jason Wang Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 4 ++-- drivers/vdpa/mlx5/core/mr.c | 6 +++--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 12 ++++++------ 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 554899a80241..e1e6e7aba50e 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -118,8 +118,8 @@ int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *io bool *change_map, unsigned int asid); int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid); -void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev); -void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid); +void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev); +void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid); int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid); diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index fde00497f4ad..00dcce190a1f 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -507,7 +507,7 @@ static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid mr->initialized = false; } -void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid) +void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) { struct mlx5_vdpa_mr *mr = &mvdev->mr; @@ -518,9 +518,9 @@ void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid) mutex_unlock(&mr->mkey_mtx); } -void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev) +void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) { - mlx5_vdpa_destroy_mr_asid(mvdev, mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]); + mlx5_vdpa_destroy_mr(mvdev, mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]); prune_iotlb(mvdev); } diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index aa4896662699..ab196c43694c 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -2644,7 +2644,7 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, goto err_mr; teardown_driver(ndev); - mlx5_vdpa_destroy_mr_asid(mvdev, asid); + mlx5_vdpa_destroy_mr(mvdev, asid); err = mlx5_vdpa_create_mr(mvdev, iotlb, asid); if (err) goto err_mr; @@ -2660,7 +2660,7 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, return 0; err_setup: - mlx5_vdpa_destroy_mr_asid(mvdev, asid); + mlx5_vdpa_destroy_mr(mvdev, asid); err_mr: return err; } @@ -2797,7 +2797,7 @@ static void mlx5_vdpa_set_status(struct vdpa_device *vdev, u8 status) err_driver: unregister_link_notifier(ndev); err_setup: - mlx5_vdpa_destroy_mr(&ndev->mvdev); + mlx5_vdpa_destroy_mr_resources(&ndev->mvdev); ndev->mvdev.status |= VIRTIO_CONFIG_S_FAILED; err_clear: up_write(&ndev->reslock); @@ -2824,7 +2824,7 @@ static int mlx5_vdpa_reset(struct vdpa_device *vdev) unregister_link_notifier(ndev); teardown_driver(ndev); clear_vqs_ready(ndev); - mlx5_vdpa_destroy_mr(&ndev->mvdev); + mlx5_vdpa_destroy_mr_resources(&ndev->mvdev); ndev->mvdev.status = 0; ndev->mvdev.suspended = false; ndev->cur_num_vqs = 0; @@ -2944,7 +2944,7 @@ static void mlx5_vdpa_free(struct vdpa_device *vdev) ndev = to_mlx5_vdpa_ndev(mvdev); free_resources(ndev); - mlx5_vdpa_destroy_mr(mvdev); + mlx5_vdpa_destroy_mr_resources(mvdev); if (!is_zero_ether_addr(ndev->config.mac)) { pfmdev = pci_get_drvdata(pci_physfn(mvdev->mdev->pdev)); mlx5_mpfs_del_mac(pfmdev, ndev->config.mac); @@ -3474,7 +3474,7 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name, err_res2: free_resources(ndev); err_mr: - mlx5_vdpa_destroy_mr(mvdev); + mlx5_vdpa_destroy_mr_resources(mvdev); err_res: mlx5_vdpa_free_resources(&ndev->mvdev); err_mpfs: From patchwork Wed Oct 18 17:14:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 155016 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4945711vqb; Wed, 18 Oct 2023 10:17:22 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEwfw1uDh0QslqvASJ4ATBQ/kS236DBLvw/KgCjHaSLYQgG2jM2mTFtrIwm1kgiXDObMUZq X-Received: by 2002:a05:6a00:194a:b0:6bd:9281:9453 with SMTP id s10-20020a056a00194a00b006bd92819453mr6462089pfk.9.1697649441997; Wed, 18 Oct 2023 10:17:21 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697649441; cv=pass; d=google.com; s=arc-20160816; b=jHYE4V5taIVvwfbGpXmY+aEeX/KGY/ETtYw4AA9MtHMBoASCe7abRgJsgzWW/j46iJ XQMh8ihqfAR4/Okafm9faeb1tCDtBmlRFcMs7GWXHMktY6pfW1xXnQh+XfP8KKJ7vrWp yCIMDhsHwh1tAmUIZEHh1X3c0Hlhczy7lLaY8MtrwB8NurMMSd5QeeM7F8L2Mz8NFQWB 5H+8OD8KXLY97rBwrZKG7nN4DSLwqLfZT8GWYbitxx3JEYB1elkACv5Bg+q57U7PfQIz Hcith0olbJCGIpsi1efneS+bScz79wtpNdeRqwbzXLAmOzKTtUhiJNc/ZmBZ4psa6RdU /SxQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Q3nj/DFK1Ax7axx5cxRwjydNEVXFy4148fMqcUXwGVE=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=PGfRiv+204K4f0N2ZSgBbqFmWIB7djvvppR+scH3X/xNBmp0K2DgE8SK1AqsIgDhLs hTwepX14/SKAncVi+JSX3mCIomZQutcxdlfPBqPDLRSDm/kzlSIzgqkKtny2W/fUsXc4 q56VTQIocF/WbuY2kEuQjFcQVpzsr+M+ODcbK58rHS2Lngzj0hVpxKsjHWPE4wZs7lH8 70BCzuX1afiJBbeRu9hfendkG4iZPXBX+w+RJhV6r3/GnBvXBzBczssYxpNRoC+uVk7A 0kQ11OSfqPHiWo++2V0xzrr53+5KMzIvvVbMc1rGd8LNiksgng+V0IY+laoKjpwOwzaQ HEKQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=r4gTpt2e; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id k69-20020a636f48000000b005859c221c6dsi2497323pgc.221.2023.10.18.10.17.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 10:17:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=r4gTpt2e; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id AAC1482157B0; Wed, 18 Oct 2023 10:17:18 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344602AbjJRRRK (ORCPT + 24 others); Wed, 18 Oct 2023 13:17:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40294 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231715AbjJRRQd (ORCPT ); Wed, 18 Oct 2023 13:16:33 -0400 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2043.outbound.protection.outlook.com [40.107.93.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDFC6D59; Wed, 18 Oct 2023 10:16:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OSBPaGY/xftjUIJl07onKTkDm6UTUIuA22yZ8DKu+QO1fmqB1Z3vs3BhEElbhrrtzo3lLUUpvNTMUNmmcKF351zWv4ULFbD4IcFiDt1jPF+he16SR1FQcyW5LHbMK/4xHri53tJRAPFLRJu+Sz/dqyA52fxJC40sYQ3lbjLeTU02Dygt58XQlwDNCR2LDbNfQuUoqq4c7NwI2U0/DfrFkaPTPNyzVB7Gw1Zz3mT/oaKfyT4sResMn6R7Om+onhU4mEJ20DI7UZHbjNvqhneUmDyK7t7/Ku45ZBBd1lwmA+w/T51TXlCzDessgNUg9wS00/L2KeQvzDwHL+2x61+6nw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Q3nj/DFK1Ax7axx5cxRwjydNEVXFy4148fMqcUXwGVE=; b=XKWolbvEQhuLIOqx1X0SWef3CQqMCmbkOsFA4RJUDo/cquQd/F2v/SwdF/aHWLwVtZQCvZvMPn209kV6o/7U5BL/y33vczIWbkPvk8k2p5E3XDER7xXq0magLcuHUA3QU1adeRsBUZdo+GY+8X+zSrVNxfO4ymtuRxSl9ndcCSd4xCmI8VqMRBTnPnDZ9zs/ebrxUIjYRJmYEYCiItTSIwmatO9+RDSMH6qD4mgR5069UUEPSniFyOqRN2CqYE8qkJ6m9pbiyiYYKnrRy10kXL5bd+3UXdpenzx3FI4zJRSU4fZG4zsnaKWWbx8wck5B2B+KQAHk1ImTWxcSP9GORg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Q3nj/DFK1Ax7axx5cxRwjydNEVXFy4148fMqcUXwGVE=; b=r4gTpt2eHep52rYmbHmxrd78dPF6fpprrJYkrKFhzWnUYNoGqC48PE7WgJgFT88mxBN1HeiG+1seQKW5DACABHTVYluDZwd6ehARqE3wYa3vMpIJeuMVujl28k/39+tUZr140P756CUkrxrQDJEYTb3FhWwtsIHG+4+J+TXEELx7Muyw1ruE/kgsGRuzc/w3R7UZIipzZdbjNCeeZMtpChhGSyBYTLpakEDJMnYdrZlewt7lnnT4vX3FuLU18mz/bBPu6WdAXbE4AA2feTFOL2LeJtfmla9YQTumDngJMT3nHgXnHqC8nlOQ3yZGwX9KDLFci2jd5XNoNtrHDo3wTQ== Received: from CY5PR22CA0061.namprd22.prod.outlook.com (2603:10b6:930:80::6) by LV3PR12MB9332.namprd12.prod.outlook.com (2603:10b6:408:20f::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.36; Wed, 18 Oct 2023 17:16:14 +0000 Received: from CY4PEPF0000FCBE.namprd03.prod.outlook.com (2603:10b6:930:80:cafe::9f) by CY5PR22CA0061.outlook.office365.com (2603:10b6:930:80::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21 via Frontend Transport; Wed, 18 Oct 2023 17:16:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000FCBE.mail.protection.outlook.com (10.167.242.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.23 via Frontend Transport; Wed, 18 Oct 2023 17:16:14 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:16:02 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:16:01 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Wed, 18 Oct 2023 10:15:59 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 10/16] vdpa/mlx5: Allow creation/deletion of any given mr struct Date: Wed, 18 Oct 2023 20:14:49 +0300 Message-ID: <20231018171456.1624030-12-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231018171456.1624030-2-dtatulea@nvidia.com> References: <20231018171456.1624030-2-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000FCBE:EE_|LV3PR12MB9332:EE_ X-MS-Office365-Filtering-Correlation-Id: 32c3215f-ec9c-47be-96ca-08dbcffdeed2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Bqa1HB4gK7K7RtQe1kOBSWjMF9TBCr1uoYaRicI4QGNh8rgpbsYqAeZT7kwCQF0iZwGanaFJtI83e3bmXUWScQ3Kj14V4Sdr/Tsj3DYbyMqLU5o6UYQhU+/em/S4+JD8754qTgSrvRetdhlT5z6dIJ9qM2f2dsrXq4xxJZrgoHVhm78yKGjk+EEVjKnoDEAWq18mpkW4y7RJlEE9lR/Yy57XoLA97/Z8W8++uGEzNPDDplZ9acD4w1f/fnXQpynC3BOi+x1ep86FfULKSa/ruY+z7/s0Esf9FFAILG2Ofqn4y8wq1TdpBtTwJ8fpOPGt6W4jImrEtBp8LGPl6jb2+46+/poPFEQKyDsry6CxFcHBpoLyHr22tDUQZ7T5UhmTtQekSQaDS+ZmaNXInRl0IGJof9HUW0pLAqVgEWlruuwH0Ay8xcdw1/i81CK2BgR7tOQsvm1Yi2cDPyvsrVimwqzXP79kwS46w8lssiEU0IDLw/9wDQ5Q0kfEims2rQwOFn9oJ8120ZrrvzoEL3vBuNrLJH/4mTh3/im3f6vErRs1xTlAfWQ7useT7WXV1qCyjEIrk4kih75qHIrRIgnDSItRrV97rn1ETAq14IuAQg3nUoy1SAUmHPwsU4BpD7clbdTpAJY8F164tj0TtSmICNTPV8XqkD2ySsTPfgfykw7L7dDe0L9Me30Pj85nxKicBdS66ToPdmYo1UVjgomNHlsVh0N81OZzUN464gIg/HQwEn9HMl1zA+teSafSNAPA X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(396003)(39860400002)(346002)(376002)(230922051799003)(1800799009)(451199024)(82310400011)(186009)(64100799003)(40470700004)(36840700001)(46966006)(40480700001)(5660300002)(36860700001)(40460700003)(7636003)(83380400001)(426003)(26005)(36756003)(6666004)(336012)(66574015)(2616005)(86362001)(356005)(47076005)(82740400003)(1076003)(2906002)(41300700001)(110136005)(316002)(478600001)(70206006)(70586007)(8936002)(54906003)(8676002)(4326008);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 17:16:14.2104 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 32c3215f-ec9c-47be-96ca-08dbcffdeed2 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000FCBE.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR12MB9332 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Wed, 18 Oct 2023 10:17:18 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780114461223432724 X-GMAIL-MSGID: 1780114461223432724 This patch adapts the mr creation/deletion code to be able to work with any given mr struct pointer. All the APIs are adapted to take an extra parameter for the mr. mlx5_vdpa_create/delete_mr doesn't need a ASID parameter anymore. The check is done in the caller instead (mlx5_set_map). This change is needed for a followup patch which will introduce an additional mr for the vq descriptor data. Signed-off-by: Dragos Tatulea Acked-by: Eugenio Pérez Acked-by: Jason Wang --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 8 +++-- drivers/vdpa/mlx5/core/mr.c | 53 ++++++++++++++---------------- drivers/vdpa/mlx5/net/mlx5_vnet.c | 10 ++++-- 3 files changed, 36 insertions(+), 35 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index e1e6e7aba50e..01d4ee58ccb1 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -116,10 +116,12 @@ int mlx5_vdpa_create_mkey(struct mlx5_vdpa_dev *mvdev, u32 *mkey, u32 *in, int mlx5_vdpa_destroy_mkey(struct mlx5_vdpa_dev *mvdev, u32 mkey); int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, bool *change_map, unsigned int asid); -int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, - unsigned int asid); +int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr, + struct vhost_iotlb *iotlb); void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev); -void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid); +void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr); int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid); diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 00dcce190a1f..6f29e8eaabb1 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -301,10 +301,13 @@ static void unmap_direct_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_direct sg_free_table(&mr->sg_head); } -static int add_direct_chain(struct mlx5_vdpa_dev *mvdev, u64 start, u64 size, u8 perm, +static int add_direct_chain(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr, + u64 start, + u64 size, + u8 perm, struct vhost_iotlb *iotlb) { - struct mlx5_vdpa_mr *mr = &mvdev->mr; struct mlx5_vdpa_direct_mr *dmr; struct mlx5_vdpa_direct_mr *n; LIST_HEAD(tmp); @@ -354,9 +357,10 @@ static int add_direct_chain(struct mlx5_vdpa_dev *mvdev, u64 start, u64 size, u8 * indirect memory key that provides access to the enitre address space given * by iotlb. */ -static int create_user_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb) +static int create_user_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr, + struct vhost_iotlb *iotlb) { - struct mlx5_vdpa_mr *mr = &mvdev->mr; struct mlx5_vdpa_direct_mr *dmr; struct mlx5_vdpa_direct_mr *n; struct vhost_iotlb_map *map; @@ -384,7 +388,7 @@ static int create_user_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb LOG_MAX_KLM_SIZE); mr->num_klms += nnuls; } - err = add_direct_chain(mvdev, ps, pe - ps, pperm, iotlb); + err = add_direct_chain(mvdev, mr, ps, pe - ps, pperm, iotlb); if (err) goto err_chain; } @@ -393,7 +397,7 @@ static int create_user_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb pperm = map->perm; } } - err = add_direct_chain(mvdev, ps, pe - ps, pperm, iotlb); + err = add_direct_chain(mvdev, mr, ps, pe - ps, pperm, iotlb); if (err) goto err_chain; @@ -489,13 +493,8 @@ static void destroy_user_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr } } -static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) +static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { - struct mlx5_vdpa_mr *mr = &mvdev->mr; - - if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] != asid) - return; - if (!mr->initialized) return; @@ -507,38 +506,33 @@ static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid mr->initialized = false; } -void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) +void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr) { - struct mlx5_vdpa_mr *mr = &mvdev->mr; - mutex_lock(&mr->mkey_mtx); - _mlx5_vdpa_destroy_mr(mvdev, asid); + _mlx5_vdpa_destroy_mr(mvdev, mr); mutex_unlock(&mr->mkey_mtx); } void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) { - mlx5_vdpa_destroy_mr(mvdev, mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]); + mlx5_vdpa_destroy_mr(mvdev, &mvdev->mr); prune_iotlb(mvdev); } static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, - struct vhost_iotlb *iotlb, - unsigned int asid) + struct mlx5_vdpa_mr *mr, + struct vhost_iotlb *iotlb) { - struct mlx5_vdpa_mr *mr = &mvdev->mr; int err; - if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] != asid) - return 0; - if (mr->initialized) return 0; if (iotlb) - err = create_user_mr(mvdev, iotlb); + err = create_user_mr(mvdev, mr, iotlb); else err = create_dma_mr(mvdev, mr); @@ -550,13 +544,14 @@ static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, return 0; } -int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, - unsigned int asid) +int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr, + struct vhost_iotlb *iotlb) { int err; mutex_lock(&mvdev->mr.mkey_mtx); - err = _mlx5_vdpa_create_mr(mvdev, iotlb, asid); + err = _mlx5_vdpa_create_mr(mvdev, mr, iotlb); mutex_unlock(&mvdev->mr.mkey_mtx); return err; } @@ -574,7 +569,7 @@ int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *io *change_map = true; } if (!*change_map) - err = _mlx5_vdpa_create_mr(mvdev, iotlb, asid); + err = _mlx5_vdpa_create_mr(mvdev, mr, iotlb); mutex_unlock(&mr->mkey_mtx); return err; @@ -603,7 +598,7 @@ int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvdev) { int err; - err = mlx5_vdpa_create_mr(mvdev, NULL, 0); + err = mlx5_vdpa_create_mr(mvdev, &mvdev->mr, NULL); if (err) return err; diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index ab196c43694c..256fdd80c321 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -2644,8 +2644,8 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, goto err_mr; teardown_driver(ndev); - mlx5_vdpa_destroy_mr(mvdev, asid); - err = mlx5_vdpa_create_mr(mvdev, iotlb, asid); + mlx5_vdpa_destroy_mr(mvdev, &mvdev->mr); + err = mlx5_vdpa_create_mr(mvdev, &mvdev->mr, iotlb); if (err) goto err_mr; @@ -2660,7 +2660,7 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, return 0; err_setup: - mlx5_vdpa_destroy_mr(mvdev, asid); + mlx5_vdpa_destroy_mr(mvdev, &mvdev->mr); err_mr: return err; } @@ -2878,6 +2878,9 @@ static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, bool change_map; int err; + if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] != asid) + goto end; + err = mlx5_vdpa_handle_set_map(mvdev, iotlb, &change_map, asid); if (err) { mlx5_vdpa_warn(mvdev, "set map failed(%d)\n", err); @@ -2890,6 +2893,7 @@ static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, return err; } +end: return mlx5_vdpa_update_cvq_iotlb(mvdev, iotlb, asid); } From patchwork Wed Oct 18 17:14:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 155017 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4945788vqb; Wed, 18 Oct 2023 10:17:30 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGUoCWxM5f1ul3cPxb4NgaDvm7WEAHeHPbM4IhZhcEb+DYS/cJqXVksqvQlNtwzSyN4S6Wh X-Received: by 2002:a05:6a20:a123:b0:159:f884:4d6e with SMTP id q35-20020a056a20a12300b00159f8844d6emr6716746pzk.40.1697649449763; Wed, 18 Oct 2023 10:17:29 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697649449; cv=pass; d=google.com; s=arc-20160816; b=hhtuuL2fmFycLcynWTmDULi6nB2fx6jjuCswwEiFtmCZoAP2u+hP3QS6TWNxMn81b4 sPBSPK3PntKQd1stxmeqZLPlzV8Ev43xnJp+/tqcC+0fCdSJUpbS3hlWsTBimvdsXQcS ouJhSJXgYmef+ZJDQOZOP1BWgd6CPE7UYuFm1c9rB+J59C6mvZZRma1foY+oYjpUtm8K pn14so6S/B9es1v8yv33pM3XG/Ix6N0NaDXa2+1PAONmVhE0wGANaW0JsdOhJw8mSnoI NP/alqTYYbepCVWtfOK8RNC9N0knPHnquulQCIPQHdxl9v51cUb50nCpxT+6eOmIB29f qqDg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=qb2hZXT5+/f8Ri3WhuaZoK6JttuOpOqJz8f5tpo4KeM=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=JCTn9ntixtXt18Avt/q8OKOvw3m+9NCOF4uczydqM/2lY+lViIaBXBdlk85ZxH9Rck xIcyXkP4vRXwB+PXvPmRx/aHdWZ0GF0bsYE3cXOITFIgPB5mnGEaa0vrupBX8Q5Fm9JQ OhS8EP5j9QL4hgT+rHZjTBTuLPQNqcuGRqCW6NLFx+N30T5mTy9Elj6j0mZ9uBcUOEgd O6dD89q/pam5jO/toQuMQ++i6i/4S6KMUO5gwyzLRcxkUkSa3u3GtlEPWR9bwYWkoChT PSo5gW2Pbq85+c/0UYJTdTDfDidbZ7Vq4uOA6DHhiiqwHHV6OB/oxyEglan9Wbl0rAnE 6TFg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=Nn1ur1k7; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id w13-20020a1709027b8d00b001c61bd7cee0si283301pll.211.2023.10.18.10.17.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 10:17:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=Nn1ur1k7; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 7789480D5876; Wed, 18 Oct 2023 10:17:27 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235362AbjJRRRQ (ORCPT + 24 others); Wed, 18 Oct 2023 13:17:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40320 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235337AbjJRRQh (ORCPT ); Wed, 18 Oct 2023 13:16:37 -0400 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2054.outbound.protection.outlook.com [40.107.223.54]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1332310C3; Wed, 18 Oct 2023 10:16:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MXxYSf60JY0aXF3lsU03Y5CUP15zTk7Q7oq7FacUJHAg/AQprZ9Cr2kXbhsv43n8FrwZvwcU0yx31VB9n4tkKW0vF8Qz7rP1RV8Z1/Hhn5nfFKOTDDVEZnCrYPYI9aZJfKoBEUNL0X6gMJgxemwPfa7bUJq8FRxTbL9+rFKTRnYTwHOPcrhY/cRXO2hiJiVeS0R5pD899XYASrvz5FX/tnmAf7cazKs+D0DcuMurBwPKjCl6uIIYD3XoSOgUdxiDqjI3rDkeebkwATIaCGgkhUiA04+3d3Q1zPj8VApcLUGVKQAjHnT78cxvTno4jHAdl6XDzEAzYH+IomhXK30WlA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qb2hZXT5+/f8Ri3WhuaZoK6JttuOpOqJz8f5tpo4KeM=; b=iNxjNVuFcXgjxWmU2+ujfPKil9V6Fiy+1ueFpiXo093K57PKgHYYERXtjZfsg1NymTMvhWsX7hKs2h5Rq1pXRg7apVjCuaAW6xqJn6D7ZbRhyki6ELdqbJEAu0fOg24bq/32E411zD9uQikT590uQAZGeuFZ91VPODxzpZOcDHzKTWHSgWUiumLuBZ0ihO7i5bGIPNflSCSnhklK7mNl2QWMZ5zB0Vhe2Qe6w4TZCzmpOHEhW4lRmhtT4VHXEjpDbbd6jjO2aRlIgc7pjZTG2dHojOiebarWNNRxHVxQ4lquzhCEWaA6OYssPhq41J0NuUYXLm0+9Ygt/MWbw4pgmw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qb2hZXT5+/f8Ri3WhuaZoK6JttuOpOqJz8f5tpo4KeM=; b=Nn1ur1k79NC5ruhUQwoIb+Pc8oXhdlGuzRbnsZ39KNyfMjmd7mDCPSIHTz6DPcbzV3o5zlX66Ga5i42rCfeBVrJe5Bzed975egex200/pmper2gYP/ndb1Yk4glmpFdlEDomFshBe3WJRIuttsPD8Q2zileR6Ze6TiWvtRca2320rHjJqzqpzkEeiejDNfBOaHJUufocyjsHyksi3Mlqw566big+ReC+dIWMFb9PWyjED/TmkZb5gjjxEGFTY8VXiq4/5krWHndLxtK6Ogh7aQe+Zc2nm3LjcpnMCxH0LLTpC94ctwGR3crotqU0PBKtY4VklSO6nwjk7ysv0P+NVA== Received: from CY5P221CA0154.NAMP221.PROD.OUTLOOK.COM (2603:10b6:930:6a::22) by PH8PR12MB7424.namprd12.prod.outlook.com (2603:10b6:510:228::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21; Wed, 18 Oct 2023 17:16:19 +0000 Received: from CY4PEPF0000FCBF.namprd03.prod.outlook.com (2603:10b6:930:6a:cafe::51) by CY5P221CA0154.outlook.office365.com (2603:10b6:930:6a::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.34 via Frontend Transport; Wed, 18 Oct 2023 17:16:19 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000FCBF.mail.protection.outlook.com (10.167.242.101) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.23 via Frontend Transport; Wed, 18 Oct 2023 17:16:19 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:16:05 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:16:05 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Wed, 18 Oct 2023 10:16:02 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 11/16] vdpa/mlx5: Move mr mutex out of mr struct Date: Wed, 18 Oct 2023 20:14:50 +0300 Message-ID: <20231018171456.1624030-13-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231018171456.1624030-2-dtatulea@nvidia.com> References: <20231018171456.1624030-2-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000FCBF:EE_|PH8PR12MB7424:EE_ X-MS-Office365-Filtering-Correlation-Id: baca3d41-04a9-4979-ead1-08dbcffdf1e2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8DzHpTO8LBf0AWSFkinLAuAlpXVQJx7pa4BZKxV05WPAi9MZpq0bkiZXNYx9kPbA/POG4BVeWWFY+UpkJIsscyz5dnD6CV/aYnfCd5PwhIwtowhM7Zu3NigdI3r5YC5rRNu2Bmr6eRzz0gdhwQDiz5TU5fNDYqYMoWZIoSzfGNMcbZhVNBcfewtYtD4T70FJoH/VsT7TQXas7+K9sczXdiJI44kVa/zWwF14caZAr0FphLx+5oKLq3NlXfKcu4tN3V7Axo/OckjCN1WuvKGIeDeLlRuZumBnw6EkaBI700RFsnyh9xSQ3gWtmfOcIWyiLJD7scs4loBaSypyHtOmQoZB9R9KDf4BbBW9qdCLsJZwg1I9YczDFiQ/yGFnM+O2MyxeX+bDKyK1rMVOR5KPWYuG3VDC2gwXu16+5MCptOO3dwYgJbBBUNJpyLHECbhZ5Gs/3BAnN/9tERap4yE7L2wtcc9cbWN1rvE8I2+tGq8aVYE0rO0TeMlFjkg59aKFixbRQXO44zc0gjfu1BOZHJt+7E5u/2uzJ6hto02R0eDjmeXI4O/WVS3lXOs15tyEOcDJEa9e60VELs5VhCxPU0T2VCNxzG4ZOrDdfJ0xl8mPPqBzh3kPV2/H/uhf3TxxI00uZII2s8Iafr7+Qu/9Tp+vfJxuvMXjfqMLWwaf2mqHeyBa1LswIao5qvnIXqLv1mrh5z00JYn7+DNMmL05UDwtI6cAZxeDmdAHuJfY+bwRG8HBb1Obfb0fQMf/X3Nb X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(136003)(376002)(396003)(346002)(230922051799003)(82310400011)(186009)(451199024)(64100799003)(1800799009)(46966006)(36840700001)(40470700004)(40460700003)(2616005)(47076005)(1076003)(426003)(66574015)(336012)(26005)(478600001)(36860700001)(8936002)(83380400001)(41300700001)(2906002)(8676002)(4326008)(6666004)(110136005)(5660300002)(70586007)(316002)(70206006)(54906003)(356005)(7636003)(40480700001)(36756003)(82740400003)(86362001)(66899024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 17:16:19.2881 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: baca3d41-04a9-4979-ead1-08dbcffdf1e2 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000FCBF.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB7424 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 18 Oct 2023 10:17:27 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780114469141091754 X-GMAIL-MSGID: 1780114469141091754 The mutex is named like it is supposed to protect only the mkey but in reality it is a global lock for all mr resources. Shift the mutex to it's rightful location (struct mlx5_vdpa_dev) and give it a more appropriate name. Signed-off-by: Dragos Tatulea Acked-by: Eugenio Pérez Acked-by: Jason Wang --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 4 ++-- drivers/vdpa/mlx5/core/mr.c | 13 +++++++------ drivers/vdpa/mlx5/core/resources.c | 6 +++--- 3 files changed, 12 insertions(+), 11 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 01d4ee58ccb1..9c6ac42c21e1 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -34,8 +34,6 @@ struct mlx5_vdpa_mr { /* state of dvq mr */ bool initialized; - /* serialize mkey creation and destruction */ - struct mutex mkey_mtx; bool user_mr; }; @@ -94,6 +92,8 @@ struct mlx5_vdpa_dev { u32 generation; struct mlx5_vdpa_mr mr; + /* serialize mr access */ + struct mutex mr_mtx; struct mlx5_control_vq cvq; struct workqueue_struct *wq; unsigned int group2asid[MLX5_VDPA_NUMVQ_GROUPS]; diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 6f29e8eaabb1..abd6a6fb122f 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -509,11 +509,11 @@ static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_ void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { - mutex_lock(&mr->mkey_mtx); + mutex_lock(&mvdev->mr_mtx); _mlx5_vdpa_destroy_mr(mvdev, mr); - mutex_unlock(&mr->mkey_mtx); + mutex_unlock(&mvdev->mr_mtx); } void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) @@ -550,9 +550,10 @@ int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, { int err; - mutex_lock(&mvdev->mr.mkey_mtx); + mutex_lock(&mvdev->mr_mtx); err = _mlx5_vdpa_create_mr(mvdev, mr, iotlb); - mutex_unlock(&mvdev->mr.mkey_mtx); + mutex_unlock(&mvdev->mr_mtx); + return err; } @@ -563,14 +564,14 @@ int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *io int err = 0; *change_map = false; - mutex_lock(&mr->mkey_mtx); + mutex_lock(&mvdev->mr_mtx); if (mr->initialized) { mlx5_vdpa_info(mvdev, "memory map update\n"); *change_map = true; } if (!*change_map) err = _mlx5_vdpa_create_mr(mvdev, mr, iotlb); - mutex_unlock(&mr->mkey_mtx); + mutex_unlock(&mvdev->mr_mtx); return err; } diff --git a/drivers/vdpa/mlx5/core/resources.c b/drivers/vdpa/mlx5/core/resources.c index d5a59c9035fb..5c5a41b64bfc 100644 --- a/drivers/vdpa/mlx5/core/resources.c +++ b/drivers/vdpa/mlx5/core/resources.c @@ -256,7 +256,7 @@ int mlx5_vdpa_alloc_resources(struct mlx5_vdpa_dev *mvdev) mlx5_vdpa_warn(mvdev, "resources already allocated\n"); return -EINVAL; } - mutex_init(&mvdev->mr.mkey_mtx); + mutex_init(&mvdev->mr_mtx); res->uar = mlx5_get_uars_page(mdev); if (IS_ERR(res->uar)) { err = PTR_ERR(res->uar); @@ -301,7 +301,7 @@ int mlx5_vdpa_alloc_resources(struct mlx5_vdpa_dev *mvdev) err_uctx: mlx5_put_uars_page(mdev, res->uar); err_uars: - mutex_destroy(&mvdev->mr.mkey_mtx); + mutex_destroy(&mvdev->mr_mtx); return err; } @@ -318,6 +318,6 @@ void mlx5_vdpa_free_resources(struct mlx5_vdpa_dev *mvdev) dealloc_pd(mvdev, res->pdn, res->uid); destroy_uctx(mvdev, res->uid); mlx5_put_uars_page(mvdev->mdev, res->uar); - mutex_destroy(&mvdev->mr.mkey_mtx); + mutex_destroy(&mvdev->mr_mtx); res->valid = false; } From patchwork Wed Oct 18 17:14:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 155019 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4946330vqb; Wed, 18 Oct 2023 10:18:20 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFrgqg21CZu8RxtYJPuhtggCtOhyLwEsivs6ylcXXEIwxpPFeJ5Eq/VPfJy9Uqm2mqkQVOw X-Received: by 2002:a05:6a20:2208:b0:17a:7c2:d4a6 with SMTP id u8-20020a056a20220800b0017a07c2d4a6mr4264692pzb.55.1697649500730; Wed, 18 Oct 2023 10:18:20 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697649500; cv=pass; d=google.com; s=arc-20160816; b=W58dm/EHGhp8Vz1caBlXFImrkyMvfLj4r+i7trNm1YlIQQKG4sEQle+lezF7NG6kJr q6swrxgmdaqIr3nESW2ZzjC5IJ47EtbIj/rz1GD0B/aA0NAXlYpzsCUo0SWLE4aP5yXF RtCeEfZVaBUYjuBrUW/wyJIzfXAfa0t1MGHBDJcWqF95Bqjo9BDEQaNAdDk7hOMIv8b/ 6k/SihBFnZsn6mz0gv2JlMttog3QbT29kz7XNJt2EggHsHfCBOsq4cucomD8fxYQVvCu s7LdagqMHNUrL/fEVZUHGQNSzJeBrpPaHKZflIMNEhsTzzDrB0WYKFslgXw4hYiXFNuE 2M/w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=j+yKp2e1QHrSlE2/K+1Q9nARtdxnRNsDTO9eHQUcPRA=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=DQDE7FyPkV5dGBhrDLwwvNL333+8TTjQHZaah2pbDDt1Ee+Fp0Aii4CynOPEenMfPh +2pX+eY5HHwCvg/+7an39ZOa3L6rQaPhfxF4NI8etvJYAyA3A+exPkQRDTZe0GugweuU Al0OlTmeICzFOtKIagsGzy4We9rdrIehmfInZXXnxKwVJyyhAS1msGmouatIbRm3pq1G CtSX7c9c2KFHmmQtWCUjhcvWyTgmLKq95V/5MNYBIl36XbVn02XtiSK5DiTN21Nyd0V/ YbcaYHcT5PBu0qgCsCmHgJzaz+GgDnDXZLj31qar3Pav4PMVQXE4niGjbAeDIpDrHwGk gpwQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=IrmamJsF; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id z1-20020a633301000000b00582f1f73c82si2279443pgz.381.2023.10.18.10.18.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 10:18:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=IrmamJsF; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 541258023A9E; Wed, 18 Oct 2023 10:18:17 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232057AbjJRRRa (ORCPT + 24 others); Wed, 18 Oct 2023 13:17:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235402AbjJRRQk (ORCPT ); Wed, 18 Oct 2023 13:16:40 -0400 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2047.outbound.protection.outlook.com [40.107.244.47]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E7F8189; Wed, 18 Oct 2023 10:16:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fsyAf+op9S9BBb9bqex+8v6EERwV5yXMgWErn813SY5rDY768Q/YM8dytFvW9mGPnswqe8Ic7DstwJ2SNHO/JK/TrkswYLxdlA/v1kOyBB9poKXSRSrq9/DPk4E4CV1S/nFgD4onFR+A6LKIrQZ76mmbEelXKJ+wrXdFJ4VMdFX0K33hEAWyhsEwsAwkElmC5GbE6cYGiCDzDALUEM5g8avLss5xZeFIaOxc4rIzzaXJU94qkOyqXe3lXQWOeSct6CZ3TDepHicvE3v+u+K3Okw87ZSFuS6GG/a+8VXaSRNMi/6+iBLgD0vNKfYAr9UqgzKTpzD3YoJDfV3agxpAtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=j+yKp2e1QHrSlE2/K+1Q9nARtdxnRNsDTO9eHQUcPRA=; b=AmRcUJYc8JAEhUhTsl8A49UVaSvInguozEh3CGGwdvr+tFGDaUWHn97OoRvZAthNLXbQmDFHdW91X2OSv9eLrJ9C/Twe52RyFvaamrzj4Wa5hz3Tb33bAbjIBlg5uOyg2seK75pRmdlkrdBm8XhLmsSQrkJufhXOD3EbuCG6qZaa17QB/j2B7jtUO2k80WoO8+094hK8PF4JuzkeCLvZ16a8cg9SwHt55m8iq1hYb6EwpWUao8hyyX19FgPw6TZpiLQktLkryUrDU11kZ2OXRckEnjK4gm+UxJrd2t5F2mTiyxAeyNoxYTwFsTvDVU5EIxFdrmVJaEcYL6J6SwWVXw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=j+yKp2e1QHrSlE2/K+1Q9nARtdxnRNsDTO9eHQUcPRA=; b=IrmamJsFjnIQg7D62DgfvfluJVuD7b+TwKTQSC/DgE+0aSJvVrhXn8WaH4LwH2pPIhRT83C5EJQO7OnPaBuo2vIK3/T+8RrbdR3LJz+oHn1xeIttx+9vvA10bW/ej94zUNAbWPnAmu1q9wSmQhLOVdSjja3GznBgNrJFrshUYy9tjVy1s++WIhEKSYKmjDKFlwBoHnD535fyfSWMjDQhHgznp3+L+WGj9CCi362e0P6tqrSMU1SbODseJ54YVj/aro8wcPPmb2O1tY1PgVAYnGvObc6uOo4Y+B1ab+95nibIM6PiwOjXvbsbfHZgh38h9WJ81FYJHFpWbaf21BOonQ== Received: from CY8P220CA0034.NAMP220.PROD.OUTLOOK.COM (2603:10b6:930:47::15) by MN0PR12MB6367.namprd12.prod.outlook.com (2603:10b6:208:3d3::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.36; Wed, 18 Oct 2023 17:16:23 +0000 Received: from CY4PEPF0000FCC5.namprd03.prod.outlook.com (2603:10b6:930:47:cafe::25) by CY8P220CA0034.outlook.office365.com (2603:10b6:930:47::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.30 via Frontend Transport; Wed, 18 Oct 2023 17:16:22 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000FCC5.mail.protection.outlook.com (10.167.242.107) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.22 via Frontend Transport; Wed, 18 Oct 2023 17:16:22 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:16:09 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:16:08 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Wed, 18 Oct 2023 10:16:05 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 12/16] vdpa/mlx5: Improve mr update flow Date: Wed, 18 Oct 2023 20:14:51 +0300 Message-ID: <20231018171456.1624030-14-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231018171456.1624030-2-dtatulea@nvidia.com> References: <20231018171456.1624030-2-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000FCC5:EE_|MN0PR12MB6367:EE_ X-MS-Office365-Filtering-Correlation-Id: 34232203-ccbe-4cb1-fa0f-08dbcffdf3ea X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: dUPA8LAZ+cXKhaCogU8k1Wwz3OSZYsXwpY/39z0byjSt0VDW/Km+ILR1RjL1UyTerXGBLBt1Nz35wZudSBdwPtQIVuKa8k/UPWusdtujMb5l1FtEM0uGg5V3Xz9fWjb2UOHjoGmf3i+LCZ/WSizclzrIvX1LPlmg+2PIrHv7ghc8vjk0VYZfrF7qab2omxFwHd2PcYYuUi7yEkU4kHsGyy30zl0yhTaTAN4QTvYtgvKSazYrgoWWjm/wr80JQTY4GOLt62Io6nIbgeLleewt/lwKAhLzuTc9Jllo6v5bSMq9jZXQneGuyWS/ADOA9wR2FyZLcz2q23adDG73rKW/kroRUMfeXTWrw0ECnu1G4Ls68ECorWqj7+z9FRhaw6q/ms59gw8WORXbfG1i9vl0h4t476DDp3rGQCY2dG+SNTIhUCXJ+sjXHljQgnSw+IVaC8gC72wFurr8LKC6e2OAkAXAACNJtG8pKe8TPO4vcLvHjTqWMVlIrYnjG/tEg0tih+un/bLYZtE9KUazRE0xJHVEhXDNbB6tQ9LjwghcBstMMMsPxywGMoYklF7nRZV37FEJaniby8jEbvSdN9rqGUmfQABeFRHmF49nA+R1AdMzut22W7QAmRVFXdC2ToPIC4MQo2RCTivI6jiE0+e2yt9U3cK0NTa9bts4zakruc1j/BqceUbcDyqF1o8nwCiKFSmKGSaP50mD4N+vbIq593nEdMWRuqPwapNN/LoR874Te8UVqjfmrihS311ygSt7MYxl4KcjFXhJ5j3h41q6Bg== X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(376002)(396003)(39860400002)(346002)(230922051799003)(451199024)(186009)(82310400011)(1800799009)(64100799003)(36840700001)(40470700004)(46966006)(40480700001)(40460700003)(70206006)(6666004)(70586007)(54906003)(110136005)(478600001)(83380400001)(36860700001)(47076005)(356005)(336012)(82740400003)(316002)(26005)(86362001)(1076003)(426003)(66574015)(2616005)(7636003)(36756003)(5660300002)(8936002)(8676002)(41300700001)(2906002)(4326008)(15650500001)(334744004);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 17:16:22.6808 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 34232203-ccbe-4cb1-fa0f-08dbcffdf3ea X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000FCC5.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6367 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Wed, 18 Oct 2023 10:18:17 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780114523317628834 X-GMAIL-MSGID: 1780114523317628834 The current flow for updating an mr works directly on mvdev->mr which makes it cumbersome to handle multiple new mr structs. This patch makes the flow more straightforward by having mlx5_vdpa_create_mr return a new mr which will update the old mr (if any). The old mr will be deleted and unlinked from mvdev. For the case when the iotlb is empty (not NULL), the old mr will be cleared. This change paves the way for adding mrs for different ASIDs. The initialized bool is no longer needed as mr is now a pointer in the mlx5_vdpa_dev struct which will be NULL when not initialized. Acked-by: Eugenio Pérez Acked-by: Jason Wang Signed-off-by: Dragos Tatulea Acked-by: Eugenio Pérez --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 14 +++-- drivers/vdpa/mlx5/core/mr.c | 87 ++++++++++++++++-------------- drivers/vdpa/mlx5/net/mlx5_vnet.c | 53 +++++++++--------- 3 files changed, 82 insertions(+), 72 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 9c6ac42c21e1..bbe4335106bd 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -31,8 +31,6 @@ struct mlx5_vdpa_mr { struct list_head head; unsigned long num_directs; unsigned long num_klms; - /* state of dvq mr */ - bool initialized; bool user_mr; }; @@ -91,7 +89,7 @@ struct mlx5_vdpa_dev { u16 max_idx; u32 generation; - struct mlx5_vdpa_mr mr; + struct mlx5_vdpa_mr *mr; /* serialize mr access */ struct mutex mr_mtx; struct mlx5_control_vq cvq; @@ -114,14 +112,14 @@ void mlx5_vdpa_free_resources(struct mlx5_vdpa_dev *mvdev); int mlx5_vdpa_create_mkey(struct mlx5_vdpa_dev *mvdev, u32 *mkey, u32 *in, int inlen); int mlx5_vdpa_destroy_mkey(struct mlx5_vdpa_dev *mvdev, u32 mkey); -int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, - bool *change_map, unsigned int asid); -int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, - struct mlx5_vdpa_mr *mr, - struct vhost_iotlb *iotlb); +struct mlx5_vdpa_mr *mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, + struct vhost_iotlb *iotlb); void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev); void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr); +void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr, + unsigned int asid); int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid); diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index abd6a6fb122f..00eff5a07152 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -495,30 +495,51 @@ static void destroy_user_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { - if (!mr->initialized) - return; - if (mr->user_mr) destroy_user_mr(mvdev, mr); else destroy_dma_mr(mvdev, mr); - - mr->initialized = false; } void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { + if (!mr) + return; + mutex_lock(&mvdev->mr_mtx); _mlx5_vdpa_destroy_mr(mvdev, mr); + if (mvdev->mr == mr) + mvdev->mr = NULL; + + mutex_unlock(&mvdev->mr_mtx); + + kfree(mr); +} + +void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *new_mr, + unsigned int asid) +{ + struct mlx5_vdpa_mr *old_mr = mvdev->mr; + + mutex_lock(&mvdev->mr_mtx); + + mvdev->mr = new_mr; + if (old_mr) { + _mlx5_vdpa_destroy_mr(mvdev, old_mr); + kfree(old_mr); + } + mutex_unlock(&mvdev->mr_mtx); + } void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) { - mlx5_vdpa_destroy_mr(mvdev, &mvdev->mr); + mlx5_vdpa_destroy_mr(mvdev, mvdev->mr); prune_iotlb(mvdev); } @@ -528,52 +549,36 @@ static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, { int err; - if (mr->initialized) - return 0; - if (iotlb) err = create_user_mr(mvdev, mr, iotlb); else err = create_dma_mr(mvdev, mr); - if (err) - return err; - - mr->initialized = true; - - return 0; + return err; } -int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, - struct mlx5_vdpa_mr *mr, - struct vhost_iotlb *iotlb) +struct mlx5_vdpa_mr *mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, + struct vhost_iotlb *iotlb) { + struct mlx5_vdpa_mr *mr; int err; + mr = kzalloc(sizeof(*mr), GFP_KERNEL); + if (!mr) + return ERR_PTR(-ENOMEM); + mutex_lock(&mvdev->mr_mtx); err = _mlx5_vdpa_create_mr(mvdev, mr, iotlb); mutex_unlock(&mvdev->mr_mtx); - return err; -} - -int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, - bool *change_map, unsigned int asid) -{ - struct mlx5_vdpa_mr *mr = &mvdev->mr; - int err = 0; + if (err) + goto out_err; - *change_map = false; - mutex_lock(&mvdev->mr_mtx); - if (mr->initialized) { - mlx5_vdpa_info(mvdev, "memory map update\n"); - *change_map = true; - } - if (!*change_map) - err = _mlx5_vdpa_create_mr(mvdev, mr, iotlb); - mutex_unlock(&mvdev->mr_mtx); + return mr; - return err; +out_err: + kfree(mr); + return ERR_PTR(err); } int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, @@ -597,11 +602,13 @@ int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvdev) { - int err; + struct mlx5_vdpa_mr *mr; - err = mlx5_vdpa_create_mr(mvdev, &mvdev->mr, NULL); - if (err) - return err; + mr = mlx5_vdpa_create_mr(mvdev, NULL); + if (IS_ERR(mr)) + return PTR_ERR(mr); + + mlx5_vdpa_update_mr(mvdev, mr, 0); return mlx5_vdpa_update_cvq_iotlb(mvdev, NULL, 0); } diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 256fdd80c321..7b878995b6aa 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -873,7 +873,7 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque MLX5_SET64(virtio_q, vq_ctx, desc_addr, mvq->desc_addr); MLX5_SET64(virtio_q, vq_ctx, used_addr, mvq->device_addr); MLX5_SET64(virtio_q, vq_ctx, available_addr, mvq->driver_addr); - MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, ndev->mvdev.mr.mkey); + MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, ndev->mvdev.mr->mkey); MLX5_SET(virtio_q, vq_ctx, umem_1_id, mvq->umem1.id); MLX5_SET(virtio_q, vq_ctx, umem_1_size, mvq->umem1.size); MLX5_SET(virtio_q, vq_ctx, umem_2_id, mvq->umem2.id); @@ -2633,7 +2633,7 @@ static void restore_channels_info(struct mlx5_vdpa_net *ndev) } static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, - struct vhost_iotlb *iotlb, unsigned int asid) + struct mlx5_vdpa_mr *new_mr, unsigned int asid) { struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev); int err; @@ -2641,27 +2641,18 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, suspend_vqs(ndev); err = save_channels_info(ndev); if (err) - goto err_mr; + return err; teardown_driver(ndev); - mlx5_vdpa_destroy_mr(mvdev, &mvdev->mr); - err = mlx5_vdpa_create_mr(mvdev, &mvdev->mr, iotlb); - if (err) - goto err_mr; + + mlx5_vdpa_update_mr(mvdev, new_mr, asid); if (!(mvdev->status & VIRTIO_CONFIG_S_DRIVER_OK) || mvdev->suspended) - goto err_mr; + return 0; restore_channels_info(ndev); err = setup_driver(mvdev); - if (err) - goto err_setup; - - return 0; -err_setup: - mlx5_vdpa_destroy_mr(mvdev, &mvdev->mr); -err_mr: return err; } @@ -2875,26 +2866,40 @@ static u32 mlx5_vdpa_get_generation(struct vdpa_device *vdev) static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid) { - bool change_map; + struct mlx5_vdpa_mr *new_mr; int err; if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] != asid) goto end; - err = mlx5_vdpa_handle_set_map(mvdev, iotlb, &change_map, asid); - if (err) { - mlx5_vdpa_warn(mvdev, "set map failed(%d)\n", err); - return err; + if (vhost_iotlb_itree_first(iotlb, 0, U64_MAX)) { + new_mr = mlx5_vdpa_create_mr(mvdev, iotlb); + if (IS_ERR(new_mr)) { + err = PTR_ERR(new_mr); + mlx5_vdpa_warn(mvdev, "create map failed(%d)\n", err); + return err; + } + } else { + /* Empty iotlbs don't have an mr but will clear the previous mr. */ + new_mr = NULL; } - if (change_map) { - err = mlx5_vdpa_change_map(mvdev, iotlb, asid); - if (err) - return err; + if (!mvdev->mr) { + mlx5_vdpa_update_mr(mvdev, new_mr, asid); + } else { + err = mlx5_vdpa_change_map(mvdev, new_mr, asid); + if (err) { + mlx5_vdpa_warn(mvdev, "change map failed(%d)\n", err); + goto out_err; + } } end: return mlx5_vdpa_update_cvq_iotlb(mvdev, iotlb, asid); + +out_err: + mlx5_vdpa_destroy_mr(mvdev, new_mr); + return err; } static int mlx5_vdpa_set_map(struct vdpa_device *vdev, unsigned int asid, From patchwork Wed Oct 18 17:14:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 155018 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4945958vqb; Wed, 18 Oct 2023 10:17:45 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHG2AzY8OH7KNIhbyaZmiiQT6SOlhHHjPUSq0j2+8FWw/vR234yX5bOWHfZR0mcAmN2aSQz X-Received: by 2002:a17:902:8218:b0:1c9:dfb8:a5a0 with SMTP id x24-20020a170902821800b001c9dfb8a5a0mr4430pln.10.1697649465142; Wed, 18 Oct 2023 10:17:45 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697649465; cv=pass; d=google.com; s=arc-20160816; b=B/F541SEbMuyEsfKpvHi51WhRx0e5N3/KEBouavYIpkNp+xoQIe8UAK4+PeIgNlqhS CL7e4/nYd7utWS1Ym8skNtbxxXugijmMcvsY/xfiLeIrz/rgMdIOGV9YS4SA9zoLvJYy P9zN4dEN8AspRdO2FGuvyBLs92JNC6EN29A53tK8w898vC2Ba9r2nOBsfViipz/9Dt7B nXdkgkOHOYZqrbXbSFGkif8FCbeIGXMjdyC3Y++qcqzFhCKsWNFJH4CFang2vAmtswEA tlprK8BWrGoHWmbyp/papjMaduYyCr8oIqL9P//fkVBT/eh/dxe9TSdE2BfrtPLeLS5W 6TWg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=g/riwkRSoi1nVotT7iLhK2hAguH78ZBh8oaLsGcPRlY=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=Tc2SJ6543bEvHnIHE8oRKEo8qolLZAdlapkj7ZU6G9jWAn8QZ6ahlu9FwQA6Yh5Rnq DA0D7GFhXR6+6BT31+HdQIj3MdVYtK8cccSUy2kMYfZqf6PPDWJ7iYY/3CBwx7V3U+gI 2GzL/uaENMPAuAMLB5ZXSQoFX+OUr/uiym8+xmpygwgOsnCzcg/uPxctzEMKT1lDJcpb uRW/QkFMd4udPyYh/MSneCqb25VQYq7hS8R/tzLzYFYHYb8kMKBBZ/pSK3QmtXUhmzIo pF+gUCcnqxeJeGslMC93JTrji5QXJSUI+bRb6PdjSl+6XOvM3dCEvzciAH/eOim+vf2s qdog== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=Bp4AuE4I; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id f11-20020a17090274cb00b001c62ca6d540si310740plt.77.2023.10.18.10.17.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 10:17:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=Bp4AuE4I; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id D399A81B895C; Wed, 18 Oct 2023 10:17:43 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232965AbjJRRRh (ORCPT + 24 others); Wed, 18 Oct 2023 13:17:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43412 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235252AbjJRRRE (ORCPT ); Wed, 18 Oct 2023 13:17:04 -0400 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2050.outbound.protection.outlook.com [40.107.237.50]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2EFFE1A2; Wed, 18 Oct 2023 10:16:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VpRj7sFrdqTJF2zTAcRS2tugm6DZx6p7nxYr3PlHDgfX0eoiiNc1o4QYowH3HqaygueV/Szx/J/6bADTqfXpSMkz1+XcgSzeeF9kVOreQe9ldyP9cZ4tdwZIAUgSA+57072/ftT26IDL3Hvtl5N/YbEmr67VJlC3W2D+DYqp2zCZY9dl+jFPdqVvkFaf5aeEnREnKrFpqx2t8xQrlR5z5ZjC8oc0Hdz5o5cUmW/f8jyAFFirazbvyxbJg7NIs5gr8NVKBNo/T6Y03RaGGw405vl6St0t6FKxec3SorkLSo38duw9wSfH9JPDuQARkd/JJ0LWqcqL/vhTk/BGJV0nFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=g/riwkRSoi1nVotT7iLhK2hAguH78ZBh8oaLsGcPRlY=; b=ktq4HaNCBVPcivo0+Q9ccTnfFQ0QOr/k9PbElV1KdpYRUQKV+Z6lC8osSxN/gjZF/1lZcBYXoB6SGVspgK9oKbLiQcRFf8eM51t0myeo3u1Rc2C0CFrdMVgidmp/h+tcHj+X1rU395h4kZjvEgJrwMK4x/tAoiC4X/h3eWwkG/u0Q3ydmDq+A2h0fP3/Rd+jEPU5eBuvWFRay1MZFyHiW9S8r+U4+lCa4O4ao4hnuSlEjZVlPSmGMfvoM18CX8ZTD89N2Cus28jQDtP4f8QSXpexSD5zOmMMX8dLPbWUzcKEgKEkxuoDrSRcdQvb8Kg27JhjvGaQ5xvOUTFNgxw44Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=g/riwkRSoi1nVotT7iLhK2hAguH78ZBh8oaLsGcPRlY=; b=Bp4AuE4Ixc4c8pHb2s38uxHpa7yRmGloy97Wrzb23CjJEsa26gdLKZofy7EWGyKoFfazyRExlSyA/VC2HWo/n7q0MxnDTsb2hW+QQmB+z0OmktY/MFqFB1a/1wmSnsHoVbMh7OWZ6ZLiGSTsu/A0EpRYur6P7NVKEUssWNrMCRvTzdotDW6PsrXXyWiFKlW/rkam3jTZUo5mNezRktoYM0G7Vgv3J/NhvqNvps/cOS9TZ7YjFsYo3UZjY+kTrc7XfY5kOWJNk1zylItMc/OYmgfkEKWW1a9wfDfM6KUmNP/jVc7PuHU17LhY2oncFoC2hLC/O89HUV6QerDd/g8xEA== Received: from CY5P221CA0143.NAMP221.PROD.OUTLOOK.COM (2603:10b6:930:6a::13) by SA0PR12MB4527.namprd12.prod.outlook.com (2603:10b6:806:73::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.36; Wed, 18 Oct 2023 17:16:26 +0000 Received: from CY4PEPF0000FCBF.namprd03.prod.outlook.com (2603:10b6:930:6a:cafe::bb) by CY5P221CA0143.outlook.office365.com (2603:10b6:930:6a::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.35 via Frontend Transport; Wed, 18 Oct 2023 17:16:25 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by CY4PEPF0000FCBF.mail.protection.outlook.com (10.167.242.101) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.23 via Frontend Transport; Wed, 18 Oct 2023 17:16:25 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:16:12 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:16:12 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Wed, 18 Oct 2023 10:16:09 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 13/16] vdpa/mlx5: Introduce mr for vq descriptor Date: Wed, 18 Oct 2023 20:14:52 +0300 Message-ID: <20231018171456.1624030-15-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231018171456.1624030-2-dtatulea@nvidia.com> References: <20231018171456.1624030-2-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000FCBF:EE_|SA0PR12MB4527:EE_ X-MS-Office365-Filtering-Correlation-Id: cd83ee9a-e141-43fa-95b0-08dbcffdf5bb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: KcKMgssP9vdVKuyhJMCqPmzmE3aNah2+q5ca8h0ifg3WWRaXJPAujkOAEndDlYZKotHAjQwotDxcFWc6Xf8AQbHckJi//p2fd7GEZfy1OuixDoC7iq1RhpX63WY9fCp7YHidIOIMY6/kB6H0QnYEQPPha63naQmMAbw8/YNy3QFSpaah4aGFyW2yAJPUgJTNVotSQwAsgxzDBSYLv1ZI7BDqCjD+VdzUlYAP5vbMBv8Cum9U5n/eQuvOAgKzHv48pAEA8+mK9cTpbXgGeZBZic7KhLqZP5/iuF2fmyjX4o/TcGYdIjU+gAGfi8QDOB6nPL2mMG6kz1cxRaM++MoKhfdhq8m73favLjB/wQ+oOrQTV2SpRk7Kem/CZtrB70WVtPqwWMAD1FyrsqonK6wSeZMabpUGxte/cdhGLMrBuBQvWi5rVFpqXms/nbM2wVm2rYv0mizshfQl4hbxrwyUO5ufQn1sOZ1jUJHyCjFPU7DouoiaT37a6HQ1dt0K0tKcd2HrQ8QecXBIAaR0W+K7u7vj1tQIxeRq47vWNvtvKtr7hoYzOfn9JcRo3zLhAK5edvISQ+gsP9oYWYpKCA5nAlP2YTrPNXG9O5ah+KoXjEK50MMv4NZAkX0Luw0V4aLPioYbC2H5+rEc2qsHZIogdkEznGFBmWbqrzVbe3Eu+qPfqkf3IOLel+tbGtbKkFnCJXXvg2K2waeiEZrdu5Nmle5g6grVogYbNNSkq2iGKgg= X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(376002)(346002)(39860400002)(136003)(396003)(230922051799003)(451199024)(1800799009)(82310400011)(186009)(64100799003)(36840700001)(40470700004)(46966006)(40460700003)(1076003)(26005)(6666004)(426003)(4326008)(2616005)(83380400001)(336012)(36860700001)(66574015)(8936002)(8676002)(316002)(5660300002)(47076005)(41300700001)(2906002)(478600001)(82740400003)(110136005)(70206006)(54906003)(70586007)(86362001)(7636003)(356005)(36756003)(40480700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 17:16:25.7881 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cd83ee9a-e141-43fa-95b0-08dbcffdf5bb X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000FCBF.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4527 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 18 Oct 2023 10:17:43 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780114485441928014 X-GMAIL-MSGID: 1780114485441928014 Introduce the vq descriptor group and mr per ASID. Until now .set_map on ASID 1 was only updating the cvq iotlb. From now on it also creates a mkey for it. The current patch doesn't use it but follow-up patches will add hardware support for mapping the vq descriptors. Acked-by: Jason Wang Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 5 +++-- drivers/vdpa/mlx5/core/mr.c | 14 +++++++++----- drivers/vdpa/mlx5/net/mlx5_vnet.c | 20 +++++++++++++------- 3 files changed, 25 insertions(+), 14 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index bbe4335106bd..ae09296f4270 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -70,11 +70,12 @@ struct mlx5_vdpa_wq_ent { enum { MLX5_VDPA_DATAVQ_GROUP, MLX5_VDPA_CVQ_GROUP, + MLX5_VDPA_DATAVQ_DESC_GROUP, MLX5_VDPA_NUMVQ_GROUPS }; enum { - MLX5_VDPA_NUM_AS = MLX5_VDPA_NUMVQ_GROUPS + MLX5_VDPA_NUM_AS = 2 }; struct mlx5_vdpa_dev { @@ -89,7 +90,7 @@ struct mlx5_vdpa_dev { u16 max_idx; u32 generation; - struct mlx5_vdpa_mr *mr; + struct mlx5_vdpa_mr *mr[MLX5_VDPA_NUM_AS]; /* serialize mr access */ struct mutex mr_mtx; struct mlx5_control_vq cvq; diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 00eff5a07152..3dee6d9bed6b 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -511,8 +511,10 @@ void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, _mlx5_vdpa_destroy_mr(mvdev, mr); - if (mvdev->mr == mr) - mvdev->mr = NULL; + for (int i = 0; i < MLX5_VDPA_NUM_AS; i++) { + if (mvdev->mr[i] == mr) + mvdev->mr[i] = NULL; + } mutex_unlock(&mvdev->mr_mtx); @@ -523,11 +525,11 @@ void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *new_mr, unsigned int asid) { - struct mlx5_vdpa_mr *old_mr = mvdev->mr; + struct mlx5_vdpa_mr *old_mr = mvdev->mr[asid]; mutex_lock(&mvdev->mr_mtx); - mvdev->mr = new_mr; + mvdev->mr[asid] = new_mr; if (old_mr) { _mlx5_vdpa_destroy_mr(mvdev, old_mr); kfree(old_mr); @@ -539,7 +541,9 @@ void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) { - mlx5_vdpa_destroy_mr(mvdev, mvdev->mr); + for (int i = 0; i < MLX5_VDPA_NUM_AS; i++) + mlx5_vdpa_destroy_mr(mvdev, mvdev->mr[i]); + prune_iotlb(mvdev); } diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 7b878995b6aa..ea76c0b4b78e 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -821,6 +821,8 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque { int inlen = MLX5_ST_SZ_BYTES(create_virtio_net_q_in); u32 out[MLX5_ST_SZ_DW(create_virtio_net_q_out)] = {}; + struct mlx5_vdpa_dev *mvdev = &ndev->mvdev; + struct mlx5_vdpa_mr *vq_mr; void *obj_context; u16 mlx_features; void *cmd_hdr; @@ -873,7 +875,9 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque MLX5_SET64(virtio_q, vq_ctx, desc_addr, mvq->desc_addr); MLX5_SET64(virtio_q, vq_ctx, used_addr, mvq->device_addr); MLX5_SET64(virtio_q, vq_ctx, available_addr, mvq->driver_addr); - MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, ndev->mvdev.mr->mkey); + vq_mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]]; + if (vq_mr) + MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, vq_mr->mkey); MLX5_SET(virtio_q, vq_ctx, umem_1_id, mvq->umem1.id); MLX5_SET(virtio_q, vq_ctx, umem_1_size, mvq->umem1.size); MLX5_SET(virtio_q, vq_ctx, umem_2_id, mvq->umem2.id); @@ -2633,7 +2637,8 @@ static void restore_channels_info(struct mlx5_vdpa_net *ndev) } static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, - struct mlx5_vdpa_mr *new_mr, unsigned int asid) + struct mlx5_vdpa_mr *new_mr, + unsigned int asid) { struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev); int err; @@ -2652,8 +2657,10 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, restore_channels_info(ndev); err = setup_driver(mvdev); + if (err) + return err; - return err; + return 0; } /* reslock must be held for this function */ @@ -2869,8 +2876,8 @@ static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, struct mlx5_vdpa_mr *new_mr; int err; - if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] != asid) - goto end; + if (asid >= MLX5_VDPA_NUM_AS) + return -EINVAL; if (vhost_iotlb_itree_first(iotlb, 0, U64_MAX)) { new_mr = mlx5_vdpa_create_mr(mvdev, iotlb); @@ -2884,7 +2891,7 @@ static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, new_mr = NULL; } - if (!mvdev->mr) { + if (!mvdev->mr[asid]) { mlx5_vdpa_update_mr(mvdev, new_mr, asid); } else { err = mlx5_vdpa_change_map(mvdev, new_mr, asid); @@ -2894,7 +2901,6 @@ static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, } } -end: return mlx5_vdpa_update_cvq_iotlb(mvdev, iotlb, asid); out_err: From patchwork Wed Oct 18 17:14:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 155023 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4947112vqb; Wed, 18 Oct 2023 10:19:36 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF7BazX472psf4mef7R4wehQ8PT80rucp6ErozAwxN4vwbNV9YDeBL/CdQ7COdqg3U17bhQ X-Received: by 2002:a17:903:70e:b0:1c8:9832:827f with SMTP id kk14-20020a170903070e00b001c89832827fmr4866236plb.20.1697649575808; Wed, 18 Oct 2023 10:19:35 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697649575; cv=pass; d=google.com; s=arc-20160816; b=QId9dpUzR0dvafp2ZECxb0ex/eXnKsK3hulFlz51VAJG6oUxjPrditNAn6BFkpymTX hk9P5IILxhdfo3dmnIdokz2+aJEBdvlVBl2wZgqhDMzAUD0sV8T/x6p0ju46fmKUjXR5 dxLEfx2grbTuMnoH9moFOzG0sSIjmRO3FdANKL8efP79xqwRpEWevjEO7CZr1xn4/v2+ +glpTGeMO3R+HjkeBfUwzRODYtoWH2W9vJdVWfqp9C65dG/HkS5BN10n0mKvPc+8yFnn 2C5pzCMdrab3N/lcdCn3g8nK4Xf9PGNN4Hm8gW0De0gVlOkSlt5JmmfhCkrmQ6fEuQFX JEYA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=fxAFjgl71dKtPW1VP7nvgdxX96LoRPCBaqis2IoV/TE=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=fmlv2ehMVa5riP9BA7qiq3cr1pHHb5T57WUkeCeTozdLwwxrwAi5ofuLW4K7fdKGgL x5sLHH1FxMmGD39WZaa8NMQSt45T4TuJvMPm41EuXc7yY5Qv+0AXO393KjMyPSPqfO/S 2H5u+C1o1eG9usZncXjm0ntnu5YSjCsuDupsi3anpHbW0Q/UH+/36fW8rk836MCrgOOp HK5O2Zs7E4al8ye6W5HC3saI6Qb9gcaKh4VvnxcYGot8EgS/VbHED6r89uP5VjnRpjqt vQ8gVPQzsak01v6Gsql3jkavxSoO2l73IkTRUa+k/yVa9zh/FZ1ylt/XgDirZlCgo1Gg dL0g== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=V4B6YJGP; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id l5-20020a170902f68500b001c1fe58a1e8si311954plg.576.2023.10.18.10.19.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 10:19:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=V4B6YJGP; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 431468085992; Wed, 18 Oct 2023 10:19:32 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235197AbjJRRRu (ORCPT + 24 others); Wed, 18 Oct 2023 13:17:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40316 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235013AbjJRRRO (ORCPT ); Wed, 18 Oct 2023 13:17:14 -0400 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2083.outbound.protection.outlook.com [40.107.92.83]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A64011709; Wed, 18 Oct 2023 10:16:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=go8LIE97kvUwdpbCQzE2tu2OWThNOG82qgbuK2LINxN6WxVypXIjOgg5pCTvS+0goRL+FdWfIqShVAjf7URtZq+CAiW5whbQFPBepOQ11Y0ISsiix/TtbzPbgE9D4oI5OcCmwsF9xIP0fmyug/d7bzldKymVWq9QtwEmn5+oKvR6NLmZW4DRsNTbfPRXaKDBa/ivlWv0TKLG7WWJj5iDHuFQ8sxXo5k1GIRjLvCskDMy+jphB6mSm49edYVkN+Q3c4uqatyqEB7FktGyWrqH3LCjuZmIQ5VadXCjNjqZm8UvCC3DRa9cIIwNnZe8DqHxd8l8SgiFedU12IcvLnjIaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fxAFjgl71dKtPW1VP7nvgdxX96LoRPCBaqis2IoV/TE=; b=T8ZLkHdQHxVxUt8sAY28oSlLqPotiti0woS8nj4k3Kcrh3W/amWDW2Ult6bFlLmZyRIr34f9FoJD9siBf4wwKOIU4ayET08KAm9Zgg4uvqNZuC3Ggc31UkRy7BW09nHtGxHnoVA9ZQ+XmmcAtu3mX1guivJDrS2SJVsIJtG76H+Alpn7ZJHYjSW6K9BwLZ43Aj7IpSweZQ9zKGmnSEo11rk5JroGTcu+vdTbux73uXgRknXoQmt+Bx1MNCOzqBPqV0DWLgj6ov7ZwwkILDzN/QPcj0lspMwJSkuHIzY0A9uT3CLNSoR+V/nMdpYYtoF+WKS/ZleWOMCg6PlfOglDrg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fxAFjgl71dKtPW1VP7nvgdxX96LoRPCBaqis2IoV/TE=; b=V4B6YJGPSSpxc63GyVPcGP643CcmZbcsaigDp7PSlbPFi3ImjiYG8DHQ9Bprc8R2VlbKDAUbNBxDfR/RZ8Q1HfKP002sHKfPP5eMkkYNXIDgsYGOyNb1eNxbddRXzU2bo6uTYtfQ2y9otABLJ7dFMEHYJ3gccPSMneNmDuODzErMuSkK2dg6y+y1FaLXMP2/ssGeSM3v1lQybH6/f0FpDU1mUEhTeHM3dDuNjiihZj5alLV7eCQAKMOLNgBv1TBXwqrJjN3g4SPM0DXn5fwfG72XFO3cpuQ+uV5x0QwIcgPNPc4MiwifHuO3W0Jvdf8B0wx/frZsfVhm8gpvAxtO8A== Received: from CH0P220CA0001.NAMP220.PROD.OUTLOOK.COM (2603:10b6:610:ef::19) by DS0PR12MB7969.namprd12.prod.outlook.com (2603:10b6:8:146::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21; Wed, 18 Oct 2023 17:16:30 +0000 Received: from DS2PEPF00003446.namprd04.prod.outlook.com (2603:10b6:610:ef:cafe::f0) by CH0P220CA0001.outlook.office365.com (2603:10b6:610:ef::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21 via Frontend Transport; Wed, 18 Oct 2023 17:16:30 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by DS2PEPF00003446.mail.protection.outlook.com (10.167.17.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.20 via Frontend Transport; Wed, 18 Oct 2023 17:16:30 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:16:16 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:16:15 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Wed, 18 Oct 2023 10:16:12 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 14/16] vdpa/mlx5: Enable hw support for vq descriptor mapping Date: Wed, 18 Oct 2023 20:14:53 +0300 Message-ID: <20231018171456.1624030-16-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231018171456.1624030-2-dtatulea@nvidia.com> References: <20231018171456.1624030-2-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PEPF00003446:EE_|DS0PR12MB7969:EE_ X-MS-Office365-Filtering-Correlation-Id: e15a2f69-5892-41f1-b374-08dbcffdf897 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: wZb4Ys7Mevhp9YBrGMgNOq/74H36lZyg0Cos9NozfBK3IEbXZa56x6fwd6bIkaDu1wK/MQy84qVUr7TTo+0U2dfAhgegeAgedHez0nhtmQYOcz92I+114CP5nTLA1Glh/AApAOwpW6wJCB6p2KkG6xtZLsFGYVjVWtdNZdr8phuRUqB/ZksybwvDqpLqD46Z+Od+aBaipPdXxJFRfFwpu5TsBZTrS56THtUr17CPaPQvPEDzesOaTU5zqdKw0cY3j8YjLRck5qLGySjcqyAZo29Z2HD+zN2eR2jxwf0DAgdB0rWFPhRKb7cSk9pP9KiO+yZq/QsoiaqVlvn/XUxopycAUsIi0JdAEsDOQHYDJSxifdj35/seggZkVA9RN56jHZYSAd5nXa16CPhgDKeplMvAScNcLiftgCzr3OjweXg8asWOESO51jdCtcQ1bP/hOFk+BxnaGfwyM+hlBihmnMhU9OXLlnWeb4Gqh7WT0qUW9UvmMFD98ArCyRzUrsoHAlydxvG8lMwHD+0pjYf2d98j4tidDxJK0zErSo4PDMZDnstOVM/k0508ikQ5qQMriBJaPzwRX+d4xf7VP3awuJlJ9grniD9orFdvQjo5G6OQxPZ60T9aF0DlGXqsfkt5zcGoEvhBHUdHZDFFMpYragjmCUR2d/vtfljtxC/vVqGA1wD2MZtzu2eZxNIR/5iGmolIOaFqFqNA8sWNeezme7xjo9i8T0dvISItNKRbgPpdA1OMfRTPs834Tqak9kW1 X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(376002)(136003)(346002)(39860400002)(396003)(230922051799003)(82310400011)(1800799009)(186009)(64100799003)(451199024)(40470700004)(46966006)(36840700001)(86362001)(36756003)(40460700003)(8936002)(40480700001)(2616005)(54906003)(70206006)(82740400003)(7636003)(26005)(316002)(356005)(110136005)(83380400001)(336012)(426003)(36860700001)(47076005)(66574015)(1076003)(6666004)(2906002)(41300700001)(70586007)(4326008)(478600001)(8676002)(5660300002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 17:16:30.5861 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e15a2f69-5892-41f1-b374-08dbcffdf897 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS2PEPF00003446.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB7969 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Wed, 18 Oct 2023 10:19:32 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780114601720210008 X-GMAIL-MSGID: 1780114601720210008 Vq descriptor mappings are supported in hardware by filling in an additional mkey which contains the descriptor mappings to the hw vq. A previous patch in this series added support for hw mkey (mr) creation for ASID 1. This patch fills in both the vq data and vq descriptor mkeys based on group ASID mapping. The feature is signaled to the vdpa core through the presence of the .get_vq_desc_group op. Acked-by: Jason Wang Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 24 +++++++++++++++++++++++- include/linux/mlx5/mlx5_ifc_vdpa.h | 7 ++++++- 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index ea76c0b4b78e..87dd0ba76899 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -823,6 +823,7 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque u32 out[MLX5_ST_SZ_DW(create_virtio_net_q_out)] = {}; struct mlx5_vdpa_dev *mvdev = &ndev->mvdev; struct mlx5_vdpa_mr *vq_mr; + struct mlx5_vdpa_mr *vq_desc_mr; void *obj_context; u16 mlx_features; void *cmd_hdr; @@ -878,6 +879,11 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque vq_mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]]; if (vq_mr) MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, vq_mr->mkey); + + vq_desc_mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP]]; + if (vq_desc_mr && MLX5_CAP_DEV_VDPA_EMULATION(mvdev->mdev, desc_group_mkey_supported)) + MLX5_SET(virtio_q, vq_ctx, desc_group_mkey, vq_desc_mr->mkey); + MLX5_SET(virtio_q, vq_ctx, umem_1_id, mvq->umem1.id); MLX5_SET(virtio_q, vq_ctx, umem_1_size, mvq->umem1.size); MLX5_SET(virtio_q, vq_ctx, umem_2_id, mvq->umem2.id); @@ -2265,6 +2271,16 @@ static u32 mlx5_vdpa_get_vq_group(struct vdpa_device *vdev, u16 idx) return MLX5_VDPA_DATAVQ_GROUP; } +static u32 mlx5_vdpa_get_vq_desc_group(struct vdpa_device *vdev, u16 idx) +{ + struct mlx5_vdpa_dev *mvdev = to_mvdev(vdev); + + if (is_ctrl_vq_idx(mvdev, idx)) + return MLX5_VDPA_CVQ_GROUP; + + return MLX5_VDPA_DATAVQ_DESC_GROUP; +} + static u64 mlx_to_vritio_features(u16 dev_features) { u64 result = 0; @@ -3165,6 +3181,7 @@ static const struct vdpa_config_ops mlx5_vdpa_ops = { .get_vq_irq = mlx5_get_vq_irq, .get_vq_align = mlx5_vdpa_get_vq_align, .get_vq_group = mlx5_vdpa_get_vq_group, + .get_vq_desc_group = mlx5_vdpa_get_vq_desc_group, /* Op disabled if not supported. */ .get_device_features = mlx5_vdpa_get_device_features, .set_driver_features = mlx5_vdpa_set_driver_features, .get_driver_features = mlx5_vdpa_get_driver_features, @@ -3263,6 +3280,7 @@ struct mlx5_vdpa_mgmtdev { struct vdpa_mgmt_dev mgtdev; struct mlx5_adev *madev; struct mlx5_vdpa_net *ndev; + struct vdpa_config_ops vdpa_ops; }; static int config_func_mtu(struct mlx5_core_dev *mdev, u16 mtu) @@ -3376,7 +3394,7 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name, max_vqs = 2; } - ndev = vdpa_alloc_device(struct mlx5_vdpa_net, mvdev.vdev, mdev->device, &mlx5_vdpa_ops, + ndev = vdpa_alloc_device(struct mlx5_vdpa_net, mvdev.vdev, mdev->device, &mgtdev->vdpa_ops, MLX5_VDPA_NUMVQ_GROUPS, MLX5_VDPA_NUM_AS, name, false); if (IS_ERR(ndev)) return PTR_ERR(ndev); @@ -3551,6 +3569,10 @@ static int mlx5v_probe(struct auxiliary_device *adev, MLX5_CAP_DEV_VDPA_EMULATION(mdev, max_num_virtio_queues) + 1; mgtdev->mgtdev.supported_features = get_supported_features(mdev); mgtdev->madev = madev; + mgtdev->vdpa_ops = mlx5_vdpa_ops; + + if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, desc_group_mkey_supported)) + mgtdev->vdpa_ops.get_vq_desc_group = NULL; err = vdpa_mgmtdev_register(&mgtdev->mgtdev); if (err) diff --git a/include/linux/mlx5/mlx5_ifc_vdpa.h b/include/linux/mlx5/mlx5_ifc_vdpa.h index 9becdc3fa503..b86d51a855f6 100644 --- a/include/linux/mlx5/mlx5_ifc_vdpa.h +++ b/include/linux/mlx5/mlx5_ifc_vdpa.h @@ -74,7 +74,11 @@ struct mlx5_ifc_virtio_q_bits { u8 reserved_at_320[0x8]; u8 pd[0x18]; - u8 reserved_at_340[0xc0]; + u8 reserved_at_340[0x20]; + + u8 desc_group_mkey[0x20]; + + u8 reserved_at_380[0x80]; }; struct mlx5_ifc_virtio_net_q_object_bits { @@ -141,6 +145,7 @@ enum { MLX5_VIRTQ_MODIFY_MASK_STATE = (u64)1 << 0, MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_PARAMS = (u64)1 << 3, MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_DUMP_ENABLE = (u64)1 << 4, + MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY = (u64)1 << 14, }; enum { From patchwork Wed Oct 18 17:14:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 155020 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4946369vqb; Wed, 18 Oct 2023 10:18:23 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEFKjpPwHyNlkbfa8W1UAyxVGn7X9+rkA3E94usesaZptMnZUy3iVvdr2WGlHLpNP2T+P9h X-Received: by 2002:a05:6a20:42a2:b0:175:59b5:5a61 with SMTP id o34-20020a056a2042a200b0017559b55a61mr6104755pzj.60.1697649503376; Wed, 18 Oct 2023 10:18:23 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697649503; cv=pass; d=google.com; s=arc-20160816; b=HHLGXrnHqvLlzem8fMIhnA4LLcPe3Qph47rw8z8Q3i2cM8QnZrGXOA2Xi0ErTIFV48 3YruA89ao61YXLJtBS325HYNr4b31M2ZcLtGW79gzsgKGHZ4BM+ftUJexpScVXiSZ5Tx 5cENU+pevUCThexTuZpuq9Y2YoFomxRyFC+Jt05sSbkxYZBFOUpG+OgMnAMC3TC54gfH 7RU3fwCXHFM/u+91LEptSRXGqHE27XjPPfTpiui0Mwp+5OweOIeHegBjd5sw5PA395AG Fc5XQaCkXkIx/QVSHWdv7HlrJ+UMn8SSXBgry99yeRebpL1uYzsQqGi3lDah9KIu3pZp 3AbA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=lN0j2IV51vKRPVldubdAaR1gNfbMIxY1heIdwiqqZ4o=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=btssBgCiyJpzZN/nS+aqfPqigvBmQWY+juX0jY4WMM+hZRykDQOlEkpDeWB3q1CLvL 92m87epDS3Zj8Nwkh8CMSMHl2DzDvj4eHEcB/+w2a1BVIL9x0HhVsCR0Eu9g7vV1Y0Dv LBnO3FEM0oqlxxhtoYIzoIGMHAI/ACCwREWn5/3moog5mp+CccA1mParKPbP8kSzXSNt l8ZAaVu1Zf0iRoLoZesMHpl6nfo0WGrM+7g6jT+WfSnoRT0uXUnelAwI4DNsF4LisoZB fA1ymdgBoMno/cnt0Pvs7ztv23WmE2LJi/ApIc2gVJH4mDjQkTid95kWzu+jGK8yr0qo dwsA== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=YkPBkZFt; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id w62-20020a636241000000b0056513361b4fsi2659393pgb.741.2023.10.18.10.18.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 10:18:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=YkPBkZFt; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 32728822918F; Wed, 18 Oct 2023 10:18:20 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231772AbjJRRRy (ORCPT + 24 others); Wed, 18 Oct 2023 13:17:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40404 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235102AbjJRRRY (ORCPT ); Wed, 18 Oct 2023 13:17:24 -0400 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2059.outbound.protection.outlook.com [40.107.237.59]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8042C1712; Wed, 18 Oct 2023 10:16:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TriHVTKDVLVqL88fTAvEOlBpYHyt3QM49N/I5eKsADbOxRFPXXICUwACO4CQghKO89F0T9bipMWDJEQqWhgxRMDhCOy3yGdl6VXxn0XOiaHkNUqfFYWDKqW/2yFYjPxIWgfTJcLN4xoFwwEwHy1CngBDr8yc9WddEKVbvz/IhjVSzzhV0YcJqSpfwbC8yxYW34t5OkGtBGfymT7vs9hNFKMQ8pUtD3GlRjHZNloBHAI9vYSginQ+51thp9l5KiCmanoaQNIHXcGSADfBYKue2t8iwzl/iO8hhPIWMmANzJZy0+mw8AGjAg09cUr15RF7Zw2wLAB/Z85NM8ziw9Rc9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=lN0j2IV51vKRPVldubdAaR1gNfbMIxY1heIdwiqqZ4o=; b=jFVngTogCTSvDo16HcOeQ3yzakNZtS2qeX3drtKdrek1YnlKDAM0Dp2k3Jd8ZzZC2j0iEuWs9oLwrVa8OJ9RZ4HDrfxp6uWwCOBA/DEQi1QIMr28bq+QVNRTf2X3TRHi1Qd5oMSilMtWWmXEBA98wDHmsVd9V5JtAbhzRqUNi8vS9VV220Olyo89vsH2xZh6c9YI6ciOGWMUOES985B2gMwSQeXAATa9X7/OlmnFGEwrwIa+ifJfHv492wbLLv4ZhT0D6KGelEN4EK/R3+cSjtVaPbCkHIlEhoE2RVnLZ+Ctb0VgLFOeDTaFhncSq+ba+FE5obzF7bSEmFiBP+689Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lN0j2IV51vKRPVldubdAaR1gNfbMIxY1heIdwiqqZ4o=; b=YkPBkZFthU2uBa1dKNAb0MUyGrxYyGVoY96v90Xiqz6wOG0jypmqjLbeKQlHPQDApuIDaZ2gnNgWc2g1pFkaDYJ7nqtgOuupjzZ9oLKnfWuiE9d5JKB29IG7v/PAZ16bY4SGBcj38SCGf1Lb9yfBcSsrtU9jo908x5wkVhM+3wFrFNs7zBUcfillycC28CokfEwRTxOvmgDdPLrresshcO4B2TUuc1Mj352Fwz1pXxknn6UhhMJBFJ7NxwV5JukQWFit/HqjSOqrVLcWne5Vnw7qKxQFLsjJXI5R283Edtbav2O3tTQAhPehAjLWqNduG5KIfvOFkiZ+DPVMqkbgaQ== Received: from CH0P220CA0017.NAMP220.PROD.OUTLOOK.COM (2603:10b6:610:ef::34) by CH3PR12MB9194.namprd12.prod.outlook.com (2603:10b6:610:19f::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6863.44; Wed, 18 Oct 2023 17:16:33 +0000 Received: from DS2PEPF00003446.namprd04.prod.outlook.com (2603:10b6:610:ef::4) by CH0P220CA0017.outlook.office365.com (2603:10b6:610:ef::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6886.37 via Frontend Transport; Wed, 18 Oct 2023 17:16:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by DS2PEPF00003446.mail.protection.outlook.com (10.167.17.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.20 via Frontend Transport; Wed, 18 Oct 2023 17:16:33 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:16:19 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:16:18 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Wed, 18 Oct 2023 10:16:16 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 15/16] vdpa/mlx5: Make iotlb helper functions more generic Date: Wed, 18 Oct 2023 20:14:54 +0300 Message-ID: <20231018171456.1624030-17-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231018171456.1624030-2-dtatulea@nvidia.com> References: <20231018171456.1624030-2-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PEPF00003446:EE_|CH3PR12MB9194:EE_ X-MS-Office365-Filtering-Correlation-Id: 383acad4-aaed-419b-0348-08dbcffdfa4c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 49+J5IwC7lVsKS+Wt2OrDxPUJrjPYTLrWiBxygjFa1qJvpTu3XzpnaK/+lWcB87TlmkPmu2iKUuxS0KmoeCnr/lC/4dYRHD+7TEAt5xceg/JJcbeEX2JQEPp71eIlrGluNp5CvA3jv+z4pTu4NtM6OFHMJ//Tm5W3dnwWxeGoWkbIFSNj2QcRsR+AEOPOlXyKef42FS96y8r8OYMe2HPJk2LF9yW8caHe1taY/V0vj2NYhJiV54Gcf03ZLXnuIiDe2061y7hhV5mbUaFlS9+CqnjN9+fklPvicm20sCZLzYdeqUVlPxm0mJ1w3HQcdTPhd0lVJahP5OVle9Ht7ZXvF6eZoSppM8nUUGEvZhEViD4xwTnwZRm5ilKgVpahGTCEhAuNE+ulNaEslmgPA8gNz3S4AX4I365lmwksvpOdMZeSoH+x238f9CE4Ya1Mcpmn+hDWkyd93z2WxoEaQgnEsYzrjl5cj17oBj2aWuJPMQJUBt0o3LMpWI6CgY5xAlKvHcU9BhgFTJeCj7MOrjxEfyqpmN3BVxnC91kN6eLZH89rjn+8YgWODk7Y7AxP7g7lKkR3NrQGXH/vbO6eYdT2VNy/b46zXWTeUxYNm3DhH1ZWFunD7kTcWUpppu05NQ613Hx1ZZCvQ9S3QAz+YSIaiy7VityD6mUMEh/NsTHqSu1K0/kdelkOVRd1xUpZwUKnYn0/biTri7zYTbLIayQzZKHjdLn3M8e1d4NJLVXNDm56Ci3oQO/OipHZF+en/OnI12/hOxk1izHtOExR4zobg== X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(376002)(136003)(39860400002)(346002)(396003)(230922051799003)(1800799009)(186009)(451199024)(64100799003)(82310400011)(36840700001)(46966006)(40470700004)(110136005)(36756003)(6666004)(26005)(1076003)(336012)(66574015)(47076005)(478600001)(2616005)(36860700001)(54906003)(83380400001)(316002)(70206006)(70586007)(426003)(7636003)(4326008)(8676002)(356005)(8936002)(41300700001)(5660300002)(82740400003)(2906002)(86362001)(40480700001)(40460700003)(41533002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 17:16:33.4455 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 383acad4-aaed-419b-0348-08dbcffdfa4c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS2PEPF00003446.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB9194 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Wed, 18 Oct 2023 10:18:20 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780114525601900786 X-GMAIL-MSGID: 1780114525601900786 They will be used in a follow-up patch. For dup_iotlb, avoid the src == dst case. This is an error. Acked-by: Jason Wang Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/core/mr.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 3dee6d9bed6b..4a3df865df40 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -454,20 +454,23 @@ static void destroy_dma_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) mlx5_vdpa_destroy_mkey(mvdev, mr->mkey); } -static int dup_iotlb(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *src) +static int dup_iotlb(struct vhost_iotlb *dst, struct vhost_iotlb *src) { struct vhost_iotlb_map *map; u64 start = 0, last = ULLONG_MAX; int err; + if (dst == src) + return -EINVAL; + if (!src) { - err = vhost_iotlb_add_range(mvdev->cvq.iotlb, start, last, start, VHOST_ACCESS_RW); + err = vhost_iotlb_add_range(dst, start, last, start, VHOST_ACCESS_RW); return err; } for (map = vhost_iotlb_itree_first(src, start, last); map; map = vhost_iotlb_itree_next(map, start, last)) { - err = vhost_iotlb_add_range(mvdev->cvq.iotlb, map->start, map->last, + err = vhost_iotlb_add_range(dst, map->start, map->last, map->addr, map->perm); if (err) return err; @@ -475,9 +478,9 @@ static int dup_iotlb(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *src) return 0; } -static void prune_iotlb(struct mlx5_vdpa_dev *mvdev) +static void prune_iotlb(struct vhost_iotlb *iotlb) { - vhost_iotlb_del_range(mvdev->cvq.iotlb, 0, ULLONG_MAX); + vhost_iotlb_del_range(iotlb, 0, ULLONG_MAX); } static void destroy_user_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) @@ -544,7 +547,7 @@ void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) for (int i = 0; i < MLX5_VDPA_NUM_AS; i++) mlx5_vdpa_destroy_mr(mvdev, mvdev->mr[i]); - prune_iotlb(mvdev); + prune_iotlb(mvdev->cvq.iotlb); } static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, @@ -596,8 +599,8 @@ int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, spin_lock(&mvdev->cvq.iommu_lock); - prune_iotlb(mvdev); - err = dup_iotlb(mvdev, iotlb); + prune_iotlb(mvdev->cvq.iotlb); + err = dup_iotlb(mvdev->cvq.iotlb, iotlb); spin_unlock(&mvdev->cvq.iommu_lock); From patchwork Wed Oct 18 17:14:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 155021 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4946448vqb; Wed, 18 Oct 2023 10:18:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFdMX+kwcuA/FK2FgU3NIWFnZtZYdho+p6l2bW/Ny4DEEa8H4AGqDA2YWRJwXjI18lMrbui X-Received: by 2002:a17:90a:317:b0:27d:635:2fa1 with SMTP id 23-20020a17090a031700b0027d06352fa1mr5484981pje.0.1697649512459; Wed, 18 Oct 2023 10:18:32 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1697649512; cv=pass; d=google.com; s=arc-20160816; b=lUeF990xrw74TLwtPuqsPl1gi7lDzHMOYoQtWDr6bIVl1egcQ83x1oVUrwD7qyx+6l MY6+Pf2NngojaIUkgsf5/gmTufyMBcPdL1h+Y4OaWgp45vpkFoXXay2A3CzhUThuJEDO gkdgBiCiYd7sC4L1MS/Mg0vXu0ldNBlhV3haL+4roI9Nqpht59m3F1ozLcn9oMykmbHW Xg1kELgQT1xh/dCoBbihvk4+D1Lc00qJkbmkdFbKYRn2dq2OeYdltYR0cowPn38bBPdL 3m9NGhrCd4ORZJPh+LTnJxDE1XdgAZL7ZH1Bqi3ud+uoz9cWaugUGV1dBPxMaxWdJ4IV snRg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1BE04TjWBIEmnEIfT4KflUMZCKQcmJpsqNZEqnvGfsQ=; fh=AwOKC3RXAMdCAfoni6jcDuW/UqHO6HngpsLpt3S/AkI=; b=uThf2rMiHKgcC5xT6AR8FYpefi7aNc0mGA4PWNfImCdpvzLXpnFvn/RrCeSdoL5kWK RdzMZzZ6lto1YqrJEYx1H2YXmiKttwfVmA01s0ShNN84xmPiWK+MkYJkJq1sDC42O9W6 9d4J1TvpMILloInZPoGtCJjIjzfJ5BeiC5PDwBYEe4Gw0AfElcUPTZB2cpPGQZrHVPsS hnJMKepwPsv+QBaABr1V4u8zOOBKruCpfKzkxIonKqT1fxONujDUW85+FPGl44D1ntsz RpHB4kiYnv8rMmY0Q2Ad1TSR6gvWT//Y8wwUyodtzpZHXkYoQsuoG7eMTRh3lZn8Be0W grWw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=EjtSpysc; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id bb20-20020a17090b009400b0027367e0c931si279533pjb.129.2023.10.18.10.18.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 10:18:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=EjtSpysc; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 53618822741C; Wed, 18 Oct 2023 10:18:30 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235403AbjJRRSH (ORCPT + 24 others); Wed, 18 Oct 2023 13:18:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40314 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232373AbjJRRRe (ORCPT ); Wed, 18 Oct 2023 13:17:34 -0400 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2063.outbound.protection.outlook.com [40.107.220.63]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D63B1171D; Wed, 18 Oct 2023 10:16:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EThE2dydt6oXrjOLoPyYsQlGJtgXpXD98CJVlB+Btnzu+Lx+FBopXGexN3NtkFy6efxrdQcb9NfGHpeaE4GFPV3zl0N40XCCYu86f97lupi/sT3I2Jpy4SnuqYc2EIWuQYPM8qbrTeDz34Y9fhrlPFgAblMhPf3sCejJgKyrzA71i2jub0k48jJngZR604l99QWcCGH/tkq9hvG9ZfJXN4LAiWuBQYi004aZQQwZqpm4x9h4GsQoa1Ixp55qR2mYhVgKqNA+TNNtnMworuf5YshC9KPBFsReU0sUSbKiX0VOUmRrs5nMwCLRuYRi5yBj+24HnGm4XSEqaJCe+jZJvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1BE04TjWBIEmnEIfT4KflUMZCKQcmJpsqNZEqnvGfsQ=; b=nUWuZ07mSd8B/UxQckhyAuilNBWhbx/36gAm+v4rUSlkKgIG5Xv0jJ0xVyE/p3gWeSC9za1XLsxSoHxb2r/k2tdca7kOyCH0XQAEMK/AUMhC8CyGInbXvhuvOpxTeIqvR3Npngpl6sIqBJQMJ60984IeJN3rRGbXPTE4qR8dOWtHefgIPcrxi8JB1DF8cU/nAd42Dv+jCTAp1BEImB+6HvuyPj9v636FTXpZDnjovRB5OXUt0m5COOCp7LZ85qHfztjqCL1+lInQQ9fKboSR7itoKt3yq1S5y1LNeqSpGP4uOoazzBH63E/piwbmiwBB29vEHhqDN/GzEBCeFOpRHA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1BE04TjWBIEmnEIfT4KflUMZCKQcmJpsqNZEqnvGfsQ=; b=EjtSpyscQSl7+4idLTwA/h2aQE3rx7z/BERL5lNmZr0e1ZiU/ymURg+JHLFCQE3xFeFws85ncQ0SsdJ4TlSlGKnX25Zxo9gA3sJ/tUyUhYHwIPQZbg5Jopgi5nE1ImYmFSNF2iV97Du80pgBjovN3XS1ekYcppW6Ly6J+lWQHzfzZuCC2dVLu1NyeQu5obiSCoFfG9seVoHlNqjNDqQ4467bG/mFJsnICXzqkSHiXklSjjGRmkoS3S9ud1YzlLaSR4ddFDcdzzMDpF0kIqxc1CcNOBTPvlAkWCSAJFWn+Ft9u45EuTqHhNZss8Z2pAK9h8DWbqWobkYilSyA6FqHUw== Received: from CH2PR17CA0026.namprd17.prod.outlook.com (2603:10b6:610:53::36) by CYYPR12MB8702.namprd12.prod.outlook.com (2603:10b6:930:c8::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.23; Wed, 18 Oct 2023 17:16:36 +0000 Received: from DS2PEPF00003445.namprd04.prod.outlook.com (2603:10b6:610:53:cafe::28) by CH2PR17CA0026.outlook.office365.com (2603:10b6:610:53::36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.21 via Frontend Transport; Wed, 18 Oct 2023 17:16:36 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by DS2PEPF00003445.mail.protection.outlook.com (10.167.17.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.20 via Frontend Transport; Wed, 18 Oct 2023 17:16:36 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:16:22 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Wed, 18 Oct 2023 10:16:22 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Wed, 18 Oct 2023 10:16:19 -0700 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v4 16/16] vdpa/mlx5: Update cvq iotlb mapping on ASID change Date: Wed, 18 Oct 2023 20:14:55 +0300 Message-ID: <20231018171456.1624030-18-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231018171456.1624030-2-dtatulea@nvidia.com> References: <20231018171456.1624030-2-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PEPF00003445:EE_|CYYPR12MB8702:EE_ X-MS-Office365-Filtering-Correlation-Id: fb129a15-674b-4721-65e9-08dbcffdfbfa X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 0+D4Zt2S2VMRqoLgjxAYWJDcziees+cNHNFOzz0jKImWmi/rdJL0W2s6NkiMSQ7Us0iw9J7nTbCpPRwcjA3oWp1eGn1GGUNCg36fASqMuWbjVjkBifviA0szgtSh/j4ALB1WtH43sYrCg3BcqRUZ/Bin1ZxFjyRH80ghqtwQh8RFZJWv0Ii2L5g7eZ9d1GOebh9zPbJZB00/vnME/DSqCtxrFR1wlLEh7WvkKoDwiopQ0e7EuJxqB2uClwsYlmEBSWBGd+VgMD+M0kWW9V1DDmt1tjHKQUQoa/uYd+wAvCM5xXBxlJrWeQNUa8ihWBZLzoeeutq9n1F5e56ehVeG8it+huFPBJs9j7zFqlggRzh9Uomewr9QoZ5nXK3SHStyzPYXtYSpzkRWRi7RQ399foXz5J7Y/oimj3jCrbq4kaRZelO1IdYrHqY+vFfMCpIJlviIUu4tzckFrxmcYLQC3qOB/hYG7KnTD8k0bUTatfRVBbKVPtQ9PNZhZQ619V4F7c8leiJJnE+VjaopCmLF7lqHKEN+xwgDwcqEOmcGhMEhfofbNyosRjbe/oWomLBNgzKdPca1PZIVo6STtI1k8FuZ4IvxubMZssyl4VVLApxuGPDq1IytKmb5gHpO+N8uyjilpVpvHa4HWYpzyBkk+0AWpN2PqsjJhBhpyWCR14LREFNb1zlG2cz3mDrWKWiZ9Oh9xDl1nYHv5XUOqcaUW4HncEUK1bSFKKgnljVUPK/oU3AlNi5yaqo7COiPf2/E X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(346002)(136003)(39860400002)(376002)(396003)(230922051799003)(186009)(1800799009)(451199024)(82310400011)(64100799003)(46966006)(40470700004)(36840700001)(36860700001)(66574015)(426003)(336012)(26005)(82740400003)(7636003)(356005)(47076005)(40460700003)(83380400001)(41300700001)(54906003)(8936002)(110136005)(70586007)(86362001)(8676002)(4326008)(5660300002)(1076003)(70206006)(316002)(2906002)(478600001)(6666004)(40480700001)(2616005)(36756003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2023 17:16:36.2817 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: fb129a15-674b-4721-65e9-08dbcffdfbfa X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS2PEPF00003445.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR12MB8702 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Wed, 18 Oct 2023 10:18:30 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780114534677166651 X-GMAIL-MSGID: 1780114534677166651 For the following sequence: - cvq group is in ASID 0 - .set_map(1, cvq_iotlb) - .set_group_asid(cvq_group, 1) ... the cvq mapping from ASID 0 will be used. This is not always correct behaviour. This patch adds support for the above mentioned flow by saving the iotlb on each .set_map and updating the cvq iotlb with it on a cvq group change. Acked-by: Jason Wang Acked-by: Eugenio Pérez Signed-off-by: Dragos Tatulea --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 2 ++ drivers/vdpa/mlx5/core/mr.c | 26 ++++++++++++++++++++++++++ drivers/vdpa/mlx5/net/mlx5_vnet.c | 9 ++++++++- 3 files changed, 36 insertions(+), 1 deletion(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index ae09296f4270..db988ced5a5d 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -32,6 +32,8 @@ struct mlx5_vdpa_mr { unsigned long num_directs; unsigned long num_klms; + struct vhost_iotlb *iotlb; + bool user_mr; }; diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 4a3df865df40..66530e28f327 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -502,6 +502,8 @@ static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_ destroy_user_mr(mvdev, mr); else destroy_dma_mr(mvdev, mr); + + vhost_iotlb_free(mr->iotlb); } void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, @@ -561,6 +563,30 @@ static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, else err = create_dma_mr(mvdev, mr); + if (err) + return err; + + mr->iotlb = vhost_iotlb_alloc(0, 0); + if (!mr->iotlb) { + err = -ENOMEM; + goto err_mr; + } + + err = dup_iotlb(mr->iotlb, iotlb); + if (err) + goto err_iotlb; + + return 0; + +err_iotlb: + vhost_iotlb_free(mr->iotlb); + +err_mr: + if (iotlb) + destroy_user_mr(mvdev, mr); + else + destroy_dma_mr(mvdev, mr); + return err; } diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 87dd0ba76899..5774f4adb7c4 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -3159,12 +3159,19 @@ static int mlx5_set_group_asid(struct vdpa_device *vdev, u32 group, unsigned int asid) { struct mlx5_vdpa_dev *mvdev = to_mvdev(vdev); + int err = 0; if (group >= MLX5_VDPA_NUMVQ_GROUPS) return -EINVAL; mvdev->group2asid[group] = asid; - return 0; + + mutex_lock(&mvdev->mr_mtx); + if (group == MLX5_VDPA_CVQ_GROUP && mvdev->mr[asid]) + err = mlx5_vdpa_update_cvq_iotlb(mvdev, mvdev->mr[asid]->iotlb, asid); + mutex_unlock(&mvdev->mr_mtx); + + return err; } static const struct vdpa_config_ops mlx5_vdpa_ops = {