From patchwork Wed Aug 2 17:12:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 130022 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f41:0:b0:3e4:2afc:c1 with SMTP id v1csp616264vqx; Wed, 2 Aug 2023 10:32:28 -0700 (PDT) X-Google-Smtp-Source: APBJJlHjtppAPDz7ZL4GWPLG7Xv8s8HJapmSTrfsjgRC/bA4PjIksWMNSH76RTGNwUlIMd1D3ZXs X-Received: by 2002:a17:902:e546:b0:1b1:9d43:ad4c with SMTP id n6-20020a170902e54600b001b19d43ad4cmr16930872plf.40.1690997548444; Wed, 02 Aug 2023 10:32:28 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1690997548; cv=pass; d=google.com; s=arc-20160816; b=TW5LSYFeUQQOvCOVyRrkHSvqxWbakrkQ2Q5roNVep8mj7KJJoaeEW7rIDpdRuqc3L5 6oYkMD3gSxlXFGxSeh81ylNg9jy9kmxWeXc0Gq+4ncDDFki64Oii+2O7qOGX1NvKenQ6 Bh9Xj8j+Wh1wffWZu6o2BYls94VDsKfqJbSACdf5b+COvafMUOQY2FhWpZRedjianuZw 5/Oz8kjSTB5BC7sbOesRBno43WY40NuuBAbsU0uzrR08co/9kR519tEyL6olVS6/EUKi 8W9RWmezU3vxB15JSxK12BIqdTfEUCEW9G13HQZ3/pVfsEKsENE0WvcmW1i4btQ5cMxK E6zg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Z3o64W6AxgCasG8EB+Q2EH9SyaH2V1AMN/IpgxvfuVs=; fh=+95HZF2HMWf/zVxErTglth2K4K2H1g2tqr9wBazu2Nw=; b=dpX4ymktsv/PvzOXnXwiQdiWxhkblpqeWjAMdT2zko7NG4OVYmLhO1gt6tXvBuYfrl kQP7kb7htRr/KZgoYa0boLlyYNQXXbZTF2aN3peKdhzGFREJym7hTdOff6QVGtcgbZDr oEGJcThYZ/DdcBkMnE7CQcQAFWk4YvlOEFflxyIH4uEv5ZKSShPpHQ5gKNvSpPGzO+bD RbaA+Nc8Rjvcsyo0CrVP1as+LaulUvz9hoq4RG5c77/n3v4wMOjJQiKpdru23a7yj3fb WHPGrJPSEmbsVvVlr+5cGZi07cEO8soHyrHRRRZbFPTq/h6PG7K/LKimluwaDVLREYKh 0Fqw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=cvCPZyaC; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a6-20020a170902ecc600b001bbfdb016a5si7601983plh.185.2023.08.02.10.32.10; Wed, 02 Aug 2023 10:32:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=cvCPZyaC; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232381AbjHBRNZ (ORCPT + 99 others); Wed, 2 Aug 2023 13:13:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38460 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232014AbjHBRNR (ORCPT ); Wed, 2 Aug 2023 13:13:17 -0400 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2056.outbound.protection.outlook.com [40.107.96.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 803C31734 for ; Wed, 2 Aug 2023 10:13:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IavSKqK64YjUSvBo4hps72Hs1aAbpsJDk2lYlUu5hk1AndYdMQ6CRs9yDTY0KHELrKo1t+dSnJVL+ggH9t7D6Nsbp/J9xWhwWSu+ri0QrnOpy5JaqcqF7uJLqCiAi4JcRXkl+zlj1AwcZd2zMHPJIH9W2Q646LmNe7BNwQkNfe7KsHbm0XHuJ2OsnYVVc6RgaKqOqHa9NQBgrm5llg+0oJrApwNxmbfmqOOcSJqSy30CfoGl0nxwF/PH32Uyh6HgOlbM2OrIWLeqkSUEe2dTzSCaIcyDPC5or5Vi89iDAZaeFBsS4jVi8PwTr6beR1fK7IJ4exlosyxxJoxkkWVd7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Z3o64W6AxgCasG8EB+Q2EH9SyaH2V1AMN/IpgxvfuVs=; b=Hb6Dtycl6qkai1ebLP7oi+5Y8Ry0K9NPaqUEujnDgeF9ya7eXREAukAcrmYHIoSKWYtsYFc9DSbuiV4I2e5jDSrMQ4DOTea5B3I9Rg85Z1XCIj3zagLRHNr+khUJmnabkrqO466IQj20ruwulfo45WGHxV1jzbXTvUhTX2bMuinAPVxfiFFUcosB6+36jWp4rceilYwTZeListn+b88/yiZ6poJetYbEwUDDalS6wvQwVmHe8zWUqvIN7RkpXEW+eEVnD3+fJUrl9quYQxAAWdlVvoDlSuXORnPxXnZIOMbfKkBxL1+rF5vqctGLCxgPKo5vS14WE8SwmUUeWBy6mQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Z3o64W6AxgCasG8EB+Q2EH9SyaH2V1AMN/IpgxvfuVs=; b=cvCPZyaCfaccG4etB2GxvLGzW7+Lk1cgP0eHYtKczFv1vvUb3nmbTvyu7uzU13SfZtOANUMlMnakdDizy2XCgT1/Qdv8POug8z2l2bth8VaG04T85axBquZ2lIMtG2Vv7Ddy1biV8XC0X0CmCWLJ31402wCXzHE8VyFe+LtnbVNn7cIzVdFtLMg1V/VgFI7NM1L3SCCsPmqSM2BdFgieO7WAyQP/QqEZDvZs6pYktlJekU1keoyMtce089clFVEYm5Q3y+3CSf7GHZKpl1lQ17Q4lqzUwoC7JOg1TaKh9tAZ55KZ6rIvcDC846/K63IiyvNVsokEj/6e7pK8zFdsgg== Received: from MW4PR03CA0096.namprd03.prod.outlook.com (2603:10b6:303:b7::11) by IA0PR12MB7555.namprd12.prod.outlook.com (2603:10b6:208:43d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6631.45; Wed, 2 Aug 2023 17:13:08 +0000 Received: from MWH0EPF000971E6.namprd02.prod.outlook.com (2603:10b6:303:b7:cafe::72) by MW4PR03CA0096.outlook.office365.com (2603:10b6:303:b7::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6652.19 via Frontend Transport; Wed, 2 Aug 2023 17:13:07 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MWH0EPF000971E6.mail.protection.outlook.com (10.167.243.74) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6652.20 via Frontend Transport; Wed, 2 Aug 2023 17:13:07 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 2 Aug 2023 10:12:53 -0700 Received: from rnnvmail203.nvidia.com (10.129.68.9) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 2 Aug 2023 10:12:53 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.9) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Wed, 2 Aug 2023 10:12:50 -0700 From: Dragos Tatulea To: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo CC: Dragos Tatulea , =?utf-8?q?Eugenio_P=C3=A9rez?= , Gal Pressman , , Subject: [PATCH 1/2] vdpa/mlx5: Fix mr->initialized semantics Date: Wed, 2 Aug 2023 20:12:18 +0300 Message-ID: <20230802171231.11001-3-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230802171231.11001-1-dtatulea@nvidia.com> References: <20230802171231.11001-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MWH0EPF000971E6:EE_|IA0PR12MB7555:EE_ X-MS-Office365-Filtering-Correlation-Id: 25d91cf9-007c-40ad-25b5-08db937bbddb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3ebzthq04FOkvko+bkA4hJChFsnc701BWXCqTZ7KAlO+zHhcFSd6zjT0c7euDAf7YyyZkZbY2AErdRGkcRmhtqrB5fBaTEYocUxWobBa69ad2Z4VQIVITWnpBTg3dDznbP5/5wbXfMYruZryzcy6fSKhloN0Qakc4FcXTKBdFKDI2rzW4KYzD9nZY2dXoSzps20Ny7mbSTxTyk2awKssEecvSxdXTMAYKzL4NO78gNn4hgIs0P6zFyUkuQNklqiavplQ7LBMzmEsBbTtXt9p0Lo3HBIet5dlPUf7Nwl/XSgVfxXWjyq8EGS9U7PYS9GzSYiDCU07JQu8ztSn12aajbc4YIagvSftCVQ/Tl4Nyqc2dF6ZV9XfR3OS0O+cKci9SNAbtuYxICn9/++p2ZVazW+Nl6XGxKo4pIcd1PxE0R+u1/12BMcramAWhwHi/GdIO9hiT83lco0Kalo7m5FK88KFkcIy1eHr5SoCAlxqZH8/khWM5zWpodsH/BmltGtmphoGHd7JgKAI0Fe5kCaKwlwCiJ5HKIfnQxG5LgqG6cfqztYEAwPnKe+l82egTF3CBFY0nwx52vnARH4BXv1rxdZDGQvEGu6wZt7q4l1dSMoG6U41LxgIh8k0ADWy55MgekFxjk7LEKKRdnF04J+yEup2KNVzhaNZVOnJ6BdSUF3zkVpcCykTM/Intial52DoSBU2oKMkEAccl9qPr1v28IUL7k0Y1z1pRjRrRZs13e/EAyFbwtlPEwoG8ylPDz2P X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230028)(4636009)(396003)(346002)(136003)(376002)(39860400002)(451199021)(82310400008)(36840700001)(46966006)(40470700004)(36756003)(2616005)(66574015)(426003)(5660300002)(8936002)(186003)(36860700001)(8676002)(47076005)(83380400001)(26005)(336012)(478600001)(4326008)(70206006)(54906003)(110136005)(316002)(6666004)(41300700001)(70586007)(86362001)(40480700001)(40460700003)(1076003)(2906002)(356005)(7636003)(82740400003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2023 17:13:07.7010 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 25d91cf9-007c-40ad-25b5-08db937bbddb X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MWH0EPF000971E6.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB7555 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773139444939134780 X-GMAIL-MSGID: 1773139444939134780 The mr->initialized flag is shared between the control vq and data vq part of the mr init/uninit. But if the control vq and data vq get placed in different ASIDs, it can happen that initializing the control vq will prevent the data vq mr from being initialized. This patch consolidates the control and data vq init parts into their own init functions. The mr->initialized will now be used for the data vq only. The control vq currently doesn't need a flag. The uninitializing part is also taken care of: mlx5_vdpa_destroy_mr got split into data and control vq functions which are now also ASID aware. Fixes: 8fcd20c30704 ("vdpa/mlx5: Support different address spaces for control and data") Signed-off-by: Dragos Tatulea Reviewed-by: Eugenio Pérez Reviewed-by: Gal Pressman --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 1 + drivers/vdpa/mlx5/core/mr.c | 97 +++++++++++++++++++++--------- 2 files changed, 71 insertions(+), 27 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 25fc4120b618..a0420be5059f 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -31,6 +31,7 @@ struct mlx5_vdpa_mr { struct list_head head; unsigned long num_directs; unsigned long num_klms; + /* state of dvq mr */ bool initialized; /* serialize mkey creation and destruction */ diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 03e543229791..4ae14a248a4b 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -489,60 +489,103 @@ static void destroy_user_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr } } -void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev) +static void _mlx5_vdpa_destroy_cvq_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) +{ + if (mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] != asid) + return; + + prune_iotlb(mvdev); +} + +static void _mlx5_vdpa_destroy_dvq_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) { struct mlx5_vdpa_mr *mr = &mvdev->mr; - mutex_lock(&mr->mkey_mtx); + if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] != asid) + return; + if (!mr->initialized) - goto out; + return; - prune_iotlb(mvdev); if (mr->user_mr) destroy_user_mr(mvdev, mr); else destroy_dma_mr(mvdev, mr); mr->initialized = false; -out: +} + +static void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid) +{ + struct mlx5_vdpa_mr *mr = &mvdev->mr; + + mutex_lock(&mr->mkey_mtx); + + _mlx5_vdpa_destroy_dvq_mr(mvdev, asid); + _mlx5_vdpa_destroy_cvq_mr(mvdev, asid); + mutex_unlock(&mr->mkey_mtx); } -static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, - struct vhost_iotlb *iotlb, unsigned int asid) +void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev) +{ + mlx5_vdpa_destroy_mr_asid(mvdev, mvdev->group2asid[MLX5_VDPA_CVQ_GROUP]); + mlx5_vdpa_destroy_mr_asid(mvdev, mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]); +} + +static int _mlx5_vdpa_create_cvq_mr(struct mlx5_vdpa_dev *mvdev, + struct vhost_iotlb *iotlb, + unsigned int asid) +{ + if (mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] != asid) + return 0; + + return dup_iotlb(mvdev, iotlb); +} + +static int _mlx5_vdpa_create_dvq_mr(struct mlx5_vdpa_dev *mvdev, + struct vhost_iotlb *iotlb, + unsigned int asid) { struct mlx5_vdpa_mr *mr = &mvdev->mr; int err; - if (mr->initialized) + if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] != asid) return 0; - if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] == asid) { - if (iotlb) - err = create_user_mr(mvdev, iotlb); - else - err = create_dma_mr(mvdev, mr); + if (mr->initialized) + return 0; - if (err) - return err; - } + if (iotlb) + err = create_user_mr(mvdev, iotlb); + else + err = create_dma_mr(mvdev, mr); - if (mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] == asid) { - err = dup_iotlb(mvdev, iotlb); - if (err) - goto out_err; - } + if (err) + return err; mr->initialized = true; + + return 0; +} + +static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, + struct vhost_iotlb *iotlb, unsigned int asid) +{ + int err; + + err = _mlx5_vdpa_create_dvq_mr(mvdev, iotlb, asid); + if (err) + return err; + + err = _mlx5_vdpa_create_cvq_mr(mvdev, iotlb, asid); + if (err) + goto out_err; + return 0; out_err: - if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] == asid) { - if (iotlb) - destroy_user_mr(mvdev, mr); - else - destroy_dma_mr(mvdev, mr); - } + _mlx5_vdpa_destroy_dvq_mr(mvdev, asid); return err; } From patchwork Wed Aug 2 17:12:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 130049 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f41:0:b0:3e4:2afc:c1 with SMTP id v1csp644495vqx; Wed, 2 Aug 2023 11:22:00 -0700 (PDT) X-Google-Smtp-Source: APBJJlGz2Vxke3AJ4He6NxD9wBIcphEeGezLEBYuOaZ6c5V+Pjm5k9bqsu71rsJE/tgtTU9xNzdD X-Received: by 2002:a17:902:e745:b0:1b8:8682:62fb with SMTP id p5-20020a170902e74500b001b8868262fbmr20259442plf.4.1691000520244; Wed, 02 Aug 2023 11:22:00 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1691000520; cv=pass; d=google.com; s=arc-20160816; b=BzxKv/di+orRFsTRPA7+MhDetMSXw5nArKWwbXvB9lweZthGY+sNq8gU7rVd7bNowf s8ZKS2iJgpH4deMzWbMvmb3m0UdbvA3227E8iSl4oziix0kd3ADVuYCKeQ7ixJoxij6/ h8zWIF9K35TOzaVgjwKxDP9cQdpq1JFc40RptlZEaxVYmfFLQpuWyeY0rIBriaDhdQY8 siRdMyLUH0Yu4u1c1fw9reFnBCAoy690zsWqpRIwbpPsukYQKAS3iyNPdVhsUrc63wsO G0Wl3CkBWmA/LMV8aSa0dReJsit5lYe0z1A77J35BnQgqalxVRKXaR8cPgTQk6mHJ0vf qExw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+UyKQTMFX1mut5/H2U/CfaadEbA4WrB66IyJaD16oIQ=; fh=UE9+Ipz6zYon8/bS9+eQEZnbHFF0pAFCW+O/IF/rNOU=; b=tyUcIZeKbPFuBmxmM8QCErs34ng/DA4C8qUBi6jCmHTQ/fwmCHlk005uASbUThXNSj CX8u9sntSrTV3FqAMpydKt3AMoQpHu4ULwezacpm7sgPJGz3OUFdcOHV/FQpFvVOlcGm CXFRXB8alq7NiWl1ZlTrgMxQRJDea4b8o49ZwI3xsDOyVez90d4fxhW11BvO+u8q9xTz A+dfuUx5VZ3G8Ezx9FMy807907SPlo2unvOgQSwMHU0dISDut2rDdPH9NHdEf6LQTsiA XguH7r0PuzIyonwqmr+NrWwpGebA4McqOOk42FvYONmDve+C8D7wwvK21JgRNvLWijPV ACKQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=dcLxT6ie; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bk13-20020a056a02028d00b0056401aef835si10630936pgb.762.2023.08.02.11.21.46; Wed, 02 Aug 2023 11:22:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=dcLxT6ie; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232164AbjHBROj (ORCPT + 99 others); Wed, 2 Aug 2023 13:14:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39886 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232155AbjHBROe (ORCPT ); Wed, 2 Aug 2023 13:14:34 -0400 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on20602.outbound.protection.outlook.com [IPv6:2a01:111:f400:7e89::602]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE7AC2D70 for ; Wed, 2 Aug 2023 10:14:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dttdudDqvQi2+hQJ3NaHb3YGylDNMZojjRe2U0NPXfIvG2qkANSrM4cbMC4RD2ZrimPa4b9CJtJhfJbn5v9s4GwMlaWJFGPE9E8D6uoD9tovU4N6qiaHMKO4xF/4aZMBGV82A8wYM3aV2wEOmddmDL7ymRSUPWsRmJnbq6WhLVATEKEg1eeqhQHynKNmD75PN5Ck6LWSFHoyYqA8HNvxbhDW+SMIIUuMzeREsuhK7M+TLowFjKHX3M/LgeSSzjUMONdpNWhboacnbKeUfwCNAZggdtdF2uOMWo6D0AGxRPOcju6InimzZKX5zHtagjteWteKIVB+yFl8rlQ3uIuGZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+UyKQTMFX1mut5/H2U/CfaadEbA4WrB66IyJaD16oIQ=; b=i+vd9N1fZ1EQsezdHymW+cek/BKhzeOZRTQurxApvnAbqJfT4msC1q7TiEjai2PEvw9XRV1eLiyomRaGcW1kaIIF1BrbpW1dUtvdXRzZ8+ZqwkrpYyJkL18UOmXff72cxsm7Q6YJLU7jy17H2eJ0AMGpyVqRDc0hZTlbmN7GR0iwsDDpNLyLV34srULRodmwb6KdFfc5AFW48VL2BhSpudmXViWm+U9sXh8Hf7N1H7LEiMZJRHirIUvIsCf1gki8TzpeqP8rLQvrdI5vaXSY1nJbHGHxiO++CsITodUR0eT/kC2rUmDoTNqSPo5EqxelDAFg+j4Y02+A59/s1virTA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+UyKQTMFX1mut5/H2U/CfaadEbA4WrB66IyJaD16oIQ=; b=dcLxT6iex1FoDKFrue6uu8T040Pt884AeBGzzd2BUFR3RaCpcGmQ5sH4+usmjW0G8Y3mUOvgSB5wRMO/iY4x26aqQcddXsSYRXnj/99tt7Gm/Heim41wTa9FuLTK6ojd3i/KqSxze+qloQHUAEJxT2H/E6ZiOOsGCFI+7jJkXmdRViOR/coovt97eWSF2u/1FU0/TAoJkywccnLvjZyHMuYvVIwjfnlTkWKEd6E28UVMIPwSfQAWz42WdVNzC0QRrp6me2e9vZsKrSKSJcl+FgNwX3QrSP3oCK270bQknNq5+SPhMM4gvZqqFvXiZ00kBlsCTruHo6nqk7k7IKd2qg== Received: from MW4P220CA0005.NAMP220.PROD.OUTLOOK.COM (2603:10b6:303:115::10) by BL1PR12MB5144.namprd12.prod.outlook.com (2603:10b6:208:316::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6652.19; Wed, 2 Aug 2023 17:13:18 +0000 Received: from MWH0EPF000971E8.namprd02.prod.outlook.com (2603:10b6:303:115:cafe::b8) by MW4P220CA0005.outlook.office365.com (2603:10b6:303:115::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6652.19 via Frontend Transport; Wed, 2 Aug 2023 17:13:18 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by MWH0EPF000971E8.mail.protection.outlook.com (10.167.243.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6652.19 via Frontend Transport; Wed, 2 Aug 2023 17:13:18 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Wed, 2 Aug 2023 10:13:05 -0700 Received: from rnnvmail203.nvidia.com (10.129.68.9) by rnnvmail203.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Wed, 2 Aug 2023 10:13:04 -0700 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.9) with Microsoft SMTP Server id 15.2.986.37 via Frontend Transport; Wed, 2 Aug 2023 10:13:03 -0700 From: Dragos Tatulea To: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo CC: =?utf-8?q?Eugenio_P=C3=A9rez?= , Gal Pressman , , Subject: [PATCH 2/2] vdpa/mlx5: Delete control vq iotlb in destroy_mr only when necessary Date: Wed, 2 Aug 2023 20:12:20 +0300 Message-ID: <20230802171231.11001-5-dtatulea@nvidia.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230802171231.11001-1-dtatulea@nvidia.com> References: <20230802171231.11001-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MWH0EPF000971E8:EE_|BL1PR12MB5144:EE_ X-MS-Office365-Filtering-Correlation-Id: 0868c003-c92f-467d-cb1b-08db937bc43e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: cdONbFcQg/XDn/bSE8yzrOwELtztIb22l3ZbDOlzuDXq53xarGoMujSj/nHqWWXg/iFrAUAKZuMGNWqfC7OTBl2sw3VUwDjmFpBJf9Ao4+ajmw8J2GRoi+KA4OiSvsGskJ3VxSFuF2BYNj5YEcSmwl+DUNNdAXj5g8p8P0EGIBEJ4Gh3GSOeObrnzxNnrNkxKnndqgqpsnIaYZwzdWwOfnHEGwTDeqKs6/TrVv5Q/UISlITGwhtR6+HZAkZvXEcjqTduZXiebyZPxX+UcHY+yxPdQ6Y1XP2mydC8qfYGOiLXLPXOjnDe9rfKjqMu6+4yGYkyMrl6UxZcVJ73t0UkIOllXTk7EzByLEqWgW4ks7+zFOjtWvKDYlRK0IJzh4gyaZ7byTC5bcgOtA52oXk9BM0WUW2kSmFYZCybMxNFjdMtAlc88Qo0djSLAUTyMTSStXhKrEfJgZO7MDPoqZwOsZ/PXktTtwvYexGq0eP+IFrHKzOr8IKMtQqq3luZcHuKx3ykJ7MQ6GcCcGy1yUIb/NffXlQ8C2lFaNOKhQkAm/4bnCGcQtdJKlT1FdrL4mFCFXVoR+KiCfbbdvH91HXVIt2Fas6gLCdXYeI7dHxU2HzhrRmTsogaYU0xD6VafHyzoiGUHrE9663M9C/70yXAHiX7qLAXKpE1+o+ERXa09aNUXS2pI22HrGDOi9jE4QFbIJ/F0dPzMX68CYra/GFD+xb/gKip5YUMs7KeixYJl4Siz9sPCgMUObX8xFILmEFh X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230028)(4636009)(136003)(39860400002)(346002)(376002)(396003)(82310400008)(451199021)(46966006)(36840700001)(40470700004)(4326008)(40460700003)(426003)(66574015)(70206006)(70586007)(186003)(2616005)(356005)(7636003)(82740400003)(336012)(2906002)(26005)(1076003)(83380400001)(36860700001)(47076005)(54906003)(110136005)(40480700001)(86362001)(478600001)(36756003)(6666004)(41300700001)(8676002)(8936002)(5660300002)(316002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2023 17:13:18.4204 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0868c003-c92f-467d-cb1b-08db937bc43e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: MWH0EPF000971E8.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5144 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_PASS,SPF_NONE,T_SCC_BODY_TEXT_LINE, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773142561985510268 X-GMAIL-MSGID: 1773142561985510268 From: Eugenio Pérez mlx5_vdpa_destroy_mr can be called from .set_map with data ASID after the control virtqueue ASID iotlb has been populated. The control vq iotlb must not be cleared, since it will not be populated again. So call the ASID aware destroy function which makes sure that the right vq resource is destroyed. Fixes: 8fcd20c30704 ("vdpa/mlx5: Support different address spaces for control and data") Signed-off-by: Eugenio Pérez Reviewed-by: Gal Pressman --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 1 + drivers/vdpa/mlx5/core/mr.c | 2 +- drivers/vdpa/mlx5/net/mlx5_vnet.c | 4 ++-- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index a0420be5059f..b53420e874ac 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -122,6 +122,7 @@ int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *io int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid); void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev); +void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid); #define mlx5_vdpa_warn(__dev, format, ...) \ dev_warn((__dev)->mdev->device, "%s:%d:(pid %d) warning: " format, __func__, __LINE__, \ diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 4ae14a248a4b..5a1971fcd87b 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -515,7 +515,7 @@ static void _mlx5_vdpa_destroy_dvq_mr(struct mlx5_vdpa_dev *mvdev, unsigned int mr->initialized = false; } -static void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid) +void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int asid) { struct mlx5_vdpa_mr *mr = &mvdev->mr; diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 9138ef2fb2c8..61c10ba5e3f5 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -2636,7 +2636,7 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, goto err_mr; teardown_driver(ndev); - mlx5_vdpa_destroy_mr(mvdev); + mlx5_vdpa_destroy_mr_asid(mvdev, asid); err = mlx5_vdpa_create_mr(mvdev, iotlb, asid); if (err) goto err_mr; @@ -2652,7 +2652,7 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, return 0; err_setup: - mlx5_vdpa_destroy_mr(mvdev); + mlx5_vdpa_destroy_mr_asid(mvdev, asid); err_mr: return err; }