From patchwork Tue Dec 5 10:46:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 173944 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp3342188vqy; Tue, 5 Dec 2023 02:46:52 -0800 (PST) X-Google-Smtp-Source: AGHT+IGah5+wYcfSBHGPShn5NsDKGlddURsKqXkQIc1R4v2DW7F+oIB38RManJQ8XJoGX6kT0LZE X-Received: by 2002:a05:6a21:a59e:b0:18c:23b0:3b16 with SMTP id gd30-20020a056a21a59e00b0018c23b03b16mr7432806pzc.60.1701773211916; Tue, 05 Dec 2023 02:46:51 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1701773211; cv=pass; d=google.com; s=arc-20160816; b=FOsqI2jAW98dC+3tkfAHBkcRFM/pB3BlU9JJ5o+0hGkdCI9yI8rMaswlPJJLB8iT7q jBGeRjvaKwnsHrm7o0+ievz9cpSQrBFFr2fCR20vFYhpbeomE3zRvFyA4ASeZ5yZ+cD8 +8rVQynuYnAQZLlIiwaQDyHX7ujWRXCT9xJd6Q9/ShM2SrB8FnsZrNZYhNdb18EQ89zR dtn3aBcp6Z7DRIt5KJXVcGTWblCYbwpQLzEDVCkjmfNPN4MQYVcnTQdK6OmlKIqc/soI ExpA41qumS7NUZ22MQDhmz8xKC6r+Yb82IwERFsTSEs6lMAl6sfKQJJYy7mJm7uS6I0o wVRA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9wS5MTVIMLhDRBwh0qlAFrF8E+Hyo1Jur/HfthTmKAA=; fh=IC8CtaUsCHH+VLkdER9IbUix5yCQzPbBUzKtIdyo0ag=; b=dFtIkYs38uVRy3YQWyyYfXLwFwnN8gO0Es6N3Oww9TWaeZxfXCLH39AQ7sa0ErGjrL h+O3ljh3x2+a5RWWfWd0Q4pvghjnEeAOj78JfpyPih9+G5vOOSselNdvo2XS4pxMbOFX yFkrp6QAOcES/5QDyiN7MR9vZkhz/fDJIQy7L01sUArG5Gb3eZsvZl7Bd4xdGr4j/XcK nizs7CnJcnAyW4utQKIuqo8aAndjGWm4DeZZ8joP5xZh7Mp6rnkAcjjd9qQNTKKC2xrx p5gSYXempkUM4OwMJw/8LQAS/x0Dk4JimohsyVTmWmoO9skgetiy85TY56k/v7GUzY7E RfqQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=QyMv4RwX; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id h2-20020a635302000000b005c661524f67si5273188pgb.26.2023.12.05.02.46.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Dec 2023 02:46:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=QyMv4RwX; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 25518803E4BA; Tue, 5 Dec 2023 02:46:50 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346506AbjLEKql (ORCPT + 99 others); Tue, 5 Dec 2023 05:46:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52462 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235101AbjLEKqg (ORCPT ); Tue, 5 Dec 2023 05:46:36 -0500 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2057.outbound.protection.outlook.com [40.107.223.57]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E9340113; Tue, 5 Dec 2023 02:46:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kd8Ji78g1bHdflSqgDqY+vNsTMkQLbWZnJYHdSUZa2iCw07yOznGMvoh2lS2CRLKWy5Hxstswd+sNH9OI8J6pP/uEwoO5HKP3IMZSOne/0i6aoe40mr5+C6WcFUGQ0p6tWpSxIOvhz1Cqpp3/iFSTU5WPJmPMJtS4Ok/aQMXx5U/9ofSjXqxILVxdNOyGdbz02W7oTJ3nKflYN1wp+fk8E8NPeJ2YoUAyAd5JnhjMnnJR2+fSB/IxcKEhMWLbXrYHavLJXDc5hn6z5jnR6kegqp6by1mC0kfMrcIbe7OIrhJ4v/KzDZmMPBVQTBZ9c9BmruPhCrJn896mbMK1ufY3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9wS5MTVIMLhDRBwh0qlAFrF8E+Hyo1Jur/HfthTmKAA=; b=FXFzlzttpF/ejoHXBoYIeurZA66qmWp13EmuXKENE7Gsy/vRmw1qUGp8TYAQfPlo5zH2cO2kFiYC8X5FjCYZFqGQF/cJSk2HQp40y5cgALWBmlZz3UYOz7t4daPburJ3AZPBHWaK5mLn2RZ3LC5KWagFqsP8o4r27XFPhj42LS1wKerVLSnRtHLdWyRZv4WHkZNNjS2wSfweNJbu5rClXHdMta0hQLHFIHlW/au7qSx48nSmwrn2GGRJULWQX1A4HsufCeav1Q//puDhhUmkrj4gN3pJA/s8SsbzhD9POMKY3XbYVUjQbdOvB1IzXP6jA7Yi24b0rFd9KkhnxQNlAA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9wS5MTVIMLhDRBwh0qlAFrF8E+Hyo1Jur/HfthTmKAA=; b=QyMv4RwX0175InpUiemWVAD0LMH+6CDeLnoFajYqBJGuA7F03MPYxfT9TiDy85eyjgSEpAhU8MqHauXJ33FcdNdB+iIiie8r29mVWxp7B2/ytnhRWl/aygY+aSMi4R+uIhcl5lV8AdlTAxhv+0CTy9CDbJOBwJWs3Sm8m/F9B+y4IKYVD3bI3AxHrrLNfVjCQcsdGlfLGehzL4KCph7GvCC9nT69D5den9MhuQQHE+oKnidQJPPCUNxxagZH2otmct6jT1knhSLZjEas4/3jZSUrZRimPSXKcoF2ZDaariTO3knwBK/JZ0Ik/5Jou7jWUlSEsByGxWjhbrQP7gDfGQ== Received: from SA1P222CA0178.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:3c4::26) by DS7PR12MB5813.namprd12.prod.outlook.com (2603:10b6:8:75::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Tue, 5 Dec 2023 10:46:38 +0000 Received: from SA2PEPF000015C9.namprd03.prod.outlook.com (2603:10b6:806:3c4:cafe::ae) by SA1P222CA0178.outlook.office365.com (2603:10b6:806:3c4::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.27 via Frontend Transport; Tue, 5 Dec 2023 10:46:38 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SA2PEPF000015C9.mail.protection.outlook.com (10.167.241.199) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7068.20 via Frontend Transport; Tue, 5 Dec 2023 10:46:38 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 5 Dec 2023 02:46:24 -0800 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 5 Dec 2023 02:46:23 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 5 Dec 2023 02:46:20 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH mlx5-vhost v2 1/8] vdpa/mlx5: Expose resumable vq capability Date: Tue, 5 Dec 2023 12:46:02 +0200 Message-ID: <20231205104609.876194-2-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231205104609.876194-1-dtatulea@nvidia.com> References: <20231205104609.876194-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF000015C9:EE_|DS7PR12MB5813:EE_ X-MS-Office365-Filtering-Correlation-Id: e4887589-6877-41c0-8a14-08dbf57f758a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: OxPwki4y9ZrRscJaZc/zhR4r6xHrMqt3SdhOOIvcRdXPaw01JTfL2GRitwHqQdFQOues5obHl3PPI4RCCRDy2/DUU5J2Agx69I7qF1aLOlCti7rnfVo8MtcMkPzS111V+mQoOtH0qUWHaUlMvR3xKYWj1IRHTwLmN8IHBB8EVxMWZ/4qe9iM0mNhIk3pDs41T2rElgAl3gY2NnNKB7z86keaJ3ENg1YGxw9q1OXW4wcGtyKiN1wFpe1bZ7HJZkorJe2yfgTXFRr+mUdKzr4iV1PrxFmfSYWN7AxpjdqUckTfbvn9SAoE23pP/ImZQlrcDbhSBPXbETKCBmxXR6BHp3IpVjrtVLo7pOBDkeojgYdnOyLXsVlDNF+ipmId1InQqUxF77aU0INSWRrmtIF1qyr3nHV0Rj2PrM1kN1ZPVElhQIhXndq9MB0wvp+h6VvSz4pqb2HV6DTYJbnSU3S429jcvLeKT3gKxwDVPs86iWjy0JBJ6FshJ2pAbrH2jr7ciD7GxEw6iV+9oZtV523UuvZPbM3myCb4/CFQL8xOXsJMt6BFTABalqlR96MOwRRCHmlSf2wXis4Ramx/gT7nO1/wbzYRH1BQTo73hNsKpo3H6pSCjSMFV4q8wYdYAefKs0h9hV34fZ6hMlplWFsiIp5xafXVEk1rl/U9dKQr/c9BfLphpaTtumiCxZf1+qTFlY999OiHo30X38IQvVsd18KYqf1FLpoZxA4hlBNAKzNmbK8XQzkKAJCkNSHK5ZcR X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(39860400002)(136003)(346002)(396003)(376002)(230922051799003)(1800799012)(186009)(82310400011)(451199024)(64100799003)(36840700001)(46966006)(40470700004)(36860700001)(40480700001)(478600001)(6666004)(110136005)(54906003)(6636002)(356005)(7636003)(70586007)(70206006)(316002)(26005)(47076005)(2616005)(1076003)(83380400001)(4326008)(8676002)(8936002)(66574015)(426003)(336012)(82740400003)(40460700003)(5660300002)(2906002)(41300700001)(86362001)(4744005)(36756003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Dec 2023 10:46:38.3316 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e4887589-6877-41c0-8a14-08dbf57f758a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF000015C9.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB5813 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Tue, 05 Dec 2023 02:46:50 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784438547596122352 X-GMAIL-MSGID: 1784438547596122352 Necessary for checking if resumable vqs are supported by the hardware. Actual support will be added in a downstream patch. Signed-off-by: Dragos Tatulea Reviewed-by: Gal Pressman Acked-by: Eugenio Pérez --- include/linux/mlx5/mlx5_ifc.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index 6f3631425f38..9eaceaf6bcb0 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -1236,7 +1236,8 @@ struct mlx5_ifc_virtio_emulation_cap_bits { u8 reserved_at_c0[0x13]; u8 desc_group_mkey_supported[0x1]; - u8 reserved_at_d4[0xc]; + u8 freeze_to_rdy_supported[0x1]; + u8 reserved_at_d5[0xb]; u8 reserved_at_e0[0x20]; From patchwork Tue Dec 5 10:46:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 173946 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp3342312vqy; Tue, 5 Dec 2023 02:47:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IEq9JtQmzuDRXY5CPOh0nDV04x8rm3zbwfkVNZZPOfM/DEfGK/yTBKRJwJqktbJC+HaAQ3I X-Received: by 2002:a17:90a:3ee2:b0:286:975f:7e36 with SMTP id k89-20020a17090a3ee200b00286975f7e36mr1264009pjc.12.1701773233885; Tue, 05 Dec 2023 02:47:13 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1701773233; cv=pass; d=google.com; s=arc-20160816; b=Vt6X7Wtpalf3+tJns8cr4k79JuAUrFZ9KoPvy5kCW6VPVN9ul0f9cOXeF+iuFLoTrH +4DW/PGvWMMqB99rF0dD0btm65dTvzaKIiSHrypcmLxtCCJtjgVAulSa7clDSOIhSjro 3zj7rmJN/P2mnGBSHPlTscdsl8i9MnJjV5Hl7cUOtWZq1er6FxzS8kjmz7Ws1zyh04hD u4uzHAdzKFz7oDhe6jWbmLJcpi9Ph2MPo4k4yU95e9/QlkB7d/PBNg69Ficx8NwLueMK xQixvU9wTRiOk1U/gla2mTtahiJunhH1+j6GWf24FcsWVqfS9yhMktbt4MVyinmuKqh/ nnVw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=An8Syir92Vj8AodKfUp5XRBa97QSN66p6UCwyLSlCOQ=; fh=IC8CtaUsCHH+VLkdER9IbUix5yCQzPbBUzKtIdyo0ag=; b=jSkTyffyOEHbuEqtrP24HCmOBn7u1Z1pDcrA1/0f3vdkzn4u54QJgJOP/s3Y1X+uej cWWWF0Ji3rypTvAGB7I8Vr9B3NpcFIKhf2dBHlWpHLC+PWVKvMbAAWHxvrCrkqkCMTaq Hyzj8WFJNOM5w/p20Qach1pAR+Lhx/76aty4c/7G7MkKeM/WCO2sdqFTalfViSHQT94F QHC0LJ38spWGMPublov+iYkyRztL5ZlVUFpnsFR9E2ouSLsafeHqAxRGrFZgYO5dFJMx H+scJLC+5i9Jrw+EfWaNQSyZ3edMxw8qRcChEUG4NlOlEkAWkyKX3RKOZ19DWPSPZacy kudQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="F/tA/RL6"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id e9-20020a170902b78900b001d00a9256adsi7476020pls.488.2023.12.05.02.47.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Dec 2023 02:47:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="F/tA/RL6"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id E180F8040398; Tue, 5 Dec 2023 02:47:09 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376415AbjLEKqo (ORCPT + 99 others); Tue, 5 Dec 2023 05:46:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52466 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235105AbjLEKqg (ORCPT ); Tue, 5 Dec 2023 05:46:36 -0500 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2043.outbound.protection.outlook.com [40.107.93.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5E9B10F; Tue, 5 Dec 2023 02:46:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cAGWuOkAPSgZf47UgUIJc/EDqaq88vapzZGIsm1X2c9yDBtBmdnfGb4RuzCGNPzOl2SA7tORlSmaT7O9zr7lruamjYpec3Et8wBTpO5/LsLo7X3r9dHy01tOo2jML/jQhuDDwNzGLrlzWRGVEsJix3XrGRPGv3NZ47Q0fGszrPkrizzWMxSykO59Q5h+ecmZnwAMNxHr6YqhHpm13qBxKOUyzledptabQYToqL7yBkyLS1aGy+pVNYm0zERJWZJYOcB19Ea7sR/ntCYHgk7GEiTdT9UYqCmhe05f0C6Z1BfTaNJZTa6ut/7+jEYRzkoOMfZy+BVnd56dCHnhikuchw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=An8Syir92Vj8AodKfUp5XRBa97QSN66p6UCwyLSlCOQ=; b=NqFRK7/LOvYjVyVzg5sUn3chK0AOxzZBr21Ic2CQcAu3OxFd+IIipeN/eDHoSgPiTljYPqbQoR9gLocCgQyiVx/npYpkd/kd4wTaM57GELM+b1FueTNIFmtEbbdh2F7BnfW874qm/g9GYeaB56LVfx1vnnYEKeFOU9o6G1p2MktDx1d3fcs/E+af+B7hs7a2iUnGBNZKy2E1Bgj6vyoNAPL/wq1vyIBrtuxxARegkxbRZpdge40yFZMHSRpHi0ZnWb4CGW2+AexQ8FQ87TCjzt5ThaVan/CResb4wkMvJl8m4xjb0JIAVaMAcMUoBBH2DXe6Skia6M0CvZmh0ged2Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=An8Syir92Vj8AodKfUp5XRBa97QSN66p6UCwyLSlCOQ=; b=F/tA/RL6s5ahgwNIe2dN2E/5SAUwEBuww7Wub/hm5Sl2wd09CWYgH4MPZRnZjZjq8gm9Jpx3Wca3nSbdB8vU2HO5kDdqHDpN9lxRStIgIr+SKbGmnoDUtgp4oQ9lpf/8oSBBwgOxD9MGIENhGGmqeKTTqW6B4NFmmDrtOEn+n8EzemdXGNNIuLSpRkzbtFjGJbwasCkq+lBf17OMtMKInDRYWA0+zn6GeBvp6vhMBvx4vVJyLbFmDJEK259q/BK9BLbF83KV8ENg92w1ZomhN7pEAYyBPSSGtayUy+f5a7qUA5kbCQgMb/2jKyRLk/jyWenF92BaC79/dnFLE6DGtw== Received: from DS0PR17CA0002.namprd17.prod.outlook.com (2603:10b6:8:191::25) by PH7PR12MB8426.namprd12.prod.outlook.com (2603:10b6:510:241::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Tue, 5 Dec 2023 10:46:39 +0000 Received: from DS3PEPF000099D6.namprd04.prod.outlook.com (2603:10b6:8:191:cafe::38) by DS0PR17CA0002.outlook.office365.com (2603:10b6:8:191::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34 via Frontend Transport; Tue, 5 Dec 2023 10:46:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DS3PEPF000099D6.mail.protection.outlook.com (10.167.17.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7068.20 via Frontend Transport; Tue, 5 Dec 2023 10:46:39 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 5 Dec 2023 02:46:27 -0800 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 5 Dec 2023 02:46:27 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 5 Dec 2023 02:46:24 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v2 2/8] vdpa/mlx5: Allow modifying multiple vq fields in one modify command Date: Tue, 5 Dec 2023 12:46:03 +0200 Message-ID: <20231205104609.876194-3-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231205104609.876194-1-dtatulea@nvidia.com> References: <20231205104609.876194-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099D6:EE_|PH7PR12MB8426:EE_ X-MS-Office365-Filtering-Correlation-Id: ef98bad9-3676-41ce-027d-08dbf57f760d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: AIBnSo+Y4nqUjO/K8pw0btR6huDh5PCkEE2r6rcvpim2+bfQN0YbBhX3RM6DeaUSmA/9EbKJrVaz1cRZziQal9HuMP2q27nD6U4xYcRf1JxuphvEos9TcAQSPOYWw9AaODKcKilG04HQMZH71vJuzPhav15GLJXaCWtPVMfP+xQAMfZbdB9wluLS/C3HJWCEvPlNGSotNlhDKygiAYb2SJSiqkSC/+l4Xe/cIaQDRxYcbk/q9dXhdZ4xgjWf60bace8w6lLz1HmiP4TsmWsY3GOwVd2+JdfsmgqukxnQzXvasS5Ap+1n5SXEBGnX3cbovcjidb42208Tqy5YZ6nXBhyiowC4Z3UOoXWvPDdzSGJo6nX4tv7aH4XVNQOElWaCbiZA+0JZgsH6CVJzQmK7yh6nD/y5MO9JtvDJnctMJH5YOF0CUh4tIEHSBE5HIMIyo87yDILrZjsnekyShw39aW7SB2AGvlgGzTMGoF4bRSrVkFLl3TFgRBf34Re5VxZtu2eEp+x98C4U18kv1lefyHFwmtQy+3cdZ9F8pLZeejKNpt/jHUDia+Lg3KpOmR16uAXZWxKGzZ7pNIFKmouCPbIdSGNTyPPVNjnSfwzWaBlw5yNUrjnLPNCwLw78XAjldes3Fgf2WCnKcavEdB3CbZ+l/J7TmnLLEE63iNqWbtXXjHCnrwTPVpPxqYh/Q/iYl8SnMI0phehN8LBrI8IXbAd+Alfit0FIxbBqQKA29e5bkzNnJkDS1RMsCvxK6sT2 X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(376002)(346002)(136003)(396003)(39860400002)(230922051799003)(451199024)(82310400011)(64100799003)(1800799012)(186009)(36840700001)(46966006)(40470700004)(8936002)(83380400001)(426003)(8676002)(4326008)(40460700003)(66899024)(82740400003)(7636003)(36756003)(41300700001)(86362001)(356005)(2906002)(47076005)(36860700001)(5660300002)(6666004)(1076003)(26005)(478600001)(40480700001)(2616005)(70206006)(54906003)(110136005)(336012)(316002)(70586007)(6636002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Dec 2023 10:46:39.1866 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ef98bad9-3676-41ce-027d-08dbf57f760d X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099D6.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB8426 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Tue, 05 Dec 2023 02:47:10 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784438570400705277 X-GMAIL-MSGID: 1784438570400705277 Add a bitmask variable that tracks hw vq field changes that are supposed to be modified on next hw vq change command. This will be useful to set multiple vq fields when resuming the vq. Signed-off-by: Dragos Tatulea Reviewed-by: Gal Pressman --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 48 +++++++++++++++++++++++++------ 1 file changed, 40 insertions(+), 8 deletions(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 26ba7da6b410..1e08a8805640 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -120,6 +120,9 @@ struct mlx5_vdpa_virtqueue { u16 avail_idx; u16 used_idx; int fw_state; + + u64 modified_fields; + struct msi_map map; /* keep last in the struct */ @@ -1181,7 +1184,19 @@ static bool is_valid_state_change(int oldstate, int newstate) } } -static int modify_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq, int state) +static bool modifiable_virtqueue_fields(struct mlx5_vdpa_virtqueue *mvq) +{ + /* Only state is always modifiable */ + if (mvq->modified_fields & ~MLX5_VIRTQ_MODIFY_MASK_STATE) + return mvq->fw_state == MLX5_VIRTIO_NET_Q_OBJECT_STATE_INIT || + mvq->fw_state == MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND; + + return true; +} + +static int modify_virtqueue(struct mlx5_vdpa_net *ndev, + struct mlx5_vdpa_virtqueue *mvq, + int state) { int inlen = MLX5_ST_SZ_BYTES(modify_virtio_net_q_in); u32 out[MLX5_ST_SZ_DW(modify_virtio_net_q_out)] = {}; @@ -1193,6 +1208,9 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque if (mvq->fw_state == MLX5_VIRTIO_NET_Q_OBJECT_NONE) return 0; + if (!modifiable_virtqueue_fields(mvq)) + return -EINVAL; + if (!is_valid_state_change(mvq->fw_state, state)) return -EINVAL; @@ -1208,17 +1226,28 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque MLX5_SET(general_obj_in_cmd_hdr, cmd_hdr, uid, ndev->mvdev.res.uid); obj_context = MLX5_ADDR_OF(modify_virtio_net_q_in, in, obj_context); - MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, - MLX5_VIRTQ_MODIFY_MASK_STATE); - MLX5_SET(virtio_net_q_object, obj_context, state, state); + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_STATE) + MLX5_SET(virtio_net_q_object, obj_context, state, state); + + MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields); err = mlx5_cmd_exec(ndev->mvdev.mdev, in, inlen, out, sizeof(out)); kfree(in); if (!err) mvq->fw_state = state; + mvq->modified_fields = 0; + return err; } +static int modify_virtqueue_state(struct mlx5_vdpa_net *ndev, + struct mlx5_vdpa_virtqueue *mvq, + unsigned int state) +{ + mvq->modified_fields |= MLX5_VIRTQ_MODIFY_MASK_STATE; + return modify_virtqueue(ndev, mvq, state); +} + static int counter_set_alloc(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq) { u32 in[MLX5_ST_SZ_DW(create_virtio_q_counters_in)] = {}; @@ -1347,7 +1376,7 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq) goto err_vq; if (mvq->ready) { - err = modify_virtqueue(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY); + err = modify_virtqueue_state(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY); if (err) { mlx5_vdpa_warn(&ndev->mvdev, "failed to modify to ready vq idx %d(%d)\n", idx, err); @@ -1382,7 +1411,7 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *m if (mvq->fw_state != MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY) return; - if (modify_virtqueue(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND)) + if (modify_virtqueue_state(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND)) mlx5_vdpa_warn(&ndev->mvdev, "modify to suspend failed\n"); if (query_virtqueue(ndev, mvq, &attr)) { @@ -1407,6 +1436,7 @@ static void teardown_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue * return; suspend_vq(ndev, mvq); + mvq->modified_fields = 0; destroy_virtqueue(ndev, mvq); dealloc_vector(ndev, mvq); counter_set_dealloc(ndev, mvq); @@ -2207,7 +2237,7 @@ static void mlx5_vdpa_set_vq_ready(struct vdpa_device *vdev, u16 idx, bool ready if (!ready) { suspend_vq(ndev, mvq); } else { - err = modify_virtqueue(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY); + err = modify_virtqueue_state(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY); if (err) { mlx5_vdpa_warn(mvdev, "modify VQ %d to ready failed (%d)\n", idx, err); ready = false; @@ -2804,8 +2834,10 @@ static void clear_vqs_ready(struct mlx5_vdpa_net *ndev) { int i; - for (i = 0; i < ndev->mvdev.max_vqs; i++) + for (i = 0; i < ndev->mvdev.max_vqs; i++) { ndev->vqs[i].ready = false; + ndev->vqs[i].modified_fields = 0; + } ndev->mvdev.cvq.ready = false; } From patchwork Tue Dec 5 10:46:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 173945 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp3342257vqy; Tue, 5 Dec 2023 02:47:04 -0800 (PST) X-Google-Smtp-Source: AGHT+IE3gH3Asmb3/XGc90yMw62lBijInxZXf2uCaCE1ydTf1edS35nLKJrawszLNOQ45w8/0O8e X-Received: by 2002:a05:6a20:2443:b0:18f:97c:8a2a with SMTP id t3-20020a056a20244300b0018f097c8a2amr7121205pzc.85.1701773223688; Tue, 05 Dec 2023 02:47:03 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1701773223; cv=pass; d=google.com; s=arc-20160816; b=KFO4o1/kRQSiPhcowUr8DbztdJTgpYRR7Oa65POTVkgb6jpnbWzkhSBiDvOa/ET0U9 y4rDjU4bUYGXTWttwL63emjBghomjh55hNoaTStRMZPQr4mkMi3WucWQOwQI2CreBs4/ eSb5yXBs8rhVBP25WndltI/FJLvv4Npl3lzqlY7HHlVjI/pe6117rwUROWeXoM+YDMPo ElLx8eeibJMFQ6/CoDeo6w0jN8LJw3LX+B+yyx0NZ16Bn6RqcWJypZQOKa8M+PWM+b+I Hc5XlI+PIEYLj/1UFMAfr/CKPQXgUQ0SUQsQvdgg9vThFcWFIfcz1kj6njPYHZlG19SC qiAw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=DOLe22Cwu8VjfCRuEQma2mo/tC+HslROt3JYWL8C5OI=; fh=IC8CtaUsCHH+VLkdER9IbUix5yCQzPbBUzKtIdyo0ag=; b=jhcCV4MCUPvOL6uYQZaHmI5cwpL9d6eihgOchdXiXHRVL2tj0PUw6O64h0pf0COkgX KG8t3Dhiu57q168IYGG7cpxTW1DFHdElYIQlTlDJTG6uBMixBTwcfKJfK81tsB7NvZ9f eorV/unPV9A/dLpGMrT169WL3G9pczM6d/NYCqzvdJpP1rC++7c0/4P6e431kSK2H8fH UYr4YzHdX2j65gZB/sElYarcQ5TWalnsuNg4Cv9DHRPlVaj7tcZAFo0t/4kLGO6t7A70 75RwZ/098W2c4TMcb14U7NNzzKvb8kaLBbX+uVIEgBhvhTNT82cCAdS5BYBsCo3h+GLZ Yacg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=Uh99H4uW; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id d15-20020a056a00198f00b006be55174f3fsi404100pfl.28.2023.12.05.02.47.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Dec 2023 02:47:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=Uh99H4uW; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 84A9180A6EF6; Tue, 5 Dec 2023 02:46:58 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376536AbjLEKqr (ORCPT + 99 others); Tue, 5 Dec 2023 05:46:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52488 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235111AbjLEKqk (ORCPT ); Tue, 5 Dec 2023 05:46:40 -0500 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2080.outbound.protection.outlook.com [40.107.223.80]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6816D135; Tue, 5 Dec 2023 02:46:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HJuR5Uvh7dWkL4X9t8luywoDmCl5CEI5rxgA7lwfZEvLmQ8hYWDqmbonYpAS3nFz/f9IKeKdhpzKhyX31sU3QKh6oFZQXkiSCE3C/rc7w2GgsQiBakEFIqbzbYnMAYQXnNeHtRRBOD5amTthDDWB4mrStuQxazznhN7XsWr++9Pakv6PA8favfPGaiaSk7xTK8WcuMVyyWDu5VhXF3nP5fL9NbMzziYhZJSCrPkb+3u5dtD33YW9EcbOI+acaR6ddaRZpgkV4e0GtyIwF66QsgNfcFV0PO5r+Zs1hg9pzM5YRHm4rJyIe7LhGXN4ws4gDtvnfeKg1/ZqkGRWZ/ggqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DOLe22Cwu8VjfCRuEQma2mo/tC+HslROt3JYWL8C5OI=; b=dvwEwpBKV66pqE6/3nWj1n1B7ROE79lUJBw0FRq5kUf+u/I1O1p8htrTkzVZmH/QuYRqEUTaGQVK0jVn0Grik6s3Uci6LPEPKCmlqRQrqlljmmjWTuK7CLL72dj+5qhBdbbbl8YA+Fmf1U/TYj6fq+AGhar4afKZ5y74tfLbdTSnEc/UINz3fb7TCM2cvURr5MBLIEHNYn7l6T7t1dKWQkLueFEcv/YQ0OMDFUjtdO3tm/d3UK9Lz40OvrQCRf5tT2C64NpZluIgzD0BjgdehDgSjC4tEIlpQ6HHMNr+Yx18d+e0JSIF6u/L3WyKtZiLrRmyQrBYOWVjCWAdGT7rCQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DOLe22Cwu8VjfCRuEQma2mo/tC+HslROt3JYWL8C5OI=; b=Uh99H4uWhyQuPocAiIlGD7vmJQbJ2s7MmMJaCxj297QqEebic0kD6z6RK8mykfgnvuwmvJ+kObiMyAzKp5cEhOtR5RJBbv1d23bFubNUg92QacBwFK7iiegUXWvUmKKMEjTOXJM9F9KsO9JiJM+a0dzGEIStbBDtZ+siHD2eV4jOVdqTDfYpcpD68MItFp8habx4guwKzLXN9/ijqpRJU07v5Jqut4etBBUcUZAXgCqa5FKqXh5OJ47UW1E7mmLsNmuUbZ5IeSPph70Wn9bvbmXv2U2Tjt0TbAThaUFPKY5w0MDqrP1JUfSkLpXdjx9nepXyGePNUIVVBoi8EUO0nQ== Received: from DS0PR17CA0006.namprd17.prod.outlook.com (2603:10b6:8:191::22) by MN0PR12MB6318.namprd12.prod.outlook.com (2603:10b6:208:3c1::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Tue, 5 Dec 2023 10:46:42 +0000 Received: from DS3PEPF000099D6.namprd04.prod.outlook.com (2603:10b6:8:191:cafe::5d) by DS0PR17CA0006.outlook.office365.com (2603:10b6:8:191::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34 via Frontend Transport; Tue, 5 Dec 2023 10:46:42 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DS3PEPF000099D6.mail.protection.outlook.com (10.167.17.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7068.20 via Frontend Transport; Tue, 5 Dec 2023 10:46:42 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 5 Dec 2023 02:46:31 -0800 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 5 Dec 2023 02:46:31 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 5 Dec 2023 02:46:28 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v2 3/8] vdpa/mlx5: Introduce per vq and device resume Date: Tue, 5 Dec 2023 12:46:04 +0200 Message-ID: <20231205104609.876194-4-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231205104609.876194-1-dtatulea@nvidia.com> References: <20231205104609.876194-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099D6:EE_|MN0PR12MB6318:EE_ X-MS-Office365-Filtering-Correlation-Id: 32d1da99-f106-405d-eac5-08dbf57f77fc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: SWNtjpexaz/8Wv341D4tK8xN+yRpVaDNDpt30PEQHW5/d46P9hqFK5JdwmlYsp9156yCF5teLnSpJW1/h6aRdjA+opDue+3ftilrwSdKU+DGZDGGv6ySfeaBXZWZjhDZyqMu4prOVwHY/LPoqViWXpgvDkSs70T8vO7WcDmN4rQpknEpUsdgNx6pBUJVlFq+kJvbrE4ANMTlz2s3l0p4O3qw7CHnOOc4G3+nOZFzgH3gq3dHhAy0+0khCEIOJ9PG1ZGOgXEZQVzpUc53iKuUY+ofUqdlbcHZTfjb1mJ2Zq1glVsYPcHPHrRFmp0+KaNdrxzg++gNp3kIf3lVlcfftH1jb1UtcTwumUuFH0SKngCdWfDp3KXCfSjJFEY0jEtotBpauTK3Dj41Ah62XWB0pvVbq6ua4bDFrXI/qUeyIubwX4xlNdJ6MR+8PU3snYTV0Z0Vo1IYdy3XUeMBe9PYLoppOjdiz745eXTU7ghPowqIBTtq3YMLmkBWsJveY3e6AMOsX4sublOj5uiCSiEobvxegIbSJubyQ0ZkYbYa9AasRblpjnHXa5k44OduX3GCoMuIRnNteciJu+MBSt8hHbz9tDt9E/LdiAThgq54HkuIcfgpPRd0FiuMA0mNVjMOYK4bWAvrlsax/3OX+kDuZMzIrqbjWwUAV+BT939YZkNymFtulN8V2S8vmPrzjxT0eoLQp6IUd666u3V/LsH5bIPLVKwdhDL6wKqRWw/2t+ysqlAUFdL3GYBLLi3Ah0FK X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(346002)(39860400002)(396003)(136003)(376002)(230922051799003)(451199024)(186009)(64100799003)(82310400011)(1800799012)(46966006)(40470700004)(36840700001)(1076003)(2616005)(41300700001)(36756003)(86362001)(66574015)(426003)(40460700003)(83380400001)(26005)(336012)(110136005)(70586007)(316002)(54906003)(6636002)(70206006)(6666004)(4326008)(8676002)(8936002)(478600001)(82740400003)(356005)(36860700001)(2906002)(7636003)(40480700001)(47076005)(5660300002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Dec 2023 10:46:42.4366 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 32d1da99-f106-405d-eac5-08dbf57f77fc X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099D6.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6318 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Tue, 05 Dec 2023 02:46:58 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784438559764282374 X-GMAIL-MSGID: 1784438559764282374 Implement vdpa vq and device resume if capability detected. Add support for suspend -> ready state change. Signed-off-by: Dragos Tatulea Reviewed-by: Gal Pressman Acked-by: Eugenio Pérez --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 69 +++++++++++++++++++++++++++---- 1 file changed, 62 insertions(+), 7 deletions(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 1e08a8805640..f8f088cced50 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -1170,7 +1170,12 @@ static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueu return err; } -static bool is_valid_state_change(int oldstate, int newstate) +static bool is_resumable(struct mlx5_vdpa_net *ndev) +{ + return ndev->mvdev.vdev.config->resume; +} + +static bool is_valid_state_change(int oldstate, int newstate, bool resumable) { switch (oldstate) { case MLX5_VIRTIO_NET_Q_OBJECT_STATE_INIT: @@ -1178,6 +1183,7 @@ static bool is_valid_state_change(int oldstate, int newstate) case MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY: return newstate == MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND; case MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND: + return resumable ? newstate == MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY : false; case MLX5_VIRTIO_NET_Q_OBJECT_STATE_ERR: default: return false; @@ -1200,6 +1206,7 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, { int inlen = MLX5_ST_SZ_BYTES(modify_virtio_net_q_in); u32 out[MLX5_ST_SZ_DW(modify_virtio_net_q_out)] = {}; + bool state_change = false; void *obj_context; void *cmd_hdr; void *in; @@ -1211,9 +1218,6 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, if (!modifiable_virtqueue_fields(mvq)) return -EINVAL; - if (!is_valid_state_change(mvq->fw_state, state)) - return -EINVAL; - in = kzalloc(inlen, GFP_KERNEL); if (!in) return -ENOMEM; @@ -1226,17 +1230,29 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, MLX5_SET(general_obj_in_cmd_hdr, cmd_hdr, uid, ndev->mvdev.res.uid); obj_context = MLX5_ADDR_OF(modify_virtio_net_q_in, in, obj_context); - if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_STATE) + + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_STATE) { + if (!is_valid_state_change(mvq->fw_state, state, is_resumable(ndev))) { + err = -EINVAL; + goto done; + } + MLX5_SET(virtio_net_q_object, obj_context, state, state); + state_change = true; + } MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields); err = mlx5_cmd_exec(ndev->mvdev.mdev, in, inlen, out, sizeof(out)); - kfree(in); - if (!err) + if (err) + goto done; + + if (state_change) mvq->fw_state = state; mvq->modified_fields = 0; +done: + kfree(in); return err; } @@ -1430,6 +1446,24 @@ static void suspend_vqs(struct mlx5_vdpa_net *ndev) suspend_vq(ndev, &ndev->vqs[i]); } +static void resume_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq) +{ + if (!mvq->initialized || !is_resumable(ndev)) + return; + + if (mvq->fw_state != MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND) + return; + + if (modify_virtqueue_state(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY)) + mlx5_vdpa_warn(&ndev->mvdev, "modify to resume failed for vq %u\n", mvq->index); +} + +static void resume_vqs(struct mlx5_vdpa_net *ndev) +{ + for (int i = 0; i < ndev->mvdev.max_vqs; i++) + resume_vq(ndev, &ndev->vqs[i]); +} + static void teardown_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq) { if (!mvq->initialized) @@ -3261,6 +3295,23 @@ static int mlx5_vdpa_suspend(struct vdpa_device *vdev) return 0; } +static int mlx5_vdpa_resume(struct vdpa_device *vdev) +{ + struct mlx5_vdpa_dev *mvdev = to_mvdev(vdev); + struct mlx5_vdpa_net *ndev; + + ndev = to_mlx5_vdpa_ndev(mvdev); + + mlx5_vdpa_info(mvdev, "resuming device\n"); + + down_write(&ndev->reslock); + mvdev->suspended = false; + resume_vqs(ndev); + register_link_notifier(ndev); + up_write(&ndev->reslock); + return 0; +} + static int mlx5_set_group_asid(struct vdpa_device *vdev, u32 group, unsigned int asid) { @@ -3317,6 +3368,7 @@ static const struct vdpa_config_ops mlx5_vdpa_ops = { .get_vq_dma_dev = mlx5_get_vq_dma_dev, .free = mlx5_vdpa_free, .suspend = mlx5_vdpa_suspend, + .resume = mlx5_vdpa_resume, /* Op disabled if not supported. */ }; static int query_mtu(struct mlx5_core_dev *mdev, u16 *mtu) @@ -3688,6 +3740,9 @@ static int mlx5v_probe(struct auxiliary_device *adev, if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, desc_group_mkey_supported)) mgtdev->vdpa_ops.get_vq_desc_group = NULL; + if (!MLX5_CAP_DEV_VDPA_EMULATION(mdev, freeze_to_rdy_supported)) + mgtdev->vdpa_ops.resume = NULL; + err = vdpa_mgmtdev_register(&mgtdev->mgtdev); if (err) goto reg_err; From patchwork Tue Dec 5 10:46:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 173950 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp3342397vqy; Tue, 5 Dec 2023 02:47:27 -0800 (PST) X-Google-Smtp-Source: AGHT+IHaYuQVG407nJTHmRNs5I3QTbWKJH1NfZmEaP6Al6U9nHfKbWSfQYIqzZByWCtEY++VFvGM X-Received: by 2002:a05:6a20:a19b:b0:18f:97c:927e with SMTP id r27-20020a056a20a19b00b0018f097c927emr2301752pzk.99.1701773247107; Tue, 05 Dec 2023 02:47:27 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1701773247; cv=pass; d=google.com; s=arc-20160816; b=yUq77W25EvYyqXEtoCr4Q4wKokF+AV5UrtMRPZ7XMCLE5EtrzLsRpwZrtdXUepbb67 qDg+sXQQzaylJabVZFSoTUpFQ73pghw5p+axnXmnwV2mKBDiu9LyN4XqRkjVLKwIPnQE hWf5HpzAj/J00uFcR0Zr6WNHRJmpYiKKma3aMoedSfaQQZP7LsZa524j6wzzFP8wmyFj hPeu3yWAIpbKYmL1Ztp4rOksAEztGouGRiy96CElIZBrvoMkcu0O0VJMV5D7TH2Bwk+y cwCG5kbYvY/DRhYtNjRts9IxGoOh4dCHrdikMuiHmFUUvotMz1B5dKYV1dmi21YhVzkx ScWQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CGyLeP1cFq3pBUJz5P3wMAmQBwNy91SdqMFyXqiZ1nM=; fh=IC8CtaUsCHH+VLkdER9IbUix5yCQzPbBUzKtIdyo0ag=; b=j8fsTWNiTGnNO+j9meodfHZDUshSBySAhOVAYMOCLArx8W60kwTC8IZ39ecLljRdJJ O6miEXeK3iTQZIPw+wI119GYslinhp6aJtNxh2mibZQo76f1npRrxY6MciJhhcnws4em 94Jbyb7adbKSdQ/bLpwHmo3Ryxx2PeXQhezRXUjDyK3YJ/0tjZTWJwjlZNZkg55tkl5j hyhfMMMKSm9NAc6HfZ6uJcL2R/IXrcKYOuzbQshZIMUCR+WQ0EcpHZnw/pRIsd/emxHp Xq2wf8FIZML7rOAbQwyOBhbttx4rzhdlLdY3D6h088fm2L323UKVsDHsEy82VyePtzXU bo6Q== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=S5znUFSZ; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id c19-20020a056a000ad300b006baae7ecfacsi9596422pfl.308.2023.12.05.02.47.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Dec 2023 02:47:27 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=S5znUFSZ; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 1E30F80403B1; Tue, 5 Dec 2023 02:47:22 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346727AbjLEKq4 (ORCPT + 99 others); Tue, 5 Dec 2023 05:46:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346605AbjLEKqs (ORCPT ); Tue, 5 Dec 2023 05:46:48 -0500 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2059.outbound.protection.outlook.com [40.107.100.59]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3815E116; Tue, 5 Dec 2023 02:46:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Nd/QRdGUnQ8xPL+rTevk8YIRcku05uvXs/pA8STLkTdMdrMLPKx4YtzETSVUhxluMPOflgFcV5z/vaJaHs7ugMAJLp2LuzWJ8F/FtevMGSTPf3cysjYEiMlE5kaZL1cmpfamtJA3jitJVCqPk36oD+e0baIVusF2xz3TnRATeczkdIl+H4MEJ37EueF3kfQa3qkGH0Ipk3Z9C4vaVPh8WSXxMaiurzm5WOPkEIQ6snsQVbiXPjAoJKsXNmVOJdSfQpP2hWCz41kCtjGcJhaGBlPjbDq5lOgNQOYapnaYkPV4uyl62MDHcF0gSjqqv/JFMTzMOp6fFoVo3qksC5ivaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CGyLeP1cFq3pBUJz5P3wMAmQBwNy91SdqMFyXqiZ1nM=; b=gEI6R/2J8QsIeczPvJuycVTvl4R0lUNsOL+CSG6Lu2/dRGiSsXrCRLv0P4FH9fOPaRW/X3h5656+Z1KeQ8cEtW9Tfs9igTXVd/IFF1h58DfY2n28TZJs0tCufezTC320Gp34MSsT6mV3VV1x3g01mqpQUZEO3+Z86mE+FtikkmFKXg11Ti1u+Oo6QVPSyPgpgpVNLODcBT9gzIVyDLFeMUkTCQpAdyNaoYg868FNLmDg5KDOJzSfg8wbb30OtauQZbSSClc9U0ypJtcsuvUFPOBSMzmtSoBG7ise2Hf8Pg6TH86zV+ZpytasNywW2Zto6UBT+64pHhClBUmeG5zvRA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CGyLeP1cFq3pBUJz5P3wMAmQBwNy91SdqMFyXqiZ1nM=; b=S5znUFSZ/SWyNvsvKeHgYthaBbub6BHP/9ttLGhwrMp9LX1YMfTQybApujyBfnfEpPczPXkW4NNTNts1M27Ger27da5vaoFCyDKb5L/mPQ8XPgyN5XEmkDLWXNkMPEJcQVdO//z3spb73WykVoJyufp2IHhO8ByDZwVI5bIYmxoiTlD4Gc7HwOWjaNY9xcFzXRK2Hvn+L+0ZdxXf7H2AuxsHTkluNC7mQbdAMrT0byiVRUkUKqRWKF80+y4EsIkM6/Go4VA2TEWcGsU7G9g+Jfvx2Utzst1aYQlLzLl2OOJMy7e/dynHbBJTUHkWpLfEt4sjkEpWrC3bYcNQtfC6OQ== Received: from PR0P264CA0168.FRAP264.PROD.OUTLOOK.COM (2603:10a6:100:1b::36) by DM4PR12MB5358.namprd12.prod.outlook.com (2603:10b6:5:39c::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Tue, 5 Dec 2023 10:46:48 +0000 Received: from SA2PEPF000015C8.namprd03.prod.outlook.com (2603:10a6:100:1b:cafe::86) by PR0P264CA0168.outlook.office365.com (2603:10a6:100:1b::36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34 via Frontend Transport; Tue, 5 Dec 2023 10:46:47 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SA2PEPF000015C8.mail.protection.outlook.com (10.167.241.198) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7068.20 via Frontend Transport; Tue, 5 Dec 2023 10:46:47 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 5 Dec 2023 02:46:35 -0800 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 5 Dec 2023 02:46:34 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 5 Dec 2023 02:46:31 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v2 4/8] vdpa/mlx5: Mark vq addrs for modification in hw vq Date: Tue, 5 Dec 2023 12:46:05 +0200 Message-ID: <20231205104609.876194-5-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231205104609.876194-1-dtatulea@nvidia.com> References: <20231205104609.876194-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF000015C8:EE_|DM4PR12MB5358:EE_ X-MS-Office365-Filtering-Correlation-Id: 2308df19-1e91-44aa-fea8-08dbf57f7afc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: SIOSg1xefXfBdhNPxegUmam1vABP2JvBQiKBvdrNplFN7FYWGNgM5QCuf7HJWjXnsiEBUuAw983kaW97FWVwDNUVTs6JMUyFea4LxBLWE7gIaqb15a/Uo9/+M8iy6/HRs4xkkoksm0FmsQYPpNk1DUy1v2XnRZlK821qWDdWgYaMsEeRFZ18QsT5KcitKqGQBlKHngvrWeBgmxtHUGFnHxa11vkvHdH6Ch3Y3f3e//54qe6jAcmn72TpUqJm7ttEAqoMjde81uB+7Fii+vczywoiLF6YFHyWG780tx7EZYU7nZd7b5t8RKkyJeddsLmoGhgFn7wQOzPFFJ4ZMZjdg7EUbtv/sUKgA0Y3pG7lAGCLkich8UlIei65M9G2Uy5ox18N0xyrowtGagUfWul7XNJDBLvL3mwlD8xGdnjOCwTx0L56HEJrCGD1sd/p/SFZk1+raDw0SHCBcphMX9Pji7UBZtyXvLkiuHsGNlURYkbDDVOJ9T2KRtD1Q5uzKCtjQPeXFIIZS6zxzOENCYd/+m1l/QFwOrB40nPwNp9aukbCTKBAffmDND6ItBTShHVZ1T6k0akrmNz6By5OgU3sCiu86nIULzcN0xy50l4b42V2zH1VX0J1H2uA2mAnefhlgwXFc6k5MPIkzb8KCv8W15XAKFRMhqUJaeEB0l97yWYvRjjjWQu7AlhbcE3lBCwHCZux8YMAitKmeRsvU7U1VS0MJpBOYEYHbnjo/Y+vQNccupHsj69XECqUOAFlr2pN X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(136003)(346002)(376002)(396003)(39860400002)(230922051799003)(451199024)(82310400011)(186009)(1800799012)(64100799003)(40470700004)(46966006)(36840700001)(40460700003)(54906003)(316002)(86362001)(478600001)(8936002)(8676002)(6636002)(4326008)(70586007)(110136005)(70206006)(41300700001)(36756003)(5660300002)(2906002)(36860700001)(356005)(7636003)(47076005)(2616005)(26005)(1076003)(6666004)(82740400003)(83380400001)(336012)(426003)(40480700001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Dec 2023 10:46:47.4703 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2308df19-1e91-44aa-fea8-08dbf57f7afc X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF000015C8.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5358 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Tue, 05 Dec 2023 02:47:22 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784438584574306047 X-GMAIL-MSGID: 1784438584574306047 Addresses get set by .set_vq_address. hw vq addresses will be updated on next modify_virtqueue. Signed-off-by: Dragos Tatulea Reviewed-by: Gal Pressman Acked-by: Eugenio Pérez --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 9 +++++++++ include/linux/mlx5/mlx5_ifc_vdpa.h | 1 + 2 files changed, 10 insertions(+) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index f8f088cced50..80e066de0866 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -1209,6 +1209,7 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, bool state_change = false; void *obj_context; void *cmd_hdr; + void *vq_ctx; void *in; int err; @@ -1230,6 +1231,7 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, MLX5_SET(general_obj_in_cmd_hdr, cmd_hdr, uid, ndev->mvdev.res.uid); obj_context = MLX5_ADDR_OF(modify_virtio_net_q_in, in, obj_context); + vq_ctx = MLX5_ADDR_OF(virtio_net_q_object, obj_context, virtio_q_context); if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_STATE) { if (!is_valid_state_change(mvq->fw_state, state, is_resumable(ndev))) { @@ -1241,6 +1243,12 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, state_change = true; } + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS) { + MLX5_SET64(virtio_q, vq_ctx, desc_addr, mvq->desc_addr); + MLX5_SET64(virtio_q, vq_ctx, used_addr, mvq->device_addr); + MLX5_SET64(virtio_q, vq_ctx, available_addr, mvq->driver_addr); + } + MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields); err = mlx5_cmd_exec(ndev->mvdev.mdev, in, inlen, out, sizeof(out)); if (err) @@ -2202,6 +2210,7 @@ static int mlx5_vdpa_set_vq_address(struct vdpa_device *vdev, u16 idx, u64 desc_ mvq->desc_addr = desc_area; mvq->device_addr = device_area; mvq->driver_addr = driver_area; + mvq->modified_fields |= MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS; return 0; } diff --git a/include/linux/mlx5/mlx5_ifc_vdpa.h b/include/linux/mlx5/mlx5_ifc_vdpa.h index b86d51a855f6..9594ac405740 100644 --- a/include/linux/mlx5/mlx5_ifc_vdpa.h +++ b/include/linux/mlx5/mlx5_ifc_vdpa.h @@ -145,6 +145,7 @@ enum { MLX5_VIRTQ_MODIFY_MASK_STATE = (u64)1 << 0, MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_PARAMS = (u64)1 << 3, MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_DUMP_ENABLE = (u64)1 << 4, + MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS = (u64)1 << 6, MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY = (u64)1 << 14, }; From patchwork Tue Dec 5 10:46:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 173947 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp3342331vqy; Tue, 5 Dec 2023 02:47:18 -0800 (PST) X-Google-Smtp-Source: AGHT+IFAAN/tpQT7PcJDQTVMMU38jF+MDYZuoQypQCP2LSet0E2IQXUN0Lpoc02AtW3v4UDEMZ4w X-Received: by 2002:a17:90a:16:b0:286:9464:1bb7 with SMTP id 22-20020a17090a001600b0028694641bb7mr1084088pja.27.1701773237889; Tue, 05 Dec 2023 02:47:17 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1701773237; cv=pass; d=google.com; s=arc-20160816; b=QcBwnWPRaf2y+U4LN4AhcLGYH3AlKBP0svxCO1FJu90FMJ20TYVv9A5LI0Qy8sWGgC GWOnZ5ihUBPS5YnZGCEc+OEa2bDIhy49y6NliHJ0+vMcxLi7lz8v23CXn3Hi02vAyl4O 7cmY3p/YK2+8PedPA1GLkiQ4QvFUKGalzma9MOV0cc3vGz7YBAH03wE1b0rhZtZQFYqW Nvp1W+syC/VHhFfihqbwNRlXqNuZTyyDPkNKU/nVLqs/q+a+pZUCtyx7cgxBpg3CdAKv Xt4wFNAHpnHU0Ni0pUwwyaK5DynQvj9CMC2IcADhjJgd5bk6nQbShHLhN+x9nCtTB4DD 7HiA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Kv7GtFG/dFKlyxKDXcQOoXI8ps+BEnx9aFSvOOlvkuo=; fh=IC8CtaUsCHH+VLkdER9IbUix5yCQzPbBUzKtIdyo0ag=; b=WYVLYRFMwZQ8DEIIqVbOdb+XuvtRekVIBhoEf4JiZ6Cts137JNe/bhhH+OZQfdg5Ek /8bGzC8Hsm0muWDKxLNhv0sdi1xlgRPSJvG6KoaY/B4WRbQuKzPt5/NKQJ+FWo2PWMao VJxopsVr0X/TEXLJhNiIAUdwEEhT15cIx7QIdOgLpDtd3GuRE3tean/yqDSUvqeIJAR7 Qz973HOb+S22aKAV2VBNJ6MSwaZHnhjGUMtZECS9KF7wxF4E2rFWw2gDzgA/YBf/RZLT izq7woxDDkmGesBQqF5b32U9Svrx75BDQzQ87h4IbBDVvF76D9T+oWZIo/S976s4t9Ac z32g== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=KcG8UknJ; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id bo13-20020a17090b090d00b00286a9514be8si3505761pjb.81.2023.12.05.02.47.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Dec 2023 02:47:17 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=KcG8UknJ; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id B37FC80BBE8A; Tue, 5 Dec 2023 02:47:16 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376493AbjLEKrD (ORCPT + 99 others); Tue, 5 Dec 2023 05:47:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376570AbjLEKq4 (ORCPT ); Tue, 5 Dec 2023 05:46:56 -0500 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1nam02on2063.outbound.protection.outlook.com [40.107.96.63]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B80A2199; Tue, 5 Dec 2023 02:47:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=J9cue5zzItLrXSH7xylix6m+bsBbVAoIJ8qV3t7etr2ict4qpYnZsfZwQ8Suc9vSfGpdySWVRc8YuTSXttMngI0/zzeceN4BuuNiFniTdF1F0LbbCXB/E2l86jHsKE6YK+SSHZcBggSMc+YN1ddl4EVNoBsCaOVoLNPBLcB4uJM8RIqpvG8HyPAXvJWTN3zVPnOt+BLPAiDQlrE6+WNNiQaC82BqOmRs/7MN75DqADJb60uLJvJJHaaci1hftmRTcYYrmdP/kXu7EmXP3ImC328ukXMZ4bZ3DAmCUgAXw7BxEJHdDamz0x5tCwFEOtkv6HBYXFeNZD1hpwyNblAbnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Kv7GtFG/dFKlyxKDXcQOoXI8ps+BEnx9aFSvOOlvkuo=; b=Eu+p6zcLzcqAqzJQgLWSRt6+cTXm1Z++YIO45zcXLpl+QbpvzBJcmhy155qlfVrq1R2HCYeVAc1brM3B0ZYRom5jQ0vQzQRSsjpt0jYpwF4ZKtPTpRrq4VgWJbrUq/MDqFVRmPYJVNVOKM8kOAGUOPiY/vOdIPORJ8HE331/Wl46rgsraEx1yv4EXaH33sQXpuPpGzz5LzL/r1NPoVV4sacY0LrVUAww0KkY0y1cSgb8jKG9UCMTZ69Rk7ddOTyfNaumRHVXFYp8ENoPqy4zKIsy6Ewm3hjVk8RJuWIxwothTbiLgK5iXdNGDbA/23pN3RqRbnQoxNXmshOKA4QLEA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Kv7GtFG/dFKlyxKDXcQOoXI8ps+BEnx9aFSvOOlvkuo=; b=KcG8UknJQtaOOELADtJzXNnHmOq+DIL+Bgb39byiUxUzDlQ07c88u/WZnlwPTHUlD/WMezpafj3Ud0jnyoHuyVip2g+MDkGnsikW5WtLNLfyAgMdnDVwWeArJK2TE/O8Mo+UZjHnk7cdLHqXa+v0wQCU3Hq9XM5KgpSWHFNuNe2X4WJyyXOZbWcNrrG8cUwY58lLGDP3+FeejaChyCiVPtft/yf328oF5RXoARZmb0nRR8oCZNqYcH/J88xE1/GXLl+e1jOkYaEEtwXpq5/DKXWp0SCjR46H7/XsDz/Nh7yxX/1YozifVlcE5A2PLwdCf/iP0WjSVfUMD1gD0iwimg== Received: from PR0P264CA0156.FRAP264.PROD.OUTLOOK.COM (2603:10a6:100:1b::24) by LV3PR12MB9439.namprd12.prod.outlook.com (2603:10b6:408:20e::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Tue, 5 Dec 2023 10:46:57 +0000 Received: from SA2PEPF000015C8.namprd03.prod.outlook.com (2603:10a6:100:1b:cafe::41) by PR0P264CA0156.outlook.office365.com (2603:10a6:100:1b::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34 via Frontend Transport; Tue, 5 Dec 2023 10:46:56 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SA2PEPF000015C8.mail.protection.outlook.com (10.167.241.198) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7068.20 via Frontend Transport; Tue, 5 Dec 2023 10:46:56 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 5 Dec 2023 02:46:38 -0800 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 5 Dec 2023 02:46:38 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 5 Dec 2023 02:46:35 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v2 5/8] vdpa/mlx5: Mark vq state for modification in hw vq Date: Tue, 5 Dec 2023 12:46:06 +0200 Message-ID: <20231205104609.876194-6-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231205104609.876194-1-dtatulea@nvidia.com> References: <20231205104609.876194-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF000015C8:EE_|LV3PR12MB9439:EE_ X-MS-Office365-Filtering-Correlation-Id: a87d189b-ced2-480c-126a-08dbf57f8021 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4rQ3UbeYE2+FYjqlHb0uA9ixd1AHNJ6JWT3PkSn0YFOEzQcHJbS+FoEa01P16BF6Ptx2tXQomyU3vu1LXududk09KjrdSDNKBsXhfIVsGWCiFQ0q/k8TgmOzCbNZIrw5UDMEkHgfoGObG0ycNPvzqGsdamT8725dfAFnl/w65l/zfXuB6Sn2B935804AYfFwxzsKy81jG/0BCmqLxCJ4BiwrNS11qjkxWRmgBcsXvT2d7tKA0/ZX94JALKxA6Mhh8VJfVs78QxKsl7FhJ/C5mzCYUN2YbHpxm5bT/s6VPa8WHkY9CpimneUqWCQggF5oFkkP+wTelwKBYL+euaMmF7hLfMsJSEucHlwKO92vTUONmU5tcVXvO2hp37nldeaXE9lW0ykL+bsB7L/R62SeQlLQn1MPWpFsoNhqqWK+cL4vARbbhIFYkADJphIrHLb9NTEXwscl0xrFYZrE8HCorYowgRop5gjarptz6eg7odjUcZjvPC6hGKd2CxAWw95xK/OZA+TjW3hx2UvcR6EiDhtCx1i+xeyLeHDaxIz0EJzaBcPCuKUGrkq3KLJqskzPkTupb38uoUl+xs3SQiPIdXhg7PTYLnbXVLhGlD+aSynRc+t1h+3xJedvFm8sgWrDeWrAieZ8rU8J+K7LSUIOIKbQwVwjLJNsgfCrXYz5JhNEyKNSmOCA8oBOdvhZCYx8WYBQWGuH3//ILJus9RDbJ2VnmhomDCNJyicNaU6kzYu/ToyShCXAX894Q13tsJ8F X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(376002)(136003)(39860400002)(346002)(230922051799003)(64100799003)(82310400011)(1800799012)(451199024)(186009)(36840700001)(40470700004)(46966006)(6666004)(36756003)(478600001)(1076003)(26005)(2616005)(70586007)(6636002)(70206006)(110136005)(336012)(54906003)(40480700001)(316002)(426003)(4326008)(8676002)(8936002)(47076005)(36860700001)(5660300002)(2906002)(7636003)(356005)(82740400003)(41300700001)(86362001)(40460700003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Dec 2023 10:46:56.0953 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a87d189b-ced2-480c-126a-08dbf57f8021 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF000015C8.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR12MB9439 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Tue, 05 Dec 2023 02:47:17 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784438574893701720 X-GMAIL-MSGID: 1784438574893701720 .set_vq_state will set the indices and mark the fields to be modified in the hw vq. Signed-off-by: Dragos Tatulea Reviewed-by: Gal Pressman Acked-by: Eugenio Pérez --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 ++++++++ include/linux/mlx5/mlx5_ifc_vdpa.h | 2 ++ 2 files changed, 10 insertions(+) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 80e066de0866..d6c8506cec8f 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -1249,6 +1249,12 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, MLX5_SET64(virtio_q, vq_ctx, available_addr, mvq->driver_addr); } + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_AVAIL_IDX) + MLX5_SET(virtio_net_q_object, obj_context, hw_available_index, mvq->avail_idx); + + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_USED_IDX) + MLX5_SET(virtio_net_q_object, obj_context, hw_used_index, mvq->used_idx); + MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields); err = mlx5_cmd_exec(ndev->mvdev.mdev, in, inlen, out, sizeof(out)); if (err) @@ -2328,6 +2334,8 @@ static int mlx5_vdpa_set_vq_state(struct vdpa_device *vdev, u16 idx, mvq->used_idx = state->split.avail_index; mvq->avail_idx = state->split.avail_index; + mvq->modified_fields |= MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_AVAIL_IDX | + MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_USED_IDX; return 0; } diff --git a/include/linux/mlx5/mlx5_ifc_vdpa.h b/include/linux/mlx5/mlx5_ifc_vdpa.h index 9594ac405740..32e712106e68 100644 --- a/include/linux/mlx5/mlx5_ifc_vdpa.h +++ b/include/linux/mlx5/mlx5_ifc_vdpa.h @@ -146,6 +146,8 @@ enum { MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_PARAMS = (u64)1 << 3, MLX5_VIRTQ_MODIFY_MASK_DIRTY_BITMAP_DUMP_ENABLE = (u64)1 << 4, MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS = (u64)1 << 6, + MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_AVAIL_IDX = (u64)1 << 7, + MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_USED_IDX = (u64)1 << 8, MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY = (u64)1 << 14, }; From patchwork Tue Dec 5 10:46:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 173948 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp3342371vqy; Tue, 5 Dec 2023 02:47:24 -0800 (PST) X-Google-Smtp-Source: AGHT+IFZDvDUxpGyWw1ik/oEQDBjA+IChx1ThMU7jZ851RmeMt20ihYIieAWXoYIsWNPaRbdGwcU X-Received: by 2002:a05:6870:8092:b0:1fb:75a:778c with SMTP id q18-20020a056870809200b001fb075a778cmr6053255oab.61.1701773244077; Tue, 05 Dec 2023 02:47:24 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1701773244; cv=pass; d=google.com; s=arc-20160816; b=oMuI5i8Hjk+CaBD+aEJokuCZrS6r0RL6dqWl9jkd4ip+ni05u6BbOljmjtseBhHyKU dxVMlkRd8RdLxqk5uay7FvE/x+qdrJ+CLggxc0OYkuWVmnKB1lL8jI0CQ+PFuoT/eey8 0+3FL3EgHo+Uy7aHR4U+JnazCr4K5w8/G4HyfsBpBbSTNJcU7j/+O8bppAi+JLDlu5w5 HE2yhVOsYB0qjBb7ngqulOTR3sj2J1eeyrMIO0CafrIhQ0Hw6Ff1qnda4GYcDfgJ6okk 9ArckRWn61leachnx1s3ksdIBPXNSbKokvv9whtYmXy0V09ODPgYg6I2Dn7XwS6s0row KsXQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XxdASW21yJ2XNUrGBMJT6rmW3BFXdPpM3w/CAWePScM=; fh=IC8CtaUsCHH+VLkdER9IbUix5yCQzPbBUzKtIdyo0ag=; b=OVXYS3v/t9IltDpPmWsFO9pNe+Tm3q3EjbuLu1Jb+OF6uJlg0iqwo6ebCHsZVGrcSI I2lFF3dspbKhS8cXc1LV/ryS6q5GuHmofHeUMd/R/ElGYZ0Mx4B9hoNiFMveCgKvHA3n EaSEZ/8QUx4oAeKiUDX3iSTdGSmsXezTeEh3Wn2u+s1SOI6uGh/yyPxZxB4VOoHIE/v5 Wi0oy3J8lQV5c+L8U9WMfSmqBJcTwfEnHTm9oJjJ0Klx4SWolXcxAyJPR+BCQEF1g7Jd opWxQzP++FUlE15+sylRqa5F0aOYkvrR7vuFCoy0IhleFUgy4fioeFAKPrzCl+5wv2m2 QHZg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=elslTy+o; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id h14-20020a63210e000000b00585463c36fbsi5605706pgh.44.2023.12.05.02.47.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Dec 2023 02:47:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=elslTy+o; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 306CB80403AB; Tue, 5 Dec 2023 02:47:18 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346675AbjLEKrH (ORCPT + 99 others); Tue, 5 Dec 2023 05:47:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38914 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376576AbjLEKq4 (ORCPT ); Tue, 5 Dec 2023 05:46:56 -0500 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2053.outbound.protection.outlook.com [40.107.93.53]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ACDD7198; Tue, 5 Dec 2023 02:47:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=feyWx9dbtVq6WJRDqetAzi+iMDblae1M6i3BX38xf7vD3YTi4FQ4Bf+QEShqOn20JMUqp9Jf2YJSngw/RSZdAfYLEVbQxTbs4zV8OqhhBdcvtMPWBOB1NS5ZXV5jh6Zp+lajqwGUPFB5zo7iE05yYPoeOpdzgblusNEvgNF3r3fVzoa6Hna8eW7DCY42j2anDYyIEFMFs5wPY2H1DEDTJ89QhmXJ0D8cPy1WMonXOBvPMPGRnWeW4Ieh4B924E2+U4FLvm0PdiIiMuk2jJGXOBxSCusn8+ZAe//txYlQklXokKON2QFyfFhttE3LPE0Fk+fEiEBKxCM1msk56xDtWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XxdASW21yJ2XNUrGBMJT6rmW3BFXdPpM3w/CAWePScM=; b=U4fJy0TgOqOW2uibam3htnFI//cNYnm9rvOQo5aMZwyUMCnuhB+f7bS22TPIndMPwzgHvMu/eVRgHzcByUrq31aUBYeaGRXwZvIbxBYZJkbV437LrERm6sGyL8Lc8JDarqwRQljNOM1aXdRon4pdC9a+oIh8sN8li6euyKKhc7kTMFU9v/3XMkiUoagPy9Mwqm0XSyCj9h4TQgZOZHqDrN4OEoEcTZedY4HSaV/G/D8UrbqGyU51J0h0dzeb1f2HLXTSxQsU9KgXiHgVyi5kv5/QpTuD1CtmjeSO9PfxAMFvsWUSssHtYn5+jH0Za/UHImQx3IbG/j+u/RrKi8py9Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XxdASW21yJ2XNUrGBMJT6rmW3BFXdPpM3w/CAWePScM=; b=elslTy+otnoyfxRr5Jc7JwawtxNoJaiS0A8zLNmbZoZURmCtcQdq9IomK60meAJ8HLXxPmEsjz7L/ZzHd9YTgoulpHrMO/ghExVJZssGzsZBZzdapYdjxrcyoq6e6suXBCP0RzKK85zdJmf1Ge1bbH43aNg/map3mb/yeDPBgOG2ZZyWPC6bflP6CH8IQppWpJ6oQOzJQs/MzebdkrNsOxNGnpNI2MDJYXWCoIgKIn1Fynremsvok0B9kFIB2hJIKQqeqBucB6G5VUdrU8KfEi7bPubv7sA0Qh+wbPB5QGZ/V1FDy2xFcY6Ol1KtvYSMnxnt/ow3DEoFfdMfQ+UnmA== Received: from SJ0PR03CA0027.namprd03.prod.outlook.com (2603:10b6:a03:33a::32) by PH7PR12MB6858.namprd12.prod.outlook.com (2603:10b6:510:1b4::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Tue, 5 Dec 2023 10:46:56 +0000 Received: from DS3PEPF000099DA.namprd04.prod.outlook.com (2603:10b6:a03:33a:cafe::8e) by SJ0PR03CA0027.outlook.office365.com (2603:10b6:a03:33a::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34 via Frontend Transport; Tue, 5 Dec 2023 10:46:56 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by DS3PEPF000099DA.mail.protection.outlook.com (10.167.17.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7068.20 via Frontend Transport; Tue, 5 Dec 2023 10:46:56 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 5 Dec 2023 02:46:42 -0800 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 5 Dec 2023 02:46:41 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 5 Dec 2023 02:46:38 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v2 6/8] vdpa/mlx5: Use vq suspend/resume during .set_map Date: Tue, 5 Dec 2023 12:46:07 +0200 Message-ID: <20231205104609.876194-7-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231205104609.876194-1-dtatulea@nvidia.com> References: <20231205104609.876194-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF000099DA:EE_|PH7PR12MB6858:EE_ X-MS-Office365-Filtering-Correlation-Id: ee727bf1-056b-4170-dec1-08dbf57f802e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: thBRXEv8w0Xhydt0DYssxXitr9Q266aJdb6WsKJCF44X/WZbWo2g+k9CkNe89U9K7Iz5jvekKsko+W6aHDxIGlV7zRCunQs9Apj4PQGvrZcpKImx1/772wzK3sWPKSsiN7o11gIizvRIbJiKoT+aYMXinbXxBk3CKgMf7asPnKvNtUqyjlu6ERLW8YS2Uv6vMzdxjFMQbDYWPXs6XWv40mqOF97kHs3eHKDx8AuR+0oirbAhdyLT5XBQXkyKPEg7hGDxfkjHQAwfjH/ZCygEq/T5fZiWx9ym8IKDaA1Z2WcMLd7iNDcCy2YPsKXfauFj4MGDvhBsqc9KtxnACB6ZM6TJh7Y2jRui/f6y9qPxQbKJxodYR3GJR9j2F425Okk4wJmxuzRniwabICAgfmMKVY1z++YCeEPphVzzckgbimFG8G7YSud9Nc1MmJ379UDdhQx96BMQx0WkBFxhbWAuFeiZfy4jfgSQDepUb4vRmYLdVvAk0z+vXtcj+2Di68aqgGKvyJFF+ygQyqD53KcdvqHaNovMkh9UipfxvS/OHk05jYIRsERwnCdwziGhHYTsGlkGhaADthL0IDEsht8ob7f8GT5Qoig9ZuQcw5zIjj9zHYRYix+W81SAJOunBGvcsY8CQtVJuaRQ8XX5zUwcWiOckjuHfosoNfO3QMAm4uSvLk6WIuN6Gy3u7qbV8nlwWX7Mpy3ZbnRwQhyk+xtVQ+C2gfPrn0XIpj8zXVZ6njqf6lEetANt8zbI21MR/w6fuANvQJk/WHiV8iA4DmnGgA== X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(136003)(39860400002)(346002)(376002)(230922051799003)(82310400011)(64100799003)(186009)(451199024)(1800799012)(40470700004)(46966006)(36840700001)(26005)(66574015)(1076003)(36860700001)(82740400003)(356005)(7636003)(47076005)(336012)(426003)(40480700001)(36756003)(2616005)(83380400001)(41300700001)(6666004)(40460700003)(15650500001)(2906002)(86362001)(54906003)(70206006)(70586007)(110136005)(5660300002)(6636002)(316002)(478600001)(8676002)(4326008)(8936002)(142923001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Dec 2023 10:46:56.1701 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ee727bf1-056b-4170-dec1-08dbf57f802e X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF000099DA.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6858 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Tue, 05 Dec 2023 02:47:18 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784438581266041181 X-GMAIL-MSGID: 1784438581266041181 Instead of tearing down and setting up vq resources, use vq suspend/resume during .set_map to speed things up a bit. The vq mr is updated with the new mapping while the vqs are suspended. If the device doesn't support resumable vqs, do the old teardown and setup dance. Signed-off-by: Dragos Tatulea Reviewed-by: Gal Pressman Acked-by: Eugenio Pérez Acked-by: Eugenio Pérez --- drivers/vdpa/mlx5/net/mlx5_vnet.c | 46 ++++++++++++++++++++++++------ include/linux/mlx5/mlx5_ifc_vdpa.h | 1 + 2 files changed, 39 insertions(+), 8 deletions(-) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index d6c8506cec8f..6a21223d97a8 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -1206,6 +1206,7 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, { int inlen = MLX5_ST_SZ_BYTES(modify_virtio_net_q_in); u32 out[MLX5_ST_SZ_DW(modify_virtio_net_q_out)] = {}; + struct mlx5_vdpa_dev *mvdev = &ndev->mvdev; bool state_change = false; void *obj_context; void *cmd_hdr; @@ -1255,6 +1256,24 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_USED_IDX) MLX5_SET(virtio_net_q_object, obj_context, hw_used_index, mvq->used_idx); + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY) { + struct mlx5_vdpa_mr *mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]]; + + if (mr) + MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, mr->mkey); + else + mvq->modified_fields &= ~MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY; + } + + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY) { + struct mlx5_vdpa_mr *mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP]]; + + if (mr && MLX5_CAP_DEV_VDPA_EMULATION(mvdev->mdev, desc_group_mkey_supported)) + MLX5_SET(virtio_q, vq_ctx, desc_group_mkey, mr->mkey); + else + mvq->modified_fields &= ~MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY; + } + MLX5_SET64(virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields); err = mlx5_cmd_exec(ndev->mvdev.mdev, in, inlen, out, sizeof(out)); if (err) @@ -2784,24 +2803,35 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev, unsigned int asid) { struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev); + bool teardown = !is_resumable(ndev); int err; suspend_vqs(ndev); - err = save_channels_info(ndev); - if (err) - return err; + if (teardown) { + err = save_channels_info(ndev); + if (err) + return err; - teardown_driver(ndev); + teardown_driver(ndev); + } mlx5_vdpa_update_mr(mvdev, new_mr, asid); + for (int i = 0; i < ndev->cur_num_vqs; i++) + ndev->vqs[i].modified_fields |= MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY | + MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY; + if (!(mvdev->status & VIRTIO_CONFIG_S_DRIVER_OK) || mvdev->suspended) return 0; - restore_channels_info(ndev); - err = setup_driver(mvdev); - if (err) - return err; + if (teardown) { + restore_channels_info(ndev); + err = setup_driver(mvdev); + if (err) + return err; + } + + resume_vqs(ndev); return 0; } diff --git a/include/linux/mlx5/mlx5_ifc_vdpa.h b/include/linux/mlx5/mlx5_ifc_vdpa.h index 32e712106e68..40371c916cf9 100644 --- a/include/linux/mlx5/mlx5_ifc_vdpa.h +++ b/include/linux/mlx5/mlx5_ifc_vdpa.h @@ -148,6 +148,7 @@ enum { MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS = (u64)1 << 6, MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_AVAIL_IDX = (u64)1 << 7, MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_USED_IDX = (u64)1 << 8, + MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY = (u64)1 << 11, MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY = (u64)1 << 14, }; From patchwork Tue Dec 5 10:46:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 173949 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp3342390vqy; Tue, 5 Dec 2023 02:47:26 -0800 (PST) X-Google-Smtp-Source: AGHT+IElrI5IjbJcThq+EDDDBoZjndb8DayF8QkpZ/6ZtBvjg081AKEUoPEOQiBqpMKqPe42wHS0 X-Received: by 2002:a17:90b:224d:b0:286:9cdc:c2d8 with SMTP id hk13-20020a17090b224d00b002869cdcc2d8mr1068697pjb.22.1701773245935; Tue, 05 Dec 2023 02:47:25 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1701773245; cv=pass; d=google.com; s=arc-20160816; b=OfLywvfHT6xQbkPReMXrIp3odCybr0ZIuNP6BMXXGrdnCGdpB1JGs4t7GxNXEbASpt 59rUMH0J64I2MBM0fVwH/lFjBa1pb6DKh52HuJPD1eyN1xcQ6jwkLRNFPbDQzbrpMWIA VLWCWMZYwnj7+8cxa2uvYS8+r1e+TZcU4Nt8oVWE3XuD00emTwKsqMAixIlFbqLBQOHq lTTkM1K7wm5I19ymY8uR47GP+iJI25RsqVxzh4wTWp/PETfydw3LagyKtgme0XF7b+FS a/GG6/MyAHq14DSUzld8ZsqFHv/dj6a6QdXqASA66DIcUbxVyOomfpfDYB8Cus7xQwKm tAeg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=SmnTSuDMlg6UrNUarJnZfl2MUGLgu9oqKhmm0RZDiAY=; fh=IC8CtaUsCHH+VLkdER9IbUix5yCQzPbBUzKtIdyo0ag=; b=xSkJ2vSal7xb5QyQ8Tr+AtJUuMRVhj74R7/2UdRL9arAYPl+2pl7avBq3m1Dfl/4OG 8cjxgw5QufpaIQX2I50FWu4t7Pz75sivzx7Clzh7fO/4it017vjfquiHd1fJAQn2wHGq 9+7hAwcJ1aqcF2ST7koJui3MehdJabJWFx+PCK7PojRnlgNnjYs3R/ydTIqHj+63wgt5 O902DQDs7S5qM9f4s/s+LHCoxh0KXrL2VKDHOPoxL+ThqaWkDcDFmLDyLETXAr1n5AKn OoUW7tKhhbZ74eWd5zFmtslo7lAbEt/1oWRlxnCTa4gwsPKc4GLu/EkgiKJhgurNaqtJ b1bQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=j2fRFkv6; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id m1-20020a170902768100b001c7347e993esi9079917pll.17.2023.12.05.02.47.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Dec 2023 02:47:25 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=j2fRFkv6; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 9DF2180BC53D; Tue, 5 Dec 2023 02:47:24 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376536AbjLEKrK (ORCPT + 99 others); Tue, 5 Dec 2023 05:47:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38930 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346637AbjLEKrD (ORCPT ); Tue, 5 Dec 2023 05:47:03 -0500 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2061.outbound.protection.outlook.com [40.107.94.61]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9D646109; Tue, 5 Dec 2023 02:47:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZAugOt3xDWDHTWQwQgOcf2TYv2BaK1DOxY4NTUBVUtuayrjGKV4l9NVGF4P99KKdwHMQiOwO81khhJZZcnshlbZWlSVFGjR2vkAQ00y/vwo4tf5UxewoX5Stq4Rj+xAw+9Z508c62Pl1Hp3eZAJ2RtLQmxIJZgnGa4c+rY8+p5//ob+fUDDV8OUtiYOfaW7TOINzA9b3qfqH+7yT1jztVaNgXVQjhr/oltVz62LooWdd4ptArbK1whvvAsoIrV/GZ7ZsGH+tpfunqlfa3u4UvNxEQve7lUIgWxhf0erCsNegXGWr2AwRK+xfJYy4mWHSPl9YdmcNZCqyDYgToTGOkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SmnTSuDMlg6UrNUarJnZfl2MUGLgu9oqKhmm0RZDiAY=; b=f+YIq94A5Vwe+fxG7eKEwXXoxqSGUNVmopxxftrlgZiZ1lxS6lkxQ/5JPF9HIIaU1L2FYQBmV/pI2kaTLYLslGwzA4hju96L8h7W/IJr1wzAe8FP1sOaxxtgCrvNkilNpGiT3letyQfMLloJWsbPG0M0OshNe/3S2Wba16lH5FKjnNWJ+llCP7A5MiSe/VD/Oo962ZBKVxQVdorEgHLdl8gCzNlIAxu+BK9FA1uDseZMM0Lb0hzDJ2kx4MTOKKVj4hMAqKt40QGE8aeBUc10cB6Im/LG7CDfAkk9slUB87DAdXN7UB6xUxu1Qj2mtBvnh1/1vXqHN53fQt0ZOsCnDg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SmnTSuDMlg6UrNUarJnZfl2MUGLgu9oqKhmm0RZDiAY=; b=j2fRFkv6N4XZzYW1y4U2EgabHDz0OYrtDEFOviUoSQ04QpURwpM3nEq6g9T4Kgw+JYOPEMQlNfiHf/B7UKE/UoPA387veYXqBlqJrUfm933W7OUQ3eSRquHOhlGDujlUlyY1R6c3x87ghJiEMYXpj4L5ZLQoyIdRP6EhEwIFnnLRtgmgIaVVmgAC//HXPQmfX5uM0kaG23LyarbaK0dwu7xTxIxzrcEcpayIjNxj+ET9vNvEe14LoF+PVlDQf2wG45jrtQT9smfVRuKxxXCi4P9kg6YRWFb1b08Xgtz2wwlHaQjcaX5b61B20E5JM5ZRnBoAPfUDMM6Zpdpd9r1RkQ== Received: from SA1P222CA0192.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:3c4::13) by SA1PR12MB6775.namprd12.prod.outlook.com (2603:10b6:806:25a::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Tue, 5 Dec 2023 10:47:04 +0000 Received: from SA2PEPF000015C9.namprd03.prod.outlook.com (2603:10b6:806:3c4:cafe::ac) by SA1P222CA0192.outlook.office365.com (2603:10b6:806:3c4::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.26 via Frontend Transport; Tue, 5 Dec 2023 10:47:04 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SA2PEPF000015C9.mail.protection.outlook.com (10.167.241.199) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7068.20 via Frontend Transport; Tue, 5 Dec 2023 10:47:04 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 5 Dec 2023 02:46:45 -0800 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 5 Dec 2023 02:46:45 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 5 Dec 2023 02:46:42 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v2 7/8] vdpa/mlx5: Introduce reference counting to mrs Date: Tue, 5 Dec 2023 12:46:08 +0200 Message-ID: <20231205104609.876194-8-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231205104609.876194-1-dtatulea@nvidia.com> References: <20231205104609.876194-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF000015C9:EE_|SA1PR12MB6775:EE_ X-MS-Office365-Filtering-Correlation-Id: 8ed7a8db-f89d-4855-e4bf-08dbf57f84fd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: JGkvOteMhvcOUByllclwWHAcwEy8meYtnfmvWJgHQci4314Xhb07c7GrYbpGBlEA2289NDHbWwsySTodFAnRWUTrgbXArkkgIMTc4vmZxxS5Oh6cTq+M+4PDoebwQ2r1C3RmZpFSgqZ3E9i83v+XP9EpoUkDvo/BqTLjNo9WrQQv6nBAgKaCgxK6MCUU+roOuMdoH8KiY4w5LtOYezJ7PI8apm+dLv4V8nlOiwluu0PxljjSuc1bwTmZwY3knfjgasqY7uPelWEJNEvHHkvVjMoEa4ki0X471GMrJo/gq2rO7tQq8P+fKPPdGsL6C7jTmt7LJpEg1gIBSVpP5MnHLQAjQKjnom9CAu3vjkmrgkyyJWH1GOWc9czorZUKKngB1Bq8Wa5Wlxc7+KIHHDUy2LtNQltBWHtZxAUqVIVumpJSX1A89S7ZG0eb+g2am/qmcGCPdVIskQbo1PFocQUb1GhG79lj5KgsCw/Lg215m7APiLgmM0idpXP4Asj2hirNK5DJb9ln3DriCw69SZhB+GrOmuhnK3sFE15XJSYoAFDN3ez5tlz9WV9qkvtWCiKOkjgFNVa16cB1wBSbVh6+9a/NuZ6CFTswRvrsYeyb46yKU4difsyPk2XqIhE/hSp3p+YNEfJq/Qxqehqp92+h1kxmGTUwJDRi/9uObTYcH1aqrbS03FDG+wnAsQNiYhq5LYQZh5uaxkGmL8dK94svO0qqoMeCj5VUlxy2zq6klcbks/2MT2iJi+pRddqbIMFM X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(346002)(136003)(39860400002)(376002)(396003)(230922051799003)(1800799012)(64100799003)(451199024)(186009)(82310400011)(40470700004)(46966006)(36840700001)(336012)(426003)(83380400001)(82740400003)(478600001)(6666004)(1076003)(26005)(40480700001)(2616005)(110136005)(316002)(4326008)(70586007)(8936002)(6636002)(54906003)(8676002)(70206006)(36860700001)(356005)(47076005)(7636003)(5660300002)(40460700003)(2906002)(41300700001)(36756003)(86362001);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Dec 2023 10:47:04.2534 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8ed7a8db-f89d-4855-e4bf-08dbf57f84fd X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF000015C9.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6775 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Tue, 05 Dec 2023 02:47:25 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784438583015220991 X-GMAIL-MSGID: 1784438583015220991 Deleting the old mr during mr update (.set_map) and then modifying the vqs with the new mr is not a good flow for firmware. The firmware expects that mkeys are deleted after there are no more vqs referencing them. Introduce reference counting for mrs to fix this. It is the only way to make sure that mkeys are not in use by vqs. An mr reference is taken when the mr is associated to the mr asid table and when the mr is linked to the vq on create/modify. The reference is released when the mkey is unlinked from the vq (trough modify/destroy) and from the mr asid table. To make things consistent, get rid of mlx5_vdpa_destroy_mr and use get/put semantics everywhere. Signed-off-by: Dragos Tatulea Reviewed-by: Gal Pressman Acked-by: Eugenio Pérez --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 8 +++-- drivers/vdpa/mlx5/core/mr.c | 50 ++++++++++++++++++++---------- drivers/vdpa/mlx5/net/mlx5_vnet.c | 45 ++++++++++++++++++++++----- 3 files changed, 78 insertions(+), 25 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 84547d998bcf..1a0d27b6e09a 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -35,6 +35,8 @@ struct mlx5_vdpa_mr { struct vhost_iotlb *iotlb; bool user_mr; + + refcount_t refcount; }; struct mlx5_vdpa_resources { @@ -118,8 +120,10 @@ int mlx5_vdpa_destroy_mkey(struct mlx5_vdpa_dev *mvdev, u32 mkey); struct mlx5_vdpa_mr *mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb); void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev); -void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, - struct mlx5_vdpa_mr *mr); +void mlx5_vdpa_get_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr); +void mlx5_vdpa_put_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr); void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr, unsigned int asid); diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 2197c46e563a..c7dc8914354a 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -498,32 +498,52 @@ static void destroy_user_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_mr *mr) { + if (WARN_ON(!mr)) + return; + if (mr->user_mr) destroy_user_mr(mvdev, mr); else destroy_dma_mr(mvdev, mr); vhost_iotlb_free(mr->iotlb); + + kfree(mr); } -void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, - struct mlx5_vdpa_mr *mr) +static void _mlx5_vdpa_put_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr) { if (!mr) return; + if (refcount_dec_and_test(&mr->refcount)) + _mlx5_vdpa_destroy_mr(mvdev, mr); +} + +void mlx5_vdpa_put_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr) +{ mutex_lock(&mvdev->mr_mtx); + _mlx5_vdpa_put_mr(mvdev, mr); + mutex_unlock(&mvdev->mr_mtx); +} - _mlx5_vdpa_destroy_mr(mvdev, mr); +static void _mlx5_vdpa_get_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr) +{ + if (!mr) + return; - for (int i = 0; i < MLX5_VDPA_NUM_AS; i++) { - if (mvdev->mr[i] == mr) - mvdev->mr[i] = NULL; - } + refcount_inc(&mr->refcount); +} +void mlx5_vdpa_get_mr(struct mlx5_vdpa_dev *mvdev, + struct mlx5_vdpa_mr *mr) +{ + mutex_lock(&mvdev->mr_mtx); + _mlx5_vdpa_get_mr(mvdev, mr); mutex_unlock(&mvdev->mr_mtx); - - kfree(mr); } void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, @@ -534,20 +554,16 @@ void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, mutex_lock(&mvdev->mr_mtx); + _mlx5_vdpa_put_mr(mvdev, old_mr); mvdev->mr[asid] = new_mr; - if (old_mr) { - _mlx5_vdpa_destroy_mr(mvdev, old_mr); - kfree(old_mr); - } mutex_unlock(&mvdev->mr_mtx); - } void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) { for (int i = 0; i < MLX5_VDPA_NUM_AS; i++) - mlx5_vdpa_destroy_mr(mvdev, mvdev->mr[i]); + mlx5_vdpa_update_mr(mvdev, NULL, i); prune_iotlb(mvdev->cvq.iotlb); } @@ -607,6 +623,8 @@ struct mlx5_vdpa_mr *mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, if (err) goto out_err; + refcount_set(&mr->refcount, 1); + return mr; out_err: @@ -651,7 +669,7 @@ int mlx5_vdpa_reset_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) if (asid >= MLX5_VDPA_NUM_AS) return -EINVAL; - mlx5_vdpa_destroy_mr(mvdev, mvdev->mr[asid]); + mlx5_vdpa_update_mr(mvdev, NULL, asid); if (asid == 0 && MLX5_CAP_GEN(mvdev->mdev, umem_uid_0)) { if (mlx5_vdpa_create_dma_mr(mvdev)) diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 6a21223d97a8..133cbb66dcfe 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -123,6 +123,9 @@ struct mlx5_vdpa_virtqueue { u64 modified_fields; + struct mlx5_vdpa_mr *vq_mr; + struct mlx5_vdpa_mr *desc_mr; + struct msi_map map; /* keep last in the struct */ @@ -946,6 +949,14 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque kfree(in); mvq->virtq_id = MLX5_GET(general_obj_out_cmd_hdr, out, obj_id); + mlx5_vdpa_get_mr(mvdev, vq_mr); + mvq->vq_mr = vq_mr; + + if (vq_desc_mr && MLX5_CAP_DEV_VDPA_EMULATION(mvdev->mdev, desc_group_mkey_supported)) { + mlx5_vdpa_get_mr(mvdev, vq_desc_mr); + mvq->desc_mr = vq_desc_mr; + } + return 0; err_cmd: @@ -972,6 +983,12 @@ static void destroy_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtq } mvq->fw_state = MLX5_VIRTIO_NET_Q_OBJECT_NONE; umems_destroy(ndev, mvq); + + mlx5_vdpa_put_mr(&ndev->mvdev, mvq->vq_mr); + mvq->vq_mr = NULL; + + mlx5_vdpa_put_mr(&ndev->mvdev, mvq->desc_mr); + mvq->desc_mr = NULL; } static u32 get_rqpn(struct mlx5_vdpa_virtqueue *mvq, bool fw) @@ -1207,6 +1224,8 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, int inlen = MLX5_ST_SZ_BYTES(modify_virtio_net_q_in); u32 out[MLX5_ST_SZ_DW(modify_virtio_net_q_out)] = {}; struct mlx5_vdpa_dev *mvdev = &ndev->mvdev; + struct mlx5_vdpa_mr *desc_mr = NULL; + struct mlx5_vdpa_mr *vq_mr = NULL; bool state_change = false; void *obj_context; void *cmd_hdr; @@ -1257,19 +1276,19 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, MLX5_SET(virtio_net_q_object, obj_context, hw_used_index, mvq->used_idx); if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY) { - struct mlx5_vdpa_mr *mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]]; + vq_mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP]]; - if (mr) - MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, mr->mkey); + if (vq_mr) + MLX5_SET(virtio_q, vq_ctx, virtio_q_mkey, vq_mr->mkey); else mvq->modified_fields &= ~MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY; } if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY) { - struct mlx5_vdpa_mr *mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP]]; + desc_mr = mvdev->mr[mvdev->group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP]]; - if (mr && MLX5_CAP_DEV_VDPA_EMULATION(mvdev->mdev, desc_group_mkey_supported)) - MLX5_SET(virtio_q, vq_ctx, desc_group_mkey, mr->mkey); + if (desc_mr && MLX5_CAP_DEV_VDPA_EMULATION(mvdev->mdev, desc_group_mkey_supported)) + MLX5_SET(virtio_q, vq_ctx, desc_group_mkey, desc_mr->mkey); else mvq->modified_fields &= ~MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY; } @@ -1282,6 +1301,18 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, if (state_change) mvq->fw_state = state; + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY) { + mlx5_vdpa_put_mr(mvdev, mvq->vq_mr); + mlx5_vdpa_get_mr(mvdev, vq_mr); + mvq->vq_mr = vq_mr; + } + + if (mvq->modified_fields & MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY) { + mlx5_vdpa_put_mr(mvdev, mvq->desc_mr); + mlx5_vdpa_get_mr(mvdev, desc_mr); + mvq->desc_mr = desc_mr; + } + mvq->modified_fields = 0; done: @@ -3095,7 +3126,7 @@ static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, return mlx5_vdpa_update_cvq_iotlb(mvdev, iotlb, asid); out_err: - mlx5_vdpa_destroy_mr(mvdev, new_mr); + mlx5_vdpa_put_mr(mvdev, new_mr); return err; } From patchwork Tue Dec 5 10:46:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Dragos Tatulea X-Patchwork-Id: 173951 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp3342406vqy; Tue, 5 Dec 2023 02:47:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IHkFPSrzGh9MJvmapb+9QX3wTPy1344f509NNAyKYImotV56SZ5hLHmxiwXy/pWP34Gv0EF X-Received: by 2002:a05:6358:63a2:b0:16f:eb38:e930 with SMTP id k34-20020a05635863a200b0016feb38e930mr6251341rwh.13.1701773248276; Tue, 05 Dec 2023 02:47:28 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1701773248; cv=pass; d=google.com; s=arc-20160816; b=GQKlg8+wqyKV4kIwLClVYOpmM+ALj7k8MGFEIupv6QO/kOApkAY4wQ7iVTyBP5I+7J rqnpIjcGaLetQ/qNFR0NPKHOv6AXU1I2KSFDvW+5pbgiVr51323+TjDQNo9kvMiCaoel sZD1e9+Jv3uGcADyGLvt090jXbnDmhhoHjCWGZ3XtOrxSi261ICjYVj6keSGpByu/ZUU VrwkZdOOlvKIZEylrgLx2/v8muphbiITvXRnVKvqYHov2flnaL0CD9R1PFtIEd+7ISys 1QCSlpfbcKX00ZN8ttBVwvbsTvul7ijSuA3TVh1ByXXM9aY6tvJyQDJ5gqNkXoNNI4Ab fzHg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=cW138V3/fZ5TxTAkYoTkM9wdibA2aaBfTJwyExACtWI=; fh=IC8CtaUsCHH+VLkdER9IbUix5yCQzPbBUzKtIdyo0ag=; b=HAE+VP/g/urIQbnsAwA1kyxwAfG/rom3lKPf0ZnoGGmUVUHHrpPDiZa5Weml6tOtv+ NtXnzJvZpEOzuaela0zAUZa7AYoXGHjd6oH7dW2mwBj7nFWc8RsMwErk+cisujY8QzI2 EterYu7ioTwQ85ZAi0kSHXDIwzXL39+6evqNZRUYkS+n5BSVPERRjoc8XoGrIsIhs7uW hK2V30Ax3JJDNmLaVGHvl6jLoxRErOXtg+luAMGWP535WMjtMo16uxjamETU0EIsalWs wJm1iklIF9VRqPJ/nMTetbWcDGHets06/xLmKmCSCR/bC1yecd3/XQNDg0ZvoutTF72J +t5g== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=XZYo0Dl1; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id d6-20020a63fd06000000b005c63ff42dbdsi373041pgh.167.2023.12.05.02.47.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Dec 2023 02:47:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=XZYo0Dl1; arc=pass (i=1 spf=pass spfdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id E4FC68040382; Tue, 5 Dec 2023 02:47:24 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346691AbjLEKrP (ORCPT + 99 others); Tue, 5 Dec 2023 05:47:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376526AbjLEKrF (ORCPT ); Tue, 5 Dec 2023 05:47:05 -0500 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2052.outbound.protection.outlook.com [40.107.94.52]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F833B9; Tue, 5 Dec 2023 02:47:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VWJfSXBxhHLPtpCWsZ35T4cNkw4LDqTxrPSpzIk5SmM70V88Vn8+Qq7INVoN/gxUBjKoO8W9VkWEzbl8jVIXM5mPlsdv/4qKiXZsGzVXXWbxWHhY3z443Y1xXmEpW5XiPQ4VBINjiyAxXlLZTDQ7jWzbZzyyvbS+4lbU06F/yIxwqP3+prONYMo0EoEWjMXXviFa452z5MnK8tLdhHBtBTab0nX3ZT1DE2i2cToitBLSvGaW7Zbo+M2rcKYWwSLlI3mojw+uUxcxEZL/0N7IT3ya+wsQXe469fuOzMavFVI08jx86M3C7SJYoBK3eBGbqIFvAXEhQs/kvV2klxw4Lg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cW138V3/fZ5TxTAkYoTkM9wdibA2aaBfTJwyExACtWI=; b=DZxZoSRzu6zaWpoWKZq2kcnsGestbSpCD5qO7x9dY59ev3GaHc/KxQW1/5Lqp5dP9zO+3yyZxUwl3f1G28oz1bXK86nr0hhP//QpD+JlfzvBAOIQz1PQgAiwBzsTqq1RnIEF232yQ5EpR8n+qjq9kIz0vjubg++Nm5OPo+SWfbNGT+mqUgvzswGRzwedMMouzmj0+mHsPkI7ic2cx6dsW7Thji8XkZiVd6eptnKnfKky8O//O6MMYFf2eD9rk4AUKoQwSrGwdnFhiKXxt8bink9vkg+LgNCCpXRlOPYPM5YI6kHwk/i05MRE6RzKN9IUtvjnT5F2Tcm06HKi+HuUUA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cW138V3/fZ5TxTAkYoTkM9wdibA2aaBfTJwyExACtWI=; b=XZYo0Dl1io6DNWXp3MQtJJgRKd1jSpg79+XMJ18iHHOwN40sucVu4Y6SRphC+LG3leOoyRSGW/gdANLPbz0rPXy1NFTS3BR1HVNX8jADM9G2IvNCw5hitbfD/r/IPjgFq6FMxoMNheVd3VWlGmnQ6UWGya00SciB10ZqKmM93P5tpnAum7kr7qQ2mUtE+rooSYR5YGubpZB3ZJI9rVkP0vlgSXPpXLbyq7cK0dPr+O3Dskx481UYW697pM+8/5F9DTuZ9ua15heQpjz8AxRJn6TNI36zQnh604A94uLNm5w4pwj+MdSI1sdJICL1iVO4Gy2VEMnfzCEZVio8agkQ9g== Received: from PH0P220CA0013.NAMP220.PROD.OUTLOOK.COM (2603:10b6:510:d3::11) by SJ2PR12MB8184.namprd12.prod.outlook.com (2603:10b6:a03:4f2::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34; Tue, 5 Dec 2023 10:47:06 +0000 Received: from SA2PEPF000015C8.namprd03.prod.outlook.com (2603:10b6:510:d3:cafe::a6) by PH0P220CA0013.outlook.office365.com (2603:10b6:510:d3::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7046.34 via Frontend Transport; Tue, 5 Dec 2023 10:47:06 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by SA2PEPF000015C8.mail.protection.outlook.com (10.167.241.198) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7068.20 via Frontend Transport; Tue, 5 Dec 2023 10:47:06 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 5 Dec 2023 02:46:49 -0800 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Tue, 5 Dec 2023 02:46:49 -0800 Received: from c-237-113-220-225.mtl.labs.mlnx (10.127.8.12) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server id 15.2.986.41 via Frontend Transport; Tue, 5 Dec 2023 02:46:46 -0800 From: Dragos Tatulea To: "Michael S . Tsirkin" , Jason Wang , Eugenio Perez Martin , Si-Wei Liu , Saeed Mahameed , Leon Romanovsky , , Gal Pressman CC: Dragos Tatulea , , , Parav Pandit , Xuan Zhuo Subject: [PATCH vhost v2 8/8] vdpa/mlx5: Add mkey leak detection Date: Tue, 5 Dec 2023 12:46:09 +0200 Message-ID: <20231205104609.876194-9-dtatulea@nvidia.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231205104609.876194-1-dtatulea@nvidia.com> References: <20231205104609.876194-1-dtatulea@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF000015C8:EE_|SJ2PR12MB8184:EE_ X-MS-Office365-Filtering-Correlation-Id: ed47a027-13f1-4325-7edf-08dbf57f8620 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8YP7t7SmWJdgN2feMpXljMTk2SBdWvrHKb0gu+BK0HWrxbX6LN+hwoM08zSlqOxg0vMPwPcjyfNPAvaTHxU1j4Bj9ZYPu8jJwG/JDlAar51TWkySrtIY8mfSIdniF296vl33W0oJGaxIiDYbPc844n7Z/PmTXQ3vL5ernW/gyUiHRMrZJk/bYa//equayAgsLb9+wWJEPUtN+g3MjOz7+xZ5RF0hqbAJ8lCpoyEr++i8fxxh3UaIAT/jp/JbuU7dyjFFzRA+vAILa/TyijExmcRI1iy+/n99gysXtWKHfqoz/JKK/ftelE7f1xgBys6S7G+CoRwYPDaflW47v5DnabRxlt8XtPrejRfqsSinML3girpFdWd6d4Fe36JzfyGOn8+oP2+BJLeSvoo7Ppa2ViufLYtpIT3y7k/ciG9CRWw5qTohyQQWuuvhWd/dSplEsh/BvO8rEBe4hhHi8euKs/T3QMVXDkxYIQLokOICOdhXJJx+1SG0+9/UIa64OdMzUBtIQmFysYq4iNudKJLXWZDsPRNgIyVqxpI1YFoDJG3XUsiBs/Ad0Gh9UxhP4nIc3ilwXk+I2G1RDGkpHCCuvb8+1fQDgOZA9I+IeC5UVeTxIyjt8oDKtTzr27XtNcR2g5sI7nm8M3uOedAMmo5bvUs1eJE4DWzCEFOQYVMJeAbVAJN0KIFAnlL1wi8U/SeCaEtpVV8Ql26WSCcrVu9j5qVm5XMkX0Vw0xOwBWwsexiEyqKgTcJJDyWwmPRHuh2i3k5pvUGY29noT+sGQpFtas5J6mLd4nwmwQrZ1Ib7dRw= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(346002)(39860400002)(396003)(136003)(376002)(230922051799003)(230173577357003)(230273577357003)(451199024)(186009)(64100799003)(82310400011)(1800799012)(46966006)(40470700004)(36840700001)(1076003)(2616005)(41300700001)(36756003)(86362001)(426003)(40460700003)(83380400001)(26005)(336012)(316002)(70586007)(110136005)(6636002)(70206006)(54906003)(6666004)(4326008)(8676002)(8936002)(478600001)(82740400003)(356005)(36860700001)(2906002)(7636003)(40480700001)(47076005)(5660300002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Dec 2023 10:47:06.1579 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ed47a027-13f1-4325-7edf-08dbf57f8620 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF000015C8.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR12MB8184 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Tue, 05 Dec 2023 02:47:25 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784438585354395843 X-GMAIL-MSGID: 1784438585354395843 Track allocated mrs in a list and show warning when leaks are detected on device free or reset. Signed-off-by: Dragos Tatulea Reviewed-by: Gal Pressman Acked-by: Eugenio Pérez --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 2 ++ drivers/vdpa/mlx5/core/mr.c | 23 +++++++++++++++++++++++ drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 ++ 3 files changed, 27 insertions(+) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index 1a0d27b6e09a..50aac8fe57ef 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -37,6 +37,7 @@ struct mlx5_vdpa_mr { bool user_mr; refcount_t refcount; + struct list_head mr_list; }; struct mlx5_vdpa_resources { @@ -95,6 +96,7 @@ struct mlx5_vdpa_dev { u32 generation; struct mlx5_vdpa_mr *mr[MLX5_VDPA_NUM_AS]; + struct list_head mr_list_head; /* serialize mr access */ struct mutex mr_mtx; struct mlx5_control_vq cvq; diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index c7dc8914354a..4758914ccf86 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -508,6 +508,8 @@ static void _mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, struct mlx5_vdpa_ vhost_iotlb_free(mr->iotlb); + list_del(&mr->mr_list); + kfree(mr); } @@ -560,12 +562,31 @@ void mlx5_vdpa_update_mr(struct mlx5_vdpa_dev *mvdev, mutex_unlock(&mvdev->mr_mtx); } +static void mlx5_vdpa_show_mr_leaks(struct mlx5_vdpa_dev *mvdev) +{ + struct mlx5_vdpa_mr *mr; + + mutex_lock(&mvdev->mr_mtx); + + list_for_each_entry(mr, &mvdev->mr_list_head, mr_list) { + + mlx5_vdpa_warn(mvdev, "mkey still alive after resource delete: " + "mr: %p, mkey: 0x%x, refcount: %u\n", + mr, mr->mkey, refcount_read(&mr->refcount)); + } + + mutex_unlock(&mvdev->mr_mtx); + +} + void mlx5_vdpa_destroy_mr_resources(struct mlx5_vdpa_dev *mvdev) { for (int i = 0; i < MLX5_VDPA_NUM_AS; i++) mlx5_vdpa_update_mr(mvdev, NULL, i); prune_iotlb(mvdev->cvq.iotlb); + + mlx5_vdpa_show_mr_leaks(mvdev); } static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, @@ -592,6 +613,8 @@ static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, if (err) goto err_iotlb; + list_add_tail(&mr->mr_list, &mvdev->mr_list_head); + return 0; err_iotlb: diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 133cbb66dcfe..778821bab7d9 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -3722,6 +3722,8 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name, if (err) goto err_mpfs; + INIT_LIST_HEAD(&mvdev->mr_list_head); + if (MLX5_CAP_GEN(mvdev->mdev, umem_uid_0)) { err = mlx5_vdpa_create_dma_mr(mvdev); if (err)