From patchwork Tue Oct 10 09:02:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 150594 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp57290vqb; Tue, 10 Oct 2023 02:06:27 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEZPyqq/8mKiE2xSE2tVtIkPj7gI9EAhXsIru1tF/MmwzxLE6GxBKv+WeSdB2ttT/UahLSf X-Received: by 2002:a05:6a00:180c:b0:690:d008:8cfc with SMTP id y12-20020a056a00180c00b00690d0088cfcmr19295533pfa.32.1696928786835; Tue, 10 Oct 2023 02:06:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696928786; cv=none; d=google.com; s=arc-20160816; b=fU0Jqu2PlBDgpEdRH0AThFyzVjmqURY66lc4uTPmacICt40JXdk2/0vBxIdtU1TKHZ 2C0WY2S5/1pCTJqnYAFlfksyUjqgOJkDAa+av1KCy3Qg2nWNDuz+e1c4um+X+DM2iVHc 7CxfuRLYemXdKFb4LHNuKGw+VmEDO+4f/FQtC/oIzY110wAyplVLRvUaUtHc6TmlSHoi 69eNYa9GT2zb/e8RnSwjqzMIlJWHBU63KH4Xk3rKFall9TQt1o1Q8ql3iOjJ6yDKs5ZG BzP5io61HYjSOr3uMIOOHys/husMp32kU17SL09Tfgq+0tiRfZ5BAz6+8SNUOdoLoT0J Dr2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=OR1Mg65jkrHjEjNGA3NuJOUmbwtflXaMlM5D2J5Mle8=; fh=FYvCndxPQizgygd8hAQJp2i0SXZazO8uVJMWtxk8XU4=; b=jfBfLttcRnok0FJEXXGAVcwzWB+9TPu/1Hzeq/1fk3escv6Fpmhb2wMNBVQg6cN3oH XiM3Ruesx7Cf6j0AlmJIrcJOPHcFDFYakeLnC9aIPDjGjzAuNgQzxOzCk5Kh1FBxTabL beM3EHf0axwb41c7r4KkLxx2QBdeSWig7vh5VT2eHiTj5TDoG3PFmUSaieYwr0Jw2Ajn /JWNCFOGDdF7L71uLtq45hJIocQqrmnswCAX+TX9hwLEGGGYeiOWVWINoUJYmZcppi1K 8kpCHhviHtxDQnUb57nVoOc4Ff4u0V8RvztmRO3C2E0uIaDk9YtAgW+sloleY8Awh96e 1F5w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=XQV0JY97; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id bs70-20020a632849000000b00565f182839asi11241196pgb.28.2023.10.10.02.06.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Oct 2023 02:06:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=XQV0JY97; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 8C7DC8329A90; Tue, 10 Oct 2023 02:06:22 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229775AbjJJJFs (ORCPT + 20 others); Tue, 10 Oct 2023 05:05:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50530 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229516AbjJJJFp (ORCPT ); Tue, 10 Oct 2023 05:05:45 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 28042BA for ; Tue, 10 Oct 2023 02:05:42 -0700 (PDT) Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 39A7xm4n017321; Tue, 10 Oct 2023 09:05:36 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-03-30; bh=OR1Mg65jkrHjEjNGA3NuJOUmbwtflXaMlM5D2J5Mle8=; b=XQV0JY97HpvsePs8RzztkmT4Rntm8557ItgXjVdFiahdkGvMiaAueu7oH0CdgzBjqzfy XkYGZVUUZ5STP1ERL8xA7L/vUKAeB9llQ64o+AFdpP5wl02QfEx55Mo0m2NW0GAaupo6 V9QVLG/XupAz6ESMzyr0BDDjRqw4y9jnx4mpndZXQJjqFGHJ8y4NpqlZa7jfvEliJunS i7LQlKz39eSHcDtLEGC/G8rE8gyGiJaDDAqBrk7rgSBCEJIEbmz/V+5JOCZAtA+3cDG7 4B1SfvxEnf1ZySgR8aPmt5TAhIglEuHC9fgvbwP+ePzt/YF8GiUd1O0mFTB+nXv+1nTq 7w== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3tjx8cck9a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 10 Oct 2023 09:05:36 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 39A6sxrD021395; Tue, 10 Oct 2023 09:05:34 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3tjwsc5cy2-2; Tue, 10 Oct 2023 09:05:34 +0000 From: Si-Wei Liu To: jasowang@redhat.com, mst@redhat.com, eperezma@redhat.com, xuanzhuo@linux.alibaba.com, dtatulea@nvidia.com Cc: virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/4] vdpa: introduce .reset_map operation callback Date: Tue, 10 Oct 2023 02:02:57 -0700 Message-Id: <1696928580-7520-2-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1696928580-7520-1-git-send-email-si-wei.liu@oracle.com> References: <1696928580-7520-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-10_04,2023-10-09_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 suspectscore=0 malwarescore=0 adultscore=0 spamscore=0 mlxlogscore=999 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2310100067 X-Proofpoint-ORIG-GUID: 2nX3W0JHB2ZaF5l04tF0rfaY7HthgzBa X-Proofpoint-GUID: 2nX3W0JHB2ZaF5l04tF0rfaY7HthgzBa X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Tue, 10 Oct 2023 02:06:22 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779358799285890919 X-GMAIL-MSGID: 1779358799285890919 Device specific IOMMU parent driver who wishes to see mapping to be decoupled from virtio or vdpa device life cycle (device reset) can use it to restore memory mapping in the device IOMMU to the initial or default state. The reset of mapping is done per address space basis. The reason why a separate .reset_map op is introduced is because this allows a simple on-chip IOMMU model without exposing too much device implementation details to the upper vdpa layer. The .dma_map/unmap or .set_map driver API is meant to be used to manipulate the IOTLB mappings, and has been abstracted in a way similar to how a real IOMMU device maps or unmaps pages for certain memory ranges. However, apart from this there also exists other mapping needs, in which case 1:1 passthrough mapping has to be used by other users (read virtio-vdpa). To ease parent/vendor driver implementation and to avoid abusing DMA ops in an unexpacted way, these on-chip IOMMU devices can start with 1:1 passthrough mapping mode initially at the he time of creation. Then the .reset_map op can be used to switch iotlb back to this initial state without having to expose a complex two-dimensional IOMMU device model. Signed-off-by: Si-Wei Liu Acked-by: Jason Wang --- include/linux/vdpa.h | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/include/linux/vdpa.h b/include/linux/vdpa.h index d376309..26ae6ae 100644 --- a/include/linux/vdpa.h +++ b/include/linux/vdpa.h @@ -327,6 +327,15 @@ struct vdpa_map_file { * @iova: iova to be unmapped * @size: size of the area * Returns integer: success (0) or error (< 0) + * @reset_map: Reset device memory mapping to the default + * state (optional) + * Needed for devices that are using device + * specific DMA translation and prefer mapping + * to be decoupled from the virtio life cycle, + * i.e. device .reset op does not reset mapping + * @vdev: vdpa device + * @asid: address space identifier + * Returns integer: success (0) or error (< 0) * @get_vq_dma_dev: Get the dma device for a specific * virtqueue (optional) * @vdev: vdpa device @@ -405,6 +414,7 @@ struct vdpa_config_ops { u64 iova, u64 size, u64 pa, u32 perm, void *opaque); int (*dma_unmap)(struct vdpa_device *vdev, unsigned int asid, u64 iova, u64 size); + int (*reset_map)(struct vdpa_device *vdev, unsigned int asid); int (*set_group_asid)(struct vdpa_device *vdev, unsigned int group, unsigned int asid); struct device *(*get_vq_dma_dev)(struct vdpa_device *vdev, u16 idx); From patchwork Tue Oct 10 09:02:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 150596 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp57747vqb; Tue, 10 Oct 2023 02:07:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE5vNsIL5xwBLEXszl0dWsHcZDJd7ROUJgelMy+Ulc3qMb496dBmVknF0G/kFvSI2Z+o/Yz X-Received: by 2002:a05:6808:1442:b0:3a7:6ff5:c628 with SMTP id x2-20020a056808144200b003a76ff5c628mr24134844oiv.11.1696928852202; Tue, 10 Oct 2023 02:07:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696928852; cv=none; d=google.com; s=arc-20160816; b=btbo7+70E84ABMsiG+5OCH7cUVmLrSG3nQl20S032E+bagZYbMghs3Wr0+5WJZ1SzV qUxm1lH3XEY0UbdNI+Qk3+IWiLkXX8iPF7f5oIkPP00mHkxIoNmwwKNe8xoCb5cRBMDM ayrmlWNMF2R15zYFZglYUiTXeylln1xNEa4ZE1/fYDnmaF2VS2viGCht2TFOVJ3tyE55 bW2dKrE3OqyvwD5NllcIWTMPzkf0wMZzPs2Vk9hxJ9wHQ+z+a0KuAyD9orX+laKopQA8 vmdBURYtRvPjKF6cIVshTTnLqlXDS6Z+dPsDjhYQ0YB8nvQdL5e9fVlsqTwznM+zyzHI KYJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=qu/sS9ZNID3yHvjNcXeEspAM15qF3bafmisnhPGsjhQ=; fh=FYvCndxPQizgygd8hAQJp2i0SXZazO8uVJMWtxk8XU4=; b=AQBQnnflaV4zaznPap1FlX8clzj1q/fZzSL0PMdYNi/To/Z7BEJeYuruQCiGNB9cG7 46IHqpKj03eqdskxKim4lVF4TOduXG8etzQIInEY6PrKnRXuq64LhLfrk+RyBgHzb8np Np+Bgm1vahDoOQb2w6VwGbDVCKfWDB7Ax3Eu2CixVBEfGiPtT+7jr43FtWiWXTxDP6cQ gNXTPOnbBxBzKJRf5/R+ys6Ep0jb8g5oFQ6kdPipC+2I1oZVUrOHWEdFfR8h6NvHKnSU uMKqugZ994LLDf2poXw3Pbkhs0oFO6aPpVlxIjC/7Wxu+R60IHNXmb++kC1+++TFICU1 aj0w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=dqPzgY4T; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id d4-20020a633604000000b00578a44df6c1si11431918pga.640.2023.10.10.02.07.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Oct 2023 02:07:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=dqPzgY4T; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 021998088698; Tue, 10 Oct 2023 02:06:18 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229868AbjJJJGF (ORCPT + 20 others); Tue, 10 Oct 2023 05:06:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229694AbjJJJFt (ORCPT ); Tue, 10 Oct 2023 05:05:49 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E5878A7 for ; Tue, 10 Oct 2023 02:05:47 -0700 (PDT) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 39A7xYjd002212; Tue, 10 Oct 2023 09:05:36 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-03-30; bh=qu/sS9ZNID3yHvjNcXeEspAM15qF3bafmisnhPGsjhQ=; b=dqPzgY4Tp0iqzSl72qjlk/AFSBvvW9sKi4q8sOEjDRyjsZjErLwOKLuLftvGSbdB6Kh1 qjT7JpYgZkX2EpkqiPIqRl/SzaLZFf83X41dph0q7GEQoUIomASpqPsehgEZLh9KYlrE rd7CLZAym29NmevKTzK9T7UOgteEFjDxX8woiXLsDZW/QJ0KuZuedWZ8SbBG3b1F/GZi He7nhIJMAxBR6SpoE5HtTEZq6vn4YYXMshSH5icoa7WpeTgw61vnH9ebHvHTO2mlgHgh xAyJ1fyIXMietd7ScqG58m45Y6CSPpapU7/tvlB0iB56oOOD15DgDSzNCjjo6gs5mVBn 2Q== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3tjyvumg51-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 10 Oct 2023 09:05:36 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 39A6sxrE021395; Tue, 10 Oct 2023 09:05:35 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3tjwsc5cy2-3; Tue, 10 Oct 2023 09:05:35 +0000 From: Si-Wei Liu To: jasowang@redhat.com, mst@redhat.com, eperezma@redhat.com, xuanzhuo@linux.alibaba.com, dtatulea@nvidia.com Cc: virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/4] vhost-vdpa: reset vendor specific mapping to initial state in .release Date: Tue, 10 Oct 2023 02:02:58 -0700 Message-Id: <1696928580-7520-3-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1696928580-7520-1-git-send-email-si-wei.liu@oracle.com> References: <1696928580-7520-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-10_04,2023-10-09_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 suspectscore=0 malwarescore=0 adultscore=0 spamscore=0 mlxlogscore=999 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2310100067 X-Proofpoint-ORIG-GUID: gxDNOQCin2U6REVSseR0sYShSUBxZDDW X-Proofpoint-GUID: gxDNOQCin2U6REVSseR0sYShSUBxZDDW X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Tue, 10 Oct 2023 02:06:18 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779358868290604037 X-GMAIL-MSGID: 1779358868290604037 Devices with on-chip IOMMU or vendor specific IOTLB implementation may need to restore iotlb mapping to the initial or default state using the .reset_map op, as it's desirable for some parent devices to solely manipulate mappings by its own, independent of virtio device state. For instance, device reset does not cause mapping go away on such IOTLB model in need of persistent mapping. Before vhost-vdpa is going away, give them a chance to reset iotlb back to the initial state in vhost_vdpa_cleanup(). Signed-off-by: Si-Wei Liu --- drivers/vhost/vdpa.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index 851535f..a3f8160 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -131,6 +131,15 @@ static struct vhost_vdpa_as *vhost_vdpa_find_alloc_as(struct vhost_vdpa *v, return vhost_vdpa_alloc_as(v, asid); } +static void vhost_vdpa_reset_map(struct vhost_vdpa *v, u32 asid) +{ + struct vdpa_device *vdpa = v->vdpa; + const struct vdpa_config_ops *ops = vdpa->config; + + if (ops->reset_map) + ops->reset_map(vdpa, asid); +} + static int vhost_vdpa_remove_as(struct vhost_vdpa *v, u32 asid) { struct vhost_vdpa_as *as = asid_to_as(v, asid); @@ -140,6 +149,13 @@ static int vhost_vdpa_remove_as(struct vhost_vdpa *v, u32 asid) hlist_del(&as->hash_link); vhost_vdpa_iotlb_unmap(v, &as->iotlb, 0ULL, 0ULL - 1, asid); + /* + * Devices with vendor specific IOMMU may need to restore + * iotlb to the initial or default state which is not done + * through device reset, as the IOTLB mapping manipulation + * could be decoupled from the virtio device life cycle. + */ + vhost_vdpa_reset_map(v, asid); kfree(as); return 0; From patchwork Tue Oct 10 09:02:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 150593 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp57197vqb; Tue, 10 Oct 2023 02:06:15 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEwEyOiqnFia2iC4CguUczpUBkeVMInm7+yTvVbl33oxIGj9nPecH0gVSvjHOuNMknxSXvR X-Received: by 2002:a05:6808:2d6:b0:3a7:3988:87ee with SMTP id a22-20020a05680802d600b003a7398887eemr17722281oid.58.1696928775590; Tue, 10 Oct 2023 02:06:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696928775; cv=none; d=google.com; s=arc-20160816; b=dhKjBHkWXihIYW2kWM2jecJL4g6mxmGPm7NVreGE/pkPUGTptqR12LNkuUOPDN2Ac5 R1r3sZsVqeJJklCaV0fstjfbNVS6JA3rD8tiVV5IcjpxUnZX90+UukZxu3gAf99HN0fC gW/umD84aHbWxHFjr+FIvobcthqKhWTgSoZBfrLdEqT+jr1ylI4DzefRqjprDS8g3Zli rgJVgwdL9Hzy5cT9Bh6tfmKpwIrhaxUWbCV6M6MbYcLrBk7bAgKOhYHq2WYxFH6X5Cel ZKJ/2k/jCWgbJTXutnZwjKQ5mVeO3GUYT4QqFEWhYqpFvm897rgmsMKQjIdfl5+iVWT7 sFCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=OMuWF+ib0rZpZceqxHvEWuT+8aI6hH6IoBKKRKDUxqQ=; fh=FYvCndxPQizgygd8hAQJp2i0SXZazO8uVJMWtxk8XU4=; b=SbaV6T93myDoJQ+bN3a6TZocVJa4jq77GsTA3NWaOxArdYHMXD4VXTzqdNd/hF+C/k AuWO19w8vSKzlfwIJX2czLscxQemjJ7R84fhgaLus4na7YcWZGq1feJl1+WSxNnqc6CN AsXWoppJTsKoGQpzWcBnJZGeAdII3f/3gfNNP8sIG1FHqGdfSvMeX4PZ9PfsDObMBs05 G4SK/3dm8fpDu6JoCI3wpTYbalH1i2AwXqc+ziFJU6IYTXgjMdrh2XWWSVNBo5pIIX74 64I6DqyKP+P7kHrJvfOWW7zvYNdpUviVRKsxrZevKlNELqFyDcUF/2zHXl9/1HmdASn7 SKwQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=cbcrn05D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id f20-20020a63f114000000b00566016fc08csi11614336pgi.83.2023.10.10.02.06.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Oct 2023 02:06:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=cbcrn05D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 119F58023749; Tue, 10 Oct 2023 02:06:10 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229516AbjJJJFw (ORCPT + 20 others); Tue, 10 Oct 2023 05:05:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33318 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229721AbjJJJFq (ORCPT ); Tue, 10 Oct 2023 05:05:46 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 28FBDCA for ; Tue, 10 Oct 2023 02:05:43 -0700 (PDT) Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 39A804bs017830; Tue, 10 Oct 2023 09:05:37 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-03-30; bh=OMuWF+ib0rZpZceqxHvEWuT+8aI6hH6IoBKKRKDUxqQ=; b=cbcrn05DehNm2FC9NaVjr7SBJLN7aIKRHY/OVM+w8R6Xj9siw6c+h5UFe5234+bR8lf6 zrskHKjNc5xMqoz7gsyzsxrdDTOrvkvqsNRkg5X8i7cJElPAdhSW84R7SmBT6STDbt1U sf02MXbqjLj3yss2lnMvEPlbRAkjV2r7ILw9a9vcFu6Ihvyhl+H61h3UMJXBFoJFQu+1 dQGcgIwTMhUoks4iNCVk9h1N4wxvJlrccaCrWnoEw3QRxXX1pcM3XYqirxLS2B5g9gqn g50FUf1ptXULdekmcjooqqm8jRPIBssvXsyG5IepZ8km/qLuNV/2CKewl2JiMetLSpgc WQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3tjx8cck9b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 10 Oct 2023 09:05:37 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 39A6sxrF021395; Tue, 10 Oct 2023 09:05:35 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3tjwsc5cy2-4; Tue, 10 Oct 2023 09:05:35 +0000 From: Si-Wei Liu To: jasowang@redhat.com, mst@redhat.com, eperezma@redhat.com, xuanzhuo@linux.alibaba.com, dtatulea@nvidia.com Cc: virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/4] vhost-vdpa: introduce IOTLB_PERSIST backend feature bit Date: Tue, 10 Oct 2023 02:02:59 -0700 Message-Id: <1696928580-7520-4-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1696928580-7520-1-git-send-email-si-wei.liu@oracle.com> References: <1696928580-7520-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-10_04,2023-10-09_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 suspectscore=0 malwarescore=0 adultscore=0 spamscore=0 mlxlogscore=999 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2310100067 X-Proofpoint-ORIG-GUID: tiCYM9t-hsTeWDq1eosaeRopCiPEuEdj X-Proofpoint-GUID: tiCYM9t-hsTeWDq1eosaeRopCiPEuEdj X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Tue, 10 Oct 2023 02:06:10 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779358787629955945 X-GMAIL-MSGID: 1779358787629955945 Userspace needs this feature flag to distinguish if vhost-vdpa iotlb in the kernel supports persistent IOTLB mapping across device reset. Without it, userspace has no way to tell apart if it's running on an older kernel, which could silently drop all iotlb mapping across vDPA reset. There are 3 cases that backend may claim this feature bit on: - parent device that has to work with platform IOMMU - parent device with on-chip IOMMU that has the expected .reset_map support in driver - parent device with vendor specific IOMMU implementation that explicitly declares the specific backend feature The reason why .reset_map is being one of the pre-condition for persistent iotlb is because without it, vhost-vdpa can't switch back iotlb to the initial state later on, especially for the on-chip IOMMU case which starts with identity mapping at device creation. virtio-vdpa requires on-chip IOMMU to perform 1:1 passthrough translation from PA to IOVA as-is to begin with, and .reset_map is the only means to turn back iotlb to the identity mapping mode after vhost-vdpa is gone. Signed-off-by: Si-Wei Liu --- drivers/vhost/vdpa.c | 15 +++++++++++++++ include/uapi/linux/vhost_types.h | 2 ++ 2 files changed, 17 insertions(+) diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index a3f8160..c92794f 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -413,6 +413,15 @@ static bool vhost_vdpa_has_desc_group(const struct vhost_vdpa *v) return ops->get_vq_desc_group; } +static bool vhost_vdpa_has_persistent_map(const struct vhost_vdpa *v) +{ + struct vdpa_device *vdpa = v->vdpa; + const struct vdpa_config_ops *ops = vdpa->config; + + return (!ops->set_map && !ops->dma_map) || ops->reset_map || + vhost_vdpa_get_backend_features(v) & BIT_ULL(VHOST_BACKEND_F_IOTLB_PERSIST); +} + static long vhost_vdpa_get_features(struct vhost_vdpa *v, u64 __user *featurep) { struct vdpa_device *vdpa = v->vdpa; @@ -725,6 +734,7 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, return -EFAULT; if (features & ~(VHOST_VDPA_BACKEND_FEATURES | BIT_ULL(VHOST_BACKEND_F_DESC_ASID) | + BIT_ULL(VHOST_BACKEND_F_IOTLB_PERSIST) | BIT_ULL(VHOST_BACKEND_F_SUSPEND) | BIT_ULL(VHOST_BACKEND_F_RESUME) | BIT_ULL(VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK))) @@ -741,6 +751,9 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, if ((features & BIT_ULL(VHOST_BACKEND_F_DESC_ASID)) && !vhost_vdpa_has_desc_group(v)) return -EOPNOTSUPP; + if ((features & BIT_ULL(VHOST_BACKEND_F_IOTLB_PERSIST)) && + !vhost_vdpa_has_persistent_map(v)) + return -EOPNOTSUPP; vhost_set_backend_features(&v->vdev, features); return 0; } @@ -796,6 +809,8 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep, features |= BIT_ULL(VHOST_BACKEND_F_RESUME); if (vhost_vdpa_has_desc_group(v)) features |= BIT_ULL(VHOST_BACKEND_F_DESC_ASID); + if (vhost_vdpa_has_persistent_map(v)) + features |= BIT_ULL(VHOST_BACKEND_F_IOTLB_PERSIST); features |= vhost_vdpa_get_backend_features(v); if (copy_to_user(featurep, &features, sizeof(features))) r = -EFAULT; diff --git a/include/uapi/linux/vhost_types.h b/include/uapi/linux/vhost_types.h index 18ad6ae..d765690 100644 --- a/include/uapi/linux/vhost_types.h +++ b/include/uapi/linux/vhost_types.h @@ -190,5 +190,7 @@ struct vhost_vdpa_iova_range { * buffers may reside. Requires VHOST_BACKEND_F_IOTLB_ASID. */ #define VHOST_BACKEND_F_DESC_ASID 0x7 +/* IOTLB don't flush memory mapping across device reset */ +#define VHOST_BACKEND_F_IOTLB_PERSIST 0x8 #endif From patchwork Tue Oct 10 09:03:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 150595 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp57653vqb; Tue, 10 Oct 2023 02:07:21 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHH5KE1d1QK6pkc9QE4Utc7CxXvvbRfnCitBsY2OQBioQAh+kq2LWwD3zWqOs9yiGSbYYuV X-Received: by 2002:a17:90b:4b4f:b0:279:856:b036 with SMTP id mi15-20020a17090b4b4f00b002790856b036mr14429682pjb.6.1696928841429; Tue, 10 Oct 2023 02:07:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1696928841; cv=none; d=google.com; s=arc-20160816; b=i8ftITwbIHULi97WQHaveIYhySqe6cDT1jZbrff1D5aIM+rsJ+bAJkv4aKhxj3Mqen fQDb8tRh9upxjIPkdIailnMLK5FEp+gK1oTKcd6Hsf8l4HDpXdZBbL4YP/BMvUUvZuIh pEmnRROCY8oNYWtJ3maW64enMvnlRQXjvgD++TwfLyCOEJ2OwFPg7uGCpIynhQvKPcfm 0ktNAPgu/3expw32kQ6U4UebvOdk4xWdpEkwfcbVEnR6s1FFi3pmuCCJUxy5AdAyr7bJ HlDj4Bt9nlg7dvhCObxJAdo9+r8LQfcQwaXHTmWFH3UJ+HeSnMCAxBXYBvx0+SRKp4Xb Mn8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=+ssyG8n1VyDRfdFSLjL3zxxCG5zJZGN6cRxBgPvs3Cs=; fh=FYvCndxPQizgygd8hAQJp2i0SXZazO8uVJMWtxk8XU4=; b=rS1Ed5Ozc/Txg/oM2NcDmcaWhnT7o5wLn99eFrGryTfjPHLSfedLAlaaJwQRgEThV+ 7NzQSXosf8d48gBC651uyka5D3N5CbzEtKnC2bJ7lY0vxUAPRgwtjpAJMo5C5IQWWcWM HqZs2tVa4JOjV56dZAZLo/M8Ab46aTnszIk8KBM2gobvYdJBMGkTLYUrKBloCWxbdK0q 87+OO/3qPCH5oaZuRUZBMoEpri2/7F7ga2e5N2qB6ndImpyIIcv7Q9h65tlQUO23jc3k F/O7MCWPDe+VYfH1jJAx0uOJ44uR3myvP0O2Vdo3NGK+2rcDPTtsO8tDQpXAJ6C6lqtj va8w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=cFWVJECg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id kk12-20020a17090b4a0c00b00274e5b20e65si12690590pjb.50.2023.10.10.02.07.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Oct 2023 02:07:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=cFWVJECg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 539F0808D692; Tue, 10 Oct 2023 02:06:13 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229805AbjJJJFy (ORCPT + 20 others); Tue, 10 Oct 2023 05:05:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33340 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229729AbjJJJFr (ORCPT ); Tue, 10 Oct 2023 05:05:47 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 29537A9 for ; Tue, 10 Oct 2023 02:05:43 -0700 (PDT) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 39A7xExN022226; Tue, 10 Oct 2023 09:05:37 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-03-30; bh=+ssyG8n1VyDRfdFSLjL3zxxCG5zJZGN6cRxBgPvs3Cs=; b=cFWVJECgHMx6UtCawOTgxY6aL92qmoIVI0c5OJNyJxzkvjNp6sHiK027WCFa7BvjFy4t uXqmSGxDSDaerxmGTKHHYuc+qvCzIfRCLo/1iQiE8kIerKjxc+h4gBpdLwGI8T3zJl7a QStu/Kh7SsLE/8fIUuxFYiNd/zA/6BQXEMx7+6skNCIm1NxnuJO5hNshDLFrhChXFkom QtCG/tajlr71sDnxDVQpEqLyMnqRzlv6RgyDafFPxCYAnQ6o9j02dHy2xLooL7I9erpV 05QuNUyaQBmV3JvPZMObx8UtSQuORxEi+7NsF/LyPzkchvRyj5yNKkEst/lCAKaph8Il lQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3tjwx24jay-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 10 Oct 2023 09:05:37 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 39A6sxrG021395; Tue, 10 Oct 2023 09:05:36 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3tjwsc5cy2-5; Tue, 10 Oct 2023 09:05:36 +0000 From: Si-Wei Liu To: jasowang@redhat.com, mst@redhat.com, eperezma@redhat.com, xuanzhuo@linux.alibaba.com, dtatulea@nvidia.com Cc: virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org Subject: [PATCH 4/4] vdpa/mlx5: implement .reset_map driver op Date: Tue, 10 Oct 2023 02:03:00 -0700 Message-Id: <1696928580-7520-5-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1696928580-7520-1-git-send-email-si-wei.liu@oracle.com> References: <1696928580-7520-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-10_04,2023-10-09_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 suspectscore=0 malwarescore=0 adultscore=0 spamscore=0 mlxlogscore=992 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2310100067 X-Proofpoint-GUID: tG7hUu3eOdF7C5akQ9-eoLWMKPyF4ijF X-Proofpoint-ORIG-GUID: tG7hUu3eOdF7C5akQ9-eoLWMKPyF4ijF X-Spam-Status: No, score=2.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Tue, 10 Oct 2023 02:06:13 -0700 (PDT) X-Spam-Level: ** X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779358856795105077 X-GMAIL-MSGID: 1779358856795105077 Since commit 6f5312f80183 ("vdpa/mlx5: Add support for running with virtio_vdpa"), mlx5_vdpa starts with preallocate 1:1 DMA MR at device creation time. This 1:1 DMA MR will be implicitly destroyed while the first .set_map call is invoked, in which case callers like vhost-vdpa will start to set up custom mappings. When the .reset callback is invoked, the custom mappings will be cleared and the 1:1 DMA MR will be re-created. In order to reduce excessive memory mapping cost in live migration, it is desirable to decouple the vhost-vdpa IOTLB abstraction from the virtio device life cycle, i.e. mappings can be kept around intact across virtio device reset. Leverage the .reset_map callback, which is meant to destroy the regular MR on the given ASID and recreate the initial DMA mapping. That way, the device .reset op can run free from having to maintain and clean up memory mappings by itself. The cvq mapping also needs to be cleared if is in the given ASID. Co-developed-by: Dragos Tatulea Signed-off-by: Dragos Tatulea Signed-off-by: Si-Wei Liu --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 1 + drivers/vdpa/mlx5/core/mr.c | 17 +++++++++++++++++ drivers/vdpa/mlx5/net/mlx5_vnet.c | 18 +++++++++++++----- 3 files changed, 31 insertions(+), 5 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index db988ce..84547d9 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -127,6 +127,7 @@ int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, unsigned int asid); int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvdev); +int mlx5_vdpa_reset_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid); #define mlx5_vdpa_warn(__dev, format, ...) \ dev_warn((__dev)->mdev->device, "%s:%d:(pid %d) warning: " format, __func__, __LINE__, \ diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 66530e28..2197c46 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -645,3 +645,20 @@ int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvdev) return mlx5_vdpa_update_cvq_iotlb(mvdev, NULL, 0); } + +int mlx5_vdpa_reset_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) +{ + if (asid >= MLX5_VDPA_NUM_AS) + return -EINVAL; + + mlx5_vdpa_destroy_mr(mvdev, mvdev->mr[asid]); + + if (asid == 0 && MLX5_CAP_GEN(mvdev->mdev, umem_uid_0)) { + if (mlx5_vdpa_create_dma_mr(mvdev)) + mlx5_vdpa_warn(mvdev, "create DMA MR failed\n"); + } else { + mlx5_vdpa_update_cvq_iotlb(mvdev, NULL, asid); + } + + return 0; +} diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 6abe023..928e71b 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -2838,7 +2838,6 @@ static int mlx5_vdpa_reset(struct vdpa_device *vdev) unregister_link_notifier(ndev); teardown_driver(ndev); clear_vqs_ready(ndev); - mlx5_vdpa_destroy_mr_resources(&ndev->mvdev); ndev->mvdev.status = 0; ndev->mvdev.suspended = false; ndev->cur_num_vqs = 0; @@ -2849,10 +2848,6 @@ static int mlx5_vdpa_reset(struct vdpa_device *vdev) init_group_to_asid_map(mvdev); ++mvdev->generation; - if (MLX5_CAP_GEN(mvdev->mdev, umem_uid_0)) { - if (mlx5_vdpa_create_dma_mr(mvdev)) - mlx5_vdpa_warn(mvdev, "create MR failed\n"); - } up_write(&ndev->reslock); return 0; @@ -2932,6 +2927,18 @@ static int mlx5_vdpa_set_map(struct vdpa_device *vdev, unsigned int asid, return err; } +static int mlx5_vdpa_reset_map(struct vdpa_device *vdev, unsigned int asid) +{ + struct mlx5_vdpa_dev *mvdev = to_mvdev(vdev); + struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev); + int err; + + down_write(&ndev->reslock); + err = mlx5_vdpa_reset_mr(mvdev, asid); + up_write(&ndev->reslock); + return err; +} + static struct device *mlx5_get_vq_dma_dev(struct vdpa_device *vdev, u16 idx) { struct mlx5_vdpa_dev *mvdev = to_mvdev(vdev); @@ -3199,6 +3206,7 @@ static int mlx5_set_group_asid(struct vdpa_device *vdev, u32 group, .set_config = mlx5_vdpa_set_config, .get_generation = mlx5_vdpa_get_generation, .set_map = mlx5_vdpa_set_map, + .reset_map = mlx5_vdpa_reset_map, .set_group_asid = mlx5_set_group_asid, .get_vq_dma_dev = mlx5_get_vq_dma_dev, .free = mlx5_vdpa_free,