From patchwork Thu Oct 26 07:14:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Si-Wei Liu X-Patchwork-Id: 158422 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp483154vqb; Thu, 26 Oct 2023 00:18:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEZQ5UcMszH9otA61v6UKmG6Twq0maInlMmDTcKYOYkVJbL5XjwJojznV+VxKNnFwPpqS36 X-Received: by 2002:a25:764a:0:b0:d81:85c9:cea9 with SMTP id r71-20020a25764a000000b00d8185c9cea9mr18307203ybc.65.1698304682390; Thu, 26 Oct 2023 00:18:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698304682; cv=none; d=google.com; s=arc-20160816; b=fhUBucsoAw1vkP4aFybfBmxbg5TJWXkS65C/fZ+lOl75CgIk8YIpvBjZFb7pDGHJjg 0kH80zlmZZR76gDZk7WwtQ5iESwKDdTtr7n0GlBM8IqIjX4RRnoXp0q72xHYN+dEYc0I 7U6iget+Q38mJU2mrqx0qahZvvg1s0jsUwr4If/8+eoEuGzjFSTqP2Leyjo9F8Od9K18 SSsPVpV5UHSOUzUYAVbVpKsEwyoSjxK7V+oRsPhWpPBgXNJjgyclsCGyYV7rGvB4aBFd cwc0zFgoJ83D6ndwHmo5U8R0erkk2HewRjSTyJbOYGBMHIkU2aBAk9DHHKuMJ95U+KMm L48w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9BXLcHlC+mpIFJm+VhDktZctOK6MnG2fZvm6HmvyjLg=; fh=ozy2CkLebIOzbpPF40kmeuF6jbAmH8vZa/HD/EKOE2c=; b=VVW80c1CHAZEFkDxRuo2B86IEsOlQdw07W3YIqSD8MqDbLBE0W5nVmLxtYd+5CUMlB H+URqqNy541ChSH6it2HIhHZc5qHKZDH/HUYxng33bj5WZKogAVE+UR2/CxLEAd6f1Uz ML+ylKPAVAMFa0yU27kOa5+514uUBdCgWxiPyyxPsrnYycpPao84uBAC5CY4kNBCYiWP vcwBYexDEoAJzl8ayF4x+DMHca4z7lhxDpBKgxJtb+qeX/mLN3U9Ofb7whn/wVD3AEwV 1favfW05E8WJjN+b2MojKUK+LtVWnffs1ZmIY3gsrVFhRWrIyoUfyniYHYntmUO0zgBC MmKA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=Rs3lqwAo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id h23-20020a25b197000000b00d815ce3c720si13187634ybj.64.2023.10.26.00.18.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Oct 2023 00:18:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=Rs3lqwAo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 06DA8822742D; Thu, 26 Oct 2023 00:17:54 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344325AbjJZHRj (ORCPT + 26 others); Thu, 26 Oct 2023 03:17:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49090 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344159AbjJZHRd (ORCPT ); Thu, 26 Oct 2023 03:17:33 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 15179129 for ; Thu, 26 Oct 2023 00:17:32 -0700 (PDT) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 39Q3FQZP004619; Thu, 26 Oct 2023 07:17:27 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2023-03-30; bh=9BXLcHlC+mpIFJm+VhDktZctOK6MnG2fZvm6HmvyjLg=; b=Rs3lqwAo8Be3B4aEEFM9rMNfoKIV486hCTweqP6nrCkz4e8FgmE/a7lr26SNqfDobjY/ W8YOmplnAw/v4QXV+57QrxcVaWv6+mYyUx+8oB404zKvoTCh3NIdG10QhaR2pQjHTJIy Hof4uaWs1gT19s/TZ7PfRyUo2co+HNr7LKxgW8Iam+PVGlcDbwau5BYSc6uCON64BPF8 SUZfUF3JddMeGbzDt0QYBsoaz5GJZnKwwyJrumsthWZYyfV890sq2DKFXpxNqTzRpS6S jQeVcc00ZGvS1jEqwzLVsfQcbKvNmGWtOiw3f2Ke0bnR1iXUN0xds/CgDc+O0d7+wW8U cQ== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3tv5e3a9uw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 26 Oct 2023 07:17:27 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 39Q6tbAF015200; Thu, 26 Oct 2023 07:17:26 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3tv537t53r-2; Thu, 26 Oct 2023 07:17:26 +0000 From: Si-Wei Liu To: jasowang@redhat.com, mst@redhat.com, eperezma@redhat.com, sgarzare@redhat.com, dtatulea@nvidia.com Cc: virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 1/7] vdpa: introduce .reset_map operation callback Date: Thu, 26 Oct 2023 00:14:34 -0700 Message-Id: <1698304480-18463-2-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1698304480-18463-1-git-send-email-si-wei.liu@oracle.com> References: <1698304480-18463-1-git-send-email-si-wei.liu@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-26_04,2023-10-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 spamscore=0 malwarescore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2310170001 definitions=main-2310260060 X-Proofpoint-GUID: iH73BH5elrVY7zO4KmEbsp8tRo1wmiLL X-Proofpoint-ORIG-GUID: iH73BH5elrVY7zO4KmEbsp8tRo1wmiLL X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Thu, 26 Oct 2023 00:17:54 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780801530560478932 X-GMAIL-MSGID: 1780801530560478932 Some device specific IOMMU parent drivers have long standing bogus behavior that mistakenly clean up the maps during .reset. By definition, this is violation to the on-chip IOMMU ops (i.e. .set_map, or .dma_map & .dma_unmap) in those offending drivers, as the removal of internal maps is completely agnostic to the upper layer, causing inconsistent view between the userspace and the kernel. Some userspace app like QEMU gets around of this brokenness by proactively removing and adding back all the maps around vdpa device reset, but such workaround actually penalize other well-behaved driver setup, where vdpa reset always comes with the associated mapping cost, especially for kernel vDPA devices (use_va=false) that have high cost on pinning. It's imperative to rectify this behavior and remove the problematic code from all those non-compliant parent drivers. The reason why a separate .reset_map op is introduced is because this allows a simple on-chip IOMMU model without exposing too much device implementation detail to the upper vdpa layer. The .dma_map/unmap or .set_map driver API is meant to be used to manipulate the IOTLB mappings, and has been abstracted in a way similar to how a real IOMMU device maps or unmaps pages for certain memory ranges. However, apart from this there also exists other mapping needs, in which case 1:1 passthrough mapping has to be used by other users (read virtio-vdpa). To ease parent/vendor driver implementation and to avoid abusing DMA ops in an unexpacted way, these on-chip IOMMU devices can start with 1:1 passthrough mapping mode initially at the time of creation. Then the .reset_map op can be used to switch iotlb back to this initial state without having to expose a complex two-dimensional IOMMU device model. The .reset_map is not a MUST for every parent that implements the .dma_map or .set_map API, because device may work with DMA ops directly by implement their own to manipulate system memory mappings, so don't have to use .reset_map to achieve a simple IOMMU device model for 1:1 passthrough mapping. Signed-off-by: Si-Wei Liu Acked-by: Eugenio Pérez Acked-by: Jason Wang --- include/linux/vdpa.h | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/include/linux/vdpa.h b/include/linux/vdpa.h index d376309b99cf..26ae6ae1eac3 100644 --- a/include/linux/vdpa.h +++ b/include/linux/vdpa.h @@ -327,6 +327,15 @@ struct vdpa_map_file { * @iova: iova to be unmapped * @size: size of the area * Returns integer: success (0) or error (< 0) + * @reset_map: Reset device memory mapping to the default + * state (optional) + * Needed for devices that are using device + * specific DMA translation and prefer mapping + * to be decoupled from the virtio life cycle, + * i.e. device .reset op does not reset mapping + * @vdev: vdpa device + * @asid: address space identifier + * Returns integer: success (0) or error (< 0) * @get_vq_dma_dev: Get the dma device for a specific * virtqueue (optional) * @vdev: vdpa device @@ -405,6 +414,7 @@ struct vdpa_config_ops { u64 iova, u64 size, u64 pa, u32 perm, void *opaque); int (*dma_unmap)(struct vdpa_device *vdev, unsigned int asid, u64 iova, u64 size); + int (*reset_map)(struct vdpa_device *vdev, unsigned int asid); int (*set_group_asid)(struct vdpa_device *vdev, unsigned int group, unsigned int asid); struct device *(*get_vq_dma_dev)(struct vdpa_device *vdev, u16 idx);