From patchwork Tue Sep 26 09:31:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 145011 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp2151655vqu; Tue, 26 Sep 2023 12:42:23 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHFt0NmGRO60hwVB9Zo9fvOqf0f5noAAS3FJ1parbOQEYyGI+X5eWOUZSnmTLWNtiQr4pNn X-Received: by 2002:a05:6358:919d:b0:134:d467:b751 with SMTP id j29-20020a056358919d00b00134d467b751mr59700rwa.21.1695757343321; Tue, 26 Sep 2023 12:42:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695757343; cv=none; d=google.com; s=arc-20160816; b=mRJ5TH5u2Vk2zd63g+kbdlPKsLUDk6FHoOns7otj8mx4i5PvcL1lQOS1CSgKuYz6u7 0cMrJwuUs90vpWZheUYXEw6QSr5NP0wKsYPZ5RmUMex9e8OJPEbaWv6lYfwJ4A4tSW3U OrmnE7Rgmk0PcXPbgUoIXe+D4F584cXfqLvSJIJ0qYhPcCubDOxSd8JMvVtb5Q84jKsN i10OkIjM3FVGSEtoQQiqnTITv5fEpH1iB6RNmccCnFZLSZaeIcyJiZEfcnevDHBLLbi9 Wewcq/jUPdOExXrq4ETOG4EZ0JYTeqfZzkuZ4M+xCSgPC6j8ZW6PNnwLyPFvveBRg2dV 8oLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=W+44INTLDs1XlSBHON/3VOQtv3YpADCCgF+hWOX1y1I=; fh=Kv+cxc+457xZThbZHBdHDWxamNAAkIiByoci0wEvMsM=; b=hbIpJ8Se8SsBSBD06lIl9dk0FkQqOq89fxPWOiCric6VvMCxK1ok4T+I3Us5Ks/G5h q2Dd+tzvV3fBSKzeZNrsGF9gtxHuAjaRaWyg1lZgbjsbnLQIXcmDtCwFjaUgjMlTeLZE Hkvi71INZdaiOTza9DVtw7DScx30HuiE9hbFLjodxHHdc0Wx7d6orLr/h+aa24GJtGQm VY/TCSYSCSBjb1dnUvtDNzc+pHeHw7hOebsYYtbT5P/ytZR744CzIi79gXde++fCXl6r JPhBpFZ3oMVoqAPexkIBwnhxzN4AkUYGILDG73gf77e9nJN8tkBPw4EOZRVgMGv6fMQl IQLg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=l60kF5QG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id bk13-20020a056a02028d00b00584a4a5be67si2736088pgb.520.2023.09.26.12.42.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Sep 2023 12:42:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=l60kF5QG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 043A08182EE8; Tue, 26 Sep 2023 02:36:13 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234236AbjIZJfZ (ORCPT + 27 others); Tue, 26 Sep 2023 05:35:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50088 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233940AbjIZJfS (ORCPT ); Tue, 26 Sep 2023 05:35:18 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1ECAFBE; Tue, 26 Sep 2023 02:35:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1695720912; x=1727256912; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=pDVskkLMVGFFDFeU0VLYi3kSiq/U4WCUpxivzmmSlTI=; b=l60kF5QGmRTwbY6/FU6Aqy9M67FiY2Bv+wYiSKj/MTfJomHbJ+EbYrw4 fOh4R66mFIAD85NcaMEJ4d638y1620grVUED8GXIIOPH+5EtoInJbsQN/ LRBi48UYTnQzeWP0dwgVLHL79vU3yoJ51tQBZtz+KvJWvTo7raCILn60g T81C3a+gD0EmOmG6bapGbufofHQ6qt3PclRDHC5M2PKFDA+d2krtLiLYl 4h1/bVssU39peP/ZhlfZxRRm+Uf6Z4x597Y2FbR6Z5Lwa4ORmV+OlIL5B mQcb8rJYLktH43ZoFr53Akp2uS1dxz2AloiYat8qFaw0VRza7S5feLzmi w==; X-IronPort-AV: E=McAfee;i="6600,9927,10843"; a="412442884" X-IronPort-AV: E=Sophos;i="6.03,177,1694761200"; d="scan'208";a="412442884" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Sep 2023 02:31:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10843"; a="725373027" X-IronPort-AV: E=Sophos;i="6.03,177,1694761200"; d="scan'208";a="725373027" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga006.jf.intel.com with ESMTP; 26 Sep 2023 02:31:24 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com, robin.murphy@arm.com, baolu.lu@linux.intel.com Cc: joro@8bytes.org, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, zhenzhong.duan@intel.com, joao.m.martins@oracle.com Subject: [RFC 1/3] vfio-iommufd: Support pasid [at|de]tach for physical VFIO devices Date: Tue, 26 Sep 2023 02:31:19 -0700 Message-Id: <20230926093121.18676-2-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230926093121.18676-1-yi.l.liu@intel.com> References: <20230926093121.18676-1-yi.l.liu@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Tue, 26 Sep 2023 02:36:13 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778130452236174619 X-GMAIL-MSGID: 1778130452236174619 From: Kevin Tian This adds pasid_at|de]tach_ioas ops for attaching hwpt to pasid of a device and the helpers for it. For now, only vfio-pci supports pasid attach/detach. Signed-off-by: Kevin Tian Signed-off-by: Yi Liu --- drivers/vfio/iommufd.c | 48 +++++++++++++++++++++++++++++++++++++ drivers/vfio/pci/vfio_pci.c | 2 ++ include/linux/vfio.h | 11 +++++++++ 3 files changed, 61 insertions(+) diff --git a/drivers/vfio/iommufd.c b/drivers/vfio/iommufd.c index 82eba6966fa5..43a702b9b4d3 100644 --- a/drivers/vfio/iommufd.c +++ b/drivers/vfio/iommufd.c @@ -119,6 +119,7 @@ int vfio_iommufd_physical_bind(struct vfio_device *vdev, if (IS_ERR(idev)) return PTR_ERR(idev); vdev->iommufd_device = idev; + xa_init(&vdev->pasid_pts); return 0; } EXPORT_SYMBOL_GPL(vfio_iommufd_physical_bind); @@ -127,6 +128,17 @@ void vfio_iommufd_physical_unbind(struct vfio_device *vdev) { lockdep_assert_held(&vdev->dev_set->lock); + if (!xa_empty(&vdev->pasid_pts)) { + void *entry; + unsigned long index; + + xa_for_each(&vdev->pasid_pts, index, entry) { + xa_erase(&vdev->pasid_pts, index); + iommufd_device_pasid_detach(vdev->iommufd_device, index); + } + xa_destroy(&vdev->pasid_pts); + } + if (vdev->iommufd_attached) { iommufd_device_detach(vdev->iommufd_device); vdev->iommufd_attached = false; @@ -168,6 +180,42 @@ void vfio_iommufd_physical_detach_ioas(struct vfio_device *vdev) } EXPORT_SYMBOL_GPL(vfio_iommufd_physical_detach_ioas); +int vfio_iommufd_physical_pasid_attach_ioas(struct vfio_device *vdev, + u32 pasid, u32 *pt_id) +{ + void *entry; + int rc; + + lockdep_assert_held(&vdev->dev_set->lock); + + if (WARN_ON(!vdev->iommufd_device)) + return -EINVAL; + + entry = xa_load(&vdev->pasid_pts, pasid); + if (xa_is_value(entry)) + rc = iommufd_device_pasid_replace(vdev->iommufd_device, pasid, pt_id); + else + rc = iommufd_device_pasid_attach(vdev->iommufd_device, pasid, pt_id); + if (rc) + return rc; + xa_store(&vdev->pasid_pts, pasid, xa_mk_value(*pt_id), GFP_KERNEL); + return 0; +} +EXPORT_SYMBOL_GPL(vfio_iommufd_physical_pasid_attach_ioas); + +void vfio_iommufd_physical_pasid_detach_ioas(struct vfio_device *vdev, u32 pasid) +{ + lockdep_assert_held(&vdev->dev_set->lock); + + if (WARN_ON(!vdev->iommufd_device) || + !xa_is_value(xa_load(&vdev->pasid_pts, pasid))) + return; + + iommufd_device_pasid_detach(vdev->iommufd_device, pasid); + xa_erase(&vdev->pasid_pts, pasid); +} +EXPORT_SYMBOL_GPL(vfio_iommufd_physical_pasid_detach_ioas); + /* * The emulated standard ops mean that vfio_device is going to use the * "mdev path" and will call vfio_pin_pages()/vfio_dma_rw(). Drivers using this diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c index cb5b7f865d58..e0198851ffd2 100644 --- a/drivers/vfio/pci/vfio_pci.c +++ b/drivers/vfio/pci/vfio_pci.c @@ -142,6 +142,8 @@ static const struct vfio_device_ops vfio_pci_ops = { .unbind_iommufd = vfio_iommufd_physical_unbind, .attach_ioas = vfio_iommufd_physical_attach_ioas, .detach_ioas = vfio_iommufd_physical_detach_ioas, + .pasid_attach_ioas = vfio_iommufd_physical_pasid_attach_ioas, + .pasid_detach_ioas = vfio_iommufd_physical_pasid_detach_ioas, }; static int vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) diff --git a/include/linux/vfio.h b/include/linux/vfio.h index 454e9295970c..7b06d1bc7cb3 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -66,6 +66,7 @@ struct vfio_device { void (*put_kvm)(struct kvm *kvm); #if IS_ENABLED(CONFIG_IOMMUFD) struct iommufd_device *iommufd_device; + struct xarray pasid_pts; u8 iommufd_attached:1; #endif u8 cdev_opened:1; @@ -83,6 +84,8 @@ struct vfio_device { * bound iommufd. Undo in unbind_iommufd if @detach_ioas is not * called. * @detach_ioas: Opposite of attach_ioas + * @pasid_attach_ioas: The pasid variation of attach_ioas + * @pasid_detach_ioas: Opposite of pasid_attach_ioas * @open_device: Called when the first file descriptor is opened for this device * @close_device: Opposite of open_device * @read: Perform read(2) on device file descriptor @@ -107,6 +110,8 @@ struct vfio_device_ops { void (*unbind_iommufd)(struct vfio_device *vdev); int (*attach_ioas)(struct vfio_device *vdev, u32 *pt_id); void (*detach_ioas)(struct vfio_device *vdev); + int (*pasid_attach_ioas)(struct vfio_device *vdev, u32 pasid, u32 *pt_id); + void (*pasid_detach_ioas)(struct vfio_device *vdev, u32 pasid); int (*open_device)(struct vfio_device *vdev); void (*close_device)(struct vfio_device *vdev); ssize_t (*read)(struct vfio_device *vdev, char __user *buf, @@ -131,6 +136,8 @@ int vfio_iommufd_physical_bind(struct vfio_device *vdev, void vfio_iommufd_physical_unbind(struct vfio_device *vdev); int vfio_iommufd_physical_attach_ioas(struct vfio_device *vdev, u32 *pt_id); void vfio_iommufd_physical_detach_ioas(struct vfio_device *vdev); +int vfio_iommufd_physical_pasid_attach_ioas(struct vfio_device *vdev, u32 pasid, u32 *pt_id); +void vfio_iommufd_physical_pasid_detach_ioas(struct vfio_device *vdev, u32 pasid); int vfio_iommufd_emulated_bind(struct vfio_device *vdev, struct iommufd_ctx *ictx, u32 *out_device_id); void vfio_iommufd_emulated_unbind(struct vfio_device *vdev); @@ -158,6 +165,10 @@ vfio_iommufd_get_dev_id(struct vfio_device *vdev, struct iommufd_ctx *ictx) ((int (*)(struct vfio_device *vdev, u32 *pt_id)) NULL) #define vfio_iommufd_physical_detach_ioas \ ((void (*)(struct vfio_device *vdev)) NULL) +#define vfio_iommufd_physical_pasid_attach_ioas \ + ((int (*)(struct vfio_device *vdev, u32 pasid, u32 *pt_id)) NULL) +#define vfio_iommufd_physical_pasid_detach_ioas \ + ((void (*)(struct vfio_device *vdev, u32 pasid)) NULL) #define vfio_iommufd_emulated_bind \ ((int (*)(struct vfio_device *vdev, struct iommufd_ctx *ictx, \ u32 *out_device_id)) NULL) From patchwork Tue Sep 26 09:31:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 144874 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp1885921vqu; Tue, 26 Sep 2023 05:53:03 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFArwYq1CSFok90zFehRG+zaoxLqbic3M/gfexZsH/eEPWUXlmE4IMM+PCuQXEtrlb3LoeF X-Received: by 2002:a05:6a20:7d82:b0:154:4246:d63b with SMTP id v2-20020a056a207d8200b001544246d63bmr8833564pzj.25.1695732783591; Tue, 26 Sep 2023 05:53:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695732783; cv=none; d=google.com; s=arc-20160816; b=BCWzfR2qTZFLQRwc2NyKZkPii/3NpoDxANavBsCuaQziHUbvpzzvZRZA2OUulBStNv APz0Pt36w+fGiqkafN8GgpS60hwZyuCxvMCjm6D+Bg06AGi/dc47iV6w97DPjZPL7JDO OWqsCZWJCue2S8tHH6wD1c+4rOCzaK3lyh8HmBI7O/NQWIibS3UIbCdKgQcIMIQG7M1U sxOv9Ma3YEpcTX8ICK4VYSXljHcBCqWeXESsgf+ZAeDA80H1+X+ad6GCZ4QkzcbCykze oNekSyDWcUMFc8Cf9N2xtfPg8RW+cE5n2aSNWN3i2fQJ8IUfIwgUVOb1bYGAH5iBkOP5 2Y/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=NTm27p+U74OB1eQzUOgYfCU/CzinSNMK0aohUfkivSA=; fh=Kv+cxc+457xZThbZHBdHDWxamNAAkIiByoci0wEvMsM=; b=n+Gin7WigDMrtXxSE3ZYxBfS5FWDGne7g39Hg2zLG0JDSwnUA+738GR5x9cnlI22Y+ lYEdU0PQG8nz7JAQoZdEnZWAssNGCWtu6YJ6H9eN0cQPj2NEXEc8dsQxn2Rv2Mif+FgX 6BRA+eVJZgwgKXjGzvMAGckr7EHyGXrlXw1M4khw5eW6rCbPwcy8rTQBS3WUv0cJob2w H+/yUmsDgQYcUkCZY/pV5DWGbAlq0QyuOQDc5ZSQazsu18QHWSmnN6Fav4dpuLEXGHtb bc/maS4XPQD2Zxi9znl4IBT9pbFkvXZxnPHhMJWoR/PxND6MCZ/f7vDhhItU3Mo6EJFL eeHw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=QdfmeT1C; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id l186-20020a6388c3000000b0056a1ed17deasi12345607pgd.675.2023.09.26.05.53.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Sep 2023 05:53:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=QdfmeT1C; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id AEE0680B19E8; Tue, 26 Sep 2023 02:35:29 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234186AbjIZJfV (ORCPT + 27 others); Tue, 26 Sep 2023 05:35:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232303AbjIZJfS (ORCPT ); Tue, 26 Sep 2023 05:35:18 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B55AAB3; Tue, 26 Sep 2023 02:35:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1695720911; x=1727256911; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ThUJ2dgRLVmBBUHWMxUL3UVe0FEzVRO2lU88vuPRrAY=; b=QdfmeT1CoH1gicu8n8otNvzmX6o1YYQuNX4L6dxMHtmVxjUpm05YpLeP MkSVuEfA4qojEopL3lobHOKAKLSEYAj7P/zO/T2Zc8BPXoaGoK8eQDMyo ZB6Icx8CvWgmlo9YjfqMQxc8Cp3Ha4L5x3Nfq8sxvf7RI/BPojlTWgPin CvT4jCdXAk91rHkWBwodS+YoUXJNLkPpm1HfvXwEr0sqUKEKVf8KYiPTz uMBZ2wQgbkm/NRL4nZ5gwrov53g5D0VhtBszCRSCGgwJc8pxGycxBbj7W un05bW8vryJbjdc0y0Bd3ipsvma8kEEGtlJz8U27pYO8MEi8yoC45Cglc w==; X-IronPort-AV: E=McAfee;i="6600,9927,10843"; a="412442898" X-IronPort-AV: E=Sophos;i="6.03,177,1694761200"; d="scan'208";a="412442898" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Sep 2023 02:31:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10843"; a="725373037" X-IronPort-AV: E=Sophos;i="6.03,177,1694761200"; d="scan'208";a="725373037" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga006.jf.intel.com with ESMTP; 26 Sep 2023 02:31:25 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com, robin.murphy@arm.com, baolu.lu@linux.intel.com Cc: joro@8bytes.org, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, zhenzhong.duan@intel.com, joao.m.martins@oracle.com Subject: [RFC 2/3] vfio: Add VFIO_DEVICE_PASID_[AT|DE]TACH_IOMMUFD_PT Date: Tue, 26 Sep 2023 02:31:20 -0700 Message-Id: <20230926093121.18676-3-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230926093121.18676-1-yi.l.liu@intel.com> References: <20230926093121.18676-1-yi.l.liu@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Tue, 26 Sep 2023 02:35:29 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778104699243790494 X-GMAIL-MSGID: 1778104699243790494 This adds ioctls for the userspace to attach a given pasid of a vfio device to/from an IOAS/HWPT. Signed-off-by: Yi Liu --- drivers/vfio/device_cdev.c | 45 +++++++++++++++++++++++++++++++ drivers/vfio/vfio.h | 4 +++ drivers/vfio/vfio_main.c | 8 ++++++ include/uapi/linux/vfio.h | 55 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 112 insertions(+) diff --git a/drivers/vfio/device_cdev.c b/drivers/vfio/device_cdev.c index e75da0a70d1f..c2ac7ed44537 100644 --- a/drivers/vfio/device_cdev.c +++ b/drivers/vfio/device_cdev.c @@ -210,6 +210,51 @@ int vfio_df_ioctl_detach_pt(struct vfio_device_file *df, return 0; } +int vfio_df_ioctl_pasid_attach_pt(struct vfio_device_file *df, + struct vfio_device_pasid_attach_iommufd_pt __user *arg) +{ + struct vfio_device *device = df->device; + struct vfio_device_pasid_attach_iommufd_pt attach; + unsigned long minsz; + int ret; + + minsz = offsetofend(struct vfio_device_pasid_attach_iommufd_pt, pt_id); + + if (copy_from_user(&attach, arg, minsz)) + return -EFAULT; + + if (attach.argsz < minsz || attach.flags) + return -EINVAL; + + mutex_lock(&device->dev_set->lock); + ret = device->ops->pasid_attach_ioas(device, attach.pasid, &attach.pt_id); + mutex_unlock(&device->dev_set->lock); + + return ret; +} + +int vfio_df_ioctl_pasid_detach_pt(struct vfio_device_file *df, + struct vfio_device_pasid_detach_iommufd_pt __user *arg) +{ + struct vfio_device *device = df->device; + struct vfio_device_pasid_detach_iommufd_pt detach; + unsigned long minsz; + + minsz = offsetofend(struct vfio_device_pasid_detach_iommufd_pt, flags); + + if (copy_from_user(&detach, arg, minsz)) + return -EFAULT; + + if (detach.argsz < minsz || detach.flags) + return -EINVAL; + + mutex_lock(&device->dev_set->lock); + device->ops->pasid_detach_ioas(device, detach.pasid); + mutex_unlock(&device->dev_set->lock); + + return 0; +} + static char *vfio_device_devnode(const struct device *dev, umode_t *mode) { return kasprintf(GFP_KERNEL, "vfio/devices/%s", dev_name(dev)); diff --git a/drivers/vfio/vfio.h b/drivers/vfio/vfio.h index 307e3f29b527..d228cdb6b345 100644 --- a/drivers/vfio/vfio.h +++ b/drivers/vfio/vfio.h @@ -353,6 +353,10 @@ int vfio_df_ioctl_attach_pt(struct vfio_device_file *df, struct vfio_device_attach_iommufd_pt __user *arg); int vfio_df_ioctl_detach_pt(struct vfio_device_file *df, struct vfio_device_detach_iommufd_pt __user *arg); +int vfio_df_ioctl_pasid_attach_pt(struct vfio_device_file *df, + struct vfio_device_pasid_attach_iommufd_pt __user *arg); +int vfio_df_ioctl_pasid_detach_pt(struct vfio_device_file *df, + struct vfio_device_pasid_detach_iommufd_pt __user *arg); #if IS_ENABLED(CONFIG_VFIO_DEVICE_CDEV) void vfio_init_device_cdev(struct vfio_device *device); diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index 40732e8ed4c6..850bbaebdd29 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -1230,6 +1230,14 @@ static long vfio_device_fops_unl_ioctl(struct file *filep, case VFIO_DEVICE_DETACH_IOMMUFD_PT: ret = vfio_df_ioctl_detach_pt(df, uptr); goto out; + + case VFIO_DEVICE_PASID_ATTACH_IOMMUFD_PT: + ret = vfio_df_ioctl_pasid_attach_pt(df, uptr); + goto out; + + case VFIO_DEVICE_PASID_DETACH_IOMMUFD_PT: + ret = vfio_df_ioctl_pasid_detach_pt(df, uptr); + goto out; } } diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 97ab68a175e0..474bb314d135 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -975,6 +975,61 @@ struct vfio_device_detach_iommufd_pt { #define VFIO_DEVICE_DETACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 20) +/* + * VFIO_DEVICE_PASID_ATTACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 21, + * struct vfio_device_pasid_attach_iommufd_pt) + * @argsz: User filled size of this data. + * @flags: Must be 0. + * @pasid: The pasid to be attached. + * @pt_id: Input the target id which can represent an ioas or a hwpt + * allocated via iommufd subsystem. + * Output the input ioas id or the attached hwpt id which could + * be the specified hwpt itself or a hwpt automatically created + * for the specified ioas by kernel during the attachment. + * + * Associate a pasid (of a cdev device) with an address space within the + * bound iommufd. Undo by VFIO_DEVICE_PASID_DETACH_IOMMUFD_PT or device fd + * close. This is only allowed on cdev fds. + * + * If a pasid is currently attached to a valid hw_pagetable (hwpt), without + * doing a VFIO_DEVICE_PASID_DETACH_IOMMUFD_PT, a second + * VFIO_DEVICE_PASID_ATTACH_IOMMUFD_PT ioctl passing in another hwpt id is + * allowed. This action, also known as a hwpt replacement, will replace the + * pasid's currently attached hwpt with a new hwpt corresponding to the given + * @pt_id. + * + * Return: 0 on success, -errno on failure. + */ +struct vfio_device_pasid_attach_iommufd_pt { + __u32 argsz; + __u32 flags; + __u32 pasid; + __u32 pt_id; +}; + +#define VFIO_DEVICE_PASID_ATTACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 21) + +/* + * VFIO_DEVICE_PASID_DETACH_IOMMUFD_PT - _IOW(VFIO_TYPE, VFIO_BASE + 22, + * struct vfio_device_pasid_detach_iommufd_pt) + * @argsz: User filled size of this data. + * @flags: Must be 0. + * @pasid: The pasid to be detached. + * + * Remove the association of a pasid (of a cdev device) and its current + * associated address space. After it, the pasid of the device should be in + * a blocking DMA state. This is only allowed on cdev fds. + * + * Return: 0 on success, -errno on failure. + */ +struct vfio_device_pasid_detach_iommufd_pt { + __u32 argsz; + __u32 flags; + __u32 pasid; +}; + +#define VFIO_DEVICE_PASID_DETACH_IOMMUFD_PT _IO(VFIO_TYPE, VFIO_BASE + 22) + /* * Provide support for setting a PCI VF Token, which is used as a shared * secret between PF and VF drivers. This feature may only be set on a From patchwork Tue Sep 26 09:31:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi Liu X-Patchwork-Id: 144931 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp1973616vqu; Tue, 26 Sep 2023 07:52:23 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGEfnu6Ka8na4/d8532SyuRxkkfoyuu8jOn4SdarXPzKDqKT3iE071RXKWZvmThG9ozOq4W X-Received: by 2002:aa7:8893:0:b0:690:2ecd:a597 with SMTP id z19-20020aa78893000000b006902ecda597mr10287291pfe.21.1695739943271; Tue, 26 Sep 2023 07:52:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695739943; cv=none; d=google.com; s=arc-20160816; b=YstuYZXf1BgrIbk6qBtM1q3HbPrJb0mW3ruBpIsJKsGWMNsY9D8KKun1Dg+Z16LuNX /razBDRCSotxwUpscV2qnlJZhBnuBD8bhmW5sSIzHvK2qnuE0nol4t6+09x1wNC6VLWU gLJLMIDdB9B8J09UWPyfwekPUhupK8SNg2DZhyK1keMuOJYGgRhCk5VxMCNYSOo1EFcr zAlA3DFXgp0TbQBo22bW8/YoYfmuw+CW9opSQgduTtuKCb7lPp23FsdePz5D/iEqldFn TubBo4yryqQt9fCtrKIsPsxtItqtlmF3vTyrWiX82yGJ3rcJoKMEKLFjNiBMy1zUHfUb 8PGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=chqNXPWoxq6NAS2n5n8XL0z9LhMbkxnIDip8oI3wTTE=; fh=Kv+cxc+457xZThbZHBdHDWxamNAAkIiByoci0wEvMsM=; b=hde++UX2vfndsxFpQr++hy+AALY9jBqjwAIHloXB+mFR5cCACWvPeJfp/HwdTKD9+V /2VPI/Ocps/cMdK4tA7FwzVjXqov0f8DELUj6NrqTHHHdLipJfhm0dtaeIGxcRyAcY9t 3X8S+tHt1F5fPsfpp9LIrgGyeYAsjQHZSf2kADADw9CS+lNr0S7J7zwpWm/0AfgVmv/C 6d0lGwFkvgDI4CmzPbRfakaUQCJOVnL0l50GqKTlSoeAVPqh0KvYUKYz6M7M77zKx2LG faENquc0SXaHee4zCzODC1lGOEB9S8LTsDYJWKNFzTVicYfO7ZRrhZNC8N215oJyf6yj 4Qqw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=CsFHfwDj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id a21-20020a63e855000000b00578d7a3a4fdsi12679850pgk.563.2023.09.26.07.52.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Sep 2023 07:52:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=CsFHfwDj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id D26E0801B67D; Tue, 26 Sep 2023 02:36:12 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233940AbjIZJfg (ORCPT + 27 others); Tue, 26 Sep 2023 05:35:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50108 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234114AbjIZJfT (ORCPT ); Tue, 26 Sep 2023 05:35:19 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 035FEF3; Tue, 26 Sep 2023 02:35:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1695720913; x=1727256913; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bZj72hwk6aGVHqUfVsQNiXvoc1GB8ZJ+RY64a2dEVnk=; b=CsFHfwDjUPio3TMeQ2KS9igb5wWbW/08xnYhqcUybO0NggZHh19kwHyS zfDr9YI0ht/JkwU0sUIZkmzL/dz3nLYJkZ4wIc47evQ6z8tA2v1zhUc5t uOfB7Khf9qO5+w0fGR4A/6N+HDM9qkxrjpvJ+1M80GXqZVEZQ4QSNxm+m 30lC/HqCC9+K8kv6MZcJw/TNjs5E2YmweYPrjJQzuzW9y6o4fCzJfPmKz A6T2NGk1UwqBmYDlKdv2LIv1flPwHNfxsHTAMqY9KdR9CIbwb7ELoTalM Bp2FZgH+OggPdYuwpW4kUhQgVMGPOg50lLIjgbwIq+xeadLWGUVZcNihI A==; X-IronPort-AV: E=McAfee;i="6600,9927,10843"; a="412442921" X-IronPort-AV: E=Sophos;i="6.03,177,1694761200"; d="scan'208";a="412442921" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Sep 2023 02:31:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10843"; a="725373049" X-IronPort-AV: E=Sophos;i="6.03,177,1694761200"; d="scan'208";a="725373049" Received: from 984fee00a4c6.jf.intel.com ([10.165.58.231]) by orsmga006.jf.intel.com with ESMTP; 26 Sep 2023 02:31:26 -0700 From: Yi Liu To: alex.williamson@redhat.com, jgg@nvidia.com, kevin.tian@intel.com, robin.murphy@arm.com, baolu.lu@linux.intel.com Cc: joro@8bytes.org, cohuck@redhat.com, eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com, yi.l.liu@intel.com, yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com, shameerali.kolothum.thodi@huawei.com, lulu@redhat.com, suravee.suthikulpanit@amd.com, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, zhenzhong.duan@intel.com, joao.m.martins@oracle.com Subject: [RFC 3/3] vfio/pci: Expose PCIe PASID capability to userspace Date: Tue, 26 Sep 2023 02:31:21 -0700 Message-Id: <20230926093121.18676-4-yi.l.liu@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230926093121.18676-1-yi.l.liu@intel.com> References: <20230926093121.18676-1-yi.l.liu@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Tue, 26 Sep 2023 02:36:12 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778112206673291284 X-GMAIL-MSGID: 1778112206673291284 This exposes PCIe PASID capability to userspace and where to emulate this capability if wants to further expose it to VM. And this only exposes PASID capability for devices which has PCIe PASID extended struture in its configuration space. While for VFs, userspace is still unable to see this capability as SR-IOV spec forbides VF to implement PASID capability extended structure. It is a TODO in future. Related discussion can be found in below links: https://lore.kernel.org/kvm/20200407095801.648b1371@w520.home/ https://lore.kernel.org/kvm/BL1PR11MB5271A60035EF591A5BE8AC878C01A@BL1PR11MB5271.namprd11.prod.outlook.com/ Signed-off-by: Yi Liu --- drivers/vfio/pci/vfio_pci_config.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c index 7e2e62ab0869..dfae5ad5ebc0 100644 --- a/drivers/vfio/pci/vfio_pci_config.c +++ b/drivers/vfio/pci/vfio_pci_config.c @@ -95,7 +95,7 @@ static const u16 pci_ext_cap_length[PCI_EXT_CAP_ID_MAX + 1] = { [PCI_EXT_CAP_ID_LTR] = PCI_EXT_CAP_LTR_SIZEOF, [PCI_EXT_CAP_ID_SECPCI] = 0, /* not yet */ [PCI_EXT_CAP_ID_PMUX] = 0, /* not yet */ - [PCI_EXT_CAP_ID_PASID] = 0, /* not yet */ + [PCI_EXT_CAP_ID_PASID] = PCI_EXT_CAP_PASID_SIZEOF, [PCI_EXT_CAP_ID_DVSEC] = 0xFF, };