mbox series

[RFC,0/7] Add SIOV virtual device support

Message ID 20231009085123.463179-1-yi.l.liu@intel.com
Headers
Series Add SIOV virtual device support |

Message

Yi Liu Oct. 9, 2023, 8:51 a.m. UTC 
  Intel SIOV allows creating virtual devices of which the vRID is
represented by a pasid of a physical device. It is called as SIOV
virtual device in this series. Such devices can be bound to an iommufd
as physical device does and then later be attached to an IOAS/hwpt
using that pasid. Such PASIDs are called as default pasid.

iommufd has already supported pasid attach[1]. So a simple way to
support SIOV virtual device attachment is to let device driver call
the iommufd_device_pasid_attach() and pass in the default pasid for
the virtual device. This should work for now, but it may have problem
if iommufd core wants to differentiate the default pasids with other
kind of pasids (e.g. pasid given by userspace). In the later forwarding
page request to userspace, the default pasids are not supposed to send
to userspace as default pasids are mainly used by the SIOV device driver.

With above reason, this series chooses to have a new API to bind the
default pasid to iommufd, and extends the iommufd_device_attach() to
convert the attachment to be pasid attach with the default pasid. Device
drivers (e.g. VFIO) that support SIOV shall call the below APIs to
interact with iommufd:

  - iommufd_device_bind_pasid(): Bind virtual device (a pasid of a device)
				 to iommufd;
  - iommufd_device_attach(): Attach a SIOV virtual device to IOAS/HWPT;
  - iommufd_device_replace(): Replace IOAS/HWPT of a SIOV virtual device;
  - iommufd_device_detach(): Detach IOAS/HWPT of a SIOV virtual device;
  - iommufd_device_unbind(): Unbind virtual device from iommufd;

For vfio devices, the device drivers that support SIOV should:

  - use below API to register vdev for SIOV virtual device
        vfio_register_pasid_iommu_dev()

  - use below API to bind vdev to iommufd in .bind_iommufd() callback
        iommufd_device_bind_pasid()

  - allocate pasid by itself before calling iommufd_device_bind_pasid()

Complete code can be found at[2]

[1] https://lore.kernel.org/linux-iommu/20230926092651.17041-1-yi.l.liu@intel.com/
[2] https://github.com/yiliu1765/iommufd/tree/iommufd_pasid_siov

Regards,
	Yi Liu

Kevin Tian (5):
  iommufd: Handle unsafe interrupts in a separate function
  iommufd: Introduce iommufd_alloc_device()
  iommufd: Add iommufd_device_bind_pasid()
  iommufd: Support attach/replace for SIOV virtual device {dev, pasid}
  vfio: Add vfio_register_pasid_iommu_dev()

Yi Liu (2):
  iommufd/selftest: Extend IOMMU_TEST_OP_MOCK_DOMAIN to pass in pasid
  iommufd/selftest: Add test coverage for SIOV virtual device

 drivers/iommu/iommufd/device.c                | 163 ++++++++++++++----
 drivers/iommu/iommufd/iommufd_private.h       |   7 +
 drivers/iommu/iommufd/iommufd_test.h          |   2 +
 drivers/iommu/iommufd/selftest.c              |  10 +-
 drivers/vfio/group.c                          |  18 ++
 drivers/vfio/vfio.h                           |   8 +
 drivers/vfio/vfio_main.c                      |  10 ++
 include/linux/iommufd.h                       |   3 +
 include/linux/vfio.h                          |   1 +
 tools/testing/selftests/iommu/iommufd.c       |  75 ++++++--
 .../selftests/iommu/iommufd_fail_nth.c        |  42 ++++-
 tools/testing/selftests/iommu/iommufd_utils.h |  21 ++-
 12 files changed, 296 insertions(+), 64 deletions(-)

Comments

Jason Gunthorpe Oct. 9, 2023, 1:21 p.m. UTC | #1 
On Mon, Oct 09, 2023 at 01:51:16AM -0700, Yi Liu wrote:
> Intel SIOV allows creating virtual devices of which the vRID is
> represented by a pasid of a physical device. It is called as SIOV
> virtual device in this series. Such devices can be bound to an iommufd
> as physical device does and then later be attached to an IOAS/hwpt
> using that pasid. Such PASIDs are called as default pasid.

I would want to see the idxd implementation too..

Jason
Tian, Kevin Oct. 9, 2023, 11:33 p.m. UTC | #2 
> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Monday, October 9, 2023 9:21 PM
> 
> On Mon, Oct 09, 2023 at 01:51:16AM -0700, Yi Liu wrote:
> > Intel SIOV allows creating virtual devices of which the vRID is
> > represented by a pasid of a physical device. It is called as SIOV
> > virtual device in this series. Such devices can be bound to an iommufd
> > as physical device does and then later be attached to an IOAS/hwpt
> > using that pasid. Such PASIDs are called as default pasid.
> 
> I would want to see the idxd implementation too..
> 

It still needs some time (and unfortunately the guy working on idxd
is currently on a long vacation).

Instead of waiting we want to seek early comments on the iommufd
changes given that part is relatively self-contained. Same as what
Reinette is doing for IMS.

Certainly this is not for merging w/o having a driver user. 😊
Cao, Yahui Nov. 22, 2023, 3:59 a.m. UTC | #3 
On 10/9/2023 9:21 PM, Jason Gunthorpe wrote:
> On Mon, Oct 09, 2023 at 01:51:16AM -0700, Yi Liu wrote:
>> Intel SIOV allows creating virtual devices of which the vRID is
>> represented by a pasid of a physical device. It is called as SIOV
>> virtual device in this series. Such devices can be bound to an iommufd
>> as physical device does and then later be attached to an IOAS/hwpt
>> using that pasid. Such PASIDs are called as default pasid.
> 
> I would want to see the idxd implementation too..
> 
> Jason

Hey Jason,

         ice(E810 NIC) driver implementation for SIOV virtual device is 
also working in progress. We are working closely with Kevin and Yi on 
the patchset.

         We'll send out the ice patch for SIOV as lan driver user 
example once it is available. (There is some format issue in the last 
email, re-send again and sorry for any inconvenience)

Thanks.
Yahui.