[RFC,3/3] vfio/pci: Expose PCIe PASID capability to userspace

  This exposes PCIe PASID capability to userspace and where to emulate this
capability if wants to further expose it to VM.

And this only exposes PASID capability for devices which has PCIe PASID
extended struture in its configuration space. While for VFs, userspace
is still unable to see this capability as SR-IOV spec forbides VF to
implement PASID capability extended structure. It is a TODO in future.
Related discussion can be found in below links:

https://lore.kernel.org/kvm/20200407095801.648b1371@w520.home/
https://lore.kernel.org/kvm/BL1PR11MB5271A60035EF591A5BE8AC878C01A@BL1PR11MB5271.namprd11.prod.outlook.com/

Signed-off-by: Yi Liu <yi.l.liu@intel.com>
---
 drivers/vfio/pci/vfio_pci_config.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
  

> From: Liu, Yi L <yi.l.liu@intel.com>
> Sent: Tuesday, September 26, 2023 5:31 PM
> 
> This exposes PCIe PASID capability to userspace and where to emulate this
> capability if wants to further expose it to VM.
> 
> And this only exposes PASID capability for devices which has PCIe PASID
> extended struture in its configuration space. While for VFs, userspace
> is still unable to see this capability as SR-IOV spec forbides VF to
> implement PASID capability extended structure. It is a TODO in future.
> Related discussion can be found in below links:
> 
> https://lore.kernel.org/kvm/20200407095801.648b1371@w520.home/
> https://lore.kernel.org/kvm/BL1PR11MB5271A60035EF591A5BE8AC878C01A
> @BL1PR11MB5271.namprd11.prod.outlook.com/
> 

Yes, we need a decision for VF case.

If the consensus is to continue exposing the PASID capability in vfio-pci
config space by developing a kernel quirk mechanism to find offset for
VF, then this patch for PF is orthogonal to that VF work and can go as it is.

But if the decision is to have a device feature for the user to enumerate
the vPASID capability and let the VMM take care of finding the vPASID
cap offset, then better we start doing that for PF too since it's not good
to have two enumeration interfaces for PF/VF respectively.

My preference is via device feature given Qemu already includes lots of
quirks for vfio-pci devices. Another reason is that when supporting vPASID
with SIOV there are some arch constraints which the driver needs to
report to the user to follow (e.g.  don't assign ENQCMD-capable sibling
vdev's to a same guest, etc.). A device feature interface can better
encapsulate everything related to vPASID in one place.

Thanks
Kevin
  

On Wed, 27 Sep 2023 08:07:54 +0000
"Tian, Kevin" <kevin.tian@intel.com> wrote:

> > From: Liu, Yi L <yi.l.liu@intel.com>
> > Sent: Tuesday, September 26, 2023 5:31 PM
> > 
> > This exposes PCIe PASID capability to userspace and where to emulate this
> > capability if wants to further expose it to VM.
> > 
> > And this only exposes PASID capability for devices which has PCIe PASID
> > extended struture in its configuration space. While for VFs, userspace
> > is still unable to see this capability as SR-IOV spec forbides VF to
> > implement PASID capability extended structure. It is a TODO in future.
> > Related discussion can be found in below links:
> > 
> > https://lore.kernel.org/kvm/20200407095801.648b1371@w520.home/
> > https://lore.kernel.org/kvm/BL1PR11MB5271A60035EF591A5BE8AC878C01A
> > @BL1PR11MB5271.namprd11.prod.outlook.com/
> >   
> 
> Yes, we need a decision for VF case.
> 
> If the consensus is to continue exposing the PASID capability in vfio-pci
> config space by developing a kernel quirk mechanism to find offset for
> VF, then this patch for PF is orthogonal to that VF work and can go as it is.
> 
> But if the decision is to have a device feature for the user to enumerate
> the vPASID capability and let the VMM take care of finding the vPASID
> cap offset, then better we start doing that for PF too since it's not good
> to have two enumeration interfaces for PF/VF respectively.

Note also that QEMU implements a lazy algorithm for exposing
capabilities, the default is to expose them, so we need to consider
existing VMs seeing a new read-only PASID capability on an assigned PF.

That might support an alternate means to expose the capability.

> My preference is via device feature given Qemu already includes lots of
> quirks for vfio-pci devices. Another reason is that when supporting vPASID
> with SIOV there are some arch constraints which the driver needs to
> report to the user to follow (e.g.  don't assign ENQCMD-capable sibling
> vdev's to a same guest, etc.).

?!

> A device feature interface can better
> encapsulate everything related to vPASID in one place.

Sorry if I don't remember, have you posted a proposal for the device
feature interface?  Thanks,

Alex
  

> From: Alex Williamson <alex.williamson@redhat.com>
> Sent: Thursday, September 28, 2023 2:53 AM
> 
> On Wed, 27 Sep 2023 08:07:54 +0000
> "Tian, Kevin" <kevin.tian@intel.com> wrote:
> 
> > > From: Liu, Yi L <yi.l.liu@intel.com>
> > > Sent: Tuesday, September 26, 2023 5:31 PM
> > >
> > > This exposes PCIe PASID capability to userspace and where to emulate
> this
> > > capability if wants to further expose it to VM.
> > >
> > > And this only exposes PASID capability for devices which has PCIe PASID
> > > extended struture in its configuration space. While for VFs, userspace
> > > is still unable to see this capability as SR-IOV spec forbides VF to
> > > implement PASID capability extended structure. It is a TODO in future.
> > > Related discussion can be found in below links:
> > >
> > > https://lore.kernel.org/kvm/20200407095801.648b1371@w520.home/
> > >
> https://lore.kernel.org/kvm/BL1PR11MB5271A60035EF591A5BE8AC878C01A
> > > @BL1PR11MB5271.namprd11.prod.outlook.com/
> > >
> >
> > Yes, we need a decision for VF case.
> >
> > If the consensus is to continue exposing the PASID capability in vfio-pci
> > config space by developing a kernel quirk mechanism to find offset for
> > VF, then this patch for PF is orthogonal to that VF work and can go as it is.
> >
> > But if the decision is to have a device feature for the user to enumerate
> > the vPASID capability and let the VMM take care of finding the vPASID
> > cap offset, then better we start doing that for PF too since it's not good
> > to have two enumeration interfaces for PF/VF respectively.
> 
> Note also that QEMU implements a lazy algorithm for exposing
> capabilities, the default is to expose them, so we need to consider
> existing VMs seeing a new read-only PASID capability on an assigned PF.
> 
> That might support an alternate means to expose the capability.

Yep. that's also a valid point.

> 
> > My preference is via device feature given Qemu already includes lots of
> > quirks for vfio-pci devices. Another reason is that when supporting vPASID
> > with SIOV there are some arch constraints which the driver needs to
> > report to the user to follow (e.g.  don't assign ENQCMD-capable sibling
> > vdev's to a same guest, etc.).
> 
> ?!

Sorry that I didn't plan to elaborate that tricky constraint before we show
the overall SIOV/vPASID implementation. Explaining it requires lots of
context and here just want to mention the potential requirement in case
we need more proofs to go this direction. 😊

> 
> > A device feature interface can better
> > encapsulate everything related to vPASID in one place.
> 
> Sorry if I don't remember, have you posted a proposal for the device
> feature interface?  Thanks,
> 

Not yet. Will do in next version.