[RFC,0/7] vdpa: Add support for iommufd

  Hi All
Really apologize for the delay, this is the draft RFC for
iommufd support for vdpa, This code provides the basic function 
for iommufd support 

The code was tested and passed in device vdpa_sim_net
The qemu code is
https://gitlab.com/lulu6/gitlabqemutmp/-/tree/iommufdRFC
The kernel code is
https://gitlab.com/lulu6/vhost/-/tree/iommufdRFC

ToDo
1. this code is out of date and needs to clean and rebase on the latest code 
2. this code has some workaround, I Skip the check for
iommu_group and CACHE_COHERENCY, also some misc issues like need to add
mutex for iommfd operations 
3. only test in emulated device, other modes not tested yet

After addressed these problems I will send out a new version for RFC. I will
provide the code in 3 weeks

Thanks
Cindy 
Signed-off-by: Cindy Lu <lulu@redhat.com>
The test step is
1. create vdpa_sim device
...
vdpa dev add name vdpa15 mgmtdev vdpasim_net
...
2. load the VM with the command
  -object iommufd,id=iommufd0 \
  -device virtio-net-pci,netdev=vhost-vdpa1,disable-legacy=on,disable-modern=off\
  -netdev type=vhost-vdpa,vhostdev=/dev/vhost-vdpa-0,id=vhost-vdpa1,iommufd=iommufd0\

3. in guest VM you can find the vdpa_sim port works well.
[root@ubuntunew ~]# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::5054:ff:fe12:3456  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:12:34:56  txqueuelen 1000  (Ethernet)
        RX packets 53  bytes 9108 (8.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 53  bytes 9108 (8.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@ubuntunew ~]# ./test.sh eth0
[  172.815279] pktgen: Packet Generator for packet performance testing. Version: 2.75
Adding queue 0 of eth0
Configuring devices eth0@0
Running... ctrl^C to stop

[root@ubuntunew ~]# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::5054:ff:fe12:3456  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:12:34:56  txqueuelen 1000  (Ethernet)
        RX packets 183455  bytes 11748533 (11.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 183473  bytes 11749685 (11.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Cindy Lu (7):
  vhost/iommufd: Add the functions support iommufd
  Kconfig: Add the new file vhost/iommufd
  vhost: Add 3 new uapi to support iommufd
  vdpa: change the map/unmap process to support iommufd
  vdpa: Add new vdpa_config_ops
  vdpa_sim :Add support for iommufd
  iommufd: Skip the CACHE_COHERENCY and iommu group check

 drivers/iommu/iommufd/device.c   |   6 +-
 drivers/vdpa/vdpa_sim/vdpa_sim.c |   8 ++
 drivers/vhost/Kconfig            |   1 +
 drivers/vhost/Makefile           |   1 +
 drivers/vhost/iommufd.c          | 151 +++++++++++++++++++++++
 drivers/vhost/vdpa.c             | 201 +++++++++++++++++++++++++++++++
 drivers/vhost/vhost.h            |  21 ++++
 include/linux/vdpa.h             |  34 +++++-
 include/uapi/linux/vhost.h       |  71 +++++++++++
 9 files changed, 490 insertions(+), 4 deletions(-)
 create mode 100644 drivers/vhost/iommufd.c
  

On Sun, Sep 24, 2023 at 1:05 AM Cindy Lu <lulu@redhat.com> wrote:
>
> Hi All
> Really apologize for the delay, this is the draft RFC for
> iommufd support for vdpa, This code provides the basic function
> for iommufd support
>
> The code was tested and passed in device vdpa_sim_net
> The qemu code is
> https://gitlab.com/lulu6/gitlabqemutmp/-/tree/iommufdRFC
> The kernel code is
> https://gitlab.com/lulu6/vhost/-/tree/iommufdRFC
>
> ToDo
> 1. this code is out of date and needs to clean and rebase on the latest code
> 2. this code has some workaround, I Skip the check for
> iommu_group and CACHE_COHERENCY, also some misc issues like need to add
> mutex for iommfd operations
> 3. only test in emulated device, other modes not tested yet
>
> After addressed these problems I will send out a new version for RFC. I will
> provide the code in 3 weeks

Something more needs to be done after a quick glance at the codes.

1) The support for device with platform IOMMU support
2) The support for multiple ASes per device

...

Thanks

>
> Thanks
> Cindy
> Signed-off-by: Cindy Lu <lulu@redhat.com>
> The test step is
> 1. create vdpa_sim device
> ...
> vdpa dev add name vdpa15 mgmtdev vdpasim_net
> ...
> 2. load the VM with the command
>   -object iommufd,id=iommufd0 \
>   -device virtio-net-pci,netdev=vhost-vdpa1,disable-legacy=on,disable-modern=off\
>   -netdev type=vhost-vdpa,vhostdev=/dev/vhost-vdpa-0,id=vhost-vdpa1,iommufd=iommufd0\
>
> 3. in guest VM you can find the vdpa_sim port works well.
> [root@ubuntunew ~]# ifconfig eth0
> eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet6 fe80::5054:ff:fe12:3456  prefixlen 64  scopeid 0x20<link>
>         ether 52:54:00:12:34:56  txqueuelen 1000  (Ethernet)
>         RX packets 53  bytes 9108 (8.8 KiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 53  bytes 9108 (8.8 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>
> [root@ubuntunew ~]# ./test.sh eth0
> [  172.815279] pktgen: Packet Generator for packet performance testing. Version: 2.75
> Adding queue 0 of eth0
> Configuring devices eth0@0
> Running... ctrl^C to stop
>
> [root@ubuntunew ~]# ifconfig eth0
> eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet6 fe80::5054:ff:fe12:3456  prefixlen 64  scopeid 0x20<link>
>         ether 52:54:00:12:34:56  txqueuelen 1000  (Ethernet)
>         RX packets 183455  bytes 11748533 (11.2 MiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 183473  bytes 11749685 (11.2 MiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>
> Cindy Lu (7):
>   vhost/iommufd: Add the functions support iommufd
>   Kconfig: Add the new file vhost/iommufd
>   vhost: Add 3 new uapi to support iommufd
>   vdpa: change the map/unmap process to support iommufd
>   vdpa: Add new vdpa_config_ops
>   vdpa_sim :Add support for iommufd
>   iommufd: Skip the CACHE_COHERENCY and iommu group check
>
>  drivers/iommu/iommufd/device.c   |   6 +-
>  drivers/vdpa/vdpa_sim/vdpa_sim.c |   8 ++
>  drivers/vhost/Kconfig            |   1 +
>  drivers/vhost/Makefile           |   1 +
>  drivers/vhost/iommufd.c          | 151 +++++++++++++++++++++++
>  drivers/vhost/vdpa.c             | 201 +++++++++++++++++++++++++++++++
>  drivers/vhost/vhost.h            |  21 ++++
>  include/linux/vdpa.h             |  34 +++++-
>  include/uapi/linux/vhost.h       |  71 +++++++++++
>  9 files changed, 490 insertions(+), 4 deletions(-)
>  create mode 100644 drivers/vhost/iommufd.c
>
> --
> 2.34.3
>
  

On Sun, Sep 24, 2023 at 01:05:33AM +0800, Cindy Lu wrote:
> Hi All
> Really apologize for the delay, this is the draft RFC for
> iommufd support for vdpa, This code provides the basic function 
> for iommufd support 
> 
> The code was tested and passed in device vdpa_sim_net
> The qemu code is
> https://gitlab.com/lulu6/gitlabqemutmp/-/tree/iommufdRFC
> The kernel code is
> https://gitlab.com/lulu6/vhost/-/tree/iommufdRFC
> 
> ToDo
> 1. this code is out of date and needs to clean and rebase on the latest code 
> 2. this code has some workaround, I Skip the check for
> iommu_group and CACHE_COHERENCY, also some misc issues like need to add
> mutex for iommfd operations 
> 3. only test in emulated device, other modes not tested yet
> 
> After addressed these problems I will send out a new version for RFC. I will
> provide the code in 3 weeks

What's the status here?
  

On Thu, Oct 26, 2023 at 2:42 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Sun, Sep 24, 2023 at 01:05:33AM +0800, Cindy Lu wrote:
> > Hi All
> > Really apologize for the delay, this is the draft RFC for
> > iommufd support for vdpa, This code provides the basic function
> > for iommufd support
> >
> > The code was tested and passed in device vdpa_sim_net
> > The qemu code is
> > https://gitlab.com/lulu6/gitlabqemutmp/-/tree/iommufdRFC
> > The kernel code is
> > https://gitlab.com/lulu6/vhost/-/tree/iommufdRFC
> >
> > ToDo
> > 1. this code is out of date and needs to clean and rebase on the latest code
> > 2. this code has some workaround, I Skip the check for
> > iommu_group and CACHE_COHERENCY, also some misc issues like need to add
> > mutex for iommfd operations
> > 3. only test in emulated device, other modes not tested yet
> >
> > After addressed these problems I will send out a new version for RFC. I will
> > provide the code in 3 weeks
>
> What's the status here?
>
Hi Michael
The code is finished, but I found some bug after adding the support for ASID,
will post the new version after this bug is fixed, should be next week
Thanks
Cindy

> --
> MST
>
  

On Thu, Oct 26, 2023 at 02:48:07PM +0800, Cindy Lu wrote:
> On Thu, Oct 26, 2023 at 2:42 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Sun, Sep 24, 2023 at 01:05:33AM +0800, Cindy Lu wrote:
> > > Hi All
> > > Really apologize for the delay, this is the draft RFC for
> > > iommufd support for vdpa, This code provides the basic function
> > > for iommufd support
> > >
> > > The code was tested and passed in device vdpa_sim_net
> > > The qemu code is
> > > https://gitlab.com/lulu6/gitlabqemutmp/-/tree/iommufdRFC
> > > The kernel code is
> > > https://gitlab.com/lulu6/vhost/-/tree/iommufdRFC
> > >
> > > ToDo
> > > 1. this code is out of date and needs to clean and rebase on the latest code
> > > 2. this code has some workaround, I Skip the check for
> > > iommu_group and CACHE_COHERENCY, also some misc issues like need to add
> > > mutex for iommfd operations
> > > 3. only test in emulated device, other modes not tested yet
> > >
> > > After addressed these problems I will send out a new version for RFC. I will
> > > provide the code in 3 weeks
> >
> > What's the status here?
> >
> Hi Michael
> The code is finished, but I found some bug after adding the support for ASID,
> will post the new version after this bug is fixed, should be next week
> Thanks
> Cindy


We'll miss this merge window then.

> > --
> > MST
> >
  

On Thu, Oct 26, 2023 at 2:49 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Thu, Oct 26, 2023 at 02:48:07PM +0800, Cindy Lu wrote:
> > On Thu, Oct 26, 2023 at 2:42 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Sun, Sep 24, 2023 at 01:05:33AM +0800, Cindy Lu wrote:
> > > > Hi All
> > > > Really apologize for the delay, this is the draft RFC for
> > > > iommufd support for vdpa, This code provides the basic function
> > > > for iommufd support
> > > >
> > > > The code was tested and passed in device vdpa_sim_net
> > > > The qemu code is
> > > > https://gitlab.com/lulu6/gitlabqemutmp/-/tree/iommufdRFC
> > > > The kernel code is
> > > > https://gitlab.com/lulu6/vhost/-/tree/iommufdRFC
> > > >
> > > > ToDo
> > > > 1. this code is out of date and needs to clean and rebase on the latest code
> > > > 2. this code has some workaround, I Skip the check for
> > > > iommu_group and CACHE_COHERENCY, also some misc issues like need to add
> > > > mutex for iommfd operations
> > > > 3. only test in emulated device, other modes not tested yet
> > > >
> > > > After addressed these problems I will send out a new version for RFC. I will
> > > > provide the code in 3 weeks
> > >
> > > What's the status here?
> > >
> > Hi Michael
> > The code is finished, but I found some bug after adding the support for ASID,
> > will post the new version after this bug is fixed, should be next week
> > Thanks
> > Cindy
>
>
> We'll miss this merge window then.
>
thanks Micheal， I will try my best. will post the new version as soon as I can
Thanks
Cindy
> > > --
> > > MST
> > >
>
  

On Thu, Oct 26, 2023 at 02:48:07PM +0800, Cindy Lu wrote:
> On Thu, Oct 26, 2023 at 2:42 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Sun, Sep 24, 2023 at 01:05:33AM +0800, Cindy Lu wrote:
> > > Hi All
> > > Really apologize for the delay, this is the draft RFC for
> > > iommufd support for vdpa, This code provides the basic function
> > > for iommufd support
> > >
> > > The code was tested and passed in device vdpa_sim_net
> > > The qemu code is
> > > https://gitlab.com/lulu6/gitlabqemutmp/-/tree/iommufdRFC
> > > The kernel code is
> > > https://gitlab.com/lulu6/vhost/-/tree/iommufdRFC
> > >
> > > ToDo
> > > 1. this code is out of date and needs to clean and rebase on the latest code
> > > 2. this code has some workaround, I Skip the check for
> > > iommu_group and CACHE_COHERENCY, also some misc issues like need to add
> > > mutex for iommfd operations
> > > 3. only test in emulated device, other modes not tested yet
> > >
> > > After addressed these problems I will send out a new version for RFC. I will
> > > provide the code in 3 weeks
> >
> > What's the status here?
> >
> Hi Michael
> The code is finished, but I found some bug after adding the support for ASID,
> will post the new version after this bug is fixed, should be next week
> Thanks
> Cindy

The week is almost gone, what's going on?


> > --
> > MST
> >
  

On Thu, Nov 2, 2023 at 6:02 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Thu, Oct 26, 2023 at 02:48:07PM +0800, Cindy Lu wrote:
> > On Thu, Oct 26, 2023 at 2:42 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Sun, Sep 24, 2023 at 01:05:33AM +0800, Cindy Lu wrote:
> > > > Hi All
> > > > Really apologize for the delay, this is the draft RFC for
> > > > iommufd support for vdpa, This code provides the basic function
> > > > for iommufd support
> > > >
> > > > The code was tested and passed in device vdpa_sim_net
> > > > The qemu code is
> > > > https://gitlab.com/lulu6/gitlabqemutmp/-/tree/iommufdRFC
> > > > The kernel code is
> > > > https://gitlab.com/lulu6/vhost/-/tree/iommufdRFC
> > > >
> > > > ToDo
> > > > 1. this code is out of date and needs to clean and rebase on the latest code
> > > > 2. this code has some workaround, I Skip the check for
> > > > iommu_group and CACHE_COHERENCY, also some misc issues like need to add
> > > > mutex for iommfd operations
> > > > 3. only test in emulated device, other modes not tested yet
> > > >
> > > > After addressed these problems I will send out a new version for RFC. I will
> > > > provide the code in 3 weeks
> > >
> > > What's the status here?
> > >
> > Hi Michael
> > The code is finished, but I found some bug after adding the support for ASID,
> > will post the new version after this bug is fixed, should be next week
> > Thanks
> > Cindy
>
> The week is almost gone, what's going on?
>
thanks, Micheal, I will send it out tomorrow
Thanks
Cindy
>
> > > --
> > > MST
> > >
>