mbox series

[vhost,v2,00/16] vdpa: Add support for vq descriptor mappings

Message ID 20230928164550.980832-2-dtatulea@nvidia.com
Headers
Series vdpa: Add support for vq descriptor mappings |

Message

Dragos Tatulea Sept. 28, 2023, 4:45 p.m. UTC 
  This patch series adds support for vq descriptor table mappings which
are used to improve vdpa live migration downtime. The improvement comes
from using smaller mappings which take less time to create and destroy
in hw.

The first part adds the vdpa core changes from Si-Wei [0].

The second part adds support in mlx5_vdpa:
- Refactor the mr code to be able to cleanly add descriptor mappings.
- Add hardware descriptor mr support.
- Properly update iotlb for cvq during ASID switch.

Changes in v2:

- The "vdpa/mlx5: Enable hw support for vq descriptor mapping" change
  was split off into two patches to avoid merge conflicts into the tree
  of Linus.

  The first patch contains only changes for mlx5_ifc.h. This must be
  applied into the mlx5-next tree [1] first. Once this patch is applied
  on mlx5-next, the change has to be pulled fom mlx5-next into the vhost
  tree and only then the remaining patches can be applied.

[0] https://lore.kernel.org/virtualization/1694248959-13369-1-git-send-email-si-wei.liu@oracle.com
[1] https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git/log/?h=mlx5-next

Dragos Tatulea (13):
  vdpa/mlx5: Expose descriptor group mkey hw capability
  vdpa/mlx5: Create helper function for dma mappings
  vdpa/mlx5: Decouple cvq iotlb handling from hw mapping code
  vdpa/mlx5: Take cvq iotlb lock during refresh
  vdpa/mlx5: Collapse "dvq" mr add/delete functions
  vdpa/mlx5: Rename mr destroy functions
  vdpa/mlx5: Allow creation/deletion of any given mr struct
  vdpa/mlx5: Move mr mutex out of mr struct
  vdpa/mlx5: Improve mr update flow
  vdpa/mlx5: Introduce mr for vq descriptor
  vdpa/mlx5: Enable hw support for vq descriptor mapping
  vdpa/mlx5: Make iotlb helper functions more generic
  vdpa/mlx5: Update cvq iotlb mapping on ASID change

Si-Wei Liu (3):
  vdpa: introduce dedicated descriptor group for virtqueue
  vhost-vdpa: introduce descriptor group backend feature
  vhost-vdpa: uAPI to get dedicated descriptor group id

 drivers/vdpa/mlx5/core/mlx5_vdpa.h |  31 +++--
 drivers/vdpa/mlx5/core/mr.c        | 191 ++++++++++++++++-------------
 drivers/vdpa/mlx5/core/resources.c |   6 +-
 drivers/vdpa/mlx5/net/mlx5_vnet.c  | 100 ++++++++++-----
 drivers/vhost/vdpa.c               |  27 ++++
 include/linux/mlx5/mlx5_ifc.h      |   8 +-
 include/linux/mlx5/mlx5_ifc_vdpa.h |   7 +-
 include/linux/vdpa.h               |  11 ++
 include/uapi/linux/vhost.h         |   8 ++
 include/uapi/linux/vhost_types.h   |   5 +
 10 files changed, 264 insertions(+), 130 deletions(-)

Comments

Leon Romanovsky Oct. 2, 2023, 10:16 a.m. UTC | #1 
On Thu, Sep 28, 2023 at 07:45:11PM +0300, Dragos Tatulea wrote:
> This patch series adds support for vq descriptor table mappings which
> are used to improve vdpa live migration downtime. The improvement comes
> from using smaller mappings which take less time to create and destroy
> in hw.
> 
> The first part adds the vdpa core changes from Si-Wei [0].
> 
> The second part adds support in mlx5_vdpa:
> - Refactor the mr code to be able to cleanly add descriptor mappings.
> - Add hardware descriptor mr support.
> - Properly update iotlb for cvq during ASID switch.
> 
> Changes in v2:
> 
> - The "vdpa/mlx5: Enable hw support for vq descriptor mapping" change
>   was split off into two patches to avoid merge conflicts into the tree
>   of Linus.
> 
>   The first patch contains only changes for mlx5_ifc.h. This must be
>   applied into the mlx5-next tree [1] first. Once this patch is applied
>   on mlx5-next, the change has to be pulled fom mlx5-next into the vhost
>   tree and only then the remaining patches can be applied.
> 
> [0] https://lore.kernel.org/virtualization/1694248959-13369-1-git-send-email-si-wei.liu@oracle.com
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git/log/?h=mlx5-next
> 
> Dragos Tatulea (13):
>   vdpa/mlx5: Expose descriptor group mkey hw capability

I prepared shared branch with this patch.
https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git/log/?h=mlx5-vhost

Thanks
Leon Romanovsky Oct. 2, 2023, 10:16 a.m. UTC | #2 
On Thu, 28 Sep 2023 19:45:11 +0300, Dragos Tatulea wrote:
> This patch series adds support for vq descriptor table mappings which
> are used to improve vdpa live migration downtime. The improvement comes
> from using smaller mappings which take less time to create and destroy
> in hw.
> 
> The first part adds the vdpa core changes from Si-Wei [0].
> 
> [...]

Applied, thanks!

[01/16] vdpa/mlx5: Expose descriptor group mkey hw capability
        https://git.kernel.org/rdma/rdma/c/d424348b060d87

Best regards,
Michael S. Tsirkin Oct. 5, 2023, 5:31 p.m. UTC | #3 
On Thu, Sep 28, 2023 at 07:45:11PM +0300, Dragos Tatulea wrote:
> This patch series adds support for vq descriptor table mappings which
> are used to improve vdpa live migration downtime. The improvement comes
> from using smaller mappings which take less time to create and destroy
> in hw.
> 
> The first part adds the vdpa core changes from Si-Wei [0].
> 
> The second part adds support in mlx5_vdpa:
> - Refactor the mr code to be able to cleanly add descriptor mappings.
> - Add hardware descriptor mr support.
> - Properly update iotlb for cvq during ASID switch.
> 
> Changes in v2:
> 
> - The "vdpa/mlx5: Enable hw support for vq descriptor mapping" change
>   was split off into two patches to avoid merge conflicts into the tree
>   of Linus.
> 
>   The first patch contains only changes for mlx5_ifc.h. This must be
>   applied into the mlx5-next tree [1] first. Once this patch is applied
>   on mlx5-next, the change has to be pulled fom mlx5-next into the vhost
>   tree and only then the remaining patches can be applied.


I get it you plan v3?

> [0] https://lore.kernel.org/virtualization/1694248959-13369-1-git-send-email-si-wei.liu@oracle.com
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git/log/?h=mlx5-next
> 
> Dragos Tatulea (13):
>   vdpa/mlx5: Expose descriptor group mkey hw capability
>   vdpa/mlx5: Create helper function for dma mappings
>   vdpa/mlx5: Decouple cvq iotlb handling from hw mapping code
>   vdpa/mlx5: Take cvq iotlb lock during refresh
>   vdpa/mlx5: Collapse "dvq" mr add/delete functions
>   vdpa/mlx5: Rename mr destroy functions
>   vdpa/mlx5: Allow creation/deletion of any given mr struct
>   vdpa/mlx5: Move mr mutex out of mr struct
>   vdpa/mlx5: Improve mr update flow
>   vdpa/mlx5: Introduce mr for vq descriptor
>   vdpa/mlx5: Enable hw support for vq descriptor mapping
>   vdpa/mlx5: Make iotlb helper functions more generic
>   vdpa/mlx5: Update cvq iotlb mapping on ASID change
> 
> Si-Wei Liu (3):
>   vdpa: introduce dedicated descriptor group for virtqueue
>   vhost-vdpa: introduce descriptor group backend feature
>   vhost-vdpa: uAPI to get dedicated descriptor group id
> 
>  drivers/vdpa/mlx5/core/mlx5_vdpa.h |  31 +++--
>  drivers/vdpa/mlx5/core/mr.c        | 191 ++++++++++++++++-------------
>  drivers/vdpa/mlx5/core/resources.c |   6 +-
>  drivers/vdpa/mlx5/net/mlx5_vnet.c  | 100 ++++++++++-----
>  drivers/vhost/vdpa.c               |  27 ++++
>  include/linux/mlx5/mlx5_ifc.h      |   8 +-
>  include/linux/mlx5/mlx5_ifc_vdpa.h |   7 +-
>  include/linux/vdpa.h               |  11 ++
>  include/uapi/linux/vhost.h         |   8 ++
>  include/uapi/linux/vhost_types.h   |   5 +
>  10 files changed, 264 insertions(+), 130 deletions(-)
> 
> -- 
> 2.41.0
Dragos Tatulea Oct. 5, 2023, 5:44 p.m. UTC | #4 
On Thu, 2023-10-05 at 13:31 -0400, Michael S. Tsirkin wrote:
> On Thu, Sep 28, 2023 at 07:45:11PM +0300, Dragos Tatulea wrote:
> > This patch series adds support for vq descriptor table mappings which
> > are used to improve vdpa live migration downtime. The improvement comes
> > from using smaller mappings which take less time to create and destroy
> > in hw.
> > 
> > The first part adds the vdpa core changes from Si-Wei [0].
> > 
> > The second part adds support in mlx5_vdpa:
> > - Refactor the mr code to be able to cleanly add descriptor mappings.
> > - Add hardware descriptor mr support.
> > - Properly update iotlb for cvq during ASID switch.
> > 
> > Changes in v2:
> > 
> > - The "vdpa/mlx5: Enable hw support for vq descriptor mapping" change
> >   was split off into two patches to avoid merge conflicts into the tree
> >   of Linus.
> > 
> >   The first patch contains only changes for mlx5_ifc.h. This must be
> >   applied into the mlx5-next tree [1] first. Once this patch is applied
> >   on mlx5-next, the change has to be pulled fom mlx5-next into the vhost
> >   tree and only then the remaining patches can be applied.
> 
> 
> I get it you plan v3?
There are some very small improvements (commit message in 13/16 and fix in
16/16) that could make a v3. The latter can be addressed as a separate patch
when moving dup_iotlb to vhost/iotlb. What do you think?

> 
> > [0]
> > https://lore.kernel.org/virtualization/1694248959-13369-1-git-send-email-si-wei.liu@oracle.com
> > [1]
> > https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git/log/?h=mlx5-next
> > 
> > Dragos Tatulea (13):
> >   vdpa/mlx5: Expose descriptor group mkey hw capability
> >   vdpa/mlx5: Create helper function for dma mappings
> >   vdpa/mlx5: Decouple cvq iotlb handling from hw mapping code
> >   vdpa/mlx5: Take cvq iotlb lock during refresh
> >   vdpa/mlx5: Collapse "dvq" mr add/delete functions
> >   vdpa/mlx5: Rename mr destroy functions
> >   vdpa/mlx5: Allow creation/deletion of any given mr struct
> >   vdpa/mlx5: Move mr mutex out of mr struct
> >   vdpa/mlx5: Improve mr update flow
> >   vdpa/mlx5: Introduce mr for vq descriptor
> >   vdpa/mlx5: Enable hw support for vq descriptor mapping
> >   vdpa/mlx5: Make iotlb helper functions more generic
> >   vdpa/mlx5: Update cvq iotlb mapping on ASID change
> > 
> > Si-Wei Liu (3):
> >   vdpa: introduce dedicated descriptor group for virtqueue
> >   vhost-vdpa: introduce descriptor group backend feature
> >   vhost-vdpa: uAPI to get dedicated descriptor group id
> > 
> >  drivers/vdpa/mlx5/core/mlx5_vdpa.h |  31 +++--
> >  drivers/vdpa/mlx5/core/mr.c        | 191 ++++++++++++++++-------------
> >  drivers/vdpa/mlx5/core/resources.c |   6 +-
> >  drivers/vdpa/mlx5/net/mlx5_vnet.c  | 100 ++++++++++-----
> >  drivers/vhost/vdpa.c               |  27 ++++
> >  include/linux/mlx5/mlx5_ifc.h      |   8 +-
> >  include/linux/mlx5/mlx5_ifc_vdpa.h |   7 +-
> >  include/linux/vdpa.h               |  11 ++
> >  include/uapi/linux/vhost.h         |   8 ++
> >  include/uapi/linux/vhost_types.h   |   5 +
> >  10 files changed, 264 insertions(+), 130 deletions(-)
> > 
> > -- 
> > 2.41.0
>
Michael S. Tsirkin Oct. 5, 2023, 7:18 p.m. UTC | #5 
On Thu, Oct 05, 2023 at 05:44:01PM +0000, Dragos Tatulea wrote:
> On Thu, 2023-10-05 at 13:31 -0400, Michael S. Tsirkin wrote:
> > On Thu, Sep 28, 2023 at 07:45:11PM +0300, Dragos Tatulea wrote:
> > > This patch series adds support for vq descriptor table mappings which
> > > are used to improve vdpa live migration downtime. The improvement comes
> > > from using smaller mappings which take less time to create and destroy
> > > in hw.
> > > 
> > > The first part adds the vdpa core changes from Si-Wei [0].
> > > 
> > > The second part adds support in mlx5_vdpa:
> > > - Refactor the mr code to be able to cleanly add descriptor mappings.
> > > - Add hardware descriptor mr support.
> > > - Properly update iotlb for cvq during ASID switch.
> > > 
> > > Changes in v2:
> > > 
> > > - The "vdpa/mlx5: Enable hw support for vq descriptor mapping" change
> > >   was split off into two patches to avoid merge conflicts into the tree
> > >   of Linus.
> > > 
> > >   The first patch contains only changes for mlx5_ifc.h. This must be
> > >   applied into the mlx5-next tree [1] first. Once this patch is applied
> > >   on mlx5-next, the change has to be pulled fom mlx5-next into the vhost
> > >   tree and only then the remaining patches can be applied.
> > 
> > 
> > I get it you plan v3?
> There are some very small improvements (commit message in 13/16 and fix in
> 16/16) that could make a v3. The latter can be addressed as a separate patch
> when moving dup_iotlb to vhost/iotlb. What do you think?


if there's a fix by all means post v3.

> > 
> > > [0]
> > > https://lore.kernel.org/virtualization/1694248959-13369-1-git-send-email-si-wei.liu@oracle.com
> > > [1]
> > > https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git/log/?h=mlx5-next
> > > 
> > > Dragos Tatulea (13):
> > >   vdpa/mlx5: Expose descriptor group mkey hw capability
> > >   vdpa/mlx5: Create helper function for dma mappings
> > >   vdpa/mlx5: Decouple cvq iotlb handling from hw mapping code
> > >   vdpa/mlx5: Take cvq iotlb lock during refresh
> > >   vdpa/mlx5: Collapse "dvq" mr add/delete functions
> > >   vdpa/mlx5: Rename mr destroy functions
> > >   vdpa/mlx5: Allow creation/deletion of any given mr struct
> > >   vdpa/mlx5: Move mr mutex out of mr struct
> > >   vdpa/mlx5: Improve mr update flow
> > >   vdpa/mlx5: Introduce mr for vq descriptor
> > >   vdpa/mlx5: Enable hw support for vq descriptor mapping
> > >   vdpa/mlx5: Make iotlb helper functions more generic
> > >   vdpa/mlx5: Update cvq iotlb mapping on ASID change
> > > 
> > > Si-Wei Liu (3):
> > >   vdpa: introduce dedicated descriptor group for virtqueue
> > >   vhost-vdpa: introduce descriptor group backend feature
> > >   vhost-vdpa: uAPI to get dedicated descriptor group id
> > > 
> > >  drivers/vdpa/mlx5/core/mlx5_vdpa.h |  31 +++--
> > >  drivers/vdpa/mlx5/core/mr.c        | 191 ++++++++++++++++-------------
> > >  drivers/vdpa/mlx5/core/resources.c |   6 +-
> > >  drivers/vdpa/mlx5/net/mlx5_vnet.c  | 100 ++++++++++-----
> > >  drivers/vhost/vdpa.c               |  27 ++++
> > >  include/linux/mlx5/mlx5_ifc.h      |   8 +-
> > >  include/linux/mlx5/mlx5_ifc_vdpa.h |   7 +-
> > >  include/linux/vdpa.h               |  11 ++
> > >  include/uapi/linux/vhost.h         |   8 ++
> > >  include/uapi/linux/vhost_types.h   |   5 +
> > >  10 files changed, 264 insertions(+), 130 deletions(-)
> > > 
> > > -- 
> > > 2.41.0
> > 
>