[v8,0/9] Refactor the SMMU's CD table ownership

  Hi all,

This series refactors stage 1 domains so that they describe a single CD
entry. These entries are now inserted into a CD table that is owned by
the arm_smmu_master instead of the domain.
This is conceptually cleaner and unblocks other features, such as
attaching domains with PASID (for unmanaged/dma domains).

This patch series was originally part of a larger patch series that
implemented the set_dev_pasid callback for non-SVA domains but is now
split into a distinct series.

This patch series is also available on gerrit with Jean's SMMU test
engine patches cherry-picked on top for testing:
https://linux-review.googlesource.com/c/linux/kernel/git/torvalds/linux/+/24742/16

Thanks,
Michael Shavit

Changes in v8:
- Rebased off of 6.6-rc1
- Drive-by update of the "Move ctx_desc out of s1_cfg" commit message to be less
  vague.
- Link to v7: https://lore.kernel.org/linux-iommu/20230831174536.103472-1-mshavit@google.com/

Changes in v7:
- Change the amr_smmu_write_ctx_desc_devices helper introduced to
  arm_smmu_update_ctx_desc_devices to distinguish from the case where
  a potentially new CD entry is written to. Add a comment to clarify
  that it is assumed that the operation can't fail and that it's
  therefore safe not to handle the return. In contrast, the case where a
  new CD entry is written-to does not use the helper and does have to
  handle failure.
- Update commit message to be more clear about locking purpose.
- Drop "Skip cd sync if CD table isn't active" commit, and remove
  related comment in the "Move CD table to arm_smmu_master patch.
- And some minor cosmetic changes based on v6 feedback.
- Link to v6: https://lore.kernel.org/all/20230816131925.2521220-1-mshavit@google.com/

Changes in v6:
- Undo removal of s1fmt and renaming of s1cdmax
- Unwind the loop in amr_smmu_write_ctx_desc_devices to NULL out the CD
  entries we succesfully wrote on failure.
- Add a comment clarifying the different usages of
  amr_smmu_write_ctx_desc_devices
- Grab the asid lock while writing the RID CD to prevent a race with
  SVA.
- Add the device to the devices list before writing the CD to the table
  and installing the CD table.
- Link to v5: https://lore.kernel.org/all/20230808171446.2187795-1-mshavit@google.com/

Changes in v5:
- Clear the 0th CD entry when the domain is detached. Not clearing it
  caused a bug in arm_smmu_write_ctx_desc which doesn't expect the entry
  to already be set.
- Added commit to clean-up now unused master parameter in
  arm_smmu_domain_finalise
- Link to v4: https://lore.kernel.org/all/20230802163328.2623773-1-mshavit@google.com/

Changes in v4:
- Added comment about the cd_table's dependency on the iommu core's
  group mutex.
- Narrowed the range of code for which the domain's init_mutex is held
  on attach since it now only protects the arm_smmu_domain_finalise
  call.
- Link to v3: https://lore.kernel.org/all/20230801183845.4026101-1-mshavit@google.com/

Changes in v3:
- Add a helper to write a CD to all masters that a domain is attached
  to.
- Fixed an issue where an arm_smmu_write_ctx_desc error return wasn't
  correctly handled by its caller.
- Link to v2: https://lore.kernel.org/all/20230731104833.800114-1-mshavit@google.com/

Changes in v2:
- Allocate CD table when it's first needed instead of on probe.
- Minor changes
- Added commit to rename remaining usages of cdcfg to cd_table
- Link to v1: https://lore.kernel.org/all/20230727182647.4106140-1-mshavit@google.com/#r

Changes in v1:
- Replace s1_cfg with arm_smmu_ctx_desc_cfg representing the CD table
- Assume that the CD table is owned by the SMMU master for most
  operations. This is forward-compatible with the nested patch series as
  these operations wouldn't be called when the installed CD table comes
  from nested domains.
- Split off as a distinct patch series from https://lore.kernel.org/all/20230621063825.268890-1-mshavit@google.com/

Michael Shavit (9):
  iommu/arm-smmu-v3: Move ctx_desc out of s1_cfg
  iommu/arm-smmu-v3: Replace s1_cfg with cdtab_cfg
  iommu/arm-smmu-v3: Encapsulate ctx_desc_cfg init in alloc_cd_tables
  iommu/arm-smmu-v3: move stall_enabled to the cd table
  iommu/arm-smmu-v3: Refactor write_ctx_desc
  iommu/arm-smmu-v3: Move CD table to arm_smmu_master
  iommu/arm-smmu-v3: Cleanup arm_smmu_domain_finalise
  iommu/arm-smmu-v3: Update comment about STE liveness
  iommu/arm-smmu-v3: Rename cdcfg to cd_table

 .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c   |  41 ++-
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c   | 251 +++++++++---------
 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h   |  17 +-
 3 files changed, 166 insertions(+), 143 deletions(-)


base-commit: 0bb80ecc33a8fb5a682236443c1e740d5c917d1d
  

On Fri, Sep 15, 2023 at 09:17:31PM +0800, Michael Shavit wrote:
> 
> Hi all,
> 
> This series refactors stage 1 domains so that they describe a single CD
> entry. These entries are now inserted into a CD table that is owned by
> the arm_smmu_master instead of the domain.
> This is conceptually cleaner and unblocks other features, such as
> attaching domains with PASID (for unmanaged/dma domains).
> 
> This patch series was originally part of a larger patch series that
> implemented the set_dev_pasid callback for non-SVA domains but is now
> split into a distinct series.
> 
> This patch series is also available on gerrit with Jean's SMMU test
> engine patches cherry-picked on top for testing:
> https://linux-review.googlesource.com/c/linux/kernel/git/torvalds/linux/+/24742/16
> 
> Thanks,
> Michael Shavit
> 
> Changes in v8:
> - Rebased off of 6.6-rc1
> - Drive-by update of the "Move ctx_desc out of s1_cfg" commit message to be less
>   vague.
> - Link to v7: https://lore.kernel.org/linux-iommu/20230831174536.103472-1-mshavit@google.com/
 
Hmm. I recall that I gave Tested-by to v7, yet some of the patches
in this v8 don't include that. Anyway, I retested with this v8:

Tested-by: Nicolin Chen <nicolinc@nvidia.com>
  

On Fri, Sep 15, 2023 at 09:17:31PM +0800, Michael Shavit wrote:
> 
> Hi all,
> 
> This series refactors stage 1 domains so that they describe a single CD
> entry. These entries are now inserted into a CD table that is owned by
> the arm_smmu_master instead of the domain.
> This is conceptually cleaner and unblocks other features, such as
> attaching domains with PASID (for unmanaged/dma domains).
> 
> This patch series was originally part of a larger patch series that
> implemented the set_dev_pasid callback for non-SVA domains but is now
> split into a distinct series.
> 
> This patch series is also available on gerrit with Jean's SMMU test
> engine patches cherry-picked on top for testing:
> https://linux-review.googlesource.com/c/linux/kernel/git/torvalds/linux/+/24742/16
> 
> Thanks,
> Michael Shavit
> 
> Michael Shavit (9):
>   iommu/arm-smmu-v3: Move ctx_desc out of s1_cfg
>   iommu/arm-smmu-v3: Replace s1_cfg with cdtab_cfg
>   iommu/arm-smmu-v3: Encapsulate ctx_desc_cfg init in alloc_cd_tables
>   iommu/arm-smmu-v3: move stall_enabled to the cd table
>   iommu/arm-smmu-v3: Refactor write_ctx_desc
>   iommu/arm-smmu-v3: Move CD table to arm_smmu_master
>   iommu/arm-smmu-v3: Cleanup arm_smmu_domain_finalise
>   iommu/arm-smmu-v3: Update comment about STE liveness
>   iommu/arm-smmu-v3: Rename cdcfg to cd_table

Will, can you take this please?

Thanks,
Jason
  

On Wed, Oct 11, 2023 at 08:48:48PM -0300, Jason Gunthorpe wrote:
> On Fri, Sep 15, 2023 at 09:17:31PM +0800, Michael Shavit wrote:
> > 
> > Hi all,
> > 
> > This series refactors stage 1 domains so that they describe a single CD
> > entry. These entries are now inserted into a CD table that is owned by
> > the arm_smmu_master instead of the domain.
> > This is conceptually cleaner and unblocks other features, such as
> > attaching domains with PASID (for unmanaged/dma domains).
> > 
> > This patch series was originally part of a larger patch series that
> > implemented the set_dev_pasid callback for non-SVA domains but is now
> > split into a distinct series.
> > 
> > This patch series is also available on gerrit with Jean's SMMU test
> > engine patches cherry-picked on top for testing:
> > https://linux-review.googlesource.com/c/linux/kernel/git/torvalds/linux/+/24742/16
> > 
> > Thanks,
> > Michael Shavit
> > 
> > Michael Shavit (9):
> >   iommu/arm-smmu-v3: Move ctx_desc out of s1_cfg
> >   iommu/arm-smmu-v3: Replace s1_cfg with cdtab_cfg
> >   iommu/arm-smmu-v3: Encapsulate ctx_desc_cfg init in alloc_cd_tables
> >   iommu/arm-smmu-v3: move stall_enabled to the cd table
> >   iommu/arm-smmu-v3: Refactor write_ctx_desc
> >   iommu/arm-smmu-v3: Move CD table to arm_smmu_master
> >   iommu/arm-smmu-v3: Cleanup arm_smmu_domain_finalise
> >   iommu/arm-smmu-v3: Update comment about STE liveness
> >   iommu/arm-smmu-v3: Rename cdcfg to cd_table
> 
> Will, can you take this please?

On my list to look at today!

Will
  

On Fri, 15 Sep 2023 21:17:31 +0800, Michael Shavit wrote:
> This series refactors stage 1 domains so that they describe a single CD
> entry. These entries are now inserted into a CD table that is owned by
> the arm_smmu_master instead of the domain.
> This is conceptually cleaner and unblocks other features, such as
> attaching domains with PASID (for unmanaged/dma domains).
> 
> This patch series was originally part of a larger patch series that
> implemented the set_dev_pasid callback for non-SVA domains but is now
> split into a distinct series.
> 
> [...]

Applied to will (for-joerg/arm-smmu/updates), thanks!

[1/9] iommu/arm-smmu-v3: Move ctx_desc out of s1_cfg
      https://git.kernel.org/will/c/987a878e09c6
[2/9] iommu/arm-smmu-v3: Replace s1_cfg with cdtab_cfg
      https://git.kernel.org/will/c/1f8588834016
[3/9] iommu/arm-smmu-v3: Encapsulate ctx_desc_cfg init in alloc_cd_tables
      https://git.kernel.org/will/c/e3aad74c51a7
[4/9] iommu/arm-smmu-v3: move stall_enabled to the cd table
      https://git.kernel.org/will/c/1228cc509fc6
[5/9] iommu/arm-smmu-v3: Refactor write_ctx_desc
      https://git.kernel.org/will/c/24503148c545
[6/9] iommu/arm-smmu-v3: Move CD table to arm_smmu_master
      https://git.kernel.org/will/c/10e4968cd511
[7/9] iommu/arm-smmu-v3: Cleanup arm_smmu_domain_finalise
      https://git.kernel.org/will/c/5e14313df2c8
[8/9] iommu/arm-smmu-v3: Update comment about STE liveness
      https://git.kernel.org/will/c/6032f58498b7
[9/9] iommu/arm-smmu-v3: Rename cdcfg to cd_table
      https://git.kernel.org/will/c/475918e9c4eb

Cheers,