mbox series

[v3,0/5] iommu/amd: AVIC Interrupt Remapping Improvements

Message ID 20230530141137.14376-1-suravee.suthikulpanit@amd.com
Headers
Series iommu/amd: AVIC Interrupt Remapping Improvements |

Message

Suravee Suthikulpanit May 30, 2023, 2:11 p.m. UTC 
  For IOMMU AVIC, the IOMMU driver needs to keep track of vcpu scheduling
changes, and updates interrupt remapping table entry (IRTE) accordingly.
The IRTE is normally cached by the hardware, which requires the IOMMU
driver to issue IOMMU IRT invalidation command and wait for completion
everytime it updates the table.

Enabling IOMMU AVIC on a large scale system with lots of vcpus and
VFIO pass-through devices running interrupt-intensive workload,
it could result in high IRT invalidation rate. In such case, the overhead
from IRT invalidation could outweigh the benefit of IRTE caching.

Therefore, introduce a new AMD IOMMU driver option "amd_iommu=irtcachedis"
to allow disabling IRTE caching, and avoid the need for IRTE invalidation.

Patch 1,2 prepare the AMD IOMMU driver to support IRT cache disabling.
Patch 3,4 introduce IRT cache disabling support
Patch 5 improves the code path in IOMMU driver for updating vcpu scheduling
for AVIC.

Thank you,
Suravee

Changes from V2
(https://lore.kernel.org/linux-iommu/rlurmw6n6eyyhtnfr6wva6azur2gvgcrdn4mvykr3nvsosj5py@ieaivyv6cqrv/T/)
* Added Reviewed-by and Sign-off-by.
* Patch 4: Reword the commit summary (per Jerry suggestion).

Changes from V1
(https://lore.kernel.org/lkml/20230509111646.369661-1-suravee.suthikulpanit@amd.com/T/)
* Patch 3: Add logic to clean up the IRTE cache disabling
  and handle kdump code path (per Alejandro)

Joao Martins (1):
  iommu/amd: Switch amd_iommu_update_ga() to use modify_irte_ga()

Suravee Suthikulpanit (4):
  iommu/amd: Remove the unused struct amd_ir_data.ref
  iommu/amd: Introduce Disable IRTE Caching Support
  iommu/amd: Do not Invalidate IRT when IRTE caching is disabled
  iommu/amd: Improving Interrupt Remapping Table Invalidation

 .../admin-guide/kernel-parameters.txt         |  1 +
 drivers/iommu/amd/amd_iommu_types.h           |  7 +-
 drivers/iommu/amd/init.c                      | 38 +++++++-
 drivers/iommu/amd/iommu.c                     | 97 ++++++++++---------
 4 files changed, 94 insertions(+), 49 deletions(-)

Comments

Suravee Suthikulpanit June 6, 2023, 2:06 p.m. UTC | #1 
Hi Joerg,

Please let me know if you have any other concerns for this series.

Thanks,
Suravee

On 5/30/2023 9:11 PM, Suravee Suthikulpanit wrote:
> For IOMMU AVIC, the IOMMU driver needs to keep track of vcpu scheduling
> changes, and updates interrupt remapping table entry (IRTE) accordingly.
> The IRTE is normally cached by the hardware, which requires the IOMMU
> driver to issue IOMMU IRT invalidation command and wait for completion
> everytime it updates the table.
> 
> Enabling IOMMU AVIC on a large scale system with lots of vcpus and
> VFIO pass-through devices running interrupt-intensive workload,
> it could result in high IRT invalidation rate. In such case, the overhead
> from IRT invalidation could outweigh the benefit of IRTE caching.
> 
> Therefore, introduce a new AMD IOMMU driver option "amd_iommu=irtcachedis"
> to allow disabling IRTE caching, and avoid the need for IRTE invalidation.
> 
> Patch 1,2 prepare the AMD IOMMU driver to support IRT cache disabling.
> Patch 3,4 introduce IRT cache disabling support
> Patch 5 improves the code path in IOMMU driver for updating vcpu scheduling
> for AVIC.
> 
> Thank you,
> Suravee
> 
> Changes from V2
> (https://lore.kernel.org/linux-iommu/rlurmw6n6eyyhtnfr6wva6azur2gvgcrdn4mvykr3nvsosj5py@ieaivyv6cqrv/T/)
> * Added Reviewed-by and Sign-off-by.
> * Patch 4: Reword the commit summary (per Jerry suggestion).
> 
> Changes from V1
> (https://lore.kernel.org/lkml/20230509111646.369661-1-suravee.suthikulpanit@amd.com/T/)
> * Patch 3: Add logic to clean up the IRTE cache disabling
>    and handle kdump code path (per Alejandro)
> 
> Joao Martins (1):
>    iommu/amd: Switch amd_iommu_update_ga() to use modify_irte_ga()
> 
> Suravee Suthikulpanit (4):
>    iommu/amd: Remove the unused struct amd_ir_data.ref
>    iommu/amd: Introduce Disable IRTE Caching Support
>    iommu/amd: Do not Invalidate IRT when IRTE caching is disabled
>    iommu/amd: Improving Interrupt Remapping Table Invalidation
> 
>   .../admin-guide/kernel-parameters.txt         |  1 +
>   drivers/iommu/amd/amd_iommu_types.h           |  7 +-
>   drivers/iommu/amd/init.c                      | 38 +++++++-
>   drivers/iommu/amd/iommu.c                     | 97 ++++++++++---------
>   4 files changed, 94 insertions(+), 49 deletions(-)
>
Joerg Roedel June 9, 2023, 12:47 p.m. UTC | #2 
On Tue, May 30, 2023 at 10:11:32AM -0400, Suravee Suthikulpanit wrote:
> Suravee Suthikulpanit (4):
>   iommu/amd: Remove the unused struct amd_ir_data.ref
>   iommu/amd: Introduce Disable IRTE Caching Support
>   iommu/amd: Do not Invalidate IRT when IRTE caching is disabled
>   iommu/amd: Improving Interrupt Remapping Table Invalidation
> 
>  .../admin-guide/kernel-parameters.txt         |  1 +
>  drivers/iommu/amd/amd_iommu_types.h           |  7 +-
>  drivers/iommu/amd/init.c                      | 38 +++++++-
>  drivers/iommu/amd/iommu.c                     | 97 ++++++++++---------
>  4 files changed, 94 insertions(+), 49 deletions(-)

Applied, thanks.