mbox series

[0/5] Increase the number of IRQ descriptors for SPARSEIRQ

Message ID 20230130005725.3517597-1-sdonthineni@nvidia.com
Headers
Series Increase the number of IRQ descriptors for SPARSEIRQ |

Message

Shanker Donthineni Jan. 30, 2023, 12:57 a.m. UTC 
  The ARM64 architecture uses SPARSEIRQ with a default value of NR_IRQS,
which is set to 64. This means that only 64+8192 IRQ descriptors are
allowed, which may not be sufficient for modern ARM64 servers that
have a large number of IO devices and GIC hardware that supports
direct vSGI and vLPI injection features.

This limitation has caused issues when attempting to launch multiple
virtual machines with GICv4.1 features, resulting in the error message
'kvm_err("VPE IRQ allocation failure\n")'. The root cause of this issue
is the ~8K IRQ descriptor limit.

To address this issue, an initial proposal was made to define NR_IRQS
to 2^19 for ARM64. However, Marc Zyngier suggested implementing a
generic solution instead of hard-coded values. Thomas Gleixner advised
to use the maple tree data structure and provided most of the necessary
functions.

For more information, refer to the discussion thread at
https://lore.kernel.org/linux-arm-kernel/20230104023738.1258925-1-sdonthineni@nvidia.com/.

This patch series converts the static memory allocation to dynamic using
the maple tree, and increases the maximum number of IRQ descriptors to
INT_MAX from NR_IRQS+8192. This change has been tested on an ARM64 server
with CONFIG_SPARSE_IRQ=y, where 256 virtual machines were launched,
creating a total of 128K+ IRQ descriptors, and IRQ injection was verified.

Tested with v6.2-rc5 along with Maple-Tree RCU mode bug fixes, as per the
information available at this link: 
 https://lore.kernel.org/all/20230109205336.3665937-1-surenb@google.com/

 [PATCH 1/41]  maple_tree: Be more cautious about dead nodes
 [PATCH 2/41]  maple_tree: Detect dead nodes in mas_start()
 [PATCH 3/41]  maple_tree: Fix freeing of nodes in rcu mode
 [PATCH 4/41]  maple_tree: remove extra smp_wmb() from mas_dead_leaves()
 [PATCH 5/41]  maple_tree: Fix write memory barrier of nodes once dead for RCU mode
 [PATCH 6/41]  maple_tree: Add smp_rmb() to dead node detection

Shanker Donthineni (5):
  genirq: Use hlist for managing resend handlers
  genirq: Allocate IRQ descriptors at boot time for !SPARSEIRQ
  genirq: Introduce two helper functions
  genirq: Use the common function irq_expand_nr_irqs()
  genirq: Use the maple tree for IRQ descriptors management

 include/linux/irqdesc.h |   3 ++
 kernel/irq/chip.c       |   1 +
 kernel/irq/internals.h  |   5 +-
 kernel/irq/irqdesc.c    | 117 ++++++++++++++++++++++------------------
 kernel/irq/resend.c     |  36 +++++++------
 5 files changed, 94 insertions(+), 68 deletions(-)