mbox series

[0/9] Linux RISC-V AIA Support

Message ID 20221111044207.1478350-1-apatel@ventanamicro.com
Headers
Series Linux RISC-V AIA Support |

Message

Anup Patel Nov. 11, 2022, 4:41 a.m. UTC 
  The RISC-V AIA specification is now frozen as-per the RISC-V international
process. The latest frozen specifcation can be found at:
https://github.com/riscv/riscv-aia/releases/download/1.0-RC1/riscv-interrupts-1.0-RC1.pdf

At a high-level, the AIA specification adds three things:
1) AIA CSRs
   - Improved local interrupt support
2) Incoming Message Signaled Interrupt Controller (IMSIC)
   - Per-HART MSI controller
   - Support MSI virtualization
   - Support IPI along with virtualization
3) Advanced Platform-Level Interrupt Controller (APLIC)
   - Wired interrupt controller
   - In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
   - In Direct-mode, injects external interrupts directly into HARTs

For an overview of the AIA specification, refer the recent AIA virtualization
talk at KVM Forum 2022:
https://static.sched.com/hosted_files/kvmforum2022/a1/AIA_Virtualization_in_KVM_RISCV_final.pdf
https://www.youtube.com/watch?v=r071dL8Z0yo

This series adds required Linux irqchip drivers for AIA and it depends on
the recent "RISC-V IPI Improvements".
(Refer, https://lore.kernel.org/lkml/20221101143400.690000-1-apatel@ventanamicro.com/t/)

To test this series, use QEMU v7.1 (or higher) and OpenSBI v1.1 (or higher).

These patches can also be found in the riscv_aia_v1 branch at:
https://github.com/avpatel/linux.git

Anup Patel (9):
  RISC-V: Add AIA related CSR defines
  RISC-V: Detect AIA CSRs from ISA string
  irqchip/riscv-intc: Add support for RISC-V AIA
  dt-bindings: Add RISC-V incoming MSI controller bindings
  irqchip: Add RISC-V incoming MSI controller driver
  dt-bindings: Add RISC-V advanced PLIC bindings
  irqchip: Add RISC-V advanced PLIC driver
  RISC-V: Select APLIC and IMSIC drivers for QEMU virt machine
  MAINTAINERS: Add entry for RISC-V AIA drivers

 .../interrupt-controller/riscv,aplic.yaml     |  136 ++
 .../interrupt-controller/riscv,imsic.yaml     |  174 +++
 MAINTAINERS                                   |   12 +
 arch/riscv/Kconfig.socs                       |    2 +
 arch/riscv/include/asm/csr.h                  |   92 ++
 arch/riscv/include/asm/hwcap.h                |    8 +
 arch/riscv/kernel/cpu.c                       |    2 +
 arch/riscv/kernel/cpufeature.c                |    2 +
 drivers/irqchip/Kconfig                       |   32 +-
 drivers/irqchip/Makefile                      |    2 +
 drivers/irqchip/irq-riscv-aplic.c             |  656 +++++++++
 drivers/irqchip/irq-riscv-imsic.c             | 1207 +++++++++++++++++
 drivers/irqchip/irq-riscv-intc.c              |   37 +-
 include/linux/irqchip/riscv-aplic.h           |  117 ++
 include/linux/irqchip/riscv-imsic.h           |   92 ++
 15 files changed, 2564 insertions(+), 7 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/interrupt-controller/riscv,aplic.yaml
 create mode 100644 Documentation/devicetree/bindings/interrupt-controller/riscv,imsic.yaml
 create mode 100644 drivers/irqchip/irq-riscv-aplic.c
 create mode 100644 drivers/irqchip/irq-riscv-imsic.c
 create mode 100644 include/linux/irqchip/riscv-aplic.h
 create mode 100644 include/linux/irqchip/riscv-imsic.h

Comments

Atish Patra Nov. 11, 2022, 9:07 a.m. UTC | #1 
On Thu, Nov 10, 2022 at 8:42 PM Anup Patel <apatel@ventanamicro.com> wrote:
>
> The RISC-V AIA specification is now frozen as-per the RISC-V international
> process. The latest frozen specifcation can be found at:
> https://github.com/riscv/riscv-aia/releases/download/1.0-RC1/riscv-interrupts-1.0-RC1.pdf
>
> At a high-level, the AIA specification adds three things:
> 1) AIA CSRs
>    - Improved local interrupt support
> 2) Incoming Message Signaled Interrupt Controller (IMSIC)
>    - Per-HART MSI controller
>    - Support MSI virtualization
>    - Support IPI along with virtualization
> 3) Advanced Platform-Level Interrupt Controller (APLIC)
>    - Wired interrupt controller
>    - In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
>    - In Direct-mode, injects external interrupts directly into HARTs
>
> For an overview of the AIA specification, refer the recent AIA virtualization
> talk at KVM Forum 2022:
> https://static.sched.com/hosted_files/kvmforum2022/a1/AIA_Virtualization_in_KVM_RISCV_final.pdf
> https://www.youtube.com/watch?v=r071dL8Z0yo
>
> This series adds required Linux irqchip drivers for AIA and it depends on
> the recent "RISC-V IPI Improvements".
> (Refer, https://lore.kernel.org/lkml/20221101143400.690000-1-apatel@ventanamicro.com/t/)
>
> To test this series, use QEMU v7.1 (or higher) and OpenSBI v1.1 (or higher).
>
> These patches can also be found in the riscv_aia_v1 branch at:
> https://github.com/avpatel/linux.git
>
> Anup Patel (9):
>   RISC-V: Add AIA related CSR defines
>   RISC-V: Detect AIA CSRs from ISA string
>   irqchip/riscv-intc: Add support for RISC-V AIA
>   dt-bindings: Add RISC-V incoming MSI controller bindings
>   irqchip: Add RISC-V incoming MSI controller driver
>   dt-bindings: Add RISC-V advanced PLIC bindings
>   irqchip: Add RISC-V advanced PLIC driver
>   RISC-V: Select APLIC and IMSIC drivers for QEMU virt machine
>   MAINTAINERS: Add entry for RISC-V AIA drivers
>
>  .../interrupt-controller/riscv,aplic.yaml     |  136 ++
>  .../interrupt-controller/riscv,imsic.yaml     |  174 +++
>  MAINTAINERS                                   |   12 +
>  arch/riscv/Kconfig.socs                       |    2 +
>  arch/riscv/include/asm/csr.h                  |   92 ++
>  arch/riscv/include/asm/hwcap.h                |    8 +
>  arch/riscv/kernel/cpu.c                       |    2 +
>  arch/riscv/kernel/cpufeature.c                |    2 +
>  drivers/irqchip/Kconfig                       |   32 +-
>  drivers/irqchip/Makefile                      |    2 +
>  drivers/irqchip/irq-riscv-aplic.c             |  656 +++++++++
>  drivers/irqchip/irq-riscv-imsic.c             | 1207 +++++++++++++++++
>  drivers/irqchip/irq-riscv-intc.c              |   37 +-
>  include/linux/irqchip/riscv-aplic.h           |  117 ++
>  include/linux/irqchip/riscv-imsic.h           |   92 ++
>  15 files changed, 2564 insertions(+), 7 deletions(-)
>  create mode 100644 Documentation/devicetree/bindings/interrupt-controller/riscv,aplic.yaml
>  create mode 100644 Documentation/devicetree/bindings/interrupt-controller/riscv,imsic.yaml
>  create mode 100644 drivers/irqchip/irq-riscv-aplic.c
>  create mode 100644 drivers/irqchip/irq-riscv-imsic.c
>  create mode 100644 include/linux/irqchip/riscv-aplic.h
>  create mode 100644 include/linux/irqchip/riscv-imsic.h
>
> --
> 2.34.1
>

I am seeing the following boot failure with your branch and upstream
qemu (tag: v7.2.0-rc0).
It seems IPIs are probably not getting delivered after a point. I saw
that IPIs are delivered in the same path earlier (via gdb).

[    0.990152] NET: Registered PF_INET6 protocol family
[    1.004885] Segment Routing with IPv6
[    1.005385] In-situ OAM (IOAM) with IPv6
[    1.006371] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    1.011609] NET: Registered PF_PACKET protocol family
[    1.014877] 9pnet: Installing 9P2000 support
[    1.015603] Key type dns_resolver registered
[    1.026396] debug_vm_pgtable: [debug_vm_pgtable         ]:
Validating architecture page table helpers
[    1.144097] EXT4-fs (vda2): recovery complete
[    1.146345] EXT4-fs (vda2): mounted filesystem with ordered data
mode. Quota mode: disabled.
[    1.147020] VFS: Mounted root (ext4 filesystem) on device 254:2.
[    1.152238] devtmpfs: mounted
[    1.183914] Freeing unused kernel image (initmem) memory: 2176K
[    1.185898] Run /sbin/init as init process
[   29.412287] rcu: INFO: rcu_sched self-detected stall on CPU
[   29.412992] rcu:     3-....: (5250 ticks this GP)
idle=fc3c/1/0x4000000000000002 softirq=78/78 fqs=2230
[   29.413427]  (t=5251 jiffies g=-1047 q=3 ncpus=8)
[   29.414199] CPU: 3 PID: 1 Comm: init Not tainted
6.1.0-rc4-00024-g5b711f2d7b91 #198
[   29.414578] Hardware name: riscv-virtio,qemu (DT)
[   29.414896] epc : smp_call_function_many_cond+0x138/0x372
[   29.415157]  ra : smp_call_function_many_cond+0x154/0x372
[   29.415318] epc : ffffffff8008d2e6 ra : ffffffff8008d302 sp :
ff2000000004bb40
[   29.415512]  gp : ffffffff812e9eb0 tp : ff600000016d8000 t0 :
ff6000007ed88610
[   29.415686]  t1 : 00000000000000ff t2 : 0000000000000002 s0 :
ff2000000004bc00
[   29.415837]  s1 : ff6000007ed85448 a0 : 0000000000000007 a1 :
00000000000000f7
[   29.416008]  a2 : 0000000000000000 a3 : 0000000000000000 a4 :
ff6000007edd1780
[   29.416188]  a5 : 0000000000000001 a6 : ffffffff812eb1c0 a7 :
ff600000016d8000
[   29.416613]  s2 : ffffffff81323c30 s3 : ffffffff812e9964 s4 :
0000000000000000
[   29.416810]  s5 : 0000000000000000 s6 : ff6000007ed85440 s7 :
0000000000000038
[   29.416997]  s8 : 0000000000000003 s9 : ffffffff81323c30 s10:
ff6000007ed85448
[   29.417159]  s11: 0000000000000008 t3 : 00ffffffad08a000 t4 :
ff60000001613e0c
[   29.417331]  t5 : 0000000000000000 t6 : 00ffffffad177fff
[   29.417482] status: 0000000200000120 badaddr: 0000000000000000
cause: 8000000000000005
[   29.417939] [<ffffffff8008d590>] on_each_cpu_cond_mask+0x20/0x32
[   29.418179] [<ffffffff80008d32>] flush_icache_all+0x38/0x40
[   29.418324] [<ffffffff80008eb2>] flush_icache_pte+0x4a/0x7a
[   29.418442] [<ffffffff80139e00>] do_set_pte+0x132/0x192
[   29.418594] [<ffffffff8010b42e>] filemap_map_pages+0x178/0x3a0
[   29.418738] [<ffffffff8013ad00>] __handle_mm_fault+0x992/0xbac
[   29.418876] [<ffffffff8013afde>] handle_mm_fault+0xc4/0x1d4
[   29.419010] [<ffffffff80008372>] do_page_fault+0x120/0x326
[   29.419145] [<ffffffff800033e6>] ret_from_exception+0x0/0xc
Atish Patra Nov. 11, 2022, 9:13 a.m. UTC | #2 
On Fri, Nov 11, 2022 at 1:07 AM Atish Patra <atishp@atishpatra.org> wrote:
>
> On Thu, Nov 10, 2022 at 8:42 PM Anup Patel <apatel@ventanamicro.com> wrote:
> >
> > The RISC-V AIA specification is now frozen as-per the RISC-V international
> > process. The latest frozen specifcation can be found at:
> > https://github.com/riscv/riscv-aia/releases/download/1.0-RC1/riscv-interrupts-1.0-RC1.pdf
> >
> > At a high-level, the AIA specification adds three things:
> > 1) AIA CSRs
> >    - Improved local interrupt support
> > 2) Incoming Message Signaled Interrupt Controller (IMSIC)
> >    - Per-HART MSI controller
> >    - Support MSI virtualization
> >    - Support IPI along with virtualization
> > 3) Advanced Platform-Level Interrupt Controller (APLIC)
> >    - Wired interrupt controller
> >    - In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
> >    - In Direct-mode, injects external interrupts directly into HARTs
> >
> > For an overview of the AIA specification, refer the recent AIA virtualization
> > talk at KVM Forum 2022:
> > https://static.sched.com/hosted_files/kvmforum2022/a1/AIA_Virtualization_in_KVM_RISCV_final.pdf
> > https://www.youtube.com/watch?v=r071dL8Z0yo
> >
> > This series adds required Linux irqchip drivers for AIA and it depends on
> > the recent "RISC-V IPI Improvements".
> > (Refer, https://lore.kernel.org/lkml/20221101143400.690000-1-apatel@ventanamicro.com/t/)
> >
> > To test this series, use QEMU v7.1 (or higher) and OpenSBI v1.1 (or higher).
> >
> > These patches can also be found in the riscv_aia_v1 branch at:
> > https://github.com/avpatel/linux.git
> >
> > Anup Patel (9):
> >   RISC-V: Add AIA related CSR defines
> >   RISC-V: Detect AIA CSRs from ISA string
> >   irqchip/riscv-intc: Add support for RISC-V AIA
> >   dt-bindings: Add RISC-V incoming MSI controller bindings
> >   irqchip: Add RISC-V incoming MSI controller driver
> >   dt-bindings: Add RISC-V advanced PLIC bindings
> >   irqchip: Add RISC-V advanced PLIC driver
> >   RISC-V: Select APLIC and IMSIC drivers for QEMU virt machine
> >   MAINTAINERS: Add entry for RISC-V AIA drivers
> >
> >  .../interrupt-controller/riscv,aplic.yaml     |  136 ++
> >  .../interrupt-controller/riscv,imsic.yaml     |  174 +++
> >  MAINTAINERS                                   |   12 +
> >  arch/riscv/Kconfig.socs                       |    2 +
> >  arch/riscv/include/asm/csr.h                  |   92 ++
> >  arch/riscv/include/asm/hwcap.h                |    8 +
> >  arch/riscv/kernel/cpu.c                       |    2 +
> >  arch/riscv/kernel/cpufeature.c                |    2 +
> >  drivers/irqchip/Kconfig                       |   32 +-
> >  drivers/irqchip/Makefile                      |    2 +
> >  drivers/irqchip/irq-riscv-aplic.c             |  656 +++++++++
> >  drivers/irqchip/irq-riscv-imsic.c             | 1207 +++++++++++++++++
> >  drivers/irqchip/irq-riscv-intc.c              |   37 +-
> >  include/linux/irqchip/riscv-aplic.h           |  117 ++
> >  include/linux/irqchip/riscv-imsic.h           |   92 ++
> >  15 files changed, 2564 insertions(+), 7 deletions(-)
> >  create mode 100644 Documentation/devicetree/bindings/interrupt-controller/riscv,aplic.yaml
> >  create mode 100644 Documentation/devicetree/bindings/interrupt-controller/riscv,imsic.yaml
> >  create mode 100644 drivers/irqchip/irq-riscv-aplic.c
> >  create mode 100644 drivers/irqchip/irq-riscv-imsic.c
> >  create mode 100644 include/linux/irqchip/riscv-aplic.h
> >  create mode 100644 include/linux/irqchip/riscv-imsic.h
> >
> > --
> > 2.34.1
> >
>
> I am seeing the following boot failure with your branch and upstream
> qemu (tag: v7.2.0-rc0).
> It seems IPIs are probably not getting delivered after a point. I saw
> that IPIs are delivered in the same path earlier (via gdb).
>
> [    0.990152] NET: Registered PF_INET6 protocol family
> [    1.004885] Segment Routing with IPv6
> [    1.005385] In-situ OAM (IOAM) with IPv6
> [    1.006371] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
> [    1.011609] NET: Registered PF_PACKET protocol family
> [    1.014877] 9pnet: Installing 9P2000 support
> [    1.015603] Key type dns_resolver registered
> [    1.026396] debug_vm_pgtable: [debug_vm_pgtable         ]:
> Validating architecture page table helpers
> [    1.144097] EXT4-fs (vda2): recovery complete
> [    1.146345] EXT4-fs (vda2): mounted filesystem with ordered data
> mode. Quota mode: disabled.
> [    1.147020] VFS: Mounted root (ext4 filesystem) on device 254:2.
> [    1.152238] devtmpfs: mounted
> [    1.183914] Freeing unused kernel image (initmem) memory: 2176K
> [    1.185898] Run /sbin/init as init process
> [   29.412287] rcu: INFO: rcu_sched self-detected stall on CPU
> [   29.412992] rcu:     3-....: (5250 ticks this GP)
> idle=fc3c/1/0x4000000000000002 softirq=78/78 fqs=2230
> [   29.413427]  (t=5251 jiffies g=-1047 q=3 ncpus=8)
> [   29.414199] CPU: 3 PID: 1 Comm: init Not tainted
> 6.1.0-rc4-00024-g5b711f2d7b91 #198
> [   29.414578] Hardware name: riscv-virtio,qemu (DT)
> [   29.414896] epc : smp_call_function_many_cond+0x138/0x372
> [   29.415157]  ra : smp_call_function_many_cond+0x154/0x372
> [   29.415318] epc : ffffffff8008d2e6 ra : ffffffff8008d302 sp :
> ff2000000004bb40
> [   29.415512]  gp : ffffffff812e9eb0 tp : ff600000016d8000 t0 :
> ff6000007ed88610
> [   29.415686]  t1 : 00000000000000ff t2 : 0000000000000002 s0 :
> ff2000000004bc00
> [   29.415837]  s1 : ff6000007ed85448 a0 : 0000000000000007 a1 :
> 00000000000000f7
> [   29.416008]  a2 : 0000000000000000 a3 : 0000000000000000 a4 :
> ff6000007edd1780
> [   29.416188]  a5 : 0000000000000001 a6 : ffffffff812eb1c0 a7 :
> ff600000016d8000
> [   29.416613]  s2 : ffffffff81323c30 s3 : ffffffff812e9964 s4 :
> 0000000000000000
> [   29.416810]  s5 : 0000000000000000 s6 : ff6000007ed85440 s7 :
> 0000000000000038
> [   29.416997]  s8 : 0000000000000003 s9 : ffffffff81323c30 s10:
> ff6000007ed85448
> [   29.417159]  s11: 0000000000000008 t3 : 00ffffffad08a000 t4 :
> ff60000001613e0c
> [   29.417331]  t5 : 0000000000000000 t6 : 00ffffffad177fff
> [   29.417482] status: 0000000200000120 badaddr: 0000000000000000
> cause: 8000000000000005
> [   29.417939] [<ffffffff8008d590>] on_each_cpu_cond_mask+0x20/0x32
> [   29.418179] [<ffffffff80008d32>] flush_icache_all+0x38/0x40
> [   29.418324] [<ffffffff80008eb2>] flush_icache_pte+0x4a/0x7a
> [   29.418442] [<ffffffff80139e00>] do_set_pte+0x132/0x192
> [   29.418594] [<ffffffff8010b42e>] filemap_map_pages+0x178/0x3a0
> [   29.418738] [<ffffffff8013ad00>] __handle_mm_fault+0x992/0xbac
> [   29.418876] [<ffffffff8013afde>] handle_mm_fault+0xc4/0x1d4
> [   29.419010] [<ffffffff80008372>] do_page_fault+0x120/0x326
> [   29.419145] [<ffffffff800033e6>] ret_from_exception+0x0/0xc
>
>

Sorry. I forgot to mention that this happens while booting Fedora on
an 8 cpu virt machine.
I am yet to reproduce this issue for 4 or 2 cpus.

>
> --
> Regards,
> Atish
Atish Patra Nov. 11, 2022, 7:01 p.m. UTC | #3 
On Fri, Nov 11, 2022 at 1:13 AM Atish Patra <atishp@atishpatra.org> wrote:
>
> On Fri, Nov 11, 2022 at 1:07 AM Atish Patra <atishp@atishpatra.org> wrote:
> >
> > On Thu, Nov 10, 2022 at 8:42 PM Anup Patel <apatel@ventanamicro.com> wrote:
> > >
> > > The RISC-V AIA specification is now frozen as-per the RISC-V international
> > > process. The latest frozen specifcation can be found at:
> > > https://github.com/riscv/riscv-aia/releases/download/1.0-RC1/riscv-interrupts-1.0-RC1.pdf
> > >
> > > At a high-level, the AIA specification adds three things:
> > > 1) AIA CSRs
> > >    - Improved local interrupt support
> > > 2) Incoming Message Signaled Interrupt Controller (IMSIC)
> > >    - Per-HART MSI controller
> > >    - Support MSI virtualization
> > >    - Support IPI along with virtualization
> > > 3) Advanced Platform-Level Interrupt Controller (APLIC)
> > >    - Wired interrupt controller
> > >    - In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
> > >    - In Direct-mode, injects external interrupts directly into HARTs
> > >
> > > For an overview of the AIA specification, refer the recent AIA virtualization
> > > talk at KVM Forum 2022:
> > > https://static.sched.com/hosted_files/kvmforum2022/a1/AIA_Virtualization_in_KVM_RISCV_final.pdf
> > > https://www.youtube.com/watch?v=r071dL8Z0yo
> > >
> > > This series adds required Linux irqchip drivers for AIA and it depends on
> > > the recent "RISC-V IPI Improvements".
> > > (Refer, https://lore.kernel.org/lkml/20221101143400.690000-1-apatel@ventanamicro.com/t/)
> > >
> > > To test this series, use QEMU v7.1 (or higher) and OpenSBI v1.1 (or higher).
> > >
> > > These patches can also be found in the riscv_aia_v1 branch at:
> > > https://github.com/avpatel/linux.git
> > >
> > > Anup Patel (9):
> > >   RISC-V: Add AIA related CSR defines
> > >   RISC-V: Detect AIA CSRs from ISA string
> > >   irqchip/riscv-intc: Add support for RISC-V AIA
> > >   dt-bindings: Add RISC-V incoming MSI controller bindings
> > >   irqchip: Add RISC-V incoming MSI controller driver
> > >   dt-bindings: Add RISC-V advanced PLIC bindings
> > >   irqchip: Add RISC-V advanced PLIC driver
> > >   RISC-V: Select APLIC and IMSIC drivers for QEMU virt machine
> > >   MAINTAINERS: Add entry for RISC-V AIA drivers
> > >
> > >  .../interrupt-controller/riscv,aplic.yaml     |  136 ++
> > >  .../interrupt-controller/riscv,imsic.yaml     |  174 +++
> > >  MAINTAINERS                                   |   12 +
> > >  arch/riscv/Kconfig.socs                       |    2 +
> > >  arch/riscv/include/asm/csr.h                  |   92 ++
> > >  arch/riscv/include/asm/hwcap.h                |    8 +
> > >  arch/riscv/kernel/cpu.c                       |    2 +
> > >  arch/riscv/kernel/cpufeature.c                |    2 +
> > >  drivers/irqchip/Kconfig                       |   32 +-
> > >  drivers/irqchip/Makefile                      |    2 +
> > >  drivers/irqchip/irq-riscv-aplic.c             |  656 +++++++++
> > >  drivers/irqchip/irq-riscv-imsic.c             | 1207 +++++++++++++++++
> > >  drivers/irqchip/irq-riscv-intc.c              |   37 +-
> > >  include/linux/irqchip/riscv-aplic.h           |  117 ++
> > >  include/linux/irqchip/riscv-imsic.h           |   92 ++
> > >  15 files changed, 2564 insertions(+), 7 deletions(-)
> > >  create mode 100644 Documentation/devicetree/bindings/interrupt-controller/riscv,aplic.yaml
> > >  create mode 100644 Documentation/devicetree/bindings/interrupt-controller/riscv,imsic.yaml
> > >  create mode 100644 drivers/irqchip/irq-riscv-aplic.c
> > >  create mode 100644 drivers/irqchip/irq-riscv-imsic.c
> > >  create mode 100644 include/linux/irqchip/riscv-aplic.h
> > >  create mode 100644 include/linux/irqchip/riscv-imsic.h
> > >
> > > --
> > > 2.34.1
> > >
> >
> > I am seeing the following boot failure with your branch and upstream
> > qemu (tag: v7.2.0-rc0).
> > It seems IPIs are probably not getting delivered after a point. I saw
> > that IPIs are delivered in the same path earlier (via gdb).
> >
> > [    0.990152] NET: Registered PF_INET6 protocol family
> > [    1.004885] Segment Routing with IPv6
> > [    1.005385] In-situ OAM (IOAM) with IPv6
> > [    1.006371] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
> > [    1.011609] NET: Registered PF_PACKET protocol family
> > [    1.014877] 9pnet: Installing 9P2000 support
> > [    1.015603] Key type dns_resolver registered
> > [    1.026396] debug_vm_pgtable: [debug_vm_pgtable         ]:
> > Validating architecture page table helpers
> > [    1.144097] EXT4-fs (vda2): recovery complete
> > [    1.146345] EXT4-fs (vda2): mounted filesystem with ordered data
> > mode. Quota mode: disabled.
> > [    1.147020] VFS: Mounted root (ext4 filesystem) on device 254:2.
> > [    1.152238] devtmpfs: mounted
> > [    1.183914] Freeing unused kernel image (initmem) memory: 2176K
> > [    1.185898] Run /sbin/init as init process
> > [   29.412287] rcu: INFO: rcu_sched self-detected stall on CPU
> > [   29.412992] rcu:     3-....: (5250 ticks this GP)
> > idle=fc3c/1/0x4000000000000002 softirq=78/78 fqs=2230
> > [   29.413427]  (t=5251 jiffies g=-1047 q=3 ncpus=8)
> > [   29.414199] CPU: 3 PID: 1 Comm: init Not tainted
> > 6.1.0-rc4-00024-g5b711f2d7b91 #198
> > [   29.414578] Hardware name: riscv-virtio,qemu (DT)
> > [   29.414896] epc : smp_call_function_many_cond+0x138/0x372
> > [   29.415157]  ra : smp_call_function_many_cond+0x154/0x372
> > [   29.415318] epc : ffffffff8008d2e6 ra : ffffffff8008d302 sp :
> > ff2000000004bb40
> > [   29.415512]  gp : ffffffff812e9eb0 tp : ff600000016d8000 t0 :
> > ff6000007ed88610
> > [   29.415686]  t1 : 00000000000000ff t2 : 0000000000000002 s0 :
> > ff2000000004bc00
> > [   29.415837]  s1 : ff6000007ed85448 a0 : 0000000000000007 a1 :
> > 00000000000000f7
> > [   29.416008]  a2 : 0000000000000000 a3 : 0000000000000000 a4 :
> > ff6000007edd1780
> > [   29.416188]  a5 : 0000000000000001 a6 : ffffffff812eb1c0 a7 :
> > ff600000016d8000
> > [   29.416613]  s2 : ffffffff81323c30 s3 : ffffffff812e9964 s4 :
> > 0000000000000000
> > [   29.416810]  s5 : 0000000000000000 s6 : ff6000007ed85440 s7 :
> > 0000000000000038
> > [   29.416997]  s8 : 0000000000000003 s9 : ffffffff81323c30 s10:
> > ff6000007ed85448
> > [   29.417159]  s11: 0000000000000008 t3 : 00ffffffad08a000 t4 :
> > ff60000001613e0c
> > [   29.417331]  t5 : 0000000000000000 t6 : 00ffffffad177fff
> > [   29.417482] status: 0000000200000120 badaddr: 0000000000000000
> > cause: 8000000000000005
> > [   29.417939] [<ffffffff8008d590>] on_each_cpu_cond_mask+0x20/0x32
> > [   29.418179] [<ffffffff80008d32>] flush_icache_all+0x38/0x40
> > [   29.418324] [<ffffffff80008eb2>] flush_icache_pte+0x4a/0x7a
> > [   29.418442] [<ffffffff80139e00>] do_set_pte+0x132/0x192
> > [   29.418594] [<ffffffff8010b42e>] filemap_map_pages+0x178/0x3a0
> > [   29.418738] [<ffffffff8013ad00>] __handle_mm_fault+0x992/0xbac
> > [   29.418876] [<ffffffff8013afde>] handle_mm_fault+0xc4/0x1d4
> > [   29.419010] [<ffffffff80008372>] do_page_fault+0x120/0x326
> > [   29.419145] [<ffffffff800033e6>] ret_from_exception+0x0/0xc
> >
> >
>
> Sorry. I forgot to mention that this happens while booting Fedora on
> an 8 cpu virt machine.
> I am yet to reproduce this issue for 4 or 2 cpus.
>

This warning splat appears with 4 cpus though.

[  126.977213] e1000e 0000:00:02.0 enp0s2: NIC Link is Up 1000 Mbps
Full Duplex, Flow Control: Rx/Tx
[  126.988275] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s2: link becomes ready
[  128.210090] ------------[ cut here ]------------
[  128.210476] WARNING: CPU: 0 PID: 294 at
drivers/irqchip/irq-riscv-aplic.c:318 aplic_msi_write_msg+0x110/0x12a
[  128.211702] Modules linked in:
[  128.212473] CPU: 0 PID: 294 Comm: (agetty) Not tainted
6.1.0-rc4-00024-g5b711f2d7b91 #198
[  128.212839] Hardware name: riscv-virtio,qemu (DT)
[  128.213459] epc : aplic_msi_write_msg+0x110/0x12a
[  128.213652]  ra : aplic_msi_write_msg+0x1c/0x12a
[  128.213823] epc : ffffffff8039eb80 ra : ffffffff8039ea8c sp :
ff2000001054bb10
[  128.213884]  gp : ffffffff812e9eb0 tp : ff600000032d3840 t0 :
0000000000000002
[  128.213945]  t1 : 0000000000000008 t2 : 0000000000000040 s0 :
ff2000001054bb30
[  128.214015]  s1 : ff600000018443a8 a0 : ff600000017e8c30 a1 :
0000000000000000
[  128.214083]  a2 : 0000000000000001 a3 : 000000000000000c a4 :
0000000000000003
[  128.214141]  a5 : 0000000000000000 a6 : 0000000000000000 a7 :
0000000000000000
[  128.214173]  s2 : ff2000001054bb40 s3 : ff600000017e8c30 s4 :
0000000200000022
[  128.214246]  s5 : ff600000017e8d78 s6 : 000000000000000c s7 :
ff600000017e8cb0
[  128.214295]  s8 : ffffffff80c5c8f0 s9 : ff6000000e97ab60 s10:
0000000000000000
[  128.214327]  s11: ff6000000e97ab78 t3 : 0000000000000004 t4 :
0000000000000002
[  128.214383]  t5 : 0000000000000000 t6 : 0000000000028000
[  128.214414] status: 0000000200000100 badaddr: 0000000000000000
cause: 0000000000000003
[  128.215267] [<ffffffff80453060>] platform_msi_write_msg+0x18/0x20
[  128.215413] [<ffffffff80060b7a>] msi_domain_deactivate+0x2a/0x48
[  128.215432] [<ffffffff8005e5a2>] __irq_domain_deactivate_irq+0x1c/0x30
[  128.215450] [<ffffffff8005e5aa>] __irq_domain_deactivate_irq+0x24/0x30
[  128.215464] [<ffffffff8005f9ea>] irq_domain_deactivate_irq+0x1c/0x30
[  128.215477] [<ffffffff8005b8f6>] free_irq+0x1d0/0x27c
[  128.219393] [<ffffffff8041be06>] univ8250_release_irq+0xa4/0xaa
[  128.219417] [<ffffffff8041e018>] serial8250_do_shutdown+0xf0/0x166
[  128.219431] [<ffffffff8041e0a8>] serial8250_shutdown+0x1a/0x22
[  128.219444] [<ffffffff804174f0>] uart_port_shutdown+0x2c/0x42
[  128.220230] [<ffffffff8041908a>] uart_shutdown+0x66/0x11e
[  128.220247] [<ffffffff8041a69a>] uart_hangup+0x5e/0x100
[  128.220260] [<ffffffff803fdaa2>] __tty_hangup.part.0+0x286/0x2ac
[  128.220859] [<ffffffff803fec6c>] tty_ioctl+0x684/0x802
[  128.220875] [<ffffffff8019ec48>] sys_ioctl+0x37e/0x95e
[  128.221284] [<ffffffff800033d8>] ret_from_syscall+0x0/0x2
[  128.221653] ---[ end trace 0000000000000000 ]---


> >
> > --
> > Regards,
> > Atish
>
>
>
> --
> Regards,
> Atish
Anup Patel Jan. 2, 2023, 10:05 a.m. UTC | #4 
On Fri, Nov 11, 2022 at 2:37 PM Atish Patra <atishp@atishpatra.org> wrote:
>
> On Thu, Nov 10, 2022 at 8:42 PM Anup Patel <apatel@ventanamicro.com> wrote:
> >
> > The RISC-V AIA specification is now frozen as-per the RISC-V international
> > process. The latest frozen specifcation can be found at:
> > https://github.com/riscv/riscv-aia/releases/download/1.0-RC1/riscv-interrupts-1.0-RC1.pdf
> >
> > At a high-level, the AIA specification adds three things:
> > 1) AIA CSRs
> >    - Improved local interrupt support
> > 2) Incoming Message Signaled Interrupt Controller (IMSIC)
> >    - Per-HART MSI controller
> >    - Support MSI virtualization
> >    - Support IPI along with virtualization
> > 3) Advanced Platform-Level Interrupt Controller (APLIC)
> >    - Wired interrupt controller
> >    - In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
> >    - In Direct-mode, injects external interrupts directly into HARTs
> >
> > For an overview of the AIA specification, refer the recent AIA virtualization
> > talk at KVM Forum 2022:
> > https://static.sched.com/hosted_files/kvmforum2022/a1/AIA_Virtualization_in_KVM_RISCV_final.pdf
> > https://www.youtube.com/watch?v=r071dL8Z0yo
> >
> > This series adds required Linux irqchip drivers for AIA and it depends on
> > the recent "RISC-V IPI Improvements".
> > (Refer, https://lore.kernel.org/lkml/20221101143400.690000-1-apatel@ventanamicro.com/t/)
> >
> > To test this series, use QEMU v7.1 (or higher) and OpenSBI v1.1 (or higher).
> >
> > These patches can also be found in the riscv_aia_v1 branch at:
> > https://github.com/avpatel/linux.git
> >
> > Anup Patel (9):
> >   RISC-V: Add AIA related CSR defines
> >   RISC-V: Detect AIA CSRs from ISA string
> >   irqchip/riscv-intc: Add support for RISC-V AIA
> >   dt-bindings: Add RISC-V incoming MSI controller bindings
> >   irqchip: Add RISC-V incoming MSI controller driver
> >   dt-bindings: Add RISC-V advanced PLIC bindings
> >   irqchip: Add RISC-V advanced PLIC driver
> >   RISC-V: Select APLIC and IMSIC drivers for QEMU virt machine
> >   MAINTAINERS: Add entry for RISC-V AIA drivers
> >
> >  .../interrupt-controller/riscv,aplic.yaml     |  136 ++
> >  .../interrupt-controller/riscv,imsic.yaml     |  174 +++
> >  MAINTAINERS                                   |   12 +
> >  arch/riscv/Kconfig.socs                       |    2 +
> >  arch/riscv/include/asm/csr.h                  |   92 ++
> >  arch/riscv/include/asm/hwcap.h                |    8 +
> >  arch/riscv/kernel/cpu.c                       |    2 +
> >  arch/riscv/kernel/cpufeature.c                |    2 +
> >  drivers/irqchip/Kconfig                       |   32 +-
> >  drivers/irqchip/Makefile                      |    2 +
> >  drivers/irqchip/irq-riscv-aplic.c             |  656 +++++++++
> >  drivers/irqchip/irq-riscv-imsic.c             | 1207 +++++++++++++++++
> >  drivers/irqchip/irq-riscv-intc.c              |   37 +-
> >  include/linux/irqchip/riscv-aplic.h           |  117 ++
> >  include/linux/irqchip/riscv-imsic.h           |   92 ++
> >  15 files changed, 2564 insertions(+), 7 deletions(-)
> >  create mode 100644 Documentation/devicetree/bindings/interrupt-controller/riscv,aplic.yaml
> >  create mode 100644 Documentation/devicetree/bindings/interrupt-controller/riscv,imsic.yaml
> >  create mode 100644 drivers/irqchip/irq-riscv-aplic.c
> >  create mode 100644 drivers/irqchip/irq-riscv-imsic.c
> >  create mode 100644 include/linux/irqchip/riscv-aplic.h
> >  create mode 100644 include/linux/irqchip/riscv-imsic.h
> >
> > --
> > 2.34.1
> >
>
> I am seeing the following boot failure with your branch and upstream
> qemu (tag: v7.2.0-rc0).
> It seems IPIs are probably not getting delivered after a point. I saw
> that IPIs are delivered in the same path earlier (via gdb).
>
> [    0.990152] NET: Registered PF_INET6 protocol family
> [    1.004885] Segment Routing with IPv6
> [    1.005385] In-situ OAM (IOAM) with IPv6
> [    1.006371] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
> [    1.011609] NET: Registered PF_PACKET protocol family
> [    1.014877] 9pnet: Installing 9P2000 support
> [    1.015603] Key type dns_resolver registered
> [    1.026396] debug_vm_pgtable: [debug_vm_pgtable         ]:
> Validating architecture page table helpers
> [    1.144097] EXT4-fs (vda2): recovery complete
> [    1.146345] EXT4-fs (vda2): mounted filesystem with ordered data
> mode. Quota mode: disabled.
> [    1.147020] VFS: Mounted root (ext4 filesystem) on device 254:2.
> [    1.152238] devtmpfs: mounted
> [    1.183914] Freeing unused kernel image (initmem) memory: 2176K
> [    1.185898] Run /sbin/init as init process
> [   29.412287] rcu: INFO: rcu_sched self-detected stall on CPU
> [   29.412992] rcu:     3-....: (5250 ticks this GP)
> idle=fc3c/1/0x4000000000000002 softirq=78/78 fqs=2230
> [   29.413427]  (t=5251 jiffies g=-1047 q=3 ncpus=8)
> [   29.414199] CPU: 3 PID: 1 Comm: init Not tainted
> 6.1.0-rc4-00024-g5b711f2d7b91 #198
> [   29.414578] Hardware name: riscv-virtio,qemu (DT)
> [   29.414896] epc : smp_call_function_many_cond+0x138/0x372
> [   29.415157]  ra : smp_call_function_many_cond+0x154/0x372
> [   29.415318] epc : ffffffff8008d2e6 ra : ffffffff8008d302 sp :
> ff2000000004bb40
> [   29.415512]  gp : ffffffff812e9eb0 tp : ff600000016d8000 t0 :
> ff6000007ed88610
> [   29.415686]  t1 : 00000000000000ff t2 : 0000000000000002 s0 :
> ff2000000004bc00
> [   29.415837]  s1 : ff6000007ed85448 a0 : 0000000000000007 a1 :
> 00000000000000f7
> [   29.416008]  a2 : 0000000000000000 a3 : 0000000000000000 a4 :
> ff6000007edd1780
> [   29.416188]  a5 : 0000000000000001 a6 : ffffffff812eb1c0 a7 :
> ff600000016d8000
> [   29.416613]  s2 : ffffffff81323c30 s3 : ffffffff812e9964 s4 :
> 0000000000000000
> [   29.416810]  s5 : 0000000000000000 s6 : ff6000007ed85440 s7 :
> 0000000000000038
> [   29.416997]  s8 : 0000000000000003 s9 : ffffffff81323c30 s10:
> ff6000007ed85448
> [   29.417159]  s11: 0000000000000008 t3 : 00ffffffad08a000 t4 :
> ff60000001613e0c
> [   29.417331]  t5 : 0000000000000000 t6 : 00ffffffad177fff
> [   29.417482] status: 0000000200000120 badaddr: 0000000000000000
> cause: 8000000000000005
> [   29.417939] [<ffffffff8008d590>] on_each_cpu_cond_mask+0x20/0x32
> [   29.418179] [<ffffffff80008d32>] flush_icache_all+0x38/0x40
> [   29.418324] [<ffffffff80008eb2>] flush_icache_pte+0x4a/0x7a
> [   29.418442] [<ffffffff80139e00>] do_set_pte+0x132/0x192
> [   29.418594] [<ffffffff8010b42e>] filemap_map_pages+0x178/0x3a0
> [   29.418738] [<ffffffff8013ad00>] __handle_mm_fault+0x992/0xbac
> [   29.418876] [<ffffffff8013afde>] handle_mm_fault+0xc4/0x1d4
> [   29.419010] [<ffffffff80008372>] do_page_fault+0x120/0x326
> [   29.419145] [<ffffffff800033e6>] ret_from_exception+0x0/0xc

This has been addressed in the RISC-V IPI improvement series.

Regards,
Anup

>
>
>
> --
> Regards,
> Atish
Anup Patel Jan. 2, 2023, 10:06 a.m. UTC | #5 
On Sat, Nov 12, 2022 at 12:31 AM Atish Patra <atishp@atishpatra.org> wrote:
>
> On Fri, Nov 11, 2022 at 1:13 AM Atish Patra <atishp@atishpatra.org> wrote:
> >
> > On Fri, Nov 11, 2022 at 1:07 AM Atish Patra <atishp@atishpatra.org> wrote:
> > >
> > > On Thu, Nov 10, 2022 at 8:42 PM Anup Patel <apatel@ventanamicro.com> wrote:
> > > >
> > > > The RISC-V AIA specification is now frozen as-per the RISC-V international
> > > > process. The latest frozen specifcation can be found at:
> > > > https://github.com/riscv/riscv-aia/releases/download/1.0-RC1/riscv-interrupts-1.0-RC1.pdf
> > > >
> > > > At a high-level, the AIA specification adds three things:
> > > > 1) AIA CSRs
> > > >    - Improved local interrupt support
> > > > 2) Incoming Message Signaled Interrupt Controller (IMSIC)
> > > >    - Per-HART MSI controller
> > > >    - Support MSI virtualization
> > > >    - Support IPI along with virtualization
> > > > 3) Advanced Platform-Level Interrupt Controller (APLIC)
> > > >    - Wired interrupt controller
> > > >    - In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
> > > >    - In Direct-mode, injects external interrupts directly into HARTs
> > > >
> > > > For an overview of the AIA specification, refer the recent AIA virtualization
> > > > talk at KVM Forum 2022:
> > > > https://static.sched.com/hosted_files/kvmforum2022/a1/AIA_Virtualization_in_KVM_RISCV_final.pdf
> > > > https://www.youtube.com/watch?v=r071dL8Z0yo
> > > >
> > > > This series adds required Linux irqchip drivers for AIA and it depends on
> > > > the recent "RISC-V IPI Improvements".
> > > > (Refer, https://lore.kernel.org/lkml/20221101143400.690000-1-apatel@ventanamicro.com/t/)
> > > >
> > > > To test this series, use QEMU v7.1 (or higher) and OpenSBI v1.1 (or higher).
> > > >
> > > > These patches can also be found in the riscv_aia_v1 branch at:
> > > > https://github.com/avpatel/linux.git
> > > >
> > > > Anup Patel (9):
> > > >   RISC-V: Add AIA related CSR defines
> > > >   RISC-V: Detect AIA CSRs from ISA string
> > > >   irqchip/riscv-intc: Add support for RISC-V AIA
> > > >   dt-bindings: Add RISC-V incoming MSI controller bindings
> > > >   irqchip: Add RISC-V incoming MSI controller driver
> > > >   dt-bindings: Add RISC-V advanced PLIC bindings
> > > >   irqchip: Add RISC-V advanced PLIC driver
> > > >   RISC-V: Select APLIC and IMSIC drivers for QEMU virt machine
> > > >   MAINTAINERS: Add entry for RISC-V AIA drivers
> > > >
> > > >  .../interrupt-controller/riscv,aplic.yaml     |  136 ++
> > > >  .../interrupt-controller/riscv,imsic.yaml     |  174 +++
> > > >  MAINTAINERS                                   |   12 +
> > > >  arch/riscv/Kconfig.socs                       |    2 +
> > > >  arch/riscv/include/asm/csr.h                  |   92 ++
> > > >  arch/riscv/include/asm/hwcap.h                |    8 +
> > > >  arch/riscv/kernel/cpu.c                       |    2 +
> > > >  arch/riscv/kernel/cpufeature.c                |    2 +
> > > >  drivers/irqchip/Kconfig                       |   32 +-
> > > >  drivers/irqchip/Makefile                      |    2 +
> > > >  drivers/irqchip/irq-riscv-aplic.c             |  656 +++++++++
> > > >  drivers/irqchip/irq-riscv-imsic.c             | 1207 +++++++++++++++++
> > > >  drivers/irqchip/irq-riscv-intc.c              |   37 +-
> > > >  include/linux/irqchip/riscv-aplic.h           |  117 ++
> > > >  include/linux/irqchip/riscv-imsic.h           |   92 ++
> > > >  15 files changed, 2564 insertions(+), 7 deletions(-)
> > > >  create mode 100644 Documentation/devicetree/bindings/interrupt-controller/riscv,aplic.yaml
> > > >  create mode 100644 Documentation/devicetree/bindings/interrupt-controller/riscv,imsic.yaml
> > > >  create mode 100644 drivers/irqchip/irq-riscv-aplic.c
> > > >  create mode 100644 drivers/irqchip/irq-riscv-imsic.c
> > > >  create mode 100644 include/linux/irqchip/riscv-aplic.h
> > > >  create mode 100644 include/linux/irqchip/riscv-imsic.h
> > > >
> > > > --
> > > > 2.34.1
> > > >
> > >
> > > I am seeing the following boot failure with your branch and upstream
> > > qemu (tag: v7.2.0-rc0).
> > > It seems IPIs are probably not getting delivered after a point. I saw
> > > that IPIs are delivered in the same path earlier (via gdb).
> > >
> > > [    0.990152] NET: Registered PF_INET6 protocol family
> > > [    1.004885] Segment Routing with IPv6
> > > [    1.005385] In-situ OAM (IOAM) with IPv6
> > > [    1.006371] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
> > > [    1.011609] NET: Registered PF_PACKET protocol family
> > > [    1.014877] 9pnet: Installing 9P2000 support
> > > [    1.015603] Key type dns_resolver registered
> > > [    1.026396] debug_vm_pgtable: [debug_vm_pgtable         ]:
> > > Validating architecture page table helpers
> > > [    1.144097] EXT4-fs (vda2): recovery complete
> > > [    1.146345] EXT4-fs (vda2): mounted filesystem with ordered data
> > > mode. Quota mode: disabled.
> > > [    1.147020] VFS: Mounted root (ext4 filesystem) on device 254:2.
> > > [    1.152238] devtmpfs: mounted
> > > [    1.183914] Freeing unused kernel image (initmem) memory: 2176K
> > > [    1.185898] Run /sbin/init as init process
> > > [   29.412287] rcu: INFO: rcu_sched self-detected stall on CPU
> > > [   29.412992] rcu:     3-....: (5250 ticks this GP)
> > > idle=fc3c/1/0x4000000000000002 softirq=78/78 fqs=2230
> > > [   29.413427]  (t=5251 jiffies g=-1047 q=3 ncpus=8)
> > > [   29.414199] CPU: 3 PID: 1 Comm: init Not tainted
> > > 6.1.0-rc4-00024-g5b711f2d7b91 #198
> > > [   29.414578] Hardware name: riscv-virtio,qemu (DT)
> > > [   29.414896] epc : smp_call_function_many_cond+0x138/0x372
> > > [   29.415157]  ra : smp_call_function_many_cond+0x154/0x372
> > > [   29.415318] epc : ffffffff8008d2e6 ra : ffffffff8008d302 sp :
> > > ff2000000004bb40
> > > [   29.415512]  gp : ffffffff812e9eb0 tp : ff600000016d8000 t0 :
> > > ff6000007ed88610
> > > [   29.415686]  t1 : 00000000000000ff t2 : 0000000000000002 s0 :
> > > ff2000000004bc00
> > > [   29.415837]  s1 : ff6000007ed85448 a0 : 0000000000000007 a1 :
> > > 00000000000000f7
> > > [   29.416008]  a2 : 0000000000000000 a3 : 0000000000000000 a4 :
> > > ff6000007edd1780
> > > [   29.416188]  a5 : 0000000000000001 a6 : ffffffff812eb1c0 a7 :
> > > ff600000016d8000
> > > [   29.416613]  s2 : ffffffff81323c30 s3 : ffffffff812e9964 s4 :
> > > 0000000000000000
> > > [   29.416810]  s5 : 0000000000000000 s6 : ff6000007ed85440 s7 :
> > > 0000000000000038
> > > [   29.416997]  s8 : 0000000000000003 s9 : ffffffff81323c30 s10:
> > > ff6000007ed85448
> > > [   29.417159]  s11: 0000000000000008 t3 : 00ffffffad08a000 t4 :
> > > ff60000001613e0c
> > > [   29.417331]  t5 : 0000000000000000 t6 : 00ffffffad177fff
> > > [   29.417482] status: 0000000200000120 badaddr: 0000000000000000
> > > cause: 8000000000000005
> > > [   29.417939] [<ffffffff8008d590>] on_each_cpu_cond_mask+0x20/0x32
> > > [   29.418179] [<ffffffff80008d32>] flush_icache_all+0x38/0x40
> > > [   29.418324] [<ffffffff80008eb2>] flush_icache_pte+0x4a/0x7a
> > > [   29.418442] [<ffffffff80139e00>] do_set_pte+0x132/0x192
> > > [   29.418594] [<ffffffff8010b42e>] filemap_map_pages+0x178/0x3a0
> > > [   29.418738] [<ffffffff8013ad00>] __handle_mm_fault+0x992/0xbac
> > > [   29.418876] [<ffffffff8013afde>] handle_mm_fault+0xc4/0x1d4
> > > [   29.419010] [<ffffffff80008372>] do_page_fault+0x120/0x326
> > > [   29.419145] [<ffffffff800033e6>] ret_from_exception+0x0/0xc
> > >
> > >
> >
> > Sorry. I forgot to mention that this happens while booting Fedora on
> > an 8 cpu virt machine.
> > I am yet to reproduce this issue for 4 or 2 cpus.
> >
>
> This warning splat appears with 4 cpus though.
>
> [  126.977213] e1000e 0000:00:02.0 enp0s2: NIC Link is Up 1000 Mbps
> Full Duplex, Flow Control: Rx/Tx
> [  126.988275] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s2: link becomes ready
> [  128.210090] ------------[ cut here ]------------
> [  128.210476] WARNING: CPU: 0 PID: 294 at
> drivers/irqchip/irq-riscv-aplic.c:318 aplic_msi_write_msg+0x110/0x12a
> [  128.211702] Modules linked in:
> [  128.212473] CPU: 0 PID: 294 Comm: (agetty) Not tainted
> 6.1.0-rc4-00024-g5b711f2d7b91 #198
> [  128.212839] Hardware name: riscv-virtio,qemu (DT)
> [  128.213459] epc : aplic_msi_write_msg+0x110/0x12a
> [  128.213652]  ra : aplic_msi_write_msg+0x1c/0x12a
> [  128.213823] epc : ffffffff8039eb80 ra : ffffffff8039ea8c sp :
> ff2000001054bb10
> [  128.213884]  gp : ffffffff812e9eb0 tp : ff600000032d3840 t0 :
> 0000000000000002
> [  128.213945]  t1 : 0000000000000008 t2 : 0000000000000040 s0 :
> ff2000001054bb30
> [  128.214015]  s1 : ff600000018443a8 a0 : ff600000017e8c30 a1 :
> 0000000000000000
> [  128.214083]  a2 : 0000000000000001 a3 : 000000000000000c a4 :
> 0000000000000003
> [  128.214141]  a5 : 0000000000000000 a6 : 0000000000000000 a7 :
> 0000000000000000
> [  128.214173]  s2 : ff2000001054bb40 s3 : ff600000017e8c30 s4 :
> 0000000200000022
> [  128.214246]  s5 : ff600000017e8d78 s6 : 000000000000000c s7 :
> ff600000017e8cb0
> [  128.214295]  s8 : ffffffff80c5c8f0 s9 : ff6000000e97ab60 s10:
> 0000000000000000
> [  128.214327]  s11: ff6000000e97ab78 t3 : 0000000000000004 t4 :
> 0000000000000002
> [  128.214383]  t5 : 0000000000000000 t6 : 0000000000028000
> [  128.214414] status: 0000000200000100 badaddr: 0000000000000000
> cause: 0000000000000003
> [  128.215267] [<ffffffff80453060>] platform_msi_write_msg+0x18/0x20
> [  128.215413] [<ffffffff80060b7a>] msi_domain_deactivate+0x2a/0x48
> [  128.215432] [<ffffffff8005e5a2>] __irq_domain_deactivate_irq+0x1c/0x30
> [  128.215450] [<ffffffff8005e5aa>] __irq_domain_deactivate_irq+0x24/0x30
> [  128.215464] [<ffffffff8005f9ea>] irq_domain_deactivate_irq+0x1c/0x30
> [  128.215477] [<ffffffff8005b8f6>] free_irq+0x1d0/0x27c
> [  128.219393] [<ffffffff8041be06>] univ8250_release_irq+0xa4/0xaa
> [  128.219417] [<ffffffff8041e018>] serial8250_do_shutdown+0xf0/0x166
> [  128.219431] [<ffffffff8041e0a8>] serial8250_shutdown+0x1a/0x22
> [  128.219444] [<ffffffff804174f0>] uart_port_shutdown+0x2c/0x42
> [  128.220230] [<ffffffff8041908a>] uart_shutdown+0x66/0x11e
> [  128.220247] [<ffffffff8041a69a>] uart_hangup+0x5e/0x100
> [  128.220260] [<ffffffff803fdaa2>] __tty_hangup.part.0+0x286/0x2ac
> [  128.220859] [<ffffffff803fec6c>] tty_ioctl+0x684/0x802
> [  128.220875] [<ffffffff8019ec48>] sys_ioctl+0x37e/0x95e
> [  128.221284] [<ffffffff800033d8>] ret_from_syscall+0x0/0x2
> [  128.221653] ---[ end trace 0000000000000000 ]---

I will address this in v2 of the AIA series.

Regards,
Anup

>
>
> > >
> > > --
> > > Regards,
> > > Atish
> >
> >
> >
> > --
> > Regards,
> > Atish
>
>
>
> --
> Regards,
> Atish