KVM: arm64: vgic: Fix soft lockup during VM teardown

  Getting intermittent CPU soft lockups during the virtual machines
teardown on a system with GICv4 features enabled. The function
__synchronize_hardirq() has been waiting for IRQD_IRQ_INPROGRESS
to be cleared forever as per the current implementation.

CPU stuck here for a long time leads to soft lockup:
  while (irqd_irq_inprogress(&desc->irq_data))
	cpu_relax();

Call trace from the lockup CPU:
 [   87.238866] watchdog: BUG: soft lockup - CPU#37 stuck for 23s!
 [   87.250025] CPU: 37 PID: 1031 Comm: qemu-system-aarch64
 [   87.358397] Call trace:
 [   87.360891]  __synchronize_hardirq+0x48/0x140
 [   87.365343]  free_irq+0x138/0x424
 [   87.368727]  vgic_v4_teardown+0xa4/0xe0
 [   87.372649]  __kvm_vgic_destroy+0x18c/0x194
 [   87.376922]  kvm_vgic_destroy+0x28/0x3c
 [   87.380839]  kvm_arch_destroy_vm+0x24/0x44
 [   87.385024]  kvm_destroy_vm+0x158/0x2c4
 [   87.388943]  kvm_vm_release+0x6c/0x98
 [   87.392681]  __fput+0x70/0x220
 [   87.395800]  ____fput+0x10/0x20
 [   87.399005]  task_work_run+0xb4/0x23c
 [   87.402746]  do_exit+0x2bc/0x8a4
 [   87.406042]  do_group_exit+0x34/0xb0
 [   87.409693]  get_signal+0x878/0x8a0
 [   87.413254]  do_notify_resume+0x138/0x1530
 [   87.417440]  el0_svc+0xdc/0xf0
 [   87.420559]  el0t_64_sync_handler+0xf0/0x11c
 [   87.424919]  el0t_64_sync+0x18c/0x190

The state of the IRQD_IRQ_INPROGRESS information is lost inside
irq_domain_activate_irq() which happens before calling free_irq().
Instrumented the code and confirmed, the IRQD state is changed from
0x10401400 to 0x10441600 instead of 0x10401600 causing problem.

Call trace from irqd_set_activated():
 [   78.983544] irqd_set_activated: lost IRQD_IRQ_INPROGRESS
                old=0x10401400, new=0x10441600
 [   78.992093] CPU: 19 PID: 1511 Comm: qemu-system-aarch64
 [   79.008461] Call trace:
 [   79.010956]  dump_backtrace.part.0+0xc8/0xe0
 [   79.015328]  show_stack+0x18/0x54
 [   79.018713]  dump_stack_lvl+0x64/0x7c
 [   79.022459]  dump_stack+0x18/0x30
 [   79.025842]  irq_domain_activate_irq+0x88/0x94
 [   79.030385]  vgic_v3_save_pending_tables+0x260/0x29c
 [   79.035463]  vgic_set_common_attr+0xac/0x23c
 [   79.039826]  vgic_v3_set_attr+0x48/0x60
 [   79.043742]  kvm_device_ioctl+0x120/0x19c
 [   79.047840]  __arm64_sys_ioctl+0x42c/0xe00
 [   79.052027]  invoke_syscall.constprop.0+0x50/0xe0
 [   79.056835]  do_el0_svc+0x58/0x180
 [   79.060308]  el0_svc+0x38/0xf0
 [   79.063425]  el0t_64_sync_handler+0xf0/0x11c
 [   79.067785]  el0t_64_sync+0x18c/0x190

irqreturn_t handle_irq_event(struct irq_desc *desc)
{
    irqd_set(&desc->irq_data, IRQD_IRQ_INPROGRESS);
    raw_spin_unlock(&desc->lock);

    ret = handle_irq_event_percpu(desc);

    raw_spin_lock(&desc->lock);
    irqd_clear(&desc->irq_data, IRQD_IRQ_INPROGRESS);
}

In this particular failed case and based on traces, the two functions
irqd_set_activated() and handle_irq_event() are concurrently modifying
IRQD state without both holding desc->lock. The irqd_set_activated()
execution path is reading memory 'state_use_accessors' in between set
& clear of IRQD_IRQ_INPROGRESS state change and writing the modified
data after executing 'irqd_clear(desc->irq_data, IRQD_IRQ_INPROGRESS)'.

To fix the lockup issue, hold desc->lock when calling functions
irq_domain_activate_irq() and irq_domain_deactivate_irq).

Signed-off-by: Shanker Donthineni <sdonthineni@nvidia.com>
---
 arch/arm64/kvm/vgic/vgic-v3.c | 6 ++++++
 arch/arm64/kvm/vgic/vgic-v4.c | 4 ++++
 2 files changed, 10 insertions(+)
  

Shanker,

Please Cc all the KVM/arm64 reviewers when sending KVM/arm64 patches.

On Wed, 18 Jan 2023 02:23:48 +0000,
Shanker Donthineni <sdonthineni@nvidia.com> wrote:
> 
> Getting intermittent CPU soft lockups during the virtual machines
> teardown on a system with GICv4 features enabled. The function
> __synchronize_hardirq() has been waiting for IRQD_IRQ_INPROGRESS
> to be cleared forever as per the current implementation.
> 
> CPU stuck here for a long time leads to soft lockup:
>   while (irqd_irq_inprogress(&desc->irq_data))
> 	cpu_relax();

Is it a soft-lockup from which the system recovers? or a livelock
which leaves the system dead?

What kernel version is that? Please provide all the relevant context
that could help analysing the issue.

> 
> Call trace from the lockup CPU:
>  [   87.238866] watchdog: BUG: soft lockup - CPU#37 stuck for 23s!
>  [   87.250025] CPU: 37 PID: 1031 Comm: qemu-system-aarch64
>  [   87.358397] Call trace:
>  [   87.360891]  __synchronize_hardirq+0x48/0x140
>  [   87.365343]  free_irq+0x138/0x424
>  [   87.368727]  vgic_v4_teardown+0xa4/0xe0
>  [   87.372649]  __kvm_vgic_destroy+0x18c/0x194
>  [   87.376922]  kvm_vgic_destroy+0x28/0x3c
>  [   87.380839]  kvm_arch_destroy_vm+0x24/0x44
>  [   87.385024]  kvm_destroy_vm+0x158/0x2c4
>  [   87.388943]  kvm_vm_release+0x6c/0x98
>  [   87.392681]  __fput+0x70/0x220
>  [   87.395800]  ____fput+0x10/0x20
>  [   87.399005]  task_work_run+0xb4/0x23c
>  [   87.402746]  do_exit+0x2bc/0x8a4
>  [   87.406042]  do_group_exit+0x34/0xb0
>  [   87.409693]  get_signal+0x878/0x8a0
>  [   87.413254]  do_notify_resume+0x138/0x1530
>  [   87.417440]  el0_svc+0xdc/0xf0
>  [   87.420559]  el0t_64_sync_handler+0xf0/0x11c
>  [   87.424919]  el0t_64_sync+0x18c/0x190
> 
> The state of the IRQD_IRQ_INPROGRESS information is lost inside
> irq_domain_activate_irq() which happens before calling free_irq().
> Instrumented the code and confirmed, the IRQD state is changed from
> 0x10401400 to 0x10441600 instead of 0x10401600 causing problem.
> 
> Call trace from irqd_set_activated():
>  [   78.983544] irqd_set_activated: lost IRQD_IRQ_INPROGRESS
>                 old=0x10401400, new=0x10441600
>  [   78.992093] CPU: 19 PID: 1511 Comm: qemu-system-aarch64
>  [   79.008461] Call trace:
>  [   79.010956]  dump_backtrace.part.0+0xc8/0xe0
>  [   79.015328]  show_stack+0x18/0x54
>  [   79.018713]  dump_stack_lvl+0x64/0x7c
>  [   79.022459]  dump_stack+0x18/0x30
>  [   79.025842]  irq_domain_activate_irq+0x88/0x94
>  [   79.030385]  vgic_v3_save_pending_tables+0x260/0x29c
>  [   79.035463]  vgic_set_common_attr+0xac/0x23c
>  [   79.039826]  vgic_v3_set_attr+0x48/0x60
>  [   79.043742]  kvm_device_ioctl+0x120/0x19c
>  [   79.047840]  __arm64_sys_ioctl+0x42c/0xe00
>  [   79.052027]  invoke_syscall.constprop.0+0x50/0xe0
>  [   79.056835]  do_el0_svc+0x58/0x180
>  [   79.060308]  el0_svc+0x38/0xf0
>  [   79.063425]  el0t_64_sync_handler+0xf0/0x11c
>  [   79.067785]  el0t_64_sync+0x18c/0x190

Are these two traces an indication of concurrent events? Or are they
far apart?

> 
> irqreturn_t handle_irq_event(struct irq_desc *desc)
> {
>     irqd_set(&desc->irq_data, IRQD_IRQ_INPROGRESS);
>     raw_spin_unlock(&desc->lock);
> 
>     ret = handle_irq_event_percpu(desc);
> 
>     raw_spin_lock(&desc->lock);
>     irqd_clear(&desc->irq_data, IRQD_IRQ_INPROGRESS);
> }

How is that relevant to this trace? Do you see this function running
concurrently with the teardown? If it matters here, it must be a VPE
doorbell, right? But you claim that this is on a GICv4 platform, while
this would only affect GICv4.1... Or are you using GICv4.1?

> 
> In this particular failed case and based on traces, the two functions
> irqd_set_activated() and handle_irq_event() are concurrently modifying
> IRQD state without both holding desc->lock. The irqd_set_activated()
> execution path is reading memory 'state_use_accessors' in between set
> & clear of IRQD_IRQ_INPROGRESS state change and writing the modified
> data after executing 'irqd_clear(desc->irq_data, IRQD_IRQ_INPROGRESS)'.
> 
> To fix the lockup issue, hold desc->lock when calling functions
> irq_domain_activate_irq() and irq_domain_deactivate_irq).

For that particular issue, the main problem is that we are abusing the
interrupt startup/teardown code. The locking is only a consequence of
this.

> 
> Signed-off-by: Shanker Donthineni <sdonthineni@nvidia.com>
> ---
>  arch/arm64/kvm/vgic/vgic-v3.c | 6 ++++++
>  arch/arm64/kvm/vgic/vgic-v4.c | 4 ++++
>  2 files changed, 10 insertions(+)
> 
> diff --git a/arch/arm64/kvm/vgic/vgic-v3.c b/arch/arm64/kvm/vgic/vgic-v3.c
> index 2074521d4a8c..e6aa909fcbe2 100644
> --- a/arch/arm64/kvm/vgic/vgic-v3.c
> +++ b/arch/arm64/kvm/vgic/vgic-v3.c
> @@ -353,22 +353,28 @@ int vgic_v3_lpi_sync_pending_status(struct kvm *kvm, struct vgic_irq *irq)
>  static void unmap_all_vpes(struct vgic_dist *dist)
>  {
>  	struct irq_desc *desc;
> +	unsigned long flags;
>  	int i;
>  
>  	for (i = 0; i < dist->its_vm.nr_vpes; i++) {
>  		desc = irq_to_desc(dist->its_vm.vpes[i]->irq);
> +		raw_spin_lock_irqsave(&desc->lock, flags);
>  		irq_domain_deactivate_irq(irq_desc_get_irq_data(desc));
> +		raw_spin_unlock_irqrestore(&desc->lock, flags);

I guess this is the guilty one, based on your analysis, and assuming
this is a v4.1 issue, not v4.

>  	}
>  }
>  
>  static void map_all_vpes(struct vgic_dist *dist)
>  {
>  	struct irq_desc *desc;
> +	unsigned long flags;
>  	int i;
>  
>  	for (i = 0; i < dist->its_vm.nr_vpes; i++) {
>  		desc = irq_to_desc(dist->its_vm.vpes[i]->irq);
> +		raw_spin_lock_irqsave(&desc->lock, flags);
>  		irq_domain_activate_irq(irq_desc_get_irq_data(desc), false);
> +		raw_spin_unlock_irqrestore(&desc->lock, flags);
>  	}
>  }
>  
> diff --git a/arch/arm64/kvm/vgic/vgic-v4.c b/arch/arm64/kvm/vgic/vgic-v4.c
> index ad06ba6c9b00..a01b8313e82c 100644
> --- a/arch/arm64/kvm/vgic/vgic-v4.c
> +++ b/arch/arm64/kvm/vgic/vgic-v4.c
> @@ -139,8 +139,10 @@ static void vgic_v4_enable_vsgis(struct kvm_vcpu *vcpu)
>  		/* Transfer the full irq state to the vPE */
>  		vgic_v4_sync_sgi_config(vpe, irq);
>  		desc = irq_to_desc(irq->host_irq);
> +		raw_spin_lock(&desc->lock);
>  		ret = irq_domain_activate_irq(irq_desc_get_irq_data(desc),
>  					      false);
> +		raw_spin_unlock(&desc->lock);

This one looks wrong. The interrupt never fires on the host (that's
the whole point of this stuff). There isn't even a handler attached to
it. How can that result in a problem?

My take on the whole thing is that we should stop playing games with
the underlying interrupt infrastructure. How about the following hack,
which is only compile-tested. Let me know if that helps.

	M.

diff --git a/arch/arm64/kvm/vgic/vgic-v3.c b/arch/arm64/kvm/vgic/vgic-v3.c
index 2074521d4a8c..2624963cb95b 100644
--- a/arch/arm64/kvm/vgic/vgic-v3.c
+++ b/arch/arm64/kvm/vgic/vgic-v3.c
@@ -350,26 +350,23 @@ int vgic_v3_lpi_sync_pending_status(struct kvm *kvm, struct vgic_irq *irq)
  * The deactivation of the doorbell interrupt will trigger the
  * unmapping of the associated vPE.
  */
-static void unmap_all_vpes(struct vgic_dist *dist)
+static void unmap_all_vpes(struct kvm *kvm)
 {
-	struct irq_desc *desc;
+	struct vgic_dist *dist = &kvm->arch.vgic;
 	int i;
 
-	for (i = 0; i < dist->its_vm.nr_vpes; i++) {
-		desc = irq_to_desc(dist->its_vm.vpes[i]->irq);
-		irq_domain_deactivate_irq(irq_desc_get_irq_data(desc));
-	}
+	for (i = 0; i < dist->its_vm.nr_vpes; i++)
+		free_irq(dist->its_vm.vpes[i]->irq, kvm_get_vcpu(kvm, i));
 }
 
-static void map_all_vpes(struct vgic_dist *dist)
+static void map_all_vpes(struct kvm *kvm)
 {
-	struct irq_desc *desc;
+	struct vgic_dist *dist = &kvm->arch.vgic;
 	int i;
 
-	for (i = 0; i < dist->its_vm.nr_vpes; i++) {
-		desc = irq_to_desc(dist->its_vm.vpes[i]->irq);
-		irq_domain_activate_irq(irq_desc_get_irq_data(desc), false);
-	}
+	for (i = 0; i < dist->its_vm.nr_vpes; i++)
+		WARN_ON(vgic_v4_request_vpe_irq(kvm_get_vcpu(kvm, i),
+						dist->its_vm.vpes[i]->irq));
 }
 
 /**
@@ -394,7 +391,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
 	 * and enabling of the doorbells have already been done.
 	 */
 	if (kvm_vgic_global_state.has_gicv4_1) {
-		unmap_all_vpes(dist);
+		unmap_all_vpes(kvm);
 		vlpi_avail = true;
 	}
 
@@ -444,7 +441,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
 
 out:
 	if (vlpi_avail)
-		map_all_vpes(dist);
+		map_all_vpes(kvm);
 
 	return ret;
 }
diff --git a/arch/arm64/kvm/vgic/vgic-v4.c b/arch/arm64/kvm/vgic/vgic-v4.c
index ad06ba6c9b00..a413718be92b 100644
--- a/arch/arm64/kvm/vgic/vgic-v4.c
+++ b/arch/arm64/kvm/vgic/vgic-v4.c
@@ -222,6 +222,11 @@ void vgic_v4_get_vlpi_state(struct vgic_irq *irq, bool *val)
 	*val = !!(*ptr & mask);
 }
 
+int vgic_v4_request_vpe_irq(struct kvm_vcpu *vcpu, int irq)
+{
+	return request_irq(irq, vgic_v4_doorbell_handler, 0, "vcpu", vcpu);
+}
+
 /**
  * vgic_v4_init - Initialize the GICv4 data structures
  * @kvm:	Pointer to the VM being initialized
@@ -283,8 +288,7 @@ int vgic_v4_init(struct kvm *kvm)
 			irq_flags &= ~IRQ_NOAUTOEN;
 		irq_set_status_flags(irq, irq_flags);
 
-		ret = request_irq(irq, vgic_v4_doorbell_handler,
-				  0, "vcpu", vcpu);
+		ret = vgic_v4_request_vpe_irq(vcpu, irq);
 		if (ret) {
 			kvm_err("failed to allocate vcpu IRQ%d\n", irq);
 			/*
diff --git a/arch/arm64/kvm/vgic/vgic.h b/arch/arm64/kvm/vgic/vgic.h
index 0c8da72953f0..23e280fa0a16 100644
--- a/arch/arm64/kvm/vgic/vgic.h
+++ b/arch/arm64/kvm/vgic/vgic.h
@@ -331,5 +331,6 @@ int vgic_v4_init(struct kvm *kvm);
 void vgic_v4_teardown(struct kvm *kvm);
 void vgic_v4_configure_vsgis(struct kvm *kvm);
 void vgic_v4_get_vlpi_state(struct vgic_irq *irq, bool *val);
+int vgic_v4_request_vpe_irq(struct kvm_vcpu *vcpu, int irq);
 
 #endif
  

On 1/18/23 05:54, Marc Zyngier wrote:
> External email: Use caution opening links or attachments
> 
> 
> Shanker,
> 
> Please Cc all the KVM/arm64 reviewers when sending KVM/arm64 patches.
> 
> On Wed, 18 Jan 2023 02:23:48 +0000,
> Shanker Donthineni <sdonthineni@nvidia.com> wrote:
>>
>> Getting intermittent CPU soft lockups during the virtual machines
>> teardown on a system with GICv4 features enabled. The function
>> __synchronize_hardirq() has been waiting for IRQD_IRQ_INPROGRESS
>> to be cleared forever as per the current implementation.
>>
>> CPU stuck here for a long time leads to soft lockup:
>>    while (irqd_irq_inprogress(&desc->irq_data))
>>        cpu_relax();
> 
> Is it a soft-lockup from which the system recovers? or a livelock
> which leaves the system dead?
> 
The system is not recovering, did a power cycle to recover.

[  211.018214] watchdog: BUG: soft lockup - CPU#8 stuck for 45s! [qemu-system-aar:1261]
[  211.026164] Modules linked in:
[  211.029283] CPU: 8 PID: 1261 Comm: qemu-system-aar Kdump: loaded Not tainted 6.0.0 #7
[  211.037338] Hardware name:  /th500_silicon, BIOS v1.1.1-4638c6c9 01/17/2023
[  211.044453] pstate: 62400009 (nZCv daif +PAN -UAO +TCO -DIT -SSBS BTYPE=--)
[  211.051586] pc : __synchronize_hardirq+0x54/0x140
[  211.056421] lr : free_irq+0x138/0x424
[  211.060160] sp : ffff80001504f990
[  211.063551] x29: ffff80001504f990 x28: 0000000000000008 x27: ffff00036c880000
[  211.070850] x26: ffff00036f888c00 x25: 0000000000000000 x24: 0000000000000001
[  211.078219] x23: ffff0000835659d0 x22: 0000000000000001 x21: 0000000000000000
[  211.085513] x20: ffff00008356591c x19: ffff0000835658a0 x18: 000000000000003f
[  211.092807] x17: ffff001feafabf28 x16: ffffffc0012c3388 x15: ffff80001504fab0
[  211.100103] x14: 0000000000000008 x13: ffff800008b41718 x12: 00000000f5257d14
[  211.107489] x11: 0000000000000040 x10: ffff8000091f1680 x9 : ffff8000091f1678
[  211.114783] x8 : 0000000000000001 x7 : 0000000000000000 x6 : 0000000000000001
[  211.122115] x5 : ffff8000083524e4 x4 : ffffffc000db5e20 x3 : 0000000081550146
[  211.129409] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000010071200
[  211.136736] Call trace:
[  211.139232]  __synchronize_hardirq+0x54/0x140
[  211.143746]  free_irq+0x138/0x424
[  211.147139]  vgic_v4_teardown+0xa4/0xe0
[  211.151072]  __kvm_vgic_destroy+0x18c/0x194
[  211.155351]  kvm_vgic_destroy+0x28/0x3c
[  211.159310]  kvm_arch_destroy_vm+0x24/0x44
[  211.163579]  kvm_destroy_vm+0x158/0x2c4
[  211.167495]  kvm_vm_release+0x6c/0x98
[  211.171257]  __fput+0x70/0x220
[  211.174373]  ____fput+0x10/0x20
[  211.177588]  task_work_run+0xb4/0x23c
[  211.181327]  do_exit+0x2bc/0x8a4
[  211.184622]  do_group_exit+0x34/0xb0
[  211.188271]  get_signal+0x878/0x8a0
[  211.191856]  do_notify_resume+0x138/0x1530
[  211.196075]  el0_svc+0xdc/0xf0
[  211.199211]  el0t_64_sync_handler+0xf0/0x11c
[  211.203635]  el0t_64_sync+0x18c/0x190
[  211.207374] Kernel panic - not syncing: softlockup: hung tasks
[  211.213335] CPU: 8 PID: 1261 Comm: qemu-system-aar Kdump: loaded Tainted: G             L     6.0.0 #7
[  211.222851] Hardware name:  /th500_silicon, BIOS v1.1.1-4638c6c9 01/17/2023
[  211.229993] Call trace:
[  211.232498]  dump_backtrace.part.0+0xc8/0xe0
[  211.236890]  show_stack+0x18/0x54
[  211.240330]  dump_stack_lvl+0x64/0x7c
[  211.244097]  dump_stack+0x18/0x30
[  211.247479]  panic+0x160/0x328
[  211.250622]  watchdog_nmi_enable+0x0/0x14
[  211.254818]  __hrtimer_run_queues+0x118/0x244
[  211.259288]  hrtimer_interrupt+0x114/0x300
[  211.263471]  arch_timer_handler_phys+0x2c/0x40
[  211.268034]  handle_percpu_devid_irq+0x84/0x138
[  211.272671]  generic_handle_domain_irq+0x2c/0x40
[  211.277394]  gic_handle_irq+0x50/0x128
[  211.281221]  call_on_irq_stack+0x2c/0x38
[  211.285226]  do_interrupt_handler+0x7c/0x88
[  211.289552]  el1_interrupt+0x34/0x54
[  211.293239]  el1h_64_irq_handler+0x18/0x20
[  211.297433]  el1h_64_irq+0x64/0x68
[  211.300935]  __synchronize_hardirq+0x54/0x140
[  211.305385]  free_irq+0x138/0x424
[  211.308767]  vgic_v4_teardown+0xa4/0xe0
[  211.312709]  __kvm_vgic_destroy+0x18c/0x194
[  211.317007]  kvm_vgic_destroy+0x28/0x3c
[  211.320927]  kvm_arch_destroy_vm+0x24/0x44
[  211.325204]  kvm_destroy_vm+0x158/0x2c4
[  211.329125]  kvm_vm_release+0x6c/0x98
[  211.332872]  __fput+0x70/0x220
[  211.335988]  ____fput+0x10/0x20
[  211.339192]  task_work_run+0xb4/0x23c
[  211.342980]  do_exit+0x2bc/0x8a4
[  211.346310]  do_group_exit+0x34/0xb0
[  211.349960]  get_signal+0x878/0x8a0
[  211.353520]  do_notify_resume+0x138/0x1530
[  211.357703]  el0_svc+0xdc/0xf0
[  211.360818]  el0t_64_sync_handler+0xf0/0x11c
[  211.365197]  el0t_64_sync+0x18c/0x190
[  211.368947] SMP: stopping secondary CPUs


> What kernel version is that? Please provide all the relevant context
> that could help analysing the issue.
> 
Tested with v6.0+ and v6.2.rc4.

>>
>> Call trace from the lockup CPU:
>>   [   87.238866] watchdog: BUG: soft lockup - CPU#37 stuck for 23s!
>>   [   87.250025] CPU: 37 PID: 1031 Comm: qemu-system-aarch64
>>   [   87.358397] Call trace:
>>   [   87.360891]  __synchronize_hardirq+0x48/0x140
>>   [   87.365343]  free_irq+0x138/0x424
>>   [   87.368727]  vgic_v4_teardown+0xa4/0xe0
>>   [   87.372649]  __kvm_vgic_destroy+0x18c/0x194
>>   [   87.376922]  kvm_vgic_destroy+0x28/0x3c
>>   [   87.380839]  kvm_arch_destroy_vm+0x24/0x44
>>   [   87.385024]  kvm_destroy_vm+0x158/0x2c4
>>   [   87.388943]  kvm_vm_release+0x6c/0x98
>>   [   87.392681]  __fput+0x70/0x220
>>   [   87.395800]  ____fput+0x10/0x20
>>   [   87.399005]  task_work_run+0xb4/0x23c
>>   [   87.402746]  do_exit+0x2bc/0x8a4
>>   [   87.406042]  do_group_exit+0x34/0xb0
>>   [   87.409693]  get_signal+0x878/0x8a0
>>   [   87.413254]  do_notify_resume+0x138/0x1530
>>   [   87.417440]  el0_svc+0xdc/0xf0
>>   [   87.420559]  el0t_64_sync_handler+0xf0/0x11c
>>   [   87.424919]  el0t_64_sync+0x18c/0x190
>>
>> The state of the IRQD_IRQ_INPROGRESS information is lost inside
>> irq_domain_activate_irq() which happens before calling free_irq().
>> Instrumented the code and confirmed, the IRQD state is changed from
>> 0x10401400 to 0x10441600 instead of 0x10401600 causing problem.
>>
>> Call trace from irqd_set_activated():
>>   [   78.983544] irqd_set_activated: lost IRQD_IRQ_INPROGRESS
>>                  old=0x10401400, new=0x10441600
>>   [   78.992093] CPU: 19 PID: 1511 Comm: qemu-system-aarch64
>>   [   79.008461] Call trace:
>>   [   79.010956]  dump_backtrace.part.0+0xc8/0xe0
>>   [   79.015328]  show_stack+0x18/0x54
>>   [   79.018713]  dump_stack_lvl+0x64/0x7c
>>   [   79.022459]  dump_stack+0x18/0x30
>>   [   79.025842]  irq_domain_activate_irq+0x88/0x94
>>   [   79.030385]  vgic_v3_save_pending_tables+0x260/0x29c
>>   [   79.035463]  vgic_set_common_attr+0xac/0x23c
>>   [   79.039826]  vgic_v3_set_attr+0x48/0x60
>>   [   79.043742]  kvm_device_ioctl+0x120/0x19c
>>   [   79.047840]  __arm64_sys_ioctl+0x42c/0xe00
>>   [   79.052027]  invoke_syscall.constprop.0+0x50/0xe0
>>   [   79.056835]  do_el0_svc+0x58/0x180
>>   [   79.060308]  el0_svc+0x38/0xf0
>>   [   79.063425]  el0t_64_sync_handler+0xf0/0x11c
>>   [   79.067785]  el0t_64_sync+0x18c/0x190
> 
> Are these two traces an indication of concurrent events? Or are they
> far apart?
> 
Concurrent.

>>
>> irqreturn_t handle_irq_event(struct irq_desc *desc)
>> {
>>      irqd_set(&desc->irq_data, IRQD_IRQ_INPROGRESS);
>>      raw_spin_unlock(&desc->lock);
>>
>>      ret = handle_irq_event_percpu(desc);
>>
>>      raw_spin_lock(&desc->lock);
>>      irqd_clear(&desc->irq_data, IRQD_IRQ_INPROGRESS);
>> }
> 
> How is that relevant to this trace? Do you see this function running
> concurrently with the teardown? If it matters here, it must be a VPE
> doorbell, right? But you claim that this is on a GICv4 platform, while
> this would only affect GICv4.1... Or are you using GICv4.1?
> 
handle_irq_event() is running concurrently with irq_domain_activate_irq()
which happens before free_irq() called. Corruption at [78.983544] and
teardown started at [87.360891].

[   78.983544] irqd_set_activated: lost IRQD_IRQ_INPROGRESS old=0x10401400, new=0x10441600

[   87.360891]  __synchronize_hardirq+0x48/0x140

Yes, I'm using GICv4.1, used these 2 functions to trace the issue.

static inline void irqd_set_activated(struct irq_data *d)
{
	atomic_t *state = (atomic_t *)(&__irqd_to_state(d));
	unsigned int new, old = __irqd_to_state(d);

	new = atomic_xchg(state, old | IRQD_ACTIVATED);	

	if ((old ^ new) & IRQD_IRQ_INPROGRESS) {
		pr_err("%s: lost IRQD_IRQ_INPROGRESS old=0x%x new=0x%x\n",
		       __func__, old, old | IRQD_ACTIVATED);
		dump_stack();
	}
}

static inline void irqd_clr_activated(struct irq_data *d)
{
	atomic_t *state = (atomic_t *)(&__irqd_to_state(d));
	unsigned int new, old = __irqd_to_state(d);

	new = atomic_xchg(state, old & ~IRQD_ACTIVATED);	

	if ((old ^ new) & IRQD_IRQ_INPROGRESS) {
		pr_err("%s: lost IRQD_IRQ_INPROGRESS old=0x%x new=0x%x\n",
		       __func__, old, old & ~IRQD_ACTIVATED);
		dump_stack();
	}
}

>>
>> In this particular failed case and based on traces, the two functions
>> irqd_set_activated() and handle_irq_event() are concurrently modifying
>> IRQD state without both holding desc->lock. The irqd_set_activated()
>> execution path is reading memory 'state_use_accessors' in between set
>> & clear of IRQD_IRQ_INPROGRESS state change and writing the modified
>> data after executing 'irqd_clear(desc->irq_data, IRQD_IRQ_INPROGRESS)'.
>>
>> To fix the lockup issue, hold desc->lock when calling functions
>> irq_domain_activate_irq() and irq_domain_deactivate_irq).
> 
> For that particular issue, the main problem is that we are abusing the
> interrupt startup/teardown code. The locking is only a consequence of
> this.
> 
>>
>> Signed-off-by: Shanker Donthineni <sdonthineni@nvidia.com>
>> ---
>>   arch/arm64/kvm/vgic/vgic-v3.c | 6 ++++++
>>   arch/arm64/kvm/vgic/vgic-v4.c | 4 ++++
>>   2 files changed, 10 insertions(+)
>>
>> diff --git a/arch/arm64/kvm/vgic/vgic-v3.c b/arch/arm64/kvm/vgic/vgic-v3.c
>> index 2074521d4a8c..e6aa909fcbe2 100644
>> --- a/arch/arm64/kvm/vgic/vgic-v3.c
>> +++ b/arch/arm64/kvm/vgic/vgic-v3.c
>> @@ -353,22 +353,28 @@ int vgic_v3_lpi_sync_pending_status(struct kvm *kvm, struct vgic_irq *irq)
>>   static void unmap_all_vpes(struct vgic_dist *dist)
>>   {
>>        struct irq_desc *desc;
>> +     unsigned long flags;
>>        int i;
>>
>>        for (i = 0; i < dist->its_vm.nr_vpes; i++) {
>>                desc = irq_to_desc(dist->its_vm.vpes[i]->irq);
>> +             raw_spin_lock_irqsave(&desc->lock, flags);
>>                irq_domain_deactivate_irq(irq_desc_get_irq_data(desc));
>> +             raw_spin_unlock_irqrestore(&desc->lock, flags);
> 
> I guess this is the guilty one, based on your analysis, and assuming
> this is a v4.1 issue, not v4.
> 
>>        }
>>   }
>>
>>   static void map_all_vpes(struct vgic_dist *dist)
>>   {
>>        struct irq_desc *desc;
>> +     unsigned long flags;
>>        int i;
>>
>>        for (i = 0; i < dist->its_vm.nr_vpes; i++) {
>>                desc = irq_to_desc(dist->its_vm.vpes[i]->irq);
>> +             raw_spin_lock_irqsave(&desc->lock, flags);
>>                irq_domain_activate_irq(irq_desc_get_irq_data(desc), false);
>> +             raw_spin_unlock_irqrestore(&desc->lock, flags);
>>        }
>>   }
>>
>> diff --git a/arch/arm64/kvm/vgic/vgic-v4.c b/arch/arm64/kvm/vgic/vgic-v4.c
>> index ad06ba6c9b00..a01b8313e82c 100644
>> --- a/arch/arm64/kvm/vgic/vgic-v4.c
>> +++ b/arch/arm64/kvm/vgic/vgic-v4.c
>> @@ -139,8 +139,10 @@ static void vgic_v4_enable_vsgis(struct kvm_vcpu *vcpu)
>>                /* Transfer the full irq state to the vPE */
>>                vgic_v4_sync_sgi_config(vpe, irq);
>>                desc = irq_to_desc(irq->host_irq);
>> +             raw_spin_lock(&desc->lock);
>>                ret = irq_domain_activate_irq(irq_desc_get_irq_data(desc),
>>                                              false);
>> +             raw_spin_unlock(&desc->lock);
> 
> This one looks wrong. The interrupt never fires on the host (that's
> the whole point of this stuff). There isn't even a handler attached to
> it. How can that result in a problem?

Thanks for catching mistake.

> 
> My take on the whole thing is that we should stop playing games with
> the underlying interrupt infrastructure. How about the following hack,
> which is only compile-tested. Let me know if that helps.
> 
>          M.
> 
> diff --git a/arch/arm64/kvm/vgic/vgic-v3.c b/arch/arm64/kvm/vgic/vgic-v3.c
> index 2074521d4a8c..2624963cb95b 100644
> --- a/arch/arm64/kvm/vgic/vgic-v3.c
> +++ b/arch/arm64/kvm/vgic/vgic-v3.c
> @@ -350,26 +350,23 @@ int vgic_v3_lpi_sync_pending_status(struct kvm *kvm, struct vgic_irq *irq)
>    * The deactivation of the doorbell interrupt will trigger the
>    * unmapping of the associated vPE.
>    */
> -static void unmap_all_vpes(struct vgic_dist *dist)
> +static void unmap_all_vpes(struct kvm *kvm)
>   {
> -       struct irq_desc *desc;
> +       struct vgic_dist *dist = &kvm->arch.vgic;
>          int i;
> 
> -       for (i = 0; i < dist->its_vm.nr_vpes; i++) {
> -               desc = irq_to_desc(dist->its_vm.vpes[i]->irq);
> -               irq_domain_deactivate_irq(irq_desc_get_irq_data(desc));
> -       }
> +       for (i = 0; i < dist->its_vm.nr_vpes; i++)
> +               free_irq(dist->its_vm.vpes[i]->irq, kvm_get_vcpu(kvm, i));
>   }
> 
> -static void map_all_vpes(struct vgic_dist *dist)
> +static void map_all_vpes(struct kvm *kvm)
>   {
> -       struct irq_desc *desc;
> +       struct vgic_dist *dist = &kvm->arch.vgic;
>          int i;
> 
> -       for (i = 0; i < dist->its_vm.nr_vpes; i++) {
> -               desc = irq_to_desc(dist->its_vm.vpes[i]->irq);
> -               irq_domain_activate_irq(irq_desc_get_irq_data(desc), false);
> -       }
> +       for (i = 0; i < dist->its_vm.nr_vpes; i++)
> +               WARN_ON(vgic_v4_request_vpe_irq(kvm_get_vcpu(kvm, i),
> +                                               dist->its_vm.vpes[i]->irq));
>   }
> 
>   /**
> @@ -394,7 +391,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
>           * and enabling of the doorbells have already been done.
>           */
>          if (kvm_vgic_global_state.has_gicv4_1) {
> -               unmap_all_vpes(dist);
> +               unmap_all_vpes(kvm);
>                  vlpi_avail = true;
>          }
> 
> @@ -444,7 +441,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
> 
>   out:
>          if (vlpi_avail)
> -               map_all_vpes(dist);
> +               map_all_vpes(kvm);
> 
>          return ret;
>   }
> diff --git a/arch/arm64/kvm/vgic/vgic-v4.c b/arch/arm64/kvm/vgic/vgic-v4.c
> index ad06ba6c9b00..a413718be92b 100644
> --- a/arch/arm64/kvm/vgic/vgic-v4.c
> +++ b/arch/arm64/kvm/vgic/vgic-v4.c
> @@ -222,6 +222,11 @@ void vgic_v4_get_vlpi_state(struct vgic_irq *irq, bool *val)
>          *val = !!(*ptr & mask);
>   }
> 
> +int vgic_v4_request_vpe_irq(struct kvm_vcpu *vcpu, int irq)
> +{
> +       return request_irq(irq, vgic_v4_doorbell_handler, 0, "vcpu", vcpu);
> +}
> +
>   /**
>    * vgic_v4_init - Initialize the GICv4 data structures
>    * @kvm:       Pointer to the VM being initialized
> @@ -283,8 +288,7 @@ int vgic_v4_init(struct kvm *kvm)
>                          irq_flags &= ~IRQ_NOAUTOEN;
>                  irq_set_status_flags(irq, irq_flags);
> 
> -               ret = request_irq(irq, vgic_v4_doorbell_handler,
> -                                 0, "vcpu", vcpu);
> +               ret = vgic_v4_request_vpe_irq(vcpu, irq);
>                  if (ret) {
>                          kvm_err("failed to allocate vcpu IRQ%d\n", irq);
>                          /*
> diff --git a/arch/arm64/kvm/vgic/vgic.h b/arch/arm64/kvm/vgic/vgic.h
> index 0c8da72953f0..23e280fa0a16 100644
> --- a/arch/arm64/kvm/vgic/vgic.h
> +++ b/arch/arm64/kvm/vgic/vgic.h
> @@ -331,5 +331,6 @@ int vgic_v4_init(struct kvm *kvm);
>   void vgic_v4_teardown(struct kvm *kvm);
>   void vgic_v4_configure_vsgis(struct kvm *kvm);
>   void vgic_v4_get_vlpi_state(struct vgic_irq *irq, bool *val);
> +int vgic_v4_request_vpe_irq(struct kvm_vcpu *vcpu, int irq);
> 
>   #endif
> 

I ran stress test launch/teardown multiple VMs for 3hrs. The issue
is not reproducible. The same test fails in 10-30min without code
changes.

> --
> Without deviation from the norm, progress is not possible.
  

[dropping the now dead old kvmarm list]

On Wed, 18 Jan 2023 19:24:01 +0000,
Shanker Donthineni <sdonthineni@nvidia.com> wrote:
> 
> 
> 
> On 1/18/23 05:54, Marc Zyngier wrote:
> > External email: Use caution opening links or attachments
> > 
> > 
> > Shanker,
> > 
> > Please Cc all the KVM/arm64 reviewers when sending KVM/arm64 patches.
> > 
> > On Wed, 18 Jan 2023 02:23:48 +0000,
> > Shanker Donthineni <sdonthineni@nvidia.com> wrote:
> >> 
> >> Getting intermittent CPU soft lockups during the virtual machines
> >> teardown on a system with GICv4 features enabled. The function
> >> __synchronize_hardirq() has been waiting for IRQD_IRQ_INPROGRESS
> >> to be cleared forever as per the current implementation.
> >> 
> >> CPU stuck here for a long time leads to soft lockup:
> >>    while (irqd_irq_inprogress(&desc->irq_data))
> >>        cpu_relax();
> > 
> > Is it a soft-lockup from which the system recovers? or a livelock
> > which leaves the system dead?
> > 
> The system is not recovering, did a power cycle to recover.

This isn't a soft-lockup then. This is at best a livelock.

> > Are these two traces an indication of concurrent events? Or are they
> > far apart?
> > 
> Concurrent.

So you can see the VM being torn down while the vgic save sequence is
still in progress?

If you can actually see that, then this is a much bigger bug than the
simple race you are describing, and we're missing a reference on the
kvm structure. This would be a *MAJOR* bug.

Please post the full traces, not snippets. The absolutely full kernel
log, the configuration, what you run, how you run it, *EVERYTHING*. I
need to be able to reproduce this.

> 
> >> 
> >> irqreturn_t handle_irq_event(struct irq_desc *desc)
> >> {
> >>      irqd_set(&desc->irq_data, IRQD_IRQ_INPROGRESS);
> >>      raw_spin_unlock(&desc->lock);
> >> 
> >>      ret = handle_irq_event_percpu(desc);
> >> 
> >>      raw_spin_lock(&desc->lock);
> >>      irqd_clear(&desc->irq_data, IRQD_IRQ_INPROGRESS);
> >> }
> > 
> > How is that relevant to this trace? Do you see this function running
> > concurrently with the teardown? If it matters here, it must be a VPE
> > doorbell, right? But you claim that this is on a GICv4 platform, while
> > this would only affect GICv4.1... Or are you using GICv4.1?
> > 
> handle_irq_event() is running concurrently with irq_domain_activate_irq()
> which happens before free_irq() called. Corruption at [78.983544] and
> teardown started at [87.360891].

But that doesn't match the description you made of concurrent
events. Does it take more than 9 seconds for the vgic state to be
saved to memory?

> 
> [   78.983544] irqd_set_activated: lost IRQD_IRQ_INPROGRESS old=0x10401400, new=0x10441600
> 
> [   87.360891]  __synchronize_hardirq+0x48/0x140
> 
> Yes, I'm using GICv4.1, used these 2 functions to trace the issue.

Then *please* be precise in your descriptions. You send people in the
wrong direction.

[...]

> I ran stress test launch/teardown multiple VMs for 3hrs. The issue
> is not reproducible. The same test fails in 10-30min without code
> changes.

That doesn't add up with the earlier description of concurrent events,
with the VM teardown and the vgic saving running in parallel. My patch
doesn't prevent this.

So either your testing is insufficient, or your description of
concurrent events is inaccurate.

	M.
  

On 1/19/23 01:11, Marc Zyngier wrote:
> External email: Use caution opening links or attachments
> 
> 
> [dropping the now dead old kvmarm list]
> 
> On Wed, 18 Jan 2023 19:24:01 +0000,
> Shanker Donthineni <sdonthineni@nvidia.com> wrote:
>>
>>
>>
>> On 1/18/23 05:54, Marc Zyngier wrote:
>>> External email: Use caution opening links or attachments
>>>
>>>
>>> Shanker,
>>>
>>> Please Cc all the KVM/arm64 reviewers when sending KVM/arm64 patches.
>>>
>>> On Wed, 18 Jan 2023 02:23:48 +0000,
>>> Shanker Donthineni <sdonthineni@nvidia.com> wrote:
>>>>
>>>> Getting intermittent CPU soft lockups during the virtual machines
>>>> teardown on a system with GICv4 features enabled. The function
>>>> __synchronize_hardirq() has been waiting for IRQD_IRQ_INPROGRESS
>>>> to be cleared forever as per the current implementation.
>>>>
>>>> CPU stuck here for a long time leads to soft lockup:
>>>>     while (irqd_irq_inprogress(&desc->irq_data))
>>>>         cpu_relax();
>>>
>>> Is it a soft-lockup from which the system recovers? or a livelock
>>> which leaves the system dead?
>>>
>> The system is not recovering, did a power cycle to recover.
> 
> This isn't a soft-lockup then. This is at best a livelock.
> 
>>> Are these two traces an indication of concurrent events? Or are they
>>> far apart?
>>>
>> Concurrent.
> 
> So you can see the VM being torn down while the vgic save sequence is
> still in progress?
> 
> If you can actually see that, then this is a much bigger bug than the
> simple race you are describing, and we're missing a reference on the
> kvm structure. This would be a *MAJOR* bug.
> 
How do we know vGIC save sequence is in progress while VM is being teardown?
I'm launching/terminating ~32 VMs in a loop to reproduce the issue.
  
> Please post the full traces, not snippets. The absolutely full kernel
> log, the configuration, what you run, how you run it, *EVERYTHING*. I
> need to be able to reproduce this.
Sure, I'll share the complete boot log messages of host kernel next run.
  
> 
>>
>>>>
>>>> irqreturn_t handle_irq_event(struct irq_desc *desc)
>>>> {
>>>>       irqd_set(&desc->irq_data, IRQD_IRQ_INPROGRESS);
>>>>       raw_spin_unlock(&desc->lock);
>>>>
>>>>       ret = handle_irq_event_percpu(desc);
>>>>
>>>>       raw_spin_lock(&desc->lock);
>>>>       irqd_clear(&desc->irq_data, IRQD_IRQ_INPROGRESS);
>>>> }
>>>
>>> How is that relevant to this trace? Do you see this function running
>>> concurrently with the teardown? If it matters here, it must be a VPE
>>> doorbell, right? But you claim that this is on a GICv4 platform, while
>>> this would only affect GICv4.1... Or are you using GICv4.1?
>>>
>> handle_irq_event() is running concurrently with irq_domain_activate_irq()
>> which happens before free_irq() called. Corruption at [78.983544] and
>> teardown started at [87.360891].
> 
> But that doesn't match the description you made of concurrent
> events. Does it take more than 9 seconds for the vgic state to be
> saved to memory?

Are there any other possibilities of corrupting IRQD_IRQ_INPROGRESS
state bit other than concurrent accesses?

I did another experiment, changed the bit manipulation of the IRQD
state to atomic operations, the issue is not reproducible.

struct irq_common_data {
-       unsigned  int		__private state_use_accessors;
+	atomic_t		__private state_use_accessors;

+#define __irqd_to_state(d)       ACCESS_PRIVATE((d)->common, state_use_accessors)
+#define __irqd_get_state(d)      atomic_read(&__irqd_to_state(d))
+#define __irqd_set_state(d, m)   atomic_or((m), &__irqd_to_state(d))
+#define __irqd_clear_state(d, m) atomic_andnot((m), &__irqd_to_state(d))
> 
>>
>> [   78.983544] irqd_set_activated: lost IRQD_IRQ_INPROGRESS old=0x10401400, new=0x10441600
>>
>> [   87.360891]  __synchronize_hardirq+0x48/0x140
>>
>> Yes, I'm using GICv4.1, used these 2 functions to trace the issue.
> 
> Then *please* be precise in your descriptions. You send people in the
> wrong direction.
> 
It was my mistake not to mention the exact GIC version.

> [...]
> 
>> I ran stress test launch/teardown multiple VMs for 3hrs. The issue
>> is not reproducible. The same test fails in 10-30min without code
>> changes.
> 
> That doesn't add up with the earlier description of concurrent events,
> with the VM teardown and the vgic saving running in parallel. My patch
> doesn't prevent this.
> 
> So either your testing is insufficient, or your description of
> concurrent events is inaccurate.
> 

I'm using the unmodified 6.2.rc3/v6.0 from kernel.org and QEMU to
reproduce the issue.

Using the below steps for launching/terminating 32 VMs in loop. The
failure is intermittent. The same issue is reproducible with KVMTOOL
also.

numvms=32
socketcnt=1
iterations=100
while [ ${iterations} -ne 0 ]; do
  iterations=$(( iterations - 1 ))
  for idx in `seq 1 ${numvms}`; do
    vmname=vm${idx}
    cpunode=$((idx % socketcnt))
    cpuset=`lscpu  | grep "NUMA node${cpunode} CPU(s)" | awk -F' ' '{ print $4 }'`
    qemu-img create -f qcow2 -F raw -b /var/lib/libvirt/images/cloudimg.raw /var/lib/libvirt/images/${vmname}.qcow2
    virt-sysprep -a /var/lib/libvirt/images/${vmname}.qcow2 --hostname ${vmname}
    virt-install \
      --name ${vmname} \
      --numatune ${cpunode},mode=preferred \
      --vcpus vcpus=16,maxvcpus=16,cpuset=${cpuset} \
      --memory 4096 \
      --import \
      --noautoconsole \
      --os-variant name=ubuntu22.04 \
      --disk path=/var/lib/libvirt/images/${vmname}.qcow2 \
      --boot loader=/usr/share/AAVMF/AAVMF_CODE.fd,loader_ro=yes,loader_type=pflash
    if [ $? -ne 0 ]; then
      echo "Failed to create vm${idx}"
      while true ; do  sleep 1; done
    fi
    sleep 5
  done

  for idx in `seq 1 ${numvms}`; do
    echo -n "Checking vm${idx} boot: "
    while true; do
      ghostname=`virsh guestinfo vm${idx} --hostname 2>/dev/null | awk '{ print $3 }'`
      [ "${ghostname}" == "vm${idx}" ] && echo "done" && break
      sleep 1
    done
  done

  for idx in `seq 1 ${numvms}`; do
   virsh domstate vm${idx} >/dev/null 2>&1 || continue
   echo "Started shutdown & undefine vm${idx}..."
   virsh shutdown "vm${idx}" --mode agent >/dev/null 2>&1
   while true; do
     virsh domstate vm${idx} 2>/dev/null 2>&1 | grep -w "^shut off" >/dev/null 2>&1
     [ $? -eq 0 ] && break
     sleep 1
     virsh shutdown "vm${idx}" --mode agent >/dev/null 2>&1
   done
   virsh undefine --nvram --remove-all-storage vm${idx} 2>/dev/null
  done
done


>          M.
> 
> --
> Without deviation from the norm, progress is not possible.
  

On Thu, 19 Jan 2023 13:00:49 +0000,
Shanker Donthineni <sdonthineni@nvidia.com> wrote:
> 
> 
> 
> On 1/19/23 01:11, Marc Zyngier wrote:
> > So you can see the VM being torn down while the vgic save sequence is
> > still in progress?
> > 
> > If you can actually see that, then this is a much bigger bug than the
> > simple race you are describing, and we're missing a reference on the
> > kvm structure. This would be a *MAJOR* bug.
> > 
> How do we know vGIC save sequence is in progress while VM is being
> teardown?  I'm launching/terminating ~32 VMs in a loop to reproduce
> the issue.

Errr... *you* know when you are issuing the save ioctl, right? You
also know when you are terminating the VM (closing its fd or killing
the VMM).

>  
> > Please post the full traces, not snippets. The absolutely full kernel
> > log, the configuration, what you run, how you run it, *EVERYTHING*. I
> > need to be able to reproduce this.
> Sure, I'll share the complete boot log messages of host kernel next run.
>  
> > 
> >> 
> >>>> 
> >>>> irqreturn_t handle_irq_event(struct irq_desc *desc)
> >>>> {
> >>>>       irqd_set(&desc->irq_data, IRQD_IRQ_INPROGRESS);
> >>>>       raw_spin_unlock(&desc->lock);
> >>>> 
> >>>>       ret = handle_irq_event_percpu(desc);
> >>>> 
> >>>>       raw_spin_lock(&desc->lock);
> >>>>       irqd_clear(&desc->irq_data, IRQD_IRQ_INPROGRESS);
> >>>> }
> >>> 
> >>> How is that relevant to this trace? Do you see this function running
> >>> concurrently with the teardown? If it matters here, it must be a VPE
> >>> doorbell, right? But you claim that this is on a GICv4 platform, while
> >>> this would only affect GICv4.1... Or are you using GICv4.1?
> >>> 
> >> handle_irq_event() is running concurrently with irq_domain_activate_irq()
> >> which happens before free_irq() called. Corruption at [78.983544] and
> >> teardown started at [87.360891].
> > 
> > But that doesn't match the description you made of concurrent
> > events. Does it take more than 9 seconds for the vgic state to be
> > saved to memory?
> 
> Are there any other possibilities of corrupting IRQD_IRQ_INPROGRESS
> state bit other than concurrent accesses?

Forget about this bit. You said that we could see the VM teardown
happening *at the same time* as the vgic state saving, despite the
vgic device holding a reference on the kvm structure. If that's the
case, this bit is the least of our worries. Think of the consequences
for a second...

[...]

> Using the below steps for launching/terminating 32 VMs in loop. The
> failure is intermittent. The same issue is reproducible with KVMTOOL
> also.

kvmtool never issue a KVM_DEV_ARM_VGIC_GRP_CTRL with the
KVM_DEV_ARM_ITS_SAVE_TABLES argument, so the code path we discussed is
never used. What is the exact problem you're observing with kvmtool
as the VMM?

	M.
  

Hi Marc,

On 1/19/23 08:01, Marc Zyngier wrote:
> External email: Use caution opening links or attachments
> 
> 
> On Thu, 19 Jan 2023 13:00:49 +0000,
> Shanker Donthineni <sdonthineni@nvidia.com> wrote:
>>
>>
>>
>> On 1/19/23 01:11, Marc Zyngier wrote:
>>> So you can see the VM being torn down while the vgic save sequence is
>>> still in progress?
>>>
>>> If you can actually see that, then this is a much bigger bug than the
>>> simple race you are describing, and we're missing a reference on the
>>> kvm structure. This would be a *MAJOR* bug.
>>>
>> How do we know vGIC save sequence is in progress while VM is being
>> teardown?  I'm launching/terminating ~32 VMs in a loop to reproduce
>> the issue.
> 
> Errr... *you* know when you are issuing the save ioctl, right? You
> also know when you are terminating the VM (closing its fd or killing
> the VMM).
> 
>>
>>> Please post the full traces, not snippets. The absolutely full kernel
>>> log, the configuration, what you run, how you run it, *EVERYTHING*. I
>>> need to be able to reproduce this.
>> Sure, I'll share the complete boot log messages of host kernel next run.
>>
>>>
>>>>
>>>>>>
>>>>>> irqreturn_t handle_irq_event(struct irq_desc *desc)
>>>>>> {
>>>>>>        irqd_set(&desc->irq_data, IRQD_IRQ_INPROGRESS);
>>>>>>        raw_spin_unlock(&desc->lock);
>>>>>>
>>>>>>        ret = handle_irq_event_percpu(desc);
>>>>>>
>>>>>>        raw_spin_lock(&desc->lock);
>>>>>>        irqd_clear(&desc->irq_data, IRQD_IRQ_INPROGRESS);
>>>>>> }
>>>>>
>>>>> How is that relevant to this trace? Do you see this function running
>>>>> concurrently with the teardown? If it matters here, it must be a VPE
>>>>> doorbell, right? But you claim that this is on a GICv4 platform, while
>>>>> this would only affect GICv4.1... Or are you using GICv4.1?
>>>>>
>>>> handle_irq_event() is running concurrently with irq_domain_activate_irq()
>>>> which happens before free_irq() called. Corruption at [78.983544] and
>>>> teardown started at [87.360891].
>>>
>>> But that doesn't match the description you made of concurrent
>>> events. Does it take more than 9 seconds for the vgic state to be
>>> saved to memory?
>>
>> Are there any other possibilities of corrupting IRQD_IRQ_INPROGRESS
>> state bit other than concurrent accesses?
> 
> Forget about this bit. You said that we could see the VM teardown
> happening *at the same time* as the vgic state saving, despite the
> vgic device holding a reference on the kvm structure. If that's the
> case, this bit is the least of our worries. Think of the consequences
> for a second...
> 
> [...]
> 
>> Using the below steps for launching/terminating 32 VMs in loop. The
>> failure is intermittent. The same issue is reproducible with KVMTOOL
>> also.
> 
> kvmtool never issue a KVM_DEV_ARM_VGIC_GRP_CTRL with the
> KVM_DEV_ARM_ITS_SAVE_TABLES argument, so the code path we discussed is
> never used. What is the exact problem you're observing with kvmtool
> as the VMM?
> 
Unfortunately I didn't capture the log messages of kvmtool based VM boot.
Saw once kernel crash. Now I'm trying to reproduce the issue with QEMU tool
and share complete log messages.

>          M.
> 
> --
> Without deviation from the norm, progress is not possible.
  

Hi Marc,

On 1/19/23 08:01, Marc Zyngier wrote:
>> On 1/19/23 01:11, Marc Zyngier wrote:
>>> So you can see the VM being torn down while the vgic save sequence is
>>> still in progress?
>>>
>>> If you can actually see that, then this is a much bigger bug than the
>>> simple race you are describing, and we're missing a reference on the
>>> kvm structure. This would be a*MAJOR*  bug.
>>>
>> How do we know vGIC save sequence is in progress while VM is being
>> teardown?  I'm launching/terminating ~32 VMs in a loop to reproduce
>> the issue.
> Errr...*you*  know when you are issuing the save ioctl, right? You
> also know when you are terminating the VM (closing its fd or killing
> the VMM).
> 

Added debug statements to trace the code patch, and tagged each log message
with 'struct kvm *'. Attached the complete kernel log messages including
debug messages.

All 32 VMs launched, time 258s to 291s
  [  258.519837] kvm_create_vm(1236) called kvm=ffff8000303e0000  --> 1st VM
  ...
  [  291.801179] kvm_create_vm(1236) called kvm=ffff800057a60000   --> 32nd VM

Test script inside VM issues poweroff command after sleeping 200sec.

Working case kvm=ffff8000303e0000:

  $ cat gicv4-debug.txt | grep ffff8000303e0000
   [  258.519837] kvm_create_vm(1236) called kvm=ffff8000303e0000
   [  258.667101] vgic_v4_init(267) called kvm=ffff8000303e0000 doorbell=140(64)
   [  517.942167] vgic_set_common_attr(263) called kvm=ffff8000303e0000
   [  517.948415] vgic_v3_save_pending_tables(397) called kvm=ffff8000303e0000
   [  517.955602] vgic_v3_save_pending_tables(448) called kvm=ffff8000303e0000
   [  518.099696] kvm_vm_release(1374) called kvm=ffff8000303e0000
   [  518.126833] vgic_v4_teardown(323) started kvm=ffff8000303e0000 doorbell=140(64)
   [  518.134677] vgic_v4_teardown(333) finished kvm=ffff8000303e0000 doorbell=140(64)

Not working case kvm=ffff80001e0a0000:

  $ cat gicv4-debug.txt | grep ffff80001e0a0000
   [  277.684981] kvm_create_vm(1236) called kvm=ffff80001e0a0000
   [  278.158511] vgic_v4_init(267) called kvm=ffff80001e0a0000 doorbell=20812(64)
   [  545.079117] vgic_set_common_attr(263) called kvm=ffff80001e0a0000
   [  545.085358] vgic_v3_save_pending_tables(397) called kvm=ffff80001e0a0000
   [  545.092580] vgic_v3_save_pending_tables(448) called kvm=ffff80001e0a0000
   [  545.099562] irq: irqd_set_activated: CPU49 IRQ20821 lost IRQD_IRQ_INPROGRESS old=0x10401400 new=0x10401600, expected=0x10441600 kvm=ffff80001e0a0000^M
   [  545.113177] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS set time [545.099561]^M
   [  545.121454] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS clr time [545.099562]^M
   [  545.129755] irq: irqd_set_activated: CPU49 IRQ20826 lost IRQD_IRQ_INPROGRESS old=0x10441400 new=0x10441600, expected=0x10401600 kvm=ffff80001e0a0000^M
   [  545.143365] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS set time [545.129754]^M
   [  545.151654] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS clr time [545.129755]^M
   [  545.163250] kvm_vm_release(1374) called kvm=ffff80001e0a0000
   [  545.169204] vgic_v4_teardown(323) started kvm=ffff80001e0a0000 doorbell=20812(64)

IRQD_IRQ_INPROGRESS is corrupted before calling kvm_vm_release(),

   Time             State
  -----------------------
   [277.684981]    VM created
   [278.158511]    Started executing GICv4 init code
   [545.092580]    VGIC save/pending called
   [545.099561]    set IRQD_IRQ_INPROGRESS
   [545.099562]    clr IRQD_IRQ_INPROGRESS & irqd_set_activated()  ---> same/close time
   [545.163250]    kvm_vm_release() called

Software:
   Linux kernel: 6.2.0-rc4
   QEMU: 7.1.0

Debug patch:

  diff --git a/arch/arm64/kvm/vgic/vgic-kvm-device.c b/arch/arm64/kvm/vgic/vgic-kvm-device.c
index edeac2380591..da922ac72dc6 100644
--- a/arch/arm64/kvm/vgic/vgic-kvm-device.c
+++ b/arch/arm64/kvm/vgic/vgic-kvm-device.c
@@ -260,6 +260,7 @@ static int vgic_set_common_attr(struct kvm_device *dev,
                                 mutex_unlock(&dev->kvm->lock);
                                 return -EBUSY;
                         }
+                       pr_err("%s(%d) called kvm=%px\n",__func__,__LINE__,dev->kvm);
                         r = vgic_v3_save_pending_tables(dev->kvm);
                         unlock_all_vcpus(dev->kvm);
                         mutex_unlock(&dev->kvm->lock);
diff --git a/arch/arm64/kvm/vgic/vgic-v3.c b/arch/arm64/kvm/vgic/vgic-v3.c
index 2074521d4a8c..91b53646f7ec 100644
--- a/arch/arm64/kvm/vgic/vgic-v3.c
+++ b/arch/arm64/kvm/vgic/vgic-v3.c
@@ -394,6 +394,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
          * and enabling of the doorbells have already been done.
          */
         if (kvm_vgic_global_state.has_gicv4_1) {
+               pr_err("%s(%d) called kvm=%px \n",__func__,__LINE__,kvm);
                 unmap_all_vpes(dist);
                 vlpi_avail = true;
         }
@@ -443,8 +444,10 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
         }

  out:
-       if (vlpi_avail)
+       if (vlpi_avail) {
+               pr_err("%s(%d) called kvm=%px \n",__func__,__LINE__,kvm);
                 map_all_vpes(dist);
+       }
diff --git a/arch/arm64/kvm/vgic/vgic-v4.c b/arch/arm64/kvm/vgic/vgic-v4.c
index ad06ba6c9b00..71d234844acd 100644
--- a/arch/arm64/kvm/vgic/vgic-v4.c
+++ b/arch/arm64/kvm/vgic/vgic-v4.c
@@ -264,10 +264,14 @@ int vgic_v4_init(struct kvm *kvm)
                 dist->its_vm.vpes = NULL;
                 return ret;
         }
+       pr_err("%s(%d) called kvm=%px doorbell=%d(%d)\n", __func__, __LINE__, kvm, dist->its_vm.vpes[0]->irq, nr_vcpus);

         kvm_for_each_vcpu(i, vcpu, kvm) {
                 int irq = dist->its_vm.vpes[i]->irq;
                 unsigned long irq_flags = DB_IRQ_FLAGS;
+               struct irq_desc *desc = irq_to_desc(irq);
+
+               desc->irq_data.dbgdata = kvm;

                 /*
                  * Don't automatically enable the doorbell, as we're
@@ -316,6 +320,8 @@ void vgic_v4_teardown(struct kvm *kvm)
         if (!its_vm->vpes)
                 return;

+       pr_err("%s(%d) started kvm=%px doorbell=%d(%d)\n", __func__, __LINE__, kvm, its_vm->vpes[0]->irq, its_vm->nr_vpes);
+
         for (i = 0; i < its_vm->nr_vpes; i++) {
                 struct kvm_vcpu *vcpu = kvm_get_vcpu(kvm, i);
                 int irq = its_vm->vpes[i]->irq;
@@ -324,6 +330,7 @@ void vgic_v4_teardown(struct kvm *kvm)
                 free_irq(irq, vcpu);
         }

+       pr_err("%s(%d) finished kvm=%px doorbell=%d(%d)\n", __func__, __LINE__, kvm, its_vm->vpes[0]->irq, its_vm->nr_vpes);
         its_free_vcpu_irqs(its_vm);
         kfree(its_vm->vpes);
         its_vm->nr_vpes = 0;
diff --git a/include/linux/irq.h b/include/linux/irq.h
index c3eb89606c2b..3e3d6ae8d59f 100644
--- a/include/linux/irq.h
+++ b/include/linux/irq.h
@@ -187,6 +187,8 @@ struct irq_data {
         struct irq_data         *parent_data;
  #endif
         void                    *chip_data;
+       unsigned long           dbgtime[2];
+       void                    *dbgdata;
  };

  /*
@@ -390,12 +392,35 @@ static inline bool irqd_is_activated(struct irq_data *d)

  static inline void irqd_set_activated(struct irq_data *d)
  {
-       __irqd_to_state(d) |= IRQD_ACTIVATED;
+       atomic_t *state = (atomic_t *)(&__irqd_to_state(d));
+       unsigned int new, old = __irqd_to_state(d);
+
+       new = atomic_xchg(state, old | IRQD_ACTIVATED);
+
+       if ((old ^ new) & IRQD_IRQ_INPROGRESS) {
+               pr_err("%s: CPU%u IRQ%u lost IRQD_IRQ_INPROGRESS old=0x%x new=0x%x, expected=0x%x kvm=%px\n",
+                      __func__, raw_smp_processor_id(), d->irq,
+                       old, old | IRQD_ACTIVATED, new | IRQD_ACTIVATED, d->dbgdata);
+               pr_err("%s: IRQD_IRQ_INPROGRESS set time [%5lu.%06u] \n", __func__, d->dbgtime[0], do_div(d->dbgtime[0], 1000000000)/1000);
+               pr_err("%s: IRQD_IRQ_INPROGRESS clr time [%5lu.%06u] \n", __func__, d->dbgtime[1], do_div(d->dbgtime[1], 1000000000)/1000);
+       }
+
  }

  static inline void irqd_clr_activated(struct irq_data *d)
  {
-       __irqd_to_state(d) &= ~IRQD_ACTIVATED;
+       atomic_t *state = (atomic_t *)(&__irqd_to_state(d));
+       unsigned int new, old = __irqd_to_state(d);
+
+       new = atomic_xchg(state, old & ~IRQD_ACTIVATED);
+
+       if ((old ^ new) & IRQD_IRQ_INPROGRESS) {
+               pr_err("%s: CPU%u IRQ%u lost IRQD_IRQ_INPROGRESS old=0x%x new=0x%x, expected=0x%x kvm=%px\n",
+                      __func__, raw_smp_processor_id(), d->irq,
+                       old, old & ~IRQD_ACTIVATED, new & ~IRQD_ACTIVATED, d->dbgdata);
+               pr_err("%s: IRQD_IRQ_INPROGRESS set time [%5lu.%06u] \n", __func__, d->dbgtime[0], do_div(d->dbgtime[0], 1000000000)/1000);
+               pr_err("%s: IRQD_IRQ_INPROGRESS clr time [%5lu.%06u] \n", __func__, d->dbgtime[1], do_div(d->dbgtime[1], 1000000000)/1000);
+       }
  }

  static inline bool irqd_is_started(struct irq_data *d)
diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c
index 9489f93b3db3..5891ec27ae93 100644
--- a/kernel/irq/handle.c
+++ b/kernel/irq/handle.c
@@ -205,12 +205,14 @@ irqreturn_t handle_irq_event(struct irq_desc *desc)

         desc->istate &= ~IRQS_PENDING;
         irqd_set(&desc->irq_data, IRQD_IRQ_INPROGRESS);
+       desc->irq_data.dbgtime[0] = sched_clock();
         raw_spin_unlock(&desc->lock);

         ret = handle_irq_event_percpu(desc);

         raw_spin_lock(&desc->lock);
         irqd_clear(&desc->irq_data, IRQD_IRQ_INPROGRESS);
+       desc->irq_data.dbgtime[1] = sched_clock();
         return ret;
  }

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 9c60384b5ae0..0fbf17ea3adb 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -1233,6 +1233,7 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
         preempt_notifier_inc();
         kvm_init_pm_notifier(kvm);

+       pr_err("%s(%d) called kvm=%px\n",__func__,__LINE__,kvm);
         return kvm;

  out_err:
@@ -1370,6 +1371,7 @@ static int kvm_vm_release(struct inode *inode, struct file *filp)
  {
         struct kvm *kvm = filp->private_data;

+       pr_err("%s(%d) called kvm=%px\n",__func__,__LINE__, kvm);
         kvm_irqfd_release(kvm);

         kvm_put_kvm(kvm);

         return ret;
  }
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd4f0]
[    0.000000] Linux version 6.0.0 (sdonthineni@SDONTHINENI-DESKTOP) (aarch64-buildroot-linux-gnu-gcc.br_real (Buildroot 2022.08-1040-g3e947bf73d-dirty) 12.2.0, GNU ld (GNU Bin3
[    0.000000] efi: EFI v2.70 by EDK II
[    0.000000] efi: RTPROP=0x2065a95a98 ACPI 2.0=0x204b8f0018 SMBIOS=0xffff0000 SMBIOS 3.0=0x2068e30000 MEMATTR=0x2064104018 ESRT=0x2063e52e18 MEMRESERVE=0x204a909d98
[    0.000000] esrt: Reserving ESRT space from 0x0000002063e52e18 to 0x0000002063e52e50.
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x000000204B8F0018 000024 (v02 NVIDIA)
[    0.000000] ACPI: XSDT 0x000000204B8FE218 000104 (v01 NVIDIA TH500    00000001 NVDA 00000001)
[    0.000000] ACPI: FACP 0x000000204B8F7518 000114 (v06 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: DSDT 0x000000204B8F7698 0016B7 (v02 NVIDIA TH500    00000001 INTL 20200717)
[    0.000000] ACPI: SDEI 0x000000204B8FFF98 000024 (v01 NVIDIA TH500    00000001 NVDA 00000001)
[    0.000000] ACPI: HEST 0x000000204B8FE998 000E88 (v01 NVIDIA TH500    00000001 NVDA 00000001)
[    0.000000] ACPI: BERT 0x000000204B8FFA98 000030 (v01 NVIDIA TH500    00000001 NVDA 00000001)
[    0.000000] ACPI: EINJ 0x000000204B8FFB18 000170 (v01 NVIDIA TH500    00000001 NVDA 00000001)
[    0.000000] ACPI: FPDT 0x000000204B8FFD98 000034 (v01 NVIDIA TH500    00000001 NVDA 00000001)
[    0.000000] ACPI: GTDT 0x000000204B8FE898 000084 (v03 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: APIC 0x000000204B8FBE98 001788 (v05 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: PPTT 0x000000204B8F9218 0021E0 (v03 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204B8F0098 0061C4 (v02 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: SPCR 0x000000204B8FFE18 000050 (v02 NVIDIA TEGRAUAR 00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204B8FE498 0000CC (v02 NVIDIA SERIAL   00000001 INTL 20200717)
[    0.000000] ACPI: SSDT 0x000000204A950018 001341 (v02 NVIDIA BPMP_S0  00000001 INTL 20200717)
[    0.000000] ACPI: MCFG 0x000000204A95FE18 00008C (v01 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204A95F398 000351 (v02 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204A95F798 000353 (v02 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204B8FDA18 000353 (v02 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204B8FDE18 000353 (v02 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204A95D818 000353 (v02 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204A95ED18 0004C2 (v02 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: IORT 0x000000204A95DC18 000433 (v06 NVIDIA NVIDIORT 00000001 ARMH 00010000)
[    0.000000] ACPI: MPAM 0x000000204B8FFE98 000084 (v01 NVIDIA NVIDMPAM 00000001 ARMH 00010000)
[    0.000000] ACPI: SRAT 0x000000204A95E298 000700 (v03 NVIDIA NVIDSRAT 00000001 ARMH 00010000)
[    0.000000] ACPI: SLIT 0x000000204A95FB98 00012C (v01 NVIDIA NVIDSLIT 00000001 ARMH 00010000)
[    0.000000] ACPI: APMT 0x000000204B8FE598 00013C (v00 NVIDIA TH500    00000001 NVDA 00000001)
[    0.000000] ACPI: SPMI 0x000000204A95FF18 000041 (v05 NVIDIA NVIDSPMI 00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204B8FE718 00007C (v02 INTEL  RamDisk  00001000 INTL 20200717)
[    0.000000] ACPI: NFIT 0x000000204A95FD18 000060 (v01 NVIDIA TH500    00000001 NVDA 00000001)
[    0.000000] ACPI: SPCR: console: pl011,mmio32,0xc280000,115200
[    0.000000] earlycon: pl11 at MMIO32 0x000000000c280000 (options '115200')
[    0.000000] printk: bootconsole [pl11] enabled
[    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x80000000-0x206bffffff]
[    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x206d400000-0x206d7fffff]
[    0.000000] ACPI: SRAT: Node 1 PXM 8 [mem 0x400000000000-0x40ffffffffff] hotplug
[    0.000000] ACPI: SRAT: Node 2 PXM 9 [mem 0x410000000000-0x41ffffffffff] hotplug
[    0.000000] ACPI: SRAT: Node 3 PXM 10 [mem 0x420000000000-0x42ffffffffff] hotplug
[    0.000000] ACPI: SRAT: Node 4 PXM 11 [mem 0x430000000000-0x43ffffffffff] hotplug
[    0.000000] ACPI: SRAT: Node 5 PXM 12 [mem 0x440000000000-0x44ffffffffff] hotplug
[    0.000000] ACPI: SRAT: Node 6 PXM 13 [mem 0x450000000000-0x45ffffffffff] hotplug
[    0.000000] ACPI: SRAT: Node 7 PXM 14 [mem 0x460000000000-0x46ffffffffff] hotplug
[    0.000000] ACPI: SRAT: Node 8 PXM 15 [mem 0x470000000000-0x47ffffffffff] hotplug
[    0.000000] NUMA: NODE_DATA [mem 0x206afaaec0-0x206afaffff]
[    0.000000] NUMA: Initmem setup node 1 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x206afa5d80-0x206afaaebf]
[    0.000000] NUMA: NODE_DATA(1) on node 0
[    0.000000] NUMA: Initmem setup node 2 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x206af9aec0-0x206af9ffff]
[    0.000000] NUMA: NODE_DATA(2) on node 0
[    0.000000] NUMA: Initmem setup node 3 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x206af95d80-0x206af9aebf]
[    0.000000] NUMA: NODE_DATA(3) on node 0
[    0.000000] NUMA: Initmem setup node 4 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x206af90c40-0x206af95d7f]
[    0.000000] NUMA: NODE_DATA(4) on node 0
[    0.000000] NUMA: Initmem setup node 5 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x206af8bb00-0x206af90c3f]
[    0.000000] NUMA: NODE_DATA(5) on node 0
[    0.000000] NUMA: Initmem setup node 6 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x206af869c0-0x206af8baff]
[    0.000000] NUMA: NODE_DATA(6) on node 0
[    0.000000] NUMA: Initmem setup node 7 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x206af81880-0x206af869bf]
[    0.000000] NUMA: NODE_DATA(7) on node 0
[    0.000000] NUMA: Initmem setup node 8 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x206af7c740-0x206af8187f]
[    0.000000] NUMA: NODE_DATA(8) on node 0
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000206d7fffff]
[    0.000000]   Device   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000fffdffff]
[    0.000000]   node   0: [mem 0x00000000fffe0000-0x00000000ffffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x000000202b65ffff]
[    0.000000]   node   0: [mem 0x000000202b660000-0x000000202b7cffff]
[    0.000000]   node   0: [mem 0x000000202b7d0000-0x0000002042c9ffff]
[    0.000000]   node   0: [mem 0x0000002042ca0000-0x000000204a7bffff]
[    0.000000]   node   0: [mem 0x000000204a7c0000-0x000000204a82ffff]
[    0.000000]   node   0: [mem 0x000000204a830000-0x000000204a87ffff]
[    0.000000]   node   0: [mem 0x000000204a880000-0x000000204aeaffff]
[    0.000000]   node   0: [mem 0x000000204aeb0000-0x000000204af4ffff]
[    0.000000]   node   0: [mem 0x000000204af50000-0x000000204af6ffff]
[    0.000000]   node   0: [mem 0x000000204af70000-0x000000204afbffff]
[    0.000000]   node   0: [mem 0x000000204afc0000-0x000000204b32ffff]
[    0.000000]   node   0: [mem 0x000000204b330000-0x000000204b3cffff]
[    0.000000]   node   0: [mem 0x000000204b3d0000-0x000000204b4effff]
[    0.000000]   node   0: [mem 0x000000204b4f0000-0x000000204b5effff]
[    0.000000]   node   0: [mem 0x000000204b5f0000-0x000000204b63ffff]
[    0.000000]   node   0: [mem 0x000000204b640000-0x000000204b83ffff]
[    0.000000]   node   0: [mem 0x000000204b840000-0x000000204b84ffff]
[    0.000000]   node   0: [mem 0x000000204b850000-0x000000204b8effff]
[    0.000000]   node   0: [mem 0x000000204b8f0000-0x000000204b8fffff]
[    0.000000]   node   0: [mem 0x000000204b900000-0x000000204baeffff]
[    0.000000]   node   0: [mem 0x000000204baf0000-0x0000002068dbffff]
[    0.000000]   node   0: [mem 0x0000002068dc0000-0x0000002068dfffff]
[    0.000000]   node   0: [mem 0x0000002068e00000-0x0000002068e1ffff]
[    0.000000]   node   0: [mem 0x0000002068e20000-0x0000002068e5ffff]
[    0.000000]   node   0: [mem 0x0000002068e60000-0x000000206bffffff]
[    0.000000]   node   0: [mem 0x000000206d400000-0x000000206d7fffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000206d7fffff]
[    0.000000] Initmem setup node 1 as memoryless
[    0.000000] Initmem setup node 2 as memoryless
[    0.000000] Initmem setup node 3 as memoryless
[    0.000000] Initmem setup node 4 as memoryless
[    0.000000] Initmem setup node 5 as memoryless
[    0.000000] Initmem setup node 6 as memoryless
[    0.000000] Initmem setup node 7 as memoryless
[    0.000000] Initmem setup node 8 as memoryless
[    0.000000] On node 0, zone Normal: 320 pages in unavailable ranges
[    0.000000] On node 0, zone Normal: 4736 pages in unavailable ranges
[    0.000000] crashkernel reserved: 0x0000001f2b600000 - 0x000000202b600000 (4096 MB)
[    0.000000] psci: probing for conduit method from ACPI.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.2
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x0 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x30000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x40000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x50000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x60000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x70000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x80000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x90000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xa0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xb0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xc0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xe0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x100000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x110000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x120000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x130000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x140000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x150000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x160000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x170000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x180000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x190000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1a0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1c0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1d0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1e0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1f0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x200000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x210000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x220000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x230000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x240000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x250000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x260000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x270000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x280000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x290000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2a0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2b0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2c0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2d0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2e0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2f0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x300000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x310000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x320000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x330000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x340000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x350000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x360000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x370000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x380000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3a0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3b0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3c0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3d0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3e0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3f0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x410000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x420000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x430000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x440000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x460000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x480000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x490000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x4a0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x4b0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x4c0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x4d0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x4e0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x4f0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x500000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x510000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x520000 -> Node 0
[    0.000000] percpu: Embedded 2 pages/cpu s42728 r8192 d80152 u131072
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: Virtualization Host Extensions
[    0.000000] CPU features: detected: Hardware dirty bit management
[    0.000000] CPU features: detected: Spectre-v4
[    0.000000] alternatives: patching kernel code
[    0.000000] Fallback order for Node 0: 0
[    0.000000] Fallback order for Node 1: 1 0
[    0.000000] Fallback order for Node 2: 2 0
[    0.000000] Fallback order for Node 3: 3 0
[    0.000000] Fallback order for Node 4: 4 0
[    0.000000] Fallback order for Node 5: 5 0
[    0.000000] Fallback order for Node 6: 6 0
[    0.000000] Fallback order for Node 7: 7 0
[    0.000000] Fallback order for Node 8: 8 0
[    0.000000] Built 9 zonelists, mobility grouping on.  Total pages: 2090052
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=(loop)/boot/Image earlycon maxcpus=72 crashkernel=4096M,high default_hugepagesz=2M nokaslr cma=0 kvm-arm.vgic_v4_enable=1
[    0.000000] Unknown kernel command line parameters "nokaslr BOOT_IMAGE=(loop)/boot/Image", will be passed to user space.
[    0.000000] printk: log_buf_len individual max cpu contribution: 4096 bytes
[    0.000000] printk: log_buf_len total cpu_extra contributions: 299008 bytes
[    0.000000] printk: log_buf_len min size: 131072 bytes
[    0.000000] printk: log_buf_len: 524288 bytes
[    0.000000] printk: early log buf free: 115304(87%)
[    0.000000] Dentry cache hash table entries: 8388608 (order: 10, 67108864 bytes, linear)
[    0.000000] Inode-cache hash table entries: 4194304 (order: 9, 33554432 bytes, linear)
[    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
[    0.000000] software IO TLB: area num 128.
[    0.000000] software IO TLB: mapped [mem 0x00000000fbfe0000-0x00000000fffe0000] (64MB)
[    0.000000] Memory: 129057280K/133894144K available (11456K kernel code, 1332K rwdata, 3776K rodata, 2240K init, 1490K bss, 4836864K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=74, Nodes=9
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=74.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=74
[    0.000000] NR_IRQS: 524288, nr_irqs: 524288, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 960 SPIs implemented
[    0.000000] GICv3: 320 Extended SPIs implemented
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI
[    0.000000] GICv3: GICv4 features: DirectLPI RVPEID Valid+Dirty
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000022080000
[    0.000000] SRAT: PXM 0 -> ITS 0 -> Node 0
[    0.000000] ITS [mem 0x22040000-0x2205ffff]
[    0.000000] ITS@0x0000000022040000: Single VMOVP capable
[    0.000000] ITS@0x0000000022040000: Using GICv4.1 mode 00000000 00000001
[    0.000000] ITS@0x0000000022040000: allocated 8192 Devices @100550000 (indirect, esz 8, psz 64K, shr 1)
[    0.000000] ITS@0x0000000022040000: allocated 32768 Interrupt Collections @100560000 (flat, esz 2, psz 64K, shr 1)
[    0.000000] ITS@0x0000000022040000: allocated 16384 Virtual CPUs @100580000 (flat, esz 8, psz 64K, shr 1)
[    0.000000] GICv3: using LPI property table @0x0000000100570000
[    0.000000] ITS: Using DirectLPI for VPE invalidation
[    0.000000] ITS: Enabling GICv4 support
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x00000001005b0000
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] kfence: initialized - using 33554432 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)
[    0.000000] arch_timer: cp15 timer(s) running at 1000.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns
[    0.008674] Console: colour dummy device 80x25
[    0.013430] mempolicy: Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl
[    0.025369] ACPI: Core revision 20220331
[    0.029638] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)
[    0.040927] pid_max: default: 75776 minimum: 592
[    0.045853] LSM: Security Framework initializing
[    0.050805] Mount-cache hash table entries: 131072 (order: 4, 1048576 bytes, linear)
[    0.059065] Mountpoint-cache hash table entries: 131072 (order: 4, 1048576 bytes, linear)
[    0.068227] cblist_init_generic: Setting adjustable number of callback queues.
[    0.075930] cblist_init_generic: Setting shift to 7 and lim to 1.
[    0.082467] rcu: Hierarchical SRCU implementation.
[    0.087558] rcu:     Max phase no-delay instances is 1000.
[    0.093199] Platform MSI: ITS@0x22040000 domain created
[    0.098756] PCI/MSI: ITS@0x22040000 domain created
[    0.103851] Remapping and enabling EFI services.
[    0.109309] smp: Bringing up secondary CPUs ...
[    0.121815] Detected PIPT I-cache on CPU1
[    0.121831] GICv3: CPU1: found redistributor 30000 region 0:0x0000000022140000
[    0.121838] GICv3: CPU1: using allocated LPI pending table @0x00000001005c0000
[    0.121850] CPU1: Booted secondary processor 0x0000030000 [0x410fd4f0]
[    0.129578] Detected PIPT I-cache on CPU2
[    0.129593] GICv3: CPU2: found redistributor 40000 region 0:0x0000000022180000
[    0.129599] GICv3: CPU2: using allocated LPI pending table @0x00000001005d0000
[    0.129611] CPU2: Booted secondary processor 0x0000040000 [0x410fd4f0]
[    0.137333] Detected PIPT I-cache on CPU3
[    0.137347] GICv3: CPU3: found redistributor 50000 region 0:0x00000000221c0000
[    0.137354] GICv3: CPU3: using allocated LPI pending table @0x00000001005e0000
[    0.137366] CPU3: Booted secondary processor 0x0000050000 [0x410fd4f0]
[    0.145051] Detected PIPT I-cache on CPU4
[    0.145064] GICv3: CPU4: found redistributor 60000 region 0:0x0000000022200000
[    0.145071] GICv3: CPU4: using allocated LPI pending table @0x00000001005f0000
[    0.145081] CPU4: Booted secondary processor 0x0000060000 [0x410fd4f0]
[    0.152764] Detected PIPT I-cache on CPU5
[    0.152777] GICv3: CPU5: found redistributor 70000 region 0:0x0000000022240000
[    0.152783] GICv3: CPU5: using allocated LPI pending table @0x0000000100600000
[    0.152794] CPU5: Booted secondary processor 0x0000070000 [0x410fd4f0]
[    0.160528] Detected PIPT I-cache on CPU6
[    0.160543] GICv3: CPU6: found redistributor 80000 region 0:0x0000000022280000
[    0.160550] GICv3: CPU6: using allocated LPI pending table @0x0000000100610000
[    0.160560] CPU6: Booted secondary processor 0x0000080000 [0x410fd4f0]
[    0.168297] Detected PIPT I-cache on CPU7
[    0.168311] GICv3: CPU7: found redistributor 90000 region 0:0x00000000222c0000
[    0.168318] GICv3: CPU7: using allocated LPI pending table @0x0000000100620000
[    0.168329] CPU7: Booted secondary processor 0x0000090000 [0x410fd4f0]
[    0.176070] Detected PIPT I-cache on CPU8
[    0.176084] GICv3: CPU8: found redistributor a0000 region 0:0x0000000022300000
[    0.176091] GICv3: CPU8: using allocated LPI pending table @0x0000000100630000
[    0.176102] CPU8: Booted secondary processor 0x00000a0000 [0x410fd4f0]
[    0.183845] Detected PIPT I-cache on CPU9
[    0.183861] GICv3: CPU9: found redistributor b0000 region 0:0x0000000022340000
[    0.183867] GICv3: CPU9: using allocated LPI pending table @0x0000000100640000
[    0.183879] CPU9: Booted secondary processor 0x00000b0000 [0x410fd4f0]
[    0.191667] Detected PIPT I-cache on CPU10
[    0.191684] GICv3: CPU10: found redistributor c0000 region 0:0x0000000022380000
[    0.191691] GICv3: CPU10: using allocated LPI pending table @0x0000000100650000
[    0.191702] CPU10: Booted secondary processor 0x00000c0000 [0x410fd4f0]
[    0.199395] Detected PIPT I-cache on CPU11
[    0.199443] GICv3: CPU11: found redistributor e0000 region 0:0x0000000022400000
[    0.199450] GICv3: CPU11: using allocated LPI pending table @0x0000000100660000
[    0.199461] CPU11: Booted secondary processor 0x00000e0000 [0x410fd4f0]
[    0.207159] Detected PIPT I-cache on CPU12
[    0.207176] GICv3: CPU12: found redistributor 100000 region 0:0x0000000022480000
[    0.207183] GICv3: CPU12: using allocated LPI pending table @0x0000000100670000
[    0.207194] CPU12: Booted secondary processor 0x0000100000 [0x410fd4f0]
[    0.214887] Detected PIPT I-cache on CPU13
[    0.214904] GICv3: CPU13: found redistributor 110000 region 0:0x00000000224c0000
[    0.214911] GICv3: CPU13: using allocated LPI pending table @0x0000000100680000
[    0.214922] CPU13: Booted secondary processor 0x0000110000 [0x410fd4f0]
[    0.222614] Detected PIPT I-cache on CPU14
[    0.222633] GICv3: CPU14: found redistributor 120000 region 0:0x0000000022500000
[    0.222640] GICv3: CPU14: using allocated LPI pending table @0x0000000100690000
[    0.222652] CPU14: Booted secondary processor 0x0000120000 [0x410fd4f0]
[    0.230339] Detected PIPT I-cache on CPU15
[    0.230358] GICv3: CPU15: found redistributor 130000 region 0:0x0000000022540000
[    0.230365] GICv3: CPU15: using allocated LPI pending table @0x00000001006a0000
[    0.230377] CPU15: Booted secondary processor 0x0000130000 [0x410fd4f0]
[    0.238060] Detected PIPT I-cache on CPU16
[    0.238079] GICv3: CPU16: found redistributor 140000 region 0:0x0000000022580000
[    0.238087] GICv3: CPU16: using allocated LPI pending table @0x00000001006b0000
[    0.238099] CPU16: Booted secondary processor 0x0000140000 [0x410fd4f0]
[    0.245789] Detected PIPT I-cache on CPU17
[    0.245808] GICv3: CPU17: found redistributor 150000 region 0:0x00000000225c0000
[    0.245815] GICv3: CPU17: using allocated LPI pending table @0x00000001006c0000
[    0.245827] CPU17: Booted secondary processor 0x0000150000 [0x410fd4f0]
[    0.253538] Detected PIPT I-cache on CPU18
[    0.253558] GICv3: CPU18: found redistributor 160000 region 0:0x0000000022600000
[    0.253565] GICv3: CPU18: using allocated LPI pending table @0x00000001006d0000
[    0.253577] CPU18: Booted secondary processor 0x0000160000 [0x410fd4f0]
[    0.261290] Detected PIPT I-cache on CPU19
[    0.261311] GICv3: CPU19: found redistributor 170000 region 0:0x0000000022640000
[    0.261319] GICv3: CPU19: using allocated LPI pending table @0x00000001006e0000
[    0.261331] CPU19: Booted secondary processor 0x0000170000 [0x410fd4f0]
[    0.269083] Detected PIPT I-cache on CPU20
[    0.269105] GICv3: CPU20: found redistributor 180000 region 0:0x0000000022680000
[    0.269112] GICv3: CPU20: using allocated LPI pending table @0x00000001006f0000
[    0.269125] CPU20: Booted secondary processor 0x0000180000 [0x410fd4f0]
[    0.276875] Detected PIPT I-cache on CPU21
[    0.276898] GICv3: CPU21: found redistributor 190000 region 0:0x00000000226c0000
[    0.276906] GICv3: CPU21: using allocated LPI pending table @0x0000000100700000
[    0.276918] CPU21: Booted secondary processor 0x0000190000 [0x410fd4f0]
[    0.284678] Detected PIPT I-cache on CPU22
[    0.284701] GICv3: CPU22: found redistributor 1a0000 region 0:0x0000000022700000
[    0.284708] GICv3: CPU22: using allocated LPI pending table @0x0000000100710000
[    0.284721] CPU22: Booted secondary processor 0x00001a0000 [0x410fd4f0]
[    0.292424] Detected PIPT I-cache on CPU23
[    0.292448] GICv3: CPU23: found redistributor 1c0000 region 0:0x0000000022780000
[    0.292456] GICv3: CPU23: using allocated LPI pending table @0x0000000100720000
[    0.292468] CPU23: Booted secondary processor 0x00001c0000 [0x410fd4f0]
[    0.300161] Detected PIPT I-cache on CPU24
[    0.300217] GICv3: CPU24: found redistributor 1d0000 region 0:0x00000000227c0000
[    0.300225] GICv3: CPU24: using allocated LPI pending table @0x0000000100730000
[    0.300237] CPU24: Booted secondary processor 0x00001d0000 [0x410fd4f0]
[    0.307923] Detected PIPT I-cache on CPU25
[    0.307947] GICv3: CPU25: found redistributor 1e0000 region 0:0x0000000022800000
[    0.307954] GICv3: CPU25: using allocated LPI pending table @0x0000000100740000
[    0.307965] CPU25: Booted secondary processor 0x00001e0000 [0x410fd4f0]
[    0.315653] Detected PIPT I-cache on CPU26
[    0.315677] GICv3: CPU26: found redistributor 1f0000 region 0:0x0000000022840000
[    0.315685] GICv3: CPU26: using allocated LPI pending table @0x0000000100750000
[    0.315696] CPU26: Booted secondary processor 0x00001f0000 [0x410fd4f0]
[    0.323390] Detected PIPT I-cache on CPU27
[    0.323416] GICv3: CPU27: found redistributor 200000 region 0:0x0000000022880000
[    0.323424] GICv3: CPU27: using allocated LPI pending table @0x0000000100760000
[    0.323435] CPU27: Booted secondary processor 0x0000200000 [0x410fd4f0]
[    0.331129] Detected PIPT I-cache on CPU28
[    0.331155] GICv3: CPU28: found redistributor 210000 region 0:0x00000000228c0000
[    0.331163] GICv3: CPU28: using allocated LPI pending table @0x0000000100770000
[    0.331175] CPU28: Booted secondary processor 0x0000210000 [0x410fd4f0]
[    0.338838] Detected PIPT I-cache on CPU29
[    0.338865] GICv3: CPU29: found redistributor 220000 region 0:0x0000000022900000
[    0.338873] GICv3: CPU29: using allocated LPI pending table @0x0000000100780000
[    0.338885] CPU29: Booted secondary processor 0x0000220000 [0x410fd4f0]
[    0.346547] Detected PIPT I-cache on CPU30
[    0.346575] GICv3: CPU30: found redistributor 230000 region 0:0x0000000022940000
[    0.346583] GICv3: CPU30: using allocated LPI pending table @0x0000000100790000
[    0.346595] CPU30: Booted secondary processor 0x0000230000 [0x410fd4f0]
[    0.354316] Detected PIPT I-cache on CPU31
[    0.354345] GICv3: CPU31: found redistributor 240000 region 0:0x0000000022980000
[    0.354353] GICv3: CPU31: using allocated LPI pending table @0x00000001007a0000
[    0.354365] CPU31: Booted secondary processor 0x0000240000 [0x410fd4f0]
[    0.362091] Detected PIPT I-cache on CPU32
[    0.362121] GICv3: CPU32: found redistributor 250000 region 0:0x00000000229c0000
[    0.362129] GICv3: CPU32: using allocated LPI pending table @0x00000001007b0000
[    0.362142] CPU32: Booted secondary processor 0x0000250000 [0x410fd4f0]
[    0.369875] Detected PIPT I-cache on CPU33
[    0.369905] GICv3: CPU33: found redistributor 260000 region 0:0x0000000022a00000
[    0.369914] GICv3: CPU33: using allocated LPI pending table @0x00000001007c0000
[    0.369926] CPU33: Booted secondary processor 0x0000260000 [0x410fd4f0]
[    0.377658] Detected PIPT I-cache on CPU34
[    0.377688] GICv3: CPU34: found redistributor 270000 region 0:0x0000000022a40000
[    0.377696] GICv3: CPU34: using allocated LPI pending table @0x00000001007d0000
[    0.377709] CPU34: Booted secondary processor 0x0000270000 [0x410fd4f0]
[    0.385463] Detected PIPT I-cache on CPU35
[    0.385496] GICv3: CPU35: found redistributor 280000 region 0:0x0000000022a80000
[    0.385505] GICv3: CPU35: using allocated LPI pending table @0x00000001007e0000
[    0.385518] CPU35: Booted secondary processor 0x0000280000 [0x410fd4f0]
[    0.393273] Detected PIPT I-cache on CPU36
[    0.393306] GICv3: CPU36: found redistributor 290000 region 0:0x0000000022ac0000
[    0.393315] GICv3: CPU36: using allocated LPI pending table @0x00000001007f0000
[    0.393328] CPU36: Booted secondary processor 0x0000290000 [0x410fd4f0]
[    0.401007] Detected PIPT I-cache on CPU37
[    0.401039] GICv3: CPU37: found redistributor 2a0000 region 0:0x0000000022b00000
[    0.401049] GICv3: CPU37: using allocated LPI pending table @0x0000000100800000
[    0.401062] CPU37: Booted secondary processor 0x00002a0000 [0x410fd4f0]
[    0.408745] Detected PIPT I-cache on CPU38
[    0.408807] GICv3: CPU38: found redistributor 2b0000 region 0:0x0000000022b40000
[    0.408817] GICv3: CPU38: using allocated LPI pending table @0x0000000100810000
[    0.408830] CPU38: Booted secondary processor 0x00002b0000 [0x410fd4f0]
[    0.416501] Detected PIPT I-cache on CPU39
[    0.416532] GICv3: CPU39: found redistributor 2c0000 region 0:0x0000000022b80000
[    0.416540] GICv3: CPU39: using allocated LPI pending table @0x0000000100820000
[    0.416553] CPU39: Booted secondary processor 0x00002c0000 [0x410fd4f0]
[    0.424213] Detected PIPT I-cache on CPU40
[    0.424242] GICv3: CPU40: found redistributor 2d0000 region 0:0x0000000022bc0000
[    0.424251] GICv3: CPU40: using allocated LPI pending table @0x0000000100830000
[    0.424264] CPU40: Booted secondary processor 0x00002d0000 [0x410fd4f0]
[    0.431958] Detected PIPT I-cache on CPU41
[    0.431992] GICv3: CPU41: found redistributor 2e0000 region 0:0x0000000022c00000
[    0.432004] GICv3: CPU41: using allocated LPI pending table @0x0000000100840000
[    0.432019] CPU41: Booted secondary processor 0x00002e0000 [0x410fd4f0]
[    0.439711] Detected PIPT I-cache on CPU42
[    0.439743] GICv3: CPU42: found redistributor 2f0000 region 0:0x0000000022c40000
[    0.439755] GICv3: CPU42: using allocated LPI pending table @0x0000000100850000
[    0.439770] CPU42: Booted secondary processor 0x00002f0000 [0x410fd4f0]
[    0.447462] Detected PIPT I-cache on CPU43
[    0.447496] GICv3: CPU43: found redistributor 300000 region 0:0x0000000022c80000
[    0.447512] GICv3: CPU43: using allocated LPI pending table @0x0000000100860000
[    0.447529] CPU43: Booted secondary processor 0x0000300000 [0x410fd4f0]
[    0.455224] Detected PIPT I-cache on CPU44
[    0.455261] GICv3: CPU44: found redistributor 310000 region 0:0x0000000022cc0000
[    0.455276] GICv3: CPU44: using allocated LPI pending table @0x0000000100870000
[    0.455294] CPU44: Booted secondary processor 0x0000310000 [0x410fd4f0]
[    0.463028] Detected PIPT I-cache on CPU45
[    0.463065] GICv3: CPU45: found redistributor 320000 region 0:0x0000000022d00000
[    0.463084] GICv3: CPU45: using allocated LPI pending table @0x0000000100880000
[    0.463104] CPU45: Booted secondary processor 0x0000320000 [0x410fd4f0]
[    0.470827] Detected PIPT I-cache on CPU46
[    0.470866] GICv3: CPU46: found redistributor 330000 region 0:0x0000000022d40000
[    0.470885] GICv3: CPU46: using allocated LPI pending table @0x0000000100890000
[    0.470905] CPU46: Booted secondary processor 0x0000330000 [0x410fd4f0]
[    0.478807] Detected PIPT I-cache on CPU47
[    0.478847] GICv3: CPU47: found redistributor 340000 region 0:0x0000000022d80000
[    0.478906] GICv3: CPU47: using allocated LPI pending table @0x00000001008a0000
[    0.478954] CPU47: Booted secondary processor 0x0000340000 [0x410fd4f0]
[    0.486866] Detected PIPT I-cache on CPU48
[    0.486903] GICv3: CPU48: found redistributor 350000 region 0:0x0000000022dc0000
[    0.486962] GICv3: CPU48: using allocated LPI pending table @0x00000001008b0000
[    0.487010] CPU48: Booted secondary processor 0x0000350000 [0x410fd4f0]
[    0.495014] Detected PIPT I-cache on CPU49
[    0.495054] GICv3: CPU49: found redistributor 360000 region 0:0x0000000022e00000
[    0.495128] GICv3: CPU49: using allocated LPI pending table @0x00000001008c0000
[    0.495186] CPU49: Booted secondary processor 0x0000360000 [0x410fd4f0]
[    0.503195] Detected PIPT I-cache on CPU50
[    0.503237] GICv3: CPU50: found redistributor 370000 region 0:0x0000000022e40000
[    0.503310] GICv3: CPU50: using allocated LPI pending table @0x00000001008d0000
[    0.503368] CPU50: Booted secondary processor 0x0000370000 [0x410fd4f0]
[    0.511033] Detected PIPT I-cache on CPU51
[    0.511104] GICv3: CPU51: found redistributor 380000 region 0:0x0000000022e80000
[    0.511116] GICv3: CPU51: using allocated LPI pending table @0x00000001008e0000
[    0.511131] CPU51: Booted secondary processor 0x0000380000 [0x410fd4f0]
[    0.518783] Detected PIPT I-cache on CPU52
[    0.518821] GICv3: CPU52: found redistributor 3a0000 region 0:0x0000000022f00000
[    0.518834] GICv3: CPU52: using allocated LPI pending table @0x00000001008f0000
[    0.518849] CPU52: Booted secondary processor 0x00003a0000 [0x410fd4f0]
[    0.526495] Detected PIPT I-cache on CPU53
[    0.526535] GICv3: CPU53: found redistributor 3b0000 region 0:0x0000000022f40000
[    0.526547] GICv3: CPU53: using allocated LPI pending table @0x0000000100900000
[    0.526563] CPU53: Booted secondary processor 0x00003b0000 [0x410fd4f0]
[    0.534426] Detected PIPT I-cache on CPU54
[    0.534467] GICv3: CPU54: found redistributor 3c0000 region 0:0x0000000022f80000
[    0.534535] GICv3: CPU54: using allocated LPI pending table @0x0000000100910000
[    0.534590] CPU54: Booted secondary processor 0x00003c0000 [0x410fd4f0]
[    0.542469] Detected PIPT I-cache on CPU55
[    0.542511] GICv3: CPU55: found redistributor 3d0000 region 0:0x0000000022fc0000
[    0.542579] GICv3: CPU55: using allocated LPI pending table @0x0000000100920000
[    0.542634] CPU55: Booted secondary processor 0x00003d0000 [0x410fd4f0]
[    0.550730] Detected PIPT I-cache on CPU56
[    0.550774] GICv3: CPU56: found redistributor 3e0000 region 0:0x0000000023000000
[    0.550909] GICv3: CPU56: using allocated LPI pending table @0x0000000100930000
[    0.551010] CPU56: Booted secondary processor 0x00003e0000 [0x410fd4f0]
[    0.559114] Detected PIPT I-cache on CPU57
[    0.559157] GICv3: CPU57: found redistributor 3f0000 region 0:0x0000000023040000
[    0.559292] GICv3: CPU57: using allocated LPI pending table @0x0000000100940000
[    0.559394] CPU57: Booted secondary processor 0x00003f0000 [0x410fd4f0]
[    0.568028] Detected PIPT I-cache on CPU58
[    0.568075] GICv3: CPU58: found redistributor 410000 region 0:0x00000000230c0000
[    0.568343] GICv3: CPU58: using allocated LPI pending table @0x0000000100950000
[    0.568538] CPU58: Booted secondary processor 0x0000410000 [0x410fd4f0]
[    0.578229] Detected PIPT I-cache on CPU59
[    0.578273] GICv3: CPU59: found redistributor 420000 region 0:0x0000000023100000
[    0.578806] GICv3: CPU59: using allocated LPI pending table @0x0000000100960000
[    0.579188] CPU59: Booted secondary processor 0x0000420000 [0x410fd4f0]
[    0.588938] Detected PIPT I-cache on CPU60
[    0.588987] GICv3: CPU60: found redistributor 430000 region 0:0x0000000023140000
[    0.589521] GICv3: CPU60: using allocated LPI pending table @0x0000000100970000
[    0.589903] CPU60: Booted secondary processor 0x0000430000 [0x410fd4f0]
[    0.600182] Detected PIPT I-cache on CPU61
[    0.600231] GICv3: CPU61: found redistributor 440000 region 0:0x0000000023180000
[    0.600907] GICv3: CPU61: using allocated LPI pending table @0x0000000100980000
[    0.601388] CPU61: Booted secondary processor 0x0000440000 [0x410fd4f0]
[    0.609624] Detected PIPT I-cache on CPU62
[    0.609673] GICv3: CPU62: found redistributor 460000 region 0:0x0000000023200000
[    0.609792] GICv3: CPU62: using allocated LPI pending table @0x0000000100990000
[    0.609883] CPU62: Booted secondary processor 0x0000460000 [0x410fd4f0]
[    0.617894] Detected PIPT I-cache on CPU63
[    0.617974] GICv3: CPU63: found redistributor 480000 region 0:0x0000000023280000
[    0.618065] GICv3: CPU63: using allocated LPI pending table @0x00000001009a0000
[    0.618137] CPU63: Booted secondary processor 0x0000480000 [0x410fd4f0]
[    0.626153] Detected PIPT I-cache on CPU64
[    0.626201] GICv3: CPU64: found redistributor 490000 region 0:0x00000000232c0000
[    0.626295] GICv3: CPU64: using allocated LPI pending table @0x00000001009b0000
[    0.626367] CPU64: Booted secondary processor 0x0000490000 [0x410fd4f0]
[    0.634756] Detected PIPT I-cache on CPU65
[    0.634805] GICv3: CPU65: found redistributor 4a0000 region 0:0x0000000023300000
[    0.634994] GICv3: CPU65: using allocated LPI pending table @0x00000001009c0000
[    0.635136] CPU65: Booted secondary processor 0x00004a0000 [0x410fd4f0]
[    0.643543] Detected PIPT I-cache on CPU66
[    0.643595] GICv3: CPU66: found redistributor 4b0000 region 0:0x0000000023340000
[    0.643785] GICv3: CPU66: using allocated LPI pending table @0x00000001009d0000
[    0.643926] CPU66: Booted secondary processor 0x00004b0000 [0x410fd4f0]
[    0.653054] Detected PIPT I-cache on CPU67
[    0.653107] GICv3: CPU67: found redistributor 4c0000 region 0:0x0000000023380000
[    0.653496] GICv3: CPU67: using allocated LPI pending table @0x00000001009e0000
[    0.653776] CPU67: Booted secondary processor 0x00004c0000 [0x410fd4f0]
[    0.662940] Detected PIPT I-cache on CPU68
[    0.662993] GICv3: CPU68: found redistributor 4d0000 region 0:0x00000000233c0000
[    0.663380] GICv3: CPU68: using allocated LPI pending table @0x00000001009f0000
[    0.663661] CPU68: Booted secondary processor 0x00004d0000 [0x410fd4f0]
[    0.673427] Detected PIPT I-cache on CPU69
[    0.673482] GICv3: CPU69: found redistributor 4e0000 region 0:0x0000000023400000
[    0.674017] GICv3: CPU69: using allocated LPI pending table @0x0000000100a00000
[    0.674397] CPU69: Booted secondary processor 0x00004e0000 [0x410fd4f0]
[    0.684194] Detected PIPT I-cache on CPU70
[    0.684247] GICv3: CPU70: found redistributor 4f0000 region 0:0x0000000023440000
[    0.684781] GICv3: CPU70: using allocated LPI pending table @0x0000000100a10000
[    0.685164] CPU70: Booted secondary processor 0x00004f0000 [0x410fd4f0]
[    0.693790] Detected PIPT I-cache on CPU71
[    0.693846] GICv3: CPU71: found redistributor 500000 region 0:0x0000000023480000
[    0.694063] GICv3: CPU71: using allocated LPI pending table @0x0000000100a20000
[    0.694225] CPU71: Booted secondary processor 0x0000500000 [0x410fd4f0]
[    0.694327] smp: Brought up 9 nodes, 72 CPUs
[    2.613053] SMP: Total of 72 processors activated.
[    2.618145] CPU features: detected: Branch Target Identification
[    2.624533] CPU features: detected: ARMv8.4 Translation Table Level
[    2.631197] CPU features: detected: Instruction cache invalidation not required for I/D coherence
[    2.640640] CPU features: detected: Data cache clean to the PoU not required for I/D coherence
[    2.649804] CPU features: detected: Common not Private translations
[    2.656469] CPU features: detected: CRC32 instructions
[    2.661930] CPU features: detected: Data cache clean to Point of Deep Persistence
[    2.669891] CPU features: detected: Data cache clean to Point of Persistence
[    2.677389] CPU features: detected: E0PD
[    2.681556] CPU features: detected: Enhanced Privileged Access Never
[    2.688314] CPU features: detected: Generic authentication (architected QARMA5 algorithm)
[    2.697015] CPU features: detected: RCpc load-acquire (LDAPR)
[    2.703125] CPU features: detected: LSE atomic instructions
[    2.709049] CPU features: detected: Privileged Access Never
[    2.714973] CPU features: detected: RAS Extension Support
[    2.720713] CPU features: detected: Speculation barrier (SB)
[    2.726731] CPU features: detected: Stage-2 Force Write-Back
[    2.732749] CPU features: detected: TLB range maintenance instructions
[    2.739693] CPU features: detected: Memory Partitioning And Monitoring
[    2.746635] CPU features: detected: Speculative Store Bypassing Safe (SSBS)
[    2.754039] CPU features: detected: Scalable Vector Extension
[    2.769269] CPU features: detected CPU65: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU6: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU3: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU9: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU46: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU45: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU67: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU21: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU4: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU69: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU20: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU71: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU58: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU50: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU29: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU38: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU19: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU43: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU57: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU34: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU56: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU64: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU68: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU12: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU7: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU5: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU8: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU70: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU63: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU39: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU44: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU66: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU11: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU31: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU13: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU35: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU1: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU22: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU32: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU28: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU27: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU49: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU55: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU33: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU16: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU60: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU61: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU10: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU23: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU15: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU48: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU24: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU14: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU30: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU2: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU18: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU17: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU59: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU47: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU62: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU0: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU42: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU25: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU53: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU40: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU41: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU54: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU36: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU26: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU52: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU37: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU51: Activity Monitors Unit (AMU)
[    3.269001] SVE: maximum available vector length 16 bytes per vector
[    3.275763] SVE: default vector length 16 bytes per vector
[    3.282028] CPU: All CPU(s) started at EL2
[    3.287429] devtmpfs: initialized
[    3.293580] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    3.303956] futex hash table entries: 32768 (order: 5, 2097152 bytes, linear)
[    3.311812] pinctrl core: initialized pinctrl subsystem
[    3.317422] SMBIOS 3.0.0 present.
[    3.320942] DMI: NVIDIA silicon, BIOS v1.1.1-4638c6c9 01/17/2023
[    3.328771] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    3.337447] DMA: preallocated 16384 KiB GFP_KERNEL pool for atomic allocations
[    3.345309] DMA: preallocated 16384 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    3.353907] DMA: preallocated 16384 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    3.362606] thermal_sys: Registered thermal governor 'step_wise'
[    3.362606] thermal_sys: Registered thermal governor 'power_allocator'
[    3.369042] cpuidle: using governor menu
[    3.380187] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    3.388295] ASID allocator initialised with 65536 entries
[    3.394037] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    3.400906] Serial: AMBA PL011 UART driver
[    3.405726] KASLR disabled on command line
[    3.410761] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[    3.417985] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page
[    3.424651] HugeTLB: registered 16.0 GiB page size, pre-allocated 0 pages
[    3.431873] HugeTLB: 16320 KiB vmemmap can be freed for a 16.0 GiB page
[    3.438908] HugeTLB: registered 512 MiB page size, pre-allocated 0 pages
[    3.446036] HugeTLB: 448 KiB vmemmap can be freed for a 512 MiB page
[    3.453205] ACPI: Added _OSI(Module Device)
[    3.457640] ACPI: Added _OSI(Processor Device)
[    3.462351] ACPI: Added _OSI(3.0 _SCP Extensions)
[    3.467337] ACPI: Added _OSI(Processor Aggregator Device)
[    3.473063] ACPI: Added _OSI(Linux-Dell-Video)
[    3.477774] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    3.483405] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[    3.490976] ACPI: 11 ACPI AML tables successfully acquired and loaded
[    3.497938] ACPI Error: AE_NOT_FOUND, While resolving a named reference package element - \_SB_.C000.C002 (20220331/dspkginit-438)
[    3.510427] ACPI Error: AE_NOT_FOUND, While resolving a named reference package element - \_SB_.C000.C040 (20220331/dspkginit-438)
[    3.523336] ACPI: Interpreter enabled
[    3.527226] ACPI: Using GIC for interrupt routing
[    3.532233] ACPI: MCFG table detected, 6 entries
[    3.537144] ACPI: IORT: SMMU-v3[11000000] Mapped to Proximity domain 0
[    3.544115] ACPI: IORT: SMMU-v3[12000000] Mapped to Proximity domain 0
[    3.551068] ACPI: IORT: SMMU-v3[15000000] Mapped to Proximity domain 0
[    3.558025] ACPI: IORT: SMMU-v3[16000000] Mapped to Proximity domain 0
[    3.564976] ACPI: IORT: SMMU-v3[5000000] Mapped to Proximity domain 0
[    3.571951] HEST: Table parsing has been initialized.
[    3.577341] sdei: SDEIv1.0 (0x0) detected in firmware.
[    3.584184] ghes_edac: This system has a very crappy BIOS: It doesn't even list the DIMMS.
[    3.592978] ghes_edac: Its SMBIOS info is wrong. It is doubtful that the error report would
[    3.601864] ghes_edac: work on such system. Use this driver with caution
[    3.609005] EDAC MC0: Giving out device to module ghes_edac.c controller ghes_edac: DEV ghes (INTERRUPT)
[    3.619202] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2561
[    3.628273] {1}[Hardware Error]: It has been corrected by h/w and requires no further action
[    3.637253] {1}[Hardware Error]: event severity: corrected
[    3.643086] {1}[Hardware Error]:  Error 0, type: corrected
[    3.648918] {1}[Hardware Error]:   section type: unknown, 6d5244f2-2712-11ec-bea7-cb3fdb95c786
[    3.658082] {1}[Hardware Error]:   section length: 0x30
[    3.663638] {1}[Hardware Error]:   00000000: 5f4d5348 455f5753 00005252 00000000  HSM_SW_ERR......
[    3.673170] {1}[Hardware Error]:   00000010: 00000000 00010000 0d000000 00000000  ................
[    3.682703] {1}[Hardware Error]:   00000020: 0d650010 00000000 c0e041d5 00000000  ..e......A......
[    3.693180] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
[    3.703614] ARMH0011:00: ttyAMA0 at MMIO 0xc280000 (irq = 34, base_baud = 0) is a SBSA
[    3.712045] printk: console [ttyAMA0] enabled
[    3.712045] printk: console [ttyAMA0] enabled
[    3.721123] printk: bootconsole [pl11] disabled
[    3.721123] printk: bootconsole [pl11] disabled
[    3.730716] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    3.737039] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    3.746690] acpi PNP0A08:00: _OSC: platform does not support [PME AER DPC]
[    3.753768] acpi PNP0A08:00: _OSC: OS now controls [PCIeCapability LTR]
[    3.760637] acpi PNP0A08:00: ECAM area [mem 0x600010000000-0x60001fffffff] reserved by PNP0C02:01
[    3.769784] acpi PNP0A08:00: ECAM at [mem 0x600010000000-0x60001fffffff] for [bus 00-ff]
[    3.778063] ACPI: Remapped I/O 0x0000600020000000 to [io  0x0000-0xffff window]
[    3.785561] PCI host bridge to bus 0000:00
[    3.789746] pci_bus 0000:00: root bus resource [mem 0x600040000000-0x6000bfffffff window] (bus address [0x40000000-0xbfffffff])
[    3.801485] pci_bus 0000:00: root bus resource [mem 0x6000c0000000-0x607fffffffff window]
[    3.809847] pci_bus 0000:00: root bus resource [io  0x0000-0xffff window]
[    3.816787] pci_bus 0000:00: root bus resource [bus 00-ff]
[    3.822407] pci 0000:00:00.0: [10de:22b2] type 01 class 0x060400
[    3.828634] pci 0000:00:00.0: PME# supported from D0 D3hot
[    3.834421] pci 0000:00:00.0: bridge window [io  0x1000-0x0fff] to [bus 01] add_size 1000
[    3.842816] pci 0000:00:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01] add_size 200000 add_align 100000
[    3.854556] pci 0000:00:00.0: bridge window [mem 0x00100000-0x000fffff] to [bus 01] add_size 200000 add_align 100000
[    3.865321] pci 0000:00:00.0: BAR 14: assigned [mem 0x600040000000-0x6000401fffff]
[    3.873061] pci 0000:00:00.0: BAR 15: assigned [mem 0x6000c0000000-0x6000c01fffff 64bit pref]
[    3.881779] pci 0000:00:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    3.888132] pci 0000:00:00.0: PCI bridge to [bus 01]
[    3.893205] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    3.899435] pci 0000:00:00.0:   bridge window [mem 0x600040000000-0x6000401fffff]
[    3.907086] pci 0000:00:00.0:   bridge window [mem 0x6000c0000000-0x6000c01fffff 64bit pref]
[    3.915716] pci_bus 0000:00: resource 4 [mem 0x600040000000-0x6000bfffffff window]
[    3.923456] pci_bus 0000:00: resource 5 [mem 0x6000c0000000-0x607fffffffff window]
[    3.931195] pci_bus 0000:00: resource 6 [io  0x0000-0xffff window]
[    3.937513] pci_bus 0000:01: resource 0 [io  0x1000-0x1fff]
[    3.943208] pci_bus 0000:01: resource 1 [mem 0x600040000000-0x6000401fffff]
[    3.950324] pci_bus 0000:01: resource 2 [mem 0x6000c0000000-0x6000c01fffff 64bit pref]
[    3.958434] ACPI: PCI Root Bridge [PCI2] (domain 0002 [bus 00-ff])
[    3.964752] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    3.974388] acpi PNP0A08:01: _OSC: platform does not support [PME AER DPC]
[    3.981461] acpi PNP0A08:01: _OSC: OS now controls [PCIeCapability LTR]
[    3.988326] acpi PNP0A08:01: ECAM area [mem 0x610010000000-0x61001fffffff] reserved by PNP0C02:02
[    3.997467] acpi PNP0A08:01: ECAM at [mem 0x610010000000-0x61001fffffff] for [bus 00-ff]
[    4.005743] ACPI: Remapped I/O 0x0000610020000000 to [io  0x10000-0x1ffff window]
[    4.013409] PCI host bridge to bus 0002:00
[    4.017593] pci_bus 0002:00: root bus resource [mem 0x610040000000-0x6100bfffffff window] (bus address [0x40000000-0xbfffffff])
[    4.029332] pci_bus 0002:00: root bus resource [mem 0x6100c0000000-0x617fffffffff window]
[    4.037694] pci_bus 0002:00: root bus resource [io  0x10000-0x1ffff window] (bus address [0x0000-0xffff])
[    4.047511] pci_bus 0002:00: root bus resource [bus 00-ff]
[    4.053127] pci 0002:00:00.0: [10de:22b2] type 01 class 0x060400
[    4.059352] pci 0002:00:00.0: PME# supported from D0 D3hot
[    4.065114] pci 0002:00:00.0: bridge window [io  0x1000-0x0fff] to [bus 01] add_size 1000
[    4.073475] pci 0002:00:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01] add_size 200000 add_align 100000
[    4.085214] pci 0002:00:00.0: bridge window [mem 0x00100000-0x000fffff] to [bus 01] add_size 200000 add_align 100000
[    4.095977] pci 0002:00:00.0: BAR 14: assigned [mem 0x610040000000-0x6100401fffff]
[    4.103716] pci 0002:00:00.0: BAR 15: assigned [mem 0x6100c0000000-0x6100c01fffff 64bit pref]
[    4.112433] pci 0002:00:00.0: BAR 13: assigned [io  0x10000-0x10fff]
[    4.118928] pci 0002:00:00.0: PCI bridge to [bus 01]
[    4.124000] pci 0002:00:00.0:   bridge window [io  0x10000-0x10fff]
[    4.130408] pci 0002:00:00.0:   bridge window [mem 0x610040000000-0x6100401fffff]
[    4.138059] pci 0002:00:00.0:   bridge window [mem 0x6100c0000000-0x6100c01fffff 64bit pref]
[    4.146689] pci_bus 0002:00: resource 4 [mem 0x610040000000-0x6100bfffffff window]
[    4.154428] pci_bus 0002:00: resource 5 [mem 0x6100c0000000-0x617fffffffff window]
[    4.162167] pci_bus 0002:00: resource 6 [io  0x10000-0x1ffff window]
[    4.168662] pci_bus 0002:01: resource 0 [io  0x10000-0x10fff]
[    4.174535] pci_bus 0002:01: resource 1 [mem 0x610040000000-0x6100401fffff]
[    4.181651] pci_bus 0002:01: resource 2 [mem 0x6100c0000000-0x6100c01fffff 64bit pref]
[    4.189756] ACPI: PCI Root Bridge [PCI4] (domain 0004 [bus 00-ff])
[    4.196074] acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    4.205708] acpi PNP0A08:02: _OSC: platform does not support [PME AER DPC]
[    4.212780] acpi PNP0A08:02: _OSC: OS now controls [PCIeCapability LTR]
[    4.219650] acpi PNP0A08:02: ECAM area [mem 0x620010000000-0x62001fffffff] reserved by PNP0C02:03
[    4.228791] acpi PNP0A08:02: ECAM at [mem 0x620010000000-0x62001fffffff] for [bus 00-ff]
[    4.237068] ACPI: Remapped I/O 0x0000620020000000 to [io  0x20000-0x2ffff window]
[    4.244763] PCI host bridge to bus 0004:00
[    4.248947] pci_bus 0004:00: root bus resource [mem 0x620040000000-0x6200bfffffff window] (bus address [0x40000000-0xbfffffff])
[    4.260686] pci_bus 0004:00: root bus resource [mem 0x6200c0000000-0x627fffffffff window]
[    4.269047] pci_bus 0004:00: root bus resource [io  0x20000-0x2ffff window] (bus address [0x0000-0xffff])
[    4.278830] pci_bus 0004:00: root bus resource [bus 00-ff]
[    4.284446] pci 0004:00:00.0: [10de:22b2] type 01 class 0x060400
[    4.290663] pci 0004:00:00.0: PME# supported from D0 D3hot
[    4.296414] pci 0004:00:00.0: bridge window [io  0x1000-0x0fff] to [bus 01] add_size 1000
[    4.304776] pci 0004:00:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01] add_size 200000 add_align 100000
[    4.316517] pci 0004:00:00.0: bridge window [mem 0x00100000-0x000fffff] to [bus 01] add_size 200000 add_align 100000
[    4.327279] pci 0004:00:00.0: BAR 14: assigned [mem 0x620040000000-0x6200401fffff]
[    4.335017] pci 0004:00:00.0: BAR 15: assigned [mem 0x6200c0000000-0x6200c01fffff 64bit pref]
[    4.343734] pci 0004:00:00.0: BAR 13: assigned [io  0x20000-0x20fff]
[    4.350229] pci 0004:00:00.0: PCI bridge to [bus 01]
[    4.355302] pci 0004:00:00.0:   bridge window [io  0x20000-0x20fff]
[    4.361709] pci 0004:00:00.0:   bridge window [mem 0x620040000000-0x6200401fffff]
[    4.369360] pci 0004:00:00.0:   bridge window [mem 0x6200c0000000-0x6200c01fffff 64bit pref]
[    4.377990] pci_bus 0004:00: resource 4 [mem 0x620040000000-0x6200bfffffff window]
[    4.385730] pci_bus 0004:00: resource 5 [mem 0x6200c0000000-0x627fffffffff window]
[    4.393469] pci_bus 0004:00: resource 6 [io  0x20000-0x2ffff window]
[    4.399963] pci_bus 0004:01: resource 0 [io  0x20000-0x20fff]
[    4.405835] pci_bus 0004:01: resource 1 [mem 0x620040000000-0x6200401fffff]
[    4.412952] pci_bus 0004:01: resource 2 [mem 0x6200c0000000-0x6200c01fffff 64bit pref]
[    4.421058] ACPI: PCI Root Bridge [PCI6] (domain 0006 [bus 00-ff])
[    4.427376] acpi PNP0A08:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    4.437009] acpi PNP0A08:03: _OSC: platform does not support [PME AER DPC]
[    4.444148] acpi PNP0A08:03: _OSC: OS now controls [PCIeCapability LTR]
[    4.451028] acpi PNP0A08:03: ECAM area [mem 0x630010000000-0x63001fffffff] reserved by PNP0C02:04
[    4.460169] acpi PNP0A08:03: ECAM at [mem 0x630010000000-0x63001fffffff] for [bus 00-ff]
[    4.468445] ACPI: Remapped I/O 0x0000630020000000 to [io  0x30000-0x3ffff window]
[    4.476110] PCI host bridge to bus 0006:00
[    4.480293] pci_bus 0006:00: root bus resource [mem 0x630040000000-0x6300bfffffff window] (bus address [0x40000000-0xbfffffff])
[    4.492033] pci_bus 0006:00: root bus resource [mem 0x6300c0000000-0x63ffffffffff window]
[    4.500394] pci_bus 0006:00: root bus resource [io  0x30000-0x3ffff window] (bus address [0x0000-0xffff])
[    4.510177] pci_bus 0006:00: root bus resource [bus 00-ff]
[    4.515791] pci 0006:00:00.0: [10de:22b2] type 01 class 0x060400
[    4.522008] pci 0006:00:00.0: PME# supported from D0 D3hot
[    4.527757] pci 0006:00:00.0: bridge window [io  0x1000-0x0fff] to [bus 01] add_size 1000
[    4.536119] pci 0006:00:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01] add_size 200000 add_align 100000
[    4.547859] pci 0006:00:00.0: bridge window [mem 0x00100000-0x000fffff] to [bus 01] add_size 200000 add_align 100000
[    4.558621] pci 0006:00:00.0: BAR 14: assigned [mem 0x630040000000-0x6300401fffff]
[    4.566360] pci 0006:00:00.0: BAR 15: assigned [mem 0x6300c0000000-0x6300c01fffff 64bit pref]
[    4.575077] pci 0006:00:00.0: BAR 13: assigned [io  0x30000-0x30fff]
[    4.581572] pci 0006:00:00.0: PCI bridge to [bus 01]
[    4.586644] pci 0006:00:00.0:   bridge window [io  0x30000-0x30fff]
[    4.593051] pci 0006:00:00.0:   bridge window [mem 0x630040000000-0x6300401fffff]
[    4.600703] pci 0006:00:00.0:   bridge window [mem 0x6300c0000000-0x6300c01fffff 64bit pref]
[    4.609333] pci_bus 0006:00: resource 4 [mem 0x630040000000-0x6300bfffffff window]
[    4.617072] pci_bus 0006:00: resource 5 [mem 0x6300c0000000-0x63ffffffffff window]
[    4.624811] pci_bus 0006:00: resource 6 [io  0x30000-0x3ffff window]
[    4.631306] pci_bus 0006:01: resource 0 [io  0x30000-0x30fff]
[    4.637178] pci_bus 0006:01: resource 1 [mem 0x630040000000-0x6300401fffff]
[    4.644294] pci_bus 0006:01: resource 2 [mem 0x6300c0000000-0x6300c01fffff 64bit pref]
[    4.652436] ACPI: PCI Root Bridge [PCI8] (domain 0008 [bus 00-ff])
[    4.658753] acpi PNP0A08:04: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    4.668388] acpi PNP0A08:04: _OSC: platform does not support [PME AER DPC]
[    4.675459] acpi PNP0A08:04: _OSC: OS now controls [PCIeCapability LTR]
[    4.682336] acpi PNP0A08:04: ECAM area [mem 0x650010000000-0x65001fffffff] reserved by PNP0C02:05
[    4.691477] acpi PNP0A08:04: ECAM at [mem 0x650010000000-0x65001fffffff] for [bus 00-ff]
[    4.699754] ACPI: Remapped I/O 0x0000650020000000 to [io  0x40000-0x4ffff window]
[    4.707418] PCI host bridge to bus 0008:00
[    4.711601] pci_bus 0008:00: root bus resource [mem 0x650040000000-0x6500bfffffff window] (bus address [0x40000000-0xbfffffff])
[    4.723340] pci_bus 0008:00: root bus resource [mem 0x6500c0000000-0x65ffffffffff window]
[    4.731701] pci_bus 0008:00: root bus resource [io  0x40000-0x4ffff window] (bus address [0x0000-0xffff])
[    4.741485] pci_bus 0008:00: root bus resource [bus 00-ff]
[    4.747095] pci 0008:00:00.0: [10de:22b9] type 01 class 0x060400
[    4.753275] pci 0008:00:00.0: PME# supported from D0 D3hot
[    4.759081] pci 0008:01:00.0: [11f8:4028] type 01 class 0x060400
[    4.766417] pci 0008:01:00.0: enabling Extended Tags
[    4.773074] pci 0008:01:00.0: PME# supported from D0 D3hot D3cold
[    4.780718] pci 0008:01:00.0: 15.753 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x1 link at 0008:00:00.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[    4.796650] pci 0008:01:00.1: [11f8:4028] type 00 class 0x058000
[    4.803418] pci 0008:01:00.1: reg 0x10: [mem 0x6500c0000000-0x6500c03fffff 64bit pref]
[    4.812071] pci 0008:01:00.1: enabling Extended Tags
[    4.818153] pci 0008:01:00.1: PME# supported from D0 D3hot D3cold
[    4.825758] pci 0008:02:00.0: [11f8:4028] type 01 class 0x060400
[    4.832707] pci 0008:02:00.0: enabling Extended Tags
[    4.839539] pci 0008:02:00.0: PME# supported from D0 D3hot D3cold
[    4.847133] pci 0008:02:01.0: [11f8:4028] type 01 class 0x060400
[    4.854046] pci 0008:02:01.0: enabling Extended Tags
[    4.860878] pci 0008:02:01.0: PME# supported from D0 D3hot D3cold
[    4.868472] pci 0008:02:02.0: [11f8:4028] type 01 class 0x060400
[    4.875385] pci 0008:02:02.0: enabling Extended Tags
[    4.882217] pci 0008:02:02.0: PME# supported from D0 D3hot D3cold
[    4.889821] pci 0008:02:03.0: [11f8:4028] type 01 class 0x060400
[    4.896733] pci 0008:02:03.0: enabling Extended Tags
[    4.903556] pci 0008:02:03.0: PME# supported from D0 D3hot D3cold
[    4.911169] pci 0008:02:04.0: [11f8:4028] type 01 class 0x060400
[    4.918081] pci 0008:02:04.0: enabling Extended Tags
[    4.924895] pci 0008:02:04.0: PME# supported from D0 D3hot D3cold
[    4.932517] pci 0008:02:05.0: [11f8:4028] type 01 class 0x060400
[    4.939423] pci 0008:02:05.0: enabling Extended Tags
[    4.946234] pci 0008:02:05.0: PME# supported from D0 D3hot D3cold
[    4.954846] pci 0008:03:00.0: [1344:51c0] type 00 class 0x010802
[    4.961169] pci 0008:03:00.0: reg 0x10: [mem 0x650041300000-0x65004133ffff 64bit]
[    4.969082] pci 0008:03:00.0: reg 0x20: [mem 0x650041340000-0x65004137ffff 64bit]
[    4.976796] pci 0008:03:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref]
[    4.984997] pci 0008:03:00.0: PME# supported from D0 D1 D3hot
[    4.991877] pci 0008:03:00.0: 15.753 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x1 link at 0008:00:00.0 (capable of 63.012 Gb/s with 16.0 GT/s PCIe x4 link)
[    5.009708] pci 0008:08:00.0: [1912:0014] type 00 class 0x0c0330
[    5.016096] pci 0008:08:00.0: reg 0x10: [mem 0x650041200000-0x650041201fff 64bit]
[    5.025170] pci 0008:08:00.0: PME# supported from D0 D3hot D3cold
[    5.033096] pci 0008:09:00.0: [10ec:8168] type 00 class 0x020000
[    5.039375] pci 0008:09:00.0: reg 0x10: [io  0x40000-0x400ff]
[    5.045540] pci 0008:09:00.0: reg 0x18: [mem 0x650041104000-0x650041104fff 64bit]
[    5.053309] pci 0008:09:00.0: reg 0x20: [mem 0x650041100000-0x650041103fff 64bit]
[    5.062316] pci 0008:09:00.0: supports D1 D2
[    5.066678] pci 0008:09:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    5.075468] pci 0008:02:00.0: bridge window [io  0x1000-0x0fff] to [bus 03] add_size 1000
[    5.083833] pci 0008:02:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 03] add_size 200000 add_align 100000
[    5.095572] pci 0008:02:00.0: bridge window [mem 0x00100000-0x001fffff] to [bus 03] add_size 100000 add_align 100000
[    5.106334] pci 0008:02:01.0: bridge window [io  0x1000-0x0fff] to [bus 04] add_size 1000
[    5.114695] pci 0008:02:01.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 04] add_size 200000 add_align 100000
[    5.126435] pci 0008:02:01.0: bridge window [mem 0x00100000-0x000fffff] to [bus 04] add_size 200000 add_align 100000
[    5.137197] pci 0008:02:02.0: bridge window [io  0x1000-0x0fff] to [bus 05] add_size 1000
[    5.145558] pci 0008:02:02.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 05] add_size 200000 add_align 100000
[    5.157297] pci 0008:02:02.0: bridge window [mem 0x00100000-0x000fffff] to [bus 05] add_size 200000 add_align 100000
[    5.168059] pci 0008:02:03.0: bridge window [io  0x1000-0x0fff] to [bus 06-07] add_size 1000
[    5.176687] pci 0008:02:03.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 06-07] add_size 200000 add_align 100000
[    5.188693] pci 0008:02:03.0: bridge window [mem 0x00100000-0x000fffff] to [bus 06-07] add_size 200000 add_align 100000
[    5.199721] pci 0008:02:04.0: bridge window [io  0x1000-0x0fff] to [bus 08] add_size 1000
[    5.208082] pci 0008:02:04.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 08] add_size 200000 add_align 100000
[    5.219822] pci 0008:02:04.0: bridge window [mem 0x00100000-0x001fffff] to [bus 08] add_size 100000 add_align 100000
[    5.230583] pci 0008:02:05.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 09] add_size 200000 add_align 100000
[    5.242323] pci 0008:02:05.0: bridge window [mem 0x00100000-0x001fffff] to [bus 09] add_size 100000 add_align 100000
[    5.253084] pci 0008:01:00.0: bridge window [io  0x1000-0x1fff] to [bus 02-09] add_size 5000
[    5.261713] pci 0008:01:00.0: bridge window [mem 0x00100000-0x006fffff 64bit pref] to [bus 02-09] add_size c00000 add_align 100000
[    5.273721] pci 0008:01:00.0: bridge window [mem 0x00100000-0x006fffff] to [bus 02-09] add_size 900000 add_align 100000
[    5.284750] pci 0008:00:00.0: bridge window [io  0x1000-0x1fff] to [bus 01-09] add_size 5000
[    5.293378] pci 0008:00:00.0: bridge window [mem 0x00400000-0x00ffffff 64bit pref] to [bus 01-09] add_size c00000 add_align 400000
[    5.305415] pci 0008:00:00.0: bridge window [mem 0x00100000-0x006fffff] to [bus 01-09] add_size 900000 add_align 100000
[    5.316445] pci 0008:00:00.0: BAR 15: assigned [mem 0x6500c0000000-0x6500c17fffff 64bit pref]
[    5.325162] pci 0008:00:00.0: BAR 14: assigned [mem 0x650040000000-0x650040efffff]
[    5.332901] pci 0008:00:00.0: BAR 13: assigned [io  0x40000-0x45fff]
[    5.339430] pci 0008:01:00.1: BAR 0: assigned [mem 0x6500c0000000-0x6500c03fffff 64bit pref]
[    5.348121] pci 0008:01:00.0: BAR 14: assigned [mem 0x650040000000-0x650040efffff]
[    5.355860] pci 0008:01:00.0: BAR 15: assigned [mem 0x6500c0400000-0x6500c15fffff 64bit pref]
[    5.364577] pci 0008:01:00.0: BAR 13: assigned [io  0x40000-0x45fff]
[    5.371074] pci 0008:02:00.0: BAR 14: assigned [mem 0x650040000000-0x6500401fffff]
[    5.378813] pci 0008:02:00.0: BAR 15: assigned [mem 0x6500c0400000-0x6500c05fffff 64bit pref]
[    5.387530] pci 0008:02:01.0: BAR 14: assigned [mem 0x650040200000-0x6500403fffff]
[    5.395269] pci 0008:02:01.0: BAR 15: assigned [mem 0x6500c0600000-0x6500c07fffff 64bit pref]
[    5.403986] pci 0008:02:02.0: BAR 14: assigned [mem 0x650040400000-0x6500405fffff]
[    5.411725] pci 0008:02:02.0: BAR 15: assigned [mem 0x6500c0800000-0x6500c09fffff 64bit pref]
[    5.420442] pci 0008:02:03.0: BAR 14: assigned [mem 0x650040600000-0x6500407fffff]
[    5.428181] pci 0008:02:03.0: BAR 15: assigned [mem 0x6500c0a00000-0x6500c0bfffff 64bit pref]
[    5.436899] pci 0008:02:04.0: BAR 14: assigned [mem 0x650040800000-0x6500409fffff]
[    5.444638] pci 0008:02:04.0: BAR 15: assigned [mem 0x6500c0c00000-0x6500c0dfffff 64bit pref]
[    5.453355] pci 0008:02:05.0: BAR 14: assigned [mem 0x650040a00000-0x650040bfffff]
[    5.461094] pci 0008:02:05.0: BAR 15: assigned [mem 0x6500c0e00000-0x6500c0ffffff 64bit pref]
[    5.469811] pci 0008:02:00.0: BAR 13: assigned [io  0x40000-0x40fff]
[    5.476306] pci 0008:02:01.0: BAR 13: assigned [io  0x41000-0x41fff]
[    5.482801] pci 0008:02:02.0: BAR 13: assigned [io  0x42000-0x42fff]
[    5.489296] pci 0008:02:03.0: BAR 13: assigned [io  0x43000-0x43fff]
[    5.495791] pci 0008:02:04.0: BAR 13: assigned [io  0x44000-0x44fff]
[    5.502285] pci 0008:02:05.0: BAR 13: assigned [io  0x45000-0x45fff]
[    5.508814] pci 0008:03:00.0: BAR 0: assigned [mem 0x650040000000-0x65004003ffff 64bit]
[    5.517064] pci 0008:03:00.0: BAR 4: assigned [mem 0x650040040000-0x65004007ffff 64bit]
[    5.525312] pci 0008:03:00.0: BAR 6: assigned [mem 0x650040080000-0x6500400bffff pref]
[    5.533439] pci 0008:02:00.0: PCI bridge to [bus 03]
[    5.538522] pci 0008:02:00.0:   bridge window [io  0x40000-0x40fff]
[    5.544947] pci 0008:02:00.0:   bridge window [mem 0x650040000000-0x6500401fffff]
[    5.552657] pci 0008:02:00.0:   bridge window [mem 0x6500c0400000-0x6500c05fffff 64bit pref]
[    5.561433] pci 0008:02:01.0: PCI bridge to [bus 04]
[    5.566523] pci 0008:02:01.0:   bridge window [io  0x41000-0x41fff]
[    5.572948] pci 0008:02:01.0:   bridge window [mem 0x650040200000-0x6500403fffff]
[    5.580652] pci 0008:02:01.0:   bridge window [mem 0x6500c0600000-0x6500c07fffff 64bit pref]
[    5.589432] pci 0008:02:02.0: PCI bridge to [bus 05]
[    5.594523] pci 0008:02:02.0:   bridge window [io  0x42000-0x42fff]
[    5.600946] pci 0008:02:02.0:   bridge window [mem 0x650040400000-0x6500405fffff]
[    5.608651] pci 0008:02:02.0:   bridge window [mem 0x6500c0800000-0x6500c09fffff 64bit pref]
[    5.617429] pci 0008:02:03.0: PCI bridge to [bus 06-07]
[    5.622787] pci 0008:02:03.0:   bridge window [io  0x43000-0x43fff]
[    5.629210] pci 0008:02:03.0:   bridge window [mem 0x650040600000-0x6500407fffff]
[    5.636915] pci 0008:02:03.0:   bridge window [mem 0x6500c0a00000-0x6500c0bfffff 64bit pref]
[    5.645703] pci 0008:08:00.0: BAR 0: assigned [mem 0x650040800000-0x650040801fff 64bit]
[    5.653952] pci 0008:02:04.0: PCI bridge to [bus 08]
[    5.659043] pci 0008:02:04.0:   bridge window [io  0x44000-0x44fff]
[    5.665467] pci 0008:02:04.0:   bridge window [mem 0x650040800000-0x6500409fffff]
[    5.673173] pci 0008:02:04.0:   bridge window [mem 0x6500c0c00000-0x6500c0dfffff 64bit pref]
[    5.681959] pci 0008:09:00.0: BAR 4: assigned [mem 0x650040a00000-0x650040a03fff 64bit]
[    5.690221] pci 0008:09:00.0: BAR 2: assigned [mem 0x650040a04000-0x650040a04fff 64bit]
[    5.698481] pci 0008:09:00.0: BAR 0: assigned [io  0x45000-0x450ff]
[    5.704909] pci 0008:02:05.0: PCI bridge to [bus 09]
[    5.709991] pci 0008:02:05.0:   bridge window [io  0x45000-0x45fff]
[    5.716415] pci 0008:02:05.0:   bridge window [mem 0x650040a00000-0x650040bfffff]
[    5.724122] pci 0008:02:05.0:   bridge window [mem 0x6500c0e00000-0x6500c0ffffff 64bit pref]
[    5.732901] pci 0008:01:00.0: PCI bridge to [bus 02-09]
[    5.738257] pci 0008:01:00.0:   bridge window [io  0x40000-0x45fff]
[    5.744681] pci 0008:01:00.0:   bridge window [mem 0x650040000000-0x650040efffff]
[    5.752554] pci 0008:01:00.0:   bridge window [mem 0x6500c0400000-0x6500c15fffff 64bit pref]
[    5.761835] pci 0008:00:00.0: PCI bridge to [bus 01-09]
[    5.767184] pci 0008:00:00.0:   bridge window [io  0x40000-0x45fff]
[    5.773590] pci 0008:00:00.0:   bridge window [mem 0x650040000000-0x650040efffff]
[    5.781241] pci 0008:00:00.0:   bridge window [mem 0x6500c0000000-0x6500c17fffff 64bit pref]
[    5.789870] pci_bus 0008:00: resource 4 [mem 0x650040000000-0x6500bfffffff window]
[    5.797609] pci_bus 0008:00: resource 5 [mem 0x6500c0000000-0x65ffffffffff window]
[    5.805348] pci_bus 0008:00: resource 6 [io  0x40000-0x4ffff window]
[    5.811843] pci_bus 0008:01: resource 0 [io  0x40000-0x45fff]
[    5.817716] pci_bus 0008:01: resource 1 [mem 0x650040000000-0x650040efffff]
[    5.824833] pci_bus 0008:01: resource 2 [mem 0x6500c0000000-0x6500c17fffff 64bit pref]
[    5.832928] pci_bus 0008:02: resource 0 [io  0x40000-0x45fff]
[    5.838800] pci_bus 0008:02: resource 1 [mem 0x650040000000-0x650040efffff]
[    5.845917] pci_bus 0008:02: resource 2 [mem 0x6500c0400000-0x6500c15fffff 64bit pref]
[    5.854012] pci_bus 0008:03: resource 0 [io  0x40000-0x40fff]
[    5.859884] pci_bus 0008:03: resource 1 [mem 0x650040000000-0x6500401fffff]
[    5.867001] pci_bus 0008:03: resource 2 [mem 0x6500c0400000-0x6500c05fffff 64bit pref]
[    5.875096] pci_bus 0008:04: resource 0 [io  0x41000-0x41fff]
[    5.880969] pci_bus 0008:04: resource 1 [mem 0x650040200000-0x6500403fffff]
[    5.888086] pci_bus 0008:04: resource 2 [mem 0x6500c0600000-0x6500c07fffff 64bit pref]
[    5.896180] pci_bus 0008:05: resource 0 [io  0x42000-0x42fff]
[    5.902053] pci_bus 0008:05: resource 1 [mem 0x650040400000-0x6500405fffff]
[    5.909169] pci_bus 0008:05: resource 2 [mem 0x6500c0800000-0x6500c09fffff 64bit pref]
[    5.917264] pci_bus 0008:06: resource 0 [io  0x43000-0x43fff]
[    5.923136] pci_bus 0008:06: resource 1 [mem 0x650040600000-0x6500407fffff]
[    5.930253] pci_bus 0008:06: resource 2 [mem 0x6500c0a00000-0x6500c0bfffff 64bit pref]
[    5.938347] pci_bus 0008:08: resource 0 [io  0x44000-0x44fff]
[    5.944220] pci_bus 0008:08: resource 1 [mem 0x650040800000-0x6500409fffff]
[    5.951337] pci_bus 0008:08: resource 2 [mem 0x6500c0c00000-0x6500c0dfffff 64bit pref]
[    5.959432] pci_bus 0008:09: resource 0 [io  0x45000-0x45fff]
[    5.965304] pci_bus 0008:09: resource 1 [mem 0x650040a00000-0x650040bfffff]
[    5.972421] pci_bus 0008:09: resource 2 [mem 0x6500c0e00000-0x6500c0ffffff 64bit pref]
[    5.980572] ACPI: PCI Root Bridge [PCI9] (domain 0009 [bus 00-ff])
[    5.986890] acpi PNP0A08:05: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    5.996530] acpi PNP0A08:05: _OSC: platform does not support [PME AER DPC]
[    6.003602] acpi PNP0A08:05: _OSC: OS now controls [PCIeCapability LTR]
[    6.010488] acpi PNP0A08:05: ECAM area [mem 0x660010000000-0x66001fffffff] reserved by PNP0C02:06
[    6.019630] acpi PNP0A08:05: ECAM at [mem 0x660010000000-0x66001fffffff] for [bus 00-ff]
[    6.027907] ACPI: Remapped I/O 0x0000660020000000 to [io  0x50000-0x5ffff window]
[    6.035576] PCI host bridge to bus 0009:00
[    6.039759] pci_bus 0009:00: root bus resource [mem 0x660040000000-0x6600bfffffff window] (bus address [0x40000000-0xbfffffff])
[    6.051498] pci_bus 0009:00: root bus resource [mem 0x6600c0000000-0x66ffffffffff window]
[    6.059859] pci_bus 0009:00: root bus resource [io  0x50000-0x5ffff window] (bus address [0x0000-0xffff])
[    6.069643] pci_bus 0009:00: root bus resource [bus 00-ff]
[    6.075321] pci 0009:00:00.0: [10de:22b1] type 01 class 0x060400
[    6.081502] pci 0009:00:00.0: PME# supported from D0 D3hot
[    6.087220] pci 0009:01:00.0: [10de:2342] type 00 class 0x030200
[    6.093367] pci 0009:01:00.0: reg 0x10: [mem 0x660040000000-0x660040ffffff]
[    6.100490] pci 0009:01:00.0: reg 0x14: [mem 0x662000000000-0x663fffffffff 64bit pref]
[    6.108591] pci 0009:01:00.0: reg 0x1c: [mem 0x664000000000-0x664001ffffff 64bit pref]
[    6.116689] pci 0009:01:00.0: reg 0x24: [io  0x50000-0x5007f]
[    6.122583] pci 0009:01:00.0: Enabling HDA controller
[    6.127813] pci 0009:01:00.0: reg 0x274: [mem 0x00000000-0x0003ffff 64bit pref]
[    6.135286] pci 0009:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x005fffff 64bit pref] (contains BAR0 for 24 VFs)
[    6.145887] pci 0009:01:00.0: 15.753 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x1 link at 0009:00:00.0 (capable of 504.112 Gb/s with 32.0 GT/s PCIe x16 link)
[    6.173643] pci 0009:00:00.0: BAR 15: assigned [mem 0x661000000000-0x663fffffffff 64bit pref]
[    6.182361] pci 0009:00:00.0: BAR 14: assigned [mem 0x660040000000-0x660040ffffff]
[    6.190100] pci 0009:00:00.0: BAR 13: assigned [io  0x50000-0x50fff]
[    6.196594] pci 0009:01:00.0: BAR 1: assigned [mem 0x662000000000-0x663fffffffff 64bit pref]
[    6.205227] pci 0009:01:00.0: BAR 3: assigned [mem 0x661000000000-0x661001ffffff 64bit pref]
[    6.213859] pci 0009:01:00.0: BAR 0: assigned [mem 0x660040000000-0x660040ffffff]
[    6.221510] pci 0009:01:00.0: BAR 7: assigned [mem 0x661002000000-0x6610025fffff 64bit pref]
[    6.230140] pci 0009:01:00.0: BAR 5: assigned [io  0x50000-0x5007f]
[    6.236547] pci 0009:00:00.0: PCI bridge to [bus 01]
[    6.241620] pci 0009:00:00.0:   bridge window [io  0x50000-0x50fff]
[    6.248026] pci 0009:00:00.0:   bridge window [mem 0x660040000000-0x660040ffffff]
[    6.255677] pci 0009:00:00.0:   bridge window [mem 0x661000000000-0x663fffffffff 64bit pref]
[    6.264306] pci_bus 0009:00: resource 4 [mem 0x660040000000-0x6600bfffffff window]
[    6.272046] pci_bus 0009:00: resource 5 [mem 0x6600c0000000-0x66ffffffffff window]
[    6.279785] pci_bus 0009:00: resource 6 [io  0x50000-0x5ffff window]
[    6.286280] pci_bus 0009:01: resource 0 [io  0x50000-0x50fff]
[    6.292152] pci_bus 0009:01: resource 1 [mem 0x660040000000-0x660040ffffff]
[    6.299269] pci_bus 0009:01: resource 2 [mem 0x661000000000-0x663fffffffff 64bit pref]
[    6.307558] iommu: Default domain type: Passthrough
[    6.312733] SCSI subsystem initialized
[    6.316580] ACPI: bus type USB registered
[    6.320685] usbcore: registered new interface driver usbfs
[    6.326295] usbcore: registered new interface driver hub
[    6.331736] usbcore: registered new device driver usb
[    6.336903] pps_core: LinuxPPS API ver. 1 registered
[    6.341976] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    6.351317] PTP clock support registered
[    6.355328] EDAC MC: Ver: 3.0.0
[    6.359042] ARM FF-A: Driver version 1.0
[    6.363048] ARM FF-A: Firmware version 1.1 found
[    6.367764] ARM FF-A: Firmware version higher than driver version, downgrading
[    6.375189] Registered efivars operations
[    6.380405] vgaarb: loaded
[    6.383209] mpam: update max_partid from 40 to 45
[    6.389204] clocksource: Switched to clocksource arch_sys_counter
[    6.395515] VFS: Disk quotas dquot_6.6.0
[    6.399538] VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
[    6.406827] pnp: PnP ACPI init
[    6.410085] system 00:00: [mem 0x600010000000-0x60001fffffff window] could not be reserved
[    6.418629] system 00:01: [mem 0x610010000000-0x61001fffffff window] could not be reserved
[    6.427097] system 00:02: [mem 0x620010000000-0x62001fffffff window] could not be reserved
[    6.435563] system 00:03: [mem 0x630010000000-0x63001fffffff window] could not be reserved
[    6.444028] system 00:04: [mem 0x650010000000-0x65001fffffff window] could not be reserved
[    6.452493] system 00:05: [mem 0x660010000000-0x66001fffffff window] could not be reserved
[    6.460950] pnp: PnP ACPI: found 6 devices
[    6.465984] NET: Registered PF_INET protocol family
[    6.471032] IP idents hash table entries: 262144 (order: 5, 2097152 bytes, linear)
[    6.480570] tcp_listen_portaddr_hash hash table entries: 65536 (order: 4, 1048576 bytes, linear)
[    6.489703] Table-perturb hash table entries: 65536 (order: 2, 262144 bytes, linear)
[    6.497623] TCP established hash table entries: 524288 (order: 6, 4194304 bytes, linear)
[    6.506138] TCP bind hash table entries: 65536 (order: 4, 1048576 bytes, linear)
[    6.513845] TCP: Hash tables configured (established 524288 bind 65536)
[    6.520658] UDP hash table entries: 65536 (order: 5, 2097152 bytes, linear)
[    6.527919] UDP-Lite hash table entries: 65536 (order: 5, 2097152 bytes, linear)
[    6.535656] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    6.541744] pci 0008:08:00.0: enabling device (0000 -> 0002)
[    6.547632] PCI: CLS 0 bytes, default 64
[    6.551689] Unpacking initramfs...
[    6.565375] hw perfevents: enabled with armv8_pmuv3_0 PMU driver, 7 counters available
[    6.573268] Freeing initrd memory: 123968K
[    6.573575] kvm [1]: IPA Size Limit: 48 bits
[    6.582058] kvm [1]: GICv4.1 support enabled
[    6.586420] kvm [1]: GICv3: no GICV resource entry
[    6.591316] kvm [1]: disabling GICv2 emulation
[    6.595862] kvm [1]: GIC system register CPU interface enabled
[    6.603354] kvm [1]: vgic interrupt IRQ9
[    6.610091] kvm [1]: VHE mode initialized successfully
[    6.615926] Initialise system trusted keyrings
[    6.620516] workingset: timestamp_bits=40 max_order=21 bucket_order=0
[    6.627767] 9p: Installing v9fs 9p2000 file system support
[    6.642447] Key type asymmetric registered
[    6.646631] Asymmetric key parser 'x509' registered
[    6.651626] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[    6.659188] io scheduler mq-deadline registered
[    6.663817] io scheduler kyber registered
[    6.669176] IPMI message handler: version 39.2
[    6.673726] ipmi device interface
[    6.677140] ipmi_si: IPMI System Interface driver
[    6.681975] ipmi_si: Unable to find any System Interface(s)
[    6.687670] ipmi_ssif: IPMI SSIF Interface driver
[    6.692495] IPMI Watchdog: driver initialized
[    6.696944] IPMI poweroff: Copyright (C) 2004 MontaVista Software - IPMI Powerdown via sys_reboot
[    6.765231] thermal LNXTHERM:00: registered as thermal_zone0
[    6.771019] ACPI: thermal: Thermal Zone [TZ00] (43 C)
[    6.833294] thermal LNXTHERM:01: registered as thermal_zone1
[    6.839081] ACPI: thermal: Thermal Zone [TZ01] (42 C)
[    6.921230] thermal LNXTHERM:02: registered as thermal_zone2
[    6.927016] ACPI: thermal: Thermal Zone [TZ02] (42 C)
[    7.009231] thermal LNXTHERM:03: registered as thermal_zone3
[    7.015016] ACPI: thermal: Thermal Zone [TZ03] (42 C)
[    7.129228] thermal LNXTHERM:04: registered as thermal_zone4
[    7.135014] ACPI: thermal: Thermal Zone [TZ04] (41 C)
[    7.269228] thermal LNXTHERM:05: registered as thermal_zone5
[    7.275014] ACPI: thermal: Thermal Zone [TZ05] (41 C)
[    7.453227] thermal LNXTHERM:06: registered as thermal_zone6
[    7.459012] ACPI: thermal: Thermal Zone [TZ06] (42 C)
[    7.673292] thermal LNXTHERM:07: registered as thermal_zone7
[    7.679078] ACPI: thermal: Thermal Zone [TZ07] (42 C)
[    7.953227] thermal LNXTHERM:08: registered as thermal_zone8
[    7.959012] ACPI: thermal: Thermal Zone [TZ08] (42 C)
[    8.273228] thermal LNXTHERM:09: registered as thermal_zone9
[    8.279014] ACPI: thermal: Thermal Zone [TZ09] (43 C)
[    8.633226] thermal LNXTHERM:0a: registered as thermal_zone10
[    8.639100] ACPI: thermal: Thermal Zone [TZ0A] (33 C)
[    9.053231] thermal LNXTHERM:0b: registered as thermal_zone11
[    9.059106] ACPI: thermal: Thermal Zone [TZ0B] (34 C)
[    9.064389] Unknown online node for memory at 0x2042cab000, assuming node 0
[    9.071508] nfit ACPI0012:00: changing numa node from -1 to 0 for nfit region [0x0000002042cab000-0x000000204a77ffff]
[    9.082362] Unknown target node for memory at 0x2042cab000, assuming node 0
[    9.089480] nfit ACPI0012:00: changing target node from -1 to 0 for nfit region [0x0000002042cab000-0x000000204a77ffff]
[    9.100582] EINJ: Error INJection is initialized.
[    9.105416] ACPI GTDT: found 1 SBSA generic Watchdog(s).
[    9.111898] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    9.118568] arm-smmu-v3 arm-smmu-v3.0.auto: option mask 0x0
[    9.124305] arm-smmu-v3 arm-smmu-v3.0.auto: ias 48-bit, oas 48-bit (features 0x00061faf)
[    9.132746] arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for cmdq
[    9.140372] arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for evtq
[    9.147795] arm-smmu-v3 arm-smmu-v3.0.auto: msi_domain absent - falling back to wired irqs
[    9.156334] arm-smmu-v3 arm-smmu-v3.1.auto: option mask 0x0
[    9.162036] arm-smmu-v3 arm-smmu-v3.1.auto: ias 48-bit, oas 48-bit (features 0x00061faf)
[    9.170474] arm-smmu-v3 arm-smmu-v3.1.auto: allocated 524288 entries for cmdq
[    9.178099] arm-smmu-v3 arm-smmu-v3.1.auto: allocated 524288 entries for evtq
[    9.185518] arm-smmu-v3 arm-smmu-v3.1.auto: msi_domain absent - falling back to wired irqs
[    9.194032] arm-smmu-v3 arm-smmu-v3.2.auto: option mask 0x0
[    9.199735] arm-smmu-v3 arm-smmu-v3.2.auto: ias 48-bit, oas 48-bit (features 0x00061faf)
[    9.208173] arm-smmu-v3 arm-smmu-v3.2.auto: allocated 524288 entries for cmdq
[    9.215796] arm-smmu-v3 arm-smmu-v3.2.auto: allocated 524288 entries for evtq
[    9.223215] arm-smmu-v3 arm-smmu-v3.2.auto: msi_domain absent - falling back to wired irqs
[    9.231762] arm-smmu-v3 arm-smmu-v3.3.auto: option mask 0x0
[    9.237466] arm-smmu-v3 arm-smmu-v3.3.auto: ias 48-bit, oas 48-bit (features 0x00061faf)
[    9.245902] arm-smmu-v3 arm-smmu-v3.3.auto: allocated 524288 entries for cmdq
[    9.253526] arm-smmu-v3 arm-smmu-v3.3.auto: allocated 524288 entries for evtq
[    9.260943] arm-smmu-v3 arm-smmu-v3.3.auto: msi_domain absent - falling back to wired irqs
[    9.269453] arm-smmu-v3 arm-smmu-v3.4.auto: option mask 0x0
[    9.275154] arm-smmu-v3 arm-smmu-v3.4.auto: ias 48-bit, oas 48-bit (features 0x00061faf)
[    9.283591] arm-smmu-v3 arm-smmu-v3.4.auto: allocated 524288 entries for cmdq
[    9.291215] arm-smmu-v3 arm-smmu-v3.4.auto: allocated 524288 entries for evtq
[    9.298635] arm-smmu-v3 arm-smmu-v3.4.auto: msi_domain absent - falling back to wired irqs
[    9.311849] loop: module loaded
[    9.315113] nd_pmem namespace0.0: unable to guarantee persistence of writes
[    9.322401]  pmem0: p1
[    9.325127] nvme 0008:03:00.0: Adding to iommu group 0
[    9.330492] nvme nvme0: pci function 0008:03:00.0
[    9.335394] tun: Universal TUN/TAP device driver, 1.6
[    9.340585] e1000: Intel(R) PRO/1000 Network Driver
[    9.345570] e1000: Copyright (c) 1999-2006 Intel Corporation.
[    9.347070] nvme nvme0: 74/0/0 default/read/poll queues
[    9.351453] e1000e: Intel(R) PRO/1000 Network Driver
[    9.361861] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    9.364602]  nvme0n1: p1 p2
[    9.367917] igb: Intel(R) Gigabit Ethernet Network Driver
[    9.376281] igb: Copyright (c) 2007-2014 Intel Corporation.
[    9.381980] igbvf: Intel(R) Gigabit Virtual Function Network Driver
[    9.388385] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    9.394439] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[    9.400667] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    9.406630] r8169 0008:09:00.0: Adding to iommu group 1
[    9.426231] r8169 0008:09:00.0 eth0: RTL8168h/8111h, 48:b0:2d:b1:4d:55, XID 541, IRQ 121
[    9.434511] r8169 0008:09:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
[    9.443262] usbcore: registered new interface driver cdc_ether
[    9.449230] usbcore: registered new interface driver r8153_ecm
[    9.455230] VFIO - User Level meta-driver version: 0.3
[    9.460546] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    9.467220] ehci-pci: EHCI PCI platform driver
[    9.471762] ehci-platform: EHCI generic platform driver
[    9.477160] xhci_hcd 0008:08:00.0: Adding to iommu group 2
[   10.592622] xhci_hcd 0008:08:00.0: xHCI Host Controller
[   10.597969] xhci_hcd 0008:08:00.0: new USB bus registered, assigned bus number 1
[   10.605568] xhci_hcd 0008:08:00.0: Zeroing 64bit base registers, expecting fault
[   10.613231] xhci_hcd 0008:08:00.0: Fault detected
[   10.623368] xhci_hcd 0008:08:00.0: hcc params 0x014051cf hci version 0x100 quirks 0x0000001100000410
[   10.633182] xhci_hcd 0008:08:00.0: xHCI Host Controller
[   10.638522] xhci_hcd 0008:08:00.0: new USB bus registered, assigned bus number 2
[   10.646084] xhci_hcd 0008:08:00.0: Host supports USB 3.0 SuperSpeed
[   10.652608] hub 1-0:1.0: USB hub found
[   10.656447] hub 1-0:1.0: 4 ports detected
[   10.660624] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[   10.668937] hub 2-0:1.0: USB hub found
[   10.672773] hub 2-0:1.0: 4 ports detected
[   10.676965] usbcore: registered new interface driver usb-storage
[   10.683176] rtc-efi rtc-efi.0: registered as rtc0
[   10.688026] rtc-efi rtc-efi.0: setting system clock to 2023-01-19T21:54:17 UTC (1674165257)
[   10.696581] i2c_dev: i2c /dev entries driver
[   10.701004] sbsa-gwdt sbsa-gwdt.0: Initialized with 10s timeout @ 1000000000 Hz, action=0.
[   10.709541] pstore: Registered efi as persistent store backend
[   10.715534] SMCCC: SOC_ID: ID = jep106:036b:0024 Revision = 0x00000101
[   10.730161] NET: Registered PF_INET6 protocol family
[   10.735503] Segment Routing with IPv6
[   10.739247] In-situ OAM (IOAM) with IPv6
[   10.743262] NET: Registered PF_PACKET protocol family
[   10.748453] sctp: Hash tables configured (bind 4096/4096)
[   10.754016] 9pnet: Installing 9P2000 support
[   10.758394] Key type dns_resolver registered
[   10.762802] NET: Registered PF_VSOCK protocol family
[   10.768933] registered taskstats version 1
[   10.773124] Loading compiled-in X.509 certificates
[   10.778108] pstore: Using crash dump compression: deflate
[   10.783642] Key type big_key registered
[   10.787872] mpam: MPAM enabled with 45 partid and 2 pmg
[   10.790576] pcieport 0000:00:00.0: Adding to iommu group 3
[   10.798963] pcieport 0002:00:00.0: Adding to iommu group 4
[   10.804689] pcieport 0004:00:00.0: Adding to iommu group 5
[   10.810406] pcieport 0006:00:00.0: Adding to iommu group 6
[   10.816129] pcieport 0008:00:00.0: Adding to iommu group 7
[   10.821835] pcieport 0008:01:00.0: Adding to iommu group 8
[   10.828985] pcieport 0008:02:00.0: Adding to iommu group 9
[   10.836790] pcieport 0008:02:01.0: Adding to iommu group 10
[   10.844885] pcieport 0008:02:02.0: Adding to iommu group 11
[   10.852759] pcieport 0008:02:03.0: Adding to iommu group 12
[   10.860756] pcieport 0008:02:04.0: Adding to iommu group 13
[   10.868676] pcieport 0008:02:05.0: Adding to iommu group 14
[   10.876713] pcieport 0009:00:00.0: Adding to iommu group 15
[   10.891236] Freeing unused kernel memory: 2240K
[   10.917183] usb 1-1: new high-speed USB device number 2 using xhci_hcd
[   10.937383] Run /init as init process
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
[   11.079768] hub 1-1:1.0: USB hub found
[   11.083879] hub 1-1:1.0: 7 ports detected
[   11.381184] usb 1-1.1: new high-speed USB device number 3 using xhci_hcd
[   11.507959] cdc_ether 1-1.1:1.0 usb0: register 'cdc_ether' at usb-0008:08:00.0-1.1, CDC Ethernet Device, 8e:40:7a:97:bc:8e
[   14.955110] random: crng init done
Saving random seed: OK
Starting network: [   15.069183] Generic FE-GE Realtek PHY r8169-8-900:00: attached PHY driver (mii_bus:phy_addr=r8169-8-900:00, irq=MAC)
[   15.281296] r8169 0008:09:00.0 eth0: Link is Down
udhcpc: started, v1.35.0
udhcpc: broadcasting discover
udhcpc: no lease, forking to background
OK
Starting dropbear sshd: OK
root@grace# [   18.861591] r8169 0008:09:00.0 eth0: Link is Up - 1Gbps/Full - flow control off
[   18.869075] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

root@grace# uname -a
Linux grace 6.2.0-rc4 #8 SMP Thu Jan 19 17:53:00 CST 2023 aarch64 GNU/Linux

root@grace# 
[  258.519837] kvm_create_vm(1236) called kvm=ffff8000303e0000
[  258.527613] kvm_create_vm(1236) called kvm=ffff800030400000
[  258.533393] kvm_vm_release(1374) called kvm=ffff800030400000
[  258.557893] kvm_create_vm(1236) called kvm=ffff8000304c0000
[  258.563627] kvm_vm_release(1374) called kvm=ffff8000304c0000
[  258.667101] vgic_v4_init(267) called kvm=ffff8000303e0000 doorbell=140(64)
[  258.853783] kvm_create_vm(1236) called kvm=ffff800031020000
[  258.861988] kvm_create_vm(1236) called kvm=ffff800031040000
[  258.867797] kvm_vm_release(1374) called kvm=ffff800031040000
[  258.873976] kvm_create_vm(1236) called kvm=ffff800031080000
[  258.879724] kvm_vm_release(1374) called kvm=ffff800031080000
[  258.978326] vgic_v4_init(267) called kvm=ffff800031020000 doorbell=1228(64)
[  259.117997] kvm_create_vm(1236) called kvm=ffff800031ae0000
[  259.126079] kvm_create_vm(1236) called kvm=ffff800031b00000
[  259.131899] kvm_vm_release(1374) called kvm=ffff800031b00000
[  259.138068] kvm_create_vm(1236) called kvm=ffff800031b40000
[  259.143844] kvm_vm_release(1374) called kvm=ffff800031b40000
[  259.246442] vgic_v4_init(267) called kvm=ffff800031ae0000 doorbell=2316(64)
[  259.393845] kvm_create_vm(1236) called kvm=ffff8000325e0000
[  259.402781] kvm_create_vm(1236) called kvm=ffff800032600000
[  259.408646] kvm_vm_release(1374) called kvm=ffff800032600000
[  259.414931] kvm_create_vm(1236) called kvm=ffff800032640000
[  259.420715] kvm_vm_release(1374) called kvm=ffff800032640000
[  259.521881] vgic_v4_init(267) called kvm=ffff8000325e0000 doorbell=3404(64)
[  259.666997] kvm_create_vm(1236) called kvm=ffff8000330c0000
[  259.675346] kvm_create_vm(1236) called kvm=ffff8000330e0000
[  259.681173] kvm_vm_release(1374) called kvm=ffff8000330e0000
[  259.687343] kvm_create_vm(1236) called kvm=ffff800033120000
[  259.693116] kvm_vm_release(1374) called kvm=ffff800033120000
[  259.783433] vgic_v4_init(267) called kvm=ffff8000330c0000 doorbell=4492(64)
[  259.953796] kvm_create_vm(1236) called kvm=ffff800033c80000
[  259.961874] kvm_create_vm(1236) called kvm=ffff800033ca0000
[  259.967680] kvm_vm_release(1374) called kvm=ffff800033ca0000
[  259.973927] kvm_create_vm(1236) called kvm=ffff800033ce0000
[  259.979714] kvm_vm_release(1374) called kvm=ffff800033ce0000
[  260.116812] vgic_v4_init(267) called kvm=ffff800033c80000 doorbell=5580(64)
[  260.744980] kvm_create_vm(1236) called kvm=ffff800034a20000
[  260.752594] kvm_create_vm(1236) called kvm=ffff800034a60000
[  260.758328] kvm_vm_release(1374) called kvm=ffff800034a60000
[  260.764358] kvm_create_vm(1236) called kvm=ffff800034a80000
[  260.770077] kvm_vm_release(1374) called kvm=ffff800034a80000
[  261.246087] vgic_v4_init(267) called kvm=ffff800034a20000 doorbell=6668(64)
[  261.664859] kvm_create_vm(1236) called kvm=ffff8000357a0000
[  261.673366] kvm_create_vm(1236) called kvm=ffff8000357e0000
[  261.679345] kvm_vm_release(1374) called kvm=ffff8000357e0000
[  261.685576] kvm_create_vm(1236) called kvm=ffff800035800000
[  261.691396] kvm_vm_release(1374) called kvm=ffff800035800000
[  262.131167] vgic_v4_init(267) called kvm=ffff8000357a0000 doorbell=7756(64)
[  262.985089] kvm_create_vm(1236) called kvm=ffff8000367a0000
[  262.995954] kvm_create_vm(1236) called kvm=ffff8000367e0000
[  263.001694] kvm_vm_release(1374) called kvm=ffff8000367e0000
[  263.008196] kvm_create_vm(1236) called kvm=ffff800036820000
[  263.013926] kvm_vm_release(1374) called kvm=ffff800036820000
[  263.520661] vgic_v4_init(267) called kvm=ffff8000367a0000 doorbell=8844(64)
[  264.045132] kvm_create_vm(1236) called kvm=ffff8000376e0000
[  264.052028] kvm_create_vm(1236) called kvm=ffff800037720000
[  264.057955] kvm_vm_release(1374) called kvm=ffff800037720000
[  264.069762] kvm_create_vm(1236) called kvm=ffff800037740000
[  264.075502] kvm_vm_release(1374) called kvm=ffff800037740000
[  264.589563] vgic_v4_init(267) called kvm=ffff8000376e0000 doorbell=9932(64)
[  264.986861] kvm_create_vm(1236) called kvm=ffff80003fec0000
[  264.997744] kvm_create_vm(1236) called kvm=ffff80003ff00000
[  265.003516] kvm_vm_release(1374) called kvm=ffff80003ff00000
[  265.013497] kvm_create_vm(1236) called kvm=ffff80003ff40000
[  265.019257] kvm_vm_release(1374) called kvm=ffff80003ff40000
[  265.487934] vgic_v4_init(267) called kvm=ffff80003fec0000 doorbell=11020(64)
[  266.353271] kvm_create_vm(1236) called kvm=ffff800050f80000
[  266.362554] kvm_create_vm(1236) called kvm=ffff800050fa0000
[  266.368305] kvm_vm_release(1374) called kvm=ffff800050fa0000
[  266.374783] kvm_create_vm(1236) called kvm=ffff800050fe0000
[  266.380501] kvm_vm_release(1374) called kvm=ffff800050fe0000
[  267.123984] vgic_v4_init(267) called kvm=ffff800050f80000 doorbell=12108(64)
[  267.862029] kvm_create_vm(1236) called kvm=ffff800016080000
[  267.871450] kvm_create_vm(1236) called kvm=ffff8000160c0000
[  267.877187] kvm_vm_release(1374) called kvm=ffff8000160c0000
[  267.889841] kvm_create_vm(1236) called kvm=ffff800016120000
[  267.895563] kvm_vm_release(1374) called kvm=ffff800016120000
[  268.414704] vgic_v4_init(267) called kvm=ffff800016080000 doorbell=13196(64)
[  269.233206] kvm_create_vm(1236) called kvm=ffff8000179a0000
[  269.250008] kvm_create_vm(1236) called kvm=ffff8000179e0000
[  269.255737] kvm_vm_release(1374) called kvm=ffff8000179e0000
[  269.293831] kvm_create_vm(1236) called kvm=ffff800017a40000
[  269.299545] kvm_vm_release(1374) called kvm=ffff800017a40000
[  269.752031] vgic_v4_init(267) called kvm=ffff8000179a0000 doorbell=14284(64)
[  270.402065] kvm_create_vm(1236) called kvm=ffff800018920000
[  270.415703] kvm_create_vm(1236) called kvm=ffff800018960000
[  270.421433] kvm_vm_release(1374) called kvm=ffff800018960000
[  270.428963] kvm_create_vm(1236) called kvm=ffff800018980000
[  270.434718] kvm_vm_release(1374) called kvm=ffff800018980000
[  271.030587] vgic_v4_init(267) called kvm=ffff800018920000 doorbell=15372(64)
[  271.941460] kvm_create_vm(1236) called kvm=ffff800019a80000
[  271.951108] kvm_create_vm(1236) called kvm=ffff800019ac0000
[  271.976587] kvm_vm_release(1374) called kvm=ffff800019ac0000
[  271.993855] kvm_create_vm(1236) called kvm=ffff800019b00000
[  272.024367] kvm_vm_release(1374) called kvm=ffff800019b00000
[  272.812285] vgic_v4_init(267) called kvm=ffff800019a80000 doorbell=16460(64)
[  274.138056] kvm_create_vm(1236) called kvm=ffff80001b020000
[  274.147482] kvm_create_vm(1236) called kvm=ffff80001b040000
[  274.153208] kvm_vm_release(1374) called kvm=ffff80001b040000
[  274.274058] kvm_create_vm(1236) called kvm=ffff80001b120000
[  274.279777] kvm_vm_release(1374) called kvm=ffff80001b120000
[  274.821977] vgic_v4_init(267) called kvm=ffff80001b020000 doorbell=17548(64)
[  275.289483] kvm_create_vm(1236) called kvm=ffff80001c1a0000
[  275.303975] kvm_create_vm(1236) called kvm=ffff80001c1e0000
[  275.309712] kvm_vm_release(1374) called kvm=ffff80001c1e0000
[  275.406027] kvm_create_vm(1236) called kvm=ffff80001c240000
[  275.411756] kvm_vm_release(1374) called kvm=ffff80001c240000
[  275.914284] vgic_v4_init(267) called kvm=ffff80001c1a0000 doorbell=18636(64)
[  276.713109] kvm_create_vm(1236) called kvm=ffff80001d240000
[  276.721696] kvm_create_vm(1236) called kvm=ffff80001d280000
[  276.727421] kvm_vm_release(1374) called kvm=ffff80001d280000
[  276.734615] kvm_create_vm(1236) called kvm=ffff80001d2a0000
[  276.740342] kvm_vm_release(1374) called kvm=ffff80001d2a0000
[  277.341173] vgic_v4_init(267) called kvm=ffff80001d240000 doorbell=19724(64)
[  277.684981] kvm_create_vm(1236) called kvm=ffff80001e0a0000
[  277.694807] kvm_create_vm(1236) called kvm=ffff80001e0e0000
[  277.700544] kvm_vm_release(1374) called kvm=ffff80001e0e0000
[  277.706794] kvm_create_vm(1236) called kvm=ffff80001e120000
[  277.712519] kvm_vm_release(1374) called kvm=ffff80001e120000
[  278.158511] vgic_v4_init(267) called kvm=ffff80001e0a0000 doorbell=20812(64)
[  278.965615] kvm_create_vm(1236) called kvm=ffff80001f0c0000
[  278.973233] kvm_create_vm(1236) called kvm=ffff80001f0e0000
[  278.979624] kvm_vm_release(1374) called kvm=ffff80001f0e0000
[  278.988106] kvm_create_vm(1236) called kvm=ffff80001f120000
[  278.994484] kvm_vm_release(1374) called kvm=ffff80001f120000
[  279.408259] vgic_v4_init(267) called kvm=ffff80001f0c0000 doorbell=21900(64)
[  280.062592] kvm_create_vm(1236) called kvm=ffff8000300a0000
[  280.089029] kvm_create_vm(1236) called kvm=ffff8000300e0000
[  280.095744] kvm_vm_release(1374) called kvm=ffff8000300e0000
[  280.102889] kvm_create_vm(1236) called kvm=ffff800030100000
[  280.108646] kvm_vm_release(1374) called kvm=ffff800030100000
[  280.590774] vgic_v4_init(267) called kvm=ffff8000300a0000 doorbell=22988(64)
[  280.905459] kvm_create_vm(1236) called kvm=ffff800033ac0000
[  280.914767] kvm_create_vm(1236) called kvm=ffff800033b00000
[  280.920505] kvm_vm_release(1374) called kvm=ffff800033b00000
[  280.926616] kvm_create_vm(1236) called kvm=ffff800033b20000
[  280.932330] kvm_vm_release(1374) called kvm=ffff800033b20000
[  281.583530] vgic_v4_init(267) called kvm=ffff800033ac0000 doorbell=24076(64)
[  282.689082] kvm_create_vm(1236) called kvm=ffff800036820000
[  282.699338] kvm_create_vm(1236) called kvm=ffff800036840000
[  282.705066] kvm_vm_release(1374) called kvm=ffff800036840000
[  282.711249] kvm_create_vm(1236) called kvm=ffff8000369c0000
[  282.716981] kvm_vm_release(1374) called kvm=ffff8000369c0000
[  283.247998] vgic_v4_init(267) called kvm=ffff800036820000 doorbell=25164(64)
[  284.104919] kvm_create_vm(1236) called kvm=ffff800050cc0000
[  284.120141] kvm_create_vm(1236) called kvm=ffff800050ce0000
[  284.126130] kvm_vm_release(1374) called kvm=ffff800050ce0000
[  284.132492] kvm_create_vm(1236) called kvm=ffff800050d20000
[  284.138297] kvm_vm_release(1374) called kvm=ffff800050d20000
[  284.646071] vgic_v4_init(267) called kvm=ffff800050cc0000 doorbell=26252(64)
[  285.211935] kvm_create_vm(1236) called kvm=ffff800051ea0000
[  285.314317] kvm_create_vm(1236) called kvm=ffff800051f40000
[  285.320257] kvm_vm_release(1374) called kvm=ffff800051f40000
[  285.326383] kvm_create_vm(1236) called kvm=ffff800051f80000
[  285.332193] kvm_vm_release(1374) called kvm=ffff800051f80000
[  285.852482] vgic_v4_init(267) called kvm=ffff800051ea0000 doorbell=27340(64)
[  286.285262] kvm_create_vm(1236) called kvm=ffff800052da0000
[  286.294007] kvm_create_vm(1236) called kvm=ffff800052dc0000
[  286.299752] kvm_vm_release(1374) called kvm=ffff800052dc0000
[  286.305854] kvm_create_vm(1236) called kvm=ffff800052e00000
[  286.311567] kvm_vm_release(1374) called kvm=ffff800052e00000
[  286.741362] vgic_v4_init(267) called kvm=ffff800052da0000 doorbell=28428(64)
[  287.413763] kvm_create_vm(1236) called kvm=ffff800053d00000
[  287.423054] kvm_create_vm(1236) called kvm=ffff800053d20000
[  287.428812] kvm_vm_release(1374) called kvm=ffff800053d20000
[  287.442072] kvm_create_vm(1236) called kvm=ffff800053d60000
[  287.447830] kvm_vm_release(1374) called kvm=ffff800053d60000
[  288.066585] vgic_v4_init(267) called kvm=ffff800053d00000 doorbell=29516(64)
[  288.581236] kvm_create_vm(1236) called kvm=ffff800054d20000
[  288.588253] kvm_create_vm(1236) called kvm=ffff800054d40000
[  288.594051] kvm_vm_release(1374) called kvm=ffff800054d40000
[  288.605777] kvm_create_vm(1236) called kvm=ffff800054d80000
[  288.611535] kvm_vm_release(1374) called kvm=ffff800054d80000
[  289.094571] vgic_v4_init(267) called kvm=ffff800054d20000 doorbell=30604(64)
[  289.795308] kvm_create_vm(1236) called kvm=ffff800055d80000
[  289.803028] kvm_create_vm(1236) called kvm=ffff800055da0000
[  289.809110] kvm_vm_release(1374) called kvm=ffff800055da0000
[  289.815422] kvm_create_vm(1236) called kvm=ffff800055de0000
[  289.821154] kvm_vm_release(1374) called kvm=ffff800055de0000
[  290.271484] vgic_v4_init(267) called kvm=ffff800055d80000 doorbell=31692(64)
[  290.589450] kvm_create_vm(1236) called kvm=ffff800056b40000
[  290.601373] kvm_create_vm(1236) called kvm=ffff800056b80000
[  290.607220] kvm_vm_release(1374) called kvm=ffff800056b80000
[  290.613773] kvm_create_vm(1236) called kvm=ffff800056bc0000
[  290.619505] kvm_vm_release(1374) called kvm=ffff800056bc0000
[  291.059917] vgic_v4_init(267) called kvm=ffff800056b40000 doorbell=32780(64)
[  291.801179] kvm_create_vm(1236) called kvm=ffff800057a60000
[  291.811298] kvm_create_vm(1236) called kvm=ffff800057aa0000
[  291.817055] kvm_vm_release(1374) called kvm=ffff800057aa0000
[  291.829798] kvm_create_vm(1236) called kvm=ffff800057ac0000
[  291.835522] kvm_vm_release(1374) called kvm=ffff800057ac0000
[  292.364622] vgic_v4_init(267) called kvm=ffff800057a60000 doorbell=33868(64)
[  517.942167] vgic_set_common_attr(263) called kvm=ffff8000303e0000
[  517.948415] vgic_v3_save_pending_tables(397) called kvm=ffff8000303e0000
[  517.955602] vgic_v3_save_pending_tables(448) called kvm=ffff8000303e0000
[  518.099696] kvm_vm_release(1374) called kvm=ffff8000303e0000
[  518.105775] vgic_set_common_attr(263) called kvm=ffff800031020000
[  518.112012] vgic_v3_save_pending_tables(397) called kvm=ffff800031020000
[  518.119195] vgic_v3_save_pending_tables(448) called kvm=ffff800031020000
[  518.126833] vgic_v4_teardown(323) started kvm=ffff8000303e0000 doorbell=140(64)
[  518.134677] vgic_v4_teardown(333) finished kvm=ffff8000303e0000 doorbell=140(64)
[  518.146755] kvm_vm_release(1374) called kvm=ffff800031020000
[  518.152869] vgic_v4_teardown(323) started kvm=ffff800031020000 doorbell=1228(64)
[  518.160829] vgic_v4_teardown(333) finished kvm=ffff800031020000 doorbell=1228(64)
[  518.723172] vgic_set_common_attr(263) called kvm=ffff8000325e0000
[  518.729460] vgic_v3_save_pending_tables(397) called kvm=ffff8000325e0000
[  518.736992] vgic_v3_save_pending_tables(448) called kvm=ffff8000325e0000
[  518.755085] kvm_vm_release(1374) called kvm=ffff8000325e0000
[  518.761731] vgic_v4_teardown(323) started kvm=ffff8000325e0000 doorbell=3404(64)
[  518.769946] vgic_v4_teardown(333) finished kvm=ffff8000325e0000 doorbell=3404(64)
[  519.506094] vgic_set_common_attr(263) called kvm=ffff800031ae0000
[  519.512335] vgic_v3_save_pending_tables(397) called kvm=ffff800031ae0000
[  519.519512] vgic_v3_save_pending_tables(448) called kvm=ffff800031ae0000
[  519.555086] kvm_vm_release(1374) called kvm=ffff800031ae0000
[  519.561047] vgic_v4_teardown(323) started kvm=ffff800031ae0000 doorbell=2316(64)
[  519.568990] vgic_v4_teardown(333) finished kvm=ffff800031ae0000 doorbell=2316(64)
[  519.762370] vgic_set_common_attr(263) called kvm=ffff8000330c0000
[  519.768607] vgic_v3_save_pending_tables(397) called kvm=ffff8000330c0000
[  519.775782] vgic_v3_save_pending_tables(448) called kvm=ffff8000330c0000
[  519.788056] kvm_vm_release(1374) called kvm=ffff8000330c0000
[  519.794232] vgic_v4_teardown(323) started kvm=ffff8000330c0000 doorbell=4492(64)
[  519.802317] vgic_v4_teardown(333) finished kvm=ffff8000330c0000 doorbell=4492(64)
[  520.182176] vgic_set_common_attr(263) called kvm=ffff800033c80000
[  520.188417] vgic_v3_save_pending_tables(397) called kvm=ffff800033c80000
[  520.195584] vgic_v3_save_pending_tables(448) called kvm=ffff800033c80000
[  520.207328] kvm_vm_release(1374) called kvm=ffff800033c80000
[  520.213361] vgic_v4_teardown(323) started kvm=ffff800033c80000 doorbell=5580(64)
[  520.220387] vgic_set_common_attr(263) called kvm=ffff800034a20000
[  520.221321] vgic_v4_teardown(333) finished kvm=ffff800033c80000 doorbell=5580(64)
[  520.227161] vgic_v3_save_pending_tables(397) called kvm=ffff800034a20000
[  520.242001] vgic_v3_save_pending_tables(448) called kvm=ffff800034a20000
[  520.251585] kvm_vm_release(1374) called kvm=ffff800034a20000
[  520.257593] vgic_v4_teardown(323) started kvm=ffff800034a20000 doorbell=6668(64)
[  520.265558] vgic_v4_teardown(333) finished kvm=ffff800034a20000 doorbell=6668(64)
[  521.395539] vgic_set_common_attr(263) called kvm=ffff8000357a0000
[  521.401782] vgic_v3_save_pending_tables(397) called kvm=ffff8000357a0000
[  521.408953] vgic_v3_save_pending_tables(448) called kvm=ffff8000357a0000
[  521.418300] kvm_vm_release(1374) called kvm=ffff8000357a0000
[  521.424240] vgic_v4_teardown(323) started kvm=ffff8000357a0000 doorbell=7756(64)
[  521.432182] vgic_v4_teardown(333) finished kvm=ffff8000357a0000 doorbell=7756(64)
[  525.002673] vgic_set_common_attr(263) called kvm=ffff8000367a0000
[  525.008914] vgic_v3_save_pending_tables(397) called kvm=ffff8000367a0000
[  525.016143] vgic_v3_save_pending_tables(448) called kvm=ffff8000367a0000
[  525.027230] kvm_vm_release(1374) called kvm=ffff8000367a0000
[  525.033257] vgic_v4_teardown(323) started kvm=ffff8000367a0000 doorbell=8844(64)
[  525.041195] vgic_v4_teardown(333) finished kvm=ffff8000367a0000 doorbell=8844(64)
[  525.090124] vgic_set_common_attr(263) called kvm=ffff8000376e0000
[  525.096372] vgic_v3_save_pending_tables(397) called kvm=ffff8000376e0000
[  525.103612] vgic_v3_save_pending_tables(448) called kvm=ffff8000376e0000
[  525.110594] irq: irqd_set_activated: CPU49 IRQ9940 lost IRQD_IRQ_INPROGRESS old=0x10401400 new=0x10401600, expected=0x10441600 kvm=ffff8000376e0000
[  525.124122] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS set time [525.110593]
[  525.132397] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS clr time [525.110595]
[  525.143493] kvm_vm_release(1374) called kvm=ffff8000376e0000
[  525.149518] vgic_v4_teardown(323) started kvm=ffff8000376e0000 doorbell=9932(64)
[  525.157503] vgic_v4_teardown(333) finished kvm=ffff8000376e0000 doorbell=9932(64)
[  529.649331] vgic_set_common_attr(263) called kvm=ffff80003fec0000
[  529.655575] vgic_v3_save_pending_tables(397) called kvm=ffff80003fec0000
[  529.662767] vgic_v3_save_pending_tables(448) called kvm=ffff80003fec0000
[  529.672309] kvm_vm_release(1374) called kvm=ffff80003fec0000
[  529.678306] vgic_v4_teardown(323) started kvm=ffff80003fec0000 doorbell=11020(64)
[  529.686330] vgic_v4_teardown(333) finished kvm=ffff80003fec0000 doorbell=11020(64)
[  530.424380] vgic_set_common_attr(263) called kvm=ffff800050f80000
[  530.430619] vgic_v3_save_pending_tables(397) called kvm=ffff800050f80000
[  530.437805] vgic_v3_save_pending_tables(448) called kvm=ffff800050f80000
[  530.469006] kvm_vm_release(1374) called kvm=ffff800050f80000
[  530.476493] vgic_v4_teardown(323) started kvm=ffff800050f80000 doorbell=12108(64)
[  530.485410] vgic_v4_teardown(333) finished kvm=ffff800050f80000 doorbell=12108(64)
[  533.584697] vgic_set_common_attr(263) called kvm=ffff800016080000
[  533.590950] vgic_v3_save_pending_tables(397) called kvm=ffff800016080000
[  533.598124] vgic_v3_save_pending_tables(448) called kvm=ffff800016080000
[  533.610606] kvm_vm_release(1374) called kvm=ffff800016080000
[  533.616952] vgic_v4_teardown(323) started kvm=ffff800016080000 doorbell=13196(64)
[  533.625146] vgic_v4_teardown(333) finished kvm=ffff800016080000 doorbell=13196(64)
[  533.725021] vgic_set_common_attr(263) called kvm=ffff8000179a0000
[  533.731258] vgic_v3_save_pending_tables(397) called kvm=ffff8000179a0000
[  533.738444] vgic_v3_save_pending_tables(448) called kvm=ffff8000179a0000
[  533.758156] kvm_vm_release(1374) called kvm=ffff8000179a0000
[  533.764364] vgic_v4_teardown(323) started kvm=ffff8000179a0000 doorbell=14284(64)
[  533.772633] vgic_v4_teardown(333) finished kvm=ffff8000179a0000 doorbell=14284(64)
[  535.981082] vgic_set_common_attr(263) called kvm=ffff800018920000
[  535.987328] vgic_v3_save_pending_tables(397) called kvm=ffff800018920000
[  535.994532] vgic_v3_save_pending_tables(448) called kvm=ffff800018920000
[  536.001494] irq: irqd_set_activated: CPU19 IRQ15375 lost IRQD_IRQ_INPROGRESS old=0x10441400 new=0x10441600, expected=0x10401600 kvm=ffff800018920000
[  536.015125] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS set time [536.001493]
[  536.023400] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS clr time [536.001493]
[  536.052598] kvm_vm_release(1374) called kvm=ffff800018920000
[  536.059510] vgic_v4_teardown(323) started kvm=ffff800018920000 doorbell=15372(64)
[  536.512174] vgic_set_common_attr(263) called kvm=ffff80001b020000
[  536.518413] vgic_v3_save_pending_tables(397) called kvm=ffff80001b020000
[  536.525626] vgic_v3_save_pending_tables(448) called kvm=ffff80001b020000
[  536.561496] kvm_vm_release(1374) called kvm=ffff80001b020000
[  536.568194] vgic_v4_teardown(323) started kvm=ffff80001b020000 doorbell=17548(64)
[  536.576982] vgic_v4_teardown(333) finished kvm=ffff80001b020000 doorbell=17548(64)
[  537.657725] vgic_set_common_attr(263) called kvm=ffff800019a80000
[  537.663958] vgic_v3_save_pending_tables(397) called kvm=ffff800019a80000
[  537.671130] vgic_v3_save_pending_tables(448) called kvm=ffff800019a80000
[  537.711212] kvm_vm_release(1374) called kvm=ffff800019a80000
[  537.718230] vgic_v4_teardown(323) started kvm=ffff800019a80000 doorbell=16460(64)
[  537.727162] vgic_v4_teardown(333) finished kvm=ffff800019a80000 doorbell=16460(64)
[  538.602096] vgic_set_common_attr(263) called kvm=ffff80001c1a0000
[  538.608378] vgic_v3_save_pending_tables(397) called kvm=ffff80001c1a0000
[  538.615831] vgic_v3_save_pending_tables(448) called kvm=ffff80001c1a0000
[  538.627140] kvm_vm_release(1374) called kvm=ffff80001c1a0000
[  538.633337] vgic_v4_teardown(323) started kvm=ffff80001c1a0000 doorbell=18636(64)
[  538.641470] vgic_v4_teardown(333) finished kvm=ffff80001c1a0000 doorbell=18636(64)
[  541.777190] vgic_set_common_attr(263) called kvm=ffff80001d240000
[  541.783431] vgic_v3_save_pending_tables(397) called kvm=ffff80001d240000
[  541.790748] vgic_v3_save_pending_tables(448) called kvm=ffff80001d240000
[  541.801177] kvm_vm_release(1374) called kvm=ffff80001d240000
[  541.807262] vgic_v4_teardown(323) started kvm=ffff80001d240000 doorbell=19724(64)
[  541.815301] vgic_v4_teardown(333) finished kvm=ffff80001d240000 doorbell=19724(64)
[  545.079117] vgic_set_common_attr(263) called kvm=ffff80001e0a0000
[  545.085358] vgic_v3_save_pending_tables(397) called kvm=ffff80001e0a0000
[  545.092580] vgic_v3_save_pending_tables(448) called kvm=ffff80001e0a0000
[  545.099562] irq: irqd_set_activated: CPU49 IRQ20821 lost IRQD_IRQ_INPROGRESS old=0x10401400 new=0x10401600, expected=0x10441600 kvm=ffff80001e0a0000
[  545.113177] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS set time [545.099561]
[  545.121454] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS clr time [545.099562]
[  545.129755] irq: irqd_set_activated: CPU49 IRQ20826 lost IRQD_IRQ_INPROGRESS old=0x10441400 new=0x10441600, expected=0x10401600 kvm=ffff80001e0a0000
[  545.143365] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS set time [545.129754]
[  545.151654] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS clr time [545.129755]
[  545.163250] kvm_vm_release(1374) called kvm=ffff80001e0a0000
[  545.169204] vgic_v4_teardown(323) started kvm=ffff80001e0a0000 doorbell=20812(64)
[  557.073632] rcu: INFO: rcu_sched self-detected stall on CPU
[  557.079331] rcu:     48-....: (5250 ticks this GP) idle=399c/1/0x4000000000000000 softirq=36358/36358 fqs=2052
[  557.089294]  (t=5254 jiffies g=106321 q=10324 ncpus=72)
[  557.094636] CPU: 48 PID: 1631 Comm: qemu-system-aar Kdump: loaded Not tainted 6.2.0-rc4 #11
[  557.103176] Hardware name: NVIDIA Ranger/th500_silicon, BIOS v1.1.1-4638c6c9 01/17/2023
[  557.111359] pstate: 63400009 (nZCv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
[  557.118478] pc : __synchronize_hardirq+0x48/0x154
[  557.123294] lr : free_irq+0x134/0x460
[  557.127034] sp : ffff800018b6f990
[  557.130417] x29: ffff800018b6f990 x28: 000000000000000d x27: 0000000000000001
[  557.137713] x26: 0000000000003c0f x25: ffff0000c82aa8a0 x24: 0000000000000001
[  557.145008] x23: ffff0000836e0cf4 x22: 0000000000000000 x21: 0000000000000001
[  557.152303] x20: ffff0000836e0cf4 x19: ffff0000836e0c60 x18: ffff800018b6f510
[  557.159598] x17: 373335313d6c6c65 x16: 62726f6f64203030 x15: 3030323938313030
[  557.166893] x14: 3038666666663d6d x13: 2934362832373335 x12: 313d6c6c6562726f
[  557.174189] x11: 0000000000000040 x10: ffff800009230a58 x9 : 0000000000000000
[  557.181484] x8 : ffff800018b6f8e0 x7 : 0000000000000000 x6 : 0000000000000001
[  557.188779] x5 : ffff800008355054 x4 : ffffffc0012c5760 x3 : 0000000081550151
[  557.196075] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000010071200
[  557.203371] Call trace:
[  557.205865]  __synchronize_hardirq+0x48/0x154
[  557.210317]  free_irq+0x134/0x460
[  557.213700]  vgic_v4_teardown+0xdc/0x138
[  557.217712]  __kvm_vgic_destroy+0x18c/0x194
[  557.221985]  kvm_vgic_destroy+0x28/0x3c
[  557.225902]  kvm_arch_destroy_vm+0x24/0x80
[  557.230087]  kvm_destroy_vm+0x158/0x2f8
[  557.234006]  kvm_put_kvm+0x60/0x80
[  557.237479]  kvm_vm_release+0x44/0x54
[  557.241221]  __fput+0x70/0x250
[  557.244342]  ____fput+0x10/0x20
[  557.247548]  task_work_run+0xc8/0x178
[  557.251290]  do_exit+0x2c8/0x8e0
[  557.254589]  do_group_exit+0x34/0xa0
[  557.258239]  get_signal+0x8bc/0x8e4
[  557.261801]  do_notify_resume+0x144/0x1220
[  557.265989]  el0_svc+0xdc/0xf0
[  557.269109]  el0t_64_sync_handler+0xf0/0x11c
[  557.273470]  el0t_64_sync+0x190/0x194
[  566.921633] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [qemu-system-aar:1976]
[  566.929560] Modules linked in:
[  566.932679] CPU: 0 PID: 1976 Comm: qemu-system-aar Kdump: loaded Not tainted 6.2.0-rc4 #11
[  566.941129] Hardware name: NVIDIA Ranger/th500_silicon, BIOS v1.1.1-4638c6c9 01/17/2023
[  566.949312] pstate: 63400009 (nZCv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
[  566.956429] pc : __synchronize_hardirq+0x54/0x154
[  566.961240] lr : free_irq+0x134/0x460
[  566.964980] sp : ffff80003252f990
[  566.968362] x29: ffff80003252f990 x28: 000000000000000d x27: 0000000000000001
[  566.975658] x26: 000000000000515a x25: ffff0000c56ace70 x24: 0000000000000001
[  566.982954] x23: ffff0000cf1126f4 x22: 0000000000000000 x21: 0000000000000001
[  566.990249] x20: ffff0000cf1126f4 x19: ffff0000cf112660 x18: ffff80003252f510
[  566.997545] x17: 313830323d6c6c65 x16: 62726f6f64203030 x15: 3030613065313030
[  567.004840] x14: 3038666666663d6d x13: 0000000000000000 x12: 0000000000000006
[  567.012136] x11: 0000000000000040 x10: ffff800009230a58 x9 : 0000000000000000
[  567.019431] x8 : ffff80003252f8e0 x7 : 0000000000000000 x6 : 0000000000000001
[  567.026727] x5 : ffff800008355054 x4 : ffffffc0019232e0 x3 : 0000000081550151
[  567.034023] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000010071200
[  567.041318] Call trace:
[  567.043812]  __synchronize_hardirq+0x54/0x154
[  567.048264]  free_irq+0x134/0x460
[  567.051648]  vgic_v4_teardown+0xdc/0x138
[  567.055655]  __kvm_vgic_destroy+0x18c/0x194
[  567.059928]  kvm_vgic_destroy+0x28/0x3c
[  567.063844]  kvm_arch_destroy_vm+0x24/0x80
[  567.068028]  kvm_destroy_vm+0x158/0x2f8
[  567.071946]  kvm_put_kvm+0x60/0x80
[  567.075418]  kvm_vm_release+0x44/0x54
[  567.079158]  __fput+0x70/0x250
[  567.082275]  ____fput+0x10/0x20
[  567.085480]  task_work_run+0xc8/0x178
[  567.089220]  do_exit+0x2c8/0x8e0
[  567.092515]  do_group_exit+0x34/0xa0
[  567.096166]  get_signal+0x8bc/0x8e4
[  567.099728]  do_notify_resume+0x144/0x1220
[  567.103912]  el0_svc+0xdc/0xf0
[  567.107030]  el0t_64_sync_handler+0xf0/0x11c
[  567.111392]  el0t_64_sync+0x190/0x194
[  567.115134] Kernel panic - not syncing: softlockup: hung tasks
[  567.121097] CPU: 0 PID: 1976 Comm: qemu-system-aar Kdump: loaded Tainted: G             L     6.2.0-rc4 #11
[  567.131058] Hardware name: NVIDIA Ranger/th500_silicon, BIOS v1.1.1-4638c6c9 01/17/2023
[  567.139241] Call trace:
[  567.141734]  dump_backtrace.part.0+0xdc/0xe8
[  567.146096]  show_stack+0x18/0x20
[  567.149479]  dump_stack_lvl+0x64/0x7c
[  567.153218]  dump_stack+0x18/0x30
[  567.156602]  panic+0x178/0x340
[  567.159718]  watchdog_timer_fn+0x28c/0x298
[  567.163907]  __hrtimer_run_queues+0x118/0x260
[  567.168358]  hrtimer_interrupt+0x114/0x31c
[  567.172541]  arch_timer_handler_phys+0x2c/0x40
[  567.177083]  handle_percpu_devid_irq+0x84/0x138
[  567.181710]  generic_handle_domain_irq+0x2c/0x40
[  567.186427]  gic_handle_irq+0x50/0x140
[  567.190257]  call_on_irq_stack+0x2c/0x38
[  567.194262]  do_interrupt_handler+0x7c/0x88
[  567.198535]  el1_interrupt+0x34/0x58
[  567.202185]  el1h_64_irq_handler+0x18/0x20
[  567.206369]  el1h_64_irq+0x64/0x68
[  567.209840]  __synchronize_hardirq+0x54/0x154
[  567.214291]  free_irq+0x134/0x460
[  567.217675]  vgic_v4_teardown+0xdc/0x138
[  567.221681]  __kvm_vgic_destroy+0x18c/0x194
[  567.225953]  kvm_vgic_destroy+0x28/0x3c
[  567.229870]  kvm_arch_destroy_vm+0x24/0x80
[  567.234053]  kvm_destroy_vm+0x158/0x2f8
[  567.237970]  kvm_put_kvm+0x60/0x80
[  567.241443]  kvm_vm_release+0x44/0x54
[  567.245182]  __fput+0x70/0x250
[  567.248298]  ____fput+0x10/0x20
[  567.251503]  task_work_run+0xc8/0x178
[  567.255243]  do_exit+0x2c8/0x8e0
[  567.258538]  do_group_exit+0x34/0xa0
[  567.262189]  get_signal+0x8bc/0x8e4
[  567.265750]  do_notify_resume+0x144/0x1220
[  567.269935]  el0_svc+0xdc/0xf0
[  567.273052]  el0t_64_sync_handler+0xf0/0x11c
[  567.277414]  el0t_64_sync+0x190/0x194
[  567.281161] SMP: stopping secondary CPUs
  

Hi Marc,

On 1/19/23 08:01, Marc Zyngier wrote:
>> On 1/19/23 01:11, Marc Zyngier wrote:
>>> So you can see the VM being torn down while the vgic save sequence is
>>> still in progress?
>>>
>>> If you can actually see that, then this is a much bigger bug than the
>>> simple race you are describing, and we're missing a reference on the
>>> kvm structure. This would be a*MAJOR*  bug.
>>>
>> How do we know vGIC save sequence is in progress while VM is being
>> teardown?  I'm launching/terminating ~32 VMs in a loop to reproduce
>> the issue.
> Errr...*you*  know when you are issuing the save ioctl, right? You
> also know when you are terminating the VM (closing its fd or killing
> the VMM).
> 

Added debug statements to trace the code patch, and tagged each log message
with 'struct kvm *'. Attached the complete kernel log messages including
debug messages.

All 32 VMs launched, time 258s to 291s
  [  258.519837] kvm_create_vm(1236) called kvm=ffff8000303e0000  --> 1st VM
  ...
  [  291.801179] kvm_create_vm(1236) called kvm=ffff800057a60000   --> 32nd VM

Test script inside VM issues poweroff command after sleeping 200sec.

Working case kvm=ffff8000303e0000:

  $ cat gicv4-debug.txt | grep ffff8000303e0000
   [  258.519837] kvm_create_vm(1236) called kvm=ffff8000303e0000
   [  258.667101] vgic_v4_init(267) called kvm=ffff8000303e0000 doorbell=140(64)
   [  517.942167] vgic_set_common_attr(263) called kvm=ffff8000303e0000
   [  517.948415] vgic_v3_save_pending_tables(397) called kvm=ffff8000303e0000
   [  517.955602] vgic_v3_save_pending_tables(448) called kvm=ffff8000303e0000
   [  518.099696] kvm_vm_release(1374) called kvm=ffff8000303e0000
   [  518.126833] vgic_v4_teardown(323) started kvm=ffff8000303e0000 doorbell=140(64)
   [  518.134677] vgic_v4_teardown(333) finished kvm=ffff8000303e0000 doorbell=140(64)

Not working case kvm=ffff80001e0a0000:

  $ cat gicv4-debug.txt | grep ffff80001e0a0000
   [  277.684981] kvm_create_vm(1236) called kvm=ffff80001e0a0000
   [  278.158511] vgic_v4_init(267) called kvm=ffff80001e0a0000 doorbell=20812(64)
   [  545.079117] vgic_set_common_attr(263) called kvm=ffff80001e0a0000
   [  545.085358] vgic_v3_save_pending_tables(397) called kvm=ffff80001e0a0000
   [  545.092580] vgic_v3_save_pending_tables(448) called kvm=ffff80001e0a0000
   [  545.099562] irq: irqd_set_activated: CPU49 IRQ20821 lost IRQD_IRQ_INPROGRESS old=0x10401400 new=0x10401600, expected=0x10441600 kvm=ffff80001e0a0000^M
   [  545.113177] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS set time [545.099561]^M
   [  545.121454] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS clr time [545.099562]^M
   [  545.129755] irq: irqd_set_activated: CPU49 IRQ20826 lost IRQD_IRQ_INPROGRESS old=0x10441400 new=0x10441600, expected=0x10401600 kvm=ffff80001e0a0000^M
   [  545.143365] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS set time [545.129754]^M
   [  545.151654] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS clr time [545.129755]^M
   [  545.163250] kvm_vm_release(1374) called kvm=ffff80001e0a0000
   [  545.169204] vgic_v4_teardown(323) started kvm=ffff80001e0a0000 doorbell=20812(64)

IRQD_IRQ_INPROGRESS is corrupted before calling kvm_vm_release(),

   Time             State
  -----------------------
   [277.684981]    VM created
   [278.158511]    Started executing GICv4 init code
   [545.092580]    VGIC save/pending called
   [545.099561]    set IRQD_IRQ_INPROGRESS
   [545.099562]    clr IRQD_IRQ_INPROGRESS & irqd_set_activated()  ---> same/close time
   [545.163250]    kvm_vm_release() called

Software:
   Linux kernel: 6.2.0-rc4
   QEMU: 7.1.0

Debug patch:

  diff --git a/arch/arm64/kvm/vgic/vgic-kvm-device.c b/arch/arm64/kvm/vgic/vgic-kvm-device.c
index edeac2380591..da922ac72dc6 100644
--- a/arch/arm64/kvm/vgic/vgic-kvm-device.c
+++ b/arch/arm64/kvm/vgic/vgic-kvm-device.c
@@ -260,6 +260,7 @@ static int vgic_set_common_attr(struct kvm_device *dev,
                                 mutex_unlock(&dev->kvm->lock);
                                 return -EBUSY;
                         }
+                       pr_err("%s(%d) called kvm=%px\n",__func__,__LINE__,dev->kvm);
                         r = vgic_v3_save_pending_tables(dev->kvm);
                         unlock_all_vcpus(dev->kvm);
                         mutex_unlock(&dev->kvm->lock);
diff --git a/arch/arm64/kvm/vgic/vgic-v3.c b/arch/arm64/kvm/vgic/vgic-v3.c
index 2074521d4a8c..91b53646f7ec 100644
--- a/arch/arm64/kvm/vgic/vgic-v3.c
+++ b/arch/arm64/kvm/vgic/vgic-v3.c
@@ -394,6 +394,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
          * and enabling of the doorbells have already been done.
          */
         if (kvm_vgic_global_state.has_gicv4_1) {
+               pr_err("%s(%d) called kvm=%px \n",__func__,__LINE__,kvm);
                 unmap_all_vpes(dist);
                 vlpi_avail = true;
         }
@@ -443,8 +444,10 @@ int vgic_v3_save_pending_tables(struct kvm *kvm)
         }

  out:
-       if (vlpi_avail)
+       if (vlpi_avail) {
+               pr_err("%s(%d) called kvm=%px \n",__func__,__LINE__,kvm);
                 map_all_vpes(dist);
+       }
diff --git a/arch/arm64/kvm/vgic/vgic-v4.c b/arch/arm64/kvm/vgic/vgic-v4.c
index ad06ba6c9b00..71d234844acd 100644
--- a/arch/arm64/kvm/vgic/vgic-v4.c
+++ b/arch/arm64/kvm/vgic/vgic-v4.c
@@ -264,10 +264,14 @@ int vgic_v4_init(struct kvm *kvm)
                 dist->its_vm.vpes = NULL;
                 return ret;
         }
+       pr_err("%s(%d) called kvm=%px doorbell=%d(%d)\n", __func__, __LINE__, kvm, dist->its_vm.vpes[0]->irq, nr_vcpus);

         kvm_for_each_vcpu(i, vcpu, kvm) {
                 int irq = dist->its_vm.vpes[i]->irq;
                 unsigned long irq_flags = DB_IRQ_FLAGS;
+               struct irq_desc *desc = irq_to_desc(irq);
+
+               desc->irq_data.dbgdata = kvm;

                 /*
                  * Don't automatically enable the doorbell, as we're
@@ -316,6 +320,8 @@ void vgic_v4_teardown(struct kvm *kvm)
         if (!its_vm->vpes)
                 return;

+       pr_err("%s(%d) started kvm=%px doorbell=%d(%d)\n", __func__, __LINE__, kvm, its_vm->vpes[0]->irq, its_vm->nr_vpes);
+
         for (i = 0; i < its_vm->nr_vpes; i++) {
                 struct kvm_vcpu *vcpu = kvm_get_vcpu(kvm, i);
                 int irq = its_vm->vpes[i]->irq;
@@ -324,6 +330,7 @@ void vgic_v4_teardown(struct kvm *kvm)
                 free_irq(irq, vcpu);
         }

+       pr_err("%s(%d) finished kvm=%px doorbell=%d(%d)\n", __func__, __LINE__, kvm, its_vm->vpes[0]->irq, its_vm->nr_vpes);
         its_free_vcpu_irqs(its_vm);
         kfree(its_vm->vpes);
         its_vm->nr_vpes = 0;
diff --git a/include/linux/irq.h b/include/linux/irq.h
index c3eb89606c2b..3e3d6ae8d59f 100644
--- a/include/linux/irq.h
+++ b/include/linux/irq.h
@@ -187,6 +187,8 @@ struct irq_data {
         struct irq_data         *parent_data;
  #endif
         void                    *chip_data;
+       unsigned long           dbgtime[2];
+       void                    *dbgdata;
  };

  /*
@@ -390,12 +392,35 @@ static inline bool irqd_is_activated(struct irq_data *d)

  static inline void irqd_set_activated(struct irq_data *d)
  {
-       __irqd_to_state(d) |= IRQD_ACTIVATED;
+       atomic_t *state = (atomic_t *)(&__irqd_to_state(d));
+       unsigned int new, old = __irqd_to_state(d);
+
+       new = atomic_xchg(state, old | IRQD_ACTIVATED);
+
+       if ((old ^ new) & IRQD_IRQ_INPROGRESS) {
+               pr_err("%s: CPU%u IRQ%u lost IRQD_IRQ_INPROGRESS old=0x%x new=0x%x, expected=0x%x kvm=%px\n",
+                      __func__, raw_smp_processor_id(), d->irq,
+                       old, old | IRQD_ACTIVATED, new | IRQD_ACTIVATED, d->dbgdata);
+               pr_err("%s: IRQD_IRQ_INPROGRESS set time [%5lu.%06u] \n", __func__, d->dbgtime[0], do_div(d->dbgtime[0], 1000000000)/1000);
+               pr_err("%s: IRQD_IRQ_INPROGRESS clr time [%5lu.%06u] \n", __func__, d->dbgtime[1], do_div(d->dbgtime[1], 1000000000)/1000);
+       }
+
  }

  static inline void irqd_clr_activated(struct irq_data *d)
  {
-       __irqd_to_state(d) &= ~IRQD_ACTIVATED;
+       atomic_t *state = (atomic_t *)(&__irqd_to_state(d));
+       unsigned int new, old = __irqd_to_state(d);
+
+       new = atomic_xchg(state, old & ~IRQD_ACTIVATED);
+
+       if ((old ^ new) & IRQD_IRQ_INPROGRESS) {
+               pr_err("%s: CPU%u IRQ%u lost IRQD_IRQ_INPROGRESS old=0x%x new=0x%x, expected=0x%x kvm=%px\n",
+                      __func__, raw_smp_processor_id(), d->irq,
+                       old, old & ~IRQD_ACTIVATED, new & ~IRQD_ACTIVATED, d->dbgdata);
+               pr_err("%s: IRQD_IRQ_INPROGRESS set time [%5lu.%06u] \n", __func__, d->dbgtime[0], do_div(d->dbgtime[0], 1000000000)/1000);
+               pr_err("%s: IRQD_IRQ_INPROGRESS clr time [%5lu.%06u] \n", __func__, d->dbgtime[1], do_div(d->dbgtime[1], 1000000000)/1000);
+       }
  }

  static inline bool irqd_is_started(struct irq_data *d)
diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c
index 9489f93b3db3..5891ec27ae93 100644
--- a/kernel/irq/handle.c
+++ b/kernel/irq/handle.c
@@ -205,12 +205,14 @@ irqreturn_t handle_irq_event(struct irq_desc *desc)

         desc->istate &= ~IRQS_PENDING;
         irqd_set(&desc->irq_data, IRQD_IRQ_INPROGRESS);
+       desc->irq_data.dbgtime[0] = sched_clock();
         raw_spin_unlock(&desc->lock);

         ret = handle_irq_event_percpu(desc);

         raw_spin_lock(&desc->lock);
         irqd_clear(&desc->irq_data, IRQD_IRQ_INPROGRESS);
+       desc->irq_data.dbgtime[1] = sched_clock();
         return ret;
  }

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 9c60384b5ae0..0fbf17ea3adb 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -1233,6 +1233,7 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
         preempt_notifier_inc();
         kvm_init_pm_notifier(kvm);

+       pr_err("%s(%d) called kvm=%px\n",__func__,__LINE__,kvm);
         return kvm;

  out_err:
@@ -1370,6 +1371,7 @@ static int kvm_vm_release(struct inode *inode, struct file *filp)
  {
         struct kvm *kvm = filp->private_data;

+       pr_err("%s(%d) called kvm=%px\n",__func__,__LINE__, kvm);
         kvm_irqfd_release(kvm);

         kvm_put_kvm(kvm);

         return ret;
  }
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd4f0]
[    0.000000] Linux version 6.0.0 (sdonthineni@SDONTHINENI-DESKTOP) (aarch64-buildroot-linux-gnu-gcc.br_real (Buildroot 2022.08-1040-g3e947bf73d-dirty) 12.2.0, GNU ld (GNU Bin3
[    0.000000] efi: EFI v2.70 by EDK II
[    0.000000] efi: RTPROP=0x2065a95a98 ACPI 2.0=0x204b8f0018 SMBIOS=0xffff0000 SMBIOS 3.0=0x2068e30000 MEMATTR=0x2064104018 ESRT=0x2063e52e18 MEMRESERVE=0x204a909d98
[    0.000000] esrt: Reserving ESRT space from 0x0000002063e52e18 to 0x0000002063e52e50.
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x000000204B8F0018 000024 (v02 NVIDIA)
[    0.000000] ACPI: XSDT 0x000000204B8FE218 000104 (v01 NVIDIA TH500    00000001 NVDA 00000001)
[    0.000000] ACPI: FACP 0x000000204B8F7518 000114 (v06 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: DSDT 0x000000204B8F7698 0016B7 (v02 NVIDIA TH500    00000001 INTL 20200717)
[    0.000000] ACPI: SDEI 0x000000204B8FFF98 000024 (v01 NVIDIA TH500    00000001 NVDA 00000001)
[    0.000000] ACPI: HEST 0x000000204B8FE998 000E88 (v01 NVIDIA TH500    00000001 NVDA 00000001)
[    0.000000] ACPI: BERT 0x000000204B8FFA98 000030 (v01 NVIDIA TH500    00000001 NVDA 00000001)
[    0.000000] ACPI: EINJ 0x000000204B8FFB18 000170 (v01 NVIDIA TH500    00000001 NVDA 00000001)
[    0.000000] ACPI: FPDT 0x000000204B8FFD98 000034 (v01 NVIDIA TH500    00000001 NVDA 00000001)
[    0.000000] ACPI: GTDT 0x000000204B8FE898 000084 (v03 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: APIC 0x000000204B8FBE98 001788 (v05 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: PPTT 0x000000204B8F9218 0021E0 (v03 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204B8F0098 0061C4 (v02 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: SPCR 0x000000204B8FFE18 000050 (v02 NVIDIA TEGRAUAR 00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204B8FE498 0000CC (v02 NVIDIA SERIAL   00000001 INTL 20200717)
[    0.000000] ACPI: SSDT 0x000000204A950018 001341 (v02 NVIDIA BPMP_S0  00000001 INTL 20200717)
[    0.000000] ACPI: MCFG 0x000000204A95FE18 00008C (v01 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204A95F398 000351 (v02 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204A95F798 000353 (v02 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204B8FDA18 000353 (v02 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204B8FDE18 000353 (v02 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204A95D818 000353 (v02 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204A95ED18 0004C2 (v02 NVIDIA TH500    00000001 ARMH 00010000)
[    0.000000] ACPI: IORT 0x000000204A95DC18 000433 (v06 NVIDIA NVIDIORT 00000001 ARMH 00010000)
[    0.000000] ACPI: MPAM 0x000000204B8FFE98 000084 (v01 NVIDIA NVIDMPAM 00000001 ARMH 00010000)
[    0.000000] ACPI: SRAT 0x000000204A95E298 000700 (v03 NVIDIA NVIDSRAT 00000001 ARMH 00010000)
[    0.000000] ACPI: SLIT 0x000000204A95FB98 00012C (v01 NVIDIA NVIDSLIT 00000001 ARMH 00010000)
[    0.000000] ACPI: APMT 0x000000204B8FE598 00013C (v00 NVIDIA TH500    00000001 NVDA 00000001)
[    0.000000] ACPI: SPMI 0x000000204A95FF18 000041 (v05 NVIDIA NVIDSPMI 00000001 ARMH 00010000)
[    0.000000] ACPI: SSDT 0x000000204B8FE718 00007C (v02 INTEL  RamDisk  00001000 INTL 20200717)
[    0.000000] ACPI: NFIT 0x000000204A95FD18 000060 (v01 NVIDIA TH500    00000001 NVDA 00000001)
[    0.000000] ACPI: SPCR: console: pl011,mmio32,0xc280000,115200
[    0.000000] earlycon: pl11 at MMIO32 0x000000000c280000 (options '115200')
[    0.000000] printk: bootconsole [pl11] enabled
[    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x80000000-0x206bffffff]
[    0.000000] ACPI: SRAT: Node 0 PXM 0 [mem 0x206d400000-0x206d7fffff]
[    0.000000] ACPI: SRAT: Node 1 PXM 8 [mem 0x400000000000-0x40ffffffffff] hotplug
[    0.000000] ACPI: SRAT: Node 2 PXM 9 [mem 0x410000000000-0x41ffffffffff] hotplug
[    0.000000] ACPI: SRAT: Node 3 PXM 10 [mem 0x420000000000-0x42ffffffffff] hotplug
[    0.000000] ACPI: SRAT: Node 4 PXM 11 [mem 0x430000000000-0x43ffffffffff] hotplug
[    0.000000] ACPI: SRAT: Node 5 PXM 12 [mem 0x440000000000-0x44ffffffffff] hotplug
[    0.000000] ACPI: SRAT: Node 6 PXM 13 [mem 0x450000000000-0x45ffffffffff] hotplug
[    0.000000] ACPI: SRAT: Node 7 PXM 14 [mem 0x460000000000-0x46ffffffffff] hotplug
[    0.000000] ACPI: SRAT: Node 8 PXM 15 [mem 0x470000000000-0x47ffffffffff] hotplug
[    0.000000] NUMA: NODE_DATA [mem 0x206afaaec0-0x206afaffff]
[    0.000000] NUMA: Initmem setup node 1 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x206afa5d80-0x206afaaebf]
[    0.000000] NUMA: NODE_DATA(1) on node 0
[    0.000000] NUMA: Initmem setup node 2 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x206af9aec0-0x206af9ffff]
[    0.000000] NUMA: NODE_DATA(2) on node 0
[    0.000000] NUMA: Initmem setup node 3 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x206af95d80-0x206af9aebf]
[    0.000000] NUMA: NODE_DATA(3) on node 0
[    0.000000] NUMA: Initmem setup node 4 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x206af90c40-0x206af95d7f]
[    0.000000] NUMA: NODE_DATA(4) on node 0
[    0.000000] NUMA: Initmem setup node 5 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x206af8bb00-0x206af90c3f]
[    0.000000] NUMA: NODE_DATA(5) on node 0
[    0.000000] NUMA: Initmem setup node 6 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x206af869c0-0x206af8baff]
[    0.000000] NUMA: NODE_DATA(6) on node 0
[    0.000000] NUMA: Initmem setup node 7 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x206af81880-0x206af869bf]
[    0.000000] NUMA: NODE_DATA(7) on node 0
[    0.000000] NUMA: Initmem setup node 8 [<memory-less node>]
[    0.000000] NUMA: NODE_DATA [mem 0x206af7c740-0x206af8187f]
[    0.000000] NUMA: NODE_DATA(8) on node 0
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000ffffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000206d7fffff]
[    0.000000]   Device   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x00000000fffdffff]
[    0.000000]   node   0: [mem 0x00000000fffe0000-0x00000000ffffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x000000202b65ffff]
[    0.000000]   node   0: [mem 0x000000202b660000-0x000000202b7cffff]
[    0.000000]   node   0: [mem 0x000000202b7d0000-0x0000002042c9ffff]
[    0.000000]   node   0: [mem 0x0000002042ca0000-0x000000204a7bffff]
[    0.000000]   node   0: [mem 0x000000204a7c0000-0x000000204a82ffff]
[    0.000000]   node   0: [mem 0x000000204a830000-0x000000204a87ffff]
[    0.000000]   node   0: [mem 0x000000204a880000-0x000000204aeaffff]
[    0.000000]   node   0: [mem 0x000000204aeb0000-0x000000204af4ffff]
[    0.000000]   node   0: [mem 0x000000204af50000-0x000000204af6ffff]
[    0.000000]   node   0: [mem 0x000000204af70000-0x000000204afbffff]
[    0.000000]   node   0: [mem 0x000000204afc0000-0x000000204b32ffff]
[    0.000000]   node   0: [mem 0x000000204b330000-0x000000204b3cffff]
[    0.000000]   node   0: [mem 0x000000204b3d0000-0x000000204b4effff]
[    0.000000]   node   0: [mem 0x000000204b4f0000-0x000000204b5effff]
[    0.000000]   node   0: [mem 0x000000204b5f0000-0x000000204b63ffff]
[    0.000000]   node   0: [mem 0x000000204b640000-0x000000204b83ffff]
[    0.000000]   node   0: [mem 0x000000204b840000-0x000000204b84ffff]
[    0.000000]   node   0: [mem 0x000000204b850000-0x000000204b8effff]
[    0.000000]   node   0: [mem 0x000000204b8f0000-0x000000204b8fffff]
[    0.000000]   node   0: [mem 0x000000204b900000-0x000000204baeffff]
[    0.000000]   node   0: [mem 0x000000204baf0000-0x0000002068dbffff]
[    0.000000]   node   0: [mem 0x0000002068dc0000-0x0000002068dfffff]
[    0.000000]   node   0: [mem 0x0000002068e00000-0x0000002068e1ffff]
[    0.000000]   node   0: [mem 0x0000002068e20000-0x0000002068e5ffff]
[    0.000000]   node   0: [mem 0x0000002068e60000-0x000000206bffffff]
[    0.000000]   node   0: [mem 0x000000206d400000-0x000000206d7fffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000206d7fffff]
[    0.000000] Initmem setup node 1 as memoryless
[    0.000000] Initmem setup node 2 as memoryless
[    0.000000] Initmem setup node 3 as memoryless
[    0.000000] Initmem setup node 4 as memoryless
[    0.000000] Initmem setup node 5 as memoryless
[    0.000000] Initmem setup node 6 as memoryless
[    0.000000] Initmem setup node 7 as memoryless
[    0.000000] Initmem setup node 8 as memoryless
[    0.000000] On node 0, zone Normal: 320 pages in unavailable ranges
[    0.000000] On node 0, zone Normal: 4736 pages in unavailable ranges
[    0.000000] crashkernel reserved: 0x0000001f2b600000 - 0x000000202b600000 (4096 MB)
[    0.000000] psci: probing for conduit method from ACPI.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.2
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x0 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x30000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x40000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x50000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x60000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x70000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x80000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x90000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xa0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xb0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xc0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0xe0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x100000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x110000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x120000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x130000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x140000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x150000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x160000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x170000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x180000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x190000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1a0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1c0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1d0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1e0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x1f0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x200000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x210000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x220000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x230000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x240000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x250000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x260000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x270000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x280000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x290000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2a0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2b0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2c0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2d0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2e0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x2f0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x300000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x310000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x320000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x330000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x340000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x350000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x360000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x370000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x380000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3a0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3b0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3c0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3d0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3e0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x3f0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x410000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x420000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x430000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x440000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x460000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x480000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x490000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x4a0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x4b0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x4c0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x4d0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x4e0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x4f0000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x500000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x510000 -> Node 0
[    0.000000] ACPI: NUMA: SRAT: PXM 0 -> MPIDR 0x520000 -> Node 0
[    0.000000] percpu: Embedded 2 pages/cpu s42728 r8192 d80152 u131072
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: Virtualization Host Extensions
[    0.000000] CPU features: detected: Hardware dirty bit management
[    0.000000] CPU features: detected: Spectre-v4
[    0.000000] alternatives: patching kernel code
[    0.000000] Fallback order for Node 0: 0
[    0.000000] Fallback order for Node 1: 1 0
[    0.000000] Fallback order for Node 2: 2 0
[    0.000000] Fallback order for Node 3: 3 0
[    0.000000] Fallback order for Node 4: 4 0
[    0.000000] Fallback order for Node 5: 5 0
[    0.000000] Fallback order for Node 6: 6 0
[    0.000000] Fallback order for Node 7: 7 0
[    0.000000] Fallback order for Node 8: 8 0
[    0.000000] Built 9 zonelists, mobility grouping on.  Total pages: 2090052
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=(loop)/boot/Image earlycon maxcpus=72 crashkernel=4096M,high default_hugepagesz=2M nokaslr cma=0 kvm-arm.vgic_v4_enable=1
[    0.000000] Unknown kernel command line parameters "nokaslr BOOT_IMAGE=(loop)/boot/Image", will be passed to user space.
[    0.000000] printk: log_buf_len individual max cpu contribution: 4096 bytes
[    0.000000] printk: log_buf_len total cpu_extra contributions: 299008 bytes
[    0.000000] printk: log_buf_len min size: 131072 bytes
[    0.000000] printk: log_buf_len: 524288 bytes
[    0.000000] printk: early log buf free: 115304(87%)
[    0.000000] Dentry cache hash table entries: 8388608 (order: 10, 67108864 bytes, linear)
[    0.000000] Inode-cache hash table entries: 4194304 (order: 9, 33554432 bytes, linear)
[    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
[    0.000000] software IO TLB: area num 128.
[    0.000000] software IO TLB: mapped [mem 0x00000000fbfe0000-0x00000000fffe0000] (64MB)
[    0.000000] Memory: 129057280K/133894144K available (11456K kernel code, 1332K rwdata, 3776K rodata, 2240K init, 1490K bss, 4836864K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=74, Nodes=9
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU event tracing is enabled.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=74.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=74
[    0.000000] NR_IRQS: 524288, nr_irqs: 524288, preallocated irqs: 0
[    0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[    0.000000] GICv3: 960 SPIs implemented
[    0.000000] GICv3: 320 Extended SPIs implemented
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI
[    0.000000] GICv3: GICv4 features: DirectLPI RVPEID Valid+Dirty
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000022080000
[    0.000000] SRAT: PXM 0 -> ITS 0 -> Node 0
[    0.000000] ITS [mem 0x22040000-0x2205ffff]
[    0.000000] ITS@0x0000000022040000: Single VMOVP capable
[    0.000000] ITS@0x0000000022040000: Using GICv4.1 mode 00000000 00000001
[    0.000000] ITS@0x0000000022040000: allocated 8192 Devices @100550000 (indirect, esz 8, psz 64K, shr 1)
[    0.000000] ITS@0x0000000022040000: allocated 32768 Interrupt Collections @100560000 (flat, esz 2, psz 64K, shr 1)
[    0.000000] ITS@0x0000000022040000: allocated 16384 Virtual CPUs @100580000 (flat, esz 8, psz 64K, shr 1)
[    0.000000] GICv3: using LPI property table @0x0000000100570000
[    0.000000] ITS: Using DirectLPI for VPE invalidation
[    0.000000] ITS: Enabling GICv4 support
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x00000001005b0000
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] kfence: initialized - using 33554432 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)
[    0.000000] arch_timer: cp15 timer(s) running at 1000.00MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns
[    0.008674] Console: colour dummy device 80x25
[    0.013430] mempolicy: Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl
[    0.025369] ACPI: Core revision 20220331
[    0.029638] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)
[    0.040927] pid_max: default: 75776 minimum: 592
[    0.045853] LSM: Security Framework initializing
[    0.050805] Mount-cache hash table entries: 131072 (order: 4, 1048576 bytes, linear)
[    0.059065] Mountpoint-cache hash table entries: 131072 (order: 4, 1048576 bytes, linear)
[    0.068227] cblist_init_generic: Setting adjustable number of callback queues.
[    0.075930] cblist_init_generic: Setting shift to 7 and lim to 1.
[    0.082467] rcu: Hierarchical SRCU implementation.
[    0.087558] rcu:     Max phase no-delay instances is 1000.
[    0.093199] Platform MSI: ITS@0x22040000 domain created
[    0.098756] PCI/MSI: ITS@0x22040000 domain created
[    0.103851] Remapping and enabling EFI services.
[    0.109309] smp: Bringing up secondary CPUs ...
[    0.121815] Detected PIPT I-cache on CPU1
[    0.121831] GICv3: CPU1: found redistributor 30000 region 0:0x0000000022140000
[    0.121838] GICv3: CPU1: using allocated LPI pending table @0x00000001005c0000
[    0.121850] CPU1: Booted secondary processor 0x0000030000 [0x410fd4f0]
[    0.129578] Detected PIPT I-cache on CPU2
[    0.129593] GICv3: CPU2: found redistributor 40000 region 0:0x0000000022180000
[    0.129599] GICv3: CPU2: using allocated LPI pending table @0x00000001005d0000
[    0.129611] CPU2: Booted secondary processor 0x0000040000 [0x410fd4f0]
[    0.137333] Detected PIPT I-cache on CPU3
[    0.137347] GICv3: CPU3: found redistributor 50000 region 0:0x00000000221c0000
[    0.137354] GICv3: CPU3: using allocated LPI pending table @0x00000001005e0000
[    0.137366] CPU3: Booted secondary processor 0x0000050000 [0x410fd4f0]
[    0.145051] Detected PIPT I-cache on CPU4
[    0.145064] GICv3: CPU4: found redistributor 60000 region 0:0x0000000022200000
[    0.145071] GICv3: CPU4: using allocated LPI pending table @0x00000001005f0000
[    0.145081] CPU4: Booted secondary processor 0x0000060000 [0x410fd4f0]
[    0.152764] Detected PIPT I-cache on CPU5
[    0.152777] GICv3: CPU5: found redistributor 70000 region 0:0x0000000022240000
[    0.152783] GICv3: CPU5: using allocated LPI pending table @0x0000000100600000
[    0.152794] CPU5: Booted secondary processor 0x0000070000 [0x410fd4f0]
[    0.160528] Detected PIPT I-cache on CPU6
[    0.160543] GICv3: CPU6: found redistributor 80000 region 0:0x0000000022280000
[    0.160550] GICv3: CPU6: using allocated LPI pending table @0x0000000100610000
[    0.160560] CPU6: Booted secondary processor 0x0000080000 [0x410fd4f0]
[    0.168297] Detected PIPT I-cache on CPU7
[    0.168311] GICv3: CPU7: found redistributor 90000 region 0:0x00000000222c0000
[    0.168318] GICv3: CPU7: using allocated LPI pending table @0x0000000100620000
[    0.168329] CPU7: Booted secondary processor 0x0000090000 [0x410fd4f0]
[    0.176070] Detected PIPT I-cache on CPU8
[    0.176084] GICv3: CPU8: found redistributor a0000 region 0:0x0000000022300000
[    0.176091] GICv3: CPU8: using allocated LPI pending table @0x0000000100630000
[    0.176102] CPU8: Booted secondary processor 0x00000a0000 [0x410fd4f0]
[    0.183845] Detected PIPT I-cache on CPU9
[    0.183861] GICv3: CPU9: found redistributor b0000 region 0:0x0000000022340000
[    0.183867] GICv3: CPU9: using allocated LPI pending table @0x0000000100640000
[    0.183879] CPU9: Booted secondary processor 0x00000b0000 [0x410fd4f0]
[    0.191667] Detected PIPT I-cache on CPU10
[    0.191684] GICv3: CPU10: found redistributor c0000 region 0:0x0000000022380000
[    0.191691] GICv3: CPU10: using allocated LPI pending table @0x0000000100650000
[    0.191702] CPU10: Booted secondary processor 0x00000c0000 [0x410fd4f0]
[    0.199395] Detected PIPT I-cache on CPU11
[    0.199443] GICv3: CPU11: found redistributor e0000 region 0:0x0000000022400000
[    0.199450] GICv3: CPU11: using allocated LPI pending table @0x0000000100660000
[    0.199461] CPU11: Booted secondary processor 0x00000e0000 [0x410fd4f0]
[    0.207159] Detected PIPT I-cache on CPU12
[    0.207176] GICv3: CPU12: found redistributor 100000 region 0:0x0000000022480000
[    0.207183] GICv3: CPU12: using allocated LPI pending table @0x0000000100670000
[    0.207194] CPU12: Booted secondary processor 0x0000100000 [0x410fd4f0]
[    0.214887] Detected PIPT I-cache on CPU13
[    0.214904] GICv3: CPU13: found redistributor 110000 region 0:0x00000000224c0000
[    0.214911] GICv3: CPU13: using allocated LPI pending table @0x0000000100680000
[    0.214922] CPU13: Booted secondary processor 0x0000110000 [0x410fd4f0]
[    0.222614] Detected PIPT I-cache on CPU14
[    0.222633] GICv3: CPU14: found redistributor 120000 region 0:0x0000000022500000
[    0.222640] GICv3: CPU14: using allocated LPI pending table @0x0000000100690000
[    0.222652] CPU14: Booted secondary processor 0x0000120000 [0x410fd4f0]
[    0.230339] Detected PIPT I-cache on CPU15
[    0.230358] GICv3: CPU15: found redistributor 130000 region 0:0x0000000022540000
[    0.230365] GICv3: CPU15: using allocated LPI pending table @0x00000001006a0000
[    0.230377] CPU15: Booted secondary processor 0x0000130000 [0x410fd4f0]
[    0.238060] Detected PIPT I-cache on CPU16
[    0.238079] GICv3: CPU16: found redistributor 140000 region 0:0x0000000022580000
[    0.238087] GICv3: CPU16: using allocated LPI pending table @0x00000001006b0000
[    0.238099] CPU16: Booted secondary processor 0x0000140000 [0x410fd4f0]
[    0.245789] Detected PIPT I-cache on CPU17
[    0.245808] GICv3: CPU17: found redistributor 150000 region 0:0x00000000225c0000
[    0.245815] GICv3: CPU17: using allocated LPI pending table @0x00000001006c0000
[    0.245827] CPU17: Booted secondary processor 0x0000150000 [0x410fd4f0]
[    0.253538] Detected PIPT I-cache on CPU18
[    0.253558] GICv3: CPU18: found redistributor 160000 region 0:0x0000000022600000
[    0.253565] GICv3: CPU18: using allocated LPI pending table @0x00000001006d0000
[    0.253577] CPU18: Booted secondary processor 0x0000160000 [0x410fd4f0]
[    0.261290] Detected PIPT I-cache on CPU19
[    0.261311] GICv3: CPU19: found redistributor 170000 region 0:0x0000000022640000
[    0.261319] GICv3: CPU19: using allocated LPI pending table @0x00000001006e0000
[    0.261331] CPU19: Booted secondary processor 0x0000170000 [0x410fd4f0]
[    0.269083] Detected PIPT I-cache on CPU20
[    0.269105] GICv3: CPU20: found redistributor 180000 region 0:0x0000000022680000
[    0.269112] GICv3: CPU20: using allocated LPI pending table @0x00000001006f0000
[    0.269125] CPU20: Booted secondary processor 0x0000180000 [0x410fd4f0]
[    0.276875] Detected PIPT I-cache on CPU21
[    0.276898] GICv3: CPU21: found redistributor 190000 region 0:0x00000000226c0000
[    0.276906] GICv3: CPU21: using allocated LPI pending table @0x0000000100700000
[    0.276918] CPU21: Booted secondary processor 0x0000190000 [0x410fd4f0]
[    0.284678] Detected PIPT I-cache on CPU22
[    0.284701] GICv3: CPU22: found redistributor 1a0000 region 0:0x0000000022700000
[    0.284708] GICv3: CPU22: using allocated LPI pending table @0x0000000100710000
[    0.284721] CPU22: Booted secondary processor 0x00001a0000 [0x410fd4f0]
[    0.292424] Detected PIPT I-cache on CPU23
[    0.292448] GICv3: CPU23: found redistributor 1c0000 region 0:0x0000000022780000
[    0.292456] GICv3: CPU23: using allocated LPI pending table @0x0000000100720000
[    0.292468] CPU23: Booted secondary processor 0x00001c0000 [0x410fd4f0]
[    0.300161] Detected PIPT I-cache on CPU24
[    0.300217] GICv3: CPU24: found redistributor 1d0000 region 0:0x00000000227c0000
[    0.300225] GICv3: CPU24: using allocated LPI pending table @0x0000000100730000
[    0.300237] CPU24: Booted secondary processor 0x00001d0000 [0x410fd4f0]
[    0.307923] Detected PIPT I-cache on CPU25
[    0.307947] GICv3: CPU25: found redistributor 1e0000 region 0:0x0000000022800000
[    0.307954] GICv3: CPU25: using allocated LPI pending table @0x0000000100740000
[    0.307965] CPU25: Booted secondary processor 0x00001e0000 [0x410fd4f0]
[    0.315653] Detected PIPT I-cache on CPU26
[    0.315677] GICv3: CPU26: found redistributor 1f0000 region 0:0x0000000022840000
[    0.315685] GICv3: CPU26: using allocated LPI pending table @0x0000000100750000
[    0.315696] CPU26: Booted secondary processor 0x00001f0000 [0x410fd4f0]
[    0.323390] Detected PIPT I-cache on CPU27
[    0.323416] GICv3: CPU27: found redistributor 200000 region 0:0x0000000022880000
[    0.323424] GICv3: CPU27: using allocated LPI pending table @0x0000000100760000
[    0.323435] CPU27: Booted secondary processor 0x0000200000 [0x410fd4f0]
[    0.331129] Detected PIPT I-cache on CPU28
[    0.331155] GICv3: CPU28: found redistributor 210000 region 0:0x00000000228c0000
[    0.331163] GICv3: CPU28: using allocated LPI pending table @0x0000000100770000
[    0.331175] CPU28: Booted secondary processor 0x0000210000 [0x410fd4f0]
[    0.338838] Detected PIPT I-cache on CPU29
[    0.338865] GICv3: CPU29: found redistributor 220000 region 0:0x0000000022900000
[    0.338873] GICv3: CPU29: using allocated LPI pending table @0x0000000100780000
[    0.338885] CPU29: Booted secondary processor 0x0000220000 [0x410fd4f0]
[    0.346547] Detected PIPT I-cache on CPU30
[    0.346575] GICv3: CPU30: found redistributor 230000 region 0:0x0000000022940000
[    0.346583] GICv3: CPU30: using allocated LPI pending table @0x0000000100790000
[    0.346595] CPU30: Booted secondary processor 0x0000230000 [0x410fd4f0]
[    0.354316] Detected PIPT I-cache on CPU31
[    0.354345] GICv3: CPU31: found redistributor 240000 region 0:0x0000000022980000
[    0.354353] GICv3: CPU31: using allocated LPI pending table @0x00000001007a0000
[    0.354365] CPU31: Booted secondary processor 0x0000240000 [0x410fd4f0]
[    0.362091] Detected PIPT I-cache on CPU32
[    0.362121] GICv3: CPU32: found redistributor 250000 region 0:0x00000000229c0000
[    0.362129] GICv3: CPU32: using allocated LPI pending table @0x00000001007b0000
[    0.362142] CPU32: Booted secondary processor 0x0000250000 [0x410fd4f0]
[    0.369875] Detected PIPT I-cache on CPU33
[    0.369905] GICv3: CPU33: found redistributor 260000 region 0:0x0000000022a00000
[    0.369914] GICv3: CPU33: using allocated LPI pending table @0x00000001007c0000
[    0.369926] CPU33: Booted secondary processor 0x0000260000 [0x410fd4f0]
[    0.377658] Detected PIPT I-cache on CPU34
[    0.377688] GICv3: CPU34: found redistributor 270000 region 0:0x0000000022a40000
[    0.377696] GICv3: CPU34: using allocated LPI pending table @0x00000001007d0000
[    0.377709] CPU34: Booted secondary processor 0x0000270000 [0x410fd4f0]
[    0.385463] Detected PIPT I-cache on CPU35
[    0.385496] GICv3: CPU35: found redistributor 280000 region 0:0x0000000022a80000
[    0.385505] GICv3: CPU35: using allocated LPI pending table @0x00000001007e0000
[    0.385518] CPU35: Booted secondary processor 0x0000280000 [0x410fd4f0]
[    0.393273] Detected PIPT I-cache on CPU36
[    0.393306] GICv3: CPU36: found redistributor 290000 region 0:0x0000000022ac0000
[    0.393315] GICv3: CPU36: using allocated LPI pending table @0x00000001007f0000
[    0.393328] CPU36: Booted secondary processor 0x0000290000 [0x410fd4f0]
[    0.401007] Detected PIPT I-cache on CPU37
[    0.401039] GICv3: CPU37: found redistributor 2a0000 region 0:0x0000000022b00000
[    0.401049] GICv3: CPU37: using allocated LPI pending table @0x0000000100800000
[    0.401062] CPU37: Booted secondary processor 0x00002a0000 [0x410fd4f0]
[    0.408745] Detected PIPT I-cache on CPU38
[    0.408807] GICv3: CPU38: found redistributor 2b0000 region 0:0x0000000022b40000
[    0.408817] GICv3: CPU38: using allocated LPI pending table @0x0000000100810000
[    0.408830] CPU38: Booted secondary processor 0x00002b0000 [0x410fd4f0]
[    0.416501] Detected PIPT I-cache on CPU39
[    0.416532] GICv3: CPU39: found redistributor 2c0000 region 0:0x0000000022b80000
[    0.416540] GICv3: CPU39: using allocated LPI pending table @0x0000000100820000
[    0.416553] CPU39: Booted secondary processor 0x00002c0000 [0x410fd4f0]
[    0.424213] Detected PIPT I-cache on CPU40
[    0.424242] GICv3: CPU40: found redistributor 2d0000 region 0:0x0000000022bc0000
[    0.424251] GICv3: CPU40: using allocated LPI pending table @0x0000000100830000
[    0.424264] CPU40: Booted secondary processor 0x00002d0000 [0x410fd4f0]
[    0.431958] Detected PIPT I-cache on CPU41
[    0.431992] GICv3: CPU41: found redistributor 2e0000 region 0:0x0000000022c00000
[    0.432004] GICv3: CPU41: using allocated LPI pending table @0x0000000100840000
[    0.432019] CPU41: Booted secondary processor 0x00002e0000 [0x410fd4f0]
[    0.439711] Detected PIPT I-cache on CPU42
[    0.439743] GICv3: CPU42: found redistributor 2f0000 region 0:0x0000000022c40000
[    0.439755] GICv3: CPU42: using allocated LPI pending table @0x0000000100850000
[    0.439770] CPU42: Booted secondary processor 0x00002f0000 [0x410fd4f0]
[    0.447462] Detected PIPT I-cache on CPU43
[    0.447496] GICv3: CPU43: found redistributor 300000 region 0:0x0000000022c80000
[    0.447512] GICv3: CPU43: using allocated LPI pending table @0x0000000100860000
[    0.447529] CPU43: Booted secondary processor 0x0000300000 [0x410fd4f0]
[    0.455224] Detected PIPT I-cache on CPU44
[    0.455261] GICv3: CPU44: found redistributor 310000 region 0:0x0000000022cc0000
[    0.455276] GICv3: CPU44: using allocated LPI pending table @0x0000000100870000
[    0.455294] CPU44: Booted secondary processor 0x0000310000 [0x410fd4f0]
[    0.463028] Detected PIPT I-cache on CPU45
[    0.463065] GICv3: CPU45: found redistributor 320000 region 0:0x0000000022d00000
[    0.463084] GICv3: CPU45: using allocated LPI pending table @0x0000000100880000
[    0.463104] CPU45: Booted secondary processor 0x0000320000 [0x410fd4f0]
[    0.470827] Detected PIPT I-cache on CPU46
[    0.470866] GICv3: CPU46: found redistributor 330000 region 0:0x0000000022d40000
[    0.470885] GICv3: CPU46: using allocated LPI pending table @0x0000000100890000
[    0.470905] CPU46: Booted secondary processor 0x0000330000 [0x410fd4f0]
[    0.478807] Detected PIPT I-cache on CPU47
[    0.478847] GICv3: CPU47: found redistributor 340000 region 0:0x0000000022d80000
[    0.478906] GICv3: CPU47: using allocated LPI pending table @0x00000001008a0000
[    0.478954] CPU47: Booted secondary processor 0x0000340000 [0x410fd4f0]
[    0.486866] Detected PIPT I-cache on CPU48
[    0.486903] GICv3: CPU48: found redistributor 350000 region 0:0x0000000022dc0000
[    0.486962] GICv3: CPU48: using allocated LPI pending table @0x00000001008b0000
[    0.487010] CPU48: Booted secondary processor 0x0000350000 [0x410fd4f0]
[    0.495014] Detected PIPT I-cache on CPU49
[    0.495054] GICv3: CPU49: found redistributor 360000 region 0:0x0000000022e00000
[    0.495128] GICv3: CPU49: using allocated LPI pending table @0x00000001008c0000
[    0.495186] CPU49: Booted secondary processor 0x0000360000 [0x410fd4f0]
[    0.503195] Detected PIPT I-cache on CPU50
[    0.503237] GICv3: CPU50: found redistributor 370000 region 0:0x0000000022e40000
[    0.503310] GICv3: CPU50: using allocated LPI pending table @0x00000001008d0000
[    0.503368] CPU50: Booted secondary processor 0x0000370000 [0x410fd4f0]
[    0.511033] Detected PIPT I-cache on CPU51
[    0.511104] GICv3: CPU51: found redistributor 380000 region 0:0x0000000022e80000
[    0.511116] GICv3: CPU51: using allocated LPI pending table @0x00000001008e0000
[    0.511131] CPU51: Booted secondary processor 0x0000380000 [0x410fd4f0]
[    0.518783] Detected PIPT I-cache on CPU52
[    0.518821] GICv3: CPU52: found redistributor 3a0000 region 0:0x0000000022f00000
[    0.518834] GICv3: CPU52: using allocated LPI pending table @0x00000001008f0000
[    0.518849] CPU52: Booted secondary processor 0x00003a0000 [0x410fd4f0]
[    0.526495] Detected PIPT I-cache on CPU53
[    0.526535] GICv3: CPU53: found redistributor 3b0000 region 0:0x0000000022f40000
[    0.526547] GICv3: CPU53: using allocated LPI pending table @0x0000000100900000
[    0.526563] CPU53: Booted secondary processor 0x00003b0000 [0x410fd4f0]
[    0.534426] Detected PIPT I-cache on CPU54
[    0.534467] GICv3: CPU54: found redistributor 3c0000 region 0:0x0000000022f80000
[    0.534535] GICv3: CPU54: using allocated LPI pending table @0x0000000100910000
[    0.534590] CPU54: Booted secondary processor 0x00003c0000 [0x410fd4f0]
[    0.542469] Detected PIPT I-cache on CPU55
[    0.542511] GICv3: CPU55: found redistributor 3d0000 region 0:0x0000000022fc0000
[    0.542579] GICv3: CPU55: using allocated LPI pending table @0x0000000100920000
[    0.542634] CPU55: Booted secondary processor 0x00003d0000 [0x410fd4f0]
[    0.550730] Detected PIPT I-cache on CPU56
[    0.550774] GICv3: CPU56: found redistributor 3e0000 region 0:0x0000000023000000
[    0.550909] GICv3: CPU56: using allocated LPI pending table @0x0000000100930000
[    0.551010] CPU56: Booted secondary processor 0x00003e0000 [0x410fd4f0]
[    0.559114] Detected PIPT I-cache on CPU57
[    0.559157] GICv3: CPU57: found redistributor 3f0000 region 0:0x0000000023040000
[    0.559292] GICv3: CPU57: using allocated LPI pending table @0x0000000100940000
[    0.559394] CPU57: Booted secondary processor 0x00003f0000 [0x410fd4f0]
[    0.568028] Detected PIPT I-cache on CPU58
[    0.568075] GICv3: CPU58: found redistributor 410000 region 0:0x00000000230c0000
[    0.568343] GICv3: CPU58: using allocated LPI pending table @0x0000000100950000
[    0.568538] CPU58: Booted secondary processor 0x0000410000 [0x410fd4f0]
[    0.578229] Detected PIPT I-cache on CPU59
[    0.578273] GICv3: CPU59: found redistributor 420000 region 0:0x0000000023100000
[    0.578806] GICv3: CPU59: using allocated LPI pending table @0x0000000100960000
[    0.579188] CPU59: Booted secondary processor 0x0000420000 [0x410fd4f0]
[    0.588938] Detected PIPT I-cache on CPU60
[    0.588987] GICv3: CPU60: found redistributor 430000 region 0:0x0000000023140000
[    0.589521] GICv3: CPU60: using allocated LPI pending table @0x0000000100970000
[    0.589903] CPU60: Booted secondary processor 0x0000430000 [0x410fd4f0]
[    0.600182] Detected PIPT I-cache on CPU61
[    0.600231] GICv3: CPU61: found redistributor 440000 region 0:0x0000000023180000
[    0.600907] GICv3: CPU61: using allocated LPI pending table @0x0000000100980000
[    0.601388] CPU61: Booted secondary processor 0x0000440000 [0x410fd4f0]
[    0.609624] Detected PIPT I-cache on CPU62
[    0.609673] GICv3: CPU62: found redistributor 460000 region 0:0x0000000023200000
[    0.609792] GICv3: CPU62: using allocated LPI pending table @0x0000000100990000
[    0.609883] CPU62: Booted secondary processor 0x0000460000 [0x410fd4f0]
[    0.617894] Detected PIPT I-cache on CPU63
[    0.617974] GICv3: CPU63: found redistributor 480000 region 0:0x0000000023280000
[    0.618065] GICv3: CPU63: using allocated LPI pending table @0x00000001009a0000
[    0.618137] CPU63: Booted secondary processor 0x0000480000 [0x410fd4f0]
[    0.626153] Detected PIPT I-cache on CPU64
[    0.626201] GICv3: CPU64: found redistributor 490000 region 0:0x00000000232c0000
[    0.626295] GICv3: CPU64: using allocated LPI pending table @0x00000001009b0000
[    0.626367] CPU64: Booted secondary processor 0x0000490000 [0x410fd4f0]
[    0.634756] Detected PIPT I-cache on CPU65
[    0.634805] GICv3: CPU65: found redistributor 4a0000 region 0:0x0000000023300000
[    0.634994] GICv3: CPU65: using allocated LPI pending table @0x00000001009c0000
[    0.635136] CPU65: Booted secondary processor 0x00004a0000 [0x410fd4f0]
[    0.643543] Detected PIPT I-cache on CPU66
[    0.643595] GICv3: CPU66: found redistributor 4b0000 region 0:0x0000000023340000
[    0.643785] GICv3: CPU66: using allocated LPI pending table @0x00000001009d0000
[    0.643926] CPU66: Booted secondary processor 0x00004b0000 [0x410fd4f0]
[    0.653054] Detected PIPT I-cache on CPU67
[    0.653107] GICv3: CPU67: found redistributor 4c0000 region 0:0x0000000023380000
[    0.653496] GICv3: CPU67: using allocated LPI pending table @0x00000001009e0000
[    0.653776] CPU67: Booted secondary processor 0x00004c0000 [0x410fd4f0]
[    0.662940] Detected PIPT I-cache on CPU68
[    0.662993] GICv3: CPU68: found redistributor 4d0000 region 0:0x00000000233c0000
[    0.663380] GICv3: CPU68: using allocated LPI pending table @0x00000001009f0000
[    0.663661] CPU68: Booted secondary processor 0x00004d0000 [0x410fd4f0]
[    0.673427] Detected PIPT I-cache on CPU69
[    0.673482] GICv3: CPU69: found redistributor 4e0000 region 0:0x0000000023400000
[    0.674017] GICv3: CPU69: using allocated LPI pending table @0x0000000100a00000
[    0.674397] CPU69: Booted secondary processor 0x00004e0000 [0x410fd4f0]
[    0.684194] Detected PIPT I-cache on CPU70
[    0.684247] GICv3: CPU70: found redistributor 4f0000 region 0:0x0000000023440000
[    0.684781] GICv3: CPU70: using allocated LPI pending table @0x0000000100a10000
[    0.685164] CPU70: Booted secondary processor 0x00004f0000 [0x410fd4f0]
[    0.693790] Detected PIPT I-cache on CPU71
[    0.693846] GICv3: CPU71: found redistributor 500000 region 0:0x0000000023480000
[    0.694063] GICv3: CPU71: using allocated LPI pending table @0x0000000100a20000
[    0.694225] CPU71: Booted secondary processor 0x0000500000 [0x410fd4f0]
[    0.694327] smp: Brought up 9 nodes, 72 CPUs
[    2.613053] SMP: Total of 72 processors activated.
[    2.618145] CPU features: detected: Branch Target Identification
[    2.624533] CPU features: detected: ARMv8.4 Translation Table Level
[    2.631197] CPU features: detected: Instruction cache invalidation not required for I/D coherence
[    2.640640] CPU features: detected: Data cache clean to the PoU not required for I/D coherence
[    2.649804] CPU features: detected: Common not Private translations
[    2.656469] CPU features: detected: CRC32 instructions
[    2.661930] CPU features: detected: Data cache clean to Point of Deep Persistence
[    2.669891] CPU features: detected: Data cache clean to Point of Persistence
[    2.677389] CPU features: detected: E0PD
[    2.681556] CPU features: detected: Enhanced Privileged Access Never
[    2.688314] CPU features: detected: Generic authentication (architected QARMA5 algorithm)
[    2.697015] CPU features: detected: RCpc load-acquire (LDAPR)
[    2.703125] CPU features: detected: LSE atomic instructions
[    2.709049] CPU features: detected: Privileged Access Never
[    2.714973] CPU features: detected: RAS Extension Support
[    2.720713] CPU features: detected: Speculation barrier (SB)
[    2.726731] CPU features: detected: Stage-2 Force Write-Back
[    2.732749] CPU features: detected: TLB range maintenance instructions
[    2.739693] CPU features: detected: Memory Partitioning And Monitoring
[    2.746635] CPU features: detected: Speculative Store Bypassing Safe (SSBS)
[    2.754039] CPU features: detected: Scalable Vector Extension
[    2.769269] CPU features: detected CPU65: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU6: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU3: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU9: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU46: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU45: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU67: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU21: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU4: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU69: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU20: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU71: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU58: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU50: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU29: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU38: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU19: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU43: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU57: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU34: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU56: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU64: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU68: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU12: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU7: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU5: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU8: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU70: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU63: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU39: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU44: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU66: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU11: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU31: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU13: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU35: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU1: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU22: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU32: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU28: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU27: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU49: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU55: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU33: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU16: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU60: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU61: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU10: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU23: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU15: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU48: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU24: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU14: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU30: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU2: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU18: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU17: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU59: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU47: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU62: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU0: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU42: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU25: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU53: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU40: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU41: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU54: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU36: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU26: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU52: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU37: Activity Monitors Unit (AMU)
[    2.769269] CPU features: detected CPU51: Activity Monitors Unit (AMU)
[    3.269001] SVE: maximum available vector length 16 bytes per vector
[    3.275763] SVE: default vector length 16 bytes per vector
[    3.282028] CPU: All CPU(s) started at EL2
[    3.287429] devtmpfs: initialized
[    3.293580] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    3.303956] futex hash table entries: 32768 (order: 5, 2097152 bytes, linear)
[    3.311812] pinctrl core: initialized pinctrl subsystem
[    3.317422] SMBIOS 3.0.0 present.
[    3.320942] DMI: NVIDIA silicon, BIOS v1.1.1-4638c6c9 01/17/2023
[    3.328771] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    3.337447] DMA: preallocated 16384 KiB GFP_KERNEL pool for atomic allocations
[    3.345309] DMA: preallocated 16384 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    3.353907] DMA: preallocated 16384 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    3.362606] thermal_sys: Registered thermal governor 'step_wise'
[    3.362606] thermal_sys: Registered thermal governor 'power_allocator'
[    3.369042] cpuidle: using governor menu
[    3.380187] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    3.388295] ASID allocator initialised with 65536 entries
[    3.394037] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    3.400906] Serial: AMBA PL011 UART driver
[    3.405726] KASLR disabled on command line
[    3.410761] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[    3.417985] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page
[    3.424651] HugeTLB: registered 16.0 GiB page size, pre-allocated 0 pages
[    3.431873] HugeTLB: 16320 KiB vmemmap can be freed for a 16.0 GiB page
[    3.438908] HugeTLB: registered 512 MiB page size, pre-allocated 0 pages
[    3.446036] HugeTLB: 448 KiB vmemmap can be freed for a 512 MiB page
[    3.453205] ACPI: Added _OSI(Module Device)
[    3.457640] ACPI: Added _OSI(Processor Device)
[    3.462351] ACPI: Added _OSI(3.0 _SCP Extensions)
[    3.467337] ACPI: Added _OSI(Processor Aggregator Device)
[    3.473063] ACPI: Added _OSI(Linux-Dell-Video)
[    3.477774] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    3.483405] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[    3.490976] ACPI: 11 ACPI AML tables successfully acquired and loaded
[    3.497938] ACPI Error: AE_NOT_FOUND, While resolving a named reference package element - \_SB_.C000.C002 (20220331/dspkginit-438)
[    3.510427] ACPI Error: AE_NOT_FOUND, While resolving a named reference package element - \_SB_.C000.C040 (20220331/dspkginit-438)
[    3.523336] ACPI: Interpreter enabled
[    3.527226] ACPI: Using GIC for interrupt routing
[    3.532233] ACPI: MCFG table detected, 6 entries
[    3.537144] ACPI: IORT: SMMU-v3[11000000] Mapped to Proximity domain 0
[    3.544115] ACPI: IORT: SMMU-v3[12000000] Mapped to Proximity domain 0
[    3.551068] ACPI: IORT: SMMU-v3[15000000] Mapped to Proximity domain 0
[    3.558025] ACPI: IORT: SMMU-v3[16000000] Mapped to Proximity domain 0
[    3.564976] ACPI: IORT: SMMU-v3[5000000] Mapped to Proximity domain 0
[    3.571951] HEST: Table parsing has been initialized.
[    3.577341] sdei: SDEIv1.0 (0x0) detected in firmware.
[    3.584184] ghes_edac: This system has a very crappy BIOS: It doesn't even list the DIMMS.
[    3.592978] ghes_edac: Its SMBIOS info is wrong. It is doubtful that the error report would
[    3.601864] ghes_edac: work on such system. Use this driver with caution
[    3.609005] EDAC MC0: Giving out device to module ghes_edac.c controller ghes_edac: DEV ghes (INTERRUPT)
[    3.619202] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2561
[    3.628273] {1}[Hardware Error]: It has been corrected by h/w and requires no further action
[    3.637253] {1}[Hardware Error]: event severity: corrected
[    3.643086] {1}[Hardware Error]:  Error 0, type: corrected
[    3.648918] {1}[Hardware Error]:   section type: unknown, 6d5244f2-2712-11ec-bea7-cb3fdb95c786
[    3.658082] {1}[Hardware Error]:   section length: 0x30
[    3.663638] {1}[Hardware Error]:   00000000: 5f4d5348 455f5753 00005252 00000000  HSM_SW_ERR......
[    3.673170] {1}[Hardware Error]:   00000010: 00000000 00010000 0d000000 00000000  ................
[    3.682703] {1}[Hardware Error]:   00000020: 0d650010 00000000 c0e041d5 00000000  ..e......A......
[    3.693180] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC.
[    3.703614] ARMH0011:00: ttyAMA0 at MMIO 0xc280000 (irq = 34, base_baud = 0) is a SBSA
[    3.712045] printk: console [ttyAMA0] enabled
[    3.712045] printk: console [ttyAMA0] enabled
[    3.721123] printk: bootconsole [pl11] disabled
[    3.721123] printk: bootconsole [pl11] disabled
[    3.730716] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    3.737039] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    3.746690] acpi PNP0A08:00: _OSC: platform does not support [PME AER DPC]
[    3.753768] acpi PNP0A08:00: _OSC: OS now controls [PCIeCapability LTR]
[    3.760637] acpi PNP0A08:00: ECAM area [mem 0x600010000000-0x60001fffffff] reserved by PNP0C02:01
[    3.769784] acpi PNP0A08:00: ECAM at [mem 0x600010000000-0x60001fffffff] for [bus 00-ff]
[    3.778063] ACPI: Remapped I/O 0x0000600020000000 to [io  0x0000-0xffff window]
[    3.785561] PCI host bridge to bus 0000:00
[    3.789746] pci_bus 0000:00: root bus resource [mem 0x600040000000-0x6000bfffffff window] (bus address [0x40000000-0xbfffffff])
[    3.801485] pci_bus 0000:00: root bus resource [mem 0x6000c0000000-0x607fffffffff window]
[    3.809847] pci_bus 0000:00: root bus resource [io  0x0000-0xffff window]
[    3.816787] pci_bus 0000:00: root bus resource [bus 00-ff]
[    3.822407] pci 0000:00:00.0: [10de:22b2] type 01 class 0x060400
[    3.828634] pci 0000:00:00.0: PME# supported from D0 D3hot
[    3.834421] pci 0000:00:00.0: bridge window [io  0x1000-0x0fff] to [bus 01] add_size 1000
[    3.842816] pci 0000:00:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01] add_size 200000 add_align 100000
[    3.854556] pci 0000:00:00.0: bridge window [mem 0x00100000-0x000fffff] to [bus 01] add_size 200000 add_align 100000
[    3.865321] pci 0000:00:00.0: BAR 14: assigned [mem 0x600040000000-0x6000401fffff]
[    3.873061] pci 0000:00:00.0: BAR 15: assigned [mem 0x6000c0000000-0x6000c01fffff 64bit pref]
[    3.881779] pci 0000:00:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    3.888132] pci 0000:00:00.0: PCI bridge to [bus 01]
[    3.893205] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    3.899435] pci 0000:00:00.0:   bridge window [mem 0x600040000000-0x6000401fffff]
[    3.907086] pci 0000:00:00.0:   bridge window [mem 0x6000c0000000-0x6000c01fffff 64bit pref]
[    3.915716] pci_bus 0000:00: resource 4 [mem 0x600040000000-0x6000bfffffff window]
[    3.923456] pci_bus 0000:00: resource 5 [mem 0x6000c0000000-0x607fffffffff window]
[    3.931195] pci_bus 0000:00: resource 6 [io  0x0000-0xffff window]
[    3.937513] pci_bus 0000:01: resource 0 [io  0x1000-0x1fff]
[    3.943208] pci_bus 0000:01: resource 1 [mem 0x600040000000-0x6000401fffff]
[    3.950324] pci_bus 0000:01: resource 2 [mem 0x6000c0000000-0x6000c01fffff 64bit pref]
[    3.958434] ACPI: PCI Root Bridge [PCI2] (domain 0002 [bus 00-ff])
[    3.964752] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    3.974388] acpi PNP0A08:01: _OSC: platform does not support [PME AER DPC]
[    3.981461] acpi PNP0A08:01: _OSC: OS now controls [PCIeCapability LTR]
[    3.988326] acpi PNP0A08:01: ECAM area [mem 0x610010000000-0x61001fffffff] reserved by PNP0C02:02
[    3.997467] acpi PNP0A08:01: ECAM at [mem 0x610010000000-0x61001fffffff] for [bus 00-ff]
[    4.005743] ACPI: Remapped I/O 0x0000610020000000 to [io  0x10000-0x1ffff window]
[    4.013409] PCI host bridge to bus 0002:00
[    4.017593] pci_bus 0002:00: root bus resource [mem 0x610040000000-0x6100bfffffff window] (bus address [0x40000000-0xbfffffff])
[    4.029332] pci_bus 0002:00: root bus resource [mem 0x6100c0000000-0x617fffffffff window]
[    4.037694] pci_bus 0002:00: root bus resource [io  0x10000-0x1ffff window] (bus address [0x0000-0xffff])
[    4.047511] pci_bus 0002:00: root bus resource [bus 00-ff]
[    4.053127] pci 0002:00:00.0: [10de:22b2] type 01 class 0x060400
[    4.059352] pci 0002:00:00.0: PME# supported from D0 D3hot
[    4.065114] pci 0002:00:00.0: bridge window [io  0x1000-0x0fff] to [bus 01] add_size 1000
[    4.073475] pci 0002:00:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01] add_size 200000 add_align 100000
[    4.085214] pci 0002:00:00.0: bridge window [mem 0x00100000-0x000fffff] to [bus 01] add_size 200000 add_align 100000
[    4.095977] pci 0002:00:00.0: BAR 14: assigned [mem 0x610040000000-0x6100401fffff]
[    4.103716] pci 0002:00:00.0: BAR 15: assigned [mem 0x6100c0000000-0x6100c01fffff 64bit pref]
[    4.112433] pci 0002:00:00.0: BAR 13: assigned [io  0x10000-0x10fff]
[    4.118928] pci 0002:00:00.0: PCI bridge to [bus 01]
[    4.124000] pci 0002:00:00.0:   bridge window [io  0x10000-0x10fff]
[    4.130408] pci 0002:00:00.0:   bridge window [mem 0x610040000000-0x6100401fffff]
[    4.138059] pci 0002:00:00.0:   bridge window [mem 0x6100c0000000-0x6100c01fffff 64bit pref]
[    4.146689] pci_bus 0002:00: resource 4 [mem 0x610040000000-0x6100bfffffff window]
[    4.154428] pci_bus 0002:00: resource 5 [mem 0x6100c0000000-0x617fffffffff window]
[    4.162167] pci_bus 0002:00: resource 6 [io  0x10000-0x1ffff window]
[    4.168662] pci_bus 0002:01: resource 0 [io  0x10000-0x10fff]
[    4.174535] pci_bus 0002:01: resource 1 [mem 0x610040000000-0x6100401fffff]
[    4.181651] pci_bus 0002:01: resource 2 [mem 0x6100c0000000-0x6100c01fffff 64bit pref]
[    4.189756] ACPI: PCI Root Bridge [PCI4] (domain 0004 [bus 00-ff])
[    4.196074] acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    4.205708] acpi PNP0A08:02: _OSC: platform does not support [PME AER DPC]
[    4.212780] acpi PNP0A08:02: _OSC: OS now controls [PCIeCapability LTR]
[    4.219650] acpi PNP0A08:02: ECAM area [mem 0x620010000000-0x62001fffffff] reserved by PNP0C02:03
[    4.228791] acpi PNP0A08:02: ECAM at [mem 0x620010000000-0x62001fffffff] for [bus 00-ff]
[    4.237068] ACPI: Remapped I/O 0x0000620020000000 to [io  0x20000-0x2ffff window]
[    4.244763] PCI host bridge to bus 0004:00
[    4.248947] pci_bus 0004:00: root bus resource [mem 0x620040000000-0x6200bfffffff window] (bus address [0x40000000-0xbfffffff])
[    4.260686] pci_bus 0004:00: root bus resource [mem 0x6200c0000000-0x627fffffffff window]
[    4.269047] pci_bus 0004:00: root bus resource [io  0x20000-0x2ffff window] (bus address [0x0000-0xffff])
[    4.278830] pci_bus 0004:00: root bus resource [bus 00-ff]
[    4.284446] pci 0004:00:00.0: [10de:22b2] type 01 class 0x060400
[    4.290663] pci 0004:00:00.0: PME# supported from D0 D3hot
[    4.296414] pci 0004:00:00.0: bridge window [io  0x1000-0x0fff] to [bus 01] add_size 1000
[    4.304776] pci 0004:00:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01] add_size 200000 add_align 100000
[    4.316517] pci 0004:00:00.0: bridge window [mem 0x00100000-0x000fffff] to [bus 01] add_size 200000 add_align 100000
[    4.327279] pci 0004:00:00.0: BAR 14: assigned [mem 0x620040000000-0x6200401fffff]
[    4.335017] pci 0004:00:00.0: BAR 15: assigned [mem 0x6200c0000000-0x6200c01fffff 64bit pref]
[    4.343734] pci 0004:00:00.0: BAR 13: assigned [io  0x20000-0x20fff]
[    4.350229] pci 0004:00:00.0: PCI bridge to [bus 01]
[    4.355302] pci 0004:00:00.0:   bridge window [io  0x20000-0x20fff]
[    4.361709] pci 0004:00:00.0:   bridge window [mem 0x620040000000-0x6200401fffff]
[    4.369360] pci 0004:00:00.0:   bridge window [mem 0x6200c0000000-0x6200c01fffff 64bit pref]
[    4.377990] pci_bus 0004:00: resource 4 [mem 0x620040000000-0x6200bfffffff window]
[    4.385730] pci_bus 0004:00: resource 5 [mem 0x6200c0000000-0x627fffffffff window]
[    4.393469] pci_bus 0004:00: resource 6 [io  0x20000-0x2ffff window]
[    4.399963] pci_bus 0004:01: resource 0 [io  0x20000-0x20fff]
[    4.405835] pci_bus 0004:01: resource 1 [mem 0x620040000000-0x6200401fffff]
[    4.412952] pci_bus 0004:01: resource 2 [mem 0x6200c0000000-0x6200c01fffff 64bit pref]
[    4.421058] ACPI: PCI Root Bridge [PCI6] (domain 0006 [bus 00-ff])
[    4.427376] acpi PNP0A08:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    4.437009] acpi PNP0A08:03: _OSC: platform does not support [PME AER DPC]
[    4.444148] acpi PNP0A08:03: _OSC: OS now controls [PCIeCapability LTR]
[    4.451028] acpi PNP0A08:03: ECAM area [mem 0x630010000000-0x63001fffffff] reserved by PNP0C02:04
[    4.460169] acpi PNP0A08:03: ECAM at [mem 0x630010000000-0x63001fffffff] for [bus 00-ff]
[    4.468445] ACPI: Remapped I/O 0x0000630020000000 to [io  0x30000-0x3ffff window]
[    4.476110] PCI host bridge to bus 0006:00
[    4.480293] pci_bus 0006:00: root bus resource [mem 0x630040000000-0x6300bfffffff window] (bus address [0x40000000-0xbfffffff])
[    4.492033] pci_bus 0006:00: root bus resource [mem 0x6300c0000000-0x63ffffffffff window]
[    4.500394] pci_bus 0006:00: root bus resource [io  0x30000-0x3ffff window] (bus address [0x0000-0xffff])
[    4.510177] pci_bus 0006:00: root bus resource [bus 00-ff]
[    4.515791] pci 0006:00:00.0: [10de:22b2] type 01 class 0x060400
[    4.522008] pci 0006:00:00.0: PME# supported from D0 D3hot
[    4.527757] pci 0006:00:00.0: bridge window [io  0x1000-0x0fff] to [bus 01] add_size 1000
[    4.536119] pci 0006:00:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 01] add_size 200000 add_align 100000
[    4.547859] pci 0006:00:00.0: bridge window [mem 0x00100000-0x000fffff] to [bus 01] add_size 200000 add_align 100000
[    4.558621] pci 0006:00:00.0: BAR 14: assigned [mem 0x630040000000-0x6300401fffff]
[    4.566360] pci 0006:00:00.0: BAR 15: assigned [mem 0x6300c0000000-0x6300c01fffff 64bit pref]
[    4.575077] pci 0006:00:00.0: BAR 13: assigned [io  0x30000-0x30fff]
[    4.581572] pci 0006:00:00.0: PCI bridge to [bus 01]
[    4.586644] pci 0006:00:00.0:   bridge window [io  0x30000-0x30fff]
[    4.593051] pci 0006:00:00.0:   bridge window [mem 0x630040000000-0x6300401fffff]
[    4.600703] pci 0006:00:00.0:   bridge window [mem 0x6300c0000000-0x6300c01fffff 64bit pref]
[    4.609333] pci_bus 0006:00: resource 4 [mem 0x630040000000-0x6300bfffffff window]
[    4.617072] pci_bus 0006:00: resource 5 [mem 0x6300c0000000-0x63ffffffffff window]
[    4.624811] pci_bus 0006:00: resource 6 [io  0x30000-0x3ffff window]
[    4.631306] pci_bus 0006:01: resource 0 [io  0x30000-0x30fff]
[    4.637178] pci_bus 0006:01: resource 1 [mem 0x630040000000-0x6300401fffff]
[    4.644294] pci_bus 0006:01: resource 2 [mem 0x6300c0000000-0x6300c01fffff 64bit pref]
[    4.652436] ACPI: PCI Root Bridge [PCI8] (domain 0008 [bus 00-ff])
[    4.658753] acpi PNP0A08:04: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    4.668388] acpi PNP0A08:04: _OSC: platform does not support [PME AER DPC]
[    4.675459] acpi PNP0A08:04: _OSC: OS now controls [PCIeCapability LTR]
[    4.682336] acpi PNP0A08:04: ECAM area [mem 0x650010000000-0x65001fffffff] reserved by PNP0C02:05
[    4.691477] acpi PNP0A08:04: ECAM at [mem 0x650010000000-0x65001fffffff] for [bus 00-ff]
[    4.699754] ACPI: Remapped I/O 0x0000650020000000 to [io  0x40000-0x4ffff window]
[    4.707418] PCI host bridge to bus 0008:00
[    4.711601] pci_bus 0008:00: root bus resource [mem 0x650040000000-0x6500bfffffff window] (bus address [0x40000000-0xbfffffff])
[    4.723340] pci_bus 0008:00: root bus resource [mem 0x6500c0000000-0x65ffffffffff window]
[    4.731701] pci_bus 0008:00: root bus resource [io  0x40000-0x4ffff window] (bus address [0x0000-0xffff])
[    4.741485] pci_bus 0008:00: root bus resource [bus 00-ff]
[    4.747095] pci 0008:00:00.0: [10de:22b9] type 01 class 0x060400
[    4.753275] pci 0008:00:00.0: PME# supported from D0 D3hot
[    4.759081] pci 0008:01:00.0: [11f8:4028] type 01 class 0x060400
[    4.766417] pci 0008:01:00.0: enabling Extended Tags
[    4.773074] pci 0008:01:00.0: PME# supported from D0 D3hot D3cold
[    4.780718] pci 0008:01:00.0: 15.753 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x1 link at 0008:00:00.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[    4.796650] pci 0008:01:00.1: [11f8:4028] type 00 class 0x058000
[    4.803418] pci 0008:01:00.1: reg 0x10: [mem 0x6500c0000000-0x6500c03fffff 64bit pref]
[    4.812071] pci 0008:01:00.1: enabling Extended Tags
[    4.818153] pci 0008:01:00.1: PME# supported from D0 D3hot D3cold
[    4.825758] pci 0008:02:00.0: [11f8:4028] type 01 class 0x060400
[    4.832707] pci 0008:02:00.0: enabling Extended Tags
[    4.839539] pci 0008:02:00.0: PME# supported from D0 D3hot D3cold
[    4.847133] pci 0008:02:01.0: [11f8:4028] type 01 class 0x060400
[    4.854046] pci 0008:02:01.0: enabling Extended Tags
[    4.860878] pci 0008:02:01.0: PME# supported from D0 D3hot D3cold
[    4.868472] pci 0008:02:02.0: [11f8:4028] type 01 class 0x060400
[    4.875385] pci 0008:02:02.0: enabling Extended Tags
[    4.882217] pci 0008:02:02.0: PME# supported from D0 D3hot D3cold
[    4.889821] pci 0008:02:03.0: [11f8:4028] type 01 class 0x060400
[    4.896733] pci 0008:02:03.0: enabling Extended Tags
[    4.903556] pci 0008:02:03.0: PME# supported from D0 D3hot D3cold
[    4.911169] pci 0008:02:04.0: [11f8:4028] type 01 class 0x060400
[    4.918081] pci 0008:02:04.0: enabling Extended Tags
[    4.924895] pci 0008:02:04.0: PME# supported from D0 D3hot D3cold
[    4.932517] pci 0008:02:05.0: [11f8:4028] type 01 class 0x060400
[    4.939423] pci 0008:02:05.0: enabling Extended Tags
[    4.946234] pci 0008:02:05.0: PME# supported from D0 D3hot D3cold
[    4.954846] pci 0008:03:00.0: [1344:51c0] type 00 class 0x010802
[    4.961169] pci 0008:03:00.0: reg 0x10: [mem 0x650041300000-0x65004133ffff 64bit]
[    4.969082] pci 0008:03:00.0: reg 0x20: [mem 0x650041340000-0x65004137ffff 64bit]
[    4.976796] pci 0008:03:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref]
[    4.984997] pci 0008:03:00.0: PME# supported from D0 D1 D3hot
[    4.991877] pci 0008:03:00.0: 15.753 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x1 link at 0008:00:00.0 (capable of 63.012 Gb/s with 16.0 GT/s PCIe x4 link)
[    5.009708] pci 0008:08:00.0: [1912:0014] type 00 class 0x0c0330
[    5.016096] pci 0008:08:00.0: reg 0x10: [mem 0x650041200000-0x650041201fff 64bit]
[    5.025170] pci 0008:08:00.0: PME# supported from D0 D3hot D3cold
[    5.033096] pci 0008:09:00.0: [10ec:8168] type 00 class 0x020000
[    5.039375] pci 0008:09:00.0: reg 0x10: [io  0x40000-0x400ff]
[    5.045540] pci 0008:09:00.0: reg 0x18: [mem 0x650041104000-0x650041104fff 64bit]
[    5.053309] pci 0008:09:00.0: reg 0x20: [mem 0x650041100000-0x650041103fff 64bit]
[    5.062316] pci 0008:09:00.0: supports D1 D2
[    5.066678] pci 0008:09:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    5.075468] pci 0008:02:00.0: bridge window [io  0x1000-0x0fff] to [bus 03] add_size 1000
[    5.083833] pci 0008:02:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 03] add_size 200000 add_align 100000
[    5.095572] pci 0008:02:00.0: bridge window [mem 0x00100000-0x001fffff] to [bus 03] add_size 100000 add_align 100000
[    5.106334] pci 0008:02:01.0: bridge window [io  0x1000-0x0fff] to [bus 04] add_size 1000
[    5.114695] pci 0008:02:01.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 04] add_size 200000 add_align 100000
[    5.126435] pci 0008:02:01.0: bridge window [mem 0x00100000-0x000fffff] to [bus 04] add_size 200000 add_align 100000
[    5.137197] pci 0008:02:02.0: bridge window [io  0x1000-0x0fff] to [bus 05] add_size 1000
[    5.145558] pci 0008:02:02.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 05] add_size 200000 add_align 100000
[    5.157297] pci 0008:02:02.0: bridge window [mem 0x00100000-0x000fffff] to [bus 05] add_size 200000 add_align 100000
[    5.168059] pci 0008:02:03.0: bridge window [io  0x1000-0x0fff] to [bus 06-07] add_size 1000
[    5.176687] pci 0008:02:03.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 06-07] add_size 200000 add_align 100000
[    5.188693] pci 0008:02:03.0: bridge window [mem 0x00100000-0x000fffff] to [bus 06-07] add_size 200000 add_align 100000
[    5.199721] pci 0008:02:04.0: bridge window [io  0x1000-0x0fff] to [bus 08] add_size 1000
[    5.208082] pci 0008:02:04.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 08] add_size 200000 add_align 100000
[    5.219822] pci 0008:02:04.0: bridge window [mem 0x00100000-0x001fffff] to [bus 08] add_size 100000 add_align 100000
[    5.230583] pci 0008:02:05.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 09] add_size 200000 add_align 100000
[    5.242323] pci 0008:02:05.0: bridge window [mem 0x00100000-0x001fffff] to [bus 09] add_size 100000 add_align 100000
[    5.253084] pci 0008:01:00.0: bridge window [io  0x1000-0x1fff] to [bus 02-09] add_size 5000
[    5.261713] pci 0008:01:00.0: bridge window [mem 0x00100000-0x006fffff 64bit pref] to [bus 02-09] add_size c00000 add_align 100000
[    5.273721] pci 0008:01:00.0: bridge window [mem 0x00100000-0x006fffff] to [bus 02-09] add_size 900000 add_align 100000
[    5.284750] pci 0008:00:00.0: bridge window [io  0x1000-0x1fff] to [bus 01-09] add_size 5000
[    5.293378] pci 0008:00:00.0: bridge window [mem 0x00400000-0x00ffffff 64bit pref] to [bus 01-09] add_size c00000 add_align 400000
[    5.305415] pci 0008:00:00.0: bridge window [mem 0x00100000-0x006fffff] to [bus 01-09] add_size 900000 add_align 100000
[    5.316445] pci 0008:00:00.0: BAR 15: assigned [mem 0x6500c0000000-0x6500c17fffff 64bit pref]
[    5.325162] pci 0008:00:00.0: BAR 14: assigned [mem 0x650040000000-0x650040efffff]
[    5.332901] pci 0008:00:00.0: BAR 13: assigned [io  0x40000-0x45fff]
[    5.339430] pci 0008:01:00.1: BAR 0: assigned [mem 0x6500c0000000-0x6500c03fffff 64bit pref]
[    5.348121] pci 0008:01:00.0: BAR 14: assigned [mem 0x650040000000-0x650040efffff]
[    5.355860] pci 0008:01:00.0: BAR 15: assigned [mem 0x6500c0400000-0x6500c15fffff 64bit pref]
[    5.364577] pci 0008:01:00.0: BAR 13: assigned [io  0x40000-0x45fff]
[    5.371074] pci 0008:02:00.0: BAR 14: assigned [mem 0x650040000000-0x6500401fffff]
[    5.378813] pci 0008:02:00.0: BAR 15: assigned [mem 0x6500c0400000-0x6500c05fffff 64bit pref]
[    5.387530] pci 0008:02:01.0: BAR 14: assigned [mem 0x650040200000-0x6500403fffff]
[    5.395269] pci 0008:02:01.0: BAR 15: assigned [mem 0x6500c0600000-0x6500c07fffff 64bit pref]
[    5.403986] pci 0008:02:02.0: BAR 14: assigned [mem 0x650040400000-0x6500405fffff]
[    5.411725] pci 0008:02:02.0: BAR 15: assigned [mem 0x6500c0800000-0x6500c09fffff 64bit pref]
[    5.420442] pci 0008:02:03.0: BAR 14: assigned [mem 0x650040600000-0x6500407fffff]
[    5.428181] pci 0008:02:03.0: BAR 15: assigned [mem 0x6500c0a00000-0x6500c0bfffff 64bit pref]
[    5.436899] pci 0008:02:04.0: BAR 14: assigned [mem 0x650040800000-0x6500409fffff]
[    5.444638] pci 0008:02:04.0: BAR 15: assigned [mem 0x6500c0c00000-0x6500c0dfffff 64bit pref]
[    5.453355] pci 0008:02:05.0: BAR 14: assigned [mem 0x650040a00000-0x650040bfffff]
[    5.461094] pci 0008:02:05.0: BAR 15: assigned [mem 0x6500c0e00000-0x6500c0ffffff 64bit pref]
[    5.469811] pci 0008:02:00.0: BAR 13: assigned [io  0x40000-0x40fff]
[    5.476306] pci 0008:02:01.0: BAR 13: assigned [io  0x41000-0x41fff]
[    5.482801] pci 0008:02:02.0: BAR 13: assigned [io  0x42000-0x42fff]
[    5.489296] pci 0008:02:03.0: BAR 13: assigned [io  0x43000-0x43fff]
[    5.495791] pci 0008:02:04.0: BAR 13: assigned [io  0x44000-0x44fff]
[    5.502285] pci 0008:02:05.0: BAR 13: assigned [io  0x45000-0x45fff]
[    5.508814] pci 0008:03:00.0: BAR 0: assigned [mem 0x650040000000-0x65004003ffff 64bit]
[    5.517064] pci 0008:03:00.0: BAR 4: assigned [mem 0x650040040000-0x65004007ffff 64bit]
[    5.525312] pci 0008:03:00.0: BAR 6: assigned [mem 0x650040080000-0x6500400bffff pref]
[    5.533439] pci 0008:02:00.0: PCI bridge to [bus 03]
[    5.538522] pci 0008:02:00.0:   bridge window [io  0x40000-0x40fff]
[    5.544947] pci 0008:02:00.0:   bridge window [mem 0x650040000000-0x6500401fffff]
[    5.552657] pci 0008:02:00.0:   bridge window [mem 0x6500c0400000-0x6500c05fffff 64bit pref]
[    5.561433] pci 0008:02:01.0: PCI bridge to [bus 04]
[    5.566523] pci 0008:02:01.0:   bridge window [io  0x41000-0x41fff]
[    5.572948] pci 0008:02:01.0:   bridge window [mem 0x650040200000-0x6500403fffff]
[    5.580652] pci 0008:02:01.0:   bridge window [mem 0x6500c0600000-0x6500c07fffff 64bit pref]
[    5.589432] pci 0008:02:02.0: PCI bridge to [bus 05]
[    5.594523] pci 0008:02:02.0:   bridge window [io  0x42000-0x42fff]
[    5.600946] pci 0008:02:02.0:   bridge window [mem 0x650040400000-0x6500405fffff]
[    5.608651] pci 0008:02:02.0:   bridge window [mem 0x6500c0800000-0x6500c09fffff 64bit pref]
[    5.617429] pci 0008:02:03.0: PCI bridge to [bus 06-07]
[    5.622787] pci 0008:02:03.0:   bridge window [io  0x43000-0x43fff]
[    5.629210] pci 0008:02:03.0:   bridge window [mem 0x650040600000-0x6500407fffff]
[    5.636915] pci 0008:02:03.0:   bridge window [mem 0x6500c0a00000-0x6500c0bfffff 64bit pref]
[    5.645703] pci 0008:08:00.0: BAR 0: assigned [mem 0x650040800000-0x650040801fff 64bit]
[    5.653952] pci 0008:02:04.0: PCI bridge to [bus 08]
[    5.659043] pci 0008:02:04.0:   bridge window [io  0x44000-0x44fff]
[    5.665467] pci 0008:02:04.0:   bridge window [mem 0x650040800000-0x6500409fffff]
[    5.673173] pci 0008:02:04.0:   bridge window [mem 0x6500c0c00000-0x6500c0dfffff 64bit pref]
[    5.681959] pci 0008:09:00.0: BAR 4: assigned [mem 0x650040a00000-0x650040a03fff 64bit]
[    5.690221] pci 0008:09:00.0: BAR 2: assigned [mem 0x650040a04000-0x650040a04fff 64bit]
[    5.698481] pci 0008:09:00.0: BAR 0: assigned [io  0x45000-0x450ff]
[    5.704909] pci 0008:02:05.0: PCI bridge to [bus 09]
[    5.709991] pci 0008:02:05.0:   bridge window [io  0x45000-0x45fff]
[    5.716415] pci 0008:02:05.0:   bridge window [mem 0x650040a00000-0x650040bfffff]
[    5.724122] pci 0008:02:05.0:   bridge window [mem 0x6500c0e00000-0x6500c0ffffff 64bit pref]
[    5.732901] pci 0008:01:00.0: PCI bridge to [bus 02-09]
[    5.738257] pci 0008:01:00.0:   bridge window [io  0x40000-0x45fff]
[    5.744681] pci 0008:01:00.0:   bridge window [mem 0x650040000000-0x650040efffff]
[    5.752554] pci 0008:01:00.0:   bridge window [mem 0x6500c0400000-0x6500c15fffff 64bit pref]
[    5.761835] pci 0008:00:00.0: PCI bridge to [bus 01-09]
[    5.767184] pci 0008:00:00.0:   bridge window [io  0x40000-0x45fff]
[    5.773590] pci 0008:00:00.0:   bridge window [mem 0x650040000000-0x650040efffff]
[    5.781241] pci 0008:00:00.0:   bridge window [mem 0x6500c0000000-0x6500c17fffff 64bit pref]
[    5.789870] pci_bus 0008:00: resource 4 [mem 0x650040000000-0x6500bfffffff window]
[    5.797609] pci_bus 0008:00: resource 5 [mem 0x6500c0000000-0x65ffffffffff window]
[    5.805348] pci_bus 0008:00: resource 6 [io  0x40000-0x4ffff window]
[    5.811843] pci_bus 0008:01: resource 0 [io  0x40000-0x45fff]
[    5.817716] pci_bus 0008:01: resource 1 [mem 0x650040000000-0x650040efffff]
[    5.824833] pci_bus 0008:01: resource 2 [mem 0x6500c0000000-0x6500c17fffff 64bit pref]
[    5.832928] pci_bus 0008:02: resource 0 [io  0x40000-0x45fff]
[    5.838800] pci_bus 0008:02: resource 1 [mem 0x650040000000-0x650040efffff]
[    5.845917] pci_bus 0008:02: resource 2 [mem 0x6500c0400000-0x6500c15fffff 64bit pref]
[    5.854012] pci_bus 0008:03: resource 0 [io  0x40000-0x40fff]
[    5.859884] pci_bus 0008:03: resource 1 [mem 0x650040000000-0x6500401fffff]
[    5.867001] pci_bus 0008:03: resource 2 [mem 0x6500c0400000-0x6500c05fffff 64bit pref]
[    5.875096] pci_bus 0008:04: resource 0 [io  0x41000-0x41fff]
[    5.880969] pci_bus 0008:04: resource 1 [mem 0x650040200000-0x6500403fffff]
[    5.888086] pci_bus 0008:04: resource 2 [mem 0x6500c0600000-0x6500c07fffff 64bit pref]
[    5.896180] pci_bus 0008:05: resource 0 [io  0x42000-0x42fff]
[    5.902053] pci_bus 0008:05: resource 1 [mem 0x650040400000-0x6500405fffff]
[    5.909169] pci_bus 0008:05: resource 2 [mem 0x6500c0800000-0x6500c09fffff 64bit pref]
[    5.917264] pci_bus 0008:06: resource 0 [io  0x43000-0x43fff]
[    5.923136] pci_bus 0008:06: resource 1 [mem 0x650040600000-0x6500407fffff]
[    5.930253] pci_bus 0008:06: resource 2 [mem 0x6500c0a00000-0x6500c0bfffff 64bit pref]
[    5.938347] pci_bus 0008:08: resource 0 [io  0x44000-0x44fff]
[    5.944220] pci_bus 0008:08: resource 1 [mem 0x650040800000-0x6500409fffff]
[    5.951337] pci_bus 0008:08: resource 2 [mem 0x6500c0c00000-0x6500c0dfffff 64bit pref]
[    5.959432] pci_bus 0008:09: resource 0 [io  0x45000-0x45fff]
[    5.965304] pci_bus 0008:09: resource 1 [mem 0x650040a00000-0x650040bfffff]
[    5.972421] pci_bus 0008:09: resource 2 [mem 0x6500c0e00000-0x6500c0ffffff 64bit pref]
[    5.980572] ACPI: PCI Root Bridge [PCI9] (domain 0009 [bus 00-ff])
[    5.986890] acpi PNP0A08:05: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
[    5.996530] acpi PNP0A08:05: _OSC: platform does not support [PME AER DPC]
[    6.003602] acpi PNP0A08:05: _OSC: OS now controls [PCIeCapability LTR]
[    6.010488] acpi PNP0A08:05: ECAM area [mem 0x660010000000-0x66001fffffff] reserved by PNP0C02:06
[    6.019630] acpi PNP0A08:05: ECAM at [mem 0x660010000000-0x66001fffffff] for [bus 00-ff]
[    6.027907] ACPI: Remapped I/O 0x0000660020000000 to [io  0x50000-0x5ffff window]
[    6.035576] PCI host bridge to bus 0009:00
[    6.039759] pci_bus 0009:00: root bus resource [mem 0x660040000000-0x6600bfffffff window] (bus address [0x40000000-0xbfffffff])
[    6.051498] pci_bus 0009:00: root bus resource [mem 0x6600c0000000-0x66ffffffffff window]
[    6.059859] pci_bus 0009:00: root bus resource [io  0x50000-0x5ffff window] (bus address [0x0000-0xffff])
[    6.069643] pci_bus 0009:00: root bus resource [bus 00-ff]
[    6.075321] pci 0009:00:00.0: [10de:22b1] type 01 class 0x060400
[    6.081502] pci 0009:00:00.0: PME# supported from D0 D3hot
[    6.087220] pci 0009:01:00.0: [10de:2342] type 00 class 0x030200
[    6.093367] pci 0009:01:00.0: reg 0x10: [mem 0x660040000000-0x660040ffffff]
[    6.100490] pci 0009:01:00.0: reg 0x14: [mem 0x662000000000-0x663fffffffff 64bit pref]
[    6.108591] pci 0009:01:00.0: reg 0x1c: [mem 0x664000000000-0x664001ffffff 64bit pref]
[    6.116689] pci 0009:01:00.0: reg 0x24: [io  0x50000-0x5007f]
[    6.122583] pci 0009:01:00.0: Enabling HDA controller
[    6.127813] pci 0009:01:00.0: reg 0x274: [mem 0x00000000-0x0003ffff 64bit pref]
[    6.135286] pci 0009:01:00.0: VF(n) BAR0 space: [mem 0x00000000-0x005fffff 64bit pref] (contains BAR0 for 24 VFs)
[    6.145887] pci 0009:01:00.0: 15.753 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x1 link at 0009:00:00.0 (capable of 504.112 Gb/s with 32.0 GT/s PCIe x16 link)
[    6.173643] pci 0009:00:00.0: BAR 15: assigned [mem 0x661000000000-0x663fffffffff 64bit pref]
[    6.182361] pci 0009:00:00.0: BAR 14: assigned [mem 0x660040000000-0x660040ffffff]
[    6.190100] pci 0009:00:00.0: BAR 13: assigned [io  0x50000-0x50fff]
[    6.196594] pci 0009:01:00.0: BAR 1: assigned [mem 0x662000000000-0x663fffffffff 64bit pref]
[    6.205227] pci 0009:01:00.0: BAR 3: assigned [mem 0x661000000000-0x661001ffffff 64bit pref]
[    6.213859] pci 0009:01:00.0: BAR 0: assigned [mem 0x660040000000-0x660040ffffff]
[    6.221510] pci 0009:01:00.0: BAR 7: assigned [mem 0x661002000000-0x6610025fffff 64bit pref]
[    6.230140] pci 0009:01:00.0: BAR 5: assigned [io  0x50000-0x5007f]
[    6.236547] pci 0009:00:00.0: PCI bridge to [bus 01]
[    6.241620] pci 0009:00:00.0:   bridge window [io  0x50000-0x50fff]
[    6.248026] pci 0009:00:00.0:   bridge window [mem 0x660040000000-0x660040ffffff]
[    6.255677] pci 0009:00:00.0:   bridge window [mem 0x661000000000-0x663fffffffff 64bit pref]
[    6.264306] pci_bus 0009:00: resource 4 [mem 0x660040000000-0x6600bfffffff window]
[    6.272046] pci_bus 0009:00: resource 5 [mem 0x6600c0000000-0x66ffffffffff window]
[    6.279785] pci_bus 0009:00: resource 6 [io  0x50000-0x5ffff window]
[    6.286280] pci_bus 0009:01: resource 0 [io  0x50000-0x50fff]
[    6.292152] pci_bus 0009:01: resource 1 [mem 0x660040000000-0x660040ffffff]
[    6.299269] pci_bus 0009:01: resource 2 [mem 0x661000000000-0x663fffffffff 64bit pref]
[    6.307558] iommu: Default domain type: Passthrough
[    6.312733] SCSI subsystem initialized
[    6.316580] ACPI: bus type USB registered
[    6.320685] usbcore: registered new interface driver usbfs
[    6.326295] usbcore: registered new interface driver hub
[    6.331736] usbcore: registered new device driver usb
[    6.336903] pps_core: LinuxPPS API ver. 1 registered
[    6.341976] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    6.351317] PTP clock support registered
[    6.355328] EDAC MC: Ver: 3.0.0
[    6.359042] ARM FF-A: Driver version 1.0
[    6.363048] ARM FF-A: Firmware version 1.1 found
[    6.367764] ARM FF-A: Firmware version higher than driver version, downgrading
[    6.375189] Registered efivars operations
[    6.380405] vgaarb: loaded
[    6.383209] mpam: update max_partid from 40 to 45
[    6.389204] clocksource: Switched to clocksource arch_sys_counter
[    6.395515] VFS: Disk quotas dquot_6.6.0
[    6.399538] VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
[    6.406827] pnp: PnP ACPI init
[    6.410085] system 00:00: [mem 0x600010000000-0x60001fffffff window] could not be reserved
[    6.418629] system 00:01: [mem 0x610010000000-0x61001fffffff window] could not be reserved
[    6.427097] system 00:02: [mem 0x620010000000-0x62001fffffff window] could not be reserved
[    6.435563] system 00:03: [mem 0x630010000000-0x63001fffffff window] could not be reserved
[    6.444028] system 00:04: [mem 0x650010000000-0x65001fffffff window] could not be reserved
[    6.452493] system 00:05: [mem 0x660010000000-0x66001fffffff window] could not be reserved
[    6.460950] pnp: PnP ACPI: found 6 devices
[    6.465984] NET: Registered PF_INET protocol family
[    6.471032] IP idents hash table entries: 262144 (order: 5, 2097152 bytes, linear)
[    6.480570] tcp_listen_portaddr_hash hash table entries: 65536 (order: 4, 1048576 bytes, linear)
[    6.489703] Table-perturb hash table entries: 65536 (order: 2, 262144 bytes, linear)
[    6.497623] TCP established hash table entries: 524288 (order: 6, 4194304 bytes, linear)
[    6.506138] TCP bind hash table entries: 65536 (order: 4, 1048576 bytes, linear)
[    6.513845] TCP: Hash tables configured (established 524288 bind 65536)
[    6.520658] UDP hash table entries: 65536 (order: 5, 2097152 bytes, linear)
[    6.527919] UDP-Lite hash table entries: 65536 (order: 5, 2097152 bytes, linear)
[    6.535656] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    6.541744] pci 0008:08:00.0: enabling device (0000 -> 0002)
[    6.547632] PCI: CLS 0 bytes, default 64
[    6.551689] Unpacking initramfs...
[    6.565375] hw perfevents: enabled with armv8_pmuv3_0 PMU driver, 7 counters available
[    6.573268] Freeing initrd memory: 123968K
[    6.573575] kvm [1]: IPA Size Limit: 48 bits
[    6.582058] kvm [1]: GICv4.1 support enabled
[    6.586420] kvm [1]: GICv3: no GICV resource entry
[    6.591316] kvm [1]: disabling GICv2 emulation
[    6.595862] kvm [1]: GIC system register CPU interface enabled
[    6.603354] kvm [1]: vgic interrupt IRQ9
[    6.610091] kvm [1]: VHE mode initialized successfully
[    6.615926] Initialise system trusted keyrings
[    6.620516] workingset: timestamp_bits=40 max_order=21 bucket_order=0
[    6.627767] 9p: Installing v9fs 9p2000 file system support
[    6.642447] Key type asymmetric registered
[    6.646631] Asymmetric key parser 'x509' registered
[    6.651626] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 243)
[    6.659188] io scheduler mq-deadline registered
[    6.663817] io scheduler kyber registered
[    6.669176] IPMI message handler: version 39.2
[    6.673726] ipmi device interface
[    6.677140] ipmi_si: IPMI System Interface driver
[    6.681975] ipmi_si: Unable to find any System Interface(s)
[    6.687670] ipmi_ssif: IPMI SSIF Interface driver
[    6.692495] IPMI Watchdog: driver initialized
[    6.696944] IPMI poweroff: Copyright (C) 2004 MontaVista Software - IPMI Powerdown via sys_reboot
[    6.765231] thermal LNXTHERM:00: registered as thermal_zone0
[    6.771019] ACPI: thermal: Thermal Zone [TZ00] (43 C)
[    6.833294] thermal LNXTHERM:01: registered as thermal_zone1
[    6.839081] ACPI: thermal: Thermal Zone [TZ01] (42 C)
[    6.921230] thermal LNXTHERM:02: registered as thermal_zone2
[    6.927016] ACPI: thermal: Thermal Zone [TZ02] (42 C)
[    7.009231] thermal LNXTHERM:03: registered as thermal_zone3
[    7.015016] ACPI: thermal: Thermal Zone [TZ03] (42 C)
[    7.129228] thermal LNXTHERM:04: registered as thermal_zone4
[    7.135014] ACPI: thermal: Thermal Zone [TZ04] (41 C)
[    7.269228] thermal LNXTHERM:05: registered as thermal_zone5
[    7.275014] ACPI: thermal: Thermal Zone [TZ05] (41 C)
[    7.453227] thermal LNXTHERM:06: registered as thermal_zone6
[    7.459012] ACPI: thermal: Thermal Zone [TZ06] (42 C)
[    7.673292] thermal LNXTHERM:07: registered as thermal_zone7
[    7.679078] ACPI: thermal: Thermal Zone [TZ07] (42 C)
[    7.953227] thermal LNXTHERM:08: registered as thermal_zone8
[    7.959012] ACPI: thermal: Thermal Zone [TZ08] (42 C)
[    8.273228] thermal LNXTHERM:09: registered as thermal_zone9
[    8.279014] ACPI: thermal: Thermal Zone [TZ09] (43 C)
[    8.633226] thermal LNXTHERM:0a: registered as thermal_zone10
[    8.639100] ACPI: thermal: Thermal Zone [TZ0A] (33 C)
[    9.053231] thermal LNXTHERM:0b: registered as thermal_zone11
[    9.059106] ACPI: thermal: Thermal Zone [TZ0B] (34 C)
[    9.064389] Unknown online node for memory at 0x2042cab000, assuming node 0
[    9.071508] nfit ACPI0012:00: changing numa node from -1 to 0 for nfit region [0x0000002042cab000-0x000000204a77ffff]
[    9.082362] Unknown target node for memory at 0x2042cab000, assuming node 0
[    9.089480] nfit ACPI0012:00: changing target node from -1 to 0 for nfit region [0x0000002042cab000-0x000000204a77ffff]
[    9.100582] EINJ: Error INJection is initialized.
[    9.105416] ACPI GTDT: found 1 SBSA generic Watchdog(s).
[    9.111898] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    9.118568] arm-smmu-v3 arm-smmu-v3.0.auto: option mask 0x0
[    9.124305] arm-smmu-v3 arm-smmu-v3.0.auto: ias 48-bit, oas 48-bit (features 0x00061faf)
[    9.132746] arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for cmdq
[    9.140372] arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for evtq
[    9.147795] arm-smmu-v3 arm-smmu-v3.0.auto: msi_domain absent - falling back to wired irqs
[    9.156334] arm-smmu-v3 arm-smmu-v3.1.auto: option mask 0x0
[    9.162036] arm-smmu-v3 arm-smmu-v3.1.auto: ias 48-bit, oas 48-bit (features 0x00061faf)
[    9.170474] arm-smmu-v3 arm-smmu-v3.1.auto: allocated 524288 entries for cmdq
[    9.178099] arm-smmu-v3 arm-smmu-v3.1.auto: allocated 524288 entries for evtq
[    9.185518] arm-smmu-v3 arm-smmu-v3.1.auto: msi_domain absent - falling back to wired irqs
[    9.194032] arm-smmu-v3 arm-smmu-v3.2.auto: option mask 0x0
[    9.199735] arm-smmu-v3 arm-smmu-v3.2.auto: ias 48-bit, oas 48-bit (features 0x00061faf)
[    9.208173] arm-smmu-v3 arm-smmu-v3.2.auto: allocated 524288 entries for cmdq
[    9.215796] arm-smmu-v3 arm-smmu-v3.2.auto: allocated 524288 entries for evtq
[    9.223215] arm-smmu-v3 arm-smmu-v3.2.auto: msi_domain absent - falling back to wired irqs
[    9.231762] arm-smmu-v3 arm-smmu-v3.3.auto: option mask 0x0
[    9.237466] arm-smmu-v3 arm-smmu-v3.3.auto: ias 48-bit, oas 48-bit (features 0x00061faf)
[    9.245902] arm-smmu-v3 arm-smmu-v3.3.auto: allocated 524288 entries for cmdq
[    9.253526] arm-smmu-v3 arm-smmu-v3.3.auto: allocated 524288 entries for evtq
[    9.260943] arm-smmu-v3 arm-smmu-v3.3.auto: msi_domain absent - falling back to wired irqs
[    9.269453] arm-smmu-v3 arm-smmu-v3.4.auto: option mask 0x0
[    9.275154] arm-smmu-v3 arm-smmu-v3.4.auto: ias 48-bit, oas 48-bit (features 0x00061faf)
[    9.283591] arm-smmu-v3 arm-smmu-v3.4.auto: allocated 524288 entries for cmdq
[    9.291215] arm-smmu-v3 arm-smmu-v3.4.auto: allocated 524288 entries for evtq
[    9.298635] arm-smmu-v3 arm-smmu-v3.4.auto: msi_domain absent - falling back to wired irqs
[    9.311849] loop: module loaded
[    9.315113] nd_pmem namespace0.0: unable to guarantee persistence of writes
[    9.322401]  pmem0: p1
[    9.325127] nvme 0008:03:00.0: Adding to iommu group 0
[    9.330492] nvme nvme0: pci function 0008:03:00.0
[    9.335394] tun: Universal TUN/TAP device driver, 1.6
[    9.340585] e1000: Intel(R) PRO/1000 Network Driver
[    9.345570] e1000: Copyright (c) 1999-2006 Intel Corporation.
[    9.347070] nvme nvme0: 74/0/0 default/read/poll queues
[    9.351453] e1000e: Intel(R) PRO/1000 Network Driver
[    9.361861] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    9.364602]  nvme0n1: p1 p2
[    9.367917] igb: Intel(R) Gigabit Ethernet Network Driver
[    9.376281] igb: Copyright (c) 2007-2014 Intel Corporation.
[    9.381980] igbvf: Intel(R) Gigabit Virtual Function Network Driver
[    9.388385] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[    9.394439] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[    9.400667] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    9.406630] r8169 0008:09:00.0: Adding to iommu group 1
[    9.426231] r8169 0008:09:00.0 eth0: RTL8168h/8111h, 48:b0:2d:b1:4d:55, XID 541, IRQ 121
[    9.434511] r8169 0008:09:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
[    9.443262] usbcore: registered new interface driver cdc_ether
[    9.449230] usbcore: registered new interface driver r8153_ecm
[    9.455230] VFIO - User Level meta-driver version: 0.3
[    9.460546] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    9.467220] ehci-pci: EHCI PCI platform driver
[    9.471762] ehci-platform: EHCI generic platform driver
[    9.477160] xhci_hcd 0008:08:00.0: Adding to iommu group 2
[   10.592622] xhci_hcd 0008:08:00.0: xHCI Host Controller
[   10.597969] xhci_hcd 0008:08:00.0: new USB bus registered, assigned bus number 1
[   10.605568] xhci_hcd 0008:08:00.0: Zeroing 64bit base registers, expecting fault
[   10.613231] xhci_hcd 0008:08:00.0: Fault detected
[   10.623368] xhci_hcd 0008:08:00.0: hcc params 0x014051cf hci version 0x100 quirks 0x0000001100000410
[   10.633182] xhci_hcd 0008:08:00.0: xHCI Host Controller
[   10.638522] xhci_hcd 0008:08:00.0: new USB bus registered, assigned bus number 2
[   10.646084] xhci_hcd 0008:08:00.0: Host supports USB 3.0 SuperSpeed
[   10.652608] hub 1-0:1.0: USB hub found
[   10.656447] hub 1-0:1.0: 4 ports detected
[   10.660624] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[   10.668937] hub 2-0:1.0: USB hub found
[   10.672773] hub 2-0:1.0: 4 ports detected
[   10.676965] usbcore: registered new interface driver usb-storage
[   10.683176] rtc-efi rtc-efi.0: registered as rtc0
[   10.688026] rtc-efi rtc-efi.0: setting system clock to 2023-01-19T21:54:17 UTC (1674165257)
[   10.696581] i2c_dev: i2c /dev entries driver
[   10.701004] sbsa-gwdt sbsa-gwdt.0: Initialized with 10s timeout @ 1000000000 Hz, action=0.
[   10.709541] pstore: Registered efi as persistent store backend
[   10.715534] SMCCC: SOC_ID: ID = jep106:036b:0024 Revision = 0x00000101
[   10.730161] NET: Registered PF_INET6 protocol family
[   10.735503] Segment Routing with IPv6
[   10.739247] In-situ OAM (IOAM) with IPv6
[   10.743262] NET: Registered PF_PACKET protocol family
[   10.748453] sctp: Hash tables configured (bind 4096/4096)
[   10.754016] 9pnet: Installing 9P2000 support
[   10.758394] Key type dns_resolver registered
[   10.762802] NET: Registered PF_VSOCK protocol family
[   10.768933] registered taskstats version 1
[   10.773124] Loading compiled-in X.509 certificates
[   10.778108] pstore: Using crash dump compression: deflate
[   10.783642] Key type big_key registered
[   10.787872] mpam: MPAM enabled with 45 partid and 2 pmg
[   10.790576] pcieport 0000:00:00.0: Adding to iommu group 3
[   10.798963] pcieport 0002:00:00.0: Adding to iommu group 4
[   10.804689] pcieport 0004:00:00.0: Adding to iommu group 5
[   10.810406] pcieport 0006:00:00.0: Adding to iommu group 6
[   10.816129] pcieport 0008:00:00.0: Adding to iommu group 7
[   10.821835] pcieport 0008:01:00.0: Adding to iommu group 8
[   10.828985] pcieport 0008:02:00.0: Adding to iommu group 9
[   10.836790] pcieport 0008:02:01.0: Adding to iommu group 10
[   10.844885] pcieport 0008:02:02.0: Adding to iommu group 11
[   10.852759] pcieport 0008:02:03.0: Adding to iommu group 12
[   10.860756] pcieport 0008:02:04.0: Adding to iommu group 13
[   10.868676] pcieport 0008:02:05.0: Adding to iommu group 14
[   10.876713] pcieport 0009:00:00.0: Adding to iommu group 15
[   10.891236] Freeing unused kernel memory: 2240K
[   10.917183] usb 1-1: new high-speed USB device number 2 using xhci_hcd
[   10.937383] Run /init as init process
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
[   11.079768] hub 1-1:1.0: USB hub found
[   11.083879] hub 1-1:1.0: 7 ports detected
[   11.381184] usb 1-1.1: new high-speed USB device number 3 using xhci_hcd
[   11.507959] cdc_ether 1-1.1:1.0 usb0: register 'cdc_ether' at usb-0008:08:00.0-1.1, CDC Ethernet Device, 8e:40:7a:97:bc:8e
[   14.955110] random: crng init done
Saving random seed: OK
Starting network: [   15.069183] Generic FE-GE Realtek PHY r8169-8-900:00: attached PHY driver (mii_bus:phy_addr=r8169-8-900:00, irq=MAC)
[   15.281296] r8169 0008:09:00.0 eth0: Link is Down
udhcpc: started, v1.35.0
udhcpc: broadcasting discover
udhcpc: no lease, forking to background
OK
Starting dropbear sshd: OK
root@grace# [   18.861591] r8169 0008:09:00.0 eth0: Link is Up - 1Gbps/Full - flow control off
[   18.869075] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

root@grace# uname -a
Linux grace 6.2.0-rc4 #8 SMP Thu Jan 19 17:53:00 CST 2023 aarch64 GNU/Linux

root@grace# 
[  258.519837] kvm_create_vm(1236) called kvm=ffff8000303e0000
[  258.527613] kvm_create_vm(1236) called kvm=ffff800030400000
[  258.533393] kvm_vm_release(1374) called kvm=ffff800030400000
[  258.557893] kvm_create_vm(1236) called kvm=ffff8000304c0000
[  258.563627] kvm_vm_release(1374) called kvm=ffff8000304c0000
[  258.667101] vgic_v4_init(267) called kvm=ffff8000303e0000 doorbell=140(64)
[  258.853783] kvm_create_vm(1236) called kvm=ffff800031020000
[  258.861988] kvm_create_vm(1236) called kvm=ffff800031040000
[  258.867797] kvm_vm_release(1374) called kvm=ffff800031040000
[  258.873976] kvm_create_vm(1236) called kvm=ffff800031080000
[  258.879724] kvm_vm_release(1374) called kvm=ffff800031080000
[  258.978326] vgic_v4_init(267) called kvm=ffff800031020000 doorbell=1228(64)
[  259.117997] kvm_create_vm(1236) called kvm=ffff800031ae0000
[  259.126079] kvm_create_vm(1236) called kvm=ffff800031b00000
[  259.131899] kvm_vm_release(1374) called kvm=ffff800031b00000
[  259.138068] kvm_create_vm(1236) called kvm=ffff800031b40000
[  259.143844] kvm_vm_release(1374) called kvm=ffff800031b40000
[  259.246442] vgic_v4_init(267) called kvm=ffff800031ae0000 doorbell=2316(64)
[  259.393845] kvm_create_vm(1236) called kvm=ffff8000325e0000
[  259.402781] kvm_create_vm(1236) called kvm=ffff800032600000
[  259.408646] kvm_vm_release(1374) called kvm=ffff800032600000
[  259.414931] kvm_create_vm(1236) called kvm=ffff800032640000
[  259.420715] kvm_vm_release(1374) called kvm=ffff800032640000
[  259.521881] vgic_v4_init(267) called kvm=ffff8000325e0000 doorbell=3404(64)
[  259.666997] kvm_create_vm(1236) called kvm=ffff8000330c0000
[  259.675346] kvm_create_vm(1236) called kvm=ffff8000330e0000
[  259.681173] kvm_vm_release(1374) called kvm=ffff8000330e0000
[  259.687343] kvm_create_vm(1236) called kvm=ffff800033120000
[  259.693116] kvm_vm_release(1374) called kvm=ffff800033120000
[  259.783433] vgic_v4_init(267) called kvm=ffff8000330c0000 doorbell=4492(64)
[  259.953796] kvm_create_vm(1236) called kvm=ffff800033c80000
[  259.961874] kvm_create_vm(1236) called kvm=ffff800033ca0000
[  259.967680] kvm_vm_release(1374) called kvm=ffff800033ca0000
[  259.973927] kvm_create_vm(1236) called kvm=ffff800033ce0000
[  259.979714] kvm_vm_release(1374) called kvm=ffff800033ce0000
[  260.116812] vgic_v4_init(267) called kvm=ffff800033c80000 doorbell=5580(64)
[  260.744980] kvm_create_vm(1236) called kvm=ffff800034a20000
[  260.752594] kvm_create_vm(1236) called kvm=ffff800034a60000
[  260.758328] kvm_vm_release(1374) called kvm=ffff800034a60000
[  260.764358] kvm_create_vm(1236) called kvm=ffff800034a80000
[  260.770077] kvm_vm_release(1374) called kvm=ffff800034a80000
[  261.246087] vgic_v4_init(267) called kvm=ffff800034a20000 doorbell=6668(64)
[  261.664859] kvm_create_vm(1236) called kvm=ffff8000357a0000
[  261.673366] kvm_create_vm(1236) called kvm=ffff8000357e0000
[  261.679345] kvm_vm_release(1374) called kvm=ffff8000357e0000
[  261.685576] kvm_create_vm(1236) called kvm=ffff800035800000
[  261.691396] kvm_vm_release(1374) called kvm=ffff800035800000
[  262.131167] vgic_v4_init(267) called kvm=ffff8000357a0000 doorbell=7756(64)
[  262.985089] kvm_create_vm(1236) called kvm=ffff8000367a0000
[  262.995954] kvm_create_vm(1236) called kvm=ffff8000367e0000
[  263.001694] kvm_vm_release(1374) called kvm=ffff8000367e0000
[  263.008196] kvm_create_vm(1236) called kvm=ffff800036820000
[  263.013926] kvm_vm_release(1374) called kvm=ffff800036820000
[  263.520661] vgic_v4_init(267) called kvm=ffff8000367a0000 doorbell=8844(64)
[  264.045132] kvm_create_vm(1236) called kvm=ffff8000376e0000
[  264.052028] kvm_create_vm(1236) called kvm=ffff800037720000
[  264.057955] kvm_vm_release(1374) called kvm=ffff800037720000
[  264.069762] kvm_create_vm(1236) called kvm=ffff800037740000
[  264.075502] kvm_vm_release(1374) called kvm=ffff800037740000
[  264.589563] vgic_v4_init(267) called kvm=ffff8000376e0000 doorbell=9932(64)
[  264.986861] kvm_create_vm(1236) called kvm=ffff80003fec0000
[  264.997744] kvm_create_vm(1236) called kvm=ffff80003ff00000
[  265.003516] kvm_vm_release(1374) called kvm=ffff80003ff00000
[  265.013497] kvm_create_vm(1236) called kvm=ffff80003ff40000
[  265.019257] kvm_vm_release(1374) called kvm=ffff80003ff40000
[  265.487934] vgic_v4_init(267) called kvm=ffff80003fec0000 doorbell=11020(64)
[  266.353271] kvm_create_vm(1236) called kvm=ffff800050f80000
[  266.362554] kvm_create_vm(1236) called kvm=ffff800050fa0000
[  266.368305] kvm_vm_release(1374) called kvm=ffff800050fa0000
[  266.374783] kvm_create_vm(1236) called kvm=ffff800050fe0000
[  266.380501] kvm_vm_release(1374) called kvm=ffff800050fe0000
[  267.123984] vgic_v4_init(267) called kvm=ffff800050f80000 doorbell=12108(64)
[  267.862029] kvm_create_vm(1236) called kvm=ffff800016080000
[  267.871450] kvm_create_vm(1236) called kvm=ffff8000160c0000
[  267.877187] kvm_vm_release(1374) called kvm=ffff8000160c0000
[  267.889841] kvm_create_vm(1236) called kvm=ffff800016120000
[  267.895563] kvm_vm_release(1374) called kvm=ffff800016120000
[  268.414704] vgic_v4_init(267) called kvm=ffff800016080000 doorbell=13196(64)
[  269.233206] kvm_create_vm(1236) called kvm=ffff8000179a0000
[  269.250008] kvm_create_vm(1236) called kvm=ffff8000179e0000
[  269.255737] kvm_vm_release(1374) called kvm=ffff8000179e0000
[  269.293831] kvm_create_vm(1236) called kvm=ffff800017a40000
[  269.299545] kvm_vm_release(1374) called kvm=ffff800017a40000
[  269.752031] vgic_v4_init(267) called kvm=ffff8000179a0000 doorbell=14284(64)
[  270.402065] kvm_create_vm(1236) called kvm=ffff800018920000
[  270.415703] kvm_create_vm(1236) called kvm=ffff800018960000
[  270.421433] kvm_vm_release(1374) called kvm=ffff800018960000
[  270.428963] kvm_create_vm(1236) called kvm=ffff800018980000
[  270.434718] kvm_vm_release(1374) called kvm=ffff800018980000
[  271.030587] vgic_v4_init(267) called kvm=ffff800018920000 doorbell=15372(64)
[  271.941460] kvm_create_vm(1236) called kvm=ffff800019a80000
[  271.951108] kvm_create_vm(1236) called kvm=ffff800019ac0000
[  271.976587] kvm_vm_release(1374) called kvm=ffff800019ac0000
[  271.993855] kvm_create_vm(1236) called kvm=ffff800019b00000
[  272.024367] kvm_vm_release(1374) called kvm=ffff800019b00000
[  272.812285] vgic_v4_init(267) called kvm=ffff800019a80000 doorbell=16460(64)
[  274.138056] kvm_create_vm(1236) called kvm=ffff80001b020000
[  274.147482] kvm_create_vm(1236) called kvm=ffff80001b040000
[  274.153208] kvm_vm_release(1374) called kvm=ffff80001b040000
[  274.274058] kvm_create_vm(1236) called kvm=ffff80001b120000
[  274.279777] kvm_vm_release(1374) called kvm=ffff80001b120000
[  274.821977] vgic_v4_init(267) called kvm=ffff80001b020000 doorbell=17548(64)
[  275.289483] kvm_create_vm(1236) called kvm=ffff80001c1a0000
[  275.303975] kvm_create_vm(1236) called kvm=ffff80001c1e0000
[  275.309712] kvm_vm_release(1374) called kvm=ffff80001c1e0000
[  275.406027] kvm_create_vm(1236) called kvm=ffff80001c240000
[  275.411756] kvm_vm_release(1374) called kvm=ffff80001c240000
[  275.914284] vgic_v4_init(267) called kvm=ffff80001c1a0000 doorbell=18636(64)
[  276.713109] kvm_create_vm(1236) called kvm=ffff80001d240000
[  276.721696] kvm_create_vm(1236) called kvm=ffff80001d280000
[  276.727421] kvm_vm_release(1374) called kvm=ffff80001d280000
[  276.734615] kvm_create_vm(1236) called kvm=ffff80001d2a0000
[  276.740342] kvm_vm_release(1374) called kvm=ffff80001d2a0000
[  277.341173] vgic_v4_init(267) called kvm=ffff80001d240000 doorbell=19724(64)
[  277.684981] kvm_create_vm(1236) called kvm=ffff80001e0a0000
[  277.694807] kvm_create_vm(1236) called kvm=ffff80001e0e0000
[  277.700544] kvm_vm_release(1374) called kvm=ffff80001e0e0000
[  277.706794] kvm_create_vm(1236) called kvm=ffff80001e120000
[  277.712519] kvm_vm_release(1374) called kvm=ffff80001e120000
[  278.158511] vgic_v4_init(267) called kvm=ffff80001e0a0000 doorbell=20812(64)
[  278.965615] kvm_create_vm(1236) called kvm=ffff80001f0c0000
[  278.973233] kvm_create_vm(1236) called kvm=ffff80001f0e0000
[  278.979624] kvm_vm_release(1374) called kvm=ffff80001f0e0000
[  278.988106] kvm_create_vm(1236) called kvm=ffff80001f120000
[  278.994484] kvm_vm_release(1374) called kvm=ffff80001f120000
[  279.408259] vgic_v4_init(267) called kvm=ffff80001f0c0000 doorbell=21900(64)
[  280.062592] kvm_create_vm(1236) called kvm=ffff8000300a0000
[  280.089029] kvm_create_vm(1236) called kvm=ffff8000300e0000
[  280.095744] kvm_vm_release(1374) called kvm=ffff8000300e0000
[  280.102889] kvm_create_vm(1236) called kvm=ffff800030100000
[  280.108646] kvm_vm_release(1374) called kvm=ffff800030100000
[  280.590774] vgic_v4_init(267) called kvm=ffff8000300a0000 doorbell=22988(64)
[  280.905459] kvm_create_vm(1236) called kvm=ffff800033ac0000
[  280.914767] kvm_create_vm(1236) called kvm=ffff800033b00000
[  280.920505] kvm_vm_release(1374) called kvm=ffff800033b00000
[  280.926616] kvm_create_vm(1236) called kvm=ffff800033b20000
[  280.932330] kvm_vm_release(1374) called kvm=ffff800033b20000
[  281.583530] vgic_v4_init(267) called kvm=ffff800033ac0000 doorbell=24076(64)
[  282.689082] kvm_create_vm(1236) called kvm=ffff800036820000
[  282.699338] kvm_create_vm(1236) called kvm=ffff800036840000
[  282.705066] kvm_vm_release(1374) called kvm=ffff800036840000
[  282.711249] kvm_create_vm(1236) called kvm=ffff8000369c0000
[  282.716981] kvm_vm_release(1374) called kvm=ffff8000369c0000
[  283.247998] vgic_v4_init(267) called kvm=ffff800036820000 doorbell=25164(64)
[  284.104919] kvm_create_vm(1236) called kvm=ffff800050cc0000
[  284.120141] kvm_create_vm(1236) called kvm=ffff800050ce0000
[  284.126130] kvm_vm_release(1374) called kvm=ffff800050ce0000
[  284.132492] kvm_create_vm(1236) called kvm=ffff800050d20000
[  284.138297] kvm_vm_release(1374) called kvm=ffff800050d20000
[  284.646071] vgic_v4_init(267) called kvm=ffff800050cc0000 doorbell=26252(64)
[  285.211935] kvm_create_vm(1236) called kvm=ffff800051ea0000
[  285.314317] kvm_create_vm(1236) called kvm=ffff800051f40000
[  285.320257] kvm_vm_release(1374) called kvm=ffff800051f40000
[  285.326383] kvm_create_vm(1236) called kvm=ffff800051f80000
[  285.332193] kvm_vm_release(1374) called kvm=ffff800051f80000
[  285.852482] vgic_v4_init(267) called kvm=ffff800051ea0000 doorbell=27340(64)
[  286.285262] kvm_create_vm(1236) called kvm=ffff800052da0000
[  286.294007] kvm_create_vm(1236) called kvm=ffff800052dc0000
[  286.299752] kvm_vm_release(1374) called kvm=ffff800052dc0000
[  286.305854] kvm_create_vm(1236) called kvm=ffff800052e00000
[  286.311567] kvm_vm_release(1374) called kvm=ffff800052e00000
[  286.741362] vgic_v4_init(267) called kvm=ffff800052da0000 doorbell=28428(64)
[  287.413763] kvm_create_vm(1236) called kvm=ffff800053d00000
[  287.423054] kvm_create_vm(1236) called kvm=ffff800053d20000
[  287.428812] kvm_vm_release(1374) called kvm=ffff800053d20000
[  287.442072] kvm_create_vm(1236) called kvm=ffff800053d60000
[  287.447830] kvm_vm_release(1374) called kvm=ffff800053d60000
[  288.066585] vgic_v4_init(267) called kvm=ffff800053d00000 doorbell=29516(64)
[  288.581236] kvm_create_vm(1236) called kvm=ffff800054d20000
[  288.588253] kvm_create_vm(1236) called kvm=ffff800054d40000
[  288.594051] kvm_vm_release(1374) called kvm=ffff800054d40000
[  288.605777] kvm_create_vm(1236) called kvm=ffff800054d80000
[  288.611535] kvm_vm_release(1374) called kvm=ffff800054d80000
[  289.094571] vgic_v4_init(267) called kvm=ffff800054d20000 doorbell=30604(64)
[  289.795308] kvm_create_vm(1236) called kvm=ffff800055d80000
[  289.803028] kvm_create_vm(1236) called kvm=ffff800055da0000
[  289.809110] kvm_vm_release(1374) called kvm=ffff800055da0000
[  289.815422] kvm_create_vm(1236) called kvm=ffff800055de0000
[  289.821154] kvm_vm_release(1374) called kvm=ffff800055de0000
[  290.271484] vgic_v4_init(267) called kvm=ffff800055d80000 doorbell=31692(64)
[  290.589450] kvm_create_vm(1236) called kvm=ffff800056b40000
[  290.601373] kvm_create_vm(1236) called kvm=ffff800056b80000
[  290.607220] kvm_vm_release(1374) called kvm=ffff800056b80000
[  290.613773] kvm_create_vm(1236) called kvm=ffff800056bc0000
[  290.619505] kvm_vm_release(1374) called kvm=ffff800056bc0000
[  291.059917] vgic_v4_init(267) called kvm=ffff800056b40000 doorbell=32780(64)
[  291.801179] kvm_create_vm(1236) called kvm=ffff800057a60000
[  291.811298] kvm_create_vm(1236) called kvm=ffff800057aa0000
[  291.817055] kvm_vm_release(1374) called kvm=ffff800057aa0000
[  291.829798] kvm_create_vm(1236) called kvm=ffff800057ac0000
[  291.835522] kvm_vm_release(1374) called kvm=ffff800057ac0000
[  292.364622] vgic_v4_init(267) called kvm=ffff800057a60000 doorbell=33868(64)
[  517.942167] vgic_set_common_attr(263) called kvm=ffff8000303e0000
[  517.948415] vgic_v3_save_pending_tables(397) called kvm=ffff8000303e0000
[  517.955602] vgic_v3_save_pending_tables(448) called kvm=ffff8000303e0000
[  518.099696] kvm_vm_release(1374) called kvm=ffff8000303e0000
[  518.105775] vgic_set_common_attr(263) called kvm=ffff800031020000
[  518.112012] vgic_v3_save_pending_tables(397) called kvm=ffff800031020000
[  518.119195] vgic_v3_save_pending_tables(448) called kvm=ffff800031020000
[  518.126833] vgic_v4_teardown(323) started kvm=ffff8000303e0000 doorbell=140(64)
[  518.134677] vgic_v4_teardown(333) finished kvm=ffff8000303e0000 doorbell=140(64)
[  518.146755] kvm_vm_release(1374) called kvm=ffff800031020000
[  518.152869] vgic_v4_teardown(323) started kvm=ffff800031020000 doorbell=1228(64)
[  518.160829] vgic_v4_teardown(333) finished kvm=ffff800031020000 doorbell=1228(64)
[  518.723172] vgic_set_common_attr(263) called kvm=ffff8000325e0000
[  518.729460] vgic_v3_save_pending_tables(397) called kvm=ffff8000325e0000
[  518.736992] vgic_v3_save_pending_tables(448) called kvm=ffff8000325e0000
[  518.755085] kvm_vm_release(1374) called kvm=ffff8000325e0000
[  518.761731] vgic_v4_teardown(323) started kvm=ffff8000325e0000 doorbell=3404(64)
[  518.769946] vgic_v4_teardown(333) finished kvm=ffff8000325e0000 doorbell=3404(64)
[  519.506094] vgic_set_common_attr(263) called kvm=ffff800031ae0000
[  519.512335] vgic_v3_save_pending_tables(397) called kvm=ffff800031ae0000
[  519.519512] vgic_v3_save_pending_tables(448) called kvm=ffff800031ae0000
[  519.555086] kvm_vm_release(1374) called kvm=ffff800031ae0000
[  519.561047] vgic_v4_teardown(323) started kvm=ffff800031ae0000 doorbell=2316(64)
[  519.568990] vgic_v4_teardown(333) finished kvm=ffff800031ae0000 doorbell=2316(64)
[  519.762370] vgic_set_common_attr(263) called kvm=ffff8000330c0000
[  519.768607] vgic_v3_save_pending_tables(397) called kvm=ffff8000330c0000
[  519.775782] vgic_v3_save_pending_tables(448) called kvm=ffff8000330c0000
[  519.788056] kvm_vm_release(1374) called kvm=ffff8000330c0000
[  519.794232] vgic_v4_teardown(323) started kvm=ffff8000330c0000 doorbell=4492(64)
[  519.802317] vgic_v4_teardown(333) finished kvm=ffff8000330c0000 doorbell=4492(64)
[  520.182176] vgic_set_common_attr(263) called kvm=ffff800033c80000
[  520.188417] vgic_v3_save_pending_tables(397) called kvm=ffff800033c80000
[  520.195584] vgic_v3_save_pending_tables(448) called kvm=ffff800033c80000
[  520.207328] kvm_vm_release(1374) called kvm=ffff800033c80000
[  520.213361] vgic_v4_teardown(323) started kvm=ffff800033c80000 doorbell=5580(64)
[  520.220387] vgic_set_common_attr(263) called kvm=ffff800034a20000
[  520.221321] vgic_v4_teardown(333) finished kvm=ffff800033c80000 doorbell=5580(64)
[  520.227161] vgic_v3_save_pending_tables(397) called kvm=ffff800034a20000
[  520.242001] vgic_v3_save_pending_tables(448) called kvm=ffff800034a20000
[  520.251585] kvm_vm_release(1374) called kvm=ffff800034a20000
[  520.257593] vgic_v4_teardown(323) started kvm=ffff800034a20000 doorbell=6668(64)
[  520.265558] vgic_v4_teardown(333) finished kvm=ffff800034a20000 doorbell=6668(64)
[  521.395539] vgic_set_common_attr(263) called kvm=ffff8000357a0000
[  521.401782] vgic_v3_save_pending_tables(397) called kvm=ffff8000357a0000
[  521.408953] vgic_v3_save_pending_tables(448) called kvm=ffff8000357a0000
[  521.418300] kvm_vm_release(1374) called kvm=ffff8000357a0000
[  521.424240] vgic_v4_teardown(323) started kvm=ffff8000357a0000 doorbell=7756(64)
[  521.432182] vgic_v4_teardown(333) finished kvm=ffff8000357a0000 doorbell=7756(64)
[  525.002673] vgic_set_common_attr(263) called kvm=ffff8000367a0000
[  525.008914] vgic_v3_save_pending_tables(397) called kvm=ffff8000367a0000
[  525.016143] vgic_v3_save_pending_tables(448) called kvm=ffff8000367a0000
[  525.027230] kvm_vm_release(1374) called kvm=ffff8000367a0000
[  525.033257] vgic_v4_teardown(323) started kvm=ffff8000367a0000 doorbell=8844(64)
[  525.041195] vgic_v4_teardown(333) finished kvm=ffff8000367a0000 doorbell=8844(64)
[  525.090124] vgic_set_common_attr(263) called kvm=ffff8000376e0000
[  525.096372] vgic_v3_save_pending_tables(397) called kvm=ffff8000376e0000
[  525.103612] vgic_v3_save_pending_tables(448) called kvm=ffff8000376e0000
[  525.110594] irq: irqd_set_activated: CPU49 IRQ9940 lost IRQD_IRQ_INPROGRESS old=0x10401400 new=0x10401600, expected=0x10441600 kvm=ffff8000376e0000
[  525.124122] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS set time [525.110593]
[  525.132397] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS clr time [525.110595]
[  525.143493] kvm_vm_release(1374) called kvm=ffff8000376e0000
[  525.149518] vgic_v4_teardown(323) started kvm=ffff8000376e0000 doorbell=9932(64)
[  525.157503] vgic_v4_teardown(333) finished kvm=ffff8000376e0000 doorbell=9932(64)
[  529.649331] vgic_set_common_attr(263) called kvm=ffff80003fec0000
[  529.655575] vgic_v3_save_pending_tables(397) called kvm=ffff80003fec0000
[  529.662767] vgic_v3_save_pending_tables(448) called kvm=ffff80003fec0000
[  529.672309] kvm_vm_release(1374) called kvm=ffff80003fec0000
[  529.678306] vgic_v4_teardown(323) started kvm=ffff80003fec0000 doorbell=11020(64)
[  529.686330] vgic_v4_teardown(333) finished kvm=ffff80003fec0000 doorbell=11020(64)
[  530.424380] vgic_set_common_attr(263) called kvm=ffff800050f80000
[  530.430619] vgic_v3_save_pending_tables(397) called kvm=ffff800050f80000
[  530.437805] vgic_v3_save_pending_tables(448) called kvm=ffff800050f80000
[  530.469006] kvm_vm_release(1374) called kvm=ffff800050f80000
[  530.476493] vgic_v4_teardown(323) started kvm=ffff800050f80000 doorbell=12108(64)
[  530.485410] vgic_v4_teardown(333) finished kvm=ffff800050f80000 doorbell=12108(64)
[  533.584697] vgic_set_common_attr(263) called kvm=ffff800016080000
[  533.590950] vgic_v3_save_pending_tables(397) called kvm=ffff800016080000
[  533.598124] vgic_v3_save_pending_tables(448) called kvm=ffff800016080000
[  533.610606] kvm_vm_release(1374) called kvm=ffff800016080000
[  533.616952] vgic_v4_teardown(323) started kvm=ffff800016080000 doorbell=13196(64)
[  533.625146] vgic_v4_teardown(333) finished kvm=ffff800016080000 doorbell=13196(64)
[  533.725021] vgic_set_common_attr(263) called kvm=ffff8000179a0000
[  533.731258] vgic_v3_save_pending_tables(397) called kvm=ffff8000179a0000
[  533.738444] vgic_v3_save_pending_tables(448) called kvm=ffff8000179a0000
[  533.758156] kvm_vm_release(1374) called kvm=ffff8000179a0000
[  533.764364] vgic_v4_teardown(323) started kvm=ffff8000179a0000 doorbell=14284(64)
[  533.772633] vgic_v4_teardown(333) finished kvm=ffff8000179a0000 doorbell=14284(64)
[  535.981082] vgic_set_common_attr(263) called kvm=ffff800018920000
[  535.987328] vgic_v3_save_pending_tables(397) called kvm=ffff800018920000
[  535.994532] vgic_v3_save_pending_tables(448) called kvm=ffff800018920000
[  536.001494] irq: irqd_set_activated: CPU19 IRQ15375 lost IRQD_IRQ_INPROGRESS old=0x10441400 new=0x10441600, expected=0x10401600 kvm=ffff800018920000
[  536.015125] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS set time [536.001493]
[  536.023400] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS clr time [536.001493]
[  536.052598] kvm_vm_release(1374) called kvm=ffff800018920000
[  536.059510] vgic_v4_teardown(323) started kvm=ffff800018920000 doorbell=15372(64)
[  536.512174] vgic_set_common_attr(263) called kvm=ffff80001b020000
[  536.518413] vgic_v3_save_pending_tables(397) called kvm=ffff80001b020000
[  536.525626] vgic_v3_save_pending_tables(448) called kvm=ffff80001b020000
[  536.561496] kvm_vm_release(1374) called kvm=ffff80001b020000
[  536.568194] vgic_v4_teardown(323) started kvm=ffff80001b020000 doorbell=17548(64)
[  536.576982] vgic_v4_teardown(333) finished kvm=ffff80001b020000 doorbell=17548(64)
[  537.657725] vgic_set_common_attr(263) called kvm=ffff800019a80000
[  537.663958] vgic_v3_save_pending_tables(397) called kvm=ffff800019a80000
[  537.671130] vgic_v3_save_pending_tables(448) called kvm=ffff800019a80000
[  537.711212] kvm_vm_release(1374) called kvm=ffff800019a80000
[  537.718230] vgic_v4_teardown(323) started kvm=ffff800019a80000 doorbell=16460(64)
[  537.727162] vgic_v4_teardown(333) finished kvm=ffff800019a80000 doorbell=16460(64)
[  538.602096] vgic_set_common_attr(263) called kvm=ffff80001c1a0000
[  538.608378] vgic_v3_save_pending_tables(397) called kvm=ffff80001c1a0000
[  538.615831] vgic_v3_save_pending_tables(448) called kvm=ffff80001c1a0000
[  538.627140] kvm_vm_release(1374) called kvm=ffff80001c1a0000
[  538.633337] vgic_v4_teardown(323) started kvm=ffff80001c1a0000 doorbell=18636(64)
[  538.641470] vgic_v4_teardown(333) finished kvm=ffff80001c1a0000 doorbell=18636(64)
[  541.777190] vgic_set_common_attr(263) called kvm=ffff80001d240000
[  541.783431] vgic_v3_save_pending_tables(397) called kvm=ffff80001d240000
[  541.790748] vgic_v3_save_pending_tables(448) called kvm=ffff80001d240000
[  541.801177] kvm_vm_release(1374) called kvm=ffff80001d240000
[  541.807262] vgic_v4_teardown(323) started kvm=ffff80001d240000 doorbell=19724(64)
[  541.815301] vgic_v4_teardown(333) finished kvm=ffff80001d240000 doorbell=19724(64)
[  545.079117] vgic_set_common_attr(263) called kvm=ffff80001e0a0000
[  545.085358] vgic_v3_save_pending_tables(397) called kvm=ffff80001e0a0000
[  545.092580] vgic_v3_save_pending_tables(448) called kvm=ffff80001e0a0000
[  545.099562] irq: irqd_set_activated: CPU49 IRQ20821 lost IRQD_IRQ_INPROGRESS old=0x10401400 new=0x10401600, expected=0x10441600 kvm=ffff80001e0a0000
[  545.113177] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS set time [545.099561]
[  545.121454] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS clr time [545.099562]
[  545.129755] irq: irqd_set_activated: CPU49 IRQ20826 lost IRQD_IRQ_INPROGRESS old=0x10441400 new=0x10441600, expected=0x10401600 kvm=ffff80001e0a0000
[  545.143365] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS set time [545.129754]
[  545.151654] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS clr time [545.129755]
[  545.163250] kvm_vm_release(1374) called kvm=ffff80001e0a0000
[  545.169204] vgic_v4_teardown(323) started kvm=ffff80001e0a0000 doorbell=20812(64)
[  557.073632] rcu: INFO: rcu_sched self-detected stall on CPU
[  557.079331] rcu:     48-....: (5250 ticks this GP) idle=399c/1/0x4000000000000000 softirq=36358/36358 fqs=2052
[  557.089294]  (t=5254 jiffies g=106321 q=10324 ncpus=72)
[  557.094636] CPU: 48 PID: 1631 Comm: qemu-system-aar Kdump: loaded Not tainted 6.2.0-rc4 #11
[  557.103176] Hardware name: NVIDIA Ranger/th500_silicon, BIOS v1.1.1-4638c6c9 01/17/2023
[  557.111359] pstate: 63400009 (nZCv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
[  557.118478] pc : __synchronize_hardirq+0x48/0x154
[  557.123294] lr : free_irq+0x134/0x460
[  557.127034] sp : ffff800018b6f990
[  557.130417] x29: ffff800018b6f990 x28: 000000000000000d x27: 0000000000000001
[  557.137713] x26: 0000000000003c0f x25: ffff0000c82aa8a0 x24: 0000000000000001
[  557.145008] x23: ffff0000836e0cf4 x22: 0000000000000000 x21: 0000000000000001
[  557.152303] x20: ffff0000836e0cf4 x19: ffff0000836e0c60 x18: ffff800018b6f510
[  557.159598] x17: 373335313d6c6c65 x16: 62726f6f64203030 x15: 3030323938313030
[  557.166893] x14: 3038666666663d6d x13: 2934362832373335 x12: 313d6c6c6562726f
[  557.174189] x11: 0000000000000040 x10: ffff800009230a58 x9 : 0000000000000000
[  557.181484] x8 : ffff800018b6f8e0 x7 : 0000000000000000 x6 : 0000000000000001
[  557.188779] x5 : ffff800008355054 x4 : ffffffc0012c5760 x3 : 0000000081550151
[  557.196075] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000010071200
[  557.203371] Call trace:
[  557.205865]  __synchronize_hardirq+0x48/0x154
[  557.210317]  free_irq+0x134/0x460
[  557.213700]  vgic_v4_teardown+0xdc/0x138
[  557.217712]  __kvm_vgic_destroy+0x18c/0x194
[  557.221985]  kvm_vgic_destroy+0x28/0x3c
[  557.225902]  kvm_arch_destroy_vm+0x24/0x80
[  557.230087]  kvm_destroy_vm+0x158/0x2f8
[  557.234006]  kvm_put_kvm+0x60/0x80
[  557.237479]  kvm_vm_release+0x44/0x54
[  557.241221]  __fput+0x70/0x250
[  557.244342]  ____fput+0x10/0x20
[  557.247548]  task_work_run+0xc8/0x178
[  557.251290]  do_exit+0x2c8/0x8e0
[  557.254589]  do_group_exit+0x34/0xa0
[  557.258239]  get_signal+0x8bc/0x8e4
[  557.261801]  do_notify_resume+0x144/0x1220
[  557.265989]  el0_svc+0xdc/0xf0
[  557.269109]  el0t_64_sync_handler+0xf0/0x11c
[  557.273470]  el0t_64_sync+0x190/0x194
[  566.921633] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [qemu-system-aar:1976]
[  566.929560] Modules linked in:
[  566.932679] CPU: 0 PID: 1976 Comm: qemu-system-aar Kdump: loaded Not tainted 6.2.0-rc4 #11
[  566.941129] Hardware name: NVIDIA Ranger/th500_silicon, BIOS v1.1.1-4638c6c9 01/17/2023
[  566.949312] pstate: 63400009 (nZCv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
[  566.956429] pc : __synchronize_hardirq+0x54/0x154
[  566.961240] lr : free_irq+0x134/0x460
[  566.964980] sp : ffff80003252f990
[  566.968362] x29: ffff80003252f990 x28: 000000000000000d x27: 0000000000000001
[  566.975658] x26: 000000000000515a x25: ffff0000c56ace70 x24: 0000000000000001
[  566.982954] x23: ffff0000cf1126f4 x22: 0000000000000000 x21: 0000000000000001
[  566.990249] x20: ffff0000cf1126f4 x19: ffff0000cf112660 x18: ffff80003252f510
[  566.997545] x17: 313830323d6c6c65 x16: 62726f6f64203030 x15: 3030613065313030
[  567.004840] x14: 3038666666663d6d x13: 0000000000000000 x12: 0000000000000006
[  567.012136] x11: 0000000000000040 x10: ffff800009230a58 x9 : 0000000000000000
[  567.019431] x8 : ffff80003252f8e0 x7 : 0000000000000000 x6 : 0000000000000001
[  567.026727] x5 : ffff800008355054 x4 : ffffffc0019232e0 x3 : 0000000081550151
[  567.034023] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000010071200
[  567.041318] Call trace:
[  567.043812]  __synchronize_hardirq+0x54/0x154
[  567.048264]  free_irq+0x134/0x460
[  567.051648]  vgic_v4_teardown+0xdc/0x138
[  567.055655]  __kvm_vgic_destroy+0x18c/0x194
[  567.059928]  kvm_vgic_destroy+0x28/0x3c
[  567.063844]  kvm_arch_destroy_vm+0x24/0x80
[  567.068028]  kvm_destroy_vm+0x158/0x2f8
[  567.071946]  kvm_put_kvm+0x60/0x80
[  567.075418]  kvm_vm_release+0x44/0x54
[  567.079158]  __fput+0x70/0x250
[  567.082275]  ____fput+0x10/0x20
[  567.085480]  task_work_run+0xc8/0x178
[  567.089220]  do_exit+0x2c8/0x8e0
[  567.092515]  do_group_exit+0x34/0xa0
[  567.096166]  get_signal+0x8bc/0x8e4
[  567.099728]  do_notify_resume+0x144/0x1220
[  567.103912]  el0_svc+0xdc/0xf0
[  567.107030]  el0t_64_sync_handler+0xf0/0x11c
[  567.111392]  el0t_64_sync+0x190/0x194
[  567.115134] Kernel panic - not syncing: softlockup: hung tasks
[  567.121097] CPU: 0 PID: 1976 Comm: qemu-system-aar Kdump: loaded Tainted: G             L     6.2.0-rc4 #11
[  567.131058] Hardware name: NVIDIA Ranger/th500_silicon, BIOS v1.1.1-4638c6c9 01/17/2023
[  567.139241] Call trace:
[  567.141734]  dump_backtrace.part.0+0xdc/0xe8
[  567.146096]  show_stack+0x18/0x20
[  567.149479]  dump_stack_lvl+0x64/0x7c
[  567.153218]  dump_stack+0x18/0x30
[  567.156602]  panic+0x178/0x340
[  567.159718]  watchdog_timer_fn+0x28c/0x298
[  567.163907]  __hrtimer_run_queues+0x118/0x260
[  567.168358]  hrtimer_interrupt+0x114/0x31c
[  567.172541]  arch_timer_handler_phys+0x2c/0x40
[  567.177083]  handle_percpu_devid_irq+0x84/0x138
[  567.181710]  generic_handle_domain_irq+0x2c/0x40
[  567.186427]  gic_handle_irq+0x50/0x140
[  567.190257]  call_on_irq_stack+0x2c/0x38
[  567.194262]  do_interrupt_handler+0x7c/0x88
[  567.198535]  el1_interrupt+0x34/0x58
[  567.202185]  el1h_64_irq_handler+0x18/0x20
[  567.206369]  el1h_64_irq+0x64/0x68
[  567.209840]  __synchronize_hardirq+0x54/0x154
[  567.214291]  free_irq+0x134/0x460
[  567.217675]  vgic_v4_teardown+0xdc/0x138
[  567.221681]  __kvm_vgic_destroy+0x18c/0x194
[  567.225953]  kvm_vgic_destroy+0x28/0x3c
[  567.229870]  kvm_arch_destroy_vm+0x24/0x80
[  567.234053]  kvm_destroy_vm+0x158/0x2f8
[  567.237970]  kvm_put_kvm+0x60/0x80
[  567.241443]  kvm_vm_release+0x44/0x54
[  567.245182]  __fput+0x70/0x250
[  567.248298]  ____fput+0x10/0x20
[  567.251503]  task_work_run+0xc8/0x178
[  567.255243]  do_exit+0x2c8/0x8e0
[  567.258538]  do_group_exit+0x34/0xa0
[  567.262189]  get_signal+0x8bc/0x8e4
[  567.265750]  do_notify_resume+0x144/0x1220
[  567.269935]  el0_svc+0xdc/0xf0
[  567.273052]  el0t_64_sync_handler+0xf0/0x11c
[  567.277414]  el0t_64_sync+0x190/0x194
[  567.281161] SMP: stopping secondary CPUs
  

On Fri, 20 Jan 2023 05:02:15 +0000,
Shanker Donthineni <sdonthineni@nvidia.com> wrote:
> 
> [1  <text/plain; UTF-8 (7bit)>]
> Hi Marc,
> 
> On 1/19/23 08:01, Marc Zyngier wrote:
> >> On 1/19/23 01:11, Marc Zyngier wrote:
> >>> So you can see the VM being torn down while the vgic save sequence is
> >>> still in progress?
> >>> 
> >>> If you can actually see that, then this is a much bigger bug than the
> >>> simple race you are describing, and we're missing a reference on the
> >>> kvm structure. This would be a*MAJOR*  bug.
> >>> 
> >> How do we know vGIC save sequence is in progress while VM is being
> >> teardown?  I'm launching/terminating ~32 VMs in a loop to reproduce
> >> the issue.
> > Errr...*you*  know when you are issuing the save ioctl, right? You
> > also know when you are terminating the VM (closing its fd or killing
> > the VMM).
> > 
> 
> Added debug statements to trace the code patch, and tagged each log message
> with 'struct kvm *'. Attached the complete kernel log messages including
> debug messages.
> 
> All 32 VMs launched, time 258s to 291s
>  [  258.519837] kvm_create_vm(1236) called kvm=ffff8000303e0000  --> 1st VM
>  ...
>  [  291.801179] kvm_create_vm(1236) called kvm=ffff800057a60000   --> 32nd VM
> 
> Test script inside VM issues poweroff command after sleeping 200sec.
> 
> Working case kvm=ffff8000303e0000:
> 
>  $ cat gicv4-debug.txt | grep ffff8000303e0000
>   [  258.519837] kvm_create_vm(1236) called kvm=ffff8000303e0000
>   [  258.667101] vgic_v4_init(267) called kvm=ffff8000303e0000 doorbell=140(64)
>   [  517.942167] vgic_set_common_attr(263) called kvm=ffff8000303e0000
>   [  517.948415] vgic_v3_save_pending_tables(397) called kvm=ffff8000303e0000
>   [  517.955602] vgic_v3_save_pending_tables(448) called kvm=ffff8000303e0000
>   [  518.099696] kvm_vm_release(1374) called kvm=ffff8000303e0000
>   [  518.126833] vgic_v4_teardown(323) started kvm=ffff8000303e0000 doorbell=140(64)
>   [  518.134677] vgic_v4_teardown(333) finished kvm=ffff8000303e0000 doorbell=140(64)
> 
> Not working case kvm=ffff80001e0a0000:
> 
>  $ cat gicv4-debug.txt | grep ffff80001e0a0000
>   [  277.684981] kvm_create_vm(1236) called kvm=ffff80001e0a0000
>   [  278.158511] vgic_v4_init(267) called kvm=ffff80001e0a0000 doorbell=20812(64)
>   [  545.079117] vgic_set_common_attr(263) called kvm=ffff80001e0a0000
>   [  545.085358] vgic_v3_save_pending_tables(397) called kvm=ffff80001e0a0000
>   [  545.092580] vgic_v3_save_pending_tables(448) called kvm=ffff80001e0a0000
>   [  545.099562] irq: irqd_set_activated: CPU49 IRQ20821 lost IRQD_IRQ_INPROGRESS old=0x10401400 new=0x10401600, expected=0x10441600 kvm=ffff80001e0a0000^M
>   [  545.113177] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS set time [545.099561]^M
>   [  545.121454] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS clr time [545.099562]^M
>   [  545.129755] irq: irqd_set_activated: CPU49 IRQ20826 lost IRQD_IRQ_INPROGRESS old=0x10441400 new=0x10441600, expected=0x10401600 kvm=ffff80001e0a0000^M
>   [  545.143365] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS set time [545.129754]^M
>   [  545.151654] irq: irqd_set_activated: IRQD_IRQ_INPROGRESS clr time [545.129755]^M
>   [  545.163250] kvm_vm_release(1374) called kvm=ffff80001e0a0000
>   [  545.169204] vgic_v4_teardown(323) started kvm=ffff80001e0a0000 doorbell=20812(64)
> 
> IRQD_IRQ_INPROGRESS is corrupted before calling kvm_vm_release(),

You keep missing my point. Yes, we have established the interrupt race
and have a way to fix it, let's move on...

What I am asking agin is: is there any overlap between any vgic ioctl
and the teardown of the VM? Do you ever see kvm_vm_release() being
called before kvm_device_release()?

Because that's the overlap I've been talking all along.

	M.
  

On 1/20/23 06:00, Marc Zyngier wrote:
> What I am asking agin is: is there any overlap between any vgic ioctl
> and the teardown of the VM? Do you ever see kvm_vm_release() being
> called before kvm_device_release()?
> 
> Because that's the overlap I've been talking all along.

I've not observed overlap of kvm_vm_release() with ioctls kvm_device_ioctl(),
kvm_vcpu_ioctl() and kvm_vm_ioctl() for 3hrs. Test created hundreds of VMs and
teardowns.


Debug code:
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 4f26b244f6d0..9c8409a894f5 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -800,7 +800,7 @@ struct kvm {
         bool dirty_ring_with_bitmap;
         bool vm_bugged;
         bool vm_dead;
-
+       atomic_t checkoverlap;
  #ifdef CONFIG_HAVE_KVM_PM_NOTIFIER
         struct notifier_block pm_notifier;
  #endif
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 9c60384b5ae0..a5a9071aeb19 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -1370,9 +1370,13 @@ static int kvm_vm_release(struct inode *inode, struct file *filp)
  {
         struct kvm *kvm = filp->private_data;

+       atomic_inc(&kvm->checkoverlap);
+       pr_err("%s(%d) start kvm=%px\n",__func__,__LINE__, kvm);
+
         kvm_irqfd_release(kvm);

         kvm_put_kvm(kvm);
+       pr_err("%s(%d) end kvm=%px\n",__func__,__LINE__, kvm);
         return 0;
  }

@@ -4073,6 +4077,8 @@ static long kvm_vcpu_ioctl(struct file *filp,
         if (unlikely(_IOC_TYPE(ioctl) != KVMIO))
                 return -EINVAL;

+if (atomic_read(&vcpu->kvm->checkoverlap)) pr_err("%s(%d) overlap kvm=%px\n", __func__, __LINE__, vcpu->kvm);
+
         /*
          * Some architectures have vcpu ioctls that are asynchronous to vcpu
          * execution; mutex_lock() would break them.
@@ -4346,6 +4352,8 @@ static long kvm_device_ioctl(struct file *filp, unsigned int ioctl,
         if (dev->kvm->mm != current->mm || dev->kvm->vm_dead)
                 return -EIO;

+       if (atomic_read(&dev->kvm->checkoverlap)) pr_err("%s(%d) overlap kvm=%px\n", __func__, __LINE__, dev->kvm);
+
         switch (ioctl) {
         case KVM_SET_DEVICE_ATTR:
                 return kvm_device_ioctl_attr(dev, dev->ops->set_attr, arg);
@@ -4731,6 +4739,8 @@ static long kvm_vm_ioctl(struct file *filp,

         if (kvm->mm != current->mm || kvm->vm_dead)
                 return -EIO;
+if (atomic_read(&kvm->checkoverlap)) pr_err("%s(%d) overlap kvm=%px\n", __func__, __LINE__, kvm);
+
         switch (ioctl) {
         case KVM_CREATE_VCPU:
                 r = kvm_vm_ioctl_create_vcpu(kvm, arg);
  

On 1/20/23 06:00, Marc Zyngier wrote:
> What I am asking agin is: is there any overlap between any vgic ioctl
> and the teardown of the VM? Do you ever see kvm_vm_release() being
> called before kvm_device_release()?
> 
> Because that's the overlap I've been talking all along.

I've not observed overlap of kvm_vm_release() with ioctls kvm_device_ioctl(),
kvm_vcpu_ioctl() and kvm_vm_ioctl() for 3hrs. Test created hundreds of VMs and
teardowns.


Debug code:
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 4f26b244f6d0..9c8409a894f5 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -800,7 +800,7 @@ struct kvm {
         bool dirty_ring_with_bitmap;
         bool vm_bugged;
         bool vm_dead;
-
+       atomic_t checkoverlap;
  #ifdef CONFIG_HAVE_KVM_PM_NOTIFIER
         struct notifier_block pm_notifier;
  #endif
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 9c60384b5ae0..a5a9071aeb19 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -1370,9 +1370,13 @@ static int kvm_vm_release(struct inode *inode, struct file *filp)
  {
         struct kvm *kvm = filp->private_data;

+       atomic_inc(&kvm->checkoverlap);
+       pr_err("%s(%d) start kvm=%px\n",__func__,__LINE__, kvm);
+
         kvm_irqfd_release(kvm);

         kvm_put_kvm(kvm);
+       pr_err("%s(%d) end kvm=%px\n",__func__,__LINE__, kvm);
         return 0;
  }

@@ -4073,6 +4077,8 @@ static long kvm_vcpu_ioctl(struct file *filp,
         if (unlikely(_IOC_TYPE(ioctl) != KVMIO))
                 return -EINVAL;

+if (atomic_read(&vcpu->kvm->checkoverlap)) pr_err("%s(%d) overlap kvm=%px\n", __func__, __LINE__, vcpu->kvm);
+
         /*
          * Some architectures have vcpu ioctls that are asynchronous to vcpu
          * execution; mutex_lock() would break them.
@@ -4346,6 +4352,8 @@ static long kvm_device_ioctl(struct file *filp, unsigned int ioctl,
         if (dev->kvm->mm != current->mm || dev->kvm->vm_dead)
                 return -EIO;

+       if (atomic_read(&dev->kvm->checkoverlap)) pr_err("%s(%d) overlap kvm=%px\n", __func__, __LINE__, dev->kvm);
+
         switch (ioctl) {
         case KVM_SET_DEVICE_ATTR:
                 return kvm_device_ioctl_attr(dev, dev->ops->set_attr, arg);
@@ -4731,6 +4739,8 @@ static long kvm_vm_ioctl(struct file *filp,

         if (kvm->mm != current->mm || kvm->vm_dead)
                 return -EIO;
+if (atomic_read(&kvm->checkoverlap)) pr_err("%s(%d) overlap kvm=%px\n", __func__, __LINE__, kvm);
+
         switch (ioctl) {
         case KVM_CREATE_VCPU:
                 r = kvm_vm_ioctl_create_vcpu(kvm, arg);
  

On Sat, 21 Jan 2023 15:35:48 +0000,
Shanker Donthineni <sdonthineni@nvidia.com> wrote:
> 
> 
> 
> On 1/20/23 06:00, Marc Zyngier wrote:
> > What I am asking agin is: is there any overlap between any vgic ioctl
> > and the teardown of the VM? Do you ever see kvm_vm_release() being
> > called before kvm_device_release()?
> > 
> > Because that's the overlap I've been talking all along.
> 
> I've not observed overlap of kvm_vm_release() with ioctls
> kvm_device_ioctl(), kvm_vcpu_ioctl() and kvm_vm_ioctl() for
> 3hrs. Test created hundreds of VMs and teardowns.

OK, that's reassuring. Thanks for looking into this.

	M.