From patchwork Sat Nov 12 13:23:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juergen Gross X-Patchwork-Id: 19174 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1269571wru; Sat, 12 Nov 2022 05:29:57 -0800 (PST) X-Google-Smtp-Source: AA0mqf72V5lfipEq3DkBOB0O4m/oBlGPWqh5F3dV4k8otUHiHzeB9zZmxSx/kr17TAGM5Nxxb4sY X-Received: by 2002:a63:4a55:0:b0:46b:27b0:c245 with SMTP id j21-20020a634a55000000b0046b27b0c245mr5347555pgl.611.1668259797566; Sat, 12 Nov 2022 05:29:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668259797; cv=none; d=google.com; s=arc-20160816; b=gbPD7Oj4CHaz93wc2dYR/AXOTM6I640+bFczsAMt1wJkOfLWRHeBNi+pwRqZFCL+I4 1/ph+yAYybbtmlynw8mO+WGt9vBWuoXeuELLSn8R32Y1dEv9CLdjNf3R8+9MZG/T68lh MDjvDEPaAVQDOXseTICQIZ7k/pLqjCHDuHvIGMx2hC7bZH0RP5ViCQs/AEHRXJ4GwlYT smMIfObvcUUGqMac3JpcFvO+za7y1hUEqb4CimDdGAd99Lf/2+1a7zqQ2/WX5EZjeU8Y C5tf3QLrefa5EKzgzgh/yfjRW/7eXnwnGVv89Tfy6+ogDBADq4jtEOxiorI1kps/MmZ6 lsrQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=Lzs4s6ThfI/Cryj0DS92Vtk9nJ4GBe5JTPtbjjKk9HI=; b=n+24Wdjc8ar6Rm6hXkQbBeIjbFVGiZrJ5hBlT6wRPsLwm4YCVzsCfhIIE04mG+gwxU LqRfm2fT8Kn7Q+kdsMgIZWP1KU21+2Wwcmt9BXX5IFNWmzdNGKZHkyk17CIpFFTwycGo XuYOn/nNY8MI65d3YHhTUAiKo72kXQM9zNqvat4R7f96Fl0wvQGX55O3j8nTGdSjHClV 1YjZYag9eTsHw8Eb4kpdLOxPioioInn3DpflP4GoVZC6l9h/jDJhiPBy+NOAECtgxYi9 KJCbSTN+XyqA3Al+PeEVrqvEM9XDJ101g8kaQqOBjsnuWen2S8x0c16cOWTfwX1bmhjE +Tpg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=dy+yrA7+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t185-20020a6381c2000000b00437ec09f591si4983050pgd.135.2022.11.12.05.29.39; Sat, 12 Nov 2022 05:29:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.com header.s=susede1 header.b=dy+yrA7+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=suse.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234292AbiKLNXo (ORCPT + 99 others); Sat, 12 Nov 2022 08:23:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230383AbiKLNXl (ORCPT ); Sat, 12 Nov 2022 08:23:41 -0500 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 224E6DF96 for ; Sat, 12 Nov 2022 05:23:40 -0800 (PST) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 8FE111F890; Sat, 12 Nov 2022 13:23:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1668259418; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=Lzs4s6ThfI/Cryj0DS92Vtk9nJ4GBe5JTPtbjjKk9HI=; b=dy+yrA7+wJLuK/hi2++hyzE90Av1bO8emPl4vdobeaup1lfBoKcNPnR5Xi/ex2wJE7jgwb ZiEDMqw8d340OavTInDgcs710dqR516tS0v76gSCQd1wuP51tcu+jb5beZkShn2qFuzoIj WvV38uMkQ0R6DiiV3RUYKd75TSHf3Kk= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 22A4C13A08; Sat, 12 Nov 2022 13:23:38 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id bWkWB1qeb2MyGQAAMHmgww (envelope-from ); Sat, 12 Nov 2022 13:23:38 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Juergen Gross , Stefano Stabellini , Russell King , Boris Ostrovsky , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Oleksandr Tyshchenko , xen-devel@lists.xenproject.org, linux-arm-kernel@lists.infradead.org, Jan Beulich Subject: [PATCH] x86/xen: call xen_set_upcall_vector() only after local apic setup Date: Sat, 12 Nov 2022 14:23:36 +0100 Message-Id: <20221112132336.698-1-jgross@suse.com> X-Mailer: git-send-email 2.35.3 MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749297185923481584?= X-GMAIL-MSGID: =?utf-8?q?1749297185923481584?= Calling xen_set_upcall_vector() before the local APIC has been initialized results in warnings when setting up the local APIC: [ 0.037322] APIC: Stale IRR: 00080000, ... Avoid this problem by moving the call of xen_set_upcall_vector() for APs from xen_cpu_up_prepare_hvm() to a new hotplug callback called on the AP before interrupts are enabled and after the local APIC has been initialized. There is already a Xen specific hotplug state defined at a matching position, but right now it is specific to Arm. So rename the CPUHP_AP_ARM_XEN_STARTING to CPUHP_AP_XEN_STARTING and use it for x86, too. On the boot processor use the x86_init.irqs.intr_mode_init hook to call xen_set_upcall_vector() only after the local APIC initialization. As interrupts are on already at the time when the hook is being used, modify xen_init_setup_upcall_vector() to disable interrupts. This is needed as the xen_percpu_upcall flag is set only after the callback has been registered, and the flag is needed for correctly acknowledging an interrupt. Fixes: b1c3497e604d ("x86/xen: Add support for HVMOP_set_evtchn_upcall_vector") Reported-by: Jan Beulich Signed-off-by: Juergen Gross --- arch/arm/xen/enlighten.c | 4 ++-- arch/x86/xen/enlighten_hvm.c | 9 --------- drivers/xen/events/events_base.c | 22 +++++++++++++++++----- include/linux/cpuhotplug.h | 2 +- 4 files changed, 20 insertions(+), 17 deletions(-) diff --git a/arch/arm/xen/enlighten.c b/arch/arm/xen/enlighten.c index 93c8ccbf2982..1778a7a7c3ed 100644 --- a/arch/arm/xen/enlighten.c +++ b/arch/arm/xen/enlighten.c @@ -529,8 +529,8 @@ static int __init xen_guest_init(void) if (xen_initial_domain()) pvclock_gtod_register_notifier(&xen_pvclock_gtod_notifier); - return cpuhp_setup_state(CPUHP_AP_ARM_XEN_STARTING, - "arm/xen:starting", xen_starting_cpu, + return cpuhp_setup_state(CPUHP_AP_XEN_STARTING, + "xen:starting", xen_starting_cpu, xen_dying_cpu); } early_initcall(xen_guest_init); diff --git a/arch/x86/xen/enlighten_hvm.c b/arch/x86/xen/enlighten_hvm.c index c1cd28e915a3..55d140cc6128 100644 --- a/arch/x86/xen/enlighten_hvm.c +++ b/arch/x86/xen/enlighten_hvm.c @@ -177,15 +177,6 @@ static int xen_cpu_up_prepare_hvm(unsigned int cpu) if (!xen_have_vector_callback) return 0; - if (xen_percpu_upcall) { - rc = xen_set_upcall_vector(cpu); - if (rc) { - WARN(1, "HVMOP_set_evtchn_upcall_vector" - " for CPU %d failed: %d\n", cpu, rc); - return rc; - } - } - if (xen_feature(XENFEAT_hvm_safe_pvclock)) xen_setup_timer(cpu); diff --git a/drivers/xen/events/events_base.c b/drivers/xen/events/events_base.c index c443f04aaad7..7609fedb2a33 100644 --- a/drivers/xen/events/events_base.c +++ b/drivers/xen/events/events_base.c @@ -39,6 +39,7 @@ #include #ifdef CONFIG_X86 +#include #include #include #include @@ -2209,16 +2210,27 @@ void xen_setup_callback_vector(void) */ static __init void xen_init_setup_upcall_vector(void) { + unsigned long flags; + + apic_intr_mode_init(); + if (!xen_have_vector_callback) return; + local_irq_save(flags); + if ((cpuid_eax(xen_cpuid_base() + 4) & XEN_HVM_CPUID_UPCALL_VECTOR) && - !xen_set_upcall_vector(0)) + !xen_set_upcall_vector(0)) { xen_percpu_upcall = true; + WARN_ON(cpuhp_setup_state(CPUHP_AP_XEN_STARTING, "xen:starting", + xen_set_upcall_vector, NULL)); + } else if (xen_feature(XENFEAT_hvm_callback_vector)) xen_setup_callback_vector(); else xen_have_vector_callback = false; + + local_irq_restore(flags); } int xen_set_upcall_vector(unsigned int cpu) @@ -2250,9 +2262,7 @@ static __init void xen_alloc_callback_vector(void) } #else void xen_setup_callback_vector(void) {} -static inline void xen_init_setup_upcall_vector(void) {} int xen_set_upcall_vector(unsigned int cpu) {} -static inline void xen_alloc_callback_vector(void) {} #endif /* CONFIG_XEN_PVHVM */ #endif /* CONFIG_X86 */ @@ -2314,9 +2324,11 @@ void __init xen_init_IRQ(void) if (xen_initial_domain()) pci_xen_initial_domain(); } - xen_init_setup_upcall_vector(); - xen_alloc_callback_vector(); +#ifdef CONFIG_XEN_PVHVM + x86_init.irqs.intr_mode_init = xen_init_setup_upcall_vector; + xen_alloc_callback_vector(); +#endif if (xen_hvm_domain()) { native_init_IRQ(); diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index f61447913db9..c232ada4e4ca 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -191,7 +191,7 @@ enum cpuhp_state { CPUHP_AP_KVM_ARM_TIMER_STARTING, /* Must be the last timer callback */ CPUHP_AP_DUMMY_TIMER_STARTING, - CPUHP_AP_ARM_XEN_STARTING, + CPUHP_AP_XEN_STARTING, CPUHP_AP_ARM_CORESIGHT_STARTING, CPUHP_AP_ARM_CORESIGHT_CTI_STARTING, CPUHP_AP_ARM64_ISNDEP_STARTING,