From patchwork Wed Nov 30 23:36:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 28083 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1215199wrr; Wed, 30 Nov 2022 15:59:41 -0800 (PST) X-Google-Smtp-Source: AA0mqf4zNDVwqV8KXb5ZHcGVNrz8gykPuKyim+5dDZG7pAfeAEwjZ35Sj5rfsqbiOGr66ne7v/gk X-Received: by 2002:a17:906:65c4:b0:7ad:d250:b907 with SMTP id z4-20020a17090665c400b007add250b907mr53331656ejn.737.1669852781393; Wed, 30 Nov 2022 15:59:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669852781; cv=none; d=google.com; s=arc-20160816; b=v1cCHH6G+HEYdjxbzbFEt7O4ErSpTKCe0kKAmIMBEDQMz01pjTUYLy7wY7gHEYKhpa xNEV87EmDHYtq9kddk4kGq/tXnorK01I8hdDKvtBNkXQAGKspkZKkuWU5uyJPwVnSJMW hD8hrFyCwbYsX9tfRDh3WEVooif4NApmf4XbwSoKCZf+41UYJ6Qdiy/3zKpHouOVsyET y7f1gwC8+nLTvKFdfDz7G0YUGkV4iDe+8uYm6YvzelVMRhNP1bvoUFKkhThL54G3ZXhU Rdo3v38Nuy6KGpZvJuMY5YLXwoUQsR1KQpD3uTWiaM4ECB2K0JEXDRec+JEAV7VHBsna VnvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=I8FwrRgfmd2UuwKL7ywz+/7+wS6AMaBUpFIfke29fZA=; b=gLZBCdnDkQKq9cqF+h+c4X5q3UTtuELa5umRy9S2uk7DrRZCyM9Pxo2FcL06i+Af8E yyCXA+JuXy1WaCoAQ0n5XnrK0sWTiEnAEVNV36hZMsE9C8s4ZWo0KIN9aMSunCKZPaqf H6S8nzcT/eNAl3oXU2VnmNdUY43EX5TAuEyoDW9JT46K9BR5P0QYxdXXueo9HTAMyyVg NmPjbl5TMcC+aVfWK+dRQVVJfavR/vm75UKahDrFjbG7UmMF7h/NM9PTPikewBb4qoy8 WjOJcRvO+y9FwftfbzO9E4MGzL9tLaj7+lKNWUYNzzduvtVyLt5NcfGeBBrPS3GYK0sR DwLw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=PNHfGcWV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cw28-20020a056402229c00b0046b5aff3c95si2188564edb.311.2022.11.30.15.59.16; Wed, 30 Nov 2022 15:59:41 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=PNHfGcWV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229801AbiK3Xg7 (ORCPT + 99 others); Wed, 30 Nov 2022 18:36:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229566AbiK3Xgz (ORCPT ); Wed, 30 Nov 2022 18:36:55 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A32071128 for ; Wed, 30 Nov 2022 15:36:54 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id l2-20020a17090add8200b00218daa1a812so222177pjv.3 for ; Wed, 30 Nov 2022 15:36:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=I8FwrRgfmd2UuwKL7ywz+/7+wS6AMaBUpFIfke29fZA=; b=PNHfGcWVuJ5EEApzCw9+x7W5a3m55Zv7MFPYgUAebhkNCmmO6tTohmCwEgw8WX9ps6 zoZT3no8rG4d0/DJFD/5rLzRcgGPMwLRSZqORTwbOBep4v/52sxg1tyUZM29i8iKITFG fa+73uOueAh2lPT0ISJfsQvpBmyjceWy18CJovXZisAFMLvvPd/clsPg1QO+1I5SNL+D FtLAz136RsueNj4MfQJ98AYiSJ0WtopplRvlsqet/EvgVruROuKUWQGF4HEn4K65BF9o 2XKVqUbHmf84ebktMA/jtRpsQY7yhjg24PERH1/ksa49K5IQ2z+ag3Pq5aiew26OWBke PDJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=I8FwrRgfmd2UuwKL7ywz+/7+wS6AMaBUpFIfke29fZA=; b=ofKJaXYS4FvZintDhVUrGNFW3lLmKUopNtUuxtgWI9agJyPxf91+AKv4xGiBuPhgtm FjU0Dui+fGZ0Tnyr8uiUp/JCBULkT/1nKSVdLwD+fy9SzmYUBCz02K+EN0IOfyfJtmBx 2zslPD/yYuG/a3cxr7tZRZ2LFPoeTeTZdmAhVCyrRbB7JISBX0zX8plVAQXuUUh0Z/F2 bL2wU9wJUslo47o1S1CPGW2kPqTBOa3++MuVekDAmj74+gdHki+y+S8Ca7oADBK+/1r5 3d72DaEC+sTpaEtpRio6KiKoHyJ+/x6zFobriXNLUYZvjdv1mph0ITMIaK6Hj7N2Feyy +/tw== X-Gm-Message-State: ANoB5pk/05zZPo5D/WeSvqbOYy7ueWfVv5jutzWuWAfLGoDEjNy+YK+f +zdgB2JVX/GBlLYamoXNiMTbOjeQrrM= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:e550:b0:184:dc59:cc9f with SMTP id n16-20020a170902e55000b00184dc59cc9fmr47087289plf.89.1669851414240; Wed, 30 Nov 2022 15:36:54 -0800 (PST) Reply-To: Sean Christopherson Date: Wed, 30 Nov 2022 23:36:47 +0000 In-Reply-To: <20221130233650.1404148-1-seanjc@google.com> Mime-Version: 1.0 References: <20221130233650.1404148-1-seanjc@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog Message-ID: <20221130233650.1404148-2-seanjc@google.com> Subject: [PATCH v4 1/4] x86/crash: Disable virt in core NMI crash handler to avoid double shootdown From: Sean Christopherson To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org Cc: "H. Peter Anvin" , linux-kernel@vger.kernel.org, "Guilherme G . Piccoli" , Vitaly Kuznetsov , Paolo Bonzini , Sean Christopherson , Andrew Cooper , Tom Lendacky , stable@vger.kernel X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750967550134386439?= X-GMAIL-MSGID: =?utf-8?q?1750967550134386439?= Disable virtualization in crash_nmi_callback() and rework the emergency_vmx_disable_all() path to do an NMI shootdown if and only if a shootdown has not already occurred. NMI crash shootdown fundamentally can't support multiple invocations as responding CPUs are deliberately put into halt state without unblocking NMIs. But, the emergency reboot path doesn't have any work of its own, it simply cares about disabling virtualization, i.e. so long as a shootdown occurred, emergency reboot doesn't care who initiated the shootdown, or when. If "crash_kexec_post_notifiers" is specified on the kernel command line, panic() will invoke crash_smp_send_stop() and result in a second call to nmi_shootdown_cpus() during native_machine_emergency_restart(). Invoke the callback _before_ disabling virtualization, as the current VMCS needs to be cleared before doing VMXOFF. Note, this results in a subtle change in ordering between disabling virtualization and stopping Intel PT on the responding CPUs. While VMX and Intel PT do interact, VMXOFF and writes to MSR_IA32_RTIT_CTL do not induce faults between one another, which is all that matters when panicking. Harden nmi_shootdown_cpus() against multiple invocations to try and capture any such kernel bugs via a WARN instead of hanging the system during a crash/dump, e.g. prior to the recent hardening of register_nmi_handler(), re-registering the NMI handler would trigger a double list_add() and hang the system if CONFIG_BUG_ON_DATA_CORRUPTION=y. list_add double add: new=ffffffff82220800, prev=ffffffff8221cfe8, next=ffffffff82220800. WARNING: CPU: 2 PID: 1319 at lib/list_debug.c:29 __list_add_valid+0x67/0x70 Call Trace: __register_nmi_handler+0xcf/0x130 nmi_shootdown_cpus+0x39/0x90 native_machine_emergency_restart+0x1c9/0x1d0 panic+0x237/0x29b Extract the disabling logic to a common helper to deduplicate code, and to prepare for doing the shootdown in the emergency reboot path if SVM is supported. Note, prior to commit ed72736183c4 ("x86/reboot: Force all cpus to exit VMX root if VMX is supported"), nmi_shootdown_cpus() was subtly protected against a second invocation by a cpu_vmx_enabled() check as the kdump handler would disable VMX if it ran first. Fixes: ed72736183c4 ("x86/reboot: Force all cpus to exit VMX root if VMX is supported) Cc: stable@vger.kernel.org Reported-by: Guilherme G. Piccoli Cc: Vitaly Kuznetsov Cc: Paolo Bonzini Link: https://lore.kernel.org/all/20220427224924.592546-2-gpiccoli@igalia.com Signed-off-by: Sean Christopherson Tested-by: Guilherme G. Piccoli --- arch/x86/include/asm/reboot.h | 2 ++ arch/x86/kernel/crash.c | 17 +-------- arch/x86/kernel/reboot.c | 65 ++++++++++++++++++++++++++++------- 3 files changed, 56 insertions(+), 28 deletions(-) diff --git a/arch/x86/include/asm/reboot.h b/arch/x86/include/asm/reboot.h index 04c17be9b5fd..bc5b4d788c08 100644 --- a/arch/x86/include/asm/reboot.h +++ b/arch/x86/include/asm/reboot.h @@ -25,6 +25,8 @@ void __noreturn machine_real_restart(unsigned int type); #define MRR_BIOS 0 #define MRR_APM 1 +void cpu_emergency_disable_virtualization(void); + typedef void (*nmi_shootdown_cb)(int, struct pt_regs*); void nmi_panic_self_stop(struct pt_regs *regs); void nmi_shootdown_cpus(nmi_shootdown_cb callback); diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c index 9730c88530fc..6b8a6aae02e3 100644 --- a/arch/x86/kernel/crash.c +++ b/arch/x86/kernel/crash.c @@ -37,7 +37,6 @@ #include #include #include -#include #include #include #include @@ -81,15 +80,6 @@ static void kdump_nmi_callback(int cpu, struct pt_regs *regs) */ cpu_crash_vmclear_loaded_vmcss(); - /* Disable VMX or SVM if needed. - * - * We need to disable virtualization on all CPUs. - * Having VMX or SVM enabled on any CPU may break rebooting - * after the kdump kernel has finished its task. - */ - cpu_emergency_vmxoff(); - cpu_emergency_svm_disable(); - /* * Disable Intel PT to stop its logging */ @@ -148,12 +138,7 @@ void native_machine_crash_shutdown(struct pt_regs *regs) */ cpu_crash_vmclear_loaded_vmcss(); - /* Booting kdump kernel with VMX or SVM enabled won't work, - * because (among other limitations) we can't disable paging - * with the virt flags. - */ - cpu_emergency_vmxoff(); - cpu_emergency_svm_disable(); + cpu_emergency_disable_virtualization(); /* * Disable Intel PT to stop its logging diff --git a/arch/x86/kernel/reboot.c b/arch/x86/kernel/reboot.c index c3636ea4aa71..374c14b3a9bf 100644 --- a/arch/x86/kernel/reboot.c +++ b/arch/x86/kernel/reboot.c @@ -528,10 +528,7 @@ static inline void kb_wait(void) } } -static void vmxoff_nmi(int cpu, struct pt_regs *regs) -{ - cpu_emergency_vmxoff(); -} +static inline void nmi_shootdown_cpus_on_restart(void); /* Use NMIs as IPIs to tell all CPUs to disable virtualization */ static void emergency_vmx_disable_all(void) @@ -554,7 +551,7 @@ static void emergency_vmx_disable_all(void) __cpu_emergency_vmxoff(); /* Halt and exit VMX root operation on the other CPUs. */ - nmi_shootdown_cpus(vmxoff_nmi); + nmi_shootdown_cpus_on_restart(); } } @@ -795,6 +792,17 @@ void machine_crash_shutdown(struct pt_regs *regs) /* This is the CPU performing the emergency shutdown work. */ int crashing_cpu = -1; +/* + * Disable virtualization, i.e. VMX or SVM, to ensure INIT is recognized during + * reboot. VMX blocks INIT if the CPU is post-VMXON, and SVM blocks INIT if + * GIF=0, i.e. if the crash occurred between CLGI and STGI. + */ +void cpu_emergency_disable_virtualization(void) +{ + cpu_emergency_vmxoff(); + cpu_emergency_svm_disable(); +} + #if defined(CONFIG_SMP) static nmi_shootdown_cb shootdown_callback; @@ -817,7 +825,14 @@ static int crash_nmi_callback(unsigned int val, struct pt_regs *regs) return NMI_HANDLED; local_irq_disable(); - shootdown_callback(cpu, regs); + if (shootdown_callback) + shootdown_callback(cpu, regs); + + /* + * Prepare the CPU for reboot _after_ invoking the callback so that the + * callback can safely use virtualization instructions, e.g. VMCLEAR. + */ + cpu_emergency_disable_virtualization(); atomic_dec(&waiting_for_crash_ipi); /* Assume hlt works */ @@ -828,18 +843,32 @@ static int crash_nmi_callback(unsigned int val, struct pt_regs *regs) return NMI_HANDLED; } -/* - * Halt all other CPUs, calling the specified function on each of them +/** + * nmi_shootdown_cpus - Stop other CPUs via NMI + * @callback: Optional callback to be invoked from the NMI handler * - * This function can be used to halt all other CPUs on crash - * or emergency reboot time. The function passed as parameter - * will be called inside a NMI handler on all CPUs. + * The NMI handler on the remote CPUs invokes @callback, if not + * NULL, first and then disables virtualization to ensure that + * INIT is recognized during reboot. + * + * nmi_shootdown_cpus() can only be invoked once. After the first + * invocation all other CPUs are stuck in crash_nmi_callback() and + * cannot respond to a second NMI. */ void nmi_shootdown_cpus(nmi_shootdown_cb callback) { unsigned long msecs; + local_irq_disable(); + /* + * Avoid certain doom if a shootdown already occurred; re-registering + * the NMI handler will cause list corruption, modifying the callback + * will do who knows what, etc... + */ + if (WARN_ON_ONCE(crash_ipi_issued)) + return; + /* Make a note of crashing cpu. Will be used in NMI callback. */ crashing_cpu = safe_smp_processor_id(); @@ -867,7 +896,17 @@ void nmi_shootdown_cpus(nmi_shootdown_cb callback) msecs--; } - /* Leave the nmi callback set */ + /* + * Leave the nmi callback set, shootdown is a one-time thing. Clearing + * the callback could result in a NULL pointer dereference if a CPU + * (finally) responds after the timeout expires. + */ +} + +static inline void nmi_shootdown_cpus_on_restart(void) +{ + if (!crash_ipi_issued) + nmi_shootdown_cpus(NULL); } /* @@ -897,6 +936,8 @@ void nmi_shootdown_cpus(nmi_shootdown_cb callback) /* No other CPUs to shoot down */ } +static inline void nmi_shootdown_cpus_on_restart(void) { } + void run_crash_ipi_callback(struct pt_regs *regs) { }