From patchwork Tue Oct 24 13:20:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 157454 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce89:0:b0:403:3b70:6f57 with SMTP id p9csp1937867vqx; Tue, 24 Oct 2023 06:22:43 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFehRzW35VDWtGAk4RoWGQ7HTsjBNIIZYTlNgm6BsHIW2U0Ko78DPYE5bOCaRXOpPnGw5+V X-Received: by 2002:a17:902:ea95:b0:1c9:d90b:c3e4 with SMTP id x21-20020a170902ea9500b001c9d90bc3e4mr9264099plb.10.1698153762813; Tue, 24 Oct 2023 06:22:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698153762; cv=none; d=google.com; s=arc-20160816; b=XZI+QeTYJjQ7h/MyV4NWYhxkxM+SgOyc9BPWl1Z9REYUG/sSbwr04WTknh4m87B1Du 8KLUReJZOZ4mETYHrptuh8HA+hKz/NoQI4LnxJIPBmceu0AX1DlnQtRqGdB7uCcE1bb4 BMZ+G5ncFaysca45zDx7/zFGUcIyjyN1SMBq+MvTr/i/fce5SQAsB2MMsz5JzAuX1Uzc 57uq9gKTNUxFkIMqv9zR1WbKkGG/5rSHU9FVFOHQr0CGicd9Gn2mO5Ckpq/OAaPPXNsA kYqA5y4uytzTVmEUrAA0r2HROspbnqFiTuYo49KxDKJNO7r/ts6pLtzCtTIJbjjFMiYp icDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:references:in-reply-to:cc:subject :to:reply-to:sender:from:dkim-signature:dkim-signature:date; bh=z8Q6m45DRvc+fLKdHVCG1nuNlm5KdR5Bik6GwrhH7jw=; fh=LWlv3U+xZ3+vQt4tlJRIlKl5VzN7iFjjE/bzRmlxKDA=; b=qndBuifr/0wOMwEoXkutHrG1mSfbFMvuPj9Vltc7oT7JKha4US721nRUs+1noWVz7/ /6wDrH8aHKksbR2jETSK5w6mempdUR8Z0nqSzIZDZOsaBl5LogR9WrArBDvRnBSeUqAp TTL65WjR7FQh9MfnbU2QdxkQLsrRhi6CIgot+5fh9cxtn8Wm1MavD0Uvfxom9lDf4efs 5Fm9XlfPCGepOoNkzTxnpdRt3gsfjkGQQESyyC0IdKcMDSRV6osPPMNJr5RxO6dAspCk z/6cbWkMPDV6MbvGm2NNdeN6bN8v8c9Sxda+u2RNEMjTD6gCHSW0GxUpjb6rHSPrAzTf MIsg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ySve58o5; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id b19-20020a170902ed1300b001c9d9050b37si8022080pld.260.2023.10.24.06.22.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Oct 2023 06:22:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=ySve58o5; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id C25A380845AE; Tue, 24 Oct 2023 06:22:23 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234677AbjJXNVp (ORCPT + 26 others); Tue, 24 Oct 2023 09:21:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37756 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234533AbjJXNU7 (ORCPT ); Tue, 24 Oct 2023 09:20:59 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 748EED7E; Tue, 24 Oct 2023 06:20:54 -0700 (PDT) Date: Tue, 24 Oct 2023 13:20:51 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1698153652; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z8Q6m45DRvc+fLKdHVCG1nuNlm5KdR5Bik6GwrhH7jw=; b=ySve58o5qnf7nSRGE4gfglzDbW4Yoj+sZvF/kZX77njbSkNuixhhsgb5PZkJH+7f3Bv3La bvzrwOF6OhpC8jyTLEfUF8aN1t+SzJD/2E6JVZKH9GCAw1fTtbRhDB2ZU7ffj9BKecINuA 1X4IZFk0hSZVLF7hnecVpr1tBsqKbZCkdYlEDhX8aqN7EnOQoT812gkwbEo98qmwMdbjhz aAaSBfKjQ8M040x3ob3U8bpiJLAF/kj02NonvidMQTWWEBEngC8p/WqoaSdEl3gq648HhV 4DUvs02Lo4ZA0lT8vKnVi4Ys6djI/EEuPZgLCdUgkbUJsQs7gBkyPTClNUukLQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1698153652; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z8Q6m45DRvc+fLKdHVCG1nuNlm5KdR5Bik6GwrhH7jw=; b=k//iEEZDTq/MhIzfxriq/Fhaokrm7r+RtfgOk4HYy9xfzVAfdB8mIfHQVu+VJ4//wNfkyY fPZXLn1NjD6GeMBg== From: "tip-bot2 for Thomas Gleixner" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: x86/microcode] x86/microcode: Rendezvous and load in NMI Cc: Thomas Gleixner , "Borislav Petkov (AMD)" , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20231002115903.489900814@linutronix.de> References: <20231002115903.489900814@linutronix.de> MIME-Version: 1.0 Message-ID: <169815365169.3135.4479318752696201093.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Tue, 24 Oct 2023 06:22:24 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778649516171008629 X-GMAIL-MSGID: 1780643279925738321 The following commit has been merged into the x86/microcode branch of tip: Commit-ID: 7eb314a22800457396f541c655697dabd71e44a7 Gitweb: https://git.kernel.org/tip/7eb314a22800457396f541c655697dabd71e44a7 Author: Thomas Gleixner AuthorDate: Mon, 02 Oct 2023 14:00:05 +02:00 Committer: Borislav Petkov (AMD) CommitterDate: Tue, 24 Oct 2023 15:05:55 +02:00 x86/microcode: Rendezvous and load in NMI stop_machine() does not prevent the spin-waiting sibling from handling an NMI, which is obviously violating the whole concept of rendezvous. Implement a static branch right in the beginning of the NMI handler which is nopped out except when enabled by the late loading mechanism. The late loader enables the static branch before stop_machine() is invoked. Each CPU has an nmi_enable in its control structure which indicates whether the CPU should go into the update routine. This is required to bridge the gap between enabling the branch and actually being at the point where it is required to enter the loader wait loop. Each CPU which arrives in the stopper thread function sets that flag and issues a self NMI right after that. If the NMI function sees the flag clear, it returns. If it's set it clears the flag and enters the rendezvous. This is safe against a real NMI which hits in between setting the flag and sending the NMI to itself. The real NMI will be swallowed by the microcode update and the self NMI will then let stuff continue. Otherwise this would end up with a spurious NMI. Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov (AMD) Link: https://lore.kernel.org/r/20231002115903.489900814@linutronix.de --- arch/x86/include/asm/microcode.h | 12 +++++++- arch/x86/kernel/cpu/microcode/core.c | 42 ++++++++++++++++++++--- arch/x86/kernel/cpu/microcode/intel.c | 1 +- arch/x86/kernel/cpu/microcode/internal.h | 3 +- arch/x86/kernel/nmi.c | 4 ++- 5 files changed, 57 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/microcode.h b/arch/x86/include/asm/microcode.h index 78f1eb2..8292482 100644 --- a/arch/x86/include/asm/microcode.h +++ b/arch/x86/include/asm/microcode.h @@ -72,4 +72,16 @@ static inline u32 intel_get_microcode_revision(void) } #endif /* !CONFIG_CPU_SUP_INTEL */ +bool microcode_nmi_handler(void); + +#ifdef CONFIG_MICROCODE_LATE_LOADING +DECLARE_STATIC_KEY_FALSE(microcode_nmi_handler_enable); +static __always_inline bool microcode_nmi_handler_enabled(void) +{ + return static_branch_unlikely(µcode_nmi_handler_enable); +} +#else +static __always_inline bool microcode_nmi_handler_enabled(void) { return false; } +#endif + #endif /* _ASM_X86_MICROCODE_H */ diff --git a/arch/x86/kernel/cpu/microcode/core.c b/arch/x86/kernel/cpu/microcode/core.c index 1c2710b..7b8ade5 100644 --- a/arch/x86/kernel/cpu/microcode/core.c +++ b/arch/x86/kernel/cpu/microcode/core.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -31,6 +32,7 @@ #include #include +#include #include #include #include @@ -265,8 +267,10 @@ struct microcode_ctrl { enum sibling_ctrl ctrl; enum ucode_state result; unsigned int ctrl_cpu; + bool nmi_enabled; }; +DEFINE_STATIC_KEY_FALSE(microcode_nmi_handler_enable); static DEFINE_PER_CPU(struct microcode_ctrl, ucode_ctrl); static atomic_t late_cpus_in; @@ -282,7 +286,8 @@ static bool wait_for_cpus(atomic_t *cnt) udelay(1); - if (!(timeout % USEC_PER_MSEC)) + /* If invoked directly, tickle the NMI watchdog */ + if (!microcode_ops->use_nmi && !(timeout % USEC_PER_MSEC)) touch_nmi_watchdog(); } /* Prevent the late comers from making progress and let them time out */ @@ -298,7 +303,8 @@ static bool wait_for_ctrl(void) if (this_cpu_read(ucode_ctrl.ctrl) != SCTRL_WAIT) return true; udelay(1); - if (!(timeout % 1000)) + /* If invoked directly, tickle the NMI watchdog */ + if (!microcode_ops->use_nmi && !(timeout % 1000)) touch_nmi_watchdog(); } return false; @@ -374,7 +380,7 @@ static void load_primary(unsigned int cpu) } } -static int load_cpus_stopped(void *unused) +static bool microcode_update_handler(void) { unsigned int cpu = smp_processor_id(); @@ -383,7 +389,29 @@ static int load_cpus_stopped(void *unused) else load_secondary(cpu); - /* No point to wait here. The CPUs will all wait in stop_machine(). */ + touch_nmi_watchdog(); + return true; +} + +bool microcode_nmi_handler(void) +{ + if (!this_cpu_read(ucode_ctrl.nmi_enabled)) + return false; + + this_cpu_write(ucode_ctrl.nmi_enabled, false); + return microcode_update_handler(); +} + +static int load_cpus_stopped(void *unused) +{ + if (microcode_ops->use_nmi) { + /* Enable the NMI handler and raise NMI */ + this_cpu_write(ucode_ctrl.nmi_enabled, true); + apic->send_IPI(smp_processor_id(), NMI_VECTOR); + } else { + /* Just invoke the handler directly */ + microcode_update_handler(); + } return 0; } @@ -404,8 +432,14 @@ static int load_late_stop_cpus(void) */ store_cpu_caps(&prev_info); + if (microcode_ops->use_nmi) + static_branch_enable_cpuslocked(µcode_nmi_handler_enable); + stop_machine_cpuslocked(load_cpus_stopped, NULL, cpu_online_mask); + if (microcode_ops->use_nmi) + static_branch_disable_cpuslocked(µcode_nmi_handler_enable); + /* Analyze the results */ for_each_cpu_and(cpu, cpu_present_mask, &cpus_booted_once_mask) { switch (per_cpu(ucode_ctrl.result, cpu)) { diff --git a/arch/x86/kernel/cpu/microcode/intel.c b/arch/x86/kernel/cpu/microcode/intel.c index e5c5ddf..905ed3b 100644 --- a/arch/x86/kernel/cpu/microcode/intel.c +++ b/arch/x86/kernel/cpu/microcode/intel.c @@ -611,6 +611,7 @@ static struct microcode_ops microcode_intel_ops = { .collect_cpu_info = collect_cpu_info, .apply_microcode = apply_microcode_late, .finalize_late_load = finalize_late_load, + .use_nmi = IS_ENABLED(CONFIG_X86_64), }; static __init void calc_llc_size_per_core(struct cpuinfo_x86 *c) diff --git a/arch/x86/kernel/cpu/microcode/internal.h b/arch/x86/kernel/cpu/microcode/internal.h index c699043..627d238 100644 --- a/arch/x86/kernel/cpu/microcode/internal.h +++ b/arch/x86/kernel/cpu/microcode/internal.h @@ -31,7 +31,8 @@ struct microcode_ops { enum ucode_state (*apply_microcode)(int cpu); int (*collect_cpu_info)(int cpu, struct cpu_signature *csig); void (*finalize_late_load)(int result); - unsigned int nmi_safe : 1; + unsigned int nmi_safe : 1, + use_nmi : 1; }; extern struct ucode_cpu_info ucode_cpu_info[]; diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c index a0c5518..a87d856 100644 --- a/arch/x86/kernel/nmi.c +++ b/arch/x86/kernel/nmi.c @@ -33,6 +33,7 @@ #include #include #include +#include #include #define CREATE_TRACE_POINTS @@ -343,6 +344,9 @@ static noinstr void default_do_nmi(struct pt_regs *regs) instrumentation_begin(); + if (microcode_nmi_handler_enabled() && microcode_nmi_handler()) + goto out; + handled = nmi_handle(NMI_LOCAL, regs); __this_cpu_add(nmi_stats.normal, handled); if (handled) {