From patchwork Fri Oct 20 11:37:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 156052 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2010:b0:403:3b70:6f57 with SMTP id fe16csp992159vqb; Fri, 20 Oct 2023 04:41:47 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG1uOIcmhf+wLMy45eXCj09Q2glJZWNVsRw4PlKrCZtftMv64p6X5lxkI5nvLHdWQdHmRFI X-Received: by 2002:a05:6870:5688:b0:1ea:30b2:3fcc with SMTP id p8-20020a056870568800b001ea30b23fccmr1665721oao.31.1697802107158; Fri, 20 Oct 2023 04:41:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697802107; cv=none; d=google.com; s=arc-20160816; b=sH10FZGWQVmspqxzflcnsRFSuv13x5lwQYdhj+syLH6IsxSMQAtcgIPb+ZQVybHJSs 4dBlHaMwAKbDoTtgWq9pRQkQGQbOHwT6RZNLaKcWSX+V+Ia6VFHKnXcCvISMnnV3dUbr DYPoElUyB3ZzrnJeD1biC35cLgsuXCS2/uhNT3ZYWl5c/IG18oeBz+O24H+XNbNy/BjV Og6TprSKyPWlFk1I5Z8KUM4BSMWb5mmZaw0KuP3kGFwTR/0xGvV4zBJhSJVOSG2EhsZq rgzRHPFjteIJhFf7sVF7DQ+lDFMLl/cXgyFc+ssIvCsdWtvulkGae1JSZbA/HAbZN+WA S2Wg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:references:in-reply-to:cc:subject :to:reply-to:sender:from:dkim-signature:dkim-signature:date; bh=kqsrCiEYSGeCDn1ja0TvVqQlLDSZTHMm0bv3Pbnyo/k=; fh=LWlv3U+xZ3+vQt4tlJRIlKl5VzN7iFjjE/bzRmlxKDA=; b=KKQDMEdqOfmoTrkCW4JAeOiMbvNxGKN0bB254qI74MJ60FBVZntbo3U3baI25I4Jru wI0qH5dNUveV+xDrIi4KUUV4U6IJTHXcSc2+NSjpPMZADa5Z5xVQ4P3x0/7b/cxRIrrw PfJ+v9bBD2PbHcOMELfk6WtVPRq3DFyRGzE+UA/reBmA77C+/ZkQHthciLQH9DxDxOsF 9r0H9LY982wbI7CXTnRgcAg5TkQg/dTBLfKvCfHEqnTXb+teelRmH8H7B1TA70JdxSVH I7594ISy/ucMtDpw8vsJWyzXNVGMqTe7VwuJjS5m2ytNWkPbgWxVX9k9+GKvZbavkiAF qMsA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=l24g0Y0m; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id h70-20020a638349000000b005acf4298a78si1840184pge.149.2023.10.20.04.41.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 04:41:47 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=l24g0Y0m; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 4D1A383AEFD3; Fri, 20 Oct 2023 04:40:35 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1377366AbjJTLjd (ORCPT + 25 others); Fri, 20 Oct 2023 07:39:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1377225AbjJTLiu (ORCPT ); Fri, 20 Oct 2023 07:38:50 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F1D6610DF; Fri, 20 Oct 2023 04:38:04 -0700 (PDT) Date: Fri, 20 Oct 2023 11:37:48 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1697801868; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kqsrCiEYSGeCDn1ja0TvVqQlLDSZTHMm0bv3Pbnyo/k=; b=l24g0Y0mcULazRSOL17m0Rf743416j/+3Q781oV+IX1CDSlB/HNl5thiEUmt5lxuK3GwvM /ErLreamRpF/PGUMpFh3jrSItLbBXhQpl5K+NDPFgeKC/bqYSWtEv1u+CK3P87B0fXzGR2 /jIvbIBub9wkgXS4L1gskYnsqMJM51qw4v69COx+P1tQ4P+3tVI0MSSJbp7PvDmnNm+slt JfTO0uTkK1J16rasoiatKHb9/1fX/umV/LzfIlMUMVQZexp2tReabpoozL8t58JuQacSlS 5ftxjFYm0n4X8Jfp5aBo37ss0xmXsae1kdsbOUpx13gTX2Re8Ws4vnkNl0q5PA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1697801868; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kqsrCiEYSGeCDn1ja0TvVqQlLDSZTHMm0bv3Pbnyo/k=; b=BhGKHevD73Y7LQho+vk3/nDutfR14IGkRx1p6vD75xjq60aLH1C0ERYKaQhu4ER5+xe97L fVWCLQLMkvngyFAQ== From: "tip-bot2 for Thomas Gleixner" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: x86/microcode] x86/microcode: Rendezvous and load in NMI Cc: Thomas Gleixner , "Borislav Petkov (AMD)" , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20231002115903.489900814@linutronix.de> References: <20231002115903.489900814@linutronix.de> MIME-Version: 1.0 Message-ID: <169780186816.3135.11385598800295455799.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 20 Oct 2023 04:40:35 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778649516171008629 X-GMAIL-MSGID: 1780274542201766884 The following commit has been merged into the x86/microcode branch of tip: Commit-ID: 8858b12182058468542cb5d0a4587895f6ec473c Gitweb: https://git.kernel.org/tip/8858b12182058468542cb5d0a4587895f6ec473c Author: Thomas Gleixner AuthorDate: Mon, 02 Oct 2023 14:00:05 +02:00 Committer: Borislav Petkov (AMD) CommitterDate: Thu, 19 Oct 2023 15:59:38 +02:00 x86/microcode: Rendezvous and load in NMI stop_machine() does not prevent the spin-waiting sibling from handling an NMI, which is obviously violating the whole concept of rendezvous. Implement a static branch right in the beginning of the NMI handler which is nopped out except when enabled by the late loading mechanism. The late loader enables the static branch before stop_machine() is invoked. Each CPU has an nmi_enable in its control structure which indicates whether the CPU should go into the update routine. This is required to bridge the gap between enabling the branch and actually being at the point where it is required to enter the loader wait loop. Each CPU which arrives in the stopper thread function sets that flag and issues a self NMI right after that. If the NMI function sees the flag clear, it returns. If it's set it clears the flag and enters the rendezvous. This is safe against a real NMI which hits in between setting the flag and sending the NMI to itself. The real NMI will be swallowed by the microcode update and the self NMI will then let stuff continue. Otherwise this would end up with a spurious NMI. Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov (AMD) Link: https://lore.kernel.org/r/20231002115903.489900814@linutronix.de --- arch/x86/include/asm/microcode.h | 12 +++++++- arch/x86/kernel/cpu/microcode/core.c | 42 ++++++++++++++++++++--- arch/x86/kernel/cpu/microcode/intel.c | 1 +- arch/x86/kernel/cpu/microcode/internal.h | 3 +- arch/x86/kernel/nmi.c | 4 ++- 5 files changed, 57 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/microcode.h b/arch/x86/include/asm/microcode.h index 78f1eb2..8292482 100644 --- a/arch/x86/include/asm/microcode.h +++ b/arch/x86/include/asm/microcode.h @@ -72,4 +72,16 @@ static inline u32 intel_get_microcode_revision(void) } #endif /* !CONFIG_CPU_SUP_INTEL */ +bool microcode_nmi_handler(void); + +#ifdef CONFIG_MICROCODE_LATE_LOADING +DECLARE_STATIC_KEY_FALSE(microcode_nmi_handler_enable); +static __always_inline bool microcode_nmi_handler_enabled(void) +{ + return static_branch_unlikely(µcode_nmi_handler_enable); +} +#else +static __always_inline bool microcode_nmi_handler_enabled(void) { return false; } +#endif + #endif /* _ASM_X86_MICROCODE_H */ diff --git a/arch/x86/kernel/cpu/microcode/core.c b/arch/x86/kernel/cpu/microcode/core.c index 1c2710b..7b8ade5 100644 --- a/arch/x86/kernel/cpu/microcode/core.c +++ b/arch/x86/kernel/cpu/microcode/core.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -31,6 +32,7 @@ #include #include +#include #include #include #include @@ -265,8 +267,10 @@ struct microcode_ctrl { enum sibling_ctrl ctrl; enum ucode_state result; unsigned int ctrl_cpu; + bool nmi_enabled; }; +DEFINE_STATIC_KEY_FALSE(microcode_nmi_handler_enable); static DEFINE_PER_CPU(struct microcode_ctrl, ucode_ctrl); static atomic_t late_cpus_in; @@ -282,7 +286,8 @@ static bool wait_for_cpus(atomic_t *cnt) udelay(1); - if (!(timeout % USEC_PER_MSEC)) + /* If invoked directly, tickle the NMI watchdog */ + if (!microcode_ops->use_nmi && !(timeout % USEC_PER_MSEC)) touch_nmi_watchdog(); } /* Prevent the late comers from making progress and let them time out */ @@ -298,7 +303,8 @@ static bool wait_for_ctrl(void) if (this_cpu_read(ucode_ctrl.ctrl) != SCTRL_WAIT) return true; udelay(1); - if (!(timeout % 1000)) + /* If invoked directly, tickle the NMI watchdog */ + if (!microcode_ops->use_nmi && !(timeout % 1000)) touch_nmi_watchdog(); } return false; @@ -374,7 +380,7 @@ static void load_primary(unsigned int cpu) } } -static int load_cpus_stopped(void *unused) +static bool microcode_update_handler(void) { unsigned int cpu = smp_processor_id(); @@ -383,7 +389,29 @@ static int load_cpus_stopped(void *unused) else load_secondary(cpu); - /* No point to wait here. The CPUs will all wait in stop_machine(). */ + touch_nmi_watchdog(); + return true; +} + +bool microcode_nmi_handler(void) +{ + if (!this_cpu_read(ucode_ctrl.nmi_enabled)) + return false; + + this_cpu_write(ucode_ctrl.nmi_enabled, false); + return microcode_update_handler(); +} + +static int load_cpus_stopped(void *unused) +{ + if (microcode_ops->use_nmi) { + /* Enable the NMI handler and raise NMI */ + this_cpu_write(ucode_ctrl.nmi_enabled, true); + apic->send_IPI(smp_processor_id(), NMI_VECTOR); + } else { + /* Just invoke the handler directly */ + microcode_update_handler(); + } return 0; } @@ -404,8 +432,14 @@ static int load_late_stop_cpus(void) */ store_cpu_caps(&prev_info); + if (microcode_ops->use_nmi) + static_branch_enable_cpuslocked(µcode_nmi_handler_enable); + stop_machine_cpuslocked(load_cpus_stopped, NULL, cpu_online_mask); + if (microcode_ops->use_nmi) + static_branch_disable_cpuslocked(µcode_nmi_handler_enable); + /* Analyze the results */ for_each_cpu_and(cpu, cpu_present_mask, &cpus_booted_once_mask) { switch (per_cpu(ucode_ctrl.result, cpu)) { diff --git a/arch/x86/kernel/cpu/microcode/intel.c b/arch/x86/kernel/cpu/microcode/intel.c index 9dffb0c..5c4c2b6 100644 --- a/arch/x86/kernel/cpu/microcode/intel.c +++ b/arch/x86/kernel/cpu/microcode/intel.c @@ -595,6 +595,7 @@ static struct microcode_ops microcode_intel_ops = { .collect_cpu_info = collect_cpu_info, .apply_microcode = apply_microcode_late, .finalize_late_load = finalize_late_load, + .use_nmi = IS_ENABLED(CONFIG_X86_64), }; static __init void calc_llc_size_per_core(struct cpuinfo_x86 *c) diff --git a/arch/x86/kernel/cpu/microcode/internal.h b/arch/x86/kernel/cpu/microcode/internal.h index c699043..627d238 100644 --- a/arch/x86/kernel/cpu/microcode/internal.h +++ b/arch/x86/kernel/cpu/microcode/internal.h @@ -31,7 +31,8 @@ struct microcode_ops { enum ucode_state (*apply_microcode)(int cpu); int (*collect_cpu_info)(int cpu, struct cpu_signature *csig); void (*finalize_late_load)(int result); - unsigned int nmi_safe : 1; + unsigned int nmi_safe : 1, + use_nmi : 1; }; extern struct ucode_cpu_info ucode_cpu_info[]; diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c index a0c5518..a87d856 100644 --- a/arch/x86/kernel/nmi.c +++ b/arch/x86/kernel/nmi.c @@ -33,6 +33,7 @@ #include #include #include +#include #include #define CREATE_TRACE_POINTS @@ -343,6 +344,9 @@ static noinstr void default_do_nmi(struct pt_regs *regs) instrumentation_begin(); + if (microcode_nmi_handler_enabled() && microcode_nmi_handler()) + goto out; + handled = nmi_handle(NMI_LOCAL, regs); __this_cpu_add(nmi_stats.normal, handled); if (handled) {