From patchwork Tue Sep 12 07:58:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 138245 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9ecd:0:b0:3f2:4152:657d with SMTP id t13csp483192vqx; Tue, 12 Sep 2023 08:22:38 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF60S3Z0sd5bik/AFLanTxZeqQ+8a+vWv3hxBcE3Yet2V6fV3Vo0GAdb8GS8sMQYn+2k230 X-Received: by 2002:a17:90a:f191:b0:26b:3751:652a with SMTP id bv17-20020a17090af19100b0026b3751652amr12104093pjb.38.1694532157768; Tue, 12 Sep 2023 08:22:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694532157; cv=none; d=google.com; s=arc-20160816; b=prt5ZkBUhzoNtChCyp2ZsQM/SZrNvhB1LP+a1MsKcw2N1VwSOyde8e5oCL8bD6v+Or nuLPY1H5t+iWWecN3dn37wOB8msW+2naSWsD+AOxdWCHCppy9pHluTCIZdhyWrdxrWOt DTTYb7J/g+DPeKA/rzPtUqUa4LPsKwVZCB1k3p9rfgmXmQVxo5p4vqsOVzU0tyKpGHq4 ijusj0gZv1+ww+1LYsJVhUxybSZ4M6ukYBN8AtHBh3IwNYHeRDjjAOlEhdT0zy86bJ+Z Kz9FEh7DCWODO7VwHxJYq7r2j3xnRq92C7C53tw5LGvVcQpl45EY8c16nNH0e7ZZaXrQ JSoA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:date:mime-version:references:subject:cc:to:from :dkim-signature:dkim-signature:message-id; bh=M0PdCWTDbxqvJEjnQT+Zz1AFGlSL0PoJj3xhmsUVjSU=; fh=u57tXYamzTrJA+Ht8n1u7SfTMptrQaIb6LVW+jsaYf4=; b=XK5l3Ja/KJjAC8KOR9pJcv2up2JYIW9knDqyfL7rR8ozZUVCevE+9uenfFXt/DCztL MPuCZyIHRRYd+/iaHKyJQ9GI3LWE7bd6ScMJh+/dg/vAInQm9Dm5YS4XtKbw9DNykZxy iZH+WxVEqvcvziHIA4DOiB0g72LO9Xa0yXB17MK/LKH8hqcFvr/f0/7vaETIWF8S2DV5 ia2LVJ0HW5+CVtzwZRwFzNmoXI3ur+as819obRvwqFKmkFCJ+X9FUFb+PUk4LyIafiUh jxJdmdtOT943GOyxJXt3u1ZJ4OXYKM5mE8xLVhw22n4E8Loe/wKsIFWQnLHpMS6HbO8/ AL/Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=WJFNhXUR; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=qkm8ylO3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id me3-20020a17090b17c300b00262ff3a4545si6139861pjb.169.2023.09.12.08.22.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 08:22:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=WJFNhXUR; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=qkm8ylO3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id D74D980A52B8; Tue, 12 Sep 2023 01:04:20 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.8 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232375AbjILIBJ (ORCPT + 39 others); Tue, 12 Sep 2023 04:01:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42608 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232319AbjILH7z (ORCPT ); Tue, 12 Sep 2023 03:59:55 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 90D762D4C for ; Tue, 12 Sep 2023 00:58:30 -0700 (PDT) Message-ID: <20230912065502.507838060@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1694505508; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=M0PdCWTDbxqvJEjnQT+Zz1AFGlSL0PoJj3xhmsUVjSU=; b=WJFNhXURunT+UUAt0yLxmQJYOM9x7pobFvhwAsGCYnxu1NtcAYYPy136l0anxbcST+2uGD dFhI+2o9Rd5WtByPHl4zHPkIlizrt18BCHsueMwrHHBVsK5ys10C25PbOFb29Qa/23m6dA 3GGQQy2rEHubTdYuKjkMTdOj4+3Qhy0slm18eG2C0hVeKTS/TGZTNDk98OvRhmu7RZe3Wh u/vLdb5QMCji8nn6iO4YFvHwfvhenCdrday43dUYkRkCqnbDQkO+iDDF5AhXhlitXHv4YW gMryfPe+wNz7j9MY2S0N9tKVOuPc4GicGEBhZRzPp0fsGZwnTuBfC7z8+ktJlg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1694505508; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=M0PdCWTDbxqvJEjnQT+Zz1AFGlSL0PoJj3xhmsUVjSU=; b=qkm8ylO36PDb1/S3GKM+jtPlcSRQCQ6iGq10TaOtVE0DDRHX9WB2UhPfJDyxVifxdM3nLP OqssZpdfx96u1rDQ== From: Thomas Gleixner To: LKML Cc: x86@kernel.org, Borislav Petkov , "Chang S. Bae" , Arjan van de Ven , Nikolay Borisov Subject: [patch V3 28/30] x86/microcode: Handle "offline" CPUs correctly References: <20230912065249.695681286@linutronix.de> MIME-Version: 1.0 Date: Tue, 12 Sep 2023 09:58:28 +0200 (CEST) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Tue, 12 Sep 2023 01:04:21 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776845751710238289 X-GMAIL-MSGID: 1776845751710238289 From: Thomas Gleixner Offline CPUs need to be parked in a safe loop when microcode update is in progress on the primary CPU. Currently offline CPUs are parked in 'mwait_play_dead()', and for Intel CPUs, its not a safe instruction, because 'mwait' instruction can be patched in the new microcode update that can cause instability. - Adds a new microcode state 'UCODE_OFFLINE' to report status on per-cpu basis. - Force NMI on the offline CPUs. Wakeup offline CPUs while the update is in progress and then return them back to 'mwait_play_dead()' after microcode update is complete. Signed-off-by: Thomas Gleixner --- arch/x86/include/asm/microcode.h | 1 arch/x86/kernel/cpu/microcode/core.c | 112 +++++++++++++++++++++++++++++-- arch/x86/kernel/cpu/microcode/internal.h | 1 arch/x86/kernel/nmi.c | 5 + 4 files changed, 113 insertions(+), 6 deletions(-) --- --- a/arch/x86/include/asm/microcode.h +++ b/arch/x86/include/asm/microcode.h @@ -71,6 +71,7 @@ static inline u32 intel_get_microcode_re #endif /* !CONFIG_CPU_SUP_INTEL */ bool microcode_nmi_handler(void); +void microcode_offline_nmi_handler(void); #ifdef CONFIG_MICROCODE_LATE_LOADING DECLARE_STATIC_KEY_FALSE(microcode_nmi_handler_enable); --- a/arch/x86/kernel/cpu/microcode/core.c +++ b/arch/x86/kernel/cpu/microcode/core.c @@ -298,8 +298,9 @@ struct ucode_ctrl { DEFINE_STATIC_KEY_FALSE(microcode_nmi_handler_enable); static DEFINE_PER_CPU(struct ucode_ctrl, ucode_ctrl); +static atomic_t late_cpus_in, offline_in_nmi; static unsigned int loops_per_usec; -static atomic_t late_cpus_in; +static cpumask_t cpu_offline_mask; static noinstr bool wait_for_cpus(atomic_t *cnt) { @@ -407,7 +408,7 @@ static noinstr void ucode_load_secondary instrumentation_end(); } -static void ucode_load_primary(unsigned int cpu) +static void __ucode_load_primary(unsigned int cpu) { struct cpumask *secondaries = topology_sibling_cpumask(cpu); enum sibling_ctrl ctrl; @@ -443,6 +444,67 @@ static void ucode_load_primary(unsigned } } +static bool ucode_kick_offline_cpus(unsigned int nr_offl) +{ + unsigned int cpu, timeout; + + for_each_cpu(cpu, &cpu_offline_mask) { + /* Enable the rendevouz handler and send NMI */ + per_cpu(ucode_ctrl.nmi_enabled, cpu) = true; + apic_send_nmi_to_offline_cpu(cpu); + } + + /* Wait for them to arrive */ + for (timeout = 0; timeout < (USEC_PER_SEC / 2); timeout++) { + if (atomic_read(&offline_in_nmi) == nr_offl) + return true; + udelay(1); + } + /* Let the others time out */ + return false; +} + +static void ucode_release_offline_cpus(void) +{ + unsigned int cpu; + + for_each_cpu(cpu, &cpu_offline_mask) + per_cpu(ucode_ctrl.ctrl, cpu) = SCTRL_DONE; +} + +static void ucode_load_primary(unsigned int cpu) +{ + unsigned int nr_offl = cpumask_weight(&cpu_offline_mask); + bool proceed = true; + + /* Kick soft-offlined SMT siblings if required */ + if (!cpu && nr_offl) + proceed = ucode_kick_offline_cpus(nr_offl); + + /* If the soft-offlined CPUs did not respond, abort */ + if (proceed) + __ucode_load_primary(cpu); + + /* Unconditionally release soft-offlined SMT siblings if required */ + if (!cpu && nr_offl) + ucode_release_offline_cpus(); +} + +/* + * Minimal stub rendevouz handler for soft-offlined CPUs which participate + * in the NMI rendevouz to protect against a concurrent NMI on affected + * CPUs. + */ +void noinstr microcode_offline_nmi_handler(void) +{ + if (!raw_cpu_read(ucode_ctrl.nmi_enabled)) + return; + raw_cpu_write(ucode_ctrl.nmi_enabled, false); + raw_cpu_write(ucode_ctrl.result, UCODE_OFFLINE); + raw_atomic_inc(&offline_in_nmi); + wait_for_ctrl(); +} + static noinstr bool microcode_update_handler(void) { unsigned int cpu = raw_smp_processor_id(); @@ -499,6 +561,7 @@ static int ucode_load_cpus_stopped(void static int ucode_load_late_stop_cpus(void) { unsigned int cpu, updated = 0, failed = 0, timedout = 0, siblings = 0; + unsigned int nr_offl, offline = 0; int old_rev = boot_cpu_data.microcode; struct cpuinfo_x86 prev_info; @@ -506,6 +569,7 @@ static int ucode_load_late_stop_cpus(voi pr_err("You should switch to early loading, if possible.\n"); atomic_set(&late_cpus_in, num_online_cpus()); + atomic_set(&offline_in_nmi, 0); loops_per_usec = loops_per_jiffy / (TICK_NSEC / 1000); /* @@ -528,6 +592,7 @@ static int ucode_load_late_stop_cpus(voi case UCODE_UPDATED: updated++; break; case UCODE_TIMEOUT: timedout++; break; case UCODE_OK: siblings++; break; + case UCODE_OFFLINE: offline++; break; default: failed++; break; } } @@ -539,6 +604,13 @@ static int ucode_load_late_stop_cpus(voi /* Nothing changed. */ if (!failed && !timedout) return 0; + + nr_offl = cpumask_weight(&cpu_offline_mask); + if (offline < nr_offl) { + pr_warn("%u offline siblings did not respond.\n", + nr_offl - atomic_read(&offline_in_nmi)); + return -EIO; + } pr_err("Microcode update failed: %u CPUs failed %u CPUs timed out\n", failed, timedout); return -EIO; @@ -572,19 +644,49 @@ static int ucode_load_late_stop_cpus(voi * modern CPUs is using MWAIT, which is also not guaranteed to be safe * against a microcode update which affects MWAIT. * - * 2) Initialize the per CPU control structure + * As soft-offlined CPUs still react on NMIs, the SMT sibling + * restriction can be lifted when the vendor driver signals to use NMI + * for rendevouz and the APIC provides a mechanism to send an NMI to a + * soft-offlined CPU. The soft-offlined CPUs are then able to + * participate in the rendezvouz in a trivial stub handler. + * + * 2) Initialize the per CPU control structure and create a cpumask + * which contains "offline"; secondary threads, so they can be handled + * correctly by a control CPU. */ static bool ucode_setup_cpus(void) { struct ucode_ctrl ctrl = { .ctrl = SCTRL_WAIT, .result = -1, }; + bool allow_smt_offline; unsigned int cpu; + allow_smt_offline = microcode_ops->nmi_safe || + (microcode_ops->use_nmi && apic->nmi_to_offline_cpu); + + cpumask_clear(&cpu_offline_mask); + for_each_cpu_and(cpu, cpu_present_mask, &cpus_booted_once_mask) { + /* + * Offline CPUs sit in one of the play_dead() functions + * with interrupts disabled, but they still react on NMIs + * and execute arbitrary code. Also MWAIT being updated + * while the offline CPU sits there is not necessarily safe + * on all CPU variants. + * + * Mark them in the offline_cpus mask which will be handled + * by CPU0 later in the update process. + * + * Ensure that the primary thread is online so that it is + * guaranteed that all cores are updated. + */ if (!cpu_online(cpu)) { - if (topology_is_primary_thread(cpu) || !microcode_ops->nmi_safe) { - pr_err("CPU %u not online\n", cpu); + if (topology_is_primary_thread(cpu) || !allow_smt_offline) { + pr_err("CPU %u not online, loading aborted\n", cpu); return false; } + cpumask_set_cpu(cpu, &cpu_offline_mask); + per_cpu(ucode_ctrl, cpu) = ctrl; + continue; } /* --- a/arch/x86/kernel/cpu/microcode/internal.h +++ b/arch/x86/kernel/cpu/microcode/internal.h @@ -17,6 +17,7 @@ enum ucode_state { UCODE_NFOUND, UCODE_ERROR, UCODE_TIMEOUT, + UCODE_OFFLINE, }; struct microcode_ops { --- a/arch/x86/kernel/nmi.c +++ b/arch/x86/kernel/nmi.c @@ -502,8 +502,11 @@ DEFINE_IDTENTRY_RAW(exc_nmi) if (IS_ENABLED(CONFIG_NMI_CHECK_CPU)) raw_atomic_long_inc(&nsp->idt_calls); - if (IS_ENABLED(CONFIG_SMP) && arch_cpu_is_offline(smp_processor_id())) + if (IS_ENABLED(CONFIG_SMP) && arch_cpu_is_offline(smp_processor_id())) { + if (microcode_nmi_handler_enabled()) + microcode_offline_nmi_handler(); return; + } if (this_cpu_read(nmi_state) != NMI_NOT_RUNNING) { this_cpu_write(nmi_state, NMI_LATCHED);