From patchwork Tue Oct 24 13:20:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 157449 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce89:0:b0:403:3b70:6f57 with SMTP id p9csp1937371vqx; Tue, 24 Oct 2023 06:21:58 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG3JevZLVR88i28TOwStQcgbpTjoEQh87rlqRpt6gqPv49Ew8cVMqei0/YemkJ65ioh6j71 X-Received: by 2002:a17:90a:193:b0:27f:bd9e:5a19 with SMTP id 19-20020a17090a019300b0027fbd9e5a19mr3771514pjc.25.1698153718517; Tue, 24 Oct 2023 06:21:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698153718; cv=none; d=google.com; s=arc-20160816; b=QoS6WDFiIGwQH5ZcnvVun1heKG/R0qiMAg9SaccMM6/QssVcslDkEHBMD6ME7jXU7o +JGmyd2DGNuksVwLx2+o1Ee4biVS3C5uX0qYkdP2N/QIpZz4SRuZ/2Zjw8GhuJkH4myL EHl7cz3W8No5ndTuGyMW7y4E/E7HbxrFj7nvsL0Z0Gn2faxgVWyPlqGTPq6u2iU4mNau IuNYUyu92swJVWu6/p0lHZjnOjJdq+bNRAkI6LGU5pgyyRJVKIUyiCzvDvicMW1fMjVm YVGEhMeg1f52+LPrakuv4jr+lOv3Z8MtcAy0J9P3hQPpg3X52BF16gjM2SJOkbAJ1aAQ ZHyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:references:in-reply-to:cc:subject :to:reply-to:sender:from:dkim-signature:dkim-signature:date; bh=3BJJkCmggxAnwtZY1XsMLY7eU371NfjGYVmCHFfvrAQ=; fh=LWlv3U+xZ3+vQt4tlJRIlKl5VzN7iFjjE/bzRmlxKDA=; b=cMkDKiWnfyQfEabbUYCROj8RR6a3/CBt1IHXNtCLIGM51PMp4ZcKFikzsTDLn/Hlmi FXXd/Mtloyu4AEQ3p5MS6NcmfP5SCiKpVa4laCl4HP7VeHzPK+m/m+wCWhXEccEk13AB LJtJG07rAeJSpzMHNALN3kKXzAmuAjVIW03M9C09qYNY2dirw4aMY44NymGh/a3XG7Wn N+ukZ9BMsx9icOcgaZ0Ki/GmItgU23Z0D8uJmB8I5F4fpIKEmqcewRRPcSUxQu2svYMJ HcSlTsJQxXqpnbvaqhmbickEuatxTZiY4hv3iaxZ1bccB4QV5q0D0vFcg9ZiHZ0WarlW fBpw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=KWwGXICq; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=Oe35n+fp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id gp19-20020a17090adf1300b0027d0db5c235si10962707pjb.166.2023.10.24.06.21.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Oct 2023 06:21:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=KWwGXICq; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=Oe35n+fp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id B3B32802C8F2; Tue, 24 Oct 2023 06:21:57 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234415AbjJXNVk (ORCPT + 26 others); Tue, 24 Oct 2023 09:21:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44190 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234490AbjJXNU6 (ORCPT ); Tue, 24 Oct 2023 09:20:58 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 53D3910F4; Tue, 24 Oct 2023 06:20:53 -0700 (PDT) Date: Tue, 24 Oct 2023 13:20:50 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1698153651; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3BJJkCmggxAnwtZY1XsMLY7eU371NfjGYVmCHFfvrAQ=; b=KWwGXICqtHYMHS3d51j8Ea0OvDMLpVp2mVLczVvTvjdaGuKoeRG9yOVM4WkjD8+og6We1l JQYQKWo7/bszGHAGcrCdwOAk7M+tpM1/LuLI/gTMLxPIwWqvfLKhyBbFu0d0Q7JNnmQm5g Wyg62agigyn2Lh7MxXLBHD4zRsuZqPSihI0kIGy76+mRdYB/8d/zqXdrOJ5SABgJ6/cTGZ gowz04q60vJVE5SLm6nyBWvINsu9zt3CGG1vM8dWktyrcMCj80VaH88Trn1jO7XKD6R81/ dNkQ+Hepxs40bpucSCaBNEmmqxjFA0S2L73Jym/2UTUf7LnwAYsh9mdu2UE4wA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1698153651; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3BJJkCmggxAnwtZY1XsMLY7eU371NfjGYVmCHFfvrAQ=; b=Oe35n+fpFVMhOxxyXb51BON6YRRBWRtfHzZHZWp54+7dPinY/+ItBDzOs+Zl9MZ/AacDo+ aC1PsSzweMdJtpAw== From: "tip-bot2 for Thomas Gleixner" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: x86/microcode] x86/microcode: Protect against instrumentation Cc: Thomas Gleixner , "Borislav Petkov (AMD)" , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20231002115903.545969323@linutronix.de> References: <20231002115903.545969323@linutronix.de> MIME-Version: 1.0 Message-ID: <169815365091.3135.7005664746879083949.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Tue, 24 Oct 2023 06:21:57 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778667852013679379 X-GMAIL-MSGID: 1780643233354065342 The following commit has been merged into the x86/microcode branch of tip: Commit-ID: 1582c0f4a21303792f523fe2839dd8433ee630c0 Gitweb: https://git.kernel.org/tip/1582c0f4a21303792f523fe2839dd8433ee630c0 Author: Thomas Gleixner AuthorDate: Mon, 02 Oct 2023 14:00:06 +02:00 Committer: Borislav Petkov (AMD) CommitterDate: Tue, 24 Oct 2023 15:05:55 +02:00 x86/microcode: Protect against instrumentation The wait for control loop in which the siblings are waiting for the microcode update on the primary thread must be protected against instrumentation as instrumentation can end up in #INT3, #DB or #PF, which then returns with IRET. That IRET reenables NMI which is the opposite of what the NMI rendezvous is trying to achieve. Signed-off-by: Thomas Gleixner Signed-off-by: Borislav Petkov (AMD) Link: https://lore.kernel.org/r/20231002115903.545969323@linutronix.de --- arch/x86/kernel/cpu/microcode/core.c | 111 +++++++++++++++++++------- 1 file changed, 83 insertions(+), 28 deletions(-) diff --git a/arch/x86/kernel/cpu/microcode/core.c b/arch/x86/kernel/cpu/microcode/core.c index 7b8ade5..6c90836 100644 --- a/arch/x86/kernel/cpu/microcode/core.c +++ b/arch/x86/kernel/cpu/microcode/core.c @@ -272,54 +272,65 @@ struct microcode_ctrl { DEFINE_STATIC_KEY_FALSE(microcode_nmi_handler_enable); static DEFINE_PER_CPU(struct microcode_ctrl, ucode_ctrl); +static unsigned int loops_per_usec; static atomic_t late_cpus_in; -static bool wait_for_cpus(atomic_t *cnt) +static noinstr bool wait_for_cpus(atomic_t *cnt) { - unsigned int timeout; + unsigned int timeout, loops; - WARN_ON_ONCE(atomic_dec_return(cnt) < 0); + WARN_ON_ONCE(raw_atomic_dec_return(cnt) < 0); for (timeout = 0; timeout < USEC_PER_SEC; timeout++) { - if (!atomic_read(cnt)) + if (!raw_atomic_read(cnt)) return true; - udelay(1); + for (loops = 0; loops < loops_per_usec; loops++) + cpu_relax(); /* If invoked directly, tickle the NMI watchdog */ - if (!microcode_ops->use_nmi && !(timeout % USEC_PER_MSEC)) + if (!microcode_ops->use_nmi && !(timeout % USEC_PER_MSEC)) { + instrumentation_begin(); touch_nmi_watchdog(); + instrumentation_end(); + } } /* Prevent the late comers from making progress and let them time out */ - atomic_inc(cnt); + raw_atomic_inc(cnt); return false; } -static bool wait_for_ctrl(void) +static noinstr bool wait_for_ctrl(void) { - unsigned int timeout; + unsigned int timeout, loops; for (timeout = 0; timeout < USEC_PER_SEC; timeout++) { - if (this_cpu_read(ucode_ctrl.ctrl) != SCTRL_WAIT) + if (raw_cpu_read(ucode_ctrl.ctrl) != SCTRL_WAIT) return true; - udelay(1); + + for (loops = 0; loops < loops_per_usec; loops++) + cpu_relax(); + /* If invoked directly, tickle the NMI watchdog */ - if (!microcode_ops->use_nmi && !(timeout % 1000)) + if (!microcode_ops->use_nmi && !(timeout % USEC_PER_MSEC)) { + instrumentation_begin(); touch_nmi_watchdog(); + instrumentation_end(); + } } return false; } -static void load_secondary(unsigned int cpu) +/* + * Protected against instrumentation up to the point where the primary + * thread completed the update. See microcode_nmi_handler() for details. + */ +static noinstr bool load_secondary_wait(unsigned int ctrl_cpu) { - unsigned int ctrl_cpu = this_cpu_read(ucode_ctrl.ctrl_cpu); - enum ucode_state ret; - /* Initial rendezvous to ensure that all CPUs have arrived */ if (!wait_for_cpus(&late_cpus_in)) { - pr_err_once("load: %d CPUs timed out\n", atomic_read(&late_cpus_in) - 1); - this_cpu_write(ucode_ctrl.result, UCODE_TIMEOUT); - return; + raw_cpu_write(ucode_ctrl.result, UCODE_TIMEOUT); + return false; } /* @@ -329,9 +340,33 @@ static void load_secondary(unsigned int cpu) * scheduler, watchdogs etc. There is no way to safely evacuate the * machine. */ - if (!wait_for_ctrl()) - panic("Microcode load: Primary CPU %d timed out\n", ctrl_cpu); + if (wait_for_ctrl()) + return true; + + instrumentation_begin(); + panic("Microcode load: Primary CPU %d timed out\n", ctrl_cpu); + instrumentation_end(); +} +/* + * Protected against instrumentation up to the point where the primary + * thread completed the update. See microcode_nmi_handler() for details. + */ +static noinstr void load_secondary(unsigned int cpu) +{ + unsigned int ctrl_cpu = raw_cpu_read(ucode_ctrl.ctrl_cpu); + enum ucode_state ret; + + if (!load_secondary_wait(ctrl_cpu)) { + instrumentation_begin(); + pr_err_once("load: %d CPUs timed out\n", + atomic_read(&late_cpus_in) - 1); + instrumentation_end(); + return; + } + + /* Primary thread completed. Allow to invoke instrumentable code */ + instrumentation_begin(); /* * If the primary succeeded then invoke the apply() callback, * otherwise copy the state from the primary thread. @@ -343,6 +378,7 @@ static void load_secondary(unsigned int cpu) this_cpu_write(ucode_ctrl.result, ret); this_cpu_write(ucode_ctrl.ctrl, SCTRL_DONE); + instrumentation_end(); } static void load_primary(unsigned int cpu) @@ -380,25 +416,43 @@ static void load_primary(unsigned int cpu) } } -static bool microcode_update_handler(void) +static noinstr bool microcode_update_handler(void) { - unsigned int cpu = smp_processor_id(); + unsigned int cpu = raw_smp_processor_id(); - if (this_cpu_read(ucode_ctrl.ctrl_cpu) == cpu) + if (raw_cpu_read(ucode_ctrl.ctrl_cpu) == cpu) { + instrumentation_begin(); load_primary(cpu); - else + instrumentation_end(); + } else { load_secondary(cpu); + } + instrumentation_begin(); touch_nmi_watchdog(); + instrumentation_end(); + return true; } -bool microcode_nmi_handler(void) +/* + * Protection against instrumentation is required for CPUs which are not + * safe against an NMI which is delivered to the secondary SMT sibling + * while the primary thread updates the microcode. Instrumentation can end + * up in #INT3, #DB and #PF. The IRET from those exceptions reenables NMI + * which is the opposite of what the NMI rendezvous is trying to achieve. + * + * The primary thread is safe versus instrumentation as the actual + * microcode update handles this correctly. It's only the sibling code + * path which must be NMI safe until the primary thread completed the + * update. + */ +bool noinstr microcode_nmi_handler(void) { - if (!this_cpu_read(ucode_ctrl.nmi_enabled)) + if (!raw_cpu_read(ucode_ctrl.nmi_enabled)) return false; - this_cpu_write(ucode_ctrl.nmi_enabled, false); + raw_cpu_write(ucode_ctrl.nmi_enabled, false); return microcode_update_handler(); } @@ -425,6 +479,7 @@ static int load_late_stop_cpus(void) pr_err("You should switch to early loading, if possible.\n"); atomic_set(&late_cpus_in, num_online_cpus()); + loops_per_usec = loops_per_jiffy / (TICK_NSEC / 1000); /* * Take a snapshot before the microcode update in order to compare and