From patchwork Tue Feb 20 01:02:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 203363 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp124611dyc; Mon, 19 Feb 2024 17:03:15 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUaHTGLapsQse0IqOmZuehQ6k/nx8i57U6QodvUqyNGGaPf9VZsrCiHr5kZe5S+nXkhFS34QqpYMoQAnMjPyrsXl1C+OA== X-Google-Smtp-Source: AGHT+IHJs70T/PpOWOpTo4fnTWZoBBIe4APxPjZ452GgD6vio6jgvxIK72E2gZbxg8sVlEi07KDU X-Received: by 2002:a17:90a:b385:b0:299:2b9a:88c4 with SMTP id e5-20020a17090ab38500b002992b9a88c4mr15681755pjr.1.1708390995032; Mon, 19 Feb 2024 17:03:15 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708390995; cv=pass; d=google.com; s=arc-20160816; b=rUKoaQKbDQNQQ+GSnSwapbr/mxJDpTYYuMbH6Sl9FkCN6/DM0Zn7MbL+7Krko6poll YQaovcEagDYmkdkZWVs92uDQcf/YnRbKQJnIF7ez2ICk0fZIHaBTLvvyKW59nkcZmGf5 X4vuJju0T/+pNn3NTGUzfJ8DhAot0AjDp/ifc/7OMRsDJ+H+k44+LFrI+eOsSHQU8oGn WInO+NGS+o9nkfZwgx+5FAtnKmNQBqc7iDwDMKIR4E3Ct3pLkra5+NMqk6cjQ3AHyLv2 zlwZapgk3jFKN6XEutcRkD93BABEGJHVivnYdomkLmT+w4NFzUmgZ3U5Hb67yGfv4jcZ VOIg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:precedence:robot-unsubscribe:robot-id :message-id:mime-version:list-unsubscribe:list-subscribe:list-id :precedence:cc:subject:to:reply-to:sender:from:dkim-signature :dkim-signature:date; bh=NSSokhVeMhI6TR2uUDBrbQMtvebxhbH6YFF+BVMKCxg=; fh=F3j5vJzznal/UB6e/yKmVNrMQh0yeZebCg5CK9TWA8g=; b=Sov7afpfGnM3aQKDrcLbYMf7AMKM3h+VdSXXhXUp9EMOxnJ+aa9bi/EaBcVwTxTBAs PdmdIGet8CLH3uRJ2m0qkS4ZwCyKrdqJxsAGRR1y96izUqhp6fL+ShEfbQnLXQ2VHeP8 OoVNEoSszPrr0iI0DwcdrOvL53oLuqgvYXJ4ibKtKeOfp6Fy4w8HA17kWe1l0KpcJMC+ rOZckbBU3e9knWp+jcNVXT1XzDLzCKfi478ff4oTqSIjEcoSeKKVRjOBYIKuRmc39qi4 Gqg77jaYq1zr5GXIijF77r3JO6hZ2M24FicE11ArBGh8SWLc6oMHLWSkhzVKo+aPbLpB 2MDA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=UJpPgxCL; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-72161-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-72161-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id nu18-20020a17090b1b1200b002997a08e32csi3749453pjb.141.2024.02.19.17.03.14 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Feb 2024 17:03:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-72161-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=UJpPgxCL; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e; arc=pass (i=1 spf=pass spfdomain=linutronix.de dkim=pass dkdomain=linutronix.de dmarc=pass fromdomain=linutronix.de); spf=pass (google.com: domain of linux-kernel+bounces-72161-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-72161-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id E152EB20B1C for ; Tue, 20 Feb 2024 01:03:13 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 66B8243AA0; Tue, 20 Feb 2024 01:02:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="UJpPgxCL"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="IttxE0/I" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 74FD5F9F8; Tue, 20 Feb 2024 01:02:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708390930; cv=none; b=Kk/Zf3baBUCoA8CbW6MiXZMDGWXUQAuqinxLr2JZmxFIFCe8nOeIB8nB/6znxThADWfyB8netsv6md/0qIxeVIFtHHphRvxD5u7zt5vr6X7Z7Gj6YslHZAdcxPZqcJBqAy+T+6Hdpic81xoUQoqurHssZA60rNjDL9bSNK91vqg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708390930; c=relaxed/simple; bh=FATTxN+hr2b1x/cLkiDexiJI3Bx6UVo9woKmdwJlpvI=; h=Date:From:To:Subject:Cc:MIME-Version:Message-ID:Content-Type; b=o9z4gfLDeyhbd06+PB9qO/Q63Y7ys51hIuYDZkqAUMx62hsqOSaVjDdSOfd5C1p2LNmZ+p2Niw+wcsfN/ozAyUteKiLrNYCg8Q2Nfrp2lRHiwA3FvISN00OXuLpxjjaPz6zq+93mDDcjVduLaPayIkDe+R9KjiDjCSgJo3mEwQQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=UJpPgxCL; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=IttxE0/I; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Date: Tue, 20 Feb 2024 01:02:06 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1708390926; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=NSSokhVeMhI6TR2uUDBrbQMtvebxhbH6YFF+BVMKCxg=; b=UJpPgxCLiPLHdL4iua0TA0HDAJH4ggphqifs6snq8rXKtWVqKGcvuDZAd+xpJtW5zHp3Dq gASE3mYcMTETZpq9ZDkUy2p4BwQ0qStyMoQWAncIa+DbE9uZ4ZYuhHDNJm/yUnINoinPgw cx0xm9YW+3Txe1EuvGcRuEOStlC4O3e48Ucp5Gb7Awh6aSN3cWIFVLSaCBI9rPE5xjiPoj hE8BB2oeq8S8QJJKCuxV/tG4CRYNGEPLXlRHHDtLvGTwgEDQdwGwE+PhqRf1JGw6JcgEcY bVf8iah/gDBIdHAvB4su6gY77coqK5zSoImwEnW5rq0DY7vCKvxtjv4VihWkFQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1708390926; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=NSSokhVeMhI6TR2uUDBrbQMtvebxhbH6YFF+BVMKCxg=; b=IttxE0/I4Zs5GadANvYT5qqM7DDrxEGnBUPiLS566gJc1AMB46BCTTpsymVGKRNYU4n30f pnlKZbQ3Ui61SfBQ== From: "tip-bot2 for Pawan Gupta" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: x86/urgent] x86/bugs: Use ALTERNATIVE() instead of mds_user_clear static key Cc: Pawan Gupta , Dave Hansen , x86@kernel.org, linux-kernel@vger.kernel.org Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-ID: <170839092600.398.4858077691839959721.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Precedence: bulk X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791377796180369576 X-GMAIL-MSGID: 1791377796180369576 The following commit has been merged into the x86/urgent branch of tip: Commit-ID: 6613d82e617dd7eb8b0c40b2fe3acea655b1d611 Gitweb: https://git.kernel.org/tip/6613d82e617dd7eb8b0c40b2fe3acea655b1d611 Author: Pawan Gupta AuthorDate: Tue, 13 Feb 2024 18:22:24 -08:00 Committer: Dave Hansen CommitterDate: Mon, 19 Feb 2024 16:31:49 -08:00 x86/bugs: Use ALTERNATIVE() instead of mds_user_clear static key The VERW mitigation at exit-to-user is enabled via a static branch mds_user_clear. This static branch is never toggled after boot, and can be safely replaced with an ALTERNATIVE() which is convenient to use in asm. Switch to ALTERNATIVE() to use the VERW mitigation late in exit-to-user path. Also remove the now redundant VERW in exc_nmi() and arch_exit_to_user_mode(). Signed-off-by: Pawan Gupta Signed-off-by: Dave Hansen Link: https://lore.kernel.org/all/20240213-delay-verw-v8-4-a6216d83edb7%40linux.intel.com --- Documentation/arch/x86/mds.rst | 38 +++++++++++++++++++-------- arch/x86/include/asm/entry-common.h | 1 +- arch/x86/include/asm/nospec-branch.h | 12 +--------- arch/x86/kernel/cpu/bugs.c | 15 ++++------- arch/x86/kernel/nmi.c | 3 +-- arch/x86/kvm/vmx/vmx.c | 2 +- 6 files changed, 34 insertions(+), 37 deletions(-) diff --git a/Documentation/arch/x86/mds.rst b/Documentation/arch/x86/mds.rst index e73fdff..c58c723 100644 --- a/Documentation/arch/x86/mds.rst +++ b/Documentation/arch/x86/mds.rst @@ -95,6 +95,9 @@ The kernel provides a function to invoke the buffer clearing: mds_clear_cpu_buffers() +Also macro CLEAR_CPU_BUFFERS can be used in ASM late in exit-to-user path. +Other than CFLAGS.ZF, this macro doesn't clobber any registers. + The mitigation is invoked on kernel/userspace, hypervisor/guest and C-state (idle) transitions. @@ -138,17 +141,30 @@ Mitigation points When transitioning from kernel to user space the CPU buffers are flushed on affected CPUs when the mitigation is not disabled on the kernel - command line. The migitation is enabled through the static key - mds_user_clear. - - The mitigation is invoked in prepare_exit_to_usermode() which covers - all but one of the kernel to user space transitions. The exception - is when we return from a Non Maskable Interrupt (NMI), which is - handled directly in do_nmi(). - - (The reason that NMI is special is that prepare_exit_to_usermode() can - enable IRQs. In NMI context, NMIs are blocked, and we don't want to - enable IRQs with NMIs blocked.) + command line. The mitigation is enabled through the feature flag + X86_FEATURE_CLEAR_CPU_BUF. + + The mitigation is invoked just before transitioning to userspace after + user registers are restored. This is done to minimize the window in + which kernel data could be accessed after VERW e.g. via an NMI after + VERW. + + **Corner case not handled** + Interrupts returning to kernel don't clear CPUs buffers since the + exit-to-user path is expected to do that anyways. But, there could be + a case when an NMI is generated in kernel after the exit-to-user path + has cleared the buffers. This case is not handled and NMI returning to + kernel don't clear CPU buffers because: + + 1. It is rare to get an NMI after VERW, but before returning to userspace. + 2. For an unprivileged user, there is no known way to make that NMI + less rare or target it. + 3. It would take a large number of these precisely-timed NMIs to mount + an actual attack. There's presumably not enough bandwidth. + 4. The NMI in question occurs after a VERW, i.e. when user state is + restored and most interesting data is already scrubbed. Whats left + is only the data that NMI touches, and that may or may not be of + any interest. 2. C-State transition diff --git a/arch/x86/include/asm/entry-common.h b/arch/x86/include/asm/entry-common.h index ce8f501..7e523bb 100644 --- a/arch/x86/include/asm/entry-common.h +++ b/arch/x86/include/asm/entry-common.h @@ -91,7 +91,6 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, static __always_inline void arch_exit_to_user_mode(void) { - mds_user_clear_cpu_buffers(); amd_clear_divider(); } #define arch_exit_to_user_mode arch_exit_to_user_mode diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index 077083e..2aa52ca 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -540,7 +540,6 @@ DECLARE_STATIC_KEY_FALSE(switch_to_cond_stibp); DECLARE_STATIC_KEY_FALSE(switch_mm_cond_ibpb); DECLARE_STATIC_KEY_FALSE(switch_mm_always_ibpb); -DECLARE_STATIC_KEY_FALSE(mds_user_clear); DECLARE_STATIC_KEY_FALSE(mds_idle_clear); DECLARE_STATIC_KEY_FALSE(switch_mm_cond_l1d_flush); @@ -575,17 +574,6 @@ static __always_inline void mds_clear_cpu_buffers(void) } /** - * mds_user_clear_cpu_buffers - Mitigation for MDS and TAA vulnerability - * - * Clear CPU buffers if the corresponding static key is enabled - */ -static __always_inline void mds_user_clear_cpu_buffers(void) -{ - if (static_branch_likely(&mds_user_clear)) - mds_clear_cpu_buffers(); -} - -/** * mds_idle_clear_cpu_buffers - Mitigation for MDS vulnerability * * Clear CPU buffers if the corresponding static key is enabled diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index bb0ab84..48d049c 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -111,9 +111,6 @@ DEFINE_STATIC_KEY_FALSE(switch_mm_cond_ibpb); /* Control unconditional IBPB in switch_mm() */ DEFINE_STATIC_KEY_FALSE(switch_mm_always_ibpb); -/* Control MDS CPU buffer clear before returning to user space */ -DEFINE_STATIC_KEY_FALSE(mds_user_clear); -EXPORT_SYMBOL_GPL(mds_user_clear); /* Control MDS CPU buffer clear before idling (halt, mwait) */ DEFINE_STATIC_KEY_FALSE(mds_idle_clear); EXPORT_SYMBOL_GPL(mds_idle_clear); @@ -252,7 +249,7 @@ static void __init mds_select_mitigation(void) if (!boot_cpu_has(X86_FEATURE_MD_CLEAR)) mds_mitigation = MDS_MITIGATION_VMWERV; - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); if (!boot_cpu_has(X86_BUG_MSBDS_ONLY) && (mds_nosmt || cpu_mitigations_auto_nosmt())) @@ -356,7 +353,7 @@ static void __init taa_select_mitigation(void) * For guests that can't determine whether the correct microcode is * present on host, enable the mitigation for UCODE_NEEDED as well. */ - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); if (taa_nosmt || cpu_mitigations_auto_nosmt()) cpu_smt_disable(false); @@ -424,7 +421,7 @@ static void __init mmio_select_mitigation(void) */ if (boot_cpu_has_bug(X86_BUG_MDS) || (boot_cpu_has_bug(X86_BUG_TAA) && boot_cpu_has(X86_FEATURE_RTM))) - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); else static_branch_enable(&mmio_stale_data_clear); @@ -484,12 +481,12 @@ static void __init md_clear_update_mitigation(void) if (cpu_mitigations_off()) return; - if (!static_key_enabled(&mds_user_clear)) + if (!boot_cpu_has(X86_FEATURE_CLEAR_CPU_BUF)) goto out; /* - * mds_user_clear is now enabled. Update MDS, TAA and MMIO Stale Data - * mitigation, if necessary. + * X86_FEATURE_CLEAR_CPU_BUF is now enabled. Update MDS, TAA and MMIO + * Stale Data mitigation, if necessary. */ if (mds_mitigation == MDS_MITIGATION_OFF && boot_cpu_has_bug(X86_BUG_MDS)) { diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c index 17e955a..3082cf2 100644 --- a/arch/x86/kernel/nmi.c +++ b/arch/x86/kernel/nmi.c @@ -563,9 +563,6 @@ nmi_restart: } if (this_cpu_dec_return(nmi_state)) goto nmi_restart; - - if (user_mode(regs)) - mds_user_clear_cpu_buffers(); } #if IS_ENABLED(CONFIG_KVM_INTEL) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 1111d9d..db8a5fe 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7227,7 +7227,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, /* L1D Flush includes CPU buffer clear to mitigate MDS */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); - else if (static_branch_unlikely(&mds_user_clear)) + else if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF)) mds_clear_cpu_buffers(); else if (static_branch_unlikely(&mmio_stale_data_clear) && kvm_arch_has_assigned_device(vcpu->kvm))