From patchwork Wed Feb 14 02:21:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 200809 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp1001740dyb; Tue, 13 Feb 2024 21:26:18 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWg2fSwUyPZFnY6fNYpfOKmp6T/pZWbshKei8MAGjvr1q75ySTAQu12INUvKgTMIfRtpdJnD2IJ4yI2GRSNgM5j4xIgaQ== X-Google-Smtp-Source: AGHT+IGwHk1/dzLV+LkvEZF35ZjgjsBk0HsE3555eqjdTghxCVo79My1CWtXRRjq/gY6WOSzl/6x X-Received: by 2002:a17:906:459:b0:a3c:eb19:8e9d with SMTP id e25-20020a170906045900b00a3ceb198e9dmr897515eja.51.1707888378230; Tue, 13 Feb 2024 21:26:18 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707888378; cv=pass; d=google.com; s=arc-20160816; b=UNkID8fwWvODU8j6kwsQPStxNUaBSeXC4hqbXzP7WW0xFjV4+9s8pPLnB5jqGOCe5n O9kRYMx92A22X1a3BdNFGFbwFGioGw2nrXeIrrxtkdZby++EMlMurc6CYzJVG4Q9EMz/ zCkzgqSlHDtIFnByZlMlHij4W1TGIYk5b/bwAwq3uJfthzzbYnKIGtdWPhAAElQRyMdV Cdz7NCFjwc7bLG6mQPcAmRF32zG4nmPN9K+JvTwq2kgGplGhbbMYPjh67m5QFpvBPhEz vpbarZsRxfVCAptHPDrzD3uZfevVRqPCFYNbhoruo7gIZSuRdYmdwpKtHcHUATcglZ6J Ga0g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:message-id:subject:cc :to:from:date:dkim-signature; bh=rGY8MgAjorTLmS0etZL+MbOeGy8EWjx6eeLXYxdDNas=; fh=D3XBFI9LD91qUIln3y5LG2psl0Zso2sXQ029rMbJMZs=; b=otXrTLcEx0av0zwjw7t02E7D1n03AY9pT2+g1tcHr76e+A4Qrxel3o+S+0WO5sv8ov wBoAV9EeDU4OsyI75qE0d5lgYIRglLkreUKO8eS6+ffKShFgjtVR53wNHFc+QUunjief dDa5S0XH0cBe8gTctm0jcAyPit8WSdYT/2qDMsqc35tRyIJTr9NvANOhVJBcBsTYgBcu znqClk+SE8EJn1eWrbyn5aOhhuoejTw1rie1dJtT5yWIG71ACPCgkstRIN/vqnc3lB3F 0C4c+o3PmNSluNQE2imgcpYHoAUtXmtZNEnkMzt4vLGA8HxatXdusu/lJVhsBR2VHxha muSg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=O86RzAzY; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-64684-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-64684-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=2; AJvYcCX+NRKh9+s73DJ9cMeF9SxHfww/E/6qd9MC8kWBrj6KnD2gTK7XG9AhnB38Bd3BnH5+jC2393wCiEd4bbC015nRdJT0xA== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id w21-20020a170906131500b00a371e56413fsi1952229ejb.745.2024.02.13.21.26.18 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Feb 2024 21:26:18 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-64684-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=O86RzAzY; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-64684-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-64684-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 4B17D1F29221 for ; Wed, 14 Feb 2024 02:22:07 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 23243C2E9; Wed, 14 Feb 2024 02:21:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="O86RzAzY" Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 65F8CB664; Wed, 14 Feb 2024 02:21:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=134.134.136.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707877300; cv=none; b=e/fQWzPhBsZ+ShF8TbwcZk2h2WvIDChcYKItAHBE8vfVZnf52iwuTySCiq2Lx3gt1aNK5LPcj/HImRCQLXbkmR+l2QWwz7ZlVMX056POwotQ6XqegG8rGfX6JjqwVkDyb6Mp1Cj7Hmp6fGQKsIkzFh69hn7B+5WFNKDWwpIdecY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707877300; c=relaxed/simple; bh=w/Py/fKUqDJ1wrKhND+FI5QwXTp0BunLn1s0v7WFZgg=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=rWzVQblEq7Omx10Tfa0nGHP5tjIP1cWwDV+p44sVu+oW1AhKKH0KshfIqnGJAMMCMREoeC6TF9wmYlilnUMOzB/MWTd8pywQJ/r7xtlvmiZYhnYQmm8nYYtmQ5ySR1ujJLSDbzG58GZIN+tnaky8fVbnL5K9FcK8jZcEDXrFUcs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=O86RzAzY; arc=none smtp.client-ip=134.134.136.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707877298; x=1739413298; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=w/Py/fKUqDJ1wrKhND+FI5QwXTp0BunLn1s0v7WFZgg=; b=O86RzAzYuaysXZG5yXbqlQbxfOQGanLdMRDXlurt4VNIWwyGTP72s9U4 AoY+CS7cnw7gGXwc01QJ7fQzIvXJvGzI+2eccAjBX95iypONQppz7VSUF 574YGNxdBDts3+A28hh491ucg52z57/AecDB+XGreNCba7sU6zzq8gJhV /GwE/wrDyQ4eW0jD0PPOJ6q5fm9QT5gVC2jwPevV19YzeATCqt+QP9S8k dRbT+Nm+MBwtWLsBgWeUPgJPU9O5rH9zgQAz2pdNqaQJidhe2tjhnAFyM oz5f1r4JpXtEumxTexoq2TXxNSoStdXRyYN5YFvMYkgGb0sX/yIlUOOTy A==; X-IronPort-AV: E=McAfee;i="6600,9927,10982"; a="396348013" X-IronPort-AV: E=Sophos;i="6.06,158,1705392000"; d="scan'208";a="396348013" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2024 18:21:37 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10982"; a="911976033" X-IronPort-AV: E=Sophos;i="6.06,158,1705392000"; d="scan'208";a="911976033" Received: from diegoavi-mobl.amr.corp.intel.com (HELO desk) ([10.255.230.185]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2024 18:21:35 -0800 Date: Tue, 13 Feb 2024 18:21:35 -0800 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com Subject: [PATCH v8 1/6] x86/bugs: Add asm helpers for executing VERW Message-ID: <20240213-delay-verw-v8-1-a6216d83edb7@linux.intel.com> X-Mailer: b4 0.12.3 References: <20240213-delay-verw-v8-0-a6216d83edb7@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240213-delay-verw-v8-0-a6216d83edb7@linux.intel.com> X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790850764130276555 X-GMAIL-MSGID: 1790850764130276555 MDS mitigation requires clearing the CPU buffers before returning to user. This needs to be done late in the exit-to-user path. Current location of VERW leaves a possibility of kernel data ending up in CPU buffers for memory accesses done after VERW such as: 1. Kernel data accessed by an NMI between VERW and return-to-user can remain in CPU buffers since NMI returning to kernel does not execute VERW to clear CPU buffers. 2. Alyssa reported that after VERW is executed, CONFIG_GCC_PLUGIN_STACKLEAK=y scrubs the stack used by a system call. Memory accesses during stack scrubbing can move kernel stack contents into CPU buffers. 3. When caller saved registers are restored after a return from function executing VERW, the kernel stack accesses can remain in CPU buffers(since they occur after VERW). To fix this VERW needs to be moved very late in exit-to-user path. In preparation for moving VERW to entry/exit asm code, create macros that can be used in asm. Also make VERW patching depend on a new feature flag X86_FEATURE_CLEAR_CPU_BUF. Reported-by: Alyssa Milburn Suggested-by: Andrew Cooper Suggested-by: Peter Zijlstra Cc: stable@kernel.org Signed-off-by: Pawan Gupta --- arch/x86/entry/entry.S | 22 ++++++++++++++++++++++ arch/x86/include/asm/cpufeatures.h | 2 +- arch/x86/include/asm/nospec-branch.h | 13 +++++++++++++ 3 files changed, 36 insertions(+), 1 deletion(-) diff --git a/arch/x86/entry/entry.S b/arch/x86/entry/entry.S index 582731f74dc8..d9feadffa972 100644 --- a/arch/x86/entry/entry.S +++ b/arch/x86/entry/entry.S @@ -6,6 +6,9 @@ #include #include #include +#include +#include +#include #include "calling.h" @@ -23,4 +26,23 @@ EXPORT_SYMBOL_GPL(entry_ibpb); .popsection +/* + * Define the VERW operand that is disguised as entry code so that + * it can be referenced with KPTI enabled. This ensure VERW can be + * used late in exit-to-user path after page tables are switched. + */ +.pushsection .entry.text, "ax" + +.align L1_CACHE_BYTES, 0xcc +SYM_CODE_START_NOALIGN(mds_verw_sel) + UNWIND_HINT_UNDEFINED + ANNOTATE_NOENDBR + .word __KERNEL_DS +.align L1_CACHE_BYTES, 0xcc +SYM_CODE_END(mds_verw_sel); +/* For KVM */ +EXPORT_SYMBOL_GPL(mds_verw_sel); + +.popsection + THUNK warn_thunk_thunk, __warn_thunk diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 315673320f6b..0343caa016a9 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -95,7 +95,7 @@ #define X86_FEATURE_SYSENTER32 ( 3*32+15) /* "" sysenter in IA32 userspace */ #define X86_FEATURE_REP_GOOD ( 3*32+16) /* REP microcode works well */ #define X86_FEATURE_AMD_LBR_V2 ( 3*32+17) /* AMD Last Branch Record Extension Version 2 */ -/* FREE, was #define X86_FEATURE_LFENCE_RDTSC ( 3*32+18) "" LFENCE synchronizes RDTSC */ +#define X86_FEATURE_CLEAR_CPU_BUF ( 3*32+18) /* "" Clear CPU buffers using VERW */ #define X86_FEATURE_ACC_POWER ( 3*32+19) /* AMD Accumulated Power Mechanism */ #define X86_FEATURE_NOPL ( 3*32+20) /* The NOPL (0F 1F) instructions */ #define X86_FEATURE_ALWAYS ( 3*32+21) /* "" Always-present feature */ diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index 8bd8ed576f91..da1b5a20dea2 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -306,6 +306,17 @@ #endif .endm +/* + * Macro to execute VERW instruction that mitigate transient data sampling + * attacks such as MDS. On affected systems a microcode update overloaded VERW + * instruction to also clear the CPU buffers. VERW clobbers CFLAGS.ZF. + * + * Note: Only the memory operand variant of VERW clears the CPU buffers. + */ +.macro CLEAR_CPU_BUFFERS + ALTERNATIVE "", __stringify(verw _ASM_RIP(mds_verw_sel)), X86_FEATURE_CLEAR_CPU_BUF +.endm + #else /* __ASSEMBLY__ */ #define ANNOTATE_RETPOLINE_SAFE \ @@ -529,6 +540,8 @@ DECLARE_STATIC_KEY_FALSE(switch_mm_cond_l1d_flush); DECLARE_STATIC_KEY_FALSE(mmio_stale_data_clear); +extern u16 mds_verw_sel; + #include /** From patchwork Wed Feb 14 02:21:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 200823 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp1022422dyb; Tue, 13 Feb 2024 22:30:25 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUACCZkNc29+ir2IVx6BxZeObA9uO25DR28UqsfA1NqZJwq6/0ManMSfrwlMrzuapwvtD+/mkJsHYVrracK/8MJl5tSCw== X-Google-Smtp-Source: AGHT+IH/2Odt0x6z3qJfmJkAhiJZ08lcWHEu0D1zhG12oqHUfrsJhOKfeKdWBamUAyuLqLTyVjNy X-Received: by 2002:a05:6214:2622:b0:685:6715:9693 with SMTP id gv2-20020a056214262200b0068567159693mr2422090qvb.8.1707892225213; Tue, 13 Feb 2024 22:30:25 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707892225; cv=pass; d=google.com; s=arc-20160816; b=MJ8N3b0kt2j2qwzws24u0d1dR7xaMdOmQXgDsEQuBFXr3QKSqiwAj5TGr7DA0ZlK46 C/qCafTd3JbyjRXcg8zAiZHXUqyUafwoAMp7hEluDPuHWSCXgIznPLAQjdkU12G/n0QU Gh2dcfmjRBYIMhmYaS8thx4QpIEF2Lvv1s4rA8C6iM1bwUv/mKMOwOHUQvbjEtfPjSPL Oxez/BzZ3ukl8PjJDDWZYHsAChgaTrvVLwr/DncxHcgY9moMx6hUTSYjJU/PsF1LRR0L /pl1AI40T5/cyX3GMBmVHuF1J0PNvyOV5pta+h0PS0RKp9UwOLa1DMkc3Vld7uCAukm2 lWmA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:message-id:subject:cc :to:from:date:dkim-signature; bh=HEHTscg2GNsby7u8dw6BhBxvk+vKsO5pQVAkBhxZbLw=; fh=BaQb903muLeLL+sUGjsVxlwp2HQODeVocZSln5PXLD8=; b=gQZ5N4cvvK1tHv8FZD6xuZcdSfy2JtmF6fF37+PUyeKLafH5rEg6l8JPZfNdQdcI8P LoQSUuBP6qGg0yHzZ0+V1LKDpEGZyCAjl+9oE+/KSzZ719FwtZV4OKtZF+Wez8pElb6c yUO8FlkvcfZGQnKa7U7wEUlFW/OQLNSvyDUXjARJDRu1Jhs9WuL4Ujjerqn6h+KE945G ryexmVWjpwmsW4SKHWcCsjoW2p9vAjBlMFMyP8wa5uwN8plO9iiH1X7qMrjcvF1SdVNz EBCrnMu2bmOOLQm1HS61FN1+qS0jrSzTrHUDBJ+qaJMO44+LFhuQ7sjU8uaaAm0wFzi8 M9rw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="G+rr/gHS"; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-64686-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-64686-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=2; AJvYcCUT5eVDCSGmqY3T8giSg+tPeKrs7n3p1BtO6yNDkwXNxXl/X4YV6TWZzOYqrepXMNHierUfgG63k2uf/z5AUUxXENATxQ== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id 7-20020a05621420e700b0068d14e993bfsi4734182qvk.85.2024.02.13.22.30.25 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Feb 2024 22:30:25 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-64686-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="G+rr/gHS"; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-64686-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-64686-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id C27621C283B0 for ; Wed, 14 Feb 2024 02:22:41 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1C0758F66; Wed, 14 Feb 2024 02:22:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="G+rr/gHS" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5AE2B947A; Wed, 14 Feb 2024 02:21:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707877317; cv=none; b=Kklaihcb5M+NSnEyL68/13y2GmIWP5pH1zOj25pqZ0lxH8FYWOysiS/mm1p/6HQVSp+gpWpr3pbW4fVTObIaZSqRNVjyVfD/DgDgOff9o819oKJqnFG16fsg+B24OxLK+Ao/r2Jw0YH054o+4+z+gWAa0ed5vawng3KpfEjlIy4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707877317; c=relaxed/simple; bh=730qVyOoAO3iHO+2lbrC+3aGZCiIPt6MdeACYblDLQY=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=lBZaUWEgh83KYEeil4SJg3mL1ahULX2Pt5+POYq9PQuUEKM6JK3sI5MlLCwe42lebUY75YEfFWrOO5TQ5cwVb85zSFnWpo7i/sp2l3RTY4Xpd8DRS6KXNbJaOU4Cqwl1vxJ18aLyT+DZssKvoOJ6VK/k0odVBiUmSFkZu478/ck= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=G+rr/gHS; arc=none smtp.client-ip=192.198.163.18 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707877315; x=1739413315; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=730qVyOoAO3iHO+2lbrC+3aGZCiIPt6MdeACYblDLQY=; b=G+rr/gHSZCF3lAAtKhVP4NbpkA8aUuEdbNX7C3KjaMwzPYdBA1IXtA1O Jzg8c3FGfBYh4Cclr3f6eUN7nIcJI0TH3PA4/RNVDOyROoWNupCczDhAg XMANTxuLL0T2UsCTzLYGaVrL/c18C+hnqMTMWoq5eEAuz8YHoMlOJ8nX4 w+IZA8RjqMFQMwwcej9mn1l9O28WeInyH/Y2O4bdc8CnT/ywhb4cinmcU 069Jcc6Sr9PnAnX3fkhwRjUQLW1MQv3mrbUMh8aSKLOA27YdlOkihI31T Yi3PPSFuXq3qjEyYzdjLMymjHrHSNWvoXZayx/kiLL0krGi41mViJ8ueL w==; X-IronPort-AV: E=McAfee;i="6600,9927,10982"; a="1784869" X-IronPort-AV: E=Sophos;i="6.06,158,1705392000"; d="scan'208";a="1784869" Received: from orviesa005.jf.intel.com ([10.64.159.145]) by fmvoesa112.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2024 18:21:53 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,158,1705392000"; d="scan'208";a="7710856" Received: from diegoavi-mobl.amr.corp.intel.com (HELO desk) ([10.255.230.185]) by orviesa005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2024 18:21:53 -0800 Date: Tue, 13 Feb 2024 18:21:52 -0800 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com Subject: [PATCH v8 2/6] x86/entry_64: Add VERW just before userspace transition Message-ID: <20240213-delay-verw-v8-2-a6216d83edb7@linux.intel.com> X-Mailer: b4 0.12.3 References: <20240213-delay-verw-v8-0-a6216d83edb7@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240213-delay-verw-v8-0-a6216d83edb7@linux.intel.com> X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790854798097192190 X-GMAIL-MSGID: 1790854798097192190 Mitigation for MDS is to use VERW instruction to clear any secrets in CPU Buffers. Any memory accesses after VERW execution can still remain in CPU buffers. It is safer to execute VERW late in return to user path to minimize the window in which kernel data can end up in CPU buffers. There are not many kernel secrets to be had after SWITCH_TO_USER_CR3. Add support for deploying VERW mitigation after user register state is restored. This helps minimize the chances of kernel data ending up into CPU buffers after executing VERW. Note that the mitigation at the new location is not yet enabled. Corner case not handled ======================= Interrupts returning to kernel don't clear CPUs buffers since the exit-to-user path is expected to do that anyways. But, there could be a case when an NMI is generated in kernel after the exit-to-user path has cleared the buffers. This case is not handled and NMI returning to kernel don't clear CPU buffers because: 1. It is rare to get an NMI after VERW, but before returning to userspace. 2. For an unprivileged user, there is no known way to make that NMI less rare or target it. 3. It would take a large number of these precisely-timed NMIs to mount an actual attack. There's presumably not enough bandwidth. 4. The NMI in question occurs after a VERW, i.e. when user state is restored and most interesting data is already scrubbed. Whats left is only the data that NMI touches, and that may or may not be of any interest. Suggested-by: Dave Hansen Cc: stable@kernel.org Signed-off-by: Pawan Gupta --- arch/x86/entry/entry_64.S | 11 +++++++++++ arch/x86/entry/entry_64_compat.S | 1 + 2 files changed, 12 insertions(+) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index dfb9b8c66123..8af2a26b24f6 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -161,6 +161,7 @@ syscall_return_via_sysret: SYM_INNER_LABEL(entry_SYSRETQ_unsafe_stack, SYM_L_GLOBAL) ANNOTATE_NOENDBR swapgs + CLEAR_CPU_BUFFERS sysretq SYM_INNER_LABEL(entry_SYSRETQ_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR @@ -571,6 +572,7 @@ SYM_INNER_LABEL(swapgs_restore_regs_and_return_to_usermode, SYM_L_GLOBAL) .Lswapgs_and_iret: swapgs + CLEAR_CPU_BUFFERS /* Assert that the IRET frame indicates user mode. */ testb $3, 8(%rsp) jnz .Lnative_iret @@ -721,6 +723,8 @@ native_irq_return_ldt: */ popq %rax /* Restore user RAX */ + CLEAR_CPU_BUFFERS + /* * RSP now points to an ordinary IRET frame, except that the page * is read-only and RSP[31:16] are preloaded with the userspace @@ -1446,6 +1450,12 @@ nmi_restore: std movq $0, 5*8(%rsp) /* clear "NMI executing" */ + /* + * Skip CLEAR_CPU_BUFFERS here, since it only helps in rare cases like + * NMI in kernel after user state is restored. For an unprivileged user + * these conditions are hard to meet. + */ + /* * iretq reads the "iret" frame and exits the NMI stack in a * single instruction. We are returning to kernel mode, so this @@ -1463,6 +1473,7 @@ SYM_CODE_START(entry_SYSCALL32_ignore) UNWIND_HINT_END_OF_STACK ENDBR mov $-ENOSYS, %eax + CLEAR_CPU_BUFFERS sysretl SYM_CODE_END(entry_SYSCALL32_ignore) diff --git a/arch/x86/entry/entry_64_compat.S b/arch/x86/entry/entry_64_compat.S index de94e2e84ecc..eabf48c4d4b4 100644 --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -270,6 +270,7 @@ SYM_INNER_LABEL(entry_SYSRETL_compat_unsafe_stack, SYM_L_GLOBAL) xorl %r9d, %r9d xorl %r10d, %r10d swapgs + CLEAR_CPU_BUFFERS sysretl SYM_INNER_LABEL(entry_SYSRETL_compat_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR From patchwork Wed Feb 14 02:22:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 200798 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp988938dyb; Tue, 13 Feb 2024 20:49:53 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVDzBXdI4JzXFNQ75KRt4EAU7MHToGldn+kAHn3nQfyZnPZf4FdaicBKM+CXWkCyTUEAF3YYpgE5W3NMgdudTFvTtETwQ== X-Google-Smtp-Source: AGHT+IF/K3LVqpo65HiejYxE7jHIvGrw83rFeAto/0Rf5xIzA8FvL3/1Xxo+1R8oOWqp1Kn6GtF8 X-Received: by 2002:a05:6214:500e:b0:68c:499f:f803 with SMTP id jo14-20020a056214500e00b0068c499ff803mr2045922qvb.30.1707886192754; Tue, 13 Feb 2024 20:49:52 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707886192; cv=pass; d=google.com; s=arc-20160816; b=VpX2BEqnu7WS5DaPKesSi+BK23oppHxH/o2Lj96I5h6niWGgUc7KQ+37/j9uNV74Ly yc8BV3ANgVQ6GDQEUyOIw0ukGr0QUsj01TaC5/FWu0sfFhe80Uj0/asNPfizxSa6EBV1 wzMYTuvup8cYYIoy7pP+xv7d7SpVCZNLKWXdviBp+EMSLWDm7844PvgUXwHsCl+HBH09 +wQ+eVnFWo1VEHaSDM3i2AQyBKwISiuAMWDAorj+p8LTh5+gZjj5sTEmLBhrBZgjHK+R L9fExtxn+8TJjv7ZH8KpQPgONxDGiFDmtpWnYjpSdkw0jlBJgmw3BwoAffLlDpuC/wXp 4b0w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:message-id:subject:cc :to:from:date:dkim-signature; bh=JDnY58EnAyBHoXuUzqB9yPaCKJZTDjl2i6ea84ttwb0=; fh=6d5qO0SIIniOZU0TMetb0yxTPGRuJdBSoyMIg4+lgUQ=; b=OwjvUK1vDwaxL8kBWPWmzR5pSesiKeE1/40Fidy3YCBGZ2a5UMQ7o4Kwpwily5qYB2 jYgvFoGCuJaejN6DhMfcxOo+k1SKEHy3JsHRuRswgZ8SHCm+9AWm4UfxQRMZBn+HeKIB pYpgK5QoQukNH37TJYoLmS/l3fhWn0sCMlfgjgMlNV2bkvorWUduCu6ahlsnnRs8k+10 /ewMHXqyTijxNkTql69fe2GK7iJ9TGweUMN/yq6U7y4fbHGI4jeWd7aHVTwMNjPCYOgm vyj0cXgavLdyFLEJnIjGj95AFnCHhG1XOjyCZsr6s3re2MozW7p1FCtqTmOj6JS0+6TT Kq7w==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="BnH/c9o+"; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-64687-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-64687-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=2; AJvYcCWlzdjFgbcdpu8eDAMHWFYZ5PQxf7S6ehUlBzcJw3/wI3+8n9Gvv5kHfxfh6Py3kBlGuJY18/ZauRx1pNiK5jsruRDXRA== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id cm19-20020a056a020a1300b005dc85dc982asi2296823pgb.832.2024.02.13.20.49.52 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Feb 2024 20:49:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-64687-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="BnH/c9o+"; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-64687-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-64687-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 921B328DF39 for ; Wed, 14 Feb 2024 02:23:01 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id AD40E10A1B; Wed, 14 Feb 2024 02:22:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="BnH/c9o+" Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B061DBE48; Wed, 14 Feb 2024 02:22:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707877331; cv=none; b=L6RPPfyEKyyxisMyZ6TDq8BEd9gcEpPadF0Eog3LEjURMEmU/cMhntNn6hscf+MY0rqzl3XnupREWEgF3nXbWQO6etO/jFYoNlMQy84QsR8ezxWm4ZCvjykAMG5MQmnsRL19KaqS1AZ/iNnGTYdMbjZ7+KTk6WZoLz1AMkCmtw0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707877331; c=relaxed/simple; bh=rWRFdEOwJ9VATethxyyPtZVSQTxJREjdJD4RCKAgKSo=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=oNMg+Ej8CI0aR33ibC975oAtR89WI25qAEzAcP3iRLW7ZGvK8chfpeyKsdaDa+zVclEU15FMlG3OYWttlx1joHmzs+hPSar/0zF34KNNI2g67ZZJzLHmACzjfSAwJrghe5FnUOPXi2LlWa3oL65ROt/hXNxhbzc/nNo1aV60vIc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=BnH/c9o+; arc=none smtp.client-ip=198.175.65.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707877329; x=1739413329; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=rWRFdEOwJ9VATethxyyPtZVSQTxJREjdJD4RCKAgKSo=; b=BnH/c9o+sfWSpPjiF3oHxAALqjiHB9jrBK0zzE6sNT6HgguYNHq0kYRT Wm2FUQ8WP7rEPnHNe8XtT0BB+1Vtb2o4KdhafcDL1k74p/GarmjLfCvCy gzGabcGNq0CR3btwz9jPFDz6VzxdPWt6Pjnv4GRLQRY7m+O6zvOwMC89X z3UWmFqFiKge1RO+xU7TepYBZgx8mU/K0bZ8cLNMMh1/Sh6YeiQGLqjWs QhoqFrvaGCUDvPX/NsyL9txFH0irdEDwMi2tisHyjq6S6Zny1FrMwcM9M ZNldme6qpIk7Dnasbx+8NEE5RN4NEkQagMoH8Mln8sYXHrubcQnQJyPa4 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10982"; a="2058739" X-IronPort-AV: E=Sophos;i="6.06,158,1705392000"; d="scan'208";a="2058739" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by orvoesa108.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2024 18:22:08 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,158,1705392000"; d="scan'208";a="3417034" Received: from diegoavi-mobl.amr.corp.intel.com (HELO desk) ([10.255.230.185]) by orviesa006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2024 18:22:08 -0800 Date: Tue, 13 Feb 2024 18:22:08 -0800 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com Subject: [PATCH v8 3/6] x86/entry_32: Add VERW just before userspace transition Message-ID: <20240213-delay-verw-v8-3-a6216d83edb7@linux.intel.com> X-Mailer: b4 0.12.3 References: <20240213-delay-verw-v8-0-a6216d83edb7@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240213-delay-verw-v8-0-a6216d83edb7@linux.intel.com> X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790848472782508941 X-GMAIL-MSGID: 1790848472782508941 As done for entry_64, add support for executing VERW late in exit to user path for 32-bit mode. Cc: stable@kernel.org Signed-off-by: Pawan Gupta --- arch/x86/entry/entry_32.S | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S index 1b0fe4b49ea0..d3a814efbff6 100644 --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -881,6 +881,7 @@ SYM_FUNC_START(entry_SYSENTER_32) BUG_IF_WRONG_CR3 no_user_check=1 popfl popl %eax + CLEAR_CPU_BUFFERS /* * Return back to the vDSO, which will pop ecx and edx. @@ -950,6 +951,7 @@ restore_all_switch_stack: /* Restore user state */ RESTORE_REGS pop=4 # skip orig_eax/error_code + CLEAR_CPU_BUFFERS .Lirq_return: /* * ARCH_HAS_MEMBARRIER_SYNC_CORE rely on IRET core serialization @@ -1142,6 +1144,7 @@ SYM_CODE_START(asm_exc_nmi) /* Not on SYSENTER stack. */ call exc_nmi + CLEAR_CPU_BUFFERS jmp .Lnmi_return .Lnmi_from_sysenter_stack: From patchwork Wed Feb 14 02:22:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 200810 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp1001794dyb; Tue, 13 Feb 2024 21:26:27 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXmPGEeFyhAyn1iW7fWl9MWYCqBOHR+J28DMnOizM+ZyIsAwt/3doUvjYCyHDp4fWvPAyV4/87oPvXPYOKw4FAIOd717A== X-Google-Smtp-Source: AGHT+IEjRIRSptku61oR+kw6R/zV/wqHc5QZl2AW1VfmDuQF3eaJG4oBZxrsmtROg3Z8Pybgd6kU X-Received: by 2002:aa7:c49a:0:b0:55f:d95c:923 with SMTP id m26-20020aa7c49a000000b0055fd95c0923mr1239878edq.25.1707888387530; Tue, 13 Feb 2024 21:26:27 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707888387; cv=pass; d=google.com; s=arc-20160816; b=ZHaizkpOysmlLdNCVF0kp46qEvJiF+SRBrUonDR3MxXUspXj3ptwpQyY0XDD/L+BbB 7CEmaNGPHF7FyJgP+HeDP4Ck5EnArhW4ZilHHoD1b5C23UmsT3mkYLf24n3UoO5oD9+o yG5foSnTCScoZ/8rfSwlS/bRt9FBWjUim5JtToM1bgja5xfHtFRTf06Xm3tmTCxGV5nu 3wNjrcFVN9JZ0Fd8U8I64zyYln4mpdkk2Tx/XVaSiIboFwTa7yAjT2Idt0FQ7IK7i0Uj qLUSjVjm5l9s5/0k3NLdPeixqUVeDq0UtSgDwBC/bymfiRvxAhteOgTVSEld2imf4GFD eTqg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:message-id:subject:cc :to:from:date:dkim-signature; bh=/D6SO2KTPnncSsHH2nw6H9riyA9PoPWY2gM/OWrPCno=; fh=i2URiyAasBjvUVs6V32bnYbeNw/U2rgL5jCcrcP1TmU=; b=n5ATKQg8zoLO8LfLMabVusAM8HpzOytSKZmQzKD2cLz0vGFg1S5Z9ypH9OVDCmM98E 1APOw9I+TfUgDiBXeZOoJqbz+d+p/4Gsc8tVgdBYi4bMo4zJculSTXRzYkaTU1zmpvWt CadP+LzDOA712q9Vq/K3Vdfj+VRImZtfbdVjgI5aSeq0iMCtJi7zoG1BaLpQ1r8FybDi wDdBSTBj8JD6tt7aPZA/qMvYiZS97wQY3PYEFfBGAOijEKJflUuZwpgcsaX34YhvSmLI jLBPNE9P0FHqP2GFQN4APF2/J/Nu/yzdLzGCxhGd/AH7I7cemm9EFplnHSjNrxo/CcJ2 Oolg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=GJJxKrxl; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-64688-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-64688-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=2; AJvYcCW5Jx3u6Ikh8IbB+GDX9GeUL5wXyOlCdSfdw6fOK66HVmzAqt2O1D2WdyoNRi0yBAjIDAOXX/439wakDw6/DM7bdKMYyA== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id v10-20020aa7d64a000000b0056385632490si46817edr.178.2024.02.13.21.26.27 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Feb 2024 21:26:27 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-64688-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=GJJxKrxl; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-64688-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-64688-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 337F11F2A319 for ; Wed, 14 Feb 2024 02:23:24 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id CD57711706; Wed, 14 Feb 2024 02:22:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="GJJxKrxl" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4FC2D10A25; Wed, 14 Feb 2024 02:22:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.10 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707877348; cv=none; b=klb90+mRi0CVPuT8sgkz/N3OIZFth6w1vhq1Ys6pO2JzxDUvAk0h5Jtn5etws0giFe6CKT4eq4h3yKBRHNMQrk1jWmAJtLTfrnqdAL3kAHyy0pepIXAXJL9wrQIapCbrFRnsvvXvn5NEoTjPuz5YlHTziv2/aaZk1k0btjX9Ho4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707877348; c=relaxed/simple; bh=+qrAwU7Ai29jF1VMtksHOHSiDckKmD+ywwD6BI5fBYw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=XVn7RCsGi7KLHX/yHIzhcgCs1weXxCVWF0PtnV2GMycYKB67ihAPq1gz0QZYlyP4n3AI73lJ6DfR+0GzOrRU3M+m73t5jPAgoHy7KdLOsO2iz67fccANUVsc8YlB5lscQEk7aBtwqdI/cSzGFRejh3VXQnmEyCvubD7isGHXFs0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=GJJxKrxl; arc=none smtp.client-ip=192.198.163.10 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707877346; x=1739413346; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=+qrAwU7Ai29jF1VMtksHOHSiDckKmD+ywwD6BI5fBYw=; b=GJJxKrxlKDAzDgtJ+zEVhIWH8ayJTMEo5Ar/1WGV43kJDdoYUDGAN933 pf4hwihtYbBTthQpGVLOkWZEv77eGijVQ15OelB0kuHbTTOEhKeFOBgmE YIPdNoS6c7IFyH89wuAtD1dmKOrN5oEbOdhTMBVR3V6DVgTfYPftKEXUJ NvCiJ1a6QxW0zWDLbL02d0REj6+BQRyHRxs6/7OM9tH/QFAXpCUkyGI8z RLSTEHK6rYTjoy4Wbn+3OB0SVTLqsyqPQ5AFzduibSNH/j8KpZSBMqYXV EFVijmudC1EhL97MJ5dK6HBHbitkMrb3MFrrQOWELWm3OIvyoLVtp+rAn A==; X-IronPort-AV: E=McAfee;i="6600,9927,10982"; a="13297980" X-IronPort-AV: E=Sophos;i="6.06,158,1705392000"; d="scan'208";a="13297980" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2024 18:22:25 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10982"; a="826284913" X-IronPort-AV: E=Sophos;i="6.06,158,1705392000"; d="scan'208";a="826284913" Received: from diegoavi-mobl.amr.corp.intel.com (HELO desk) ([10.255.230.185]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2024 18:22:24 -0800 Date: Tue, 13 Feb 2024 18:22:24 -0800 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com Subject: [PATCH v8 4/6] x86/bugs: Use ALTERNATIVE() instead of mds_user_clear static key Message-ID: <20240213-delay-verw-v8-4-a6216d83edb7@linux.intel.com> X-Mailer: b4 0.12.3 References: <20240213-delay-verw-v8-0-a6216d83edb7@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240213-delay-verw-v8-0-a6216d83edb7@linux.intel.com> X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790850773484119979 X-GMAIL-MSGID: 1790850773484119979 The VERW mitigation at exit-to-user is enabled via a static branch mds_user_clear. This static branch is never toggled after boot, and can be safely replaced with an ALTERNATIVE() which is convenient to use in asm. Switch to ALTERNATIVE() to use the VERW mitigation late in exit-to-user path. Also remove the now redundant VERW in exc_nmi() and arch_exit_to_user_mode(). Cc: stable@kernel.org Signed-off-by: Pawan Gupta --- Documentation/arch/x86/mds.rst | 38 +++++++++++++++++++++++++----------- arch/x86/include/asm/entry-common.h | 1 - arch/x86/include/asm/nospec-branch.h | 12 ------------ arch/x86/kernel/cpu/bugs.c | 15 ++++++-------- arch/x86/kernel/nmi.c | 3 --- arch/x86/kvm/vmx/vmx.c | 2 +- 6 files changed, 34 insertions(+), 37 deletions(-) diff --git a/Documentation/arch/x86/mds.rst b/Documentation/arch/x86/mds.rst index e73fdff62c0a..c58c72362911 100644 --- a/Documentation/arch/x86/mds.rst +++ b/Documentation/arch/x86/mds.rst @@ -95,6 +95,9 @@ The kernel provides a function to invoke the buffer clearing: mds_clear_cpu_buffers() +Also macro CLEAR_CPU_BUFFERS can be used in ASM late in exit-to-user path. +Other than CFLAGS.ZF, this macro doesn't clobber any registers. + The mitigation is invoked on kernel/userspace, hypervisor/guest and C-state (idle) transitions. @@ -138,17 +141,30 @@ Mitigation points When transitioning from kernel to user space the CPU buffers are flushed on affected CPUs when the mitigation is not disabled on the kernel - command line. The migitation is enabled through the static key - mds_user_clear. - - The mitigation is invoked in prepare_exit_to_usermode() which covers - all but one of the kernel to user space transitions. The exception - is when we return from a Non Maskable Interrupt (NMI), which is - handled directly in do_nmi(). - - (The reason that NMI is special is that prepare_exit_to_usermode() can - enable IRQs. In NMI context, NMIs are blocked, and we don't want to - enable IRQs with NMIs blocked.) + command line. The mitigation is enabled through the feature flag + X86_FEATURE_CLEAR_CPU_BUF. + + The mitigation is invoked just before transitioning to userspace after + user registers are restored. This is done to minimize the window in + which kernel data could be accessed after VERW e.g. via an NMI after + VERW. + + **Corner case not handled** + Interrupts returning to kernel don't clear CPUs buffers since the + exit-to-user path is expected to do that anyways. But, there could be + a case when an NMI is generated in kernel after the exit-to-user path + has cleared the buffers. This case is not handled and NMI returning to + kernel don't clear CPU buffers because: + + 1. It is rare to get an NMI after VERW, but before returning to userspace. + 2. For an unprivileged user, there is no known way to make that NMI + less rare or target it. + 3. It would take a large number of these precisely-timed NMIs to mount + an actual attack. There's presumably not enough bandwidth. + 4. The NMI in question occurs after a VERW, i.e. when user state is + restored and most interesting data is already scrubbed. Whats left + is only the data that NMI touches, and that may or may not be of + any interest. 2. C-State transition diff --git a/arch/x86/include/asm/entry-common.h b/arch/x86/include/asm/entry-common.h index ce8f50192ae3..7e523bb3d2d3 100644 --- a/arch/x86/include/asm/entry-common.h +++ b/arch/x86/include/asm/entry-common.h @@ -91,7 +91,6 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, static __always_inline void arch_exit_to_user_mode(void) { - mds_user_clear_cpu_buffers(); amd_clear_divider(); } #define arch_exit_to_user_mode arch_exit_to_user_mode diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index da1b5a20dea2..fc3a8a3c7ffe 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -533,7 +533,6 @@ DECLARE_STATIC_KEY_FALSE(switch_to_cond_stibp); DECLARE_STATIC_KEY_FALSE(switch_mm_cond_ibpb); DECLARE_STATIC_KEY_FALSE(switch_mm_always_ibpb); -DECLARE_STATIC_KEY_FALSE(mds_user_clear); DECLARE_STATIC_KEY_FALSE(mds_idle_clear); DECLARE_STATIC_KEY_FALSE(switch_mm_cond_l1d_flush); @@ -567,17 +566,6 @@ static __always_inline void mds_clear_cpu_buffers(void) asm volatile("verw %[ds]" : : [ds] "m" (ds) : "cc"); } -/** - * mds_user_clear_cpu_buffers - Mitigation for MDS and TAA vulnerability - * - * Clear CPU buffers if the corresponding static key is enabled - */ -static __always_inline void mds_user_clear_cpu_buffers(void) -{ - if (static_branch_likely(&mds_user_clear)) - mds_clear_cpu_buffers(); -} - /** * mds_idle_clear_cpu_buffers - Mitigation for MDS vulnerability * diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index a78892b0f823..b34c21d73239 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -111,9 +111,6 @@ DEFINE_STATIC_KEY_FALSE(switch_mm_cond_ibpb); /* Control unconditional IBPB in switch_mm() */ DEFINE_STATIC_KEY_FALSE(switch_mm_always_ibpb); -/* Control MDS CPU buffer clear before returning to user space */ -DEFINE_STATIC_KEY_FALSE(mds_user_clear); -EXPORT_SYMBOL_GPL(mds_user_clear); /* Control MDS CPU buffer clear before idling (halt, mwait) */ DEFINE_STATIC_KEY_FALSE(mds_idle_clear); EXPORT_SYMBOL_GPL(mds_idle_clear); @@ -252,7 +249,7 @@ static void __init mds_select_mitigation(void) if (!boot_cpu_has(X86_FEATURE_MD_CLEAR)) mds_mitigation = MDS_MITIGATION_VMWERV; - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); if (!boot_cpu_has(X86_BUG_MSBDS_ONLY) && (mds_nosmt || cpu_mitigations_auto_nosmt())) @@ -356,7 +353,7 @@ static void __init taa_select_mitigation(void) * For guests that can't determine whether the correct microcode is * present on host, enable the mitigation for UCODE_NEEDED as well. */ - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); if (taa_nosmt || cpu_mitigations_auto_nosmt()) cpu_smt_disable(false); @@ -424,7 +421,7 @@ static void __init mmio_select_mitigation(void) */ if (boot_cpu_has_bug(X86_BUG_MDS) || (boot_cpu_has_bug(X86_BUG_TAA) && boot_cpu_has(X86_FEATURE_RTM))) - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); else static_branch_enable(&mmio_stale_data_clear); @@ -484,12 +481,12 @@ static void __init md_clear_update_mitigation(void) if (cpu_mitigations_off()) return; - if (!static_key_enabled(&mds_user_clear)) + if (!boot_cpu_has(X86_FEATURE_CLEAR_CPU_BUF)) goto out; /* - * mds_user_clear is now enabled. Update MDS, TAA and MMIO Stale Data - * mitigation, if necessary. + * X86_FEATURE_CLEAR_CPU_BUF is now enabled. Update MDS, TAA and MMIO + * Stale Data mitigation, if necessary. */ if (mds_mitigation == MDS_MITIGATION_OFF && boot_cpu_has_bug(X86_BUG_MDS)) { diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c index db1c1848a1e6..604d8c996d9c 100644 --- a/arch/x86/kernel/nmi.c +++ b/arch/x86/kernel/nmi.c @@ -564,9 +564,6 @@ DEFINE_IDTENTRY_RAW(exc_nmi) } if (this_cpu_dec_return(nmi_state)) goto nmi_restart; - - if (user_mode(regs)) - mds_user_clear_cpu_buffers(); } #if IS_ENABLED(CONFIG_KVM_INTEL) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index fd19ef314bcb..40594eae2cd3 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7230,7 +7230,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, /* L1D Flush includes CPU buffer clear to mitigate MDS */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); - else if (static_branch_unlikely(&mds_user_clear)) + else if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF)) mds_clear_cpu_buffers(); else if (static_branch_unlikely(&mmio_stale_data_clear) && kvm_arch_has_assigned_device(vcpu->kvm)) From patchwork Wed Feb 14 02:22:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 200828 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp1025568dyb; Tue, 13 Feb 2024 22:39:52 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCW6Wg1tzR6W6i1lOdZiFO+K946nOY2GBa3YA7/n1pcyIfE6xO9WRSy+8mcJ0ujFSYsGfpW7asN8KPq4Awlr/ekOLoWFWQ== X-Google-Smtp-Source: AGHT+IEXfhH1a3ForXpAd1qnKUkoRN7tgq6d6kx/dxYGsuvtJR+JGtCZd5ivO6XrUJVdq6kDUlFo X-Received: by 2002:a17:902:f54e:b0:1da:2122:be72 with SMTP id h14-20020a170902f54e00b001da2122be72mr1968274plf.62.1707892791919; Tue, 13 Feb 2024 22:39:51 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707892791; cv=pass; d=google.com; s=arc-20160816; b=TeItfqVwewlC9WUrG2E95XhjQc5Nz2AIJYfRtc/bDt4+40xLx2FNHlevkSAqYLNhcu LJSsWdT07Zm9ANpygu2nrr+4hgVrIs6avNv7eEapZ+o2ac2nczGnNIj44mKFdSKj1FBw shwJFcF2dnV9FczzQKaWMTuY2XS/PRPKm1TnYhbE5gw4tvcGn9ZAzjzhOMo+GBN+YfLS oi4b0J3nxhwo6Q8VjkOHjcuLyvCUf3OgIZo5Ot5Go46Qs7bmsOtOga9YDbxRbZ1vimOX OhGUUaSPst6LrXen7ejAnPPKKmc4t6FMwpVy8PZxc0NBPm4KqErfxl3cd+oBGsnO53gf 5GQA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:message-id:subject:cc :to:from:date:dkim-signature; bh=/7UK3x3tKrFCfYJG7YwyIIYKLN4qC7RJ/CJ7sQXwahs=; fh=hG4UUp1oC+LwLbUDqxfiPve8iqQ6gt9x8REnzohJjCA=; b=AxQuMwXvei4gqxgFY8BvLb5k40p7+VZqkBF70A8yEmbZJDmUWxyvXdad20Pl4WdLkR oAQRilLmMxRtIp7NqAhZub2RBBG456/wO/bH8E76gZm7OieUh0LhPCs9xHYFUZLytN65 vK6kIz3z0krhrzixXxBsE1KxOHYkoPINhqbFhK7k6Z+Pa89htomhPsXfDF6wgir9ajNZ IDYGo3luvWLvzsDwshHestS5UW6Mnz9Dry3UL0f1FKNp6GL2yUB7onkE0eyw4IArGGnx n/EUaJGQBFssp0pslrI3wVmf/nM7P5ojqqfmqN2HuAMQLNNC7EKxEfQceDHgMPnW7ZpK r2Xg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=XHiS2cTA; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-64690-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-64690-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Forwarded-Encrypted: i=2; AJvYcCV7VCvNcVIcdYd+7UfjTWFOV/3r833ljbHfEKjwv/LyrWCe0xEeRtjdzV8+S3ZDtX2SYa+BlfD7WlsfRhQcCVqsc1xHGw== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id l18-20020a170902d35200b001d981d92655si3255740plk.308.2024.02.13.22.39.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Feb 2024 22:39:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-64690-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=XHiS2cTA; arc=pass (i=1 dkim=pass dkdomain=intel.com dmarc=pass fromdomain=linux.intel.com); spf=pass (google.com: domain of linux-kernel+bounces-64690-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-64690-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id EBA3528F274 for ; Wed, 14 Feb 2024 02:24:02 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8D325D30B; Wed, 14 Feb 2024 02:23:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="XHiS2cTA" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6BEF7BA22; Wed, 14 Feb 2024 02:22:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.7 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707877380; cv=none; b=EZTEpkuDSk6x+IrnK5ASQWBw6ve/cCQYy5ZMpIrceTz+t8TFXluSXSkNu/rDv4A3mlJ9hFIvAg3l+LpWCzq+eHMgz9RvEwClu3zWNeaEUnMnQuOjuXnWYc0OuEW5hMaQzN9sI4i5ch7Nro4ErBMhgX3aRrVW25HiTYeVKyNpMgk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707877380; c=relaxed/simple; bh=mYU4BqeqoU+8q8eBooFhBgcQd6WMmusiCtW20ZTS80g=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=IOZr2ZF3Pa3mUNvMgMWkd2TMmO9l2dJNBLLdePVoLtcuxKGvVG8+S6CLpFC3BZ/ot2SS3FuBHkZLXoyEUTzNTleej59UZe/MhUtbC5hXMAoYt9/N0v2/FzxM3CDu2r4amwFz8lAfDW0ain9Ubgxv1VDiTN7uE08IQnN0xHdMFV0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=XHiS2cTA; arc=none smtp.client-ip=192.198.163.7 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707877378; x=1739413378; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=mYU4BqeqoU+8q8eBooFhBgcQd6WMmusiCtW20ZTS80g=; b=XHiS2cTAOLhWqZY7Ze4kn/BbNSLHUBmbzJ1Nq1Kdb8ecU1J58aCzZls3 bRduvcxXRe11axdwF6e1B1oVpCccvVMhOKKxRTpF35vQ13ufR3QDAQC+y 1SMCDO2TSO5yHAHEXoZIxi2ADvIl6k04QrQEk5sb5wr+X/UlIr/kY+I6j oRP2miooBHaGm+iRWe+cz+96TbEHiELc/rl4EglTsgIIf/0755Vzy8brz Cj2g4J6mqw9uJxP8emuj0ZIn781WFVSs+x+GdMyt26tss1P9fd4HMGfEf A46yGohgmaoylMX1FG8SrDjYBdUxGA5XyGyIaDmMppayC8TTDrluQlpO9 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10982"; a="27359354" X-IronPort-AV: E=Sophos;i="6.06,158,1705392000"; d="scan'208";a="27359354" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2024 18:22:57 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,158,1705392000"; d="scan'208";a="26229202" Received: from diegoavi-mobl.amr.corp.intel.com (HELO desk) ([10.255.230.185]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2024 18:22:56 -0800 Date: Tue, 13 Feb 2024 18:22:56 -0800 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com Subject: [PATCH v8 6/6] KVM: VMX: Move VERW closer to VMentry for MDS mitigation Message-ID: <20240213-delay-verw-v8-6-a6216d83edb7@linux.intel.com> X-Mailer: b4 0.12.3 References: <20240213-delay-verw-v8-0-a6216d83edb7@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240213-delay-verw-v8-0-a6216d83edb7@linux.intel.com> X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790855392392217449 X-GMAIL-MSGID: 1790855392392217449 During VMentry VERW is executed to mitigate MDS. After VERW, any memory access like register push onto stack may put host data in MDS affected CPU buffers. A guest can then use MDS to sample host data. Although likelihood of secrets surviving in registers at current VERW callsite is less, but it can't be ruled out. Harden the MDS mitigation by moving the VERW mitigation late in VMentry path. Note that VERW for MMIO Stale Data mitigation is unchanged because of the complexity of per-guest conditional VERW which is not easy to handle that late in asm with no GPRs available. If the CPU is also affected by MDS, VERW is unconditionally executed late in asm regardless of guest having MMIO access. Cc: stable@kernel.org Signed-off-by: Pawan Gupta Acked-by: Sean Christopherson --- arch/x86/kvm/vmx/vmenter.S | 3 +++ arch/x86/kvm/vmx/vmx.c | 20 ++++++++++++++++---- 2 files changed, 19 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index ef7cfbad4d57..2bfbf758d061 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -161,6 +161,9 @@ SYM_FUNC_START(__vmx_vcpu_run) /* Load guest RAX. This kills the @regs pointer! */ mov VCPU_RAX(%_ASM_AX), %_ASM_AX + /* Clobbers EFLAGS.ZF */ + CLEAR_CPU_BUFFERS + /* Check EFLAGS.CF from the VMX_RUN_VMRESUME bit test above. */ jnc .Lvmlaunch diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 40594eae2cd3..305237dcba88 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -389,7 +389,16 @@ static __always_inline void vmx_enable_fb_clear(struct vcpu_vmx *vmx) static void vmx_update_fb_clear_dis(struct kvm_vcpu *vcpu, struct vcpu_vmx *vmx) { - vmx->disable_fb_clear = (host_arch_capabilities & ARCH_CAP_FB_CLEAR_CTRL) && + /* + * Disable VERW's behavior of clearing CPU buffers for the guest if the + * CPU isn't affected by MDS/TAA, and the host hasn't forcefully enabled + * the mitigation. Disabling the clearing behavior provides a + * performance boost for guests that aren't aware that manually clearing + * CPU buffers is unnecessary, at the cost of MSR accesses on VM-Entry + * and VM-Exit. + */ + vmx->disable_fb_clear = !cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF) && + (host_arch_capabilities & ARCH_CAP_FB_CLEAR_CTRL) && !boot_cpu_has_bug(X86_BUG_MDS) && !boot_cpu_has_bug(X86_BUG_TAA); @@ -7227,11 +7236,14 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, guest_state_enter_irqoff(); - /* L1D Flush includes CPU buffer clear to mitigate MDS */ + /* + * L1D Flush includes CPU buffer clear to mitigate MDS, but VERW + * mitigation for MDS is done late in VMentry and is still + * executed in spite of L1D Flush. This is because an extra VERW + * should not matter much after the big hammer L1D Flush. + */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); - else if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF)) - mds_clear_cpu_buffers(); else if (static_branch_unlikely(&mmio_stale_data_clear) && kvm_arch_has_assigned_device(vcpu->kvm)) mds_clear_cpu_buffers();