From patchwork Fri Oct 20 20:44:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 156310 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2010:b0:403:3b70:6f57 with SMTP id fe16csp1316086vqb; Fri, 20 Oct 2023 13:47:11 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH+sF9H/JtyPCUconbXDzoyZ/Dy8eG2L9eZN4Q3mg9jmfszTNiPBnbz9fD7pOKUO4g28COD X-Received: by 2002:a17:90a:ff11:b0:27d:3c62:6342 with SMTP id ce17-20020a17090aff1100b0027d3c626342mr4214729pjb.4.1697834831284; Fri, 20 Oct 2023 13:47:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697834831; cv=none; d=google.com; s=arc-20160816; b=TZsn5K3fVpKtBMJRuW/NF71IkjyDjkZYc9sgXffGmf4EOtlGavgu6m0Jjuu6sU+27H vpwllQ0UM3kawWyiBTpeFewpFmrQh4yK3Wko1Y02CNyCWMXeBu4SJ71nhHFBEN6DNco4 cWgJG5miKFfr4cppKP2nqn6PbjzA4g+IfKPXPAzsbABHR8CBXP2YfXRN48jc91NYzgUy 8idMrudKqds2sXriRBUuggC/mEYmijJ/8kcH82D+9/v3/iRZEAEyF9vOT8WGZoTOtnqS 9nfd1mvdPWq1r0XCRcwfgLFbLIVpKz6dsxHMU7MNWPpRNLKAH3zwKOUo4XIEchOXBp83 rQOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=/uDv/Lq6ZhIV3GDsMiEcK+HSOZGbupGLMBcESjRYziI=; fh=2nScFibjZ9o9nwcLK7AVDq2RD1oZc3mBH5lwaxEPwe4=; b=Zsp4y6dR5ieUB+W4HtaG8/6PmFXsmk3+7Oko+F+/VyRFMmCqrLdVXwBkpvms70V/Ne Dk+GSXxDLpb3RUHCRetg4XTZGu9jDomw3wYoKAN1fxCo1mWkccdROnQn8EGQKLCxxVb0 w9JKOngA4nS1GIZt6exIHj2AcT4DX3OGebYgxuNciKqnDkKUVNvSQUZnvO7RNq/vIVle u0iWDWk8Ix0cNiQJr0L1FvQ++mC7GD8i8Jobq32uf7UDOCWZCNqYdxqiqtRo2C7vEHtp AwDS+uqit8ReaUdd+ECTTwSNRQHropJOvN6HOAL8GQacYuuY34pkn+2EmEhGcl4f6Hc/ 2pRQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=FoJYlN+M; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id bf20-20020a17090b0b1400b0027921228848si5081922pjb.133.2023.10.20.13.47.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 13:47:11 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=FoJYlN+M; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id B79638090EAF; Fri, 20 Oct 2023 13:45:33 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230383AbjJTUpB (ORCPT + 26 others); Fri, 20 Oct 2023 16:45:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229983AbjJTUpA (ORCPT ); Fri, 20 Oct 2023 16:45:00 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 95A26E8; Fri, 20 Oct 2023 13:44:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697834698; x=1729370698; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=xzkWuakHUz1psAcA/doANbYYppEAuf8OEljg8J5mWNg=; b=FoJYlN+MSoMMbtFKT8O1yBnhE4HHU4tl+Jz+nluseou204nITad+MgMF qLSwEhFvebYakO9/PNM/k6Q8wl9pG/UNOb7pXBmRPqd/g+U9wZJ4MxD93 pkmByX0qXXZluiZl5R3ZdwgbdtO7hEdl02btHiFYyodLRzcO9RNs1IkZz syPsl81eZNeps8IkgLxf9Ht8e7oZ3k4/9OiB1Y6R/DMOnRphxgZMZY/MA 1YahkF/PiOEYNBuCKfd91i/lqClESprZbmpzkuEUZnTrS1d71BDNfMNTt qkpkwiXY4Q7ib8Fs2W32CCQ0xe/kekgYnYPirQPK1LjctRmmXjxiqT/OS Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="371640195" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="371640195" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:44:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="931117471" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="931117471" Received: from hkchanda-mobl.amr.corp.intel.com (HELO desk) ([10.209.90.113]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:44:57 -0700 Date: Fri, 20 Oct 2023 13:44:57 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta , Alyssa Milburn Subject: [PATCH 1/6] x86/bugs: Add asm helpers for executing VERW Message-ID: <20231020-delay-verw-v1-1-cff54096326d@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 20 Oct 2023 13:45:33 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780308855887674140 X-GMAIL-MSGID: 1780308855887674140 MDS mitigation requires clearing the CPU buffers before returning to user. This needs to be done late in the exit-to-user path. Current location of VERW leaves a possibility of kernel data ending up in CPU buffers for memory accesses done after VERW such as: 1. Kernel data accessed by an NMI between VERW and return-to-user can remain in CPU buffers ( since NMI returning to kernel does not execute VERW to clear CPU buffers. 2. Alyssa reported that after VERW is executed, CONFIG_GCC_PLUGIN_STACKLEAK=y scrubs the stack used by a system call. Memory accesses during stack scrubbing can move kernel stack contents into CPU buffers. 3. When caller saved registers are restored after a return from function executing VERW, the kernel stack accesses can remain in CPU buffers(since they occur after VERW). To fix this VERW needs to be moved very late in exit-to-user path. In preparation for moving VERW to entry/exit asm code, create macros that can be used in asm. Also make them depend on a new feature flag X86_FEATURE_USER_CLEAR_CPU_BUF. Reported-by: Alyssa Milburn Signed-off-by: Pawan Gupta --- arch/x86/include/asm/cpufeatures.h | 2 +- arch/x86/include/asm/nospec-branch.h | 25 +++++++++++++++++++++++++ 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 58cb9495e40f..3f018dfedd5f 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -308,10 +308,10 @@ #define X86_FEATURE_SMBA (11*32+21) /* "" Slow Memory Bandwidth Allocation */ #define X86_FEATURE_BMEC (11*32+22) /* "" Bandwidth Monitoring Event Configuration */ #define X86_FEATURE_USER_SHSTK (11*32+23) /* Shadow stack support for user mode applications */ - #define X86_FEATURE_SRSO (11*32+24) /* "" AMD BTB untrain RETs */ #define X86_FEATURE_SRSO_ALIAS (11*32+25) /* "" AMD BTB untrain RETs through aliasing */ #define X86_FEATURE_IBPB_ON_VMEXIT (11*32+26) /* "" Issue an IBPB only on VMEXIT */ +#define X86_FEATURE_USER_CLEAR_CPU_BUF (11*32+27) /* "" Clear CPU buffers before returning to user */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */ diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index c55cc243592e..e1b623a27e1b 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -111,6 +111,24 @@ #define RESET_CALL_DEPTH_FROM_CALL #endif +/* + * Macro to execute VERW instruction to mitigate transient data sampling + * attacks such as MDS. On affected systems a microcode update overloaded VERW + * instruction to also clear the CPU buffers. + * + * Note: Only the memory operand variant of VERW clears the CPU buffers. To + * handle the case when VERW is executed after user registers are restored, use + * RIP to point the memory operand to a part NOPL instruction that contains + * __KERNEL_DS. + */ +#define __EXEC_VERW(m) verw _ASM_RIP(m) + +#define EXEC_VERW \ + __EXEC_VERW(551f); \ + /* nopl __KERNEL_DS(%rax) */ \ + .byte 0x0f, 0x1f, 0x80, 0x00, 0x00; \ +551: .word __KERNEL_DS; \ + /* * Fill the CPU return stack buffer. * @@ -329,6 +347,13 @@ #endif .endm +/* Clear CPU buffers before returning to user */ +.macro USER_CLEAR_CPU_BUFFERS + ALTERNATIVE "jmp .Lskip_verw_\@;", "", X86_FEATURE_USER_CLEAR_CPU_BUF + EXEC_VERW +.Lskip_verw_\@: +.endm + #else /* __ASSEMBLY__ */ #define ANNOTATE_RETPOLINE_SAFE \ From patchwork Fri Oct 20 20:45:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 156311 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2010:b0:403:3b70:6f57 with SMTP id fe16csp1316128vqb; Fri, 20 Oct 2023 13:47:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEnZMVln4FMZVR2IdCNoDcHr/0s59DGgAiRT1JZW+bOouDTwvHjUOr7GyELJ8kj356fQWcf X-Received: by 2002:a05:6a00:1402:b0:6b6:1216:d8fe with SMTP id l2-20020a056a00140200b006b61216d8femr2888462pfu.27.1697834836392; Fri, 20 Oct 2023 13:47:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697834836; cv=none; d=google.com; s=arc-20160816; b=a7TVK5P2h4JjRLPfvHdHx2dSkibRuMGmCJxKwwtVyM1skketUsBwqRBMcH1P9FMDoo 13hmN4KYvWgI4p5pBP+SkqZkelg6eBznF4FCJ5Rnko9sRMNxMiB2Fd8xVCXcfAEesURe vdAKdgPkzEN0f2Ghw2wNrC30sp7DYUyiXK058gWJAiq1UWE2Y5W+CqWgyrGSKkX5hrUY sOU+GcGEdfA5+aBdfg5cRAoAjIsF6bZGkVh9rqpSIxjqaBbF6bgKH4ZzkrJmwubpyX1K efcz324j3i2zW4O0UmDKBOxVgMI2g66zgKvOpqT5kADU6dzGq0HaWOUE1y50doQ6WGwp 4uoA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=+4bXi0i3JM+1tb66mtUEbuj02CZdqBs8bh0Zx/rzKMA=; fh=pfRkWPzvKcXRDb8bzZOHP7Lfu+YA9O7iyInvCf011xY=; b=isrT+5DHGTHiPGjm+nr/Be3F9GEaEdemkgGZuJw/iGVp+pSdB5iCSuRa4588DkAoJM ISxYr/go4gkZoMHwaZ5DkBT2V2O9CLB1ubr2bQb1DgbViKhAY4YC6akYkVenMJAOZiSM iAzr3lf/K3brzEhWgbCxeXlHeZCcuIdl+0eEBTtQIZXYh6Jf76ROrHBcycKEM0kzBmWF zJ9T14LVvo3+tXJhSF6ccRR/TXzl8R9gNvicVzL9GOuAfYaksoFkNJYzLFCgITzHttDV JKietFna9j3X+f405LndfBF0kzwyVJkeUtYRBpf7fmCQ/Qi3Heaf6YJNmBHVid29l9NP 13GA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=JtQId4Tq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id v202-20020a6361d3000000b005891f742957si2494818pgb.570.2023.10.20.13.47.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 13:47:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=JtQId4Tq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 15DC98095F4D; Fri, 20 Oct 2023 13:45:31 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230464AbjJTUpK (ORCPT + 26 others); Fri, 20 Oct 2023 16:45:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231160AbjJTUpI (ORCPT ); Fri, 20 Oct 2023 16:45:08 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA248D72; Fri, 20 Oct 2023 13:45:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697834704; x=1729370704; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=uGr8ClT2h4Zv+dGb/ooumBjLtEvJ/qtZo0hY1GnJHiw=; b=JtQId4Tq+iQT8NvWsje/7gQ+4wjmiFSQjQoSYO3NTHzd+QxGHX2E4WYI femvCycQI5en2W0y0weDxNYO9IStc15L2wbz5a8mtpUja2R15r/IOGqJM FWgiWue0E/yxhsNcDLd1YC4lgNAJ3tCO8eekbtpNfReSDNKFL6gYSn7O+ 8jj4E3GQQRsdsqHasxx8I/cwY6nV5khn990lJ3qTVpPvb2QNF+gNTh+RJ 7z3LMW5LTxWCI2uqal+hxPNCQlW1RWiDcpumRNzdwsAv1FFZBOa+EYQKu AjhJ5cDllDFVcH0WdcebcdhLvf+6eEeSQ/ul4MWOguoCiM6JddsF5JQbU Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="453048206" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="453048206" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="5217189" Received: from hkchanda-mobl.amr.corp.intel.com (HELO desk) ([10.209.90.113]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:43:52 -0700 Date: Fri, 20 Oct 2023 13:45:03 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta , Dave Hansen Subject: [PATCH 2/6] x86/entry_64: Add VERW just before userspace transition Message-ID: <20231020-delay-verw-v1-2-cff54096326d@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Fri, 20 Oct 2023 13:45:31 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780308861366336221 X-GMAIL-MSGID: 1780308861366336221 Mitigation for MDS is to use VERW instruction to clear any secrets in CPU Buffers. Any memory accesses after VERW execution can still remain in CPU buffers. It is safer to execute VERW late in return to user path to minimize the window in which kernel data can end up in CPU buffers. There are not many kernel secrets to be had after SWITCH_TO_USER_CR3. Add support for deploying VERW mitigation after user register state is restored. This helps minimize the chances of kernel data ending up into CPU buffers after executing VERW. Note that the mitigation at the new location is not yet enabled. Suggested-by: Dave Hansen Signed-off-by: Pawan Gupta --- arch/x86/entry/entry_64.S | 14 ++++++++++++++ arch/x86/entry/entry_64_compat.S | 2 ++ 2 files changed, 16 insertions(+) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 43606de22511..e72ac30f0714 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -223,6 +223,8 @@ syscall_return_via_sysret: SYM_INNER_LABEL(entry_SYSRETQ_unsafe_stack, SYM_L_GLOBAL) ANNOTATE_NOENDBR swapgs + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS sysretq SYM_INNER_LABEL(entry_SYSRETQ_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR @@ -663,6 +665,10 @@ SYM_INNER_LABEL(swapgs_restore_regs_and_return_to_usermode, SYM_L_GLOBAL) /* Restore RDI. */ popq %rdi swapgs + + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS + jmp .Lnative_iret @@ -774,6 +780,9 @@ native_irq_return_ldt: */ popq %rax /* Restore user RAX */ + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS + /* * RSP now points to an ordinary IRET frame, except that the page * is read-only and RSP[31:16] are preloaded with the userspace @@ -1502,6 +1511,9 @@ nmi_restore: std movq $0, 5*8(%rsp) /* clear "NMI executing" */ + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS + /* * iretq reads the "iret" frame and exits the NMI stack in a * single instruction. We are returning to kernel mode, so this @@ -1520,6 +1532,8 @@ SYM_CODE_START(ignore_sysret) UNWIND_HINT_END_OF_STACK ENDBR mov $-ENOSYS, %eax + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS sysretl SYM_CODE_END(ignore_sysret) #endif diff --git a/arch/x86/entry/entry_64_compat.S b/arch/x86/entry/entry_64_compat.S index 70150298f8bd..d2ccd9148239 100644 --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -271,6 +271,8 @@ SYM_INNER_LABEL(entry_SYSRETL_compat_unsafe_stack, SYM_L_GLOBAL) xorl %r9d, %r9d xorl %r10d, %r10d swapgs + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS sysretl SYM_INNER_LABEL(entry_SYSRETL_compat_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR From patchwork Fri Oct 20 20:45:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 156312 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2010:b0:403:3b70:6f57 with SMTP id fe16csp1316131vqb; Fri, 20 Oct 2023 13:47:16 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHdTNczhcuRN6cawN21Da7fQHd58kdVyCRLAfyjqUQwpeiEFdx+RlAr/Up3TY10QnAfxdlN X-Received: by 2002:a92:d481:0:b0:357:5179:6453 with SMTP id p1-20020a92d481000000b0035751796453mr2854205ilg.32.1697834836552; Fri, 20 Oct 2023 13:47:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697834836; cv=none; d=google.com; s=arc-20160816; b=C8TLFfds2c2Fz9t7Tcq9fofcQygDGIb/Cp0ctbydeAiibzlOhhqy28v1D5bolfxOTc nyEfJb9fMoCpDrSeOdjoMwcGmn+69xWnp7ByzW3zVHYMNdvU6UeYnVS+IyCug1SaMYX0 CWIXjGmiJQL/ytQzo6/XZfqA1gNG+YO3dNZ43Gb2KdY5nJDgifEe8u+NJbep8De7GTT/ malt584JG62lMrIOFiT9PFDvqNgPOog3TEDmmaki0Hg8wFwcAq7rPx9xmZF1yMxG+hQF rIT7WUIUhB9AG85m58YSNssUyHD0grHFLHXUgbWYQbJgKUS5XJ65MoGqssJ+NJrdQTGb 6dTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=/363Ck6ML1XM7ZYyVTDe7YNHEqM+JbJAFbfKlaD1Q1k=; fh=lbJsPO+ZRDN9baDq/BZnnNH6CGOpZtNT0YCi3hapTHQ=; b=rf3daz8+QKn/KKT0hbRZDqNDWLdxlSbWzyzxlgJxtxHmOQ7hC7/ci60dUIsyvDW5mV vQRSQh5O9XulXhuiWIftX1Qf7luJ8FB4mG6TFw9d1S3AOp5xYtMjOwymFcgFC3HWu+ka 3bJWaZuVyAYjlQJo9kBvrRt7U5MsWTLZxavOFFUgwCtH2aO4jMbAD4fQzh4sCMm2ZN1v rdupLDYexauIDfMRdbyM0WW5UcDKyMd79ByEcqFyVp0VomoXTXo3CyfKAU5dZ9ZaSOMy w4vpTvL0XFM271B01kt0VIzXk0IzgjlHrIFPtbx5GlAEJbipgUZrJtmAgtBNCrJK9LdW UzDg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=TgfSmSbN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id bv67-20020a632e46000000b005b065bb6822si2401518pgb.286.2023.10.20.13.47.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 13:47:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=TgfSmSbN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 321B08097A37; Fri, 20 Oct 2023 13:45:35 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232345AbjJTUpV (ORCPT + 26 others); Fri, 20 Oct 2023 16:45:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55942 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230408AbjJTUpP (ORCPT ); Fri, 20 Oct 2023 16:45:15 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D711D73; Fri, 20 Oct 2023 13:45:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697834711; x=1729370711; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=0qW8gkQ9cQn9nJIAFp1D4GChim4j3ri4y26baGhJi2U=; b=TgfSmSbN2UWoYOR68QBC3ieQZ05CW1/XqLHaJxayGBCoJn/5/ybDPij/ EYG4rXAafabNr1tzgTAlVhftdr7xGEwHdld8P7HZBI9rg0vdFPTs+XkP+ 4dYd+5V1U0XAO0ZsU0LzeGILbka7tIEj17kwG0izuM0XibxtlSQlVlMnP szjlDwHWh9XBMCE6VdHcEZUKJ17aRpCfHp8GdfuKuH23km6dyy6yXqVVP un76a1JOnAPFcfe+GXBCDZkEiLesavYqajCpNi1MwUE102s5vROJ7S3mx gzaYTnX1dY0T3aoJ0x0JAFSOInqsiPsj5FZG+RouyItpLFdxnucZgAzpz w==; X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="371640238" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="371640238" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:10 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="931117494" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="931117494" Received: from hkchanda-mobl.amr.corp.intel.com (HELO desk) ([10.209.90.113]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:09 -0700 Date: Fri, 20 Oct 2023 13:45:09 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta Subject: [PATCH 3/6] x86/entry_32: Add VERW just before userspace transition Message-ID: <20231020-delay-verw-v1-3-cff54096326d@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Fri, 20 Oct 2023 13:45:35 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780308861838952734 X-GMAIL-MSGID: 1780308861838952734 As done for entry_64, add support for executing VERW late in exit to user path for 32-bit mode. Signed-off-by: Pawan Gupta --- arch/x86/entry/entry_32.S | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S index 6e6af42e044a..bbf77d2aab2e 100644 --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -886,6 +886,9 @@ SYM_FUNC_START(entry_SYSENTER_32) popfl popl %eax + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS + /* * Return back to the vDSO, which will pop ecx and edx. * Don't bother with DS and ES (they already contain __USER_DS). @@ -954,6 +957,9 @@ restore_all_switch_stack: /* Restore user state */ RESTORE_REGS pop=4 # skip orig_eax/error_code + + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS .Lirq_return: /* * ARCH_HAS_MEMBARRIER_SYNC_CORE rely on IRET core serialization @@ -1146,6 +1152,8 @@ SYM_CODE_START(asm_exc_nmi) /* Not on SYSENTER stack. */ call exc_nmi + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS jmp .Lnmi_return .Lnmi_from_sysenter_stack: From patchwork Fri Oct 20 20:45:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 156308 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2010:b0:403:3b70:6f57 with SMTP id fe16csp1315744vqb; Fri, 20 Oct 2023 13:46:20 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGRV9jjC8TtI2cKTeu+TxHHnIo/CstPGxfPm3Ql3BKgv2q/ybr/Swd86+6W9y+weFQf2Yx4 X-Received: by 2002:a05:6358:2492:b0:145:6433:8224 with SMTP id m18-20020a056358249200b0014564338224mr3189931rwc.18.1697834779743; Fri, 20 Oct 2023 13:46:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697834779; cv=none; d=google.com; s=arc-20160816; b=cu38DNLYrXCxEJ1hlHjQ3kZ4JfzDI4CtgHb39F+428Dp1f5oO7nK+verP8+U8gawsr wnaTR8VQpT1L9dmJzMnK3UnpUSLM00FJUpbY62R2/A39BpHr/pyRzMRriqx1Akk/cWvH MhYCpLWKfT7pZzRJ+VgSvwsAqkQEn+RF3cH+LUETUkjBVe77qWwnl/tpjkteF2siMLKV lqa1IZgXOwz43jLNFXu7CbXNNIPHo/axA1pi+cH7lJ52OOqCZc0hHw4imw7Jxl6JiZKA YKzAIBdEctGTFqa6E6LTE17C3DtFJv6NeiUeTqjyKguTfD9mnumkVpkYyOAs1XKb6JXv yI6A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=7MyDsa+/xdrOJl3PMZkZTDY2pBpdmSGneeLJ6XWSfD4=; fh=lbJsPO+ZRDN9baDq/BZnnNH6CGOpZtNT0YCi3hapTHQ=; b=ZH5mBlWTqG4fqq49wYLWtk1EMZK8FbzofxmkjBYhhe1nb1Rbr8GRxfixwa+3bAc4BO YHqSTy/GJC7OMapJakHX+o26nAzWy3ocNwpZfAAyAcK2lPgb/7PkAIy5IpbjcOPH8PNn MrrOcBWjOIMhvODz7K8Fau444WCNAyRhy/8p+3wjZX53fSr8/8CdJkdjg8zAWKEJqB8f QLtf2kFXbnWelESxMNW2cNTdekmgMdwIK96aqoO3M33BEsfXVpAtVFzDxhBbBlgkV/VY DOqH/SU7pM3VIA4bzap5ZnnHkC17rSAeRYQewzfcYNb9mi6U+PB+f7SlmJR8a5svwt++ T3tw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=DhHJxzTA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id bz39-20020a056a02062700b005648b1d63aasi2785318pgb.553.2023.10.20.13.46.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 13:46:19 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=DhHJxzTA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id AEB8B806E5FD; Fri, 20 Oct 2023 13:46:15 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230522AbjJTUpg (ORCPT + 26 others); Fri, 20 Oct 2023 16:45:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39936 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231354AbjJTUp1 (ORCPT ); Fri, 20 Oct 2023 16:45:27 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5D92610F0; Fri, 20 Oct 2023 13:45:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697834718; x=1729370718; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=mbuqw2cUwhH3Vh8gF1I6goIWRvTTaGe6ZC14bjh7nw8=; b=DhHJxzTAGENVxsznas6zwauTC7DN77uoWtPYE5NUabvSohsCIctjepGm H7Nws6PDyrKxdhlxgy4Ym6ectHNtR+EFwAyaP9FC7ewp5Wxak/yCFSJET e3ZHJ2/qYC03Y3m73LvWNyWnMz/LZ4EZoO2mADKNzKsOfGxjbCsT+go02 C5EPek/QUdo5zOYc3tV2JPuWathnX6D+UthtHwq5brVfq68xYVOMTqBAz 5GiNzkfkQpuTs4Uw/lUmA57tKOm2agvw0fFQDhdUI2VqdLvKnbhj0GDZW m7FZ5Qzoutnbg8lYe2q3ZXcONx4C/2v1BMPjjUP0G0ugJq464ppDCsXSq g==; X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="365909393" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="365909393" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="848201701" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="848201701" Received: from hkchanda-mobl.amr.corp.intel.com (HELO desk) ([10.209.90.113]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:16 -0700 Date: Fri, 20 Oct 2023 13:45:15 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta Subject: [PATCH 4/6] x86/bugs: Use ALTERNATIVE() instead of mds_user_clear static key Message-ID: <20231020-delay-verw-v1-4-cff54096326d@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 20 Oct 2023 13:46:15 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780308801905727145 X-GMAIL-MSGID: 1780308801905727145 The VERW mitigation at exit-to-user is enabled via a static branch mds_user_clear. This static branch is never toggled after boot, and can be safely replaced with an ALTERNATIVE() which is convenient to use in asm. Switch to ALTERNATIVE() to use the VERW mitigation late in exit-to-user path. Also remove the now redundant VERW in exc_nmi() and arch_exit_to_user_mode(). Signed-off-by: Pawan Gupta --- Documentation/arch/x86/mds.rst | 20 +++++++++----------- arch/x86/include/asm/entry-common.h | 1 - arch/x86/include/asm/nospec-branch.h | 11 ----------- arch/x86/kernel/cpu/bugs.c | 10 +++++----- arch/x86/kernel/nmi.c | 2 -- arch/x86/kvm/vmx/vmx.c | 2 +- 6 files changed, 15 insertions(+), 31 deletions(-) diff --git a/Documentation/arch/x86/mds.rst b/Documentation/arch/x86/mds.rst index e73fdff62c0a..05090f33f65e 100644 --- a/Documentation/arch/x86/mds.rst +++ b/Documentation/arch/x86/mds.rst @@ -95,6 +95,9 @@ The kernel provides a function to invoke the buffer clearing: mds_clear_cpu_buffers() +Also macro USER_CLEAR_CPU_BUFFERS is meant to be used in ASM late in +exit-to-user path. This macro works for cases where GPRs can't be clobbered. + The mitigation is invoked on kernel/userspace, hypervisor/guest and C-state (idle) transitions. @@ -138,18 +141,13 @@ Mitigation points When transitioning from kernel to user space the CPU buffers are flushed on affected CPUs when the mitigation is not disabled on the kernel - command line. The migitation is enabled through the static key - mds_user_clear. - - The mitigation is invoked in prepare_exit_to_usermode() which covers - all but one of the kernel to user space transitions. The exception - is when we return from a Non Maskable Interrupt (NMI), which is - handled directly in do_nmi(). - - (The reason that NMI is special is that prepare_exit_to_usermode() can - enable IRQs. In NMI context, NMIs are blocked, and we don't want to - enable IRQs with NMIs blocked.) + command line. The migitation is enabled through the feature flag + X86_FEATURE_USER_CLEAR_CPU_BUF. + The mitigation is invoked just before transitioning to userspace after + user registers are restored. This is done to minimize the window in + which kernel data could be accessed after VERW e.g. via an NMI after + VERW. 2. C-State transition ^^^^^^^^^^^^^^^^^^^^^ diff --git a/arch/x86/include/asm/entry-common.h b/arch/x86/include/asm/entry-common.h index ce8f50192ae3..7e523bb3d2d3 100644 --- a/arch/x86/include/asm/entry-common.h +++ b/arch/x86/include/asm/entry-common.h @@ -91,7 +91,6 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, static __always_inline void arch_exit_to_user_mode(void) { - mds_user_clear_cpu_buffers(); amd_clear_divider(); } #define arch_exit_to_user_mode arch_exit_to_user_mode diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index e1b623a27e1b..501c26ecd722 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -595,17 +595,6 @@ static __always_inline void mds_clear_cpu_buffers(void) asm volatile("verw %[ds]" : : [ds] "m" (ds) : "cc"); } -/** - * mds_user_clear_cpu_buffers - Mitigation for MDS and TAA vulnerability - * - * Clear CPU buffers if the corresponding static key is enabled - */ -static __always_inline void mds_user_clear_cpu_buffers(void) -{ - if (static_branch_likely(&mds_user_clear)) - mds_clear_cpu_buffers(); -} - /** * mds_idle_clear_cpu_buffers - Mitigation for MDS vulnerability * diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index 10499bcd4e39..75fc0a70877f 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -252,7 +252,7 @@ static void __init mds_select_mitigation(void) if (!boot_cpu_has(X86_FEATURE_MD_CLEAR)) mds_mitigation = MDS_MITIGATION_VMWERV; - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_USER_CLEAR_CPU_BUF); if (!boot_cpu_has(X86_BUG_MSBDS_ONLY) && (mds_nosmt || cpu_mitigations_auto_nosmt())) @@ -356,7 +356,7 @@ static void __init taa_select_mitigation(void) * For guests that can't determine whether the correct microcode is * present on host, enable the mitigation for UCODE_NEEDED as well. */ - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_USER_CLEAR_CPU_BUF); if (taa_nosmt || cpu_mitigations_auto_nosmt()) cpu_smt_disable(false); @@ -424,7 +424,7 @@ static void __init mmio_select_mitigation(void) */ if (boot_cpu_has_bug(X86_BUG_MDS) || (boot_cpu_has_bug(X86_BUG_TAA) && boot_cpu_has(X86_FEATURE_RTM))) - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_USER_CLEAR_CPU_BUF); else static_branch_enable(&mmio_stale_data_clear); @@ -484,11 +484,11 @@ static void __init md_clear_update_mitigation(void) if (cpu_mitigations_off()) return; - if (!static_key_enabled(&mds_user_clear)) + if (!boot_cpu_has(X86_FEATURE_USER_CLEAR_CPU_BUF)) goto out; /* - * mds_user_clear is now enabled. Update MDS, TAA and MMIO Stale Data + * X86_FEATURE_USER_CLEAR_CPU_BUF is now enabled. Update MDS, TAA and MMIO Stale Data * mitigation, if necessary. */ if (mds_mitigation == MDS_MITIGATION_OFF && diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c index a0c551846b35..ebfff8dca661 100644 --- a/arch/x86/kernel/nmi.c +++ b/arch/x86/kernel/nmi.c @@ -551,8 +551,6 @@ DEFINE_IDTENTRY_RAW(exc_nmi) if (this_cpu_dec_return(nmi_state)) goto nmi_restart; - if (user_mode(regs)) - mds_user_clear_cpu_buffers(); if (IS_ENABLED(CONFIG_NMI_CHECK_CPU)) { WRITE_ONCE(nsp->idt_seq, nsp->idt_seq + 1); WARN_ON_ONCE(nsp->idt_seq & 0x1); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 72e3943f3693..c16297a49e4d 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7229,7 +7229,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, /* L1D Flush includes CPU buffer clear to mitigate MDS */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); - else if (static_branch_unlikely(&mds_user_clear)) + else if (cpu_feature_enabled(X86_FEATURE_USER_CLEAR_CPU_BUF)) mds_clear_cpu_buffers(); else if (static_branch_unlikely(&mmio_stale_data_clear) && kvm_arch_has_assigned_device(vcpu->kvm)) From patchwork Fri Oct 20 20:45:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 156307 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2010:b0:403:3b70:6f57 with SMTP id fe16csp1315713vqb; Fri, 20 Oct 2023 13:46:15 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGHt/GYYHXAATzrGHAhoXfzdhtL8Kbt2CUUzEAbG7XnETrpS9Ay8heGY99nb57tKm8lQVGW X-Received: by 2002:a17:903:247:b0:1c3:64f9:45ad with SMTP id j7-20020a170903024700b001c364f945admr3810220plh.48.1697834774897; Fri, 20 Oct 2023 13:46:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697834774; cv=none; d=google.com; s=arc-20160816; b=d1MxuVtxMwW06Y/6rX3GLWOu2dtF/E40KqiOg0nPg7W1N21Oym8MrNV9NfLAxAu7SH BVBc5bkxx/5JfrlTJIr9uD+poxFl/FwqippxkpM6ziVxBPKYUrZvXoDZk5Z/K5ALD8fJ m5fNlvvE4TIVWH/Y3VjKMbysifGU1NU2JPdo7RNvUFypIpUpCM3pLqa09jrJE7OSoV/E A3N3TK5HET6FvQdjP2oqgPW8JmXYEjAv+u8Yup35dER+3lfaeKlgm/QSmoPAFH1DvT58 mPVoyZUhWblLlx0HVuC+jeEL9oBeqfw0jb+ZBmv34psUi/bVSCzDrSiJu/MzxEprnpPz bfwA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=p/tOFzXLLWREH1+FIFM9uUiLRon5jnteFfuqnh9KevE=; fh=lbJsPO+ZRDN9baDq/BZnnNH6CGOpZtNT0YCi3hapTHQ=; b=FailHB6fww7Xdy4bg6ZGWZ4EQjQw+nZ3lTPkVIH8WQwrxyCNrtgTTQwWhSSr3uUemQ EN9rWj5xpmAtR5q3jT3XH1gv8dvl8nILoHO/gf5MtTHPck9xgsXNZm9RhOM3kGQ3ulIo jdUiDInuhWFqhJUSeAE20Zl3vFxvgaicvx3uyZvH78JaiMSMvqJcR+bnofvWZ9Ry4Ucy 5JhigD2tkqDyemZ3odssvdHBo0E89t8Xs3ZL2phMgulZavwZ8qVMCXaLFDQndeXt/5dd CYcymp+zfaaWLkpDrSkaqsM4HOwHUXWxRcUTU+KUno4yaEf99REEGjAjATUtoTFoXued jQ7g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=DYdLoAt5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id j17-20020a170902c3d100b001c45291b6ebsi2340354plj.272.2023.10.20.13.46.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 13:46:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=DYdLoAt5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 02F7B8266520; Fri, 20 Oct 2023 13:46:07 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232788AbjJTUpt (ORCPT + 26 others); Fri, 20 Oct 2023 16:45:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231792AbjJTUpf (ORCPT ); Fri, 20 Oct 2023 16:45:35 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CDFEF1703; Fri, 20 Oct 2023 13:45:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697834726; x=1729370726; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=MYGTJ5cvnXWg6uwlltOdXM4ZAEsPGA4t+1Vpxa3NQlw=; b=DYdLoAt55s+cLosXIgh1ArQH+Bz73e2aIAgqNFFYlcz7uMpCMGb2WDaJ Bqgn9zPbX9GZZAp+J3ge3DIa4Lgleem+ZA0BKZXs35KPBWmY1a/a7jKBH ODBEUqQKL2ddwyFbCPCHf/y2K3m/0vW4rJpVBDUBSERW6sGj62SfJULwZ of4BP6YD28YaFQcA+M/Ru5hRTn/4NcBFZDfzQlNJJ6mYVMFSFLJBbCHXC HoAnYNbz0gWrERcUcZ21wY6IR1FXQLVFWZ7+dUkx45cyM9AwKsatG6j8h fQR8GpL6PqQm4PVsDjqsMEPTqEtSFurcyuZ0nHzudJac9KtpAqWI5mLRD A==; X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="365909429" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="365909429" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="848201758" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="848201758" Received: from hkchanda-mobl.amr.corp.intel.com (HELO desk) ([10.209.90.113]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:23 -0700 Date: Fri, 20 Oct 2023 13:45:22 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta Subject: [PATCH 5/6] x86/bugs: Cleanup mds_user_clear Message-ID: <20231020-delay-verw-v1-5-cff54096326d@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 20 Oct 2023 13:46:07 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780308796547573123 X-GMAIL-MSGID: 1780308796547573123 There are no more users of mds_user_clear static key, remove it. Signed-off-by: Pawan Gupta --- arch/x86/include/asm/nospec-branch.h | 1 - arch/x86/kernel/cpu/bugs.c | 3 --- 2 files changed, 4 deletions(-) diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index 501c26ecd722..520689325014 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -563,7 +563,6 @@ DECLARE_STATIC_KEY_FALSE(switch_to_cond_stibp); DECLARE_STATIC_KEY_FALSE(switch_mm_cond_ibpb); DECLARE_STATIC_KEY_FALSE(switch_mm_always_ibpb); -DECLARE_STATIC_KEY_FALSE(mds_user_clear); DECLARE_STATIC_KEY_FALSE(mds_idle_clear); DECLARE_STATIC_KEY_FALSE(switch_mm_cond_l1d_flush); diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index 75fc0a70877f..cf2181b45055 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -111,9 +111,6 @@ DEFINE_STATIC_KEY_FALSE(switch_mm_cond_ibpb); /* Control unconditional IBPB in switch_mm() */ DEFINE_STATIC_KEY_FALSE(switch_mm_always_ibpb); -/* Control MDS CPU buffer clear before returning to user space */ -DEFINE_STATIC_KEY_FALSE(mds_user_clear); -EXPORT_SYMBOL_GPL(mds_user_clear); /* Control MDS CPU buffer clear before idling (halt, mwait) */ DEFINE_STATIC_KEY_FALSE(mds_idle_clear); EXPORT_SYMBOL_GPL(mds_idle_clear); From patchwork Fri Oct 20 20:45:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 156309 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2010:b0:403:3b70:6f57 with SMTP id fe16csp1315853vqb; Fri, 20 Oct 2023 13:46:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGeJ0M2npZnlZqRYXKedb4T64J5ZS5JMUc4VfZkocl2JhruHWOoVLUJG9bd2pfmCgrEnmPX X-Received: by 2002:a05:6358:52c8:b0:166:d637:a2db with SMTP id z8-20020a05635852c800b00166d637a2dbmr3786648rwz.9.1697834792074; Fri, 20 Oct 2023 13:46:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697834792; cv=none; d=google.com; s=arc-20160816; b=kfKXMcGURBS5s8OO/NG5oB/8Gz1Sy/YrqCQlCm90K93z3Ta/T3F9ZR/zPeZsTArj3s O/MBHM0n71kLR2zAdxTIfIJleX3Mv0E8SjZ8/NJOyDZFZo/yDkGfZiivE6jN9pJOLyqK NjK+EJl0limH77kzbNCre7x8K5T/f5Kc+SsANLW0pQGzBZL+B6pJzV30lX5D+8I2jmE9 KJzRTkDIOtVf0n7eA5fCEAkSqFoaKsFXNXuUK2+VWEHBOnUHdU/DZm7xSRZXok4Xj7Ct 2H+Iax68nm2tXeC+jhn37UhxhR9a2EGjb5Cwv9kECp9XL2E7hET9McymeX2J4LBEiz/Q woXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=0+7uwXB+8WqXbDyxiJ57dmAUbeDIaySL2qn3LxaZELU=; fh=lbJsPO+ZRDN9baDq/BZnnNH6CGOpZtNT0YCi3hapTHQ=; b=UmZverm4kSRT1qcxbziI3C6H0bsPqVKkBCf98xkRHg0jhdp9/PMg8RjVcFE//t3xIw Kofv7LSSrEoq5u1FtP8a6FmLkFsi1gvqKGRN3zWCd77cm5uO4M1QBO++uSohckmJdyDY yeuuCk12s59Kv+TlMcq+HkYUOVuJinRevYU5a5+fU4jRSGWnZG8VNYy+LtqIaZWwLKRM Kez+mW5ODRBurrLr0Qa65LqY/F548n4j7sN37sopwsrSEK2DVPy8fEmEnd1JSqfd2Tfo 3JoWfup3vw+PLIhMb9oxonJFqO6f0MZvhnYtpVVm26ijSdbduzG6BJJbUirJ8Ec7mo7a q/0w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="F/rxuNem"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id 199-20020a6301d0000000b00578af1e2f3csi2554272pgb.426.2023.10.20.13.46.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 20 Oct 2023 13:46:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="F/rxuNem"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 8821D81C179C; Fri, 20 Oct 2023 13:46:29 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231338AbjJTUp6 (ORCPT + 26 others); Fri, 20 Oct 2023 16:45:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40002 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231160AbjJTUpp (ORCPT ); Fri, 20 Oct 2023 16:45:45 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 264701717; Fri, 20 Oct 2023 13:45:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697834732; x=1729370732; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=efcm4OHP2Uz572Un+YAEz/A+CIucgALlZ4w/Rew7Rqg=; b=F/rxuNemKnaWdJ9lk7HdzRn0wZJOjnk0pEjMJE2qTWjQYYbfxpO9ni6z stLc0qUV/lcjFfZubfAZirqtcbolK1gCJ8aVvhfqHc+A5z1QzGruG6RFr B7HWY9JJSzKbDI36O7KjjwaFlg7zp5X6IFNnks6c24/xD9z34v6HrdH6R FK01NAw09dGSJp1dEZut39X7CGRJDRdMaDc4v4YzoOM9XqrNEOUSYqshw ZECUt0iyK9UUuLd2E0WsRiSziaKCYsdIqdZpOlUVuybAkmVETWrUetOR8 ZxnpYG0JqA57GH6bnBV5i4cvBDdMpQ0KKot8/tyjr3OgyP/uzKWVJSgVl w==; X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="365909446" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="365909446" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="848201802" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="848201802" Received: from hkchanda-mobl.amr.corp.intel.com (HELO desk) ([10.209.90.113]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:30 -0700 Date: Fri, 20 Oct 2023 13:45:29 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta Subject: [PATCH 6/6] KVM: VMX: Move VERW closer to VMentry for MDS mitigation Message-ID: <20231020-delay-verw-v1-6-cff54096326d@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 20 Oct 2023 13:46:29 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780308814888062778 X-GMAIL-MSGID: 1780308814888062778 During VMentry VERW is executed to mitigate MDS. After VERW, any memory access like register push onto stack may put host data in MDS affected CPU buffers. A guest can then use MDS to sample host data. Although likelihood of secrets surviving in registers at current VERW callsite is less, but it can't be ruled out. Harden the MDS mitigation by moving the VERW mitigation late in VMentry path. Note that VERW for MMIO Stale Data mitigation is unchanged because of the complexity of per-guest conditional VERW which is not easy to handle that late in asm with no GPRs available. If the CPU is also affected by MDS, VERW is unconditionally executed late in asm regardless of guest having MMIO access. Signed-off-by: Pawan Gupta Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmenter.S | 9 +++++++++ arch/x86/kvm/vmx/vmx.c | 10 +++++++--- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index be275a0410a8..efa716cf4727 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -1,6 +1,7 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include #include +#include #include #include #include @@ -31,6 +32,8 @@ #define VCPU_R15 __VCPU_REGS_R15 * WORD_SIZE #endif +#define GUEST_CLEAR_CPU_BUFFERS USER_CLEAR_CPU_BUFFERS + .macro VMX_DO_EVENT_IRQOFF call_insn call_target /* * Unconditionally create a stack frame, getting the correct RSP on the @@ -177,10 +180,16 @@ SYM_FUNC_START(__vmx_vcpu_run) * the 'vmx_vmexit' label below. */ .Lvmresume: + /* Mitigate CPU data sampling attacks .e.g. MDS */ + GUEST_CLEAR_CPU_BUFFERS + vmresume jmp .Lvmfail .Lvmlaunch: + /* Mitigate CPU data sampling attacks .e.g. MDS */ + GUEST_CLEAR_CPU_BUFFERS + vmlaunch jmp .Lvmfail diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index c16297a49e4d..e3d0eda292c3 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7226,13 +7226,17 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, guest_state_enter_irqoff(); - /* L1D Flush includes CPU buffer clear to mitigate MDS */ + /* + * L1D Flush includes CPU buffer clear to mitigate MDS, but VERW + * mitigation for MDS is done late in VMentry and is still executed + * inspite of L1D Flush. This is because an extra VERW should not matter + * much after the big hammer L1D Flush. + */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); - else if (cpu_feature_enabled(X86_FEATURE_USER_CLEAR_CPU_BUF)) - mds_clear_cpu_buffers(); else if (static_branch_unlikely(&mmio_stale_data_clear) && kvm_arch_has_assigned_device(vcpu->kvm)) + /* MMIO mitigation is mutually exclusive to MDS mitigation later in asm */ mds_clear_cpu_buffers(); vmx_disable_fb_clear(vmx);