From patchwork Wed Oct 25 20:52:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 158296 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp232032vqb; Wed, 25 Oct 2023 13:54:58 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEtJSHV1NXB5L14Uutb3t5SRi0xJV7Zty62c+T6abaWQ8Nhw+LjnAb4rpcdaLwHgLDOcBr4 X-Received: by 2002:a05:6830:1b79:b0:6bc:8afe:8a15 with SMTP id d25-20020a0568301b7900b006bc8afe8a15mr13980788ote.38.1698267298221; Wed, 25 Oct 2023 13:54:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698267298; cv=none; d=google.com; s=arc-20160816; b=sRu5O74vwxWBAQh/9cF5jWgIPGVbkVVRrU0yZHyJHnEteuMDrtXrSYuf90pu66jWxj y0CZ/3N8q/4nuHjDHvuPjTXIrjmjngqJOkourSJkGvx9VNVCEtTMwTZaqD1uoB8DoWy8 EkHlG5bqyObSTeGr1Ya0RTEMN2b4z04cYGpkvq9H+jzQ6RduVpxDQc5j2cTPxVgauO5W 9bjTUTdx/ods/WHKmCWNTm/TQW0ufdp91AFAhRsgF0gmSnMw7ZEqVsJZyRhV6Kc1RH9u Zdv6hBqC6RF3YefxcO1bTUfcjWhFdshE64Mw84kh6qsdH0fOGc4AbT3YpXRlyEJxH6gX qJJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=v66TrwN0sy6aehif6gWbz27zeRRHZQY9Z/gfXbjKhf0=; fh=qERIqab1IS0kFX4A0/Z/ByLJ99QIXEw3Uf6Pmmx8DsE=; b=VF3x+V37g8MF7KIAuiNR+zzcINRZrIbBOC9qi6+WMObGsgKRuzi5cpG3sjOpKJ/rQn Qy1kQk/4CqgeXtpw7Zcnndq/Ml+JfOH7Qi3obV1VAvCdEhSnGjtOCUD93u479hH2P2mI 5eLk+jDoFokvmcfGTvt+BeuIHGS/DnC6uCQMnorOm1silkVfs8x6rRiNVH6a5GT6hBeo FL1BCgxsG4tfe21mjwajqHYGT3iuZfsp28y6uHXWBbHg9G3K5y2rYtC72ooq8TGx/+oX s+cwSKLKpYqKPrlHyB93ai6/s+L6avxDS+aervhMEgMwrmZIdkHzp8h9P/cD9nsBXFjm GLmg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=RB6+4Btn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id q129-20020a0de787000000b0059eae2bc317si12507917ywe.217.2023.10.25.13.54.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 13:54:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=RB6+4Btn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id BB45A80620AE; Wed, 25 Oct 2023 13:53:33 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234796AbjJYUxG (ORCPT + 25 others); Wed, 25 Oct 2023 16:53:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232329AbjJYUxA (ORCPT ); Wed, 25 Oct 2023 16:53:00 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD02E184; Wed, 25 Oct 2023 13:52:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698267175; x=1729803175; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=bKrk4oUB1RyJhM26Xb3XGAzoBI+ZL+b1OrWpIw4mqdA=; b=RB6+4Btni8ROXTTL1LKark9z43ex0ID6SLYJovFW2r6FhpqAsAUBXK0d b5J5DMVM4K0j9//ZtFBZvjzJEaf4tUBDo2vmgyjt+cjatFbpdhunDP1kk mJTPBBR1Awa8Oz5QUF8xYsrVeZbh33BZ0Csekrwim99wgzw9mJcKmYasu zh7Se2GPUKGHLsauDAi6dlR0ni1vb6Mir89850XYahJiSJuhF4k98RPMw bDjQkY/57lkhIw7C82oxNBebmtfJdtSDl41v0tIvMoGKcMNmzT6xJwMjU MqqTlSvuiBR44eQ3d2X0dHQUP8/OPGg0BgTtn/1Wfv0wUvqdIyrnAFVSR A==; X-IronPort-AV: E=McAfee;i="6600,9927,10874"; a="390255480" X-IronPort-AV: E=Sophos;i="6.03,250,1694761200"; d="scan'208";a="390255480" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Oct 2023 13:52:55 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.03,250,1694761200"; d="scan'208";a="259602" Received: from kkomeyli-mobl.amr.corp.intel.com (HELO desk) ([10.251.29.139]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Oct 2023 13:52:43 -0700 Date: Wed, 25 Oct 2023 13:52:51 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta , Alyssa Milburn , Andrew Cooper Subject: [PATCH v3 1/6] x86/bugs: Add asm helpers for executing VERW Message-ID: <20231025-delay-verw-v3-1-52663677ee35@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231025-delay-verw-v3-0-52663677ee35@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231025-delay-verw-v3-0-52663677ee35@linux.intel.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 25 Oct 2023 13:53:34 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780762330390121979 X-GMAIL-MSGID: 1780762330390121979 MDS mitigation requires clearing the CPU buffers before returning to user. This needs to be done late in the exit-to-user path. Current location of VERW leaves a possibility of kernel data ending up in CPU buffers for memory accesses done after VERW such as: 1. Kernel data accessed by an NMI between VERW and return-to-user can remain in CPU buffers ( since NMI returning to kernel does not execute VERW to clear CPU buffers. 2. Alyssa reported that after VERW is executed, CONFIG_GCC_PLUGIN_STACKLEAK=y scrubs the stack used by a system call. Memory accesses during stack scrubbing can move kernel stack contents into CPU buffers. 3. When caller saved registers are restored after a return from function executing VERW, the kernel stack accesses can remain in CPU buffers(since they occur after VERW). To fix this VERW needs to be moved very late in exit-to-user path. In preparation for moving VERW to entry/exit asm code, create macros that can be used in asm. Also make them depend on a new feature flag X86_FEATURE_CLEAR_CPU_BUF. Reported-by: Alyssa Milburn Suggested-by: Andrew Cooper Suggested-by: Peter Zijlstra Signed-off-by: Pawan Gupta Signed-off-by: Pawan Gupta --- arch/x86/entry/entry.S | 16 ++++++++++++++++ arch/x86/include/asm/cpufeatures.h | 2 +- arch/x86/include/asm/nospec-branch.h | 15 +++++++++++++++ 3 files changed, 32 insertions(+), 1 deletion(-) diff --git a/arch/x86/entry/entry.S b/arch/x86/entry/entry.S index bfb7bcb362bc..f8ba0c0b6e60 100644 --- a/arch/x86/entry/entry.S +++ b/arch/x86/entry/entry.S @@ -6,6 +6,9 @@ #include #include #include +#include +#include +#include .pushsection .noinstr.text, "ax" @@ -20,3 +23,16 @@ SYM_FUNC_END(entry_ibpb) EXPORT_SYMBOL_GPL(entry_ibpb); .popsection + +.pushsection .entry.text, "ax" + +.align L1_CACHE_BYTES, 0xcc +SYM_CODE_START_NOALIGN(mds_verw_sel) + UNWIND_HINT_UNDEFINED + ANNOTATE_NOENDBR + .word __KERNEL_DS +SYM_CODE_END(mds_verw_sel); +/* For KVM */ +EXPORT_SYMBOL_GPL(mds_verw_sel); + +.popsection diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 58cb9495e40f..f21fc0f12737 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -308,10 +308,10 @@ #define X86_FEATURE_SMBA (11*32+21) /* "" Slow Memory Bandwidth Allocation */ #define X86_FEATURE_BMEC (11*32+22) /* "" Bandwidth Monitoring Event Configuration */ #define X86_FEATURE_USER_SHSTK (11*32+23) /* Shadow stack support for user mode applications */ - #define X86_FEATURE_SRSO (11*32+24) /* "" AMD BTB untrain RETs */ #define X86_FEATURE_SRSO_ALIAS (11*32+25) /* "" AMD BTB untrain RETs through aliasing */ #define X86_FEATURE_IBPB_ON_VMEXIT (11*32+26) /* "" Issue an IBPB only on VMEXIT */ +#define X86_FEATURE_CLEAR_CPU_BUF (11*32+27) /* "" Clear CPU buffers */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */ diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index c55cc243592e..005e69f93115 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -329,6 +329,21 @@ #endif .endm +/* + * Macros to execute VERW instruction that mitigate transient data sampling + * attacks such as MDS. On affected systems a microcode update overloaded VERW + * instruction to also clear the CPU buffers. VERW clobbers CFLAGS.ZF. + * + * Note: Only the memory operand variant of VERW clears the CPU buffers. + */ +.macro EXEC_VERW + verw _ASM_RIP(mds_verw_sel) +.endm + +.macro CLEAR_CPU_BUFFERS + ALTERNATIVE "", __stringify(EXEC_VERW), X86_FEATURE_CLEAR_CPU_BUF +.endm + #else /* __ASSEMBLY__ */ #define ANNOTATE_RETPOLINE_SAFE \ From patchwork Wed Oct 25 20:52:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 158297 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp232344vqb; Wed, 25 Oct 2023 13:55:42 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFh3ECjZ5LahF+BzmUzqoT8EjAWZd+XIckjCxHN5aXJc5uDgerecQabQMRvdwoThFWO+Dci X-Received: by 2002:a81:6984:0:b0:5ac:fe2d:5edf with SMTP id e126-20020a816984000000b005acfe2d5edfmr7575422ywc.44.1698267342398; Wed, 25 Oct 2023 13:55:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698267342; cv=none; d=google.com; s=arc-20160816; b=QTSx33jgI7qc3ucn9T9zr6GwyHic2lV7/caTalFtFLJ9BqjPItpVHc97Fiwisd9dRQ w7maGzvtIUHSfqUO3OnQoaa8Az3+A5J0P4kbtBVWV0rEbsp+U4YP5/YkeGD0Gp568eaZ l4EsvUBSl9wClrOeBFxOhUO7Fu71WEtZNo5PUlGMN03l48LY+1ZVDHZbiOB38NoAQeBt +tOJ9EAgfaciqmwOeVKFHjqLX7FT54IKx7tE8i+CwK8/FqUC0pXElxu3wKEKsXVigeJJ 6L58jwXUWqYytsBeowAiMn0zGTtoNa6ujIWTDKf/PdeUjytxpaHrnhvx7Xkub9UpyeSf 4flw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=JXW6ncWciaghnfyTdoACNbmv3XQBwz+zUjQClT1iM5A=; fh=pfRkWPzvKcXRDb8bzZOHP7Lfu+YA9O7iyInvCf011xY=; b=LQ3I7sdKsZ9WlcQTGCKrLR+67PhA8NACaKs9hl9YN5JJUw6PjUJRuht6uT33gcPNlZ Rlj3EmoF+xEqNVJSnavwDXZn/0KFgdmnM9R8qTNEKZSBZ32ujzppnNsPT9nE06kTWIay 3tf0NUw/YidRXkRUiRVHd7QgZS2fFa7DUmz3IvJAwsi/ZQJmEFsP3chI1xH/veZnMwKZ 3+eqLtJz42zzgqJ10Hy38ISdPzox1RuWUCpufmrQBDRQoBkOF+Vy8ajE60ZqSy3ippP3 W35cXfepdVKesysL081Da2MNmSSQ6xQygeRPdtU5T7WfGh2OHBfX2RcMwOfXakdfZKak S1NQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=O8hipnOI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id r8-20020a815d08000000b0059eb9ebb8d7si12757761ywb.383.2023.10.25.13.55.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 13:55:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=O8hipnOI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 0A231806AFE4; Wed, 25 Oct 2023 13:53:30 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234967AbjJYUxS (ORCPT + 25 others); Wed, 25 Oct 2023 16:53:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43854 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234816AbjJYUxH (ORCPT ); Wed, 25 Oct 2023 16:53:07 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 78DCB18B; Wed, 25 Oct 2023 13:53:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698267184; x=1729803184; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=KI9R1ykioS10T+zkw0Cw1ATpow7yvlFUWGh3i+Y0oPk=; b=O8hipnOIeYo9aSZqmLQaUd78Iv8mPL7gxnkozwNG5tpLu2jpSnt6AGk9 Gu/EazGF8lsjzQQKqIvnrHf6KXd66/h0rTEDa4QsEY8NCYVWUxEDK7Zgc ptUgtlmxDT7rQO5Dt4VKXfuBSzz6Iq3lHqgdcI3dFpZCK97qSMjkAy9sB yO/8DxTvatxcJLJ+nmvGEu6pbzDusJILFPcnpllLPmH3TJXXcFh/xzi+1 HPRgHVTL6iPd1wgsMz5oSh77ae8m1jmptnV3b6Tm/Ct7TI64kIm9HgTdc Y+j2r7FF26F0TPZ8wyr8U/P0uDGWXpqi3PCJGgwGp04Une6xdeo0JFC/v g==; X-IronPort-AV: E=McAfee;i="6600,9927,10874"; a="390255502" X-IronPort-AV: E=Sophos;i="6.03,250,1694761200"; d="scan'208";a="390255502" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Oct 2023 13:53:01 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.03,250,1694761200"; d="scan'208";a="259627" Received: from kkomeyli-mobl.amr.corp.intel.com (HELO desk) ([10.251.29.139]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Oct 2023 13:52:50 -0700 Date: Wed, 25 Oct 2023 13:52:58 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta , Dave Hansen Subject: [PATCH v3 2/6] x86/entry_64: Add VERW just before userspace transition Message-ID: <20231025-delay-verw-v3-2-52663677ee35@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231025-delay-verw-v3-0-52663677ee35@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231025-delay-verw-v3-0-52663677ee35@linux.intel.com> X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 25 Oct 2023 13:53:30 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780762377291195886 X-GMAIL-MSGID: 1780762377291195886 Mitigation for MDS is to use VERW instruction to clear any secrets in CPU Buffers. Any memory accesses after VERW execution can still remain in CPU buffers. It is safer to execute VERW late in return to user path to minimize the window in which kernel data can end up in CPU buffers. There are not many kernel secrets to be had after SWITCH_TO_USER_CR3. Add support for deploying VERW mitigation after user register state is restored. This helps minimize the chances of kernel data ending up into CPU buffers after executing VERW. Note that the mitigation at the new location is not yet enabled. Corner case not handled ======================= Interrupts returning to kernel don't clear CPUs buffers since the exit-to-user path is expected to do that anyways. But, there could be a case when an NMI is generated in kernel after the exit-to-user path has cleared the buffers. This case is not handled and NMI returning to kernel don't clear CPU buffers because: 1. It is rare to get an NMI after VERW, but before returning to userspace. 2. For an unprivileged user, there is no known way to make that NMI less rare or target it. 3. It would take a large number of these precisely-timed NMIs to mount an actual attack. There's presumably not enough bandwidth. 4. The NMI in question occurs after a VERW, i.e. when user state is restored and most interesting data is already scrubbed. Whats left is only the data that NMI touches, and that may or may not be of any interest. Suggested-by: Dave Hansen Signed-off-by: Pawan Gupta --- arch/x86/entry/entry_64.S | 11 +++++++++++ arch/x86/entry/entry_64_compat.S | 1 + 2 files changed, 12 insertions(+) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 43606de22511..9f97a8bd11e8 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -223,6 +223,7 @@ syscall_return_via_sysret: SYM_INNER_LABEL(entry_SYSRETQ_unsafe_stack, SYM_L_GLOBAL) ANNOTATE_NOENDBR swapgs + CLEAR_CPU_BUFFERS sysretq SYM_INNER_LABEL(entry_SYSRETQ_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR @@ -663,6 +664,7 @@ SYM_INNER_LABEL(swapgs_restore_regs_and_return_to_usermode, SYM_L_GLOBAL) /* Restore RDI. */ popq %rdi swapgs + CLEAR_CPU_BUFFERS jmp .Lnative_iret @@ -774,6 +776,8 @@ native_irq_return_ldt: */ popq %rax /* Restore user RAX */ + CLEAR_CPU_BUFFERS + /* * RSP now points to an ordinary IRET frame, except that the page * is read-only and RSP[31:16] are preloaded with the userspace @@ -1502,6 +1506,12 @@ nmi_restore: std movq $0, 5*8(%rsp) /* clear "NMI executing" */ + /* + * Skip CLEAR_CPU_BUFFERS here, since it only helps in rare cases like + * NMI in kernel after user state is restored. For an unprivileged user + * these conditions are hard to meet. + */ + /* * iretq reads the "iret" frame and exits the NMI stack in a * single instruction. We are returning to kernel mode, so this @@ -1520,6 +1530,7 @@ SYM_CODE_START(ignore_sysret) UNWIND_HINT_END_OF_STACK ENDBR mov $-ENOSYS, %eax + CLEAR_CPU_BUFFERS sysretl SYM_CODE_END(ignore_sysret) #endif diff --git a/arch/x86/entry/entry_64_compat.S b/arch/x86/entry/entry_64_compat.S index 70150298f8bd..245697eb8485 100644 --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -271,6 +271,7 @@ SYM_INNER_LABEL(entry_SYSRETL_compat_unsafe_stack, SYM_L_GLOBAL) xorl %r9d, %r9d xorl %r10d, %r10d swapgs + CLEAR_CPU_BUFFERS sysretl SYM_INNER_LABEL(entry_SYSRETL_compat_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR From patchwork Wed Oct 25 20:53:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 158294 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp231664vqb; Wed, 25 Oct 2023 13:53:57 -0700 (PDT) X-Google-Smtp-Source: AGHT+IECCxEgaqWqk64A/oiFvxR7d0ibT8eZ+EgV6SsMYadk/4HyEF/unt8gbx3Y+olZ5DpNrK0p X-Received: by 2002:a05:6358:9d85:b0:168:eaa2:a6c4 with SMTP id d5-20020a0563589d8500b00168eaa2a6c4mr8069356rwo.8.1698267236758; Wed, 25 Oct 2023 13:53:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698267236; cv=none; d=google.com; s=arc-20160816; b=TVz4b1MOGmh5yZe2dTN/eGq/dVryhkq8DTdo+Rsz9tfYH22QgGQyh1f/+6cRU12DfV IQMviAEgpjb+CzzIc2FWGfHKB//hY4B6l3UAvwZDOO0fhi5YUeU7MHE++f1JvuKj3gzZ AD4kfAIyO3UfQdeReU9EaNZlMDkCHlb8rhXeXOg0pDtteUOrdIzchlXaIjM/gY6YbIGq RJ5VFePqxbvmYPVZA7XoAcLpA3NiFD8bMeWNFbXwwgmPuNRiLuDMtLp+pmcGcP4LIvBx dI7YUqfNpxy/d27lTId3L/tiEQR+YuocEvYD830/Y6WICGU1Z336TuF1FdMfbc/8FmSC llNQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=UtZ6k9Oyj9C76UgVXziiWcsI59b09aZmqIi0GzVC7eo=; fh=lbJsPO+ZRDN9baDq/BZnnNH6CGOpZtNT0YCi3hapTHQ=; b=wLYiA3Mo6iOjsTIOZkrD/cX7DsidxRyzelpFan3lxTikYJ1BhDMMRLw63f43449b0l gT/ZJKF27NtfCn72ooJVc23gT9rwTiXwWPdykNStu7MT/qDdgFywuDEGQ8HccB/Pt34o E+wj6/FNA3HbiHBVoXnKdcuoMtTBo9PVh08b5p50ll7EHPmwNOOWx1qYgfepatuVhDW3 WCd/5Lim9cfz021AmJNWSDTSABd8KIyJ8O7tXAYfLNbgUF5Wci3tetOAHKgnfk60M2je ssMb2sOOncv39iyx7p5907tmnBezrH6mnE8CO1fx9B/G26i9u7vn5pJy3SRbIs6XypCB 3DTA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=P2Tp2FfU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id s5-20020a818205000000b0059575fba61csi13115553ywf.446.2023.10.25.13.53.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 13:53:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=P2Tp2FfU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 2F83581DF0D7; Wed, 25 Oct 2023 13:53:54 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232789AbjJYUxX (ORCPT + 25 others); Wed, 25 Oct 2023 16:53:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233221AbjJYUxQ (ORCPT ); Wed, 25 Oct 2023 16:53:16 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A67DD50; Wed, 25 Oct 2023 13:53:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698267189; x=1729803189; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=fKL8O9LfX0J38E00Mtc5QFOreN8mewIuIipoFxbNi84=; b=P2Tp2FfUFAD2G856ryNsSeuasFVWSN9mlO29Mzm6TXU9ZG3jZddBNnL0 iGGH07YiN/qXtZIEfQiXxXGdYDAI8afP5+0t2up+wxtTFuJms+JVY/BpK s8viFGeHdMczxgNHn+K5jMu5BMEQwdL5WFDh7GhQBhBh6f1JsQjlvByIH TBPaHalfMd7wxIg7aAhXUlFNBE2IdWGGsyzqEXEcWjHAjqyBSgyIRCtF/ 5pGWzEbw4XDbG7rfWNNPecsk247lBQWv2ZW4nIRxEJTqqee9q23ndY04H np7VvRRSMU0erv7RU3KoBxnsHSPRtFsMJwGeMbttwwU66DHVBwd/lQ0s4 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10874"; a="390255526" X-IronPort-AV: E=Sophos;i="6.03,250,1694761200"; d="scan'208";a="390255526" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Oct 2023 13:53:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.03,250,1694761200"; d="scan'208";a="259651" Received: from kkomeyli-mobl.amr.corp.intel.com (HELO desk) ([10.251.29.139]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Oct 2023 13:52:57 -0700 Date: Wed, 25 Oct 2023 13:53:05 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta Subject: [PATCH v3 3/6] x86/entry_32: Add VERW just before userspace transition Message-ID: <20231025-delay-verw-v3-3-52663677ee35@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231025-delay-verw-v3-0-52663677ee35@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231025-delay-verw-v3-0-52663677ee35@linux.intel.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Wed, 25 Oct 2023 13:53:54 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780762265947727392 X-GMAIL-MSGID: 1780762265947727392 As done for entry_64, add support for executing VERW late in exit to user path for 32-bit mode. Signed-off-by: Pawan Gupta --- arch/x86/entry/entry_32.S | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S index 6e6af42e044a..74a4358c7f45 100644 --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -885,6 +885,7 @@ SYM_FUNC_START(entry_SYSENTER_32) BUG_IF_WRONG_CR3 no_user_check=1 popfl popl %eax + CLEAR_CPU_BUFFERS /* * Return back to the vDSO, which will pop ecx and edx. @@ -954,6 +955,7 @@ restore_all_switch_stack: /* Restore user state */ RESTORE_REGS pop=4 # skip orig_eax/error_code + CLEAR_CPU_BUFFERS .Lirq_return: /* * ARCH_HAS_MEMBARRIER_SYNC_CORE rely on IRET core serialization @@ -1146,6 +1148,7 @@ SYM_CODE_START(asm_exc_nmi) /* Not on SYSENTER stack. */ call exc_nmi + CLEAR_CPU_BUFFERS jmp .Lnmi_return .Lnmi_from_sysenter_stack: From patchwork Wed Oct 25 20:53:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 158295 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp231826vqb; Wed, 25 Oct 2023 13:54:26 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEn7Bq+IzbZiWjgyeWndiIwQDA7mw1lcAWZuFGlYNJTho6lI7VhAKYEEvmu/0U1zWeEtPSU X-Received: by 2002:a25:d7c7:0:b0:d9a:4839:68fe with SMTP id o190-20020a25d7c7000000b00d9a483968femr14944607ybg.43.1698267265992; Wed, 25 Oct 2023 13:54:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698267265; cv=none; d=google.com; s=arc-20160816; b=F841uStJtGiD1TdQgo5lJhFLG0uJxO+FdpDDOqKBf2m/rQ35jVNsxodXh2ihqb8Jx2 f+2X/cFT9Wpxgrx8X1Y8reRv0w8lGU7FtnjXBfO4zdkLL1begi77vU+LQGC3+NDjz0QW S4un9s7tYlM7NzF5159NjU3oQrgcg9eXJjhnMS0W2T9NUpniR+Q1YzFxVsk8zRLOSRpL dN7KLOncJ9XV7L4h9Iqd2sEcyp76oQyGQOUepF5fBUX3ljXMQS3d1nsrAbrArE9JxTB4 1Oy5uWpddRtoe48qmbq+7T3VzhvMTsrWrC0aO090N6ki9SG5UY5rw0si3Rs/rlnVlFKq OfHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=Ge1mFZSSZx6Zx0C+6un4ZIu+Adp8APpkNLRsMYIkDkA=; fh=lbJsPO+ZRDN9baDq/BZnnNH6CGOpZtNT0YCi3hapTHQ=; b=vlpPF9/sauoM7tOIqJQRptElt4kTzzRb/9P5wgCSQQPxQ00rTrNjiITa18DT0W+E0e /akPxZ0kZmOy+LBSn9dkwyh6ervZDm52taTeYlhAYWzrDcScBY+9+u0efDxAFCx7rLSo nYSCifayU03ui8PIXzzciq35RJdpw56iw8bMj8W0upCQo/sRnQkBWZMz0rlmLBp29xDW N5meIWqW8W37FLRUVhx0lylWgLeV0x1mNv8duHD4QOlDZODfTwNCcq6G4I++w4fHZvfS HqeeW0KbOENUTcZjYrU58dpFm5lj4oRNENce5UZzmJD6Lu1qYUxpQz/JL4XuCiCqlmGz hFYA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="HP/Jicyd"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id 187-20020a2515c4000000b00d7ba10667b0si4221483ybv.536.2023.10.25.13.54.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 13:54:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="HP/Jicyd"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id DFE9B81DF0DF; Wed, 25 Oct 2023 13:54:21 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235071AbjJYUxm (ORCPT + 25 others); Wed, 25 Oct 2023 16:53:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43942 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229814AbjJYUx0 (ORCPT ); Wed, 25 Oct 2023 16:53:26 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5B5F31AD; Wed, 25 Oct 2023 13:53:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698267196; x=1729803196; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=wkq/Le2P/oy/c1LxizAjXnanmzK5N4i8jkBb77ryAII=; b=HP/JicydR3wupsu7JuM1n+dy0gfQnUZ+5YYNPgmCwGqjfFXyB3hF0wsk 3/Rc+bM9/rn6zqaam9S6XpxDiVp/3H49ihuItRnEFtskJeM3uEPoXPW2H VvPcU5i7XEO5vDxbbkfDvbpD4PnUdxwtVjVaNlnNAfIyXkqduZKkUkmzc 0utF7aOoz8jUhIuB2alqCb4hTQei+JrBDRtQbXGc9tqBzKBWOBl9b4Ofc r0ygYjUGc1p4BNPFVW/ZVXyep1SEnTkNvULLNrECV8FNzOIUN7ngt1jAg /KweT6+zPfyO2rvHlA+YvtpgvyP9dk4p/8JyKRcewI99aWquAcnLI6FyF Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10874"; a="367610029" X-IronPort-AV: E=Sophos;i="6.03,250,1694761200"; d="scan'208";a="367610029" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Oct 2023 13:53:15 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10874"; a="902683225" X-IronPort-AV: E=Sophos;i="6.03,250,1694761200"; d="scan'208";a="902683225" Received: from kkomeyli-mobl.amr.corp.intel.com (HELO desk) ([10.251.29.139]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Oct 2023 13:50:50 -0700 Date: Wed, 25 Oct 2023 13:53:11 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta Subject: [PATCH v3 4/6] x86/bugs: Use ALTERNATIVE() instead of mds_user_clear static key Message-ID: <20231025-delay-verw-v3-4-52663677ee35@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231025-delay-verw-v3-0-52663677ee35@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231025-delay-verw-v3-0-52663677ee35@linux.intel.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Wed, 25 Oct 2023 13:54:21 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780762296467549398 X-GMAIL-MSGID: 1780762296467549398 The VERW mitigation at exit-to-user is enabled via a static branch mds_user_clear. This static branch is never toggled after boot, and can be safely replaced with an ALTERNATIVE() which is convenient to use in asm. Switch to ALTERNATIVE() to use the VERW mitigation late in exit-to-user path. Also remove the now redundant VERW in exc_nmi() and arch_exit_to_user_mode(). Signed-off-by: Pawan Gupta --- Documentation/arch/x86/mds.rst | 39 ++++++++++++++++++++++++++---------- arch/x86/include/asm/entry-common.h | 1 - arch/x86/include/asm/nospec-branch.h | 12 ----------- arch/x86/kernel/cpu/bugs.c | 15 ++++++-------- arch/x86/kernel/nmi.c | 2 -- arch/x86/kvm/vmx/vmx.c | 2 +- 6 files changed, 35 insertions(+), 36 deletions(-) diff --git a/Documentation/arch/x86/mds.rst b/Documentation/arch/x86/mds.rst index e73fdff62c0a..34b9e476078c 100644 --- a/Documentation/arch/x86/mds.rst +++ b/Documentation/arch/x86/mds.rst @@ -95,6 +95,9 @@ The kernel provides a function to invoke the buffer clearing: mds_clear_cpu_buffers() +Also macro CLEAR_CPU_BUFFERS is meant to be used in ASM late in exit-to-user +path. This macro works for cases where GPRs can't be clobbered. + The mitigation is invoked on kernel/userspace, hypervisor/guest and C-state (idle) transitions. @@ -138,17 +141,31 @@ Mitigation points When transitioning from kernel to user space the CPU buffers are flushed on affected CPUs when the mitigation is not disabled on the kernel - command line. The migitation is enabled through the static key - mds_user_clear. - - The mitigation is invoked in prepare_exit_to_usermode() which covers - all but one of the kernel to user space transitions. The exception - is when we return from a Non Maskable Interrupt (NMI), which is - handled directly in do_nmi(). - - (The reason that NMI is special is that prepare_exit_to_usermode() can - enable IRQs. In NMI context, NMIs are blocked, and we don't want to - enable IRQs with NMIs blocked.) + command line. The mitigation is enabled through the feature flag + X86_FEATURE_CLEAR_CPU_BUF. + + The mitigation is invoked just before transitioning to userspace after + user registers are restored. This is done to minimize the window in + which kernel data could be accessed after VERW e.g. via an NMI after + VERW. + + Corner case not handled + ^^^^^^^^^^^^^^^^^^^^^^^ + Interrupts returning to kernel don't clear CPUs buffers since the + exit-to-user path is expected to do that anyways. But, there could be + a case when an NMI is generated in kernel after the exit-to-user path + has cleared the buffers. This case is not handled and NMI returning to + kernel don't clear CPU buffers because: + + 1. It is rare to get an NMI after VERW, but before returning to userspace. + 2. For an unprivileged user, there is no known way to make that NMI + less rare or target it. + 3. It would take a large number of these precisely-timed NMIs to mount + an actual attack. There's presumably not enough bandwidth. + 4. The NMI in question occurs after a VERW, i.e. when user state is + restored and most interesting data is already scrubbed. Whats left + is only the data that NMI touches, and that may or may not be of + any interest. 2. C-State transition diff --git a/arch/x86/include/asm/entry-common.h b/arch/x86/include/asm/entry-common.h index ce8f50192ae3..7e523bb3d2d3 100644 --- a/arch/x86/include/asm/entry-common.h +++ b/arch/x86/include/asm/entry-common.h @@ -91,7 +91,6 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, static __always_inline void arch_exit_to_user_mode(void) { - mds_user_clear_cpu_buffers(); amd_clear_divider(); } #define arch_exit_to_user_mode arch_exit_to_user_mode diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index 005e69f93115..12b8e86678bf 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -553,7 +553,6 @@ DECLARE_STATIC_KEY_FALSE(switch_to_cond_stibp); DECLARE_STATIC_KEY_FALSE(switch_mm_cond_ibpb); DECLARE_STATIC_KEY_FALSE(switch_mm_always_ibpb); -DECLARE_STATIC_KEY_FALSE(mds_user_clear); DECLARE_STATIC_KEY_FALSE(mds_idle_clear); DECLARE_STATIC_KEY_FALSE(switch_mm_cond_l1d_flush); @@ -585,17 +584,6 @@ static __always_inline void mds_clear_cpu_buffers(void) asm volatile("verw %[ds]" : : [ds] "m" (ds) : "cc"); } -/** - * mds_user_clear_cpu_buffers - Mitigation for MDS and TAA vulnerability - * - * Clear CPU buffers if the corresponding static key is enabled - */ -static __always_inline void mds_user_clear_cpu_buffers(void) -{ - if (static_branch_likely(&mds_user_clear)) - mds_clear_cpu_buffers(); -} - /** * mds_idle_clear_cpu_buffers - Mitigation for MDS vulnerability * diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index 10499bcd4e39..00aab0c0937f 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -111,9 +111,6 @@ DEFINE_STATIC_KEY_FALSE(switch_mm_cond_ibpb); /* Control unconditional IBPB in switch_mm() */ DEFINE_STATIC_KEY_FALSE(switch_mm_always_ibpb); -/* Control MDS CPU buffer clear before returning to user space */ -DEFINE_STATIC_KEY_FALSE(mds_user_clear); -EXPORT_SYMBOL_GPL(mds_user_clear); /* Control MDS CPU buffer clear before idling (halt, mwait) */ DEFINE_STATIC_KEY_FALSE(mds_idle_clear); EXPORT_SYMBOL_GPL(mds_idle_clear); @@ -252,7 +249,7 @@ static void __init mds_select_mitigation(void) if (!boot_cpu_has(X86_FEATURE_MD_CLEAR)) mds_mitigation = MDS_MITIGATION_VMWERV; - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); if (!boot_cpu_has(X86_BUG_MSBDS_ONLY) && (mds_nosmt || cpu_mitigations_auto_nosmt())) @@ -356,7 +353,7 @@ static void __init taa_select_mitigation(void) * For guests that can't determine whether the correct microcode is * present on host, enable the mitigation for UCODE_NEEDED as well. */ - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); if (taa_nosmt || cpu_mitigations_auto_nosmt()) cpu_smt_disable(false); @@ -424,7 +421,7 @@ static void __init mmio_select_mitigation(void) */ if (boot_cpu_has_bug(X86_BUG_MDS) || (boot_cpu_has_bug(X86_BUG_TAA) && boot_cpu_has(X86_FEATURE_RTM))) - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); else static_branch_enable(&mmio_stale_data_clear); @@ -484,12 +481,12 @@ static void __init md_clear_update_mitigation(void) if (cpu_mitigations_off()) return; - if (!static_key_enabled(&mds_user_clear)) + if (!boot_cpu_has(X86_FEATURE_CLEAR_CPU_BUF)) goto out; /* - * mds_user_clear is now enabled. Update MDS, TAA and MMIO Stale Data - * mitigation, if necessary. + * X86_FEATURE_CLEAR_CPU_BUF is now enabled. Update MDS, TAA and MMIO + * Stale Data mitigation, if necessary. */ if (mds_mitigation == MDS_MITIGATION_OFF && boot_cpu_has_bug(X86_BUG_MDS)) { diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c index a0c551846b35..ebfff8dca661 100644 --- a/arch/x86/kernel/nmi.c +++ b/arch/x86/kernel/nmi.c @@ -551,8 +551,6 @@ DEFINE_IDTENTRY_RAW(exc_nmi) if (this_cpu_dec_return(nmi_state)) goto nmi_restart; - if (user_mode(regs)) - mds_user_clear_cpu_buffers(); if (IS_ENABLED(CONFIG_NMI_CHECK_CPU)) { WRITE_ONCE(nsp->idt_seq, nsp->idt_seq + 1); WARN_ON_ONCE(nsp->idt_seq & 0x1); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 72e3943f3693..24e8694b83fc 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7229,7 +7229,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, /* L1D Flush includes CPU buffer clear to mitigate MDS */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); - else if (static_branch_unlikely(&mds_user_clear)) + else if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF)) mds_clear_cpu_buffers(); else if (static_branch_unlikely(&mmio_stale_data_clear) && kvm_arch_has_assigned_device(vcpu->kvm)) From patchwork Wed Oct 25 20:53:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 158299 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp232504vqb; Wed, 25 Oct 2023 13:56:03 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGtKOcZcX9jXWaY3JiEBdBf9FMdRbm3Ur7O+CH2ZlzrBOsdsugQ/MFKEFjCPkTf3l9raBJv X-Received: by 2002:a0d:e743:0:b0:5a7:e445:fad9 with SMTP id q64-20020a0de743000000b005a7e445fad9mr14719385ywe.35.1698267363504; Wed, 25 Oct 2023 13:56:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698267363; cv=none; d=google.com; s=arc-20160816; b=ctZn5TZlTPkWkaJKd6G39CbQPX5WEtQ69KuGAjpA6rI0J3nfP2eNlaNO8VRAVo+AYU hIWX4ELQIbDOOKcpQeyGXEmr6SbdWxJ/Otn6ax1ZCtBedB+HKD/g9vVzPu8wSn4H0iJT PltHS4QjESzE7FzA4uhf5urFrst48v9p7jNCc06ZEsC7s4J95qR/K2JLHAFHIZcRB9gH QypNW5LjiwwswPcQ8pOJVGoxcgu7ECK4+Axj5IRzuUN0KipAR6ZcT1OG3/s81YmU5gtU X/5JmvcsFeTlHQw0oNZ9kBQHNKWyE/ic9apTkCKr9fx3ci47cfP8+ABxk/tzHQYcFx1h sZTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=ilqGXAQjD0Zw0p+0QS0PhsqpTUKzsxY97u90wwaRZ8Y=; fh=uQdKXzBD0hDCTt1Q3txvwrtjZoD8HE2wNgMpnRCfUnM=; b=OqoPprGVle0z/rdobjLqUQJxDxv3OTjqEQbXB9Rb3vbJMuINP9r/sF3kW35IulCRkb 3aa3scQ5JGKuPa24SzwB/hoceH7xZXvrfWoY3Ew1aa+8X2Z1UVeEYFXy9JNly5Jtnpuz FL9u6WywGAkT//bdW7HKKrFTkcPZBEST2hB29sFZaFadeP6qLmcPYuHNl6zgvTmPWzO8 4S82kvi7vzgHHZ9E0v27zzUTlVpWyeYlJpTggC6XRfJdAmgra47M+IpZEvnPjhn0KSFo 0ic19KFrbaUMFbLpqMsOy+3+wLxIC3t4hiaIbgr0H3qwLcrutAHwYDCkmkvKg5rl2WWU Uqmw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=SnBjG1ji; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id z63-20020a0dd742000000b005a7b91e234dsi13010018ywd.54.2023.10.25.13.56.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 13:56:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=SnBjG1ji; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id DB64E80B5A2F; Wed, 25 Oct 2023 13:54:40 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232076AbjJYUyE (ORCPT + 25 others); Wed, 25 Oct 2023 16:54:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32868 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235019AbjJYUxl (ORCPT ); Wed, 25 Oct 2023 16:53:41 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EAB6618D; Wed, 25 Oct 2023 13:53:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698267212; x=1729803212; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=rpFgire7Svqru/HfpQ1b16RkVx0842HOL7xbgXY2vaQ=; b=SnBjG1jifPpdEo/40nUwikKXsnXNh6s3yRp+o9DkmYK5mtrG1ef1ZipH CM015dD3HDo2BbVhd1lQMJTdIN/IwSGFo8U1SwpMJF2GN+z/TUf9HRK1l IHW4bWClvhj4Y7GzYHMlYhpAjhOKVE9axCXZZG6spJ5r4+dzO6I4wkfjl PwvWudKA1b4tabZHcslIKJlmkhqtR0R+6jzPEUratKaJMD2ZKdkd92ShG JNzttVr2Pci5SBj/g4oajf5bp7oYH5uG/h76u08s+K8osKkYCr/lvJr4Y 2At5X5m2dffXuh2uv+HS9v5Sdso8jN3U6pKuui4FOnLQDiM5JVS3OoMyn A==; X-IronPort-AV: E=McAfee;i="6600,9927,10874"; a="386277340" X-IronPort-AV: E=Sophos;i="6.03,250,1694761200"; d="scan'208";a="386277340" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Oct 2023 13:53:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.03,250,1694761200"; d="scan'208";a="217513" Received: from kkomeyli-mobl.amr.corp.intel.com (HELO desk) ([10.251.29.139]) by orviesa002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Oct 2023 13:52:46 -0700 Date: Wed, 25 Oct 2023 13:53:18 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta , Nikolay Borisov Subject: [PATCH v3 5/6] KVM: VMX: Use BT+JNC, i.e. EFLAGS.CF to select VMRESUME vs. VMLAUNCH Message-ID: <20231025-delay-verw-v3-5-52663677ee35@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231025-delay-verw-v3-0-52663677ee35@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231025-delay-verw-v3-0-52663677ee35@linux.intel.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 25 Oct 2023 13:54:40 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780762399092606421 X-GMAIL-MSGID: 1780762399092606421 From: Sean Christopherson Use EFLAGS.CF instead of EFLAGS.ZF to track whether to use VMRESUME versus VMLAUNCH. Freeing up EFLAGS.ZF will allow doing VERW, which clobbers ZF, for MDS mitigations as late as possible without needing to duplicate VERW for both paths. Reviewed-by: Nikolay Borisov Signed-off-by: Sean Christopherson Signed-off-by: Pawan Gupta --- arch/x86/kvm/vmx/run_flags.h | 7 +++++-- arch/x86/kvm/vmx/vmenter.S | 6 +++--- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/vmx/run_flags.h b/arch/x86/kvm/vmx/run_flags.h index edc3f16cc189..6a9bfdfbb6e5 100644 --- a/arch/x86/kvm/vmx/run_flags.h +++ b/arch/x86/kvm/vmx/run_flags.h @@ -2,7 +2,10 @@ #ifndef __KVM_X86_VMX_RUN_FLAGS_H #define __KVM_X86_VMX_RUN_FLAGS_H -#define VMX_RUN_VMRESUME (1 << 0) -#define VMX_RUN_SAVE_SPEC_CTRL (1 << 1) +#define VMX_RUN_VMRESUME_SHIFT 0 +#define VMX_RUN_SAVE_SPEC_CTRL_SHIFT 1 + +#define VMX_RUN_VMRESUME BIT(VMX_RUN_VMRESUME_SHIFT) +#define VMX_RUN_SAVE_SPEC_CTRL BIT(VMX_RUN_SAVE_SPEC_CTRL_SHIFT) #endif /* __KVM_X86_VMX_RUN_FLAGS_H */ diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index be275a0410a8..b3b13ec04bac 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -139,7 +139,7 @@ SYM_FUNC_START(__vmx_vcpu_run) mov (%_ASM_SP), %_ASM_AX /* Check if vmlaunch or vmresume is needed */ - test $VMX_RUN_VMRESUME, %ebx + bt $VMX_RUN_VMRESUME_SHIFT, %ebx /* Load guest registers. Don't clobber flags. */ mov VCPU_RCX(%_ASM_AX), %_ASM_CX @@ -161,8 +161,8 @@ SYM_FUNC_START(__vmx_vcpu_run) /* Load guest RAX. This kills the @regs pointer! */ mov VCPU_RAX(%_ASM_AX), %_ASM_AX - /* Check EFLAGS.ZF from 'test VMX_RUN_VMRESUME' above */ - jz .Lvmlaunch + /* Check EFLAGS.CF from the VMX_RUN_VMRESUME bit test above. */ + jnc .Lvmlaunch /* * After a successful VMRESUME/VMLAUNCH, control flow "magically" From patchwork Wed Oct 25 20:53:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 158298 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp232495vqb; Wed, 25 Oct 2023 13:56:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHZ2nBczk6LMVW88nLX1/5Il7Js74aS4V3MLoJeRnoewh6+qFNFgc/5REaM59U1S9VnH+fg X-Received: by 2002:a05:6102:1087:b0:45a:b9d3:ef73 with SMTP id s7-20020a056102108700b0045ab9d3ef73mr5477936vsr.29.1698267361961; Wed, 25 Oct 2023 13:56:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698267361; cv=none; d=google.com; s=arc-20160816; b=TaearIsnjBE90bD/ENgVZ/9ETwLw3DTDyTaWu1bEAzvsxGBJGqmDnNrSTaO/bWtfQh i5wS6yuMyPEfCU8sQzWniJ9klBkr7QtU8bXu6SUSqeJHsXsmC9Lan46MFVfQD+Og/3/N hM1Ml70HyRL7R1oTGKexDB5eFcwWZpvxVc8lCVt3IKsFQZXveat/HlE6MZynBgg3OyZh UUfjTF+2es6gilajynUzWz+KLNBgHHK63t6hosDy71ojcC9/jOTK+DSCTYC1jjL+Z3lj fyu0qrU3g+2NpG+0kclsGCxJe+jfDZs+i6cnYnuZoHLS8gC/Jk1tAGa4SOYhhzCXn6Jt XZ0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=fgO/aRS1b0dehQC4Ja3hlWXAeBfKY7QdZ9bqVRKNjw0=; fh=lbJsPO+ZRDN9baDq/BZnnNH6CGOpZtNT0YCi3hapTHQ=; b=uW0dpsoF8eP1YtMZ/I+f2sYuEOialuKwZztU8mTMj9xeG2rjkwXMJDMIeCGJok0I0r 48/8p22KzelBrX1xpfIGXG/kmgrst/yFS7aN5CpAjeX7tZWq9hVjLJJWQQdufBfmx3Pw StCH51YDeAgNZS1l6FSTzo/HFRr60ZdKGPnMNQ2vO+5GFSW5+bHmc5qKBujMqhrTI1j/ zz2eTaNDsclNSLZ7l00KNEpA8zlDly8/hOy9WBNOkVYZcEQNrGq8HvmUbwHhM4I0TMcx hYgW/GErumqoIlONzGIWQoV7lyD/0N/pEt43zy8R0x1INXQW1JCMCCCidy1cR0fsn9xA FT0Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=B7z6gfUR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id q11-20020a056902150b00b00d9ac1b2bb7esi15363345ybu.255.2023.10.25.13.56.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Oct 2023 13:56:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=B7z6gfUR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 3D41180A440E; Wed, 25 Oct 2023 13:54:33 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235026AbjJYUyB (ORCPT + 25 others); Wed, 25 Oct 2023 16:54:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43942 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234602AbjJYUxj (ORCPT ); Wed, 25 Oct 2023 16:53:39 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3508BD4F; Wed, 25 Oct 2023 13:53:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698267209; x=1729803209; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=ZayczDqX1KVxTZCaG/Mto+AoTwEtKqxRVewv1TUrl9g=; b=B7z6gfUR0iRhssWVQRgUdvY70iijnpQcZd5BjLhHUuUxPCdFyJPaOi5G N8ijmU9RM5zJjru0U3QruA2A64Md9E91SVRFNn5LVRPDHU6s9265dongw bQHO9SbKIbC+UkTtzc/1k+zH5UaTx6hju9PV2JXkIWFZqWzg0m5qQxaxI SNSGcsxMQX1RFg99lTpEuWjYZbuax2v3++KEp2luYOGoz4po2NUfFrEZF SsgR33ifSADt1rYKArw/IpU6bBi5VXKFXdXhVyzL6eElGseYmrgWwFfkD G713srsINhFY7Ic9h717qeb9BAKM+GyRAA2/uB535BuG0ZX9zytuKfCOj A==; X-IronPort-AV: E=McAfee;i="6600,9927,10874"; a="367610052" X-IronPort-AV: E=Sophos;i="6.03,250,1694761200"; d="scan'208";a="367610052" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Oct 2023 13:53:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10874"; a="902683314" X-IronPort-AV: E=Sophos;i="6.03,250,1694761200"; d="scan'208";a="902683314" Received: from kkomeyli-mobl.amr.corp.intel.com (HELO desk) ([10.251.29.139]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Oct 2023 13:51:03 -0700 Date: Wed, 25 Oct 2023 13:53:24 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta Subject: [PATCH v3 6/6] KVM: VMX: Move VERW closer to VMentry for MDS mitigation Message-ID: <20231025-delay-verw-v3-6-52663677ee35@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231025-delay-verw-v3-0-52663677ee35@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231025-delay-verw-v3-0-52663677ee35@linux.intel.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 25 Oct 2023 13:54:33 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780762397523017002 X-GMAIL-MSGID: 1780762397523017002 During VMentry VERW is executed to mitigate MDS. After VERW, any memory access like register push onto stack may put host data in MDS affected CPU buffers. A guest can then use MDS to sample host data. Although likelihood of secrets surviving in registers at current VERW callsite is less, but it can't be ruled out. Harden the MDS mitigation by moving the VERW mitigation late in VMentry path. Note that VERW for MMIO Stale Data mitigation is unchanged because of the complexity of per-guest conditional VERW which is not easy to handle that late in asm with no GPRs available. If the CPU is also affected by MDS, VERW is unconditionally executed late in asm regardless of guest having MMIO access. Signed-off-by: Pawan Gupta --- arch/x86/kvm/vmx/vmenter.S | 3 +++ arch/x86/kvm/vmx/vmx.c | 10 +++++++--- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index b3b13ec04bac..139960deb736 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -161,6 +161,9 @@ SYM_FUNC_START(__vmx_vcpu_run) /* Load guest RAX. This kills the @regs pointer! */ mov VCPU_RAX(%_ASM_AX), %_ASM_AX + /* Clobbers EFLAGS.ZF */ + CLEAR_CPU_BUFFERS + /* Check EFLAGS.CF from the VMX_RUN_VMRESUME bit test above. */ jnc .Lvmlaunch diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 24e8694b83fc..2d149589cf5b 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7226,13 +7226,17 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, guest_state_enter_irqoff(); - /* L1D Flush includes CPU buffer clear to mitigate MDS */ + /* + * L1D Flush includes CPU buffer clear to mitigate MDS, but VERW + * mitigation for MDS is done late in VMentry and is still + * executed inspite of L1D Flush. This is because an extra VERW + * should not matter much after the big hammer L1D Flush. + */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); - else if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF)) - mds_clear_cpu_buffers(); else if (static_branch_unlikely(&mmio_stale_data_clear) && kvm_arch_has_assigned_device(vcpu->kvm)) + /* MMIO mitigation is mutually exclusive with MDS mitigation later in asm */ mds_clear_cpu_buffers(); vmx_disable_fb_clear(vmx);