From patchwork Fri Oct 27 14:38:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 158968 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp652481vqb; Fri, 27 Oct 2023 07:39:44 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH7ASHknY1eOwPJSGa8uBNM4e3w/Pm9tk5mAOXWQui4x5ZzkhAEdAqA3iUujKE3j+4aJ6N1 X-Received: by 2002:a81:c105:0:b0:5a1:fb1d:740a with SMTP id f5-20020a81c105000000b005a1fb1d740amr2424413ywi.51.1698417584447; Fri, 27 Oct 2023 07:39:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698417584; cv=none; d=google.com; s=arc-20160816; b=CfkJ8wBejWKOkNaNfGv8lf6tXVd6F2LUqmdLap2LeXj2SuZ084s+Fd5Jbq1/L6SNka JlFNGXX0ElP9knz0LFJ50nh2JPJq7xeI9z6yBH+s+S9+yUUsReQQvLVi15V5+iyJz7gm 2d6bopHEDy8LlHNUkjt5UoGYyGI7E1zowoNdyGd8j2PM1fyxZr7MoHQWonoPIbRvXGkq THWQY20J35WZsdw29P2CWNM1Sq4BspLMqeeFW36kQEiPuKOkIg/D06vLiT8HC240QTpC 5diufguU3o8KS1E/yODZq7MkxxsLNr498gt6VRsbdziJz72hEcYbKaVRwKMr0pozXmTN 1hrQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=LILw1Z2vts/SpWBB8vwvL7y8EItpKATIVasNhXBL/rA=; fh=nr44bLQvVCZBc0nIZdewn81p1e8Bet4/2V+MdlvYUZA=; b=HmdwnOGk+BBKu0LjLx7C8zKppIMiuXnSW8XfxjaQ/P0M1ZUR6r8LNywzxLlqrubPxz Ncf5mMyFv89fUSbTrbaxCu3Cg2WygsLN58oSDsUc5g5W7R6Z82OCYKjd4bt3mo8383Ja ZVGsGfWwhlQxJ2vxgvfPN/vyhM956M8Cmy58PIpGyEd4NpIU1KpRp18YR7BFCGiuutib RgHIwt0Z6Pkysp2iGTHjuiz2rfmmBKLR2+NM8v5p68AFKoBTF2cPz3gVU1KJC8iHVug1 Qg89Ixie5ymhZ+61N0myRNo6YIjLxjoUzw2uOW32f1ZWQfmTir4HTViwDJTNDQKLckiD wrwg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=W91o8T+O; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id p16-20020a0de610000000b005a4f6575a70si2760210ywe.62.2023.10.27.07.39.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Oct 2023 07:39:44 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=W91o8T+O; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 67C4982EA163; Fri, 27 Oct 2023 07:39:33 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346065AbjJ0OjG (ORCPT + 25 others); Fri, 27 Oct 2023 10:39:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345876AbjJ0OjB (ORCPT ); Fri, 27 Oct 2023 10:39:01 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 537E2DE; Fri, 27 Oct 2023 07:38:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698417539; x=1729953539; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=PE5fU1agwGvIYR6WvOaWH9GVoG/TbRgWJmFahYBu+3E=; b=W91o8T+OFJe/GWkmC5yEGIfpYejsOutjiDN0hOcrJ8nxXEKCSzJDDKiw v05OKruP/gBwbcBL4eDc4rfgudKP6Nyj/7mHi8ZDZpvWO+tbGuuSbVlyl eort2OLe81g79DRQxFh8ak0xtHazucaBBmOQls+2D1kR5pZZYBfn5BipL GtJVM07eytSsKUnaV5efvkYdam0RLyBDr8Umxq27000uY2f5hymqrAlhk 61EPGQLz7a/iI0YtkRMa7SqpjVOUUMSnOtww7C74TliLqQt5S4je+jkke acyT9ouHYoO6w0BN/laURUYHE4dAz2Cwug0ukLXtftRIMuCuV70kkw6ml Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10876"; a="386670113" X-IronPort-AV: E=Sophos;i="6.03,256,1694761200"; d="scan'208";a="386670113" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2023 07:38:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.03,256,1694761200"; d="scan'208";a="812619" Received: from dmnassar-mobl.amr.corp.intel.com (HELO desk) ([10.212.203.39]) by orviesa002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2023 07:38:06 -0700 Date: Fri, 27 Oct 2023 07:38:40 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Greg Kroah-Hartman , Pawan Gupta , Alyssa Milburn Subject: [PATCH v4 1/6] x86/bugs: Add asm helpers for executing VERW Message-ID: <20231027-delay-verw-v4-1-9a3622d4bcf7@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231027-delay-verw-v4-0-9a3622d4bcf7@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231027-delay-verw-v4-0-9a3622d4bcf7@linux.intel.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 27 Oct 2023 07:39:33 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780919916892350987 X-GMAIL-MSGID: 1780919916892350987 MDS mitigation requires clearing the CPU buffers before returning to user. This needs to be done late in the exit-to-user path. Current location of VERW leaves a possibility of kernel data ending up in CPU buffers for memory accesses done after VERW such as: 1. Kernel data accessed by an NMI between VERW and return-to-user can remain in CPU buffers ( since NMI returning to kernel does not execute VERW to clear CPU buffers. 2. Alyssa reported that after VERW is executed, CONFIG_GCC_PLUGIN_STACKLEAK=y scrubs the stack used by a system call. Memory accesses during stack scrubbing can move kernel stack contents into CPU buffers. 3. When caller saved registers are restored after a return from function executing VERW, the kernel stack accesses can remain in CPU buffers(since they occur after VERW). To fix this VERW needs to be moved very late in exit-to-user path. In preparation for moving VERW to entry/exit asm code, create macros that can be used in asm. Also make them depend on a new feature flag X86_FEATURE_CLEAR_CPU_BUF. Reported-by: Alyssa Milburn Suggested-by: Andrew Cooper Suggested-by: Peter Zijlstra Signed-off-by: Pawan Gupta --- arch/x86/entry/entry.S | 17 +++++++++++++++++ arch/x86/include/asm/cpufeatures.h | 2 +- arch/x86/include/asm/nospec-branch.h | 15 +++++++++++++++ 3 files changed, 33 insertions(+), 1 deletion(-) diff --git a/arch/x86/entry/entry.S b/arch/x86/entry/entry.S index bfb7bcb362bc..8dc84bb9dc0b 100644 --- a/arch/x86/entry/entry.S +++ b/arch/x86/entry/entry.S @@ -6,6 +6,9 @@ #include #include #include +#include +#include +#include .pushsection .noinstr.text, "ax" @@ -20,3 +23,17 @@ SYM_FUNC_END(entry_ibpb) EXPORT_SYMBOL_GPL(entry_ibpb); .popsection + +.pushsection .entry.text, "ax" + +.align L1_CACHE_BYTES, 0xcc +SYM_CODE_START_NOALIGN(mds_verw_sel) + UNWIND_HINT_UNDEFINED + ANNOTATE_NOENDBR + .word __KERNEL_DS +.align L1_CACHE_BYTES, 0xcc +SYM_CODE_END(mds_verw_sel); +/* For KVM */ +EXPORT_SYMBOL_GPL(mds_verw_sel); + +.popsection diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 58cb9495e40f..f21fc0f12737 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -308,10 +308,10 @@ #define X86_FEATURE_SMBA (11*32+21) /* "" Slow Memory Bandwidth Allocation */ #define X86_FEATURE_BMEC (11*32+22) /* "" Bandwidth Monitoring Event Configuration */ #define X86_FEATURE_USER_SHSTK (11*32+23) /* Shadow stack support for user mode applications */ - #define X86_FEATURE_SRSO (11*32+24) /* "" AMD BTB untrain RETs */ #define X86_FEATURE_SRSO_ALIAS (11*32+25) /* "" AMD BTB untrain RETs through aliasing */ #define X86_FEATURE_IBPB_ON_VMEXIT (11*32+26) /* "" Issue an IBPB only on VMEXIT */ +#define X86_FEATURE_CLEAR_CPU_BUF (11*32+27) /* "" Clear CPU buffers */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */ diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index c55cc243592e..005e69f93115 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -329,6 +329,21 @@ #endif .endm +/* + * Macros to execute VERW instruction that mitigate transient data sampling + * attacks such as MDS. On affected systems a microcode update overloaded VERW + * instruction to also clear the CPU buffers. VERW clobbers CFLAGS.ZF. + * + * Note: Only the memory operand variant of VERW clears the CPU buffers. + */ +.macro EXEC_VERW + verw _ASM_RIP(mds_verw_sel) +.endm + +.macro CLEAR_CPU_BUFFERS + ALTERNATIVE "", __stringify(EXEC_VERW), X86_FEATURE_CLEAR_CPU_BUF +.endm + #else /* __ASSEMBLY__ */ #define ANNOTATE_RETPOLINE_SAFE \ From patchwork Fri Oct 27 14:38:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 158966 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp652286vqb; Fri, 27 Oct 2023 07:39:28 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGat+ttpoxqodONLJfsR4fw6mzspw2Y/BrUfB+ueNfS7z1WOdz88SlfDxmniJ7JwRbvIOlu X-Received: by 2002:a5b:e88:0:b0:d9c:aa17:2ae3 with SMTP id z8-20020a5b0e88000000b00d9caa172ae3mr2707967ybr.64.1698417568041; Fri, 27 Oct 2023 07:39:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698417568; cv=none; d=google.com; s=arc-20160816; b=tD3Tjpc/lazGyok1rciHY9oFF97Qb1nY3KAbDPHen6shXQSMAQUZKfUa+oqRnawSBe OREKXPsL+5HVDu7aqsSKcf+9XJxLBEDhgLYf+jathn2yz0cwpTd1UOtOHAjyh+8NF8mr 394nr1M5dfb2uUn9tbMRlIs2XTislmAvQyjlggVf9cgEBI/sbU8NqhRie5UmETxf1oP6 Yp2ZTv1hVKXQREjcA21pTDHDOCc5r+ougzA8ScO7Lb1GWwKI+7XeIKPRg39fE5Ip7Anx I2wSkTJROBzI9ij3C3Y/U68zSwCsrK3X4qHeoO/x0vW0J0VeSDuBfBFWe5bsW/J3pEDh 89uQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=JXW6ncWciaghnfyTdoACNbmv3XQBwz+zUjQClT1iM5A=; fh=CwSpNuzJZ4SYflAY8X/5Th1WavevelKKrCkLrndW6AM=; b=BmMVRykCcY97NmZtOOK2Qg+ZgpqPkwUmDRBypREqL16yvLIcHW6hMddWhASSehEDWf f+pBGw05GFdmhZgpPlkXUqObBsgGc9RACirAkj3uZt6OUNjiEbWFU22gy6IRJA9XZaVf w1Up7cO5fo0PB34Vjf/XwhnYDaw4QyWlpA0+NYIHw1e13oVXcOm1lPS6plgiPLQ1cFFw Psyy4uF+7yULYT+94vtXWcwL8XutfxFV/uoCWra1JNSFAPUWFh3qj+jMrurhaIs1cMnf dWXz37XYkqb1h3dFaERZjCFrRLqMPnJPzw2XjTL3HdVJJlaS2yld+T+vxVBgSxN3+OCE olVQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="Cuxy/yew"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id p205-20020a2542d6000000b00da0c581666bsi2821722yba.524.2023.10.27.07.39.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Oct 2023 07:39:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="Cuxy/yew"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 37803837B2E0; Fri, 27 Oct 2023 07:39:26 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346049AbjJ0OjM (ORCPT + 25 others); Fri, 27 Oct 2023 10:39:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345977AbjJ0OjC (ORCPT ); Fri, 27 Oct 2023 10:39:02 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D3D89E3; Fri, 27 Oct 2023 07:38:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698417540; x=1729953540; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=KI9R1ykioS10T+zkw0Cw1ATpow7yvlFUWGh3i+Y0oPk=; b=Cuxy/yewMll0iq1rxeyEa6vjsqrhzGCpXCTYzXRhZhOD1CrjXYM+QQDo xWzOk8Q6zbTI4v7MBgRvrCOp16n4TiGOZ9TGkn4Qn/aKhzzrTGZ1iZ0vH hqY9zn2GVYvUEHAVDA837Q1mlr6wZb6ElKbF1q7+3uakaij2vNqYhgeJF WN2ZRtXcmwnuRLw0xJswtxGFKXMgCYWl4PauGNp/0IcAY22WuQUJgQuPM 5KXcKHVSlD8eKcbxm6xLADsgj9Pu6dWP6pcE41nigDwHYCfeW4sIlUfmU sILB/8SxCoaPzmEqgtIY1peQdQQXvxpPWJgWY1FJ3/5Wyh11hSOXt+qZR w==; X-IronPort-AV: E=McAfee;i="6600,9927,10876"; a="386670135" X-IronPort-AV: E=Sophos;i="6.03,256,1694761200"; d="scan'208";a="386670135" Received: from orviesa002.jf.intel.com ([10.64.159.142]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2023 07:38:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.03,256,1694761200"; d="scan'208";a="812623" Received: from dmnassar-mobl.amr.corp.intel.com (HELO desk) ([10.212.203.39]) by orviesa002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2023 07:38:12 -0700 Date: Fri, 27 Oct 2023 07:38:47 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Greg Kroah-Hartman , Pawan Gupta , Dave Hansen Subject: [PATCH v4 2/6] x86/entry_64: Add VERW just before userspace transition Message-ID: <20231027-delay-verw-v4-2-9a3622d4bcf7@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231027-delay-verw-v4-0-9a3622d4bcf7@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231027-delay-verw-v4-0-9a3622d4bcf7@linux.intel.com> X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 27 Oct 2023 07:39:26 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780919899656758159 X-GMAIL-MSGID: 1780919899656758159 Mitigation for MDS is to use VERW instruction to clear any secrets in CPU Buffers. Any memory accesses after VERW execution can still remain in CPU buffers. It is safer to execute VERW late in return to user path to minimize the window in which kernel data can end up in CPU buffers. There are not many kernel secrets to be had after SWITCH_TO_USER_CR3. Add support for deploying VERW mitigation after user register state is restored. This helps minimize the chances of kernel data ending up into CPU buffers after executing VERW. Note that the mitigation at the new location is not yet enabled. Corner case not handled ======================= Interrupts returning to kernel don't clear CPUs buffers since the exit-to-user path is expected to do that anyways. But, there could be a case when an NMI is generated in kernel after the exit-to-user path has cleared the buffers. This case is not handled and NMI returning to kernel don't clear CPU buffers because: 1. It is rare to get an NMI after VERW, but before returning to userspace. 2. For an unprivileged user, there is no known way to make that NMI less rare or target it. 3. It would take a large number of these precisely-timed NMIs to mount an actual attack. There's presumably not enough bandwidth. 4. The NMI in question occurs after a VERW, i.e. when user state is restored and most interesting data is already scrubbed. Whats left is only the data that NMI touches, and that may or may not be of any interest. Suggested-by: Dave Hansen Signed-off-by: Pawan Gupta --- arch/x86/entry/entry_64.S | 11 +++++++++++ arch/x86/entry/entry_64_compat.S | 1 + 2 files changed, 12 insertions(+) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 43606de22511..9f97a8bd11e8 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -223,6 +223,7 @@ syscall_return_via_sysret: SYM_INNER_LABEL(entry_SYSRETQ_unsafe_stack, SYM_L_GLOBAL) ANNOTATE_NOENDBR swapgs + CLEAR_CPU_BUFFERS sysretq SYM_INNER_LABEL(entry_SYSRETQ_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR @@ -663,6 +664,7 @@ SYM_INNER_LABEL(swapgs_restore_regs_and_return_to_usermode, SYM_L_GLOBAL) /* Restore RDI. */ popq %rdi swapgs + CLEAR_CPU_BUFFERS jmp .Lnative_iret @@ -774,6 +776,8 @@ native_irq_return_ldt: */ popq %rax /* Restore user RAX */ + CLEAR_CPU_BUFFERS + /* * RSP now points to an ordinary IRET frame, except that the page * is read-only and RSP[31:16] are preloaded with the userspace @@ -1502,6 +1506,12 @@ nmi_restore: std movq $0, 5*8(%rsp) /* clear "NMI executing" */ + /* + * Skip CLEAR_CPU_BUFFERS here, since it only helps in rare cases like + * NMI in kernel after user state is restored. For an unprivileged user + * these conditions are hard to meet. + */ + /* * iretq reads the "iret" frame and exits the NMI stack in a * single instruction. We are returning to kernel mode, so this @@ -1520,6 +1530,7 @@ SYM_CODE_START(ignore_sysret) UNWIND_HINT_END_OF_STACK ENDBR mov $-ENOSYS, %eax + CLEAR_CPU_BUFFERS sysretl SYM_CODE_END(ignore_sysret) #endif diff --git a/arch/x86/entry/entry_64_compat.S b/arch/x86/entry/entry_64_compat.S index 70150298f8bd..245697eb8485 100644 --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -271,6 +271,7 @@ SYM_INNER_LABEL(entry_SYSRETL_compat_unsafe_stack, SYM_L_GLOBAL) xorl %r9d, %r9d xorl %r10d, %r10d swapgs + CLEAR_CPU_BUFFERS sysretl SYM_INNER_LABEL(entry_SYSRETL_compat_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR From patchwork Fri Oct 27 14:38:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 158969 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp652485vqb; Fri, 27 Oct 2023 07:39:45 -0700 (PDT) X-Google-Smtp-Source: AGHT+IERTVk3V2NO15hSMBpcCpAvbknyk6uVFO8wHcyXO39BShwMcDJfdWhCmv7pyb4f2U43y8DW X-Received: by 2002:a81:ad1d:0:b0:5a8:1a54:ba4b with SMTP id l29-20020a81ad1d000000b005a81a54ba4bmr11388860ywh.13.1698417585161; Fri, 27 Oct 2023 07:39:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698417585; cv=none; d=google.com; s=arc-20160816; b=Y9Kj2vRFKo1tmzJlfqoPtsRKelQnNTP9DCdsQ3ON6uPE1aDlPb2q/qm+I5ZDefNtMD X/8A7kNKWIK7/l6w3C8xH2GyL+dw5/OW9EWsLq6hy5rJfX0Ku+oqpgDW1+9izRNCSEnl VrWuVhrF5P3/+wLmLdwmXcR0H+q6Ww4A0ai3OA2lFoOLSirDoDvwgHjTtEJOei27TV7n bwCGkcb6X7PYj2ZqCjyAc1ObWG2jB+sMLosp/SqV210Q36Xnt0//+lFbt1aw5SVXhf/a BX43cCCP9rUs4+xbMcFet7bSGH2NAfu2CT8wFINXcqX6y/FZ1cAVGcOUxxI8B0sXN/qQ yAUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=UtZ6k9Oyj9C76UgVXziiWcsI59b09aZmqIi0GzVC7eo=; fh=9bGxxPI+IiFttPs3QhfhBAZDnxipaVmWpf+QSj2Bc/s=; b=EFIA8OdueVO/JEjLzAG6OVtk1iKEjuFeWmiCsQr4ZIvILFCFNTbnVuvbMsGcuOBWxH JtJYd2o3sU0FLDMUwq+IFJ64C0NBPWM5CdqkLvwpn4GM5jqbOkMrS11lVzyR1ajac+1x YsywMhc1h7QkEW8dSPUY6IRaq67tNn/dAKcweWpNlZhLX27NgGAQL95HqNCZvt2EwTYI ucmmBosSIxB93dMQXFhGin/oUL0kG30eV861UgwC5Jl5wTaSovXjze2ahklb7FwmdZ25 6eWmSRfbCEE7vjB7OZSC1nMgiGeXbpGIq9147KsYcruMgpk02EeoSMrQwds17De+iKXV +Ixw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=lcTOp6Ug; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id y7-20020a818807000000b005af6d13d25fsi2587271ywf.300.2023.10.27.07.39.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Oct 2023 07:39:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=lcTOp6Ug; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 915DC80A1A29; Fri, 27 Oct 2023 07:39:29 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345875AbjJ0Oi7 (ORCPT + 25 others); Fri, 27 Oct 2023 10:38:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57666 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346132AbjJ0Oi5 (ORCPT ); Fri, 27 Oct 2023 10:38:57 -0400 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 026CBD7; Fri, 27 Oct 2023 07:38:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698417535; x=1729953535; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=fKL8O9LfX0J38E00Mtc5QFOreN8mewIuIipoFxbNi84=; b=lcTOp6UgzEM6Kf2icDAPp8eLvbjv1T8ASaIqPXW6M0HZtEJe5sTiymwh Qc7R0PTSEpTTW1n1WwIjny1mDKps4glpB6KjAjrufN+8Vbg/Q0WS/GhMZ +NSoV0ytTMZKmO8YIBFopGxPVIY4rNSRO4/HS2w3EU/29VL0c9sBdBcyx jGws/LS+7kTD8jyv9i0aXlt74O+HoaBJkusX+9gszJHs3/Sood0q95rWm JwXUglzIKudXx7c9ZdsPJ+lDCUQ3SVYUxWoWlSE35p/4K12ikrZivkrXu xCDEG53jCelfRqXzBD8ITmu88ygrYczzU2v1hVkHiyN984iERbBEV51JM g==; X-IronPort-AV: E=McAfee;i="6600,9927,10876"; a="606700" X-IronPort-AV: E=Sophos;i="6.03,256,1694761200"; d="scan'208";a="606700" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2023 07:38:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10876"; a="736094436" X-IronPort-AV: E=Sophos;i="6.03,256,1694761200"; d="scan'208";a="736094436" Received: from dmnassar-mobl.amr.corp.intel.com (HELO desk) ([10.212.203.39]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2023 07:38:53 -0700 Date: Fri, 27 Oct 2023 07:38:53 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Greg Kroah-Hartman , Pawan Gupta Subject: [PATCH v4 3/6] x86/entry_32: Add VERW just before userspace transition Message-ID: <20231027-delay-verw-v4-3-9a3622d4bcf7@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231027-delay-verw-v4-0-9a3622d4bcf7@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231027-delay-verw-v4-0-9a3622d4bcf7@linux.intel.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 27 Oct 2023 07:39:29 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780919917658898848 X-GMAIL-MSGID: 1780919917658898848 As done for entry_64, add support for executing VERW late in exit to user path for 32-bit mode. Signed-off-by: Pawan Gupta --- arch/x86/entry/entry_32.S | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S index 6e6af42e044a..74a4358c7f45 100644 --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -885,6 +885,7 @@ SYM_FUNC_START(entry_SYSENTER_32) BUG_IF_WRONG_CR3 no_user_check=1 popfl popl %eax + CLEAR_CPU_BUFFERS /* * Return back to the vDSO, which will pop ecx and edx. @@ -954,6 +955,7 @@ restore_all_switch_stack: /* Restore user state */ RESTORE_REGS pop=4 # skip orig_eax/error_code + CLEAR_CPU_BUFFERS .Lirq_return: /* * ARCH_HAS_MEMBARRIER_SYNC_CORE rely on IRET core serialization @@ -1146,6 +1148,7 @@ SYM_CODE_START(asm_exc_nmi) /* Not on SYSENTER stack. */ call exc_nmi + CLEAR_CPU_BUFFERS jmp .Lnmi_return .Lnmi_from_sysenter_stack: From patchwork Fri Oct 27 14:38:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 158970 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp653086vqb; Fri, 27 Oct 2023 07:40:38 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEgqGA/JuNBEtSP+F4lGPwQZfzdA4hmcuRX4eHXaKBDMKWMKc11vRe17WEi072GRj7H3cci X-Received: by 2002:a25:7105:0:b0:da0:aa1b:80f3 with SMTP id m5-20020a257105000000b00da0aa1b80f3mr9275758ybc.10.1698417638203; Fri, 27 Oct 2023 07:40:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698417638; cv=none; d=google.com; s=arc-20160816; b=0cdsFVNb37I5EL6iPrzmsk2vkNdXi35OxxDdJ5Bshkv/vnMtFI3FO7iD9mqXUAw04M qYZMvUkbGeexTVrPEGSqwky147OXKrPv1uQFjA99On9QLyud1ouRLkia/5KcN6oHxjfN cVfVSZPGRd+pOBzN0jnf6D+0aCeyg6AjW6SsQo+NBF+1UpBOUzNAgZMZLT7yNGoLeqJG XEGJkVxA2rR29npRdSUz/eT4pdZnzRswfYNnYqYdNb0TVBLbYttMkoLsmUE/QmNkcmjf JbCDw6HUOxwWKSO3ObUYfxTIfoErbHAY6r55ysqyEI97jxSME2nnmYzq+yfAyf8REqKo 7bOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=7ib1O0KtQFcIuzsnAxnO/yCusjbnTobTJA8lJBPj/ZM=; fh=9bGxxPI+IiFttPs3QhfhBAZDnxipaVmWpf+QSj2Bc/s=; b=vjZ3U4DBpgL43XNE3lQtUfb2kXgkav8TKWFVKivU5qX37WuCIRHJ5IPy/LMAJzy6gJ ZXKmfle87Sff295rXs5MzPJ+W0xrEPdwVIWcG6j1CdAYYVWScVxhb9djYqgxRp+x4vdU ldpXEJYNlEwy7NamQeXvM080GKvZzsLSPUcWe05eI42Fvzk8scvotf3EDdamPDSIaSOH JZNms8vK+QsbWb6cSZNO8i8y7IRm6RGF1eeFqdrO4KMU42sMm1feeusT7QKvY4FemDsZ 09eHe0JlukXW2YZ++ENmAUzwG6gERCPml6q0a8JvbxbIAgxLc9ztKjU2Fa8xiuxD2qPC xY6Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Z95Un8nI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id 125-20020a251383000000b00d9a40a999edsi2534005ybt.38.2023.10.27.07.40.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Oct 2023 07:40:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Z95Un8nI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 26AF5808ED8B; Fri, 27 Oct 2023 07:40:34 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346117AbjJ0OjJ (ORCPT + 25 others); Fri, 27 Oct 2023 10:39:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57818 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346018AbjJ0OjE (ORCPT ); Fri, 27 Oct 2023 10:39:04 -0400 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B4E3418F; Fri, 27 Oct 2023 07:39:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698417541; x=1729953541; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=xfVDUL3C6EU2sR+yZZNhlJ+3NJv0rEqWLQFJQSLp6Pk=; b=Z95Un8nIP2g5kboWzTwtTofMVI/z2wIQxNOO+Zvcd+vh5xecf+AdjqBe eLNJS/qfe8XHAO/xd1r3K7uaBOMlEYONxxHSbRLa22Q4rhyEGGBoKF4SH E2TzHHwTYsseEyQnnizLpdWT0F0o9d9Dx3z92bAtwiPVSr5Bz466fROBA 8T1rswWygQZCxl1fs+vvpSuL7xYgVcnYqdAQjNLRj5XPq7oW1gDA4UXBy 4vQum3xhkJYha5ybCD6apZoqrart2aHEjAaPOkxgYHnIO4+Lmc3Pw2Gjd lsyXIKDPviG313r8d5z6AMOUbZCvh3kYBEJKBSBoI5R6sQVFt979k2mDc A==; X-IronPort-AV: E=McAfee;i="6600,9927,10876"; a="606722" X-IronPort-AV: E=Sophos;i="6.03,256,1694761200"; d="scan'208";a="606722" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2023 07:39:00 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10876"; a="736094439" X-IronPort-AV: E=Sophos;i="6.03,256,1694761200"; d="scan'208";a="736094439" Received: from dmnassar-mobl.amr.corp.intel.com (HELO desk) ([10.212.203.39]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2023 07:38:59 -0700 Date: Fri, 27 Oct 2023 07:38:59 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Greg Kroah-Hartman , Pawan Gupta Subject: [PATCH v4 4/6] x86/bugs: Use ALTERNATIVE() instead of mds_user_clear static key Message-ID: <20231027-delay-verw-v4-4-9a3622d4bcf7@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231027-delay-verw-v4-0-9a3622d4bcf7@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231027-delay-verw-v4-0-9a3622d4bcf7@linux.intel.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 27 Oct 2023 07:40:34 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780919973210332312 X-GMAIL-MSGID: 1780919973210332312 The VERW mitigation at exit-to-user is enabled via a static branch mds_user_clear. This static branch is never toggled after boot, and can be safely replaced with an ALTERNATIVE() which is convenient to use in asm. Switch to ALTERNATIVE() to use the VERW mitigation late in exit-to-user path. Also remove the now redundant VERW in exc_nmi() and arch_exit_to_user_mode(). Signed-off-by: Pawan Gupta --- Documentation/arch/x86/mds.rst | 38 +++++++++++++++++++++++++----------- arch/x86/include/asm/entry-common.h | 1 - arch/x86/include/asm/nospec-branch.h | 12 ------------ arch/x86/kernel/cpu/bugs.c | 15 ++++++-------- arch/x86/kernel/nmi.c | 2 -- arch/x86/kvm/vmx/vmx.c | 2 +- 6 files changed, 34 insertions(+), 36 deletions(-) diff --git a/Documentation/arch/x86/mds.rst b/Documentation/arch/x86/mds.rst index e73fdff62c0a..a5c5091b9ccd 100644 --- a/Documentation/arch/x86/mds.rst +++ b/Documentation/arch/x86/mds.rst @@ -95,6 +95,9 @@ The kernel provides a function to invoke the buffer clearing: mds_clear_cpu_buffers() +Also macro CLEAR_CPU_BUFFERS is meant to be used in ASM late in exit-to-user +path. This macro works for cases where GPRs can't be clobbered. + The mitigation is invoked on kernel/userspace, hypervisor/guest and C-state (idle) transitions. @@ -138,17 +141,30 @@ Mitigation points When transitioning from kernel to user space the CPU buffers are flushed on affected CPUs when the mitigation is not disabled on the kernel - command line. The migitation is enabled through the static key - mds_user_clear. - - The mitigation is invoked in prepare_exit_to_usermode() which covers - all but one of the kernel to user space transitions. The exception - is when we return from a Non Maskable Interrupt (NMI), which is - handled directly in do_nmi(). - - (The reason that NMI is special is that prepare_exit_to_usermode() can - enable IRQs. In NMI context, NMIs are blocked, and we don't want to - enable IRQs with NMIs blocked.) + command line. The mitigation is enabled through the feature flag + X86_FEATURE_CLEAR_CPU_BUF. + + The mitigation is invoked just before transitioning to userspace after + user registers are restored. This is done to minimize the window in + which kernel data could be accessed after VERW e.g. via an NMI after + VERW. + + **Corner case not handled** + Interrupts returning to kernel don't clear CPUs buffers since the + exit-to-user path is expected to do that anyways. But, there could be + a case when an NMI is generated in kernel after the exit-to-user path + has cleared the buffers. This case is not handled and NMI returning to + kernel don't clear CPU buffers because: + + 1. It is rare to get an NMI after VERW, but before returning to userspace. + 2. For an unprivileged user, there is no known way to make that NMI + less rare or target it. + 3. It would take a large number of these precisely-timed NMIs to mount + an actual attack. There's presumably not enough bandwidth. + 4. The NMI in question occurs after a VERW, i.e. when user state is + restored and most interesting data is already scrubbed. Whats left + is only the data that NMI touches, and that may or may not be of + any interest. 2. C-State transition diff --git a/arch/x86/include/asm/entry-common.h b/arch/x86/include/asm/entry-common.h index ce8f50192ae3..7e523bb3d2d3 100644 --- a/arch/x86/include/asm/entry-common.h +++ b/arch/x86/include/asm/entry-common.h @@ -91,7 +91,6 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, static __always_inline void arch_exit_to_user_mode(void) { - mds_user_clear_cpu_buffers(); amd_clear_divider(); } #define arch_exit_to_user_mode arch_exit_to_user_mode diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index 005e69f93115..12b8e86678bf 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -553,7 +553,6 @@ DECLARE_STATIC_KEY_FALSE(switch_to_cond_stibp); DECLARE_STATIC_KEY_FALSE(switch_mm_cond_ibpb); DECLARE_STATIC_KEY_FALSE(switch_mm_always_ibpb); -DECLARE_STATIC_KEY_FALSE(mds_user_clear); DECLARE_STATIC_KEY_FALSE(mds_idle_clear); DECLARE_STATIC_KEY_FALSE(switch_mm_cond_l1d_flush); @@ -585,17 +584,6 @@ static __always_inline void mds_clear_cpu_buffers(void) asm volatile("verw %[ds]" : : [ds] "m" (ds) : "cc"); } -/** - * mds_user_clear_cpu_buffers - Mitigation for MDS and TAA vulnerability - * - * Clear CPU buffers if the corresponding static key is enabled - */ -static __always_inline void mds_user_clear_cpu_buffers(void) -{ - if (static_branch_likely(&mds_user_clear)) - mds_clear_cpu_buffers(); -} - /** * mds_idle_clear_cpu_buffers - Mitigation for MDS vulnerability * diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index 10499bcd4e39..00aab0c0937f 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -111,9 +111,6 @@ DEFINE_STATIC_KEY_FALSE(switch_mm_cond_ibpb); /* Control unconditional IBPB in switch_mm() */ DEFINE_STATIC_KEY_FALSE(switch_mm_always_ibpb); -/* Control MDS CPU buffer clear before returning to user space */ -DEFINE_STATIC_KEY_FALSE(mds_user_clear); -EXPORT_SYMBOL_GPL(mds_user_clear); /* Control MDS CPU buffer clear before idling (halt, mwait) */ DEFINE_STATIC_KEY_FALSE(mds_idle_clear); EXPORT_SYMBOL_GPL(mds_idle_clear); @@ -252,7 +249,7 @@ static void __init mds_select_mitigation(void) if (!boot_cpu_has(X86_FEATURE_MD_CLEAR)) mds_mitigation = MDS_MITIGATION_VMWERV; - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); if (!boot_cpu_has(X86_BUG_MSBDS_ONLY) && (mds_nosmt || cpu_mitigations_auto_nosmt())) @@ -356,7 +353,7 @@ static void __init taa_select_mitigation(void) * For guests that can't determine whether the correct microcode is * present on host, enable the mitigation for UCODE_NEEDED as well. */ - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); if (taa_nosmt || cpu_mitigations_auto_nosmt()) cpu_smt_disable(false); @@ -424,7 +421,7 @@ static void __init mmio_select_mitigation(void) */ if (boot_cpu_has_bug(X86_BUG_MDS) || (boot_cpu_has_bug(X86_BUG_TAA) && boot_cpu_has(X86_FEATURE_RTM))) - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); else static_branch_enable(&mmio_stale_data_clear); @@ -484,12 +481,12 @@ static void __init md_clear_update_mitigation(void) if (cpu_mitigations_off()) return; - if (!static_key_enabled(&mds_user_clear)) + if (!boot_cpu_has(X86_FEATURE_CLEAR_CPU_BUF)) goto out; /* - * mds_user_clear is now enabled. Update MDS, TAA and MMIO Stale Data - * mitigation, if necessary. + * X86_FEATURE_CLEAR_CPU_BUF is now enabled. Update MDS, TAA and MMIO + * Stale Data mitigation, if necessary. */ if (mds_mitigation == MDS_MITIGATION_OFF && boot_cpu_has_bug(X86_BUG_MDS)) { diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c index a0c551846b35..ebfff8dca661 100644 --- a/arch/x86/kernel/nmi.c +++ b/arch/x86/kernel/nmi.c @@ -551,8 +551,6 @@ DEFINE_IDTENTRY_RAW(exc_nmi) if (this_cpu_dec_return(nmi_state)) goto nmi_restart; - if (user_mode(regs)) - mds_user_clear_cpu_buffers(); if (IS_ENABLED(CONFIG_NMI_CHECK_CPU)) { WRITE_ONCE(nsp->idt_seq, nsp->idt_seq + 1); WARN_ON_ONCE(nsp->idt_seq & 0x1); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 72e3943f3693..24e8694b83fc 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7229,7 +7229,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, /* L1D Flush includes CPU buffer clear to mitigate MDS */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); - else if (static_branch_unlikely(&mds_user_clear)) + else if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF)) mds_clear_cpu_buffers(); else if (static_branch_unlikely(&mmio_stale_data_clear) && kvm_arch_has_assigned_device(vcpu->kvm)) From patchwork Fri Oct 27 14:39:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 158967 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp652376vqb; Fri, 27 Oct 2023 07:39:35 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHi8LHYZuX9GUVQ+D0opyx+n6cyuVcpBv06BqE6R3jVMCyuzOYQEbuHeb+vwimKcqYWv3wt X-Received: by 2002:a25:8b0d:0:b0:da0:6876:a8eb with SMTP id i13-20020a258b0d000000b00da06876a8ebmr2497528ybl.46.1698417575123; Fri, 27 Oct 2023 07:39:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698417575; cv=none; d=google.com; s=arc-20160816; b=qshldcx+vtCbXnylQk1eK/NVFpnGBxw4y/zMp34LtSP0i8K/gcj5tZQZXJ46CNdDBB gN4bwJlxtVnHSoAiEpW4syAukHopRcd2zY/5hnsOV5nXdEpFFwrPo2Y5BkdcsbiLXArM HnozpVsQdy85stYh4FcHZfKr5nah978YPL4tY7eXfAzyVYTTBpFQdCkgYX5QLwDMCmOR zsuo7+K9mQEzuB2msaEuuj2q3NDrS2KEO2t+ghe02wnKZ4V9KPyCWOa3pCp1YR8gZeox oGOP865BzpJuiS6mP8RcZc+iAfwmRF6BVWkQMF635nXcD6ZJxuCtoGnAgT7H4/9vq1yq V8eg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=ilqGXAQjD0Zw0p+0QS0PhsqpTUKzsxY97u90wwaRZ8Y=; fh=9bGxxPI+IiFttPs3QhfhBAZDnxipaVmWpf+QSj2Bc/s=; b=sXUVpgl7OYjM5m3SxJw9Qrj1kWdsfFEOr8ZIx5iheJPikUvTVllep3B2ubFRgKZjJK /AK0etmPBGN5ABdqGWHMmxVBtfcX89540JNNmLVNd/ekKHRrZCIJsMVpwUOpQCHkoniz xYFXi5BLN7ERPqtsAmoFVbct2FCX0uryFGHI/8esbyoI/Weg57DygI9/i/iTyDgOg9S5 zMvYJTxCdlg6kcf740uGT8NN9Yn+McsNmEiOfmD7XxWGlooZ4XfSAtE5pn+t0GJsKWLn vcCrbBwLFwnJYtZKVN+66X3L2IbGkpYcniKkbnk1Y9T6NSz25mBh2/Fxn1+LdlAsxLxR +UcQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=hHufNRh+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id b145-20020a25cb97000000b00d9a56bfe8c9si3312619ybg.418.2023.10.27.07.39.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Oct 2023 07:39:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=hHufNRh+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id D64BF837B2EB; Fri, 27 Oct 2023 07:39:33 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345977AbjJ0OjO (ORCPT + 25 others); Fri, 27 Oct 2023 10:39:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54752 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346135AbjJ0OjK (ORCPT ); Fri, 27 Oct 2023 10:39:10 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BB9D2D72; Fri, 27 Oct 2023 07:39:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698417547; x=1729953547; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=rpFgire7Svqru/HfpQ1b16RkVx0842HOL7xbgXY2vaQ=; b=hHufNRh+NLW/FgsdNwC6c6agifg4xg51awEAQIrprz+89E75AAsigGyP DlvtX4f6HiXQMi0Sm040LMlR/RHXBwoyBJ/zRj8XXDwDBAwWJja7fY34E K/GdO1Byc8TK0RRuOt3ziPQNWJcQLKEKJkMUgmdti9C8yokm+3TY+VJZa FABVIUeQ7PNiu+Zqqs8Ntmb8dEYrSfS8CZgIIWELY1a32ecfbrwbgYd2l cnjOqKnnZ5yGoiQ/UZZ2elR8Rnl4b+80aw+IOA9Cl/r/92/xYa6+tHKFE 8/g3A/y8ZUTZdegnNBeXyP4h2m/TDo5/bb7k68z1Dwn5IIai+t5DGdalV w==; X-IronPort-AV: E=McAfee;i="6600,9927,10876"; a="367994398" X-IronPort-AV: E=Sophos;i="6.03,256,1694761200"; d="scan'208";a="367994398" Received: from fmviesa001.fm.intel.com ([10.60.135.141]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2023 07:39:06 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.03,256,1694761200"; d="scan'208";a="7658585" Received: from dmnassar-mobl.amr.corp.intel.com (HELO desk) ([10.212.203.39]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2023 07:38:54 -0700 Date: Fri, 27 Oct 2023 07:39:05 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Greg Kroah-Hartman , Pawan Gupta Subject: [PATCH v4 5/6] KVM: VMX: Use BT+JNC, i.e. EFLAGS.CF to select VMRESUME vs. VMLAUNCH Message-ID: <20231027-delay-verw-v4-5-9a3622d4bcf7@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231027-delay-verw-v4-0-9a3622d4bcf7@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231027-delay-verw-v4-0-9a3622d4bcf7@linux.intel.com> X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 27 Oct 2023 07:39:33 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780919907522586351 X-GMAIL-MSGID: 1780919907522586351 From: Sean Christopherson Use EFLAGS.CF instead of EFLAGS.ZF to track whether to use VMRESUME versus VMLAUNCH. Freeing up EFLAGS.ZF will allow doing VERW, which clobbers ZF, for MDS mitigations as late as possible without needing to duplicate VERW for both paths. Reviewed-by: Nikolay Borisov Signed-off-by: Sean Christopherson Signed-off-by: Pawan Gupta --- arch/x86/kvm/vmx/run_flags.h | 7 +++++-- arch/x86/kvm/vmx/vmenter.S | 6 +++--- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/vmx/run_flags.h b/arch/x86/kvm/vmx/run_flags.h index edc3f16cc189..6a9bfdfbb6e5 100644 --- a/arch/x86/kvm/vmx/run_flags.h +++ b/arch/x86/kvm/vmx/run_flags.h @@ -2,7 +2,10 @@ #ifndef __KVM_X86_VMX_RUN_FLAGS_H #define __KVM_X86_VMX_RUN_FLAGS_H -#define VMX_RUN_VMRESUME (1 << 0) -#define VMX_RUN_SAVE_SPEC_CTRL (1 << 1) +#define VMX_RUN_VMRESUME_SHIFT 0 +#define VMX_RUN_SAVE_SPEC_CTRL_SHIFT 1 + +#define VMX_RUN_VMRESUME BIT(VMX_RUN_VMRESUME_SHIFT) +#define VMX_RUN_SAVE_SPEC_CTRL BIT(VMX_RUN_SAVE_SPEC_CTRL_SHIFT) #endif /* __KVM_X86_VMX_RUN_FLAGS_H */ diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index be275a0410a8..b3b13ec04bac 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -139,7 +139,7 @@ SYM_FUNC_START(__vmx_vcpu_run) mov (%_ASM_SP), %_ASM_AX /* Check if vmlaunch or vmresume is needed */ - test $VMX_RUN_VMRESUME, %ebx + bt $VMX_RUN_VMRESUME_SHIFT, %ebx /* Load guest registers. Don't clobber flags. */ mov VCPU_RCX(%_ASM_AX), %_ASM_CX @@ -161,8 +161,8 @@ SYM_FUNC_START(__vmx_vcpu_run) /* Load guest RAX. This kills the @regs pointer! */ mov VCPU_RAX(%_ASM_AX), %_ASM_AX - /* Check EFLAGS.ZF from 'test VMX_RUN_VMRESUME' above */ - jz .Lvmlaunch + /* Check EFLAGS.CF from the VMX_RUN_VMRESUME bit test above. */ + jnc .Lvmlaunch /* * After a successful VMRESUME/VMLAUNCH, control flow "magically" From patchwork Fri Oct 27 14:39:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 158971 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp653952vqb; Fri, 27 Oct 2023 07:40:56 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH1Er5GvItPg+ReluhCWNE36UPLLLV4JeTtCPGDyEhAMzLU8656z2K2G+AmcGn3k7hIaMEX X-Received: by 2002:a05:6870:c58b:b0:1d6:5c40:11b6 with SMTP id ba11-20020a056870c58b00b001d65c4011b6mr3605110oab.14.1698417656616; Fri, 27 Oct 2023 07:40:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698417656; cv=none; d=google.com; s=arc-20160816; b=v6MHWIANkDVYedH/AHxrHjWuraudBPpAU/VwcBq147mTfm6o/gluUUEYE0Uy19YjTr cI7Wq0DsV8NoWWxPDJmgtY3F0Q5DXdDfDa6f2p54YPf73yym9ubh0NPgaHqh2C9W4cLw HLddK6j+BoGQZnhzEEnVZAqC5E2bRIw6dVjRxG3j/FTkDRx5oYXCJG60aRVdK9LTBizV y1jFbX/QZYxLOY3Wa9DA56uVfE03RbTmxZvcB7ZDCeP7BoxgXvreYVduLq9YfHNjYy4a xns3qbKz90ZzncgBKBTiqWih6NpB2z1diesc8X7BeFOT92tdQo/5+n8vRk0L6fNTFEHa x0nA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=zk7u2nr/8T5QhQ4lWBIy52Kjfo5l444VWypB6DxfrGY=; fh=9bGxxPI+IiFttPs3QhfhBAZDnxipaVmWpf+QSj2Bc/s=; b=rIR4n+5aPAt6IHHSh0vwO2QivX/hVC5rEsOyDd3e0OzT66oYIVCPiKhq59RR+tvZZb ommpzMsq7PyeADFJ1Croe6b5stpRxvOZMZB7rzHMHooerkZkpow8gyN7HVT19IsoW/pz N4ZwOSRJf4h4aj+Ahq3OBvkhcN39Uk6dOltKAQcqD/8NNeOnD/Max6T1LbQ5Ksd2i59T W2w8hnpilgVnNFF6m7kJvoWe3zJdIjZxW3ItHVKNVNtfNrp1U4Z25fqdcjsPkK9X6irS /Ar261d1CZy8gRwtdZWa5jGTcGXwWn1QCwAfDaApw8JPyzofZIMeuHSleWUIR7c8kX/f Hoig== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=RBWjJBXC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id s185-20020a8177c2000000b0059eb9ebb8d7si2583891ywc.383.2023.10.27.07.40.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Oct 2023 07:40:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=RBWjJBXC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 1D4248076037; Fri, 27 Oct 2023 07:40:18 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346172AbjJ0Oj3 (ORCPT + 25 others); Fri, 27 Oct 2023 10:39:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60582 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346132AbjJ0OjS (ORCPT ); Fri, 27 Oct 2023 10:39:18 -0400 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3195610DB; Fri, 27 Oct 2023 07:39:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698417555; x=1729953555; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=mjKmJXB8oVMEQJ55YBl9y3vDRhOMcEONKyySoTTCMIQ=; b=RBWjJBXCODCf+1lv6IFxLpNKiOi0W249ivbKQlHK2/AL30xAd3AzvK3N LmE+gJNT20pwF1y2G0kPQ8yRr7npy3fqqKEK8JspS7AchJI0e3DsFklDg nudiPdPXRc/PSktFDkRKtCC1kqVgW+Rna6xtk12fp4s4RbVnB0Gx/47s9 2aXPq0G180BDJPAZTRTWuERI+CbM9NvNhbRzznIl40LPPDew52YbMh1ut LRX+E0w8DQpHhwO3leAl2lOZoAJeY+LKotw9zGJwNi1t5gD3KNvIBjBki reXRi/gQLDwyR9PVNUyP6W9LUmyPht3gNXBeNSnUBSZXJQ+ZFYa9ZGdL7 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10876"; a="606760" X-IronPort-AV: E=Sophos;i="6.03,256,1694761200"; d="scan'208";a="606760" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2023 07:39:13 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10876"; a="736094456" X-IronPort-AV: E=Sophos;i="6.03,256,1694761200"; d="scan'208";a="736094456" Received: from dmnassar-mobl.amr.corp.intel.com (HELO desk) ([10.212.203.39]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2023 07:39:12 -0700 Date: Fri, 27 Oct 2023 07:39:12 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Greg Kroah-Hartman , Pawan Gupta Subject: [PATCH v4 6/6] KVM: VMX: Move VERW closer to VMentry for MDS mitigation Message-ID: <20231027-delay-verw-v4-6-9a3622d4bcf7@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231027-delay-verw-v4-0-9a3622d4bcf7@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231027-delay-verw-v4-0-9a3622d4bcf7@linux.intel.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 27 Oct 2023 07:40:18 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780919992655207620 X-GMAIL-MSGID: 1780919992655207620 During VMentry VERW is executed to mitigate MDS. After VERW, any memory access like register push onto stack may put host data in MDS affected CPU buffers. A guest can then use MDS to sample host data. Although likelihood of secrets surviving in registers at current VERW callsite is less, but it can't be ruled out. Harden the MDS mitigation by moving the VERW mitigation late in VMentry path. Note that VERW for MMIO Stale Data mitigation is unchanged because of the complexity of per-guest conditional VERW which is not easy to handle that late in asm with no GPRs available. If the CPU is also affected by MDS, VERW is unconditionally executed late in asm regardless of guest having MMIO access. Signed-off-by: Pawan Gupta --- arch/x86/kvm/vmx/vmenter.S | 3 +++ arch/x86/kvm/vmx/vmx.c | 19 ++++++++++++++----- 2 files changed, 17 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index b3b13ec04bac..139960deb736 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -161,6 +161,9 @@ SYM_FUNC_START(__vmx_vcpu_run) /* Load guest RAX. This kills the @regs pointer! */ mov VCPU_RAX(%_ASM_AX), %_ASM_AX + /* Clobbers EFLAGS.ZF */ + CLEAR_CPU_BUFFERS + /* Check EFLAGS.CF from the VMX_RUN_VMRESUME bit test above. */ jnc .Lvmlaunch diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 24e8694b83fc..a05c6b80b06c 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7226,16 +7226,24 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, guest_state_enter_irqoff(); - /* L1D Flush includes CPU buffer clear to mitigate MDS */ + /* + * L1D Flush includes CPU buffer clear to mitigate MDS, but VERW + * mitigation for MDS is done late in VMentry and is still + * executed in spite of L1D Flush. This is because an extra VERW + * should not matter much after the big hammer L1D Flush. + */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); - else if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF)) - mds_clear_cpu_buffers(); else if (static_branch_unlikely(&mmio_stale_data_clear) && kvm_arch_has_assigned_device(vcpu->kvm)) mds_clear_cpu_buffers(); - vmx_disable_fb_clear(vmx); + /* + * Optimize the latency of VERW in guests for MMIO mitigation. Skip + * the optimization when MDS mitigation(later in asm) is enabled. + */ + if (!cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF)) + vmx_disable_fb_clear(vmx); if (vcpu->arch.cr2 != native_read_cr2()) native_write_cr2(vcpu->arch.cr2); @@ -7248,7 +7256,8 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, vmx->idt_vectoring_info = 0; - vmx_enable_fb_clear(vmx); + if (!cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF)) + vmx_enable_fb_clear(vmx); if (unlikely(vmx->fail)) { vmx->exit_reason.full = 0xdead;