From patchwork Thu Jan 11 08:56:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 187247 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2411:b0:101:2151:f287 with SMTP id m17csp1314793dyi; Thu, 11 Jan 2024 00:59:09 -0800 (PST) X-Google-Smtp-Source: AGHT+IHtjVFi8LFoN39haQdXZTxcFw+Zb54lTGgnG6OvuR4eBk09LXxrvgAgoDAUqZhiR2azpFBU X-Received: by 2002:aa7:9254:0:b0:6d9:30d4:a88a with SMTP id 20-20020aa79254000000b006d930d4a88amr773394pfp.62.1704963549044; Thu, 11 Jan 2024 00:59:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704963549; cv=none; d=google.com; s=arc-20160816; b=qaiRguvQCOYoJ78kzJeX2NmbEy4KkQqpC9iAwEBo8PM2MLN8rzJxPDhClD998VML+T /leEnsXVtiOxx3DcztkGvRlNw3tTrxKung9bnYnAqK/z4klXgZkcgZhCoqax6vRmHKpN rRmzJmQQ9/6F7kVw9T4ACKe2uy/BJYYF6Me8vqbWTbjJEWyZF9p3uzhz29JNaIaTq3BU idCFVwt4C8ZUuRKikgDK6oPM7TGs+tV4zuCU5iY6gwRyMe4s+xJ5VMsuZJGVLQ8/pxsv kDJiXDBQ9fuuw+RAL26ct45ni97G+x4CXS3MhVoaWecAZg3HUG3rroU9k12c1ECs6xO6 fdVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:message-id:subject:cc :to:from:date:dkim-signature; bh=AAWYqQo49rSUP/liphSBHP0is1wvO5riamGNWMX1lTw=; fh=mM7j0AJ/S24jynU3+f8ZuQxBzhzn0ovRqHTZp+moQSw=; b=SH7lwhFus9dRQtWTLINbZOrcBLiiMITOwWUEfreQizgZsjZn67Fj8Ny0mPanatH0SR Yfg5xVgQ84zDAv7bp2R9tZw1nfsqTm7V6jgyI+ncen15XOMpJrX8eBxyMTEb0+Fkvoad F59a+457WmHq/M7JcBL5RArzOWVaTSKvXYtSS0CdrboIAc+PgUk+gNAGZHTMfSqR7aPG SUNGFZF+FpKTDfcIHpM8hdy7Hi/QaBJZJt79i8voLb+48AidoTLzcW0vv/Dq4DkscfAs dNmjdWs6NmGO9W89ZmxGutiHaC5fmRZwwSP6y5rd5KCj9dGgpegTDsxQhrs6NLEASd6y 1qYA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Bn0OsbXO; spf=pass (google.com: domain of linux-kernel+bounces-23261-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-23261-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id l7-20020a635707000000b005cdf55cb5b5si623375pgb.840.2024.01.11.00.59.08 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jan 2024 00:59:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-23261-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Bn0OsbXO; spf=pass (google.com: domain of linux-kernel+bounces-23261-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-23261-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id EAE56B234D1 for ; Thu, 11 Jan 2024 08:59:01 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4CA6B15AC3; Thu, 11 Jan 2024 08:56:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Bn0OsbXO" Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA456154BF; Thu, 11 Jan 2024 08:56:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1704963387; x=1736499387; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=/z2uYKhSyYcTU8/yd1Vq7zQ21LOb4cUygjDFi2E+u98=; b=Bn0OsbXOuHcmjyxhW4ej8etCRCRyt9chbCRy9agDpxsKaXyftP2GRFnW lPjPn00CMKTKqWh6RN4o2ChGYm6nAKQmug8E/qHeeEWqIWT1mh9YffXgd RuoYPSggzSTg1Yu7CVNlqIG8uGMCnE7sZlJttoG1nhj2XPeJwyOTzqxB0 Haa+3PvmXZRlvJL986Hv6OgqBa+KMSzy+emyJhjt3zKLHtfYsh3aKA8ks GFEG9M/MgMTWUA1jY1uFjT/QSczUVEcomPqtP17LDbAcXQhlfsp84zGtt VxP+pji+CbDsR7+GkfzNuDxTk3O7tk9TlKQa2eGDjTFUU49cn6+gFIJkF g==; X-IronPort-AV: E=McAfee;i="6600,9927,10949"; a="12141738" X-IronPort-AV: E=Sophos;i="6.04,185,1695711600"; d="scan'208";a="12141738" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2024 00:56:26 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.04,185,1695711600"; d="scan'208";a="30929545" Received: from ericwong-mobl2.amr.corp.intel.com (HELO desk) ([10.209.43.169]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2024 00:56:24 -0800 Date: Thu, 11 Jan 2024 00:56:24 -0800 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta , Alyssa Milburn Subject: [PATCH v5 1/6] x86/bugs: Add asm helpers for executing VERW Message-ID: <20240111-delay-verw-v5-1-a3b234933ea6@linux.intel.com> X-Mailer: b4 0.12.3 References: <20240111-delay-verw-v5-0-a3b234933ea6@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240111-delay-verw-v5-0-a3b234933ea6@linux.intel.com> X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787783858130561200 X-GMAIL-MSGID: 1787783858130561200 MDS mitigation requires clearing the CPU buffers before returning to user. This needs to be done late in the exit-to-user path. Current location of VERW leaves a possibility of kernel data ending up in CPU buffers for memory accesses done after VERW such as: 1. Kernel data accessed by an NMI between VERW and return-to-user can remain in CPU buffers since NMI returning to kernel does not execute VERW to clear CPU buffers. 2. Alyssa reported that after VERW is executed, CONFIG_GCC_PLUGIN_STACKLEAK=y scrubs the stack used by a system call. Memory accesses during stack scrubbing can move kernel stack contents into CPU buffers. 3. When caller saved registers are restored after a return from function executing VERW, the kernel stack accesses can remain in CPU buffers(since they occur after VERW). To fix this VERW needs to be moved very late in exit-to-user path. In preparation for moving VERW to entry/exit asm code, create macros that can be used in asm. Also make VERW patching depend on a new feature flag X86_FEATURE_CLEAR_CPU_BUF. Reported-by: Alyssa Milburn Suggested-by: Andrew Cooper Suggested-by: Peter Zijlstra Signed-off-by: Pawan Gupta --- arch/x86/entry/entry.S | 22 ++++++++++++++++++++++ arch/x86/include/asm/cpufeatures.h | 2 +- arch/x86/include/asm/nospec-branch.h | 15 +++++++++++++++ 3 files changed, 38 insertions(+), 1 deletion(-) diff --git a/arch/x86/entry/entry.S b/arch/x86/entry/entry.S index 8c8d38f0cb1d..bd8e77c5a375 100644 --- a/arch/x86/entry/entry.S +++ b/arch/x86/entry/entry.S @@ -6,6 +6,9 @@ #include #include #include +#include +#include +#include .pushsection .noinstr.text, "ax" @@ -20,3 +23,22 @@ SYM_FUNC_END(entry_ibpb) EXPORT_SYMBOL_GPL(entry_ibpb); .popsection + +/* + * Defines the VERW operand that is disguised as entry code so that + * it can be referenced with KPTI enabled. This ensures VERW can be + * used late in exit-to-user path after page tables are switched. + */ +.pushsection .entry.text, "ax" + +.align L1_CACHE_BYTES, 0xcc +SYM_CODE_START_NOALIGN(mds_verw_sel) + UNWIND_HINT_UNDEFINED + ANNOTATE_NOENDBR + .word __KERNEL_DS +.align L1_CACHE_BYTES, 0xcc +SYM_CODE_END(mds_verw_sel); +/* For KVM */ +EXPORT_SYMBOL_GPL(mds_verw_sel); + +.popsection diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 4af140cf5719..79a7e81b9458 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -308,10 +308,10 @@ #define X86_FEATURE_SMBA (11*32+21) /* "" Slow Memory Bandwidth Allocation */ #define X86_FEATURE_BMEC (11*32+22) /* "" Bandwidth Monitoring Event Configuration */ #define X86_FEATURE_USER_SHSTK (11*32+23) /* Shadow stack support for user mode applications */ - #define X86_FEATURE_SRSO (11*32+24) /* "" AMD BTB untrain RETs */ #define X86_FEATURE_SRSO_ALIAS (11*32+25) /* "" AMD BTB untrain RETs through aliasing */ #define X86_FEATURE_IBPB_ON_VMEXIT (11*32+26) /* "" Issue an IBPB only on VMEXIT */ +#define X86_FEATURE_CLEAR_CPU_BUF (11*32+27) /* "" Clear CPU buffers using VERW */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */ diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index f93e9b96927a..4ea4c310db52 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -315,6 +315,21 @@ #endif .endm +/* + * Macros to execute VERW instruction that mitigate transient data sampling + * attacks such as MDS. On affected systems a microcode update overloaded VERW + * instruction to also clear the CPU buffers. VERW clobbers CFLAGS.ZF. + * + * Note: Only the memory operand variant of VERW clears the CPU buffers. + */ +.macro EXEC_VERW + verw _ASM_RIP(mds_verw_sel) +.endm + +.macro CLEAR_CPU_BUFFERS + ALTERNATIVE "", __stringify(EXEC_VERW), X86_FEATURE_CLEAR_CPU_BUF +.endm + #else /* __ASSEMBLY__ */ #define ANNOTATE_RETPOLINE_SAFE \ From patchwork Thu Jan 11 08:56:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 187249 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2411:b0:101:2151:f287 with SMTP id m17csp1314894dyi; Thu, 11 Jan 2024 00:59:29 -0800 (PST) X-Google-Smtp-Source: AGHT+IFwJxZSk92xTa/Qwh95JBihrUH/ABXT/Y1hb4OK771DSHT9HDGUImNQtY54B73IJqMN2YBG X-Received: by 2002:a17:90a:a118:b0:28c:5a10:f31d with SMTP id s24-20020a17090aa11800b0028c5a10f31dmr755359pjp.45.1704963569103; Thu, 11 Jan 2024 00:59:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704963569; cv=none; d=google.com; s=arc-20160816; b=EmlE4DBvWrBR0VjE0DNePDhRYPxQ4r66MZkG8dhZhzK9tVUdbIVXmCY4KjPXW0X+lo 6JAwE84Fvsc6PbMXxzJbAjmHoMl5Ex+i9sQ+I2GMY8g1ix4+BIV2/dyl3kqc2SDYYAkJ iJK4Xpkafk7bX/AP5Kaa87n9oCEVgRBtKSYRyvRLe6UlsMIDVODhJUxEnOGot5bcDpda WkS3P7OysJtDXJqfiu6Dm5W6W6+1AlcVfl2/380NG7wQI7lNKLXlyfyUTrN66FG2Uf/i 8fk9qByYE8LQ+bfRICrhGH3Qprk+VPqqKSd0ar/c7f52qXaIvoYA7uJE/145ctFEnA6f FKZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:message-id:subject:cc :to:from:date:dkim-signature; bh=khozrtzAj3vwjyUrXpWv9rTDxIFxnIQL0DQ40domI8A=; fh=9zb6IElBBOwPDhWsAkR8DeNi2jqlz/RyBa2WbyJnsJM=; b=zc464h0Z+6FdX31rUEeCJeC9JOZABznP6G3t8QXMztNqiOhqKbRc28tGgFeX86tacm uK/f49Xtm0haOQoMlaB5j0NPeVNjwo0Jg0rRkWuyvkuzidb/98oYwCcwn2XtDIOfPo6N h9RUWAIi+f9yOcYkZ0qrPYLkDq3QN2bMHa9Y/IkO/g9Xz4LKB9OqkgTVJGqaRrnywI2o AYokCsCBBQHfcnZeeUPLh6gGQ5s0FMj3euIM8ngvW8l+MmvP4eAjOMzhWrUMxT/KcRHg rTI437iA8Xmazb0/7lM/jWfr8wBmWsiaej+OiagVn51OAz+rEAyENM04o5Up9X7a2A2E Tnyw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=R5pjxTc5; spf=pass (google.com: domain of linux-kernel+bounces-23263-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-23263-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id pi14-20020a17090b1e4e00b0028dd2f9fea4si2291310pjb.80.2024.01.11.00.59.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jan 2024 00:59:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-23263-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=R5pjxTc5; spf=pass (google.com: domain of linux-kernel+bounces-23263-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-23263-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 7C703287C44 for ; Thu, 11 Jan 2024 08:59:28 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A5AD215E90; Thu, 11 Jan 2024 08:56:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="R5pjxTc5" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8248B15AD6; Thu, 11 Jan 2024 08:56:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1704963393; x=1736499393; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=u0ACNJnmCWkNPcbKiCWkDdCakrm+/qCMT4yvny0P32Q=; b=R5pjxTc5pSokC0/W59iU6/CbcmCpGUPd9Ao+MJsxTbypYBQOIIVSDGAy Q72i4lW++SjqTu091No022FCNueNC/c1TPnUYYiqV81+yrzyp4OtFav2+ 2BOr/FpX6iCmdUtDxMuKXvOd7HV4dUtpcLL/eb+Y7vt3iLU8s43p/UeJ5 /3jC+PIZ3TS3692OoOTOiGF236R3qDTcBFKrhF77Z+LAlkoG/BI8Kgvcf N16m/wVEzmP1zpFDr2ynClfYyogxe3hvHyJ6zGkUwV203IUoyohJMW90B I/sUJCV5lW9Y19VLjWX5TnvzWdpH40AzrXIC2zz3DEgGI8VoRyMEe5hB7 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10949"; a="6144907" X-IronPort-AV: E=Sophos;i="6.04,185,1695711600"; d="scan'208";a="6144907" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2024 00:56:32 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10949"; a="905877631" X-IronPort-AV: E=Sophos;i="6.04,185,1695711600"; d="scan'208";a="905877631" Received: from ericwong-mobl2.amr.corp.intel.com (HELO desk) ([10.209.43.169]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2024 00:56:30 -0800 Date: Thu, 11 Jan 2024 00:56:30 -0800 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta , Dave Hansen Subject: [PATCH v5 2/6] x86/entry_64: Add VERW just before userspace transition Message-ID: <20240111-delay-verw-v5-2-a3b234933ea6@linux.intel.com> X-Mailer: b4 0.12.3 References: <20240111-delay-verw-v5-0-a3b234933ea6@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240111-delay-verw-v5-0-a3b234933ea6@linux.intel.com> X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787783879151932541 X-GMAIL-MSGID: 1787783879151932541 Mitigation for MDS is to use VERW instruction to clear any secrets in CPU Buffers. Any memory accesses after VERW execution can still remain in CPU buffers. It is safer to execute VERW late in return to user path to minimize the window in which kernel data can end up in CPU buffers. There are not many kernel secrets to be had after SWITCH_TO_USER_CR3. Add support for deploying VERW mitigation after user register state is restored. This helps minimize the chances of kernel data ending up into CPU buffers after executing VERW. Note that the mitigation at the new location is not yet enabled. Corner case not handled ======================= Interrupts returning to kernel don't clear CPUs buffers since the exit-to-user path is expected to do that anyways. But, there could be a case when an NMI is generated in kernel after the exit-to-user path has cleared the buffers. This case is not handled and NMI returning to kernel don't clear CPU buffers because: 1. It is rare to get an NMI after VERW, but before returning to userspace. 2. For an unprivileged user, there is no known way to make that NMI less rare or target it. 3. It would take a large number of these precisely-timed NMIs to mount an actual attack. There's presumably not enough bandwidth. 4. The NMI in question occurs after a VERW, i.e. when user state is restored and most interesting data is already scrubbed. Whats left is only the data that NMI touches, and that may or may not be of any interest. Suggested-by: Dave Hansen Signed-off-by: Pawan Gupta --- arch/x86/entry/entry_64.S | 11 +++++++++++ arch/x86/entry/entry_64_compat.S | 1 + 2 files changed, 12 insertions(+) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index de6469dffe3a..bdb17fad5d04 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -161,6 +161,7 @@ syscall_return_via_sysret: SYM_INNER_LABEL(entry_SYSRETQ_unsafe_stack, SYM_L_GLOBAL) ANNOTATE_NOENDBR swapgs + CLEAR_CPU_BUFFERS sysretq SYM_INNER_LABEL(entry_SYSRETQ_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR @@ -601,6 +602,7 @@ SYM_INNER_LABEL(swapgs_restore_regs_and_return_to_usermode, SYM_L_GLOBAL) /* Restore RDI. */ popq %rdi swapgs + CLEAR_CPU_BUFFERS jmp .Lnative_iret @@ -712,6 +714,8 @@ native_irq_return_ldt: */ popq %rax /* Restore user RAX */ + CLEAR_CPU_BUFFERS + /* * RSP now points to an ordinary IRET frame, except that the page * is read-only and RSP[31:16] are preloaded with the userspace @@ -1438,6 +1442,12 @@ nmi_restore: std movq $0, 5*8(%rsp) /* clear "NMI executing" */ + /* + * Skip CLEAR_CPU_BUFFERS here, since it only helps in rare cases like + * NMI in kernel after user state is restored. For an unprivileged user + * these conditions are hard to meet. + */ + /* * iretq reads the "iret" frame and exits the NMI stack in a * single instruction. We are returning to kernel mode, so this @@ -1455,6 +1465,7 @@ SYM_CODE_START(entry_SYSCALL32_ignore) UNWIND_HINT_END_OF_STACK ENDBR mov $-ENOSYS, %eax + CLEAR_CPU_BUFFERS sysretl SYM_CODE_END(entry_SYSCALL32_ignore) diff --git a/arch/x86/entry/entry_64_compat.S b/arch/x86/entry/entry_64_compat.S index de94e2e84ecc..eabf48c4d4b4 100644 --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -270,6 +270,7 @@ SYM_INNER_LABEL(entry_SYSRETL_compat_unsafe_stack, SYM_L_GLOBAL) xorl %r9d, %r9d xorl %r10d, %r10d swapgs + CLEAR_CPU_BUFFERS sysretl SYM_INNER_LABEL(entry_SYSRETL_compat_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR From patchwork Thu Jan 11 08:56:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 187250 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2411:b0:101:2151:f287 with SMTP id m17csp1315011dyi; Thu, 11 Jan 2024 00:59:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IEJjleol+stbi9+wx1Bu/FznSWAjb21kkmx6P8sYPp67DpS0DusKkx5ZnwpUTIU7KTWy7T8 X-Received: by 2002:a17:90b:1014:b0:28c:16bb:712d with SMTP id gm20-20020a17090b101400b0028c16bb712dmr513266pjb.48.1704963588097; Thu, 11 Jan 2024 00:59:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704963588; cv=none; d=google.com; s=arc-20160816; b=Npa1YYKF2eBkk2fGeTuP5ul257rIpub9W/5zExW6L8htFXGVFvES8fQ2xTZtxnv+Y3 pReCgSb9/S2jzij8+dasB3aLtP+y1bkPuKQr4IuRLpkz0oHC7b58YFQX0z8/13My1rW4 3TzK2EPzMmbzzZVC2tW7vzBgmMJ2ZQ9v5/W1RKhICypMu9Xr9HOdDAsqaCIVzgCALTia k0UQQQA7EVvH8uC8SrHpQvpg9dN997PxIArCnEa2j4CyCk4JnEOBBC+jDTbZwOJkSzk1 SbGG8ETgs1rUxYIFdUkICtTVDRzsDlQMJYGTGTRCwby1aD6OxOfmwyHm8X6rqtlM5zpp EDOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:message-id:subject:cc :to:from:date:dkim-signature; bh=rzZbi2kNM6RYCVjIUYiVz1/256dCuMd9sPMX2hlHtQM=; fh=ER2H6g+88FnZd3JyhT6ewHH56pdK2QWfWr77z0IHoNE=; b=XZwWgH3NiqmxfJX+vOZkIe2sa/xD5oEW3a/wVNF4nHH1Eb8m13Zb6sipNlQcb36dMJ NsyfE/wa1VwS0JCNk7CwjkXgaDJFtph/g7rQ3UIQx742FQL76m7QqskAPoX40G93gWKg yWUMU7WE1KgS3MmE6qect1FaZjlHR023TSlP5hEkRpPqOyqQ1YeNoROHIwU+WDNYzWWd nmJtL3bILMSUdMvMypjpsRO2suELPZNguZl9RlR/ZYkGAMlrZbaRXk+yiAuCVL73oln8 CX8VX4Y9kuxUS0AkalPYh/QuewYBCUcv9MZrNN0au1twHPiIDXNudxtQiHsNLS0E8691 SgSg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=DP8agTYy; spf=pass (google.com: domain of linux-kernel+bounces-23264-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-23264-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id mm4-20020a17090b358400b0028bfce45881si3230130pjb.101.2024.01.11.00.59.47 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jan 2024 00:59:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-23264-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=DP8agTYy; spf=pass (google.com: domain of linux-kernel+bounces-23264-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-23264-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 28BF0287BB6 for ; Thu, 11 Jan 2024 08:59:47 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 3AF921641A; Thu, 11 Jan 2024 08:56:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="DP8agTYy" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3E2BD15E9B; Thu, 11 Jan 2024 08:56:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1704963398; x=1736499398; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=SvyQgb7E1GjbbhEZauDFduuMMCCNLr60ulcJzDeIqyo=; b=DP8agTYy0b5+FhTki9djPAUS+/vamp2KB8DIxtDRqMpktfXwTKgzbxeh ol2Q9CHdXn8+W9rnIfU7+0Hm8WxgpPhjG3FgbqudzDGO5m1gTJX8Nvlfd S3TBATBzVImTM9KmdEE4TABSLI3bul4DoE5G560+sX+yNAMs96mRsI2TV /iiW8QiV7V8z/g+8+Bzj94m8nzEO7xEoOck3flZWJ3xuHr1I1m6K+XmBc cx5Jv10HfQVrhwrGIn6mJgKtneMC/P511tyj/rCYlyCEY/mrchADfh18o 1nq6Nd7GHb1NLsygLgLOJUPq6uoLRlrNsgkmki0/STpT6gcteg6UPeAPk w==; X-IronPort-AV: E=McAfee;i="6600,9927,10949"; a="6144925" X-IronPort-AV: E=Sophos;i="6.04,185,1695711600"; d="scan'208";a="6144925" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2024 00:56:38 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10949"; a="905877668" X-IronPort-AV: E=Sophos;i="6.04,185,1695711600"; d="scan'208";a="905877668" Received: from ericwong-mobl2.amr.corp.intel.com (HELO desk) ([10.209.43.169]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2024 00:56:36 -0800 Date: Thu, 11 Jan 2024 00:56:36 -0800 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta Subject: [PATCH v5 3/6] x86/entry_32: Add VERW just before userspace transition Message-ID: <20240111-delay-verw-v5-3-a3b234933ea6@linux.intel.com> X-Mailer: b4 0.12.3 References: <20240111-delay-verw-v5-0-a3b234933ea6@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240111-delay-verw-v5-0-a3b234933ea6@linux.intel.com> X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787783898999136821 X-GMAIL-MSGID: 1787783898999136821 As done for entry_64, add support for executing VERW late in exit to user path for 32-bit mode. Signed-off-by: Pawan Gupta --- arch/x86/entry/entry_32.S | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S index c73047bf9f4b..fba427646805 100644 --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -885,6 +885,7 @@ SYM_FUNC_START(entry_SYSENTER_32) BUG_IF_WRONG_CR3 no_user_check=1 popfl popl %eax + CLEAR_CPU_BUFFERS /* * Return back to the vDSO, which will pop ecx and edx. @@ -954,6 +955,7 @@ restore_all_switch_stack: /* Restore user state */ RESTORE_REGS pop=4 # skip orig_eax/error_code + CLEAR_CPU_BUFFERS .Lirq_return: /* * ARCH_HAS_MEMBARRIER_SYNC_CORE rely on IRET core serialization @@ -1146,6 +1148,7 @@ SYM_CODE_START(asm_exc_nmi) /* Not on SYSENTER stack. */ call exc_nmi + CLEAR_CPU_BUFFERS jmp .Lnmi_return .Lnmi_from_sysenter_stack: From patchwork Thu Jan 11 08:56:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 187251 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2411:b0:101:2151:f287 with SMTP id m17csp1315204dyi; Thu, 11 Jan 2024 01:00:09 -0800 (PST) X-Google-Smtp-Source: AGHT+IGNSxUvQ+ZVLZg2dG5WCeQJBf7KjncC9Ne1tJ7uzHl/yBGCwcNGp6AQHgQfYtZBC8syjDXN X-Received: by 2002:a05:6402:326:b0:557:7f96:1d12 with SMTP id q6-20020a056402032600b005577f961d12mr348172edw.44.1704963609393; Thu, 11 Jan 2024 01:00:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704963609; cv=none; d=google.com; s=arc-20160816; b=TIPk4UZ2ot5g8DmiX3de1IXEE8e9x+LkA8yxvEGBIFPjDev7tjT5mp78p8H3cQ24zH 4vBmSHcnWWA+TSkV1bl5o1qng6zJtxuKorZ6B7UJewvuiibQglj98fX1IRK3E29shAK/ /DvVGVKedsjDpgb4zXSRpYsprzmrznDqnxhsI8oB7paYutjtzdpGDOIGL6BpYDIOOCAO TZSR/NMLbSjyPAr8xRla6w2F2T+lG28H9hEwyz3qxSseyqw9t7Qe9mSGqv6O1Mn2T2GU HJPQhWjvK4CvNwLZXqdWmJNwcVx1BUH9RtGTerd8Cs3d48jNh9c16t3V/LG3gXa1wESj 8rjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:message-id:subject:cc :to:from:date:dkim-signature; bh=VNeP+Gd3Qzgj2+RZtpjYxL5WhsUJTdm0Wic9ltoyClk=; fh=ER2H6g+88FnZd3JyhT6ewHH56pdK2QWfWr77z0IHoNE=; b=rdFHjg9Q+T3n9i3oK5hv6YF6SNJGiaG5sab0+gNl8T1XX5HEikKFCQ7d3epletkkY1 2mqk6kXXXbEGq1rdV8Tvyw0EU4dJ1uO81rDM2Sgj+VE1dTiOw8GZTIg1UbFQUKmI+3BT 045WsLDwEjdn5k9WE4SowL4KVeuNMfdlQQ7UQI96aH8bPosqdsO5khFU49MbCdcTWPbL QCKs/M5Gg85srPML14yRDpXcIuZMqqOf5GNgOB00HTIWQXad6J3iy6gNcEEHJpKWlXm0 3BFNNcgsOD5eqctMEZV7frHqmRuU3Xk/NElB/+yclsIyos5xwIMY+1aW4X8/fX3hWrw3 U3GA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=URRf9CyH; spf=pass (google.com: domain of linux-kernel+bounces-23265-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-23265-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id d9-20020a05640208c900b00543670dada0si340335edz.217.2024.01.11.01.00.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jan 2024 01:00:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-23265-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=URRf9CyH; spf=pass (google.com: domain of linux-kernel+bounces-23265-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-23265-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id D180C1F24F44 for ; Thu, 11 Jan 2024 09:00:08 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id AC0C8168AD; Thu, 11 Jan 2024 08:56:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="URRf9CyH" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 82AB816413; Thu, 11 Jan 2024 08:56:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1704963405; x=1736499405; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=avBVslkO75CqSDbvGNj2leB8duki0KQBBwT6ocfVkB8=; b=URRf9CyH4TO637Dy7NLZ0Lpuf76EdXCtsbXVvabZLoY9eTbMolVo6esl xwrmEC3Ac2fQLLkh5IaauLnYs7/DvxcK+G2KArzM/4TOQGWiIB9zsj2QA LwmKbu1DewuIatlIyxGDuBUws854jdXxzuLGnNZ12SxjY/FGj3h+s8oQR q/XXVzPcQppo6Nqb63fqyvuTfNCPa2xdyyOg9qfmRDyiF7O+FKlDStWLH nw0R+utlxstxOXCv3I6ON9gUJiFUaRVmcB+1pyTWPDPLMy7k1mcs02xv0 ySOakoC2U7NCqQQM+Vk6iCtdlEbqapsveKXQpQijRcUKIdTcTTGiYBsqA g==; X-IronPort-AV: E=McAfee;i="6600,9927,10949"; a="6144940" X-IronPort-AV: E=Sophos;i="6.04,185,1695711600"; d="scan'208";a="6144940" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2024 00:56:44 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10949"; a="905877743" X-IronPort-AV: E=Sophos;i="6.04,185,1695711600"; d="scan'208";a="905877743" Received: from ericwong-mobl2.amr.corp.intel.com (HELO desk) ([10.209.43.169]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2024 00:56:43 -0800 Date: Thu, 11 Jan 2024 00:56:43 -0800 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta Subject: [PATCH v5 4/6] x86/bugs: Use ALTERNATIVE() instead of mds_user_clear static key Message-ID: <20240111-delay-verw-v5-4-a3b234933ea6@linux.intel.com> X-Mailer: b4 0.12.3 References: <20240111-delay-verw-v5-0-a3b234933ea6@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240111-delay-verw-v5-0-a3b234933ea6@linux.intel.com> X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787783921635433706 X-GMAIL-MSGID: 1787783921635433706 The VERW mitigation at exit-to-user is enabled via a static branch mds_user_clear. This static branch is never toggled after boot, and can be safely replaced with an ALTERNATIVE() which is convenient to use in asm. Switch to ALTERNATIVE() to use the VERW mitigation late in exit-to-user path. Also remove the now redundant VERW in exc_nmi() and arch_exit_to_user_mode(). Signed-off-by: Pawan Gupta --- Documentation/arch/x86/mds.rst | 38 +++++++++++++++++++++++++----------- arch/x86/include/asm/entry-common.h | 1 - arch/x86/include/asm/nospec-branch.h | 12 ------------ arch/x86/kernel/cpu/bugs.c | 15 ++++++-------- arch/x86/kernel/nmi.c | 3 --- arch/x86/kvm/vmx/vmx.c | 2 +- 6 files changed, 34 insertions(+), 37 deletions(-) diff --git a/Documentation/arch/x86/mds.rst b/Documentation/arch/x86/mds.rst index e73fdff62c0a..c58c72362911 100644 --- a/Documentation/arch/x86/mds.rst +++ b/Documentation/arch/x86/mds.rst @@ -95,6 +95,9 @@ The kernel provides a function to invoke the buffer clearing: mds_clear_cpu_buffers() +Also macro CLEAR_CPU_BUFFERS can be used in ASM late in exit-to-user path. +Other than CFLAGS.ZF, this macro doesn't clobber any registers. + The mitigation is invoked on kernel/userspace, hypervisor/guest and C-state (idle) transitions. @@ -138,17 +141,30 @@ Mitigation points When transitioning from kernel to user space the CPU buffers are flushed on affected CPUs when the mitigation is not disabled on the kernel - command line. The migitation is enabled through the static key - mds_user_clear. - - The mitigation is invoked in prepare_exit_to_usermode() which covers - all but one of the kernel to user space transitions. The exception - is when we return from a Non Maskable Interrupt (NMI), which is - handled directly in do_nmi(). - - (The reason that NMI is special is that prepare_exit_to_usermode() can - enable IRQs. In NMI context, NMIs are blocked, and we don't want to - enable IRQs with NMIs blocked.) + command line. The mitigation is enabled through the feature flag + X86_FEATURE_CLEAR_CPU_BUF. + + The mitigation is invoked just before transitioning to userspace after + user registers are restored. This is done to minimize the window in + which kernel data could be accessed after VERW e.g. via an NMI after + VERW. + + **Corner case not handled** + Interrupts returning to kernel don't clear CPUs buffers since the + exit-to-user path is expected to do that anyways. But, there could be + a case when an NMI is generated in kernel after the exit-to-user path + has cleared the buffers. This case is not handled and NMI returning to + kernel don't clear CPU buffers because: + + 1. It is rare to get an NMI after VERW, but before returning to userspace. + 2. For an unprivileged user, there is no known way to make that NMI + less rare or target it. + 3. It would take a large number of these precisely-timed NMIs to mount + an actual attack. There's presumably not enough bandwidth. + 4. The NMI in question occurs after a VERW, i.e. when user state is + restored and most interesting data is already scrubbed. Whats left + is only the data that NMI touches, and that may or may not be of + any interest. 2. C-State transition diff --git a/arch/x86/include/asm/entry-common.h b/arch/x86/include/asm/entry-common.h index ce8f50192ae3..7e523bb3d2d3 100644 --- a/arch/x86/include/asm/entry-common.h +++ b/arch/x86/include/asm/entry-common.h @@ -91,7 +91,6 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, static __always_inline void arch_exit_to_user_mode(void) { - mds_user_clear_cpu_buffers(); amd_clear_divider(); } #define arch_exit_to_user_mode arch_exit_to_user_mode diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index 4ea4c310db52..0a8fa023a804 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -544,7 +544,6 @@ DECLARE_STATIC_KEY_FALSE(switch_to_cond_stibp); DECLARE_STATIC_KEY_FALSE(switch_mm_cond_ibpb); DECLARE_STATIC_KEY_FALSE(switch_mm_always_ibpb); -DECLARE_STATIC_KEY_FALSE(mds_user_clear); DECLARE_STATIC_KEY_FALSE(mds_idle_clear); DECLARE_STATIC_KEY_FALSE(switch_mm_cond_l1d_flush); @@ -576,17 +575,6 @@ static __always_inline void mds_clear_cpu_buffers(void) asm volatile("verw %[ds]" : : [ds] "m" (ds) : "cc"); } -/** - * mds_user_clear_cpu_buffers - Mitigation for MDS and TAA vulnerability - * - * Clear CPU buffers if the corresponding static key is enabled - */ -static __always_inline void mds_user_clear_cpu_buffers(void) -{ - if (static_branch_likely(&mds_user_clear)) - mds_clear_cpu_buffers(); -} - /** * mds_idle_clear_cpu_buffers - Mitigation for MDS vulnerability * diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index bb0ab8466b91..48d049cd74e7 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -111,9 +111,6 @@ DEFINE_STATIC_KEY_FALSE(switch_mm_cond_ibpb); /* Control unconditional IBPB in switch_mm() */ DEFINE_STATIC_KEY_FALSE(switch_mm_always_ibpb); -/* Control MDS CPU buffer clear before returning to user space */ -DEFINE_STATIC_KEY_FALSE(mds_user_clear); -EXPORT_SYMBOL_GPL(mds_user_clear); /* Control MDS CPU buffer clear before idling (halt, mwait) */ DEFINE_STATIC_KEY_FALSE(mds_idle_clear); EXPORT_SYMBOL_GPL(mds_idle_clear); @@ -252,7 +249,7 @@ static void __init mds_select_mitigation(void) if (!boot_cpu_has(X86_FEATURE_MD_CLEAR)) mds_mitigation = MDS_MITIGATION_VMWERV; - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); if (!boot_cpu_has(X86_BUG_MSBDS_ONLY) && (mds_nosmt || cpu_mitigations_auto_nosmt())) @@ -356,7 +353,7 @@ static void __init taa_select_mitigation(void) * For guests that can't determine whether the correct microcode is * present on host, enable the mitigation for UCODE_NEEDED as well. */ - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); if (taa_nosmt || cpu_mitigations_auto_nosmt()) cpu_smt_disable(false); @@ -424,7 +421,7 @@ static void __init mmio_select_mitigation(void) */ if (boot_cpu_has_bug(X86_BUG_MDS) || (boot_cpu_has_bug(X86_BUG_TAA) && boot_cpu_has(X86_FEATURE_RTM))) - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_CLEAR_CPU_BUF); else static_branch_enable(&mmio_stale_data_clear); @@ -484,12 +481,12 @@ static void __init md_clear_update_mitigation(void) if (cpu_mitigations_off()) return; - if (!static_key_enabled(&mds_user_clear)) + if (!boot_cpu_has(X86_FEATURE_CLEAR_CPU_BUF)) goto out; /* - * mds_user_clear is now enabled. Update MDS, TAA and MMIO Stale Data - * mitigation, if necessary. + * X86_FEATURE_CLEAR_CPU_BUF is now enabled. Update MDS, TAA and MMIO + * Stale Data mitigation, if necessary. */ if (mds_mitigation == MDS_MITIGATION_OFF && boot_cpu_has_bug(X86_BUG_MDS)) { diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c index 17e955ab69fe..3082cf24b69e 100644 --- a/arch/x86/kernel/nmi.c +++ b/arch/x86/kernel/nmi.c @@ -563,9 +563,6 @@ DEFINE_IDTENTRY_RAW(exc_nmi) } if (this_cpu_dec_return(nmi_state)) goto nmi_restart; - - if (user_mode(regs)) - mds_user_clear_cpu_buffers(); } #if IS_ENABLED(CONFIG_KVM_INTEL) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index be20a60047b1..bdcf2c041e0c 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7229,7 +7229,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, /* L1D Flush includes CPU buffer clear to mitigate MDS */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); - else if (static_branch_unlikely(&mds_user_clear)) + else if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF)) mds_clear_cpu_buffers(); else if (static_branch_unlikely(&mmio_stale_data_clear) && kvm_arch_has_assigned_device(vcpu->kvm)) From patchwork Thu Jan 11 08:56:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 187252 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2411:b0:101:2151:f287 with SMTP id m17csp1315409dyi; Thu, 11 Jan 2024 01:00:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IGzvyhgWBMI+EWI4b1WXu8p+aI9sv743llniASqr6QMoykPxndx7/5FGQkNX1st/4rBZi02 X-Received: by 2002:a2e:6e02:0:b0:2cc:d030:1557 with SMTP id j2-20020a2e6e02000000b002ccd0301557mr211673ljc.107.1704963628693; Thu, 11 Jan 2024 01:00:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704963628; cv=none; d=google.com; s=arc-20160816; b=irzDWJ57zbDoqjI5AiYkrUzBled8RYMsbPAFCpn4eigQ0voVgfOVTGPQX48rCJozJM 3SMTf4oiQerJX8l5sa6Lglq55g9f6BB6JakpdAJZbpLX1/qVXux1PcTUln9L9MpU4O+G LftWmVjJyVrdibUdMhfkXikBrmOv3auCnsbClNv+KB6m7H/w2tJt65cx4c8ggCxeqYKe Hby05OM4sz/Cq0cTXvKa8N2EwP3kMblIX446R9kilZN62O3cwyaZrz+gbR+3b2GEzGDf sUOCUtIZWE8sbeJRYCdy0ulYneO/v5fEKGretyzQ3qFfidBuxnQ411C5PjMFHRCnesOJ jBnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:message-id:subject:cc :to:from:date:dkim-signature; bh=ilqGXAQjD0Zw0p+0QS0PhsqpTUKzsxY97u90wwaRZ8Y=; fh=ER2H6g+88FnZd3JyhT6ewHH56pdK2QWfWr77z0IHoNE=; b=WcDHxcDMOPcQL1K/b5cYmAHbDL2Ctb0VBVbFhqRN/kHGrrZ5/2qR6W2TXGMql0elw7 9iWToZojQEcPVEQRbcZks/2XLVsO+iwFPTc9owwbEWmC9ToeSAkGxM8c7RpJPTENwNBm a8ESLSfy0/vaEn2Cib7xWfpRnAtS8nU/ebuz2t7ZwqXq+NC0O2Ee8zIxoENaTYBRrK22 Ev7nfzJIq6VISKEyVztGN03nU7bSMzShV7Z5piP2RNnTKYIan6Zc6KRZwtKDSxyGJoSW Ju6AeiXwXRSTxSnCzsRFDirxXbgWhFRK8URcIdZKetDgQXUli+XVJMvDGiO1E8Wt4vnq h+0A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=kr8xUQ7C; spf=pass (google.com: domain of linux-kernel+bounces-23266-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-23266-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id i10-20020a056402054a00b0054c943a07bfsi331690edx.373.2024.01.11.01.00.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jan 2024 01:00:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-23266-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=kr8xUQ7C; spf=pass (google.com: domain of linux-kernel+bounces-23266-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-23266-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 4C3471F24EE7 for ; Thu, 11 Jan 2024 09:00:28 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C7B68168DE; Thu, 11 Jan 2024 08:56:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="kr8xUQ7C" Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C709E168B4; Thu, 11 Jan 2024 08:56:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1704963411; x=1736499411; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=rpFgire7Svqru/HfpQ1b16RkVx0842HOL7xbgXY2vaQ=; b=kr8xUQ7C6jzdJ1hOc7IPykyiApURBUHV2KNrxu0w/RE9rPE/lcC13EM1 /jEzRe/R+JpDOcvDyPeIJ/WRQSzAu/2+1+Hxcm5iDmpdHWPa/8Q0vqDJh mPZlW83XhSWpzRk8rD80wj9XFJ7kJwaIqOOR2tLMFEiqsA6uuOxcc+bwN KSPPFJMn6tnlBK9O6b1vvhf+hHDJNe2mHHDbKPacLSizygosoFurIKo8D Up3OTcX8fg3Zk5bWpPWg82s5FxrlKpyhiN4fiWLLVbrW53kjDQV4qwAKy alatnx4c+viQbi7LR3VfCj1+9NYmFWtgeko68e87B6JtaNNe42ggmkKg+ A==; X-IronPort-AV: E=McAfee;i="6600,9927,10949"; a="6144963" X-IronPort-AV: E=Sophos;i="6.04,185,1695711600"; d="scan'208";a="6144963" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmvoesa104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2024 00:56:50 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10949"; a="905877772" X-IronPort-AV: E=Sophos;i="6.04,185,1695711600"; d="scan'208";a="905877772" Received: from ericwong-mobl2.amr.corp.intel.com (HELO desk) ([10.209.43.169]) by orsmga004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2024 00:56:49 -0800 Date: Thu, 11 Jan 2024 00:56:49 -0800 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta Subject: [PATCH v5 5/6] KVM: VMX: Use BT+JNC, i.e. EFLAGS.CF to select VMRESUME vs. VMLAUNCH Message-ID: <20240111-delay-verw-v5-5-a3b234933ea6@linux.intel.com> X-Mailer: b4 0.12.3 References: <20240111-delay-verw-v5-0-a3b234933ea6@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240111-delay-verw-v5-0-a3b234933ea6@linux.intel.com> X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787783942057339972 X-GMAIL-MSGID: 1787783942057339972 From: Sean Christopherson Use EFLAGS.CF instead of EFLAGS.ZF to track whether to use VMRESUME versus VMLAUNCH. Freeing up EFLAGS.ZF will allow doing VERW, which clobbers ZF, for MDS mitigations as late as possible without needing to duplicate VERW for both paths. Reviewed-by: Nikolay Borisov Signed-off-by: Sean Christopherson Signed-off-by: Pawan Gupta --- arch/x86/kvm/vmx/run_flags.h | 7 +++++-- arch/x86/kvm/vmx/vmenter.S | 6 +++--- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/vmx/run_flags.h b/arch/x86/kvm/vmx/run_flags.h index edc3f16cc189..6a9bfdfbb6e5 100644 --- a/arch/x86/kvm/vmx/run_flags.h +++ b/arch/x86/kvm/vmx/run_flags.h @@ -2,7 +2,10 @@ #ifndef __KVM_X86_VMX_RUN_FLAGS_H #define __KVM_X86_VMX_RUN_FLAGS_H -#define VMX_RUN_VMRESUME (1 << 0) -#define VMX_RUN_SAVE_SPEC_CTRL (1 << 1) +#define VMX_RUN_VMRESUME_SHIFT 0 +#define VMX_RUN_SAVE_SPEC_CTRL_SHIFT 1 + +#define VMX_RUN_VMRESUME BIT(VMX_RUN_VMRESUME_SHIFT) +#define VMX_RUN_SAVE_SPEC_CTRL BIT(VMX_RUN_SAVE_SPEC_CTRL_SHIFT) #endif /* __KVM_X86_VMX_RUN_FLAGS_H */ diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index be275a0410a8..b3b13ec04bac 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -139,7 +139,7 @@ SYM_FUNC_START(__vmx_vcpu_run) mov (%_ASM_SP), %_ASM_AX /* Check if vmlaunch or vmresume is needed */ - test $VMX_RUN_VMRESUME, %ebx + bt $VMX_RUN_VMRESUME_SHIFT, %ebx /* Load guest registers. Don't clobber flags. */ mov VCPU_RCX(%_ASM_AX), %_ASM_CX @@ -161,8 +161,8 @@ SYM_FUNC_START(__vmx_vcpu_run) /* Load guest RAX. This kills the @regs pointer! */ mov VCPU_RAX(%_ASM_AX), %_ASM_AX - /* Check EFLAGS.ZF from 'test VMX_RUN_VMRESUME' above */ - jz .Lvmlaunch + /* Check EFLAGS.CF from the VMX_RUN_VMRESUME bit test above. */ + jnc .Lvmlaunch /* * After a successful VMRESUME/VMLAUNCH, control flow "magically" From patchwork Thu Jan 11 08:56:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 187254 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2411:b0:101:2151:f287 with SMTP id m17csp1316112dyi; Thu, 11 Jan 2024 01:01:34 -0800 (PST) X-Google-Smtp-Source: AGHT+IHAHhHYW7gm4oFWXG/N3zD7azNwQA9eYF9hxy738rcqMvoFPTY7nI9BnxlhVwU5XghDpdtn X-Received: by 2002:a05:6a20:271e:b0:19a:260b:ec26 with SMTP id u30-20020a056a20271e00b0019a260bec26mr581045pze.61.1704963694174; Thu, 11 Jan 2024 01:01:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704963694; cv=none; d=google.com; s=arc-20160816; b=PxBSuDvJw1kMqgCLKiI0eTRW/uln22Z+n3SOr+yGxPlC/FWMMNA/FInklyBnTYECab jWlQx+JVYvD7vNI0S78JtoQa5zAlL/nXxxlFxO9plmn6wveOXs14Y7YIsJy7zx0gS509 R5pPrJeL8LOaC37ukRPYeoZC+neowdxqnpgeoy6XClB2UizS6qYnnaAozm+2ftR39McN CGv+gubiVG9cGAuFyyvB6C5v9N+7Q4IhHWibkAhd0G/E0jKLIqW9NwGhlC4MFJOjDyvV aztylvIQi9ijDFbiFfy01A9X837v83vHAJuoLbgjVp5Vn6ssRlhbjc/4AbT/dzk61dqE D4Zg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:message-id:subject:cc :to:from:date:dkim-signature; bh=6gkq0m3ZW4hQGEKSGvfJPClu+To7TEynIn/wij5+DxA=; fh=ER2H6g+88FnZd3JyhT6ewHH56pdK2QWfWr77z0IHoNE=; b=Dvx2Xtb39E6GeXPgm89mxfQyCMIv42ZXb01Ag3ekZPqSeThKN37GWByTYQQpmHusHs z0ylk62pg/CxM1gZxIBwKtGqgiD5bUn+0K/ycGtJ58Mogt2FOFAGM4nqeQae26THqiBm F+qMd97kHw8VQXPdsb6skVZj8QrOE/VHFtkTSrc1qp/grJUuf1eahzXHY8LYnNcG1pc1 KF3zpF6x7y6Eq09lIvRVecH4f29pjfq6e2hsNkegn6rLQzqL2pB19qFtPYKxJy81mZDK pFZtNOdNFH8d5SENxWyzPYDeqJ7FaDDmxULoevwNBjW5VKIpTPtZQzN9/UFFNSmJEehj Mslg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=PDSj3YIs; spf=pass (google.com: domain of linux-kernel+bounces-23267-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-23267-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id k128-20020a628486000000b006dae558afa8si643396pfd.60.2024.01.11.01.01.34 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 11 Jan 2024 01:01:34 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-23267-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=PDSj3YIs; spf=pass (google.com: domain of linux-kernel+bounces-23267-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-23267-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 2A5E5282730 for ; Thu, 11 Jan 2024 09:00:46 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 58C95125CA; Thu, 11 Jan 2024 08:57:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="PDSj3YIs" Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 44747FBFA; Thu, 11 Jan 2024 08:56:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1704963418; x=1736499418; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=UUWlIy1VxeScjKQ6UmhSu9xVQRv3RDvrQHXl6p2NOP0=; b=PDSj3YIscoybxYQTFyj6cCLIm1/V92LXdVhNyscGV+swhxytjswRqkT/ IldtAGVivK7vYAddyNx3PCnAinONfgcs6/anBo4bjdZqCUkzWp2ECv2pG Pc0mq1PCYdnMdTY4Zmaac1nzThCqfoTq6m9nZLrGMAAs5sD3dhCntCubf KL2/FWXWHcotSaZ5AbYyDp7XFBaOcXMr795JQkEtlPO8bEdfWHM0jlhX2 4kiKG1h9i8iqbt2RFVnFNYB9Knsecvo9NOZx0cMrtkSO1T/MzercEJJQU FNMYuUSj4OkeIikbFq61nf8c2xW5tk2h0WOHGofFz1gKnHkchmhORRFwf Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10949"; a="429967807" X-IronPort-AV: E=Sophos;i="6.04,185,1695711600"; d="scan'208";a="429967807" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2024 00:56:57 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10949"; a="785916920" X-IronPort-AV: E=Sophos;i="6.04,185,1695711600"; d="scan'208";a="785916920" Received: from ericwong-mobl2.amr.corp.intel.com (HELO desk) ([10.209.43.169]) by fmsmga007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jan 2024 00:56:55 -0800 Date: Thu, 11 Jan 2024 00:56:55 -0800 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta Subject: [PATCH v5 6/6] KVM: VMX: Move VERW closer to VMentry for MDS mitigation Message-ID: <20240111-delay-verw-v5-6-a3b234933ea6@linux.intel.com> X-Mailer: b4 0.12.3 References: <20240111-delay-verw-v5-0-a3b234933ea6@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240111-delay-verw-v5-0-a3b234933ea6@linux.intel.com> X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787784010810247809 X-GMAIL-MSGID: 1787784010810247809 During VMentry VERW is executed to mitigate MDS. After VERW, any memory access like register push onto stack may put host data in MDS affected CPU buffers. A guest can then use MDS to sample host data. Although likelihood of secrets surviving in registers at current VERW callsite is less, but it can't be ruled out. Harden the MDS mitigation by moving the VERW mitigation late in VMentry path. Note that VERW for MMIO Stale Data mitigation is unchanged because of the complexity of per-guest conditional VERW which is not easy to handle that late in asm with no GPRs available. If the CPU is also affected by MDS, VERW is unconditionally executed late in asm regardless of guest having MMIO access. Signed-off-by: Pawan Gupta --- arch/x86/kvm/vmx/vmenter.S | 3 +++ arch/x86/kvm/vmx/vmx.c | 20 +++++++++++++++++--- 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index b3b13ec04bac..139960deb736 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -161,6 +161,9 @@ SYM_FUNC_START(__vmx_vcpu_run) /* Load guest RAX. This kills the @regs pointer! */ mov VCPU_RAX(%_ASM_AX), %_ASM_AX + /* Clobbers EFLAGS.ZF */ + CLEAR_CPU_BUFFERS + /* Check EFLAGS.CF from the VMX_RUN_VMRESUME bit test above. */ jnc .Lvmlaunch diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index bdcf2c041e0c..8defba8e417b 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -387,6 +387,17 @@ static __always_inline void vmx_enable_fb_clear(struct vcpu_vmx *vmx) static void vmx_update_fb_clear_dis(struct kvm_vcpu *vcpu, struct vcpu_vmx *vmx) { + /* + * FB_CLEAR_CTRL is to optimize VERW latency in guests when host is + * affected by MMIO Stale Data, but not by MDS/TAA. When + * X86_FEATURE_CLEAR_CPU_BUF is enabled, system is likely affected by + * MDS/TAA. Skip the optimization for such a case. + */ + if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF)) { + vmx->disable_fb_clear = false; + return; + } + vmx->disable_fb_clear = (host_arch_capabilities & ARCH_CAP_FB_CLEAR_CTRL) && !boot_cpu_has_bug(X86_BUG_MDS) && !boot_cpu_has_bug(X86_BUG_TAA); @@ -7226,11 +7237,14 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, guest_state_enter_irqoff(); - /* L1D Flush includes CPU buffer clear to mitigate MDS */ + /* + * L1D Flush includes CPU buffer clear to mitigate MDS, but VERW + * mitigation for MDS is done late in VMentry and is still + * executed in spite of L1D Flush. This is because an extra VERW + * should not matter much after the big hammer L1D Flush. + */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); - else if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF)) - mds_clear_cpu_buffers(); else if (static_branch_unlikely(&mmio_stale_data_clear) && kvm_arch_has_assigned_device(vcpu->kvm)) mds_clear_cpu_buffers();