From patchwork Fri Oct 27 14:39:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 158971 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:d641:0:b0:403:3b70:6f57 with SMTP id cy1csp653952vqb; Fri, 27 Oct 2023 07:40:56 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH1Er5GvItPg+ReluhCWNE36UPLLLV4JeTtCPGDyEhAMzLU8656z2K2G+AmcGn3k7hIaMEX X-Received: by 2002:a05:6870:c58b:b0:1d6:5c40:11b6 with SMTP id ba11-20020a056870c58b00b001d65c4011b6mr3605110oab.14.1698417656616; Fri, 27 Oct 2023 07:40:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698417656; cv=none; d=google.com; s=arc-20160816; b=v6MHWIANkDVYedH/AHxrHjWuraudBPpAU/VwcBq147mTfm6o/gluUUEYE0Uy19YjTr cI7Wq0DsV8NoWWxPDJmgtY3F0Q5DXdDfDa6f2p54YPf73yym9ubh0NPgaHqh2C9W4cLw HLddK6j+BoGQZnhzEEnVZAqC5E2bRIw6dVjRxG3j/FTkDRx5oYXCJG60aRVdK9LTBizV y1jFbX/QZYxLOY3Wa9DA56uVfE03RbTmxZvcB7ZDCeP7BoxgXvreYVduLq9YfHNjYy4a xns3qbKz90ZzncgBKBTiqWih6NpB2z1diesc8X7BeFOT92tdQo/5+n8vRk0L6fNTFEHa x0nA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=zk7u2nr/8T5QhQ4lWBIy52Kjfo5l444VWypB6DxfrGY=; fh=9bGxxPI+IiFttPs3QhfhBAZDnxipaVmWpf+QSj2Bc/s=; b=rIR4n+5aPAt6IHHSh0vwO2QivX/hVC5rEsOyDd3e0OzT66oYIVCPiKhq59RR+tvZZb ommpzMsq7PyeADFJ1Croe6b5stpRxvOZMZB7rzHMHooerkZkpow8gyN7HVT19IsoW/pz N4ZwOSRJf4h4aj+Ahq3OBvkhcN39Uk6dOltKAQcqD/8NNeOnD/Max6T1LbQ5Ksd2i59T W2w8hnpilgVnNFF6m7kJvoWe3zJdIjZxW3ItHVKNVNtfNrp1U4Z25fqdcjsPkK9X6irS /Ar261d1CZy8gRwtdZWa5jGTcGXwWn1QCwAfDaApw8JPyzofZIMeuHSleWUIR7c8kX/f Hoig== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=RBWjJBXC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id s185-20020a8177c2000000b0059eb9ebb8d7si2583891ywc.383.2023.10.27.07.40.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Oct 2023 07:40:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=RBWjJBXC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 1D4248076037; Fri, 27 Oct 2023 07:40:18 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346172AbjJ0Oj3 (ORCPT + 25 others); Fri, 27 Oct 2023 10:39:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60582 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346132AbjJ0OjS (ORCPT ); Fri, 27 Oct 2023 10:39:18 -0400 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3195610DB; Fri, 27 Oct 2023 07:39:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698417555; x=1729953555; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=mjKmJXB8oVMEQJ55YBl9y3vDRhOMcEONKyySoTTCMIQ=; b=RBWjJBXCODCf+1lv6IFxLpNKiOi0W249ivbKQlHK2/AL30xAd3AzvK3N LmE+gJNT20pwF1y2G0kPQ8yRr7npy3fqqKEK8JspS7AchJI0e3DsFklDg nudiPdPXRc/PSktFDkRKtCC1kqVgW+Rna6xtk12fp4s4RbVnB0Gx/47s9 2aXPq0G180BDJPAZTRTWuERI+CbM9NvNhbRzznIl40LPPDew52YbMh1ut LRX+E0w8DQpHhwO3leAl2lOZoAJeY+LKotw9zGJwNi1t5gD3KNvIBjBki reXRi/gQLDwyR9PVNUyP6W9LUmyPht3gNXBeNSnUBSZXJQ+ZFYa9ZGdL7 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10876"; a="606760" X-IronPort-AV: E=Sophos;i="6.03,256,1694761200"; d="scan'208";a="606760" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2023 07:39:13 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10876"; a="736094456" X-IronPort-AV: E=Sophos;i="6.03,256,1694761200"; d="scan'208";a="736094456" Received: from dmnassar-mobl.amr.corp.intel.com (HELO desk) ([10.212.203.39]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2023 07:39:12 -0700 Date: Fri, 27 Oct 2023 07:39:12 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com, Andrew Cooper , Nikolay Borisov Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Greg Kroah-Hartman , Pawan Gupta Subject: [PATCH v4 6/6] KVM: VMX: Move VERW closer to VMentry for MDS mitigation Message-ID: <20231027-delay-verw-v4-6-9a3622d4bcf7@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231027-delay-verw-v4-0-9a3622d4bcf7@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231027-delay-verw-v4-0-9a3622d4bcf7@linux.intel.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 27 Oct 2023 07:40:18 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780919992655207620 X-GMAIL-MSGID: 1780919992655207620 During VMentry VERW is executed to mitigate MDS. After VERW, any memory access like register push onto stack may put host data in MDS affected CPU buffers. A guest can then use MDS to sample host data. Although likelihood of secrets surviving in registers at current VERW callsite is less, but it can't be ruled out. Harden the MDS mitigation by moving the VERW mitigation late in VMentry path. Note that VERW for MMIO Stale Data mitigation is unchanged because of the complexity of per-guest conditional VERW which is not easy to handle that late in asm with no GPRs available. If the CPU is also affected by MDS, VERW is unconditionally executed late in asm regardless of guest having MMIO access. Signed-off-by: Pawan Gupta --- arch/x86/kvm/vmx/vmenter.S | 3 +++ arch/x86/kvm/vmx/vmx.c | 19 ++++++++++++++----- 2 files changed, 17 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index b3b13ec04bac..139960deb736 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -161,6 +161,9 @@ SYM_FUNC_START(__vmx_vcpu_run) /* Load guest RAX. This kills the @regs pointer! */ mov VCPU_RAX(%_ASM_AX), %_ASM_AX + /* Clobbers EFLAGS.ZF */ + CLEAR_CPU_BUFFERS + /* Check EFLAGS.CF from the VMX_RUN_VMRESUME bit test above. */ jnc .Lvmlaunch diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 24e8694b83fc..a05c6b80b06c 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7226,16 +7226,24 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, guest_state_enter_irqoff(); - /* L1D Flush includes CPU buffer clear to mitigate MDS */ + /* + * L1D Flush includes CPU buffer clear to mitigate MDS, but VERW + * mitigation for MDS is done late in VMentry and is still + * executed in spite of L1D Flush. This is because an extra VERW + * should not matter much after the big hammer L1D Flush. + */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); - else if (cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF)) - mds_clear_cpu_buffers(); else if (static_branch_unlikely(&mmio_stale_data_clear) && kvm_arch_has_assigned_device(vcpu->kvm)) mds_clear_cpu_buffers(); - vmx_disable_fb_clear(vmx); + /* + * Optimize the latency of VERW in guests for MMIO mitigation. Skip + * the optimization when MDS mitigation(later in asm) is enabled. + */ + if (!cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF)) + vmx_disable_fb_clear(vmx); if (vcpu->arch.cr2 != native_read_cr2()) native_write_cr2(vcpu->arch.cr2); @@ -7248,7 +7256,8 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, vmx->idt_vectoring_info = 0; - vmx_enable_fb_clear(vmx); + if (!cpu_feature_enabled(X86_FEATURE_CLEAR_CPU_BUF)) + vmx_enable_fb_clear(vmx); if (unlikely(vmx->fail)) { vmx->exit_reason.full = 0xdead;