diff mbox series

[v10,068/108] KVM: TDX: Implement TDX vcpu enter/exit path

Message ID	7e4e5bb55c1c346694719078d8ddc3e69c1964b3.1667110240.git.isaku.yamahata@intel.com
State	New
Headers	Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; From: isaku.yamahata@intel.com To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: isaku.yamahata@intel.com, isaku.yamahata@gmail.com, Paolo Bonzini <pbonzini@redhat.com>, erdemaktas@google.com, Sean Christopherson <seanjc@google.com>, Sagi Shahar <sagis@google.com>, David Matlack <dmatlack@google.com> Subject: [PATCH v10 068/108] KVM: TDX: Implement TDX vcpu enter/exit path Date: Sat, 29 Oct 2022 23:23:09 -0700 Message-Id: <7e4e5bb55c1c346694719078d8ddc3e69c1964b3.1667110240.git.isaku.yamahata@intel.com> In-Reply-To: <cover.1667110240.git.isaku.yamahata@intel.com> References: <cover.1667110240.git.isaku.yamahata@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk
Series	KVM TDX basic feature support \| [v10,000/108] KVM TDX basic feature support [v10,001/108] KVM: VMX: Move out vmx_x86_ops to 'main.c' to wrap VMX and TDX [v10,002/108] KVM: x86: Refactor KVM VMX module init/exit functions [v10,003/108] KVM: TDX: Add placeholders for TDX VM/vcpu structure [v10,004/108] x86/virt/tdx: Add a helper function to return system wide info about TDX module [v10,005/108] KVM: TDX: Initialize the TDX module when loading the KVM intel kernel module [v10,006/108] KVM: x86: Introduce vm_type to differentiate default VMs from confidential VMs [v10,007/108] KVM: TDX: Make TDX VM type supported [v10,008/108,MARKER] The start of TDX KVM patch series: TDX architectural definitions [v10,009/108] KVM: TDX: Define TDX architectural definitions [v10,010/108] KVM: TDX: Add TDX "architectural" error codes [v10,011/108] KVM: TDX: Add C wrapper functions for SEAMCALLs to the TDX module [v10,012/108] KVM: TDX: Add helper functions to print TDX SEAMCALL error [v10,013/108,MARKER] The start of TDX KVM patch series: TD VM creation/destruction [v10,014/108] KVM: TDX: Stub in tdx.h with structs, accessors, and VMCS helpers [v10,015/108] x86/cpu: Add helper functions to allocate/free TDX private host key id [v10,016/108] KVM: TDX: create/destroy VM structure [v10,017/108] KVM: TDX: Refuse to unplug the last cpu on the package [v10,018/108] KVM: TDX: x86: Add ioctl to get TDX systemwide parameters [v10,019/108] KVM: TDX: Add place holder for TDX VM specific mem_enc_op ioctl [v10,020/108] KVM: Support KVM_CAP_MAX_VCPUS for KVM_ENABLE_CAP [v10,021/108] KVM: TDX: initialize VM with TDX specific parameters [v10,022/108] KVM: TDX: Make pmu_intel.c ignore guest TD case [v10,023/108,MARKER] The start of TDX KVM patch series: TD vcpu creation/destruction [v10,024/108] KVM: TDX: allocate/free TDX vcpu structure [v10,025/108] KVM: TDX: Do TDX specific vcpu initialization [v10,026/108] KVM: TDX: Use private memory for TDX [v10,027/108,MARKER] The start of TDX KVM patch series: KVM MMU GPA shared bits [v10,028/108] KVM: x86/mmu: introduce config for PRIVATE KVM MMU [v10,029/108] KVM: x86/mmu: Add address conversion functions for TDX shared bit of GPA [v10,030/108,MARKER] The start of TDX KVM patch series: KVM TDP refactoring for TDX [v10,031/108] KVM: x86/mmu: Replace hardcoded value 0 for the initial value for SPTE [v10,032/108] KVM: x86/mmu: Make sync_page not use hard-coded 0 as the initial SPTE value [v10,033/108] KVM: x86/mmu: Allow non-zero value for non-present SPTE and removed SPTE [v10,034/108] KVM: x86/mmu: Add Suppress VE bit to shadow_mmio_{value, mask} [v10,035/108] KVM: x86/mmu: Track shadow MMIO value on a per-VM basis [v10,036/108] KVM: TDX: Enable mmio spte caching always for TDX [v10,037/108] KVM: x86/mmu: Disallow fast page fault on private GPA [v10,038/108] KVM: x86/mmu: Allow per-VM override of the TDP max page level [v10,039/108] KVM: VMX: Introduce test mode related to EPT violation VE [v10,040/108,MARKER] The start of TDX KVM patch series: KVM TDP MMU hooks [v10,041/108] KVM: x86/tdp_mmu: refactor kvm_tdp_mmu_map() [v10,042/108] KVM: x86/tdp_mmu: Init role member of struct kvm_mmu_page at allocation [v10,043/108] KVM: x86/mmu: Require TDP MMU for TDX [v10,044/108] KVM: x86/mmu: Add a new is_private member for union kvm_mmu_page_role [v10,045/108] KVM: x86/mmu: Add a private pointer to struct kvm_mmu_page [v10,046/108] KVM: Add flags to struct kvm_gfn_range [v10,047/108] KVM: x86/tdp_mmu: Don't zap private pages for unsupported cases [v10,048/108] KVM: x86/tdp_mmu: Make handle_changed_spte() return value [v10,049/108] KVM: x86/tdp_mmu: Support TDX private mapping for TDP MMU [v10,050/108,MARKER] The start of TDX KVM patch series: TDX EPT violation [v10,051/108] KVM: x86/mmu: Disallow dirty logging for x86 TDX [v10,052/108] KVM: x86/tdp_mmu: Ignore unsupported mmu operation on private GFNs [v10,053/108] KVM: VMX: Split out guts of EPT violation to common/exposed function [v10,054/108] KVM: VMX: Move setting of EPT MMU masks to common VT-x code [v10,055/108] KVM: TDX: Add load_mmu_pgd method for TDX [v10,056/108] KVM: TDX: don't request KVM_REQ_APIC_PAGE_RELOAD [v10,057/108] KVM: x86/VMX: introduce vmx tlb_remote_flush and tlb_remote_flush_with_range [v10,058/108] KVM: TDX: TDP MMU TDX support [v10,059/108,MARKER] The start of TDX KVM patch series: KVM TDP MMU MapGPA [v10,060/108] KVM: Add functions to set GFN to private or shared [v10,061/108] KVM: x86/mmu: Introduce kvm_mmu_map_tdp_page() for use by TDX [v10,062/108] KVM: x86/tdp_mmu: implement MapGPA hypercall for TDX [v10,063/108,MARKER] The start of TDX KVM patch series: TD finalization [v10,064/108] KVM: TDX: Create initial guest memory [v10,065/108] KVM: TDX: Finalize VM initialization [v10,066/108,MARKER] The start of TDX KVM patch series: TD vcpu enter/exit [v10,067/108] KVM: TDX: Add helper assembly function to TDX vcpu [v10,068/108] KVM: TDX: Implement TDX vcpu enter/exit path [v10,069/108] KVM: TDX: vcpu_run: save/restore host state(host kernel gs) [v10,070/108] KVM: TDX: restore host xsave state when exit from the guest TD [v10,071/108] KVM: x86: Allow to update cached values in kvm_user_return_msrs w/o wrmsr [v10,072/108] KVM: TDX: restore user ret MSRs [v10,073/108,MARKER] The start of TDX KVM patch series: TD vcpu exits/interrupts/hypercalls [v10,074/108] KVM: TDX: complete interrupts after tdexit [v10,075/108] KVM: TDX: restore debug store when TD exit [v10,076/108] KVM: TDX: handle vcpu migration over logical processor [v10,077/108] KVM: x86: Add a switch_db_regs flag to handle TDX's auto-switched behavior [v10,078/108] KVM: TDX: Add support for find pending IRQ in a protected local APIC [v10,079/108] KVM: x86: Assume timer IRQ was injected if APIC state is proteced [v10,080/108] KVM: TDX: remove use of struct vcpu_vmx from posted_interrupt.c [v10,081/108] KVM: TDX: Implement interrupt injection [v10,082/108] KVM: TDX: Implements vcpu request_immediate_exit [v10,083/108] KVM: TDX: Implement methods to inject NMI [v10,084/108] KVM: VMX: Modify NMI and INTR handlers to take intr_info as function argument [v10,085/108] KVM: VMX: Move NMI/exception handler to common helper [v10,086/108] KVM: x86: Split core of hypercall emulation to helper function [v10,087/108] KVM: TDX: Add a place holder to handle TDX VM exit [v10,088/108] KVM: TDX: Retry seamcall when TDX_OPERAND_BUSY with operand SEPT [v10,089/108] KVM: TDX: handle EXIT_REASON_OTHER_SMI [v10,090/108] KVM: TDX: handle ept violation/misconfig exit [v10,091/108] KVM: TDX: handle EXCEPTION_NMI and EXTERNAL_INTERRUPT [v10,092/108] KVM: TDX: Add a place holder for handler of TDX hypercalls (TDG.VP.VMCALL) [v10,093/108] KVM: TDX: handle KVM hypercall with TDG.VP.VMCALL [v10,094/108] KVM: TDX: Handle TDX PV CPUID hypercall [v10,095/108] KVM: TDX: Handle TDX PV HLT hypercall [v10,096/108] KVM: TDX: Handle TDX PV port io hypercall [v10,097/108] KVM: TDX: Handle TDX PV MMIO hypercall [v10,098/108] KVM: TDX: Implement callbacks for MSR operations for TDX [v10,099/108] KVM: TDX: Handle TDX PV rdmsr/wrmsr hypercall [v10,100/108] KVM: TDX: Handle TDX PV report fatal error hypercall [v10,101/108] KVM: TDX: Handle TDX PV map_gpa hypercall [v10,102/108] KVM: TDX: Handle TDG.VP.VMCALL<GetTdVmCallInfo> hypercall [v10,103/108] KVM: TDX: Silently discard SMI request [v10,104/108] KVM: TDX: Silently ignore INIT/SIPI [v10,105/108] KVM: TDX: Add methods to ignore accesses to CPU state [v10,106/108] Documentation/virt/kvm: Document on Trust Domain Extensions(TDX) [v10,107/108] KVM: x86: design documentation on TDX support of x86 KVM TDP MMU [v10,108/108,MARKER] the end of (the first phase of) TDX KVM patch series

Commit Message

Isaku Yamahata Oct. 30, 2022, 6:23 a.m. UTC

  From: Isaku Yamahata <isaku.yamahata@intel.com>

This patch implements running TDX vcpu.  Once vcpu runs on the logical
processor (LP), the TDX vcpu is associated with it.  When the TDX vcpu
moves to another LP, the TDX vcpu needs to flush its status on the LP.
When destroying TDX vcpu, it needs to complete flush and flush cpu memory
cache.  Track which LP the TDX vcpu run and flush it as necessary.

Do nothing on sched_in event as TDX doesn't support pause loop.

TDX vcpu execution requires restoring PMU debug store after returning back
to KVM because the TDX module unconditionally resets the value.  To reuse
the existing code, export perf_restore_debug_store.

Signed-off-by: Isaku Yamahata <isaku.yamahata@intel.com>
---
 arch/x86/kvm/vmx/main.c    | 21 +++++++++++++++++++--
 arch/x86/kvm/vmx/tdx.c     | 32 ++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/tdx.h     | 33 +++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/x86_ops.h |  2 ++
 arch/x86/kvm/x86.c         |  1 +
 5 files changed, 87 insertions(+), 2 deletions(-)

diff mbox series

Patch

diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c
index 3163915e2e3d..252c33820271 100644
--- a/arch/x86/kvm/vmx/main.c
+++ b/arch/x86/kvm/vmx/main.c
@@ -100,6 +100,23 @@  static void vt_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
 	return vmx_vcpu_reset(vcpu, init_event);
 }
 
+static int vt_vcpu_pre_run(struct kvm_vcpu *vcpu)
+{
+	if (is_td_vcpu(vcpu))
+		/* Unconditionally continue to vcpu_run(). */
+		return 1;
+
+	return vmx_vcpu_pre_run(vcpu);
+}
+
+static fastpath_t vt_vcpu_run(struct kvm_vcpu *vcpu)
+{
+	if (is_td_vcpu(vcpu))
+		return tdx_vcpu_run(vcpu);
+
+	return vmx_vcpu_run(vcpu);
+}
+
 static void vt_flush_tlb_all(struct kvm_vcpu *vcpu)
 {
 	if (is_td_vcpu(vcpu))
@@ -232,8 +249,8 @@  struct kvm_x86_ops vt_x86_ops __initdata = {
 	.flush_tlb_gva = vt_flush_tlb_gva,
 	.flush_tlb_guest = vt_flush_tlb_guest,
 
-	.vcpu_pre_run = vmx_vcpu_pre_run,
-	.vcpu_run = vmx_vcpu_run,
+	.vcpu_pre_run = vt_vcpu_pre_run,
+	.vcpu_run = vt_vcpu_run,
 	.handle_exit = vmx_handle_exit,
 	.skip_emulated_instruction = vmx_skip_emulated_instruction,
 	.update_emulated_instruction = vmx_update_emulated_instruction,
diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
index cce6ccd4a0be..2f57f62eb103 100644
--- a/arch/x86/kvm/vmx/tdx.c
+++ b/arch/x86/kvm/vmx/tdx.c
@@ -10,6 +10,9 @@ 
 #include "x86.h"
 #include "mmu.h"
 
+#include <trace/events/kvm.h>
+#include "trace.h"
+
 #undef pr_fmt
 #define pr_fmt(fmt) "tdx: " fmt
 
@@ -412,6 +415,35 @@  void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
 	vcpu->kvm->vm_bugged = true;
 }
 
+u64 __tdx_vcpu_run(hpa_t tdvpr, void *regs, u32 regs_mask);
+
+static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu *vcpu,
+					struct vcpu_tdx *tdx)
+{
+	guest_enter_irqoff();
+	tdx->exit_reason.full = __tdx_vcpu_run(tdx->tdvpr.pa, vcpu->arch.regs, 0);
+	guest_exit_irqoff();
+}
+
+fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+
+	if (unlikely(vcpu->kvm->vm_bugged)) {
+		tdx->exit_reason.full = TDX_NON_RECOVERABLE_VCPU;
+		return EXIT_FASTPATH_NONE;
+	}
+
+	trace_kvm_entry(vcpu);
+
+	tdx_vcpu_enter_exit(vcpu, tdx);
+
+	vcpu->arch.regs_avail &= ~VMX_REGS_LAZY_LOAD_SET;
+	trace_kvm_exit(vcpu, KVM_ISA_VMX);
+
+	return EXIT_FASTPATH_NONE;
+}
+
 void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level)
 {
 	td_vmcs_write64(to_tdx(vcpu), SHARED_EPT_POINTER, root_hpa & PAGE_MASK);
diff --git a/arch/x86/kvm/vmx/tdx.h b/arch/x86/kvm/vmx/tdx.h
index 686da2321683..064e1f2f61d5 100644
--- a/arch/x86/kvm/vmx/tdx.h
+++ b/arch/x86/kvm/vmx/tdx.h
@@ -31,12 +31,45 @@  struct kvm_tdx {
 	u64 tsc_offset;
 };
 
+union tdx_exit_reason {
+	struct {
+		/* 31:0 mirror the VMX Exit Reason format */
+		u64 basic		: 16;
+		u64 reserved16		: 1;
+		u64 reserved17		: 1;
+		u64 reserved18		: 1;
+		u64 reserved19		: 1;
+		u64 reserved20		: 1;
+		u64 reserved21		: 1;
+		u64 reserved22		: 1;
+		u64 reserved23		: 1;
+		u64 reserved24		: 1;
+		u64 reserved25		: 1;
+		u64 bus_lock_detected	: 1;
+		u64 enclave_mode	: 1;
+		u64 smi_pending_mtf	: 1;
+		u64 smi_from_vmx_root	: 1;
+		u64 reserved30		: 1;
+		u64 failed_vmentry	: 1;
+
+		/* 63:32 are TDX specific */
+		u64 details_l1		: 8;
+		u64 class		: 8;
+		u64 reserved61_48	: 14;
+		u64 non_recoverable	: 1;
+		u64 error		: 1;
+	};
+	u64 full;
+};
+
 struct vcpu_tdx {
 	struct kvm_vcpu	vcpu;
 
 	struct tdx_td_page tdvpr;
 	struct tdx_td_page *tdvpx;
 
+	union tdx_exit_reason exit_reason;
+
 	bool vcpu_initialized;
 
 	/*
diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h
index 5fccd98be06e..ccae338dcfdd 100644
--- a/arch/x86/kvm/vmx/x86_ops.h
+++ b/arch/x86/kvm/vmx/x86_ops.h
@@ -147,6 +147,7 @@  void tdx_vm_free(struct kvm *kvm);
 int tdx_vcpu_create(struct kvm_vcpu *vcpu);
 void tdx_vcpu_free(struct kvm_vcpu *vcpu);
 void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event);
+fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu);
 
 int tdx_vm_ioctl(struct kvm *kvm, void __user *argp);
 int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp);
@@ -169,6 +170,7 @@  static inline void tdx_vm_free(struct kvm *kvm) {}
 static inline int tdx_vcpu_create(struct kvm_vcpu *vcpu) { return -EOPNOTSUPP; }
 static inline void tdx_vcpu_free(struct kvm_vcpu *vcpu) {}
 static inline void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event) {}
+static inline fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu) { return EXIT_FASTPATH_NONE; }
 
 static inline int tdx_vm_ioctl(struct kvm *kvm, void __user *argp) { return -EOPNOTSUPP; }
 static inline int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp) { return -EOPNOTSUPP; }
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 5dadd0f9a10e..3662f64f3b5e 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -307,6 +307,7 @@  const struct kvm_stats_header kvm_vcpu_stats_header = {
 };
 
 u64 __read_mostly host_xcr0;
+EXPORT_SYMBOL_GPL(host_xcr0);
 
 static struct kmem_cache *x86_emulator_cache;