From patchwork Fri Feb 23 20:42:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 205662 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp835748dyb; Fri, 23 Feb 2024 12:46:33 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUx6xqa8pK/voglg2pY1fnAHKcx3x0sKKFHjlbz+HkiX+D09jYfn1vpCDBtU059/zDHjebKoQz5WHlTBKvlx3/up5fQ9w== X-Google-Smtp-Source: AGHT+IEhXeH+tLYoCfeo12Yne6ErBQwhDZ/GXp4pcWhTRvJdxq+bEFmpEProy3wQI9HdnD3GOOV8 X-Received: by 2002:a05:620a:1248:b0:787:b595:8af2 with SMTP id a8-20020a05620a124800b00787b5958af2mr942058qkl.73.1708721192982; Fri, 23 Feb 2024 12:46:32 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708721192; cv=pass; d=google.com; s=arc-20160816; b=WM4+auTWI5MpIybDTyZIG/1HLf4QBl128j2kQiMDz0h3JkixTD4jgdOnDkoM2KvzA+ +XWhgXZGFdovurvC2fFWOyiMFbnZS7aORCWdAgV8X0bCkpYjiKoTbHv4zucJ3S2D1/NY gKXKHTmy1gABnue8+3tgEcFXmH3WqU0UoUbMQjZ5IcRjwcRiudxwZtZE6kTiHNAJCc85 05orWpozy5tABSQK8hziGgR4QAITBUcQb5o4yNNdR+7LIhBp6HGBLrMjwYJ2Z0g/yx66 9QclzCtlK+r3Xxv3dhgFubfQoDfIG+bOuKvqfZo3TnhVqS0TwskjvFdj+6atDOA31e0V Vt6g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :reply-to:dkim-signature; bh=fjPJ3Hja7aXkfyUdgxA5xTLuvAk9amyqKFMcOEhsGqM=; fh=GQw9bD8DgeW7uBUryaq0WKLuaHq6gqJJRTolK8AHjII=; b=jyLT262gJ01lkTMiWtsas+50Qwwkoy0wxhZ8mC0ZPUtN6WHXwtr7ImvOSB9SjJghtV guQ9LchEZI18am2eJwBHujlkWR4L0jV7FGyRTMq2tSBqdOuUp0r3j2D28T5rJqDOeuXg MKGXcl+V/71zB1h5JLlPNEcWkeTtq9EV4MO5D+b5RobA2JRoMdixMX2wDNQy4hVBg73u NzbUk1yXcu+b23DBwC2bU2oJ1d84d/sJh9lRabv2bfx1yAv2e+F7S+aId/Nb3tpNU9m7 T0SH1tAJutiZFXOuncrUqdyvtfJ5b6G3QKf6vhfE4b1jSDixR03ASX/xnJi6wfCTS9M8 L2yg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=CmekOdEZ; arc=pass (i=1 spf=pass spfdomain=flex--seanjc.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-79132-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-79132-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id l21-20020a05620a28d500b00787b434d4cesi2034240qkp.639.2024.02.23.12.46.32 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Feb 2024 12:46:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-79132-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=CmekOdEZ; arc=pass (i=1 spf=pass spfdomain=flex--seanjc.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-79132-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-79132-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id B23681C22031 for ; Fri, 23 Feb 2024 20:46:32 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C5A84158D83; Fri, 23 Feb 2024 20:42:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="CmekOdEZ" Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E410157E97 for ; Fri, 23 Feb 2024 20:42:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708720967; cv=none; b=FJtJMUw9hGrdEF0YfqSGpIB1TZi1cgc+/J5QCjF0Oure4TsdbHFM6TW5+ns2XuPGr1lrKYIDc3tmIZCNdcb73baqg6EelBXOAh7JqCQI4z8L1X+Vt9vnA0T5FEWRKuFvNM7YZ7+4AAC3rXL285BZzmzBlTE9bVnh/z23o0egei4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708720967; c=relaxed/simple; bh=2NQqn0GS1NXkZhFdQnHQDhlZap/bPRB2/FTisNgRF48=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ASyAgHHwn1YT9an3mOHlrmlcHRF4dwJl7TVL/w31WfZmOuZRG0REDyqvfCeCfVjIkaD+M8ljOsO8AIcDywfELb7SBlDlbU87yp91GXMaNW5zeNV9M9ifEY+5Im16DH2+5G2/bn79TQ+ABwHVZI80j77KTBmQ0KJP2ux463FXRxI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=CmekOdEZ; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6085e433063so13382887b3.1 for ; Fri, 23 Feb 2024 12:42:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708720964; x=1709325764; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=fjPJ3Hja7aXkfyUdgxA5xTLuvAk9amyqKFMcOEhsGqM=; b=CmekOdEZhBlSR+m+qkvetN4vXy6aAs4K2Aqz7+dBGhTrSRGmVUfZcCJCaLRS6Tg1EQ +AqV9BdVyxHw7Jh6D1GOnbKsjjy1n6HknPWPbMCAGizEakeSVkNi/TY6W4Xo0tOVZb9Y BWupji0dF0NJDVzfNEvYfc1zQxDFrXOue+lO6dTtT/bftQiurEtQ3BbOeEo+CWEXA/8K TMCiEJygm01gxqW5BTqTU/qfSAmul6B3MnNlHotgMdPwLz+f25jNcxep49Hduc8CZU/9 h/sX7rS+e6MDCaTX697ZC/gOKrv2VbiqVhSzL5CTiWHIUbzSNbJq5ld/8t2XfSiOmhXM 9pdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708720964; x=1709325764; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=fjPJ3Hja7aXkfyUdgxA5xTLuvAk9amyqKFMcOEhsGqM=; b=ua0HCKSj6lhgH+iFDb1YEWjXMoAYXlislouhePCwvjyEbbtWYspnF7I1QTct+WOkP2 7qyZn6u6Pe7Zlf77VG/8LvLs3CjmApL0BXbUaUkS7ahobLuqV3QXGPlenR60L4gOVIoQ GmCpteaCopbP7cFBfMKsNRtGiVb7qvxVEnPvvP81144HOpITnxLoEO7Y3HVUOnRpyzGz +z2kKIFFEU+iIxzxutEm3Q0iTurWPDFI8AN+TWpYVd1hvTXrCSbWavBZQUte7hXOPZIq EenyhoAiihVJZhRalH+5vWC9gkUWrrWvYATnosUy2qPjBfL0YYtLLDGdYyCEBzIdGgui pihQ== X-Forwarded-Encrypted: i=1; AJvYcCUweSXGwsuVWF+yUSUF/2cGxLMjhVepb8zeYyFjw1LmtBIddO9YnsEs98RvMafgUXUp411K8J+IjR3tyMlvNxx6sv7mdlJ0pk0S3DCt X-Gm-Message-State: AOJu0Yx9vbaePu46LWvSznIAlvZ+Z3xdLarNRSDTYpzoVz857fqM+Yij bdZUkgJZHLf+NxWqqtLhBV2tlR9l6o4sd+WY4EkgRDAqYHUWjPKOgFUvs7EUVrcljhLJ439hGFM n1w== X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a05:6902:1243:b0:dcd:3172:7265 with SMTP id t3-20020a056902124300b00dcd31727265mr244887ybu.8.1708720964754; Fri, 23 Feb 2024 12:42:44 -0800 (PST) Reply-To: Sean Christopherson Date: Fri, 23 Feb 2024 12:42:30 -0800 In-Reply-To: <20240223204233.3337324-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240223204233.3337324-1-seanjc@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240223204233.3337324-6-seanjc@google.com> Subject: [PATCH 5/8] KVM: SVM: Save/restore non-volatile GPRs in SEV-ES VMRUN via host save area From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Tom Lendacky , Michael Roth , Alexey Kardashevskiy X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791724033674320614 X-GMAIL-MSGID: 1791724033674320614 Use the host save area to save/restore non-volatile (callee-saved) registers in __svm_sev_es_vcpu_run() to take advantage of hardware loading all registers from the save area on #VMEXIT. KVM still needs to save the registers it wants restored, but the loads are handled automatically by hardware. Aside from less assembly code, letting hardware do the restoration means stack frames are preserved for the entirety of __svm_sev_es_vcpu_run(). Opportunistically add a comment to call out why @svm needs to be saved across VMRUN->#VMEXIT, as it's not easy to decipher that from the macro hell. Cc: Tom Lendacky Cc: Michael Roth Cc: Alexey Kardashevskiy Signed-off-by: Sean Christopherson --- arch/x86/kvm/svm/svm.c | 17 +++++++++------- arch/x86/kvm/svm/svm.h | 3 ++- arch/x86/kvm/svm/vmenter.S | 41 +++++++++++++++++++++----------------- 3 files changed, 35 insertions(+), 26 deletions(-) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index e90b429c84f1..e7c8a48e36eb 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -1503,6 +1503,11 @@ static void svm_vcpu_free(struct kvm_vcpu *vcpu) __free_pages(virt_to_page(svm->msrpm), get_order(MSRPM_SIZE)); } +static struct sev_es_save_area *sev_es_host_save_area(struct svm_cpu_data *sd) +{ + return page_address(sd->save_area) + 0x400; +} + static void svm_prepare_switch_to_guest(struct kvm_vcpu *vcpu) { struct vcpu_svm *svm = to_svm(vcpu); @@ -1519,12 +1524,8 @@ static void svm_prepare_switch_to_guest(struct kvm_vcpu *vcpu) * or subsequent vmload of host save area. */ vmsave(sd->save_area_pa); - if (sev_es_guest(vcpu->kvm)) { - struct sev_es_save_area *hostsa; - hostsa = (struct sev_es_save_area *)(page_address(sd->save_area) + 0x400); - - sev_es_prepare_switch_to_guest(hostsa); - } + if (sev_es_guest(vcpu->kvm)) + sev_es_prepare_switch_to_guest(sev_es_host_save_area(sd)); if (tsc_scaling) __svm_write_tsc_multiplier(vcpu->arch.tsc_scaling_ratio); @@ -4101,6 +4102,7 @@ static fastpath_t svm_exit_handlers_fastpath(struct kvm_vcpu *vcpu) static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_ctrl_intercepted) { + struct svm_cpu_data *sd = per_cpu_ptr(&svm_data, vcpu->cpu); struct vcpu_svm *svm = to_svm(vcpu); guest_state_enter_irqoff(); @@ -4108,7 +4110,8 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_ctrl_in amd_clear_divider(); if (sev_es_guest(vcpu->kvm)) - __svm_sev_es_vcpu_run(svm, spec_ctrl_intercepted); + __svm_sev_es_vcpu_run(svm, spec_ctrl_intercepted, + sev_es_host_save_area(sd)); else __svm_vcpu_run(svm, spec_ctrl_intercepted); diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 8ef95139cd24..b98cced44e48 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -697,7 +697,8 @@ void sev_es_unmap_ghcb(struct vcpu_svm *svm); /* vmenter.S */ -void __svm_sev_es_vcpu_run(struct vcpu_svm *svm, bool spec_ctrl_intercepted); +void __svm_sev_es_vcpu_run(struct vcpu_svm *svm, bool spec_ctrl_intercepted, + struct sev_es_save_area *hostsa); void __svm_vcpu_run(struct vcpu_svm *svm, bool spec_ctrl_intercepted); #define DEFINE_KVM_GHCB_ACCESSORS(field) \ diff --git a/arch/x86/kvm/svm/vmenter.S b/arch/x86/kvm/svm/vmenter.S index edbaadaacba7..e92953427100 100644 --- a/arch/x86/kvm/svm/vmenter.S +++ b/arch/x86/kvm/svm/vmenter.S @@ -292,23 +292,35 @@ SYM_FUNC_START(__svm_vcpu_run) SYM_FUNC_END(__svm_vcpu_run) #ifdef CONFIG_KVM_AMD_SEV + + +#ifdef CONFIG_X86_64 +#define SEV_ES_GPRS_BASE 0x300 +#define SEV_ES_RBX (SEV_ES_GPRS_BASE + __VCPU_REGS_RBX * WORD_SIZE) +#define SEV_ES_RBP (SEV_ES_GPRS_BASE + __VCPU_REGS_RBP * WORD_SIZE) +#define SEV_ES_R12 (SEV_ES_GPRS_BASE + __VCPU_REGS_R12 * WORD_SIZE) +#define SEV_ES_R13 (SEV_ES_GPRS_BASE + __VCPU_REGS_R13 * WORD_SIZE) +#define SEV_ES_R14 (SEV_ES_GPRS_BASE + __VCPU_REGS_R14 * WORD_SIZE) +#define SEV_ES_R15 (SEV_ES_GPRS_BASE + __VCPU_REGS_R15 * WORD_SIZE) +#endif + /** * __svm_sev_es_vcpu_run - Run a SEV-ES vCPU via a transition to SVM guest mode * @svm: struct vcpu_svm * * @spec_ctrl_intercepted: bool */ SYM_FUNC_START(__svm_sev_es_vcpu_run) - push %rbp - push %r15 - push %r14 - push %r13 - push %r12 - push %rbx - /* - * Save variables needed after vmexit on the stack, in inverse - * order compared to when they are needed. + * Save non-volatile (callee-saved) registers to the host save area. + * Except for RAX and RSP, all GPRs are restored on #VMEXIT, but not + * saved on VMRUN. */ + mov %rbp, SEV_ES_RBP (%rdx) + mov %r15, SEV_ES_R15 (%rdx) + mov %r14, SEV_ES_R14 (%rdx) + mov %r13, SEV_ES_R13 (%rdx) + mov %r12, SEV_ES_R12 (%rdx) + mov %rbx, SEV_ES_RBX (%rdx) /* Accessed directly from the stack in RESTORE_HOST_SPEC_CTRL. */ push %rsi @@ -316,7 +328,7 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) /* Save @svm. */ push %rdi - /* Clobbers RAX, RCX, RDX. */ + /* Clobbers RAX, RCX, RDX (@hostsa). */ RESTORE_GUEST_SPEC_CTRL /* Get svm->current_vmcb->pa into RAX. */ @@ -338,7 +350,7 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) FILL_RETURN_BUFFER %rax, RSB_CLEAR_LOOPS, X86_FEATURE_RETPOLINE #endif - /* Clobbers RAX, RCX, RDX. */ + /* Clobbers RAX, RCX, RDX, consumes RDI (@svm). */ RESTORE_HOST_SPEC_CTRL /* @@ -353,13 +365,6 @@ SYM_FUNC_START(__svm_sev_es_vcpu_run) /* "Pop" and discard @spec_ctrl_intercepted. */ pop %rax - pop %rbx - - pop %r12 - pop %r13 - pop %r14 - pop %r15 - pop %rbp RET RESTORE_GUEST_SPEC_CTRL_BODY