From patchwork Tue Mar 21 19:40:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 73028 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp1975730wrt; Tue, 21 Mar 2023 12:47:42 -0700 (PDT) X-Google-Smtp-Source: AK7set9BuxJdCi4AKKlFIvbEJwnpg+vxcTETCu1GD7OcW86ae/fGlHOUD/kzoSHmv14lSU0F19yx X-Received: by 2002:a05:6a20:7b22:b0:db:1b41:6fe with SMTP id s34-20020a056a207b2200b000db1b4106femr930539pzh.6.1679428061698; Tue, 21 Mar 2023 12:47:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679428061; cv=none; d=google.com; s=arc-20160816; b=VJJriY4TvwOXcOBle5uKP64W7J2G5ee48xmmi9N5wU0/9uCYGK01GYyGg1buZXiD++ 6lQwpQSGQQxvDoBSeeXYJPXmUnhlMxlLjn+hjv8T0fhOvRb5hqcO/s7BY7Sssa6CV6fH gN2XGC5fPKnTfcHEil0qCdyI6I0ovc3oI0zEnZlTly/Hlq5kovRq2wdKOKA5mkRnoRpp c3bMTtRXNQrgmovUbeeXNA94THhDOAjSii58cJAbdlvx1AMbW7cJbH+g3J1WPcsEh9d+ Rzgi+xTXZ43zi45mBLrvLi1eIzv4w0q28NQee1K/LPj+RzYOs5ipzuWwfKzxmpRUZFgz xBYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=l3gQK6+MXZzclW4d0GGLnTn3p9hIu0ffrS0A+Sd8IEM=; b=nM15SyW+kR9iI5lQoCZBQK6D96KIrkYXVHajReoMSCNgPagKkEYkaBStAjmHueKzzS sGRS3c+aw8N11PDZxFf2pMfSHeiiLY9ImzWDDELAzET6SjjrwgKlOWgecI3vOsV+4Wzt cbfJdEJn6v5T7GzUsZLHkRjYX8UT7FAdu4CPi5qUdePHrQzROxUtEwIHLglrCe/1vVnr B/Qy4CFBXC290zIb/LeHyUVASn8xZbtaguWo79u/wn5eecV/DTRbYfJ8ewILJQT/YZTd AwzCh7VleIRsUYnZjghv1MCWo8u7xQaPM0Uo8WkxV7RnHrPPJGSc319mp1uKRwkTK5un yNfQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=iDIMZpNV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 7-20020a630307000000b004df43225600si14153839pgd.709.2023.03.21.12.47.29; Tue, 21 Mar 2023 12:47:41 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=iDIMZpNV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230473AbjCUTmR (ORCPT + 99 others); Tue, 21 Mar 2023 15:42:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229890AbjCUTmF (ORCPT ); Tue, 21 Mar 2023 15:42:05 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8D8A52FCE0 for ; Tue, 21 Mar 2023 12:41:06 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id r29so14855623wra.13 for ; Tue, 21 Mar 2023 12:41:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1679427642; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=l3gQK6+MXZzclW4d0GGLnTn3p9hIu0ffrS0A+Sd8IEM=; b=iDIMZpNVu/rJq2uyMdfotQHp+XfP7xkWQekYnCD7Mpayj6O5zMcoQHrk2DVTxASKAf WFIrUo72HOrWyBGizUQkIydoteGJDnDt6pVL8Q9Hmee1ZnkGR6PA+KRBQqgB0sDendod I9iUk3S02eV9sWO5GDS4kWCFiwfLf3JN3iO/MK8vIIq4ZcmVqIrUY/r/8ythxjfz2lLC X9cfoGa0pKOBUlBjmq8KK6B8nZFkKLqUbvybSbx7J7P0xzxdDvNmmhN0KDqab6tBmyof a8InYJmLone6yIGk1ZEHbHK2HsAJDO98363Iyfc6eMehMiiptxIk4tTU15ym77+l1ipJ T9vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679427642; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l3gQK6+MXZzclW4d0GGLnTn3p9hIu0ffrS0A+Sd8IEM=; b=ZU19xL4XOULQK3AZi6sqxS03TNkJEv2kVrX4Zi1fluaW2gSoY0rtaja3GoAKL/+nXw MSXrguIleGBhyNhZ643wp6qL5sd4yewmp7t3et99Sk67o54+VKR+EJtOjdv3Oi5p2ECc N0Lh5Qrcg2N7RXrkb/2zm1l12WBN9zlGOV3DgmyjtwToP1BFf+OF+R6GH+JLKnLoZBvo Gur4zfIAolF847RUrnuw0VqTRvJDs1M5XVtb9vX7h39zrcTRz6gKj0NnxZyjEdTiQhok CLa6TyftR3Rv98qLEh6cCc+3lkjPxuuC7ghhI2dyo3pudUN47lO1FZdhY0qCwgyIQ0p9 XAGw== X-Gm-Message-State: AO0yUKWiNaApB4d/XxoRkFRh795v2GYO/HBv0GCZjFwq9r1dxymzfG4d h5iomuyF751XPzJ5wV9HD4zB9w== X-Received: by 2002:adf:db86:0:b0:2d5:5610:e7b4 with SMTP id u6-20020adfdb86000000b002d55610e7b4mr3135776wri.64.1679427642103; Tue, 21 Mar 2023 12:40:42 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b566:0:7a8a:d679:ba4e:61cf]) by smtp.gmail.com with ESMTPSA id a3-20020adffb83000000b002c561805a4csm12005026wrr.45.2023.03.21.12.40.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Mar 2023 12:40:41 -0700 (PDT) From: Usama Arif To: dwmw2@infradead.org, tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com Cc: piotrgorski@cachyos.org, oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com, thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de, fam.zheng@bytedance.com, punit.agrawal@bytedance.com, simon.evans@bytedance.com, liangma@liangbit.com, gpiccoli@igalia.com, David Woodhouse , Sabin Rapan , Usama Arif Subject: [PATCH v16 8/8] x86/smpboot: Allow parallel bringup for SEV-ES Date: Tue, 21 Mar 2023 19:40:08 +0000 Message-Id: <20230321194008.785922-9-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230321194008.785922-1-usama.arif@bytedance.com> References: <20230321194008.785922-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761007958837078885?= X-GMAIL-MSGID: =?utf-8?q?1761007958837078885?= From: David Woodhouse Enable parallel bringup for SEV-ES guests. The APs can't actually execute the CPUID instruction directly during early startup, but they can make the GHCB call directly instead, just as the VC trap handler would do. Thanks to Sabin for talking me through the way this works. Suggested-by: Sabin Rapan Signed-off-by: David Woodhouse Signed-off-by: Usama Arif Reviewed-by: Tom Lendacky --- arch/x86/coco/core.c | 5 ++++ arch/x86/include/asm/coco.h | 1 + arch/x86/include/asm/sev-common.h | 3 +++ arch/x86/include/asm/smp.h | 5 +++- arch/x86/kernel/head_64.S | 30 ++++++++++++++++++++++++ arch/x86/kernel/smpboot.c | 39 ++++++++++++++++++++++++++----- 6 files changed, 76 insertions(+), 7 deletions(-) diff --git a/arch/x86/coco/core.c b/arch/x86/coco/core.c index 49b44f881484..0bab38efb15a 100644 --- a/arch/x86/coco/core.c +++ b/arch/x86/coco/core.c @@ -129,6 +129,11 @@ u64 cc_mkdec(u64 val) } EXPORT_SYMBOL_GPL(cc_mkdec); +enum cc_vendor cc_get_vendor(void) +{ + return vendor; +} + __init void cc_set_vendor(enum cc_vendor v) { vendor = v; diff --git a/arch/x86/include/asm/coco.h b/arch/x86/include/asm/coco.h index 3d98c3a60d34..0428d9712c96 100644 --- a/arch/x86/include/asm/coco.h +++ b/arch/x86/include/asm/coco.h @@ -12,6 +12,7 @@ enum cc_vendor { }; void cc_set_vendor(enum cc_vendor v); +enum cc_vendor cc_get_vendor(void); void cc_set_mask(u64 mask); #ifdef CONFIG_ARCH_HAS_CC_PLATFORM diff --git a/arch/x86/include/asm/sev-common.h b/arch/x86/include/asm/sev-common.h index b63be696b776..0abf8a39cee1 100644 --- a/arch/x86/include/asm/sev-common.h +++ b/arch/x86/include/asm/sev-common.h @@ -70,6 +70,7 @@ /* GHCBData[63:12] */ \ (((u64)(v) & GENMASK_ULL(63, 12)) >> 12) +#ifndef __ASSEMBLY__ /* * SNP Page State Change Operation * @@ -161,6 +162,8 @@ struct snp_psc_desc { #define GHCB_RESP_CODE(v) ((v) & GHCB_MSR_INFO_MASK) +#endif /* __ASSEMBLY__ */ + /* * Error codes related to GHCB input that can be communicated back to the guest * by setting the lower 32-bits of the GHCB SW_EXITINFO1 field to 2. diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index defe76ee9e64..1584f04a7007 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -204,7 +204,10 @@ extern unsigned int smpboot_control; /* Control bits for startup_64 */ #define STARTUP_APICID_CPUID_0B 0x80000000 #define STARTUP_APICID_CPUID_01 0x40000000 +#define STARTUP_APICID_SEV_ES 0x20000000 -#define STARTUP_PARALLEL_MASK (STARTUP_APICID_CPUID_01 | STARTUP_APICID_CPUID_0B) +#define STARTUP_PARALLEL_MASK (STARTUP_APICID_CPUID_01 | \ + STARTUP_APICID_CPUID_0B | \ + STARTUP_APICID_SEV_ES) #endif /* _ASM_X86_SMP_H */ diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index ff3a5f008d8a..9c38849fcac8 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -26,6 +26,7 @@ #include #include #include +#include /* * We are not able to switch in one step to the final KERNEL ADDRESS SPACE @@ -242,6 +243,7 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * * Bit 31 STARTUP_APICID_CPUID_0B flag (use CPUID 0x0b) * Bit 30 STARTUP_APICID_CPUID_01 flag (use CPUID 0x01) + * Bit 29 STARTUP_APICID_SEV_ES flag (CPUID 0x0b via GHCB MSR) * Bit 0-24 CPU# if STARTUP_APICID_CPUID_xx flags are not set */ movl smpboot_control(%rip), %ecx @@ -249,6 +251,10 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) jnz .Luse_cpuid_0b testl $STARTUP_APICID_CPUID_01, %ecx jnz .Luse_cpuid_01 +#ifdef CONFIG_AMD_MEM_ENCRYPT + testl $STARTUP_APICID_SEV_ES, %ecx + jnz .Luse_sev_cpuid_0b +#endif andl $0x0FFFFFFF, %ecx jmp .Lsetup_cpu @@ -259,6 +265,30 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) shr $24, %edx jmp .Lsetup_AP +#ifdef CONFIG_AMD_MEM_ENCRYPT +.Luse_sev_cpuid_0b: + /* Set the GHCB MSR to request CPUID 0xB_EDX */ + movl $MSR_AMD64_SEV_ES_GHCB, %ecx + movl $(GHCB_CPUID_REQ_EDX << 30) | GHCB_MSR_CPUID_REQ, %eax + movl $0x0B, %edx + wrmsr + + /* Perform GHCB MSR protocol */ + rep; vmmcall /* vmgexit */ + + /* + * Get the result. After the RDMSR: + * EAX should be 0xc0000005 + * EDX should have the CPUID register value and since EDX + * is the target register, no need to move the result. + */ + rdmsr + andl $GHCB_MSR_INFO_MASK, %eax + cmpl $GHCB_MSR_CPUID_RESP, %eax + jne 1f + jmp .Lsetup_AP +#endif + .Luse_cpuid_0b: mov $0x0B, %eax xorl %ecx, %ecx diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 0dba5c247be0..ef37356ab695 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -85,6 +85,7 @@ #include #include #include +#include /* representing HT siblings of each logical CPU */ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_map); @@ -1513,15 +1514,36 @@ void __init smp_prepare_cpus_common(void) * We can do 64-bit AP bringup in parallel if the CPU reports its APIC * ID in CPUID (either leaf 0x0B if we need the full APIC ID in X2APIC * mode, or leaf 0x01 if 8 bits are sufficient). Otherwise it's too - * hard. And not for SEV-ES guests because they can't use CPUID that - * early. + * hard. */ static bool prepare_parallel_bringup(void) { - if (IS_ENABLED(CONFIG_X86_32) || cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) + bool has_sev_es = false; + + if (IS_ENABLED(CONFIG_X86_32)) return false; - if (x2apic_mode) { + /* + * Encrypted guests other than SEV-ES (in the future) will need to + * implement an early way of finding the APIC ID, since they will + * presumably block direct CPUID too. Be kind to our future selves + * by warning here instead of just letting them break. Parallel + * startup doesn't have to be in the first round of enabling patches + * for any such technology. + */ + if (cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT)) { + switch (cc_get_vendor()) { + case CC_VENDOR_AMD: + has_sev_es = true; + break; + + default: + pr_info("Disabling parallel bringup due to guest state encryption\n"); + return false; + } + } + + if (x2apic_mode || has_sev_es) { if (boot_cpu_data.cpuid_level < 0x0b) return false; @@ -1530,8 +1552,13 @@ static bool prepare_parallel_bringup(void) return false; } - pr_debug("Using CPUID 0xb for parallel CPU startup\n"); - smpboot_control = STARTUP_APICID_CPUID_0B; + if (has_sev_es) { + pr_debug("Using SEV-ES CPUID 0xb for parallel CPU startup\n"); + smpboot_control = STARTUP_APICID_SEV_ES; + } else { + pr_debug("Using CPUID 0xb for parallel CPU startup\n"); + smpboot_control = STARTUP_APICID_CPUID_0B; + } } else { /* Without X2APIC, what's in CPUID 0x01 should suffice. */ if (boot_cpu_data.cpuid_level < 0x01)