From patchwork Tue Feb 13 12:41:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 200417 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp507507dyb; Tue, 13 Feb 2024 04:44:16 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWGi50tVHOvOFf927WCz8Cm6Q15fJzuelLKfc1YbT27M8hxOeILdxCBJv3LNV6O7v66Vahdl/kxXYBH7EWWK7GNq2oprw== X-Google-Smtp-Source: AGHT+IHEcmt8KI86aIewozkc6zpYKaMR27WNHlKelVx9q6K4QIfJMTsQnNHaZKBIrpxofagU3nuu X-Received: by 2002:a17:906:34c7:b0:a36:c466:52ea with SMTP id h7-20020a17090634c700b00a36c46652eamr6573757ejb.75.1707828256155; Tue, 13 Feb 2024 04:44:16 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707828256; cv=pass; d=google.com; s=arc-20160816; b=fnx2+ZVi+kff/X5kHs12DFg30g2R8JfiBH9vVA+I+WD9z9bX3ZZvKNxNTE5ctlGETA EBk/OG7ATNQ4gqKRWrw5ddJXCuDnm9ATe1VdPNQUuPGtbvag5tUYU78Gnz6ll+bseg8Y IDzSTEsSpTkgwScgstxuYC6cFZ6xIUFTDppgS+Pz7306ppHlKrlOr3IannYiHjX05x1d JwLAKNI1/8fsPyeGpgnLimx63LmlEiIm79OyLSvf0hgmlhaxjoEvH1jCud8zwosuCkAU A8dcBpnYsKBSYiIY7lYZBcWBVP4dd43OHww+YnWeKqMZ8OF2D/prBRM7GL3UPTXGThNi uHPg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=l9PGLkbb8+qoug2ZrrnwKu6zhcSycBxmHA7JvTLYhJc=; fh=pT8WZEymctz/R/C04LxqwIz6sTk4RG4QlCrB6Gx50eo=; b=uMI+W5aWg+RM/KlknWr4D8IEuJf4WTQH4RuTsB5Gi7rkqnt5JLp+cU9nHnxHvbZB4f PBsTMH7AcDM+rtdz7SlE9mvMH9zoGB2rsRS7G4vXiKw6pb3iHYk8seTAZm1MF/bEb2UO wctuAWgI+UYKpuHjJmx+r96VyEiTP934NU9GhmhIrcylKjJMgeBscneQk+CT3ucQQHBe qUTIwjpdB28UeKICe0Ot6wYgVRm1zYwH1jG474Ue1sr51zRq1ILFMf9qVeVA4hC8DDN/ y2FK2kxaz7RES9+lb3h6nHDwroi9CL3S4cSxBZAcM7unTUxWK3bQM/QT/wQ1hIwRoAe9 Eraw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=r4kCWYai; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63526-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63526-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com X-Forwarded-Encrypted: i=2; AJvYcCVtABIIKRmEIwLvWYL8oC6mlF/JcXc4kG3Uh5VhHUFPIFpCRZiH9lnrBINPFw2iU2+5yDw5NIUSFXy5m41/L2ptwB8efQ== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id gw24-20020a170906f15800b00a38273c1419si1207319ejb.699.2024.02.13.04.44.15 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Feb 2024 04:44:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-63526-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=r4kCWYai; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63526-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63526-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 941F81F25398 for ; Tue, 13 Feb 2024 12:44:15 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2BDE7537EF; Tue, 13 Feb 2024 12:42:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="r4kCWYai" Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6058B52F7C for ; Tue, 13 Feb 2024 12:42:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828138; cv=none; b=AhEBAqwMi6u3t65HFhgWQJ6ocJv8dogOYm0t6J9gVIrXT17jLBOi3d2II3s3VCmE0omsQKBPKo7EP1os8lnEemCt6i6+pUTSpcXKENITWhxb/zNfd4Yk/l/CUCNggYxVdHp7c6yoYbgpDVdl3FSmmpcA7jNdDwDjaPb8ZHZ5YDY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828138; c=relaxed/simple; bh=V43RwcksINAVGLrcC9zNozM9sOBz5XCNbWyT4FUAcEc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Haw2Z0gHxWYmtl38lFbcLKkUNAliDFslA83AF/9iXs+fILKloYWMk9+dbmikjK9Q/yv112f6od7xpTdEMicR7TkKij6AXlBnloT9FhLDfL0/P4dRjWbsQpuXnwQEf8+LfYC8j4o+qbHV3c1gALcyPWGT45ZSo6uDbG/TrRqUn4s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=r4kCWYai; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dcc73148612so1072501276.1 for ; Tue, 13 Feb 2024 04:42:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707828135; x=1708432935; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=l9PGLkbb8+qoug2ZrrnwKu6zhcSycBxmHA7JvTLYhJc=; b=r4kCWYai+0z+MLg4WaOkHuJmakMAZ91VFPT+1f408MTofIGUD3p/+pL28YA0p6yjft owaf+C+2dj8bL8gawo2hWlD9l7eoy/9K62lSt96Iss77DODdXy2HVhVhwq6Twvh+Fxuv 1m9ToZNWOSH2epX0uxyGpfVC1F7kcuqu4LNPRACokkycXD5Dj4DtTOKUWsQNnUWhPeLT yrkRt6Ag8abtKIut6lCVdWUI7Yw2HRhMFU3wJzzwk4ND3gIAvNxcRINNUn1OT5IwzdQw bpFJW9eMqS6AwkzRpxF8rFSWDDoKdlTSmoaNznZyEw+YWCK7D44AeqE+hesjt82Vx6G5 68LQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707828135; x=1708432935; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=l9PGLkbb8+qoug2ZrrnwKu6zhcSycBxmHA7JvTLYhJc=; b=vjkgvJwK9s2qCw53kv0kl1McYliy2KhI8xetEj3ELcaF/g2RJrxW0DFx2cpUE3jfaw rC2Xmuu2cW3f/SxUMjgRCVdApvuvxdRdhnBhPGHbz0LoSXQsVqGvVdvksD5zC/7KhaF4 CX7fD51TeTvRn4YrAjbcpEw68Ghb9VhehkpmuUNJfx/LrJ17hZPi2pc8UZRf5u85WJ+D vEXQTf6wO1IxW6knO/7iS0GpjHWW2T9Uszfhbv/9awAbrGQAKJozSJCwFqELCwXWKFf8 5g06kghFXap8gQbFgtuiZOtg/hsGCUPv1LoL56QYVzhFQlWanaVQX/wbUQi6kiR5kS2w vrEA== X-Gm-Message-State: AOJu0YzZdhLm5cy1ci6a9T0XR8dsqSFr/yIFsEjrBK4iaGDoog8GPB/j 5wveed8AkzfJoaMI8tLUy7o40Bkjon0726GT0rdt8ogM9YvggWfuwzPWvb1CwNlW6MO9OLO6TYj pB1aMv4ScXshWGcOBb5C/iT1utk7FgMv2844EITg4IWf0nEzcHMoKhJh+bIzEOQRUFyAv4tOR29 y90nNOlNGIirNXcRXQv7D6BEXeq+lcTw== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:c702:0:b0:dcc:9f24:692b with SMTP id w2-20020a25c702000000b00dcc9f24692bmr44649ybe.13.1707828135418; Tue, 13 Feb 2024 04:42:15 -0800 (PST) Date: Tue, 13 Feb 2024 13:41:50 +0100 In-Reply-To: <20240213124143.1484862-13-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240213124143.1484862-13-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3664; i=ardb@kernel.org; h=from:subject; bh=CIHrtmfTIHf1nWX5y4HvnUdj+zcQXCO5URm4j2f8Yss=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfV0cl/Gzc7Pj0827Z2x7tTrdlcrJTWWepkfL7+1Tvbfv L0+de/ajlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRZENGhp1XJhx+OvtJ0OQd t66/4Tg4Z981saPGcaoTv3fmC4oZKr1gZNjyhGdrtnH11bPX511KKZxqfenAYamIjWsevWM+0XN NcyYnAA== X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240213124143.1484862-19-ardb+git@google.com> Subject: [PATCH v4 06/11] x86/startup_64: Simplify virtual switch on primary boot From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790787721226915954 X-GMAIL-MSGID: 1790787721226915954 From: Ard Biesheuvel The secondary startup code is used on the primary boot path as well, but in this case, the initial part runs from a 1:1 mapping, until an explicit cross-jump is made to the kernel virtual mapping of the same code. On the secondary boot path, this jump is pointless as the code already executes from the mapping targeted by the jump. So combine this cross-jump with the jump from startup_64() into the common boot path. This simplifies the execution flow, and clearly separates code that runs from a 1:1 mapping from code that runs from the kernel virtual mapping. Note that this requires a page table switch, so hoist the CR3 assignment into startup_64() as well. And since absolute symbol references will no longer be permitted in .head.text once we enable the associated build time checks, a RIP-relative memory operand is used in the JMP instruction, referring to an absolute constant in the .init.rodata section. Given that the secondary startup code does not require a special placement inside the executable, move it to the .noinstr.text section. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head_64.S | 42 ++++++++++---------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 6dcc2f7f4108..3fed0aafcb41 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -39,7 +39,6 @@ L4_START_KERNEL = l4_index(__START_KERNEL_map) L3_START_KERNEL = pud_index(__START_KERNEL_map) - .text __HEAD .code64 SYM_CODE_START_NOALIGN(startup_64) @@ -126,9 +125,21 @@ SYM_CODE_START_NOALIGN(startup_64) call sev_verify_cbit #endif - jmp 1f + /* + * Switch to early_top_pgt which still has the identity mappings + * present. + */ + movq %rax, %cr3 + + /* Branch to the common startup code at its kernel virtual address */ + ANNOTATE_RETPOLINE_SAFE + jmp *0f(%rip) SYM_CODE_END(startup_64) + __INITRODATA +0: .quad common_startup_64 + + .section .noinstr.text, "ax" SYM_CODE_START(secondary_startup_64) UNWIND_HINT_END_OF_STACK ANNOTATE_NOENDBR @@ -174,8 +185,15 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) #ifdef CONFIG_AMD_MEM_ENCRYPT addq sme_me_mask(%rip), %rax #endif + /* + * Switch to the init_top_pgt here, away from the trampoline_pgd and + * unmap the identity mapped ranges. + */ + movq %rax, %cr3 -1: +SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) + UNWIND_HINT_END_OF_STACK + ANNOTATE_NOENDBR /* Create a mask of CR4 bits to preserve */ movl $(X86_CR4_PAE | X86_CR4_LA57), %edx @@ -194,16 +212,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) btsl $X86_CR4_PSE_BIT, %ecx movq %rcx, %cr4 - /* - * Switch to new page-table - * - * For the boot CPU this switches to early_top_pgt which still has the - * identity mappings present. The secondary CPUs will switch to the - * init_top_pgt here, away from the trampoline_pgd and unmap the - * identity mapped ranges. - */ - movq %rax, %cr3 - /* * Do a global TLB flush after the CR3 switch to make sure the TLB * entries from the identity mapping are flushed. @@ -211,14 +219,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) btsl $X86_CR4_PGE_BIT, %ecx movq %rcx, %cr4 - /* Ensure I am executing from virtual addresses */ - movq $1f, %rax - ANNOTATE_RETPOLINE_SAFE - jmp *%rax -1: - UNWIND_HINT_END_OF_STACK - ANNOTATE_NOENDBR // above - #ifdef CONFIG_SMP /* * For parallel boot, the APIC ID is read from the APIC, and then