From patchwork Thu Jan 25 11:28:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 192021 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2553:b0:103:945f:af90 with SMTP id p19csp1573740dyi; Thu, 25 Jan 2024 03:35:18 -0800 (PST) X-Google-Smtp-Source: AGHT+IE74N1WRkVyTnmUpcha04sXxRbNkZPksXrV7Q/X3lMo0plDJ2sxmiYrDU81Cdsf9ZLbHgB2 X-Received: by 2002:a05:6871:2b1c:b0:214:c782:8acb with SMTP id dr28-20020a0568712b1c00b00214c7828acbmr751907oac.27.1706182518267; Thu, 25 Jan 2024 03:35:18 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706182518; cv=pass; d=google.com; s=arc-20160816; b=lae09wxskBql8/jfotyAPRNnJzrZCOrf43ZPBzexFGTF9wUd6dMIC6deaGsafrNkgF ICtbZZtdAksOx5UbDirnIUF+RjVFRdPPWpU2mBliWHSgKBq3VK7GCkX1fNCN1QsgQd83 DN5A2Z5uQbUBZZYPY1It5butEo57OKDOSgyyjyrsQBAhLNPIC21imyFhY1/DI2F3vzvb XFWZqeI7H+UXyiuPBoovoDv649MjCtLkiGlZy2XnFZffNhR9cJqdWOQhq461JpPAhnGB uZwRyVTM5zkfphEUl4cppH+S/CxJEBBcwt2vugyBENeZW45uFxGU7YOuCAb36AP9w3z4 6vMw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=rJoL2/hFc4gkGtzEyAanpsFO97W15h/UiL7Bxz9040w=; fh=Mjd69IxHltS/Jba8SYEPy4aDMrScq2KBDxG2XjETtWk=; b=wyECZ64jcWzxxDzELYYVl47BJu5PRwSbMViWrca6FWf5sNNNXZCRdKF0LNgK5/OKpY 8klHhFihdkOqUm3BMdInI0PtOok0vf5/IexiLx4+1eZZAnVYegvYKUWJCAbL+wH9VmXA VGPvDfJ8FumqzmfC7gcEaXWo/NlQLxY1dtgNdLwn8HafbRcuEp7pBaio5DPhMcy4CQ5p IemMobQcXPZJC8sXmfjErkT/tV+lidD95/jJiknx9caLCL7XSObTmLyAH085nBwU6p5D Br12lS5wOvsx4lKvyq7r1d65hao/HI+n1Ae/h+FTHLETR1v3DVjZVK94NJ9jljWh4Gp2 fMQg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=mnH1Vm06; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-38509-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-38509-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id bs123-20020a632881000000b005d44d025258si3443347pgb.629.2024.01.25.03.35.18 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Jan 2024 03:35:18 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-38509-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=mnH1Vm06; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-38509-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-38509-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 5750E285888 for ; Thu, 25 Jan 2024 11:34:17 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 16A644F885; Thu, 25 Jan 2024 11:32:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="mnH1Vm06" Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F6A04655C for ; Thu, 25 Jan 2024 11:32:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706182371; cv=none; b=R7n1GaZXshBzVZwOdSd9siIsBpyKH6WI7MmyPdahYE25av7DnwFeY61PlB/JNRjyGIvLQUVzKSmTaw1y+Vaj7uVS8oc7g5LXRIab8dnHPLjbQmSsjb0sI+3kjfcZjByumyHDbZOYl7I917KJAOrJLuCj5nsFJFp9P+y7+rbUUBg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706182371; c=relaxed/simple; bh=g6w4YKbdWdUPESXVeFJEr3y36v0ES/R9hznFmwq1kAk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=k3O300VXClCh1cWvWKa+9zjE/PdUs4lQWz2ZIAxd1y61fV1SU0GJLKCy4VK/wRgA3pkk/OsOYDeoFioTFRWkPETBe9wAub2qJDvZu4l7pOggX9pYQUXJTbEsljHbnXVsWq+CTd+sFIjATKyMBahQbfARYkZx7kl9P73xKw2qwxU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=mnH1Vm06; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5ff84361ac3so7796337b3.1 for ; Thu, 25 Jan 2024 03:32:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706182369; x=1706787169; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rJoL2/hFc4gkGtzEyAanpsFO97W15h/UiL7Bxz9040w=; b=mnH1Vm06WU4xdC8JIDlrHVpu+RgHWAlQ3lbx6dgdi3Zsh01r7+gLgT0q2XLlfncgl/ Rbn9GGt7zgBMwyijWQ+1t60q/ULLQHFkz97jbxWD2+QpZxVInRQSD8pPsDFd/Q7ENqut /w3FmxdTn2Mqxf8NjMVckX5qyuhUTauirJY9lo5/cC3AKxNWrqmKmfKXN+UqOxjFRn5K ha2DQwzA8yPdI6h6BnddYSmJWeAepqVI2UAazsG9qeBFWo/DbZWXS3qDEPISWqLJ9JUV 4UhjM40ihUMaNwKbCXyN6QwKNZeWN13py64Y84n4spENfgR4k9SpjqMuSMd4yrsQsg9I dE1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706182369; x=1706787169; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rJoL2/hFc4gkGtzEyAanpsFO97W15h/UiL7Bxz9040w=; b=OroyN/e6Qu9SEXOYHvna9Kxypm0eWgUBfXg/Kq9IhZiB5j0pDa/b7GYW8zl4btrh0v IYITajAnbyLSoFMSX+nUaFBib04Zy6kplY8A3MmPN8xJrjoL0C6vp3y+0eGJxRC8pp42 tLkkO938rJxCwWxOzv2+XyApY7/HV91NlwhtSEeIMc76p/X3YSRiVZKOPh4U+gZ5pRtc AiI4bXdDJAKqX6YZFfK+1Wx3/yTfO7lhH95IRlc5tnt7OtGPxhxKq2hJB1Z++l5FCw0G e78DHEHAdXwTbE2jrbLpKMNl04PU6lYgBgi6IOJljwipvR6qT+AOmep3+K1iTP9G7sxR bmEA== X-Gm-Message-State: AOJu0YzUO5mo/MW1J88zOSFwFnmwaWIAsl+H8Vkoxrj7igahjsMB0iCS Y+ymV6I5NrW9sSiXodVd8K+TiDX6C3YIYs6qjg90SOsNxh9T57kQgIKuIaIo5uWhs82H5iHGxpn rhX3I/3b5C0O6BpHlDstP5n8RRQQjdj7bEhbSiO08CSxojoz/1oZmH0kwjp0PQJVmQTjG09gm8g dIIo++dZMl3+pvD7modLcqjWRu2vEzKA== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:2509:b0:dc2:51f6:9168 with SMTP id dt9-20020a056902250900b00dc251f69168mr447093ybb.2.1706182369248; Thu, 25 Jan 2024 03:32:49 -0800 (PST) Date: Thu, 25 Jan 2024 12:28:24 +0100 In-Reply-To: <20240125112818.2016733-19-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240125112818.2016733-19-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3319; i=ardb@kernel.org; h=from:subject; bh=t2swCS3/CT0QPpEAks7skAwmTJcH02OXVCoaa0u+RH0=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXWT6z2rFV3aYh/+hMbW2pT79zN9nPJ7uduxr3f6Xjc+2 PFLP721o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEyksYORYWFhVXFRQ+RcS7Hf vydLr3wt9ekEW7PGnoatB5+F3An6UMvw32GH9BnTl5Fz+AP4/srfvS7w5cfdXe5Tl62+rRU9Qes OPz8A X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240125112818.2016733-24-ardb+git@google.com> Subject: [PATCH v2 05/17] x86/startup_64: Simplify virtual switch on primary boot From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789062039748723630 X-GMAIL-MSGID: 1789062039748723630 From: Ard Biesheuvel The secondary startup code is used on the primary boot path as well, but in this case, the initial part runs from a 1:1 mapping, until an explicit cross-jump is made to the kernel virtual mapping of the same code. On the secondary boot path, this jump is pointless as the code already executes from the mapping targeted by the jump. So combine this cross-jump with the jump from startup_64() into the common boot path. This simplifies the execution flow, and clearly separates code that runs from a 1:1 mapping from code that runs from the kernel virtual mapping. Note that this requires a page table switch, so hoist the CR3 assignment into startup_64() as well. Given that the secondary startup code does not require a special placement inside the executable, move it to the .text section. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head_64.S | 41 +++++++++----------- 1 file changed, 19 insertions(+), 22 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 2d361e0ac74e..399241dcdbb5 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -39,7 +39,6 @@ L4_START_KERNEL = l4_index(__START_KERNEL_map) L3_START_KERNEL = pud_index(__START_KERNEL_map) - .text __HEAD .code64 SYM_CODE_START_NOALIGN(startup_64) @@ -128,9 +127,19 @@ SYM_CODE_START_NOALIGN(startup_64) call sev_verify_cbit #endif - jmp 1f + /* + * Switch to early_top_pgt which still has the identity mappings + * present. + */ + movq %rax, %cr3 + + /* Branch to the common startup code at its kernel virtual address */ + movq $common_startup_64, %rax + ANNOTATE_RETPOLINE_SAFE + jmp *%rax SYM_CODE_END(startup_64) + .text SYM_CODE_START(secondary_startup_64) UNWIND_HINT_END_OF_STACK ANNOTATE_NOENDBR @@ -176,8 +185,15 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) #ifdef CONFIG_AMD_MEM_ENCRYPT addq sme_me_mask(%rip), %rax #endif + /* + * Switch to the init_top_pgt here, away from the trampoline_pgd and + * unmap the identity mapped ranges. + */ + movq %rax, %cr3 -1: +SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) + UNWIND_HINT_END_OF_STACK + ANNOTATE_NOENDBR // above /* * Define a mask of CR4 bits to preserve. PAE and LA57 cannot be @@ -195,17 +211,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) */ orl $X86_CR4_MCE, %edx #endif - - /* - * Switch to new page-table - * - * For the boot CPU this switches to early_top_pgt which still has the - * identity mappings present. The secondary CPUs will switch to the - * init_top_pgt here, away from the trampoline_pgd and unmap the - * identity mapped ranges. - */ - movq %rax, %cr3 - /* * Do a global TLB flush after the CR3 switch to make sure the TLB * entries from the identity mapping are flushed. @@ -216,14 +221,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) movq %rcx, %cr4 jc 0b - /* Ensure I am executing from virtual addresses */ - movq $1f, %rax - ANNOTATE_RETPOLINE_SAFE - jmp *%rax -1: - UNWIND_HINT_END_OF_STACK - ANNOTATE_NOENDBR // above - #ifdef CONFIG_SMP /* * For parallel boot, the APIC ID is read from the APIC, and then