From patchwork Mon Feb 26 14:29:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 206679 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2113455dyb; Mon, 26 Feb 2024 06:31:28 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUHjy6CQg73+/rhC7IwzDs0oFN0LURf8hrJwnpaFxb3Qjega9GEOBZzD3ZsJkMT1Kx0/Es3u/hgRMc6bfaifmMND6XyTQ== X-Google-Smtp-Source: AGHT+IH/Pu7o9i2auxPNyDrZbKsT3pFjpQlflxoylU8HX6WETmE6N24P14mw2LA49ABnWpguE4jH X-Received: by 2002:a05:6102:3713:b0:470:44e4:27d0 with SMTP id s19-20020a056102371300b0047044e427d0mr5457012vst.6.1708957888536; Mon, 26 Feb 2024 06:31:28 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708957888; cv=pass; d=google.com; s=arc-20160816; b=chQ0jCtHuU8b3vQCEXKbtWrXefLjf9dvk/LjyhGt2pI2zJFe1KYT2Ed6PEZ7uYOEYT b4TdFEZac0HatbhQXdoXzb3FCdsrqg4xcoKs1ns/LwzMaQe/n3maJaiNrlKcm4k0msLK YJnHZcY7EU29o68FBs23flYX39dyydJNikFuSyRZnh7GyrvupH2zFTjtOspCYTgTf59g ktwE2ff+SAmohUK/p0JOy2EGfLdYqNHxcQPRUTiOq6V0mAyeQyOtH3XlINL7BN420EPj 9ehKlzvsgOibuDd2gnpuOlZ+1CjDpWYd6kT05EzAATNAiBJvJ9Zz+1mvmy6D82E8SsDr VrCQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=HUEqNAX+crIm6ixnlTe40pRvCdb8ZI8NNnngiA9aRDQ=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=e9Gz3mZtzrgczCOhVeQ267P/bac/4/GJFLvgeJ2RO7h16P3i+HJ2fj3/ieMJU3++y9 J/8aBdaaid9fe4kM41i0vzMs7u9yTEHF22vRWK5gt+RO2iL44AUmO9tMZpS5lpnBesWa YvlZNVF/NRlEOVQNaplLQQ7Rs3kfXw+BLBTTRyh+wqiURdsFjEf0s9+PtUmBHZEd279j nChFb8FGZ0/cBbiSYeMXdLoxVeC/FJK1JiHrR8YV9wwji0oewiENxQLL9BHDyjjZRi/w Y2CorLToqv/HcGiNe5YRoYioXxr4dPfhJJBfd5Hl1Z6bRHoGB4UoRIz24tPmNh0ph1tw oOQw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=NJFT1AwJ; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81625-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81625-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id gd8-20020a056102624800b0046d21d6e822si562754vsb.133.2024.02.26.06.31.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 06:31:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-81625-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=NJFT1AwJ; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81625-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81625-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id DC02C1C21D74 for ; Mon, 26 Feb 2024 14:31:24 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7230D12C520; Mon, 26 Feb 2024 14:30:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="NJFT1AwJ" Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8330F53E28 for ; Mon, 26 Feb 2024 14:30:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957812; cv=none; b=WbS8kGoMNNFwjq9dDxFcEDdEIm45u9jW9QdHVHXQBQKTpe0qIDplooPykf+fgi3tfQ3tQe7e/IWPNjd3F6V1RdQL1epseP4xsV+EH+oY9zEmSxVn9VFoicw5ZooMtangwCV4yM8GIPRj2zZ8DPDVhPaeGJMdgXF282EFj+Y4vnE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957812; c=relaxed/simple; bh=+SAmI2opGAKocGzeZEPFdFE24u4ggFbULQvVpI+lmmo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=FEcjRvUvaCUa6zvACdkIMjUrnAsyrlgderKZ6eE31urFyCzvHDi9XWa3LAB2uKc4Yv8iiMETAKnBHX++6F4yedzNXX1qJQfnB5E7pnCR4BZwzpGbKVOGNQAhRK9G1yNpGlHRrr9sLr0nqGku5xH/UcxbICMzVUuLIsklZE8qynQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=NJFT1AwJ; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-33d6eb5e5a9so2319021f8f.3 for ; Mon, 26 Feb 2024 06:30:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708957809; x=1709562609; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=HUEqNAX+crIm6ixnlTe40pRvCdb8ZI8NNnngiA9aRDQ=; b=NJFT1AwJg/QdP/N9y9wIu3Y+VcOPAgca8O+z7aYJPaY/Widy/qcYsaFvx3sNc7ozB9 iyrkE47WtIPAzkP/ng22SeUz7U04syim6dTIeMc1NGXnW7zblRPGY4YJycmjNnwRR6ty KpXydcgDbm2lpfSMsglYSXzYnGXZn0skReqtQqaOqcWgicZvpRwFG+h8dVcdd1u13KYp EOpZ55cLzAaO0dliXZEBx0NOm53lf3qonIOA4uQOip66nk5Oa5W2TEYWlptbScHKb2YY a+u/neMpPxTr1l5BtBlG8fE+J1htEUEDgXSJ+zXEWNefqBftidoOqX1fv2kR6V3jUY8j Lmig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708957809; x=1709562609; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=HUEqNAX+crIm6ixnlTe40pRvCdb8ZI8NNnngiA9aRDQ=; b=G+H3VtGznaOkZnuGY7VfmD4IaXUY9Huy1iTIN06NqZ0T7fqG1JJHANp6hHXg2UMHlY nlW6xUHRHt3Ova7n4maUJW6WR/lNMzFYUkoR+2QTvqb5GlRRjWRomhhR1y/GVljbWab7 6sMP93x/uklIUfdZ2YTNO3EoNHTWbOm5keoHL6FGTeMKsz4MLdgEAfaF+CIWm9Mzdqvj nwxX03SEPw7zL3myVbtFWCPJDdToaOOUJQfvtpY+nQUIYor86LEi5x6Qphv6VJ7qClZj 2LapMtGlvaxmWlvsi6SJ9qvgensY4bz4fg63tlYeED77zgA61DJRzjkoKLUdSVbNXvRu hWag== X-Gm-Message-State: AOJu0YzKbOY7nXiHRqK9NEukYsxRYzLhS3AlzJC5RtPAmR2OCWiKxifW EUGvlcEX+yK9etIDca8/+GG2MFnjX5d6nnybJ5JYlIXbmIbSAlQO+OQkdzyE+IX8KGpTty20hMl dm5Pf2G21PqfzIYF3Kk+3LV1IPQQmTh2lP4W81EYdyaCHQvDMcoJyTYfl7k4GNSchCXeGFuTKWC cK8cTIAJTnexNnED1Q8Q6B6LXn9IvOaA== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:5d85:0:b0:33d:ae58:8e1e with SMTP id ci5-20020a5d5d85000000b0033dae588e1emr14897wrb.12.1708957808755; Mon, 26 Feb 2024 06:30:08 -0800 (PST) Date: Mon, 26 Feb 2024 15:29:57 +0100 In-Reply-To: <20240226142952.64769-12-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240226142952.64769-12-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240226142952.64769-16-ardb+git@google.com> Subject: [PATCH v6 04/10] x86/startup_64: Simplify virtual switch on primary boot From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791972227434425582 X-GMAIL-MSGID: 1791972227434425582 From: Ard Biesheuvel The secondary startup code is used on the primary boot path as well, but in this case, the initial part runs from a 1:1 mapping, until an explicit cross-jump is made to the kernel virtual mapping of the same code. On the secondary boot path, this jump is pointless as the code already executes from the mapping targeted by the jump. So combine this cross-jump with the jump from startup_64() into the common boot path. This simplifies the execution flow, and clearly separates code that runs from a 1:1 mapping from code that runs from the kernel virtual mapping. Note that this requires a page table switch, so hoist the CR3 assignment into startup_64() as well. And since absolute symbol references will no longer be permitted in .head.text once we enable the associated build time checks, a RIP-relative memory operand is used in the JMP instruction, referring to an absolute constant in the .init.rodata section. Given that the secondary startup code does not require a special placement inside the executable, move it to the .text section. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head_64.S | 42 ++++++++++---------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index c451a72bc92b..87929f615048 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -39,7 +39,6 @@ L4_START_KERNEL = l4_index(__START_KERNEL_map) L3_START_KERNEL = pud_index(__START_KERNEL_map) - .text __HEAD .code64 SYM_CODE_START_NOALIGN(startup_64) @@ -126,9 +125,21 @@ SYM_CODE_START_NOALIGN(startup_64) call sev_verify_cbit #endif - jmp 1f + /* + * Switch to early_top_pgt which still has the identity mappings + * present. + */ + movq %rax, %cr3 + + /* Branch to the common startup code at its kernel virtual address */ + ANNOTATE_RETPOLINE_SAFE + jmp *0f(%rip) SYM_CODE_END(startup_64) + __INITRODATA +0: .quad common_startup_64 + + .text SYM_CODE_START(secondary_startup_64) UNWIND_HINT_END_OF_STACK ANNOTATE_NOENDBR @@ -174,8 +185,15 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) #ifdef CONFIG_AMD_MEM_ENCRYPT addq sme_me_mask(%rip), %rax #endif + /* + * Switch to the init_top_pgt here, away from the trampoline_pgd and + * unmap the identity mapped ranges. + */ + movq %rax, %cr3 -1: +SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) + UNWIND_HINT_END_OF_STACK + ANNOTATE_NOENDBR /* * Create a mask of CR4 bits to preserve. Omit PGE in order to clean @@ -199,30 +217,12 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) btsl $X86_CR4_PSE_BIT, %ecx movq %rcx, %cr4 - /* - * Switch to new page-table - * - * For the boot CPU this switches to early_top_pgt which still has the - * identity mappings present. The secondary CPUs will switch to the - * init_top_pgt here, away from the trampoline_pgd and unmap the - * identity mapped ranges. - */ - movq %rax, %cr3 - /* * Set CR4.PGE to re-enable global translations. */ btsl $X86_CR4_PGE_BIT, %ecx movq %rcx, %cr4 - /* Ensure I am executing from virtual addresses */ - movq $1f, %rax - ANNOTATE_RETPOLINE_SAFE - jmp *%rax -1: - UNWIND_HINT_END_OF_STACK - ANNOTATE_NOENDBR // above - #ifdef CONFIG_SMP /* * For parallel boot, the APIC ID is read from the APIC, and then