From patchwork Wed Feb 21 11:35:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 204166 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp978469dyc; Wed, 21 Feb 2024 03:37:33 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUx9LggdAVJ2X4zJO840TwzobHau6nKS9RqtqTJHtUqq3KytNF1/NEKrHQCxaj7k2TKtsicMjQrXPgFjUwyJ7cSYVrVtg== X-Google-Smtp-Source: AGHT+IGo0c1sa5oU+lBWIoR+c56y/NXPnUzMarZVm17O+wqV0eFJ7/+XmfhBNNUp9/U0YEqRBrYN X-Received: by 2002:a05:6a00:4fd1:b0:6e0:41fa:7a15 with SMTP id le17-20020a056a004fd100b006e041fa7a15mr20777315pfb.22.1708515453704; Wed, 21 Feb 2024 03:37:33 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708515453; cv=pass; d=google.com; s=arc-20160816; b=JvE1AdU8MlsvI5O/k9jTqMhXouNS7z6pJZlzKU5itE/86SCehcD2KDuzrfdbf/KVnb oCSa4H6LyU517y9ZKUlROz2f1XAE9y8nvFT7uFSSZ7yp0ctVPMzA9fdGy99hIaiyOX8L XOcgRpk9/DxXS+EInQ9VAhAjqI0y0fujI9ZtpNm3wTVpGKLOEOhJujQIvCMS/V4sw0H5 9I4nDsxY6WaVOib8FllK89BBFiRDyPNCZ8XSCDjOZQLAdQSd4Z6HQRkov2cR35Mt+vlA TDLlAnBsLvkLwO6Y0Oicekt9jnU4YRhRUfuQphO6zWanUDBbOkEr8clfpSlTUhNpLVsW j+Rg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=qW/A0wjJqrnttxBCkESXpcpyFK56vfk0Gs3tCOMJv4Y=; fh=fMhMPvo8duafGkM4xZHdH7fULbau3etefTGsKa+Qy7I=; b=0WYaS9tOcdm5CNp3gh8L1POASXy0LaR1gHvSaK1eIpO364ViSUngOyBpBFbDneuUFV 1WCdvDAfgM0FjQp5V2NGCtOVsSYNzr4MCZvdyW1HGVHKtnI4sCIrOwO97pT65Ju8ezO4 0Rt4e9hmYVvsy3SdLJJf0SwqAGJLkl6FMg74cs1Te8sQO/E8G4zhYsvS1yVLi0U3cWc3 ovnpcGMdQXzYk1EyGE5meXi+A0qy307neoNje7HurVOMbZLaqOL6haQgm7CcjeV9nGss v5Tzj2XSb18qnsyXmuHNiUKi3EWmIRY69TzVIsEJ9g/qoR3AGf4e9CWod3ePzkWAfCHQ B3lA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=1f2YhjBm; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74631-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74631-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id y124-20020a636482000000b005d8505c96e1si7869757pgb.423.2024.02.21.03.37.33 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 03:37:33 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74631-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=1f2YhjBm; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74631-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74631-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 96BF6B23E5B for ; Wed, 21 Feb 2024 11:37:04 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id AAAB1487B5; Wed, 21 Feb 2024 11:35:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="1f2YhjBm" Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 41E6145970 for ; Wed, 21 Feb 2024 11:35:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515339; cv=none; b=shH3O+rnizTsgznP205mbpgp+98yB+ujL/zMn2sNgbMkEu+LieAbwQmmOSntmCnAc5GTU1/QQb/+j7GLmMgsz6QKCg98hHpgrgbwbapt7dIyRPDJ5hzTIz/G5OzYlTfl/4T9jy9dNvgCBRP4LirunC1J/sT3ddUuicYugSHCnqA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515339; c=relaxed/simple; bh=py9Y+djzPQO6EmF7F9NJ1i5hAa8K3oSBQk/6uhAH298=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=gPUPLE0oPDHH2XP3RhtuocBLzffz7qRkUcnl4PHg80RtbSOIdYSxw6ezvpOYKlA4I8K44mRzODamBXC0aBtjXhB5avoB5Snnz4Jps5ihfti3ykdm2Hpk5WpLQta9LcVJlwdja3ZrQSPY2SjD5hAliUwIPlJ7Vqo3E6Zm8mVhyyA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=1f2YhjBm; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-608749ea095so17629897b3.0 for ; Wed, 21 Feb 2024 03:35:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708515337; x=1709120137; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qW/A0wjJqrnttxBCkESXpcpyFK56vfk0Gs3tCOMJv4Y=; b=1f2YhjBm6qxcDIoKgvJYITX4UJSnUl2ga3VSCfiEWq2akVTbwHmciSfRplrv7kJsJc isidw9jqQUoKrnCgNXNxFLtce0n2e0Rwy4ATkD7eAzaQySqK9S0G9gXD5t0qes7fd7rj L9k1W/7q/qRkly/cmuG5xY8vPU2WwjcoBNV4xRoMwwP1ZXQAXUXV1S1lS0WK4ME9MROW vylr6uVZWkIF4p/m4BSZd0ULedY/orp9K3cBt7ujVaexWpagTSE0hhQSaBZpcdCWE4BX X0JUd0i+lE0mSWMqglwK6gnGy90MYplm1x9C2LJD3u14DFQvMndPZEHRAcHxn9pGf5q3 ZvaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708515337; x=1709120137; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qW/A0wjJqrnttxBCkESXpcpyFK56vfk0Gs3tCOMJv4Y=; b=QaXXHRcRWcg7UihwSAFoVXGW0cMl35TUrh0c1pSw7iZ546WQG8Jg5Zh6C9gJH48dWR PGlq4M7/M+84AmvZ2ntBJ4ndTLfUxucgH3mrp+OeeKcxpkUPrzqg2Ic98IxKL6uiNDmI pOhM7+0qQFGWONGYeVv+fE7evAMyKrscijAJJA/JVBM291U/9ibTWpzI1pBXHKgxmXfQ 8gE/GuCuY2m+Arop9dMl2GjNreYgDp9SJ2h5vteqDO7BvO9BfV1gkzhfxOzAuBF1puI/ y71F9Ut+MNrqlrTGE7/ncU1EBeCH+HhjNrCGX6l8mH6BDvV4R8OqlJXIhTiO2igAuj+U YCEA== X-Gm-Message-State: AOJu0YxbcrINbaDowDBSeN/H+qpvjVWWslIFdJWM8rCpFdBqr1vxUEHO TrnK+EyJ/hbrMzBae5A//tWCmWip5/jgtRdBLlXYV6np8uAfq7mf2UO0FNFxi/uTxpXeS6NdidK aqibGQ54DgB8gWhlvyCNi8GmAwlU0J9iz4EANC2KzpxSYmR4zXepQIiBAJ5udMuNmNftA7B8yoV a43LVPnzmuEdp4Uikc4tzy1okPs9tgNQ== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a0d:e904:0:b0:607:8f65:5433 with SMTP id s4-20020a0de904000000b006078f655433mr4327050ywe.4.1708515336661; Wed, 21 Feb 2024 03:35:36 -0800 (PST) Date: Wed, 21 Feb 2024 12:35:08 +0100 In-Reply-To: <20240221113506.2565718-18-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240221113506.2565718-18-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6347; i=ardb@kernel.org; h=from:subject; bh=znXuPAQkSUWGkxlryxVinIdceSOSj8FHbeDMB1myZ+Q=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXq/dflrwLZVwhePf5QQEj9QfXG/7v/HGuIPGUQaL4k9 aSVsZJ7RykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjIPj9GhvVz7Ln//KirE/HO rxD0imif0jvh897z5Y2batlfK2p+dGVk2LPefEc0wx0r3Q1pNjeEfbl4ux32JMmnPYsL7BZdLKv JBwA= X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221113506.2565718-19-ardb+git@google.com> Subject: [PATCH v5 01/16] x86/startup_64: Simplify global variable accesses in GDT/IDT programming From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791508300437027560 X-GMAIL-MSGID: 1791508300437027560 From: Ard Biesheuvel There are two code paths in the startup code to program an IDT: one that runs from the 1:1 mapping and one that runs from the virtual kernel mapping. Currently, these are strictly separate because fixup_pointer() is used on the 1:1 path, which will produce the wrong value when used while executing from the virtual kernel mapping. Switch to RIP_REL_REF() so that the two code paths can be merged. Also, move the GDT and IDT descriptors to the stack so that they can be referenced directly, rather than via RIP_REL_REF(). Rename startup_64_setup_env() to startup_64_setup_gdt_idt() while at it, to make the call from assembler self-documenting. Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/setup.h | 2 +- arch/x86/kernel/head64.c | 75 ++++++++------------ arch/x86/kernel/head_64.S | 4 +- 3 files changed, 32 insertions(+), 49 deletions(-) diff --git a/arch/x86/include/asm/setup.h b/arch/x86/include/asm/setup.h index 5c83729c8e71..e61e68d71cba 100644 --- a/arch/x86/include/asm/setup.h +++ b/arch/x86/include/asm/setup.h @@ -48,7 +48,7 @@ extern unsigned long saved_video_mode; extern void reserve_standard_io_resources(void); extern void i386_reserve_resources(void); extern unsigned long __startup_64(unsigned long physaddr, struct boot_params *bp); -extern void startup_64_setup_env(unsigned long physbase); +extern void startup_64_setup_gdt_idt(void); extern void early_setup_idt(void); extern void __init do_early_exception(struct pt_regs *regs, int trapnr); diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index dc0956067944..1d6865eafe6a 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -22,6 +22,7 @@ #include #include +#include #include #include #include @@ -76,15 +77,6 @@ static struct desc_struct startup_gdt[GDT_ENTRIES] __initdata = { [GDT_ENTRY_KERNEL_DS] = GDT_ENTRY_INIT(DESC_DATA64, 0, 0xfffff), }; -/* - * Address needs to be set at runtime because it references the startup_gdt - * while the kernel still uses a direct mapping. - */ -static struct desc_ptr startup_gdt_descr __initdata = { - .size = sizeof(startup_gdt)-1, - .address = 0, -}; - static void __head *fixup_pointer(void *ptr, unsigned long physaddr) { return ptr - (void *)_text + (void *)physaddr; @@ -569,62 +561,52 @@ void __init __noreturn x86_64_start_reservations(char *real_mode_data) */ static gate_desc bringup_idt_table[NUM_EXCEPTION_VECTORS] __page_aligned_data; -static struct desc_ptr bringup_idt_descr = { - .size = (NUM_EXCEPTION_VECTORS * sizeof(gate_desc)) - 1, - .address = 0, /* Set at runtime */ -}; - -static void set_bringup_idt_handler(gate_desc *idt, int n, void *handler) +/* This may run while still in the direct mapping */ +static void __head startup_64_load_idt(void *vc_handler) { -#ifdef CONFIG_AMD_MEM_ENCRYPT + struct desc_ptr desc = { + .address = (unsigned long)&RIP_REL_REF(bringup_idt_table), + .size = sizeof(bringup_idt_table) - 1, + }; struct idt_data data; - gate_desc desc; - - init_idt_data(&data, n, handler); - idt_init_desc(&desc, &data); - native_write_idt_entry(idt, n, &desc); -#endif -} + gate_desc idt_desc; -/* This runs while still in the direct mapping */ -static void __head startup_64_load_idt(unsigned long physbase) -{ - struct desc_ptr *desc = fixup_pointer(&bringup_idt_descr, physbase); - gate_desc *idt = fixup_pointer(bringup_idt_table, physbase); - - - if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) { - void *handler; - - /* VMM Communication Exception */ - handler = fixup_pointer(vc_no_ghcb, physbase); - set_bringup_idt_handler(idt, X86_TRAP_VC, handler); + /* @vc_handler is set only for a VMM Communication Exception */ + if (vc_handler) { + init_idt_data(&data, X86_TRAP_VC, vc_handler); + idt_init_desc(&idt_desc, &data); + native_write_idt_entry((gate_desc *)desc.address, X86_TRAP_VC, &idt_desc); } - desc->address = (unsigned long)idt; - native_load_idt(desc); + native_load_idt(&desc); } /* This is used when running on kernel addresses */ void early_setup_idt(void) { - /* VMM Communication Exception */ + void *handler = NULL; + if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) { setup_ghcb(); - set_bringup_idt_handler(bringup_idt_table, X86_TRAP_VC, vc_boot_ghcb); + handler = vc_boot_ghcb; } - bringup_idt_descr.address = (unsigned long)bringup_idt_table; - native_load_idt(&bringup_idt_descr); + startup_64_load_idt(handler); } /* * Setup boot CPU state needed before kernel switches to virtual addresses. */ -void __head startup_64_setup_env(unsigned long physbase) +void __head startup_64_setup_gdt_idt(void) { + void *handler = NULL; + + struct desc_ptr startup_gdt_descr = { + .address = (unsigned long)&RIP_REL_REF(startup_gdt), + .size = sizeof(startup_gdt) - 1, + }; + /* Load GDT */ - startup_gdt_descr.address = (unsigned long)fixup_pointer(startup_gdt, physbase); native_load_gdt(&startup_gdt_descr); /* New GDT is live - reload data segment registers */ @@ -632,5 +614,8 @@ void __head startup_64_setup_env(unsigned long physbase) "movl %%eax, %%ss\n" "movl %%eax, %%es\n" : : "a"(__KERNEL_DS) : "memory"); - startup_64_load_idt(physbase); + if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) + handler = &RIP_REL_REF(vc_no_ghcb); + + startup_64_load_idt(handler); } diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index d4918d03efb4..3cac98c61066 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -68,8 +68,6 @@ SYM_CODE_START_NOALIGN(startup_64) /* Set up the stack for verify_cpu() */ leaq (__end_init_task - PTREGS_SIZE)(%rip), %rsp - leaq _text(%rip), %rdi - /* Setup GSBASE to allow stack canary access for C code */ movl $MSR_GS_BASE, %ecx leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx @@ -77,7 +75,7 @@ SYM_CODE_START_NOALIGN(startup_64) shrq $32, %rdx wrmsr - call startup_64_setup_env + call startup_64_setup_gdt_idt /* Now switch to __KERNEL_CS so IRET works reliably */ pushq $__KERNEL_CS From patchwork Wed Feb 21 11:35:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 204168 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp978561dyc; Wed, 21 Feb 2024 03:37:47 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVJcVjE1zaGRj3PODAzxXMn8eGdHQNMVHC9qlO/G31Lo58LyjhkjFvOQs9++ugyERGT9/8GhQWvb3lE0kO62qdTemu40A== X-Google-Smtp-Source: AGHT+IETutsDe+m1/stNkWncmK4aLflFgG1netiCsY4gqAGYAXpBh46XN0mzo0DjhVM1q3hlhqcU X-Received: by 2002:a9d:69c4:0:b0:6e4:3775:683b with SMTP id v4-20020a9d69c4000000b006e43775683bmr14356946oto.34.1708515466996; Wed, 21 Feb 2024 03:37:46 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708515466; cv=pass; d=google.com; s=arc-20160816; b=NYcdsNehJPAxtjVdH1B9UQWRzWyE9b29C4ZkIOiV4wu+VvuftRDJMeoCpSjfLVdB4n xKCOpCGgG2/ceMXGcepFT3fIugHJ+Q+lMAzvBKVcS4aqU7BaKlcgoQoxY3gk9yjWJEpN TStC710IS5hTTh3D4FgTKxBMZlzagR1yWlictTY1d/pJr0REz1+UagQjy+LItQGy8ftC J7YD8vPPRPFslRH91GGWyaU6NJPOowsQAepnq2sPwZTeF58djHChc5oAZNcS+J5ZTKeG sYDIdGk4X8vlvMreG8foZJ8FYLlGIobWjIujzkaM/986fZrfISf+VmBiCHyWgBI5L99o sU5A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=gwOWSXeXshx2pDY4YsRb8c4OJtdf4hRb8S0itmSW07o=; fh=fMhMPvo8duafGkM4xZHdH7fULbau3etefTGsKa+Qy7I=; b=nWxcs5cGJwFaXojqZY6DiycVjMvYwanz0HYMz0gNoQ1GQ8Y62bnf9KdPDc/9KDeTIp j6ZWiDk9RD5PK5Vk9CiLc+RwNWjXZGMHM25Zxj8tHBhuuJ2eJVWmMXO0jLVgml1/t9Ha USEVz+/ot8ueEkSJLoTiZbZhvmEgt5OEsl0MUgxvakHiMBO+tcZpciCkC9lT7p56ahio HqO7U9C3HO8b8/kSeslyo1pQgggz02GGJlXJY+cS1Yu8njUWdlbV79/F2QjFwpAOLagk 32ooB8TXxjoCAcdpyIiRr4Z1ibRtrj2oAePjq42+szDu2aPR2n1S9/s1Auncp8ArnBpO zUtQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=GFm1YhzK; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74632-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74632-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id b22-20020a637156000000b005e1c5c9a11esi6045819pgn.230.2024.02.21.03.37.46 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 03:37:46 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74632-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=GFm1YhzK; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74632-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74632-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 09FF8B241AD for ; Wed, 21 Feb 2024 11:37:16 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id AFC034C630; Wed, 21 Feb 2024 11:35:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GFm1YhzK" Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E8B6481B4 for ; Wed, 21 Feb 2024 11:35:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515342; cv=none; b=jT+Xl97dQuJIdwVUimGXz7MamZxLlly5oWiR0sINxodwBmc/Tjw0S5for6DG6/3Ksi7v0ev37FVo1qSg+tUTxtUn64jxcRz4ZOhfRGXo/aEnFo/H7yi30ufWqXRySY/quGjiXa9vckLuv0sR9zDSEo+T62Cnf0rUJ78dv4X3Il8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515342; c=relaxed/simple; bh=/B1OP6vbuA5phGZhjfg4uQ2vHMJ2nJmV3L2eYZwfJcM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=T+YILn2zkPvEeng6U89PlhnP3x0JR2LKa2oEHtgKPZjk5/IR4fECChnuk0KoFVLC5MIt4BjDCwZhGF7uKmdX6AzdBKBwoUeZZIZzJ+IaQFPJ8dyqLumHWcTqD1K/f7/lSv8vqBm6D/AGAAvl72A6aBpLy2W8owY22WARQzBYhkw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GFm1YhzK; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-33d6eb5e5a9so860064f8f.3 for ; Wed, 21 Feb 2024 03:35:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708515339; x=1709120139; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gwOWSXeXshx2pDY4YsRb8c4OJtdf4hRb8S0itmSW07o=; b=GFm1YhzKXCOxWJ742d1dqiKQB7zcXqKlcJY29YH5vgmp2CT2n/lZZ7q5jZ//A8FmYu UOY+2+kkyFTpnIWYAorBz47xlqkdP9wiO2VjdIp18JRtXbY/S5wP3cvhcr1LdPKwXweE lIlRYHcnx4oS4uhh3urfQ2jCVxBuvri37Q2nTInwL7lA4zecL85vrHaYop89H2iDurNz 34ae5Cg6xoeYRxtTQ1Ifd08bC3pfUKcYgNC2TB16kYFBBEMH8lvVzTzTst+Nf+bWWIHs grcfCCnmkI5JjRuWPky9iaBOzDqyNCFyeA9jx9QOexBMhlZguS6A9SJHJLpEhqGQRnAX 42QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708515339; x=1709120139; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gwOWSXeXshx2pDY4YsRb8c4OJtdf4hRb8S0itmSW07o=; b=D4v/NljHIixoaJgbnQd00P7yZqQrRS042piSvPSlXjsWlm64vr1c3KeVjoVBckDH+B a4E03nDFY1zBZ8i77nD2RFIFaqn0YdxJEJnkD9QU93RzS6Zms6QELi2NFRQjR6XSexKZ F4DbfREoj28z8O3vbD5wgnlJRbgrSnXlr5aw6evDOhLm+ciYviE8zyoByQEeRcBqqFDL 9ELXh/A5WRUQbI7W1unaypLiSIvgTdHO+1FpEo4Xiniz5Ddr9N3xPimvjca5j7uLD1eA iC+IDl80iHbTbtIBn0cLFXtHAOHg7TqIIR3n2MmoicgYHZspaFo1HABt3o8fXZDnHD0h t2hg== X-Gm-Message-State: AOJu0YxvrnP8CHqIKAdJ0wWWqrYu84h1Ue2HUyuijoGHDw/WETii0T/4 J6W1Ll2+NJqEyg4qlDTVQouCsowtio5Yu3J5ltepAnbfWh5GbcRP7j7Qtmy1i4j0GLEkdVFS/Q8 lu54rFH4rDGqwcefvUOcHFNRvNibNtdU2YeXciyiBH1pgCNAz0c3ZJeDsfbOM8CpnZAkZQY1Gvy EKSnyK+fJn6HmIla4bXxnX0QteNWwxVw== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:1c03:b0:33d:804f:7bb3 with SMTP id ba3-20020a0560001c0300b0033d804f7bb3mr2030wrb.7.1708515339021; Wed, 21 Feb 2024 03:35:39 -0800 (PST) Date: Wed, 21 Feb 2024 12:35:09 +0100 In-Reply-To: <20240221113506.2565718-18-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240221113506.2565718-18-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1451; i=ardb@kernel.org; h=from:subject; bh=HV5cVzg2bsj7zm/lufPEjuRMFtl5r6GlxXU96Wc4Zjs=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXq/TfS/Ie8Tyfs4TiZ2dY8/eqXuOJb6hbTC/44bpZM/ NZ6Myeko5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAExk2llGhlXHMqck8tmt/CX0 Ov6+atSy9U7+7FpOJUqvY+44crG28jH8D31+U3aHolG6bniHk2Phz7Pu3be/MxQ4L/q8NmmSX3w YDwA= X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221113506.2565718-20-ardb+git@google.com> Subject: [PATCH v5 02/16] x86/startup_64: Use RIP_REL_REF() to assign phys_base From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791508314537331794 X-GMAIL-MSGID: 1791508314537331794 From: Ard Biesheuvel phys_base is assigned from code that executes from a 1:1 mapping so it cannot use a plain access from C. Replace the use of fixup_pointer() with RIP_REL_REF(), which is better and simpler. While at it, move the assignment to before the addition of the SME mask so there is no need to subtract it again, and drop the unnecessary addition (phys_base is statically initialized to 0x0) Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head64.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 1d6865eafe6a..f98f5b6a06b5 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -192,6 +192,7 @@ unsigned long __head __startup_64(unsigned long physaddr, * and the address I am actually running at. */ load_delta = physaddr - (unsigned long)(_text - __START_KERNEL_map); + RIP_REL_REF(phys_base) = load_delta; /* Is the address not 2M aligned? */ if (load_delta & ~PMD_MASK) @@ -301,12 +302,6 @@ unsigned long __head __startup_64(unsigned long physaddr, for (; i < PTRS_PER_PMD; i++) pmd[i] &= ~_PAGE_PRESENT; - /* - * Fixup phys_base - remove the memory encryption mask to obtain - * the true physical address. - */ - *fixup_long(&phys_base, physaddr) += load_delta - sme_get_me_mask(); - return sme_postprocess_startup(bp, pmd); } From patchwork Wed Feb 21 11:35:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 204170 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp978692dyc; Wed, 21 Feb 2024 03:38:07 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWQEXLr4ovje1uultmSMKj1HQZtxSulpNdPh0z9kFqBG75p1CuHBOLYyzfgYcI7yeT8X/PBIf9mOJxbT9DR9t1INAM/aQ== X-Google-Smtp-Source: AGHT+IGkYZA0kyTjcIeBxzSt2bruFVU5r9ASNDFo1kDe0/0HeghXK6fS83ODENe86MWBlxgQEG7/ X-Received: by 2002:a05:6358:3a04:b0:178:f895:5978 with SMTP id g4-20020a0563583a0400b00178f8955978mr25806288rwe.10.1708515487072; Wed, 21 Feb 2024 03:38:07 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708515487; cv=pass; d=google.com; s=arc-20160816; b=Xq7wHONHa2oKfTPFPHbQtXtnTcFozFIyathEgbOqZwva68Z2sGQQ9s1R8mCNTfQKZQ LizRfr9MyzftR94liTFNtyou8ZL5pDjuyJdyVDUphtveIEqcPbsVuU8rxAv0l0cvUw48 UqtghqYPDed6gDugz6xQxPJ2iJHVyQrmE2m2jWfDenLu/LwAdxFlvHSrRfPiBWnNueoa egD9r6Dmn94eXfbCnIy6Urm7vcLFPlec9kruKny0GxW/04RGVwPAQq7JZ2zvUYH7pltL TIJdu4w67PPqjKN1oxWely1Mj/wB5yVSYfPJ3FhqmkdHxzFBsJuMMFi2JdIgPaUoMSaC NmPA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=8Y5gC7csmOryHPxIGBtz1GA+4X+0n6cre/zcvSeIDBs=; fh=fMhMPvo8duafGkM4xZHdH7fULbau3etefTGsKa+Qy7I=; b=Ki9S2iPa2ENwN2buIz9LSxbWWG6hRxhlZLznHctjGCeIwzV3I+EYWuBN7Vgbf4OXQd ExraptQrAgiphAHuJkjfoOnvJsPM5rEbEojINFNCT0vkXEKVXx+ga4xCNIPX3gra3zah S6LSf3YVTZANp7Qr6dNK/9fCjTSus+CD6UfIWsuJnnonZZLqM+sXCAFSX8XFWA2fv+Zb O6VV0BxyNcDWPjfVBbPMdjmYyGG1xXof2RzSencuwsrk4GhZaMNqyKIVpNfWydhS3lw6 O7tn2Kush5/wPp0CKOVpP3fOY8sojoYLqKBdu5rUCSjQa3LT1uRA8EfGMhix8Ar5DxKD jIqg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=uMRV0uIH; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74633-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74633-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id m30-20020a63581e000000b005dc8554d500si8000835pgb.62.2024.02.21.03.38.06 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 03:38:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74633-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=uMRV0uIH; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74633-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74633-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 13E29B24152 for ; Wed, 21 Feb 2024 11:37:31 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2138045BFE; Wed, 21 Feb 2024 11:35:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="uMRV0uIH" Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A4136487B3 for ; Wed, 21 Feb 2024 11:35:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515345; cv=none; b=TlPV4cJ9f0eACMdge60EtlRxeVWaC37+UBx7xQv4Q4UNBRcViYkEq/zjg9eSNJy61Gltxdv5UB2/T5uURXRqKfmiExivLlrRrBtAJ+fk5jB9pkUL6mJOx+GVaWVnkSd8P1vBNJjI+JX2sELD3xe7HF8e2s7diJCyRqDDbqZAntM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515345; c=relaxed/simple; bh=2RZc+OzKZCl8YvmipTaTRDsbfIBsnndPUmVKA1yKSts=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=PMmKTVg72e8bSwooiXxYh4oUWOszikw/UYQH1adzRk1JvfF55yHl8QjYyNyIDxS9RyjSaYf05y/pmurI+AX0gDeeBvWkwbXfG9fkRqs2/7PGrhfz61h6lAhQu3RqK3X1Q90fQkE7w9/lUHYtlZR7f8qBf8qAX0OMrB63gVrS3rg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=uMRV0uIH; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-603c0e020a6so51050867b3.0 for ; Wed, 21 Feb 2024 03:35:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708515342; x=1709120142; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8Y5gC7csmOryHPxIGBtz1GA+4X+0n6cre/zcvSeIDBs=; b=uMRV0uIHjXPynQUi/G1NZCFhb53bVLW+n95V///OG5499cUBjiIhG6wME9hKF2h0RT uohTMmKBYwXZ89nm/hKdD9Zdmk64sy9YhMy3E0I8JPVhJZjGavl/pTJrYUzH6PYQCaKu 96w+LKfO3nWpX2TLm9WyVRRXUPKtRCVCaS4Rb1tI47jfXaSjcLM37ta5+k0I18y63c12 qiDuFO//ARXQIaMg166xpSbZ+jM3nJa89DKZ3pr9WtGF1mFE+he8L53i6iQVEoQHiWt7 0SzDlReV3V6w40D50HPPfqu3p/lQd4faBRvMfeK9EI9YmHTovfr8QRQyW9+1aMYD+QcM z39Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708515342; x=1709120142; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8Y5gC7csmOryHPxIGBtz1GA+4X+0n6cre/zcvSeIDBs=; b=b7Np/8XZlLT8/1O1pA6dDhHwDPDwcnirawB0zlqPGH0DsxgsWYhYFvlcoYqtvkVjC4 vlzsEjDxr3rYvg+euhvtIfdLnF22QMyBNXhXuEwfm8TWw+OmbXvH+B/T0DETs9alKC7h pBe5fTFgZO9FJvT3Rh4/uRh/s25L9JFpeo7NaZhUau4CEBaJdAbymThimeo3IotI3V7E u1jYFrAjyHT7Kt3ygieYWXsIdfOHQp6Ez3yomV3HS6TrVM6hEZDemJENrSgsziVfL07B 7CEV0OoXtZfjh7qZ5IgLr2rhTtIWhpuXUzSrfStno4LrJS4bOuejtATVisvXd2n+SYwj 3b4Q== X-Gm-Message-State: AOJu0YzLo75slrLpt2X/XsT6+W//dfXxWAInBJth9gp9/1ppJRIpg5I6 Ts78ZIwuqWPUzN5EcS87ciX/pDt6KFPfxf0Ov9Cp3jaYMqfDV/ajtxk14Ya2DtshAAi6TdFelIv hFQQEXQMf3BGjR7xjzQmBXi0tFfBJCYvwHsx3Kdg5VuVpnP1J917+xhsizS9+ttDn9LbinN+DOX paj7y+nIyBb/2QZlhOslNLwBm9LfgS4A== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:c06:0:b0:dbe:387d:a8ef with SMTP id 6-20020a250c06000000b00dbe387da8efmr680600ybm.1.1708515341658; Wed, 21 Feb 2024 03:35:41 -0800 (PST) Date: Wed, 21 Feb 2024 12:35:10 +0100 In-Reply-To: <20240221113506.2565718-18-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240221113506.2565718-18-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1903; i=ardb@kernel.org; h=from:subject; bh=swM7c8C7m6s+syovcWAlr4I1pvzKmPTAbfvGRxDNNBo=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXq/bf72s8c5X6zfe4kzhLF88Ur7P10Jv1+c5Ij3Kphl 3trnr9SRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZiISwXDP+Vl63+6nT136ee+ a7WfF7mxtuzU745WC96z5anM3cc8hWkM/0Nz2E7NSCsXY+vlOrVpn+zruNaY9l8JFYocrFU2Tpo M/AA= X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221113506.2565718-21-ardb+git@google.com> Subject: [PATCH v5 03/16] x86/startup_64: Use RIP_REL_REF() to access early_dynamic_pgts[] From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791508335194189828 X-GMAIL-MSGID: 1791508335194189828 From: Ard Biesheuvel early_dynamic_pgts[] and next_early_pgt are accessed from code that executes from a 1:1 mapping so it cannot use a plain access from C. Replace the use of fixup_pointer() with RIP_REL_REF(), which is better and simpler. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head64.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index f98f5b6a06b5..2ac904110f6a 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -170,6 +170,7 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp, pmdv unsigned long __head __startup_64(unsigned long physaddr, struct boot_params *bp) { + pmd_t (*early_pgts)[PTRS_PER_PMD] = RIP_REL_REF(early_dynamic_pgts); unsigned long load_delta, *p; unsigned long pgtable_flags; pgdval_t *pgd; @@ -179,7 +180,6 @@ unsigned long __head __startup_64(unsigned long physaddr, pteval_t *mask_ptr; bool la57; int i; - unsigned int *next_pgt_ptr; la57 = check_la57_support(physaddr); @@ -231,15 +231,14 @@ unsigned long __head __startup_64(unsigned long physaddr, * it avoids problems around wraparound. */ - next_pgt_ptr = fixup_pointer(&next_early_pgt, physaddr); - pud = fixup_pointer(early_dynamic_pgts[(*next_pgt_ptr)++], physaddr); - pmd = fixup_pointer(early_dynamic_pgts[(*next_pgt_ptr)++], physaddr); + pud = &early_pgts[0]->pmd; + pmd = &early_pgts[1]->pmd; + RIP_REL_REF(next_early_pgt) = 2; pgtable_flags = _KERNPG_TABLE_NOENC + sme_get_me_mask(); if (la57) { - p4d = fixup_pointer(early_dynamic_pgts[(*next_pgt_ptr)++], - physaddr); + p4d = &early_pgts[RIP_REL_REF(next_early_pgt)++]->pmd; i = (physaddr >> PGDIR_SHIFT) % PTRS_PER_PGD; pgd[i + 0] = (pgdval_t)p4d + pgtable_flags; From patchwork Wed Feb 21 11:35:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 204167 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp978522dyc; Wed, 21 Feb 2024 03:37:41 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXqNyG+zj58mFoWIv6aYnIk0OU36V4B8H3AuJWQhumD3zUC8T6pNn4I88qa/OCTpUnIjO6dmJTZAPkowV6Rspyl3m2Z4w== X-Google-Smtp-Source: AGHT+IEl3Qrcc3VLHpw1sd9rMeCagWl/ctP+eDCAbbWxJbgPTTFjD7YR0uk5CA961j0lBa0Hhpgq X-Received: by 2002:a05:6218:2615:b0:17b:f62:b45 with SMTP id oy21-20020a056218261500b0017b0f620b45mr19651824rwc.12.1708515460743; Wed, 21 Feb 2024 03:37:40 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708515460; cv=pass; d=google.com; s=arc-20160816; b=GT9Zvw4Hh93dLG54oiPlhLBhC/PGdO0D7pIeVVwEOWMwu2FA4iVpUyCkaym8lqpYYs t1KPHbaT00jwIVLAV/QmRFTHjyirZgUaJLIf0eQauY8YbX0Q/mTbd8NnW1aCpLCaidUA LW7TilhWgrGKZNRPTWczmh18mP6SymYCvrORQin2iD5Aimc/nUwyuQIEnEAVHkKeNYzn RQ/YI5VNWdIelX/s6ASbxx2hAz5fXmy+7cfy4D7lejXfEq+4OVUWcf6Zyg81RbZPoglC c/Grlta+q0MlIbKBq0pHsnmC2Rl1PpeRf2OeA/hP2rWFhOHfn62dnyz7DI1xgJrJOJvl TqyA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=0rqzUwS6BdxGjI3H1JUFmWLh2DB/nlQLPYKogLCLbB8=; fh=fMhMPvo8duafGkM4xZHdH7fULbau3etefTGsKa+Qy7I=; b=Rj0coEv/QiqQiA8xub6+TZUDr1ZERq1zcQIFbwgZ3ozGtE7J3YJIvjIod+HBptpK1V I2+FtWMGVclpDBZ0e2YIdDG5pJ4cTKBXnwTz6nx0PMgQWpJJy8VnYqXjQQwOFQNEzWQg Eq4O3VqoqgeTqXCoYkzNEZZOdgMPo6BHv0PCIrDuTh4FkBhlSU5d7PooYsH0BwrlA1X5 L0J9yO3iIiTHrieiuPMYu7pLCh1T9IsCYJ+gwYlaL2oKLXv273j3SWRh1L3+II3Gnm80 w7ic6x4HaDAP1uu1l96hGkJbxXMbxx6P5JEvb4CxdzqoXeYdxI+cZ+VggcUaTcjoX2ie 5pvg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=ysv8a2Xv; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74634-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74634-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id cj22-20020a056a00299600b006e48313db5asi2239836pfb.83.2024.02.21.03.37.40 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 03:37:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74634-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=ysv8a2Xv; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74634-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74634-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 88C3B283ABF for ; Wed, 21 Feb 2024 11:37:40 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id F23E24CB4A; Wed, 21 Feb 2024 11:35:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ysv8a2Xv" Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC6314C634 for ; Wed, 21 Feb 2024 11:35:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515347; cv=none; b=P6XlCJjqZnYzzv9q6+mVUFHYU+2wrsBSWEVMrhBTwURHielBrUNKzQ6yEd2EUssnY/6JrA40onD91yDUmjc5Kk3dDO6j1TJr7FLSDDXL9Smb9FC5+6HDl/XeyHGGIi5sjOXC/WHmVJXtOryxf5biLqEZEfWlS0Cs4XL+gyO2Lx8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515347; c=relaxed/simple; bh=6Y7B3eAU7WFGxIfJIx2em1X2j1KbkoJBGgvBRFQVJuk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ctCkypFDhxQBR4w+n8Y9dZDufkA4NCFvgBHqcs5cfsKv2DFZ2gWBh07LF+7lql5IiW5jvZdR7dWzGw7u0m5v22x6CdPAlk6964j77xeOMYtYZGZNq3QZ1NrJwJ2yDBPAX4jQ7/pVA/zVOQgWvR+vPSl0/vcIhWtaec522CgsI1k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ysv8a2Xv; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6088a755299so4077387b3.1 for ; Wed, 21 Feb 2024 03:35:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708515345; x=1709120145; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0rqzUwS6BdxGjI3H1JUFmWLh2DB/nlQLPYKogLCLbB8=; b=ysv8a2XvMPZsZwkeZTnIkW8Dbj6ADJcYD6YqvfAITSGcBc13xHyDbI1xO6jTYs4Hre DNu2TS74LhzOWhjDEz3FvtS0eQ8k336xSXvbj2ucGOG6wrrgsZAhFKbRjJFO4HEYACEo MJJU7bowIeaScEJjDX0mY+iP5VcpFw/JH2hH0KxEJUlDplgBuRzERISP6eJkTYvMBO8O 2yFjjlpWHViUzlN6/UYiN0/9dtQ++68Tt4CfEHqaTl1d92HDDjmIrMguX7BF5t1AR7bL pkvm1Gr5VpeFK5/9JD4KCWWWZ1Vy9PNWb+QpDSnG/Mxfdb/5w5W1HUXsj4fOvgTNhrIk J5jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708515345; x=1709120145; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0rqzUwS6BdxGjI3H1JUFmWLh2DB/nlQLPYKogLCLbB8=; b=jHw4pezBzsMpe7Nlm9RkSUbNPlpCC8cmFCy8HXwXKry1iEi/GipaK5reKl2b7hNRfU RRFLTzUhMKTAVnCssLKDwwIggrC8AptD6Gkm3+GzZgBUqQ+cPeUmbbrJFTMP26f7d/Lf 5Koco5XmFJq5YPAzWVpzF08/lmi4wNMYSGiNz83nMIWBTTNHIlO61xrE/fKAnMLFaWgM rmH5qXlmhd0uMpwoxXdFYxM6wEvI7Fu69XB48kbFUh31x6yx6lnLpj+dakJ6PcLtvwnf Xj17l48uV0NV8RvGsLm2hYl++JyzrE8Nu40o90jmqOtLfweiEhu4f49BcvdqJcReezNl b7Zw== X-Gm-Message-State: AOJu0Yw0bq42u8n+U8ijpYkZa0G4ohUha9N2KLkrWC+bDJigkJqZga1r NZHPJfflgkxi1edr7m9HaDCKp+Hiw8FDp/IrDiCd/8Wjc+lo3NvN5ZHF0mPwudx7elIEhqpJt8a DHss4yu6dBZ8USg6Q82Ki37SqSu1pjhTjxk8/kCaZcaQmSR2Bw//zWItecA9bQvK0RAS4Kc4gtQ Shv9653MeCMn00y7045hz5+6idEYYlIw== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:5748:0:b0:607:c418:33ba with SMTP id l69-20020a815748000000b00607c41833bamr4147129ywb.8.1708515344377; Wed, 21 Feb 2024 03:35:44 -0800 (PST) Date: Wed, 21 Feb 2024 12:35:11 +0100 In-Reply-To: <20240221113506.2565718-18-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240221113506.2565718-18-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1147; i=ardb@kernel.org; h=from:subject; bh=e7Puh3E2XK1VJqqP8geKZexKLK94zY042KFjfKEULUs=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXq/bd/VsVJTi+bPPmYuH9fB1t473PjspnToo1YUsxUq 2rZ/8p1lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgIlcn8fwP59By9ztn+WWpIp/ Ox3ZZ2pacQjfWxM61eabxvsZwdx2xxj+2ZrtjxF+psy/dvm6zgKZQ2ypEpVS2Y8L2oKWh7RmbCt kBgA= X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221113506.2565718-22-ardb+git@google.com> Subject: [PATCH v5 04/16] x86/startup_64: Use RIP_REL_REF() to access __supported_pte_mask From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791508307867700271 X-GMAIL-MSGID: 1791508307867700271 From: Ard Biesheuvel __supported_pte_mask is accessed from code that executes from a 1:1 mapping so it cannot use a plain access from C. Replace the use of fixup_pointer() with RIP_REL_REF(), which is better and simpler. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head64.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 2ac904110f6a..e2573ddae32f 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -177,7 +177,6 @@ unsigned long __head __startup_64(unsigned long physaddr, p4dval_t *p4d; pudval_t *pud; pmdval_t *pmd, pmd_entry; - pteval_t *mask_ptr; bool la57; int i; @@ -259,8 +258,7 @@ unsigned long __head __startup_64(unsigned long physaddr, pmd_entry = __PAGE_KERNEL_LARGE_EXEC & ~_PAGE_GLOBAL; /* Filter out unsupported __PAGE_KERNEL_* bits: */ - mask_ptr = fixup_pointer(&__supported_pte_mask, physaddr); - pmd_entry &= *mask_ptr; + pmd_entry &= RIP_REL_REF(__supported_pte_mask); pmd_entry += sme_get_me_mask(); pmd_entry += physaddr; From patchwork Wed Feb 21 11:35:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 204169 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp978594dyc; Wed, 21 Feb 2024 03:37:52 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVRPTf2YVv3ed3i537LrMTYHi1K189YBxVmZGtrNR7GboSzckegJG2+XwuuFuYPQVv75RdXUC6RYcolEwi+jmIs5/LIew== X-Google-Smtp-Source: AGHT+IGy3Dqg00gCsVomidgPS9Lg5HDsWIuN3JLGIKkIo8A0vGVnftKAcWYbTyETZgV2GjJ52dxE X-Received: by 2002:a05:6a21:6da3:b0:1a0:b5cb:36d4 with SMTP id wl35-20020a056a216da300b001a0b5cb36d4mr5832612pzb.19.1708515472380; Wed, 21 Feb 2024 03:37:52 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708515472; cv=pass; d=google.com; s=arc-20160816; b=mtCIudapUXzvL4fXHAWXXVHCEbInipupcAGtbYRkJualAOyOZ0G2VkE9CMPXAYoci2 GGUWQS0Sib/ArKvMP75JKfr+stQf7sEH2X+QqYbv2bNB7natrP2jJO791mYdTddQ2/+4 LbQ1g/RjLu2iN3xoHdiK/k2dEwJee935M0mD9B8gQrApP1MvUqdqLk0auWd1MT9jZkOX zLayyNjD3EctVDuBKoIMiujcG5m/oguBZDTieJ1vAIgpg9yeWIXGu38KMere7LPLdFm0 O2EFglVZHOaueGAx8hrFZ50+kF+qactQWXwHUDGWZ0KSlZPEmGShQJ/288+/7TRy1gz6 DA/A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=mh8YQtdjDFBeR5IIfvtfOkH3iYneTElPPQWzRP3dQ5U=; fh=fMhMPvo8duafGkM4xZHdH7fULbau3etefTGsKa+Qy7I=; b=K+jEVUGGyjEsvjkOPBcYwj4WhUUoQi7EHEozOJhQFksSbfcIlyovteL3waeAGHU7kM F7IiYV6rc5Jqi1yQzS396NQ5NEdcDc7yL06ZRL5Qkgvfk9TWlPy0l+313eAvO29bPslT pCmHu/IK8n1Ly8I0qvIH47xX/6c0cJzQdHwL24rBsG3bZpUobyFZTYIKAjc9XSmMq+db naPDKo2Hf8A7YMXRnzIW1aHgxFLRs1EIazTqH+93acXc5hwlDFmeTllQuDkEr2mPN0pI Phu5VMSPHnAVKlfM4sHprannHflImIImK1D34wdJxIltqpVR6koNyt8cZy4xIm1SCWb9 XQKw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=yoKn8IIl; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74635-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74635-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id b20-20020a656694000000b005d8e1f10c3asi7730857pgw.877.2024.02.21.03.37.52 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 03:37:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74635-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=yoKn8IIl; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74635-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74635-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 05911284301 for ; Wed, 21 Feb 2024 11:37:52 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5C84C4D10A; Wed, 21 Feb 2024 11:35:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="yoKn8IIl" Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 369054174F for ; Wed, 21 Feb 2024 11:35:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515349; cv=none; b=Ll3ZSfkVIzjcnD3DcU+ZvVCuUxXQPIv5xsUp80NZMSfcZimUjKkzi/x32DROuZFkmMfzQDdr8cA38LogUkhAWwPsUeyo+anB9UWGl8PqAli842FKBXDA/qxQx9YHBjmitiIMcCuqBkjjUmBI+0AdjUTmlXjdfX33JGz4uD7aAeM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515349; c=relaxed/simple; bh=GAlPtq8JkQJ3A2WVw8c46QbR1WtxNjQfx1w8AVpPTjc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=kzMTYKZnkG+1bJho1PtFODPzrkZvK7v8IqtjjpoVROoba2R9VDyLoqDe+Za0DG4ZGA1Zprecyc/POp2uXgeoaCRZMW5y1shI65L0Kc3ZUeLO8Vlkkd2AMaNNx6nqHqBiUuOzzgTHx7W8gA3hVcWbOJhFN7xZUEj+4T+N4VeI4xA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=yoKn8IIl; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-608575317f8so35420697b3.3 for ; Wed, 21 Feb 2024 03:35:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708515347; x=1709120147; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mh8YQtdjDFBeR5IIfvtfOkH3iYneTElPPQWzRP3dQ5U=; b=yoKn8IIlNUE26g5hYhsL1NTTncQyij9u7eThJAI1n31mtRwVIUbMO+3bu8DBrMeK1/ /9tmcPZhQOXj2cPSN20U0RoUnEgKacAlF91tOZRn7oJJ7dRQyJlZtogAx4jeXUTGXS2w h3L4azD7aePLTJHCavuwYWbDOVXMSLm7W5YUsRspBZtW9UBqza77wkMZhBVLG2w3tJoW qOAkYTzo1cZUumRgQl7o8S4r5zKKKbnZqkCGn4P5I4lK+9Pn42eq2OZLxppNWrUHxKJt Ybe2qaTZA3zbvj/mLGqnpeYeYSmb+mfGi5UpB4X08PB6+4pSlA+2V0VjgRmFIqpQ71aB MSFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708515347; x=1709120147; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mh8YQtdjDFBeR5IIfvtfOkH3iYneTElPPQWzRP3dQ5U=; b=esypS/nf8o5iLZBXA5lk3GewpZQaefr1ib6oOVheIZ1/ykmP9gBxAW9EsgG83eceUh EKdm6zajxhflrs/gOKhIETv9XP2WbweKFmd14P4KPanVIoKxqEICqZ6J9pN4S1lZpKpt exs9HvGAd8oVVpyMbCevsj/A2M1B30beXpb7VpwDzCx047iYAiQWuursr0sw9K/O/afN hM/ZpRsZTfwt2uBKAioJWMWpQGmb1KlpoBSbHRK7X30780oJSIQfqNopy/iOpeBbh/Pt +EdHFnE1B/s4pouRiWjRNPXRWifky6Yjvbggw4w9FTeChrUzPgJjYEeIJlQgqtADomWa Z8ag== X-Gm-Message-State: AOJu0YzCR9fHM1WOqpUiXM4Oi6Ft2HDgXlaib8v8qBNu8V9y7rNGm789 CnO9AR2shB6K/j1e95d2I3LWYg3erjUAVCcrrGoR4q/QVSNpjcPz9jkLW3dso9KesQvIxjg32GH sp/y/V2NbItSeIwgBIfvBNsmTtt3CirPFwXhBcSxwNFhbOxs1BPFX22IveHqfVShyUNg/6LLcGP MdHcq9D8TW0BTS4Qk2D6z6c2+9r4gHvQ== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a0d:e6c5:0:b0:608:801a:e66e with SMTP id p188-20020a0de6c5000000b00608801ae66emr232627ywe.3.1708515346559; Wed, 21 Feb 2024 03:35:46 -0800 (PST) Date: Wed, 21 Feb 2024 12:35:12 +0100 In-Reply-To: <20240221113506.2565718-18-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240221113506.2565718-18-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=1582; i=ardb@kernel.org; h=from:subject; bh=cvzedKkAgUYV71bwW+wRLG9NvEsRD1LqBxLDjio6avo=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXq/XcfP02y/sG1PGlJDc9qpSi1Q69MXf6u41urqrbMb sH7yR81OkpZGMQ4GGTFFFkEZv99t/P0RKla51myMHNYmUCGMHBxCsBEJmxi+B/G7rLguUZMl3+R 0vtNEibBlX9ZVJt10g8o/TM9Ily2zY2RYZFpSYf4SxEfXrMP20O46g6+2tbuk+T93lr0pKL5u+8 PGQA= X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221113506.2565718-23-ardb+git@google.com> Subject: [PATCH v5 05/16] x86/startup_64: Use RIP_REL_REF() to access early page tables From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791508319886233081 X-GMAIL-MSGID: 1791508319886233081 From: Ard Biesheuvel The early statically allocated page tables are populated from code that executes from a 1:1 mapping so it cannot use plain accesses from C. Replace the use of fixup_pointer() with RIP_REL_REF(), which is better and simpler. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head64.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index e2573ddae32f..7e2c9b581d58 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -215,13 +215,11 @@ unsigned long __head __startup_64(unsigned long physaddr, p4d[511] += load_delta; } - pud = fixup_pointer(level3_kernel_pgt, physaddr); - pud[510] += load_delta; - pud[511] += load_delta; + RIP_REL_REF(level3_kernel_pgt)[PTRS_PER_PUD - 2].pud += load_delta; + RIP_REL_REF(level3_kernel_pgt)[PTRS_PER_PUD - 1].pud += load_delta; - pmd = fixup_pointer(level2_fixmap_pgt, physaddr); for (i = FIXMAP_PMD_TOP; i > FIXMAP_PMD_TOP - FIXMAP_PMD_NUM; i--) - pmd[i] += load_delta; + RIP_REL_REF(level2_fixmap_pgt)[i].pmd += load_delta; /* * Set up the identity mapping for the switchover. These @@ -284,7 +282,7 @@ unsigned long __head __startup_64(unsigned long physaddr, * error, causing the BIOS to halt the system. */ - pmd = fixup_pointer(level2_kernel_pgt, physaddr); + pmd = &RIP_REL_REF(level2_kernel_pgt)->pmd; /* invalidate pages before the kernel image */ for (i = 0; i < pmd_index((unsigned long)_text); i++) From patchwork Wed Feb 21 11:35:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 204171 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp978729dyc; Wed, 21 Feb 2024 03:38:13 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWIUjhlrimbQuqEEfN4azGU0QAn9jRK4G5ubk2Tv+VZA3FmBKwkncMSbGVnRJioOPBSSS8kz/2UV812/Grvwv4kNympgg== X-Google-Smtp-Source: AGHT+IFR/DjjpXI3sy+1wOJWr9Ki94cOfWeG46Ewl8m/ck/93F9aqJbpD4Y097kUAnnzHDxh2LDE X-Received: by 2002:a05:622a:49:b0:42c:6fb6:8d3b with SMTP id y9-20020a05622a004900b0042c6fb68d3bmr17705826qtw.66.1708515493033; Wed, 21 Feb 2024 03:38:13 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708515493; cv=pass; d=google.com; s=arc-20160816; b=w1AQOkzP2ArG2EYWiAYZITgLrc9SwXoLd3tEHO0srCDLZpJP4MlKPymQrAf8BsduuW OuL6xq/6d8SEGKUafDAmqSZNde+V7F4JtftGumWGlBEqHUoepHCVEecBByU+fswInkUE 1vzNRolBjRjWQ+cTEBS2Xyugf7pJbZoXrpQxnFtiBBNE4iRaBgISUz0E/7RNHcSrdAaE UCkIrRFkfCugbPlTMNEaIIgrVry4SAiT4CHVHYdcfzPxS87SDlXNhrHhhQMn9zl7FlBB BEbcgJNnL/+pAFlpVWJZ4r2JOACeK/QSq7OIXCRvfIocissKkO33cUiICuDE87wqBDH7 LOsg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=OQqG5//oJ4Rct3kZcJJxz5fvctqzzrLZGfPsxPM3FmU=; fh=fMhMPvo8duafGkM4xZHdH7fULbau3etefTGsKa+Qy7I=; b=pDcqCy9hcMdSgGP9g4RpEpA1AXn+Tx/xdqQIxUrdLU4b9lfoK/QXyWcXXVLgqhAsQb qgNztjhMfp6QYq8thF7NhKVFyBdFcyE6kjR7rlkmCiqc8FJPA6oaA++xO0fgGbMmwpDW 8JwdUQBM4EpC7A1sR5Z5Bq1RU79IoPnuQ+70xzN8RDuITsqFW+9FN4tpoP0wwpJ8IF0D 3CZKBYW3n4PiQDUaCo08n0CBc31+5jeXig0TxMXPbJJbxrR+rOBKHjUm/sw0kF7EZZHW dP/s5iXbPYxeuLgpOuEAW9HKqTzG3hW8wzPgjpATecgXJzIBd78hBps1rQPdSP6+JMb0 7nwA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=QsV+4JpE; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74636-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74636-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id g15-20020ac87d0f000000b0042e390c97f7si859138qtb.82.2024.02.21.03.38.12 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 03:38:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74636-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=QsV+4JpE; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74636-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74636-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id C08CE1C22FF5 for ; Wed, 21 Feb 2024 11:38:12 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A67874D9F7; Wed, 21 Feb 2024 11:35:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="QsV+4JpE" Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB13B4CDF9 for ; Wed, 21 Feb 2024 11:35:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515352; cv=none; b=n4R8WPhZsM2qNyUxRfaydgX4tjnbRRe3vda5q+gJ8ihQIRb4WG672n5c1bkWSye70OSS8l70yQzSGHKpvRRp+D0GovZzU4foNMNsg+sMh7wlMbw1xq8hGg+Xt6LSa7ndLZ+1mm/KT/Le5RoUPB8Rj/KLaa+AojWTas+8m4bVuxY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515352; c=relaxed/simple; bh=98RocWEs5r3eqouBXzUZIZpcbKGYcTds+8Ikdcp77uM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=PNeYgp9n5cyDixcAwfLqa9OvOKEOUynj3z9i8UTWUbcDR8b21uRw4AfJ1acGvj/qxUEFVu7m9p2cM/Sh1hY7Fx0awLMfggtZfoNtDW94O+iW0kYt7akexSB4b5HP3LhWkahBoMazCyYXXi/FMQuIbkUTYyEad3EcpmDj+dKZq2E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=QsV+4JpE; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-410d0660929so32252975e9.0 for ; Wed, 21 Feb 2024 03:35:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708515349; x=1709120149; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OQqG5//oJ4Rct3kZcJJxz5fvctqzzrLZGfPsxPM3FmU=; b=QsV+4JpEv4dcyIya22HAbt2otOOKNabd/tyK846x1RtNSKXYeWrciWgTm7mKTPKfBk 02Bb9IVCB0XOlKs/vr3cl0JsjbC+q5HPMh895FFZBaHQaBUorKyzzEsPng6UJG21x5m9 YFDei7PpC1qHwSjfomKwJYUwnqH2WM+uKUG8IPeBzPlmNC9p1e/Dd7I1/uYnV9wFCBuU zWzkxoxBUhrM+vHFOPxoGpmF/7p8Hya3lea8SWO+1mW8qOjLPTpK4nLnw2e1s0nXR7Ev IaJ4gNalIUMSkw+ALpx98cmXyd/1KE+FcKVc6b3FSSH9siVs7na8x/zrD42Vgcay+1Ar ZCmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708515349; x=1709120149; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OQqG5//oJ4Rct3kZcJJxz5fvctqzzrLZGfPsxPM3FmU=; b=eqJXk3ZhffxHxMAxjZJH7IUcS5hgH8ojVoryURumZpOuvAeDUvwKObgWQniBMP58WH sbY1sXISYNgkA6w46JvV1niuIe2EPnnN831mhKVoHqGn8AwAIbos3Hsex/GUYJM4KOBZ /+JpgJ8ZGxnqXVKPbdcn4RD/tC2lXt3KUGenGyFclMw3LRe1eqLXHqaa/iI8WfYt7f+j kaHo/1kusAoS1lrT6nTV97/y1w8lzRGVBUmzYfO7Y3ftWgaw288lOvrg1Tb51qfhTtUa 8rE9jXGmwAXzMGuW0E56B+A9kocFn0j8a9FBTVdfQKR0oCSkg6KxgOU7ujC1KfwE5xGh hOJg== X-Gm-Message-State: AOJu0YzEFH+uAAEIBcn2cSg3L30AG+kobK6sPFhjYUX/LIsHrOPSPDxl 5HaO531jqBhLqrkFx6Lp8TjUK/3UhU2K1xEoQnA7zh0tn8Y6x88+NCc5xEneSWPtlFWp9vyOPp+ GyAaVwbMGYgEBeA2C5A4FckxbYjOAtS0cWHFjykHTBA+MCFB1xYGP88zNXmPk+DjLpIHmulaUrn MM9AVd7ao+7Sew/3acwVbt37Md+EmQ2Q== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:11:b0:411:bc74:195f with SMTP id g17-20020a05600c001100b00411bc74195fmr184621wmc.2.1708515349006; Wed, 21 Feb 2024 03:35:49 -0800 (PST) Date: Wed, 21 Feb 2024 12:35:13 +0100 In-Reply-To: <20240221113506.2565718-18-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240221113506.2565718-18-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3506; i=ardb@kernel.org; h=from:subject; bh=FS+JqcMc/MiCOd6PCGmTrr43k29kYK2wA+gFcF5Sq7k=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXq/fca9UsD/iQ9/atd2jT1UMuJC+kqF22uPVv69lO06 7zp++xXdZSyMIhxMMiKKbIIzP77bufpiVK1zrNkYeawMoEMYeDiFICJLDvO8N+/Vjiw7BXTwRca zh+5VXYePPr2M+vh159WGVQ4KGb1qc1lZGi+u9OA44l9zbxqL8fcm21lu4tWeCt0nbx97Nbp2WV /HnEDAA== X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221113506.2565718-24-ardb+git@google.com> Subject: [PATCH v5 06/16] x86/startup_64: Use RIP_REL_REF() to access early_top_pgt[] From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791508341618223212 X-GMAIL-MSGID: 1791508341618223212 From: Ard Biesheuvel early_top_pgt[] is assigned from code that executes from a 1:1 mapping so it cannot use a plain access from C. Replace the use of fixup_pointer() with RIP_REL_REF(), which is better and simpler. For legibility and to align with the code that populates the lower page table levels, statically initialize the root level page table with an entry pointing to level3_kernel_pgt[], and overwrite it when needed to enable 5-level paging. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head64.c | 21 +++++++++----------- arch/x86/kernel/head_64.S | 3 ++- 2 files changed, 11 insertions(+), 13 deletions(-) diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 7e2c9b581d58..72351c3121a6 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -77,6 +77,7 @@ static struct desc_struct startup_gdt[GDT_ENTRIES] __initdata = { [GDT_ENTRY_KERNEL_DS] = GDT_ENTRY_INIT(DESC_DATA64, 0, 0xfffff), }; +#ifdef CONFIG_X86_5LEVEL static void __head *fixup_pointer(void *ptr, unsigned long physaddr) { return ptr - (void *)_text + (void *)physaddr; @@ -87,7 +88,6 @@ static unsigned long __head *fixup_long(void *ptr, unsigned long physaddr) return fixup_pointer(ptr, physaddr); } -#ifdef CONFIG_X86_5LEVEL static unsigned int __head *fixup_int(void *ptr, unsigned long physaddr) { return fixup_pointer(ptr, physaddr); @@ -165,14 +165,14 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp, pmdv * doesn't have to generate PC-relative relocations when accessing globals from * that function. Clang actually does not generate them, which leads to * boot-time crashes. To work around this problem, every global pointer must - * be adjusted using fixup_pointer(). + * be accessed using RIP_REL_REF(). */ unsigned long __head __startup_64(unsigned long physaddr, struct boot_params *bp) { pmd_t (*early_pgts)[PTRS_PER_PMD] = RIP_REL_REF(early_dynamic_pgts); - unsigned long load_delta, *p; unsigned long pgtable_flags; + unsigned long load_delta; pgdval_t *pgd; p4dval_t *p4d; pudval_t *pud; @@ -202,17 +202,14 @@ unsigned long __head __startup_64(unsigned long physaddr, /* Fixup the physical addresses in the page table */ - pgd = fixup_pointer(early_top_pgt, physaddr); - p = pgd + pgd_index(__START_KERNEL_map); - if (la57) - *p = (unsigned long)level4_kernel_pgt; - else - *p = (unsigned long)level3_kernel_pgt; - *p += _PAGE_TABLE_NOENC - __START_KERNEL_map + load_delta; + pgd = &RIP_REL_REF(early_top_pgt)->pgd; + pgd[pgd_index(__START_KERNEL_map)] += load_delta; if (la57) { - p4d = fixup_pointer(level4_kernel_pgt, physaddr); - p4d[511] += load_delta; + p4d = (p4dval_t *)&RIP_REL_REF(level4_kernel_pgt); + p4d[MAX_PTRS_PER_P4D - 1] += load_delta; + + pgd[pgd_index(__START_KERNEL_map)] = (pgdval_t)p4d | _PAGE_TABLE_NOENC; } RIP_REL_REF(level3_kernel_pgt)[PTRS_PER_PUD - 2].pud += load_delta; diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 3cac98c61066..fb2a98c29094 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -653,7 +653,8 @@ SYM_CODE_END(vc_no_ghcb) .balign 4 SYM_DATA_START_PTI_ALIGNED(early_top_pgt) - .fill 512,8,0 + .fill 511,8,0 + .quad level3_kernel_pgt - __START_KERNEL_map + _PAGE_TABLE_NOENC .fill PTI_USER_PGD_FILL,8,0 SYM_DATA_END(early_top_pgt) From patchwork Wed Feb 21 11:35:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 204172 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp978777dyc; Wed, 21 Feb 2024 03:38:22 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXW82DMWctyzlgzglQ8KwKTZNJ/gNMFnTL4r43OWeRgbiyUACpwcpGBvGPANGZPNCoetnzPDhKYxxE1qu3ZZQe/A36dtw== X-Google-Smtp-Source: AGHT+IEPWymECC3TJQIMELFc5aYUh3FqZAmKNgN5g+kGqNGploWJDz/0YJ/dmn7x/Lauq060EE3j X-Received: by 2002:aa7:cccd:0:b0:564:4e61:76cf with SMTP id y13-20020aa7cccd000000b005644e6176cfmr6796414edt.10.1708515502701; Wed, 21 Feb 2024 03:38:22 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708515502; cv=pass; d=google.com; s=arc-20160816; b=gaGtmgPqkfs7TxXVE9patEHgXmxLEi9iVP/oVFkU/7F+G/Dpxf7Bdv/VzY7r2ggf/3 2uJTEpQ1ffqRpps89uO1Q8zY1CQqMYhTCZQ33no3hnMF6PLob/9/ZFFWoU43EAV98mmf +m78eA+NLbdvWv+7dX/ltbwyKdFea+Ago/pohmZGPO8cY0+V5X6tJaA+Ubw5dEpNg79a 2NfV7MaIEqitlmiTexEYRw0P3a++IbJ6fbYTa57AGYvuX/nowc6+ZQQ/ZMaDspUZTOnb ZbdDq5c9t9u+GvyS1YS5phy1DHkicjDmcbWoZQ/0tuUeDjY++asMs2juauztEL512s+3 R8UQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=EkxcLENiSlDOPoQNco4VFPfOwd/QcirP1cMQb5AjsLI=; fh=fMhMPvo8duafGkM4xZHdH7fULbau3etefTGsKa+Qy7I=; b=V0+/EttialycaLrzGeCumgd8w0EjqIC0X0srG/VFycHKzgat6DETw22mWAuC+YeYdq zbg8I8dVSlYWZmTwgm29cMnU7WeSAQAz1KD3a6anod0fx+wa+a4x2PhIbXCd+9Pb1RxT NbBw+xSCRctoIH0x5NMdEae9tYi90csWnaJ9MJ91uUQ7t7Tl8StYjsQUyQr/Zv9N0PGz GiARTzi6/Jm/bOov16mBwkpKBa1AaCrEvX68lVrtlMo5QW0ljAGRBW41+eHnPBy7fuYD UmzToZYldLNh2yFRpkTyt6xERYgW7de73yAPdxf0UlUgPIQhdSf9YdNHA0cHzWSbgF4n 4uDA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=HUdSHzvF; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74637-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74637-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id w22-20020aa7dcd6000000b005640ff07cd1si4221517edu.309.2024.02.21.03.38.22 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 03:38:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74637-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=HUdSHzvF; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74637-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74637-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 12E5C1F23581 for ; Wed, 21 Feb 2024 11:38:22 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4216F4DA0F; Wed, 21 Feb 2024 11:35:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="HUdSHzvF" Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A4ADE4D134 for ; Wed, 21 Feb 2024 11:35:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515354; cv=none; b=qPQFGOF9Pd/edIXIWN4IAhciO4gv49m09I9UXz7pIv08Km9zun1C7L/FCm+tIV1VohpM3dVNxtqW4bW+Do22cCQH/MvTmAkynZhkNUMVF0r6hulcobwpZAfCqMg1ZpaV7DhHb9Rk/mtB5O2BGqQmutl5wpYHSuPXwHOdkHInBEw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515354; c=relaxed/simple; bh=+vNI1MmLFGS1GivHerZyjgoY4sNAT6wDfnGDYRsl6Po=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=L0fId5mne7m7chEKaNQwpMETmLPp/GY1oOKf3t3bPRJKUUbyWSYevRTw6UWgqDXK7Jk5sr6Vu3Ah4PG7gxKgnFksJqiWjyIa1OwTDHGZecS2TDxJMRDFLwhryap47dDTapmONJ287n3c8VA2vQ17FkNcoK72WTxU7SfakRlNjkI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=HUdSHzvF; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60832a48684so43910207b3.1 for ; Wed, 21 Feb 2024 03:35:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708515351; x=1709120151; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=EkxcLENiSlDOPoQNco4VFPfOwd/QcirP1cMQb5AjsLI=; b=HUdSHzvFKPPDstZtNL3AuROXQglCVOQW91cKv2q4pk+2jsrpvNUxc6qdc4t1/mPRX5 AlB+Cr5BCSgREX05P0N5nOYVKEFQm8RfZXG1ya6ffkcRoNiOXpTJTrNQClXDp6lI7D2D k8KDOgLGC6L3M4pZgtaEpkKet2jWRduZXsl/Ybzf0ggM4OGkJdQ4ghu+idGoug8xLJsu KEbCSokgZjWd5j4CJEctAhPwwHBPsn6R+/0hew/rxEPd425BhMyKaBe6gsK0sqyx1gZG 0bz3mrfsKf/x3lT/Z/+3nBIePxWybSuo+78weNarM2CHbv6sLEIbpzTXW2VbJ4s0q8AY Q7wQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708515351; x=1709120151; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=EkxcLENiSlDOPoQNco4VFPfOwd/QcirP1cMQb5AjsLI=; b=prNmuSEhFM/Dc6F9KrBn7FMJ/9COoRkJ1wvOiQWAMX8Cj4C7q7ddXkSdi8Wdj2XK9J eanOmAQQ36H9kG5HPeypMTVUos/vNcajEDsRI4ZZp/XkCXQEqdue3OdtyMPZpNpcO6VH OalYwWC/luiAVf6Qg0NnYARWuv9SAcwoy4orTMLohWC/3V8bOYzL9X/Lo0ayybtDEuF0 8w0zX46IH2xnkUNg9yDB3MLrC2Jjegqx37ibEHkXQXnIO2WRO8H90oeLVe+rBQhzPspw 0HsjDsQ4xbXE5AScmsEJY0Ru4NBNdlVlpyU6aOC0MAk+o31CzLactyxUAK7j0NmGrHgM 41rA== X-Gm-Message-State: AOJu0YwFakY13X16OFmxke1hm4GCmkmkGfYr/Qjz9+s2YydG0jQYvUOv UI3F217ptvt/+KylHlU0pudJI5yIYB8FRLn6i+bYqgb0LaxjDXaUjq1l+WRUGp3lfgAkn0deIu3 sNSpyjM1g0xmuxm11GITvvCMs9mglT2ZERZ6K9TOI3+wESYFlt1uR/BLkzC/VkVMPgGCPgLr+ej t7Io5OWHMFCc8zNOxmEZTW4NboPyenzA== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a0d:d790:0:b0:608:1fec:1dd8 with SMTP id z138-20020a0dd790000000b006081fec1dd8mr1741929ywd.6.1708515351508; Wed, 21 Feb 2024 03:35:51 -0800 (PST) Date: Wed, 21 Feb 2024 12:35:14 +0100 In-Reply-To: <20240221113506.2565718-18-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240221113506.2565718-18-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2455; i=ardb@kernel.org; h=from:subject; bh=yaS/9ovMSw8F0QV6XZaoG3EypsAj0QA2cGCCi/EyQdM=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXq/Q+6elITfjUkXFxSXNX0UuVCU/Pf1rdHz2ydK3dXK 7z/yoWTHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAij18wMvQIb7OtV6/2qrBx Cv98yNPx7toM9qoPhkYHVx1mi2xTXs7I8NctSEVKctq3g9P4dr+z/LD31RzWlr/pAolrNTynt09 3ZgAA X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221113506.2565718-25-ardb+git@google.com> Subject: [PATCH v5 07/16] x86/startup_64: Simplify CR4 handling in startup code From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791508351743268648 X-GMAIL-MSGID: 1791508351743268648 From: Ard Biesheuvel When paging is enabled, the CR4.PAE and CR4.LA57 control bits cannot be changed, and so they can simply be preserved rather than reason about whether or not they need to be set. CR4.MCE should be preserved unless the kernel was built without CONFIG_X86_MCE, in which case it must be cleared. CR4.PSE should be set explicitly, regardless of whether or not it was set before. CR4.PGE is set explicitly, and then cleared and set again after programming CR3 in order to flush TLB entries based on global translations. This makes the first assignment redundant, and can therefore be omitted. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head_64.S | 24 +++++++------------- 1 file changed, 8 insertions(+), 16 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index fb2a98c29094..426f6fdc0075 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -185,6 +185,8 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) addq $(init_top_pgt - __START_KERNEL_map), %rax 1: + /* Create a mask of CR4 bits to preserve */ + movl $(X86_CR4_PAE | X86_CR4_LA57), %edx #ifdef CONFIG_X86_MCE /* * Preserve CR4.MCE if the kernel will enable #MC support. @@ -193,20 +195,13 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * configured will crash the system regardless of the CR4.MCE value set * here. */ - movq %cr4, %rcx - andl $X86_CR4_MCE, %ecx -#else - movl $0, %ecx + orl $X86_CR4_MCE, %edx #endif + movq %cr4, %rcx + andl %edx, %ecx - /* Enable PAE mode, PSE, PGE and LA57 */ - orl $(X86_CR4_PAE | X86_CR4_PSE | X86_CR4_PGE), %ecx -#ifdef CONFIG_X86_5LEVEL - testb $1, __pgtable_l5_enabled(%rip) - jz 1f - orl $X86_CR4_LA57, %ecx -1: -#endif + /* Even if ignored in long mode, set PSE uniformly on all logical CPUs. */ + btsl $X86_CR4_PSE_BIT, %ecx movq %rcx, %cr4 /* Setup early boot stage 4-/5-level pagetables. */ @@ -226,11 +221,8 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * Do a global TLB flush after the CR3 switch to make sure the TLB * entries from the identity mapping are flushed. */ - movq %cr4, %rcx - movq %rcx, %rax - xorq $X86_CR4_PGE, %rcx + btsl $X86_CR4_PGE_BIT, %ecx movq %rcx, %cr4 - movq %rax, %cr4 /* Ensure I am executing from virtual addresses */ movq $1f, %rax From patchwork Wed Feb 21 11:35:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 204173 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp978921dyc; Wed, 21 Feb 2024 03:38:43 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVjTYHWbci6PuQv+y34oQE8f9p201D9WdxT5AI80WWx3rx8CY4tWWDuh0MfSVaUGIwPmJCjlqjzN7/Jej1XgIoR7VowkA== X-Google-Smtp-Source: AGHT+IEA87oocIoCvV4VUplCZ0nF2TgPKUgmQOyZCZ16QXFV13lw1Txisk7AwJYdikhl6//LYMU+ X-Received: by 2002:a05:6a20:e605:b0:1a0:686b:afec with SMTP id my5-20020a056a20e60500b001a0686bafecmr14963132pzb.10.1708515523488; Wed, 21 Feb 2024 03:38:43 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708515523; cv=pass; d=google.com; s=arc-20160816; b=xCRA4qwK9XtikoPmQdmPT58fn2odoAS276jhLEwM8ND9vWQwdeGS/dCymM1nmfDNV0 wh63yiRgqU6XRQ3UXawvdeW/Ueru1cfNzO3mD3FeEPL3agAs1ad9oWvto9NXnqp+vVut pIXdsDTnvMxO1t7o2O25XJ5EEO/rBr8s50JsBwvy89fUG8owKG+2PSIQ9P3c2/LsDOM1 OEMmjASlP/BZXNPs1T5eYu/xWRy0hyg1OblntLocMF84VG66XQQrfQ5Q/OHt7CqMxGQV 91tZa7a6c++5g6OHMpZZOA8CuQBCuS3VXGKPaliCCCB89xcPEJgTVK0/TFe2Y09J1m9h pnnA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=gMaXkJDSba/rz2ejEwTszeJPuP18TapiV6QGh3kcGw8=; fh=fMhMPvo8duafGkM4xZHdH7fULbau3etefTGsKa+Qy7I=; b=EI2SRC3soRsLO8ZFmf8E/QP465wTUHqogJPkdYXJ9ko4JK/sArETxGF/yvQbn+GU4v B95yX4dkMcXQgqsz5ABUkiNXYGchtuTTErCJZN5c3KZ7NywGhlgHQU4jGCcfL1Erlabs NiNfIfZFkpThYe89jfi/vUZPCD/SHOevNuVw5qNNO8QxSCxMaobs9alrwfKVcZynyS1C XpQjAhy8fq71Ml/7gaI4qjmyzDJZYWEmBqcXlC74ZvHjnaAFSPzmFFhG3ludPSSLVZva fqaUbGIOTP6lwz1cXRxMfBmwAdHj9/B8DbukyFlE4CpcasZ8PXTByJXru7N0g7aezMuv rWKg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=mwF6MfSe; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74638-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74638-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id x62-20020a638641000000b005ceb4a70188si7838369pgd.375.2024.02.21.03.38.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 03:38:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74638-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=mwF6MfSe; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74638-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74638-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 1249E283802 for ; Wed, 21 Feb 2024 11:38:43 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 004014CDEB; Wed, 21 Feb 2024 11:36:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="mwF6MfSe" Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B4A444D9F8 for ; Wed, 21 Feb 2024 11:35:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515357; cv=none; b=tJvK7DJrnuqQ7XLKdgJuM/ywZdnlPwtaRw1lB05j/rSV56qm4AdHOzDVxma71u0XD3jGMq8PaKE6niw7ey8/XnNghI27z8N9N7EiHcGMOUvovfjYoEjuUjmzJPcMBRF3mIttMDPduyHUubu8dLnO4fJLIMOT5ji8iUl7seIHxTM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515357; c=relaxed/simple; bh=IKzSvD/LgGiOgyH8nDLzJUWhN/QyWc5LsEQ67nR9hkM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=HTHqK1KqlT0F97fvHmwAVMbbgL1V+pCV4ivXFeJNCscmAwNxyX25zIWyXnTl8uR/N5VXAzDCAy9UIqd0hSHG28AW7/uz4CKpmAXgCOYrH3ph/Ns5B7EGiTsZkNnSC5m0IMV6aRhnCJITwhEMYBVDYi6LN55gpbVcPOMLEwX7UQ8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=mwF6MfSe; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-33d782af89eso429220f8f.0 for ; Wed, 21 Feb 2024 03:35:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708515354; x=1709120154; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gMaXkJDSba/rz2ejEwTszeJPuP18TapiV6QGh3kcGw8=; b=mwF6MfSeJzcARIy20xygLsVbqpermBNOVRlehzAQ/Ekcw1vTyPZNoQ0zkHohP99j7A q2M+Zsl2KzXwq5g4Kyzo/RViIjQAZ0k7rSnHw4mkfj3yhCIjEsFDHkhOJ2ONfEltlTsb pZ9eStKVUcS4ZI1XJa4ez6n/v5869ge1NgSYtZF7QTf3FWbitU+8jUSpqStDbIc72FXR Gi+k9T09Y7+Is/dMAACWtMpuyaYFkHOl2H/qZyYT1kNaTtruJBJRPMApkq2Zr5jx82VT 6TDc2F5gLJYo/bQNbEuoLg+xm6piWeuQQF2Qol/JEpZ+2ZveDZeq54yjsb04Mv2crxfl Db2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708515354; x=1709120154; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gMaXkJDSba/rz2ejEwTszeJPuP18TapiV6QGh3kcGw8=; b=PeV5UipWoNnQVT+mdq5Kzzc5Nm7LlfPNDFw2JBoDqsfY1u7pKkVvB0uVPWoR7PIGkD GNDELQldG+grVqu874TMLg5q8RBN8RCr0qMs/lmkxdlZC1yqEVsiKYz7lfoZm2I1pRMw B1+BDF2JtybqwYR0xDTNzu9fZtV/ncjond9+giiyCjnrBCRllBIotwXyZQ15eIwZ2yRW RIvgGx3dE1LtGshA//U3Nk5EilCSaZDD2pUc0kmxEU+hDkVAjoKp21eaG8G4OVovAcMF Oaf05fX7pDZ6yZJAd4775HbKTpBxlo1w3e2UbWCHB0h0bXAcwoEbfuO9YzQOMAmAibHL KEAg== X-Gm-Message-State: AOJu0YwkMp6N9INPg0jFont3uQ6acNApi+yzOMTeGqLORh57AZJVH415 nLSs6YOPvwUZy0IkPIt3ktzmtMIexJ9Cv6eQKDoACXWy97Bwn7Ali01B/XEQSF2yfBJ8pscwbQv tVsbV1G8kTRWQ79RRR6zFLhGbSgK4cNN/JtVi3yZ+NIdDgtPQc92gHO+NY+/rAzjiIZkdjhdKwf MiE6CqErk932/p7T12vfdGKI4XKdgCMg== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:5a8f:0:b0:33d:1720:87e5 with SMTP id bp15-20020a5d5a8f000000b0033d172087e5mr31048wrb.7.1708515353734; Wed, 21 Feb 2024 03:35:53 -0800 (PST) Date: Wed, 21 Feb 2024 12:35:15 +0100 In-Reply-To: <20240221113506.2565718-18-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240221113506.2565718-18-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3963; i=ardb@kernel.org; h=from:subject; bh=94XZSCg7mCe31gx3p7nGco09u3mc8zaP+76NhJPF6Y4=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXq/Q/T726tcz/f/oVTLUTl/2l27j+Pc11F7+nPCk591 RSXHezZUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACbSu52R4eCDV/8SvmkZ+zZJ 7Vv3NmFm4aOAZ0oPxJpFP1+Tlll/r5qR4RHzlsOMhtcT5jyK6uDS/rHj/b4luWd+coh0iKfMOip nxQkA X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221113506.2565718-26-ardb+git@google.com> Subject: [PATCH v5 08/16] x86/startup_64: Defer assignment of 5-level paging global variables From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791508373349467495 X-GMAIL-MSGID: 1791508373349467495 From: Ard Biesheuvel Assigning the 5-level paging related global variables from the earliest C code using explicit references that use the 1:1 translation of memory is unnecessary, as the startup code itself does not rely on them to create the initial page tables, and this is all it should be doing. So defer these assignments to the primary C entry code that executes via the ordinary kernel virtual mapping. Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/pgtable_64_types.h | 2 +- arch/x86/kernel/head64.c | 44 +++++++------------- 2 files changed, 15 insertions(+), 31 deletions(-) diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm/pgtable_64_types.h index 38b54b992f32..9053dfe9fa03 100644 --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -21,9 +21,9 @@ typedef unsigned long pgprotval_t; typedef struct { pteval_t pte; } pte_t; typedef struct { pmdval_t pmd; } pmd_t; -#ifdef CONFIG_X86_5LEVEL extern unsigned int __pgtable_l5_enabled; +#ifdef CONFIG_X86_5LEVEL #ifdef USE_EARLY_PGTABLE_L5 /* * cpu_feature_enabled() is not available in early boot code. diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 72351c3121a6..deaaea3280d9 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -23,6 +23,7 @@ #include #include +#include #include #include #include @@ -77,24 +78,11 @@ static struct desc_struct startup_gdt[GDT_ENTRIES] __initdata = { [GDT_ENTRY_KERNEL_DS] = GDT_ENTRY_INIT(DESC_DATA64, 0, 0xfffff), }; -#ifdef CONFIG_X86_5LEVEL -static void __head *fixup_pointer(void *ptr, unsigned long physaddr) -{ - return ptr - (void *)_text + (void *)physaddr; -} - -static unsigned long __head *fixup_long(void *ptr, unsigned long physaddr) +static inline bool check_la57_support(void) { - return fixup_pointer(ptr, physaddr); -} - -static unsigned int __head *fixup_int(void *ptr, unsigned long physaddr) -{ - return fixup_pointer(ptr, physaddr); -} + if (!IS_ENABLED(CONFIG_X86_5LEVEL)) + return false; -static bool __head check_la57_support(unsigned long physaddr) -{ /* * 5-level paging is detected and enabled at kernel decompression * stage. Only check if it has been enabled there. @@ -102,21 +90,8 @@ static bool __head check_la57_support(unsigned long physaddr) if (!(native_read_cr4() & X86_CR4_LA57)) return false; - *fixup_int(&__pgtable_l5_enabled, physaddr) = 1; - *fixup_int(&pgdir_shift, physaddr) = 48; - *fixup_int(&ptrs_per_p4d, physaddr) = 512; - *fixup_long(&page_offset_base, physaddr) = __PAGE_OFFSET_BASE_L5; - *fixup_long(&vmalloc_base, physaddr) = __VMALLOC_BASE_L5; - *fixup_long(&vmemmap_base, physaddr) = __VMEMMAP_BASE_L5; - return true; } -#else -static bool __head check_la57_support(unsigned long physaddr) -{ - return false; -} -#endif static unsigned long __head sme_postprocess_startup(struct boot_params *bp, pmdval_t *pmd) { @@ -180,7 +155,7 @@ unsigned long __head __startup_64(unsigned long physaddr, bool la57; int i; - la57 = check_la57_support(physaddr); + la57 = check_la57_support(); /* Is the address too large? */ if (physaddr >> MAX_PHYSMEM_BITS) @@ -465,6 +440,15 @@ asmlinkage __visible void __init __noreturn x86_64_start_kernel(char * real_mode (__START_KERNEL & PGDIR_MASK))); BUILD_BUG_ON(__fix_to_virt(__end_of_fixed_addresses) <= MODULES_END); + if (check_la57_support()) { + __pgtable_l5_enabled = 1; + pgdir_shift = 48; + ptrs_per_p4d = 512; + page_offset_base = __PAGE_OFFSET_BASE_L5; + vmalloc_base = __VMALLOC_BASE_L5; + vmemmap_base = __VMEMMAP_BASE_L5; + } + cr4_init_shadow(); /* Kill off the identity-map trampoline */ From patchwork Wed Feb 21 11:35:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 204174 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp978976dyc; Wed, 21 Feb 2024 03:38:49 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWwz1/GjXXZoeO+xN2f4QTKS/yzBUik7EyXEjNLogLNwUm74yIJPjxEfAiAMZ04T+sIAa/T9c3F3Bgyisrw6pHb2/duOw== X-Google-Smtp-Source: AGHT+IHWAWBSb7Jkd661UVnglULQ9VQ8MtkANoH02seiOZ8aeQ2ppgMBveiAtYZasgLePqqdGyhO X-Received: by 2002:a05:6871:3a2a:b0:21e:e0d3:41b8 with SMTP id pu42-20020a0568713a2a00b0021ee0d341b8mr7883249oac.51.1708515529674; Wed, 21 Feb 2024 03:38:49 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708515529; cv=pass; d=google.com; s=arc-20160816; b=QWcdbocxhgaZbP65eFCz1rmdSmkYo6+KjNId6+osxCFK6w2gQTOM97ljv3A//ik2jW TeIgWUCSeDIhAZWvTQ9LkFAloLxfi02iW/SgCzX5IrLst1RSVNnuNAIrmbf5y65zLnch QpOWIELeoOHJegrZ5Gj4/xlqwsKOpioSFo27QEYfS4+rDA6KdXTLChgeS2V5fGc5O5GD prtl39WOmpH2KB9DaVFDms5VbFUiMt+rVceAKTTwj+r3Fu1ySNBehdNBlAw2ybc3pD8e MVryjS5dZLVillWJ7f7SfkAFkZ7K4SMZ7qQBFMOYr1qHjCKN7vne/i9RM5N9Tch0SUhh SSHQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=StiHl4wqKJLROSGDCkKhKFhOVcRJSTp0SGrZr/uAC14=; fh=fMhMPvo8duafGkM4xZHdH7fULbau3etefTGsKa+Qy7I=; b=h8eHt/NB2wnoA4sD/nbmVEu+0ocvUT1eV69amjTY7QcRxeU/4uvZoAGZjUvRSuyoNb jv/P9YGQVj1J3YT/kiDMAWOwvUgqvy9YINGxSCDybV7li8u3+0NNko8CVnWVnsdyyrRu 9oCRKBhkhITXx40qes5H/l9TUitHKi61owf0bUcZqqH8FR+qY4KBfOFJnFVo05SN0EIO 5W6RBnaSXdtQjfqT5OUx488xVPVzoKtvIhXJ9na3E9SaA6uUCRmM4a0r01VBG72dHNkY evps94xTCvFqcFC64oXJPGgkFuycGWeB+FdTqmGfXn2djlszTeKjyet59GKiDoTskJt0 2MDw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=nOEY4Zuw; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74639-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74639-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id x62-20020a638641000000b005ceb4a70188si7838369pgd.375.2024.02.21.03.38.49 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 03:38:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74639-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=nOEY4Zuw; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74639-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74639-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 71BDD2845D4 for ; Wed, 21 Feb 2024 11:38:49 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0F3994DA18; Wed, 21 Feb 2024 11:36:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="nOEY4Zuw" Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5CAA24DA12 for ; Wed, 21 Feb 2024 11:35:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515359; cv=none; b=Y6slJFDhNImJsUzX/r/pJODdgdNS0jGRV73kWz0eV37yicBDJ+jaZvV7nCyecUHD1lvz2yBehexWGZG8oJy3vStI23MH+4vGXz7PaKuIV5bf8N2usU+SLYmq57xYcBtAlvBCtPs7irRCP/QJHbkGBgeR1G81fd6fLVM+aUAOQZ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515359; c=relaxed/simple; bh=sFg81TTWm8wXNN7PrKjakH1R2yINKbC6VHscAn/WI6g=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=HcFjS5yBAkU95JUCkrHssFRJD99joelXs9uHl99a/FCzmuGgcdQaCrk024YJLLWHmlcZJe37p/nrZjRJHDdket93wpUz0uUFJgWZ9aJw0O+nXyGxspJsUcwDwWYISLzefCt6ZSwJ0DUCf0VMJeDEdz1DP25uuRSI/a4dVn2A6aQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=nOEY4Zuw; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc74ac7d015so8247302276.0 for ; Wed, 21 Feb 2024 03:35:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708515356; x=1709120156; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=StiHl4wqKJLROSGDCkKhKFhOVcRJSTp0SGrZr/uAC14=; b=nOEY4ZuwYnUemy5QE8lfocHWUrTUBMlf7x2hSNcNuKDt0JVRMtBGocvfZUoG8yd6QW IvFaKhOuDnTxRwG3f2ClgiC0OSs1qVW9wKn/JYqPIXM5sDQOjCMJcx5lEFQMxVmkbsG4 DXmqtlhGXMZMJbqSho8m4yiXLSgJMJE5F+McItHYyo248tCjyAfiK28zFYFksuD/sEP7 ftjbbhNhvdNYUnpk4ExA1AYDfYKJbYUjsyw9yaaVeKhyR8q2MK73WEk+7N42hRJ8zHhZ 49BZ2o1yREoTEcB5zdn6F5Jje0arYAFecz6I1qtppAJYhrs+RTOUZ5exO/keAuU3J7DA zhDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708515356; x=1709120156; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=StiHl4wqKJLROSGDCkKhKFhOVcRJSTp0SGrZr/uAC14=; b=d1KBQn61ikUXykpnXOmjcH4g6FOpieL3fYDrqGlk0PDGZnmHHd6m0AzMy+ehRRATB+ znIhrNiithZaTlpTMWCBPZnk842FOjAXIE9Tw00taHZZVu5Gx1J8izIccfrlxi3D5BiJ QdqpW1Y87wR3bZNdmHxQwoCBSrMizY2igDd9fSGQNUF0Ks374heaLPfMH/OfgurswZYG WHRHy+B5bxLC7cWlUvYULGb8sXrnsxWZ8d3EvWv/8/soM5Qf8jbH9+4y3cdW5N7c6HAA mq461XaQjsLdrMBIzuQORi5apNVDnMSRPjRRIdVscDj5c67G8LjDr9l5nriizEa6e510 ve5w== X-Gm-Message-State: AOJu0YwLhBvdOuuWuQwt+QMR9MkPMggV54Ipkmwh+zFnl8bJuhreHQM/ jI1pVR11PXlmWniWFR+ibdXz3Oh9dYguSIJURF++uCHgEmPNc5/ZQutAZpWLN5b4VPB8vGlj34r SU8wp9Or2a0u2tV7Ek5KNhmMbTRcDGpmbgb0eS2BXMxMctoOw81sswKY2FM+jLIHrD3kC8sgrQA Uh6pczYYBVWeE0TplSY7fIf78WhVQJ4A== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1001:b0:dcc:79ab:e522 with SMTP id w1-20020a056902100100b00dcc79abe522mr731868ybt.11.1708515356342; Wed, 21 Feb 2024 03:35:56 -0800 (PST) Date: Wed, 21 Feb 2024 12:35:16 +0100 In-Reply-To: <20240221113506.2565718-18-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240221113506.2565718-18-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2709; i=ardb@kernel.org; h=from:subject; bh=l8yZBrOFoYa/63WN0vjjrRhrZ18bZIs3VDCL1kGdQvs=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXq/Y/SAgpzDwQo/t+8UsaGgZGnNTtB0tP6SOMEvj7r9 bnBk8M7SlkYxDgYZMUUWQRm/3238/REqVrnWbIwc1iZQIYwcHEKwESuczEy7HkrtmrlmlsVE/6y rebZrWbY2Ru8c/L2xheu/8PPScle6mP4za68hrVhTciUbfzS0/YXybbfFL7BPKVxwclLnTWynnf ZmQE= X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221113506.2565718-27-ardb+git@google.com> Subject: [PATCH v5 09/16] x86/startup_64: Simplify calculation of initial page table address From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791508380038488095 X-GMAIL-MSGID: 1791508380038488095 From: Ard Biesheuvel Determining the address of the initial page table to program into CR3 involves: - taking the physical address - adding the SME encryption mask On the primary entry path, the code is mapped using a 1:1 virtual to physical translation, so the physical address can be taken directly using a RIP-relative LEA instruction. On the secondary entry path, the address can be obtained by taking the offset from the virtual kernel base (__START_kernel_map) and adding the physical kernel base. This is implemented in a slightly confusing way, so clean this up. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head_64.S | 25 ++++++-------------- 1 file changed, 7 insertions(+), 18 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 426f6fdc0075..b92031d7e006 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -111,13 +111,11 @@ SYM_CODE_START_NOALIGN(startup_64) call __startup_64 /* Form the CR3 value being sure to include the CR3 modifier */ - addq $(early_top_pgt - __START_KERNEL_map), %rax + leaq early_top_pgt(%rip), %rcx + addq %rcx, %rax #ifdef CONFIG_AMD_MEM_ENCRYPT mov %rax, %rdi - mov %rax, %r14 - - addq phys_base(%rip), %rdi /* * For SEV guests: Verify that the C-bit is correct. A malicious @@ -126,12 +124,6 @@ SYM_CODE_START_NOALIGN(startup_64) * the next RET instruction. */ call sev_verify_cbit - - /* - * Restore CR3 value without the phys_base which will be added - * below, before writing %cr3. - */ - mov %r14, %rax #endif jmp 1f @@ -171,18 +163,18 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) /* Clear %R15 which holds the boot_params pointer on the boot CPU */ xorq %r15, %r15 + /* Derive the runtime physical address of init_top_pgt[] */ + movq phys_base(%rip), %rax + addq $(init_top_pgt - __START_KERNEL_map), %rax + /* * Retrieve the modifier (SME encryption mask if SME is active) to be * added to the initial pgdir entry that will be programmed into CR3. */ #ifdef CONFIG_AMD_MEM_ENCRYPT - movq sme_me_mask, %rax -#else - xorq %rax, %rax + addq sme_me_mask(%rip), %rax #endif - /* Form the CR3 value being sure to include the CR3 modifier */ - addq $(init_top_pgt - __START_KERNEL_map), %rax 1: /* Create a mask of CR4 bits to preserve */ @@ -204,9 +196,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) btsl $X86_CR4_PSE_BIT, %ecx movq %rcx, %cr4 - /* Setup early boot stage 4-/5-level pagetables. */ - addq phys_base(%rip), %rax - /* * Switch to new page-table * From patchwork Wed Feb 21 11:35:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 204175 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp979085dyc; Wed, 21 Feb 2024 03:39:10 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUoqt1nQX8g3HcYHYjytUuT650mMe9WbREEekjATx7XMGVMBy4UM3kWz/HXUjvjnEU6IxkpDeCXrc6zmyonHuYAfId0Pg== X-Google-Smtp-Source: AGHT+IG7FzXCJJx5SvesPnIMa9G95GmoLg7In0e6xpvE1lYnnBrLpKhoGrotPzyiV+Oe1FQJq4lA X-Received: by 2002:aa7:824f:0:b0:6e1:1e05:e0f5 with SMTP id e15-20020aa7824f000000b006e11e05e0f5mr15999617pfn.12.1708515550268; Wed, 21 Feb 2024 03:39:10 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708515550; cv=pass; d=google.com; s=arc-20160816; b=y1jHnc47ykdhZpN6/72GWIMSeHcg5zYHLDPD/Y2j0Ew9o2Ett3U7qqJsam/zCtz3Nu ErwHqqi4ERor+Pi9prWQWYt1vgJsCJuu+ig4VMPXK+b53YhlIX6TGHBjE9VyDVERVSYK 7wf+UhseDTAjcIcXNbZbys5W9Sm4oo6jsrrywI+iUvxkAnvXzBRkJ8IwBqfYKFG6XgUQ /haC7mJ/4Sl/WcnHAlwFr4KKmU6RW+WEIs5wpLCmSHkXWTExN75Hc8FIJwnIqQn436o0 Lp4o0+TNw0pPRSpw8KvzCN4Khx5qDvltlwEvrAwUsyKq/tXxJrD39F2ordmVHnRsAOBW vRMA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=PTqJ99P6i6SBAIAH+IAXdMn7P2pIU1w2UuJtp90ikvg=; fh=fMhMPvo8duafGkM4xZHdH7fULbau3etefTGsKa+Qy7I=; b=Gtmh28aHHYT0k7EaCZM9ncpPWD4lo3dIFIWHW09pD1XcFFJcnTQfhIHh7jvN9342jD 0RQQHGrtRk/soAGK0IHauT6lTp/pzg/1tAUeAVUJ7LYyLxP/QwYps9xA3iPftmpiPOYX pofRBd1Yu31092GCNtHPSYtctmHE+VOVhikn6nwhuOuHI6k3446lk3s5YCq9TgAF/xQB THM9aHZ8yibABcBlxM0JcpN8w/pbRjASqXNhrvfdaYMUNJMoUuPzSNLmST48cqStZVm+ X/DL6FXVtB0pHsNo0cDQZwwnSV2xZiaeg24pYZMlN2tNC/pEg0P2plu5F560yd0W7YfG IG/A==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=uOOGWx0T; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74640-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74640-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id bw38-20020a056a0204a600b005dc80a2a308si7994347pgb.436.2024.02.21.03.39.10 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 03:39:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74640-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=uOOGWx0T; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74640-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74640-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 17D182846F4 for ; Wed, 21 Feb 2024 11:39:10 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id CB3654EB45; Wed, 21 Feb 2024 11:36:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="uOOGWx0T" Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9237146535 for ; Wed, 21 Feb 2024 11:36:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515362; cv=none; b=oUcw+fsO9PM7YYDrt6jiRW5GinLD6IUjxFvCupVfrdcpF4uKjl3mnYGFbUsja597D7sgBHmRy9TJKYW6M9PtJkS+rIFuDrayr3gNhPXWWiajYPnOUlKt56VPuaeYQxH7tZWt6jSfdXXTrEpHiHkO6YBoi9UuJPIbaqZgaoCKQJU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515362; c=relaxed/simple; bh=G1DG06D2aqp4pwBOMLdzbyqBLE1i1IBpz8O+pTeBX18=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=WD+gifH4v85ak8moibKkc9zQhwGCWrSHfQqesf/ADVG0QyhzV7q+75sk0O1oAr63UaiTdJjCmrAzbOtpywHe4yVIE+0vLLWvV5u9NK82LrRq9LIhiA5og3C9XlHUqVAS25ZXfO3bEzF1osuNuuyApZhN5YZB2xLVaIElZ7CPnxA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=uOOGWx0T; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-41256ffe281so19047205e9.1 for ; Wed, 21 Feb 2024 03:36:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708515359; x=1709120159; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PTqJ99P6i6SBAIAH+IAXdMn7P2pIU1w2UuJtp90ikvg=; b=uOOGWx0Tsum9AyZZbdXGqFrO0o/LYVm3Px09l2rC2lcQKiC3SJL3rMqfC5dYMQPRNI 3t6mdzeWMijdlayAr1Q7DcXz1EGT7VJpA8tTIjzorGDUYtaXFFbyceFcBoTc7XIM+O0E dmbRkSKBkIQPmv1l2ZQo8r10yQUiqmGeJV0Dp1Fd0U2j6SbtQcU08SZcyU1g19vEZ/2U zuqG+wNQyrMXIPH4hfCc51ch4vXr6ZyEF95nLKt24tywjn95gSXIcNN6yESOEXjactrf Kppdpku6a/57HrEP4D5JcX9gGViIlfK4fbaE5OnVQUFhHzPRE9nIGVe1M3KXJK7hUlB3 3pCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708515359; x=1709120159; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PTqJ99P6i6SBAIAH+IAXdMn7P2pIU1w2UuJtp90ikvg=; b=oxPFcD6HsJFK8Bbgnjfylg6jYkOMT91G7KkiT4fDG5J8F4ZF5TGHFDOZhGQ1KyHbX4 OyhOsoypU+UbA/sF1utgkWb6DU9RDw6gR63qv3QDCeZCiTcrcz+ruv1ZFujLcQkBUEA0 1NrP+8aqMjUrYeWylF7OzOyzi4WbavbSlMPE1IZb1U4kKtU+4RCZE4wAD77gqq2jtTkb CxtXIRrQKcoILWZ+jEjuxp/eYy2DDck3xx3qWApfbYgTADNmbe9SfDDJKIE18nWOr7ue rvMUSbyn4rzx/2PK3K75EjLc5Jp4bwv15HQP9Sd+t6Jh7dO06H6hOS5AP7Obzle31d2a h88Q== X-Gm-Message-State: AOJu0YybzeT9C6x00M1wkEiqwpUprMHEITfaWsU862aFkXJ5rcyN2Qo6 osXx0OcyED1DRzvN8IKIl7E8RUSn2kil6SnwzxMJsxilcI8GROmHae9LuuNMPjoi2u9JTB9pgbm h1Yy12w1Ds0sOdJf1ka6AdSzKSkRJ62GubRw/aNnsDShspB+N1JRgSuirLviHMzR0QiqXi1blxt 9xoeEeuuBQNCKRMFE7vomk/2OLDbpU0A== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:1f17:b0:412:72d3:cfdb with SMTP id bd23-20020a05600c1f1700b0041272d3cfdbmr10352wmb.2.1708515358491; Wed, 21 Feb 2024 03:35:58 -0800 (PST) Date: Wed, 21 Feb 2024 12:35:17 +0100 In-Reply-To: <20240221113506.2565718-18-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240221113506.2565718-18-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=4361; i=ardb@kernel.org; h=from:subject; bh=i7YTwnF8QsOAYHIVebyuOmPcW0rncRMhxzgGUyIgY38=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXq/U/vplsk/HzNuOys4hb23w4nvt/7HBDut6vewlGn6 NxOleWtHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiIYsYGS7F3dGZ7vRpnUPq w7kBIQdLeRyv/F3AEvfgbefPCUflymwZGfYpGMxXyxfjX+4aMyFzvdIWPnWn14yXL3sF7Tljten 9XxYA X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221113506.2565718-28-ardb+git@google.com> Subject: [PATCH v5 10/16] x86/startup_64: Simplify virtual switch on primary boot From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791508401759905551 X-GMAIL-MSGID: 1791508401759905551 From: Ard Biesheuvel The secondary startup code is used on the primary boot path as well, but in this case, the initial part runs from a 1:1 mapping, until an explicit cross-jump is made to the kernel virtual mapping of the same code. On the secondary boot path, this jump is pointless as the code already executes from the mapping targeted by the jump. So combine this cross-jump with the jump from startup_64() into the common boot path. This simplifies the execution flow, and clearly separates code that runs from a 1:1 mapping from code that runs from the kernel virtual mapping. Note that this requires a page table switch, so hoist the CR3 assignment into startup_64() as well. And since absolute symbol references will no longer be permitted in .head.text once we enable the associated build time checks, a RIP-relative memory operand is used in the JMP instruction, referring to an absolute constant in the .init.rodata section. Given that the secondary startup code does not require a special placement inside the executable, move it to the .noinstr.text section. This requires the use of a subsection so that the payload is placed after the page aligned Xen hypercall page, as otherwise, objtool will complain about the resulting JMP instruction emitted by the assembler being unreachable. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head64.c | 2 +- arch/x86/kernel/head_64.S | 43 ++++++++++---------- 2 files changed, 23 insertions(+), 22 deletions(-) diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index deaaea3280d9..0b827cbf6ee4 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -553,7 +553,7 @@ static void __head startup_64_load_idt(void *vc_handler) } /* This is used when running on kernel addresses */ -void early_setup_idt(void) +void noinstr early_setup_idt(void) { void *handler = NULL; diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index b92031d7e006..03268bf0214a 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -39,7 +39,6 @@ L4_START_KERNEL = l4_index(__START_KERNEL_map) L3_START_KERNEL = pud_index(__START_KERNEL_map) - .text __HEAD .code64 SYM_CODE_START_NOALIGN(startup_64) @@ -126,9 +125,22 @@ SYM_CODE_START_NOALIGN(startup_64) call sev_verify_cbit #endif - jmp 1f + /* + * Switch to early_top_pgt which still has the identity mappings + * present. + */ + movq %rax, %cr3 + + /* Branch to the common startup code at its kernel virtual address */ + ANNOTATE_RETPOLINE_SAFE + jmp *0f(%rip) SYM_CODE_END(startup_64) + __INITRODATA +0: .quad common_startup_64 + + .section .noinstr.text, "ax" + .subsection 1 SYM_CODE_START(secondary_startup_64) UNWIND_HINT_END_OF_STACK ANNOTATE_NOENDBR @@ -174,8 +186,15 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) #ifdef CONFIG_AMD_MEM_ENCRYPT addq sme_me_mask(%rip), %rax #endif + /* + * Switch to the init_top_pgt here, away from the trampoline_pgd and + * unmap the identity mapped ranges. + */ + movq %rax, %cr3 -1: +SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) + UNWIND_HINT_END_OF_STACK + ANNOTATE_NOENDBR /* Create a mask of CR4 bits to preserve */ movl $(X86_CR4_PAE | X86_CR4_LA57), %edx @@ -196,16 +215,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) btsl $X86_CR4_PSE_BIT, %ecx movq %rcx, %cr4 - /* - * Switch to new page-table - * - * For the boot CPU this switches to early_top_pgt which still has the - * identity mappings present. The secondary CPUs will switch to the - * init_top_pgt here, away from the trampoline_pgd and unmap the - * identity mapped ranges. - */ - movq %rax, %cr3 - /* * Do a global TLB flush after the CR3 switch to make sure the TLB * entries from the identity mapping are flushed. @@ -213,14 +222,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) btsl $X86_CR4_PGE_BIT, %ecx movq %rcx, %cr4 - /* Ensure I am executing from virtual addresses */ - movq $1f, %rax - ANNOTATE_RETPOLINE_SAFE - jmp *%rax -1: - UNWIND_HINT_END_OF_STACK - ANNOTATE_NOENDBR // above - #ifdef CONFIG_SMP /* * For parallel boot, the APIC ID is read from the APIC, and then From patchwork Wed Feb 21 11:35:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 204176 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp979141dyc; Wed, 21 Feb 2024 03:39:24 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWxepK9Z7CYb0tAzOQCfLcrHJyd5tk91Zzo2fW4cwj8ejpwrYQjdWDkyP/C3AxbMj7uxASsVgX46tNUcS/RuFSC0S3/Rg== X-Google-Smtp-Source: AGHT+IFF6X4My7omfBert9d6HuMYaIf5d1+f/yA8CBlG5us7q9vs05PGXP9M12x10hj1Zt8qFFjS X-Received: by 2002:a05:6871:823:b0:21a:34e1:bfd3 with SMTP id q35-20020a056871082300b0021a34e1bfd3mr17137354oap.25.1708515563753; Wed, 21 Feb 2024 03:39:23 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708515563; cv=pass; d=google.com; s=arc-20160816; b=NyR7KRvls9tEs0skwGhnCxpUDcs3270a5s8cCXhQdVGIQRtKQbZWhrdvOnxdTD4pz5 E8Wk4f21NlOI2r/dKQ8G7X6WzyV9pZu3ZhhnVssUAeJQyhF69HaL5P/fxuO6o5NXNDGO p6kuAKdygj+7jhJk1lQbINXlub+mjT94nCPfUYJakhEYexnv/kGF0slPxu1PytH61D9p 0AzQKsmEUlLitl96HXHG2zcDkSkc57/pBAAlmyCHJB1wq85cYrqV267CyK/2yDql6mFX ljcd0vxRINOTFilY8+6jNQBT+H2ATovKg2enkjq0yj6/7fXjxCxRZlTj7o6zxy3XY7cx 7+Mg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=tevHFXJ+OOPc1/guz1Xb2v+32aOKjnUq6Qv962Lc8AE=; fh=fMhMPvo8duafGkM4xZHdH7fULbau3etefTGsKa+Qy7I=; b=VqPVLJOIDVkVoYmtppTxG24PcPPk445O29td2c3BZtyYb36Zdn9vn1LoyCck3RNGts uUdszPUQvA37CbhjlVE9JwGPPPECGLLVQoqPnUqRdvbwvIZidfBxWYvrVzJxXPa8XVuU uKw/MOzl5GFd9bKsXF7qpNnPPtxSsFjdrHaEXzIyfMbsI+LPoGpNZiUkgu5EkO2lEqiX bhx72YTuuoymCOvTF2W4stQVdP6Lduoo1qrGKPoikH2/sVbueXIpa9e0T/I4gEYfGdWH J++zxK7qNVT6iS20zsU0YE2sa1TknnVtwoxbxSg45uhbN62akwj0MNYRbh5cEyXtCcTb Hi4g==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=xWTU47Dk; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74641-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74641-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id bw38-20020a056a0204a600b005dc80a2a308si7994347pgb.436.2024.02.21.03.39.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 03:39:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74641-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=xWTU47Dk; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74641-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74641-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 890032836AA for ; Wed, 21 Feb 2024 11:39:23 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 927634E1DC; Wed, 21 Feb 2024 11:36:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="xWTU47Dk" Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8DF5D4E1BF for ; Wed, 21 Feb 2024 11:36:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515364; cv=none; b=gdIFjXQVi0lh63bMcxkXIq5+ngFQ4x2pXY3r0iXmyGXdURk1aTbWQ/Bb4VN/yD+NdIdFieb9/VBelpYP85oBrtcr8aDB5eFMkgCEwrmVf0n9wf/pSxVc20amz8wOmes8YzTqwdzhZz+v4Uu0/xzYobWn1R9yNTW5ixX1RRVdfuc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515364; c=relaxed/simple; bh=iv8WXooCR9hoiH3pMKa/9TfgmsuasaiYtcPtFFzw3QI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=SndZMASpAwZtA2R4SLvbd4cBjm7NHnquIlxiRsU7x54GSxT4iCzA1gNtf7et4RZfOqb6EvdixKC6yLL312dmd8xRpbZG9Bt9osBJWKHEF2JVshE1g2f8/rBGU2TXfWe3WZc3tkuICQCEc2+XHTyDzhukZeIysGW4+cwFEhDLMzU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=xWTU47Dk; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-33d60ac6781so1120646f8f.0 for ; Wed, 21 Feb 2024 03:36:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708515361; x=1709120161; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tevHFXJ+OOPc1/guz1Xb2v+32aOKjnUq6Qv962Lc8AE=; b=xWTU47DkRFKzKWef6Xy3xIpCrckMs8ylSGlcHq6QG2fGWjUr25x7s0JdtdL4v+iT4k /CJIZ/jWUUXvGNfbgu1unMo/ZbrSI9vY85IykkoTZjLCm4FD+p1T6vW/s/P614pfHMJz nlUv+GOQjbXf5GfqvM4B5IFw6yZdRtIa5V3mLWVX6KMplqdp8Ykgo7B1Q4sOaIfbOkP7 HepGp0lf1xSk9v/4U3Mrp2+TjCKNKSvOrM32hOq0qKntB/tiOyS2VrjMX54deER415ek MAhmGMDg5eoKvIPePnzOMOjIqsw+B0KqYmWg/xT6d7LN+UGqEis5P+yHGJlKCfwUVPE/ Ugzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708515361; x=1709120161; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tevHFXJ+OOPc1/guz1Xb2v+32aOKjnUq6Qv962Lc8AE=; b=CalSoDkRQgbGSrxcrK9ByeKbN5yNTW7pQC2htk3PCh/zCvTmS8+v0QRiP+0Fg7MhdC +ZgqHPYQFRbG5FG7+QDmsr/nXGpEh9POqhHO7kYbCl6tP+B1jjosQzitLa/smG0P95oE 8EanDf8QYtHQ+wtV9zoCFxih/+KIlq/J1roZ1QEoRK8eE5f9jL7MUqDRyEI2R5HY82Oh OoZct6l+O95rQDejeZhc1DCXhXtCquMfRMZUQ0lHetsa/aVh7wDciiP8QYEiG0Ve4eIS FS36hEzwDSGVzk2k/BQBmNmOSYfzayNccJZsxbDZwQ2NcvykrEwv9d3bn1oskEDJDTXv pS4A== X-Gm-Message-State: AOJu0YwvwoMQpEvAxC5hRzfWpz7kRZAtfR+swdfqrmGU6X/zyMWWLYjf LwBDWfJrBnprNSII+X8/5J2unD3OZfyMcj/AQuWK+117iPw/wzmB+tunFv3AbvZ8N1/ldZdbCJy KGyniS2K0x8rgktmLU0565yuMNWZyepOcTQzPxvun1es+KldvaBPXWDmH7n+MDteIcRpb8m+nUw Kp1RPC9JCX9pnguXs8HYlDtygNkv8JHA== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:6d8f:0:b0:33d:4dc7:ee2 with SMTP id l15-20020a5d6d8f000000b0033d4dc70ee2mr57756wrs.5.1708515360743; Wed, 21 Feb 2024 03:36:00 -0800 (PST) Date: Wed, 21 Feb 2024 12:35:18 +0100 In-Reply-To: <20240221113506.2565718-18-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240221113506.2565718-18-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3958; i=ardb@kernel.org; h=from:subject; bh=PHNTEzBLgbyUsxeGtN9QWmwRz8NSWIe83iLY3QrvkGo=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXq/U9ZK1dd2mLOnCwavPTfm2Pn2/Vrn9zLPzbvtSdvW JNl16R7HaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiPr6MDE0nLl1Z69QwX87h H+/V5a/uLhVSuvys+d217dnHq66W6eYw/C+Zf6v8zv6ogAtOaxmU4y/sNfu1tKlEzUAxfNnSZ89 u/eUAAA== X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221113506.2565718-29-ardb+git@google.com> Subject: [PATCH v5 11/16] x86/sme: Avoid SME/SVE related checks on non-SME/SVE platforms From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791508415828182446 X-GMAIL-MSGID: 1791508415828182446 From: Ard Biesheuvel Reorganize the early SME/SVE init code so that SME/SVE related calls are deferred until it has been determined that the platform actually supports this, and so those calls could actually make sense. This removes logic from the early boot path that executes from the 1:1 mapping when booting a CONFIG_AMD_MEM_ENCRYPT=y kernel on a system that does not implement that (i.e., 99% of distro kernels) Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/mem_encrypt.h | 4 ++-- arch/x86/kernel/head64.c | 6 +++--- arch/x86/mm/mem_encrypt_identity.c | 8 +++----- 3 files changed, 8 insertions(+), 10 deletions(-) diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index b31eb9fd5954..b1437ba0b3b8 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -48,7 +48,7 @@ void __init sme_unmap_bootdata(char *real_mode_data); void __init sme_early_init(void); void __init sme_encrypt_kernel(struct boot_params *bp); -void __init sme_enable(struct boot_params *bp); +void sme_enable(struct boot_params *bp); int __init early_set_memory_decrypted(unsigned long vaddr, unsigned long size); int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long size); @@ -82,7 +82,7 @@ static inline void __init sme_unmap_bootdata(char *real_mode_data) { } static inline void __init sme_early_init(void) { } static inline void __init sme_encrypt_kernel(struct boot_params *bp) { } -static inline void __init sme_enable(struct boot_params *bp) { } +static inline void sme_enable(struct boot_params *bp) { } static inline void sev_es_init_vc_handling(void) { } diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 0b827cbf6ee4..b33f47489505 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -98,9 +98,6 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp, pmdv unsigned long vaddr, vaddr_end; int i; - /* Encrypt the kernel and related (if SME is active) */ - sme_encrypt_kernel(bp); - /* * Clear the memory encryption mask from the .bss..decrypted section. * The bss section will be memset to zero later in the initialization so @@ -108,6 +105,9 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp, pmdv * attribute. */ if (sme_get_me_mask()) { + /* Encrypt the kernel and related */ + sme_encrypt_kernel(bp); + vaddr = (unsigned long)__start_bss_decrypted; vaddr_end = (unsigned long)__end_bss_decrypted; diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index 0166ab1780cc..7ddcf960e92a 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -45,6 +45,7 @@ #include #include #include +#include #include #include "mm_internal.h" @@ -502,18 +503,15 @@ void __init sme_encrypt_kernel(struct boot_params *bp) native_write_cr3(__native_read_cr3()); } -void __init sme_enable(struct boot_params *bp) +void __head sme_enable(struct boot_params *bp) { const char *cmdline_ptr, *cmdline_arg, *cmdline_on; unsigned int eax, ebx, ecx, edx; unsigned long feature_mask; unsigned long me_mask; char buffer[16]; - bool snp; u64 msr; - snp = snp_init(bp); - /* Check for the SME/SEV support leaf */ eax = 0x80000000; ecx = 0; @@ -546,7 +544,7 @@ void __init sme_enable(struct boot_params *bp) feature_mask = (msr & MSR_AMD64_SEV_ENABLED) ? AMD_SEV_BIT : AMD_SME_BIT; /* The SEV-SNP CC blob should never be present unless SEV-SNP is enabled. */ - if (snp && !(msr & MSR_AMD64_SEV_SNP_ENABLED)) + if (snp_init(bp) && !(msr & MSR_AMD64_SEV_SNP_ENABLED)) snp_abort(); /* Check if memory encryption is enabled */ From patchwork Wed Feb 21 11:35:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 204177 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp979234dyc; Wed, 21 Feb 2024 03:39:37 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCX52zGdOQ6FqwddcDZes2Qxl8tgiX0GKwNz36Ar8P3E8yXv+wA0wggbbLwM/31nazP79nFbjTxigniny1O4l5n9mrDBgg== X-Google-Smtp-Source: AGHT+IGiBKnwikTPAYolxj4mL02q/6gaJwkWz3/Z5gOrRplHLKwDyMB346mpnrjpQbPk94ExQcV6 X-Received: by 2002:a05:6402:202a:b0:565:59a:a103 with SMTP id ay10-20020a056402202a00b00565059aa103mr164585edb.33.1708515577397; Wed, 21 Feb 2024 03:39:37 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708515577; cv=pass; d=google.com; s=arc-20160816; b=HQD3mMPi98lb9KRqNSBby9eg5gEIfVM0QIBZ8wVg38Wk2v0x1O5WT5gKGCSzEUy4HF 8Kwy+lhF5CpBYYIg94S9vTv88VM9/0acHTwVamr7bYyHnZrgqKx/YPKRh/PJka1kjs7K umEn6aVD26QE48AvxBIyuRfiPglJWlB00N4TMUFnq8d9eLYUDpyUbWYTila/Knm/Iqcg jl93l5lcrVn2gOCjjlrzz8ceNBM6WegjpQk+JAKjMJc7OzUS04gZE6lPul1URS2ehgZi iyy5dvcgFp8cBr/NrLJyFBUK0GaXq82shiv4Q42QhdRGBGPYeO/WWsYm1IS1svb0LZxO R4qg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=ygqZ0HdVfjjnya+uCKEPe+HOAs/11yLIprq08KYDjCU=; fh=fMhMPvo8duafGkM4xZHdH7fULbau3etefTGsKa+Qy7I=; b=paIlCY3uh8/dsMGdtEJqgUaQnhqk2GJHsbpLR27n8Eq20g7ozprFunjoEZGO8m0Bv0 DV6mhbeaSXey7SoOstxCO162hprWCLEeiutrQtFoJX+hCyaJIt0NNq+y8Afrn5X/UsUF ZKfY9fzu+3osiebkUAnOlS6BGcbbevKexCcqHtP0/8qo+Kn9snu0gbXdYJsoAKSnQe6Z br1Zi5sYx92B1onMgzDzKHAkmvqdwb1oSMzusvuAa3sDr84/gPqAIiWQvopWUUGt0Bxb e0kQSWTnbKKwv+CdaBrEeBFkoJ/zx4SAScH8FjiwU7ABPMZSPJHqPMLYQ3i55r8FXwY0 /x4g==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=JlK8D5pv; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74642-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74642-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id 27-20020a50875b000000b00564a94db4c2si2008213edv.564.2024.02.21.03.39.37 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 03:39:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74642-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=JlK8D5pv; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74642-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74642-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id D39BD1F2385D for ; Wed, 21 Feb 2024 11:39:36 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 40F6C4F611; Wed, 21 Feb 2024 11:36:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="JlK8D5pv" Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 86F824E1DE for ; Wed, 21 Feb 2024 11:36:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515366; cv=none; b=h69xODYXBHwZ/vF7pPcY2fIsmY7oZGXjmvOmyEFx3TUMXYGfwiyby4Korv02ywdU/bZM4FhszIu7ge1fpJ5g9Xh342a+DYIKT/WLf42mnNA1kb9ZMzvf1q9EgavXGVn5BXJUTh5kI410LOmCu6LpG+tubuDt7BpMG24wX1SltXo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515366; c=relaxed/simple; bh=cyP4xFLZtWmSglaQt1bRPcLQO2AlvE55S6UyeZvMgMM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=miCNVbVoPeTLkkA6JaERCq0j/HBnHrtVOarm2Qn/rBaFvku+WSh/XuRFovucgaoRVAYXuOKmWecLcdkKk48V2thUH/fgn0PkaylEKNZ/CxvHOeC+lzr5kMWPfOYa2ZFcoQR2xHle1/YZz++nMZwabmaD4NLrXXYJJyzOxkJcySk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=JlK8D5pv; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-33d6eb5e5a9so860189f8f.3 for ; Wed, 21 Feb 2024 03:36:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708515363; x=1709120163; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ygqZ0HdVfjjnya+uCKEPe+HOAs/11yLIprq08KYDjCU=; b=JlK8D5pvk7Isl6HzBVdYrAm+zaNAlQxeZDLUziDxhFpQBDJ56Zjnx21H64Ku7KZuc1 CVfkqzNxy14GGjCSaw4GbPcMzv5BMq3c/xfU4MEvmLed8Clsg8yavTsqLncMlh4lUWrb 8q2tbO9TMckC0lYN++2lBPVnKQyNd0i+5zXtIFoZNCBXwqfMWuER5eH3VEFFIJucgwCC hKvAfWJ81UAIAQmTG7UaLpu40yCP69Uphl5+L9uJQLvG+cZ9Z8ZLBv1gaFK3C3zzBOsX s9gOSC+0FQEVSerpb0L/dCNWbj8JTVRQRKrRUF4h4vEGmrSC/FNfOP3AeIqzd8OlL7XX OBAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708515363; x=1709120163; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ygqZ0HdVfjjnya+uCKEPe+HOAs/11yLIprq08KYDjCU=; b=L2EH8dBkeTTVY6z7A9IxH+Ghbyc6DlNkFwj55f4MEvETFJqZN8QQ8WDa3XRVkYTYdS Jw+0Pq0f/FEpLtsjjS4u2ltrvYbOSpx9o2LdDCHLib8/xGRLgdiiA1YFbzSMDAzVLTUV wsIxc8etqt98A0smHOcAX/EsOEIH5SNvfKsDcRlhmsVU+2gIQxBHdLESt0BzkL8IyQyQ EV4+bZ0Gb+KLqvXqWp2PElzVCRCzq4VOi3V1EhMXbeL3pmcfMAi11uRJ+3xiJGGqcWDe pnQZV17PrGGRLZc1dHKXfYvtZcWA8kMZ9wWj26/vQqAUJclYXSl/QbonTuaNNx0ySVAe 6jGw== X-Gm-Message-State: AOJu0Yxbv3esqYDwgZO5FJRg40O8HAgg+H9diZjkD2yfOyuN85ghF6Qe EtK1rIwp1j5E03c2VknT2LmHKuxxuwKedL9oGBiqISUO4t1RoDMjhyM3M7xixaOtsewQ4XrDH3v yLo7VgphUrxeJnEbJl78s8y5V5KbaGgDVlWmeNzFSM2m+ZvR1B01W2xbPsIA7cTzWaWOFVWPLJa HGXeqb/7SDpWhRffxpZteP3MKQuqE+Jw== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:68e:b0:33d:6211:4699 with SMTP id bo14-20020a056000068e00b0033d62114699mr12133wrb.0.1708515362858; Wed, 21 Feb 2024 03:36:02 -0800 (PST) Date: Wed, 21 Feb 2024 12:35:19 +0100 In-Reply-To: <20240221113506.2565718-18-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240221113506.2565718-18-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2183; i=ardb@kernel.org; h=from:subject; bh=FIfYcIT0zte0p7QudMXkiQVk0sZSk5MJFoxtVIwXDvE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXq/c9nhK593//oPlvM+uJnf5Nc71pP9q7bFnDn2vKpe juuWl3h7yhlYRDjYJAVU2QRmP333c7TE6VqnWfJwsxhZQIZwsDFKQATefaLkaGrnXW+jfYSrgsP p9+equA17UpPH4dg3J10EcZptlf7eCUYfrN0v79eFNqz2qvx2ofQ+R1Z77et35Z4RktOxt5mUal rEB8A X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221113506.2565718-30-ardb+git@google.com> Subject: [PATCH v5 12/16] efi/libstub: Add generic support for parsing mem_encrypt= From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791508430250822530 X-GMAIL-MSGID: 1791508430250822530 From: Ard Biesheuvel Parse the mem_encrypt= command line parameter from the EFI stub if CONFIG_ARCH_HAS_MEM_ENCRYPT=y, so that it can be passed to the early boot code by the arch code in the stub. This avoids the need for the core kernel to do any string parsing very early in the boot. Signed-off-by: Ard Biesheuvel --- drivers/firmware/efi/libstub/efi-stub-helper.c | 8 ++++++++ drivers/firmware/efi/libstub/efistub.h | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/firmware/efi/libstub/efi-stub-helper.c b/drivers/firmware/efi/libstub/efi-stub-helper.c index bfa30625f5d0..3dc2f9aaf08d 100644 --- a/drivers/firmware/efi/libstub/efi-stub-helper.c +++ b/drivers/firmware/efi/libstub/efi-stub-helper.c @@ -24,6 +24,8 @@ static bool efi_noinitrd; static bool efi_nosoftreserve; static bool efi_disable_pci_dma = IS_ENABLED(CONFIG_EFI_DISABLE_PCI_DMA); +int efi_mem_encrypt; + bool __pure __efi_soft_reserve_enabled(void) { return !efi_nosoftreserve; @@ -75,6 +77,12 @@ efi_status_t efi_parse_options(char const *cmdline) efi_noinitrd = true; } else if (IS_ENABLED(CONFIG_X86_64) && !strcmp(param, "no5lvl")) { efi_no5lvl = true; + } else if (IS_ENABLED(CONFIG_ARCH_HAS_MEM_ENCRYPT) && + !strcmp(param, "mem_encrypt") && val) { + if (parse_option_str(val, "on")) + efi_mem_encrypt = 1; + else if (parse_option_str(val, "off")) + efi_mem_encrypt = -1; } else if (!strcmp(param, "efi") && val) { efi_nochunk = parse_option_str(val, "nochunk"); efi_novamap |= parse_option_str(val, "novamap"); diff --git a/drivers/firmware/efi/libstub/efistub.h b/drivers/firmware/efi/libstub/efistub.h index 212687c30d79..a1c6ab24cd99 100644 --- a/drivers/firmware/efi/libstub/efistub.h +++ b/drivers/firmware/efi/libstub/efistub.h @@ -37,8 +37,8 @@ extern bool efi_no5lvl; extern bool efi_nochunk; extern bool efi_nokaslr; extern int efi_loglevel; +extern int efi_mem_encrypt; extern bool efi_novamap; - extern const efi_system_table_t *efi_system_table; typedef union efi_dxe_services_table efi_dxe_services_table_t; From patchwork Wed Feb 21 11:35:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 204178 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp979336dyc; Wed, 21 Feb 2024 03:39:52 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUlyr/uqeYrJqYjpbuSpp9/vOzwPAtR4aWt53NGx8qu1zrQ9FRMexZM9mvwl06HaZAKriV/D7WaOBwSTxbpWoDfuOxm7Q== X-Google-Smtp-Source: AGHT+IG9jUoor8IpEI/xYkO6z33N9ocF4BZ+HCmQkO3QG8VqIDxJMPo8J9NsCWzKJpNNLkt+1UUO X-Received: by 2002:a05:622a:216:b0:42d:ae52:76a3 with SMTP id b22-20020a05622a021600b0042dae5276a3mr20535537qtx.1.1708515591956; Wed, 21 Feb 2024 03:39:51 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708515591; cv=pass; d=google.com; s=arc-20160816; b=TBsWo32SR9JIJT3oD3ZoyPaBaQMojxX+HWZSU4JE7iWCuQIpNbGnvzvaYFliIAA1k1 qbe1clSv9heqDB7mBhAvtZFtQH3xMmhGkJ/iHcR5TC/FBKfR5cuRoyR7lJmp5Azz3yDg 4NWuAjoW5CZ28+jxOgovgdnMKIYTGFCatLvb9GEVkjq+HTCnCdWEUnQpjPlMwBM0WpM4 /MeJiD0tBwthGH1zhTAxl7iqutLQ3b9V0kp4RGKTSrUWZSYq3hngfiskd+oeziOFsJrf pH2NKh/1n179JTX3l4tHBOh578LxzwZownoHH8OgjH6EXr99eyLVIadDDdtd9fdAHc+y Omzg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=xDNgWt2lMCvtDV/Eee3l1XVOGlo7HH2KTDyTgOKacbQ=; fh=fMhMPvo8duafGkM4xZHdH7fULbau3etefTGsKa+Qy7I=; b=nyG1pBon/TCBny92B7wejnKJPShud9YD4Ja7LSRZ7bFfW6EcvT6+NibvOeFY4YYzh6 7yHL+HPhJ2CktjZsDfXyrnmx9Z0uhn+Kklzg3evsk8nNzpztx1n1oqARZHl+qdMNY4gx x+wSixAojwei0uQY6/rlApYXrghnpPwEAsZdsOy94JH2tnZuK8U0S6CNhPXlVwSbZbGm pEHrGUKQiB059UiF84e4eex0jXKAU8S2fMwBQ/hjmfrpbx43BIa7br7eikSptJ/enOFg F04bmXhFJaS0IxbW3bFIVY07YseaTwwmxEb9b9XK4joJl+1wAQCf5Y5Bkpuv6EdoGSjo oxkg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=VzRFF5lF; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74643-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74643-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id h4-20020a37c444000000b007859eda4ba9si10423090qkm.750.2024.02.21.03.39.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 03:39:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74643-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=VzRFF5lF; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74643-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74643-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 6E03C1C23000 for ; Wed, 21 Feb 2024 11:39:51 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2EB4850A6E; Wed, 21 Feb 2024 11:36:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="VzRFF5lF" Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C10034654F for ; Wed, 21 Feb 2024 11:36:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515368; cv=none; b=HViae1dUNttAvW3zrxe84P3Arhthl5R06bvVuk1WYtqvopPUiQ6jLaCT5mIk4Lxzoni51ApkPDt58KiNGG8kicHgb73ZFEpsGk+i4m8YCtC5Tle53Ia6w4ai1AxGPik0chOFm6cuRCY3F73U7puYhsjP7bxUw6yYniVtGawZJ7o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515368; c=relaxed/simple; bh=S4NGlxmVOhrQxFAGepaTMi748KWyZv6XTKqvhleJQH8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=FJE6LI/C/1sikmKRhot1ic9JJN0NKQ2e57LQU3nR6HiRQUdT8YAepNpXc6AKwvc5xGu6wYQNTiYEpyRSVn77zCzM/R9xldd5Goy9cddpIWX7SPkr382f4rQS6hE61sbHqmE78NlGRzsJR9cVZ2QNLLUeqSeQsr/JgUd9+89PIZg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=VzRFF5lF; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dccc49ef73eso7569175276.2 for ; Wed, 21 Feb 2024 03:36:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708515366; x=1709120166; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xDNgWt2lMCvtDV/Eee3l1XVOGlo7HH2KTDyTgOKacbQ=; b=VzRFF5lFzKAg9Vc2zjiOzfHdwKTen7ie8RZIuA97pLx1vi/taaK5ogLKf8DIXdNVgN hocoTWaRZaAZAU1W+4aQW6SsUndmmFLDyQMrDTOXhdkKlulr2jm8edFAVtQqHpcD4H8x 8OCefRoiEz6rFOEk1LkssoCak/D5/J4kfwUHVwsyQi//MDgcVSMakenlK/Y79p5QlETM o1tbaXsmw7X1S0LJlh0YMSpZPnQOPg6Af551M/rog1YFa0tMvURRUo08+1RAc+IV8emt 8UgYUGNnRv4vOPTWrjgjxPh9dGz9tOg45ys/oiDOKn8RbHwCwYcJyh4bJj0PmuF8CpBy nvSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708515366; x=1709120166; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xDNgWt2lMCvtDV/Eee3l1XVOGlo7HH2KTDyTgOKacbQ=; b=U1Lp/8XdSPETx8Gzbl3rNeEpIMnrbk/n1XIpRY1tNbflsTDZUEtcandbZbIj1R2LeZ CIphtXaOE4ELIO4xmjGh+UXfpueOflcCE4wi+cys3FjKVuhMfIt9WFO/+ej3KAokTh70 vHgSWls/VtDcp3XIkOiS5nMXkREyIJkCGWxN3GOwjPJsbbfQ65ZYxhkuaU3vdOmm0dK8 JxyxcgDic0yT2t3gEdopXPdMfsecWnG0pb/EYfp4nHLHiM7aovR88OvhouA+SRhefqI9 YRXe5p6rDLr+XraJOmmjpVEEO0kAwrnF+MlZSgS5bqd66orkHOABb+n7OSCJLs0gr8yl DHtw== X-Gm-Message-State: AOJu0Yzz+IVYwvG34mi6EwSBLc41o49vv+9PoPN8/HXWd9l/3FsOUTnd RKmUS2mqlHrly418xR+EJULM2uKgCMJd2Cja3EUXuhrL/AnVMilb6vN+GBwyDhXt7LVeZyfCJOL agabrhETqgYttKQb3fLszX729EGtFvQ4j1i/S0VvWFtIr8lW94W+R2R1CxMMxkHktEsCkz/2XUJ qLqSih9MKOYC/T64o1QADauJBTFrK1nA== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1887:b0:dc2:2ace:860 with SMTP id cj7-20020a056902188700b00dc22ace0860mr1032065ybb.2.1708515365485; Wed, 21 Feb 2024 03:36:05 -0800 (PST) Date: Wed, 21 Feb 2024 12:35:20 +0100 In-Reply-To: <20240221113506.2565718-18-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240221113506.2565718-18-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=7014; i=ardb@kernel.org; h=from:subject; bh=oEjbaEakolZKtY3/OEd7gfiAzx2qVYvdQU90/PQTOX8=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXq/S+hVVb/nl/wOGHDrbkl6L3ogmRx+xl7VtflzZl9w MeTPedVRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjI08+MDM2+pjLOKhaPn23+ vvn/+rBYbvHl1ddYdgu4yZ2MOD1fZC7Db1bvuq+H0+5u/XZwi+wn3u/3lEuvn9p5VYfP68zHM77 ZPcwA X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221113506.2565718-31-ardb+git@google.com> Subject: [PATCH v5 13/16] x86/boot: Move mem_encrypt= parsing to the decompressor From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791508445279060449 X-GMAIL-MSGID: 1791508445279060449 From: Ard Biesheuvel The early SME/SEV code parses the command line very early, in order to decide whether or not memory encryption should be enabled, which needs to occur even before the initial page tables are created. This is problematic for a number of reasons: - this early code runs from the 1:1 mapping provided by the decompressor or firmware, which uses a different translation than the one assumed by the linker, and so the code needs to be built in a special way; - parsing external input while the entire kernel image is still mapped writable is a bad idea in general, and really does not belong in security minded code; - the current code ignores the built-in command line entirely (although this appears to be the case for the entire decompressor) Given that the decompressor/EFI stub is an intrinsic part of the x86 bootable kernel image, move the command line parsing there and out of the core kernel. This removes the need to build lib/cmdline.o in a special way, or to use RIP-relative LEA instructions in inline asm blocks. This involves a new xloadflag in the setup header to indicate that mem_encrypt=on appeared on the kernel command line. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/misc.c | 15 +++++++++ arch/x86/include/uapi/asm/bootparam.h | 1 + arch/x86/lib/Makefile | 13 -------- arch/x86/mm/mem_encrypt_identity.c | 32 ++------------------ drivers/firmware/efi/libstub/x86-stub.c | 3 ++ 5 files changed, 22 insertions(+), 42 deletions(-) diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c index b99e08e6815b..6c5c190a4d86 100644 --- a/arch/x86/boot/compressed/misc.c +++ b/arch/x86/boot/compressed/misc.c @@ -357,6 +357,19 @@ unsigned long decompress_kernel(unsigned char *outbuf, unsigned long virt_addr, return entry; } +/* + * Set the memory encryption xloadflag based on the mem_encrypt= command line + * parameter, if provided. + */ +static void parse_mem_encrypt(struct setup_header *hdr) +{ + int on = cmdline_find_option_bool("mem_encrypt=on"); + int off = cmdline_find_option_bool("mem_encrypt=off"); + + if (on > off) + hdr->xloadflags |= XLF_MEM_ENCRYPTION; +} + /* * The compressed kernel image (ZO), has been moved so that its position * is against the end of the buffer used to hold the uncompressed kernel @@ -387,6 +400,8 @@ asmlinkage __visible void *extract_kernel(void *rmode, unsigned char *output) /* Clear flags intended for solely in-kernel use. */ boot_params_ptr->hdr.loadflags &= ~KASLR_FLAG; + parse_mem_encrypt(&boot_params_ptr->hdr); + sanitize_boot_params(boot_params_ptr); if (boot_params_ptr->screen_info.orig_video_mode == 7) { diff --git a/arch/x86/include/uapi/asm/bootparam.h b/arch/x86/include/uapi/asm/bootparam.h index 01d19fc22346..eeea058cf602 100644 --- a/arch/x86/include/uapi/asm/bootparam.h +++ b/arch/x86/include/uapi/asm/bootparam.h @@ -38,6 +38,7 @@ #define XLF_EFI_KEXEC (1<<4) #define XLF_5LEVEL (1<<5) #define XLF_5LEVEL_ENABLED (1<<6) +#define XLF_MEM_ENCRYPTION (1<<7) #ifndef __ASSEMBLY__ diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile index ea3a28e7b613..f0dae4fb6d07 100644 --- a/arch/x86/lib/Makefile +++ b/arch/x86/lib/Makefile @@ -14,19 +14,6 @@ ifdef CONFIG_KCSAN CFLAGS_REMOVE_delay.o = $(CC_FLAGS_FTRACE) endif -# Early boot use of cmdline; don't instrument it -ifdef CONFIG_AMD_MEM_ENCRYPT -KCOV_INSTRUMENT_cmdline.o := n -KASAN_SANITIZE_cmdline.o := n -KCSAN_SANITIZE_cmdline.o := n - -ifdef CONFIG_FUNCTION_TRACER -CFLAGS_REMOVE_cmdline.o = -pg -endif - -CFLAGS_cmdline.o := -fno-stack-protector -fno-jump-tables -endif - inat_tables_script = $(srctree)/arch/x86/tools/gen-insn-attr-x86.awk inat_tables_maps = $(srctree)/arch/x86/lib/x86-opcode-map.txt quiet_cmd_inat_tables = GEN $@ diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index 7ddcf960e92a..0180fbbcc940 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -43,7 +43,6 @@ #include #include -#include #include #include #include @@ -96,9 +95,6 @@ struct sme_populate_pgd_data { */ static char sme_workarea[2 * PMD_SIZE] __section(".init.scratch"); -static char sme_cmdline_arg[] __initdata = "mem_encrypt"; -static char sme_cmdline_on[] __initdata = "on"; - static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) { unsigned long pgd_start, pgd_end, pgd_size; @@ -505,11 +501,9 @@ void __init sme_encrypt_kernel(struct boot_params *bp) void __head sme_enable(struct boot_params *bp) { - const char *cmdline_ptr, *cmdline_arg, *cmdline_on; unsigned int eax, ebx, ecx, edx; unsigned long feature_mask; unsigned long me_mask; - char buffer[16]; u64 msr; /* Check for the SME/SEV support leaf */ @@ -549,6 +543,9 @@ void __head sme_enable(struct boot_params *bp) /* Check if memory encryption is enabled */ if (feature_mask == AMD_SME_BIT) { + if (!(bp->hdr.xloadflags & XLF_MEM_ENCRYPTION)) + return; + /* * No SME if Hypervisor bit is set. This check is here to * prevent a guest from trying to enable SME. For running as a @@ -568,31 +565,8 @@ void __head sme_enable(struct boot_params *bp) msr = __rdmsr(MSR_AMD64_SYSCFG); if (!(msr & MSR_AMD64_SYSCFG_MEM_ENCRYPT)) return; - } else { - /* SEV state cannot be controlled by a command line option */ - goto out; } - /* - * Fixups have not been applied to phys_base yet and we're running - * identity mapped, so we must obtain the address to the SME command - * line argument data using rip-relative addressing. - */ - asm ("lea sme_cmdline_arg(%%rip), %0" - : "=r" (cmdline_arg) - : "p" (sme_cmdline_arg)); - asm ("lea sme_cmdline_on(%%rip), %0" - : "=r" (cmdline_on) - : "p" (sme_cmdline_on)); - - cmdline_ptr = (const char *)((u64)bp->hdr.cmd_line_ptr | - ((u64)bp->ext_cmd_line_ptr << 32)); - - if (cmdline_find_option(cmdline_ptr, cmdline_arg, buffer, sizeof(buffer)) < 0 || - strncmp(buffer, cmdline_on, sizeof(buffer))) - return; - -out: RIP_REL_REF(sme_me_mask) = me_mask; physical_mask &= ~me_mask; cc_vendor = CC_VENDOR_AMD; diff --git a/drivers/firmware/efi/libstub/x86-stub.c b/drivers/firmware/efi/libstub/x86-stub.c index 0d510c9a06a4..9a25ec16b344 100644 --- a/drivers/firmware/efi/libstub/x86-stub.c +++ b/drivers/firmware/efi/libstub/x86-stub.c @@ -879,6 +879,9 @@ void __noreturn efi_stub_entry(efi_handle_t handle, } } + if (efi_mem_encrypt > 0) + hdr->xloadflags |= XLF_MEM_ENCRYPTION; + status = efi_decompress_kernel(&kernel_entry); if (status != EFI_SUCCESS) { efi_err("Failed to decompress kernel\n"); From patchwork Wed Feb 21 11:35:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 204179 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp979589dyc; Wed, 21 Feb 2024 03:40:31 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXUQixIDKuHniLaI4fyNOyBFa444D7Z+TNjSczMCXKX1snaHiz5f/klZgNSWwfOYOzSjLTx6wIL/4w0laq8o7FqUo/FjQ== X-Google-Smtp-Source: AGHT+IETsRQYnJoq/ZoHCD+k0WgyGEue8evuudn5VdPZp7sLx7Z7JPEvB1PXn9fnVcAyI1oXu24a X-Received: by 2002:a05:6870:5b9b:b0:21e:623d:1824 with SMTP id em27-20020a0568705b9b00b0021e623d1824mr16818238oab.8.1708515631029; Wed, 21 Feb 2024 03:40:31 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708515630; cv=pass; d=google.com; s=arc-20160816; b=QcVps9kA5e/k0VijyEzfArfUxcvTPkoMDdDmSgvvtawN7vHgcLAeU6Xyn7frjjbHQw Upm+O6hHpGycp9BrukTAnsqYm8/aipUNK2VI9e732+xabCVDjUCehz5yHjr8ijCGRy7L Ggsqfn4+vAbto6D57JrS2ZK+qvfryW/rfgLb6DbKCIbTkf7yAM9JCGMQDah7fosjID1i r5dD6LzevVryXcw71BeW6t8SuCrYmRRuBPM2o4mc3G0SEV9m0pNBHXGaupiMhC+k8H7R p4qh4AIjj++fnp1K82wI6NjHstsezyNeYgzVJ8dZHOCCfNMrNh67/ZhYHoq03sFTBgPy jaXA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=aHBDJ41pn3iOna4rTXtxPzeJ6vfMVADf3iDfGB7g+R4=; fh=fMhMPvo8duafGkM4xZHdH7fULbau3etefTGsKa+Qy7I=; b=nPkge41Xw4dMy7aRnk8ByuIRIeXXdqdsfm/98J9+Vf5DF0J6XDPpvmYLI/aQY5o1L5 hSwGg2UgkAGq4l+tRtSXRbKQQzqYNE8GJnCvUwZ+DivyptKn7vxcqEmI4LlbdSaXQb1j 0oJHsZgMy/EnJdj10pA/IyJv5UU4jyzRaAKZatyPl3JKqTmEgHxV4tXunBcFm9JSaCRH 8rAKCM0uSYkwpz2BwPZ5Pbls88haKiNQNYjlF3afwNaRWiy5MC+VM9c49nW9R4cYxQc0 vK0zDpQwnD2MOeh/hABTPyeqN2/7hC+EZIWfGApwgMhbzV9+VzpIjInHPTSzzz7cOWAB BTbA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=PpXvLEuM; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74644-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74644-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id l62-20020a639141000000b005dc87994572si8059006pge.346.2024.02.21.03.40.30 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 03:40:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74644-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=PpXvLEuM; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74644-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74644-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 93E76283786 for ; Wed, 21 Feb 2024 11:40:02 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1E42B4654F; Wed, 21 Feb 2024 11:36:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="PpXvLEuM" Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AF3234F61C for ; Wed, 21 Feb 2024 11:36:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515371; cv=none; b=o1osFfUSpxT10KYKLPqCJCwIRV0KZ7217q4K2S0rPaOoDOpTI4XunxiF4MMFaYVe/h/VBBBs/hqxLMbaUhQ/rChONHKzJxhNJUp9fMkaF6zNhKnsLT4tPOuQtML73liNBJ5rcApUmlpIXF088DcR04cjjM4Z742UP81JQxBnrX8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515371; c=relaxed/simple; bh=WZPNjU+tKWbG1/7EwXfp1HjOJvLOdEPU1KuRuZ22CGg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=TnQsPmvoCXzPoQqe1O6175RmaSjk/ZpC4W2VLPXFDW+JvlQ7BGqzE7CYrbQc2ISppedc4wa7jl8tJL50FclKJKPUGvAHz68YO0BesyjRuAwIAfIpWWLkDsSEA9ApDwXBJ2P4/2qQL6ITwxd6unPIOHyZm31++VnnoxI4dHj4O2I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=PpXvLEuM; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-607cd6c11d7so95880437b3.0 for ; Wed, 21 Feb 2024 03:36:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708515368; x=1709120168; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=aHBDJ41pn3iOna4rTXtxPzeJ6vfMVADf3iDfGB7g+R4=; b=PpXvLEuMPM38gThvJ/6wj9PxYDyk7+h3DzW+dUxCFhEmIaDfO/O7HkjBPZdwnWjPmG FCvL8/67tc5gF4VF/1jBsEDOqW5MAivIASZYunc8kWdluLzidjBGRtuJj2i//g0SXnFF StbRwWUmu9qyezvb9Ybl0Iy+NFpLa0rmmVaPdUNbMkNVqlu5LmSIR45Z5mUNkC0AoE8W nYXmm9utes/CY10liKfc7KrAV7b5qH7R6EdgbZEaOQIZUxvvsvtKvDVDKr4OrcZm2PSn myAwnFknOSPZkMkDLjYV+w67IHEfK7mHtP/RET6oQw0S4cOimqrdI2+AEDckh5kXzuVM 5nag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708515368; x=1709120168; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aHBDJ41pn3iOna4rTXtxPzeJ6vfMVADf3iDfGB7g+R4=; b=T5pkqJQWXBuATABxtDOnlPa9VEI4i3ROADJXMdbCRVGnqI/h+ThNDYqCDs+Du4NXD8 8iTMwH/ITnZh4lwF3DnnxiPO+817VUh4HUmEsS2wpz1zH3T+3c6Hh0WD8OIKL6b79ilN ZYQv6F5sI8j2+zXOlRFwoSgb2bDM47FO+vN4NexoD7dDnx7wSOgkqm3MpdH1AH2Esyg/ 1k67XVPK1faSo8FhmS2gguKYrgtCz38WJcZW54eIIFld1Vpb0/ovHqhMvFZ7AlPv12Nh /mm9PsIqwXxq+7qao3Y7y+98PsmW68Eau4QrSxJd1VdSPmzbWd/M2wyVYC8p12JQ10Kt /9jw== X-Gm-Message-State: AOJu0Yyh1aILTWvJSRWOsQS+0HTYC3vEvFPwKldzzfWCEnetl1zvp9w/ rvosp6eWkUJ6u9yLrkKl6ZkT8vUd/ZWo1jjACYvh74frliLSVQErq0GQqdeIHJxBvdgF4pYTrTB E4J7dAkwyMxnt1guWuuM+pLS1UyLHGemCWstB8/9HePUJEDxncEpOIoY16ixVnTfKWMSAL+vXrk pwla9w6FNBKF8cwacCn2SJYrTyaPFPgg== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:b2a7:0:b0:dcc:4785:b51e with SMTP id k39-20020a25b2a7000000b00dcc4785b51emr727479ybj.12.1708515368119; Wed, 21 Feb 2024 03:36:08 -0800 (PST) Date: Wed, 21 Feb 2024 12:35:21 +0100 In-Reply-To: <20240221113506.2565718-18-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240221113506.2565718-18-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=7470; i=ardb@kernel.org; h=from:subject; bh=gAHUpSDGEhZGPmJ7iJbqEh4bp+wwMJqFxMSwv/jUH1s=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXq/a+RU19XXOhsNn20b8ZGvf/ZyurTjzOFMn3Uupqxq LFbrSmwo5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAExkoTrD//SojHvRAk8W7pCT vV3PW/42OqTXeOu2fQGHcpgPFKdqtTMynO2vZfljffNVm9olH/9Fv1+o1F5bW3/HVnF1LL/2eQ9 OFgA= X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221113506.2565718-32-ardb+git@google.com> Subject: [PATCH v5 14/16] x86/sme: Move early SME kernel encryption handling into .head.text From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791508486072442251 X-GMAIL-MSGID: 1791508486072442251 From: Ard Biesheuvel The .head.text section is the initial primary entrypoint of the core kernel, and is entered with the CPU executing from a 1:1 mapping of memory. Such code must never access global variables using absolute references, as these are based on the kernel virtual mapping which is not active yet at this point. Given that the SME startup code is also called from this early execution context, move it into .head.text as well. This will allow more thorough build time checks in the future to ensure that early startup code only uses RIP-relative references to global variables. Also replace some occurrences of __pa_symbol() [which relies on the compiler generating an absolute reference, which is not guaranteed] and an open coded RIP-relative access with RIP_REL_REF(). Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/mem_encrypt.h | 4 +- arch/x86/mm/mem_encrypt_identity.c | 40 ++++++++------------ 2 files changed, 18 insertions(+), 26 deletions(-) diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index b1437ba0b3b8..f922b682b9b4 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -47,7 +47,7 @@ void __init sme_unmap_bootdata(char *real_mode_data); void __init sme_early_init(void); -void __init sme_encrypt_kernel(struct boot_params *bp); +void sme_encrypt_kernel(struct boot_params *bp); void sme_enable(struct boot_params *bp); int __init early_set_memory_decrypted(unsigned long vaddr, unsigned long size); @@ -81,7 +81,7 @@ static inline void __init sme_unmap_bootdata(char *real_mode_data) { } static inline void __init sme_early_init(void) { } -static inline void __init sme_encrypt_kernel(struct boot_params *bp) { } +static inline void sme_encrypt_kernel(struct boot_params *bp) { } static inline void sme_enable(struct boot_params *bp) { } static inline void sev_es_init_vc_handling(void) { } diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index 0180fbbcc940..174a7192c9cb 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -41,6 +41,7 @@ #include #include +#include #include #include #include @@ -95,7 +96,7 @@ struct sme_populate_pgd_data { */ static char sme_workarea[2 * PMD_SIZE] __section(".init.scratch"); -static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) +static void __head sme_clear_pgd(struct sme_populate_pgd_data *ppd) { unsigned long pgd_start, pgd_end, pgd_size; pgd_t *pgd_p; @@ -110,7 +111,7 @@ static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) memset(pgd_p, 0, pgd_size); } -static pud_t __init *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) +static pud_t __head *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) { pgd_t *pgd; p4d_t *p4d; @@ -147,7 +148,7 @@ static pud_t __init *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) return pud; } -static void __init sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) +static void __head sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) { pud_t *pud; pmd_t *pmd; @@ -163,7 +164,7 @@ static void __init sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) set_pmd(pmd, __pmd(ppd->paddr | ppd->pmd_flags)); } -static void __init sme_populate_pgd(struct sme_populate_pgd_data *ppd) +static void __head sme_populate_pgd(struct sme_populate_pgd_data *ppd) { pud_t *pud; pmd_t *pmd; @@ -189,7 +190,7 @@ static void __init sme_populate_pgd(struct sme_populate_pgd_data *ppd) set_pte(pte, __pte(ppd->paddr | ppd->pte_flags)); } -static void __init __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) +static void __head __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) { while (ppd->vaddr < ppd->vaddr_end) { sme_populate_pgd_large(ppd); @@ -199,7 +200,7 @@ static void __init __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) } } -static void __init __sme_map_range_pte(struct sme_populate_pgd_data *ppd) +static void __head __sme_map_range_pte(struct sme_populate_pgd_data *ppd) { while (ppd->vaddr < ppd->vaddr_end) { sme_populate_pgd(ppd); @@ -209,7 +210,7 @@ static void __init __sme_map_range_pte(struct sme_populate_pgd_data *ppd) } } -static void __init __sme_map_range(struct sme_populate_pgd_data *ppd, +static void __head __sme_map_range(struct sme_populate_pgd_data *ppd, pmdval_t pmd_flags, pteval_t pte_flags) { unsigned long vaddr_end; @@ -233,22 +234,22 @@ static void __init __sme_map_range(struct sme_populate_pgd_data *ppd, __sme_map_range_pte(ppd); } -static void __init sme_map_range_encrypted(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_encrypted(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_ENC, PTE_FLAGS_ENC); } -static void __init sme_map_range_decrypted(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_decrypted(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_DEC, PTE_FLAGS_DEC); } -static void __init sme_map_range_decrypted_wp(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_decrypted_wp(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_DEC_WP, PTE_FLAGS_DEC_WP); } -static unsigned long __init sme_pgtable_calc(unsigned long len) +static unsigned long __head sme_pgtable_calc(unsigned long len) { unsigned long entries = 0, tables = 0; @@ -285,7 +286,7 @@ static unsigned long __init sme_pgtable_calc(unsigned long len) return entries + tables; } -void __init sme_encrypt_kernel(struct boot_params *bp) +void __head sme_encrypt_kernel(struct boot_params *bp) { unsigned long workarea_start, workarea_end, workarea_len; unsigned long execute_start, execute_end, execute_len; @@ -320,9 +321,8 @@ void __init sme_encrypt_kernel(struct boot_params *bp) * memory from being cached. */ - /* Physical addresses gives us the identity mapped virtual addresses */ - kernel_start = __pa_symbol(_text); - kernel_end = ALIGN(__pa_symbol(_end), PMD_SIZE); + kernel_start = (unsigned long)RIP_REL_REF(_text); + kernel_end = ALIGN((unsigned long)RIP_REL_REF(_end), PMD_SIZE); kernel_len = kernel_end - kernel_start; initrd_start = 0; @@ -339,14 +339,6 @@ void __init sme_encrypt_kernel(struct boot_params *bp) } #endif - /* - * We're running identity mapped, so we must obtain the address to the - * SME encryption workarea using rip-relative addressing. - */ - asm ("lea sme_workarea(%%rip), %0" - : "=r" (workarea_start) - : "p" (sme_workarea)); - /* * Calculate required number of workarea bytes needed: * executable encryption area size: @@ -356,7 +348,7 @@ void __init sme_encrypt_kernel(struct boot_params *bp) * pagetable structures for the encryption of the kernel * pagetable structures for workarea (in case not currently mapped) */ - execute_start = workarea_start; + execute_start = workarea_start = (unsigned long)RIP_REL_REF(sme_workarea); execute_end = execute_start + (PAGE_SIZE * 2) + PMD_SIZE; execute_len = execute_end - execute_start; From patchwork Wed Feb 21 11:35:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 204180 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp979668dyc; Wed, 21 Feb 2024 03:40:45 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXuYGb11+nM9n3h+7XuMFJXh+58La8bP/xrjk7xO5IORViITpzXOvVD66i9+dQEwyTXBktz8FZ20mNXDD0M3Gndzd7sBw== X-Google-Smtp-Source: AGHT+IEKV2Kce2K35nhcIlsUH3I828qjTbJV7T9ebM5zSIw+QDTdcjEkTaoRPsxl+/8WkhakUxuO X-Received: by 2002:aa7:9819:0:b0:6e4:67bd:10fa with SMTP id e25-20020aa79819000000b006e467bd10famr6992055pfl.0.1708515645022; Wed, 21 Feb 2024 03:40:45 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708515645; cv=pass; d=google.com; s=arc-20160816; b=dUg1V9r/ZMLiyJMc0T4MTAOU5ETrdV8h1uuEPdEh+r4r7OsKbBrODyGgz5haGp8Hgh LeTchLaCHH+BZVmUgAupEcmeVmq2Lb58x+0Vfm7Hu2bvY0N7MzBkku/AvM7YdhHc9SYJ 0cwZ6h25HOKUMK0heDkJxu+CIpkLspbsiU75H3OD9rSizR7j9uLNHZPQ2fTlc+HA5JmP hezhnvdbgPg+xbP907h5xFk4YPT9BrKvbReI3G9m2HwbeiRfQczeM5KBKR6Fb1n0uW08 hxkKQIbfYe7KeqbH2+est3rzjaI7gSVVTObvnr7rs9fF0Fw5j5JVaY++qTtQ+N3ZLSnO lGNw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=o1RTMbx8aPOdGUmQigoxFTqq7Alxv8e0fhW/7jCZ7hc=; fh=fMhMPvo8duafGkM4xZHdH7fULbau3etefTGsKa+Qy7I=; b=bDLlPdnnkiHtVtVfixcmSLd6xJIXaEEv95zL+UeBy6SfIJjs2oAoS1LYiCgSRZQo54 A64xZdskEhozIFJ6tA3m+2ed1LkY0IRwrKjPB5pNtLgZzn03fbmfHM98IG9AB2gPJ3BS dxkuTxw+vEwRZSHurGGmqtxlLaFxOtXU+4e+1GKHpuZ8q0aPghcACx36Vo0Q1Aqlvlr9 5VY3jhjolmWFfpQw15JfIJIvTXEl44KXZhoxDMFFLNFJynwGWlbYwD5WaMfYveW7v5HO K+m9dbftdDn4qaMga87sIk6nmy4By+qriUEMdxhIz4tdui4kpYI3qqZMpm5gxfQBa2+5 eIkw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b="fLT/YX0e"; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74645-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74645-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id x28-20020aa79a5c000000b006e284fdf218si7675650pfj.178.2024.02.21.03.40.44 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 03:40:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74645-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b="fLT/YX0e"; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74645-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74645-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 08AB2285276 for ; Wed, 21 Feb 2024 11:40:24 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C663D47A55; Wed, 21 Feb 2024 11:36:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="fLT/YX0e" Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1D0C646424 for ; Wed, 21 Feb 2024 11:36:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515376; cv=none; b=qSS0szPhjmfux98QLZQx0xXT9IAuWZpHN8gVf2cp2DwAevEr3BEn8vNN5f9Kl7u1oIxdAUYURDIqg3UOgdSrEeSnOUAA+DnY5Esmqcr23B1vAqr0/zdcxPjl+QJmI2J+q24fIPYMxSAsViFmILResaT3jVBViZuMbfizjCx+bYw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515376; c=relaxed/simple; bh=dRE0REehaE79mQAOudLPe6thvzCULsyHIPQfD/woex8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=IvyskZXL2deWUelzrshGul0HPozM/FdtLN+IqnifClPUMKwztfyI4X3tqncx+rh1cXDlvWtOFY2LTVv2+z++Cs2y8w5i9usybW6ro2yd+HQ7DUjnDjuSAWYEOiAy1lvVkhKnJI+CE4zavenWy18TXWrvqOf8CQ6lvWPzWvNn+2M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=fLT/YX0e; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-33d51bb9353so1411308f8f.2 for ; Wed, 21 Feb 2024 03:36:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708515372; x=1709120172; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=o1RTMbx8aPOdGUmQigoxFTqq7Alxv8e0fhW/7jCZ7hc=; b=fLT/YX0e4chPUubpXwC0TiRwEMV4uyxTkoSIYf4GlhfXFHW1v364iiVM0S3467ldkG 9IZDxigEnAosVskv2/hwsbI3id0E5mOox7Az1x/rwjy8r8vA1zozD2u6+h+s3x1rNehh n4nG054R7I8guua93NlbTPrzi9VIlaOlMRR3C58GNMELWwi7dr42HjXc62eG7VnXMFbd Fq/2sTqshJGe/1gCzzp2TqkV34blrrz3Lc1LAP1ja7G9X72wyQIcH9rIJ+zDyBL252On 2wp60/f7WD7GXjZ8Os0lFoa8cHxYqcKE6RRJPTwpbfjIgWcIr7nuCPghe/0VBp9bIh5d XLSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708515372; x=1709120172; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=o1RTMbx8aPOdGUmQigoxFTqq7Alxv8e0fhW/7jCZ7hc=; b=wL8uf8/nHGllSJeoyMh9wAHvZG3vH0v1gz71DRgWm5Ym0PPZEkmy05YpE6DzpEwiHE YDoUqr0G6bW12T4F7DTJ0vO7i+49Hzio+F3PWdZ4htFMfOOhHYjHa8iR7sM87+Ia7GMO GgQUOicM0TC3HSs7AGUlk1KNLctqZTD/obxmpKKPgRlacGZ9CrlhE+OSCzM5Q99M6cg0 34X63bSZT49gvs+WDe4QWe+uHLVxMdmneselMb6va4u4fQQHaih3yy4MxNsA4GSZxeMF xjhubGNnAGEarLVDZml7OscxrqtRz6vGhtnkvoazkHCliL/95AIzuZxuoM77iBUMLO4y gBtw== X-Gm-Message-State: AOJu0Yy+2FzS+IU5D9pYT/YFULjh4UjVNhvNpO+Fv+q3TgOkHUQdiRoo mrqIzwUzgABoV3nWtjZdjVt0b7SzIjl/DmeUKaJzn5EqXhyyIiqofYJUbCAFxiL/XeHeIqgOI0D N/xqbskUL7tRuf0nKKbKcHiNjtrHmuZRYNr0ROsCuRqFNsq/hjtSctNB8O/DMDCaxp6lqdFqRav AkEcm2as0AX/d3PgdbziEek/DbcxOL8Q== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:1e8f:b0:33d:269e:1327 with SMTP id dd15-20020a0560001e8f00b0033d269e1327mr25078wrb.3.1708515370396; Wed, 21 Feb 2024 03:36:10 -0800 (PST) Date: Wed, 21 Feb 2024 12:35:22 +0100 In-Reply-To: <20240221113506.2565718-18-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240221113506.2565718-18-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=7382; i=ardb@kernel.org; h=from:subject; bh=LAOkk6glECx/a1Ernwj65OY7RC++QF53CdqPGnTSEFg=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXq/a9zp7AHbTt95WHF51ipif/fyz7k4uw6+lfOhzWta 4nT/X+pHaUsDGIcDLJiiiwCs/++23l6olSt8yxZmDmsTCBDGLg4BWAiCvMYGa5L3HbU9bzJxPI9 2lxkOpMmX7/gqZcsp/RPRq3SEok8HM/w35GDq+sKW+6KvmlOpiGSCbe+5ygfuSIVw/S3ob1itv0 6BgA= X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221113506.2565718-33-ardb+git@google.com> Subject: [PATCH v5 15/16] x86/sev: Move early startup code into .head.text section From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791508501225092806 X-GMAIL-MSGID: 1791508501225092806 From: Ard Biesheuvel In preparation for implementing rigorous build time checks to enforce that only code that can support it will be called from the early 1:1 mapping of memory, move SEV init code that is called in this manner to the .head.text section. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/sev.c | 3 +++ arch/x86/include/asm/sev.h | 10 ++++----- arch/x86/kernel/sev-shared.c | 23 +++++++++----------- arch/x86/kernel/sev.c | 14 +++++++----- 4 files changed, 26 insertions(+), 24 deletions(-) diff --git a/arch/x86/boot/compressed/sev.c b/arch/x86/boot/compressed/sev.c index 073291832f44..bea0719d70f2 100644 --- a/arch/x86/boot/compressed/sev.c +++ b/arch/x86/boot/compressed/sev.c @@ -116,6 +116,9 @@ static bool fault_in_kernel_space(unsigned long address) #undef __init #define __init +#undef __head +#define __head + #define __BOOT_COMPRESSED /* Basic instruction decoding support needed */ diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h index bed95e1f4d52..cf671138feef 100644 --- a/arch/x86/include/asm/sev.h +++ b/arch/x86/include/asm/sev.h @@ -213,16 +213,16 @@ static inline int pvalidate(unsigned long vaddr, bool rmp_psize, bool validate) struct snp_guest_request_ioctl; void setup_ghcb(void); -void __init early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, - unsigned long npages); -void __init early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr, - unsigned long npages); +void early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, + unsigned long npages); +void early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr, + unsigned long npages); void __init snp_prep_memory(unsigned long paddr, unsigned int sz, enum psc_op op); void snp_set_memory_shared(unsigned long vaddr, unsigned long npages); void snp_set_memory_private(unsigned long vaddr, unsigned long npages); void snp_set_wakeup_secondary_cpu(void); bool snp_init(struct boot_params *bp); -void __init __noreturn snp_abort(void); +void __noreturn snp_abort(void); int snp_issue_guest_request(u64 exit_code, struct snp_req_data *input, struct snp_guest_request_ioctl *rio); void snp_accept_memory(phys_addr_t start, phys_addr_t end); u64 snp_get_unsupported_features(u64 status); diff --git a/arch/x86/kernel/sev-shared.c b/arch/x86/kernel/sev-shared.c index ae79f9505298..0bd7ccbe8732 100644 --- a/arch/x86/kernel/sev-shared.c +++ b/arch/x86/kernel/sev-shared.c @@ -93,7 +93,8 @@ static bool __init sev_es_check_cpu_features(void) return true; } -static void __noreturn sev_es_terminate(unsigned int set, unsigned int reason) +static void __head __noreturn +sev_es_terminate(unsigned int set, unsigned int reason) { u64 val = GHCB_MSR_TERM_REQ; @@ -330,13 +331,7 @@ static int sev_cpuid_hv(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid */ static const struct snp_cpuid_table *snp_cpuid_get_table(void) { - void *ptr; - - asm ("lea cpuid_table_copy(%%rip), %0" - : "=r" (ptr) - : "p" (&cpuid_table_copy)); - - return ptr; + return &RIP_REL_REF(cpuid_table_copy); } /* @@ -395,7 +390,7 @@ static u32 snp_cpuid_calc_xsave_size(u64 xfeatures_en, bool compacted) return xsave_size; } -static bool +static bool __head snp_cpuid_get_validated_func(struct cpuid_leaf *leaf) { const struct snp_cpuid_table *cpuid_table = snp_cpuid_get_table(); @@ -532,7 +527,8 @@ static int snp_cpuid_postprocess(struct ghcb *ghcb, struct es_em_ctxt *ctxt, * Returns -EOPNOTSUPP if feature not enabled. Any other non-zero return value * should be treated as fatal by caller. */ -static int snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf) +static int __head +snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf) { const struct snp_cpuid_table *cpuid_table = snp_cpuid_get_table(); @@ -574,7 +570,7 @@ static int snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_le * page yet, so it only supports the MSR based communication with the * hypervisor and only the CPUID exit-code. */ -void __init do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code) +void __head do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code) { unsigned int subfn = lower_bits(regs->cx, 32); unsigned int fn = lower_bits(regs->ax, 32); @@ -1025,7 +1021,8 @@ struct cc_setup_data { * Search for a Confidential Computing blob passed in as a setup_data entry * via the Linux Boot Protocol. */ -static struct cc_blob_sev_info *find_cc_blob_setup_data(struct boot_params *bp) +static __head +struct cc_blob_sev_info *find_cc_blob_setup_data(struct boot_params *bp) { struct cc_setup_data *sd = NULL; struct setup_data *hdr; @@ -1052,7 +1049,7 @@ static struct cc_blob_sev_info *find_cc_blob_setup_data(struct boot_params *bp) * mapping needs to be updated in sync with all the changes to virtual memory * layout and related mapping facilities throughout the boot process. */ -static void __init setup_cpuid_table(const struct cc_blob_sev_info *cc_info) +static void __head setup_cpuid_table(const struct cc_blob_sev_info *cc_info) { const struct snp_cpuid_table *cpuid_table_fw, *cpuid_table; int i; diff --git a/arch/x86/kernel/sev.c b/arch/x86/kernel/sev.c index 1ef7ae806a01..33c14aa1f06c 100644 --- a/arch/x86/kernel/sev.c +++ b/arch/x86/kernel/sev.c @@ -25,6 +25,7 @@ #include #include +#include #include #include #include @@ -682,8 +683,9 @@ static u64 __init get_jump_table_addr(void) return ret; } -static void early_set_pages_state(unsigned long vaddr, unsigned long paddr, - unsigned long npages, enum psc_op op) +static void __head +early_set_pages_state(unsigned long vaddr, unsigned long paddr, + unsigned long npages, enum psc_op op) { unsigned long paddr_end; u64 val; @@ -739,7 +741,7 @@ static void early_set_pages_state(unsigned long vaddr, unsigned long paddr, sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_PSC); } -void __init early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, +void __head early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, unsigned long npages) { /* @@ -2062,7 +2064,7 @@ bool __init handle_vc_boot_ghcb(struct pt_regs *regs) * * Scan for the blob in that order. */ -static __init struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp) +static __head struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp) { struct cc_blob_sev_info *cc_info; @@ -2088,7 +2090,7 @@ static __init struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp) return cc_info; } -bool __init snp_init(struct boot_params *bp) +bool __head snp_init(struct boot_params *bp) { struct cc_blob_sev_info *cc_info; @@ -2110,7 +2112,7 @@ bool __init snp_init(struct boot_params *bp) return true; } -void __init __noreturn snp_abort(void) +void __head __noreturn snp_abort(void) { sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SNP_UNSUPPORTED); } From patchwork Wed Feb 21 11:35:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 204181 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:693c:2685:b0:108:e6aa:91d0 with SMTP id mn5csp979699dyc; Wed, 21 Feb 2024 03:40:51 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCW40ZeSS8LSGiNaMTS+liyAvlgUbYSi6jaPAUriAIfgC3mMAkXJdmwvUiuaNbRfrfu1eEsrTJJ8ARkp+3kX5RoTBLs7vg== X-Google-Smtp-Source: AGHT+IGXnYaZRbITgLjevIZr0QEKbVAIPsYnbntTastYHNd0aN4ueOiVaXu+54pANm9J6yj/NOVX X-Received: by 2002:a17:90a:d504:b0:299:63bd:c17d with SMTP id t4-20020a17090ad50400b0029963bdc17dmr14678434pju.2.1708515650884; Wed, 21 Feb 2024 03:40:50 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708515650; cv=pass; d=google.com; s=arc-20160816; b=FHC2ORHJfID858np10Dd+kb8FEnp2YHOPRMHE87OAEDCtERR12OubbgdtgDzYedFeo nslfLwFiL+mNN60QFtDqTnvikgtcAXZKPGzCpNrK2VoZQVRKyuE52N9+fU97SQJa5+3C 2I5XpJd8zhuT3Q8x3Pl63c/yLESIXmqmIeuPXYtDgo4fyOVD3nZLr5dAGGF+bvUJG21n LFV+gIe3McSi0ABM9jm+XIo/1Uy3rRnS/bVKkEkiUzSau/c95ghlLFVuQlcLTtSJ0S6C sSUAfvZ1YPU5wWyF4vuwr73xnXalKjOO7VWGlRM+NvsvKlfQZ9BWnItdqlKXUlAee+wc nAgg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=k0sTN6yoDasV9agD6pu51ddL9MvEfa3HQq1IU+fzHuI=; fh=fMhMPvo8duafGkM4xZHdH7fULbau3etefTGsKa+Qy7I=; b=s53mV3UbnoPLyhSvM9PkDOcCbpiKWyZ04GWM8IZmCxh8Vl4bG99UKNMhEYwht5Bw0z FVwwdTWhwVpfYwx88/B5E4QPzPB62aXgVxR+xpHnhD4EaXjDFglpo6bGWTFfJkfNs3Ew AToLCspLDHBoUnCtdIAU178mzX6XmV0P65FPoABlQAePgX78ciW84KSEmN7NAqCqvt20 UL26czSjN/+L+mCDqr5C5I6g+jMiWf6eWlD0eMYbHmNOzqnC7Ldgk1NSeAQVFUJqZdIb Z6OZJFNV0Kpmk+9vw+TwpWwA2DgJfV8FAicJ/NKqlPbO35Djwna7k4IhdjOSTSYYTroT f5ew==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=nHzQDN2E; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74646-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74646-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id qb18-20020a17090b281200b0029315abe561si8058659pjb.127.2024.02.21.03.40.50 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 03:40:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-74646-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=nHzQDN2E; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-74646-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-74646-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 2E1C82838B3 for ; Wed, 21 Feb 2024 11:40:32 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E785F5380D; Wed, 21 Feb 2024 11:36:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="nHzQDN2E" Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8CDF446B9A for ; Wed, 21 Feb 2024 11:36:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515377; cv=none; b=BtOGdmAdWHjM8kpxDApv6G2Rbt0r1e4gfqJHyrTfEIZxs24bRrmvpkAgRt0jyXSHQlwY/3S0sqlkGjyutrdjxFZEMAm/pUep5MxdzP1HJiDG1n+xP+1acvb/04W+eo/l9/IGtaEGImweTiZSoAxN2kCQrscVer7yTv3Tw/YwXAI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708515377; c=relaxed/simple; bh=9RvLk+Y/2nEp04FCrDQs7qdDqW4evXdzWDP9s3Ohwys=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Xxejy//dideuwInGaUZEB25aC2PROOGVpC8Bx1K5C9z1WeDmJRfQr5ueDg8YApyfFiS/7goco2dmNtu2fVDVRGgody76nba9LE4MRPUC2b6kfyy1qArS4TfcwIZGKwQzxerYzDLiW2voTd0caH3V2pGemajwIgnzQHrstM6mzl8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=nHzQDN2E; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-608808c9646so8299607b3.2 for ; Wed, 21 Feb 2024 03:36:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708515374; x=1709120174; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=k0sTN6yoDasV9agD6pu51ddL9MvEfa3HQq1IU+fzHuI=; b=nHzQDN2E//v0W3b2dHEtEXRRm1NTgcD0mJdbKJZELZAvEGqc1JhjZCSygxfHf7m+rR n4K/8dNsi3OQ2p7Rq7TVZQtqqEVsrpECsgPpptIH7gkHQAO33P/s57uYRH810yaqmUhN XiXLGgn+27jFHnGi3d6hWzmIbP/1rIVVjWMNSnCSIBbSGW35Ahhw/6tMkAmG6l4r0bnz uYhboSvcp89jQPpoYSg6D3r1AFNCX8/q29DKQ8TMRTMghBklZIHtj3ZcBhzRtvhVkeFL MHSWtZU1sZF4H2IG7FnzNT2bQsPF6mGu5ldaB2dJdZhT6j3EEHLdlsPCRCXHiFDnDvdj RQZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708515374; x=1709120174; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=k0sTN6yoDasV9agD6pu51ddL9MvEfa3HQq1IU+fzHuI=; b=EHX24LnLAy7CU0ee0kICkzqixyO9UcSIxK+417ySPZno+N7Eudi3xk4CSfSkWBeI2N U9Oze2qinXMyMQ17MEQDLFdRTFjldYy4s5uKTOXAq+cfLvPejIO6W+QCINwwnWH+MLh3 BInbgpm7IeoC1NryoBl5ob184yH8ap0ez/oTQgBrhKc/oGeXp/tzRlLVBRAMVAJKjHdV kUAXJzPlGP2yHri95gQy6ClNnlbwJ3JxbFrhhdT6JZ3URqawKBDxWBSDq3JM5UU3ZLlf lc7+FAFepFYZjkcQhUEe+t0NO9MtSZ3FTiUwJ6aRXBNXx1QyU8Lh+5capCuKu4mGIIim vcZQ== X-Gm-Message-State: AOJu0YyXvKE377fkEjBHtoykZE4hk7AMKnnfFFow/sxvbSQ5fBl6m0Kk qYWOZWRFxlREF43CYIXiY7ytZAUQFLYc2cpceEsfMRAaz4xsXpYCf2gEk8vDRqsbzYn5mPXHhWx CEKMJi8f3kaL9X0f88RqgkktYhLVmMVK0DzzQZgiPcj1UFq3FuJ3nFPUA/bjBB2yVbOSHp731B3 ubnX6VWQek6KCrB40u26vsb/VAwL4NKA== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a0d:c946:0:b0:602:b7ce:f382 with SMTP id l67-20020a0dc946000000b00602b7cef382mr3316293ywd.7.1708515374295; Wed, 21 Feb 2024 03:36:14 -0800 (PST) Date: Wed, 21 Feb 2024 12:35:23 +0100 In-Reply-To: <20240221113506.2565718-18-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240221113506.2565718-18-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10347; i=ardb@kernel.org; h=from:subject; bh=W61hwMuIbwuGpPHigUY6x0LoOWZL0dAdP6qiyZqzCxY=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXq/W95xu/OveqIrnvltGR6qfaMTjYduQc77eUDJh4Nc pKcpV/QUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACayr4ThN5uiqViuT2fydCXx lXu61aKf7vwtaLSAVyA9JcRPstmijeG/d9fR7tQHnkntrWbi1VosjDMM1n4IkYlb86az6Zf0dXY +AA== X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240221113506.2565718-34-ardb+git@google.com> Subject: [PATCH v5 16/16] x86/startup_64: Drop global variables keeping track of LA57 state From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791508507317346100 X-GMAIL-MSGID: 1791508507317346100 From: Ard Biesheuvel On x86_64, the core kernel is entered in long mode, which implies that paging is enabled. This means that the CR4.LA57 control bit is guaranteed to be in sync with the number of paging levels used by the kernel, and there is no need to store this in a variable. There is also no need to use variables for storing the calculations of pgdir_shift and ptrs_per_p4d, as they are easily determined on the fly. This removes the need for two different sources of truth for determining whether 5-level paging is in use: CR4.LA57 always reflects the actual state, and never changes from the point of view of the 64-bit core kernel. The only potential concern is the cost of CR4 accesses, which can be mitigated using alternatives patching based on feature detection. Note that even the decompressor does not manipulate any page tables before updating CR4.LA57, so it can also avoid the associated global variables entirely. However, as it does not implement alternatives patching, the associated ELF sections need to be discarded. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/misc.h | 4 -- arch/x86/boot/compressed/pgtable_64.c | 12 ------ arch/x86/boot/compressed/vmlinux.lds.S | 1 + arch/x86/include/asm/pgtable_64_types.h | 43 ++++++++++---------- arch/x86/kernel/cpu/common.c | 2 - arch/x86/kernel/head64.c | 33 +-------------- arch/x86/mm/kasan_init_64.c | 3 -- arch/x86/mm/mem_encrypt_identity.c | 9 ---- 8 files changed, 25 insertions(+), 82 deletions(-) diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h index bc2f0f17fb90..2b15ddd0e177 100644 --- a/arch/x86/boot/compressed/misc.h +++ b/arch/x86/boot/compressed/misc.h @@ -16,9 +16,6 @@ #define __NO_FORTIFY -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 - /* * Boot stub deals with identity mappings, physical and virtual addresses are * the same, so override these defines. @@ -178,7 +175,6 @@ static inline int count_immovable_mem_regions(void) { return 0; } #endif /* ident_map_64.c */ -extern unsigned int __pgtable_l5_enabled, pgdir_shift, ptrs_per_p4d; extern void kernel_add_identity_map(unsigned long start, unsigned long end); /* Used by PAGE_KERN* macros: */ diff --git a/arch/x86/boot/compressed/pgtable_64.c b/arch/x86/boot/compressed/pgtable_64.c index 51f957b24ba7..ae72f53f5e77 100644 --- a/arch/x86/boot/compressed/pgtable_64.c +++ b/arch/x86/boot/compressed/pgtable_64.c @@ -9,13 +9,6 @@ #define BIOS_START_MIN 0x20000U /* 128K, less than this is insane */ #define BIOS_START_MAX 0x9f000U /* 640K, absolute maximum */ -#ifdef CONFIG_X86_5LEVEL -/* __pgtable_l5_enabled needs to be in .data to avoid being cleared along with .bss */ -unsigned int __section(".data") __pgtable_l5_enabled; -unsigned int __section(".data") pgdir_shift = 39; -unsigned int __section(".data") ptrs_per_p4d = 1; -#endif - /* Buffer to preserve trampoline memory */ static char trampoline_save[TRAMPOLINE_32BIT_SIZE]; @@ -125,11 +118,6 @@ asmlinkage void configure_5level_paging(struct boot_params *bp, void *pgtable) native_cpuid_eax(0) >= 7 && (native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) { l5_required = true; - - /* Initialize variables for 5-level paging */ - __pgtable_l5_enabled = 1; - pgdir_shift = 48; - ptrs_per_p4d = 512; } /* diff --git a/arch/x86/boot/compressed/vmlinux.lds.S b/arch/x86/boot/compressed/vmlinux.lds.S index 083ec6d7722a..06358bb067fe 100644 --- a/arch/x86/boot/compressed/vmlinux.lds.S +++ b/arch/x86/boot/compressed/vmlinux.lds.S @@ -81,6 +81,7 @@ SECTIONS *(.dynamic) *(.dynsym) *(.dynstr) *(.dynbss) *(.hash) *(.gnu.hash) *(.note.*) + *(.altinstructions .altinstr_replacement) } .got.plt (INFO) : { diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm/pgtable_64_types.h index 9053dfe9fa03..2fac8ba9564a 100644 --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -6,7 +6,10 @@ #ifndef __ASSEMBLY__ #include +#include +#include #include +#include /* * These are used to make use of C type-checking.. @@ -21,28 +24,24 @@ typedef unsigned long pgprotval_t; typedef struct { pteval_t pte; } pte_t; typedef struct { pmdval_t pmd; } pmd_t; -extern unsigned int __pgtable_l5_enabled; - -#ifdef CONFIG_X86_5LEVEL -#ifdef USE_EARLY_PGTABLE_L5 -/* - * cpu_feature_enabled() is not available in early boot code. - * Use variable instead. - */ -static inline bool pgtable_l5_enabled(void) +static __always_inline __pure bool pgtable_l5_enabled(void) { - return __pgtable_l5_enabled; -} -#else -#define pgtable_l5_enabled() cpu_feature_enabled(X86_FEATURE_LA57) -#endif /* USE_EARLY_PGTABLE_L5 */ + unsigned long r; + bool ret; -#else -#define pgtable_l5_enabled() 0 -#endif /* CONFIG_X86_5LEVEL */ + if (!IS_ENABLED(CONFIG_X86_5LEVEL)) + return false; -extern unsigned int pgdir_shift; -extern unsigned int ptrs_per_p4d; + asm(ALTERNATIVE_TERNARY( + "movq %%cr4, %[reg] \n\t btl %[la57], %k[reg]" CC_SET(c), + %P[feat], "stc", "clc") + : [reg] "=&r" (r), CC_OUT(c) (ret) + : [feat] "i" (X86_FEATURE_LA57), + [la57] "i" (X86_CR4_LA57_BIT) + : "cc"); + + return ret; +} #endif /* !__ASSEMBLY__ */ @@ -53,7 +52,7 @@ extern unsigned int ptrs_per_p4d; /* * PGDIR_SHIFT determines what a top-level page table entry can map */ -#define PGDIR_SHIFT pgdir_shift +#define PGDIR_SHIFT (pgtable_l5_enabled() ? 48 : 39) #define PTRS_PER_PGD 512 /* @@ -61,7 +60,7 @@ extern unsigned int ptrs_per_p4d; */ #define P4D_SHIFT 39 #define MAX_PTRS_PER_P4D 512 -#define PTRS_PER_P4D ptrs_per_p4d +#define PTRS_PER_P4D (pgtable_l5_enabled() ? 512 : 1) #define P4D_SIZE (_AC(1, UL) << P4D_SHIFT) #define P4D_MASK (~(P4D_SIZE - 1)) @@ -76,6 +75,8 @@ extern unsigned int ptrs_per_p4d; #define PTRS_PER_PGD 512 #define MAX_PTRS_PER_P4D 1 +#define MAX_POSSIBLE_PHYSMEM_BITS 46 + #endif /* CONFIG_X86_5LEVEL */ /* diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 9e35e276c55a..d88e4be88868 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1,6 +1,4 @@ // SPDX-License-Identifier: GPL-2.0-only -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 #include #include diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index b33f47489505..348fd69252ac 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -7,9 +7,6 @@ #define DISABLE_BRANCH_PROFILING -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include @@ -52,14 +49,6 @@ extern pmd_t early_dynamic_pgts[EARLY_DYNAMIC_PAGE_TABLES][PTRS_PER_PMD]; static unsigned int __initdata next_early_pgt; pmdval_t early_pmd_flags = __PAGE_KERNEL_LARGE & ~(_PAGE_GLOBAL | _PAGE_NX); -#ifdef CONFIG_X86_5LEVEL -unsigned int __pgtable_l5_enabled __ro_after_init; -unsigned int pgdir_shift __ro_after_init = 39; -EXPORT_SYMBOL(pgdir_shift); -unsigned int ptrs_per_p4d __ro_after_init = 1; -EXPORT_SYMBOL(ptrs_per_p4d); -#endif - #ifdef CONFIG_DYNAMIC_MEMORY_LAYOUT unsigned long page_offset_base __ro_after_init = __PAGE_OFFSET_BASE_L4; EXPORT_SYMBOL(page_offset_base); @@ -78,21 +67,6 @@ static struct desc_struct startup_gdt[GDT_ENTRIES] __initdata = { [GDT_ENTRY_KERNEL_DS] = GDT_ENTRY_INIT(DESC_DATA64, 0, 0xfffff), }; -static inline bool check_la57_support(void) -{ - if (!IS_ENABLED(CONFIG_X86_5LEVEL)) - return false; - - /* - * 5-level paging is detected and enabled at kernel decompression - * stage. Only check if it has been enabled there. - */ - if (!(native_read_cr4() & X86_CR4_LA57)) - return false; - - return true; -} - static unsigned long __head sme_postprocess_startup(struct boot_params *bp, pmdval_t *pmd) { unsigned long vaddr, vaddr_end; @@ -155,7 +129,7 @@ unsigned long __head __startup_64(unsigned long physaddr, bool la57; int i; - la57 = check_la57_support(); + la57 = pgtable_l5_enabled(); /* Is the address too large? */ if (physaddr >> MAX_PHYSMEM_BITS) @@ -440,10 +414,7 @@ asmlinkage __visible void __init __noreturn x86_64_start_kernel(char * real_mode (__START_KERNEL & PGDIR_MASK))); BUILD_BUG_ON(__fix_to_virt(__end_of_fixed_addresses) <= MODULES_END); - if (check_la57_support()) { - __pgtable_l5_enabled = 1; - pgdir_shift = 48; - ptrs_per_p4d = 512; + if (pgtable_l5_enabled()) { page_offset_base = __PAGE_OFFSET_BASE_L5; vmalloc_base = __VMALLOC_BASE_L5; vmemmap_base = __VMEMMAP_BASE_L5; diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c index 0302491d799d..85ae1ef840cc 100644 --- a/arch/x86/mm/kasan_init_64.c +++ b/arch/x86/mm/kasan_init_64.c @@ -2,9 +2,6 @@ #define DISABLE_BRANCH_PROFILING #define pr_fmt(fmt) "kasan: " fmt -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index 174a7192c9cb..3da15e9a8c7d 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -27,15 +27,6 @@ #undef CONFIG_PARAVIRT_XXL #undef CONFIG_PARAVIRT_SPINLOCKS -/* - * This code runs before CPU feature bits are set. By default, the - * pgtable_l5_enabled() function uses bit X86_FEATURE_LA57 to determine if - * 5-level paging is active, so that won't work here. USE_EARLY_PGTABLE_L5 - * is provided to handle this situation and, instead, use a variable that - * has been set by the early boot code. - */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include