From patchwork Tue Feb 13 12:41:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 200413 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp506791dyb; Tue, 13 Feb 2024 04:42:55 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWyVmqY8yfato6Apy5NPuvkmEqhusd0A/sTSrQsnJ+BLeWyao3hpd9OvJJGlDuTi0zbaLbL89KEh2+sU+SC9ieMJcli6w== X-Google-Smtp-Source: AGHT+IHEom/DehM9dv9+KvTGHSHJ15GP0EqVMV+9TjBvznQocCOsU/EId/F4LRlULTxw2YXOhR16 X-Received: by 2002:a25:10a:0:b0:dcc:7ad6:24df with SMTP id 10-20020a25010a000000b00dcc7ad624dfmr1621736ybb.17.1707828174934; Tue, 13 Feb 2024 04:42:54 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707828174; cv=pass; d=google.com; s=arc-20160816; b=kgGo+Vo3n8/U/BlJRUm90YpTxoegb0hAo0aY8m0j83bMayk4mRUPuydk9go+QQqKHF tayu83B1t18Eygz+tKJpWMTu/O4gCy8SJbajQdOFPwhf+8qU0fwy15XDLIfye/LaSKyY 2l78B7Oya9ERCBez8WzEs9XM3T1Zbq/mO63bryG1rtV38c8PUG3QcCZ/Ju4p3XpD5xEy Rr4U2xUo2mVQ7AaaSLBG2wupvH7yoZ85n5DR6gIQ7WvZ5dL2thm9DIzOn3wk0XWp700z dE9yKoY9FqsB/afzxOTnoqgwSZ+WX3QcW6ClQK3WuWeWXJA1VFaIDRNenQkfX++XrqrL 4lzA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=IntT6izjYhpSpfiju4QdrJP91shTnHbBbhRa2UK23MU=; fh=sYYh7IHoiLQAt2N95IJYBhNn1u5yjXy9LpFytBqIPx8=; b=uLfVWQg0S9STp25BRKsncGecf9YHKWefsZg3+SACsdDtFaPlr7A9PWs/oMmwJ52OZ2 uw2Cj1j+ozyssYbiWU+yG4P7YtIWnUi1YnFQGjFy7/bVhFTBlHrFv0nRWMP7ToCZRUV0 9uupz0uFppN+AZST1+pB7QMwk5fbjg1ZQoyp+bVUrRnAz/8mRrn0sEKKD4yQS3keCmtS iazTL13CxsTLk3FIDjUyg2YSsSkdfB8vBy1VMzDIgw1MPDiWyNZygFcAbjfaOzlk60q7 1lVb2I9trfZA8ALJ0A0Btmkh5WZd8WnhHGZYThOvR1hO3EXLl0SMzulkMS70QVdXF5ma 0cHw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=pAmQAn5V; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63521-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63521-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com X-Forwarded-Encrypted: i=2; AJvYcCUYOUxidBiucakYdE/KHILwnfVxbEgSeq1ByvURNS/LbWEOgCf4eGXgGFKX549FpBpBKubafcw1WEGRYXq8XFAGuxhDjA== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id v3-20020a05620a122300b0078717de605asi2111993qkj.359.2024.02.13.04.42.54 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Feb 2024 04:42:54 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-63521-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=pAmQAn5V; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63521-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63521-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id A43561C237E2 for ; Tue, 13 Feb 2024 12:42:54 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 818AD51013; Tue, 13 Feb 2024 12:42:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="pAmQAn5V" Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4EECE3FE27 for ; Tue, 13 Feb 2024 12:42:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828126; cv=none; b=AFi/oi8edBZR9Y2erK0j88hDAqe/iCn2v5Qn6RewElSRFxlbr6k7Nk4NOVoLA0rSq/d+WEvKuvI0zPFxeuqFZKaerxBRpjHGll8m3sGQm3XRgvn4ghVcYBe5ShB/QfL94XSTr21oTOeXhOyR+gf9kX/3gYFkJaa3QSkrh+/0r9k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828126; c=relaxed/simple; bh=YMvF0ORfuPi3XAD0k0pDL8v73CKkqlg6+lbGX9kyTzw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=gQsPoesAssF7ZNt+Osmcglm8SHJVA0rJGbb13WZz+22CJN7sj6R5vfbgpqa987FVY07ktvMSXOTm+XhbiJ7col1QIwmp+iK0Kikh5kAxDlBA7yPchKfvo/glAQglpcxgkbLX+M1LxrRW0Zv/RKWvJnU5YDtmjK0PAurMbKcb2AQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=pAmQAn5V; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-40e435a606aso27287515e9.3 for ; Tue, 13 Feb 2024 04:42:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707828123; x=1708432923; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IntT6izjYhpSpfiju4QdrJP91shTnHbBbhRa2UK23MU=; b=pAmQAn5VoM14YoCzbhlVh1q1bcvPTbZt3lgoqXK8nnMG9szFe/AjcdWa7L/gpZQUj9 vxWXO6lTkT+A+KVnExB9AgrQwM0iqsRzeTSfcfLzq4Tk0IJx7wYnEzHDlYV6XCcZakvR xZbwceDjSF42Z6CrQMwRu4oELsMTzUTIkFxsG+hoW0iYTIDlpLXQcJrfp9B+3yz+2ojl oiLq7WeS8qPXy3l+/Dr6BA9tmGphjLdJa5IBeQiyWNl6s+5Nd4hgB+v+6ma1B4fKR82Y BbtqQp3NOF9XhBm6/ezEPruQbFCQbi0f9EyJKFyIwDAPTqaU34F9LxbYLWy7sE8BLchf ykNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707828123; x=1708432923; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IntT6izjYhpSpfiju4QdrJP91shTnHbBbhRa2UK23MU=; b=E7cBhVUMuvKJ/+rMTnouPgycs5H/qtQaCGmWiWIvorQcZHikB3T9YLGHB6Bk8awPFG YNVZe84c9AwfSTnsAk8ealbejuoQ5JL9QFvS3Ols9iVdkWv8Id+EWo/jysB1IbGKHbdP xfDAI+tkt42aCUQW93eoSjmKVtduOCf4VoPpWmdP66iLvOr9ynvrICWekIN03aRaHKoX FDrhQw/bslaIi4biquazou2wZRNQzJcZGWNfa85KaSQhxYHeqqcpMACJbvfj02pT3SNu dRoML7vRO0BYOi0d5hdVqTaiIxoUsP+SAqXE6RI7GHLQkW8nvWvccx9mvdqF1ZDUOPcy ENUA== X-Gm-Message-State: AOJu0Yxnp10rOmsjq7/f25NOTHXaQc9TDR0WYLhTmPNPuHclCWjjxqXJ QvdSmgSQVpAz7bebSUbc6mccw4BnVNtM+QpJDbjqSaoCQ94I3/vkE5yWrw9hrkdKQrwcbeOYXKZ 5M/7sw1gd06dechoFLGmS7mh3fn1i6h9JgCO/iyi8zFLUhOZ/C9cmWs+dKGUPueDD1vmktsysnb sPHYqQ9lptOC+dXwu59Jebh63E8EAaxQ== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:518a:b0:411:a802:700d with SMTP id fa10-20020a05600c518a00b00411a802700dmr70870wmb.2.1707828123104; Tue, 13 Feb 2024 04:42:03 -0800 (PST) Date: Tue, 13 Feb 2024 13:41:45 +0100 In-Reply-To: <20240213124143.1484862-13-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240213124143.1484862-13-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6089; i=ardb@kernel.org; h=from:subject; bh=BsdoewkQfg4v5NoT6opMjWi0Vrg2mA8a392y9Jxs7FU=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfV0cueOKzuMyz5W71o5TYp9kdW5Wa0xu5br/SkOjb++u +9vHS9XRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjIkguMDDu+bZC0ZT+Yvm5u Ise2zLtZBuoPpiZea4wzc1/5zzhy1XOG/2G3FvnknSyvm3xqo5Hi+U/3rS7NzF25v9om6r3Ekno 9Nx4A X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240213124143.1484862-14-ardb+git@google.com> Subject: [PATCH v4 01/11] x86/startup_64: Simplify global variable accesses in GDT/IDT programming From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790787636619132398 X-GMAIL-MSGID: 1790787636619132398 From: Ard Biesheuvel There are two code paths in the startup code to program an IDT: one that runs from the 1:1 mapping and one that runs from the virtual kernel mapping. Currently, these are strictly separate because fixup_pointer() is used on the 1:1 path, which will produce the wrong value when used while executing from the virtual kernel mapping. Switch to RIP_REL_REF() so that the two code paths can be merged. Also, move the GDT and IDT descriptors to the stack so that they can be referenced directly, rather than via RIP_REL_REF(). Rename startup_64_setup_env() to startup_64_setup_gdt_idt() while at it, to make the call from assembler self-documenting. Signed-off-by: Ard Biesheuvel Signed-off-by: Ard Biesheuvel Signed-off-by: Borislav Petkov (AMD) --- arch/x86/include/asm/setup.h | 2 +- arch/x86/kernel/head64.c | 56 +++++++------------- arch/x86/kernel/head_64.S | 4 +- 3 files changed, 22 insertions(+), 40 deletions(-) diff --git a/arch/x86/include/asm/setup.h b/arch/x86/include/asm/setup.h index 5c83729c8e71..e61e68d71cba 100644 --- a/arch/x86/include/asm/setup.h +++ b/arch/x86/include/asm/setup.h @@ -48,7 +48,7 @@ extern unsigned long saved_video_mode; extern void reserve_standard_io_resources(void); extern void i386_reserve_resources(void); extern unsigned long __startup_64(unsigned long physaddr, struct boot_params *bp); -extern void startup_64_setup_env(unsigned long physbase); +extern void startup_64_setup_gdt_idt(void); extern void early_setup_idt(void); extern void __init do_early_exception(struct pt_regs *regs, int trapnr); diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index dc0956067944..9d7f12829f2d 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -22,6 +22,7 @@ #include #include +#include #include #include #include @@ -76,15 +77,6 @@ static struct desc_struct startup_gdt[GDT_ENTRIES] __initdata = { [GDT_ENTRY_KERNEL_DS] = GDT_ENTRY_INIT(DESC_DATA64, 0, 0xfffff), }; -/* - * Address needs to be set at runtime because it references the startup_gdt - * while the kernel still uses a direct mapping. - */ -static struct desc_ptr startup_gdt_descr __initdata = { - .size = sizeof(startup_gdt)-1, - .address = 0, -}; - static void __head *fixup_pointer(void *ptr, unsigned long physaddr) { return ptr - (void *)_text + (void *)physaddr; @@ -569,12 +561,7 @@ void __init __noreturn x86_64_start_reservations(char *real_mode_data) */ static gate_desc bringup_idt_table[NUM_EXCEPTION_VECTORS] __page_aligned_data; -static struct desc_ptr bringup_idt_descr = { - .size = (NUM_EXCEPTION_VECTORS * sizeof(gate_desc)) - 1, - .address = 0, /* Set at runtime */ -}; - -static void set_bringup_idt_handler(gate_desc *idt, int n, void *handler) +static void __head set_bringup_idt_handler(gate_desc *idt, int n, void *handler) { #ifdef CONFIG_AMD_MEM_ENCRYPT struct idt_data data; @@ -586,45 +573,42 @@ static void set_bringup_idt_handler(gate_desc *idt, int n, void *handler) #endif } -/* This runs while still in the direct mapping */ -static void __head startup_64_load_idt(unsigned long physbase) +/* This may run while still in the direct mapping */ +static void __head startup_64_load_idt(void *handler) { - struct desc_ptr *desc = fixup_pointer(&bringup_idt_descr, physbase); - gate_desc *idt = fixup_pointer(bringup_idt_table, physbase); - - - if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) { - void *handler; + struct desc_ptr desc = { + .address = (unsigned long)&RIP_REL_REF(bringup_idt_table), + .size = sizeof(bringup_idt_table) - 1, + }; + gate_desc *idt = (gate_desc *)desc.address; + if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) /* VMM Communication Exception */ - handler = fixup_pointer(vc_no_ghcb, physbase); set_bringup_idt_handler(idt, X86_TRAP_VC, handler); - } - desc->address = (unsigned long)idt; - native_load_idt(desc); + native_load_idt(&desc); } /* This is used when running on kernel addresses */ void early_setup_idt(void) { - /* VMM Communication Exception */ - if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) { + if (IS_ENABLED(CONFIG_AMD_MEM_ENCRYPT)) setup_ghcb(); - set_bringup_idt_handler(bringup_idt_table, X86_TRAP_VC, vc_boot_ghcb); - } - bringup_idt_descr.address = (unsigned long)bringup_idt_table; - native_load_idt(&bringup_idt_descr); + startup_64_load_idt(vc_boot_ghcb); } /* * Setup boot CPU state needed before kernel switches to virtual addresses. */ -void __head startup_64_setup_env(unsigned long physbase) +void __head startup_64_setup_gdt_idt(void) { + struct desc_ptr startup_gdt_descr = { + .address = (unsigned long)&RIP_REL_REF(startup_gdt), + .size = sizeof(startup_gdt) - 1, + }; + /* Load GDT */ - startup_gdt_descr.address = (unsigned long)fixup_pointer(startup_gdt, physbase); native_load_gdt(&startup_gdt_descr); /* New GDT is live - reload data segment registers */ @@ -632,5 +616,5 @@ void __head startup_64_setup_env(unsigned long physbase) "movl %%eax, %%ss\n" "movl %%eax, %%es\n" : : "a"(__KERNEL_DS) : "memory"); - startup_64_load_idt(physbase); + startup_64_load_idt(&RIP_REL_REF(vc_no_ghcb)); } diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index d4918d03efb4..3cac98c61066 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -68,8 +68,6 @@ SYM_CODE_START_NOALIGN(startup_64) /* Set up the stack for verify_cpu() */ leaq (__end_init_task - PTREGS_SIZE)(%rip), %rsp - leaq _text(%rip), %rdi - /* Setup GSBASE to allow stack canary access for C code */ movl $MSR_GS_BASE, %ecx leaq INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx @@ -77,7 +75,7 @@ SYM_CODE_START_NOALIGN(startup_64) shrq $32, %rdx wrmsr - call startup_64_setup_env + call startup_64_setup_gdt_idt /* Now switch to __KERNEL_CS so IRET works reliably */ pushq $__KERNEL_CS From patchwork Tue Feb 13 12:41:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 200416 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp507394dyb; Tue, 13 Feb 2024 04:44:03 -0800 (PST) X-Google-Smtp-Source: AGHT+IE65antjV7E7JMWvy7V8fOYCXVA5GmMGexvwkHFg2CxqKK7fkCuwI2aDUopJVA85NKLAT19 X-Received: by 2002:a05:6358:1902:b0:17a:ce18:8f65 with SMTP id w2-20020a056358190200b0017ace188f65mr10473270rwm.11.1707828243379; Tue, 13 Feb 2024 04:44:03 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707828243; cv=pass; d=google.com; s=arc-20160816; b=vn35TkNJmXueh/492UbiwimbuClh3x08IS3/PYMpfeR2lvMHd9ZbbnmNUQB5VEEafP 5Jo840FdBnxOSnATU1zhysw5luvCa2FtqJdL/U8Y0uckcyhqXtRrwUhV87goeIrDRiPc 6cducuDs4W6Jvqd1LVFuvymarph+QqzIAbvd4V5/6TcsRQti63M2W8bGBGehB1y7pPzB k0iEY1hqd9kJW5VHGKCr22y4NHSTWRXGWywGJkfJqHd3IBH7ETgPMiciZUefSZaXYW2E noICqFTgi4CMCA22pw2DYAPq0X2xtoub2QDIQNt8NUAZLUOGu0RZbl7cOEfwd60SDi+5 TqzQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=pi1QTrH02KU6Nn/lWHnRc1l99YL4s4zfhgW+K3Dfm0A=; fh=rryESkiiKKA1MmemdJKHhw/LXem+qqALA1sx3kqAiQI=; b=v0tDYHmLp7hIxzbRXy2Popb53n1WDb6VQVqKTSQOdjG/QLyW3y+Hu9oN12L+j6f1Ah DKPT9fozlMGDW2VhUS5nZh+NCz+QKjnE4pKpiPPKp4EYAtGmMyCWrId56sSmG8XZ681C yT6rJx0DqYGBcyLJv4TyuyFTKU81uNmC8Q5sSMs6FWzdlhNy3/lNeFcA8RKH7eAKS9zl lDtKrKQI8oLioUJqTQfIieP1RTgcduftRUWsfbaFw3bc1QssNY9zyWGHGkhwT6TUeEb/ C762q5y990evWqVLEjedS8Tnw3RLDL/Rq4k3esN5ikyTlbAdX1iRGxpE50Dyow36ZbjW EbnA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=tgBMO1XO; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63522-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63522-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com X-Forwarded-Encrypted: i=2; AJvYcCX0Oawo/qd4AA492VXDnMBDAxCBbGV+1HxScGbVYopKAzX6dKJhEHVgyX7QqK8Z4KDdaSj2wExuHyqjpdFaW2pZJP+9Zw== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id j186-20020a638bc3000000b005c220a94525si1972082pge.90.2024.02.13.04.44.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Feb 2024 04:44:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-63522-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=tgBMO1XO; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63522-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63522-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 62679B23B33 for ; Tue, 13 Feb 2024 12:43:18 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7E95B524B7; Tue, 13 Feb 2024 12:42:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="tgBMO1XO" Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D5044F887 for ; Tue, 13 Feb 2024 12:42:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828129; cv=none; b=HKhKFY02+uTj6LZDL8XaLKPOx/vEcoSe9ccH3xus+wI9zSWPX7f8KF3SQ2ICmBv0SpF+QP7vblQhaubHB+Phfby5L4Bb7LUaqS4Ax7sBPatIVUN3eaafBMeHzOyGxizKjLTOx+LKMAnvi5yJAmyLczrx1dqOgTdquGmHmP5Hh6o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828129; c=relaxed/simple; bh=DJj3BFuLzBkxhSjsLhYR2ws+biY4+9pzQT+zUXENl3c=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=DamQPl2f4m9z/8kSU7rep8MLHKfhj9qeVFfRG2AxFcfXX1/Wrzvl3o22B5t9ughWlP0HqPPabjeV1DbFnStKrW67kK/5OOPG2vYn/jYPpPIZTD3dF5+0luOYAc4gNRiAwvejuQAIpWn9+tn4zJnh7khVPCfBAW7/IeZg4zLVTxQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=tgBMO1XO; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-410dfd0e1caso10453095e9.0 for ; Tue, 13 Feb 2024 04:42:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707828126; x=1708432926; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pi1QTrH02KU6Nn/lWHnRc1l99YL4s4zfhgW+K3Dfm0A=; b=tgBMO1XOvMtOSxZQjNJtNl0WV2GoZkDuFTRzKPzl/jJloykXHc+//5GN5s5zeBqFEq V0HyaDuty3HSP7+PX7OXO3RhDd0JocG3vImWakEIrC1DVsIGVHl4okdHa5ONqWPyh/zI JTO93RRdpKXaIwA9MQod+Qju5E1WiFC6NaF+QhPbYNw+6pG2IpkG08LcSLo8tcvCUK05 Fy9k3vbQaXVFGvICqEWJ+v90mH5lAP8iOSSLl04dcOTsuO1IEn+XWz5+CyzZKTi31XGR Xe7GIKC96Xj4Iq45Mu3TIUcDitd8g0hqbsQ71ngBhV9WgP60xWmbBPm0FZEdRuUQfmPJ mCLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707828126; x=1708432926; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pi1QTrH02KU6Nn/lWHnRc1l99YL4s4zfhgW+K3Dfm0A=; b=D6QgeCFONUx0AbTotcvdBFOdp1dFmevu0Chrb2aDS9mtkdzgLo2D4Dx9E7xQnui7JD BCy2WAjbByHv7HXO7NrGFRKoWbXg3v7wV6tsrA3ZGeaaVWxtm5vPLsLUPAqD/KNjgv2t tIKVhnLL+p0xqaGT4YB5AwqqfwZB2nyGsr7WTGz+alnNLOGuJqDbGmYYnNPTaaOHYZE/ KbFEwIZcvNHKttL194bSWHVnIbHQK0wWxkNAtfs8FGvraHhYX/b/L06jlSArIGMyd0lU IuZ43Xo5pWDjG0IbpvF5tBaQ1HKKZoPre2gT8nOeoImNLmuuCxT3SIGYdSc1Q8mJfXW+ hf3A== X-Gm-Message-State: AOJu0YyZAf/6OoCE8UZtEc8xQnFKasPNdcdvjpC7y8ETfAHLTvZP/sHk ROkzl7lDgYGg5SRukf/NlRgheIRkcHnvaA76p/j5dT2VZjJI+4IlSjKJaBAEpWK/5ywD0zooAhd z4/HXHtNUQzoxax4m+7RuTrMXnDdujiS6UdrOj0si3kiSyq3ffJoUk6pUgj4b1bLsUoOacW7wY0 cz2AZu4dg48YT6DTCnCTn2rB+IKJs+xw== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:518f:b0:411:63af:64a with SMTP id fa15-20020a05600c518f00b0041163af064amr8527wmb.2.1707828125840; Tue, 13 Feb 2024 04:42:05 -0800 (PST) Date: Tue, 13 Feb 2024 13:41:46 +0100 In-Reply-To: <20240213124143.1484862-13-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240213124143.1484862-13-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6745; i=ardb@kernel.org; h=from:subject; bh=OERD3a266KSHXOboqoPZ/suMs8HVBE4QuXB3+uJqMiI=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfV0ctcjcR/xndn+O15duyZz/9+hBPe2oDCmepa35w473 WW7Pye3o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAExkYhIjw5nk2K6wbEedOOE5 Aee2v49UfdsekcA1pVqA9/uTXQrPtzIy7F3qo793SfOOimuX7v0o8mtZGZDpkXyn6GYXm8W8lwt VuQA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240213124143.1484862-15-ardb+git@google.com> Subject: [PATCH v4 02/11] x86/startup_64: Replace pointer fixups with RIP-relative references From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790787708033618669 X-GMAIL-MSGID: 1790787708033618669 From: Ard Biesheuvel The code in __startup64() runs from a 1:1 mapping of memory, and uses explicit pointer arithmetic to convert symbol references generated by the compiler into references that work correctly via this 1:1 mapping. This relies on the compiler generating absolute symbol references, which will be resolved by the linker using the kernel virtual mapping. However, the compiler may just as well emit RIP-relative references, even when not operating in PIC mode, and so this explicit pointer arithmetic is fragile and should be avoided. The fixup routines also take a 'physical base' argument which needs to be passed around as well. Convert these pointer fixups to RIP-relative references, which are guaranteed to produce the correct values without any explicit arithmetic, removing the need to pass around the physical load address. It also makes the code substantially easier to understand. Replace bare 510/511 constants with the appropriate symbolic constants while at it. Note that pgd_index(__START_KERNEL_map) always produces the value 511, regardless of the number of paging levels used, so a symbolic constant is used here as well. The remaining fixup_int()/fixup_long() calls related to 5-level paging will be removed in a subsequent patch. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head64.c | 57 ++++++++------------ 1 file changed, 21 insertions(+), 36 deletions(-) diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 9d7f12829f2d..4b08e321d168 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -77,6 +77,7 @@ static struct desc_struct startup_gdt[GDT_ENTRIES] __initdata = { [GDT_ENTRY_KERNEL_DS] = GDT_ENTRY_INIT(DESC_DATA64, 0, 0xfffff), }; +#ifdef CONFIG_X86_5LEVEL static void __head *fixup_pointer(void *ptr, unsigned long physaddr) { return ptr - (void *)_text + (void *)physaddr; @@ -87,7 +88,6 @@ static unsigned long __head *fixup_long(void *ptr, unsigned long physaddr) return fixup_pointer(ptr, physaddr); } -#ifdef CONFIG_X86_5LEVEL static unsigned int __head *fixup_int(void *ptr, unsigned long physaddr) { return fixup_pointer(ptr, physaddr); @@ -164,22 +164,21 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp, pmdv /* Code in __startup_64() can be relocated during execution, but the compiler * doesn't have to generate PC-relative relocations when accessing globals from * that function. Clang actually does not generate them, which leads to - * boot-time crashes. To work around this problem, every global pointer must - * be adjusted using fixup_pointer(). + * boot-time crashes. To work around this problem, every global variable must + * be accessed using RIP_REL_REF(). */ unsigned long __head __startup_64(unsigned long physaddr, struct boot_params *bp) { - unsigned long load_delta, *p; + pmd_t (*early_pgts)[PTRS_PER_PMD] = RIP_REL_REF(early_dynamic_pgts); unsigned long pgtable_flags; + unsigned long load_delta; pgdval_t *pgd; p4dval_t *p4d; pudval_t *pud; pmdval_t *pmd, pmd_entry; - pteval_t *mask_ptr; bool la57; int i; - unsigned int *next_pgt_ptr; la57 = check_la57_support(physaddr); @@ -192,6 +191,7 @@ unsigned long __head __startup_64(unsigned long physaddr, * and the address I am actually running at. */ load_delta = physaddr - (unsigned long)(_text - __START_KERNEL_map); + RIP_REL_REF(phys_base) = load_delta; /* Is the address not 2M aligned? */ if (load_delta & ~PMD_MASK) @@ -201,25 +201,19 @@ unsigned long __head __startup_64(unsigned long physaddr, load_delta += sme_get_me_mask(); /* Fixup the physical addresses in the page table */ - - pgd = fixup_pointer(early_top_pgt, physaddr); - p = pgd + pgd_index(__START_KERNEL_map); - if (la57) - *p = (unsigned long)level4_kernel_pgt; - else - *p = (unsigned long)level3_kernel_pgt; - *p += _PAGE_TABLE_NOENC - __START_KERNEL_map + load_delta; - if (la57) { - p4d = fixup_pointer(level4_kernel_pgt, physaddr); - p4d[511] += load_delta; + p4d = (p4dval_t *)&RIP_REL_REF(level4_kernel_pgt); + p4d[MAX_PTRS_PER_P4D - 1] += load_delta; } - pud = fixup_pointer(level3_kernel_pgt, physaddr); - pud[510] += load_delta; - pud[511] += load_delta; + pud = &RIP_REL_REF(level3_kernel_pgt)->pud; + pud[PTRS_PER_PUD - 2] += load_delta; + pud[PTRS_PER_PUD - 1] += load_delta; + + pgd = &RIP_REL_REF(early_top_pgt)->pgd; + pgd[PTRS_PER_PGD - 1] = (pgdval_t)(la57 ? p4d : pud) | _PAGE_TABLE_NOENC; - pmd = fixup_pointer(level2_fixmap_pgt, physaddr); + pmd = &RIP_REL_REF(level2_fixmap_pgt)->pmd; for (i = FIXMAP_PMD_TOP; i > FIXMAP_PMD_TOP - FIXMAP_PMD_NUM; i--) pmd[i] += load_delta; @@ -230,16 +224,14 @@ unsigned long __head __startup_64(unsigned long physaddr, * it avoids problems around wraparound. */ - next_pgt_ptr = fixup_pointer(&next_early_pgt, physaddr); - pud = fixup_pointer(early_dynamic_pgts[(*next_pgt_ptr)++], physaddr); - pmd = fixup_pointer(early_dynamic_pgts[(*next_pgt_ptr)++], physaddr); + pud = &early_pgts[0]->pmd; + pmd = &early_pgts[1]->pmd; + p4d = &early_pgts[2]->pmd; + RIP_REL_REF(next_early_pgt) = 3; pgtable_flags = _KERNPG_TABLE_NOENC + sme_get_me_mask(); if (la57) { - p4d = fixup_pointer(early_dynamic_pgts[(*next_pgt_ptr)++], - physaddr); - i = (physaddr >> PGDIR_SHIFT) % PTRS_PER_PGD; pgd[i + 0] = (pgdval_t)p4d + pgtable_flags; pgd[i + 1] = (pgdval_t)p4d + pgtable_flags; @@ -259,8 +251,7 @@ unsigned long __head __startup_64(unsigned long physaddr, pmd_entry = __PAGE_KERNEL_LARGE_EXEC & ~_PAGE_GLOBAL; /* Filter out unsupported __PAGE_KERNEL_* bits: */ - mask_ptr = fixup_pointer(&__supported_pte_mask, physaddr); - pmd_entry &= *mask_ptr; + pmd_entry &= RIP_REL_REF(__supported_pte_mask); pmd_entry += sme_get_me_mask(); pmd_entry += physaddr; @@ -286,7 +277,7 @@ unsigned long __head __startup_64(unsigned long physaddr, * error, causing the BIOS to halt the system. */ - pmd = fixup_pointer(level2_kernel_pgt, physaddr); + pmd = &RIP_REL_REF(level2_kernel_pgt)->pmd; /* invalidate pages before the kernel image */ for (i = 0; i < pmd_index((unsigned long)_text); i++) @@ -301,12 +292,6 @@ unsigned long __head __startup_64(unsigned long physaddr, for (; i < PTRS_PER_PMD; i++) pmd[i] &= ~_PAGE_PRESENT; - /* - * Fixup phys_base - remove the memory encryption mask to obtain - * the true physical address. - */ - *fixup_long(&phys_base, physaddr) += load_delta - sme_get_me_mask(); - return sme_postprocess_startup(bp, pmd); } From patchwork Tue Feb 13 12:41:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 200414 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp507039dyb; Tue, 13 Feb 2024 04:43:23 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWleunWe5o0KcEsuzLt6ukslJ1YMd9QgCqvgPMoCPApH/H2mV1wHKZbtF9vBz8VxbigjX9qbJWItIjE4ypbuRin8Ezk5w== X-Google-Smtp-Source: AGHT+IErRgzw5ExbEQKu18xTODECAMnM3sjBlVFbNW275wrQX/FqJptPcKWCOKDrEopxrFPlDuRn X-Received: by 2002:a05:6a20:c792:b0:19e:4e41:7ae0 with SMTP id hk18-20020a056a20c79200b0019e4e417ae0mr3171650pzb.20.1707828203022; Tue, 13 Feb 2024 04:43:23 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707828203; cv=pass; d=google.com; s=arc-20160816; b=LhorKTgrUJwOoXq2bIyUVb1yozXcYMviEF3MI/CvcBhBZXZUX8EGF17E55CHDAwoxG TAEQqXquLMoH5EqU8LXDfIyXrPNacbwQLl9sDktysD2Y0WT8nVTMc9eJ564zW3J3O3yg BTVFzOca5aH4XjlPW8A21hWJWp5EqdOnf1GdX1i3YBvDC4kWC/pXUY9y5JPpsZtXpANL nTRnnLbv5YozTFjasClq89zEwC2vtUzIHRck2FuF++1A2sb0XmBIKa0NWGgCp2VYrXV+ y3E39t94kTPVC9oXwnkxtJn70W/N6ZuNkX4kN4PxYY9G29LtdFVX7Sh9U+evRtP21o7e A+WQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=R8PzBwStBQmNo4IaP5da+Ydr5S+nPOaVv6NiNBRmMEI=; fh=53uUiUcD6b+VxcaUX5H0rX8jbUzS/Fzb7T+QgZy+5PA=; b=1Bn+1GKWv+4wAOoVODWBAL+Jnl+YnFhCEUy3Klt/InarWv1ZH8dyJSIXCeczzsPYss FkYXjuzRYXx4ZADm69gQHxKHwm6GFkXW59DJ5qKgRJo5shBZDWPn/qJ+EA3YOZJFffqu CQdjohAC2oiLKkP0on3yzLbdxeo5S/7qwyP9pZWpR20RxBiaZ70EM491Nlz8BX9hXEOG XnhncetbLjfUElbdHo6nx+0UPtqZfi7mfMXP0yBB7ms4+C/NTfiwom+QqxLnPTgqUz+w qSEeVXx9ovfv1NneTUjjZ86fTZeT00tud1PMchCcrGLLlx0U1FMlejJcYXbgFf+ddhxl OObQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=Ulj1QaRz; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63523-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63523-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com X-Forwarded-Encrypted: i=2; AJvYcCVBgBvNXsk3MJ+wUgk7/Qjla+YHewTKXtNt0OzsympdLMZ04CdRsVhwYANsdx1cgQHUjhd8Wi9OB0Lt3/wjgsSY0W/mLA== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id l28-20020a63ba5c000000b005d8b7ca8ffdsi1913788pgu.327.2024.02.13.04.43.22 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Feb 2024 04:43:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-63523-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=Ulj1QaRz; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63523-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63523-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id BE3C7281FBD for ; Tue, 13 Feb 2024 12:43:22 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 52584524B8; Tue, 13 Feb 2024 12:42:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Ulj1QaRz" Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D037C50A7F for ; Tue, 13 Feb 2024 12:42:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828131; cv=none; b=QfqKn314PJzzYHflP51gR2wEnOopq0vT7jlNgM9SF+84vD5IRVV0m6sdeMP10oezSxj/8VFZKEpyamw49mGZSVoLJV+98DgSSZZtZhbPz3T3Krwlw2j5IzNJP0/DG8fgRLZAmw5mvgzcxjgVvBSoodwkbtQoJzqrbOVSidQ1CXI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828131; c=relaxed/simple; bh=jPHNmEDl0ryiAggUWuXvb8LuTJ6Z4Zpf6+3+QYXDVB4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=XjFo8aYcn5T+HQHmqUSxeBQ1ciQ7n08ZxExpQO9ZWHqDRYytQpLQL2Hxkd2v9IwCV7ctEBLpHllh7bNHhq+KijreSjVuMAkNd9zqCQ4OAKqCZOTQCkX8BCf/SpnaBQlS0t7hfySNN4PLjHWFLHIZNcQ13bXH5RiylygyZVzWl2U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Ulj1QaRz; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcc0bcf9256so1704171276.3 for ; Tue, 13 Feb 2024 04:42:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707828129; x=1708432929; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=R8PzBwStBQmNo4IaP5da+Ydr5S+nPOaVv6NiNBRmMEI=; b=Ulj1QaRzhF/2z0kHbIl0qORR2B/5JZX1j9bvKaguuYs0KnvmnLFoA5G8B8vyRMTTX2 aNjqRPen6MrgyFLbYrFpnK7PbEmvI3OpkyKmxG8AoBL7zY2RYzYyYCL9LzZK2QbT6qi/ D+2KAkUtuW4eCR9a4dbHSylD7EH5GgkG9LFnSIEF4oCu/f63FQSzqfIJJlzSMDeeirTN KsF9Q0gmZeftSDy0VgUbZ2BKIJzr4B+EGoNEwlARhLEpbX1oG0sbW5RC/LAMCEY098j6 65DEqwagXx65SCaCFhL9Bjupvc+EXJgFdpDS/HoZdqQHhC8nk0oqJdX/4GyscKuYAAZM gm+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707828129; x=1708432929; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=R8PzBwStBQmNo4IaP5da+Ydr5S+nPOaVv6NiNBRmMEI=; b=evbW2X0QK/9P6gUvtIQMzemR5M9B7bGsvkSVrshYXUk1FfAISa4KPxmMtkPk1nVYtZ CbQnuO+P4Z1dDJUneEEWXbnmiir3fTcMeYGKGhwQd12q8cnk6zGL9LhbKlcOq1vPGFg9 MG8mBfEDDWveOc5wKHCoUNx941zHGS+erY9ODzW93uCYRLchOZ6yOnpBXJSmlYMh+5U2 c7gLJKIYDF5UX4g9Y6T7mnd32EJyLCiIydwkZaxYq2pTdav9MGlRXyn/u3t7GCgndFtg xkWJxCQN+IMvBGSl1K3EW8Oc4W7MdviD7G1Nnx1jkjKdkj0nJc6YZDd6DAG5DhkYYR+G UoYg== X-Gm-Message-State: AOJu0Yx7vbLIwLyRuidhhtxZNxesRtf8GqILc61eAFfzHdr0/j5NHCyj 1zig+r7r7LW/K8yA+1LspN8MXEmxlKL6Pbd3C0sl8WcyONigUc+6NdTW0wgt7/WKPLWGu/oyXf1 NBVONBSEAM+BNboPo1TxZDlnY8+9Gf0IUIkJwOouSGmTnLj5Pgf+QRh4bQQ4IrgIiseShl0UG40 DAW651YxacwQCMP7oYgqwi9QQvkyRDwQ== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1028:b0:dc7:3189:4e75 with SMTP id x8-20020a056902102800b00dc731894e75mr346289ybt.3.1707828128483; Tue, 13 Feb 2024 04:42:08 -0800 (PST) Date: Tue, 13 Feb 2024 13:41:47 +0100 In-Reply-To: <20240213124143.1484862-13-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240213124143.1484862-13-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2373; i=ardb@kernel.org; h=from:subject; bh=4k+FBUPb8cUOIJZ1D8SGO5j46VqaiuS2qc/hfQO7nqA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfV0cnf5JtewZYpJvdc2m4UXi63ZycLU4s3Y8/T5sZxrm bOVH1h1lLIwiHEwyIopsgjM/vtu5+mJUrXOs2Rh5rAygQxh4OIUgImsYWX4K/p4j/cbl04206aX nas2ySWs8X9we3KI9D3lOXJvToscymJkmOA068FH/ZhrTr+kemP/3SvaKS3atlwuVc9F9fTLNa6 7+AA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240213124143.1484862-16-ardb+git@google.com> Subject: [PATCH v4 03/11] x86/startup_64: Simplify CR4 handling in startup code From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790787665607066014 X-GMAIL-MSGID: 1790787665607066014 From: Ard Biesheuvel When paging is enabled, the CR4.PAE and CR4.LA57 control bits cannot be changed, and so they can simply be preserved rather than reason about whether or not they need to be set. CR4.MCE should be preserved unless the kernel was built without CONFIG_X86_MCE, in which case it must be cleared. CR4.PSE should be set explicitly, regardless of whether or not it was set before. CR4.PGE is set explicitly, and then cleared and set again after programming CR3 in order to flush TLB entries based on global translations. This makes the first assignment redundant, and can therefore be omitted. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head_64.S | 24 ++++++-------------- 1 file changed, 7 insertions(+), 17 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 3cac98c61066..7e76cc0b442a 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -185,6 +185,8 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) addq $(init_top_pgt - __START_KERNEL_map), %rax 1: + /* Create a mask of CR4 bits to preserve */ + movl $(X86_CR4_PAE | X86_CR4_LA57), %edx #ifdef CONFIG_X86_MCE /* * Preserve CR4.MCE if the kernel will enable #MC support. @@ -193,20 +195,11 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * configured will crash the system regardless of the CR4.MCE value set * here. */ - movq %cr4, %rcx - andl $X86_CR4_MCE, %ecx -#else - movl $0, %ecx -#endif - - /* Enable PAE mode, PSE, PGE and LA57 */ - orl $(X86_CR4_PAE | X86_CR4_PSE | X86_CR4_PGE), %ecx -#ifdef CONFIG_X86_5LEVEL - testb $1, __pgtable_l5_enabled(%rip) - jz 1f - orl $X86_CR4_LA57, %ecx -1: + orl $X86_CR4_MCE, %edx #endif + movq %cr4, %rcx + andl %edx, %ecx + btsl $X86_CR4_PSE_BIT, %ecx movq %rcx, %cr4 /* Setup early boot stage 4-/5-level pagetables. */ @@ -226,11 +219,8 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * Do a global TLB flush after the CR3 switch to make sure the TLB * entries from the identity mapping are flushed. */ - movq %cr4, %rcx - movq %rcx, %rax - xorq $X86_CR4_PGE, %rcx + btsl $X86_CR4_PGE_BIT, %ecx movq %rcx, %cr4 - movq %rax, %cr4 /* Ensure I am executing from virtual addresses */ movq $1f, %rax From patchwork Tue Feb 13 12:41:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 200415 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp507217dyb; Tue, 13 Feb 2024 04:43:43 -0800 (PST) X-Google-Smtp-Source: AGHT+IEi7UT2/0rJ9ualPP02homoZLuXa2m1JKqEpsH+hcIFVAZMyfmSX5cZImk9vn+2X51P8OvE X-Received: by 2002:a17:906:e257:b0:a3c:9206:608f with SMTP id gq23-20020a170906e25700b00a3c9206608fmr4798254ejb.0.1707828223256; Tue, 13 Feb 2024 04:43:43 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707828223; cv=pass; d=google.com; s=arc-20160816; b=rFU4vk0OgyNFaJc34mFsux9CmNazRrMdl/IdibXSzb8vGodVK1BITauZz7hpqDDmKD LmvjCJ4im2ChfhnXUUfCd1Og1RQ2bfhSNnkO2Iou+bDR1d3vQ0pJ5Bg4089FD8gULVtQ N9fwZ7xSda5AWm8OqtVDtDDcRcORm5aYXTst+ri19SG5oi/m5IDNCTU94XeSA0k03hb7 s/AcR8xUP5Ozac+7ZMUJNtvRCET5ksR8vQGo+HZ1x23HvoRJizb/v8R4RItW8f6H4NTl Panjh8TkiXOnF9WTposCcy9OW7921B38f2vl6r7pQ3mpl8w6SXNiBf4Ify9GNVG5jOM9 cfDA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=SCp/zoIzhRhhb9JUvWveq8Y/lRzJCxLp1biyCfwnfq0=; fh=e+xdvUvoO6swtC40e28a1eDsy5sMVkWerbuU+KL0HAQ=; b=KuKxQli87doGJ88EvfGjxtrUR8HjE/nckgL1vMyXFtiZ4y1bYP1hmigmc9bn5PvYjo HH+fGZuGI7TmrSKCop2XlfI0yf/5Jtpqb7Q3Wve35ZAWRq62N5l8LEosOx+lnJDxMR3p AGno0nnJAth5hUDx94fmusM9nFta1yFkMFvldT4JqJaIPSxksgXwh4RcpIh/nfMESlKL 1hDZ3Ml+O3aCACvdAOueXsmjEUp/Idn3Y6fo5PpxU6w9jagJEccFh3gLGoF/KI7S9E8g bjzVmfKigI5kZOmgHOIHK/h9EcJmRhiHVt+9WkRxOovfpoEeuFWVfEPAib3M84vWh1iS CQeA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=vqgCL7dM; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63524-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63524-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com X-Forwarded-Encrypted: i=2; AJvYcCXUUhhw+QD0Kyz7s6XF6NwsfzUJfrlOm1huvJ0WxcbNAR5+aPQkvsUKjqsuo1TT4qIBAkHvO4I14E+LiLrofscKq96g1w== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id h24-20020a17090619d800b00a3d0411ebc8si522084ejd.63.2024.02.13.04.43.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Feb 2024 04:43:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-63524-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=vqgCL7dM; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63524-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63524-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id AC30F1F259FE for ; Tue, 13 Feb 2024 12:43:42 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8FB5F52F94; Tue, 13 Feb 2024 12:42:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="vqgCL7dM" Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE68651C42 for ; Tue, 13 Feb 2024 12:42:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828134; cv=none; b=D6Wqp02rya+GeyxUYSt6bXSxCf74dHrXEgbgPcEcTo3RdFIiuUz0cP7up/6Qn0uwxaNt+RVr1Ji4T5Wp1bT12IxBSo9KWwBPZ96W4eKOra2UJIwIb8MMLJyPGyVmefxzw2Mz6GX3VXUxZpAVvVHY1CgRZGZDv15k8imnPAuW4y8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828134; c=relaxed/simple; bh=y9rxsCe7t3+BoRvciY7zzDK+lsJfA46YxMM4xo0hlg0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=JRiuChH6kBZjNdpXX05p04y+1mDtPxmpwGuozihydOlA9hUw69Y3x7hqYBcpPwwyz8Nq+uKQoGbIC5tIaKufVlRgRN8o9KU4s/uYlTKvecj3Y7ixAjclVmXFO9yyhropNowQkm75qaTd7129AxTSZRgvPfxo2vTOXY727T4wX+c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=vqgCL7dM; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6077f931442so14216357b3.3 for ; Tue, 13 Feb 2024 04:42:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707828131; x=1708432931; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SCp/zoIzhRhhb9JUvWveq8Y/lRzJCxLp1biyCfwnfq0=; b=vqgCL7dMY/OFJl8YlLz32pwDD/XuxbQr5hLcfE4Gq0b+kPvmqrflXFRbpGLnIszomv BsnEKi82TtH8Sjs9c0g06lplf5Cxmryt9EQIRlvy9sc8WkgxbxPfH8+03WdFZkiFdbIU SS6H2buQAdnY82a8LZs8nBuVFtyM5Ii3gdBYb7DQoAyI1ZVKNOSXC0+8XbQ1bUqlhpjw fdxTva6JI6b8IapQF3mrfkuVPcHE+cmpvTThBGYLHfdAvazfo+ac5LyBLirMx51fY60i H3v7L9OFgccwAFEQ1fo3emfMczMcfCF+u4MsZXdVGKIzqAm3i6+Fxm7b4XilbJUi6eEt 54FA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707828131; x=1708432931; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SCp/zoIzhRhhb9JUvWveq8Y/lRzJCxLp1biyCfwnfq0=; b=f8U8y4Vs4vBexR/JrVUfL/X/qw6wRRpo5gyVF1s/Phb/nzuh9L9gMvqjYDxDdLIPap WLfS0L3QeJlU0DPGcyWLxw6w6NFr5SjNzwk4ZYCnXwP5rzUs5wemDrB3dkhDsFDU6fzx Lmbibw1ZDsZoS9y2SZ2XEW/RO/hjFY83LbJggoylyvC2aHYLjZQ2DsK/ePWIcdWpYfht mPmf6JaKDnS2NTu6bZmMhxmLz9o+aun3vZl/G5r3Ip3uL23haZaaxg6G+td7zR+lqiK/ hVAXgntUMBqTXloUal8gsRnqnnUQAP9t5BWRIumbNl2vRb+mxXvbm0VP8qGo9cxbczar QdAg== X-Gm-Message-State: AOJu0Ywrwq5QulubFrCn6h2IBR+tdO9eaMGdQmymGZqAon/aBv2DgBfm HRCzdmYSbkxf54LS4p87J7woW0Q2vjLwJqEzddMCLAUglSNztFyNrQI0Pee8PVpqK1o8uNTs/M1 KPc4kIXxbs1dsYCcAlRxhZJ1Wv0cpX2vRwcbGqwR4sLDUq5w9H12ELK/WQ+xqnnJi/rwmtJwvTU SFpRuoYRj4cil4GqZQEhiZfyRGwS+ljg== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:6d05:0:b0:607:9963:2dd4 with SMTP id i5-20020a816d05000000b0060799632dd4mr90739ywc.9.1707828130803; Tue, 13 Feb 2024 04:42:10 -0800 (PST) Date: Tue, 13 Feb 2024 13:41:48 +0100 In-Reply-To: <20240213124143.1484862-13-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240213124143.1484862-13-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3321; i=ardb@kernel.org; h=from:subject; bh=z98YWbAEHVe7Ntbv2WAOVJQ9O8vTY0tfpKYTE0xX9/E=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfV0ck841z3bDM4lraYBmROE2A0KGp8WRh3hbUm0T2C9Z bj2lENHKQuDGAeDrJgii8Dsv+92np4oVes8SxZmDisTyBAGLk4BmMj+XYwM8/bZqcV2fOwPq966 6h2LxWvenZlbOGWbAk95ne/M2CKXxshwU41PiU1nTfbReNcy8diKz7P1jp4JEnhypeRl+8m1ga9 ZAA== X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240213124143.1484862-17-ardb+git@google.com> Subject: [PATCH v4 04/11] x86/startup_64: Defer assignment of 5-level paging global variables From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790787687168931377 X-GMAIL-MSGID: 1790787687168931377 From: Ard Biesheuvel Assigning the 5-level paging related global variables from the earliest C code using explicit references that use the 1:1 translation of memory is unnecessary, as the startup code itself does not rely on them to create the initial page tables, and this is all it should be doing. So defer these assignments to the primary C entry code that executes via the ordinary kernel virtual mapping. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head64.c | 44 +++++++------------- 1 file changed, 14 insertions(+), 30 deletions(-) diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 4b08e321d168..4bcbd4ae2dc6 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -23,6 +23,7 @@ #include #include +#include #include #include #include @@ -77,24 +78,11 @@ static struct desc_struct startup_gdt[GDT_ENTRIES] __initdata = { [GDT_ENTRY_KERNEL_DS] = GDT_ENTRY_INIT(DESC_DATA64, 0, 0xfffff), }; -#ifdef CONFIG_X86_5LEVEL -static void __head *fixup_pointer(void *ptr, unsigned long physaddr) -{ - return ptr - (void *)_text + (void *)physaddr; -} - -static unsigned long __head *fixup_long(void *ptr, unsigned long physaddr) +static inline bool check_la57_support(void) { - return fixup_pointer(ptr, physaddr); -} - -static unsigned int __head *fixup_int(void *ptr, unsigned long physaddr) -{ - return fixup_pointer(ptr, physaddr); -} + if (!IS_ENABLED(CONFIG_X86_5LEVEL)) + return false; -static bool __head check_la57_support(unsigned long physaddr) -{ /* * 5-level paging is detected and enabled at kernel decompression * stage. Only check if it has been enabled there. @@ -102,21 +90,8 @@ static bool __head check_la57_support(unsigned long physaddr) if (!(native_read_cr4() & X86_CR4_LA57)) return false; - *fixup_int(&__pgtable_l5_enabled, physaddr) = 1; - *fixup_int(&pgdir_shift, physaddr) = 48; - *fixup_int(&ptrs_per_p4d, physaddr) = 512; - *fixup_long(&page_offset_base, physaddr) = __PAGE_OFFSET_BASE_L5; - *fixup_long(&vmalloc_base, physaddr) = __VMALLOC_BASE_L5; - *fixup_long(&vmemmap_base, physaddr) = __VMEMMAP_BASE_L5; - return true; } -#else -static bool __head check_la57_support(unsigned long physaddr) -{ - return false; -} -#endif static unsigned long __head sme_postprocess_startup(struct boot_params *bp, pmdval_t *pmd) { @@ -180,7 +155,7 @@ unsigned long __head __startup_64(unsigned long physaddr, bool la57; int i; - la57 = check_la57_support(physaddr); + la57 = check_la57_support(); /* Is the address too large? */ if (physaddr >> MAX_PHYSMEM_BITS) @@ -463,6 +438,15 @@ asmlinkage __visible void __init __noreturn x86_64_start_kernel(char * real_mode (__START_KERNEL & PGDIR_MASK))); BUILD_BUG_ON(__fix_to_virt(__end_of_fixed_addresses) <= MODULES_END); + if (check_la57_support()) { + __pgtable_l5_enabled = 1; + pgdir_shift = 48; + ptrs_per_p4d = 512; + page_offset_base = __PAGE_OFFSET_BASE_L5; + vmalloc_base = __VMALLOC_BASE_L5; + vmemmap_base = __VMEMMAP_BASE_L5; + } + cr4_init_shadow(); /* Kill off the identity-map trampoline */ From patchwork Tue Feb 13 12:41:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 200417 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp507507dyb; Tue, 13 Feb 2024 04:44:16 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWGi50tVHOvOFf927WCz8Cm6Q15fJzuelLKfc1YbT27M8hxOeILdxCBJv3LNV6O7v66Vahdl/kxXYBH7EWWK7GNq2oprw== X-Google-Smtp-Source: AGHT+IHEcmt8KI86aIewozkc6zpYKaMR27WNHlKelVx9q6K4QIfJMTsQnNHaZKBIrpxofagU3nuu X-Received: by 2002:a17:906:34c7:b0:a36:c466:52ea with SMTP id h7-20020a17090634c700b00a36c46652eamr6573757ejb.75.1707828256155; Tue, 13 Feb 2024 04:44:16 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707828256; cv=pass; d=google.com; s=arc-20160816; b=fnx2+ZVi+kff/X5kHs12DFg30g2R8JfiBH9vVA+I+WD9z9bX3ZZvKNxNTE5ctlGETA EBk/OG7ATNQ4gqKRWrw5ddJXCuDnm9ATe1VdPNQUuPGtbvag5tUYU78Gnz6ll+bseg8Y IDzSTEsSpTkgwScgstxuYC6cFZ6xIUFTDppgS+Pz7306ppHlKrlOr3IannYiHjX05x1d JwLAKNI1/8fsPyeGpgnLimx63LmlEiIm79OyLSvf0hgmlhaxjoEvH1jCud8zwosuCkAU A8dcBpnYsKBSYiIY7lYZBcWBVP4dd43OHww+YnWeKqMZ8OF2D/prBRM7GL3UPTXGThNi uHPg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=l9PGLkbb8+qoug2ZrrnwKu6zhcSycBxmHA7JvTLYhJc=; fh=pT8WZEymctz/R/C04LxqwIz6sTk4RG4QlCrB6Gx50eo=; b=uMI+W5aWg+RM/KlknWr4D8IEuJf4WTQH4RuTsB5Gi7rkqnt5JLp+cU9nHnxHvbZB4f PBsTMH7AcDM+rtdz7SlE9mvMH9zoGB2rsRS7G4vXiKw6pb3iHYk8seTAZm1MF/bEb2UO wctuAWgI+UYKpuHjJmx+r96VyEiTP934NU9GhmhIrcylKjJMgeBscneQk+CT3ucQQHBe qUTIwjpdB28UeKICe0Ot6wYgVRm1zYwH1jG474Ue1sr51zRq1ILFMf9qVeVA4hC8DDN/ y2FK2kxaz7RES9+lb3h6nHDwroi9CL3S4cSxBZAcM7unTUxWK3bQM/QT/wQ1hIwRoAe9 Eraw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=r4kCWYai; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63526-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63526-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com X-Forwarded-Encrypted: i=2; AJvYcCVtABIIKRmEIwLvWYL8oC6mlF/JcXc4kG3Uh5VhHUFPIFpCRZiH9lnrBINPFw2iU2+5yDw5NIUSFXy5m41/L2ptwB8efQ== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id gw24-20020a170906f15800b00a38273c1419si1207319ejb.699.2024.02.13.04.44.15 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Feb 2024 04:44:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-63526-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=r4kCWYai; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63526-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63526-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 941F81F25398 for ; Tue, 13 Feb 2024 12:44:15 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2BDE7537EF; Tue, 13 Feb 2024 12:42:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="r4kCWYai" Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6058B52F7C for ; Tue, 13 Feb 2024 12:42:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828138; cv=none; b=AhEBAqwMi6u3t65HFhgWQJ6ocJv8dogOYm0t6J9gVIrXT17jLBOi3d2II3s3VCmE0omsQKBPKo7EP1os8lnEemCt6i6+pUTSpcXKENITWhxb/zNfd4Yk/l/CUCNggYxVdHp7c6yoYbgpDVdl3FSmmpcA7jNdDwDjaPb8ZHZ5YDY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828138; c=relaxed/simple; bh=V43RwcksINAVGLrcC9zNozM9sOBz5XCNbWyT4FUAcEc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Haw2Z0gHxWYmtl38lFbcLKkUNAliDFslA83AF/9iXs+fILKloYWMk9+dbmikjK9Q/yv112f6od7xpTdEMicR7TkKij6AXlBnloT9FhLDfL0/P4dRjWbsQpuXnwQEf8+LfYC8j4o+qbHV3c1gALcyPWGT45ZSo6uDbG/TrRqUn4s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=r4kCWYai; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dcc73148612so1072501276.1 for ; Tue, 13 Feb 2024 04:42:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707828135; x=1708432935; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=l9PGLkbb8+qoug2ZrrnwKu6zhcSycBxmHA7JvTLYhJc=; b=r4kCWYai+0z+MLg4WaOkHuJmakMAZ91VFPT+1f408MTofIGUD3p/+pL28YA0p6yjft owaf+C+2dj8bL8gawo2hWlD9l7eoy/9K62lSt96Iss77DODdXy2HVhVhwq6Twvh+Fxuv 1m9ToZNWOSH2epX0uxyGpfVC1F7kcuqu4LNPRACokkycXD5Dj4DtTOKUWsQNnUWhPeLT yrkRt6Ag8abtKIut6lCVdWUI7Yw2HRhMFU3wJzzwk4ND3gIAvNxcRINNUn1OT5IwzdQw bpFJW9eMqS6AwkzRpxF8rFSWDDoKdlTSmoaNznZyEw+YWCK7D44AeqE+hesjt82Vx6G5 68LQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707828135; x=1708432935; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=l9PGLkbb8+qoug2ZrrnwKu6zhcSycBxmHA7JvTLYhJc=; b=vjkgvJwK9s2qCw53kv0kl1McYliy2KhI8xetEj3ELcaF/g2RJrxW0DFx2cpUE3jfaw rC2Xmuu2cW3f/SxUMjgRCVdApvuvxdRdhnBhPGHbz0LoSXQsVqGvVdvksD5zC/7KhaF4 CX7fD51TeTvRn4YrAjbcpEw68Ghb9VhehkpmuUNJfx/LrJ17hZPi2pc8UZRf5u85WJ+D vEXQTf6wO1IxW6knO/7iS0GpjHWW2T9Uszfhbv/9awAbrGQAKJozSJCwFqELCwXWKFf8 5g06kghFXap8gQbFgtuiZOtg/hsGCUPv1LoL56QYVzhFQlWanaVQX/wbUQi6kiR5kS2w vrEA== X-Gm-Message-State: AOJu0YzZdhLm5cy1ci6a9T0XR8dsqSFr/yIFsEjrBK4iaGDoog8GPB/j 5wveed8AkzfJoaMI8tLUy7o40Bkjon0726GT0rdt8ogM9YvggWfuwzPWvb1CwNlW6MO9OLO6TYj pB1aMv4ScXshWGcOBb5C/iT1utk7FgMv2844EITg4IWf0nEzcHMoKhJh+bIzEOQRUFyAv4tOR29 y90nNOlNGIirNXcRXQv7D6BEXeq+lcTw== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:c702:0:b0:dcc:9f24:692b with SMTP id w2-20020a25c702000000b00dcc9f24692bmr44649ybe.13.1707828135418; Tue, 13 Feb 2024 04:42:15 -0800 (PST) Date: Tue, 13 Feb 2024 13:41:50 +0100 In-Reply-To: <20240213124143.1484862-13-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240213124143.1484862-13-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=3664; i=ardb@kernel.org; h=from:subject; bh=CIHrtmfTIHf1nWX5y4HvnUdj+zcQXCO5URm4j2f8Yss=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfV0cl/Gzc7Pj0827Z2x7tTrdlcrJTWWepkfL7+1Tvbfv L0+de/ajlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRZENGhp1XJhx+OvtJ0OQd t66/4Tg4Z981saPGcaoTv3fmC4oZKr1gZNjyhGdrtnH11bPX511KKZxqfenAYamIjWsevWM+0XN NcyYnAA== X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240213124143.1484862-19-ardb+git@google.com> Subject: [PATCH v4 06/11] x86/startup_64: Simplify virtual switch on primary boot From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790787721226915954 X-GMAIL-MSGID: 1790787721226915954 From: Ard Biesheuvel The secondary startup code is used on the primary boot path as well, but in this case, the initial part runs from a 1:1 mapping, until an explicit cross-jump is made to the kernel virtual mapping of the same code. On the secondary boot path, this jump is pointless as the code already executes from the mapping targeted by the jump. So combine this cross-jump with the jump from startup_64() into the common boot path. This simplifies the execution flow, and clearly separates code that runs from a 1:1 mapping from code that runs from the kernel virtual mapping. Note that this requires a page table switch, so hoist the CR3 assignment into startup_64() as well. And since absolute symbol references will no longer be permitted in .head.text once we enable the associated build time checks, a RIP-relative memory operand is used in the JMP instruction, referring to an absolute constant in the .init.rodata section. Given that the secondary startup code does not require a special placement inside the executable, move it to the .noinstr.text section. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head_64.S | 42 ++++++++++---------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 6dcc2f7f4108..3fed0aafcb41 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -39,7 +39,6 @@ L4_START_KERNEL = l4_index(__START_KERNEL_map) L3_START_KERNEL = pud_index(__START_KERNEL_map) - .text __HEAD .code64 SYM_CODE_START_NOALIGN(startup_64) @@ -126,9 +125,21 @@ SYM_CODE_START_NOALIGN(startup_64) call sev_verify_cbit #endif - jmp 1f + /* + * Switch to early_top_pgt which still has the identity mappings + * present. + */ + movq %rax, %cr3 + + /* Branch to the common startup code at its kernel virtual address */ + ANNOTATE_RETPOLINE_SAFE + jmp *0f(%rip) SYM_CODE_END(startup_64) + __INITRODATA +0: .quad common_startup_64 + + .section .noinstr.text, "ax" SYM_CODE_START(secondary_startup_64) UNWIND_HINT_END_OF_STACK ANNOTATE_NOENDBR @@ -174,8 +185,15 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) #ifdef CONFIG_AMD_MEM_ENCRYPT addq sme_me_mask(%rip), %rax #endif + /* + * Switch to the init_top_pgt here, away from the trampoline_pgd and + * unmap the identity mapped ranges. + */ + movq %rax, %cr3 -1: +SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) + UNWIND_HINT_END_OF_STACK + ANNOTATE_NOENDBR /* Create a mask of CR4 bits to preserve */ movl $(X86_CR4_PAE | X86_CR4_LA57), %edx @@ -194,16 +212,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) btsl $X86_CR4_PSE_BIT, %ecx movq %rcx, %cr4 - /* - * Switch to new page-table - * - * For the boot CPU this switches to early_top_pgt which still has the - * identity mappings present. The secondary CPUs will switch to the - * init_top_pgt here, away from the trampoline_pgd and unmap the - * identity mapped ranges. - */ - movq %rax, %cr3 - /* * Do a global TLB flush after the CR3 switch to make sure the TLB * entries from the identity mapping are flushed. @@ -211,14 +219,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) btsl $X86_CR4_PGE_BIT, %ecx movq %rcx, %cr4 - /* Ensure I am executing from virtual addresses */ - movq $1f, %rax - ANNOTATE_RETPOLINE_SAFE - jmp *%rax -1: - UNWIND_HINT_END_OF_STACK - ANNOTATE_NOENDBR // above - #ifdef CONFIG_SMP /* * For parallel boot, the APIC ID is read from the APIC, and then From patchwork Tue Feb 13 12:41:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 200418 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp507689dyb; Tue, 13 Feb 2024 04:44:38 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCW1dji8ZhNwlGnHpZsULfwxaW0wEBYL/zjs3hQqrvnQuqATQzuOUL3zAX0udTXZbQvQDJeNKz9sAFHcn20APRH48qeY1Q== X-Google-Smtp-Source: AGHT+IFc3g24UiTN+pUsv+EBi199OJhO9d5JUKZeeFwJRk/Prvg5r0VTuWomvUCN/WUkXMQbqwFc X-Received: by 2002:aa7:c890:0:b0:560:5c58:dfc4 with SMTP id p16-20020aa7c890000000b005605c58dfc4mr6849038eds.1.1707828278236; Tue, 13 Feb 2024 04:44:38 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707828278; cv=pass; d=google.com; s=arc-20160816; b=wbd4HXDF6osu2P8d4AcIHqzQyIx6IA2EX5HqgWDc4v98sNZjXQA9uBeIy4B5+RRYg+ 27gfkVckqWx+AginlYOqRofXRfb4BXo14xMbAAI3FKunhnsDoN9lyDDwLptKOdvy2lVQ 7GP0acqyzIEQSkXkdKJV7zIljpVL1vZRz0zLbWWeZmEY9AzlFDjHqHR5PGsMU0Ug+EAW KtzekOb2gJC0RL9KwXd+YUJKPfWfK+0nhgIZGN2NIP+bqCEjecL/hMDqljKA5kCxl5cX CzbZwFTueoACaj06cIF3A0pd/8HwA7oGuf6QxcmxL/7yb8IeGxNRbiZiXau1CWaxNiWB ynoQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=tmJq7yGJtig/x243U4iqbWlHcuxPy6IsUUTLst4tAO8=; fh=QW7hIY1X2EJnS71asFIGzt0qltcBJKclWfhAKj8y0M4=; b=R1HEvFCMcv5F8wyZRoVI6Xh2/BimzUlldgA4juBVBwzP8ox4YCnPwYh7WVmmTO2xzq ix0ZijqHidym0LVZPmh8XZR3U6wwvYnZqJS+l9QdCaNibfeY6fmFyCtbls+gpgqn6UAV yyQLRxwbpnY6CJBDuWbOwV8uCSrf6IzbgKxAiDieSIYvO9ZvX2o2SG8wXmz7KcjSMwNR vXRSfz0nJAagLdJbuzXd/ZgWu2f5AZa5Funepo7BbqzuXFZzIXplqza+q4i+TDQeHciR If9124+M1QN9RD4ALY6I9TA12/9RSDEiHZZUvoQBMPVCzOorkYCw6s+1VeS/Q5+IQhD1 jEPw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=OBtFeaYy; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63528-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63528-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com X-Forwarded-Encrypted: i=2; AJvYcCWtOICW17RGgomJp93ZBFIm26xriiORVQEg4DD0bTZjA05UnlU1v2TfdUEO/LYxMTNKnn7qHYKuzHzJ0Hn6Glgfk5p4iA== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id e5-20020a50fb85000000b0055f43795b28si3807911edq.196.2024.02.13.04.44.38 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Feb 2024 04:44:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-63528-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=OBtFeaYy; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63528-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63528-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id A3C371F26292 for ; Tue, 13 Feb 2024 12:44:37 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E3BCB5465B; Tue, 13 Feb 2024 12:42:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="OBtFeaYy" Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D94253395 for ; Tue, 13 Feb 2024 12:42:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828140; cv=none; b=Ctysf7dIAVoHcsBNaslDqlT0BImhtuLp4Wkq7adQD0Wrpr82Ts0N7H0bxGqQp4MaluLvwZxazqaSQA3HDMb6iHzqMVlFhgcc5wKIFNsJ4IvppfAqbrGnPr3/svoCs9ys6JGh/YoKjvG+a1NhvNPCEZ3I6DC3M/uYm349GW31r8o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828140; c=relaxed/simple; bh=CEh5RLJcC6IgrEQHXZSi19euYte1lfU4DaXYg20d8/k=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=cx3hk2vO2qifpWbfKFsiuZwB2L8LPqCWpwWeZCCgMIOxt5I4G1w2YdFIbC8ptbcbIxQVUA4Ahhhhf7kYFFtL4a4c6PWCO4ank2GQrOjJNTS475pF/xwgzNv/JLnH/C8NgPezQyXuGEzERgKuo2rmbWdKYd0iFL6emgnxIl1dMfQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=OBtFeaYy; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-33ce09c0644so192889f8f.1 for ; Tue, 13 Feb 2024 04:42:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707828138; x=1708432938; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tmJq7yGJtig/x243U4iqbWlHcuxPy6IsUUTLst4tAO8=; b=OBtFeaYyc/GR1Y7iEWhFsoTNTYn1fgVS6z3vXpcwP3xc0GVxIxWjLg5dOiihfO/mrJ S0o0i5Q/XHlFJlILlr2PGigl6d6yDZfdZkzhvZtk+HDbwgXoRxz0XS+PdUw2AXFKIBOM cTsqdOHhIrJKAwotz3j8ROlR8IHiThkG9xNGJYog3pcXA6tAZc8IpSraGhmUTD/cTDjL l/qG83RCgB+wC7sMP1IoYQ2qNA3vAyX3IfqJ+HnCgfPwlVEHkaAEL7ICENDHtvMW1hY1 2SExgSE9jcZYLX6iCuArzXWvLnJjgoHFP/RcQ4p+BjEnYx4RVrRcCFAvDmwLOefY6B4v Jrig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707828138; x=1708432938; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tmJq7yGJtig/x243U4iqbWlHcuxPy6IsUUTLst4tAO8=; b=gkdwZlXnMepM12T83wdGXZWS+hSoBVMZp+HjS/AMmRMUfnuU3CIx63Tw8iMHEUgBzS 2LCxcu0Oah8ZQAz13njZj44Idbn/GS72jRwYRfWc6tyZnWYvvnfSUUK9UqdWMjpMq4nH vIYSZ7y3gLY4qIf9G/yHOBY9UYoIEDPvhKDyASgMcDITPVYBc+bnloRRMyZs5GlcGLTs Q1WOW2hQ329AW3G1EJev1AR0XbOsKeb1sZ1WtImxHkEYXLfIwYuDqSy8FJYJQ/1MKQZ6 aXzOgj/bCrWi10QnaylUq00aB4QC3bE2oYGpUXEQ8Jb9iS/9wSVlfZdUgV5eDTwBEOma B2Tw== X-Gm-Message-State: AOJu0YyF+2RvpeAJANv0TWUkyoiUpON2HocLOjx4Ke6wGNimnAReL54r aiICNV7KoNlayUyket9Kak1ZlmYQZcc0Bqt0ogdIjF3i68aaGQZoMAx1Mvw1Q6G4eZimpRr8Alf 6fzrtQe78OMZldnbZ/UvH88TS5rZTK9ZrfT50CKAyB+8Qj7k6Jv2XWvwPr555a71TntbTTHKMqc Lq87v5qUieRf7LNlpZh9t6moDMPaFZFQ== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:1e0c:b0:33b:4a51:f0ed with SMTP id bj12-20020a0560001e0c00b0033b4a51f0edmr18827wrb.4.1707828137638; Tue, 13 Feb 2024 04:42:17 -0800 (PST) Date: Tue, 13 Feb 2024 13:41:51 +0100 In-Reply-To: <20240213124143.1484862-13-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240213124143.1484862-13-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=2179; i=ardb@kernel.org; h=from:subject; bh=SH5BCZpD7eT9rYWqZ8AiKPxoLpu9BlwPcI3lpvrnWyw=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfV0cn/7uaRP4tGTt8a01H99Oak74PEDlrXnFvfZ+AsXH pF9buLQUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACbycw7Df++Drjf+P264pyab 4jnl+emnm7K3nNr8cPvT223G2e5TkmYxMhw42hws/q71rUN8xDQV3cq5AQunJUxU/O+a+Fn8zG3 +aFYA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240213124143.1484862-20-ardb+git@google.com> Subject: [PATCH v4 07/11] efi/libstub: Add generic support for parsing mem_encrypt= From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790787744694062087 X-GMAIL-MSGID: 1790787744694062087 From: Ard Biesheuvel Parse the mem_encrypt= command line parameter from the EFI stub if CONFIG_ARCH_HAS_MEM_ENCRYPT=y, so that it can be passed to the early boot code by the arch code in the stub. This avoids the need for the core kernel to do any string parsing very early in the boot. Signed-off-by: Ard Biesheuvel --- drivers/firmware/efi/libstub/efi-stub-helper.c | 8 ++++++++ drivers/firmware/efi/libstub/efistub.h | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/firmware/efi/libstub/efi-stub-helper.c b/drivers/firmware/efi/libstub/efi-stub-helper.c index bfa30625f5d0..3dc2f9aaf08d 100644 --- a/drivers/firmware/efi/libstub/efi-stub-helper.c +++ b/drivers/firmware/efi/libstub/efi-stub-helper.c @@ -24,6 +24,8 @@ static bool efi_noinitrd; static bool efi_nosoftreserve; static bool efi_disable_pci_dma = IS_ENABLED(CONFIG_EFI_DISABLE_PCI_DMA); +int efi_mem_encrypt; + bool __pure __efi_soft_reserve_enabled(void) { return !efi_nosoftreserve; @@ -75,6 +77,12 @@ efi_status_t efi_parse_options(char const *cmdline) efi_noinitrd = true; } else if (IS_ENABLED(CONFIG_X86_64) && !strcmp(param, "no5lvl")) { efi_no5lvl = true; + } else if (IS_ENABLED(CONFIG_ARCH_HAS_MEM_ENCRYPT) && + !strcmp(param, "mem_encrypt") && val) { + if (parse_option_str(val, "on")) + efi_mem_encrypt = 1; + else if (parse_option_str(val, "off")) + efi_mem_encrypt = -1; } else if (!strcmp(param, "efi") && val) { efi_nochunk = parse_option_str(val, "nochunk"); efi_novamap |= parse_option_str(val, "novamap"); diff --git a/drivers/firmware/efi/libstub/efistub.h b/drivers/firmware/efi/libstub/efistub.h index 212687c30d79..a1c6ab24cd99 100644 --- a/drivers/firmware/efi/libstub/efistub.h +++ b/drivers/firmware/efi/libstub/efistub.h @@ -37,8 +37,8 @@ extern bool efi_no5lvl; extern bool efi_nochunk; extern bool efi_nokaslr; extern int efi_loglevel; +extern int efi_mem_encrypt; extern bool efi_novamap; - extern const efi_system_table_t *efi_system_table; typedef union efi_dxe_services_table efi_dxe_services_table_t; From patchwork Tue Feb 13 12:41:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 200420 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp508543dyb; Tue, 13 Feb 2024 04:46:12 -0800 (PST) X-Google-Smtp-Source: AGHT+IECula5I80morrLLcoRHFi1XB6vgmpKPKyrhD6NitwIY6iTDr2VILP4O2a+RoDPfPelvYeM X-Received: by 2002:a05:6359:4c1a:b0:17a:e6f8:3253 with SMTP id kj26-20020a0563594c1a00b0017ae6f83253mr3620171rwc.26.1707828372640; Tue, 13 Feb 2024 04:46:12 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707828372; cv=pass; d=google.com; s=arc-20160816; b=MtzGmDvRQVCIfiaDtmTRIyL5wV/R5XUfXtlInEyPn7gO1u9CC13Z6gGEOELp0r1orQ EHBQZhtRmxGwm3FrZo63GOazyKuuSBXQPXhaNcV03iJJJO0Ibi6Z2mLqQ8EegTZKdKI6 zEG0YUyqYDqPbH23m2e4ObdvImotLKznOyAb+WRzpN89xDCO3f6IH9kofsqWz4f5Lx/u JjxzUgUTHqqG8NsurPAjaBKiTCvAMrFnCUw6gB9SDdZaCeDf0ooJ3CR1Z1AYyjBLyAT1 SusheYxYCRbnfh5sgAh+XHh4SrV54DRFUjn3inHA40zaNAktgRw6xRkuB7cVWMXCcGfG B0vg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=ll+Sw/hEe0JqYepYgLfYwl4wc3XtSlKf28OWCE7h1Rg=; fh=l3vG1Bhnmk84KbfuKqxMQaRTKzTxO19j1K9yhl3hK68=; b=txyBDvnLgX/a/q8bIYRPzyx8inV92cfGOedW9AjIEEGFcfvK3f6aZG75LIH8WpCWow NmaNz/TxDr7Bi43fNrSOPyEvsh63ZiG302/biXqZuG0dZOGWjhDqIoebaFNkhzapSdls CKnLseQpQycflksm2/PEMUjJuywkA3U9etLVDIAJ7AYw8vVj+XOriYJ9x3+yteD9oD6x M/yDPjaA3BeLE9EUgSLZmJ3eVltRv4lfmWXdAgb/n1KolX9N/HQsCAiHzJtBOccjt3pd /GGOyMKUcC3bJ7bd68SeY5+GL8JhUmbPAXFPE7b8jlf3BoGGI9QIJAzxGq/GTZjSmiGc X+Xw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=mtAkottt; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63530-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63530-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com X-Forwarded-Encrypted: i=2; AJvYcCUiXrI/Sy7D1sSoFGAVSWcgNJE79PBJER4iPMRo5L5IQCwaWE73OspAjdHaCkSfJvmNRewUKwhdeJvFOZNawhWDafR7oQ== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id c16-20020a05622a025000b0042c4b28efd3si3084167qtx.520.2024.02.13.04.46.12 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Feb 2024 04:46:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-63530-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=mtAkottt; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63530-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63530-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 9F0BE1C224B8 for ; Tue, 13 Feb 2024 12:45:07 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1B61B54BF6; Tue, 13 Feb 2024 12:42:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="mtAkottt" Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D70B53E27 for ; Tue, 13 Feb 2024 12:42:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828145; cv=none; b=Y73ztLdo52YiB42MbpiOJ94UBkJLol0fo2SBLm5PazfkXCSBNN3KuWYAEwZ/pl2mNZfRgshSEn2yCkM4EERPIgcuGS9GqP6Ppfdqsap58K5vDNDwXykkqiqjPPSOaeLDB1YXqAgR5j+cFn2O2n3r9/mk+QqTC3d7guat7CMHEi8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828145; c=relaxed/simple; bh=HgCoBP1Ar8xIzQYCsjFkSLzQsHIbfD1/fXOVPSZuf2E=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=T7jYcDZY9wFijEOpMYNtJafoMMrvwa6cgk46Asym/OnnOU2JNyH2DCM/jzxpG8Avfib2lFhhxcJqtkhWg3m1O+aDbtfKfyyknpV9ETM0K/AsstALQtIXFc0IOEjam4AKeVJZQ51b0dhA2oX6+X1vOXa/QlcY0PXBsAiqPYOiCsQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=mtAkottt; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dcbee93a3e1so1792626276.3 for ; Tue, 13 Feb 2024 04:42:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707828142; x=1708432942; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ll+Sw/hEe0JqYepYgLfYwl4wc3XtSlKf28OWCE7h1Rg=; b=mtAkotttdHWN9Hum8bT+dAmEG6P25a6eT+07DCKNkvJYvvxVT1XfOl+kxIyNeHlx35 nwX3ZG/EMaxb6KOdFfzmxgXOhYo13AK5M4A3W1lGsHwcS8aYK+l8MewVgEta/aLSe9n0 p1TLnZuHhKj12IkHz1sKx4TTFwhuZakSgKPs0pxm2O/5imHTfoePRDNj0xbnun3e7sIp eJCrvNluNJiVb9cDzuL0VBTURGDGnrDdzjVRAvaK0hJm6Xl7ro2qxbf0DdBxM8jCPQkY 4AbUKHzjGqOfJR2HkscY2FodFHfXkMwnF/xIRN3SZoL88Yw1YZCqHYjGqYn5fy8lVfhH UXHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707828142; x=1708432942; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ll+Sw/hEe0JqYepYgLfYwl4wc3XtSlKf28OWCE7h1Rg=; b=LZuErG63Fm+QrgHCbs9mzRoRBc7bEAyllNKM83NQgeecyIow8UQoMhKEN6E32QIIU9 AyXPiMYzi4Ke+dEqG5jQocmRiYvaPsU+U2oSPpwG6jlvejjLJyNQ8goXlc1Hqxaha9tU c2t3tfHVs/ywPHR539wWKzZhzUzQed8+AMhlKZQ4flPq211C4W030sizRhbdyBhtkHQf rgVS3kOJKEJ8nStHJpnMbFuSFKC4Pr/LzLJwB7G6FucJar/b6YJR3jDFLaNcpiU9s3sj 3EnV413d2zRdXodxOhJWBgGHHAIRNj9rYCzfoBYqfA6jGmKLDu3xfA3IB0/abpGcLsZ6 +mdQ== X-Gm-Message-State: AOJu0YxnvDe1vV+Xr3iZFmzUCgwVRQwgSFg7Wdr/xuuGhP+YjOgZrlKd iyPtjAh4MMF9K+zKZPh8A2LNR1kOQp7T00YlPf7ewb240JvOKBP77GsB/3VUQwyGEuR7EQVF7vC FVLXFb0ieVwk06aOFzqaNcdmPGV2bdHl2VVwhcWw/kIj1JJQEluaEO7WlC3Jkrte7KWoxJhVE7Y 3QfCPtelmIgzSXj1Ef7+Z35mUR/QV5pg== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:10c1:b0:dc6:e823:9edb with SMTP id w1-20020a05690210c100b00dc6e8239edbmr268852ybu.12.1707828142047; Tue, 13 Feb 2024 04:42:22 -0800 (PST) Date: Tue, 13 Feb 2024 13:41:53 +0100 In-Reply-To: <20240213124143.1484862-13-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240213124143.1484862-13-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=7964; i=ardb@kernel.org; h=from:subject; bh=FdrHvH/vioxw/rDTZ5Ktwi3pb2GEozrF10ixhskLdq4=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfV08qQlq84+38H7dulr9Y5axXtblTMs4kOyV/Me0FmQL c2aEizQUcrCIMbBICumyCIw+++7nacnStU6z5KFmcPKBDKEgYtTACaypJiR4bLGz8mWNsr2K6b8 y97wV/jQE73OnAkvtrH29+vUd0uwf2FkWL9q/8pj7XGV4uzbmf9sefyCfc23XbrT1uu1b1aR+/j HiBMA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240213124143.1484862-22-ardb+git@google.com> Subject: [PATCH v4 09/11] x86/sme: Move early SME kernel encryption handling into .head.text From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790787843860132687 X-GMAIL-MSGID: 1790787843860132687 From: Ard Biesheuvel The .head.text section is the initial primary entrypoint of the core kernel, and is entered with the CPU executing from a 1:1 mapping of memory. Such code must never access global variables using absolute references, as these are based on the kernel virtual mapping which is not active yet at this point. Given that the SME startup code is also called from this early execution context, move it into .head.text as well. This will allow more thorough build time checks in the future to ensure that early startup code only uses RIP-relative references to global variables. Also replace some occurrences of __pa_symbol() [which relies on the compiler generating an absolute reference, which is not guaranteed] and an open coded RIP-relative access with RIP_REL_REF(). Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/mem_encrypt.h | 8 ++-- arch/x86/mm/mem_encrypt_identity.c | 42 ++++++++------------ 2 files changed, 21 insertions(+), 29 deletions(-) diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index b31eb9fd5954..f922b682b9b4 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -47,8 +47,8 @@ void __init sme_unmap_bootdata(char *real_mode_data); void __init sme_early_init(void); -void __init sme_encrypt_kernel(struct boot_params *bp); -void __init sme_enable(struct boot_params *bp); +void sme_encrypt_kernel(struct boot_params *bp); +void sme_enable(struct boot_params *bp); int __init early_set_memory_decrypted(unsigned long vaddr, unsigned long size); int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long size); @@ -81,8 +81,8 @@ static inline void __init sme_unmap_bootdata(char *real_mode_data) { } static inline void __init sme_early_init(void) { } -static inline void __init sme_encrypt_kernel(struct boot_params *bp) { } -static inline void __init sme_enable(struct boot_params *bp) { } +static inline void sme_encrypt_kernel(struct boot_params *bp) { } +static inline void sme_enable(struct boot_params *bp) { } static inline void sev_es_init_vc_handling(void) { } diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index d210c7fc8fa2..64b5005d49e5 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -41,6 +41,7 @@ #include #include +#include #include #include #include @@ -94,7 +95,7 @@ struct sme_populate_pgd_data { */ static char sme_workarea[2 * PMD_SIZE] __section(".init.scratch"); -static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) +static void __head sme_clear_pgd(struct sme_populate_pgd_data *ppd) { unsigned long pgd_start, pgd_end, pgd_size; pgd_t *pgd_p; @@ -109,7 +110,7 @@ static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) memset(pgd_p, 0, pgd_size); } -static pud_t __init *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) +static pud_t __head *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) { pgd_t *pgd; p4d_t *p4d; @@ -146,7 +147,7 @@ static pud_t __init *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) return pud; } -static void __init sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) +static void __head sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) { pud_t *pud; pmd_t *pmd; @@ -162,7 +163,7 @@ static void __init sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) set_pmd(pmd, __pmd(ppd->paddr | ppd->pmd_flags)); } -static void __init sme_populate_pgd(struct sme_populate_pgd_data *ppd) +static void __head sme_populate_pgd(struct sme_populate_pgd_data *ppd) { pud_t *pud; pmd_t *pmd; @@ -188,7 +189,7 @@ static void __init sme_populate_pgd(struct sme_populate_pgd_data *ppd) set_pte(pte, __pte(ppd->paddr | ppd->pte_flags)); } -static void __init __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) +static void __head __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) { while (ppd->vaddr < ppd->vaddr_end) { sme_populate_pgd_large(ppd); @@ -198,7 +199,7 @@ static void __init __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) } } -static void __init __sme_map_range_pte(struct sme_populate_pgd_data *ppd) +static void __head __sme_map_range_pte(struct sme_populate_pgd_data *ppd) { while (ppd->vaddr < ppd->vaddr_end) { sme_populate_pgd(ppd); @@ -208,7 +209,7 @@ static void __init __sme_map_range_pte(struct sme_populate_pgd_data *ppd) } } -static void __init __sme_map_range(struct sme_populate_pgd_data *ppd, +static void __head __sme_map_range(struct sme_populate_pgd_data *ppd, pmdval_t pmd_flags, pteval_t pte_flags) { unsigned long vaddr_end; @@ -232,22 +233,22 @@ static void __init __sme_map_range(struct sme_populate_pgd_data *ppd, __sme_map_range_pte(ppd); } -static void __init sme_map_range_encrypted(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_encrypted(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_ENC, PTE_FLAGS_ENC); } -static void __init sme_map_range_decrypted(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_decrypted(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_DEC, PTE_FLAGS_DEC); } -static void __init sme_map_range_decrypted_wp(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_decrypted_wp(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_DEC_WP, PTE_FLAGS_DEC_WP); } -static unsigned long __init sme_pgtable_calc(unsigned long len) +static unsigned long __head sme_pgtable_calc(unsigned long len) { unsigned long entries = 0, tables = 0; @@ -284,7 +285,7 @@ static unsigned long __init sme_pgtable_calc(unsigned long len) return entries + tables; } -void __init sme_encrypt_kernel(struct boot_params *bp) +void __head sme_encrypt_kernel(struct boot_params *bp) { unsigned long workarea_start, workarea_end, workarea_len; unsigned long execute_start, execute_end, execute_len; @@ -319,9 +320,8 @@ void __init sme_encrypt_kernel(struct boot_params *bp) * memory from being cached. */ - /* Physical addresses gives us the identity mapped virtual addresses */ - kernel_start = __pa_symbol(_text); - kernel_end = ALIGN(__pa_symbol(_end), PMD_SIZE); + kernel_start = (unsigned long)RIP_REL_REF(_text); + kernel_end = ALIGN((unsigned long)RIP_REL_REF(_end), PMD_SIZE); kernel_len = kernel_end - kernel_start; initrd_start = 0; @@ -338,14 +338,6 @@ void __init sme_encrypt_kernel(struct boot_params *bp) } #endif - /* - * We're running identity mapped, so we must obtain the address to the - * SME encryption workarea using rip-relative addressing. - */ - asm ("lea sme_workarea(%%rip), %0" - : "=r" (workarea_start) - : "p" (sme_workarea)); - /* * Calculate required number of workarea bytes needed: * executable encryption area size: @@ -355,7 +347,7 @@ void __init sme_encrypt_kernel(struct boot_params *bp) * pagetable structures for the encryption of the kernel * pagetable structures for workarea (in case not currently mapped) */ - execute_start = workarea_start; + execute_start = workarea_start = (unsigned long)RIP_REL_REF(sme_workarea); execute_end = execute_start + (PAGE_SIZE * 2) + PMD_SIZE; execute_len = execute_end - execute_start; @@ -498,7 +490,7 @@ void __init sme_encrypt_kernel(struct boot_params *bp) native_write_cr3(__native_read_cr3()); } -void __init sme_enable(struct boot_params *bp) +void __head sme_enable(struct boot_params *bp) { unsigned int eax, ebx, ecx, edx; unsigned long feature_mask; From patchwork Tue Feb 13 12:41:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 200419 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp508513dyb; Tue, 13 Feb 2024 04:46:10 -0800 (PST) X-Google-Smtp-Source: AGHT+IGqKEi+4TmeVR9F9f3rldoikLB6AkT1hBGPmD62QMRIsfnXCbwXAi9Tu+cjOzZYPfncdNJm X-Received: by 2002:a05:6a20:94c3:b0:19e:a30e:7333 with SMTP id ht3-20020a056a2094c300b0019ea30e7333mr8612216pzb.25.1707828369933; Tue, 13 Feb 2024 04:46:09 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707828369; cv=pass; d=google.com; s=arc-20160816; b=IU6Zi6QOQk7RpAaQfHewvzExMP/H+W5dlMYfpYCgOCB/zgDUx0qZccUYfmY8PeNUJm tyVIrCDy98ekr3oSoy+hb2flttRnUFeL9VeD1Jfm/0rrtNC2CrXwq2TGz6z4nAhT4QZ0 +vpqH9NJOEgFdsHxn0yvgbBGcECiY40j24im3bHDOyi9U2bXTHc09I3YI9jHDE2bCtsQ Z6JmSMekD6L2Bc7wWU1OaQdu7Vzbg2alfkvjH/POuGk0RA0C+MBdfGsU9ESL1awjrHA0 2A2slr53UEt3UAt7Dty4OgRoHWAIfrdTAHlKC+ei/oQCmqSkeEB7PCJz2HOT0ZQGbrIC w2LA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=vDwWzx+ZIZln/anz+aVU2C0rhpdbzTDoyUXdax0Fzpk=; fh=hRq1KNJ0+z3LZ6tMV/YsjPkWILYnjUZDXBX1QIRuTnc=; b=CLf98brdMgoXt07VpOvin5HGbzxiPpkZ0OiEnITeY3PxNR5oH0Tj9/HKxfvqBGCiqe /WZTa8l8OjKuYd4skUBM0nmjUiNhohs5ZIPqatZ0YGXHMySNUsnJQuq/xUQe056mM9Zd TpKI3OY3+JB9bNlH9MiEnfsyQiO73rGsdwGTfDQvkNUjht2OcgDLWMH5fLfLI3eZq0ZW RHL/ZcjPBq/PWSW7ribVU11T3vePos3DEm4lgcLlwt9JevxBej+pRdm50BfqyfAh0+80 qqEE4ejUADBnlzfUQsvN0BVuGr2V3IPTqj9bK2Yc9yJCKM34mVUiM56KIYJXXiEjyZVX itzQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=OI+Sua7s; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63531-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63531-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com X-Forwarded-Encrypted: i=2; AJvYcCVVY8q8oiAcDSFPIaNqRvlxtNMRN2Paixq6qWeaHwW5nBq+H5jVtiDhzuP7wyUJzDbucKAH2nbCZ3a8deMFJVAZ+hxf0Q== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id 71-20020a63014a000000b005dc833ef4e5si1263407pgb.64.2024.02.13.04.46.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Feb 2024 04:46:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-63531-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=OI+Sua7s; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63531-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63531-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id D3127280847 for ; Tue, 13 Feb 2024 12:45:30 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D6E1D55775; Tue, 13 Feb 2024 12:42:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="OI+Sua7s" Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 58E9C54BD7 for ; Tue, 13 Feb 2024 12:42:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828149; cv=none; b=nhQ8O7UvKHSm8IXGaR1Gv9sVb9SZpnDosb/U3O1CxlKxB/9iCi56AODnSP80DeCGjrb/J0Q9u5kmYNEV8oiTMH/qbh9D2KwivSwrh1fyYiQJ44O70bOKnXRGs72IW5Se/o/nFhSNek1eFheb5EllQ3ZIqOfh9DyAETiEGVigLac= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828149; c=relaxed/simple; bh=CKqAkVncBI5L3gG4K3ZJpuo/XUJZHNhpt3gf71HxdC0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=JyCCg+AEN04CXrycLSPI51x35D9/hiABaLlZnA/qma4I1Ntc34QSmhcnUS0MuyHvs4ZuPv6t6bWCH6JAw3UfYEiJHHe5OYNsoYsGNCd9Ksw1Rk/4Ep+xsDi7FLm9LXoJgNrTFPg+njNs/p6Z9DPtjV9kUEBcCviksu0rVwV+M78= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=OI+Sua7s; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-4102b934ba0so24892045e9.2 for ; Tue, 13 Feb 2024 04:42:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707828144; x=1708432944; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=vDwWzx+ZIZln/anz+aVU2C0rhpdbzTDoyUXdax0Fzpk=; b=OI+Sua7sUU4PKsaALvZwj2d7Pgsbhjr0vEAw1fcZi6Ri16fqRDSrHNTYge2vXXccys lpqEhTLKSaAgJvDmFyaS1UEpMFvKf/+wvHY0Jc+hrYxjC3wnQw6PXJn5b5Umtfn3Cnhj 9CJiAbzOS+1E1xZGhCFjkh030z+n6gjc0WdNkqYthCIbwwuDFQOYQNLD+IoAEMmV7X4W FxOs8nzpZOeRO3ul6251ZnXRHfTPcGCPqNTIjZ1WAmF9WkIQVHCedGh8MxdGb3ZIS9IA kJmyN5ElWVvs9RLpf1NQ7v76PsPYsgsCKX+L5ORsp2rTKk5XdVYD6B2ETaVU1R7fm5aB Srqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707828144; x=1708432944; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vDwWzx+ZIZln/anz+aVU2C0rhpdbzTDoyUXdax0Fzpk=; b=EG7VlEKFCQ20B/O5Jyn+7aXCmJcXQBAd/K5gTycrt258O5gbQpH8+IwYb88swQrbzk ENbYsWhPjkNq80pAavt0rq4QY1weOdEjn+5QRRLrWSaAfKoi8dm6o9jS9LMeyDjpTM+j XTD1Pv4NatNPMznX0fMAW38uzqOTVYeKQ+BockEUiz99fbj1SutYFxmx6AdVJC96UKmL 0SKmfmKBOrpQjyOk8OxJ+AaAo3ZlmPxPU84FAnsQHJM6Q9gDq8RHrqZpyr1IbgOy7RFX TzotVXS0S7/IxOM6XgegUrOQylDfr+66M+PREfbHa4djCOUOhoRJR5KG0Soq8qIzd6CB klBA== X-Gm-Message-State: AOJu0Yyja9PW8XEvSeSoPdEM2d33s8cFXCnzH+lfNNxnx1ClhbOmYcyb zv5iNoT+GD9jxSjlqZZzWPn/3W4qpulYgM3rFfTaKaCR4/CZmevL+Fy9QNZ5pYagK86Y8okD9oZ p2fPQA1QsRSnrayAasW/jpb0eh2Ze+86c16tpxtdFrFfezIx8KqbTmtnLNUBVWFFcHS7RqoPjXc 9HsR7rjbyVdptIZdnn7kweHjUlr/S+VQ== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:1753:b0:33b:5e9f:c62e with SMTP id m19-20020a056000175300b0033b5e9fc62emr31121wrf.0.1707828144184; Tue, 13 Feb 2024 04:42:24 -0800 (PST) Date: Tue, 13 Feb 2024 13:41:54 +0100 In-Reply-To: <20240213124143.1484862-13-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240213124143.1484862-13-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=7378; i=ardb@kernel.org; h=from:subject; bh=2k7RKXQoIArwBAJHl4Eso59xaiH+DoTxxKoPT8FFXXc=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfV08mQmR4fd8z7E76yaGqsvWu+xtGCDSmZwRVt3ZkcSQ 23ygpSOUhYGMQ4GWTFFFoHZf9/tPD1RqtZ5lizMHFYmkCEMXJwCMJGXcxj+F1rbm+Ys9sk+liz3 Najh2WTd8973Vi88d7rsM/8CHi3No4wMfYv09jxksrsQ/uDhXafWzeYaO5tW/jDyaTO8tF5Akuc XOwA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240213124143.1484862-23-ardb+git@google.com> Subject: [PATCH v4 10/11] x86/sev: Move early startup code into .head.text section From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790787840293396365 X-GMAIL-MSGID: 1790787840293396365 From: Ard Biesheuvel In preparation for implementing rigorous build time checks to enforce that only code that can support it will be called from the early 1:1 mapping of memory, move SEV init code that is called in this manner to the .head.text section. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/sev.c | 3 +++ arch/x86/include/asm/sev.h | 10 ++++----- arch/x86/kernel/sev-shared.c | 23 +++++++++----------- arch/x86/kernel/sev.c | 14 +++++++----- 4 files changed, 26 insertions(+), 24 deletions(-) diff --git a/arch/x86/boot/compressed/sev.c b/arch/x86/boot/compressed/sev.c index 073291832f44..bea0719d70f2 100644 --- a/arch/x86/boot/compressed/sev.c +++ b/arch/x86/boot/compressed/sev.c @@ -116,6 +116,9 @@ static bool fault_in_kernel_space(unsigned long address) #undef __init #define __init +#undef __head +#define __head + #define __BOOT_COMPRESSED /* Basic instruction decoding support needed */ diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h index bed95e1f4d52..cf671138feef 100644 --- a/arch/x86/include/asm/sev.h +++ b/arch/x86/include/asm/sev.h @@ -213,16 +213,16 @@ static inline int pvalidate(unsigned long vaddr, bool rmp_psize, bool validate) struct snp_guest_request_ioctl; void setup_ghcb(void); -void __init early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, - unsigned long npages); -void __init early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr, - unsigned long npages); +void early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, + unsigned long npages); +void early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr, + unsigned long npages); void __init snp_prep_memory(unsigned long paddr, unsigned int sz, enum psc_op op); void snp_set_memory_shared(unsigned long vaddr, unsigned long npages); void snp_set_memory_private(unsigned long vaddr, unsigned long npages); void snp_set_wakeup_secondary_cpu(void); bool snp_init(struct boot_params *bp); -void __init __noreturn snp_abort(void); +void __noreturn snp_abort(void); int snp_issue_guest_request(u64 exit_code, struct snp_req_data *input, struct snp_guest_request_ioctl *rio); void snp_accept_memory(phys_addr_t start, phys_addr_t end); u64 snp_get_unsupported_features(u64 status); diff --git a/arch/x86/kernel/sev-shared.c b/arch/x86/kernel/sev-shared.c index ae79f9505298..0bd7ccbe8732 100644 --- a/arch/x86/kernel/sev-shared.c +++ b/arch/x86/kernel/sev-shared.c @@ -93,7 +93,8 @@ static bool __init sev_es_check_cpu_features(void) return true; } -static void __noreturn sev_es_terminate(unsigned int set, unsigned int reason) +static void __head __noreturn +sev_es_terminate(unsigned int set, unsigned int reason) { u64 val = GHCB_MSR_TERM_REQ; @@ -330,13 +331,7 @@ static int sev_cpuid_hv(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid */ static const struct snp_cpuid_table *snp_cpuid_get_table(void) { - void *ptr; - - asm ("lea cpuid_table_copy(%%rip), %0" - : "=r" (ptr) - : "p" (&cpuid_table_copy)); - - return ptr; + return &RIP_REL_REF(cpuid_table_copy); } /* @@ -395,7 +390,7 @@ static u32 snp_cpuid_calc_xsave_size(u64 xfeatures_en, bool compacted) return xsave_size; } -static bool +static bool __head snp_cpuid_get_validated_func(struct cpuid_leaf *leaf) { const struct snp_cpuid_table *cpuid_table = snp_cpuid_get_table(); @@ -532,7 +527,8 @@ static int snp_cpuid_postprocess(struct ghcb *ghcb, struct es_em_ctxt *ctxt, * Returns -EOPNOTSUPP if feature not enabled. Any other non-zero return value * should be treated as fatal by caller. */ -static int snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf) +static int __head +snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf) { const struct snp_cpuid_table *cpuid_table = snp_cpuid_get_table(); @@ -574,7 +570,7 @@ static int snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_le * page yet, so it only supports the MSR based communication with the * hypervisor and only the CPUID exit-code. */ -void __init do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code) +void __head do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code) { unsigned int subfn = lower_bits(regs->cx, 32); unsigned int fn = lower_bits(regs->ax, 32); @@ -1025,7 +1021,8 @@ struct cc_setup_data { * Search for a Confidential Computing blob passed in as a setup_data entry * via the Linux Boot Protocol. */ -static struct cc_blob_sev_info *find_cc_blob_setup_data(struct boot_params *bp) +static __head +struct cc_blob_sev_info *find_cc_blob_setup_data(struct boot_params *bp) { struct cc_setup_data *sd = NULL; struct setup_data *hdr; @@ -1052,7 +1049,7 @@ static struct cc_blob_sev_info *find_cc_blob_setup_data(struct boot_params *bp) * mapping needs to be updated in sync with all the changes to virtual memory * layout and related mapping facilities throughout the boot process. */ -static void __init setup_cpuid_table(const struct cc_blob_sev_info *cc_info) +static void __head setup_cpuid_table(const struct cc_blob_sev_info *cc_info) { const struct snp_cpuid_table *cpuid_table_fw, *cpuid_table; int i; diff --git a/arch/x86/kernel/sev.c b/arch/x86/kernel/sev.c index 1ef7ae806a01..33c14aa1f06c 100644 --- a/arch/x86/kernel/sev.c +++ b/arch/x86/kernel/sev.c @@ -25,6 +25,7 @@ #include #include +#include #include #include #include @@ -682,8 +683,9 @@ static u64 __init get_jump_table_addr(void) return ret; } -static void early_set_pages_state(unsigned long vaddr, unsigned long paddr, - unsigned long npages, enum psc_op op) +static void __head +early_set_pages_state(unsigned long vaddr, unsigned long paddr, + unsigned long npages, enum psc_op op) { unsigned long paddr_end; u64 val; @@ -739,7 +741,7 @@ static void early_set_pages_state(unsigned long vaddr, unsigned long paddr, sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_PSC); } -void __init early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, +void __head early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, unsigned long npages) { /* @@ -2062,7 +2064,7 @@ bool __init handle_vc_boot_ghcb(struct pt_regs *regs) * * Scan for the blob in that order. */ -static __init struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp) +static __head struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp) { struct cc_blob_sev_info *cc_info; @@ -2088,7 +2090,7 @@ static __init struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp) return cc_info; } -bool __init snp_init(struct boot_params *bp) +bool __head snp_init(struct boot_params *bp) { struct cc_blob_sev_info *cc_info; @@ -2110,7 +2112,7 @@ bool __init snp_init(struct boot_params *bp) return true; } -void __init __noreturn snp_abort(void) +void __head __noreturn snp_abort(void) { sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SNP_UNSUPPORTED); } From patchwork Tue Feb 13 12:41:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 200421 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp508565dyb; Tue, 13 Feb 2024 04:46:15 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWmAJzJIM8GcKV1zb1s35+0qvYb4eUOxv28VYP13RiETnstcPyNYRL6PEZHkLCn3RKOwBf8bue0r8XrRN7oecDACcmPRg== X-Google-Smtp-Source: AGHT+IHB2N7PC2rH1B6d4rQ6GW6gzCnj0Qgeg8FD8mZEUAGER4VK2yUrAmIr57jSB/WXrvdy4SkW X-Received: by 2002:a17:90b:3115:b0:296:6a37:c528 with SMTP id gc21-20020a17090b311500b002966a37c528mr5311650pjb.24.1707828375007; Tue, 13 Feb 2024 04:46:15 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707828374; cv=pass; d=google.com; s=arc-20160816; b=ciuR5dlJgli8fFPwGP/O0XB5NyZM6vst1y5lXSXmAbs9cBKNNONd8BTTjdcEiHRYRJ 78/llNHXtXk2vnU6RJXuQT5FqFEcruwqnd8MPD8Etaj99DbCIZJiN4aicQBU1pEIOiNs dx9dh5jAthzVbzRQ30O4Y7bIw2cvXYUPzHXu7LciylV/Xe/FSO7q5pFF7Dcj1oxJScNc yFyB4iMjOjTIexKqKzrogXbQ+dGrTDoS+aIdq3DdBSpLPqzHbBIsqqfKWfSk/Y2h1ZbI VQhl9JdSJiQd4XaTjId27rhoYhJva4MKp/sReYjcRZ+fAKUvAFsN4f5jVry0wDY/OnAi s8QQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=3vFuPudIhOhGYqZnhT5ckh2P37048hBhXa1jox3ZGCc=; fh=gri5vWgKwW/b0wEFecprAkdjnivBVpxP1uN66Y5BBq8=; b=h512q516dz3bNWfbmgnVShwTAl0fZ6cLsrOWxg75NxjkB6cpcWEdIK0M2ZeIVUptIo Dxmj9lu3kteKshTNV/7Nz3jN9G8ZXeRs5A9csZXbmPRudSbRAYc1m7mb5tuJmseSVfNh mnjZNZZN9/GuCXOSW8X+4Kt3OU5mvufX1ZVWJluFQEjricePBTlFd0+LVqUAit8HRn8u XB5LzzRROOL9oYUuetRBrrEpb+jPzTiEP80rCfTtaU4apWTp+3DPcdc9QMxzwGg4LFMg OQSMu0TzmZOv88wR+iR+ryALMrkNMJd9k4JlvTEIxjNwiWiyS6tsxjHYczalpqH3I3UJ ng1g==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=MRfgM7yY; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63532-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63532-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com X-Forwarded-Encrypted: i=2; AJvYcCXu7YblB7k5LQgcboa9sOzoQ3UygM7uX7opOlzRu9yEhiO/aiFuD1CgeguOcSU354UYcqxk0PxR/hDX8wKgioJlLCzRSw== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id nh2-20020a17090b364200b002971af6cd46si1989291pjb.133.2024.02.13.04.46.14 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Feb 2024 04:46:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-63532-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=MRfgM7yY; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-63532-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63532-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 8846E285890 for ; Tue, 13 Feb 2024 12:45:36 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D1F0F5578A; Tue, 13 Feb 2024 12:42:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MRfgM7yY" Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B73F654BDF for ; Tue, 13 Feb 2024 12:42:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828149; cv=none; b=Q4oRe2YkvwSMbBrXqMKJbWb/3D+d6XXYyn39/wnvu7qTe/4Wia5+614WkCiMKV1dhlPwbGwtbLi6ENcSoXYT7TBos+lzgc3yFqkM1ZwkLL1w6P74a2cRFq7VKugt6eis/erEvoLERBpvMGrRMk/te04c6h5tQ51F/8AjaEdgBtk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707828149; c=relaxed/simple; bh=rh+KrQSf0b7lqlGXVSURT3xNxNpwVXF3RneXnbnQKA4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=BXH+gpJYhYrDYzISLjkYBgHgSbADsVzMuSFIB6QK0sXqGKY/M1s++g/utyMNosdaMt0xiEPGVU2k92zzWvy/pS1SbpY4mJfh3KVxELn9suW9cUNQjF8JgUDoWjaO/6v1OBoyqhsby5OjgKpaKlAG4d3YAOSbUGNWQM9pOxo7oP8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=MRfgM7yY; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-410727c32bdso3978925e9.1 for ; Tue, 13 Feb 2024 04:42:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707828146; x=1708432946; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3vFuPudIhOhGYqZnhT5ckh2P37048hBhXa1jox3ZGCc=; b=MRfgM7yYvgtxGf3DgibcVC5NsI2HNCbGPI2g2khZXJ0IM+zG/9wimAh1kRhvhbiIvG oIZpv+XiALLQUtfOeKU8gJ9AN7A9DQlvvMr2VlBRn+yTMAcB8to945plKbqMbe2AUA3J YNb0t1BRxoQxHhL3axUKTv2LT746ykS7JFUZaKTrXZ/GVKwQS35ezF8GMVlnYr57Sr5u qJ2C4kLZ9uC9IgmjV3/l43XmPxFxK8qsu2RJhdFwy2nQfEQcuyUGGw1pVUuAQUhxQrC4 2cH3n790yXCVG26awdUrOVa2AbfLVY8ITgk7g8Ec8qCFoulnu/OHd92p2RCGby7yRWO+ 3Tug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707828146; x=1708432946; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3vFuPudIhOhGYqZnhT5ckh2P37048hBhXa1jox3ZGCc=; b=oj+GxjJw4r+PmQiPqKiqXUzcEhPt+JtAuKrUw4MQDyOrGccIA+qtCxeV3oH84GbRzn Ito22FpueK1N7qdHq985eAVissfkEfBcgXUlp6HzXJ16tDbc/P1C41UT+yipVnmIF6kF NvLY1EoSX5pQuMSxMBYWOlOURqGsy9R/r3xE1J/Bk7fHbAkHHyFUHi8CnV3AkxVj4Bmr 0MGAt0CcycdKYpteM5ZtC7NElknIion3bjZzRdpHY7/xN2Nt9L/pxw6ITaZ9G0KwJ88E OtY5Vy2GxQtmqGHlZnaAij1e10lHbRQJwtjHRFhOKASP8tPcm/PSlmj33iy/q+Zb222S wYWQ== X-Gm-Message-State: AOJu0YyPw+GpvF290rZcDMeKlQIUKw+VmH/8+KFxS8rKNoK0j2zi3W+a 0bf47knQVUyKEtp9+0r/dGKPucwarvHGSkJB4XJHdjV704Rnm8aVLkdqEGLIzUNAVR9YLJDhyh0 ZD3YMi/pJMUEw+csskcXIZhMo/JaFwNnOuWI849i9hq9q7FdMgr0ZrYBkX4i0Yv5Reo4BQjEfYd ZidoYI0DV6kjHPRSyYm2HmBHmJnLnLMw== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:c89:b0:411:d2c4:5998 with SMTP id fj9-20020a05600c0c8900b00411d2c45998mr6065wmb.6.1707828146355; Tue, 13 Feb 2024 04:42:26 -0800 (PST) Date: Tue, 13 Feb 2024 13:41:55 +0100 In-Reply-To: <20240213124143.1484862-13-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240213124143.1484862-13-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=10523; i=ardb@kernel.org; h=from:subject; bh=9YEkQOcO7b0gYqcHXm2fnYoXEzE3Uz6Y26otXMUNnOQ=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfV08pQ1PwyXtqyf8nCBSezmoPWz3d4/Xvl6870Z8Xvms ZTx/LYT6ShlYRDjYJAVU2QRmP333c7TE6VqnWfJwsxhZQIZwsDFKQAT+cbI8E9nja/NA3/bL3OU 8x09nJ6dTVs9MWjmLJtjZptZHvLVhIQw/DNfu1znrZ8CL0PBOs4zHy/aiUvUaV942/NpU4/+1Ry Lm9wA X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240213124143.1484862-24-ardb+git@google.com> Subject: [PATCH v4 11/11] x86/startup_64: Drop global variables keeping track of LA57 state From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Arnd Bergmann , Nathan Chancellor , Nick Desaulniers , Justin Stitt , Kees Cook , Brian Gerst , linux-arch@vger.kernel.org, llvm@lists.linux.dev X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790787845673276713 X-GMAIL-MSGID: 1790787845673276713 From: Ard Biesheuvel On x86_64, the core kernel is entered in long mode, which implies that paging is enabled. This means that the CR4.LA57 control bit is guaranteed to be in sync with the number of paging levels used by the kernel, and there is no need to store this in a variable. There is also no need to use variables for storing the calculations of pgdir_shift and ptrs_per_p4d, as they are easily determined on the fly. This removes the need for two different sources of truth for determining whether 5-level paging is in use: CR4.LA57 always reflects the actual state, and never changes from the point of view of the 64-bit core kernel. The only potential concern is the cost of CR4 accesses, which can be mitigated using alternatives patching based on feature detection. Note that even the decompressor does not manipulate any page tables before updating CR4.LA57, so it can also avoid the associated global variables entirely. However, as it does not implement alternatives patching, the associated ELF sections need to be discarded. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/misc.h | 4 -- arch/x86/boot/compressed/pgtable_64.c | 12 ---- arch/x86/boot/compressed/vmlinux.lds.S | 1 + arch/x86/include/asm/pgtable_64_types.h | 58 ++++++++------------ arch/x86/kernel/cpu/common.c | 2 - arch/x86/kernel/head64.c | 33 +---------- arch/x86/mm/kasan_init_64.c | 3 - arch/x86/mm/mem_encrypt_identity.c | 9 --- 8 files changed, 27 insertions(+), 95 deletions(-) diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h index bc2f0f17fb90..2b15ddd0e177 100644 --- a/arch/x86/boot/compressed/misc.h +++ b/arch/x86/boot/compressed/misc.h @@ -16,9 +16,6 @@ #define __NO_FORTIFY -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 - /* * Boot stub deals with identity mappings, physical and virtual addresses are * the same, so override these defines. @@ -178,7 +175,6 @@ static inline int count_immovable_mem_regions(void) { return 0; } #endif /* ident_map_64.c */ -extern unsigned int __pgtable_l5_enabled, pgdir_shift, ptrs_per_p4d; extern void kernel_add_identity_map(unsigned long start, unsigned long end); /* Used by PAGE_KERN* macros: */ diff --git a/arch/x86/boot/compressed/pgtable_64.c b/arch/x86/boot/compressed/pgtable_64.c index 51f957b24ba7..ae72f53f5e77 100644 --- a/arch/x86/boot/compressed/pgtable_64.c +++ b/arch/x86/boot/compressed/pgtable_64.c @@ -9,13 +9,6 @@ #define BIOS_START_MIN 0x20000U /* 128K, less than this is insane */ #define BIOS_START_MAX 0x9f000U /* 640K, absolute maximum */ -#ifdef CONFIG_X86_5LEVEL -/* __pgtable_l5_enabled needs to be in .data to avoid being cleared along with .bss */ -unsigned int __section(".data") __pgtable_l5_enabled; -unsigned int __section(".data") pgdir_shift = 39; -unsigned int __section(".data") ptrs_per_p4d = 1; -#endif - /* Buffer to preserve trampoline memory */ static char trampoline_save[TRAMPOLINE_32BIT_SIZE]; @@ -125,11 +118,6 @@ asmlinkage void configure_5level_paging(struct boot_params *bp, void *pgtable) native_cpuid_eax(0) >= 7 && (native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) { l5_required = true; - - /* Initialize variables for 5-level paging */ - __pgtable_l5_enabled = 1; - pgdir_shift = 48; - ptrs_per_p4d = 512; } /* diff --git a/arch/x86/boot/compressed/vmlinux.lds.S b/arch/x86/boot/compressed/vmlinux.lds.S index 083ec6d7722a..06358bb067fe 100644 --- a/arch/x86/boot/compressed/vmlinux.lds.S +++ b/arch/x86/boot/compressed/vmlinux.lds.S @@ -81,6 +81,7 @@ SECTIONS *(.dynamic) *(.dynsym) *(.dynstr) *(.dynbss) *(.hash) *(.gnu.hash) *(.note.*) + *(.altinstructions .altinstr_replacement) } .got.plt (INFO) : { diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm/pgtable_64_types.h index 38b54b992f32..6a57bfdff52b 100644 --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -6,7 +6,10 @@ #ifndef __ASSEMBLY__ #include +#include +#include #include +#include /* * These are used to make use of C type-checking.. @@ -21,63 +24,50 @@ typedef unsigned long pgprotval_t; typedef struct { pteval_t pte; } pte_t; typedef struct { pmdval_t pmd; } pmd_t; -#ifdef CONFIG_X86_5LEVEL -extern unsigned int __pgtable_l5_enabled; - -#ifdef USE_EARLY_PGTABLE_L5 -/* - * cpu_feature_enabled() is not available in early boot code. - * Use variable instead. - */ -static inline bool pgtable_l5_enabled(void) +static __always_inline __pure bool pgtable_l5_enabled(void) { - return __pgtable_l5_enabled; -} -#else -#define pgtable_l5_enabled() cpu_feature_enabled(X86_FEATURE_LA57) -#endif /* USE_EARLY_PGTABLE_L5 */ + unsigned long r; + bool ret; -#else -#define pgtable_l5_enabled() 0 -#endif /* CONFIG_X86_5LEVEL */ + if (!IS_ENABLED(CONFIG_X86_5LEVEL)) + return false; + + asm(ALTERNATIVE_TERNARY( + "movq %%cr4, %[reg] \n\t btl %[la57], %k[reg]" CC_SET(c), + %P[feat], "stc", "clc") + : [reg] "=&r" (r), CC_OUT(c) (ret) + : [feat] "i" (X86_FEATURE_LA57), + [la57] "i" (X86_CR4_LA57_BIT) + : "cc"); -extern unsigned int pgdir_shift; -extern unsigned int ptrs_per_p4d; + return ret; +} #endif /* !__ASSEMBLY__ */ #define SHARED_KERNEL_PMD 0 -#ifdef CONFIG_X86_5LEVEL - /* * PGDIR_SHIFT determines what a top-level page table entry can map */ -#define PGDIR_SHIFT pgdir_shift +#define PGDIR_SHIFT (pgtable_l5_enabled() ? 48 : 39) #define PTRS_PER_PGD 512 /* * 4th level page in 5-level paging case */ #define P4D_SHIFT 39 +#ifdef CONFIG_X86_5LEVEL #define MAX_PTRS_PER_P4D 512 -#define PTRS_PER_P4D ptrs_per_p4d +#else +#define MAX_PTRS_PER_P4D 1 +#endif +#define PTRS_PER_P4D (pgtable_l5_enabled() ? 512 : 1) #define P4D_SIZE (_AC(1, UL) << P4D_SHIFT) #define P4D_MASK (~(P4D_SIZE - 1)) #define MAX_POSSIBLE_PHYSMEM_BITS 52 -#else /* CONFIG_X86_5LEVEL */ - -/* - * PGDIR_SHIFT determines what a top-level page table entry can map - */ -#define PGDIR_SHIFT 39 -#define PTRS_PER_PGD 512 -#define MAX_PTRS_PER_P4D 1 - -#endif /* CONFIG_X86_5LEVEL */ - /* * 3rd level page */ diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 9e35e276c55a..d88e4be88868 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1,6 +1,4 @@ // SPDX-License-Identifier: GPL-2.0-only -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 #include #include diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 4bcbd4ae2dc6..aee99cfda4eb 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -7,9 +7,6 @@ #define DISABLE_BRANCH_PROFILING -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include @@ -52,14 +49,6 @@ extern pmd_t early_dynamic_pgts[EARLY_DYNAMIC_PAGE_TABLES][PTRS_PER_PMD]; static unsigned int __initdata next_early_pgt; pmdval_t early_pmd_flags = __PAGE_KERNEL_LARGE & ~(_PAGE_GLOBAL | _PAGE_NX); -#ifdef CONFIG_X86_5LEVEL -unsigned int __pgtable_l5_enabled __ro_after_init; -unsigned int pgdir_shift __ro_after_init = 39; -EXPORT_SYMBOL(pgdir_shift); -unsigned int ptrs_per_p4d __ro_after_init = 1; -EXPORT_SYMBOL(ptrs_per_p4d); -#endif - #ifdef CONFIG_DYNAMIC_MEMORY_LAYOUT unsigned long page_offset_base __ro_after_init = __PAGE_OFFSET_BASE_L4; EXPORT_SYMBOL(page_offset_base); @@ -78,21 +67,6 @@ static struct desc_struct startup_gdt[GDT_ENTRIES] __initdata = { [GDT_ENTRY_KERNEL_DS] = GDT_ENTRY_INIT(DESC_DATA64, 0, 0xfffff), }; -static inline bool check_la57_support(void) -{ - if (!IS_ENABLED(CONFIG_X86_5LEVEL)) - return false; - - /* - * 5-level paging is detected and enabled at kernel decompression - * stage. Only check if it has been enabled there. - */ - if (!(native_read_cr4() & X86_CR4_LA57)) - return false; - - return true; -} - static unsigned long __head sme_postprocess_startup(struct boot_params *bp, pmdval_t *pmd) { unsigned long vaddr, vaddr_end; @@ -155,7 +129,7 @@ unsigned long __head __startup_64(unsigned long physaddr, bool la57; int i; - la57 = check_la57_support(); + la57 = pgtable_l5_enabled(); /* Is the address too large? */ if (physaddr >> MAX_PHYSMEM_BITS) @@ -438,10 +412,7 @@ asmlinkage __visible void __init __noreturn x86_64_start_kernel(char * real_mode (__START_KERNEL & PGDIR_MASK))); BUILD_BUG_ON(__fix_to_virt(__end_of_fixed_addresses) <= MODULES_END); - if (check_la57_support()) { - __pgtable_l5_enabled = 1; - pgdir_shift = 48; - ptrs_per_p4d = 512; + if (pgtable_l5_enabled()) { page_offset_base = __PAGE_OFFSET_BASE_L5; vmalloc_base = __VMALLOC_BASE_L5; vmemmap_base = __VMEMMAP_BASE_L5; diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c index 0302491d799d..85ae1ef840cc 100644 --- a/arch/x86/mm/kasan_init_64.c +++ b/arch/x86/mm/kasan_init_64.c @@ -2,9 +2,6 @@ #define DISABLE_BRANCH_PROFILING #define pr_fmt(fmt) "kasan: " fmt -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index 64b5005d49e5..a857945af177 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -27,15 +27,6 @@ #undef CONFIG_PARAVIRT_XXL #undef CONFIG_PARAVIRT_SPINLOCKS -/* - * This code runs before CPU feature bits are set. By default, the - * pgtable_l5_enabled() function uses bit X86_FEATURE_LA57 to determine if - * 5-level paging is active, so that won't work here. USE_EARLY_PGTABLE_L5 - * is provided to handle this situation and, instead, use a variable that - * has been set by the early boot code. - */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include