From patchwork Mon Feb 26 14:29:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 206681 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2113976dyb; Mon, 26 Feb 2024 06:32:07 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWN+Lb2ipX6fuZNev4K9rgikt0Unj2+YjH24CjGZIkMkwyQ8HjxfAEkVGZkZ2CCyQtrmsVgwxRChl7LyqrHoZwZXsWrpA== X-Google-Smtp-Source: AGHT+IGXnT+f0baZosuUsEnXaiIVqKvNVnlR5gqkCahjdWGptQXgtEfKrIEhciRtw4H1yedjeSUV X-Received: by 2002:a05:6a21:3102:b0:1a1:863:e783 with SMTP id yz2-20020a056a21310200b001a10863e783mr763780pzb.17.1708957927347; Mon, 26 Feb 2024 06:32:07 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708957927; cv=pass; d=google.com; s=arc-20160816; b=z+ZCb57FJuyQYVHnrdfrCl/EFXggfY/ihWCZnTkL6gb5pwv0Cia+u5/Ag8ZTqj9FsH gMXsRd2qevsy/wsX0luzU3Zupo7b6eWzs4IzYphIvF3oWaFt3EaMf3Y8ZsIShnV4S1uR B2x6FzVq6PgS+c0j2hOCmWWSscTh4ANLR6xO3oj7TfqP6EN1D6bN+z9GQRAmSzO7Yv5m oenz9b3FbDV5klW2DVaOyYoXfvt6Jeq5W1/bDwSMrCqOrNVy6sRPW6DRMbRa59U187XL K9DZejh9n3bLY3OJtaVdJ6QDXTDXusH0GZqLQRDnbCNLF2gurb9lIc7slbSpOZYQWn6X Y3NA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=CpT/v58LZa+6DN+5ZZ1kc+YtmkGbFtn0oji87VtEKmc=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=s4WLHGVHFOppLs5Eh2ji6twxkJcNrkII2UR/9WEiluvckxyCqweasPfbzt1xPBhqgJ Fl5JOiNNqGrxDusTJVy8mpbnpNCLJ8rdnXmUSU1zU0uXMj+/D9nNxEaHIsChx2ZHfDb4 ZQnwYlHU4oVkl3gj4vImdnnA68i1173usH1PROyP2eBS/v4umBwskAy/cxSoVclctW9Y j/7QD3hOT5Gv6jfV3lVni4G6MPl3i5tgmgd+KdtBF1gJjhLAh5j0/qi0wMpAtKyc38jb jotORmtRiJlz2tHHUPjg/j3GX5q4FfVdfU4YYqV0Yyq15iKIln9d/CpYLeEMvMlNZCui DP9g==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=dbLixok0; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81622-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81622-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id f11-20020a63e30b000000b005ce025116bbsi3727579pgh.858.2024.02.26.06.32.07 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 06:32:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-81622-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=dbLixok0; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81622-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81622-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 8F45B28AEDC for ; Mon, 26 Feb 2024 14:30:35 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0AA3F605C8; Mon, 26 Feb 2024 14:30:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="dbLixok0" Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E844A12AAC4 for ; Mon, 26 Feb 2024 14:30:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957804; cv=none; b=qEJzLkwJTEjNok8i25o63GygvXE99hPNUt/3dJOEZf4iHm8XrwZ13hZF4R72gjGzhNlypFWQreP9Aj+6VjTTYTJNB7aDhg+A48EP97KeSk4dUAFrsKMdeAbp7W9yVlJldzYbJZP9OxJzH6YhgUjxuf0S+qzJUnIqB/Gmtot1F50= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957804; c=relaxed/simple; bh=wkfgi3tF51NfZa7UfUiPY8DHKdeYcx95mPJdjnoKkNE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=jPug9fe0L+Q3pSYbYVPXzXHExV8Fq+h7HUx6jWmQljKs/4PSZUE/cVBQZlioIgm40V6pJT0WNk8xW9/RfV7PR2X9lrJK9QosaHYFd5jG0H13Ghxp9e2FegPzNIvJvakypf5IS8teX1cexi2E1QTh3ElAMdZVTlDLx46XJoHEjSs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=dbLixok0; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-608d6e7314bso28507027b3.3 for ; Mon, 26 Feb 2024 06:30:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708957802; x=1709562602; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=CpT/v58LZa+6DN+5ZZ1kc+YtmkGbFtn0oji87VtEKmc=; b=dbLixok0jF3K3eFbyGlBl51OgWZHSZN4xu+kyb9DRD8dcCkAEXPgqnMkUKVcBw5XSE k9SS+SRZNR/qfjgUpdJ1xxuh59QXFA9J/MXlNUgjK+CtozRn69iGHwnncCfqSayZqw4r PgDCaUNULDumN4h8Ato3TjBtmr6Lp4cV61iwlPkfk5Zu2lZXgPE0iA4YBt0pfDEI2r7A UM7bfLKq3oWrLvJdhcYBUDkUoLhw1Ny8MN30eFoDN7RAvH9qIQQBTlTLQP1/zcsITIZ5 PSQq+li96fapPwJNieRvCTxwuUa+ZDKkYAx7/KYfEFK31K3/kne+T/uaXUzUsg4bO8rx VclQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708957802; x=1709562602; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=CpT/v58LZa+6DN+5ZZ1kc+YtmkGbFtn0oji87VtEKmc=; b=hM4ZDp8oStyTK6YJpdEeUYzAndoAE2cow1YlGZVTZ9ojfgwTRHNimTCV5OmW//gchE RXn9IfR9e7hPZs5hQXwJOFNvn6O2VP36rmL70a/xlY97SYgooaeMQ1hHQd4Vc6L1+y9u tOqU+wkoQhkRttiHSNTglxNhnxBvzOPDcji65zRSKzExddT4lC+GvkUfx0mW26Xk6CL7 LMvxdKpULESmAwFgaJ8fTrS7T/F+5CCg+0ikIU//+rD6huCmjCOORa60tWowgNQsxb85 4c/8///xwYV3XdkippGVckaCY9mPazT0QVIfnsRL4+ypHXtgsL6iyACVBAsnUnnOhSem 39mQ== X-Gm-Message-State: AOJu0YxZdGuSOVO69haOxkuoO19gVt2w2xhF17MsNBMbpJ5DJBq/z7OX vRfV6dvwwZAs0ABpDYL9obg8FBHFijWYPozG33MNUN/9JNSqY1nAgNjFIgTYnQHPH6yGiJDQUjU ceYB5wNoKEoMoJwt6HWbjmUtgrTedovHdC1/F/GrsfUpusNZsy8Tmh7sBUT4H425ZL2OmsiJoQC VPP/vqGrFPG3NgemorY7hg6y7bJ2NUgQ== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:100b:b0:dcc:8be2:7cb0 with SMTP id w11-20020a056902100b00b00dcc8be27cb0mr244198ybt.0.1708957801956; Mon, 26 Feb 2024 06:30:01 -0800 (PST) Date: Mon, 26 Feb 2024 15:29:54 +0100 In-Reply-To: <20240226142952.64769-12-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240226142952.64769-12-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240226142952.64769-13-ardb+git@google.com> Subject: [PATCH v6 01/10] x86/startup_64: Simplify CR4 handling in startup code From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791972267490260412 X-GMAIL-MSGID: 1791972267490260412 From: Ard Biesheuvel When paging is enabled, the CR4.PAE and CR4.LA57 control bits cannot be changed, and so they can simply be preserved rather than reason about whether or not they need to be set. CR4.MCE should be preserved unless the kernel was built without CONFIG_X86_MCE, in which case it must be cleared. CR4.PSE should be set explicitly, regardless of whether or not it was set before. CR4.PGE is set explicitly, and then cleared and set again after programming CR3 in order to flush TLB entries based on global translations. This makes the first assignment redundant, and can therefore be omitted. So clear PGE by omitting it from the preserve mask, and set it again explicitly after switching to the new page tables. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head_64.S | 30 ++++++++------------ 1 file changed, 12 insertions(+), 18 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index d295bf68bf94..1b054585bfd1 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -185,6 +185,11 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) addq $(init_top_pgt - __START_KERNEL_map), %rax 1: + /* + * Create a mask of CR4 bits to preserve. Omit PGE in order to clean + * global 1:1 translations from the TLBs. + */ + movl $(X86_CR4_PAE | X86_CR4_LA57), %edx #ifdef CONFIG_X86_MCE /* * Preserve CR4.MCE if the kernel will enable #MC support. @@ -193,20 +198,13 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * configured will crash the system regardless of the CR4.MCE value set * here. */ - movq %cr4, %rcx - andl $X86_CR4_MCE, %ecx -#else - movl $0, %ecx + orl $X86_CR4_MCE, %edx #endif + movq %cr4, %rcx + andl %edx, %ecx - /* Enable PAE mode, PSE, PGE and LA57 */ - orl $(X86_CR4_PAE | X86_CR4_PSE | X86_CR4_PGE), %ecx -#ifdef CONFIG_X86_5LEVEL - testb $1, __pgtable_l5_enabled(%rip) - jz 1f - orl $X86_CR4_LA57, %ecx -1: -#endif + /* Even if ignored in long mode, set PSE uniformly on all logical CPUs. */ + btsl $X86_CR4_PSE_BIT, %ecx movq %rcx, %cr4 /* Setup early boot stage 4-/5-level pagetables. */ @@ -223,14 +221,10 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) movq %rax, %cr3 /* - * Do a global TLB flush after the CR3 switch to make sure the TLB - * entries from the identity mapping are flushed. + * Set CR4.PGE to re-enable global translations. */ - movq %cr4, %rcx - movq %rcx, %rax - xorq $X86_CR4_PGE, %rcx + btsl $X86_CR4_PGE_BIT, %ecx movq %rcx, %cr4 - movq %rax, %cr4 /* Ensure I am executing from virtual addresses */ movq $1f, %rax From patchwork Mon Feb 26 14:29:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 206682 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2114094dyb; Mon, 26 Feb 2024 06:32:16 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCV8FUk1wreGK25ar2FrVe9ljFCnJWbi0npK4N1a4wZ1fPeDR7BEMydb0SE6yNWjdX6ndCREEz4byeLGodFNYajypVxqkA== X-Google-Smtp-Source: AGHT+IFTLfa544Yj2ttSIeUDtGtzhf580soYfM0jFO+EYJaAO03Wgvo4LbtBP715BaMHJ2SxTdaA X-Received: by 2002:a05:6e02:1e0b:b0:365:b9e:bdb7 with SMTP id g11-20020a056e021e0b00b003650b9ebdb7mr10087751ila.22.1708957936173; Mon, 26 Feb 2024 06:32:16 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708957936; cv=pass; d=google.com; s=arc-20160816; b=i7NKKbH3A3EYC3LbbYS7dz2Sp+IoDgDQvgV5u0ZW4JGNn91w6whz/BRkYSTibCG34X rnbd1RxaJJv67+okiyTrfbZs7TF8jqk36CqI6kuZpIdiAhmgsIdXWwsRNJm8/PBPAAQb jZroQGDfZuLt7uY9PUrDgV7RZt8AxlCQLJeNIUgeMdkSNs9eGMmVrQQFz/QRm7Kz40/q YYgaJKQEWLJVM3TCfimagojXuPZuzsgGivheLbzaeiX3xNEdtmxZ484X8fxuL1EMfV/K F96gkIgky3t7JxHnZQnVTWCeNvzNpR1qsJr2lCiMakPZ9rvN/PiKZ9Bns+jQBORUEh/3 y/xA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=gMaXkJDSba/rz2ejEwTszeJPuP18TapiV6QGh3kcGw8=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=Uag8XcoEK+wwZ3TwwIFXAMkjMMkskm8W7Je5LKNi6gjb0dQUafkhQcZzrcjAZXM1qB HCUpgmMt8mUg+tB4P3WZRv0OGHy7trWXNdL9chEoshABFwvTGkly/LV8IEQmhodD3pcE qTWx2tDi6x0IDPbXqfYKgzNCakNaB0YLCoHpXthF1s18NUs2BceoCnVdvkluOxwtgyC4 GIMEwPsXUQnMbqfJEeD1Fh8je8bbg01BFj3QozF+x6m97ET9I+kQcvjKM4ibTncRrFwJ 91TzMRcDR6G0lcgzCEAQCYKpkWC+601BKpmxtP3yDis+5/lVJxb2fJQ8P6f84SJd6AFd zj9Q==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=FZh6eQ2G; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81623-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81623-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id a20-20020a631a54000000b005dbed76d0afsi3787554pgm.667.2024.02.26.06.32.16 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 06:32:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-81623-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=FZh6eQ2G; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81623-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81623-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 457E728E451 for ; Mon, 26 Feb 2024 14:30:49 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 3D71112BEAB; Mon, 26 Feb 2024 14:30:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="FZh6eQ2G" Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 42E8912B157 for ; Mon, 26 Feb 2024 14:30:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957807; cv=none; b=amAk4BCVbyFank+Em2to2MSIYbV8/5GhpUipt2Xzwm3UGEKzon9as7208KSnzpueZM6VBWWDdWLwi50agQy0KusIANz/TPJDaa8GUSiqDnRrlrXOGUWzQ+PCfzzHzUIcerHFGlnLklVQSPBsRs5c5bhXQ6QpEda418IiJrIEMkc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957807; c=relaxed/simple; bh=IKzSvD/LgGiOgyH8nDLzJUWhN/QyWc5LsEQ67nR9hkM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Bl1Fs7k0R+nA4Ve1rIICx0sHZOKBsy4C4Cx0GqC26y0FL7qyu88KEhDyVy/8bNgTdcf4wTD2hskQPwCq54fMLWuZgtCWJEvJ4/U6HD6t67nbjgjbMKYgRCXG8DyOKrK12/FH1r1aZANBxwfDfl25pWKr0KrFsiJU5wD0CqFYeds= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=FZh6eQ2G; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcc05887ee9so4214550276.1 for ; Mon, 26 Feb 2024 06:30:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708957804; x=1709562604; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gMaXkJDSba/rz2ejEwTszeJPuP18TapiV6QGh3kcGw8=; b=FZh6eQ2GuYuEiRfPS5iThakFFTPnnFvfuV9ztUKWjWCDNZWEoNiwoPNbH0bonYHmpw NLz2js5RqV+HBTiMVlZtD/pXnl8rTc9z4O7xXYx9uKPp/fdnv2uNBRTM2ybzQMX+KPmh TyICpKQL3HTGhKKeJAOFg243/QrHgyQ7QtVbBNaGB6L3ac75fImGWBj63MCxoUI2PqI8 MB0UbYAwh+/FgY0DTJMD1lt+gs4YtjCoTLYDaWg+ugfzm5idbn9ce1xH0O1SXh3aEgFt vb6xU3JXO4qcfXj8V5gILncxbzZjBy70krfQtplvC3ohwOtC8gQqmsGQra5c0U2hcenx eHJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708957804; x=1709562604; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gMaXkJDSba/rz2ejEwTszeJPuP18TapiV6QGh3kcGw8=; b=s1vYli1gJLGusbRLQyLoXL2o74gdSlkT5byurkpBCZt4vjm6MVEFvTGBcUf3T3oBjp 2QueToW3mwXRVRpfYj5mySn2Gr4HJgsw5bVA4Dsf9OaY576xOoZfUW1A5bHl88xjHgQP F4FLwhHp8ClvGckX+NsOD0zRVo0AmfR5I7uhD/CYlZ+hUCBbMYmXaqfFq37fp5g3h2QD zxwdZZ+eHg7fkKVTqbDi0aI5+MyYf+cTe8vrth6gtdDflZ6sHlRRiqMdbjK6Pqqg8FWg SwIC8byqFLmAnTB669HYZrGlSmULpGBEEBvd2uj+ezRqt1pR9RXSpZQUJ7wKvqC5sSoX pXEw== X-Gm-Message-State: AOJu0YwC1YIa+F7J4cKzbREUzXbRGjB7nT+sC1YYMWyMOivOYmbUj1s1 +H/O3iRw1nUBeYSuH96HrtKb1ZDQSfDYYqae4CaU1Volg5x4Vrqd5GPk/BsjC6akVQDPTZK4nE1 lmncHJXN390LOnS25u7uHBqPkQS8NobUgNVBWCHY8ZFZ4Qv2rx0UfCtz5YVAGgLn0yDjAsZkFFx gq99i6wjdzKs71aR07lpB2omD/D2U/YA== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1003:b0:dc7:3189:4e75 with SMTP id w3-20020a056902100300b00dc731894e75mr252518ybt.3.1708957804161; Mon, 26 Feb 2024 06:30:04 -0800 (PST) Date: Mon, 26 Feb 2024 15:29:55 +0100 In-Reply-To: <20240226142952.64769-12-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240226142952.64769-12-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240226142952.64769-14-ardb+git@google.com> Subject: [PATCH v6 02/10] x86/startup_64: Defer assignment of 5-level paging global variables From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791972277319773646 X-GMAIL-MSGID: 1791972277319773646 From: Ard Biesheuvel Assigning the 5-level paging related global variables from the earliest C code using explicit references that use the 1:1 translation of memory is unnecessary, as the startup code itself does not rely on them to create the initial page tables, and this is all it should be doing. So defer these assignments to the primary C entry code that executes via the ordinary kernel virtual mapping. Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/pgtable_64_types.h | 2 +- arch/x86/kernel/head64.c | 44 +++++++------------- 2 files changed, 15 insertions(+), 31 deletions(-) diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm/pgtable_64_types.h index 38b54b992f32..9053dfe9fa03 100644 --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -21,9 +21,9 @@ typedef unsigned long pgprotval_t; typedef struct { pteval_t pte; } pte_t; typedef struct { pmdval_t pmd; } pmd_t; -#ifdef CONFIG_X86_5LEVEL extern unsigned int __pgtable_l5_enabled; +#ifdef CONFIG_X86_5LEVEL #ifdef USE_EARLY_PGTABLE_L5 /* * cpu_feature_enabled() is not available in early boot code. diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 72351c3121a6..deaaea3280d9 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -23,6 +23,7 @@ #include #include +#include #include #include #include @@ -77,24 +78,11 @@ static struct desc_struct startup_gdt[GDT_ENTRIES] __initdata = { [GDT_ENTRY_KERNEL_DS] = GDT_ENTRY_INIT(DESC_DATA64, 0, 0xfffff), }; -#ifdef CONFIG_X86_5LEVEL -static void __head *fixup_pointer(void *ptr, unsigned long physaddr) -{ - return ptr - (void *)_text + (void *)physaddr; -} - -static unsigned long __head *fixup_long(void *ptr, unsigned long physaddr) +static inline bool check_la57_support(void) { - return fixup_pointer(ptr, physaddr); -} - -static unsigned int __head *fixup_int(void *ptr, unsigned long physaddr) -{ - return fixup_pointer(ptr, physaddr); -} + if (!IS_ENABLED(CONFIG_X86_5LEVEL)) + return false; -static bool __head check_la57_support(unsigned long physaddr) -{ /* * 5-level paging is detected and enabled at kernel decompression * stage. Only check if it has been enabled there. @@ -102,21 +90,8 @@ static bool __head check_la57_support(unsigned long physaddr) if (!(native_read_cr4() & X86_CR4_LA57)) return false; - *fixup_int(&__pgtable_l5_enabled, physaddr) = 1; - *fixup_int(&pgdir_shift, physaddr) = 48; - *fixup_int(&ptrs_per_p4d, physaddr) = 512; - *fixup_long(&page_offset_base, physaddr) = __PAGE_OFFSET_BASE_L5; - *fixup_long(&vmalloc_base, physaddr) = __VMALLOC_BASE_L5; - *fixup_long(&vmemmap_base, physaddr) = __VMEMMAP_BASE_L5; - return true; } -#else -static bool __head check_la57_support(unsigned long physaddr) -{ - return false; -} -#endif static unsigned long __head sme_postprocess_startup(struct boot_params *bp, pmdval_t *pmd) { @@ -180,7 +155,7 @@ unsigned long __head __startup_64(unsigned long physaddr, bool la57; int i; - la57 = check_la57_support(physaddr); + la57 = check_la57_support(); /* Is the address too large? */ if (physaddr >> MAX_PHYSMEM_BITS) @@ -465,6 +440,15 @@ asmlinkage __visible void __init __noreturn x86_64_start_kernel(char * real_mode (__START_KERNEL & PGDIR_MASK))); BUILD_BUG_ON(__fix_to_virt(__end_of_fixed_addresses) <= MODULES_END); + if (check_la57_support()) { + __pgtable_l5_enabled = 1; + pgdir_shift = 48; + ptrs_per_p4d = 512; + page_offset_base = __PAGE_OFFSET_BASE_L5; + vmalloc_base = __VMALLOC_BASE_L5; + vmemmap_base = __VMEMMAP_BASE_L5; + } + cr4_init_shadow(); /* Kill off the identity-map trampoline */ From patchwork Mon Feb 26 14:29:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 206678 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2113330dyb; Mon, 26 Feb 2024 06:31:20 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCU9C1XnEVZc9uNAtIqitYYEDBnekmbV9lMrrMQBEwCeTUdDsOEmUlGIiz/9nnmA3ISPSWnuBMFGuxHh8X1PK2hVmrHmSA== X-Google-Smtp-Source: AGHT+IFHqu3LVlEqW/079BV7qqu0pjIJbOjTflxkei6KBVPuVkBJZC/edAapheYOE7ikN98vZyvU X-Received: by 2002:aa7:d498:0:b0:565:ec92:58ee with SMTP id b24-20020aa7d498000000b00565ec9258eemr1938442edr.4.1708957880088; Mon, 26 Feb 2024 06:31:20 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708957880; cv=pass; d=google.com; s=arc-20160816; b=snHIvN2uI+9SBRfQOQqddlLw+tfq6cOhN4XxVIurZV1QF24SOJU3xp5/R2DgigsuC7 3LoRWir6DuZmczKiNVvZQhysubUyfpts/2MNaufqDcltA/zRoxobIDzR0vX9NRXaY2PA fvNPLQvIuwpI7iTBjc5aNscdsxJ+/smfSUz+/Y9nfTyeursFqlz3GPyaFuIadIyZjU9V VClEJftld1pGMrY8qm0GOGrRhQzyR2ZVbctYxypG/JH50KEyoPxrD1q70BqO9hGX0Mqu u3A74adnOXKNEReKqZd22PPLsjIakY/HSp+7NiK+al8PxTxfobci6ecUuLC9Q/MXv5oO mpPA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=ggv/9mL4EwQQhS5S3/vq6rfpKgE+aVW7oD/WPNHnqXE=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=SiNfjugF/X0WYZZj8/XVg9dEzbin31hwuSFNzVdnZzR/alVDlQGSGolOzK40MR5/EI 2VbMttgoIBmPsIo45WMKglYJMSDmndqAlOY/wwNpjvPzXKsBXoZynNi3njxx+UE1Bqyq 7x/eLKsc6CJHjRVZNFrdYFkXDXGEohguaXHnZePI2EydjvhScMxMXGBkEHp+O5m8OhfP uTw6wlymqsB20tArTD07q1s59vF/LatqaLbnfTPcck8XddaG9ePwUN1NYJJ4uhPOOQFg EH29e2tbKHg4o5lEJG30cNDybEBFvCAGx6pG7NqM6Fxy5cyM2cm+TB8bfIZwJZfT2vfD k+KQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=TeoOGVgj; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81624-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81624-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id r21-20020a50d695000000b00564ab985328si2106426edi.218.2024.02.26.06.31.19 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 06:31:20 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-81624-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=TeoOGVgj; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81624-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81624-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id C8D6F1F2C6A2 for ; Mon, 26 Feb 2024 14:31:11 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 3E07112BF03; Mon, 26 Feb 2024 14:30:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="TeoOGVgj" Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 710DE12BE89 for ; Mon, 26 Feb 2024 14:30:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957811; cv=none; b=nwXNeU5it/U4GdJzVwrLpiVf0x6RtglOuDzw5yYp+Yktpn8gbFIBBoBn9hhw281+0wAupsp7NwqkIdMTU+fMa3WX7icKkoTgzdbyqMbWr2NPqpZJ0dxP3o+n9CkUK0Cet9GTfiM14mFHA94J00agDAdhOZM84ISNcJQ3Ndy6e1U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957811; c=relaxed/simple; bh=BbfBvgfVi77/QhkFi7YZNDyhrbZynGkOkpHyBXM04sE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=aXapUhoF+z4WHE6qKgby3s9eRk+zHPUGkazqR38yineU+epf6OUFLxLGqLz/MtmbTgMjXNaufPJ5TmFppz8iUZWqcn83tUe3wFGUiwPDrbTxh6WlnzcHyC1GYjfY3P8gdKqheVNzwpjj1D3tgPRgpk6PIH0ZSc3OjXmWtEdOk2s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=TeoOGVgj; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-33d308b0c76so1190085f8f.0 for ; Mon, 26 Feb 2024 06:30:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708957807; x=1709562607; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ggv/9mL4EwQQhS5S3/vq6rfpKgE+aVW7oD/WPNHnqXE=; b=TeoOGVgjf/+/FxX2ZXGvSn3YschiMOIY421Bc3MbAxB1fs+z7Ai2bReIGNE7NyZthr Jai2f5KdqOyOLumqHAcztnK2y0H0m2PNJ5E95BR2Clpxt48NBwpeSOZ8CD5aBoxXKO9R SKz3hHvPbThw6S1vHzJbBjChv3zmNMmBaeS8M8tOFqJFPe2nwB1Md9N/jdW9nCVzSP2i 7skBL9wNRahxahNbHctFDh399EiumcT5Iz/S9km0fpN3A6dQm8pf3KgoejaA62AThbH8 vZLVJ5JzOJ1kMxE4XsnITYPeZr7AXo4uRA6t96j8bHkJt6ECLe+m338bAmWbWAO5BZBg j5MQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708957807; x=1709562607; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ggv/9mL4EwQQhS5S3/vq6rfpKgE+aVW7oD/WPNHnqXE=; b=AElduKEJa06EId7DM/+dMQ15FgkKiA+wM+2H2/eXoEZUUQOhlPEU7rz49NINUF/7WN TZ6q5a/K0TfUAbeCrdVPvqdfdCSmtx9ZYyJR5dGiv+7IvOrxRvzme1vlfcvsp1WhzhmW QrdeIwlRhJVWgqEP1T/Ehik45ic7c4wRVPIdwrH+IoQ+HRZKeRVbz628rRUC3h4VyqpW u5E4xspJvba6/7Zavz75qvYBd3YxSLwV/NUKaHu2jwcEuo3ckdy+Q0ZyTtJkN3o4jr5K lvxrS9+GhwwqW1I197lIt/BdPU/gSzqPWJxhjfRdiynKESnIjh3CYK+0ZdXGEY9zkYcj miIA== X-Gm-Message-State: AOJu0YzBuzIIVpbsLVpnrSVXEVCimTueMkOtvLFmh3YBXfZwwNxawVkg yKrvcea4rXh8gA2Oi9FcbRfFXyfd52n1LZZSS8fUA6y0+eXq9W5tTpVi3Q/xct5w2K/nYLK75x+ Ely+z48bvUf4w6LT3D4+ATu4RKS3wN1bGOwGeZBIqF7mtKANK+4gVzXNRiAOXVvRdS4y5/Tzmlj 6bhjEDE6rakqFPawUbaPPUUWlJ51PnsQ== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:4c8f:0:b0:33d:b376:89f0 with SMTP id z15-20020a5d4c8f000000b0033db37689f0mr21108wrs.4.1708957806552; Mon, 26 Feb 2024 06:30:06 -0800 (PST) Date: Mon, 26 Feb 2024 15:29:56 +0100 In-Reply-To: <20240226142952.64769-12-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240226142952.64769-12-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240226142952.64769-15-ardb+git@google.com> Subject: [PATCH v6 03/10] x86/startup_64: Simplify calculation of initial page table address From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791972218008899165 X-GMAIL-MSGID: 1791972218008899165 From: Ard Biesheuvel Determining the address of the initial page table to program into CR3 involves: - taking the physical address - adding the SME encryption mask On the primary entry path, the code is mapped using a 1:1 virtual to physical translation, so the physical address can be taken directly using a RIP-relative LEA instruction. On the secondary entry path, the address can be obtained by taking the offset from the virtual kernel base (__START_kernel_map) and adding the physical kernel base. This is implemented in a slightly confusing way, so clean this up. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head_64.S | 25 ++++++-------------- 1 file changed, 7 insertions(+), 18 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 1b054585bfd1..c451a72bc92b 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -111,13 +111,11 @@ SYM_CODE_START_NOALIGN(startup_64) call __startup_64 /* Form the CR3 value being sure to include the CR3 modifier */ - addq $(early_top_pgt - __START_KERNEL_map), %rax + leaq early_top_pgt(%rip), %rcx + addq %rcx, %rax #ifdef CONFIG_AMD_MEM_ENCRYPT mov %rax, %rdi - mov %rax, %r14 - - addq phys_base(%rip), %rdi /* * For SEV guests: Verify that the C-bit is correct. A malicious @@ -126,12 +124,6 @@ SYM_CODE_START_NOALIGN(startup_64) * the next RET instruction. */ call sev_verify_cbit - - /* - * Restore CR3 value without the phys_base which will be added - * below, before writing %cr3. - */ - mov %r14, %rax #endif jmp 1f @@ -171,18 +163,18 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) /* Clear %R15 which holds the boot_params pointer on the boot CPU */ xorq %r15, %r15 + /* Derive the runtime physical address of init_top_pgt[] */ + movq phys_base(%rip), %rax + addq $(init_top_pgt - __START_KERNEL_map), %rax + /* * Retrieve the modifier (SME encryption mask if SME is active) to be * added to the initial pgdir entry that will be programmed into CR3. */ #ifdef CONFIG_AMD_MEM_ENCRYPT - movq sme_me_mask, %rax -#else - xorq %rax, %rax + addq sme_me_mask(%rip), %rax #endif - /* Form the CR3 value being sure to include the CR3 modifier */ - addq $(init_top_pgt - __START_KERNEL_map), %rax 1: /* @@ -207,9 +199,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) btsl $X86_CR4_PSE_BIT, %ecx movq %rcx, %cr4 - /* Setup early boot stage 4-/5-level pagetables. */ - addq phys_base(%rip), %rax - /* * Switch to new page-table * From patchwork Mon Feb 26 14:29:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 206679 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2113455dyb; Mon, 26 Feb 2024 06:31:28 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUHjy6CQg73+/rhC7IwzDs0oFN0LURf8hrJwnpaFxb3Qjega9GEOBZzD3ZsJkMT1Kx0/Es3u/hgRMc6bfaifmMND6XyTQ== X-Google-Smtp-Source: AGHT+IH/Pu7o9i2auxPNyDrZbKsT3pFjpQlflxoylU8HX6WETmE6N24P14mw2LA49ABnWpguE4jH X-Received: by 2002:a05:6102:3713:b0:470:44e4:27d0 with SMTP id s19-20020a056102371300b0047044e427d0mr5457012vst.6.1708957888536; Mon, 26 Feb 2024 06:31:28 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708957888; cv=pass; d=google.com; s=arc-20160816; b=chQ0jCtHuU8b3vQCEXKbtWrXefLjf9dvk/LjyhGt2pI2zJFe1KYT2Ed6PEZ7uYOEYT b4TdFEZac0HatbhQXdoXzb3FCdsrqg4xcoKs1ns/LwzMaQe/n3maJaiNrlKcm4k0msLK YJnHZcY7EU29o68FBs23flYX39dyydJNikFuSyRZnh7GyrvupH2zFTjtOspCYTgTf59g ktwE2ff+SAmohUK/p0JOy2EGfLdYqNHxcQPRUTiOq6V0mAyeQyOtH3XlINL7BN420EPj 9ehKlzvsgOibuDd2gnpuOlZ+1CjDpWYd6kT05EzAATNAiBJvJ9Zz+1mvmy6D82E8SsDr VrCQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=HUEqNAX+crIm6ixnlTe40pRvCdb8ZI8NNnngiA9aRDQ=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=e9Gz3mZtzrgczCOhVeQ267P/bac/4/GJFLvgeJ2RO7h16P3i+HJ2fj3/ieMJU3++y9 J/8aBdaaid9fe4kM41i0vzMs7u9yTEHF22vRWK5gt+RO2iL44AUmO9tMZpS5lpnBesWa YvlZNVF/NRlEOVQNaplLQQ7Rs3kfXw+BLBTTRyh+wqiURdsFjEf0s9+PtUmBHZEd279j nChFb8FGZ0/cBbiSYeMXdLoxVeC/FJK1JiHrR8YV9wwji0oewiENxQLL9BHDyjjZRi/w Y2CorLToqv/HcGiNe5YRoYioXxr4dPfhJJBfd5Hl1Z6bRHoGB4UoRIz24tPmNh0ph1tw oOQw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=NJFT1AwJ; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81625-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81625-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id gd8-20020a056102624800b0046d21d6e822si562754vsb.133.2024.02.26.06.31.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 06:31:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-81625-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=NJFT1AwJ; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81625-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81625-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id DC02C1C21D74 for ; Mon, 26 Feb 2024 14:31:24 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7230D12C520; Mon, 26 Feb 2024 14:30:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="NJFT1AwJ" Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8330F53E28 for ; Mon, 26 Feb 2024 14:30:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957812; cv=none; b=WbS8kGoMNNFwjq9dDxFcEDdEIm45u9jW9QdHVHXQBQKTpe0qIDplooPykf+fgi3tfQ3tQe7e/IWPNjd3F6V1RdQL1epseP4xsV+EH+oY9zEmSxVn9VFoicw5ZooMtangwCV4yM8GIPRj2zZ8DPDVhPaeGJMdgXF282EFj+Y4vnE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957812; c=relaxed/simple; bh=+SAmI2opGAKocGzeZEPFdFE24u4ggFbULQvVpI+lmmo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=FEcjRvUvaCUa6zvACdkIMjUrnAsyrlgderKZ6eE31urFyCzvHDi9XWa3LAB2uKc4Yv8iiMETAKnBHX++6F4yedzNXX1qJQfnB5E7pnCR4BZwzpGbKVOGNQAhRK9G1yNpGlHRrr9sLr0nqGku5xH/UcxbICMzVUuLIsklZE8qynQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=NJFT1AwJ; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-33d6eb5e5a9so2319021f8f.3 for ; Mon, 26 Feb 2024 06:30:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708957809; x=1709562609; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=HUEqNAX+crIm6ixnlTe40pRvCdb8ZI8NNnngiA9aRDQ=; b=NJFT1AwJg/QdP/N9y9wIu3Y+VcOPAgca8O+z7aYJPaY/Widy/qcYsaFvx3sNc7ozB9 iyrkE47WtIPAzkP/ng22SeUz7U04syim6dTIeMc1NGXnW7zblRPGY4YJycmjNnwRR6ty KpXydcgDbm2lpfSMsglYSXzYnGXZn0skReqtQqaOqcWgicZvpRwFG+h8dVcdd1u13KYp EOpZ55cLzAaO0dliXZEBx0NOm53lf3qonIOA4uQOip66nk5Oa5W2TEYWlptbScHKb2YY a+u/neMpPxTr1l5BtBlG8fE+J1htEUEDgXSJ+zXEWNefqBftidoOqX1fv2kR6V3jUY8j Lmig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708957809; x=1709562609; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=HUEqNAX+crIm6ixnlTe40pRvCdb8ZI8NNnngiA9aRDQ=; b=G+H3VtGznaOkZnuGY7VfmD4IaXUY9Huy1iTIN06NqZ0T7fqG1JJHANp6hHXg2UMHlY nlW6xUHRHt3Ova7n4maUJW6WR/lNMzFYUkoR+2QTvqb5GlRRjWRomhhR1y/GVljbWab7 6sMP93x/uklIUfdZ2YTNO3EoNHTWbOm5keoHL6FGTeMKsz4MLdgEAfaF+CIWm9Mzdqvj nwxX03SEPw7zL3myVbtFWCPJDdToaOOUJQfvtpY+nQUIYor86LEi5x6Qphv6VJ7qClZj 2LapMtGlvaxmWlvsi6SJ9qvgensY4bz4fg63tlYeED77zgA61DJRzjkoKLUdSVbNXvRu hWag== X-Gm-Message-State: AOJu0YzKbOY7nXiHRqK9NEukYsxRYzLhS3AlzJC5RtPAmR2OCWiKxifW EUGvlcEX+yK9etIDca8/+GG2MFnjX5d6nnybJ5JYlIXbmIbSAlQO+OQkdzyE+IX8KGpTty20hMl dm5Pf2G21PqfzIYF3Kk+3LV1IPQQmTh2lP4W81EYdyaCHQvDMcoJyTYfl7k4GNSchCXeGFuTKWC cK8cTIAJTnexNnED1Q8Q6B6LXn9IvOaA== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a5d:5d85:0:b0:33d:ae58:8e1e with SMTP id ci5-20020a5d5d85000000b0033dae588e1emr14897wrb.12.1708957808755; Mon, 26 Feb 2024 06:30:08 -0800 (PST) Date: Mon, 26 Feb 2024 15:29:57 +0100 In-Reply-To: <20240226142952.64769-12-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240226142952.64769-12-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240226142952.64769-16-ardb+git@google.com> Subject: [PATCH v6 04/10] x86/startup_64: Simplify virtual switch on primary boot From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791972227434425582 X-GMAIL-MSGID: 1791972227434425582 From: Ard Biesheuvel The secondary startup code is used on the primary boot path as well, but in this case, the initial part runs from a 1:1 mapping, until an explicit cross-jump is made to the kernel virtual mapping of the same code. On the secondary boot path, this jump is pointless as the code already executes from the mapping targeted by the jump. So combine this cross-jump with the jump from startup_64() into the common boot path. This simplifies the execution flow, and clearly separates code that runs from a 1:1 mapping from code that runs from the kernel virtual mapping. Note that this requires a page table switch, so hoist the CR3 assignment into startup_64() as well. And since absolute symbol references will no longer be permitted in .head.text once we enable the associated build time checks, a RIP-relative memory operand is used in the JMP instruction, referring to an absolute constant in the .init.rodata section. Given that the secondary startup code does not require a special placement inside the executable, move it to the .text section. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head_64.S | 42 ++++++++++---------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index c451a72bc92b..87929f615048 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -39,7 +39,6 @@ L4_START_KERNEL = l4_index(__START_KERNEL_map) L3_START_KERNEL = pud_index(__START_KERNEL_map) - .text __HEAD .code64 SYM_CODE_START_NOALIGN(startup_64) @@ -126,9 +125,21 @@ SYM_CODE_START_NOALIGN(startup_64) call sev_verify_cbit #endif - jmp 1f + /* + * Switch to early_top_pgt which still has the identity mappings + * present. + */ + movq %rax, %cr3 + + /* Branch to the common startup code at its kernel virtual address */ + ANNOTATE_RETPOLINE_SAFE + jmp *0f(%rip) SYM_CODE_END(startup_64) + __INITRODATA +0: .quad common_startup_64 + + .text SYM_CODE_START(secondary_startup_64) UNWIND_HINT_END_OF_STACK ANNOTATE_NOENDBR @@ -174,8 +185,15 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) #ifdef CONFIG_AMD_MEM_ENCRYPT addq sme_me_mask(%rip), %rax #endif + /* + * Switch to the init_top_pgt here, away from the trampoline_pgd and + * unmap the identity mapped ranges. + */ + movq %rax, %cr3 -1: +SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) + UNWIND_HINT_END_OF_STACK + ANNOTATE_NOENDBR /* * Create a mask of CR4 bits to preserve. Omit PGE in order to clean @@ -199,30 +217,12 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) btsl $X86_CR4_PSE_BIT, %ecx movq %rcx, %cr4 - /* - * Switch to new page-table - * - * For the boot CPU this switches to early_top_pgt which still has the - * identity mappings present. The secondary CPUs will switch to the - * init_top_pgt here, away from the trampoline_pgd and unmap the - * identity mapped ranges. - */ - movq %rax, %cr3 - /* * Set CR4.PGE to re-enable global translations. */ btsl $X86_CR4_PGE_BIT, %ecx movq %rcx, %cr4 - /* Ensure I am executing from virtual addresses */ - movq $1f, %rax - ANNOTATE_RETPOLINE_SAFE - jmp *%rax -1: - UNWIND_HINT_END_OF_STACK - ANNOTATE_NOENDBR // above - #ifdef CONFIG_SMP /* * For parallel boot, the APIC ID is read from the APIC, and then From patchwork Mon Feb 26 14:29:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 206680 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2113654dyb; Mon, 26 Feb 2024 06:31:40 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXsOAa/MH0pyBiIHveUKVtg+tRojqkmFlB21+ODSZ5/H3iJpHluNnkcH2AriR6WEMtDk0tm5davqdvYYr2+IJ4oDszCeg== X-Google-Smtp-Source: AGHT+IGKdoYeYDO+sFL2cSw24kfodiyk8YPSXH8ewPjaWNBZUTo54Rb9UhWDEKJz+eLa7WuAN8mZ X-Received: by 2002:a67:f1d7:0:b0:471:e1ab:781b with SMTP id v23-20020a67f1d7000000b00471e1ab781bmr3436384vsm.4.1708957900636; Mon, 26 Feb 2024 06:31:40 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708957900; cv=pass; d=google.com; s=arc-20160816; b=V6dZLLFzqCBeqo485NjGr+mkHwSTBGF5IQoGkpLIsKp60O6pd4iN6+vVP+bEQPiNd3 9UxT4/yroNhGwib6qrGXG8CBZcACGmDUvxLDZjgUzuHff//iuIrMPhgOd5GXHCVzbSTF ZNEIzeY4xwIlvW8Nj8Z0ptT4sq7SDRdBdbvz5o0wofPNU90wd5KP6iLCiUTxhhlgSdHA kA+XVFzE3BY6mdGajAXy8tDTsiF5oNkKIDJLr9Cbj2+PWYfgB0TriTxBu1i3CF80ew31 QfEwMLZItzkgcs1dN3UQD3sDphYXguHcL1GbUGAiRjx1Tf6/3yo4YZ2SpiMag1NuFmf1 jLlQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=XsB2jfEOHuf9jjlesr7NzHbxyyk9Sr8XgRsNb8f+ZY8=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=VLxkzfXHEf/gVvombB2u/pej/xqGPT4zjInT/djHamll9AZ0bvbMroUmaZ5kW6Wd0B VFPHoifORPFznAEGTiz3EEVAzB1Yzoj9BCr1zeNxI9LUVROBFQWU4jHvc+VstCDIseBV wg+2yPcyVBx1zvumqWw3pKz4gOCfYtOu2Hqi1B5fsuiEZuzfMfHLpFoCLx/dQkxX0XuP HbRgyLHSydtIqpmMYszhOp32B772oHBEkYaMD2L4ZgVNPrBTaGhsJtUhRshe9J/iNb7F BDQ6CsRkPDaivdceDrJ7SMJaT6Bbuqk+w1vVoSk+Sxn8UimmYS1oeW+yCPpMlPElnb47 yxQQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=OoHW0QlT; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81626-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81626-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id jg19-20020a056102181300b0046ed3ee2283si488060vsb.390.2024.02.26.06.31.40 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 06:31:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-81626-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=OoHW0QlT; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81626-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81626-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 570EC1C2202B for ; Mon, 26 Feb 2024 14:31:40 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 22CB712BF1E; Mon, 26 Feb 2024 14:30:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="OoHW0QlT" Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4AF8712AAFD for ; Mon, 26 Feb 2024 14:30:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957814; cv=none; b=rndO+nmJsNVSKgLGiWhJvHGnjhKu/DZE5UyRjrEehcUeQpUs5N+zdVEo2J5oHfDBovxLNr0xdeMixUiKG6n5K761nWoNmFPRB3q1Ihn63kdPhPsG0looOqy7CqaaKmAhtCZRlcQsS1Ah5VOOSXzZHhgTpUeoyVUxmo/x+9FZzUw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957814; c=relaxed/simple; bh=Utd3yQPUC0h02QRLBQM2S7ZyErEnyX3W+vykofMJQFM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Q/gtesRNHsCZMLpcW7We0h2GIhi9GbXvYOb7anuI1ewQAjYAwk2Srbn6l5gWzC+BxgpFSJnWaFeKebk3kcOilMF1moiDcRKr7SanbzWx0bDThOIbQNw0eF8MtLHDdSH/X0rnHeV5uam2SAS/2nof7Bc999P47dqNqUsxAKZUyBg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=OoHW0QlT; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcc4563611cso5512061276.3 for ; Mon, 26 Feb 2024 06:30:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708957811; x=1709562611; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XsB2jfEOHuf9jjlesr7NzHbxyyk9Sr8XgRsNb8f+ZY8=; b=OoHW0QlTt+6ui2iCVYPBLT60E9F9CByU5XvLUZd5ublCIc8NAc28sOP39TW48T+bDE FSANlgvTTmtwUy/WW6mKWCCDX/x7bduQ6f+0f86+GaRoOONh6mNUoLYTfCP2jWLXCDrG gwYZcycXA+tSOen9u5hKa66iPPCHvQnjDciF96ep3n/6HSFFRfVAanmLBVrSHfTc89ZD C7j9gdVamb/hJu9VbuV2eno1/9UNkfJ0F2Yo0QL4v2C9MfVaCTNcXqr+s8DaG3C0XsSu /T6jNO+n8K506tnzad9UbvXsXoLcsew5NxQCf4gsp4EpziemDWJPBRq4xFABkUIsiUbD wTqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708957811; x=1709562611; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XsB2jfEOHuf9jjlesr7NzHbxyyk9Sr8XgRsNb8f+ZY8=; b=PlyywtjX+O8yqzh//nLALbH6WaH35UepzZNg0MZb92RIzrpeaAQ3sP7iVb8Pncqhbo d6/41EmAAfsN8gE19JUQ83n7GvZ8UcgrveswmrK8a/DeSo7dVZ6ZQYWBKtvNTqGDjKzj 5SyTbEkulTg6gMi+dnbzOLTKoPWS0PVNlyIRc1I6XNplCBHQJ25eTVZqeLtt7aJS6PL9 Udi1uRVJYSuOWKnXxZ0XqoyFpfO4aHEpI+XbiAHRvrJ5DyZHZaYkUym5odpB6ds1qUkC V6mvtpNaUKG4KJFERCH1wmVa/IGocw+Ft/HYvUe1796HJ8DVLzq2Yaj7ZFLRVTAi7BhN VVbg== X-Gm-Message-State: AOJu0YwHyYOKkINmmOVsvxp2iXKy8bZNLTJ7aSF40Xx/r7qYpuQ6OLiS Fb97FOJmIE54JelsvWeo+8b8+0BQTH5RjHBQq+BRoSmZsfPJg6pWFSbbFraagja43qQgKG2ZDGC in/IHr9so79/tzWbZDVQG7oc/VRSo/4sXTA8SRkiVTMhcyN2WC4y+OHPWeNrBqCrruHHnbt3Na5 o6Wqov0dB/mJ7deykSnYReH0gjbt3kEQ== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1009:b0:dc7:5aad:8965 with SMTP id w9-20020a056902100900b00dc75aad8965mr2159556ybt.0.1708957811244; Mon, 26 Feb 2024 06:30:11 -0800 (PST) Date: Mon, 26 Feb 2024 15:29:58 +0100 In-Reply-To: <20240226142952.64769-12-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240226142952.64769-12-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240226142952.64769-17-ardb+git@google.com> Subject: [PATCH v6 05/10] x86/sme: Avoid SME/SVE related checks on non-SME/SVE platforms From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791972239605257613 X-GMAIL-MSGID: 1791972239605257613 From: Ard Biesheuvel Reorganize the early SME/SVE init code so that SME/SVE related calls are deferred until it has been determined that the platform actually supports this, and so those calls could actually make sense. This removes logic from the early boot path that executes from the 1:1 mapping when booting a CONFIG_AMD_MEM_ENCRYPT=y kernel on a system that does not implement that (i.e., 99% of distro kernels) Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/mem_encrypt.h | 4 ++-- arch/x86/kernel/head64.c | 6 +++--- arch/x86/mm/mem_encrypt_identity.c | 8 +++----- 3 files changed, 8 insertions(+), 10 deletions(-) diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index b31eb9fd5954..b1437ba0b3b8 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -48,7 +48,7 @@ void __init sme_unmap_bootdata(char *real_mode_data); void __init sme_early_init(void); void __init sme_encrypt_kernel(struct boot_params *bp); -void __init sme_enable(struct boot_params *bp); +void sme_enable(struct boot_params *bp); int __init early_set_memory_decrypted(unsigned long vaddr, unsigned long size); int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long size); @@ -82,7 +82,7 @@ static inline void __init sme_unmap_bootdata(char *real_mode_data) { } static inline void __init sme_early_init(void) { } static inline void __init sme_encrypt_kernel(struct boot_params *bp) { } -static inline void __init sme_enable(struct boot_params *bp) { } +static inline void sme_enable(struct boot_params *bp) { } static inline void sev_es_init_vc_handling(void) { } diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index deaaea3280d9..f37278d1cf85 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -98,9 +98,6 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp, pmdv unsigned long vaddr, vaddr_end; int i; - /* Encrypt the kernel and related (if SME is active) */ - sme_encrypt_kernel(bp); - /* * Clear the memory encryption mask from the .bss..decrypted section. * The bss section will be memset to zero later in the initialization so @@ -108,6 +105,9 @@ static unsigned long __head sme_postprocess_startup(struct boot_params *bp, pmdv * attribute. */ if (sme_get_me_mask()) { + /* Encrypt the kernel and related */ + sme_encrypt_kernel(bp); + vaddr = (unsigned long)__start_bss_decrypted; vaddr_end = (unsigned long)__end_bss_decrypted; diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index 0166ab1780cc..7ddcf960e92a 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -45,6 +45,7 @@ #include #include #include +#include #include #include "mm_internal.h" @@ -502,18 +503,15 @@ void __init sme_encrypt_kernel(struct boot_params *bp) native_write_cr3(__native_read_cr3()); } -void __init sme_enable(struct boot_params *bp) +void __head sme_enable(struct boot_params *bp) { const char *cmdline_ptr, *cmdline_arg, *cmdline_on; unsigned int eax, ebx, ecx, edx; unsigned long feature_mask; unsigned long me_mask; char buffer[16]; - bool snp; u64 msr; - snp = snp_init(bp); - /* Check for the SME/SEV support leaf */ eax = 0x80000000; ecx = 0; @@ -546,7 +544,7 @@ void __init sme_enable(struct boot_params *bp) feature_mask = (msr & MSR_AMD64_SEV_ENABLED) ? AMD_SEV_BIT : AMD_SME_BIT; /* The SEV-SNP CC blob should never be present unless SEV-SNP is enabled. */ - if (snp && !(msr & MSR_AMD64_SEV_SNP_ENABLED)) + if (snp_init(bp) && !(msr & MSR_AMD64_SEV_SNP_ENABLED)) snp_abort(); /* Check if memory encryption is enabled */ From patchwork Mon Feb 26 14:29:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 206685 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2114310dyb; Mon, 26 Feb 2024 06:32:34 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUVdDxcKyqDKQKm2b2zWSbnBA+z1AGhpt4PLRvnyoHLE+MfZWm2mFFbaBTm4hMUd2Tv/sALZyGpEC5zg3TcGNXPaXFgJQ== X-Google-Smtp-Source: AGHT+IHzvULrIgZ17c4H2ZJqLVD1eDPNqdAqK0PhcJM4Fnt2ElIbkmeL7R4BNDYCH89aGp/WhR1g X-Received: by 2002:aa7:c3c8:0:b0:565:f90c:1263 with SMTP id l8-20020aa7c3c8000000b00565f90c1263mr1587049edr.9.1708957954360; Mon, 26 Feb 2024 06:32:34 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708957954; cv=pass; d=google.com; s=arc-20160816; b=KTUSwFLqDKBXafZGuBOJoxbSVcJUquOKncHkX1/VYF9euBdxtFX8i+YHWYsV0t14L6 giFguKLZ0jMd63DavnivIorEaH+uN94AZADYT+2OfjzA1rl51o6Bz290Mrs8h9XgUaVC qusssqdAZRpAzxNakvPi8VaWuyEy6cB2zHOPVnR+j9lZIjesEpggIBVCXc2pfQCa9hTF Bx+kMNnORnzTrl5hHE2SSxO4RFRuH45NMK6jN0UOykYpf9eup9WrOXKpkqAZoGJmKx+f dJ8IbnvXXWHHdhOZJjpaUxSCS06guHNnJLJisx+EZ5tcJ4iEL8jhGLPhF6BcnAA7JM0D 3/xg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=P75XCjLnB2v5ig5LKSiFYmfdvurR1I1hUffUvu8kLLQ=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=z9u7itQ0sZOPli6aW7P89AePnMApvl5lTjMhW7LOmCa75ptRSuLzLx5qTGdezXvjtT 0wIGkwThMEq7PdvrI5zkr0P1i3PQ+MElNN8WXDvYYqSNhjIgdMuUtMupUyNYZuvn/EFx XCEChmQinwWBYxjOlGOWcE1yB8DORpW388XqHz7Mztz25h68wcfHzemcrbvF2VgVfrUO wQOktHpD54Q+TND7yjeaWih3Ls9/JZ519asm21saz/t28teAr+pL4CfjwaUUBIjSQxS5 TPpBvKEiAWCAREkCNOMcH/MI0kEHKMMnzdo//r9tMcMS1Xi5DfkUGrGRgpz0KOp0LBuk tcpg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=RkVllO0y; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81628-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81628-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id cm9-20020a0564020c8900b00564cff54b86si2135622edb.511.2024.02.26.06.32.34 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 06:32:34 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-81628-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=RkVllO0y; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81628-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81628-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id B77CB1F2CC11 for ; Mon, 26 Feb 2024 14:31:55 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id EB87C12C7E2; Mon, 26 Feb 2024 14:30:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="RkVllO0y" Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA02F12BE9A for ; Mon, 26 Feb 2024 14:30:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957816; cv=none; b=svDtmr9luhJNWaLtZcHbWqHP6W+lM0zvF8IqM1GCAXQKc2R5xEfoZXR+gnqtpNuoxich7rbBfmgdVEsGnxUAWQL6qK6+ccDEvdA1L4DIrhgigUTB2HhpYV/aaqgsValEDHzUV0DkGsoXmTwOHMDkchO8qUZ9rSULhfioKQPI3U4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957816; c=relaxed/simple; bh=e/CqAzmvAxdJ7SZXVlU7ObmDh2K2WYB6MEyzEq28Se4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=G/34nLjQNCjp5pOTsB24ERR68bpbx2n71rJ/tDIECdWZiahV3Ow2KHYJlsvHydkuwTaizhndAHGGd3YctDPz56hBHdFG755GPnHVNwmuGg7cbMMHk41EyYTRfzwTHrKS+4e4EfSDY7ALhlrzaYgv4Yhw98yhLNwKcMTjVZ9yG98= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=RkVllO0y; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc6b26ce0bbso5303803276.1 for ; Mon, 26 Feb 2024 06:30:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708957814; x=1709562614; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=P75XCjLnB2v5ig5LKSiFYmfdvurR1I1hUffUvu8kLLQ=; b=RkVllO0yYRo8m+z55GwrpzIHoVKZpVvF8roY26eblnUDDqJLy7JP5gwkO9Z8zIHHYF FsF6WiOHbsFhacOqcY1txwBGeG6q9LIrDzJH9/ScWtflI2PK7gD0Nh039V0o3Fkjeq19 wCRfxxJXGpjO5qJT9PmMG9TXh3vLyGWq6xllpo6QLdqCHO7jqBVFQhiBgbCvO979ykWU q5jBKX/uzeqkP87605JJFUfozyKClaB25F8FhxOZlpfrWRAGpb1ca4uH9YeNwXnxQcHZ fWcXapVPNkAXugut3aFcPMz/5phkB6GF1EXsS5CwiR6Ro/OFWTOwrjbg3wm2VYWJjUM/ lY5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708957814; x=1709562614; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=P75XCjLnB2v5ig5LKSiFYmfdvurR1I1hUffUvu8kLLQ=; b=eTzzlpcoXTcbl8wxQWm9TQkRqttqnf+xdjyY7OqJmhsf+fG/xOUdjBkROYzTI3/T2y vZ1eRl/ENHAQBDWL1VnLBmH4pv3SfGgu2IY0LzYe1ozabFmo4TU1RtefTNkFmzjHXg96 mZerkC2/oQwmFbyW9Hc//6XNeiGiCuAxwMXY/b2AQEvcq7JJJe2eZJE/mlVIHVkyPax8 G/G2hocgY/gfpV8ljzZoJmk+wCSqDDC7yx3fHmVfq5QPJGf6SM9+wV2p/wKKTrtJEdcv Z1N/t9ceXzkT3iOdMj3pb5mtdZaTxTeom7hj/fyw5EoCzbjoLxgmTMzbE0kOkCCb/uUY uyBQ== X-Gm-Message-State: AOJu0YwRe6SkBggq6VhBQA+XbtANqjoG81Lh9OSWZl9381aHK6CgQ0Gf RM/9gBF/Zjwx8zPNsHIOyUWtgbcf0DyjTnvHuvC3/XQ33+GWUSNjOqZVc2ISpJoYY+3zzv4Istk PXt52MlcHR9GCFoiFV9KACet+nGoOoFxHg3t8qdx3bXbLacaptUCct7c/Q9IMoXC0FG1X9t3Q9g /+iIZ0QAvrcIIPnS05UHa3KIfXxbKrIw== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:69ce:0:b0:dc7:9218:df47 with SMTP id e197-20020a2569ce000000b00dc79218df47mr2124240ybc.5.1708957813722; Mon, 26 Feb 2024 06:30:13 -0800 (PST) Date: Mon, 26 Feb 2024 15:29:59 +0100 In-Reply-To: <20240226142952.64769-12-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240226142952.64769-12-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240226142952.64769-18-ardb+git@google.com> Subject: [PATCH v6 06/10] efi/libstub: Add generic support for parsing mem_encrypt= From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791972296009531652 X-GMAIL-MSGID: 1791972296009531652 From: Ard Biesheuvel Parse the mem_encrypt= command line parameter from the EFI stub if CONFIG_ARCH_HAS_MEM_ENCRYPT=y, so that it can be passed to the early boot code by the arch code in the stub. This avoids the need for the core kernel to do any string parsing very early in the boot. Signed-off-by: Ard Biesheuvel --- drivers/firmware/efi/libstub/efi-stub-helper.c | 8 ++++++++ drivers/firmware/efi/libstub/efistub.h | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/firmware/efi/libstub/efi-stub-helper.c b/drivers/firmware/efi/libstub/efi-stub-helper.c index bfa30625f5d0..3dc2f9aaf08d 100644 --- a/drivers/firmware/efi/libstub/efi-stub-helper.c +++ b/drivers/firmware/efi/libstub/efi-stub-helper.c @@ -24,6 +24,8 @@ static bool efi_noinitrd; static bool efi_nosoftreserve; static bool efi_disable_pci_dma = IS_ENABLED(CONFIG_EFI_DISABLE_PCI_DMA); +int efi_mem_encrypt; + bool __pure __efi_soft_reserve_enabled(void) { return !efi_nosoftreserve; @@ -75,6 +77,12 @@ efi_status_t efi_parse_options(char const *cmdline) efi_noinitrd = true; } else if (IS_ENABLED(CONFIG_X86_64) && !strcmp(param, "no5lvl")) { efi_no5lvl = true; + } else if (IS_ENABLED(CONFIG_ARCH_HAS_MEM_ENCRYPT) && + !strcmp(param, "mem_encrypt") && val) { + if (parse_option_str(val, "on")) + efi_mem_encrypt = 1; + else if (parse_option_str(val, "off")) + efi_mem_encrypt = -1; } else if (!strcmp(param, "efi") && val) { efi_nochunk = parse_option_str(val, "nochunk"); efi_novamap |= parse_option_str(val, "novamap"); diff --git a/drivers/firmware/efi/libstub/efistub.h b/drivers/firmware/efi/libstub/efistub.h index c04b82ea40f2..fc18fd649ed7 100644 --- a/drivers/firmware/efi/libstub/efistub.h +++ b/drivers/firmware/efi/libstub/efistub.h @@ -37,8 +37,8 @@ extern bool efi_no5lvl; extern bool efi_nochunk; extern bool efi_nokaslr; extern int efi_loglevel; +extern int efi_mem_encrypt; extern bool efi_novamap; - extern const efi_system_table_t *efi_system_table; typedef union efi_dxe_services_table efi_dxe_services_table_t; From patchwork Mon Feb 26 14:30:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 206683 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2114114dyb; Mon, 26 Feb 2024 06:32:18 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCX53xiNPxqbnbtZBsuT7vW3gVhzPnaW8KdPcqlFu2Y4IuXORl1WmHyv8INKqve7LB4FA7Jp3PVlhnItz2f6aqpuf3wbjQ== X-Google-Smtp-Source: AGHT+IFDf0Jo+3iUcaPFgQHOjuEOl29d0XrGeIPlvHLQ1RWYY9vc9h/7cCY66nClVpNtmdl812f1 X-Received: by 2002:a1f:4901:0:b0:4cd:20ea:35b1 with SMTP id w1-20020a1f4901000000b004cd20ea35b1mr3093043vka.2.1708957937814; Mon, 26 Feb 2024 06:32:17 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708957937; cv=pass; d=google.com; s=arc-20160816; b=xJaTz9GTpDQfdXpVe4Xc3SqW/ycNPeKVYIauw8aMH40+w6JJvQVKqFvQ3TcZ5Oiwa3 qdZTC7dcCqSCCzcUfpYWlsUu/51G5SOZhlcWffnCqx5gUwuT6UI7SQCCi3mhOriJQcsw Z4lV1esplY2rzp4ikeZwzsgqlRljy9ALXZLBx3Y8B4TP3awNr19hspWIUq8Ov2SS+j/t YPQ5FkuQTA/tn8rJfd/G68MT26U7bCi0+e4wvpiu6f4BzlACl8y4MMsedF8CMFVtJk/k NeWZd14o6oKDRjim07M7EbvOkawERdH/qOED/EFx+IAe95HTFURTqQ65Dgm5DinYZxPH b20g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=Fy8tET2K6LDamiTE0gAaW0kSto4HrKOnbqYQZfUBPUA=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=bUKw71PZOJrW7MboeYNXbJmuvgdTTgNY/hXJ1Kh1sUkIyWzvwqAvk+zIwJ/1JqKBAo ZcPbRPNF5QqDAl24JQayWvaJ3AyD9yeLbF+SCzk9tByFhS1UcSulj2Gew33euLtQBkkP y02LvarB27lRKHT5m4GKA2eok4mWStBoQSAL5Mnmc+rNfcHKIgxpsgTSqie6sD5E3snp 8TCCIqtF6cMLZCuEXK5leBYJZ3DYDtmhp3nenROCErIdYQuc/xrcLVIXVugtyDY/Yxzk MaKOOBsZEyxVw5mlP0wwRqLNf/Em76HwKfLl5thWosqsqgV8JXpzg2e1lR+S2E/tZVrj OMrQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=tSQd7RNj; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81629-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81629-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id ck17-20020a05612232d100b004b732036b15si522218vkb.8.2024.02.26.06.32.17 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 06:32:17 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-81629-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=tSQd7RNj; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81629-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81629-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 872CC1C25D87 for ; Mon, 26 Feb 2024 14:32:17 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7180560DCB; Mon, 26 Feb 2024 14:30:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="tSQd7RNj" Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6FF6512C53A for ; Mon, 26 Feb 2024 14:30:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957819; cv=none; b=qrTXvRvsa6UpV8IWCbVYLX+JojQc4cLYnEyy4R4ZLgMmIGNyDNQAnsHOHgWMIxQc2SFa+0ziDQHPftHUO00N6+cAfQ5MrpfJuFnpzujth9dS03PpYXAml51buEaYYBinOPeC4UI2sUGSNpR04IYQ3KDjUHQPsVBh8xdm+44Erow= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957819; c=relaxed/simple; bh=wyT52OsDK09PjTBuO+ZGHUjEriBzY6xk8cyGBo14qT0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=KjJtiAe/zO/Us9qa3VIRAsaDJsSWaYGWHwveiiKZqz8o9wyrIo4BbknAW2m6cuS/McaBMTzU58L8Ch/UYMBAsbbqm4CILjetnZEx7IWS7k/2IwPK4IFTr3KJKHY0B8cCFJbpNXN8Yd3zYsA+eUaMErcXZD9mlBsUgGe5UHZUXC0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=tSQd7RNj; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc64f63d768so4893978276.2 for ; Mon, 26 Feb 2024 06:30:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708957816; x=1709562616; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Fy8tET2K6LDamiTE0gAaW0kSto4HrKOnbqYQZfUBPUA=; b=tSQd7RNj9Dge8kN3BNdVRPP2jjAKFeCdqZZT0XOWfjjsROJ5G6gfDwGo+U6eGcqGwN LYQfbtNXsUIbrVft/jMhzLEBTN0emCYR7zhJgdemwrzN3BiWXdN6SRYBIi4HqxVVm66E 2rpTMiMkxsctghPro4DSxoQ0zFXC1TvIt+Xy8euTHhpfXPsQttf+LeN1jip6ezOprGli WV6Cxps8Di+US7iAVoWTAoZexjkTv74NdmNRpYFSQxckAtLU3pIpNouGVogWJe099I/F uKm8xjSUJ8KM2uB9YZFyASVMuNh9TSM7orrE+37lp+xU/DExrZ9Ol7cdBWPIXF5KMq7k Xqyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708957816; x=1709562616; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Fy8tET2K6LDamiTE0gAaW0kSto4HrKOnbqYQZfUBPUA=; b=WZUxXgtYxL0zJeF1gGZwqrsPrKVTXhYZf85llhJWf9IjKE9Qu4QtOLNQ2x+ezGYMIz yDF0JWRYvCy7Wue875cSeZ5saYVqriVVG2hXJyXJhbkyk0BST+IOc3bnZguQgQNDiinE gdxroaGz357xIuxkV5Dy28PLtUhFDVxIgOCCI7jJS47JZWVfuGbXgMisk6lPUPsXyIZ6 uojWGEXX6+uI/KohBYvIU7YM5ZG8iIsxNohIK5SfUF9qIw/IaxKWvnT3Nz8/YUMWsShv VR77uklMBtp3IsonxEXQHh5N9c0nlMaP3XEv0wZkHVy2qr49EMOAP4RSDDkyY5j5HvHQ ybEQ== X-Gm-Message-State: AOJu0YyQ/e6GH/Fw3si/vyXqYGR67GDXDajQRG6aiA/TPi03+bWkfcOW bSmxOkWiSvc9XrIUgQr7WFlj47RyL8/d3Iip57LA3g+Y4O2N/Yi8mp7X9OM//tWxg8gE7rAax/U kWRNRGpp6hg3riXnyeH10S+LTEJiGi6rD8RHp0D94iqYXsfDga4oHeS8miAxqqfurNBvjRULMPD n7jkCAVEXM0lISYF6IcsGptnuBbNBUXw== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1893:b0:dc6:fa35:b42 with SMTP id cj19-20020a056902189300b00dc6fa350b42mr2134857ybb.2.1708957816031; Mon, 26 Feb 2024 06:30:16 -0800 (PST) Date: Mon, 26 Feb 2024 15:30:00 +0100 In-Reply-To: <20240226142952.64769-12-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240226142952.64769-12-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240226142952.64769-19-ardb+git@google.com> Subject: [PATCH v6 07/10] x86/boot: Move mem_encrypt= parsing to the decompressor From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791972278636566377 X-GMAIL-MSGID: 1791972278636566377 From: Ard Biesheuvel The early SME/SEV code parses the command line very early, in order to decide whether or not memory encryption should be enabled, which needs to occur even before the initial page tables are created. This is problematic for a number of reasons: - this early code runs from the 1:1 mapping provided by the decompressor or firmware, which uses a different translation than the one assumed by the linker, and so the code needs to be built in a special way; - parsing external input while the entire kernel image is still mapped writable is a bad idea in general, and really does not belong in security minded code; - the current code ignores the built-in command line entirely (although this appears to be the case for the entire decompressor) Given that the decompressor/EFI stub is an intrinsic part of the x86 bootable kernel image, move the command line parsing there and out of the core kernel. This removes the need to build lib/cmdline.o in a special way, or to use RIP-relative LEA instructions in inline asm blocks. This involves a new xloadflag in the setup header to indicate that mem_encrypt=on appeared on the kernel command line. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/misc.c | 15 +++++++++ arch/x86/include/uapi/asm/bootparam.h | 1 + arch/x86/lib/Makefile | 13 -------- arch/x86/mm/mem_encrypt_identity.c | 32 ++------------------ drivers/firmware/efi/libstub/x86-stub.c | 3 ++ 5 files changed, 22 insertions(+), 42 deletions(-) diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c index bd6857a9f15a..408507e305be 100644 --- a/arch/x86/boot/compressed/misc.c +++ b/arch/x86/boot/compressed/misc.c @@ -371,6 +371,19 @@ unsigned long decompress_kernel(unsigned char *outbuf, unsigned long virt_addr, return entry; } +/* + * Set the memory encryption xloadflag based on the mem_encrypt= command line + * parameter, if provided. + */ +static void parse_mem_encrypt(struct setup_header *hdr) +{ + int on = cmdline_find_option_bool("mem_encrypt=on"); + int off = cmdline_find_option_bool("mem_encrypt=off"); + + if (on > off) + hdr->xloadflags |= XLF_MEM_ENCRYPTION; +} + /* * The compressed kernel image (ZO), has been moved so that its position * is against the end of the buffer used to hold the uncompressed kernel @@ -401,6 +414,8 @@ asmlinkage __visible void *extract_kernel(void *rmode, unsigned char *output) /* Clear flags intended for solely in-kernel use. */ boot_params_ptr->hdr.loadflags &= ~KASLR_FLAG; + parse_mem_encrypt(&boot_params_ptr->hdr); + sanitize_boot_params(boot_params_ptr); if (boot_params_ptr->screen_info.orig_video_mode == 7) { diff --git a/arch/x86/include/uapi/asm/bootparam.h b/arch/x86/include/uapi/asm/bootparam.h index 01d19fc22346..eeea058cf602 100644 --- a/arch/x86/include/uapi/asm/bootparam.h +++ b/arch/x86/include/uapi/asm/bootparam.h @@ -38,6 +38,7 @@ #define XLF_EFI_KEXEC (1<<4) #define XLF_5LEVEL (1<<5) #define XLF_5LEVEL_ENABLED (1<<6) +#define XLF_MEM_ENCRYPTION (1<<7) #ifndef __ASSEMBLY__ diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile index ea3a28e7b613..f0dae4fb6d07 100644 --- a/arch/x86/lib/Makefile +++ b/arch/x86/lib/Makefile @@ -14,19 +14,6 @@ ifdef CONFIG_KCSAN CFLAGS_REMOVE_delay.o = $(CC_FLAGS_FTRACE) endif -# Early boot use of cmdline; don't instrument it -ifdef CONFIG_AMD_MEM_ENCRYPT -KCOV_INSTRUMENT_cmdline.o := n -KASAN_SANITIZE_cmdline.o := n -KCSAN_SANITIZE_cmdline.o := n - -ifdef CONFIG_FUNCTION_TRACER -CFLAGS_REMOVE_cmdline.o = -pg -endif - -CFLAGS_cmdline.o := -fno-stack-protector -fno-jump-tables -endif - inat_tables_script = $(srctree)/arch/x86/tools/gen-insn-attr-x86.awk inat_tables_maps = $(srctree)/arch/x86/lib/x86-opcode-map.txt quiet_cmd_inat_tables = GEN $@ diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index 7ddcf960e92a..0180fbbcc940 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -43,7 +43,6 @@ #include #include -#include #include #include #include @@ -96,9 +95,6 @@ struct sme_populate_pgd_data { */ static char sme_workarea[2 * PMD_SIZE] __section(".init.scratch"); -static char sme_cmdline_arg[] __initdata = "mem_encrypt"; -static char sme_cmdline_on[] __initdata = "on"; - static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) { unsigned long pgd_start, pgd_end, pgd_size; @@ -505,11 +501,9 @@ void __init sme_encrypt_kernel(struct boot_params *bp) void __head sme_enable(struct boot_params *bp) { - const char *cmdline_ptr, *cmdline_arg, *cmdline_on; unsigned int eax, ebx, ecx, edx; unsigned long feature_mask; unsigned long me_mask; - char buffer[16]; u64 msr; /* Check for the SME/SEV support leaf */ @@ -549,6 +543,9 @@ void __head sme_enable(struct boot_params *bp) /* Check if memory encryption is enabled */ if (feature_mask == AMD_SME_BIT) { + if (!(bp->hdr.xloadflags & XLF_MEM_ENCRYPTION)) + return; + /* * No SME if Hypervisor bit is set. This check is here to * prevent a guest from trying to enable SME. For running as a @@ -568,31 +565,8 @@ void __head sme_enable(struct boot_params *bp) msr = __rdmsr(MSR_AMD64_SYSCFG); if (!(msr & MSR_AMD64_SYSCFG_MEM_ENCRYPT)) return; - } else { - /* SEV state cannot be controlled by a command line option */ - goto out; } - /* - * Fixups have not been applied to phys_base yet and we're running - * identity mapped, so we must obtain the address to the SME command - * line argument data using rip-relative addressing. - */ - asm ("lea sme_cmdline_arg(%%rip), %0" - : "=r" (cmdline_arg) - : "p" (sme_cmdline_arg)); - asm ("lea sme_cmdline_on(%%rip), %0" - : "=r" (cmdline_on) - : "p" (sme_cmdline_on)); - - cmdline_ptr = (const char *)((u64)bp->hdr.cmd_line_ptr | - ((u64)bp->ext_cmd_line_ptr << 32)); - - if (cmdline_find_option(cmdline_ptr, cmdline_arg, buffer, sizeof(buffer)) < 0 || - strncmp(buffer, cmdline_on, sizeof(buffer))) - return; - -out: RIP_REL_REF(sme_me_mask) = me_mask; physical_mask &= ~me_mask; cc_vendor = CC_VENDOR_AMD; diff --git a/drivers/firmware/efi/libstub/x86-stub.c b/drivers/firmware/efi/libstub/x86-stub.c index 99429bc4b0c7..0336ed175e67 100644 --- a/drivers/firmware/efi/libstub/x86-stub.c +++ b/drivers/firmware/efi/libstub/x86-stub.c @@ -884,6 +884,9 @@ void __noreturn efi_stub_entry(efi_handle_t handle, } } + if (efi_mem_encrypt > 0) + hdr->xloadflags |= XLF_MEM_ENCRYPTION; + status = efi_decompress_kernel(&kernel_entry); if (status != EFI_SUCCESS) { efi_err("Failed to decompress kernel\n"); From patchwork Mon Feb 26 14:30:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 206684 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2114216dyb; Mon, 26 Feb 2024 06:32:26 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXtoMkn8onlFBLOWbrK6422VceBycSdT/nyVtl8XnqkgPXwJssfTdP6zIYUEn9oK0lK0NwckDpv65R0Wq6oQ4ClfZ9sUQ== X-Google-Smtp-Source: AGHT+IEDQ+ZsMs/a3TFUphWVEN3BC6TeAP/L2mTD82HFmrm8rSj4oapdF0090TI2d8LpBur5/N4y X-Received: by 2002:a05:6808:201b:b0:3c1:5557:453d with SMTP id q27-20020a056808201b00b003c15557453dmr9446160oiw.27.1708957946495; Mon, 26 Feb 2024 06:32:26 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708957946; cv=pass; d=google.com; s=arc-20160816; b=GY9va4RZq8zRCSaJUZ5Xq1KTNVJ4/1CHaTeaIF3RviKE4nlJ8mPfJO28auddH1KRM/ VZikeYi4dRP5QgcXPHN9J1ZTKOsUlSUChYNdH22T0IR9w+hBCb2222MShSY6lJrf2TBa gGSD9V4Um+VwlepNzqGCGEXtZLSl1ShvSCiRLZ8n/3y02j+X8SIi3CTalTcyFfXQ9cBA +7iqycAeXaHMusD5c7d9yitCewgd+LxE5YttkSorqb8DZEu+Z9UlTx4vCWK4Hnm2FLyH 4w3rWUVye2RZGg8EzPQDH8dyoGZxo8CSUvYZva/3p/INN76xHR0N/h25MvSK+HIr8/bV XhAQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=aHBDJ41pn3iOna4rTXtxPzeJ6vfMVADf3iDfGB7g+R4=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=Jtjih811NtVuIoogaetr2ckpAul2J5bJ62IEoZYOizcYyrKs0aXy+oWi478b7UEq1C Jw2IC3j8mfBnds0FY0lhHeMBF65iCZKv9JVqMvX/sZE7OGytXWg1/KSbq2qVmFmi8B7j GH5P/rk5xSF2sDoL+bX5lb/e0RCdt7t7WL3vy2k/1yIrzWXUT7CiMMKIgTMj0pdpIqA+ D+0pn2m3He3Ln+PRuOFoC4D9A50GoOcfRQJbaZ2EbPS+yp53WRlOE8qyLebvwTA/vfe/ w0rfvbTm7BlKERHyjmVRZRaSOpFl52hIsiS/5E1QJR7NjyDY0sqqCEmTqf3UtDtIkt1f N3Dg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=2QEPb+Mj; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81630-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81630-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id z22-20020ab05656000000b007d8e55da2f0si518545uaa.135.2024.02.26.06.32.26 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 06:32:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-81630-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=2QEPb+Mj; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81630-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81630-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 3329F1C28918 for ; Mon, 26 Feb 2024 14:32:26 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1624C12CD89; Mon, 26 Feb 2024 14:30:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="2QEPb+Mj" Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 61FAB12C55D for ; Mon, 26 Feb 2024 14:30:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957821; cv=none; b=H2SltgSxdgHq3p+tuAZnjTpVJqcvHHhrE4BiQV/fSNJPXjbU5sTxUnlUicl7JBLVIJeUcWf73hcMHjaIke1Hic8zrdU3IBRpde0ueGQ3PIlB+PVFgg62XYBZexFvOZV448w+sivXm7qvA3qFdB5G/gRsnXqZvqdWKUZF87nPliQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957821; c=relaxed/simple; bh=WZPNjU+tKWbG1/7EwXfp1HjOJvLOdEPU1KuRuZ22CGg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Bm7V92jCUs1jbnVf2S6aLPSpeYfRE8yIOztToaqHwNR2tkjy1JchPuhRfNcCnGSOUt+uUPPpN/LMFNLUq8vLG9HKVOfuNgp/kdqKfiIvYjodMVXWv4p8dSc1WPQUjILJ4IBMeqClh74rqPqq9q9/HkxUWbzhOM5V//bvimEec74= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=2QEPb+Mj; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcd94cc48a1so4271653276.3 for ; Mon, 26 Feb 2024 06:30:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708957818; x=1709562618; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=aHBDJ41pn3iOna4rTXtxPzeJ6vfMVADf3iDfGB7g+R4=; b=2QEPb+MjnI56wEfb45TsVm3qQ9DWpl1NDS/j2MgF8PlRNmOKCcXGcvwpLz1e9I6sqx hO4m69w2NtRCo3MyTwaQwt8IVdXEWI6coOB4IG6/MgQaX0u4GEKgrdjnflL/ymJkZ+F0 vKUhSNIF4Zf6yt/NV8Q0I1dsujzFFzxL265oXueGWwufVlrqWrOEVfqH0fMkAlDzj9cd Z66lQLsSQ7+uT5rONPHqjvtH/enY17ms1I6xfcCj+d4gvGFX/I3MSWVJb3KdOdo3/cnL /KcuLqe3AujVzukS99Ns+GjDqzW8oGZeIqcCAA+/v9qr+9R48Qp4NhmQuXlkWCT0/amw P11w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708957818; x=1709562618; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aHBDJ41pn3iOna4rTXtxPzeJ6vfMVADf3iDfGB7g+R4=; b=kE0Uq60hCoHmPK/YF0EvU3jCE6qXVacDxd9eWOrobzfRtf/zYk9tBcn0ZqRdJGtPps kUfISCXbyxJib6Hxjtsk0Uo/y0MdiHi6T1rv2MhCCwOxALfNRgPWgHd3gnRmLAGg7Xu/ rkX+kGZmGmULDef7tcov/Uk9GmbWlnOPp5K8qXnsg6fxzsKI/ORF0QDjBnTLE9h2XDyP p3VQNsbch4yPXQ+FqCjuLEiHBk7rBj+Lblkx3auhjliifjnK7jm4AgboWrtVsBXP255I IRuoIbNT0XjBWAQNoOAQ9KKXbFtlx3wJ0xXJ6e+InzVcswuqzRBMqc+IFcUerytQ07VV qklg== X-Gm-Message-State: AOJu0Yz15EXecM9LadPmmSQCShzCWF6PFZDpFYiKRhjht7c9VGhy/L61 RJ+4Nxdfg4CbHnQg/LbbQ3ji0qZdeZPNVSOB7iUQ0I8R+LK2QhwczsUXIhyvnRGVadMIFo6qJnb fVWNHVnaGlqdhPP32BG0A42F4AgEtLfNnB4/s/uHRxSxUMPwI3PjyVHaHiJIGJ3DeNI5oOMs1YJ P5C0jkatCvkWzv4Lk2w3EGspQR3/a2bw== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1003:b0:dc6:e5e9:f3af with SMTP id w3-20020a056902100300b00dc6e5e9f3afmr2087407ybt.9.1708957818551; Mon, 26 Feb 2024 06:30:18 -0800 (PST) Date: Mon, 26 Feb 2024 15:30:01 +0100 In-Reply-To: <20240226142952.64769-12-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240226142952.64769-12-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240226142952.64769-20-ardb+git@google.com> Subject: [PATCH v6 08/10] x86/sme: Move early SME kernel encryption handling into .head.text From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791972287747087009 X-GMAIL-MSGID: 1791972287747087009 From: Ard Biesheuvel The .head.text section is the initial primary entrypoint of the core kernel, and is entered with the CPU executing from a 1:1 mapping of memory. Such code must never access global variables using absolute references, as these are based on the kernel virtual mapping which is not active yet at this point. Given that the SME startup code is also called from this early execution context, move it into .head.text as well. This will allow more thorough build time checks in the future to ensure that early startup code only uses RIP-relative references to global variables. Also replace some occurrences of __pa_symbol() [which relies on the compiler generating an absolute reference, which is not guaranteed] and an open coded RIP-relative access with RIP_REL_REF(). Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/mem_encrypt.h | 4 +- arch/x86/mm/mem_encrypt_identity.c | 40 ++++++++------------ 2 files changed, 18 insertions(+), 26 deletions(-) diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index b1437ba0b3b8..f922b682b9b4 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -47,7 +47,7 @@ void __init sme_unmap_bootdata(char *real_mode_data); void __init sme_early_init(void); -void __init sme_encrypt_kernel(struct boot_params *bp); +void sme_encrypt_kernel(struct boot_params *bp); void sme_enable(struct boot_params *bp); int __init early_set_memory_decrypted(unsigned long vaddr, unsigned long size); @@ -81,7 +81,7 @@ static inline void __init sme_unmap_bootdata(char *real_mode_data) { } static inline void __init sme_early_init(void) { } -static inline void __init sme_encrypt_kernel(struct boot_params *bp) { } +static inline void sme_encrypt_kernel(struct boot_params *bp) { } static inline void sme_enable(struct boot_params *bp) { } static inline void sev_es_init_vc_handling(void) { } diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index 0180fbbcc940..174a7192c9cb 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -41,6 +41,7 @@ #include #include +#include #include #include #include @@ -95,7 +96,7 @@ struct sme_populate_pgd_data { */ static char sme_workarea[2 * PMD_SIZE] __section(".init.scratch"); -static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) +static void __head sme_clear_pgd(struct sme_populate_pgd_data *ppd) { unsigned long pgd_start, pgd_end, pgd_size; pgd_t *pgd_p; @@ -110,7 +111,7 @@ static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) memset(pgd_p, 0, pgd_size); } -static pud_t __init *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) +static pud_t __head *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) { pgd_t *pgd; p4d_t *p4d; @@ -147,7 +148,7 @@ static pud_t __init *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) return pud; } -static void __init sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) +static void __head sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) { pud_t *pud; pmd_t *pmd; @@ -163,7 +164,7 @@ static void __init sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) set_pmd(pmd, __pmd(ppd->paddr | ppd->pmd_flags)); } -static void __init sme_populate_pgd(struct sme_populate_pgd_data *ppd) +static void __head sme_populate_pgd(struct sme_populate_pgd_data *ppd) { pud_t *pud; pmd_t *pmd; @@ -189,7 +190,7 @@ static void __init sme_populate_pgd(struct sme_populate_pgd_data *ppd) set_pte(pte, __pte(ppd->paddr | ppd->pte_flags)); } -static void __init __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) +static void __head __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) { while (ppd->vaddr < ppd->vaddr_end) { sme_populate_pgd_large(ppd); @@ -199,7 +200,7 @@ static void __init __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) } } -static void __init __sme_map_range_pte(struct sme_populate_pgd_data *ppd) +static void __head __sme_map_range_pte(struct sme_populate_pgd_data *ppd) { while (ppd->vaddr < ppd->vaddr_end) { sme_populate_pgd(ppd); @@ -209,7 +210,7 @@ static void __init __sme_map_range_pte(struct sme_populate_pgd_data *ppd) } } -static void __init __sme_map_range(struct sme_populate_pgd_data *ppd, +static void __head __sme_map_range(struct sme_populate_pgd_data *ppd, pmdval_t pmd_flags, pteval_t pte_flags) { unsigned long vaddr_end; @@ -233,22 +234,22 @@ static void __init __sme_map_range(struct sme_populate_pgd_data *ppd, __sme_map_range_pte(ppd); } -static void __init sme_map_range_encrypted(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_encrypted(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_ENC, PTE_FLAGS_ENC); } -static void __init sme_map_range_decrypted(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_decrypted(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_DEC, PTE_FLAGS_DEC); } -static void __init sme_map_range_decrypted_wp(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_decrypted_wp(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_DEC_WP, PTE_FLAGS_DEC_WP); } -static unsigned long __init sme_pgtable_calc(unsigned long len) +static unsigned long __head sme_pgtable_calc(unsigned long len) { unsigned long entries = 0, tables = 0; @@ -285,7 +286,7 @@ static unsigned long __init sme_pgtable_calc(unsigned long len) return entries + tables; } -void __init sme_encrypt_kernel(struct boot_params *bp) +void __head sme_encrypt_kernel(struct boot_params *bp) { unsigned long workarea_start, workarea_end, workarea_len; unsigned long execute_start, execute_end, execute_len; @@ -320,9 +321,8 @@ void __init sme_encrypt_kernel(struct boot_params *bp) * memory from being cached. */ - /* Physical addresses gives us the identity mapped virtual addresses */ - kernel_start = __pa_symbol(_text); - kernel_end = ALIGN(__pa_symbol(_end), PMD_SIZE); + kernel_start = (unsigned long)RIP_REL_REF(_text); + kernel_end = ALIGN((unsigned long)RIP_REL_REF(_end), PMD_SIZE); kernel_len = kernel_end - kernel_start; initrd_start = 0; @@ -339,14 +339,6 @@ void __init sme_encrypt_kernel(struct boot_params *bp) } #endif - /* - * We're running identity mapped, so we must obtain the address to the - * SME encryption workarea using rip-relative addressing. - */ - asm ("lea sme_workarea(%%rip), %0" - : "=r" (workarea_start) - : "p" (sme_workarea)); - /* * Calculate required number of workarea bytes needed: * executable encryption area size: @@ -356,7 +348,7 @@ void __init sme_encrypt_kernel(struct boot_params *bp) * pagetable structures for the encryption of the kernel * pagetable structures for workarea (in case not currently mapped) */ - execute_start = workarea_start; + execute_start = workarea_start = (unsigned long)RIP_REL_REF(sme_workarea); execute_end = execute_start + (PAGE_SIZE * 2) + PMD_SIZE; execute_len = execute_end - execute_start; From patchwork Mon Feb 26 14:30:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 206725 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2131372dyb; Mon, 26 Feb 2024 07:01:47 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXx2o9PYMRBkySoq+CBL0rwW+QC2IIBuUVBexd5PD/FQ161x6HtvblSPXtw6LAgher67FuXt3XL2PXEhiL1XkuQhaL9QA== X-Google-Smtp-Source: AGHT+IEiOGKQI+qsy/Dtpf+b63bBJY2ClLDL0eDbcF7OTyRrHdGQEpEiLmYsxLnfw/AGLc+9E/pH X-Received: by 2002:a17:902:e74e:b0:1db:e7dc:302f with SMTP id p14-20020a170902e74e00b001dbe7dc302fmr8566564plf.17.1708959707078; Mon, 26 Feb 2024 07:01:47 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708959707; cv=pass; d=google.com; s=arc-20160816; b=i86ijbRdNai2HA4Frg5x/gAUQMb1tfIQhcakD2ujnuMBrUfXnx8i90Yf2QdefIx2c4 d5sN5vvtFfBGO2iLLQjkaWSOLYSz9sQg5WjoHThBogv4WPYSCqPq6Q/Xi082UTcvttEE Gk4S5tK3Bx+mF2VfKjYZdi4CMiiAcXAxwv93f8UTq9D+MVW/hb8oXAiwwchMp+PP3Why IWUzi5YseX87zu1OwC/+ooyriT587lHOQWek1BW/yu3hqIfPqKjqJJ+izq47drBS/X/9 vPjm1+yQGHnX3YHUR+5KmRLfAMXaFudzfSljjRvDU7bFvpOjltOMnpvvzlqxg6o+B/kU uNbA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=/TXiDK1bbJQrKgG2noWrRi9jNDoQ/eaPvMqRiBLOTG4=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=cpDTgC1QzQOeaoS+9yWb3moYYQcZsr6XptfKeiJXW84WA6vh+Wn7WUdiDAuhC8k5/C kn9ic7w5ubIxdiwpLKuei1EoK+i8c6w6UHEGGzSjnsSkuL68upywi9M/F466uqiw/tR5 6GHzJ0t6VvdrJLFmNa86Bm9w86SbZIoVDyWyzELmMFKxsqeGuoDKFPepR0OmlZuCk9v9 xF3mfFiSj8Nw8hPFDhHnld2e1Y2paQHVMgOMYcV9mp62JlFLTO1qMM5JqUWHWPic9cSO uvzPxI6BHVEDEjoPKeiD8LZDTZpksoCqKdQ27oqWolZQ1JeAY3E1rc0xJHrzzisHQI95 /3WA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b="O/qpqrhV"; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81633-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81633-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id l3-20020a170903244300b001dc5cf7ab7bsi3891649pls.336.2024.02.26.07.01.46 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 07:01:47 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-81633-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b="O/qpqrhV"; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-81633-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-81633-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id CAA5BB27F60 for ; Mon, 26 Feb 2024 14:33:07 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4CB8612DD83; Mon, 26 Feb 2024 14:30:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="O/qpqrhV" Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46E2C12CD8E for ; Mon, 26 Feb 2024 14:30:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957827; cv=none; b=iOYDv0KSUTwfPolDJ9wz5NKO70LeMJSV+SGRofhrrQH5rKD1TvSzAmr53Ae+VmaYLGNogl/LLnB5W2PWr5aizliON5/abcdkKncADm73DE1kFovx84lTyJyfUKF5dLuYR1QCDz7c/YNomhVsXholZaIwqpE4eDcnlIYT73srNW4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708957827; c=relaxed/simple; bh=ygL+DuLObxVpRG3AaH/e/nRfBK1uDJORHrJVSpAA6HM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=fXNcWTmBzFbGpZqyESeWVd+0Kb8Cb0YfVh0ST5eIr7HKW/5QZ27r7gBNho8EpgjZrTrzn4/rkpf2mn5tMFI0PmTgdF5pnwT0LDv4S05lwRaDI+aWT8O0iQoOkBpPTKveK3wlErBpk+em/EEHO1XB2G7B7Jl3KPHwTkuNYgcdRvc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=O/qpqrhV; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-608d6e7314bso28517077b3.3 for ; Mon, 26 Feb 2024 06:30:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708957823; x=1709562623; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/TXiDK1bbJQrKgG2noWrRi9jNDoQ/eaPvMqRiBLOTG4=; b=O/qpqrhV87GK6Tu99IM/797Ky1liiYyqvPMS1U6xhJueJRnc5R2HeLJTawTSvfB/Co 3i8SCBaJBXMkd0J9pI+G4+UzcpmY5WVvr/VlzdITKNbjBXhbLkNWwIzaPEfrLo9aovBs BF6tiVgu7s3JOpezpDMC4tGQxr/Bdr9OMaUaDJlCYOCeDWQDYbvH3+SMyKBI0PLBIB/6 7AOLRnwaVebTgLaw0lZTlW5SWTP8KW5W7iRyFjW/b3VIYVQiMWGsL7POpXE27PFqeRbM MxuRoh4p+ARYUnIvedwXdsBmZ+8R7zqOYtnTmuG7aS0bB3IIHcPoeVSJHQ4pQQ8MPVmP x/og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708957823; x=1709562623; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/TXiDK1bbJQrKgG2noWrRi9jNDoQ/eaPvMqRiBLOTG4=; b=I/OhMRpcVtrRCypLt0iVz4FjX1KxSY27rwhjXCnQz0798JteXDF3R5MdOO3OWBZFkF BF7aIr1NvSIoduN8I4BotfUaf19XfHsAI16yKUpO2xcnNzoUymxNtKRCoNfrfyU062kE zMX6XeO5J23LE45d68iTxYVqiiqkcTz3szTbEArKmRj7KbXkhXqws70qZ2enUGEtmr5I 9vdJqdd4lxzojUhNYde7c+2roW1ck+9rJMfYfp6mptzNCjGKmPt/zdhIpeeiYpiTF7ML VXGROeGBKjZ8sr2JWgc7GMcHqyK8kV1gsXM1ZakvLnzAJdKpDzo7iznBmRTdIgZuDrwQ GIEg== X-Gm-Message-State: AOJu0Yy0D10iWoWMhLGTjS0mPSfg4PASuJ5pD5oulBL/0KzGHoWz4k9P tdLxWKciyzdbCyo3obi/Z7GFLMDV7+JraG7ZdsmdTnA2Hftg9R0x0pbHWgEAJMj0/CSB+ENktDa 4cbWtALOtsRjDZXS6ob0hNtXKQVA17IMgrW2P4xfhdi4c17YJtzHzZXKKzIVVHU3Nrs1i/e4Vwk zMHkvtg9cuQNgz/5E/zBfIV/hKmKWS9g== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a0d:e842:0:b0:608:c5d0:74c6 with SMTP id r63-20020a0de842000000b00608c5d074c6mr1392569ywe.2.1708957823299; Mon, 26 Feb 2024 06:30:23 -0800 (PST) Date: Mon, 26 Feb 2024 15:30:03 +0100 In-Reply-To: <20240226142952.64769-12-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240226142952.64769-12-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240226142952.64769-22-ardb+git@google.com> Subject: [PATCH v6 10/10] x86/startup_64: Drop global variables keeping track of LA57 state From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1791974133647991989 X-GMAIL-MSGID: 1791974133647991989 From: Ard Biesheuvel On x86_64, the core kernel is entered in long mode, which implies that paging is enabled. This means that the CR4.LA57 control bit is guaranteed to be in sync with the number of paging levels used by the kernel, and there is no need to store this in a variable. There is also no need to use variables for storing the calculations of pgdir_shift and ptrs_per_p4d, as they are easily determined on the fly. This removes the need for two different sources of truth for determining whether 5-level paging is in use: CR4.LA57 always reflects the actual state, and never changes from the point of view of the 64-bit core kernel. The only potential concern is the cost of CR4 accesses, which can be mitigated using alternatives patching based on feature detection. Note that even the decompressor does not manipulate any page tables before updating CR4.LA57, so it can also avoid the associated global variables entirely. However, as it does not implement alternatives patching, the associated ELF sections need to be discarded. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/misc.h | 4 -- arch/x86/boot/compressed/pgtable_64.c | 12 ------ arch/x86/boot/compressed/vmlinux.lds.S | 1 + arch/x86/include/asm/pgtable_64_types.h | 43 ++++++++++---------- arch/x86/kernel/cpu/common.c | 2 - arch/x86/kernel/head64.c | 33 +-------------- arch/x86/mm/kasan_init_64.c | 3 -- arch/x86/mm/mem_encrypt_identity.c | 9 ---- 8 files changed, 25 insertions(+), 82 deletions(-) diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h index b353a7be380c..e4ab7b4d8698 100644 --- a/arch/x86/boot/compressed/misc.h +++ b/arch/x86/boot/compressed/misc.h @@ -16,9 +16,6 @@ #define __NO_FORTIFY -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 - /* * Boot stub deals with identity mappings, physical and virtual addresses are * the same, so override these defines. @@ -181,7 +178,6 @@ static inline int count_immovable_mem_regions(void) { return 0; } #endif /* ident_map_64.c */ -extern unsigned int __pgtable_l5_enabled, pgdir_shift, ptrs_per_p4d; extern void kernel_add_identity_map(unsigned long start, unsigned long end); /* Used by PAGE_KERN* macros: */ diff --git a/arch/x86/boot/compressed/pgtable_64.c b/arch/x86/boot/compressed/pgtable_64.c index 51f957b24ba7..ae72f53f5e77 100644 --- a/arch/x86/boot/compressed/pgtable_64.c +++ b/arch/x86/boot/compressed/pgtable_64.c @@ -9,13 +9,6 @@ #define BIOS_START_MIN 0x20000U /* 128K, less than this is insane */ #define BIOS_START_MAX 0x9f000U /* 640K, absolute maximum */ -#ifdef CONFIG_X86_5LEVEL -/* __pgtable_l5_enabled needs to be in .data to avoid being cleared along with .bss */ -unsigned int __section(".data") __pgtable_l5_enabled; -unsigned int __section(".data") pgdir_shift = 39; -unsigned int __section(".data") ptrs_per_p4d = 1; -#endif - /* Buffer to preserve trampoline memory */ static char trampoline_save[TRAMPOLINE_32BIT_SIZE]; @@ -125,11 +118,6 @@ asmlinkage void configure_5level_paging(struct boot_params *bp, void *pgtable) native_cpuid_eax(0) >= 7 && (native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) { l5_required = true; - - /* Initialize variables for 5-level paging */ - __pgtable_l5_enabled = 1; - pgdir_shift = 48; - ptrs_per_p4d = 512; } /* diff --git a/arch/x86/boot/compressed/vmlinux.lds.S b/arch/x86/boot/compressed/vmlinux.lds.S index 083ec6d7722a..06358bb067fe 100644 --- a/arch/x86/boot/compressed/vmlinux.lds.S +++ b/arch/x86/boot/compressed/vmlinux.lds.S @@ -81,6 +81,7 @@ SECTIONS *(.dynamic) *(.dynsym) *(.dynstr) *(.dynbss) *(.hash) *(.gnu.hash) *(.note.*) + *(.altinstructions .altinstr_replacement) } .got.plt (INFO) : { diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm/pgtable_64_types.h index 9053dfe9fa03..2fac8ba9564a 100644 --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -6,7 +6,10 @@ #ifndef __ASSEMBLY__ #include +#include +#include #include +#include /* * These are used to make use of C type-checking.. @@ -21,28 +24,24 @@ typedef unsigned long pgprotval_t; typedef struct { pteval_t pte; } pte_t; typedef struct { pmdval_t pmd; } pmd_t; -extern unsigned int __pgtable_l5_enabled; - -#ifdef CONFIG_X86_5LEVEL -#ifdef USE_EARLY_PGTABLE_L5 -/* - * cpu_feature_enabled() is not available in early boot code. - * Use variable instead. - */ -static inline bool pgtable_l5_enabled(void) +static __always_inline __pure bool pgtable_l5_enabled(void) { - return __pgtable_l5_enabled; -} -#else -#define pgtable_l5_enabled() cpu_feature_enabled(X86_FEATURE_LA57) -#endif /* USE_EARLY_PGTABLE_L5 */ + unsigned long r; + bool ret; -#else -#define pgtable_l5_enabled() 0 -#endif /* CONFIG_X86_5LEVEL */ + if (!IS_ENABLED(CONFIG_X86_5LEVEL)) + return false; -extern unsigned int pgdir_shift; -extern unsigned int ptrs_per_p4d; + asm(ALTERNATIVE_TERNARY( + "movq %%cr4, %[reg] \n\t btl %[la57], %k[reg]" CC_SET(c), + %P[feat], "stc", "clc") + : [reg] "=&r" (r), CC_OUT(c) (ret) + : [feat] "i" (X86_FEATURE_LA57), + [la57] "i" (X86_CR4_LA57_BIT) + : "cc"); + + return ret; +} #endif /* !__ASSEMBLY__ */ @@ -53,7 +52,7 @@ extern unsigned int ptrs_per_p4d; /* * PGDIR_SHIFT determines what a top-level page table entry can map */ -#define PGDIR_SHIFT pgdir_shift +#define PGDIR_SHIFT (pgtable_l5_enabled() ? 48 : 39) #define PTRS_PER_PGD 512 /* @@ -61,7 +60,7 @@ extern unsigned int ptrs_per_p4d; */ #define P4D_SHIFT 39 #define MAX_PTRS_PER_P4D 512 -#define PTRS_PER_P4D ptrs_per_p4d +#define PTRS_PER_P4D (pgtable_l5_enabled() ? 512 : 1) #define P4D_SIZE (_AC(1, UL) << P4D_SHIFT) #define P4D_MASK (~(P4D_SIZE - 1)) @@ -76,6 +75,8 @@ extern unsigned int ptrs_per_p4d; #define PTRS_PER_PGD 512 #define MAX_PTRS_PER_P4D 1 +#define MAX_POSSIBLE_PHYSMEM_BITS 46 + #endif /* CONFIG_X86_5LEVEL */ /* diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 9e35e276c55a..d88e4be88868 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1,6 +1,4 @@ // SPDX-License-Identifier: GPL-2.0-only -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 #include #include diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index f37278d1cf85..a65babef6148 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -7,9 +7,6 @@ #define DISABLE_BRANCH_PROFILING -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include @@ -52,14 +49,6 @@ extern pmd_t early_dynamic_pgts[EARLY_DYNAMIC_PAGE_TABLES][PTRS_PER_PMD]; static unsigned int __initdata next_early_pgt; pmdval_t early_pmd_flags = __PAGE_KERNEL_LARGE & ~(_PAGE_GLOBAL | _PAGE_NX); -#ifdef CONFIG_X86_5LEVEL -unsigned int __pgtable_l5_enabled __ro_after_init; -unsigned int pgdir_shift __ro_after_init = 39; -EXPORT_SYMBOL(pgdir_shift); -unsigned int ptrs_per_p4d __ro_after_init = 1; -EXPORT_SYMBOL(ptrs_per_p4d); -#endif - #ifdef CONFIG_DYNAMIC_MEMORY_LAYOUT unsigned long page_offset_base __ro_after_init = __PAGE_OFFSET_BASE_L4; EXPORT_SYMBOL(page_offset_base); @@ -78,21 +67,6 @@ static struct desc_struct startup_gdt[GDT_ENTRIES] __initdata = { [GDT_ENTRY_KERNEL_DS] = GDT_ENTRY_INIT(DESC_DATA64, 0, 0xfffff), }; -static inline bool check_la57_support(void) -{ - if (!IS_ENABLED(CONFIG_X86_5LEVEL)) - return false; - - /* - * 5-level paging is detected and enabled at kernel decompression - * stage. Only check if it has been enabled there. - */ - if (!(native_read_cr4() & X86_CR4_LA57)) - return false; - - return true; -} - static unsigned long __head sme_postprocess_startup(struct boot_params *bp, pmdval_t *pmd) { unsigned long vaddr, vaddr_end; @@ -155,7 +129,7 @@ unsigned long __head __startup_64(unsigned long physaddr, bool la57; int i; - la57 = check_la57_support(); + la57 = pgtable_l5_enabled(); /* Is the address too large? */ if (physaddr >> MAX_PHYSMEM_BITS) @@ -440,10 +414,7 @@ asmlinkage __visible void __init __noreturn x86_64_start_kernel(char * real_mode (__START_KERNEL & PGDIR_MASK))); BUILD_BUG_ON(__fix_to_virt(__end_of_fixed_addresses) <= MODULES_END); - if (check_la57_support()) { - __pgtable_l5_enabled = 1; - pgdir_shift = 48; - ptrs_per_p4d = 512; + if (pgtable_l5_enabled()) { page_offset_base = __PAGE_OFFSET_BASE_L5; vmalloc_base = __VMALLOC_BASE_L5; vmemmap_base = __VMEMMAP_BASE_L5; diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c index 0302491d799d..85ae1ef840cc 100644 --- a/arch/x86/mm/kasan_init_64.c +++ b/arch/x86/mm/kasan_init_64.c @@ -2,9 +2,6 @@ #define DISABLE_BRANCH_PROFILING #define pr_fmt(fmt) "kasan: " fmt -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index 174a7192c9cb..3da15e9a8c7d 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -27,15 +27,6 @@ #undef CONFIG_PARAVIRT_XXL #undef CONFIG_PARAVIRT_SPINLOCKS -/* - * This code runs before CPU feature bits are set. By default, the - * pgtable_l5_enabled() function uses bit X86_FEATURE_LA57 to determine if - * 5-level paging is active, so that won't work here. USE_EARLY_PGTABLE_L5 - * is provided to handle this situation and, instead, use a variable that - * has been set by the early boot code. - */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include