From patchwork Tue Feb 27 15:19:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 207274 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2765607dyb; Tue, 27 Feb 2024 07:19:52 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCV2OONfGfmMm/f518EDgv++tCB/eNo3Pf6IHXrc+9vyPLVQUJuqkApR2o31AttRy/GCHlvq9PecSXxbjFM6V+bhbo3jSQ== X-Google-Smtp-Source: AGHT+IFEyMEGgg1VJkoSotn6/1/Zb93xxjCglrEl+nvKjKSRCqxONnwbVtMZfl9mf9Z1MoZAX4ea X-Received: by 2002:a17:906:36cd:b0:a3e:f79e:ce56 with SMTP id b13-20020a17090636cd00b00a3ef79ece56mr6513496ejc.45.1709047192136; Tue, 27 Feb 2024 07:19:52 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709047192; cv=pass; d=google.com; s=arc-20160816; b=Blpw0ttFgoOHM72aUiyPxeh0Pthq/X0MM/6kwQSN7P9MDFjcvIYzXYlSkq2dpQ/Bod tqznfXpxgqTHcSB8kOImJ2pofE3Ytc6DXfQBmEbbv+RFKtC7mUS12MbrRLJ5nzyu0z7z EmktcdSiii+PK5r5dCYxv0vWpQm3O8MoJvgrB8uqOWs8QK9CDBRTmUS5ogd6xpJhcxE5 7RhMpsZfItqMzjuGgMq0PT5UU0/8kcQGLwRcy1+rKrY8AgjpqS1rE8ThU9VFXBED5+Ey ioFnG9xH7DUvyoXyIw2FPrpiKcJP/f2i5YkbREZpKsz75dgzLcTUCi5JW+ZmbmslfPIC F9Gg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=V+kR9nuu+f3rXEtZJvvepNzOYr6GJN2sdntJZoeWS+k=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=OkbWlPw5as1YNqyIRHOLmPwE/+WHnMDFpo5Uz6/SDK8Gz6M1ScLSIYFCPdLj31acdh XUmf7D1nfp0n8wgpSXzsZ8MjwpX1bGSLMXjSPGOBOXjl6Dp002/Hk35uXRv7eeiG6aTZ G+vL1hwLc0AiIkUStQ4wpwmE3MvWdviXEO9ce+EMqee+cMgTwbr4rQMtn3crOxVt4ZQ+ esa39+YFWTxODLyFuxJrFLo7Y48zaYY3THdt59KE+gEMxI+SzECbWyogAboRXjeCkfzQ WBzi8GwAyktVBk5Rdoc/CXUOBtGqwfXCGuNgsoBwSXdAYNpupY7rk2tFiToUhFcfAl5l +tBw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=0B8jN3Gb; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83487-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83487-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id s9-20020a170906454900b00a3f43e48052si787607ejq.525.2024.02.27.07.19.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 07:19:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-83487-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=0B8jN3Gb; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83487-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83487-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 9177B1F231CF for ; Tue, 27 Feb 2024 15:19:51 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4D3C4145FEF; Tue, 27 Feb 2024 15:19:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="0B8jN3Gb" Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 25CA41420B3 for ; Tue, 27 Feb 2024 15:19:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047167; cv=none; b=YF6MYRpNPMgt5or1Zeehll2AC+Ko+Wy2WQXqx08B3BlvSD8NSYh47NphqwrN56ix/quK9Vab/WIqCvMWYU7OqQ/m3n0OSlL1+B97KE1+VEa8yESjHJdVtzV9mbIum6xqWAFYBvZVm5eks1rSKN6I2IxZmFw/GjGiOzD+wFgfxy8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047167; c=relaxed/simple; bh=CBHnFGty3UwF5woU9FH0wlEAG8G1OTGQbjOXf2dVQ3Q=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Ht1jKxsAeGE6Tz1VN1VeBTqAnLnS17Ulkf6l88kHAq+1QSFgGzUEeDLAmlIEyx9xZjMvkeumNasKT1NkVIcfKb+tsXzEyFk+0wZn3K8fs4OHgVy02NxRaeXU6njgQG/vB/1iEdnmH6wM3UrJoTbyS+72dRYGTawheiQ65pGsTeQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=0B8jN3Gb; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcd94cc48a1so6246143276.3 for ; Tue, 27 Feb 2024 07:19:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709047165; x=1709651965; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=V+kR9nuu+f3rXEtZJvvepNzOYr6GJN2sdntJZoeWS+k=; b=0B8jN3GbRsIqivMwq0YSRtTau2HHIC4XYnHolHZfqPKfiacyigtI0UZ1tNL0He350u aQIihibi9DM7mfR9SSl/dgE3DbulMGwApuboZ/tgENVGeAB07oV7mYKk59/IQRS+1Mdf vlwpGvD/0lWAgT+OeBQDd4aVyk9R1O01zomOP7jOv3OK+mMPWZvTXasMA12LOL1WD4FN 1rc1EEN71h4YrbbW4+S7CQ/ll9x92Tv8FEECEd/k5PJcZa9aPPZDRPugshnFA8rRzraw 2/Kwo+k0u9DHbvLg8B3ngyYiQDEgw8RrNKaCE4+Cy4Yh3J+gBopT4KhMtdZPzETDPjCQ VGWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709047165; x=1709651965; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=V+kR9nuu+f3rXEtZJvvepNzOYr6GJN2sdntJZoeWS+k=; b=KEzkvMsRC97iJP59xMEPnaRybBh3DcDp6cHPpbrKY83tOPzB9iw8Dt8L+PcnGzmtF+ uFEq0oxVkPggDHt103swYqF49KJYP26EtkjcRKwOUkPaono2emeRU4Vdl/CtOcPo2VMd XN6UUpAUq/Wmtrbc0vhMLs5rkIZtMNgIL6xzuPoR98Xp9x/b93DXLMGfzEKWysUAkTQ3 L0VqiKLIPvF0CgY9AGzEXmlkV8MYJWhxxCzOcnrJEem8KEr81BnX94wdwIVyATTbC+Iz CrWAEQLyqa4x8y09BbwsFAPDj/rpzbP6aJ7x0pThx2YQKxlJ34w97SybaDdxhGKMgfwU mPhA== X-Gm-Message-State: AOJu0Yx2UqeiAg8Tc8DqfruJc890ttn3K88vLYDjVGha8cFxJQvm/yNC JOczAfn/UM30krQ5hHvEb9ThKUzQPZ4kLgRzwsHxYBViiLcRqug9EbDKQZ5u48MucnbHBiuGlqv +CyhnHxzvYGzdpq3HKE2b9bwUylW23VeULyiunqEhpteQaQFMU7Ek+9hqVVI3as5KQDZ3upX3JJ Od5mZEpYGj4ICm3QEvpK1kLRbdmlEoew== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:4ac1:0:b0:dcf:b5b8:f825 with SMTP id x184-20020a254ac1000000b00dcfb5b8f825mr178068yba.0.1709047165048; Tue, 27 Feb 2024 07:19:25 -0800 (PST) Date: Tue, 27 Feb 2024 16:19:09 +0100 In-Reply-To: <20240227151907.387873-11-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240227151907.387873-11-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc1.240.g4c46232300-goog Message-ID: <20240227151907.387873-12-ardb+git@google.com> Subject: [PATCH v7 1/9] x86/startup_64: Simplify CR4 handling in startup code From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792065868860924965 X-GMAIL-MSGID: 1792065868860924965 From: Ard Biesheuvel When paging is enabled, the CR4.PAE and CR4.LA57 control bits cannot be changed, and so they can simply be preserved rather than reason about whether or not they need to be set. CR4.MCE should be preserved unless the kernel was built without CONFIG_X86_MCE, in which case it must be cleared. CR4.PSE should be set explicitly, regardless of whether or not it was set before. CR4.PGE is set explicitly, and then cleared and set again after programming CR3 in order to flush TLB entries based on global translations. This makes the first assignment redundant, and can therefore be omitted. So clear PGE by omitting it from the preserve mask, and set it again explicitly after switching to the new page tables. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/head_64.S | 30 ++++++++------------ 1 file changed, 12 insertions(+), 18 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index d295bf68bf94..1b054585bfd1 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -185,6 +185,11 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) addq $(init_top_pgt - __START_KERNEL_map), %rax 1: + /* + * Create a mask of CR4 bits to preserve. Omit PGE in order to clean + * global 1:1 translations from the TLBs. + */ + movl $(X86_CR4_PAE | X86_CR4_LA57), %edx #ifdef CONFIG_X86_MCE /* * Preserve CR4.MCE if the kernel will enable #MC support. @@ -193,20 +198,13 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) * configured will crash the system regardless of the CR4.MCE value set * here. */ - movq %cr4, %rcx - andl $X86_CR4_MCE, %ecx -#else - movl $0, %ecx + orl $X86_CR4_MCE, %edx #endif + movq %cr4, %rcx + andl %edx, %ecx - /* Enable PAE mode, PSE, PGE and LA57 */ - orl $(X86_CR4_PAE | X86_CR4_PSE | X86_CR4_PGE), %ecx -#ifdef CONFIG_X86_5LEVEL - testb $1, __pgtable_l5_enabled(%rip) - jz 1f - orl $X86_CR4_LA57, %ecx -1: -#endif + /* Even if ignored in long mode, set PSE uniformly on all logical CPUs. */ + btsl $X86_CR4_PSE_BIT, %ecx movq %rcx, %cr4 /* Setup early boot stage 4-/5-level pagetables. */ @@ -223,14 +221,10 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) movq %rax, %cr3 /* - * Do a global TLB flush after the CR3 switch to make sure the TLB - * entries from the identity mapping are flushed. + * Set CR4.PGE to re-enable global translations. */ - movq %cr4, %rcx - movq %rcx, %rax - xorq $X86_CR4_PGE, %rcx + btsl $X86_CR4_PGE_BIT, %ecx movq %rcx, %cr4 - movq %rax, %cr4 /* Ensure I am executing from virtual addresses */ movq $1f, %rax From patchwork Tue Feb 27 15:19:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 207275 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2765764dyb; Tue, 27 Feb 2024 07:20:04 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWiKqq0qfdvScQAC3uTAobI5bhT50/bVkeG7p7e0z7KPWQtPWr79FK2ObhVibr3R7XRUA009W1ROTCVvY83Hs8f45Biag== X-Google-Smtp-Source: AGHT+IGMvXjB/PbUINszNnTIZ0r5hKxPx0e10gV6gCbZTC/zxmvNe8wlZ9RFQjiUDnrALkL4M0ts X-Received: by 2002:a05:6214:f6f:b0:68c:c0e9:1ea5 with SMTP id iy15-20020a0562140f6f00b0068cc0e91ea5mr2510462qvb.46.1709047203905; Tue, 27 Feb 2024 07:20:03 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709047203; cv=pass; d=google.com; s=arc-20160816; b=zgMmxu4CFPvlWT13dg35MdyQ72YC2KAey0VHcaebME1M7FpLsxSd1ZV0dvF8WyIuQ6 abKTamE1j1OaDi98VkcQz3uqLQUVo4b4oxWpfWsmEmJHesJepNr2lUGwMlmGC/ySfcos LR+55KH7AOn4UD946trdttTr8qd6Yda0eVn1bT1fNVC1gTwXvA4A1f9L/EQO7SbcXS2o cu8x5u6vl5/PkPlZ9EoUVi/aJsXUzmO/Hk+456UxtJc+soJljiACTg8Ht+kZoe/rS22C G5bbsd5MnA0macVDaKLXaKIl0EzlSvp7aApDDCqOKPFHCfbY8p7WM3m3Q01hYjGQPPQz eTvA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=Y5GeRQDtmrQKN4psfZmd6RkmjsdFQnHB8sOV6dww9xU=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=yoDVA/goCGFZ6DU+TFU7v07u5wW+D+3BS8uHEQjHfUFrS5yfAEaaMNLS81zERcFP6g iqv4EBIufyxudDEiybxM0C85ArlP+U8mgbd0SjuUUiEzcmncpne/jpJq/xE3eRHUv20D LOlT7lBlNJFjoCIwj9kO9wKjz0mRwbWUBRtYUV8I+9uFt3hv0ulW+i6hwGPZZrH0V+57 g1trCGVBjIEos/Pd1tsa8HP5SMSoSbwXAM1pK+AmU95JTd2l6KWM52H/Lyjrx40/eiUC M7npULAz+0+e5dr4CpeAnljE6xenlWQl/3bAa2q8hcAG5HQGrKLwb9sUNTtSns1b3q7x 5WlA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=rReYyReM; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83488-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83488-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id jl14-20020ad45e8e000000b0068f632a4411si7658892qvb.457.2024.02.27.07.20.03 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 07:20:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-83488-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=rReYyReM; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83488-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83488-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id ADFBE1C23429 for ; Tue, 27 Feb 2024 15:20:03 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 3B2FC1468F1; Tue, 27 Feb 2024 15:19:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="rReYyReM" Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B8987145356 for ; Tue, 27 Feb 2024 15:19:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047170; cv=none; b=jSGOftHebmwxT8ci03GUvVUkXqPKB593GNMlpfPJpoK/unejq6cQik2Kubb2ghIlp+zpRSlJ3kM0wXJyc9XD9LmH9bAODvSkgq/ilssKb0FFLvkSCYjOGRZNW5n9/clwyRQX/TrvDJ4+nUblXyoi2bIIdkssG/JX3ABBQmHeeng= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047170; c=relaxed/simple; bh=X6/OzZGiJqFpxqpC0njbNWYz6ODH6zW8Z1ABMSbDTS4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=WUMB0nV7stk+Zi11rW4hAiRijIYxK+Ypwu+o7m+mVjKkS3JHgtbxXghZqOKBrn3XBNZb2YBs0RpWfdW3/0kJnLmWjGHUpmq4PElhfwtRCRrKRe8n2HfJfZamSvLAOgnxZ+KMq2diWHBIdxig7Z4NTejWy+gWkaKVDIGVLw8asw4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=rReYyReM; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dbf618042daso6691509276.0 for ; Tue, 27 Feb 2024 07:19:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709047168; x=1709651968; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Y5GeRQDtmrQKN4psfZmd6RkmjsdFQnHB8sOV6dww9xU=; b=rReYyReMccWVAE3OSLOF4pa/aTGkPQB/+kIIUW8bT2A32mF44Lr9B0Nh+ZMfOTYp0z OashlJ/oTPlgiRqm3CwTurtEh/DoFiSzfDUF8IPhboblwkqzhZUeroaN4xtOYQ9O9KtK bRAtnprD3e9fRLE3nmFCG+EmPY9cg5YVzXwpDHPxukI1kAvE35mMOOqeQ32aBr8dB3DI aH51/PPmOynLegTSxagLpEj8yDc33B0xMlWXcbI+n2IcRZO3d/XCNhem1lPsiw3H2DAY uOxVbwrDBExy9VHQCRSjEClAChX6lawqkPprZytm77XcuL3m35K2OvJ0SRkjOKxNf5sA l85A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709047168; x=1709651968; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Y5GeRQDtmrQKN4psfZmd6RkmjsdFQnHB8sOV6dww9xU=; b=qcxQgphbD09koAcBK3YUD8NolTNkJqXdVEbNtUbK6rX13pjN//YJJnlja0xKT0WzxN 0d1XBZ8lHoEJyGg7ArTqKGhhAyX7u1p2gPT5ArgS5pK4T1c6Yt4DIGxTzjsuSCDv1T0X RKQbqccHI/hf+td3O2Clpkfk3uYDRAU7CW9OcAo9sSC0NyjDyckNmQsvaIQP9FlqtxtL glvkvoy2lt0MgoZLXfoIxEZc8WxNm0smST6fpLj5BOtP6lk19T7WtshWtuA7G32KBBCO I342wbEZq8ADnEOBmds+TevycyMZYszgGBOpE+QG5nk5BY8lmNrUOHW1adDK2HxmlqqR ySHw== X-Gm-Message-State: AOJu0YzmJCBUhTi9Xo7Q3+FfluiSfeLON3P92alWy1Z6QCv5QazNlNim p4V4LHqSQftP7jwuqGkQGekbVbhgKMM0Q7u0ugYNikDNTF3OFkYBVxaDT2GDmpIZNrDwGYaKsq/ L2qZnahMRjQ4XVed1gqfpffcMrs0+nJt/HSc5rM8d7Ac0m48u7+gZ4QlyT0kW9VQKdQXypHBg3L LiRmBpzDswnK2o3ArSEvYGWU9GB/XAlg== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1005:b0:dc2:5273:53f9 with SMTP id w5-20020a056902100500b00dc2527353f9mr134018ybt.1.1709047167684; Tue, 27 Feb 2024 07:19:27 -0800 (PST) Date: Tue, 27 Feb 2024 16:19:10 +0100 In-Reply-To: <20240227151907.387873-11-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240227151907.387873-11-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc1.240.g4c46232300-goog Message-ID: <20240227151907.387873-13-ardb+git@google.com> Subject: [PATCH v7 2/9] x86/startup_64: Defer assignment of 5-level paging global variables From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792065880931845528 X-GMAIL-MSGID: 1792065880931845528 From: Ard Biesheuvel Assigning the 5-level paging related global variables from the earliest C code using explicit references that use the 1:1 translation of memory is unnecessary, as the startup code itself does not rely on them to create the initial page tables, and this is all it should be doing. So defer these assignments to the primary C entry code that executes via the ordinary kernel virtual mapping. Signed-off-by: Ard Biesheuvel Reviewed-by: Borislav Petkov (AMD) --- arch/x86/include/asm/pgtable_64_types.h | 2 +- arch/x86/kernel/head64.c | 44 +++++++------------- 2 files changed, 15 insertions(+), 31 deletions(-) diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm/pgtable_64_types.h index 38b54b992f32..9053dfe9fa03 100644 --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -21,9 +21,9 @@ typedef unsigned long pgprotval_t; typedef struct { pteval_t pte; } pte_t; typedef struct { pmdval_t pmd; } pmd_t; -#ifdef CONFIG_X86_5LEVEL extern unsigned int __pgtable_l5_enabled; +#ifdef CONFIG_X86_5LEVEL #ifdef USE_EARLY_PGTABLE_L5 /* * cpu_feature_enabled() is not available in early boot code. diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index 72351c3121a6..deaaea3280d9 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -23,6 +23,7 @@ #include #include +#include #include #include #include @@ -77,24 +78,11 @@ static struct desc_struct startup_gdt[GDT_ENTRIES] __initdata = { [GDT_ENTRY_KERNEL_DS] = GDT_ENTRY_INIT(DESC_DATA64, 0, 0xfffff), }; -#ifdef CONFIG_X86_5LEVEL -static void __head *fixup_pointer(void *ptr, unsigned long physaddr) -{ - return ptr - (void *)_text + (void *)physaddr; -} - -static unsigned long __head *fixup_long(void *ptr, unsigned long physaddr) +static inline bool check_la57_support(void) { - return fixup_pointer(ptr, physaddr); -} - -static unsigned int __head *fixup_int(void *ptr, unsigned long physaddr) -{ - return fixup_pointer(ptr, physaddr); -} + if (!IS_ENABLED(CONFIG_X86_5LEVEL)) + return false; -static bool __head check_la57_support(unsigned long physaddr) -{ /* * 5-level paging is detected and enabled at kernel decompression * stage. Only check if it has been enabled there. @@ -102,21 +90,8 @@ static bool __head check_la57_support(unsigned long physaddr) if (!(native_read_cr4() & X86_CR4_LA57)) return false; - *fixup_int(&__pgtable_l5_enabled, physaddr) = 1; - *fixup_int(&pgdir_shift, physaddr) = 48; - *fixup_int(&ptrs_per_p4d, physaddr) = 512; - *fixup_long(&page_offset_base, physaddr) = __PAGE_OFFSET_BASE_L5; - *fixup_long(&vmalloc_base, physaddr) = __VMALLOC_BASE_L5; - *fixup_long(&vmemmap_base, physaddr) = __VMEMMAP_BASE_L5; - return true; } -#else -static bool __head check_la57_support(unsigned long physaddr) -{ - return false; -} -#endif static unsigned long __head sme_postprocess_startup(struct boot_params *bp, pmdval_t *pmd) { @@ -180,7 +155,7 @@ unsigned long __head __startup_64(unsigned long physaddr, bool la57; int i; - la57 = check_la57_support(physaddr); + la57 = check_la57_support(); /* Is the address too large? */ if (physaddr >> MAX_PHYSMEM_BITS) @@ -465,6 +440,15 @@ asmlinkage __visible void __init __noreturn x86_64_start_kernel(char * real_mode (__START_KERNEL & PGDIR_MASK))); BUILD_BUG_ON(__fix_to_virt(__end_of_fixed_addresses) <= MODULES_END); + if (check_la57_support()) { + __pgtable_l5_enabled = 1; + pgdir_shift = 48; + ptrs_per_p4d = 512; + page_offset_base = __PAGE_OFFSET_BASE_L5; + vmalloc_base = __VMALLOC_BASE_L5; + vmemmap_base = __VMEMMAP_BASE_L5; + } + cr4_init_shadow(); /* Kill off the identity-map trampoline */ From patchwork Tue Feb 27 15:19:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 207276 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2766071dyb; Tue, 27 Feb 2024 07:20:31 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVeuUkMcLbPiqsLxdNaFPrQ99HXEyDNBVYQvvDhzadrmMVRB6ejE44g+sspOhynmccDi9qBf83RzoZe9D6/M9p6YIEIXQ== X-Google-Smtp-Source: AGHT+IEGd4SMVcDXtd+A4xe40SsXOTMvzqkx6SdBS85/sYDEx2nxP6/9GG9Yl/9XIsbDgu+VW7OF X-Received: by 2002:a05:6870:d1d1:b0:21e:f5a7:5b7 with SMTP id b17-20020a056870d1d100b0021ef5a705b7mr12171709oac.29.1709047231180; Tue, 27 Feb 2024 07:20:31 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709047231; cv=pass; d=google.com; s=arc-20160816; b=igV6HFqkFpmNVc6o9O8MktwJ/8OwG+5Hp1mM0y/XAdqJRgisBNGdbGLRO06oy/3OmT ZqWiuTGVe1x0SGwW/9oSZrgbWqDyATaIzEbF+YkGb/CC+DzrpJ/hhWTMxwcI1faK4bHc hcc9UPjOVwO8wTamTsr5kCbmFa5PNI++5lvbIkX21318MNV857ioHN5O3oa9R4dbvR89 Ai5UupJvpBzXW0M0lOy/pqy4mYVDcrCp6cFazfokQEHqi2BnUFILcawhtfN2fwv7ww4j GcWQKl4rxoZ9IowVIrLTffEglu7aTG9KFZXNZ9ol2KAedb8zesEow41RyEqytkONoAs3 u1QQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=h4D8aoZuYpE3xE2uEwQyHY1rJvF5ooGRo/8T5tb1R2Q=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=Rcx6SY9D7S/xpZhLYrZyWZ9+Efi2HSrhjIzmAgHrihkikqatAUMJdagNQhl8ZErVNq ChFHDNfaKYGlTp7XgJlpJuTy0y7B+yFKxkj4ATVfW3wUrqRMas+37DLVsCtyyZcBWTy2 i9kgqjLR6mewYjJKfVrlCQchD0zQaIn14weUPYdO84mxz4EbrPx3BD5YFDHOvI4o6rvK GBuxltCauDfim4mq4UCTbAwK2BlujtEaasAbB6lQrKqFJFeDDkqX816LlbIP3LXUP3ws 32wgDRfF2VabMJuoml2vKSEiiDpcI5Sig7fohuW8DdvfEDxLp29AeencU0jez/HdUd90 jTZw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b="VfHB6/Sh"; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83489-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83489-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id o22-20020ac841d6000000b0042e88222ba7si4101036qtm.754.2024.02.27.07.20.31 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 07:20:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-83489-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b="VfHB6/Sh"; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83489-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83489-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 4C49B1C23764 for ; Tue, 27 Feb 2024 15:20:17 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2BB48146E83; Tue, 27 Feb 2024 15:19:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="VfHB6/Sh" Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BD457146013 for ; Tue, 27 Feb 2024 15:19:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047173; cv=none; b=EwZu+BpPcug+1gKeH3BcFNnR+eEsEoVpRUPmp0n7NUsEMWscdXFnQRVhUJzlNHilzRXMHCUV33c+qbH63eqbA+vhA8PzToLsJTEYrnJxXqpEPNCs3GvbAaZfUx+YI1ou/XDxokj4p9nXquQlUhGtAWWsAiAvOvedtWZr5Ptea6o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047173; c=relaxed/simple; bh=cVqN7V5PAyNKuwnktsymYIoCY8DOsy5NnF9aYCziV6E=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Qfxm/3p/3U4d5GFpq/3a8iY7dawNK2I/tVxyUTYeAzHjbvNEmLwrPwuv9nqnth9Si3N7ouUCkrl++tut1682+MNWSIc5j4d9iJs5egiE4N13dn6+do5AwGAd7e7HVjOjQxOr65JfEVwtDecE0oTwn81smDoXMQOCSPP4hbUJBD8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=VfHB6/Sh; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-33d7e755f52so2147894f8f.1 for ; Tue, 27 Feb 2024 07:19:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709047170; x=1709651970; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=h4D8aoZuYpE3xE2uEwQyHY1rJvF5ooGRo/8T5tb1R2Q=; b=VfHB6/ShuugA2Wjzm/07GsI1VBIolgjiPf3jtDWhVf8Xlr9QTxKuJt6+zJvt68Ke/x ETt2uR52qJ/tOPdCQmVv1PcHnyD5zLK20ByuaAoD4a9oSW/+u+rksbk8XB12SoJ6RBNp 8Itza8++ghuhYERnJuQXN0QXWlb9UfHVsW7OdHP1ouGsGXNKiMM4vMTJ5Gzj+vvapFS/ uUV8nOBJhYlZpO3pjMmUhtHDrvP7y/vhXHtPwCf4gJXKVFmxGJHbtDOvtbUi2Y76jza/ XCYTdXoG9d9ab9JPJc5zzvSfZ+zmayC8PoaEh2/jGfsvG9M5wRJXZA3Tw1EMSKiA5SOI MyXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709047170; x=1709651970; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=h4D8aoZuYpE3xE2uEwQyHY1rJvF5ooGRo/8T5tb1R2Q=; b=csyFdt1RcWli02cbpMNQySIfQBI/tYudNN7AsS3i5q78VGHE3O2fqXOyIq5fcRVt13 NtB6nUj8YYkKEe5N95/6s4FPUV0aJgTNDEGhDiCkU7ykNQqbbGb6FB6wVorq6j6oLcTc JiQt2Bw1if7Jbrz4NnBTJGEK/F2yMD2K1CHEYrF9IfSf5nTfaO2cQY+sBB6wtm/hQQ/Q 9Tmu+S6jw/tGYjKEmvqVCX8jVtAjTP/ieUysfG/qH8yR+05YAZj1uY3z3Ud7c72tpIPV ROhE7u3G8DvWm/OxgLGyQCF+vGN6sXkh0yj0Xyn4Fa3fZ0tR8VrJMNP8H30eHLhvKDub 3QGg== X-Gm-Message-State: AOJu0Yxa3XMTrixHf5Az480bJY5oRtgm1QI2ll5RLBBMF8Due095UADE zhYKJwJ4n3uFPVWqy/AsVQUhiAnPYNLbN5Dvn6wBeBNJ2m3XwiFQKDQ8lroLh3TfjoBX/LHII86 K1ktxAb+TOSOtcCEDht/K6jF5+tBDuQ3dZiZgK6/03drNKWb8f/vyuiBgvQVh7+lITkOY3Wigqg 5dzn2Pp4SNKryRyEZsBUOldqKzuWUiBQ== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:224:b0:33d:a18e:4eef with SMTP id l4-20020a056000022400b0033da18e4eefmr19570wrz.2.1709047169980; Tue, 27 Feb 2024 07:19:29 -0800 (PST) Date: Tue, 27 Feb 2024 16:19:11 +0100 In-Reply-To: <20240227151907.387873-11-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240227151907.387873-11-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc1.240.g4c46232300-goog Message-ID: <20240227151907.387873-14-ardb+git@google.com> Subject: [PATCH v7 3/9] x86/startup_64: Simplify calculation of initial page table address From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792065909355423523 X-GMAIL-MSGID: 1792065909355423523 From: Ard Biesheuvel Determining the address of the initial page table to program into CR3 involves: - taking the physical address - adding the SME encryption mask On the primary entry path, the code is mapped using a 1:1 virtual to physical translation, so the physical address can be taken directly using a RIP-relative LEA instruction. On the secondary entry path, the address can be obtained by taking the offset from the virtual kernel base (__START_kernel_map) and adding the physical kernel base. This is implemented in a slightly confusing way, so clean this up. Signed-off-by: Ard Biesheuvel Reviewed-by: Borislav Petkov (AMD) --- arch/x86/kernel/head_64.S | 25 ++++++-------------- 1 file changed, 7 insertions(+), 18 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 1b054585bfd1..c451a72bc92b 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -111,13 +111,11 @@ SYM_CODE_START_NOALIGN(startup_64) call __startup_64 /* Form the CR3 value being sure to include the CR3 modifier */ - addq $(early_top_pgt - __START_KERNEL_map), %rax + leaq early_top_pgt(%rip), %rcx + addq %rcx, %rax #ifdef CONFIG_AMD_MEM_ENCRYPT mov %rax, %rdi - mov %rax, %r14 - - addq phys_base(%rip), %rdi /* * For SEV guests: Verify that the C-bit is correct. A malicious @@ -126,12 +124,6 @@ SYM_CODE_START_NOALIGN(startup_64) * the next RET instruction. */ call sev_verify_cbit - - /* - * Restore CR3 value without the phys_base which will be added - * below, before writing %cr3. - */ - mov %r14, %rax #endif jmp 1f @@ -171,18 +163,18 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) /* Clear %R15 which holds the boot_params pointer on the boot CPU */ xorq %r15, %r15 + /* Derive the runtime physical address of init_top_pgt[] */ + movq phys_base(%rip), %rax + addq $(init_top_pgt - __START_KERNEL_map), %rax + /* * Retrieve the modifier (SME encryption mask if SME is active) to be * added to the initial pgdir entry that will be programmed into CR3. */ #ifdef CONFIG_AMD_MEM_ENCRYPT - movq sme_me_mask, %rax -#else - xorq %rax, %rax + addq sme_me_mask(%rip), %rax #endif - /* Form the CR3 value being sure to include the CR3 modifier */ - addq $(init_top_pgt - __START_KERNEL_map), %rax 1: /* @@ -207,9 +199,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) btsl $X86_CR4_PSE_BIT, %ecx movq %rcx, %cr4 - /* Setup early boot stage 4-/5-level pagetables. */ - addq phys_base(%rip), %rax - /* * Switch to new page-table * From patchwork Tue Feb 27 15:19:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 207277 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2766182dyb; Tue, 27 Feb 2024 07:20:41 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUl3eQPJXV/3WQmu9jJh1fefnw0T7I92im5JzK/XmGdOYgs27SQDeCbFNR0CPXTt5BItV3Zb2zBxqAKCUBvVQYr2tV42A== X-Google-Smtp-Source: AGHT+IHnkBwO9+sQhGg3MIIzlnXEKSLFjRZCcXjPv8dadj1j7ZIyRUIR4+vu/Ni85GTwdcKxjTO3 X-Received: by 2002:a05:622a:309:b0:42d:f63d:98fe with SMTP id q9-20020a05622a030900b0042df63d98femr11930685qtw.47.1709047241416; Tue, 27 Feb 2024 07:20:41 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709047241; cv=pass; d=google.com; s=arc-20160816; b=fg48xGAgNPTo+6KudHBnOi6kNQk0vfvd9wXU8vUyUFxzP4xVY6OK5cfqA597KhmiOj NyweMEXVt+eJZ44tAESY4P+FTym1NfpUdzAb1FDkYElfMUBDtbNLdFUkEL09NhNCMr4u zH0G85qz0KlJ2FaPTXCBe+jSWwHD4KF27Hn8AL8pldVGUGybYpYh+zGihrrNfBr26lr6 4gNh+dgF63NAPT9Fqo7BBKkgAQkyhXRILDnzmIC6Ft0fI8H1O3EQfQ4hoD8u9jcMarwy NhVM88udIJz7sgSx3LhGFfYNDnhkJwpxY5h/CFWcj2CCCLTnVdj4Uw4P2/HZtttwj5K/ /0Iw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=KEFw4Uv2SMq6PunMM0SQo+ZA+grVXwI4qkkkBFDNiX4=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=NwTx3rarNvjvbbWpuGi2cxJ7fmzbJTPQjwAY0sI6AcJMtlDTiRHBId+RVVd2Xusb5U 7sZNUCV5BZREZfWpR9b46bvFVGbxBzn0xooAbJiCVYOcXbuviPSlOlF/aOhQO56lr3Sa MgssPTr9J74W7ebDhZzd83R2wFz17WfmFZ38SCGD8P+H2MUkpc/3V0CHkBw04lnvqm3U GsjLRJh3DVXyXZA/mV8Jw0Rgsn96HrURLFd4JI0SjrsY5T0AmleFgw/m3e/Dh3SFrMu8 BsIfNjIyiFEVqzrPiHBk0Cv8vdMkbckSseAFRJ4oHp4Ok27rcMTUfNV0hbKa2zTNFV1v FFTQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b="hju/C2NM"; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83490-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83490-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id n16-20020ac85a10000000b0042e130452a9si8264082qta.27.2024.02.27.07.20.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 07:20:41 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-83490-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b="hju/C2NM"; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83490-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83490-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 2064E1C21171 for ; Tue, 27 Feb 2024 15:20:34 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A6F621482FA; Tue, 27 Feb 2024 15:19:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="hju/C2NM" Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 15230145356 for ; Tue, 27 Feb 2024 15:19:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047175; cv=none; b=SYc4EnkK7rMfAU1mDB/fYQJKAH3h0xpxUQcRpEdBeai2wGS8VjtyqVeaRswayOu9hUnP/aRiqSAigtgISx0GYi1y4jrFLkvo1K3BdmHuzT2U48pg8fGUTKHX52Xfm911SML4YAScJEbcQbd2KBDW56ye8fHbQZZa3oQMnbxMVJg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047175; c=relaxed/simple; bh=/S719xaU9xP8uHOi68ntsc/Hv6QyoUaD6Ro7ASZe5Gs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=nHw14P5cJVhTv1kNuHDYQDKSBtvmxOorvV80FAaVwDZZZwGPQQECRj27TZi4BZIJBlMC6VLtyJS1/HHN9yNzNWaVjdbzHEJ5vEBLh11NGpLTk0KqhJMOpVTYGs36EG7CaJ0G3h/bzCoDJIjXMCONO0CD+zvRsv8j6QS7k/aR2SY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=hju/C2NM; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-412b0d34a42so1109735e9.0 for ; Tue, 27 Feb 2024 07:19:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709047172; x=1709651972; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KEFw4Uv2SMq6PunMM0SQo+ZA+grVXwI4qkkkBFDNiX4=; b=hju/C2NMxsKzTO45beITfx7SppNhRSwKOdEpnwju6ECRVoAJBwLQt7lsqVWNVgKWAk 0DnoEnGP7EoXbEqVbbieuZr53R5Emfsb0mHucG+i6aMxstlYpMLwmwjaH+YVVYhF/off +XjbY/5K5g9EiHWqfGn10FvFoLUyl1maOpm6LwXlt/zagR89LapAyAX6cus/ugwNiGJe FR8t3w4JJQ/1jzCMXHSGghkjCQVFY5Ym62JaKCv5rrEWKt5Ecrs/83OiKdMM3dA12uJX p3wOEJujm+WCBDCh+hOySZQhMg838SorSZXyWXd2hypsOWgyf2LGJYxnaAqnFJBS/PJo dgwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709047172; x=1709651972; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KEFw4Uv2SMq6PunMM0SQo+ZA+grVXwI4qkkkBFDNiX4=; b=LDyvLlG0XzzEX4M6AcQSPZKolObL6j+1AYWNZcH7gzk9t5YkL2fipMW2JML66xhBXM zow1fDlmOhsHKH/CncKtyzRJkQBnLd12vwedTMFdOOD/n5/agj23lnG5M5AlVp2raS+N X5VhAOiSLmSQpouHuK0fX4Lzos5au9cnLxO317VZFQaA+nnU5X+qGQKVnsLEtntIKp/x FNZEXbP5N0dmGAFdQ+fnsrcvIGFLTjjuDiCqnUAqpP5sZmCfs8LytkQFviF7fi5Acr0y G8HHbquCgSfGW2Gb+QWJXyHRXdh2YNq27FtTx/pcH+qlAPHkBTD2v1DJC9UVLjbeLmIR Xc8g== X-Gm-Message-State: AOJu0YxexueXlJU6fm6rzEBS2AQNxCl1uVwNpsCwl2BwfYu6PBiM0hsy zACEkDuHTbI8QOLGx9IN4bgVOEupYez2XaqOGMS26eWmENSps4ynkjXvkSigGGkUaSh3wzqCIqn oOJyBoPaUzNn3ig97FjhcUtynu33qZEzzZ/bVJShpt+oIVEjTaFtx/JsEUMy3JAOmOlvMFEkwH2 FvFAZR2IUk151a+/tLdWIur+S34Bf61g== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:600c:54e7:b0:412:c62:2dd9 with SMTP id jb7-20020a05600c54e700b004120c622dd9mr51036wmb.2.1709047172165; Tue, 27 Feb 2024 07:19:32 -0800 (PST) Date: Tue, 27 Feb 2024 16:19:12 +0100 In-Reply-To: <20240227151907.387873-11-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240227151907.387873-11-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc1.240.g4c46232300-goog Message-ID: <20240227151907.387873-15-ardb+git@google.com> Subject: [PATCH v7 4/9] x86/startup_64: Simplify virtual switch on primary boot From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792065920255849131 X-GMAIL-MSGID: 1792065920255849131 From: Ard Biesheuvel The secondary startup code is used on the primary boot path as well, but in this case, the initial part runs from a 1:1 mapping, until an explicit cross-jump is made to the kernel virtual mapping of the same code. On the secondary boot path, this jump is pointless as the code already executes from the mapping targeted by the jump. So combine this cross-jump with the jump from startup_64() into the common boot path. This simplifies the execution flow, and clearly separates code that runs from a 1:1 mapping from code that runs from the kernel virtual mapping. Note that this requires a page table switch, so hoist the CR3 assignment into startup_64() as well. And since absolute symbol references will no longer be permitted in .head.text once we enable the associated build time checks, a RIP-relative memory operand is used in the JMP instruction, referring to an absolute constant in the .init.rodata section. Given that the secondary startup code does not require a special placement inside the executable, move it to the .text section. Signed-off-by: Ard Biesheuvel Reviewed-by: Borislav Petkov (AMD) --- arch/x86/kernel/head_64.S | 42 ++++++++++---------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index c451a72bc92b..87929f615048 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -39,7 +39,6 @@ L4_START_KERNEL = l4_index(__START_KERNEL_map) L3_START_KERNEL = pud_index(__START_KERNEL_map) - .text __HEAD .code64 SYM_CODE_START_NOALIGN(startup_64) @@ -126,9 +125,21 @@ SYM_CODE_START_NOALIGN(startup_64) call sev_verify_cbit #endif - jmp 1f + /* + * Switch to early_top_pgt which still has the identity mappings + * present. + */ + movq %rax, %cr3 + + /* Branch to the common startup code at its kernel virtual address */ + ANNOTATE_RETPOLINE_SAFE + jmp *0f(%rip) SYM_CODE_END(startup_64) + __INITRODATA +0: .quad common_startup_64 + + .text SYM_CODE_START(secondary_startup_64) UNWIND_HINT_END_OF_STACK ANNOTATE_NOENDBR @@ -174,8 +185,15 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) #ifdef CONFIG_AMD_MEM_ENCRYPT addq sme_me_mask(%rip), %rax #endif + /* + * Switch to the init_top_pgt here, away from the trampoline_pgd and + * unmap the identity mapped ranges. + */ + movq %rax, %cr3 -1: +SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) + UNWIND_HINT_END_OF_STACK + ANNOTATE_NOENDBR /* * Create a mask of CR4 bits to preserve. Omit PGE in order to clean @@ -199,30 +217,12 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL) btsl $X86_CR4_PSE_BIT, %ecx movq %rcx, %cr4 - /* - * Switch to new page-table - * - * For the boot CPU this switches to early_top_pgt which still has the - * identity mappings present. The secondary CPUs will switch to the - * init_top_pgt here, away from the trampoline_pgd and unmap the - * identity mapped ranges. - */ - movq %rax, %cr3 - /* * Set CR4.PGE to re-enable global translations. */ btsl $X86_CR4_PGE_BIT, %ecx movq %rcx, %cr4 - /* Ensure I am executing from virtual addresses */ - movq $1f, %rax - ANNOTATE_RETPOLINE_SAFE - jmp *%rax -1: - UNWIND_HINT_END_OF_STACK - ANNOTATE_NOENDBR // above - #ifdef CONFIG_SMP /* * For parallel boot, the APIC ID is read from the APIC, and then From patchwork Tue Feb 27 15:19:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 207278 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2766223dyb; Tue, 27 Feb 2024 07:20:45 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXoVgfOjuRDFQCEeHub7Ta1+wH8+X0yJSt390C7oA7AUrvNonvNXXNB9yf1DuTVo64pi3DyQm7taHHf3daY41aOPboV5Q== X-Google-Smtp-Source: AGHT+IFxAfDzQMdvP8uXx5ZZ1h06Cz6xOXyF5DjM1/kYnHxynbeIIOkHu6HSKtKHI1reD8oy/sRi X-Received: by 2002:a0c:f54e:0:b0:68f:8ddd:aad4 with SMTP id p14-20020a0cf54e000000b0068f8dddaad4mr2575181qvm.35.1709047244838; Tue, 27 Feb 2024 07:20:44 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709047244; cv=pass; d=google.com; s=arc-20160816; b=Hpmu3PW1Wn1SQfyK6hj2KjL8PGgZr/Ze0RXOhEm18GC3VvnuqXwc/7GcDyW6jxnCFa cvZVw/ITqFbx+wOZyR+0WP/iwFGYXo8bFJOGupgRzifG55h4lcH3gonotE3GTTxGOJRS 79vvlSNgM2Gkkt9nttjDXZNCAnR6VJW71YUNqkLgNh/lwLZjzh4bmfR/a7FRZfqmGmHk TsAkLsHWrSRZMPs+WxXaBlKsRy/aRxVYQlou5zLnIzTC21ilop/gon1rcnHAXXBaz54p RIOMtz2xv3qhocKHiRt9kew4vsjxrDwrvcMTcC680OOi1L3ce/iEmTp+9/Qc+gitrVoU x9lQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=rh1bAQtKMaIJdR87zlJE2orPGq4tRSmOiy/LVJJ6Lnk=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=jHNHYb8VO1oIp669v/MteAQpuxVyPI65lYUqPm0nkqSrhijEHEZ0aBEQaLC/oCFoY5 +ik5Iwm4Lz92B9+AT4HAnlyfJKE1RsCdTM4TX08Uf4Teobx2uQAGARiqwcm0S13DTQp2 exzx6wED6V5vhUvguGKtxmAOfyw0qwMTuSwSDjFsBcOEqooi4pfJyWZLdsysx+JI33nG 6O2hoSGQAF9dZCqnrgX1iAEjLgh3EZ9I9tO9Z2JblcSnd4Ue9eRPx+db61r4HQgEkECA 4b75tvTKwVAHUKbrx/PsKcSs+f2eI8t6KDVsTjtxEYLx37t05EqTs2zSSU45+z1v1uTU tyvQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=YutPEzYp; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83491-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83491-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id gc13-20020a056214230d00b0068f610acb50si7663102qvb.613.2024.02.27.07.20.44 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 07:20:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-83491-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=YutPEzYp; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83491-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83491-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 9790A1C22BE1 for ; Tue, 27 Feb 2024 15:20:40 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B2ACA145356; Tue, 27 Feb 2024 15:19:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="YutPEzYp" Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8DE78146E60 for ; Tue, 27 Feb 2024 15:19:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047177; cv=none; b=BFMtE45LkOJri2bDgPzbteskeUR/JC5q13cR7urGI8ozwhwr96NHjVgqYW9Gq/INdCsCf1V8B/Ad2mzuyJAuZW8LN33EJ8EcZuon6UpNqIEYqstLeFQ76z2vwzqIplt2q36zXiBQdgMM8uaoxXPFeLr/KaSyS1WScvUWw4aEef8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047177; c=relaxed/simple; bh=bLDe3KG0XYavXQhiwPBF6RL6baBIW+tOuHgPkuSkJWI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=J81MdG6rlRJeFGLOJquETNdc679yrAvUFV8Rb4Pq6UbWVQuzBrkQMEx/1nTk0keC7hFT5qNB+5HJ69qWjCHctdqHLV6zZKFiiC1uT/zwhhIBNl61y5j0GVA2udxljT8qiqbsa/iF5L0L2kwf+IkVAOyBnH/J8TV+E7VlzXenoKE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=YutPEzYp; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc6b26eef6cso6039049276.3 for ; Tue, 27 Feb 2024 07:19:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709047174; x=1709651974; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rh1bAQtKMaIJdR87zlJE2orPGq4tRSmOiy/LVJJ6Lnk=; b=YutPEzYp6GlsFGql46at+AqpHrmrJn/FU90UQO4nH4yBZJ7LBKJg6M8YdHUEJ8BlXV Qx5W305Zls+L4m1JcVOGCp/fix7Y9PIQoROKUVb/9hm+dhWGIVT80fWYUJ4LsmA2yGe0 O+jmLktpJo1yilBjcM5Bd9KB5RP+qeV4r32sd7XmRkIt+LbpZmgJKfQ6WwzBzKkTGYMX 2ijC1GR9KdV05vZfAkaheYxH2F06uBawJUXlYJ25A7N65rtiKQsIGAa3hjgQc0QxRHcA JTd7aPRRwjiafXbrc5KGM7bt+va19Yn0CxXnqgXheVP3/3XQYqAoa1lag4IQbqJJqAaw 3gRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709047174; x=1709651974; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rh1bAQtKMaIJdR87zlJE2orPGq4tRSmOiy/LVJJ6Lnk=; b=dohPSC0EebcgvDeqVZdqqxRItLpfGz4ahhLbqpCF0ka09a09litwL6SOCOW+lRofRo 2DE0tkV0xAi2Gbc5/OwZXCcsXb4EALEN1KmM22h6GDw2ec6xh/rm61SIAxmXSIwB2gJb D0ICj4LGhXwf/tL/6VdcaDtrcIYqRTMihw1zgGIsWfYn5/3VEI8C8A5F+kXgnItRdMa7 e1IhLYoclEGIS75bq+6j+nv+Z15nhEtETTjUoChq/NQyK8Kr/9O3ub9d7uXWsHEzErM/ lPaTz14tUukiq4QIjrNoOp93kb6e1nTsaBeL3ZYDJUrnjis0wH8bh+51cuDzNe04aOhC IcDw== X-Gm-Message-State: AOJu0YzO5O9J/wXNgnggmYi0J3PpH5z/F8gfPNYaHvCHbphFlc1lQOZU YkRPqpUch1zb00xQm5rHA2FbnKyB3QrPed8WjOnz6QvxTu2BxOsb3PNLZrJuP708gGGJ3QCV/Zs d4K+SQpBiXb6OnHlm/xjP2YCuSs5COazV91Qs05q4ywIMEdiW0LiVdYHeMg1LH6aW9VPuX0WojU VSWCuQBrgtsxnoXWkDc4Hm7DvPemkigw== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:2484:b0:dc7:4af0:8c6c with SMTP id ds4-20020a056902248400b00dc74af08c6cmr133232ybb.6.1709047174691; Tue, 27 Feb 2024 07:19:34 -0800 (PST) Date: Tue, 27 Feb 2024 16:19:13 +0100 In-Reply-To: <20240227151907.387873-11-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240227151907.387873-11-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc1.240.g4c46232300-goog Message-ID: <20240227151907.387873-16-ardb+git@google.com> Subject: [PATCH v7 5/9] efi/libstub: Add generic support for parsing mem_encrypt= From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792065923942933533 X-GMAIL-MSGID: 1792065923942933533 From: Ard Biesheuvel Parse the mem_encrypt= command line parameter from the EFI stub if CONFIG_ARCH_HAS_MEM_ENCRYPT=y, so that it can be passed to the early boot code by the arch code in the stub. This avoids the need for the core kernel to do any string parsing very early in the boot. Signed-off-by: Ard Biesheuvel Reviewed-by: Borislav Petkov (AMD) --- drivers/firmware/efi/libstub/efi-stub-helper.c | 8 ++++++++ drivers/firmware/efi/libstub/efistub.h | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/firmware/efi/libstub/efi-stub-helper.c b/drivers/firmware/efi/libstub/efi-stub-helper.c index bfa30625f5d0..3dc2f9aaf08d 100644 --- a/drivers/firmware/efi/libstub/efi-stub-helper.c +++ b/drivers/firmware/efi/libstub/efi-stub-helper.c @@ -24,6 +24,8 @@ static bool efi_noinitrd; static bool efi_nosoftreserve; static bool efi_disable_pci_dma = IS_ENABLED(CONFIG_EFI_DISABLE_PCI_DMA); +int efi_mem_encrypt; + bool __pure __efi_soft_reserve_enabled(void) { return !efi_nosoftreserve; @@ -75,6 +77,12 @@ efi_status_t efi_parse_options(char const *cmdline) efi_noinitrd = true; } else if (IS_ENABLED(CONFIG_X86_64) && !strcmp(param, "no5lvl")) { efi_no5lvl = true; + } else if (IS_ENABLED(CONFIG_ARCH_HAS_MEM_ENCRYPT) && + !strcmp(param, "mem_encrypt") && val) { + if (parse_option_str(val, "on")) + efi_mem_encrypt = 1; + else if (parse_option_str(val, "off")) + efi_mem_encrypt = -1; } else if (!strcmp(param, "efi") && val) { efi_nochunk = parse_option_str(val, "nochunk"); efi_novamap |= parse_option_str(val, "novamap"); diff --git a/drivers/firmware/efi/libstub/efistub.h b/drivers/firmware/efi/libstub/efistub.h index c04b82ea40f2..fc18fd649ed7 100644 --- a/drivers/firmware/efi/libstub/efistub.h +++ b/drivers/firmware/efi/libstub/efistub.h @@ -37,8 +37,8 @@ extern bool efi_no5lvl; extern bool efi_nochunk; extern bool efi_nokaslr; extern int efi_loglevel; +extern int efi_mem_encrypt; extern bool efi_novamap; - extern const efi_system_table_t *efi_system_table; typedef union efi_dxe_services_table efi_dxe_services_table_t; From patchwork Tue Feb 27 15:19:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 207279 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2766351dyb; Tue, 27 Feb 2024 07:20:54 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVljRvy56D1+VCH21ZnxVGAU6XabZyTIBWrTAcpuFJks7qlMgZEhd/Jl0nmEoE4PWwjuVVHZr3rriSsPREujccFUuUm2w== X-Google-Smtp-Source: AGHT+IH+ceVMjfbw7AhWyLW+F798qVhkbrOOqBg8kChayWIvgj3N+D7GqC7nw0LKXrpP0qlC58iI X-Received: by 2002:ac8:578e:0:b0:42e:4f7e:5fb1 with SMTP id v14-20020ac8578e000000b0042e4f7e5fb1mr18091033qta.18.1709047254614; Tue, 27 Feb 2024 07:20:54 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709047254; cv=pass; d=google.com; s=arc-20160816; b=CVhBs5LRaDWasAQZIkrgmQc3wLcv1mAVSeQAkbA5A6JuPMGrlA6jMiazf8IOJ3IoW+ NEM/f1K0bor+t+bGPsyJVZnjIa9PNo6C6cg+n9Px/yV9iP+kskfiPzoKp/3+Hl9mnGz5 lfLfSStDBeGdXD6rVLCF5PRZMRstvaHFvZy7psM0HHHF2n3FAvoyCdnPsOljbQZ4xs2w JyS1dPeixTTTMoN9+uOnFI2gpSw8dpZds2a5CIF0QD4dcNPPEYw6Ps8WMZIILkc1zTSB nkn7KPSeYcP+1k+qTNyC+vZmwLFsUYMAJrfvvN94xEoOjjBANQTyGTtmdaxzKRU/iwU1 UM3A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=qc1vZT6QqkQkEMHN6UDx4N3IKREdTVsL1hpJl21eLz4=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=a1U4V+Ar8NnONF1bImX+L1HhMo8yAS/S8FDYW0+FnS1/TuUmbfhQycWwCzZ5GLYtCx 2jZFroc1pPiMNy4thNo3bPF1QqfEwLCwW5TSP7vYkEV7udzt0j8bA0oaqZILbEzQKAJc Rb9YW368wlpSU7ZZHnPitEmXssfkR+sZNobwPRh4j1JQbTQ2YMFDaKBTUVEe/7sveEe2 aIzfVmxoJmKIUPl4NG1QsvTQWufNbkWBdX+s01upgGmf2LhhFJKCpGB2v6bjnowHBXH1 z2AXDo9zzf9bRU4DY1EiwMrV6bQzYOSQz0hN2psZ6babbx/uosOfo7jTl492f/UohOHf 5R3g==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b="W9JoFN/e"; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83492-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83492-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id v20-20020a05622a189400b0042ea1461cefsi1059424qtc.99.2024.02.27.07.20.54 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 07:20:54 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-83492-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b="W9JoFN/e"; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83492-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83492-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 5846A1C22033 for ; Tue, 27 Feb 2024 15:20:54 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2EF721487DA; Tue, 27 Feb 2024 15:19:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="W9JoFN/e" Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 143C71482EF for ; Tue, 27 Feb 2024 15:19:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047179; cv=none; b=WVkUBwFZ3TW+wNabqqnf0efVbKB1UEOen7LDNU0keU/0MXJqPj9OkK13GS044/nE/DIF+O8RQL+OTW3sTu3QWHnOSRQESEUwSg5XDe6LfdAl/6sMT7LWTYLbsLr/DK0IPX2Y3FCj9W/vp6H7h8F9ZREV4l0IlzNwiJLL7cIVuFw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047179; c=relaxed/simple; bh=A2R9NwPOj178rTXF4jj1Oe0QhM2YkwIT8ZkVkD7sOlw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=r4go51ZMkZl3HqkTA3sx+nQGuNBawGec2xTCzHH79x4GOxR38Xo8Xo1Bh4hmDQY4gKoUM5g6vKvitzIUcaAWuN4qhguZfExwUvfipxxXdIRKiWKbbYuQUK+BbqQR2Sgewx+BpF+DIIa/Ptjsh5TsPTQDfpwc3GveOz9tpyfc/iU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=W9JoFN/e; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-608ab197437so69590887b3.1 for ; Tue, 27 Feb 2024 07:19:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709047177; x=1709651977; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qc1vZT6QqkQkEMHN6UDx4N3IKREdTVsL1hpJl21eLz4=; b=W9JoFN/eVGSVF8fxpkDJ2vCAflR+BZLg155+Nnk28l0b4i8z4PZAu96kZ1lFLtOi+i 6J6kXGbJ9fBeNP0Ed8gJoYVgGrUnF+w2ilQnODci6G7VLSNbOiNuzQLERjZwgNoMSCpy 4BWAEysHLDet29kaMd+nO0JmFFn50xonK73gyGbbL0dmuK4du27k46X2lO0j0MCpf6/W ndj4HiihGHqaoma7vfFX952ltNdrokKuW6FDSvYCZhIwi03lkgGafoVYZjNY8R/RS01P pFnnuJj/xwJKyEPkKbPSyux+Sdb3EID5d56wEZ8ZUhQQ2ogQxZVf4MWOqNxNmlzXQM8P Axdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709047177; x=1709651977; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qc1vZT6QqkQkEMHN6UDx4N3IKREdTVsL1hpJl21eLz4=; b=MEQmpCn1iBkPPz5bw2qCyzekQ4f0YXVQQ+3GIDOW+Qt54dfQqOYb5QEIBS9obf5cKT GNz+ugJY+jP0mtI5vg7sLTp2U0QQ2bunIWn5graShNiyiu0fIPXtR0giN8sPZK77MOdG MBfvdkQW2HgttIWT3xdIRAcmWjytIp6dQsdqvFBfa3iY18HiStz8rlor4JYSlHSMeJjj swSPxeLrIPxAFCB3QreAnJAHjZAERFaoOlwHSCy/7+uctNzEVW/MM8rZnTMFTz0SAcc1 FN+hUKw16uJvq52F7AO2FAS4TouxVPHb0kQpIdqtkFatj404QNsSxHHPFny9Wt9bYyvC Q+ZA== X-Gm-Message-State: AOJu0YyvWuo0Vbgy0gokv9VWQQNruG8+0LjZeSsFhHIS4QaZ3yFFchN8 q+zjfKhKFCzpTg2QdAW1XaXM54UN4suPY5r6Qck6vUoo9qbfXUREvFINwLTGHSR1niZgaqJtAdQ Jw4x0ynLFvA9VPRx5k6/AJYpMCKwY6Gn2nXaPoxMRiYfxAETMzZkw0zHQ9+RftiK0uY0RuFO+tt TU/7XD9u56Mpgu3FnAmkbDzyV2LBXeBA== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:48c8:0:b0:608:de34:9583 with SMTP id v191-20020a8148c8000000b00608de349583mr527840ywa.7.1709047177011; Tue, 27 Feb 2024 07:19:37 -0800 (PST) Date: Tue, 27 Feb 2024 16:19:14 +0100 In-Reply-To: <20240227151907.387873-11-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240227151907.387873-11-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc1.240.g4c46232300-goog Message-ID: <20240227151907.387873-17-ardb+git@google.com> Subject: [PATCH v7 6/9] x86/boot: Move mem_encrypt= parsing to the decompressor From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792065934360962739 X-GMAIL-MSGID: 1792065934360962739 From: Ard Biesheuvel The early SME/SEV code parses the command line very early, in order to decide whether or not memory encryption should be enabled, which needs to occur even before the initial page tables are created. This is problematic for a number of reasons: - this early code runs from the 1:1 mapping provided by the decompressor or firmware, which uses a different translation than the one assumed by the linker, and so the code needs to be built in a special way; - parsing external input while the entire kernel image is still mapped writable is a bad idea in general, and really does not belong in security minded code; - the current code ignores the built-in command line entirely (although this appears to be the case for the entire decompressor) Given that the decompressor/EFI stub is an intrinsic part of the x86 bootable kernel image, move the command line parsing there and out of the core kernel. This removes the need to build lib/cmdline.o in a special way, or to use RIP-relative LEA instructions in inline asm blocks. This involves a new xloadflag in the setup header to indicate that mem_encrypt=on appeared on the kernel command line. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/misc.c | 15 +++++++++ arch/x86/include/uapi/asm/bootparam.h | 1 + arch/x86/lib/Makefile | 13 -------- arch/x86/mm/mem_encrypt_identity.c | 32 ++------------------ drivers/firmware/efi/libstub/x86-stub.c | 3 ++ 5 files changed, 22 insertions(+), 42 deletions(-) diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c index bd6857a9f15a..408507e305be 100644 --- a/arch/x86/boot/compressed/misc.c +++ b/arch/x86/boot/compressed/misc.c @@ -371,6 +371,19 @@ unsigned long decompress_kernel(unsigned char *outbuf, unsigned long virt_addr, return entry; } +/* + * Set the memory encryption xloadflag based on the mem_encrypt= command line + * parameter, if provided. + */ +static void parse_mem_encrypt(struct setup_header *hdr) +{ + int on = cmdline_find_option_bool("mem_encrypt=on"); + int off = cmdline_find_option_bool("mem_encrypt=off"); + + if (on > off) + hdr->xloadflags |= XLF_MEM_ENCRYPTION; +} + /* * The compressed kernel image (ZO), has been moved so that its position * is against the end of the buffer used to hold the uncompressed kernel @@ -401,6 +414,8 @@ asmlinkage __visible void *extract_kernel(void *rmode, unsigned char *output) /* Clear flags intended for solely in-kernel use. */ boot_params_ptr->hdr.loadflags &= ~KASLR_FLAG; + parse_mem_encrypt(&boot_params_ptr->hdr); + sanitize_boot_params(boot_params_ptr); if (boot_params_ptr->screen_info.orig_video_mode == 7) { diff --git a/arch/x86/include/uapi/asm/bootparam.h b/arch/x86/include/uapi/asm/bootparam.h index 01d19fc22346..eeea058cf602 100644 --- a/arch/x86/include/uapi/asm/bootparam.h +++ b/arch/x86/include/uapi/asm/bootparam.h @@ -38,6 +38,7 @@ #define XLF_EFI_KEXEC (1<<4) #define XLF_5LEVEL (1<<5) #define XLF_5LEVEL_ENABLED (1<<6) +#define XLF_MEM_ENCRYPTION (1<<7) #ifndef __ASSEMBLY__ diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile index ea3a28e7b613..f0dae4fb6d07 100644 --- a/arch/x86/lib/Makefile +++ b/arch/x86/lib/Makefile @@ -14,19 +14,6 @@ ifdef CONFIG_KCSAN CFLAGS_REMOVE_delay.o = $(CC_FLAGS_FTRACE) endif -# Early boot use of cmdline; don't instrument it -ifdef CONFIG_AMD_MEM_ENCRYPT -KCOV_INSTRUMENT_cmdline.o := n -KASAN_SANITIZE_cmdline.o := n -KCSAN_SANITIZE_cmdline.o := n - -ifdef CONFIG_FUNCTION_TRACER -CFLAGS_REMOVE_cmdline.o = -pg -endif - -CFLAGS_cmdline.o := -fno-stack-protector -fno-jump-tables -endif - inat_tables_script = $(srctree)/arch/x86/tools/gen-insn-attr-x86.awk inat_tables_maps = $(srctree)/arch/x86/lib/x86-opcode-map.txt quiet_cmd_inat_tables = GEN $@ diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index 0166ab1780cc..d210c7fc8fa2 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -43,7 +43,6 @@ #include #include -#include #include #include @@ -95,9 +94,6 @@ struct sme_populate_pgd_data { */ static char sme_workarea[2 * PMD_SIZE] __section(".init.scratch"); -static char sme_cmdline_arg[] __initdata = "mem_encrypt"; -static char sme_cmdline_on[] __initdata = "on"; - static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) { unsigned long pgd_start, pgd_end, pgd_size; @@ -504,11 +500,9 @@ void __init sme_encrypt_kernel(struct boot_params *bp) void __init sme_enable(struct boot_params *bp) { - const char *cmdline_ptr, *cmdline_arg, *cmdline_on; unsigned int eax, ebx, ecx, edx; unsigned long feature_mask; unsigned long me_mask; - char buffer[16]; bool snp; u64 msr; @@ -551,6 +545,9 @@ void __init sme_enable(struct boot_params *bp) /* Check if memory encryption is enabled */ if (feature_mask == AMD_SME_BIT) { + if (!(bp->hdr.xloadflags & XLF_MEM_ENCRYPTION)) + return; + /* * No SME if Hypervisor bit is set. This check is here to * prevent a guest from trying to enable SME. For running as a @@ -570,31 +567,8 @@ void __init sme_enable(struct boot_params *bp) msr = __rdmsr(MSR_AMD64_SYSCFG); if (!(msr & MSR_AMD64_SYSCFG_MEM_ENCRYPT)) return; - } else { - /* SEV state cannot be controlled by a command line option */ - goto out; } - /* - * Fixups have not been applied to phys_base yet and we're running - * identity mapped, so we must obtain the address to the SME command - * line argument data using rip-relative addressing. - */ - asm ("lea sme_cmdline_arg(%%rip), %0" - : "=r" (cmdline_arg) - : "p" (sme_cmdline_arg)); - asm ("lea sme_cmdline_on(%%rip), %0" - : "=r" (cmdline_on) - : "p" (sme_cmdline_on)); - - cmdline_ptr = (const char *)((u64)bp->hdr.cmd_line_ptr | - ((u64)bp->ext_cmd_line_ptr << 32)); - - if (cmdline_find_option(cmdline_ptr, cmdline_arg, buffer, sizeof(buffer)) < 0 || - strncmp(buffer, cmdline_on, sizeof(buffer))) - return; - -out: RIP_REL_REF(sme_me_mask) = me_mask; physical_mask &= ~me_mask; cc_vendor = CC_VENDOR_AMD; diff --git a/drivers/firmware/efi/libstub/x86-stub.c b/drivers/firmware/efi/libstub/x86-stub.c index 99429bc4b0c7..0336ed175e67 100644 --- a/drivers/firmware/efi/libstub/x86-stub.c +++ b/drivers/firmware/efi/libstub/x86-stub.c @@ -884,6 +884,9 @@ void __noreturn efi_stub_entry(efi_handle_t handle, } } + if (efi_mem_encrypt > 0) + hdr->xloadflags |= XLF_MEM_ENCRYPTION; + status = efi_decompress_kernel(&kernel_entry); if (status != EFI_SUCCESS) { efi_err("Failed to decompress kernel\n"); From patchwork Tue Feb 27 15:19:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 207293 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2787278dyb; Tue, 27 Feb 2024 07:56:17 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVZf+VijdQQa5JpaneGtYGDKgukP1J9c20Z+La8FLqY2o0DXLTAPbw9tNtpWxX83Q8SIJNyj0FLlHvni+e2R7B9l2gTdg== X-Google-Smtp-Source: AGHT+IEEKIZ1fGcDsIY1ygdfDNdVwOISLRm8xJSG1TU9AX7RtYOky5tIKnYTEGCwlZMn+HSpBywZ X-Received: by 2002:a17:903:496:b0:1dc:3ab7:cc78 with SMTP id jj22-20020a170903049600b001dc3ab7cc78mr9185405plb.29.1709049376879; Tue, 27 Feb 2024 07:56:16 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709049376; cv=pass; d=google.com; s=arc-20160816; b=s2cy0jtpvcsJaQUOEibbSzErBJe3M8SRaaAlRIEu3yvwGJirGbeXP5w7hZGBD9wLFu 9G2HUO5jH7XZrN2+lrERrhKWBTfOUbMeFrV3hAiy2A7dsSE3Ag9vo2Cdq5TeQBizwZ3f dASM7PulGnzyrfbogJY8DOt6oWnhFnDY3nJmptkPlphD9C1MMyiI0rONY3ZfmH0maRPg SflXQvERGLUa3Ij8UtLkoWTYNuRsGh/FKSnBrIdVySdtBbw6HutyAq+cdB3QWdl4FTrR bqVlg2mL5fSZQz13wIRcGkVyfB6CVIfwxVx875jhcgbj491uX3l9hQMMsODVNhm/IX+Z jqmQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=RryqSy7p/WJgmbuDaBcFTKEm0duOKiPOTBDTU9e1Dgc=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=PK4nU2GS77XMWM9B+XUVWTnFEZE/e3UmjqRp0dK47NeHTjCdULEkz++4d8N7UhdBw6 5fNf1AX053M3d++pnN0ka+RjO5j6YzDGViJNg0dyiQp9xnkX4H5LvaZkyOvqt2sVuEsJ 6/I6y7Ln3Q4103gyFxMwFlL5jeNOU+Kk6QZ92m8fFh7yZ8NTn1VHrvMIPTkIFNzvon+s e5A1Qhz5LTb8gx6rP1+9E7jHQoND/MamisB//LLKci9IhQKruMnNEdNYfvj0bqoQ8ZVN hwLaSxo9bz3jHT6sTbDPHknZDeFnMmr/09XO0Gx4Ew6mU+y2C84Ir4fKUawzmASd61kr fcNA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=jfvqHmZr; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83493-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83493-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id q10-20020a170902a3ca00b001dbc1a37d6dsi1511105plb.462.2024.02.27.07.56.16 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 07:56:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-83493-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=jfvqHmZr; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83493-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83493-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id F1BC0B2CCA1 for ; Tue, 27 Feb 2024 15:21:12 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 92A1E149009; Tue, 27 Feb 2024 15:19:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="jfvqHmZr" Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4F44145346 for ; Tue, 27 Feb 2024 15:19:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047182; cv=none; b=UrsKnRKm74GP1kpq7svD1Qe/kPHz6jtpNvFZZtJS3s1/7bEhNsVj70X++WsfHRzCCCxV8SQZNS3qvHX0jJY7Sg1T+SwVB7qf9mREc/o9U0SiX9NxD9KKrn90Yoge93vBTSUZ73k6EQ+xdMp0RCx93HND/MDwId+QUwACAZrdRj8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047182; c=relaxed/simple; bh=n+M7WCortQTRWGbUCG2GXIYZAvcAmGDrlP8/0sO/Gq4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=dqDk+by19xt1pWJ4U2iVRZ3mvCyHrNqwf1hw+r2Hox9gkAvCOaWxyjFOii8Js+OetgoLzmcGhzSrhxa9/SfHowhkIvGmpQ5SZQj6OcKRWIchyJOXGDJ38yyBUy68ZzyyhaCOfrVVrgT618o6sSJ5O24T0Z3P7hiW6Un7WiAzONY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=jfvqHmZr; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-33d9f425eaaso1828563f8f.3 for ; Tue, 27 Feb 2024 07:19:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709047179; x=1709651979; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=RryqSy7p/WJgmbuDaBcFTKEm0duOKiPOTBDTU9e1Dgc=; b=jfvqHmZryq7pGm9kN4o1/Yh7bhvXh4T3tk7Q2KZR6wG+kyzrdVdwywSdCyYo2je8Uq IV111pHYJg22fmqbkCKQ5HlSAmqBGCrpmtruvwd+1bVlBeWGmTG4P/WVvsiLbGk0f3uj EP6G61yRa04cVwLu9wHYLgM8i2P00QGLIPbeFWtPi1L2IFCUkglUDv9m0FHcl1Y3hz+u DMczkI6QXyre6gjw85clLHsgtUfMcdPvsyS8v/YvR1i8DoBEJ9IPkxN2u0vuwit7PHjW 0sJxxblJZrB7KkOqU1b0dfOUL9UHShkTB0mSnWLji3AfQ9OTOaZ/hHsUZgWPx7x4lUSv frmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709047179; x=1709651979; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RryqSy7p/WJgmbuDaBcFTKEm0duOKiPOTBDTU9e1Dgc=; b=vCYWr+oPj8tdnupGmL8wIq8D2JzgEWTQ6F2im+BFf1fvQOT51ruIC7sYN8/Lhx2VVR Q1UJRYt97WP4ihYbrVCgveEwuCyzJoOHCNyWF2/cLMWZTalp/ETwMxfbVwjwxT8BZPru OiV6Rirgnw0DQ6nW38RbgIsIz8ojR4Uhjh//8IoRCNEzrWuZx7vZHX3K3iOVM7FeVR/K csRaXN3E8r/WRTCpNdso5jFgVSkhpllAjsuwVkVCUsu+kWBcBbQ85rmW/+3nrhhw+A+x /s0Ljyg3eXIQizp4g1Dp97jv+v/ASFNob2NSvOXyTk9AU+OCBiIeRVf9ZB4o8rXB8u3t i3nA== X-Gm-Message-State: AOJu0YxWGetFvKu2+UUpZbmUXDRRkCsEdjdCO2jKzdCIU8+V5/Q8fKsP z5prk+LtPrE2t7yDYbct9+RI66tYrHRWOlY+U4s3vPK+3dnPboktv5QqxdaXIiPf8S3oFlRAqRK YvvRmMtPQmlxewv6/3v45ImGGWRhKlqG6u9iFMvAlQ0vZLWTRXmdbspgyhS/AeOdRjVkLAROQ9h 7XPhCAverpYrhSvDgDyDE/wL7fmK+shg== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:1d97:b0:33d:f5e5:ac4a with SMTP id bk23-20020a0560001d9700b0033df5e5ac4amr1631wrb.4.1709047179516; Tue, 27 Feb 2024 07:19:39 -0800 (PST) Date: Tue, 27 Feb 2024 16:19:15 +0100 In-Reply-To: <20240227151907.387873-11-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240227151907.387873-11-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc1.240.g4c46232300-goog Message-ID: <20240227151907.387873-18-ardb+git@google.com> Subject: [PATCH v7 7/9] x86/sme: Move early SME kernel encryption handling into .head.text From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792068159578186468 X-GMAIL-MSGID: 1792068159578186468 From: Ard Biesheuvel The .head.text section is the initial primary entrypoint of the core kernel, and is entered with the CPU executing from a 1:1 mapping of memory. Such code must never access global variables using absolute references, as these are based on the kernel virtual mapping which is not active yet at this point. Given that the SME startup code is also called from this early execution context, move it into .head.text as well. This will allow more thorough build time checks in the future to ensure that early startup code only uses RIP-relative references to global variables. Also replace some occurrences of __pa_symbol() [which relies on the compiler generating an absolute reference, which is not guaranteed] and an open coded RIP-relative access with RIP_REL_REF(). Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/mem_encrypt.h | 8 ++-- arch/x86/mm/mem_encrypt_identity.c | 42 ++++++++------------ 2 files changed, 21 insertions(+), 29 deletions(-) diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index b31eb9fd5954..f922b682b9b4 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -47,8 +47,8 @@ void __init sme_unmap_bootdata(char *real_mode_data); void __init sme_early_init(void); -void __init sme_encrypt_kernel(struct boot_params *bp); -void __init sme_enable(struct boot_params *bp); +void sme_encrypt_kernel(struct boot_params *bp); +void sme_enable(struct boot_params *bp); int __init early_set_memory_decrypted(unsigned long vaddr, unsigned long size); int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long size); @@ -81,8 +81,8 @@ static inline void __init sme_unmap_bootdata(char *real_mode_data) { } static inline void __init sme_early_init(void) { } -static inline void __init sme_encrypt_kernel(struct boot_params *bp) { } -static inline void __init sme_enable(struct boot_params *bp) { } +static inline void sme_encrypt_kernel(struct boot_params *bp) { } +static inline void sme_enable(struct boot_params *bp) { } static inline void sev_es_init_vc_handling(void) { } diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index d210c7fc8fa2..64b5005d49e5 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -41,6 +41,7 @@ #include #include +#include #include #include #include @@ -94,7 +95,7 @@ struct sme_populate_pgd_data { */ static char sme_workarea[2 * PMD_SIZE] __section(".init.scratch"); -static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) +static void __head sme_clear_pgd(struct sme_populate_pgd_data *ppd) { unsigned long pgd_start, pgd_end, pgd_size; pgd_t *pgd_p; @@ -109,7 +110,7 @@ static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) memset(pgd_p, 0, pgd_size); } -static pud_t __init *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) +static pud_t __head *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) { pgd_t *pgd; p4d_t *p4d; @@ -146,7 +147,7 @@ static pud_t __init *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) return pud; } -static void __init sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) +static void __head sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) { pud_t *pud; pmd_t *pmd; @@ -162,7 +163,7 @@ static void __init sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) set_pmd(pmd, __pmd(ppd->paddr | ppd->pmd_flags)); } -static void __init sme_populate_pgd(struct sme_populate_pgd_data *ppd) +static void __head sme_populate_pgd(struct sme_populate_pgd_data *ppd) { pud_t *pud; pmd_t *pmd; @@ -188,7 +189,7 @@ static void __init sme_populate_pgd(struct sme_populate_pgd_data *ppd) set_pte(pte, __pte(ppd->paddr | ppd->pte_flags)); } -static void __init __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) +static void __head __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) { while (ppd->vaddr < ppd->vaddr_end) { sme_populate_pgd_large(ppd); @@ -198,7 +199,7 @@ static void __init __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) } } -static void __init __sme_map_range_pte(struct sme_populate_pgd_data *ppd) +static void __head __sme_map_range_pte(struct sme_populate_pgd_data *ppd) { while (ppd->vaddr < ppd->vaddr_end) { sme_populate_pgd(ppd); @@ -208,7 +209,7 @@ static void __init __sme_map_range_pte(struct sme_populate_pgd_data *ppd) } } -static void __init __sme_map_range(struct sme_populate_pgd_data *ppd, +static void __head __sme_map_range(struct sme_populate_pgd_data *ppd, pmdval_t pmd_flags, pteval_t pte_flags) { unsigned long vaddr_end; @@ -232,22 +233,22 @@ static void __init __sme_map_range(struct sme_populate_pgd_data *ppd, __sme_map_range_pte(ppd); } -static void __init sme_map_range_encrypted(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_encrypted(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_ENC, PTE_FLAGS_ENC); } -static void __init sme_map_range_decrypted(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_decrypted(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_DEC, PTE_FLAGS_DEC); } -static void __init sme_map_range_decrypted_wp(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_decrypted_wp(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_DEC_WP, PTE_FLAGS_DEC_WP); } -static unsigned long __init sme_pgtable_calc(unsigned long len) +static unsigned long __head sme_pgtable_calc(unsigned long len) { unsigned long entries = 0, tables = 0; @@ -284,7 +285,7 @@ static unsigned long __init sme_pgtable_calc(unsigned long len) return entries + tables; } -void __init sme_encrypt_kernel(struct boot_params *bp) +void __head sme_encrypt_kernel(struct boot_params *bp) { unsigned long workarea_start, workarea_end, workarea_len; unsigned long execute_start, execute_end, execute_len; @@ -319,9 +320,8 @@ void __init sme_encrypt_kernel(struct boot_params *bp) * memory from being cached. */ - /* Physical addresses gives us the identity mapped virtual addresses */ - kernel_start = __pa_symbol(_text); - kernel_end = ALIGN(__pa_symbol(_end), PMD_SIZE); + kernel_start = (unsigned long)RIP_REL_REF(_text); + kernel_end = ALIGN((unsigned long)RIP_REL_REF(_end), PMD_SIZE); kernel_len = kernel_end - kernel_start; initrd_start = 0; @@ -338,14 +338,6 @@ void __init sme_encrypt_kernel(struct boot_params *bp) } #endif - /* - * We're running identity mapped, so we must obtain the address to the - * SME encryption workarea using rip-relative addressing. - */ - asm ("lea sme_workarea(%%rip), %0" - : "=r" (workarea_start) - : "p" (sme_workarea)); - /* * Calculate required number of workarea bytes needed: * executable encryption area size: @@ -355,7 +347,7 @@ void __init sme_encrypt_kernel(struct boot_params *bp) * pagetable structures for the encryption of the kernel * pagetable structures for workarea (in case not currently mapped) */ - execute_start = workarea_start; + execute_start = workarea_start = (unsigned long)RIP_REL_REF(sme_workarea); execute_end = execute_start + (PAGE_SIZE * 2) + PMD_SIZE; execute_len = execute_end - execute_start; @@ -498,7 +490,7 @@ void __init sme_encrypt_kernel(struct boot_params *bp) native_write_cr3(__native_read_cr3()); } -void __init sme_enable(struct boot_params *bp) +void __head sme_enable(struct boot_params *bp) { unsigned int eax, ebx, ecx, edx; unsigned long feature_mask; From patchwork Tue Feb 27 15:19:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 207280 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2766648dyb; Tue, 27 Feb 2024 07:21:24 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXLv5apRhUUAIpo5nY7ZpRePcr9aR3rFmMPTb84HWB9+rPFHXJrTZwMsZMyz1WiFVU6ef7ZGnsl4BRzXHOGewrKz9T3nQ== X-Google-Smtp-Source: AGHT+IEV5OgIemjD0lOAFJMVOjqVMEBpCrOyQrZdMSRxz8yENyVICX3V5R/c/Y5afvS42iLGiGhI X-Received: by 2002:a05:6358:6586:b0:17b:c8d:f3a7 with SMTP id x6-20020a056358658600b0017b0c8df3a7mr14312844rwh.32.1709047284114; Tue, 27 Feb 2024 07:21:24 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709047284; cv=pass; d=google.com; s=arc-20160816; b=RTx87cKNTvwbiDGQPlFmrcIDLQueBbxBpN5PJRVzYXCD08a/bvuqS2D68Zl75e27VD HeZu4/alJrQIbv2pbKpRK8HdmYt9qtIhw45xhW8SjFImYg66+uRs7RyJC/6bVSpX41UF yefXlEygmHC+qhRgSUcT1sC6gBYKU5S3jbt3NmJRLENiA1o8/Ase5VTITAHdsyF7B1Gp zoSq12Feg3nxKfJmM4sZX42gsHqgxYk7TknAKoIUxj6Li/bLaxCx+ui6CjRO6qvZ4WqW D+61jGttVgf/Ye3CqQ85aT3Tla/59b9ctmZ9/jxBKYCkZQROSRZQq0xxhZLwlzXz+rRO TjnA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=UkoA/X/6iL1vFPkw1X4qRiTS802RjEN54BVOjq0PpUU=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=hLl3oK6DvEFcqeiKrEdm3neRpKSx6CZf+0DfNNPYWywlkyqDRlSbhyFsbFo5Ht58Ur N+TsdjnZ5oUWvlSHRRucBuveB8qQjvEtG+C1OP+zuRTY8hQUNjcbSd9KOII6DMbqIRFY w9XO+cAkBH57kzzD2nmw3MelU3gWxHfTHoiGzxbaRjy7W4MIImZz4GVxSRtHr0YebIyJ UDXD4dlxaf8awH6ziWLbHL+ywM8JVNxwZeP4ZkALM1koWrjcda1mNOC224AAvYFFAFss w8A5inVqPBjYYGzZaHY107yxEEXbr79hVSKADgJ9mnpyUqPHLot4vm8bXgWIMQv06sBc g4mA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=XxXr6N5J; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83494-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83494-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id vy6-20020a05620a490600b00787cc06dc29si5772685qkn.319.2024.02.27.07.21.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 07:21:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-83494-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=XxXr6N5J; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83494-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83494-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id CEB5D1C22214 for ; Tue, 27 Feb 2024 15:21:23 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5CC50149001; Tue, 27 Feb 2024 15:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="XxXr6N5J" Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE0B11487D3 for ; Tue, 27 Feb 2024 15:19:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047184; cv=none; b=daY1RcXB1AJOo17GCNadDXZRXxt1g1MMjlR+StADVGkRGX2sNl8eRnVqt33w20z9cFVOp7NiO1j7uDTs4yPWdCoxtauoe/UMM24V1ht9zkLBSSkRkbByyd29mHf8n3o8F10UHm353jYJMlEcsUQIP+qZWzQbiJF9sPpCHFYb2+E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047184; c=relaxed/simple; bh=6uCfH14imCu/66Z1+Fy+FGTIoV3qpsx11QnjUiEPzRU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=LhJ1Qx1XfzwcfLmOL2nmoYvcPxbz/9nEE14KMQ54rmY0MSd6tDLu68s6K65Smz9LpoCDrjt/VtqkSRDfIHCXNdr6fWaRmt0BQqjj8yj29KprwHXwaoUZ3Z702ibx21+h2aRs8GTIks4HTPV2BzDA7QtrortBn88LaMs8o23ur7Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=XxXr6N5J; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60810219282so52993047b3.0 for ; Tue, 27 Feb 2024 07:19:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709047181; x=1709651981; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UkoA/X/6iL1vFPkw1X4qRiTS802RjEN54BVOjq0PpUU=; b=XxXr6N5JFHCCCrMqDeowftOILg0K1Y02uPTFfPWmveqE+FnLjdTYBa78+9dj8E3iju l4ttlpW8GZh3uyEzFDMbvRFSvODVu5+r2h8NgO6kXSSmuZrl28OJ2TBcuF6JQf+cLZKj 3azqSCnzYTPHP1xhnMCWm8NIpHH63X6NzN2+D3lMEEeZ3T2uPY1CQV7MhO+J14/Kj1Ik uDGeGbPWYdx2bzrZ20PFhqL8z7f6sAC6kFEBEGRq7p9oi6hN4riioMh0K24/OIQ0N5ga vQMv7xfZIvbHv4JBq/N+nzPxD/el9FxyNAt9F5UGjAk/kxEP6TIQ5Qp27O+tAP+bICC2 M0UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709047181; x=1709651981; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UkoA/X/6iL1vFPkw1X4qRiTS802RjEN54BVOjq0PpUU=; b=VR/MdYSyozxz9jt31fZd3tb/XKe9V5Sj40G6VaW5QouBdDIQnJU3T7t4S3wbzMj8XF n+pOUdNGoFS5ojLkjx5Lii8rhsEKdpTwvIUZjRmXixqqFrTUzMLCk6A5mpfFCc5UDopx +AK+jcr5nrgnTRUUEjvoULjbEsQ5Do5XjU0NFyBqAdSbE70+4GGKDa8wIHcFMMw1fNPF lfiW2YU6WyDiVQDtRHjV5NP7K+W5vVK5uIXXlF175kvZTHu59wr5GAu8U6x5b5Mts/vy T9iUAzV6S3NCQE0ttqogRpy/m7YE0nStBRyTVPo+ilq71vgXJzm9NbFVVpN4TwJ8wx/u ZrcA== X-Gm-Message-State: AOJu0Yxl8QoDupJmpysr/mI4b4Cojbp1vbieEl3I9D3FOOlqyjqvTBuN cpigZIvZa3T/5Dp7rR6E15YxgHq52c1iafqlge60BO1XDAGh+JMbtVLpIPpPKq/CQLwvUNSEMDj r49tgN1YAGglj6H9JYhqwbSF8NJC52obWZXQPzSp+TwFk2attCOdMtKe3qp3vLBq+36Z2JTMcRH gJFQ+bH9sNmAWr2qI8usgcMuME/dko/g== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1889:b0:dc6:e20f:80cb with SMTP id cj9-20020a056902188900b00dc6e20f80cbmr97731ybb.3.1709047181682; Tue, 27 Feb 2024 07:19:41 -0800 (PST) Date: Tue, 27 Feb 2024 16:19:16 +0100 In-Reply-To: <20240227151907.387873-11-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240227151907.387873-11-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc1.240.g4c46232300-goog Message-ID: <20240227151907.387873-19-ardb+git@google.com> Subject: [PATCH v7 8/9] x86/sev: Move early startup code into .head.text section From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792065964641476865 X-GMAIL-MSGID: 1792065964641476865 From: Ard Biesheuvel In preparation for implementing rigorous build time checks to enforce that only code that can support it will be called from the early 1:1 mapping of memory, move SEV init code that is called in this manner to the .head.text section. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/sev.c | 3 +++ arch/x86/include/asm/sev.h | 10 ++++----- arch/x86/kernel/sev-shared.c | 23 +++++++++----------- arch/x86/kernel/sev.c | 14 +++++++----- 4 files changed, 26 insertions(+), 24 deletions(-) diff --git a/arch/x86/boot/compressed/sev.c b/arch/x86/boot/compressed/sev.c index 073291832f44..bea0719d70f2 100644 --- a/arch/x86/boot/compressed/sev.c +++ b/arch/x86/boot/compressed/sev.c @@ -116,6 +116,9 @@ static bool fault_in_kernel_space(unsigned long address) #undef __init #define __init +#undef __head +#define __head + #define __BOOT_COMPRESSED /* Basic instruction decoding support needed */ diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h index bed95e1f4d52..cf671138feef 100644 --- a/arch/x86/include/asm/sev.h +++ b/arch/x86/include/asm/sev.h @@ -213,16 +213,16 @@ static inline int pvalidate(unsigned long vaddr, bool rmp_psize, bool validate) struct snp_guest_request_ioctl; void setup_ghcb(void); -void __init early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, - unsigned long npages); -void __init early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr, - unsigned long npages); +void early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, + unsigned long npages); +void early_snp_set_memory_shared(unsigned long vaddr, unsigned long paddr, + unsigned long npages); void __init snp_prep_memory(unsigned long paddr, unsigned int sz, enum psc_op op); void snp_set_memory_shared(unsigned long vaddr, unsigned long npages); void snp_set_memory_private(unsigned long vaddr, unsigned long npages); void snp_set_wakeup_secondary_cpu(void); bool snp_init(struct boot_params *bp); -void __init __noreturn snp_abort(void); +void __noreturn snp_abort(void); int snp_issue_guest_request(u64 exit_code, struct snp_req_data *input, struct snp_guest_request_ioctl *rio); void snp_accept_memory(phys_addr_t start, phys_addr_t end); u64 snp_get_unsupported_features(u64 status); diff --git a/arch/x86/kernel/sev-shared.c b/arch/x86/kernel/sev-shared.c index ae79f9505298..0bd7ccbe8732 100644 --- a/arch/x86/kernel/sev-shared.c +++ b/arch/x86/kernel/sev-shared.c @@ -93,7 +93,8 @@ static bool __init sev_es_check_cpu_features(void) return true; } -static void __noreturn sev_es_terminate(unsigned int set, unsigned int reason) +static void __head __noreturn +sev_es_terminate(unsigned int set, unsigned int reason) { u64 val = GHCB_MSR_TERM_REQ; @@ -330,13 +331,7 @@ static int sev_cpuid_hv(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid */ static const struct snp_cpuid_table *snp_cpuid_get_table(void) { - void *ptr; - - asm ("lea cpuid_table_copy(%%rip), %0" - : "=r" (ptr) - : "p" (&cpuid_table_copy)); - - return ptr; + return &RIP_REL_REF(cpuid_table_copy); } /* @@ -395,7 +390,7 @@ static u32 snp_cpuid_calc_xsave_size(u64 xfeatures_en, bool compacted) return xsave_size; } -static bool +static bool __head snp_cpuid_get_validated_func(struct cpuid_leaf *leaf) { const struct snp_cpuid_table *cpuid_table = snp_cpuid_get_table(); @@ -532,7 +527,8 @@ static int snp_cpuid_postprocess(struct ghcb *ghcb, struct es_em_ctxt *ctxt, * Returns -EOPNOTSUPP if feature not enabled. Any other non-zero return value * should be treated as fatal by caller. */ -static int snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf) +static int __head +snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_leaf *leaf) { const struct snp_cpuid_table *cpuid_table = snp_cpuid_get_table(); @@ -574,7 +570,7 @@ static int snp_cpuid(struct ghcb *ghcb, struct es_em_ctxt *ctxt, struct cpuid_le * page yet, so it only supports the MSR based communication with the * hypervisor and only the CPUID exit-code. */ -void __init do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code) +void __head do_vc_no_ghcb(struct pt_regs *regs, unsigned long exit_code) { unsigned int subfn = lower_bits(regs->cx, 32); unsigned int fn = lower_bits(regs->ax, 32); @@ -1025,7 +1021,8 @@ struct cc_setup_data { * Search for a Confidential Computing blob passed in as a setup_data entry * via the Linux Boot Protocol. */ -static struct cc_blob_sev_info *find_cc_blob_setup_data(struct boot_params *bp) +static __head +struct cc_blob_sev_info *find_cc_blob_setup_data(struct boot_params *bp) { struct cc_setup_data *sd = NULL; struct setup_data *hdr; @@ -1052,7 +1049,7 @@ static struct cc_blob_sev_info *find_cc_blob_setup_data(struct boot_params *bp) * mapping needs to be updated in sync with all the changes to virtual memory * layout and related mapping facilities throughout the boot process. */ -static void __init setup_cpuid_table(const struct cc_blob_sev_info *cc_info) +static void __head setup_cpuid_table(const struct cc_blob_sev_info *cc_info) { const struct snp_cpuid_table *cpuid_table_fw, *cpuid_table; int i; diff --git a/arch/x86/kernel/sev.c b/arch/x86/kernel/sev.c index 1ef7ae806a01..33c14aa1f06c 100644 --- a/arch/x86/kernel/sev.c +++ b/arch/x86/kernel/sev.c @@ -25,6 +25,7 @@ #include #include +#include #include #include #include @@ -682,8 +683,9 @@ static u64 __init get_jump_table_addr(void) return ret; } -static void early_set_pages_state(unsigned long vaddr, unsigned long paddr, - unsigned long npages, enum psc_op op) +static void __head +early_set_pages_state(unsigned long vaddr, unsigned long paddr, + unsigned long npages, enum psc_op op) { unsigned long paddr_end; u64 val; @@ -739,7 +741,7 @@ static void early_set_pages_state(unsigned long vaddr, unsigned long paddr, sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_PSC); } -void __init early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, +void __head early_snp_set_memory_private(unsigned long vaddr, unsigned long paddr, unsigned long npages) { /* @@ -2062,7 +2064,7 @@ bool __init handle_vc_boot_ghcb(struct pt_regs *regs) * * Scan for the blob in that order. */ -static __init struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp) +static __head struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp) { struct cc_blob_sev_info *cc_info; @@ -2088,7 +2090,7 @@ static __init struct cc_blob_sev_info *find_cc_blob(struct boot_params *bp) return cc_info; } -bool __init snp_init(struct boot_params *bp) +bool __head snp_init(struct boot_params *bp) { struct cc_blob_sev_info *cc_info; @@ -2110,7 +2112,7 @@ bool __init snp_init(struct boot_params *bp) return true; } -void __init __noreturn snp_abort(void) +void __head __noreturn snp_abort(void) { sev_es_terminate(SEV_TERM_SET_GEN, GHCB_SNP_UNSUPPORTED); } From patchwork Tue Feb 27 15:19:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 207281 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2767019dyb; Tue, 27 Feb 2024 07:21:58 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWdSeHjT9inL0MWcFmEGIyE2E2PUipa+egNNF9T6xHs+ZD2td3lMQq0QWksuWY9ofkp2fYwAPtQyDhkiTZjj1r37Q3RTQ== X-Google-Smtp-Source: AGHT+IECc216FqvxxyvZCyMU44U9mHdIUraLvyok8j2ooGCBelC0qxFlRZQxlZnDkBkHQy68ed87 X-Received: by 2002:a17:906:918:b0:a3f:ae09:5f8a with SMTP id i24-20020a170906091800b00a3fae095f8amr9034643ejd.14.1709047318509; Tue, 27 Feb 2024 07:21:58 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709047318; cv=pass; d=google.com; s=arc-20160816; b=gw2q3cSdOS0zHBY3Xso5SD6LCC+clc/+wxfKs+lhG4uqtb7fIpbm4YywGEB+uyR69o lfAgO9EnMK1fee5AdUt9a04k5FTQYy+KtminvmCTLQnjQ9pNQrHoEOcMptU5GURyo0/4 DBK5jut5DnehQ15rRhdkmTXsFo3ejklL7tckt/nYZI6grsvUt4TlVdBA1mmeJu7RPJqQ e/dkHcdRW89DKQbARKlrwiXXhoJLznCEti01yT5o+8T4EztakW9UchfLnfpKtEvPL93T HAgiXd/EjYKKqh1LwJ1c7bR9eSXzW/e9Yt2Pjap6XTsCHFPFsEc/Vuz1dNGwfQcRjzFk wsUA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=ZYuGtKHJMVquxPkGaFI0DQjYYMZ/RRITtvpZGgefhyk=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=1Cc6SV3BF3jnBrYcYJ7S0ypRDqFq/cUeKuBf5fTMzn4g6Plz8kouOLtoUXXPIw+lx/ iSq2NH0wwl39f91zOXHpP+Qx01+nV+7iJc08shBEQLCBbgq8dw2BSBMzTXpimwvIP3X/ uvYmpvFkvBWRJPpbgPT9kBAq7BdRD6moJOiMiyXC+Nxl60IJ/jEtyZQGq8cR93l3cyuF XpOC50GBKOEGICmaPIzXMbwXEZFXAx/p6UcSkSM4onDCtLr54dFJoxVkValNeWmb8plc H8he1fLBJ9p+kCU+6e0Zec6wIzDxp+l8KiG9vYtuTulzyaGCu71u/gXwSX+1d/6F4VgK LmMw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=0zkaYKEt; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83495-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83495-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id b26-20020a170906491a00b00a430d2df4e2si831022ejq.827.2024.02.27.07.21.58 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 07:21:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-83495-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=0zkaYKEt; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83495-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83495-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 13A3D1F23F60 for ; Tue, 27 Feb 2024 15:21:51 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id DA4BF149E10; Tue, 27 Feb 2024 15:19:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="0zkaYKEt" Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A78FD148FF6 for ; Tue, 27 Feb 2024 15:19:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047187; cv=none; b=C4gqx9c1c8z2x9j9CGlo2ob0PR39B3O/wXLIzQU3vrH7e/alyNkrzPUzvU/gBFRz993ycUPPnEEqzwSYlZ+kNEyCSyUqqRjx4eqEAOruu7VZMvNn4TZoXJ0mxbrODK3lD9Yk8/nj6n9lPDDOTDBuAzrsBZovx3d4SRLi7NZp2Bg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047187; c=relaxed/simple; bh=qSjwtFzc0l5zAcSrUDslu+YsQKRi8QzKlxiS3GF6OKk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=cosFl6wHtaPoAJawRP7XLycUfnIGpa+t35mOmjkTiaE0AwD0EEGiqebxvF4mDuP+iCu19vx9lOcTkEhZfQkWsiT7fdLNUkckrHIbA2hBZNDDtNrRS086ZG3QfguQCNZtUmOsZTyxhbcUkJcZWHGgs67ocihzh5IOKRoQeoAZ0Gk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=0zkaYKEt; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-33d23c8694dso2020830f8f.1 for ; Tue, 27 Feb 2024 07:19:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709047184; x=1709651984; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZYuGtKHJMVquxPkGaFI0DQjYYMZ/RRITtvpZGgefhyk=; b=0zkaYKEt6VGQR04lMGpSiyjYy49rSyKkd+qUITHwg/WOBNYR2nbH6+InWoO5HlbBId axHvxbiqpazuYYcry2KCVqFwCns0shpmL/m3Ce08xQn/zIYR5mARC/ZaM4TGW4N6CrNT wQnAftOvio6vi+eMxRgk7OXElLbMxBfXiP5N1CcSWR0eu7RH7LwZ6prlfiovfMgZHfUQ a8tcHopNhBbngjyFEOArChy+0SW+k4RjOIkzyPJTjNty5OIZ0GC74rcI3lVJpkUQhXLZ Z9v8BBncwbEOwX76ER0+a76JLUfDMkIfWIt4Iq5AsF5Yio4epnb0vxqgQwZ2U7bgjZCR IjlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709047184; x=1709651984; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZYuGtKHJMVquxPkGaFI0DQjYYMZ/RRITtvpZGgefhyk=; b=XCKpboIMnc20BxhLVqVbQS1Lp70NV1GA7GKVVtzxOm36bQ1vycvxv5Jv5783tWIYBN nO4rkeYtCBwC4ECgCoXXKfArG/1s61ExZsO/bkd3Iu9rfk84wTtVN2mRk3HRErPc5JBw jV3/8kBol5YxqeMo+qxICwmoRG6Rv7rD9OiczpwfwItiA03MnOXUVVXxhrLZOOga1ptb XNqKRupJ4YkC8j1ODPkLZ11QSxnqFGSMKsDYAia51bTiaGargYmLj0Mdb6//4wglK2uq ci/FDhLiAnTkY9tZqvl6G/W6nPistPUDABfeR3Vm54kGDG+K5ut4ajwZECQ7yDqfhkRt hZMA== X-Gm-Message-State: AOJu0Yx5XLbLlIyZUmaLFeGN8Z5bgOAwGREPR8T5yBQLooB0khwDQ2/x zeEppUxGyk59uhP/ZQJm2icGNVQBGBbtVzzf0wc6b8m/2SOdzML72t1ap3xmj6fR+wx4iIZtdwr 8PbB5l/v3ymb6eVDUZThNdLcB6MyTmYFM0/Yo0N3pr/rtczzWwh8qckFGYd2mL6el5ssBBN3lPN D4kVTQo28xKPpnQkFUMLwA/UUWBjO9TA== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:253:b0:33d:2808:f647 with SMTP id m19-20020a056000025300b0033d2808f647mr59319wrz.3.1709047184002; Tue, 27 Feb 2024 07:19:44 -0800 (PST) Date: Tue, 27 Feb 2024 16:19:17 +0100 In-Reply-To: <20240227151907.387873-11-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240227151907.387873-11-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc1.240.g4c46232300-goog Message-ID: <20240227151907.387873-20-ardb+git@google.com> Subject: [PATCH v7 9/9] x86/startup_64: Drop global variables keeping track of LA57 state From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792066001033984044 X-GMAIL-MSGID: 1792066001033984044 From: Ard Biesheuvel On x86_64, the core kernel is entered in long mode, which implies that paging is enabled. This means that the CR4.LA57 control bit is guaranteed to be in sync with the number of paging levels used by the kernel, and there is no need to store this in a variable. There is also no need to use variables for storing the calculations of pgdir_shift and ptrs_per_p4d, as they are easily determined on the fly. This removes the need for two different sources of truth for determining whether 5-level paging is in use: CR4.LA57 always reflects the actual state, and never changes from the point of view of the 64-bit core kernel. The only potential concern is the cost of CR4 accesses, which can be mitigated using alternatives patching based on feature detection. Note that even the decompressor does not manipulate any page tables before updating CR4.LA57, so it can also avoid the associated global variables entirely. However, as it does not implement alternatives patching, the associated ELF sections need to be discarded. Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/misc.h | 4 -- arch/x86/boot/compressed/pgtable_64.c | 12 ------ arch/x86/boot/compressed/vmlinux.lds.S | 1 + arch/x86/include/asm/pgtable_64_types.h | 43 ++++++++++---------- arch/x86/kernel/cpu/common.c | 2 - arch/x86/kernel/head64.c | 33 +-------------- arch/x86/mm/kasan_init_64.c | 3 -- arch/x86/mm/mem_encrypt_identity.c | 9 ---- 8 files changed, 25 insertions(+), 82 deletions(-) diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/misc.h index b353a7be380c..e4ab7b4d8698 100644 --- a/arch/x86/boot/compressed/misc.h +++ b/arch/x86/boot/compressed/misc.h @@ -16,9 +16,6 @@ #define __NO_FORTIFY -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 - /* * Boot stub deals with identity mappings, physical and virtual addresses are * the same, so override these defines. @@ -181,7 +178,6 @@ static inline int count_immovable_mem_regions(void) { return 0; } #endif /* ident_map_64.c */ -extern unsigned int __pgtable_l5_enabled, pgdir_shift, ptrs_per_p4d; extern void kernel_add_identity_map(unsigned long start, unsigned long end); /* Used by PAGE_KERN* macros: */ diff --git a/arch/x86/boot/compressed/pgtable_64.c b/arch/x86/boot/compressed/pgtable_64.c index 51f957b24ba7..ae72f53f5e77 100644 --- a/arch/x86/boot/compressed/pgtable_64.c +++ b/arch/x86/boot/compressed/pgtable_64.c @@ -9,13 +9,6 @@ #define BIOS_START_MIN 0x20000U /* 128K, less than this is insane */ #define BIOS_START_MAX 0x9f000U /* 640K, absolute maximum */ -#ifdef CONFIG_X86_5LEVEL -/* __pgtable_l5_enabled needs to be in .data to avoid being cleared along with .bss */ -unsigned int __section(".data") __pgtable_l5_enabled; -unsigned int __section(".data") pgdir_shift = 39; -unsigned int __section(".data") ptrs_per_p4d = 1; -#endif - /* Buffer to preserve trampoline memory */ static char trampoline_save[TRAMPOLINE_32BIT_SIZE]; @@ -125,11 +118,6 @@ asmlinkage void configure_5level_paging(struct boot_params *bp, void *pgtable) native_cpuid_eax(0) >= 7 && (native_cpuid_ecx(7) & (1 << (X86_FEATURE_LA57 & 31)))) { l5_required = true; - - /* Initialize variables for 5-level paging */ - __pgtable_l5_enabled = 1; - pgdir_shift = 48; - ptrs_per_p4d = 512; } /* diff --git a/arch/x86/boot/compressed/vmlinux.lds.S b/arch/x86/boot/compressed/vmlinux.lds.S index 083ec6d7722a..06358bb067fe 100644 --- a/arch/x86/boot/compressed/vmlinux.lds.S +++ b/arch/x86/boot/compressed/vmlinux.lds.S @@ -81,6 +81,7 @@ SECTIONS *(.dynamic) *(.dynsym) *(.dynstr) *(.dynbss) *(.hash) *(.gnu.hash) *(.note.*) + *(.altinstructions .altinstr_replacement) } .got.plt (INFO) : { diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm/pgtable_64_types.h index 9053dfe9fa03..2fac8ba9564a 100644 --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -6,7 +6,10 @@ #ifndef __ASSEMBLY__ #include +#include +#include #include +#include /* * These are used to make use of C type-checking.. @@ -21,28 +24,24 @@ typedef unsigned long pgprotval_t; typedef struct { pteval_t pte; } pte_t; typedef struct { pmdval_t pmd; } pmd_t; -extern unsigned int __pgtable_l5_enabled; - -#ifdef CONFIG_X86_5LEVEL -#ifdef USE_EARLY_PGTABLE_L5 -/* - * cpu_feature_enabled() is not available in early boot code. - * Use variable instead. - */ -static inline bool pgtable_l5_enabled(void) +static __always_inline __pure bool pgtable_l5_enabled(void) { - return __pgtable_l5_enabled; -} -#else -#define pgtable_l5_enabled() cpu_feature_enabled(X86_FEATURE_LA57) -#endif /* USE_EARLY_PGTABLE_L5 */ + unsigned long r; + bool ret; -#else -#define pgtable_l5_enabled() 0 -#endif /* CONFIG_X86_5LEVEL */ + if (!IS_ENABLED(CONFIG_X86_5LEVEL)) + return false; -extern unsigned int pgdir_shift; -extern unsigned int ptrs_per_p4d; + asm(ALTERNATIVE_TERNARY( + "movq %%cr4, %[reg] \n\t btl %[la57], %k[reg]" CC_SET(c), + %P[feat], "stc", "clc") + : [reg] "=&r" (r), CC_OUT(c) (ret) + : [feat] "i" (X86_FEATURE_LA57), + [la57] "i" (X86_CR4_LA57_BIT) + : "cc"); + + return ret; +} #endif /* !__ASSEMBLY__ */ @@ -53,7 +52,7 @@ extern unsigned int ptrs_per_p4d; /* * PGDIR_SHIFT determines what a top-level page table entry can map */ -#define PGDIR_SHIFT pgdir_shift +#define PGDIR_SHIFT (pgtable_l5_enabled() ? 48 : 39) #define PTRS_PER_PGD 512 /* @@ -61,7 +60,7 @@ extern unsigned int ptrs_per_p4d; */ #define P4D_SHIFT 39 #define MAX_PTRS_PER_P4D 512 -#define PTRS_PER_P4D ptrs_per_p4d +#define PTRS_PER_P4D (pgtable_l5_enabled() ? 512 : 1) #define P4D_SIZE (_AC(1, UL) << P4D_SHIFT) #define P4D_MASK (~(P4D_SIZE - 1)) @@ -76,6 +75,8 @@ extern unsigned int ptrs_per_p4d; #define PTRS_PER_PGD 512 #define MAX_PTRS_PER_P4D 1 +#define MAX_POSSIBLE_PHYSMEM_BITS 46 + #endif /* CONFIG_X86_5LEVEL */ /* diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 9e35e276c55a..d88e4be88868 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1,6 +1,4 @@ // SPDX-License-Identifier: GPL-2.0-only -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 #include #include diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index deaaea3280d9..789ed2c53527 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -7,9 +7,6 @@ #define DISABLE_BRANCH_PROFILING -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include @@ -52,14 +49,6 @@ extern pmd_t early_dynamic_pgts[EARLY_DYNAMIC_PAGE_TABLES][PTRS_PER_PMD]; static unsigned int __initdata next_early_pgt; pmdval_t early_pmd_flags = __PAGE_KERNEL_LARGE & ~(_PAGE_GLOBAL | _PAGE_NX); -#ifdef CONFIG_X86_5LEVEL -unsigned int __pgtable_l5_enabled __ro_after_init; -unsigned int pgdir_shift __ro_after_init = 39; -EXPORT_SYMBOL(pgdir_shift); -unsigned int ptrs_per_p4d __ro_after_init = 1; -EXPORT_SYMBOL(ptrs_per_p4d); -#endif - #ifdef CONFIG_DYNAMIC_MEMORY_LAYOUT unsigned long page_offset_base __ro_after_init = __PAGE_OFFSET_BASE_L4; EXPORT_SYMBOL(page_offset_base); @@ -78,21 +67,6 @@ static struct desc_struct startup_gdt[GDT_ENTRIES] __initdata = { [GDT_ENTRY_KERNEL_DS] = GDT_ENTRY_INIT(DESC_DATA64, 0, 0xfffff), }; -static inline bool check_la57_support(void) -{ - if (!IS_ENABLED(CONFIG_X86_5LEVEL)) - return false; - - /* - * 5-level paging is detected and enabled at kernel decompression - * stage. Only check if it has been enabled there. - */ - if (!(native_read_cr4() & X86_CR4_LA57)) - return false; - - return true; -} - static unsigned long __head sme_postprocess_startup(struct boot_params *bp, pmdval_t *pmd) { unsigned long vaddr, vaddr_end; @@ -155,7 +129,7 @@ unsigned long __head __startup_64(unsigned long physaddr, bool la57; int i; - la57 = check_la57_support(); + la57 = pgtable_l5_enabled(); /* Is the address too large? */ if (physaddr >> MAX_PHYSMEM_BITS) @@ -440,10 +414,7 @@ asmlinkage __visible void __init __noreturn x86_64_start_kernel(char * real_mode (__START_KERNEL & PGDIR_MASK))); BUILD_BUG_ON(__fix_to_virt(__end_of_fixed_addresses) <= MODULES_END); - if (check_la57_support()) { - __pgtable_l5_enabled = 1; - pgdir_shift = 48; - ptrs_per_p4d = 512; + if (pgtable_l5_enabled()) { page_offset_base = __PAGE_OFFSET_BASE_L5; vmalloc_base = __VMALLOC_BASE_L5; vmemmap_base = __VMEMMAP_BASE_L5; diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c index 0302491d799d..85ae1ef840cc 100644 --- a/arch/x86/mm/kasan_init_64.c +++ b/arch/x86/mm/kasan_init_64.c @@ -2,9 +2,6 @@ #define DISABLE_BRANCH_PROFILING #define pr_fmt(fmt) "kasan: " fmt -/* cpu_feature_enabled() cannot be used this early */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index 64b5005d49e5..a857945af177 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -27,15 +27,6 @@ #undef CONFIG_PARAVIRT_XXL #undef CONFIG_PARAVIRT_SPINLOCKS -/* - * This code runs before CPU feature bits are set. By default, the - * pgtable_l5_enabled() function uses bit X86_FEATURE_LA57 to determine if - * 5-level paging is active, so that won't work here. USE_EARLY_PGTABLE_L5 - * is provided to handle this situation and, instead, use a variable that - * has been set by the early boot code. - */ -#define USE_EARLY_PGTABLE_L5 - #include #include #include