From patchwork Tue Feb 27 15:19:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 207293 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:a81b:b0:108:e6aa:91d0 with SMTP id bq27csp2787278dyb; Tue, 27 Feb 2024 07:56:17 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVZf+VijdQQa5JpaneGtYGDKgukP1J9c20Z+La8FLqY2o0DXLTAPbw9tNtpWxX83Q8SIJNyj0FLlHvni+e2R7B9l2gTdg== X-Google-Smtp-Source: AGHT+IEEKIZ1fGcDsIY1ygdfDNdVwOISLRm8xJSG1TU9AX7RtYOky5tIKnYTEGCwlZMn+HSpBywZ X-Received: by 2002:a17:903:496:b0:1dc:3ab7:cc78 with SMTP id jj22-20020a170903049600b001dc3ab7cc78mr9185405plb.29.1709049376879; Tue, 27 Feb 2024 07:56:16 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709049376; cv=pass; d=google.com; s=arc-20160816; b=s2cy0jtpvcsJaQUOEibbSzErBJe3M8SRaaAlRIEu3yvwGJirGbeXP5w7hZGBD9wLFu 9G2HUO5jH7XZrN2+lrERrhKWBTfOUbMeFrV3hAiy2A7dsSE3Ag9vo2Cdq5TeQBizwZ3f dASM7PulGnzyrfbogJY8DOt6oWnhFnDY3nJmptkPlphD9C1MMyiI0rONY3ZfmH0maRPg SflXQvERGLUa3Ij8UtLkoWTYNuRsGh/FKSnBrIdVySdtBbw6HutyAq+cdB3QWdl4FTrR bqVlg2mL5fSZQz13wIRcGkVyfB6CVIfwxVx875jhcgbj491uX3l9hQMMsODVNhm/IX+Z jqmQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:message-id:references:mime-version :list-unsubscribe:list-subscribe:list-id:precedence:in-reply-to:date :dkim-signature; bh=RryqSy7p/WJgmbuDaBcFTKEm0duOKiPOTBDTU9e1Dgc=; fh=GwInpN0wyaUEy1HIz2WqUneHJB46QWkS/TNrezKinns=; b=PK4nU2GS77XMWM9B+XUVWTnFEZE/e3UmjqRp0dK47NeHTjCdULEkz++4d8N7UhdBw6 5fNf1AX053M3d++pnN0ka+RjO5j6YzDGViJNg0dyiQp9xnkX4H5LvaZkyOvqt2sVuEsJ 6/I6y7Ln3Q4103gyFxMwFlL5jeNOU+Kk6QZ92m8fFh7yZ8NTn1VHrvMIPTkIFNzvon+s e5A1Qhz5LTb8gx6rP1+9E7jHQoND/MamisB//LLKci9IhQKruMnNEdNYfvj0bqoQ8ZVN hwLaSxo9bz3jHT6sTbDPHknZDeFnMmr/09XO0Gx4Ew6mU+y2C84Ir4fKUawzmASd61kr fcNA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=jfvqHmZr; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83493-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83493-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id q10-20020a170902a3ca00b001dbc1a37d6dsi1511105plb.462.2024.02.27.07.56.16 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 07:56:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-83493-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=jfvqHmZr; arc=pass (i=1 spf=pass spfdomain=flex--ardb.bounces.google.com dkim=pass dkdomain=google.com dmarc=pass fromdomain=google.com); spf=pass (google.com: domain of linux-kernel+bounces-83493-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-83493-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id F1BC0B2CCA1 for ; Tue, 27 Feb 2024 15:21:12 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 92A1E149009; Tue, 27 Feb 2024 15:19:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="jfvqHmZr" Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4F44145346 for ; Tue, 27 Feb 2024 15:19:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047182; cv=none; b=UrsKnRKm74GP1kpq7svD1Qe/kPHz6jtpNvFZZtJS3s1/7bEhNsVj70X++WsfHRzCCCxV8SQZNS3qvHX0jJY7Sg1T+SwVB7qf9mREc/o9U0SiX9NxD9KKrn90Yoge93vBTSUZ73k6EQ+xdMp0RCx93HND/MDwId+QUwACAZrdRj8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709047182; c=relaxed/simple; bh=n+M7WCortQTRWGbUCG2GXIYZAvcAmGDrlP8/0sO/Gq4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=dqDk+by19xt1pWJ4U2iVRZ3mvCyHrNqwf1hw+r2Hox9gkAvCOaWxyjFOii8Js+OetgoLzmcGhzSrhxa9/SfHowhkIvGmpQ5SZQj6OcKRWIchyJOXGDJ38yyBUy68ZzyyhaCOfrVVrgT618o6sSJ5O24T0Z3P7hiW6Un7WiAzONY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=jfvqHmZr; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ardb.bounces.google.com Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-33d9f425eaaso1828563f8f.3 for ; Tue, 27 Feb 2024 07:19:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709047179; x=1709651979; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=RryqSy7p/WJgmbuDaBcFTKEm0duOKiPOTBDTU9e1Dgc=; b=jfvqHmZryq7pGm9kN4o1/Yh7bhvXh4T3tk7Q2KZR6wG+kyzrdVdwywSdCyYo2je8Uq IV111pHYJg22fmqbkCKQ5HlSAmqBGCrpmtruvwd+1bVlBeWGmTG4P/WVvsiLbGk0f3uj EP6G61yRa04cVwLu9wHYLgM8i2P00QGLIPbeFWtPi1L2IFCUkglUDv9m0FHcl1Y3hz+u DMczkI6QXyre6gjw85clLHsgtUfMcdPvsyS8v/YvR1i8DoBEJ9IPkxN2u0vuwit7PHjW 0sJxxblJZrB7KkOqU1b0dfOUL9UHShkTB0mSnWLji3AfQ9OTOaZ/hHsUZgWPx7x4lUSv frmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709047179; x=1709651979; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RryqSy7p/WJgmbuDaBcFTKEm0duOKiPOTBDTU9e1Dgc=; b=vCYWr+oPj8tdnupGmL8wIq8D2JzgEWTQ6F2im+BFf1fvQOT51ruIC7sYN8/Lhx2VVR Q1UJRYt97WP4ihYbrVCgveEwuCyzJoOHCNyWF2/cLMWZTalp/ETwMxfbVwjwxT8BZPru OiV6Rirgnw0DQ6nW38RbgIsIz8ojR4Uhjh//8IoRCNEzrWuZx7vZHX3K3iOVM7FeVR/K csRaXN3E8r/WRTCpNdso5jFgVSkhpllAjsuwVkVCUsu+kWBcBbQ85rmW/+3nrhhw+A+x /s0Ljyg3eXIQizp4g1Dp97jv+v/ASFNob2NSvOXyTk9AU+OCBiIeRVf9ZB4o8rXB8u3t i3nA== X-Gm-Message-State: AOJu0YxWGetFvKu2+UUpZbmUXDRRkCsEdjdCO2jKzdCIU8+V5/Q8fKsP z5prk+LtPrE2t7yDYbct9+RI66tYrHRWOlY+U4s3vPK+3dnPboktv5QqxdaXIiPf8S3oFlRAqRK YvvRmMtPQmlxewv6/3v45ImGGWRhKlqG6u9iFMvAlQ0vZLWTRXmdbspgyhS/AeOdRjVkLAROQ9h 7XPhCAverpYrhSvDgDyDE/wL7fmK+shg== X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:1d97:b0:33d:f5e5:ac4a with SMTP id bk23-20020a0560001d9700b0033df5e5ac4amr1631wrb.4.1709047179516; Tue, 27 Feb 2024 07:19:39 -0800 (PST) Date: Tue, 27 Feb 2024 16:19:15 +0100 In-Reply-To: <20240227151907.387873-11-ardb+git@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240227151907.387873-11-ardb+git@google.com> X-Mailer: git-send-email 2.44.0.rc1.240.g4c46232300-goog Message-ID: <20240227151907.387873-18-ardb+git@google.com> Subject: [PATCH v7 7/9] x86/sme: Move early SME kernel encryption handling into .head.text From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: Ard Biesheuvel , Kevin Loughlin , Tom Lendacky , Dionna Glaze , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Andy Lutomirski , Brian Gerst X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792068159578186468 X-GMAIL-MSGID: 1792068159578186468 From: Ard Biesheuvel The .head.text section is the initial primary entrypoint of the core kernel, and is entered with the CPU executing from a 1:1 mapping of memory. Such code must never access global variables using absolute references, as these are based on the kernel virtual mapping which is not active yet at this point. Given that the SME startup code is also called from this early execution context, move it into .head.text as well. This will allow more thorough build time checks in the future to ensure that early startup code only uses RIP-relative references to global variables. Also replace some occurrences of __pa_symbol() [which relies on the compiler generating an absolute reference, which is not guaranteed] and an open coded RIP-relative access with RIP_REL_REF(). Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/mem_encrypt.h | 8 ++-- arch/x86/mm/mem_encrypt_identity.c | 42 ++++++++------------ 2 files changed, 21 insertions(+), 29 deletions(-) diff --git a/arch/x86/include/asm/mem_encrypt.h b/arch/x86/include/asm/mem_encrypt.h index b31eb9fd5954..f922b682b9b4 100644 --- a/arch/x86/include/asm/mem_encrypt.h +++ b/arch/x86/include/asm/mem_encrypt.h @@ -47,8 +47,8 @@ void __init sme_unmap_bootdata(char *real_mode_data); void __init sme_early_init(void); -void __init sme_encrypt_kernel(struct boot_params *bp); -void __init sme_enable(struct boot_params *bp); +void sme_encrypt_kernel(struct boot_params *bp); +void sme_enable(struct boot_params *bp); int __init early_set_memory_decrypted(unsigned long vaddr, unsigned long size); int __init early_set_memory_encrypted(unsigned long vaddr, unsigned long size); @@ -81,8 +81,8 @@ static inline void __init sme_unmap_bootdata(char *real_mode_data) { } static inline void __init sme_early_init(void) { } -static inline void __init sme_encrypt_kernel(struct boot_params *bp) { } -static inline void __init sme_enable(struct boot_params *bp) { } +static inline void sme_encrypt_kernel(struct boot_params *bp) { } +static inline void sme_enable(struct boot_params *bp) { } static inline void sev_es_init_vc_handling(void) { } diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index d210c7fc8fa2..64b5005d49e5 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -41,6 +41,7 @@ #include #include +#include #include #include #include @@ -94,7 +95,7 @@ struct sme_populate_pgd_data { */ static char sme_workarea[2 * PMD_SIZE] __section(".init.scratch"); -static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) +static void __head sme_clear_pgd(struct sme_populate_pgd_data *ppd) { unsigned long pgd_start, pgd_end, pgd_size; pgd_t *pgd_p; @@ -109,7 +110,7 @@ static void __init sme_clear_pgd(struct sme_populate_pgd_data *ppd) memset(pgd_p, 0, pgd_size); } -static pud_t __init *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) +static pud_t __head *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) { pgd_t *pgd; p4d_t *p4d; @@ -146,7 +147,7 @@ static pud_t __init *sme_prepare_pgd(struct sme_populate_pgd_data *ppd) return pud; } -static void __init sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) +static void __head sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) { pud_t *pud; pmd_t *pmd; @@ -162,7 +163,7 @@ static void __init sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) set_pmd(pmd, __pmd(ppd->paddr | ppd->pmd_flags)); } -static void __init sme_populate_pgd(struct sme_populate_pgd_data *ppd) +static void __head sme_populate_pgd(struct sme_populate_pgd_data *ppd) { pud_t *pud; pmd_t *pmd; @@ -188,7 +189,7 @@ static void __init sme_populate_pgd(struct sme_populate_pgd_data *ppd) set_pte(pte, __pte(ppd->paddr | ppd->pte_flags)); } -static void __init __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) +static void __head __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) { while (ppd->vaddr < ppd->vaddr_end) { sme_populate_pgd_large(ppd); @@ -198,7 +199,7 @@ static void __init __sme_map_range_pmd(struct sme_populate_pgd_data *ppd) } } -static void __init __sme_map_range_pte(struct sme_populate_pgd_data *ppd) +static void __head __sme_map_range_pte(struct sme_populate_pgd_data *ppd) { while (ppd->vaddr < ppd->vaddr_end) { sme_populate_pgd(ppd); @@ -208,7 +209,7 @@ static void __init __sme_map_range_pte(struct sme_populate_pgd_data *ppd) } } -static void __init __sme_map_range(struct sme_populate_pgd_data *ppd, +static void __head __sme_map_range(struct sme_populate_pgd_data *ppd, pmdval_t pmd_flags, pteval_t pte_flags) { unsigned long vaddr_end; @@ -232,22 +233,22 @@ static void __init __sme_map_range(struct sme_populate_pgd_data *ppd, __sme_map_range_pte(ppd); } -static void __init sme_map_range_encrypted(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_encrypted(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_ENC, PTE_FLAGS_ENC); } -static void __init sme_map_range_decrypted(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_decrypted(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_DEC, PTE_FLAGS_DEC); } -static void __init sme_map_range_decrypted_wp(struct sme_populate_pgd_data *ppd) +static void __head sme_map_range_decrypted_wp(struct sme_populate_pgd_data *ppd) { __sme_map_range(ppd, PMD_FLAGS_DEC_WP, PTE_FLAGS_DEC_WP); } -static unsigned long __init sme_pgtable_calc(unsigned long len) +static unsigned long __head sme_pgtable_calc(unsigned long len) { unsigned long entries = 0, tables = 0; @@ -284,7 +285,7 @@ static unsigned long __init sme_pgtable_calc(unsigned long len) return entries + tables; } -void __init sme_encrypt_kernel(struct boot_params *bp) +void __head sme_encrypt_kernel(struct boot_params *bp) { unsigned long workarea_start, workarea_end, workarea_len; unsigned long execute_start, execute_end, execute_len; @@ -319,9 +320,8 @@ void __init sme_encrypt_kernel(struct boot_params *bp) * memory from being cached. */ - /* Physical addresses gives us the identity mapped virtual addresses */ - kernel_start = __pa_symbol(_text); - kernel_end = ALIGN(__pa_symbol(_end), PMD_SIZE); + kernel_start = (unsigned long)RIP_REL_REF(_text); + kernel_end = ALIGN((unsigned long)RIP_REL_REF(_end), PMD_SIZE); kernel_len = kernel_end - kernel_start; initrd_start = 0; @@ -338,14 +338,6 @@ void __init sme_encrypt_kernel(struct boot_params *bp) } #endif - /* - * We're running identity mapped, so we must obtain the address to the - * SME encryption workarea using rip-relative addressing. - */ - asm ("lea sme_workarea(%%rip), %0" - : "=r" (workarea_start) - : "p" (sme_workarea)); - /* * Calculate required number of workarea bytes needed: * executable encryption area size: @@ -355,7 +347,7 @@ void __init sme_encrypt_kernel(struct boot_params *bp) * pagetable structures for the encryption of the kernel * pagetable structures for workarea (in case not currently mapped) */ - execute_start = workarea_start; + execute_start = workarea_start = (unsigned long)RIP_REL_REF(sme_workarea); execute_end = execute_start + (PAGE_SIZE * 2) + PMD_SIZE; execute_len = execute_end - execute_start; @@ -498,7 +490,7 @@ void __init sme_encrypt_kernel(struct boot_params *bp) native_write_cr3(__native_read_cr3()); } -void __init sme_enable(struct boot_params *bp) +void __head sme_enable(struct boot_params *bp) { unsigned int eax, ebx, ecx, edx; unsigned long feature_mask;