From patchwork Tue Mar 7 14:05:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 65572 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2460681wrd; Tue, 7 Mar 2023 06:28:29 -0800 (PST) X-Google-Smtp-Source: AK7set+Exksq/Gmjtq0oh0ObF66eOUqL1m+yDG3fzgDB+qPbLOzdK4WLwAOf+EG5jJDyDiI96nHC X-Received: by 2002:a17:902:e84c:b0:19e:7628:9165 with SMTP id t12-20020a170902e84c00b0019e76289165mr17727158plg.4.1678199308683; Tue, 07 Mar 2023 06:28:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678199308; cv=none; d=google.com; s=arc-20160816; b=uozZweC6NlC0D+OwPITIMHUHYIHQitnh6XJ7raCh8u38bhBs8F+M7No22CAtqEvChO Tkw7WgN4A5WPKbcHyyjonKIoqsFi5AcmaKluRE9qdtRnh2E8et2UWobweQC04a+ZBkV3 vygdvlBbbqAeQ/4KzIe4UnWbjKO/sOCojrt6/Rrkch84TasHk/iAJqlUvcLnpL3HbyfA /YbPA2s5mIt6Guxu8KRBSCSEb9yJ6oi8crm6hlEWjDdrHd5mfe13QWiYX48BiCFPYs+K STBS/jcv6MkJrpsJERVvXriy5+LuMcmX++sZ2x+O+RXVhWzCjVjChwssYJ3w9DFpWqmS ucGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=8kSAVVdedda3pxQwXl89ztdYSSaj7gkwqHX/aKzNYuU=; b=zzYHGGiNcfeJi0fChzJmhz8CbhfxvF5cTO9q0l+1xZYsjdyBdYPqn6p6ROkU1I/kAp znKsYJ8cOS4RfVfGVrLbf9GnbyC9hiICEFFQ527r9AhRRfFfargkCg81KpKPn1AntSEr VI0HGjYdkFViIh9CdKiJV+EgPLZEVP3wAnpI8Cq1qNIRv5lyAjQnuDRb3FlkoEWfIgld I5sa0edA+GHNgCRIwDis46/wVbo+P9vwJ0D2wtZfxeVcQzkmF/mJYsCe4LqYd47lUjKA 689atG+pnxkXZjF9tFq0+rb/gX2OBTmxMVT9oH14DNO5Hqaiyy3PiHthVHzSALJBFgnd h72Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="P/g1FxQv"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f17-20020a170902ce9100b0019266fcbd20si13610993plg.623.2023.03.07.06.28.16; Tue, 07 Mar 2023 06:28:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="P/g1FxQv"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231410AbjCGOMF (ORCPT + 99 others); Tue, 7 Mar 2023 09:12:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49126 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231446AbjCGOKX (ORCPT ); Tue, 7 Mar 2023 09:10:23 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2B84888A1 for ; Tue, 7 Mar 2023 06:08:46 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4BC896135E for ; Tue, 7 Mar 2023 14:08:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BA955C4339E; Tue, 7 Mar 2023 14:08:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1678198118; bh=jq0ZfsvwfvGD7dc7+4ei8T6xxeCdjcLuVSzc5ozEYYU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=P/g1FxQvtQ6Djb0efLquHx4sC5zwNZTzw6ha7e7dDSnGZtiJkcdGLhAJOIbO7Udrr c/pOatQglTTG4auyFwSaRThUdUosTsqq8d0NXzEt3QV9CzO9jtP37D1/KPzds7wH+X iP0lQkhdXjUMc+gIAoZpf0kTBUVYuSr0BYnrH6c2QUDuAl/7jsVa/FpkBUsHMTLxM1 VuXoIvbr6ldY+JSpVneCzpiOuUWgYh6w2DJ8eLlEmMyTGEe6LsmKBzDyoOSkp9EjIV 7kIVdgnn0PwRwNNqe9Jz7RuZdjmXLDXFUrz41yln2eVd24ywrvCeY5wWzD1qdtalDJ /RcDxpv06whmg== From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook Subject: [PATCH v3 50/60] arm64: mm: Add 5 level paging support to fixmap and swapper handling Date: Tue, 7 Mar 2023 15:05:12 +0100 Message-Id: <20230307140522.2311461-51-ardb@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230307140522.2311461-1-ardb@kernel.org> References: <20230307140522.2311461-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6571; i=ardb@kernel.org; h=from:subject; bh=jq0ZfsvwfvGD7dc7+4ei8T6xxeCdjcLuVSzc5ozEYYU=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIYXdpUX48+cq3RMuB7/WPMm/4j+p8uWBmeqSBgce7tNIk Um+O+FrRykLgxgHg6yYIovA7L/vdp6eKFXrPEsWZg4rE8gQBi5OAZjIG3aG/4UHXR/4de2Ysdhj Btuf1Jipx7veRyn9luW25rl0RXo1635Ghmtid2bd9fz36cClqTu/2B3bu2/eKtOSN4zfXx36c39 ShgMHAA== X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759719517905293971?= X-GMAIL-MSGID: =?utf-8?q?1759719517905293971?= Add support for using 5 levels of paging in the fixmap, as well as in the kernel page table handling code which uses fixmaps internally. This also handles the case where a 5 level build runs on hardware that only supports 4 levels of paging. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/fixmap.h | 1 + arch/arm64/include/asm/pgtable.h | 45 ++++++++++++-- arch/arm64/mm/mmu.c | 64 +++++++++++++++++--- 3 files changed, 96 insertions(+), 14 deletions(-) diff --git a/arch/arm64/include/asm/fixmap.h b/arch/arm64/include/asm/fixmap.h index 71ed5fdf718bd0fd..8d4ec7bf74afb7a6 100644 --- a/arch/arm64/include/asm/fixmap.h +++ b/arch/arm64/include/asm/fixmap.h @@ -90,6 +90,7 @@ enum fixed_addresses { FIX_PTE, FIX_PMD, FIX_PUD, + FIX_P4D, FIX_PGD, __end_of_fixed_addresses diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index c667073e3f56755d..a286ecc447d33b24 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -613,12 +613,12 @@ static inline bool pud_table(pud_t pud) { return true; } PUD_TYPE_TABLE) #endif -extern pgd_t init_pg_dir[PTRS_PER_PGD]; +extern pgd_t init_pg_dir[]; extern pgd_t init_pg_end[]; -extern pgd_t swapper_pg_dir[PTRS_PER_PGD]; -extern pgd_t idmap_pg_dir[PTRS_PER_PGD]; -extern pgd_t tramp_pg_dir[PTRS_PER_PGD]; -extern pgd_t reserved_pg_dir[PTRS_PER_PGD]; +extern pgd_t swapper_pg_dir[]; +extern pgd_t idmap_pg_dir[]; +extern pgd_t tramp_pg_dir[]; +extern pgd_t reserved_pg_dir[]; extern void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd); @@ -883,12 +883,47 @@ static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long addr) return p4d_offset_lockless(pgdp, READ_ONCE(*pgdp), addr); } +static inline p4d_t *p4d_set_fixmap(unsigned long addr) +{ + if (!pgtable_l5_enabled()) + return NULL; + return (p4d_t *)set_fixmap_offset(FIX_P4D, addr); +} + +static inline p4d_t *p4d_set_fixmap_offset(pgd_t *pgdp, unsigned long addr) +{ + if (!pgtable_l5_enabled()) + return pgd_to_folded_p4d(pgdp, addr); + return p4d_set_fixmap(p4d_offset_phys(pgdp, addr)); +} + +static inline void p4d_clear_fixmap(void) +{ + if (pgtable_l5_enabled()) + clear_fixmap(FIX_P4D); +} + +/* use ONLY for statically allocated translation tables */ +static inline p4d_t *p4d_offset_kimg(pgd_t *pgdp, u64 addr) +{ + if (!pgtable_l5_enabled()) + return pgd_to_folded_p4d(pgdp, addr); + return (p4d_t *)__phys_to_kimg(p4d_offset_phys(pgdp, addr)); +} + #define pgd_page(pgd) pfn_to_page(__phys_to_pfn(__pgd_to_phys(pgd))) #else static inline bool pgtable_l5_enabled(void) { return false; } +/* Match p4d_offset folding in */ +#define p4d_set_fixmap(addr) NULL +#define p4d_set_fixmap_offset(p4dp, addr) ((p4d_t *)p4dp) +#define p4d_clear_fixmap() + +#define p4d_offset_kimg(dir,addr) ((p4d_t *)dir) + #endif /* CONFIG_PGTABLE_LEVELS > 4 */ #define pgd_ERROR(e) \ diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 6426df5211456cfe..77cd163375124c6a 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -315,15 +315,14 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, } while (addr = next, addr != end); } -static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, +static void alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; - pud_t *pudp; - p4d_t *p4dp = p4d_offset(pgdp, addr); p4d_t p4d = READ_ONCE(*p4dp); + pud_t *pudp; if (p4d_none(p4d)) { p4dval_t p4dval = P4D_TYPE_TABLE | P4D_TABLE_UXN; @@ -371,6 +370,46 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, pud_clear_fixmap(); } +static void alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + unsigned long next; + pgd_t pgd = READ_ONCE(*pgdp); + p4d_t *p4dp; + + if (pgd_none(pgd)) { + pgdval_t pgdval = PGD_TYPE_TABLE | PGD_TABLE_UXN; + phys_addr_t p4d_phys; + + if (flags & NO_EXEC_MAPPINGS) + pgdval |= PGD_TABLE_PXN; + BUG_ON(!pgtable_alloc); + p4d_phys = pgtable_alloc(P4D_SHIFT); + __pgd_populate(pgdp, p4d_phys, pgdval); + pgd = READ_ONCE(*pgdp); + } + BUG_ON(pgd_bad(pgd)); + + p4dp = p4d_set_fixmap_offset(pgdp, addr); + do { + p4d_t old_p4d = READ_ONCE(*p4dp); + + next = p4d_addr_end(addr, end); + + alloc_init_pud(p4dp, addr, next, phys, prot, + pgtable_alloc, flags); + + BUG_ON(p4d_val(old_p4d) != 0 && + p4d_val(old_p4d) != READ_ONCE(p4d_val(*p4dp))); + + phys += next - addr; + } while (p4dp++, addr = next, addr != end); + + p4d_clear_fixmap(); +} + static void __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, @@ -393,7 +432,7 @@ static void __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, do { next = pgd_addr_end(addr, end); - alloc_init_pud(pgdp, addr, next, phys, prot, pgtable_alloc, + alloc_init_p4d(pgdp, addr, next, phys, prot, pgtable_alloc, flags); phys += next - addr; } while (pgdp++, addr = next, addr != end); @@ -1120,10 +1159,19 @@ void vmemmap_free(unsigned long start, unsigned long end, } #endif /* CONFIG_MEMORY_HOTPLUG */ -static inline pud_t *fixmap_pud(unsigned long addr) +static inline p4d_t *fixmap_p4d(unsigned long addr) { pgd_t *pgdp = pgd_offset_k(addr); - p4d_t *p4dp = p4d_offset(pgdp, addr); + pgd_t pgd = READ_ONCE(*pgdp); + + BUG_ON(pgd_none(pgd) || pgd_bad(pgd)); + + return p4d_offset_kimg(pgdp, addr); +} + +static inline pud_t *fixmap_pud(unsigned long addr) +{ + p4d_t *p4dp = fixmap_p4d(addr); p4d_t p4d = READ_ONCE(*p4dp); BUG_ON(p4d_none(p4d) || p4d_bad(p4d)); @@ -1154,14 +1202,12 @@ static inline pte_t *fixmap_pte(unsigned long addr) */ void __init early_fixmap_init(void) { - pgd_t *pgdp; p4d_t *p4dp, p4d; pud_t *pudp; pmd_t *pmdp; unsigned long addr = FIXADDR_START; - pgdp = pgd_offset_k(addr); - p4dp = p4d_offset(pgdp, addr); + p4dp = fixmap_p4d(addr); p4d = READ_ONCE(*p4dp); if (CONFIG_PGTABLE_LEVELS > 3 && !(p4d_none(p4d) || p4d_page_paddr(p4d) == __pa_symbol(bm_pud))) {