From patchwork Tue Mar 14 10:13:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evgeniy Baskov X-Patchwork-Id: 69476 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp1678229wrd; Tue, 14 Mar 2023 03:37:55 -0700 (PDT) X-Google-Smtp-Source: AK7set/PkS4C/NjMLG++SjzNJPZhHc7PsQ2786m+IR0q+C3WxzPxjnnN4E5hYMDfHIpohihB6xYe X-Received: by 2002:a17:90b:4a85:b0:237:b5d4:c0cc with SMTP id lp5-20020a17090b4a8500b00237b5d4c0ccmr37390825pjb.39.1678790274884; Tue, 14 Mar 2023 03:37:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1678790274; cv=none; d=google.com; s=arc-20160816; b=fRQMZYyPkBV9iBPvL33q+bJTy3uOldpVnRw5Muxj29YkMWQGZXc88c+4v0UZVwI0cN pMfb/5K5p4dOCr1++ROePX1U1IBjSLdjeweZOgCDp00DLrCXr762A0iGDw/iwBmMEcVb SJNaikEYURPV6otjWqgIRUXmEb8pakH8dS1AFFKJlENzBoIT8iISI2HPlZlT9FNiHUpx x31HTgogotQG/ZwtS2YPqIAwcNQPz96Hzoit+D58Jqtx0o3J2ekXm5K6K2g37jRjECsH FdeHXIsfOHF+B08bZreRKE93XZtp/y4Z01Tz+OH1MwuKN0N53KP0jtyYrg32tjZu0UJ0 wcuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-filter; bh=cYTZ/gmVk3OpdumhLwrj4VTAssL6GMlQFuk7LngrsRk=; b=tnlLxRnUIcgY/5elpnLTkwS2yfd9QyFWTL7KGlPfjTyng1d8xYTcMrJSfb6ybGXzI2 E98R5+XZWqHTP8CMZI+KAGKU86pArmGnuEbWGeekvpommu8Y73IllHUbXI/1A6hW+98m 7zwyGvzFbfRbP2lerMa0XIh4z4KlhA5U/UjnUpNsXODz5tS4mZcFHrLPx7XuOF+5DkSu BCyXbGlOcu3ZduvKBXqQo2E7r5/eLf2ajdzUncD+a5p9bdqnOaybw9Ce0t5cT1rcxFU3 +0rbGpfBcHXg2JseOrshG2sm+0WFgBvJaHaJO8qDNp5UdTunefUC2q94N/xvFUmexkHS g72A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ispras.ru header.s=default header.b=NELyDBpv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ispras.ru Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id jg21-20020a17090326d500b0019ca3bea310si2100264plb.303.2023.03.14.03.37.42; Tue, 14 Mar 2023 03:37:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@ispras.ru header.s=default header.b=NELyDBpv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=ispras.ru Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230313AbjCNKXB (ORCPT + 99 others); Tue, 14 Mar 2023 06:23:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48468 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229534AbjCNKWa (ORCPT ); Tue, 14 Mar 2023 06:22:30 -0400 Received: from mail.ispras.ru (mail.ispras.ru [83.149.199.84]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 548089BA73; Tue, 14 Mar 2023 03:22:02 -0700 (PDT) Received: from localhost.localdomain (unknown [83.149.199.65]) by mail.ispras.ru (Postfix) with ESMTPSA id 794554076B3D; Tue, 14 Mar 2023 10:14:01 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 mail.ispras.ru 794554076B3D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ispras.ru; s=default; t=1678788841; bh=cYTZ/gmVk3OpdumhLwrj4VTAssL6GMlQFuk7LngrsRk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NELyDBpvQwFHy68bV8QD1zrh4bGl7AhMizmoQ4t78vurCJjaVdPKe69IQdGCaR+3k UnR6hH0HLWvrbM01dGZ7RZDk73b0ygR+VzqzdNOpnBldldEVnD2hMO2ZwW6/GW+3LQ +hrXCP8D/8tyKVF5+nfuTGAKP2MBmUNNL24N9XjE= From: Evgeniy Baskov To: Ard Biesheuvel Cc: Evgeniy Baskov , Borislav Petkov , Andy Lutomirski , Dave Hansen , Ingo Molnar , Peter Zijlstra , Thomas Gleixner , Alexey Khoroshilov , Peter Jones , Gerd Hoffmann , "Limonciello, Mario" , joeyli , lvc-project@linuxtesting.org, x86@kernel.org, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org, kernel test robot Subject: [PATCH v5 05/27] x86/boot: Support 4KB pages for identity mapping Date: Tue, 14 Mar 2023 13:13:32 +0300 Message-Id: <7ab57fb851bb0afa3724edf556a1cf24037babbc.1678785672.git.baskov@ispras.ru> X-Mailer: git-send-email 2.39.2 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1760339191204023068?= X-GMAIL-MSGID: =?utf-8?q?1760339191204023068?= Current identity mapping code only supports 2M and 1G pages. 4KB pages are desirable for better memory protection granularity in compressed kernel code. Change identity mapping code to support 4KB pages and memory remapping with different attributes. Tested-by: Mario Limonciello Signed-off-by: Evgeniy Baskov Warnings caused by the previous version were Reported-by: kernel test robot Link: https://lore.kernel.org/oe-kbuild-all/202303081758.wjUcKEAW-lkp@intel.com/ --- arch/x86/include/asm/init.h | 8 +- arch/x86/mm/ident_map.c | 185 +++++++++++++++++++++++++++++------- 2 files changed, 160 insertions(+), 33 deletions(-) diff --git a/arch/x86/include/asm/init.h b/arch/x86/include/asm/init.h index 5f1d3c421f68..d4e790435bac 100644 --- a/arch/x86/include/asm/init.h +++ b/arch/x86/include/asm/init.h @@ -8,10 +8,16 @@ struct x86_mapping_info { unsigned long page_flag; /* page flag for PMD or PUD entry */ unsigned long offset; /* ident mapping offset */ bool direct_gbpages; /* PUD level 1GB page support */ + bool allow_4kpages; /* Allow more granular mappings with 4K pages */ unsigned long kernpg_flag; /* kernel pagetable flag override */ }; int kernel_ident_mapping_init(struct x86_mapping_info *info, pgd_t *pgd_page, - unsigned long pstart, unsigned long pend); + unsigned long pstart, unsigned long pend); +pte_t *ident_split_large_pmd(struct x86_mapping_info *info, + pmd_t *pmdp, unsigned long page_addr); +pmd_t *ident_split_large_pud(struct x86_mapping_info *info, + pud_t *pudp, unsigned long page_addr); + #endif /* _ASM_X86_INIT_H */ diff --git a/arch/x86/mm/ident_map.c b/arch/x86/mm/ident_map.c index 968d7005f4a7..662e794a325d 100644 --- a/arch/x86/mm/ident_map.c +++ b/arch/x86/mm/ident_map.c @@ -4,24 +4,127 @@ * included by both the compressed kernel and the regular kernel. */ -static void ident_pmd_init(struct x86_mapping_info *info, pmd_t *pmd_page, - unsigned long addr, unsigned long end) +static void ident_pte_init(struct x86_mapping_info *info, pte_t *pte_page, + unsigned long addr, unsigned long end, + unsigned long flags) { - addr &= PMD_MASK; - for (; addr < end; addr += PMD_SIZE) { + addr &= PAGE_MASK; + for (; addr < end; addr += PAGE_SIZE) { + pte_t *pte = pte_page + pte_index(addr); + + set_pte(pte, __pte((addr - info->offset) | flags)); + } +} + +pte_t *ident_split_large_pmd(struct x86_mapping_info *info, + pmd_t *pmdp, unsigned long page_addr) +{ + unsigned long pmd_addr, page_flags; + pte_t *pte; + + pte = (pte_t *)info->alloc_pgt_page(info->context); + if (!pte) + return NULL; + + pmd_addr = page_addr & PMD_MASK; + + /* Not a large page - clear PSE flag */ + page_flags = pmd_flags(*pmdp) & ~_PSE; + ident_pte_init(info, pte, pmd_addr, pmd_addr + PMD_SIZE, page_flags); + + return pte; +} + +static int ident_pmd_init(struct x86_mapping_info *info, pmd_t *pmd_page, + unsigned long addr, unsigned long end, + unsigned long flags) +{ + unsigned long next; + bool new_table = 0; + + for (; addr < end; addr = next) { pmd_t *pmd = pmd_page + pmd_index(addr); + pte_t *pte; - if (pmd_present(*pmd)) + next = (addr & PMD_MASK) + PMD_SIZE; + if (next > end) + next = end; + + /* + * Use 2M pages if 4k pages are not allowed or + * we are not mapping extra, i.e. address and size are aligned. + */ + + if (!info->allow_4kpages || + (!(addr & ~PMD_MASK) && next == addr + PMD_SIZE)) { + + pmd_t pmdval; + + addr &= PMD_MASK; + pmdval = __pmd((addr - info->offset) | flags | _PSE); + set_pmd(pmd, pmdval); continue; + } + + /* + * If currently mapped page is large, we need to split it. + * The case when we don't can remap 2M page to 2M page + * with different flags is already covered above. + * + * If there's nothing mapped to desired address, + * we need to allocate new page table. + */ - set_pmd(pmd, __pmd((addr - info->offset) | info->page_flag)); + if (pmd_large(*pmd)) { + pte = ident_split_large_pmd(info, pmd, addr); + new_table = 1; + } else if (!pmd_present(*pmd)) { + pte = (pte_t *)info->alloc_pgt_page(info->context); + new_table = 1; + } else { + pte = pte_offset_kernel(pmd, 0); + new_table = 0; + } + + if (!pte) + return -ENOMEM; + + ident_pte_init(info, pte, addr, next, flags); + + if (new_table) + set_pmd(pmd, __pmd(__pa(pte) | info->kernpg_flag)); } + + return 0; } + +pmd_t *ident_split_large_pud(struct x86_mapping_info *info, + pud_t *pudp, unsigned long page_addr) +{ + unsigned long pud_addr, page_flags; + pmd_t *pmd; + + pmd = (pmd_t *)info->alloc_pgt_page(info->context); + if (!pmd) + return NULL; + + pud_addr = page_addr & PUD_MASK; + + /* Not a large page - clear PSE flag */ + page_flags = pud_flags(*pudp) & ~_PSE; + ident_pmd_init(info, pmd, pud_addr, pud_addr + PUD_SIZE, page_flags); + + return pmd; +} + + static int ident_pud_init(struct x86_mapping_info *info, pud_t *pud_page, unsigned long addr, unsigned long end) { unsigned long next; + bool new_table = 0; + int result; for (; addr < end; addr = next) { pud_t *pud = pud_page + pud_index(addr); @@ -31,28 +134,39 @@ static int ident_pud_init(struct x86_mapping_info *info, pud_t *pud_page, if (next > end) next = end; + /* Use 1G pages only if forced, even if they are supported. */ if (info->direct_gbpages) { pud_t pudval; - - if (pud_present(*pud)) - continue; + unsigned long flags; addr &= PUD_MASK; - pudval = __pud((addr - info->offset) | info->page_flag); + flags = info->page_flag | _PSE; + pudval = __pud((addr - info->offset) | flags); + set_pud(pud, pudval); continue; } - if (pud_present(*pud)) { + if (pud_large(*pud)) { + pmd = ident_split_large_pud(info, pud, addr); + new_table = 1; + } else if (!pud_present(*pud)) { + pmd = (pmd_t *)info->alloc_pgt_page(info->context); + new_table = 1; + } else { pmd = pmd_offset(pud, 0); - ident_pmd_init(info, pmd, addr, next); - continue; + new_table = 0; } - pmd = (pmd_t *)info->alloc_pgt_page(info->context); + if (!pmd) return -ENOMEM; - ident_pmd_init(info, pmd, addr, next); - set_pud(pud, __pud(__pa(pmd) | info->kernpg_flag)); + + result = ident_pmd_init(info, pmd, addr, next, info->page_flag); + if (result) + return result; + + if (new_table) + set_pud(pud, __pud(__pa(pmd) | info->kernpg_flag)); } return 0; @@ -63,6 +177,7 @@ static int ident_p4d_init(struct x86_mapping_info *info, p4d_t *p4d_page, { unsigned long next; int result; + bool new_table = 0; for (; addr < end; addr = next) { p4d_t *p4d = p4d_page + p4d_index(addr); @@ -72,15 +187,14 @@ static int ident_p4d_init(struct x86_mapping_info *info, p4d_t *p4d_page, if (next > end) next = end; - if (p4d_present(*p4d)) { + if (!p4d_present(*p4d)) { + pud = (pud_t *)info->alloc_pgt_page(info->context); + new_table = 1; + } else { pud = pud_offset(p4d, 0); - result = ident_pud_init(info, pud, addr, next); - if (result) - return result; - - continue; + new_table = 0; } - pud = (pud_t *)info->alloc_pgt_page(info->context); + if (!pud) return -ENOMEM; @@ -88,19 +202,22 @@ static int ident_p4d_init(struct x86_mapping_info *info, p4d_t *p4d_page, if (result) return result; - set_p4d(p4d, __p4d(__pa(pud) | info->kernpg_flag)); + if (new_table) + set_p4d(p4d, __p4d(__pa(pud) | info->kernpg_flag)); } return 0; } -int kernel_ident_mapping_init(struct x86_mapping_info *info, pgd_t *pgd_page, - unsigned long pstart, unsigned long pend) +int kernel_ident_mapping_init(struct x86_mapping_info *info, + pgd_t *pgd_page, unsigned long pstart, + unsigned long pend) { unsigned long addr = pstart + info->offset; unsigned long end = pend + info->offset; unsigned long next; int result; + bool new_table; /* Set the default pagetable flags if not supplied */ if (!info->kernpg_flag) @@ -117,20 +234,24 @@ int kernel_ident_mapping_init(struct x86_mapping_info *info, pgd_t *pgd_page, if (next > end) next = end; - if (pgd_present(*pgd)) { + if (!pgd_present(*pgd)) { + p4d = (p4d_t *)info->alloc_pgt_page(info->context); + new_table = 1; + } else { p4d = p4d_offset(pgd, 0); - result = ident_p4d_init(info, p4d, addr, next); - if (result) - return result; - continue; + new_table = 0; } - p4d = (p4d_t *)info->alloc_pgt_page(info->context); if (!p4d) return -ENOMEM; + result = ident_p4d_init(info, p4d, addr, next); if (result) return result; + + if (!new_table) + continue; + if (pgtable_l5_enabled()) { set_pgd(pgd, __pgd(__pa(p4d) | info->kernpg_flag)); } else {