From patchwork Mon Jan 23 10:09:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 47096 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1525479wrn; Mon, 23 Jan 2023 02:13:21 -0800 (PST) X-Google-Smtp-Source: AMrXdXubo9iEtz87trWEB87le6DbC0tWWOC/CXxgZuqP0P4i7qKMjhYwQwQBpZcYbKT2VVlsMUFN X-Received: by 2002:a05:6a21:151b:b0:b8:4a1a:641a with SMTP id nq27-20020a056a21151b00b000b84a1a641amr27852454pzb.38.1674468801661; Mon, 23 Jan 2023 02:13:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674468801; cv=none; d=google.com; s=arc-20160816; b=mOkhD5CdhRRdSG8Gklfxcwil1GLO9VMCNjTgoRJoAuwEHBRPLuaSPKCVcOexnHsWyR FuXst7Iyo75VsAQ7Fx1cSwPQFgcGfJ+sy+8mAf+WNhfP//iY3ulgV3SdqcypTfrKqcmx 0VDkHlPGO1NYwxDAXrmeBjAywnr0cDi8vvZQcsrcHG2WidDTDP6OZss2XxFMx/zBpXco WpXT+OsENfFDpoQFth1281JxStZLzFDMeYJMvu3yx4uEvDhOOvNcq7DAWSuM3ocijg/b QOi4KMnoXEu6+1FeKKJuCJDLbTJk0LTuXFLUlmM6areishAXKLWoYTfVNy+mJMzJkKhh qmCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=J+Fj80rLiudnEryzZYSTURSW9/j8riSTbxetIYC+pOE=; b=u+yCaN6YOJXle/Ib15cM+biVfsPiHmNtIX37PpHx0m1llm/SJwHxQLYt8tzMHOpoZt cHyUyF2laYUyVxQ1lEk/DKPPd16x9ItBL1EZnIbQnPZQ8Dw1LvrU77AlM2N2eH39LA4v MFoQqlDuzf5QgiVM3w7DXvZlWLsOL+k/CgqfLeotPMuO4u0t/vOF0Z/tCbIB1qD0v6Bi wHys/x+2g7E+5bpu04Y+Y3AL2ILyXzQd8q1QcXcz6Xd40ekAN640nkVqxPeCzGswjybc Vf0crK7ckzuhGvxxOwFCSw4b5QfV7bltwBdnzhN6oEBbxwRy81y52PMElh5uVporIt1r zK7A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@rivosinc-com.20210112.gappssmtp.com header.s=20210112 header.b=oQYgkSxo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w14-20020a63a74e000000b004910ae954casi51652229pgo.421.2023.01.23.02.13.09; Mon, 23 Jan 2023 02:13:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@rivosinc-com.20210112.gappssmtp.com header.s=20210112 header.b=oQYgkSxo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231788AbjAWKLo (ORCPT + 99 others); Mon, 23 Jan 2023 05:11:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231557AbjAWKLn (ORCPT ); Mon, 23 Jan 2023 05:11:43 -0500 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6987EE38B for ; Mon, 23 Jan 2023 02:11:05 -0800 (PST) Received: by mail-wm1-x32e.google.com with SMTP id fl11-20020a05600c0b8b00b003daf72fc844so10220971wmb.0 for ; Mon, 23 Jan 2023 02:11:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J+Fj80rLiudnEryzZYSTURSW9/j8riSTbxetIYC+pOE=; b=oQYgkSxo8lUwnug5gjITUFTSqMP0Vb7qbI281YTS/l2HZPgykNRyelArwjj6v0yVrM L+bMZTUdy308VydriDgyUglGjJpX8WUnZaP/Kgv8VAroJXAvYOVh3UgpBki4lFGghWbK rPG7lxI/AAlwSNZNRQ5UpJ7k3nifvaY8P0rrluNphEf0JcAWtSV7U79oaE3rHs5xjKBj 6+eBD3qRdOQXD3MOtMHk8ukonWqr2SV8WPi5qLBdsyYs3r92epPt4DTj2qMmPRgPJKcs x2B9ihuhtvj8CeeUMKLUi/WiMpp9d0YAr6p9TZqyjccuo/GC/TEXATuzir95V8OlpQlE Jmhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J+Fj80rLiudnEryzZYSTURSW9/j8riSTbxetIYC+pOE=; b=Fe1Rk48eLCS+Ak7qTwU8pNBLTKXCg+4eRjMgzx2J3J7nn4kP+BvpiWLbaUJE0aBIGe 8cTVU+hdugv8kAT15AcT+vBMuUi1ll7G7DOGf/3La7OZ2W5L2z3ls6hD0XTxqfYPo+n9 zNQWpU1+GUcIowHCnpucEkZpKHaCxuYmo466p/T66RbMQC00Wa3uxnbGBP4JR2rvpYun i4/VKslacWAdGKULrRZKlPcToLKhVSOwM+1QVdvvkFNCL/r90GwVAvWAuXFtoK4f96cz 7UqoH2V+D/8QNXRVqbeGcfXPKKu+97OB80IEiJJlvrJQ2E+igzoXOOsEcVDwHImTi9ie x79A== X-Gm-Message-State: AFqh2kqyJmADHkyRlTKL1Dth3EUdnm5y+OMQsw3dEu/B3rvnMMF36xHK YBXzUSFN+SeZ1JgyP7+Q6E6Dow== X-Received: by 2002:a05:600c:4f08:b0:3db:9e3:3bf1 with SMTP id l8-20020a05600c4f0800b003db09e33bf1mr22181752wmq.31.1674468654715; Mon, 23 Jan 2023 02:10:54 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-450-160.w2-7.abo.wanadoo.fr. [2.7.42.160]) by smtp.gmail.com with ESMTPSA id p7-20020a05600c468700b003db0bb81b6asm10803053wmo.1.2023.01.23.02.10.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Jan 2023 02:10:54 -0800 (PST) From: Alexandre Ghiti To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Ard Biesheuvel , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-efi@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v2 1/6] riscv: Split early and final KASAN population functions Date: Mon, 23 Jan 2023 11:09:46 +0100 Message-Id: <20230123100951.810807-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230123100951.810807-1-alexghiti@rivosinc.com> References: <20230123100951.810807-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755807798508615115?= X-GMAIL-MSGID: =?utf-8?q?1755807798508615115?= This is a preliminary work that allows to make the code more understandable. Signed-off-by: Alexandre Ghiti --- arch/riscv/mm/kasan_init.c | 187 +++++++++++++++++++++++-------------- 1 file changed, 117 insertions(+), 70 deletions(-) diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c index e1226709490f..9a5211ca8368 100644 --- a/arch/riscv/mm/kasan_init.c +++ b/arch/riscv/mm/kasan_init.c @@ -95,23 +95,13 @@ static void __init kasan_populate_pmd(pud_t *pud, unsigned long vaddr, unsigned } static void __init kasan_populate_pud(pgd_t *pgd, - unsigned long vaddr, unsigned long end, - bool early) + unsigned long vaddr, unsigned long end) { phys_addr_t phys_addr; pud_t *pudp, *base_pud; unsigned long next; - if (early) { - /* - * We can't use pgd_page_vaddr here as it would return a linear - * mapping address but it is not mapped yet, but when populating - * early_pg_dir, we need the physical address and when populating - * swapper_pg_dir, we need the kernel virtual address so use - * pt_ops facility. - */ - base_pud = pt_ops.get_pud_virt(pfn_to_phys(_pgd_pfn(*pgd))); - } else if (pgd_none(*pgd)) { + if (pgd_none(*pgd)) { base_pud = memblock_alloc(PTRS_PER_PUD * sizeof(pud_t), PAGE_SIZE); memcpy(base_pud, (void *)kasan_early_shadow_pud, sizeof(pud_t) * PTRS_PER_PUD); @@ -130,16 +120,10 @@ static void __init kasan_populate_pud(pgd_t *pgd, next = pud_addr_end(vaddr, end); if (pud_none(*pudp) && IS_ALIGNED(vaddr, PUD_SIZE) && (next - vaddr) >= PUD_SIZE) { - if (early) { - phys_addr = __pa(((uintptr_t)kasan_early_shadow_pmd)); - set_pud(pudp, pfn_pud(PFN_DOWN(phys_addr), PAGE_TABLE)); + phys_addr = memblock_phys_alloc(PUD_SIZE, PUD_SIZE); + if (phys_addr) { + set_pud(pudp, pfn_pud(PFN_DOWN(phys_addr), PAGE_KERNEL)); continue; - } else { - phys_addr = memblock_phys_alloc(PUD_SIZE, PUD_SIZE); - if (phys_addr) { - set_pud(pudp, pfn_pud(PFN_DOWN(phys_addr), PAGE_KERNEL)); - continue; - } } } @@ -152,35 +136,22 @@ static void __init kasan_populate_pud(pgd_t *pgd, * it entirely, memblock could allocate a page at a physical address * where KASAN is not populated yet and then we'd get a page fault. */ - if (!early) - set_pgd(pgd, pfn_pgd(PFN_DOWN(__pa(base_pud)), PAGE_TABLE)); + set_pgd(pgd, pfn_pgd(PFN_DOWN(__pa(base_pud)), PAGE_TABLE)); } static void __init kasan_populate_p4d(pgd_t *pgd, - unsigned long vaddr, unsigned long end, - bool early) + unsigned long vaddr, unsigned long end) { phys_addr_t phys_addr; p4d_t *p4dp, *base_p4d; unsigned long next; - if (early) { - /* - * We can't use pgd_page_vaddr here as it would return a linear - * mapping address but it is not mapped yet, but when populating - * early_pg_dir, we need the physical address and when populating - * swapper_pg_dir, we need the kernel virtual address so use - * pt_ops facility. - */ - base_p4d = pt_ops.get_p4d_virt(pfn_to_phys(_pgd_pfn(*pgd))); - } else { - base_p4d = (p4d_t *)pgd_page_vaddr(*pgd); - if (base_p4d == lm_alias(kasan_early_shadow_p4d)) { - base_p4d = memblock_alloc(PTRS_PER_PUD * sizeof(p4d_t), PAGE_SIZE); - memcpy(base_p4d, (void *)kasan_early_shadow_p4d, - sizeof(p4d_t) * PTRS_PER_P4D); - } - } + base_p4d = (p4d_t *)pgd_page_vaddr(*pgd); + if (base_p4d == lm_alias(kasan_early_shadow_p4d)) { + base_p4d = memblock_alloc(PTRS_PER_PUD * sizeof(p4d_t), PAGE_SIZE); + memcpy(base_p4d, (void *)kasan_early_shadow_p4d, + sizeof(p4d_t) * PTRS_PER_P4D); + } p4dp = base_p4d + p4d_index(vaddr); @@ -188,20 +159,14 @@ static void __init kasan_populate_p4d(pgd_t *pgd, next = p4d_addr_end(vaddr, end); if (p4d_none(*p4dp) && IS_ALIGNED(vaddr, P4D_SIZE) && (next - vaddr) >= P4D_SIZE) { - if (early) { - phys_addr = __pa(((uintptr_t)kasan_early_shadow_pud)); - set_p4d(p4dp, pfn_p4d(PFN_DOWN(phys_addr), PAGE_TABLE)); + phys_addr = memblock_phys_alloc(P4D_SIZE, P4D_SIZE); + if (phys_addr) { + set_p4d(p4dp, pfn_p4d(PFN_DOWN(phys_addr), PAGE_KERNEL)); continue; - } else { - phys_addr = memblock_phys_alloc(P4D_SIZE, P4D_SIZE); - if (phys_addr) { - set_p4d(p4dp, pfn_p4d(PFN_DOWN(phys_addr), PAGE_KERNEL)); - continue; - } } } - kasan_populate_pud((pgd_t *)p4dp, vaddr, next, early); + kasan_populate_pud((pgd_t *)p4dp, vaddr, next); } while (p4dp++, vaddr = next, vaddr != end); /* @@ -210,8 +175,7 @@ static void __init kasan_populate_p4d(pgd_t *pgd, * it entirely, memblock could allocate a page at a physical address * where KASAN is not populated yet and then we'd get a page fault. */ - if (!early) - set_pgd(pgd, pfn_pgd(PFN_DOWN(__pa(base_p4d)), PAGE_TABLE)); + set_pgd(pgd, pfn_pgd(PFN_DOWN(__pa(base_p4d)), PAGE_TABLE)); } #define kasan_early_shadow_pgd_next (pgtable_l5_enabled ? \ @@ -219,16 +183,15 @@ static void __init kasan_populate_p4d(pgd_t *pgd, (pgtable_l4_enabled ? \ (uintptr_t)kasan_early_shadow_pud : \ (uintptr_t)kasan_early_shadow_pmd)) -#define kasan_populate_pgd_next(pgdp, vaddr, next, early) \ +#define kasan_populate_pgd_next(pgdp, vaddr, next) \ (pgtable_l5_enabled ? \ - kasan_populate_p4d(pgdp, vaddr, next, early) : \ + kasan_populate_p4d(pgdp, vaddr, next) : \ (pgtable_l4_enabled ? \ - kasan_populate_pud(pgdp, vaddr, next, early) : \ + kasan_populate_pud(pgdp, vaddr, next) : \ kasan_populate_pmd((pud_t *)pgdp, vaddr, next))) static void __init kasan_populate_pgd(pgd_t *pgdp, - unsigned long vaddr, unsigned long end, - bool early) + unsigned long vaddr, unsigned long end) { phys_addr_t phys_addr; unsigned long next; @@ -237,11 +200,7 @@ static void __init kasan_populate_pgd(pgd_t *pgdp, next = pgd_addr_end(vaddr, end); if (IS_ALIGNED(vaddr, PGDIR_SIZE) && (next - vaddr) >= PGDIR_SIZE) { - if (early) { - phys_addr = __pa((uintptr_t)kasan_early_shadow_pgd_next); - set_pgd(pgdp, pfn_pgd(PFN_DOWN(phys_addr), PAGE_TABLE)); - continue; - } else if (pgd_page_vaddr(*pgdp) == + if (pgd_page_vaddr(*pgdp) == (unsigned long)lm_alias(kasan_early_shadow_pgd_next)) { /* * pgdp can't be none since kasan_early_init @@ -258,7 +217,95 @@ static void __init kasan_populate_pgd(pgd_t *pgdp, } } - kasan_populate_pgd_next(pgdp, vaddr, next, early); + kasan_populate_pgd_next(pgdp, vaddr, next); + } while (pgdp++, vaddr = next, vaddr != end); +} + +static void __init kasan_early_populate_pud(p4d_t *p4dp, + unsigned long vaddr, + unsigned long end) +{ + pud_t *pudp, *base_pud; + phys_addr_t phys_addr; + unsigned long next; + + if (!pgtable_l4_enabled) { + pudp = (pud_t *)p4dp; + } else { + base_pud = pt_ops.get_pud_virt(pfn_to_phys(_p4d_pfn(*p4dp))); + pudp = base_pud + pud_index(vaddr); + } + + do { + next = pud_addr_end(vaddr, end); + + if (pud_none(*pudp) && IS_ALIGNED(vaddr, PUD_SIZE) && + (next - vaddr) >= PUD_SIZE) { + phys_addr = __pa((uintptr_t)kasan_early_shadow_pmd); + set_pud(pudp, pfn_pud(PFN_DOWN(phys_addr), PAGE_TABLE)); + continue; + } + + BUG(); + } while (pudp++, vaddr = next, vaddr != end); +} + +static void __init kasan_early_populate_p4d(pgd_t *pgdp, + unsigned long vaddr, + unsigned long end) +{ + p4d_t *p4dp, *base_p4d; + phys_addr_t phys_addr; + unsigned long next; + + /* + * We can't use pgd_page_vaddr here as it would return a linear + * mapping address but it is not mapped yet, but when populating + * early_pg_dir, we need the physical address and when populating + * swapper_pg_dir, we need the kernel virtual address so use + * pt_ops facility. + * Note that this test is then completely equivalent to + * p4dp = p4d_offset(pgdp, vaddr) + */ + if (!pgtable_l5_enabled) { + p4dp = (p4d_t *)pgdp; + } else { + base_p4d = pt_ops.get_p4d_virt(pfn_to_phys(_pgd_pfn(*pgdp))); + p4dp = base_p4d + p4d_index(vaddr); + } + + do { + next = p4d_addr_end(vaddr, end); + + if (p4d_none(*p4dp) && IS_ALIGNED(vaddr, P4D_SIZE) && + (next - vaddr) >= P4D_SIZE) { + phys_addr = __pa((uintptr_t)kasan_early_shadow_pud); + set_p4d(p4dp, pfn_p4d(PFN_DOWN(phys_addr), PAGE_TABLE)); + continue; + } + + kasan_early_populate_pud(p4dp, vaddr, next); + } while (p4dp++, vaddr = next, vaddr != end); +} + +static void __init kasan_early_populate_pgd(pgd_t *pgdp, + unsigned long vaddr, + unsigned long end) +{ + phys_addr_t phys_addr; + unsigned long next; + + do { + next = pgd_addr_end(vaddr, end); + + if (pgd_none(*pgdp) && IS_ALIGNED(vaddr, PGDIR_SIZE) && + (next - vaddr) >= PGDIR_SIZE) { + phys_addr = __pa((uintptr_t)kasan_early_shadow_p4d); + set_pgd(pgdp, pfn_pgd(PFN_DOWN(phys_addr), PAGE_TABLE)); + continue; + } + + kasan_early_populate_p4d(pgdp, vaddr, next); } while (pgdp++, vaddr = next, vaddr != end); } @@ -295,16 +342,16 @@ asmlinkage void __init kasan_early_init(void) PAGE_TABLE)); } - kasan_populate_pgd(early_pg_dir + pgd_index(KASAN_SHADOW_START), - KASAN_SHADOW_START, KASAN_SHADOW_END, true); + kasan_early_populate_pgd(early_pg_dir + pgd_index(KASAN_SHADOW_START), + KASAN_SHADOW_START, KASAN_SHADOW_END); local_flush_tlb_all(); } void __init kasan_swapper_init(void) { - kasan_populate_pgd(pgd_offset_k(KASAN_SHADOW_START), - KASAN_SHADOW_START, KASAN_SHADOW_END, true); + kasan_early_populate_pgd(pgd_offset_k(KASAN_SHADOW_START), + KASAN_SHADOW_START, KASAN_SHADOW_END); local_flush_tlb_all(); } @@ -314,7 +361,7 @@ static void __init kasan_populate(void *start, void *end) unsigned long vaddr = (unsigned long)start & PAGE_MASK; unsigned long vend = PAGE_ALIGN((unsigned long)end); - kasan_populate_pgd(pgd_offset_k(vaddr), vaddr, vend, false); + kasan_populate_pgd(pgd_offset_k(vaddr), vaddr, vend); local_flush_tlb_all(); memset(start, KASAN_SHADOW_INIT, end - start);