From patchwork Thu Dec 14 07:39:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nanyong Sun X-Patchwork-Id: 178519 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:3b04:b0:fb:cd0c:d3e with SMTP id c4csp8362339dys; Wed, 13 Dec 2023 23:01:42 -0800 (PST) X-Google-Smtp-Source: AGHT+IGiLcDVLhsqX2qGNAQdQvNrJ6dtsrzhV7AFLR/Kn1zxkE0RTCE45Yq+QT75QxGhmkT76FoA X-Received: by 2002:a9d:74c1:0:b0:6d9:d05c:8fe with SMTP id a1-20020a9d74c1000000b006d9d05c08femr8624177otl.35.1702537302446; Wed, 13 Dec 2023 23:01:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702537302; cv=none; d=google.com; s=arc-20160816; b=mkbUTcT4Z4zOxHCjgDEwwa0nLb/nrye2fYinPJ2r641RIy92KYT66/p3KVaU48ogHC NtVCX7bduys81E+WEBdiUtFyNC5xTxXBHVMKb8cidrs998kusvWLpVoANkQ/XCN9STxR 06aAP8dxTfMCU7ZxMpjbembNw+lVvV2MndQ/rPvWIxTkuK3N3KhRC1UHHVIXcBG56t5X TX9MkYLg5oiaBeNX4PXGy5R6i/aEplQbC1YMWCFd6vjDCVlAUwVeM9curL1CAwI3QERX 8tNj63n38GUMTjngk1Xsm3UQmoY9aHcG+J9UkElqSsQUyAcwWorgJck59CcY8C82+8wv AUjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=OgJlxUQDILr7vk9eqhUikkWa51wWaYZkIB9AjYwmae4=; fh=IBRY2OfN0iTWW2ZjSSKvUJynIiV/JLaiDZe55wN3MqA=; b=kpJYWeb7CKoCLTO1w9D/0v6au0ur9GjSVC2jeU0JTE8+ALjuo3U0I+wNmjcgrzg8Ib ajIDNRUivXJulYrYkB17JlZzmh383oVG/Ipy/EAnVYWfx5pLqzC6tuCZfXNiN9WYBfY3 9MTBQ2dpFBXfbV9RWdE4JPQDNP6PdRQMKS0klLPK3H7nC8W1UlchRgJlNwXu2x/ZUgl9 G4EgEHLrM6E2DDgyHMQz9n1mDuf2FQOJiKyRQSA9oX0qbIKeYBnqdiw5e/nKXGAEmzRz 62KgUEdk94uz1KwFQLB+ZHfHKUSfVGeXHuoNsvA2NWm75uC0OQdjBUTPmhx+nnG1Sk3H fPiQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id r9-20020a63ec49000000b005c658c16acdsi5967244pgj.196.2023.12.13.23.01.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Dec 2023 23:01:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 91F738029C52; Wed, 13 Dec 2023 23:01:40 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1443250AbjLNHBZ (ORCPT + 99 others); Thu, 14 Dec 2023 02:01:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50036 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234344AbjLNHBU (ORCPT ); Thu, 14 Dec 2023 02:01:20 -0500 Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [45.249.212.190]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6EF64E8 for ; Wed, 13 Dec 2023 23:01:26 -0800 (PST) Received: from mail.maildlp.com (unknown [172.19.88.214]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4SrN5g0gYQz29fy0; Thu, 14 Dec 2023 14:40:39 +0800 (CST) Received: from kwepemm000003.china.huawei.com (unknown [7.193.23.66]) by mail.maildlp.com (Postfix) with ESMTPS id 5007C1A0190; Thu, 14 Dec 2023 14:41:47 +0800 (CST) Received: from huawei.com (10.175.113.32) by kwepemm000003.china.huawei.com (7.193.23.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Thu, 14 Dec 2023 14:41:46 +0800 From: Nanyong Sun To: , , , , , CC: , , , , , Subject: [PATCH 1/3] mm: HVO: introduce helper function to update and flush pgtable Date: Thu, 14 Dec 2023 15:39:10 +0800 Message-ID: <20231214073912.1938330-2-sunnanyong@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231214073912.1938330-1-sunnanyong@huawei.com> References: <20231214073912.1938330-1-sunnanyong@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.113.32] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To kwepemm000003.china.huawei.com (7.193.23.66) X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 13 Dec 2023 23:01:40 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785239754499309638 X-GMAIL-MSGID: 1785239754499309638 Add pmd/pte update and tlb flush helper function to update page table. This refactoring patch is designed to facilitate each architecture to implement its own special logic in preparation for the arm64 architecture to follow the necessary break-before-make sequence when updating page tables. Signed-off-by: Nanyong Sun Reviewed-by: Muchun Song --- mm/hugetlb_vmemmap.c | 55 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 43 insertions(+), 12 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 87818ee7f01d..49e8b351def3 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -45,6 +45,37 @@ struct vmemmap_remap_walk { unsigned long flags; }; +#ifndef vmemmap_update_pmd +static inline void vmemmap_update_pmd(unsigned long start, + pmd_t *pmd, pte_t *pgtable) +{ + pmd_populate_kernel(&init_mm, pmd, pgtable); +} +#endif + +#ifndef vmemmap_update_pte +static inline void vmemmap_update_pte(unsigned long addr, + pte_t *pte, pte_t entry) +{ + set_pte_at(&init_mm, addr, pte, entry); +} +#endif + +#ifndef flush_tlb_vmemmap_all +static inline void flush_tlb_vmemmap_all(void) +{ + flush_tlb_all(); +} +#endif + +#ifndef flush_tlb_vmemmap_range +static inline void flush_tlb_vmemmap_range(unsigned long start, + unsigned long end) +{ + flush_tlb_kernel_range(start, end); +} +#endif + static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush) { pmd_t __pmd; @@ -87,9 +118,9 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush) /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); - pmd_populate_kernel(&init_mm, pmd, pgtable); + vmemmap_update_pmd(start, pmd, pgtable); if (flush) - flush_tlb_kernel_range(start, start + PMD_SIZE); + flush_tlb_vmemmap_range(start, start + PMD_SIZE); } else { pte_free_kernel(&init_mm, pgtable); } @@ -217,7 +248,7 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, } while (pgd++, addr = next, addr != end); if (walk->remap_pte && !(walk->flags & VMEMMAP_REMAP_NO_TLB_FLUSH)) - flush_tlb_kernel_range(start, end); + flush_tlb_vmemmap_range(start, end); return 0; } @@ -263,15 +294,15 @@ static void vmemmap_remap_pte(pte_t *pte, unsigned long addr, /* * Makes sure that preceding stores to the page contents from - * vmemmap_remap_free() become visible before the set_pte_at() - * write. + * vmemmap_remap_free() become visible before the + * vmemmap_update_pte() write. */ smp_wmb(); } entry = mk_pte(walk->reuse_page, pgprot); list_add(&page->lru, walk->vmemmap_pages); - set_pte_at(&init_mm, addr, pte, entry); + vmemmap_update_pte(addr, pte, entry); } /* @@ -310,10 +341,10 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, /* * Makes sure that preceding stores to the page contents become visible - * before the set_pte_at() write. + * before the vmemmap_update_pte() write. */ smp_wmb(); - set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); + vmemmap_update_pte(addr, pte, mk_pte(page, pgprot)); } /** @@ -576,7 +607,7 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, } if (restored) - flush_tlb_all(); + flush_tlb_vmemmap_all(); if (!ret) ret = restored; return ret; @@ -744,7 +775,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l break; } - flush_tlb_all(); + flush_tlb_vmemmap_all(); list_for_each_entry(folio, folio_list, lru) { int ret = __hugetlb_vmemmap_optimize_folio(h, folio, @@ -760,7 +791,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l * allowing more vmemmap remaps to occur. */ if (ret == -ENOMEM && !list_empty(&vmemmap_pages)) { - flush_tlb_all(); + flush_tlb_vmemmap_all(); free_vmemmap_page_list(&vmemmap_pages); INIT_LIST_HEAD(&vmemmap_pages); __hugetlb_vmemmap_optimize_folio(h, folio, @@ -769,7 +800,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l } } - flush_tlb_all(); + flush_tlb_vmemmap_all(); free_vmemmap_page_list(&vmemmap_pages); } From patchwork Thu Dec 14 07:39:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nanyong Sun X-Patchwork-Id: 178517 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:3b04:b0:fb:cd0c:d3e with SMTP id c4csp8360712dys; Wed, 13 Dec 2023 22:57:25 -0800 (PST) X-Google-Smtp-Source: AGHT+IHG6oAJyp1soY1lCylA7/PrdiMSP2/RMo/SzI2t1TiKwuBPP5j2389ewQH85ajq4yY4PoRU X-Received: by 2002:a05:6e02:218a:b0:35d:7acc:c455 with SMTP id j10-20020a056e02218a00b0035d7accc455mr13168134ila.64.1702537045724; Wed, 13 Dec 2023 22:57:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702537045; cv=none; d=google.com; s=arc-20160816; b=HzgndY+InXC3+dFheD7U0ZdVwZAxDopPLvvm0qbJw5vX7Up2RAOmSDpQhDR5llZT4l CB3YSCkJ/ddm25oUXTKn/caVfgvq1/kJJ4EvRscYwzz9eKqAPbJxBc7JBi1Z7duY9zlz LDp9MNHy/PMX6DsnFe6o+g9bFMMDYqTUFokvZy92EhPEJfc/FoZ9YtavntYY4kQZm30t XjQZA+0YWmnxTVdTN03qndcFygMRzcR2x6IgQQg9ukBYH0qP6hvn/hYmfKmd4amSVUCv UbTGXa+Ju35gtYUoppf1Svdb9/tMSOqFJKZaQLscPXptkklvae91IKSY6z1uUAcdgk/J 6WJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=RTOem4NWFP8g4Yoo/TlqaSw/FCROfYkCnwy3hQYUVc4=; fh=IBRY2OfN0iTWW2ZjSSKvUJynIiV/JLaiDZe55wN3MqA=; b=HQ6ECTitrO544BWxp1ZtpZ0tysJU8gtH2QPPoafSgdZCFPm3m4jWOeDHn27/IExPDe wHiu0JJUowFp9uLZ8v0HMqnDT4I27c0bT6kiSES3WQd86E19VuF0anmiJc5l+7Sptla+ ZV1X614O1eauRQTIucdHRkaansVZPheHGYhC/GuvWwLyiRaKOlzf0ICWxqGs7tFYIxX/ 64jrMh40/dUsB43gBd6TdDFLPfE4huhHpBUdsN1DOoReFx53J8AIf6Pan2RHrI+GrOVJ PUBtuux64bKx2odMDOCyYM5/BPFRxxT8fAJXgg+5veQCyVSM/Ud0ZZSkmsgZCYIm/OOh QbMA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id i3-20020a170902c94300b001cffce3a2e2si11048399pla.426.2023.12.13.22.57.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Dec 2023 22:57:25 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 8E66182DC637; Wed, 13 Dec 2023 22:57:20 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234317AbjLNG5H (ORCPT + 99 others); Thu, 14 Dec 2023 01:57:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59530 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229629AbjLNG5G (ORCPT ); Thu, 14 Dec 2023 01:57:06 -0500 X-Greylist: delayed 923 seconds by postgrey-1.37 at lindbergh.monkeyblade.net; Wed, 13 Dec 2023 22:57:11 PST Received: from szxga05-in.huawei.com (szxga05-in.huawei.com [45.249.212.191]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E662CB9 for ; Wed, 13 Dec 2023 22:57:11 -0800 (PST) Received: from mail.maildlp.com (unknown [172.19.163.17]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4SrN2q1Gx2z1FDx3; Thu, 14 Dec 2023 14:38:11 +0800 (CST) Received: from kwepemm000003.china.huawei.com (unknown [7.193.23.66]) by mail.maildlp.com (Postfix) with ESMTPS id 4017C1A0597; Thu, 14 Dec 2023 14:41:48 +0800 (CST) Received: from huawei.com (10.175.113.32) by kwepemm000003.china.huawei.com (7.193.23.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Thu, 14 Dec 2023 14:41:47 +0800 From: Nanyong Sun To: , , , , , CC: , , , , , Subject: [PATCH 2/3] arm64: mm: HVO: support BBM of vmemmap pgtable safely Date: Thu, 14 Dec 2023 15:39:11 +0800 Message-ID: <20231214073912.1938330-3-sunnanyong@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231214073912.1938330-1-sunnanyong@huawei.com> References: <20231214073912.1938330-1-sunnanyong@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.113.32] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To kwepemm000003.china.huawei.com (7.193.23.66) X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 13 Dec 2023 22:57:20 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785239485206294822 X-GMAIL-MSGID: 1785239485206294822 Implement vmemmap_update_pmd and vmemmap_update_pte on arm64 to do BBM(break-before-make) logic when change the page table of vmemmap address, they will under the init_mm.page_table_lock. If a translation fault of vmemmap address concurrently happened after pte/pmd cleared, vmemmap page fault handler will acquire the init_mm.page_table_lock to wait for vmemmap update to complete, by then the virtual address is valid again, so PF can return and access can continue. In other case, do the traditional kernel fault. Implement flush_tlb_vmemmap_all and flush_tlb_vmemmap_range on arm64 with nothing to do because tlb already flushed in every single BBM. Signed-off-by: Nanyong Sun --- arch/arm64/include/asm/esr.h | 4 ++ arch/arm64/include/asm/mmu.h | 20 ++++++++ arch/arm64/mm/fault.c | 94 ++++++++++++++++++++++++++++++++++++ arch/arm64/mm/mmu.c | 28 +++++++++++ 4 files changed, 146 insertions(+) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index ae35939f395b..1c63256efd25 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -116,6 +116,10 @@ #define ESR_ELx_FSC_SERROR (0x11) #define ESR_ELx_FSC_ACCESS (0x08) #define ESR_ELx_FSC_FAULT (0x04) +#define ESR_ELx_FSC_FAULT_L0 (0x04) +#define ESR_ELx_FSC_FAULT_L1 (0x05) +#define ESR_ELx_FSC_FAULT_L2 (0x06) +#define ESR_ELx_FSC_FAULT_L3 (0x07) #define ESR_ELx_FSC_PERM (0x0C) #define ESR_ELx_FSC_SEA_TTW0 (0x14) #define ESR_ELx_FSC_SEA_TTW1 (0x15) diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 2fcf51231d6e..fcec5827f54f 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -76,5 +76,25 @@ extern bool kaslr_requires_kpti(void); #define INIT_MM_CONTEXT(name) \ .pgd = init_pg_dir, +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +void vmemmap_update_pmd(unsigned long start, pmd_t *pmd, pte_t *pgtable); +#define vmemmap_update_pmd vmemmap_update_pmd +void vmemmap_update_pte(unsigned long addr, pte_t *pte, pte_t entry); +#define vmemmap_update_pte vmemmap_update_pte + +static inline void flush_tlb_vmemmap_all(void) +{ + /* do nothing, already flushed tlb in every single BBM */ +} +#define flush_tlb_vmemmap_all flush_tlb_vmemmap_all + +static inline void flush_tlb_vmemmap_range(unsigned long start, + unsigned long end) +{ + /* do nothing, already flushed tlb in every single BBM */ +} +#define flush_tlb_vmemmap_range flush_tlb_vmemmap_range +#endif + #endif /* !__ASSEMBLY__ */ #endif diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 460d799e1296..7066a273c1e0 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -368,6 +368,97 @@ static bool is_el1_mte_sync_tag_check_fault(unsigned long esr) return false; } +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +static inline bool is_vmemmap_address(unsigned long addr) +{ + return (addr >= VMEMMAP_START) && (addr < VMEMMAP_END); +} + +static inline bool vmemmap_fault_may_fixup(unsigned long addr, + unsigned long esr) +{ + if (!is_vmemmap_address(addr)) + return false; + + /* + * Only try to handle translation fault level 2 or level 3, + * because hugetlb vmemmap optimize only clear pmd or pte. + */ + switch (esr & ESR_ELx_FSC) { + case ESR_ELx_FSC_FAULT_L2: + case ESR_ELx_FSC_FAULT_L3: + return true; + default: + return false; + } +} + +/* + * PMD mapped vmemmap should has been split as PTE mapped + * by HVO now, here we only check this case, other cases + * should fail. + * Also should check the addr is healthy enough that will not cause + * a level2 or level3 translation fault again after page fault + * handled with success, so we need check both bits[1:0] of PMD and + * PTE as ARM Spec mentioned below: + * A Translation fault is generated if bits[1:0] of a translation + * table descriptor identify the descriptor as either a Fault + * encoding or a reserved encoding. + */ +static inline bool vmemmap_addr_healthy(unsigned long addr) +{ + pgd_t *pgdp; + p4d_t *p4dp; + pud_t *pudp, pud; + pmd_t *pmdp, pmd; + pte_t *ptep, pte; + + pgdp = pgd_offset_k(addr); + if (pgd_none(READ_ONCE(*pgdp))) + return false; + + p4dp = p4d_offset(pgdp, addr); + if (p4d_none(READ_ONCE(*p4dp))) + return false; + + pudp = pud_offset(p4dp, addr); + pud = READ_ONCE(*pudp); + if (pud_none(pud)) + return false; + + pmdp = pmd_offset(pudp, addr); + pmd = READ_ONCE(*pmdp); + if (!pmd_table(pmd)) + return false; + + ptep = pte_offset_kernel(pmdp, addr); + pte = READ_ONCE(*ptep); + return (pte_val(pte) & PTE_TYPE_MASK) == PTE_TYPE_PAGE; +} + +static bool vmemmap_handle_page_fault(unsigned long addr, + unsigned long esr) +{ + bool ret = false; + + if (likely(!vmemmap_fault_may_fixup(addr, esr))) + return false; + + spin_lock(&init_mm.page_table_lock); + if (vmemmap_addr_healthy(addr)) + ret = true; + spin_unlock(&init_mm.page_table_lock); + + return ret; +} +#else +static inline bool vmemmap_handle_page_fault(unsigned long addr, + unsigned long esr) +{ + return false; +} +#endif /*CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ + static bool is_translation_fault(unsigned long esr) { return (esr & ESR_ELx_FSC_TYPE) == ESR_ELx_FSC_FAULT; @@ -409,6 +500,9 @@ static void __do_kernel_fault(unsigned long addr, unsigned long esr, kfence_handle_page_fault(addr, esr & ESR_ELx_WNR, regs)) return; + if (vmemmap_handle_page_fault(addr, esr)) + return; + msg = "paging request"; } diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 15f6347d23b6..81a600ccac7c 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1146,6 +1146,34 @@ int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node, return 1; } +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +/* + * In the window between the page table entry is cleared and filled + * with a new value, other threads have the opportunity to concurrently + * access the vmemmap area then page translation fault occur. + * Therefore, we need to ensure that the init_mm.page_table_lock is held + * to synchronize the vmemmap page fault handling which will wait for + * this lock to be released to ensure that the page table entry has been + * refreshed with a new valid value. + */ +void vmemmap_update_pmd(unsigned long start, pmd_t *pmd, pte_t *pgtable) +{ + lockdep_assert_held(&init_mm.page_table_lock); + pmd_clear(pmd); + flush_tlb_kernel_range(start, start + PMD_SIZE); + pmd_populate_kernel(&init_mm, pmd, pgtable); +} + +void vmemmap_update_pte(unsigned long addr, pte_t *pte, pte_t entry) +{ + spin_lock(&init_mm.page_table_lock); + pte_clear(&init_mm, addr, pte); + flush_tlb_kernel_range(addr, addr + PAGE_SIZE); + set_pte_at(&init_mm, addr, pte, entry); + spin_unlock(&init_mm.page_table_lock); +} +#endif + int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap) { From patchwork Thu Dec 14 07:39:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nanyong Sun X-Patchwork-Id: 178518 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:3b04:b0:fb:cd0c:d3e with SMTP id c4csp8362257dys; Wed, 13 Dec 2023 23:01:33 -0800 (PST) X-Google-Smtp-Source: AGHT+IE411+UF2zSBSp3Lhzw0bB9nSXpB1FfOAiq/h3+2djPtSykxzA7o9ENsAITA841jmFoysok X-Received: by 2002:a17:90a:d143:b0:286:6cd8:ef04 with SMTP id t3-20020a17090ad14300b002866cd8ef04mr12480920pjw.28.1702537293566; Wed, 13 Dec 2023 23:01:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702537293; cv=none; d=google.com; s=arc-20160816; b=xS9MRA8ef8+DLtAoxyshKZ2cQfQ1035dc78j15HrSOft4QQXjb0K222e84wPUmg1Cr Z5eZV7sbmTFAWe9p01G4F3NWv5CJLxeF9riwtYvQC3eRoothD4CgfX1r/bIfTvjx26vP +1vP4MV9Uo5e1XwcM9dFwn9U3mbJ+giebpGgo370pA4UuHeINGfrwj+/NZOw3QnGcMCD /6MLpStcRG2lhZ7LbEYGzMb/sbBNSCAV3jplQYBXV7+5Z0JLtchWUo7jCNZMrKCixmzR 7i6JWE1D2YKYPYfKLXxrvfdTwVCO3VSW101kW+CIyVq91UVI4RCmPivRCQ9kg9LQ1rR3 VAfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=RbqiuXsHA2FdxHJz22f/UdGbBmiJ/DwpGUMHSWGjULA=; fh=IBRY2OfN0iTWW2ZjSSKvUJynIiV/JLaiDZe55wN3MqA=; b=YfKgzconX/aB5sFEnHDlAxuUzkJUvFlXNOK+712DZuMjVzXKHixmkI+GTDt/JP8uKt QkUR7bWOyaTs97Hsd6vg9F0X4auMcJXTmejjXrgEk0JFa0qt5nUNd8SSGEJWYk+kKXrs javxAne8IG0lqe7vDNIfB2iYH40OdkHp9axdP0rIq3HhTHkUq6wYFo5UcIB5obEmm2C8 5cu58hn9vGSnHnX9bxWWbWZnbSgrXk2ytarclS/oIbRa30aFMDocniNv9o/8xJaaJITP OM1PXNazZDWEP8fRBj767KTpubu8gTo6V7GGrisXB5n4bcV1XMgr57NgEmYWXLmYwIwN AR5w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id 4-20020a631544000000b005c201808430si10842209pgv.120.2023.12.13.23.01.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Dec 2023 23:01:33 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 3BE028029DD2; Wed, 13 Dec 2023 23:01:32 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1443267AbjLNHBX (ORCPT + 99 others); Thu, 14 Dec 2023 02:01:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50030 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234331AbjLNHBU (ORCPT ); Thu, 14 Dec 2023 02:01:20 -0500 Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [45.249.212.190]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F314F5 for ; Wed, 13 Dec 2023 23:01:26 -0800 (PST) Received: from mail.maildlp.com (unknown [172.19.162.112]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4SrN5h6bP5z29g0v; Thu, 14 Dec 2023 14:40:40 +0800 (CST) Received: from kwepemm000003.china.huawei.com (unknown [7.193.23.66]) by mail.maildlp.com (Postfix) with ESMTPS id 26D4514038F; Thu, 14 Dec 2023 14:41:49 +0800 (CST) Received: from huawei.com (10.175.113.32) by kwepemm000003.china.huawei.com (7.193.23.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Thu, 14 Dec 2023 14:41:48 +0800 From: Nanyong Sun To: , , , , , CC: , , , , , Subject: [PATCH 3/3] arm64: mm: Re-enable OPTIMIZE_HUGETLB_VMEMMAP Date: Thu, 14 Dec 2023 15:39:12 +0800 Message-ID: <20231214073912.1938330-4-sunnanyong@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231214073912.1938330-1-sunnanyong@huawei.com> References: <20231214073912.1938330-1-sunnanyong@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.113.32] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To kwepemm000003.china.huawei.com (7.193.23.66) X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 13 Dec 2023 23:01:32 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785239745284264976 X-GMAIL-MSGID: 1785239745284264976 Now update of vmemmap page table can follow the rule of break-before-make safely for arm64 architecture, re-enable HVO on arm64. Signed-off-by: Nanyong Sun Reviewed-by: Muchun Song --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 7b071a00425d..43e3d5576fb2 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -103,6 +103,7 @@ config ARM64 select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_HUGE_PMD_SHARE if ARM64_4K_PAGES || (ARM64_16K_PAGES && !ARM64_VA_BITS_36) + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_NO_INSTR select ARCH_WANTS_THP_SWAP if ARM64_4K_PAGES