From patchwork Sun Jun 18 05:30:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yajun Deng X-Patchwork-Id: 109575 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2323576vqr; Sat, 17 Jun 2023 23:50:23 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4qhk8EQaOAPQj2Ntqx/rUmIq4S7gduRs3/YYOFI6rOEqajxKSl7m+nTIp9AMkf5UzNubIA X-Received: by 2002:a05:6a20:e60a:b0:116:d50e:de38 with SMTP id my10-20020a056a20e60a00b00116d50ede38mr5107684pzb.31.1687071022882; Sat, 17 Jun 2023 23:50:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687071022; cv=none; d=google.com; s=arc-20160816; b=RcdwVCDjxQoUuFftv+Gu2KQDPqZY1ONZbtwYlJwImzvGq2L86DfOeW41e4hPiY4VKC 1CW9PCRp+Vws6QiDdNCizv3gpLK20fRvTV4wXaePU7CF3I1Z31A1PvU1qmCtZkjOmr2Z 2bTNwK03Nfuc/UgVsKQxJ9/Sug7weMA4lzqzMcZsq4GG0HAJV47yf0OOPIiwRL6m35Ac Qyzc2vvgSMS13LF4n5GrXz+L4NDo9gYFJYEDqP0AOtE36/1xbZgQoacRXdadBppIQcSv sEu5QJkW4m/Kl4f6eUjU0nGK2dcIwOaVEd7Sc0+IcLfXs6LWMiKC261JNWa3/6njpBP6 gR8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=aBYmdBEH1qB16YD6TQS5RIcWSGINKfp80AeFzQ6XOsg=; b=lr8ugzN/66Z9w+IGVdMgB5nt9fGWSd3PYrKueomSGXQ8XukgROsYdVYMIyPjUiMcro NJzGxr9qv7qHoeUmr/eEEesIunYCyjL38NJYBgRo/HWxfoyH3dcg9s6eBfXCO9I70MXd Us8XnyqLJsV640x+lebzQRj4ZiBLIDrMxX8K33z+I9J6s24majlKYELcCeacZoNv/ENI oaLkVMHngQi00AHUpswz0PhlQMn70CkviNwjDHQFvgsc3Pl7lwuBux5BcMJe54fCSI1H WfEFjQVlW5RH3YXOBSXB6T+CEhUGBDRw4rg20We9TUd+rBCwHV/FjWDecUEsDlauprJQ qGMg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=LeJfEXum; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z7-20020a626507000000b006664122cc6csi8776202pfb.404.2023.06.17.23.50.08; Sat, 17 Jun 2023 23:50:22 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=LeJfEXum; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229545AbjFRFaz (ORCPT + 99 others); Sun, 18 Jun 2023 01:30:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40230 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229574AbjFRFaU (ORCPT ); Sun, 18 Jun 2023 01:30:20 -0400 Received: from out-62.mta0.migadu.com (out-62.mta0.migadu.com [91.218.175.62]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 17CAC123 for ; Sat, 17 Jun 2023 22:30:17 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1687066215; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=aBYmdBEH1qB16YD6TQS5RIcWSGINKfp80AeFzQ6XOsg=; b=LeJfEXumxxsSmSrwA0/6hbN57mvPaEOYLtVkUp8a/TJOPWUAPG8qb6FLyim8+6JeHI+yCG aKNFs7on1MViZ4jWWaAkOl1l9kvJ7LsMDQMeYm8dLPGtsN4CfrmVqXsZ2m3CEyJ1cEsDmV Niape9HU++wF2Rz6h2u8UEW0rlZnJLA= From: Yajun Deng To: akpm@linux-foundation.org, rppt@kernel.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yajun Deng , kernel test robot Subject: [PATCH v3] mm: pass nid to reserve_bootmem_region() Date: Sun, 18 Jun 2023 13:30:00 +0800 Message-Id: <20230618053000.2930604-1-yajun.deng@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1769022184618573994?= X-GMAIL-MSGID: =?utf-8?q?1769022184618573994?= early_pfn_to_nid() is called frequently in init_reserved_page(), it returns the node id of the PFN. These PFN are probably from the same memory region, they have the same node id. It's not necessary to call early_pfn_to_nid() for each PFN. Pass nid to reserve_bootmem_region() and drop the call to early_pfn_to_nid() in init_reserved_page(). Also, set nid on all reserved pages before doing this, as some reserved memory regions may not be set nid. The most beneficial function is memmap_init_reserved_pages() if CONFIG_DEFERRED_STRUCT_PAGE_INIT is enabled. The following data was tested on an x86 machine with 190GB of RAM. before: memmap_init_reserved_pages() 67ms after: memmap_init_reserved_pages() 20ms Signed-off-by: Yajun Deng Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-kbuild-all/202306160145.juJMr3Bi-lkp@intel.com Reviewed-by: Mike Rapoport (IBM) --- include/linux/mm.h | 3 ++- mm/memblock.c | 31 +++++++++++++++++++++---------- mm/mm_init.c | 27 +++++++++++++++------------ 3 files changed, 38 insertions(+), 23 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index fdd966b11f79..a7a0e692d44d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2960,7 +2960,8 @@ extern unsigned long free_reserved_area(void *start, void *end, extern void adjust_managed_page_count(struct page *page, long count); -extern void reserve_bootmem_region(phys_addr_t start, phys_addr_t end); +extern void reserve_bootmem_region(phys_addr_t start, + phys_addr_t end, int nid); /* Free the reserved page into the buddy system, so it gets managed. */ static inline void free_reserved_page(struct page *page) diff --git a/mm/memblock.c b/mm/memblock.c index ff0da1858778..f9e61e565a53 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -2091,19 +2091,30 @@ static void __init memmap_init_reserved_pages(void) { struct memblock_region *region; phys_addr_t start, end; - u64 i; + int nid; + + /* + * set nid on all reserved pages and also treat struct + * pages for the NOMAP regions as PageReserved + */ + for_each_mem_region(region) { + nid = memblock_get_region_node(region); + start = region->base; + end = start + region->size; + + if (memblock_is_nomap(region)) + reserve_bootmem_region(start, end, nid); + + memblock_set_node(start, end, &memblock.reserved, nid); + } /* initialize struct pages for the reserved regions */ - for_each_reserved_mem_range(i, &start, &end) - reserve_bootmem_region(start, end); + for_each_reserved_mem_region(region) { + nid = memblock_get_region_node(region); + start = region->base; + end = start + region->size; - /* and also treat struct pages for the NOMAP regions as PageReserved */ - for_each_mem_region(region) { - if (memblock_is_nomap(region)) { - start = region->base; - end = start + region->size; - reserve_bootmem_region(start, end); - } + reserve_bootmem_region(start, end, nid); } } diff --git a/mm/mm_init.c b/mm/mm_init.c index d393631599a7..06cd5f93f9e7 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -646,10 +646,8 @@ static inline void pgdat_set_deferred_range(pg_data_t *pgdat) } /* Returns true if the struct page for the pfn is initialised */ -static inline bool __meminit early_page_initialised(unsigned long pfn) +static inline bool __meminit early_page_initialised(unsigned long pfn, int nid) { - int nid = early_pfn_to_nid(pfn); - if (node_online(nid) && pfn >= NODE_DATA(nid)->first_deferred_pfn) return false; @@ -695,15 +693,14 @@ defer_init(int nid, unsigned long pfn, unsigned long end_pfn) return false; } -static void __meminit init_reserved_page(unsigned long pfn) +static void __meminit init_reserved_page(unsigned long pfn, int nid) { pg_data_t *pgdat; - int nid, zid; + int zid; - if (early_page_initialised(pfn)) + if (early_page_initialised(pfn, nid)) return; - nid = early_pfn_to_nid(pfn); pgdat = NODE_DATA(nid); for (zid = 0; zid < MAX_NR_ZONES; zid++) { @@ -717,7 +714,7 @@ static void __meminit init_reserved_page(unsigned long pfn) #else static inline void pgdat_set_deferred_range(pg_data_t *pgdat) {} -static inline bool early_page_initialised(unsigned long pfn) +static inline bool early_page_initialised(unsigned long pfn, int nid) { return true; } @@ -727,7 +724,7 @@ static inline bool defer_init(int nid, unsigned long pfn, unsigned long end_pfn) return false; } -static inline void init_reserved_page(unsigned long pfn) +static inline void init_reserved_page(unsigned long pfn, int nid) { } #endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ @@ -738,7 +735,8 @@ static inline void init_reserved_page(unsigned long pfn) * marks the pages PageReserved. The remaining valid pages are later * sent to the buddy page allocator. */ -void __meminit reserve_bootmem_region(phys_addr_t start, phys_addr_t end) +void __meminit reserve_bootmem_region(phys_addr_t start, + phys_addr_t end, int nid) { unsigned long start_pfn = PFN_DOWN(start); unsigned long end_pfn = PFN_UP(end); @@ -747,7 +745,7 @@ void __meminit reserve_bootmem_region(phys_addr_t start, phys_addr_t end) if (pfn_valid(start_pfn)) { struct page *page = pfn_to_page(start_pfn); - init_reserved_page(start_pfn); + init_reserved_page(start_pfn, nid); /* Avoid false-positive PageTail() */ INIT_LIST_HEAD(&page->lru); @@ -2579,7 +2577,12 @@ void __init set_dma_reserve(unsigned long new_dma_reserve) void __init memblock_free_pages(struct page *page, unsigned long pfn, unsigned int order) { - if (!early_page_initialised(pfn)) + int nid = 0; + + if (IS_ENABLED(CONFIG_DEFERRED_STRUCT_PAGE_INIT)) + nid = early_pfn_to_nid(pfn); + + if (!early_page_initialised(pfn, nid)) return; if (!kmsan_memblock_free_pages(page, order)) { /* KMSAN will take care of these pages. */