From patchwork Wed Mar 8 09:41:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 66131 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp237983wrd; Wed, 8 Mar 2023 01:57:30 -0800 (PST) X-Google-Smtp-Source: AK7set9eJkqBnw83ltcFRK5gaVqBczr4oItrcFwMPAdvwohAURuTlFS6An9sOBKmyFH+5l7wZySh X-Received: by 2002:a17:906:9f21:b0:88a:7037:855e with SMTP id fy33-20020a1709069f2100b0088a7037855emr19642544ejc.9.1678269450376; Wed, 08 Mar 2023 01:57:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678269450; cv=none; d=google.com; s=arc-20160816; b=NmuYh+w/k0DD8ggzSTlnQ+b7P8jWHrU/CfoQWhYNz3wTphbxk+RYj98hxKE7H3qXgM j6NRatUtjbIgh/8H5/7IKry9Bb1uy2Pc7gHfp/uGZcPVkpeMRdlrs29wPixZiYhI7d8G c+AEMsHV0Q+VNiAxh8zUM0xS4C+XXA2dEpCSIdICg5SDrfMRCI7cO/+ns1/lHSX2CrNx CBMs28Rf1GGp6z44B/813UMYR7Sq9YNPHUC2xzfcVI09/NaaJfb2jZ+151gGT7M99xeQ bAbmDqD0kYqwmOvlPeVm7rpPxjVAriJs1I/nmmrg/MHgabQhZ32F2fdxybCpfHGY54NM s33Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=7mWerlB7iI5JwLACExHKsBgwf4sqipB5x9+UyIghBDI=; b=RpFo6oHl4+J81PeFlq6W736M7Cipw9RhP9KIHbXuNv0ChLMWXwpims0MjldkEbnkfw RfcHsAabF/F6vc7W831fKT1vHxzpK9DPi9fMWlrPOI6vgYHe0FeeYddD+7+4R94l02JZ NM9tUjwY83GEQuSYkAqRrvKgpX2pDNtQkGpUDYeZwtBUhZl0vJNOJr17lvJ927fsEQ02 jAoRTdP6KOjc/B5ytCIkZcH1UywIJqqJUA4IhMhVDmCNVKK3Yv9VCVevl/t7Mi8/F0Yr OuSKMZbmTLyl8y0MsjmX24KA238xqzQ26KCZAYs1Qq0CTqijSwk5UwahfXGdPJ0LUXI8 WS+g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=g2gtD1xh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qk6-20020a1709077f8600b008e01978a238si6501928ejc.52.2023.03.08.01.57.07; Wed, 08 Mar 2023 01:57:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=g2gtD1xh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230122AbjCHJlx (ORCPT + 99 others); Wed, 8 Mar 2023 04:41:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51864 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229929AbjCHJlh (ORCPT ); Wed, 8 Mar 2023 04:41:37 -0500 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 62B71B1ED1 for ; Wed, 8 Mar 2023 01:41:28 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 7B9C2CE1F10 for ; Wed, 8 Mar 2023 09:41:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2E1AAC4339B; Wed, 8 Mar 2023 09:41:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1678268484; bh=yWy9YBr3QRwlwUhGDvVMLg0rTuhOsa1avYZumFur5Fc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=g2gtD1xhW3u8pPRbZe/MH1SYrnQyB3Pg3cb3uhQHhN6UnNrqhop4CTR6FmDAcS2St YCjcuOxw+R2wN9J/DVCCiwFM1JQcWLMMdg1T7gVK2z2YpXDqJruWy/AlgmPjaCy147 JGJvvZPr9xvET4zf6lp6RmFzW7BMjIPm/A2p+scS/WAQF9qM78NbN+qaLFs0wooc6t H+Ixc0EkDbJU229EW9sLUolO7nPNsXja4Aq4l/mDMesoTliZWImHdvrCLOpMy+NyMW VntaGV1oKqnqu8ipRu6ukpj4mvrmJREPGAWAm85CMkMG5ERTOgugoBXvyVkB9UWYpg bkJ8fmxQCrpmg== From: Mike Rapoport To: linux-mm@kvack.org Cc: Andrew Morton , Dave Hansen , Mike Rapoport , Peter Zijlstra , Rick Edgecombe , Song Liu , Thomas Gleixner , Vlastimil Babka , linux-kernel@vger.kernel.org, x86@kernel.org Subject: [RFC PATCH 1/5] mm: intorduce __GFP_UNMAPPED and unmapped_alloc() Date: Wed, 8 Mar 2023 11:41:02 +0200 Message-Id: <20230308094106.227365-2-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230308094106.227365-1-rppt@kernel.org> References: <20230308094106.227365-1-rppt@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759793067494279991?= X-GMAIL-MSGID: =?utf-8?q?1759793067494279991?= From: "Mike Rapoport (IBM)" When set_memory or set_direct_map APIs used to change attribute or permissions for chunks of several pages, the large PMD that maps these pages in the direct map must be split. Fragmenting the direct map in such manner causes TLB pressure and, eventually, performance degradation. To avoid excessive direct map fragmentation, add ability to allocate "unmapped" pages with __GFP_UNMAPPED flag that will cause removal of the allocated pages from the direct map and use a cache of the unmapped pages. This cache is replenished with higher order pages with preference for PMD_SIZE pages when possible so that there will be fewer splits of large pages in the direct map. The cache is implemented as a buddy allocator, so it can serve high order allocations of unmapped pages. Signed-off-by: Mike Rapoport (IBM) Signed-off-by: Kent Overstreet --- arch/x86/Kconfig | 3 + include/linux/gfp_types.h | 11 +- include/linux/page-flags.h | 6 + include/linux/pageblock-flags.h | 28 +++++ include/trace/events/mmflags.h | 10 +- mm/Kconfig | 4 + mm/Makefile | 1 + mm/internal.h | 22 ++++ mm/page_alloc.c | 29 ++++- mm/unmapped-alloc.c | 215 ++++++++++++++++++++++++++++++++ 10 files changed, 325 insertions(+), 4 deletions(-) create mode 100644 mm/unmapped-alloc.c diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index a825bf031f49..735f691d449c 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -35,6 +35,9 @@ config X86_64 select ARCH_HAS_ELFCORE_COMPAT select ZONE_DMA32 +config ARCH_WANTS_GFP_UNMAPPED + def_bool y if X86_64 + config FORCE_DYNAMIC_FTRACE def_bool y depends on X86_32 diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 5088637fe5c2..234122b434dd 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -60,6 +60,12 @@ typedef unsigned int __bitwise gfp_t; #else #define ___GFP_NOLOCKDEP 0 #endif +#ifdef CONFIG_UNMAPPED_ALLOC +#define ___GFP_UNMAPPED 0x10000000u +#else +#define ___GFP_UNMAPPED 0 +#endif + /* If the above are modified, __GFP_BITS_SHIFT may need updating */ /* @@ -101,12 +107,15 @@ typedef unsigned int __bitwise gfp_t; * node with no fallbacks or placement policy enforcements. * * %__GFP_ACCOUNT causes the allocation to be accounted to kmemcg. + * + * %__GFP_UNMAPPED removes the allocated pages from the direct map. */ #define __GFP_RECLAIMABLE ((__force gfp_t)___GFP_RECLAIMABLE) #define __GFP_WRITE ((__force gfp_t)___GFP_WRITE) #define __GFP_HARDWALL ((__force gfp_t)___GFP_HARDWALL) #define __GFP_THISNODE ((__force gfp_t)___GFP_THISNODE) #define __GFP_ACCOUNT ((__force gfp_t)___GFP_ACCOUNT) +#define __GFP_UNMAPPED ((__force gfp_t)___GFP_UNMAPPED) /** * DOC: Watermark modifiers @@ -252,7 +261,7 @@ typedef unsigned int __bitwise gfp_t; #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) /* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT (27 + IS_ENABLED(CONFIG_LOCKDEP)) +#define __GFP_BITS_SHIFT (28 + IS_ENABLED(CONFIG_LOCKDEP)) #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) /** diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index a7e3a3405520..e66dbfdba8f2 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -922,6 +922,7 @@ static inline bool is_page_hwpoison(struct page *page) #define PG_offline 0x00000100 #define PG_table 0x00000200 #define PG_guard 0x00000400 +#define PG_unmapped 0x00000800 #define PageType(page, flag) \ ((page->page_type & (PAGE_TYPE_BASE | flag)) == PAGE_TYPE_BASE) @@ -992,6 +993,11 @@ PAGE_TYPE_OPS(Table, table) */ PAGE_TYPE_OPS(Guard, guard) +/* + * Marks pages in free lists of unmapped allocator. + */ +PAGE_TYPE_OPS(Unmapped, unmapped) + extern bool is_free_buddy_page(struct page *page); PAGEFLAG(Isolated, isolated, PF_ANY); diff --git a/include/linux/pageblock-flags.h b/include/linux/pageblock-flags.h index 5f1ae07d724b..bb6f5ec83881 100644 --- a/include/linux/pageblock-flags.h +++ b/include/linux/pageblock-flags.h @@ -21,6 +21,9 @@ enum pageblock_bits { /* 3 bits required for migrate types */ PB_migrate_skip,/* If set the block is skipped by compaction */ + PB_unmapped = 7,/* if set the block has pages unmapped in the direct + map */ + /* * Assume the bits will always align on a word. If this assumption * changes then get/set pageblock needs updating. @@ -95,4 +98,29 @@ static inline void set_pageblock_skip(struct page *page) } #endif /* CONFIG_COMPACTION */ +#ifdef CONFIG_UNMAPPED_ALLOC +#define get_pageblock_unmapped(page) \ + get_pfnblock_flags_mask(page, page_to_pfn(page), \ + (1 << (PB_unmapped))) +#define clear_pageblock_unmapped(page) \ + set_pfnblock_flags_mask(page, 0, page_to_pfn(page), \ + (1 << PB_unmapped)) +#define set_pageblock_unmapped(page) \ + set_pfnblock_flags_mask(page, (1 << PB_unmapped), \ + page_to_pfn(page), \ + (1 << PB_unmapped)) +#else /* CONFIG_UNMAPPED_ALLOC */ +static inline bool get_pageblock_unmapped(struct page *page) +{ + return false; +} +static inline void clear_pageblock_unmapped(struct page *page) +{ +} +static inline void set_pageblock_unmapped(struct page *page) +{ +} +#endif /* CONFIG_UNMAPPED_ALLOC */ + + #endif /* PAGEBLOCK_FLAGS_H */ diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index 9db52bc4ce19..951d294a3840 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -61,9 +61,17 @@ #define __def_gfpflag_names_kasan #endif +#ifdef CONFIG_UNMAPPED_ALLOC +#define __def_gfpflag_names_unmapped \ + , gfpflag_string(__GFP_UNMAPPED) +#else +#define __def_gfpflag_names_unmapped +#endif + #define show_gfp_flags(flags) \ (flags) ? __print_flags(flags, "|", \ - __def_gfpflag_names __def_gfpflag_names_kasan \ + __def_gfpflag_names __def_gfpflag_names_kasan \ + __def_gfpflag_names_unmapped \ ) : "none" #ifdef CONFIG_MMU diff --git a/mm/Kconfig b/mm/Kconfig index 4751031f3f05..404be73e00e8 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1202,6 +1202,10 @@ config LRU_GEN_STATS This option has a per-memcg and per-node memory overhead. # } +config UNMAPPED_ALLOC + def_bool y + depends on ARCH_WANTS_GFP_UNMAPPED + source "mm/damon/Kconfig" endmenu diff --git a/mm/Makefile b/mm/Makefile index 8e105e5b3e29..9143303295af 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -138,3 +138,4 @@ obj-$(CONFIG_IO_MAPPING) += io-mapping.o obj-$(CONFIG_HAVE_BOOTMEM_INFO_NODE) += bootmem_info.o obj-$(CONFIG_GENERIC_IOREMAP) += ioremap.o obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o +obj-$(CONFIG_UNMAPPED_ALLOC) += unmapped-alloc.o diff --git a/mm/internal.h b/mm/internal.h index 7920a8b7982e..8d84cceab467 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1042,4 +1042,26 @@ struct vma_prepare { struct vm_area_struct *remove; struct vm_area_struct *remove2; }; + +/* + * mm/unmapped-alloc.c + */ +#ifdef CONFIG_UNMAPPED_ALLOC +int unmapped_alloc_init(void); +struct page *unmapped_pages_alloc(gfp_t gfp, int order); +void unmapped_pages_free(struct page *page, int order); +#else +static inline int unmapped_alloc_init(void) +{ + return 0; +} + +static inline struct page *unmapped_pages_alloc(gfp_t gfp, int order) +{ + return NULL; +} + +static inline void unmapped_pages_free(struct page *page, int order) {} +#endif + #endif /* __MM_INTERNAL_H */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ac1fc986af44..01f18e7529b0 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -589,7 +589,7 @@ void set_pfnblock_flags_mask(struct page *page, unsigned long flags, unsigned long bitidx, word_bitidx; unsigned long word; - BUILD_BUG_ON(NR_PAGEBLOCK_BITS != 4); + BUILD_BUG_ON(NR_PAGEBLOCK_BITS != 8); BUILD_BUG_ON(MIGRATE_TYPES > (1 << PB_migratetype_bits)); bitmap = get_pageblock_bitmap(page, pfn); @@ -746,6 +746,13 @@ static inline bool pcp_allowed_order(unsigned int order) return false; } +static inline bool pcp_allowed(unsigned int order, gfp_t gfp) +{ + if (unlikely(gfp & __GFP_UNMAPPED)) + return false; + return pcp_allowed_order(order); +} + static inline void free_the_page(struct page *page, unsigned int order) { if (pcp_allowed_order(order)) /* Via pcp? */ @@ -1460,6 +1467,11 @@ static __always_inline bool free_pages_prepare(struct page *page, PAGE_SIZE << order); } + if (get_pageblock_unmapped(page)) { + unmapped_pages_free(page, order); + return false; + } + kernel_poison_pages(page, 1 << order); /* @@ -3525,6 +3537,13 @@ void free_unref_page_list(struct list_head *list) /* Prepare pages for freeing */ list_for_each_entry_safe(page, next, list, lru) { unsigned long pfn = page_to_pfn(page); + + if (get_pageblock_unmapped(page)) { + list_del(&page->lru); + unmapped_pages_free(page, 0); + continue; + } + if (!free_unref_page_prepare(page, pfn, 0)) { list_del(&page->lru); continue; @@ -3856,7 +3875,7 @@ struct page *rmqueue(struct zone *preferred_zone, */ WARN_ON_ONCE((gfp_flags & __GFP_NOFAIL) && (order > 1)); - if (likely(pcp_allowed_order(order))) { + if (likely(pcp_allowed(order, gfp_flags))) { /* * MIGRATE_MOVABLE pcplist could have the pages on CMA area and * we need to skip it when CMA area isn't allowed. @@ -5581,6 +5600,11 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, &alloc_gfp, &alloc_flags)) return NULL; + if (alloc_gfp & __GFP_UNMAPPED) { + page = unmapped_pages_alloc(gfp, order); + goto out; + } + /* * Forbid the first pass from falling back to types that fragment * memory until all local zones are considered. @@ -8437,6 +8461,7 @@ void __init free_area_init(unsigned long *max_zone_pfn) } memmap_init(); + unmapped_alloc_init(); } static int __init cmdline_parse_core(char *p, unsigned long *core, diff --git a/mm/unmapped-alloc.c b/mm/unmapped-alloc.c new file mode 100644 index 000000000000..fb2d54204a3c --- /dev/null +++ b/mm/unmapped-alloc.c @@ -0,0 +1,215 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include +#include +#include + +#include + +#include "internal.h" + +struct unmapped_free_area { + struct list_head free_list; + spinlock_t lock; + unsigned long nr_free; + unsigned long nr_cached; +}; + +static struct unmapped_free_area free_area[MAX_ORDER]; + +static inline void add_to_free_list(struct page *page, unsigned int order) +{ + struct unmapped_free_area *area = &free_area[order]; + + list_add(&page->buddy_list, &area->free_list); + area->nr_free++; +} + +static inline void del_page_from_free_list(struct page *page, unsigned int order) +{ + list_del(&page->buddy_list); + __ClearPageUnmapped(page); + set_page_private(page, 0); + free_area[order].nr_free--; +} + +static inline void set_unmapped_order(struct page *page, unsigned int order) +{ + set_page_private(page, order); + __SetPageUnmapped(page); +} + +static inline bool page_is_unmapped_buddy(struct page *page, struct page *buddy, + unsigned int order) +{ + if (!PageUnmapped(buddy)) + return false; + + if (buddy_order(buddy) != order) + return false; + + return true; +} + +static struct page *find_unmapped_buddy_page_pfn(struct page *page, + unsigned long pfn, + unsigned int order, + unsigned long *buddy_pfn) +{ + unsigned long __buddy_pfn = __find_buddy_pfn(pfn, order); + struct page *buddy; + + buddy = page + (__buddy_pfn - pfn); + if (buddy_pfn) + *buddy_pfn = __buddy_pfn; + + if (page_is_unmapped_buddy(page, buddy, order)) + return buddy; + + return NULL; +} + +static inline void __free_one_page(struct page *page, unsigned int order, + bool cache_refill) +{ + unsigned long pfn = page_to_pfn(page); + unsigned long buddy_pfn; + unsigned long combined_pfn; + struct page *buddy; + unsigned long flags; + + spin_lock_irqsave(&free_area->lock, flags); + + if (cache_refill) { + set_pageblock_unmapped(page); + free_area[order].nr_cached++; + } + + while (order < MAX_ORDER - 1) { + buddy = find_unmapped_buddy_page_pfn(page, pfn, order, + &buddy_pfn); + if (!buddy) + break; + + del_page_from_free_list(buddy, order); + combined_pfn = buddy_pfn & pfn; + page = page + (combined_pfn - pfn); + pfn = combined_pfn; + order++; + } + + set_unmapped_order(page, order); + add_to_free_list(page, order); + spin_unlock_irqrestore(&free_area->lock, flags); +} + +static inline void expand(struct page *page, int low, int high) +{ + unsigned long size = 1 << high; + + while (high > low) { + high--; + size >>= 1; + + add_to_free_list(&page[size], high); + set_unmapped_order(&page[size], high); + } +} + +static struct page *__rmqueue_smallest(unsigned int order) +{ + unsigned int current_order; + struct unmapped_free_area *area; + struct page *page = NULL; + unsigned long flags; + + spin_lock_irqsave(&free_area->lock, flags); + + /* Find a page of the appropriate size in the preferred list */ + for (current_order = order; current_order < MAX_ORDER; ++current_order) { + area = &free_area[current_order]; + page = list_first_entry_or_null(&area->free_list, struct page, + lru); + if (!page) + continue; + + del_page_from_free_list(page, current_order); + expand(page, order, current_order); + + break; + } + + spin_unlock_irqrestore(&free_area->lock, flags); + + return page; +} + +/* FIXME: have PMD_ORDER at last available in include/linux */ +#define PMD_ORDER (PMD_SHIFT - PAGE_SHIFT) + +struct page *unmapped_pages_alloc(gfp_t gfp, int order) +{ + + int cache_order = PMD_ORDER; + struct page *page; + + page = __rmqueue_smallest(order); + if (page) + goto out; + + gfp &= ~__GFP_UNMAPPED; + while (cache_order >= order) { + page = alloc_pages(gfp | __GFP_ZERO, cache_order); + if (page) + break; + cache_order--; + } + + if (page) { + unsigned long addr = (unsigned long)page_address(page); + unsigned long nr_pages = (1 << order); + unsigned long size = PAGE_SIZE * nr_pages; + + /* FIXME: set_direct_map_invalid_noflush may fail */ + for (int i = 0; i < nr_pages; i++) + set_direct_map_invalid_noflush(page + i); + + flush_tlb_kernel_range(addr, addr + size); + + /* + * FIXME: have this under lock so that allocation running + * in parallel won't steal all pages from the newly cached + * ones + */ + __free_one_page(page, cache_order, true); + page = __rmqueue_smallest(order); + + } + +out: + if (page) { + /* FIXME: __prep_new_page() expects page count of 0 */ + set_page_count(page, 0); + post_alloc_hook(page, order, GFP_KERNEL); + } + + return page; +} + +void unmapped_pages_free(struct page *page, int order) +{ + __free_one_page(page, order, false); +} + +int unmapped_alloc_init(void) +{ + for (int order = 0; order < MAX_ORDER; order++) { + struct unmapped_free_area *area = &free_area[order]; + INIT_LIST_HEAD(&area->free_list); + spin_lock_init(&area->lock); + } + + return 0; +} From patchwork Wed Mar 8 09:41:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 66133 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp238363wrd; Wed, 8 Mar 2023 01:58:44 -0800 (PST) X-Google-Smtp-Source: AK7set+ApVLf7/Zp52ko9hH66XfSwhC2h7TQ51e3u06jn4d2x3PG4oulHtfq6eYBz9rL00msmGxj X-Received: by 2002:aa7:cf8f:0:b0:4aa:a390:bf4a with SMTP id z15-20020aa7cf8f000000b004aaa390bf4amr17861809edx.20.1678269524147; Wed, 08 Mar 2023 01:58:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678269524; cv=none; d=google.com; s=arc-20160816; b=uSzfNKUvmEw72AtpIfQv84kJXFmY2xc4YDyYIWpvm05TXP6+wAhO4FK1LQ1HBgJO5g Xt6pboyL1OPtYEz2mYAzgtyGjt/3e263yxXGouvy5UuzH0XwhfOJlQY45KRiGjanbsEs ZLJlgaN6frMsvjpo3Dgpsx+GL6mtsaLgB3Dhxg/Zt8qxuJeyUR/A2a8J/7NcD2HZb4YH CO5je4P4ogDkMsyIstKm04LszZvf5Yn1T7+g5cT+uljIu85IYl+36Botn2hBGLDehh9s lVHyWJ26UVoLsJKMo1QdyU2Nbn+xpIv5qNfTm+17adqUb7qEYJvoUVTC+kj3Z2UJ3PX0 2iAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=TCY97IgQdv+eVDd67Oi2qOqfRry5DWncVhqaOuA69eE=; b=YGVzxHkUhQioP7Aw/oMjo4SD3JgO6vZ25iJpNgMrBBasSJhkMRE1tG2hXbGBxogMDn RW+RLnG1WHF3z4ZFEWULI7RbOp5kt8O7fq74XuPeoQwXzSHniZI/ipmY9nFDR/WHpTsN JNXCMBQzyGzhlW2i5kzbUncWKnBjo9Uj0zQ0VMxdjB5YfPQkseP41M3yz7eGyE6c4rmB VCFG8T9myPY8fNRdu4duOksEYtkRsZLKEHixkmzCdyiinNrI9DRggMbdbuHCB3QcjcJw tiv7+w6gl0fWphJhGXDbNFmMd+qxt6IdvtGd1IrlLnWr++SByWJotEQ0khRBNqBQ1pCD U8Gw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=a4zMp2Yn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x16-20020aa7dad0000000b004bdf14718c0si16240278eds.99.2023.03.08.01.58.21; Wed, 08 Mar 2023 01:58:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=a4zMp2Yn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230094AbjCHJlt (ORCPT + 99 others); Wed, 8 Mar 2023 04:41:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229943AbjCHJlh (ORCPT ); Wed, 8 Mar 2023 04:41:37 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E85CB3291 for ; Wed, 8 Mar 2023 01:41:31 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 12714B81C17 for ; Wed, 8 Mar 2023 09:41:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6E1ACC4339E; Wed, 8 Mar 2023 09:41:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1678268488; bh=nxbvrZboLg9dZQ5/M7cVi+ZACd7kAmzqj1YWR08+kKk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=a4zMp2Yn7HDfIOkHs2LHrjV9yaXYJYlXB/aKnJzLJ/PF5ZrlqK+R5tPO7fbFxNUxs IOg7AHUKv7exmJro4ZoTEYmbvSPOoqa73BwAI14TlGIR/CFSFZ+7ZTm1F0MbW4ELt4 6jMh8OZGtB42IpkD0kOZcf1Woaq6cEJMzUTm34te4vtlC80/UgPKBHW7wvG4rsoFI6 CQmqk2kwLGUSJ4nEjQYF3tw0ODzEx2YVylfv2/ajUcNhSCG99D43hWVU28PhQ7/TUw 8u+3ARAC6O8BItMDilZ/vzwy8gnulVEiyO5ZbkV8lD3YZSazdBAqfut4kR1L6muYwQ DRYHul/m47P0w== From: Mike Rapoport To: linux-mm@kvack.org Cc: Andrew Morton , Dave Hansen , Mike Rapoport , Peter Zijlstra , Rick Edgecombe , Song Liu , Thomas Gleixner , Vlastimil Babka , linux-kernel@vger.kernel.org, x86@kernel.org Subject: [RFC PATCH 2/5] mm/unmapped_alloc: add debugfs file similar to /proc/pagetypeinfo Date: Wed, 8 Mar 2023 11:41:03 +0200 Message-Id: <20230308094106.227365-3-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230308094106.227365-1-rppt@kernel.org> References: <20230308094106.227365-1-rppt@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759793144653826563?= X-GMAIL-MSGID: =?utf-8?q?1759793144653826563?= From: "Mike Rapoport (IBM)" Present statistics about unmapped_alloc in debugfs Signed-off-by: Mike Rapoport (IBM) --- mm/unmapped-alloc.c | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/mm/unmapped-alloc.c b/mm/unmapped-alloc.c index fb2d54204a3c..f74640e9ce9f 100644 --- a/mm/unmapped-alloc.c +++ b/mm/unmapped-alloc.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include @@ -213,3 +214,37 @@ int unmapped_alloc_init(void) return 0; } + +static int unmapped_alloc_debug_show(struct seq_file *m, void *private) +{ + int order; + + seq_printf(m, "MAX_ORDER: %d\n", MAX_ORDER); + seq_putc(m, '\n'); + + seq_printf(m, "%-10s", "Order:"); + for (order = 0; order < MAX_ORDER; ++order) + seq_printf(m, "%5d ", order); + seq_putc(m, '\n'); + + seq_printf(m, "%-10s", "Free:"); + for (order = 0; order < MAX_ORDER; ++order) + seq_printf(m, "%5lu ", free_area[order].nr_free); + seq_putc(m, '\n'); + + seq_printf(m, "%-10s", "Cached:"); + for (order = 0; order < MAX_ORDER; ++order) + seq_printf(m, "%5lu ", free_area[order].nr_cached); + seq_putc(m, '\n'); + + return 0; +} +DEFINE_SHOW_ATTRIBUTE(unmapped_alloc_debug); + +static int __init unmapped_alloc_init_late(void) +{ + debugfs_create_file("unmapped_alloc", 0444, NULL, + NULL, &unmapped_alloc_debug_fops); + return 0; +} +late_initcall(unmapped_alloc_init_late); From patchwork Wed Mar 8 09:41:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 66128 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp234512wrd; Wed, 8 Mar 2023 01:45:23 -0800 (PST) X-Google-Smtp-Source: AK7set+quzuk349VcK3wYsrL9Hk8iR6Nx19bKmAeiouf4xsQ7p9CiryG8ieJmSg/efMor1nTJ6lm X-Received: by 2002:a17:906:9451:b0:8e1:d996:dca2 with SMTP id z17-20020a170906945100b008e1d996dca2mr16163005ejx.64.1678268723777; Wed, 08 Mar 2023 01:45:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678268723; cv=none; d=google.com; s=arc-20160816; b=Y/dRMi0eusyJqCfBpH/PdgG9JKi7nXcvpfLWpSq7yK+tf7fQVVtfGA0d3MBcuHjfW8 7AGDPq4EwYXqQj2AlbOlxjV0q+/AQQsHjJVJfx5H1i3PIGtEbPDYOxLN4tq6XUG8nPEm j4+YPvgl0B1+W3kByUJdvnlt9+RyiWuC1TgkccBERUrauK4Li9JlFcuIghHrHdWx+5mw vF/rC59ExBmGtmQYp7fLD/rzkhebg/g/bREWCsYHA7PWK65Fa1kwkCeMs7dKqzEX5F74 ieDddial0wyTUfhzs02oIuTjsb2H+L8e0lZ7sFGJO71F6GqkV4wUlP8yImVZeo4kKsyq T/Lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=h8mhSnmMoDufRPSIQ3CwA06wj1H+ncDCjkrSqDXN2qo=; b=RnmrGT7w4Uxx1pgiUXAb4avcQ4tjPvOh7yMykEychwqQ3F0bPks/Azx8tKCgOHe3pU fLGerUEVfOjXcpXqQKvCv69S2GXB94wVM7E433msqZo2XE/EUC0iChcxyo3LQNU9g10B fy9bZXmE1ZCIY9tAcFplJFiQmo4B+jkzSeTjd7db4244YCgNx1XlkKc+JDkHD4trKnaf cNefuHv45O72fgDPB3zihoXSQFcIISNCR8s4TuHvRsIJNxQv74xMiv6w43oecDCczB25 z3Mm3d11G26nmFtnogBDFBoFRGjV0V3ri10JfSspfLCcqlPltDpR+G5EuNzdSzoGR8vq MdcQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=V6FAK2+t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ch10-20020a170906c2ca00b008c4de1a4c2asi4527404ejb.718.2023.03.08.01.44.49; Wed, 08 Mar 2023 01:45:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=V6FAK2+t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230131AbjCHJl5 (ORCPT + 99 others); Wed, 8 Mar 2023 04:41:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51972 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229565AbjCHJlj (ORCPT ); Wed, 8 Mar 2023 04:41:39 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3497B53E1 for ; Wed, 8 Mar 2023 01:41:34 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 72C626170A for ; Wed, 8 Mar 2023 09:41:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 78284C433A4; Wed, 8 Mar 2023 09:41:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1678268492; bh=8DLovPxnQy1OCrJZdJAEGK7t0D40yssgi5Mxz0RgWek=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=V6FAK2+t6gSHB91y9VkD/0zOfXhNUhq+sMc5AX4IQuRsdMaWCpYJCn1tHJeqYnOLY RWQ8R3BcCx3KjA7HQiX4FWRwiXskDus+nGuTU/h7Wao7lKZmWu32p94flaMQoKhkjv nL3jFqNz9nbN+CXPwNbIimjP38J1yfRIKOOiHu1l6hqqXu2wHBe6fikO0N0rwDw4rV jpSOfwYDrUpzaXm9lANUnmHiYWQ34T2LFwbVrf8z2Ilhgv3JEnwwcpcBlhfjLrjayl W8CcmajJxQHjkRZKa9pmK4E32ysqR4p5Fy+FD17Vl8xGoLMRnRfcR+TZrgX9jnjemr Axe7yl8DldgkA== From: Mike Rapoport To: linux-mm@kvack.org Cc: Andrew Morton , Dave Hansen , Mike Rapoport , Peter Zijlstra , Rick Edgecombe , Song Liu , Thomas Gleixner , Vlastimil Babka , linux-kernel@vger.kernel.org, x86@kernel.org Subject: [RFC PATCH 3/5] mm/unmapped_alloc: add shrinker Date: Wed, 8 Mar 2023 11:41:04 +0200 Message-Id: <20230308094106.227365-4-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230308094106.227365-1-rppt@kernel.org> References: <20230308094106.227365-1-rppt@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759792305424416393?= X-GMAIL-MSGID: =?utf-8?q?1759792305424416393?= From: "Mike Rapoport (IBM)" Allow shrinking unmapped caches when there is a memory pressure. Signed-off-by: Mike Rapoport(IBM) --- mm/internal.h | 2 ++ mm/page_alloc.c | 12 ++++++- mm/unmapped-alloc.c | 86 ++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 98 insertions(+), 2 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 8d84cceab467..aa2934594dcf 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1064,4 +1064,6 @@ static inline struct page *unmapped_pages_alloc(gfp_t gfp, int order) static inline void unmapped_pages_free(struct page *page, int order) {} #endif +void __free_unmapped_page(struct page *page, unsigned int order); + #endif /* __MM_INTERNAL_H */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 01f18e7529b0..ece213fac27a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -123,6 +123,11 @@ typedef int __bitwise fpi_t; */ #define FPI_SKIP_KASAN_POISON ((__force fpi_t)BIT(2)) +/* + * Free pages from the unmapped cache + */ +#define FPI_UNMAPPED ((__force fpi_t)BIT(3)) + /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ static DEFINE_MUTEX(pcp_batch_high_lock); #define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8) @@ -1467,7 +1472,7 @@ static __always_inline bool free_pages_prepare(struct page *page, PAGE_SIZE << order); } - if (get_pageblock_unmapped(page)) { + if (!(fpi_flags & FPI_UNMAPPED) && get_pageblock_unmapped(page)) { unmapped_pages_free(page, order); return false; } @@ -1636,6 +1641,11 @@ static void free_one_page(struct zone *zone, spin_unlock_irqrestore(&zone->lock, flags); } +void __free_unmapped_page(struct page *page, unsigned int order) +{ + __free_pages_ok(page, order, FPI_UNMAPPED | FPI_TO_TAIL); +} + static void __meminit __init_single_page(struct page *page, unsigned long pfn, unsigned long zone, int nid) { diff --git a/mm/unmapped-alloc.c b/mm/unmapped-alloc.c index f74640e9ce9f..89f54383df92 100644 --- a/mm/unmapped-alloc.c +++ b/mm/unmapped-alloc.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include @@ -16,6 +17,7 @@ struct unmapped_free_area { spinlock_t lock; unsigned long nr_free; unsigned long nr_cached; + unsigned long nr_released; }; static struct unmapped_free_area free_area[MAX_ORDER]; @@ -204,6 +206,82 @@ void unmapped_pages_free(struct page *page, int order) __free_one_page(page, order, false); } +static unsigned long unmapped_alloc_count_objects(struct shrinker *sh, + struct shrink_control *sc) +{ + unsigned long pages_to_free = 0; + + for (int order = 0; order < MAX_ORDER; order++) + pages_to_free += (free_area[order].nr_free << order); + + return pages_to_free; +} + +static unsigned long scan_free_area(struct shrink_control *sc, int order) +{ + struct unmapped_free_area *area = &free_area[order]; + unsigned long nr_pages = (1 << order); + unsigned long pages_freed = 0; + unsigned long flags; + struct page *page; + + spin_lock_irqsave(&area->lock, flags); + while (pages_freed < sc->nr_to_scan) { + + page = list_first_entry_or_null(&area->free_list, struct page, + lru); + if (!page) + break; + + del_page_from_free_list(page, order); + expand(page, order, order); + + area->nr_released++; + + if (order == pageblock_order) + clear_pageblock_unmapped(page); + + spin_unlock_irqrestore(&area->lock, flags); + + for (int i = 0; i < nr_pages; i++) + set_direct_map_default_noflush(page + i); + + __free_unmapped_page(page, order); + + pages_freed += nr_pages; + + cond_resched(); + spin_lock_irqsave(&area->lock, flags); + } + + spin_unlock_irqrestore(&area->lock, flags); + + return pages_freed; +} + +static unsigned long unmapped_alloc_scan_objects(struct shrinker *shrinker, + struct shrink_control *sc) +{ + sc->nr_scanned = 0; + + for (int order = 0; order < MAX_ORDER; order++) { + sc->nr_scanned += scan_free_area(sc, order); + + if (sc->nr_scanned >= sc->nr_to_scan) + break; + + sc->nr_to_scan -= sc->nr_scanned; + } + + return sc->nr_scanned ? sc->nr_scanned : SHRINK_STOP; +} + +static struct shrinker shrinker = { + .count_objects = unmapped_alloc_count_objects, + .scan_objects = unmapped_alloc_scan_objects, + .seeks = DEFAULT_SEEKS, +}; + int unmapped_alloc_init(void) { for (int order = 0; order < MAX_ORDER; order++) { @@ -237,6 +315,11 @@ static int unmapped_alloc_debug_show(struct seq_file *m, void *private) seq_printf(m, "%5lu ", free_area[order].nr_cached); seq_putc(m, '\n'); + seq_printf(m, "%-10s", "Released:"); + for (order = 0; order < MAX_ORDER; ++order) + seq_printf(m, "%5lu ", free_area[order].nr_released); + seq_putc(m, '\n'); + return 0; } DEFINE_SHOW_ATTRIBUTE(unmapped_alloc_debug); @@ -245,6 +328,7 @@ static int __init unmapped_alloc_init_late(void) { debugfs_create_file("unmapped_alloc", 0444, NULL, NULL, &unmapped_alloc_debug_fops); - return 0; + + return register_shrinker(&shrinker, "mm-unmapped-alloc"); } late_initcall(unmapped_alloc_init_late); From patchwork Wed Mar 8 09:41:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 66129 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp236193wrd; Wed, 8 Mar 2023 01:50:40 -0800 (PST) X-Google-Smtp-Source: AK7set9OIgfYk3KPZCTGo/F9NkuwoQ4ycYVSYsqhYXBRg95Gb2KLJAiwHcawTA52WUcvE97INC3P X-Received: by 2002:a17:907:c282:b0:884:930:b017 with SMTP id tk2-20020a170907c28200b008840930b017mr21583949ejc.60.1678269040732; Wed, 08 Mar 2023 01:50:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678269040; cv=none; d=google.com; s=arc-20160816; b=wYSGLbPQR1DaPYW9MNDp75uHM4tkPbxLB1K4j6X28lo3Qt0E5NdQkaaMN3puDFVeSJ zms7e/IEB60HURhFvX7QIUmUqAoa2qRmOVeId/ZFq6JCt7bRuVte6qrK9o9iGkhF0JqV pvPrYnDYIwwqAXUFnZ6dBYVR2XCwSTmUU1a+PzMzZzS8PrUBs+mRKAy4S6FLrh8M8Kgi 9NDgp/euI+62DPTZm+7xd80bCEhO8wI8zWHkeSodFBBUGUIXtIp4bMvYkEEiW7fKr1fu b+MwFRpmHbydHlXznf7XLrcKLj/H5lLFRtNKGgn9thHUV+FpQSkn0TQ/pTwbeVo3xhwd tcjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9oLvwGB5cLXtiQHKzRF/jATRlN7YLETOtgmzRmuPXRE=; b=Sg0fnIfOrm/LG+s2BJgk+PllfE4tLzGxhOG6QGXwWQ5XIZ/R4/9kJ1qUjuGpyyYPnl SkpkLG3/XN8EcJ2VtCQBsNzuJgC6WdWEkqwXq3OrNSq+2oh49JKTYWZuJaNqDk0tGrVE Ledeq/m+bcs3IilytiEI9Ljxgk5DEAt+7nYvZi8OBRIdh2+KbhaeHw0EJhQMR2qTpFNb acvJDN9aLfHQb4tyiy4Mjj/18+Ms47+VUYJTTZTgL/s0G2KogVRQAdu9YctKpPyNrDYl wvEwm4ngxOQ0MWhb4rXtwl7+GxJIU95G7WqwpmBGI7bRogX1qREpoykaNnTyzJIQ4KX1 D+BQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Pkmvdm5Y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d9-20020a170906304900b008d20a6a9d98si145473ejd.279.2023.03.08.01.50.17; Wed, 08 Mar 2023 01:50:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=Pkmvdm5Y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229986AbjCHJmO (ORCPT + 99 others); Wed, 8 Mar 2023 04:42:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52102 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229925AbjCHJlm (ORCPT ); Wed, 8 Mar 2023 04:41:42 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E0355B2571 for ; Wed, 8 Mar 2023 01:41:37 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 75F1761723 for ; Wed, 8 Mar 2023 09:41:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7AA97C433A0; Wed, 8 Mar 2023 09:41:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1678268496; bh=GyhXyClHrKgOJtAqFaxPxOscSnZByfdcCSJ4z9LZSNs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Pkmvdm5Ytxnt5h8jxKV6y+9spytNeNwCgY/lva+XKpCfpv2UDmFkQKXBEAPoZcPry ht6/BanBBGO/Ng+UwA4oJEes3D7Do1tSgu0fOyzsrsuFhVAaq8Fsf0yp5RspB2D/Lu lH3YM2i5lENWXxwTFZya+4DVuWQsaPy6gdGfUDbtqepQy68XZdfv9XAzN+HqlVtRDZ ekAyOV3eAfFITQHTYyjo73txVENSz2Zeqo+39fJfmKQc0sbRYnTO1sCtQWFCpyp6Em ZjhRFv88SpQ75C8j8pLGdjLo+iZuPCKs/sCi988FtUl7CBJI2C/MuBNLv8SxEFu5VZ OMu/uN9ykVk5w== From: Mike Rapoport To: linux-mm@kvack.org Cc: Andrew Morton , Dave Hansen , Mike Rapoport , Peter Zijlstra , Rick Edgecombe , Song Liu , Thomas Gleixner , Vlastimil Babka , linux-kernel@vger.kernel.org, x86@kernel.org Subject: [RFC PATCH 4/5] EXPERIMENTAL: x86: use __GFP_UNMAPPED for modele_alloc() Date: Wed, 8 Mar 2023 11:41:05 +0200 Message-Id: <20230308094106.227365-5-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230308094106.227365-1-rppt@kernel.org> References: <20230308094106.227365-1-rppt@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759792638086342519?= X-GMAIL-MSGID: =?utf-8?q?1759792638086342519?= From: "Mike Rapoport (IBM)" Signed-off-by: Mike Rapoport (IBM) --- arch/x86/kernel/module.c | 2 +- mm/vmalloc.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index 84ad0e61ba6e..845ed70ba5ab 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -67,7 +67,7 @@ static unsigned long int get_module_load_offset(void) void *module_alloc(unsigned long size) { - gfp_t gfp_mask = GFP_KERNEL; + gfp_t gfp_mask = GFP_KERNEL | __GFP_UNMAPPED; void *p; if (PAGE_ALIGN(size) > MODULES_LEN) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index ef910bf349e1..84220ec45ec2 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2892,7 +2892,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid, * to fails, fallback to a single page allocator that is * more permissive. */ - if (!order) { + if (!order && !(gfp & __GFP_UNMAPPED)) { gfp_t bulk_gfp = gfp & ~__GFP_NOFAIL; while (nr_allocated < nr_pages) { From patchwork Wed Mar 8 09:41:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 66130 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp236763wrd; Wed, 8 Mar 2023 01:52:48 -0800 (PST) X-Google-Smtp-Source: AK7set94kn6ix/n6/+ZI92muoX29zOTf7S17PDNy7eX8gQkP+Pb/UIsCDep/0k7k+ZO0pSjEGACq X-Received: by 2002:a17:907:8a0c:b0:87b:3d29:2982 with SMTP id sc12-20020a1709078a0c00b0087b3d292982mr22449060ejc.11.1678269168047; Wed, 08 Mar 2023 01:52:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678269168; cv=none; d=google.com; s=arc-20160816; b=goFk1QnZzrrSJ14pTTzp/RaLmaq2EtsYGL4gBfC+qP+EqKPblO5DiRpuM3JW9L9vWm QaQrHGjsRhRdYIC2BR5/Hgd4RPPPylajfqkLxslIFj4fcTKoXPadcqksVm4VGTMK8eHM iJAFrx4Kk99+ashJLzbxo/ZWgdOB3Fee/sjwe2Pu1gBjWirSJvPljyVP3noowX3V+M9a Yu0T2W60cx5T2I5am7oqmJpuwCEyK1HbHZ2jRKoNBp4EB1uEhB/CpQqmbIKWCmJ6Rczt 5mZUz12ZLcpfuH8Vn6z5BL/UxFdXIdDgQYu3tP85CEpizQ++M3P9xf+Avg2MErnnokW5 lhfg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=gMbdMo/pBqYIxP3tjUm5UhHWCZe8JSSf6Iu/9RrXzi8=; b=bsRSLO7fSSQgYZQK9maYd6pSzb+AvbDVADtNcyFl33S7bYzgO4YOooGKER85m8E666 A+LE+xAD3Yg46YYni0JlvVKa2Zh6rcOQUth4XXVV+N87Wit6NnFHSK8PzisOzHqrTO0K XHKeSgl2Fpi5cGB4CywxYPFk8D3gws/JQueSwRSqAiOB8gDiS2h1peo+wOHD7REmw8Ho BEHaNV01pFeQiVIxMs8OHfuI3vDvqJrzl10vZtDLMWIyqB+6MG6O8TOI9hxAemMxT36h 2icrOkAn6zlIarNk3jwd8LORqbwNRrZk1sfq/TzEn35n72d/iyFqmBXF8Z4KR5DfxrSC PaNQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=leVKbWeo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gz10-20020a170906f2ca00b009076178f532si11885978ejb.911.2023.03.08.01.52.24; Wed, 08 Mar 2023 01:52:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=leVKbWeo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230180AbjCHJmT (ORCPT + 99 others); Wed, 8 Mar 2023 04:42:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230171AbjCHJmA (ORCPT ); Wed, 8 Mar 2023 04:42:00 -0500 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9FA09B7185 for ; Wed, 8 Mar 2023 01:41:43 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 3B119B81C0B for ; Wed, 8 Mar 2023 09:41:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 80E84C433D2; Wed, 8 Mar 2023 09:41:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1678268500; bh=FOZyjqFesuEnEwpUW1WYq+F0fxn2qMQwF+XDs3hZBQk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=leVKbWeo/7qOftIstG5B9Wgf9MRXnwDGvaQUihQL6sWuanLYUQozOUwMx4mEYxlep SK+6GKBux58RzySSLw+ASPQ2LI0d6QdcBWHG4l2uuE9FqQyRWANesgRaYuCvprRtL2 994aEBWZuA/D8wD0+MLrSqO/8JSauc/E8A48GBC4UFCF0Mteg/IOFl9IH+v5CGnkmX 78RxsHHsDeqbtilOWnIuJihopZEwO+M4DIWXi6/YmWDg7iUB7fVn9xfYNiaJ5WWZ4p tvTaCpCyYZdmV2vKLa1pPMyu6s3qL5dgi4J4Ywy5j0orqUR0qA92pNM5IFhFjn/keq 6l9vE6tUdD9qw== From: Mike Rapoport To: linux-mm@kvack.org Cc: Andrew Morton , Dave Hansen , Mike Rapoport , Peter Zijlstra , Rick Edgecombe , Song Liu , Thomas Gleixner , Vlastimil Babka , linux-kernel@vger.kernel.org, x86@kernel.org Subject: [RFC PATCH 5/5] EXPERIMENTAL: mm/secretmem: use __GFP_UNMAPPED Date: Wed, 8 Mar 2023 11:41:06 +0200 Message-Id: <20230308094106.227365-6-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230308094106.227365-1-rppt@kernel.org> References: <20230308094106.227365-1-rppt@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759792771052504592?= X-GMAIL-MSGID: =?utf-8?q?1759792771052504592?= From: "Mike Rapoport (IBM)" Signed-off-by: Mike Rapoport (IBM) --- mm/secretmem.c | 26 +------------------------- 1 file changed, 1 insertion(+), 25 deletions(-) diff --git a/mm/secretmem.c b/mm/secretmem.c index 0b502625cd30..f66dfd16a0c3 100644 --- a/mm/secretmem.c +++ b/mm/secretmem.c @@ -53,7 +53,6 @@ static vm_fault_t secretmem_fault(struct vm_fault *vmf) struct inode *inode = file_inode(vmf->vma->vm_file); pgoff_t offset = vmf->pgoff; gfp_t gfp = vmf->gfp_mask; - unsigned long addr; struct page *page; vm_fault_t ret; int err; @@ -66,38 +65,22 @@ static vm_fault_t secretmem_fault(struct vm_fault *vmf) retry: page = find_lock_page(mapping, offset); if (!page) { - page = alloc_page(gfp | __GFP_ZERO); + page = alloc_page(gfp | __GFP_ZERO | __GFP_UNMAPPED); if (!page) { ret = VM_FAULT_OOM; goto out; } - err = set_direct_map_invalid_noflush(page); - if (err) { - put_page(page); - ret = vmf_error(err); - goto out; - } - __SetPageUptodate(page); err = add_to_page_cache_lru(page, mapping, offset, gfp); if (unlikely(err)) { put_page(page); - /* - * If a split of large page was required, it - * already happened when we marked the page invalid - * which guarantees that this call won't fail - */ - set_direct_map_default_noflush(page); if (err == -EEXIST) goto retry; ret = vmf_error(err); goto out; } - - addr = (unsigned long)page_address(page); - flush_tlb_kernel_range(addr, addr + PAGE_SIZE); } vmf->page = page; @@ -150,15 +133,8 @@ static int secretmem_migrate_folio(struct address_space *mapping, return -EBUSY; } -static void secretmem_free_folio(struct folio *folio) -{ - set_direct_map_default_noflush(&folio->page); - folio_zero_segment(folio, 0, folio_size(folio)); -} - const struct address_space_operations secretmem_aops = { .dirty_folio = noop_dirty_folio, - .free_folio = secretmem_free_folio, .migrate_folio = secretmem_migrate_folio, };