From patchwork Fri Nov 18 10:17:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 22253 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp105717wrr; Fri, 18 Nov 2022 02:19:42 -0800 (PST) X-Google-Smtp-Source: AA0mqf6y+DnndqvhB2JWYEZ9mwce1kP5BhEUA9tjwjwx5rxhaf8cbfTBtkviter45UJrB+FbMCLi X-Received: by 2002:a17:902:dc81:b0:186:59e9:20f6 with SMTP id n1-20020a170902dc8100b0018659e920f6mr7315470pld.39.1668766782247; Fri, 18 Nov 2022 02:19:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668766782; cv=none; d=google.com; s=arc-20160816; b=zsmB/JfEQtZVRhQdkvORDGlUPYpO/B9qGUfciI9i2xETqvSOGSul+RjSLfad2Z9yqd nFqtTTUDWc3QXQOyXahfdD5LWFB9fa+psPNPoCpz2A3a6KesJNFd/CMY41D1oj53KDCS M3BFZxWtNFjti99JAJR0Sg6qojT61AnrkdawAbA3NEoYn+m8GUYTWaHIaOjEG0M7S1fT yvWjzZpoVgdJ64sgfuVRjqZHlupCr+AMwlc6puWTCNH1rTsHS2KRqELEn5KZBacBVQLR aCADYAmjnz9mkJix3MZ0eHZ1bIsmIy8Yo1iPIJvRO7wPyEs0UyjZGKIZ6pXIUtl1C7BE /UmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=KdSm7BloM6XAV0NZVdWTK0GO7g+qUM1tjh6RpcjUiXI=; b=CM5iOol2JjGz8SIxAZSP+PiA8Wk50ieRTq9I6/zeBZtDM6cQPv91C+11h1YE8HBJV6 QLrsPgatXR7pzz7vkNfxTvsQd20itHpDOdRuh+WclEshEtE6WUznk6XfvoDLcnnMBuyV ZfyxPLry5hw/u82h1rcHF/9b4sz8sTZ+UFeIwD4JsLEahfr7lrLUKsIygVWjteSX3agr qQCsZbUl9Z6H6qpun9kWYBYPHo67JPhXeh/euzDnvlS5jiFKR8k7agmUqpea+8Ho08yI /BBauWEnNgSyurq4OFK6fQkCg7l3FZwnQtXFX1pGiuVewApefTBOvUYFjc6o7XdAYiG1 sK5g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id me2-20020a17090b17c200b00213f8a9fd79si8493154pjb.130.2022.11.18.02.19.29; Fri, 18 Nov 2022 02:19:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241383AbiKRKRl (ORCPT + 99 others); Fri, 18 Nov 2022 05:17:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46046 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241435AbiKRKRi (ORCPT ); Fri, 18 Nov 2022 05:17:38 -0500 Received: from outbound-smtp29.blacknight.com (outbound-smtp29.blacknight.com [81.17.249.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 73A846153F for ; Fri, 18 Nov 2022 02:17:36 -0800 (PST) Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp29.blacknight.com (Postfix) with ESMTPS id 2883ABEC19 for ; Fri, 18 Nov 2022 10:17:35 +0000 (GMT) Received: (qmail 3723 invoked from network); 18 Nov 2022 10:17:35 -0000 Received: from unknown (HELO morpheus.112glenside.lan) (mgorman@techsingularity.net@[84.203.198.246]) by 81.17.254.9 with ESMTPA; 18 Nov 2022 10:17:34 -0000 From: Mel Gorman To: Andrew Morton Cc: Hugh Dickins , Yu Zhao , Vlastimil Babka , Marcelo Tosatti , Michal Hocko , Marek Szyprowski , LKML , Linux-MM , Mel Gorman Subject: [PATCH 1/2] mm/page_alloc: Always remove pages from temporary list Date: Fri, 18 Nov 2022 10:17:13 +0000 Message-Id: <20221118101714.19590-2-mgorman@techsingularity.net> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221118101714.19590-1-mgorman@techsingularity.net> References: <20221118101714.19590-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749828797302745957?= X-GMAIL-MSGID: =?utf-8?q?1749828797302745957?= free_unref_page_list() has neglected to remove pages properly from the list of pages to free since forever. It works by coincidence because list_add happened to do the right thing adding the pages to just the PCP lists. However, a later patch added pages to either the PCP list or the zone list but only properly deleted the page from the list in one path leading to list corruption and a subsequent failure. As a preparation patch, always delete the pages from one list properly before adding to another. On its own, this fixes nothing although it adds a fractional amount of overhead but is critical to the next patch. Reported-by: Hugh Dickins Signed-off-by: Mel Gorman Reviewed-by: Vlastimil Babka --- mm/page_alloc.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 218b28ee49ed..1ec54173b8d4 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3546,6 +3546,8 @@ void free_unref_page_list(struct list_head *list) list_for_each_entry_safe(page, next, list, lru) { struct zone *zone = page_zone(page); + list_del(&page->lru); + /* Different zone, different pcp lock. */ if (zone != locked_zone) { if (pcp) From patchwork Fri Nov 18 10:17:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 22254 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp106329wrr; Fri, 18 Nov 2022 02:21:25 -0800 (PST) X-Google-Smtp-Source: AA0mqf5b269HAEvz73P6ov584lcZ+yod03jDy0IItKvRZ5eFNrnhbo8mb4AkFa9z5HE3+HVx11uU X-Received: by 2002:aa7:9ae4:0:b0:572:725f:33e6 with SMTP id y4-20020aa79ae4000000b00572725f33e6mr7132396pfp.46.1668766884978; Fri, 18 Nov 2022 02:21:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668766884; cv=none; d=google.com; s=arc-20160816; b=sAXeZwmpTt6pZc8yrBGXGudouibCAnJnMDSRyO5/gVOl/selXkRK0Oss2AdZdWvb33 CuS8ahOShauicrsyfR2PLw/HnnlztLR/WL2vJxV8QKzUIOHz0KJk7t4c/1JoGrBNcQnw aUGxnByefMVrnh8pZgtrUy3YVeF7sB9xcU8/zlkt4i74a1aUakyVh5E4iKagckbAENpk CVFsS1uEeWVyU/bQRnrk/OX6OSI/LlAGJuBHkGqV5OEqfSHlLuTaiKInj+0mJScJ5IPs tEnezxBaR2BuBo2NKYH6idKeZJki5H6j3uHnUYlFp7GGjv75epz4MEewS+z/q0o7kvyA 2Fcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=LXjn7HDDSK3IvRMrp/g0+gnaY9PS3TTEkjgvY+0VDq4=; b=FvOxXlXpWg5xZgWYi0H17t3ey5Rk4NO5l1KXx7gmk0n4wRSCnpLmeyJiBVp/0lvq3h GYO67XAkaX3V6thpRi+QDocB6U6b/0z3xsPJAitg//BqgHcYYaBxTOy3GxQEcQyOkT6H /uY9eBDXDrJd6Uzs9ISKBwMX0GXxGuEjP/4ao85+1P8p11C2fBjsYCFWusq/5s4s67oF eMxcePuaYR0gXyGo68Z8ZYaiTaxyG8NX6YbB1/mJbI3oWzK6/Tn2NT0LBq/PRMVtBngv f+ZsoM3gBL97FZTziT8US6zG9UGe5kDpNP6RAVrdILXFYCU30lCeVDgfht8fBMVDLuQf WgRg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s23-20020a170902b19700b00186a06a3396si3018220plr.153.2022.11.18.02.21.11; Fri, 18 Nov 2022 02:21:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241478AbiKRKRz (ORCPT + 99 others); Fri, 18 Nov 2022 05:17:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241526AbiKRKRt (ORCPT ); Fri, 18 Nov 2022 05:17:49 -0500 Received: from outbound-smtp19.blacknight.com (outbound-smtp19.blacknight.com [46.22.139.246]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 95D7C9038B for ; Fri, 18 Nov 2022 02:17:46 -0800 (PST) Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp19.blacknight.com (Postfix) with ESMTPS id 4D0D51C456C for ; Fri, 18 Nov 2022 10:17:45 +0000 (GMT) Received: (qmail 4454 invoked from network); 18 Nov 2022 10:17:45 -0000 Received: from unknown (HELO morpheus.112glenside.lan) (mgorman@techsingularity.net@[84.203.198.246]) by 81.17.254.9 with ESMTPA; 18 Nov 2022 10:17:45 -0000 From: Mel Gorman To: Andrew Morton Cc: Hugh Dickins , Yu Zhao , Vlastimil Babka , Marcelo Tosatti , Michal Hocko , Marek Szyprowski , LKML , Linux-MM , Mel Gorman Subject: [PATCH 2/2] mm/page_alloc: Leave IRQs enabled for per-cpu page allocations Date: Fri, 18 Nov 2022 10:17:14 +0000 Message-Id: <20221118101714.19590-3-mgorman@techsingularity.net> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221118101714.19590-1-mgorman@techsingularity.net> References: <20221118101714.19590-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_LOW, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749828905185246973?= X-GMAIL-MSGID: =?utf-8?q?1749828905185246973?= The pcp_spin_lock_irqsave protecting the PCP lists is IRQ-safe as a task allocating from the PCP must not re-enter the allocator from IRQ context. In each instance where IRQ-reentrancy is possible, the lock is acquired using pcp_spin_trylock_irqsave() even though IRQs are disabled and re-entrancy is impossible. Demote the lock to pcp_spin_lock avoids an IRQ disable/enable in the common case at the cost of some IRQ allocations taking a slower path. If the PCP lists need to be refilled, the zone lock still needs to disable IRQs but that will only happen on PCP refill and drain. If an IRQ is raised when a PCP allocation is in progress, the trylock will fail and fallback to using the buddy lists directly. Note that this may not be a universal win if an interrupt-intensive workload also allocates heavily from interrupt context and contends heavily on the zone->lock as a result. [yuzhao@google.com: Reported lockdep issue on IO completion from softirq] [hughd@google.com: Fix list corruption, lock improvements, micro-optimsations] Signed-off-by: Mel Gorman Reviewed-by: Vlastimil Babka Reported-by: Vlastimil Babka Signed-off-by: Mel Gorman Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka Reviewed-by: Vlastimil Babka --- mm/page_alloc.c | 122 +++++++++++++++++++++--------------------------- 1 file changed, 53 insertions(+), 69 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 1ec54173b8d4..323fec05c4c6 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -170,21 +170,12 @@ static DEFINE_MUTEX(pcp_batch_high_lock); _ret; \ }) -#define pcpu_spin_lock_irqsave(type, member, ptr, flags) \ +#define pcpu_spin_trylock(type, member, ptr) \ ({ \ type *_ret; \ pcpu_task_pin(); \ _ret = this_cpu_ptr(ptr); \ - spin_lock_irqsave(&_ret->member, flags); \ - _ret; \ -}) - -#define pcpu_spin_trylock_irqsave(type, member, ptr, flags) \ -({ \ - type *_ret; \ - pcpu_task_pin(); \ - _ret = this_cpu_ptr(ptr); \ - if (!spin_trylock_irqsave(&_ret->member, flags)) { \ + if (!spin_trylock(&_ret->member)) { \ pcpu_task_unpin(); \ _ret = NULL; \ } \ @@ -197,27 +188,16 @@ static DEFINE_MUTEX(pcp_batch_high_lock); pcpu_task_unpin(); \ }) -#define pcpu_spin_unlock_irqrestore(member, ptr, flags) \ -({ \ - spin_unlock_irqrestore(&ptr->member, flags); \ - pcpu_task_unpin(); \ -}) - /* struct per_cpu_pages specific helpers. */ #define pcp_spin_lock(ptr) \ pcpu_spin_lock(struct per_cpu_pages, lock, ptr) -#define pcp_spin_lock_irqsave(ptr, flags) \ - pcpu_spin_lock_irqsave(struct per_cpu_pages, lock, ptr, flags) - -#define pcp_spin_trylock_irqsave(ptr, flags) \ - pcpu_spin_trylock_irqsave(struct per_cpu_pages, lock, ptr, flags) +#define pcp_spin_trylock(ptr) \ + pcpu_spin_trylock(struct per_cpu_pages, lock, ptr) #define pcp_spin_unlock(ptr) \ pcpu_spin_unlock(lock, ptr) -#define pcp_spin_unlock_irqrestore(ptr, flags) \ - pcpu_spin_unlock_irqrestore(lock, ptr, flags) #ifdef CONFIG_USE_PERCPU_NUMA_NODE_ID DEFINE_PER_CPU(int, numa_node); EXPORT_PER_CPU_SYMBOL(numa_node); @@ -1547,6 +1527,7 @@ static void free_pcppages_bulk(struct zone *zone, int count, struct per_cpu_pages *pcp, int pindex) { + unsigned long flags; int min_pindex = 0; int max_pindex = NR_PCP_LISTS - 1; unsigned int order; @@ -1562,8 +1543,7 @@ static void free_pcppages_bulk(struct zone *zone, int count, /* Ensure requested pindex is drained first. */ pindex = pindex - 1; - /* Caller must hold IRQ-safe pcp->lock so IRQs are disabled. */ - spin_lock(&zone->lock); + spin_lock_irqsave(&zone->lock, flags); isolated_pageblocks = has_isolate_pageblock(zone); while (count > 0) { @@ -1611,7 +1591,7 @@ static void free_pcppages_bulk(struct zone *zone, int count, } while (count > 0 && !list_empty(list)); } - spin_unlock(&zone->lock); + spin_unlock_irqrestore(&zone->lock, flags); } static void free_one_page(struct zone *zone, @@ -3125,10 +3105,10 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, unsigned long count, struct list_head *list, int migratetype, unsigned int alloc_flags) { + unsigned long flags; int i, allocated = 0; - /* Caller must hold IRQ-safe pcp->lock so IRQs are disabled. */ - spin_lock(&zone->lock); + spin_lock_irqsave(&zone->lock, flags); for (i = 0; i < count; ++i) { struct page *page = __rmqueue(zone, order, migratetype, alloc_flags); @@ -3162,7 +3142,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, * pages added to the pcp list. */ __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order)); - spin_unlock(&zone->lock); + spin_unlock_irqrestore(&zone->lock, flags); return allocated; } @@ -3179,16 +3159,9 @@ void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp) batch = READ_ONCE(pcp->batch); to_drain = min(pcp->count, batch); if (to_drain > 0) { - unsigned long flags; - - /* - * free_pcppages_bulk expects IRQs disabled for zone->lock - * so even though pcp->lock is not intended to be IRQ-safe, - * it's needed in this context. - */ - spin_lock_irqsave(&pcp->lock, flags); + spin_lock(&pcp->lock); free_pcppages_bulk(zone, to_drain, pcp, 0); - spin_unlock_irqrestore(&pcp->lock, flags); + spin_unlock(&pcp->lock); } } #endif @@ -3202,12 +3175,9 @@ static void drain_pages_zone(unsigned int cpu, struct zone *zone) pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu); if (pcp->count) { - unsigned long flags; - - /* See drain_zone_pages on why this is disabling IRQs */ - spin_lock_irqsave(&pcp->lock, flags); + spin_lock(&pcp->lock); free_pcppages_bulk(zone, pcp->count, pcp, 0); - spin_unlock_irqrestore(&pcp->lock, flags); + spin_unlock(&pcp->lock); } } @@ -3473,7 +3443,6 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, */ void free_unref_page(struct page *page, unsigned int order) { - unsigned long flags; unsigned long __maybe_unused UP_flags; struct per_cpu_pages *pcp; struct zone *zone; @@ -3501,10 +3470,10 @@ void free_unref_page(struct page *page, unsigned int order) zone = page_zone(page); pcp_trylock_prepare(UP_flags); - pcp = pcp_spin_trylock_irqsave(zone->per_cpu_pageset, flags); + pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (pcp) { free_unref_page_commit(zone, pcp, page, migratetype, order); - pcp_spin_unlock_irqrestore(pcp, flags); + pcp_spin_unlock(pcp); } else { free_one_page(zone, page, pfn, order, migratetype, FPI_NONE); } @@ -3516,10 +3485,10 @@ void free_unref_page(struct page *page, unsigned int order) */ void free_unref_page_list(struct list_head *list) { + unsigned long __maybe_unused UP_flags; struct page *page, *next; struct per_cpu_pages *pcp = NULL; struct zone *locked_zone = NULL; - unsigned long flags; int batch_count = 0; int migratetype; @@ -3550,11 +3519,26 @@ void free_unref_page_list(struct list_head *list) /* Different zone, different pcp lock. */ if (zone != locked_zone) { - if (pcp) - pcp_spin_unlock_irqrestore(pcp, flags); + if (pcp) { + pcp_spin_unlock(pcp); + pcp_trylock_finish(UP_flags); + } + /* + * trylock is necessary as pages may be getting freed + * from IRQ or SoftIRQ context after an IO completion. + */ + pcp_trylock_prepare(UP_flags); + pcp = pcp_spin_trylock(zone->per_cpu_pageset); + if (!pcp) { + pcp_trylock_finish(UP_flags); + free_one_page(zone, page, page_to_pfn(page), + 0, migratetype, FPI_NONE); + locked_zone = NULL; + continue; + } locked_zone = zone; - pcp = pcp_spin_lock_irqsave(locked_zone->per_cpu_pageset, flags); + batch_count = 0; } /* @@ -3569,18 +3553,23 @@ void free_unref_page_list(struct list_head *list) free_unref_page_commit(zone, pcp, page, migratetype, 0); /* - * Guard against excessive IRQ disabled times when we get - * a large list of pages to free. + * Guard against excessive lock hold times when freeing + * a large list of pages. Lock will be reacquired if + * necessary on the next iteration. */ if (++batch_count == SWAP_CLUSTER_MAX) { - pcp_spin_unlock_irqrestore(pcp, flags); + pcp_spin_unlock(pcp); + pcp_trylock_finish(UP_flags); batch_count = 0; - pcp = pcp_spin_lock_irqsave(locked_zone->per_cpu_pageset, flags); + pcp = NULL; + locked_zone = NULL; } } - if (pcp) - pcp_spin_unlock_irqrestore(pcp, flags); + if (pcp) { + pcp_spin_unlock(pcp); + pcp_trylock_finish(UP_flags); + } } /* @@ -3781,15 +3770,11 @@ static struct page *rmqueue_pcplist(struct zone *preferred_zone, struct per_cpu_pages *pcp; struct list_head *list; struct page *page; - unsigned long flags; unsigned long __maybe_unused UP_flags; - /* - * spin_trylock may fail due to a parallel drain. In the future, the - * trylock will also protect against IRQ reentrancy. - */ + /* spin_trylock may fail due to a parallel drain or IRQ reentrancy. */ pcp_trylock_prepare(UP_flags); - pcp = pcp_spin_trylock_irqsave(zone->per_cpu_pageset, flags); + pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (!pcp) { pcp_trylock_finish(UP_flags); return NULL; @@ -3803,7 +3788,7 @@ static struct page *rmqueue_pcplist(struct zone *preferred_zone, pcp->free_factor >>= 1; list = &pcp->lists[order_to_pindex(migratetype, order)]; page = __rmqueue_pcplist(zone, order, migratetype, alloc_flags, pcp, list); - pcp_spin_unlock_irqrestore(pcp, flags); + pcp_spin_unlock(pcp); pcp_trylock_finish(UP_flags); if (page) { __count_zid_vm_events(PGALLOC, page_zonenum(page), 1 << order); @@ -5371,7 +5356,6 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, struct page **page_array) { struct page *page; - unsigned long flags; unsigned long __maybe_unused UP_flags; struct zone *zone; struct zoneref *z; @@ -5453,9 +5437,9 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, if (unlikely(!zone)) goto failed; - /* Is a parallel drain in progress? */ + /* spin_trylock may fail due to a parallel drain or IRQ reentrancy. */ pcp_trylock_prepare(UP_flags); - pcp = pcp_spin_trylock_irqsave(zone->per_cpu_pageset, flags); + pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (!pcp) goto failed_irq; @@ -5474,7 +5458,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, if (unlikely(!page)) { /* Try and allocate at least one page */ if (!nr_account) { - pcp_spin_unlock_irqrestore(pcp, flags); + pcp_spin_unlock(pcp); goto failed_irq; } break; @@ -5489,7 +5473,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, nr_populated++; } - pcp_spin_unlock_irqrestore(pcp, flags); + pcp_spin_unlock(pcp); pcp_trylock_finish(UP_flags); __count_zid_vm_events(PGALLOC, zone_idx(zone), nr_account);