From patchwork Mon Jun 19 11:07:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 109925 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2936921vqr; Mon, 19 Jun 2023 04:27:51 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5lp14XynB8fFo2ToGmYfGgWQZVcijddo/dw3D5QeF8WN7ftmhW5iloUdWRVw0qeNmrasdo X-Received: by 2002:a05:6808:302c:b0:39a:a4ab:22f2 with SMTP id ay44-20020a056808302c00b0039aa4ab22f2mr10720596oib.56.1687174071620; Mon, 19 Jun 2023 04:27:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687174071; cv=none; d=google.com; s=arc-20160816; b=KQCF1dsBb7SmDIzwPrF7eCvqaVFWcfugp7UIhj+08YnOHNxCAD6mukxwl+fOfrVfeN ATi+fndM/u9HYWhRVVYqPS9ZQ7aFpzVUpluTpQC1KSz4sAR/fyPXlh6YU4oJMx031mqd FPr7jzAi1YvD0s+NLvWwXTUARpXiIJRLGJD3kMgUMCz6pm32E/QxfZCPFEF+PSOXnpaQ srl3l802wrzF2sAZK/LIEa1YdmbGm7qMmk3E4kV3GOk7Pd4z8/cGNlijcMEe8cALcxfx 0UFPyc0tab16U4U1anW6kHd64h+dp9TsUSNCRbbf1pymZMamhfUdII/VaqpatTlRQ1eB uEsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from; bh=D/iy/3iUmKUNJSTSg9So9IxsM+qD2mx3VX+zehrMy0M=; b=aUuZLCqKe2HfxG2OCQw1MPuO3iLS2XOBvePt4Od76NTfUIcxIYsDpQMwkkKQsGhyP2 tMijXpQO0RQc3ThljauAPjaUDO1/f64i6gMkBQatUAbSrhqnJKbiy6y7FaMpsKd88mCe WkuDXlm+CUZPC+p+RznVtcOL2Zufj9HtQQqULaDXTDFEjJH0uwjT/cdbwkacIwW80Upa /6o2ZKonYtkCgWkeIE0AVaXO7SA8Pp6MgQcn2ZDHv4ug4nzRrzTkN0D2BwE9isAt+IUN tbO3My4ACNh0hM7XpYkVRFt1nnM91g5NPYjOuEja+nVeVl7T9SSsEXi6PWZ4lv39VnCi ZqNQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u20-20020a63b554000000b005346b8dae84si7247490pgo.787.2023.06.19.04.27.37; Mon, 19 Jun 2023 04:27:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232075AbjFSKyi (ORCPT + 99 others); Mon, 19 Jun 2023 06:54:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56824 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230477AbjFSKyP (ORCPT ); Mon, 19 Jun 2023 06:54:15 -0400 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF06F10C8 for ; Mon, 19 Jun 2023 03:52:47 -0700 (PDT) Received: from dggpemm500001.china.huawei.com (unknown [172.30.72.56]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4Ql66c1rRYzqThG; Mon, 19 Jun 2023 18:52:40 +0800 (CST) Received: from localhost.localdomain.localdomain (10.175.113.25) by dggpemm500001.china.huawei.com (7.185.36.107) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Mon, 19 Jun 2023 18:52:45 +0800 From: Kefeng Wang To: CC: , , , , , Kefeng Wang Subject: [PATCH -next v2 1/2] mm: compaction: convert to use a folio in isolate_migratepages_block() Date: Mon, 19 Jun 2023 19:07:17 +0800 Message-ID: <20230619110718.65679-1-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 X-Originating-IP: [10.175.113.25] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To dggpemm500001.china.huawei.com (7.185.36.107) X-CFilter-Loop: Reflected X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1769130238986224951?= X-GMAIL-MSGID: =?utf-8?q?1769130238986224951?= Directly use a folio instead of page_folio() when page successfully isolated (hugepage and movable page) and after folio_get_nontail_page(), which removes several calls to compound_head(). Signed-off-by: Kefeng Wang Reviewed-by: Baolin Wang --- v2: - update comments and use node_stat_mod_folio, per Matthew Wilcox - add missed PageLRU conversion and rebase on next-20230619 mm/compaction.c | 84 ++++++++++++++++++++++++++----------------------- 1 file changed, 44 insertions(+), 40 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 6149a2d324be..0334eefe4bfa 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -795,6 +795,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, struct lruvec *lruvec; unsigned long flags = 0; struct lruvec *locked = NULL; + struct folio *folio = NULL; struct page *page = NULL, *valid_page = NULL; struct address_space *mapping; unsigned long start_pfn = low_pfn; @@ -891,7 +892,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if (!valid_page && pageblock_aligned(low_pfn)) { if (!isolation_suitable(cc, page)) { low_pfn = end_pfn; - page = NULL; + folio = NULL; goto isolate_abort; } valid_page = page; @@ -923,7 +924,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, * Hugepage was successfully isolated and placed * on the cc->migratepages list. */ - low_pfn += compound_nr(page) - 1; + folio = page_folio(page); + low_pfn += folio_nr_pages(folio) - 1; goto isolate_success_no_list; } @@ -991,8 +993,10 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, locked = NULL; } - if (isolate_movable_page(page, mode)) + if (isolate_movable_page(page, mode)) { + folio = page_folio(page); goto isolate_success; + } } goto isolate_fail; @@ -1003,7 +1007,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, * sure the page is not being freed elsewhere -- the * page release code relies on it. */ - if (unlikely(!get_page_unless_zero(page))) + folio = folio_get_nontail_page(page); + if (unlikely(!folio)) goto isolate_fail; /* @@ -1011,8 +1016,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, * so avoid taking lru_lock and isolating it unnecessarily in an * admittedly racy check. */ - mapping = page_mapping(page); - if (!mapping && (page_count(page) - 1) > total_mapcount(page)) + mapping = folio_mapping(folio); + if (!mapping && (folio_ref_count(folio) - 1) > folio_mapcount(folio)) goto isolate_fail_put; /* @@ -1023,11 +1028,11 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, goto isolate_fail_put; /* Only take pages on LRU: a check now makes later tests safe */ - if (!PageLRU(page)) + if (!folio_test_lru(folio)) goto isolate_fail_put; /* Compaction might skip unevictable pages but CMA takes them */ - if (!(mode & ISOLATE_UNEVICTABLE) && PageUnevictable(page)) + if (!(mode & ISOLATE_UNEVICTABLE) && folio_test_unevictable(folio)) goto isolate_fail_put; /* @@ -1036,10 +1041,10 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, * it will be able to migrate without blocking - clean pages * for the most part. PageWriteback would require blocking. */ - if ((mode & ISOLATE_ASYNC_MIGRATE) && PageWriteback(page)) + if ((mode & ISOLATE_ASYNC_MIGRATE) && folio_test_writeback(folio)) goto isolate_fail_put; - if ((mode & ISOLATE_ASYNC_MIGRATE) && PageDirty(page)) { + if ((mode & ISOLATE_ASYNC_MIGRATE) && folio_test_dirty(folio)) { bool migrate_dirty; /* @@ -1051,22 +1056,22 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, * the page lock until after the page is removed * from the page cache. */ - if (!trylock_page(page)) + if (!folio_trylock(folio)) goto isolate_fail_put; - mapping = page_mapping(page); + mapping = folio_mapping(folio); migrate_dirty = !mapping || mapping->a_ops->migrate_folio; - unlock_page(page); + folio_unlock(folio); if (!migrate_dirty) goto isolate_fail_put; } - /* Try isolate the page */ - if (!TestClearPageLRU(page)) + /* Try isolate the folio */ + if (!folio_test_clear_lru(folio)) goto isolate_fail_put; - lruvec = folio_lruvec(page_folio(page)); + lruvec = folio_lruvec(folio); /* If we already hold the lock, we can skip some rechecking */ if (lruvec != locked) { @@ -1076,7 +1081,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, compact_lock_irqsave(&lruvec->lru_lock, &flags, cc); locked = lruvec; - lruvec_memcg_debug(lruvec, page_folio(page)); + lruvec_memcg_debug(lruvec, folio); /* * Try get exclusive access under lock. If marked for @@ -1092,34 +1097,33 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, } /* - * Page become compound since the non-locked check, - * and it's on LRU. It can only be a THP so the order - * is safe to read and it's 0 for tail pages. + * folio become large since the non-locked check, + * and it's on LRU. */ - if (unlikely(PageCompound(page) && !cc->alloc_contig)) { - low_pfn += compound_nr(page) - 1; - nr_scanned += compound_nr(page) - 1; - SetPageLRU(page); + if (unlikely(folio_test_large(folio) && !cc->alloc_contig)) { + low_pfn += folio_nr_pages(folio) - 1; + nr_scanned += folio_nr_pages(folio) - 1; + folio_set_lru(folio); goto isolate_fail_put; } } - /* The whole page is taken off the LRU; skip the tail pages. */ - if (PageCompound(page)) - low_pfn += compound_nr(page) - 1; + /* The folio is taken off the LRU */ + if (folio_test_large(folio)) + low_pfn += folio_nr_pages(folio) - 1; /* Successfully isolated */ - del_page_from_lru_list(page, lruvec); - mod_node_page_state(page_pgdat(page), - NR_ISOLATED_ANON + page_is_file_lru(page), - thp_nr_pages(page)); + lruvec_del_folio(lruvec, folio); + node_stat_mod_folio(folio, + NR_ISOLATED_ANON + folio_is_file_lru(folio), + folio_nr_pages(folio)); isolate_success: - list_add(&page->lru, &cc->migratepages); + list_add(&folio->lru, &cc->migratepages); isolate_success_no_list: - cc->nr_migratepages += compound_nr(page); - nr_isolated += compound_nr(page); - nr_scanned += compound_nr(page) - 1; + cc->nr_migratepages += folio_nr_pages(folio); + nr_isolated += folio_nr_pages(folio); + nr_scanned += folio_nr_pages(folio) - 1; /* * Avoid isolating too much unless this block is being @@ -1141,7 +1145,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, unlock_page_lruvec_irqrestore(locked, flags); locked = NULL; } - put_page(page); + folio_put(folio); isolate_fail: if (!skip_on_failure && ret != -ENOMEM) @@ -1182,14 +1186,14 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if (unlikely(low_pfn > end_pfn)) low_pfn = end_pfn; - page = NULL; + folio = NULL; isolate_abort: if (locked) unlock_page_lruvec_irqrestore(locked, flags); - if (page) { - SetPageLRU(page); - put_page(page); + if (folio) { + folio_set_lru(folio); + folio_put(folio); } /* From patchwork Mon Jun 19 11:07:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 109928 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2937853vqr; Mon, 19 Jun 2023 04:29:26 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5CysKQXsxgfP31WN30zNkmOaBcz2jKObr4OKMgCLKLNSGo+U0HnxctEyLUN1OvCvBQ3SlN X-Received: by 2002:a05:6808:641:b0:39c:7a71:945 with SMTP id z1-20020a056808064100b0039c7a710945mr10080153oih.56.1687174166276; Mon, 19 Jun 2023 04:29:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687174166; cv=none; d=google.com; s=arc-20160816; b=NiCKU12bkGHDL/NxHCo/SHvQ33Z0VbyFXfObRUXRiuPJuCgQPoh0Q+ZTn62Ocd15/r nR3yO6AZhhR9GOGG5OWdjUCYOZkty3LiCk8OrJDt1bH644/NFi9U0fbOywGn5u4FgESu ziLfeRvezNmNzqxLAvFKB7BeT7Vbm7x60n8xlPLtjGZdzdVLQIsrtapBnwIkTh1FRsxv iDBc6atvbGxo8WRUAapsfwu6SWuGkPfOvqIWQba2GlqTVNB3ZW526VhEVLY9Y+9BGGuM bOqS/GW7I7yLosvZDmpRL7Tkkdw3xWwAzvwqdh2gH8BnwMZ4vgGVGfotPKV1yiko4CNI Jisw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=kSeVltLVwL4S4NL52guaL9XBMWx3z7eLOSLgWslw1ZM=; b=qAY2SwaYaaEX41zuJkhTGErKqtk85hgPDq2ZERwF2NSPREkosnAG5I8m2NedCSWmil 3VYDuOd2P27GYaCecOCiaDGaeOM+ZQayhfAVxdmqLzw/I/OOQjy8Jf/fcrmTRulNtoBH vOMMy/EWFXkVq8PB9mC2yVb4yPy5aTpVnaxheq62IY2A+63vHTmpliY1BNar3V2UL6BM hgj3GaY0as2evFTGwS0c8Apnpbf0qt4Wh5XmZdbYv61Gudl/27tvDtixoqDx0JOH9RsG 1DwSnufggTJ9OPBLUh8fVY/mnl6Abjla0tI4lIZdBEIA8V3ozrC7M775A4OnTUK7tQsb s0mQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r12-20020a170902c60c00b001b51c68c828si7917200plr.211.2023.06.19.04.29.12; Mon, 19 Jun 2023 04:29:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232338AbjFSKyf (ORCPT + 99 others); Mon, 19 Jun 2023 06:54:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56802 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232295AbjFSKyO (ORCPT ); Mon, 19 Jun 2023 06:54:14 -0400 Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BD0F51723 for ; Mon, 19 Jun 2023 03:52:47 -0700 (PDT) Received: from dggpemm500001.china.huawei.com (unknown [172.30.72.55]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4Ql6364fZ0zMp0X; Mon, 19 Jun 2023 18:49:38 +0800 (CST) Received: from localhost.localdomain.localdomain (10.175.113.25) by dggpemm500001.china.huawei.com (7.185.36.107) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Mon, 19 Jun 2023 18:52:45 +0800 From: Kefeng Wang To: CC: , , , , , Kefeng Wang Subject: [PATCH -next v2 2/2] mm: kill [add|del]_page_to_lru_list() Date: Mon, 19 Jun 2023 19:07:18 +0800 Message-ID: <20230619110718.65679-2-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230619110718.65679-1-wangkefeng.wang@huawei.com> References: <20230619110718.65679-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.113.25] X-ClientProxiedBy: dggems703-chm.china.huawei.com (10.3.19.180) To dggpemm500001.china.huawei.com (7.185.36.107) X-CFilter-Loop: Reflected X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1769130338649246892?= X-GMAIL-MSGID: =?utf-8?q?1769130338649246892?= Now no one call [add|del]_page_to_lru_list(), let's drop unused page interfaces. Acked-by: Yu Zhao Signed-off-by: Kefeng Wang --- v2: add ACK include/linux/mm_inline.h | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 08c2bcefcb2b..21d6c72bcc71 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -323,12 +323,6 @@ void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio) list_add(&folio->lru, &lruvec->lists[lru]); } -static __always_inline void add_page_to_lru_list(struct page *page, - struct lruvec *lruvec) -{ - lruvec_add_folio(lruvec, page_folio(page)); -} - static __always_inline void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) { @@ -357,12 +351,6 @@ void lruvec_del_folio(struct lruvec *lruvec, struct folio *folio) -folio_nr_pages(folio)); } -static __always_inline void del_page_from_lru_list(struct page *page, - struct lruvec *lruvec) -{ - lruvec_del_folio(lruvec, page_folio(page)); -} - #ifdef CONFIG_ANON_VMA_NAME /* * mmap_lock should be read-locked when calling anon_vma_name(). Caller should