From patchwork Mon Jun 12 14:34:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 106677 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2669330vqr; Mon, 12 Jun 2023 08:30:02 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7HvuUgCDQFoqfrH0JOX+WlpX4GCagE5ho5WDTinGJfwM8VqAmuTG6WG1qN7dc+Ed4jHD/J X-Received: by 2002:a05:6a21:680d:b0:10e:a8e6:35c5 with SMTP id wr13-20020a056a21680d00b0010ea8e635c5mr8510160pzb.59.1686583802146; Mon, 12 Jun 2023 08:30:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1686583802; cv=none; d=google.com; s=arc-20160816; b=vC7fOlSKVUI9Gxz42a1DCKBI+iozmKY2+VITYhb47CROlW66/Dqfjactgi8S6xzeAV LluH2tPyijZeJslqod9zamandXUUeqFEz66gbbwTUK9UM5GAx18Z0YTV7aoyQbvN524a VIlonXOD+o82uVV4WwEfNUgpkXEp3WrPBU5eaEZm0wSodi7VD3umDtvV0ZI0rrCOTVWX Wd0SFK36t5CUNLzvImPvHKtqT5h7ZvcC84xLPXRGFb80nDVoKrOAoCOm9TcsLxiQUpHl 3E2dM3qUGwgfGZn1e8CfljMX7rE06MupM2VechxHAqGiVJacVzs+xW8ic1wHJl3JRp37 8u/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from; bh=uYsiE51xOQE97372+4MzK4Vf9RgTPz6ZL1P5whhNwOs=; b=lDVN9ALGcHVgLWzNcusG5LVPJAPu/tCh/lSB9NgN5U32XHeTT4/3SgtypSIBqJb+ud 2Z7Jl/qvos7fK5bxzaqD+7/FOwmOSlWK9eBtECSiwK8Y0cfBWAMbNT7L3GgUyjgAZ5k4 Em+NAqMV/sgn7bXN4X98LMFrih98JtSpb4DC7X3tnbACT/gK8B4Wr2F1AZPwiGNBQ+8m chQqY+oR4fs6MwMAi9TlvTatVQpMuKFdoRg80Jiwe8pNjGAyBwKRAzB3GDEPtixDt+mw 8gj9JLNgJNNJReC954ocFNTfZIYUi4vIwicfj5CduXPznpFaytghAcaauwlkAwfywPOU h+ZQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t69-20020a638148000000b00518cf8916e2si7218011pgd.415.2023.06.12.08.29.50; Mon, 12 Jun 2023 08:30:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237318AbjFLOmy (ORCPT + 99 others); Mon, 12 Jun 2023 10:42:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52200 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239376AbjFLOka (ORCPT ); Mon, 12 Jun 2023 10:40:30 -0400 Received: from szxga08-in.huawei.com (szxga08-in.huawei.com [45.249.212.255]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8DE21BD for ; Mon, 12 Jun 2023 07:40:27 -0700 (PDT) Received: from dggpemm500001.china.huawei.com (unknown [172.30.72.53]) by szxga08-in.huawei.com (SkyGuard) with ESMTP id 4QftwP2VmGz18M14; Mon, 12 Jun 2023 22:14:13 +0800 (CST) Received: from localhost.localdomain.localdomain (10.175.113.25) by dggpemm500001.china.huawei.com (7.185.36.107) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Mon, 12 Jun 2023 22:19:07 +0800 From: Kefeng Wang To: CC: , , , , , Kefeng Wang Subject: [PATCH -next 1/2] mm: compaction: convert to use a folio in isolate_migratepages_block() Date: Mon, 12 Jun 2023 22:34:13 +0800 Message-ID: <20230612143414.186389-1-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.35.3 MIME-Version: 1.0 X-Originating-IP: [10.175.113.25] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpemm500001.china.huawei.com (7.185.36.107) X-CFilter-Loop: Reflected X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1768510659793287004?= X-GMAIL-MSGID: =?utf-8?q?1768511296988677776?= Directly use a folio instead of page_folio() when page successfully isolated (hugepage and movable page) and after folio_get_nontail_page(), which removes several calls to compound_head(). Signed-off-by: Kefeng Wang --- mm/compaction.c | 71 ++++++++++++++++++++++++++----------------------- 1 file changed, 38 insertions(+), 33 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 3398ef3a55fe..5d3f0aaa6785 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -831,6 +831,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, struct lruvec *lruvec; unsigned long flags = 0; struct lruvec *locked = NULL; + struct folio *folio = NULL; struct page *page = NULL, *valid_page = NULL; struct address_space *mapping; unsigned long start_pfn = low_pfn; @@ -927,7 +928,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if (!valid_page && pageblock_aligned(low_pfn)) { if (!isolation_suitable(cc, page)) { low_pfn = end_pfn; - page = NULL; + folio = NULL; goto isolate_abort; } valid_page = page; @@ -959,7 +960,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, * Hugepage was successfully isolated and placed * on the cc->migratepages list. */ - low_pfn += compound_nr(page) - 1; + folio = page_folio(page); + low_pfn += folio_nr_pages(folio) - 1; goto isolate_success_no_list; } @@ -1027,8 +1029,10 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, locked = NULL; } - if (isolate_movable_page(page, mode)) + if (isolate_movable_page(page, mode)) { + folio = page_folio(page); goto isolate_success; + } } goto isolate_fail; @@ -1039,7 +1043,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, * sure the page is not being freed elsewhere -- the * page release code relies on it. */ - if (unlikely(!get_page_unless_zero(page))) + folio = folio_get_nontail_page(page); + if (unlikely(!folio)) goto isolate_fail; /* @@ -1047,7 +1052,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, * from long term pinning preventing it from migrating, * so avoid taking lru_lock and isolating it unnecessarily. */ - mapping = page_mapping(page); + mapping = folio_mapping(folio); if (!cc->alloc_contig && page_has_extra_refs(page, mapping)) goto isolate_fail_put; @@ -1063,7 +1068,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, goto isolate_fail_put; /* Compaction might skip unevictable pages but CMA takes them */ - if (!(mode & ISOLATE_UNEVICTABLE) && PageUnevictable(page)) + if (!(mode & ISOLATE_UNEVICTABLE) && folio_test_unevictable(folio)) goto isolate_fail_put; /* @@ -1072,10 +1077,10 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, * it will be able to migrate without blocking - clean pages * for the most part. PageWriteback would require blocking. */ - if ((mode & ISOLATE_ASYNC_MIGRATE) && PageWriteback(page)) + if ((mode & ISOLATE_ASYNC_MIGRATE) && folio_test_writeback(folio)) goto isolate_fail_put; - if ((mode & ISOLATE_ASYNC_MIGRATE) && PageDirty(page)) { + if ((mode & ISOLATE_ASYNC_MIGRATE) && folio_test_dirty(folio)) { bool migrate_dirty; /* @@ -1087,22 +1092,22 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, * the page lock until after the page is removed * from the page cache. */ - if (!trylock_page(page)) + if (!folio_trylock(folio)) goto isolate_fail_put; - mapping = page_mapping(page); + mapping = folio_mapping(folio); migrate_dirty = !mapping || mapping->a_ops->migrate_folio; - unlock_page(page); + folio_unlock(folio); if (!migrate_dirty) goto isolate_fail_put; } /* Try isolate the page */ - if (!TestClearPageLRU(page)) + if (!folio_test_clear_lru(folio)) goto isolate_fail_put; - lruvec = folio_lruvec(page_folio(page)); + lruvec = folio_lruvec(folio); /* If we already hold the lock, we can skip some rechecking */ if (lruvec != locked) { @@ -1112,7 +1117,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, compact_lock_irqsave(&lruvec->lru_lock, &flags, cc); locked = lruvec; - lruvec_memcg_debug(lruvec, page_folio(page)); + lruvec_memcg_debug(lruvec, folio); /* * Try get exclusive access under lock. If marked for @@ -1132,30 +1137,30 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, * and it's on LRU. It can only be a THP so the order * is safe to read and it's 0 for tail pages. */ - if (unlikely(PageCompound(page) && !cc->alloc_contig)) { - low_pfn += compound_nr(page) - 1; - nr_scanned += compound_nr(page) - 1; - SetPageLRU(page); + if (unlikely(folio_test_large(folio) && !cc->alloc_contig)) { + low_pfn += folio_nr_pages(folio) - 1; + nr_scanned += folio_nr_pages(folio) - 1; + folio_set_lru(folio); goto isolate_fail_put; } } /* The whole page is taken off the LRU; skip the tail pages. */ - if (PageCompound(page)) - low_pfn += compound_nr(page) - 1; + if (folio_test_large(folio)) + low_pfn += folio_nr_pages(folio) - 1; /* Successfully isolated */ - del_page_from_lru_list(page, lruvec); - mod_node_page_state(page_pgdat(page), - NR_ISOLATED_ANON + page_is_file_lru(page), - thp_nr_pages(page)); + lruvec_del_folio(lruvec, folio); + mod_node_page_state(folio_pgdat(folio), + NR_ISOLATED_ANON + folio_is_file_lru(folio), + folio_nr_pages(folio)); isolate_success: - list_add(&page->lru, &cc->migratepages); + list_add(&folio->lru, &cc->migratepages); isolate_success_no_list: - cc->nr_migratepages += compound_nr(page); - nr_isolated += compound_nr(page); - nr_scanned += compound_nr(page) - 1; + cc->nr_migratepages += folio_nr_pages(folio); + nr_isolated += folio_nr_pages(folio); + nr_scanned += folio_nr_pages(folio) - 1; /* * Avoid isolating too much unless this block is being @@ -1177,7 +1182,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, unlock_page_lruvec_irqrestore(locked, flags); locked = NULL; } - put_page(page); + folio_put(folio); isolate_fail: if (!skip_on_failure && ret != -ENOMEM) @@ -1218,14 +1223,14 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if (unlikely(low_pfn > end_pfn)) low_pfn = end_pfn; - page = NULL; + folio = NULL; isolate_abort: if (locked) unlock_page_lruvec_irqrestore(locked, flags); - if (page) { - SetPageLRU(page); - put_page(page); + if (folio) { + folio_set_lru(folio); + folio_put(folio); } /* From patchwork Mon Jun 12 14:34:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kefeng Wang X-Patchwork-Id: 106666 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2660663vqr; Mon, 12 Jun 2023 08:16:39 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6CPofuwF8QrBy7pG8BgvwLADOFPqQdLwJY98BY08Rl9BiM4lXk29ep+iY2D1vnoKSbe9dN X-Received: by 2002:a17:907:809:b0:977:cbaf:bc56 with SMTP id wv9-20020a170907080900b00977cbafbc56mr10436483ejb.43.1686582998947; Mon, 12 Jun 2023 08:16:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1686582998; cv=none; d=google.com; s=arc-20160816; b=F9IGRHr3R8JgWONI/ht3xDqvavXFRjZhIOzW3/RU+EADznYtGbB9ByOdGaccuYHKoF 6v0jNLcaRAjxTjWUdBTepjrWtlAqPnX5OY7Uznt1CnajXWcgGvzaK3HPjix+mrc9Bb85 2mDAgQ294cIlbG0hMNvoJtegLt/DpkKmrJvji1FgdbAXGhWF3y6Ty1BnmsG/rX5CRlHU 084pz62drP/+fE4+S7wjnCE3PVlKRR3YazaiKTLxlxmNEKQU24u0iy/8KbY0r7+dexc+ sa0yymE/nUGh6ExAr0CPvkmUUs7lZBcE+NlPDyYXL0wAmAjllqt9D9/LtMRiYXV/Y0Ub RYkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=y9oD1no1Jl5ED5oaZg6gUA0vIFcsSBeD07uHzX7PsR0=; b=oubHJoEnW1r/F+HRb82V93hViZ/ayyjtBRyXQuxTwMnQzEZkmuHN9I+wudUN4fJ3cy BQitDDSvorGwglOPAX/d6QZDOaqiLqYsbHvOJmsDc2bmEHIHizVeQa2hC7B3bvDJvkem wt1e92XtteJvrlWvRiQmsPNH7bK51Hybmo5HGC9xuqfLMagvbg0Yj9baad382o0zh7Mr RlHclCTUMW789tXllIgz16730R9cs1USW/UOtZW4Fo/gblDdCZ/PJDp2wJ4e7YoFDoQP /B6+kXgTESElbvDaiEAo4Y688b/9aHLQ3Cntq/eQAbomukIhNJG7vb+Oddf4nn2/vnUr 7S9w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c6-20020a17090603c600b009745ecdb8b4si4940422eja.183.2023.06.12.08.16.13; Mon, 12 Jun 2023 08:16:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237568AbjFLOko (ORCPT + 99 others); Mon, 12 Jun 2023 10:40:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52226 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235308AbjFLOkf (ORCPT ); Mon, 12 Jun 2023 10:40:35 -0400 Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 42E0493 for ; Mon, 12 Jun 2023 07:40:33 -0700 (PDT) Received: from dggpemm500001.china.huawei.com (unknown [172.30.72.56]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4Qfv1W5KKgzTl09; Mon, 12 Jun 2023 22:18:39 +0800 (CST) Received: from localhost.localdomain.localdomain (10.175.113.25) by dggpemm500001.china.huawei.com (7.185.36.107) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Mon, 12 Jun 2023 22:19:07 +0800 From: Kefeng Wang To: CC: , , , , , Kefeng Wang Subject: [PATCH -next 2/2] mm: kill [add|del]_page_to_lru_list() Date: Mon, 12 Jun 2023 22:34:14 +0800 Message-ID: <20230612143414.186389-2-wangkefeng.wang@huawei.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230612143414.186389-1-wangkefeng.wang@huawei.com> References: <20230612143414.186389-1-wangkefeng.wang@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.113.25] X-ClientProxiedBy: dggems705-chm.china.huawei.com (10.3.19.182) To dggpemm500001.china.huawei.com (7.185.36.107) X-CFilter-Loop: Reflected X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1768510454329331183?= X-GMAIL-MSGID: =?utf-8?q?1768510454329331183?= Now no one call [add|del]_page_to_lru_list(), let's drop unused page interfaces. Signed-off-by: Kefeng Wang Acked-by: Yu Zhao --- include/linux/mm_inline.h | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 0e1d239a882c..e9cdeb290841 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -323,12 +323,6 @@ void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio) list_add(&folio->lru, &lruvec->lists[lru]); } -static __always_inline void add_page_to_lru_list(struct page *page, - struct lruvec *lruvec) -{ - lruvec_add_folio(lruvec, page_folio(page)); -} - static __always_inline void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio) { @@ -357,12 +351,6 @@ void lruvec_del_folio(struct lruvec *lruvec, struct folio *folio) -folio_nr_pages(folio)); } -static __always_inline void del_page_from_lru_list(struct page *page, - struct lruvec *lruvec) -{ - lruvec_del_folio(lruvec, page_folio(page)); -} - #ifdef CONFIG_ANON_VMA_NAME /* * mmap_lock should be read-locked when calling anon_vma_name(). Caller should