From patchwork Wed Nov 9 01:23:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 17302 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp76816wru; Tue, 8 Nov 2022 17:28:30 -0800 (PST) X-Google-Smtp-Source: AMsMyM7RpnxCI8HiRBzFvovADFMI1sqrRsQrM4DWpNSnf0M5WK1BYgiDn0MZL5SZiloc2rgQS254 X-Received: by 2002:aa7:df08:0:b0:461:d9a2:b247 with SMTP id c8-20020aa7df08000000b00461d9a2b247mr58540171edy.54.1667957310223; Tue, 08 Nov 2022 17:28:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1667957310; cv=none; d=google.com; s=arc-20160816; b=lPdV2TxddDruNfwL7Ui2hizyQ5/7fYPOS1RrJkPqFAX0bvhQj6vwM+wWUK5mDy+JBP zwqQ4MAoso10djeOcHZbumQMThDd13ITtYc1g6rtKwIjmr5DBmAR/dmDsOaZAfY+oOKi U0Z/623jXQcYRpVpIlL+H3E8JQ6+QjDzYsCDE7GWnmIMWuQqt1OVL1kc36qILmi10LDK oNUr+WqXB1YJr/fXPdVc/5sp0I2SkJUVEfT9Vu0aS/AHQ/ddXQOTwmYEODVBIQfpan0L D/0n+VDepQXzePdTNQ6Qt2ksYoP4U/LmIaMorT6eGmI3fM23MhoHVvWrOn1+5DqB87xQ 8/TA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CBLC+SqCx7ZsTil+7GtwHtGMMzbj+OEzoriZkFFJwyE=; b=lYwcheAm7PnJg15SK2A4e7xryhWFLOIc+L0uq2oW5jdqp+wdRrGcdUsluPs+kSHuNn riHvxDvsgfJ2o5u822SeRysrk0HBl/+CJn/hM+OCI3EH8lnpWkMnZhce+h+JP7wJ2uGl U4023bKb9cZA4RpEJPvQ0zduwSmhMOI3QGcdsDP1VoFc5by+lYmt1qZCp64HZfaTvRFm delVvfOnee1Sd6daH34ZYSb7a836779k3pW6o1HbX/FBmZn+ubKXephuU1NA3MbNWMYm 3d30tItllcBGuafWSg7+1tkAHv3Y2q9YgliGjcCpeIkH4v3fUPDOGxNhNr+5Mvn9ZK+h lPPA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=YydOWPO9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j8-20020aa7c0c8000000b00461e685ce72si1906172edp.116.2022.11.08.17.27.41; Tue, 08 Nov 2022 17:28:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=YydOWPO9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229851AbiKIBYH (ORCPT + 99 others); Tue, 8 Nov 2022 20:24:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57876 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229682AbiKIBYE (ORCPT ); Tue, 8 Nov 2022 20:24:04 -0500 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C57055E9ED for ; Tue, 8 Nov 2022 17:24:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1667957043; x=1699493043; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=yoCjLY5a568iSDEpolTlkW/LjTBOESOZ2uhJ1D8VbnI=; b=YydOWPO9z3R0X+Sq97cSIh0pZNyFb1frpE6ElNB8Pm0OaQDcdNJ9dTJm 0ja2eZhF22Dbtk3JE/Wzu2zxOArBt4jUMp3xu6Ac22EEGLKTYV4Pqz2Wk 21c0eEbyNCavliL5Du8MtO4otrGunHsyKSJkxbILHw47cdhvS9Yc0QvYH ZlBBETLhSrr0ZbckDcR1B8D5Ph3Arxmtr2Tt34Ob3mUSeSUkJPdiYj8P0 kZ1Lni1FpXVZO7dy2Shs+seY6FKIiU5OXXxej+JBuIAFw+4aoC6IPuhe+ UG3+D6Gi8O8HZrsXusdtTVmFfOPYIkUlgPR/U5K8tXkoBjgZkwqE+v3yk Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10525"; a="298374467" X-IronPort-AV: E=Sophos;i="5.96,149,1665471600"; d="scan'208";a="298374467" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Nov 2022 17:24:01 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10525"; a="881731723" X-IronPort-AV: E=Sophos;i="5.96,149,1665471600"; d="scan'208";a="881731723" Received: from yhuang6-mobl2.sh.intel.com ([10.238.6.69]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Nov 2022 17:23:58 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Yang Shi , Zi Yan , Matthew Wilcox , Baolin Wang , Oscar Salvador Subject: [PATCH -V2 1/2] migrate: convert unmap_and_move() to use folios Date: Wed, 9 Nov 2022 09:23:47 +0800 Message-Id: <20221109012348.93849-2-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221109012348.93849-1-ying.huang@intel.com> References: <20221109012348.93849-1-ying.huang@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748980004432864099?= X-GMAIL-MSGID: =?utf-8?q?1748980004432864099?= Quite straightforward, the page functions are converted to corresponding folio functions. Same for comments. Signed-off-by: "Huang, Ying" Reviewed-by: Yang Shi Reviewed-by: Zi Yan Reviewed-by: Matthew Wilcox (Oracle) Reviewed-by: Baolin Wang Cc: Oscar Salvador --- mm/migrate.c | 54 ++++++++++++++++++++++++++-------------------------- 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 4aea647a0180..eed8082664f2 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1150,79 +1150,79 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, } /* - * Obtain the lock on page, remove all ptes and migrate the page - * to the newly allocated page in newpage. + * Obtain the lock on folio, remove all ptes and migrate the folio + * to the newly allocated folio in dst. */ static int unmap_and_move(new_page_t get_new_page, free_page_t put_new_page, - unsigned long private, struct page *page, + unsigned long private, struct folio *src, int force, enum migrate_mode mode, enum migrate_reason reason, struct list_head *ret) { - struct folio *dst, *src = page_folio(page); + struct folio *dst; int rc = MIGRATEPAGE_SUCCESS; struct page *newpage = NULL; - if (!thp_migration_supported() && PageTransHuge(page)) + if (!thp_migration_supported() && folio_test_transhuge(src)) return -ENOSYS; - if (page_count(page) == 1) { - /* Page was freed from under us. So we are done. */ - ClearPageActive(page); - ClearPageUnevictable(page); + if (folio_ref_count(src) == 1) { + /* Folio was freed from under us. So we are done. */ + folio_clear_active(src); + folio_clear_unevictable(src); /* free_pages_prepare() will clear PG_isolated. */ goto out; } - newpage = get_new_page(page, private); + newpage = get_new_page(&src->page, private); if (!newpage) return -ENOMEM; dst = page_folio(newpage); - newpage->private = 0; + dst->private = 0; rc = __unmap_and_move(src, dst, force, mode); if (rc == MIGRATEPAGE_SUCCESS) - set_page_owner_migrate_reason(newpage, reason); + set_page_owner_migrate_reason(&dst->page, reason); out: if (rc != -EAGAIN) { /* - * A page that has been migrated has all references - * removed and will be freed. A page that has not been + * A folio that has been migrated has all references + * removed and will be freed. A folio that has not been * migrated will have kept its references and be restored. */ - list_del(&page->lru); + list_del(&src->lru); } /* * If migration is successful, releases reference grabbed during - * isolation. Otherwise, restore the page to right list unless + * isolation. Otherwise, restore the folio to right list unless * we want to retry. */ if (rc == MIGRATEPAGE_SUCCESS) { /* - * Compaction can migrate also non-LRU pages which are + * Compaction can migrate also non-LRU folios which are * not accounted to NR_ISOLATED_*. They can be recognized - * as __PageMovable + * as __folio_test_movable */ - if (likely(!__PageMovable(page))) - mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON + - page_is_file_lru(page), -thp_nr_pages(page)); + if (likely(!__folio_test_movable(src))) + mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON + + folio_is_file_lru(src), -folio_nr_pages(src)); if (reason != MR_MEMORY_FAILURE) /* - * We release the page in page_handle_poison. + * We release the folio in page_handle_poison. */ - put_page(page); + folio_put(src); } else { if (rc != -EAGAIN) - list_add_tail(&page->lru, ret); + list_add_tail(&src->lru, ret); if (put_new_page) - put_new_page(newpage, private); + put_new_page(&dst->page, private); else - put_page(newpage); + folio_put(dst); } return rc; @@ -1459,7 +1459,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, &ret_pages); else rc = unmap_and_move(get_new_page, put_new_page, - private, page, pass > 2, mode, + private, page_folio(page), pass > 2, mode, reason, &ret_pages); /* * The rules are: From patchwork Wed Nov 9 01:23:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 17303 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp77074wru; Tue, 8 Nov 2022 17:29:11 -0800 (PST) X-Google-Smtp-Source: AMsMyM7O75JPZqKcI2XoOZ9Ymv6Obm8MleL+nfBLgZm20aFs9/lAYl8Rn3c+OB4v0iImKxGWjArH X-Received: by 2002:a05:6a00:234c:b0:56c:f6e6:976 with SMTP id j12-20020a056a00234c00b0056cf6e60976mr58776840pfj.32.1667957351654; Tue, 08 Nov 2022 17:29:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1667957351; cv=none; d=google.com; s=arc-20160816; b=u4gG9NP1RIqTSFuGY9a9pBM4Pr/2Th9ck2aQyeiAiYIO20Xjf2NexmMtsbPUScaW6i 0TCvmXHMHMJd+sW16vaq+lQAoNaKS26NY1R5wc2/D4Yyq/pBwdSNxgsHQ04Sh/zCgxJb 3DGVI8q7M7tm3uGaa8t2MopLNlOBofDJTfd5QoUwWK8FkZdmAVecAk07cvUaHIt6Z0DT LrUHCS/E+DpNDYVnuF6qV3+S/Jpb8UqTlalIR6A7lpskuA4iwj4cNvwqK1Q9Y7YrNEIB qBzzcURDIhHUDa1ZJfAxNX7qcwxW58eDmTx9ybnzNKLKR0eSiU5MuPlg9Ci+k4Git1Qc /kew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jnGjJ9D3Zht/2Ru+FKMzaMZCHYBVCLEqTWWYhHSAhvg=; b=cgCDKhAegO/otO3vwpf50U287LA81P3UdzdQitYEMXlQU8rV63zCt/IMGWt5sBUM6d /1Qrt7ge4L7uHeCdxSNfjojjh1rfsXe7ROaPflqsGD+Z3iPwNDQ4wzdBSRCtL2xzWYD3 sM46zb9xxx5Da0sOylkJYVaBSVu65Gn2HMAPMO/DqpxmP3aDrSsFiEqLO+f2eofzMeUa RdSILXSXHQ4vjaiAdd+8eEU6nOzIzWTwpjBOh1cWZvMM0qXPe2tFdngbxT2CVQ7Yc7iS lnxd4NEzzvY+9ydhSubiJH012/xGpU5KtTouUa5TeI1dgznAFLZr5AutwP1jNfeb5WjY Yssg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=PP4KP2kz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g4-20020a1709026b4400b0017b2fe7de33si15026304plt.10.2022.11.08.17.28.57; Tue, 08 Nov 2022 17:29:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=PP4KP2kz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229948AbiKIBYL (ORCPT + 99 others); Tue, 8 Nov 2022 20:24:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57896 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229854AbiKIBYG (ORCPT ); Tue, 8 Nov 2022 20:24:06 -0500 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 01FDC5EFA8 for ; Tue, 8 Nov 2022 17:24:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1667957045; x=1699493045; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=MsjtziYQ+UZEvmg+xXZfnXrSP/L1eZFlxuzHKGo0q18=; b=PP4KP2kzbv0PUxDrgyBIbMgIQY7IEmEuTqTKEIP8oqjJbUAwp99a1DHF t8adsmrE0hNhmFV+X1550g4EgP9fCHyQtqgcbN7pUBoesDPjcBI3ZxLI2 POxwEE9dKOSWGAAFFDUNSaaBu071xMfVqoKyjjEHg1oH+TZnzxoPos8M/ vhyk5u8Fa9VpHO1bFdITmx2JqKn8kaB9KJUGMqnc98puDZPz0zZEJOgZc 5KKf/XgB2GSRFJ+SIGMT0CrhDo7UqFTYX180QHxs5BbE7fHFrkthKGhvt zInEfnGYS094foajiIeDobfVf/tae01IO9lxn580Q+X0Xgn6u38++vjgf A==; X-IronPort-AV: E=McAfee;i="6500,9779,10525"; a="298374475" X-IronPort-AV: E=Sophos;i="5.96,149,1665471600"; d="scan'208";a="298374475" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Nov 2022 17:24:04 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10525"; a="881731739" X-IronPort-AV: E=Sophos;i="5.96,149,1665471600"; d="scan'208";a="881731739" Received: from yhuang6-mobl2.sh.intel.com ([10.238.6.69]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Nov 2022 17:24:01 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox Subject: [PATCH -V2 2/2] migrate: convert migrate_pages() to use folios Date: Wed, 9 Nov 2022 09:23:48 +0800 Message-Id: <20221109012348.93849-3-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221109012348.93849-1-ying.huang@intel.com> References: <20221109012348.93849-1-ying.huang@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748980047902843916?= X-GMAIL-MSGID: =?utf-8?q?1748980047902843916?= Quite straightforward, the page functions are converted to corresponding folio functions. Same for comments. THP specific code are converted to be large folio. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Reviewed-by: Baolin Wang Tested-by: Baolin Wang --- mm/migrate.c | 210 +++++++++++++++++++++++++++------------------------ 1 file changed, 112 insertions(+), 98 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index eed8082664f2..8e4088aab27a 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1373,231 +1373,245 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, return rc; } -static inline int try_split_thp(struct page *page, struct list_head *split_pages) +static inline int try_split_folio(struct folio *folio, struct list_head *split_folios) { int rc; - lock_page(page); - rc = split_huge_page_to_list(page, split_pages); - unlock_page(page); + folio_lock(folio); + rc = split_folio_to_list(folio, split_folios); + folio_unlock(folio); if (!rc) - list_move_tail(&page->lru, split_pages); + list_move_tail(&folio->lru, split_folios); return rc; } /* - * migrate_pages - migrate the pages specified in a list, to the free pages + * migrate_pages - migrate the folios specified in a list, to the free folios * supplied as the target for the page migration * - * @from: The list of pages to be migrated. - * @get_new_page: The function used to allocate free pages to be used - * as the target of the page migration. - * @put_new_page: The function used to free target pages if migration + * @from: The list of folios to be migrated. + * @get_new_page: The function used to allocate free folios to be used + * as the target of the folio migration. + * @put_new_page: The function used to free target folios if migration * fails, or NULL if no special handling is necessary. * @private: Private data to be passed on to get_new_page() * @mode: The migration mode that specifies the constraints for - * page migration, if any. - * @reason: The reason for page migration. - * @ret_succeeded: Set to the number of normal pages migrated successfully if + * folio migration, if any. + * @reason: The reason for folio migration. + * @ret_succeeded: Set to the number of folios migrated successfully if * the caller passes a non-NULL pointer. * - * The function returns after 10 attempts or if no pages are movable any more - * because the list has become empty or no retryable pages exist any more. - * It is caller's responsibility to call putback_movable_pages() to return pages + * The function returns after 10 attempts or if no folios are movable any more + * because the list has become empty or no retryable folios exist any more. + * It is caller's responsibility to call putback_movable_pages() to return folios * to the LRU or free list only if ret != 0. * - * Returns the number of {normal page, THP, hugetlb} that were not migrated, or - * an error code. The number of THP splits will be considered as the number of - * non-migrated THP, no matter how many subpages of the THP are migrated successfully. + * Returns the number of {normal folio, large folio, hugetlb} that were not + * migrated, or an error code. The number of large folio splits will be + * considered as the number of non-migrated large folio, no matter how many + * split folios of the large folio are migrated successfully. */ int migrate_pages(struct list_head *from, new_page_t get_new_page, free_page_t put_new_page, unsigned long private, enum migrate_mode mode, int reason, unsigned int *ret_succeeded) { int retry = 1; + int large_retry = 1; int thp_retry = 1; int nr_failed = 0; int nr_failed_pages = 0; int nr_retry_pages = 0; int nr_succeeded = 0; int nr_thp_succeeded = 0; + int nr_large_failed = 0; int nr_thp_failed = 0; int nr_thp_split = 0; int pass = 0; + bool is_large = false; bool is_thp = false; - struct page *page; - struct page *page2; - int rc, nr_subpages; - LIST_HEAD(ret_pages); - LIST_HEAD(thp_split_pages); + struct folio *folio, *folio2; + int rc, nr_pages; + LIST_HEAD(ret_folios); + LIST_HEAD(split_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); - bool no_subpage_counting = false; + bool no_split_folio_counting = false; trace_mm_migrate_pages_start(mode, reason); -thp_subpage_migration: - for (pass = 0; pass < 10 && (retry || thp_retry); pass++) { +split_folio_migration: + for (pass = 0; pass < 10 && (retry || large_retry); pass++) { retry = 0; + large_retry = 0; thp_retry = 0; nr_retry_pages = 0; - list_for_each_entry_safe(page, page2, from, lru) { + list_for_each_entry_safe(folio, folio2, from, lru) { /* - * THP statistics is based on the source huge page. - * Capture required information that might get lost - * during migration. + * Large folio statistics is based on the source large + * folio. Capture required information that might get + * lost during migration. */ - is_thp = PageTransHuge(page) && !PageHuge(page); - nr_subpages = compound_nr(page); + is_large = folio_test_large(folio) && !folio_test_hugetlb(folio); + is_thp = is_large && folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); cond_resched(); - if (PageHuge(page)) + if (folio_test_hugetlb(folio)) rc = unmap_and_move_huge_page(get_new_page, - put_new_page, private, page, - pass > 2, mode, reason, - &ret_pages); + put_new_page, private, + &folio->page, pass > 2, mode, + reason, + &ret_folios); else rc = unmap_and_move(get_new_page, put_new_page, - private, page_folio(page), pass > 2, mode, - reason, &ret_pages); + private, folio, pass > 2, mode, + reason, &ret_folios); /* * The rules are: - * Success: non hugetlb page will be freed, hugetlb - * page will be put back + * Success: non hugetlb folio will be freed, hugetlb + * folio will be put back * -EAGAIN: stay on the from list * -ENOMEM: stay on the from list * -ENOSYS: stay on the from list - * Other errno: put on ret_pages list then splice to + * Other errno: put on ret_folios list then splice to * from list */ switch(rc) { /* - * THP migration might be unsupported or the - * allocation could've failed so we should - * retry on the same page with the THP split - * to base pages. + * Large folio migration might be unsupported or + * the allocation could've failed so we should retry + * on the same folio with the large folio split + * to normal folios. * - * Sub-pages are put in thp_split_pages, and + * Split folios are put in split_folios, and * we will migrate them after the rest of the * list is processed. */ case -ENOSYS: - /* THP migration is unsupported */ - if (is_thp) { - nr_thp_failed++; - if (!try_split_thp(page, &thp_split_pages)) { - nr_thp_split++; + /* Large folio migration is unsupported */ + if (is_large) { + nr_large_failed++; + nr_thp_failed += is_thp; + if (!try_split_folio(folio, &split_folios)) { + nr_thp_split += is_thp; break; } /* Hugetlb migration is unsupported */ - } else if (!no_subpage_counting) { + } else if (!no_split_folio_counting) { nr_failed++; } - nr_failed_pages += nr_subpages; - list_move_tail(&page->lru, &ret_pages); + nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, &ret_folios); break; case -ENOMEM: /* * When memory is low, don't bother to try to migrate - * other pages, just exit. + * other folios, just exit. */ - if (is_thp) { - nr_thp_failed++; - /* THP NUMA faulting doesn't split THP to retry. */ + if (is_large) { + nr_large_failed++; + nr_thp_failed += is_thp; + /* Large folio NUMA faulting doesn't split to retry. */ if (!nosplit) { - int ret = try_split_thp(page, &thp_split_pages); + int ret = try_split_folio(folio, &split_folios); if (!ret) { - nr_thp_split++; + nr_thp_split += is_thp; break; } else if (reason == MR_LONGTERM_PIN && ret == -EAGAIN) { /* - * Try again to split THP to mitigate - * the failure of longterm pinning. + * Try again to split large folio to + * mitigate the failure of longterm pinning. */ - thp_retry++; - nr_retry_pages += nr_subpages; + large_retry++; + thp_retry += is_thp; + nr_retry_pages += nr_pages; break; } } - } else if (!no_subpage_counting) { + } else if (!no_split_folio_counting) { nr_failed++; } - nr_failed_pages += nr_subpages + nr_retry_pages; + nr_failed_pages += nr_pages + nr_retry_pages; /* - * There might be some subpages of fail-to-migrate THPs - * left in thp_split_pages list. Move them back to migration + * There might be some split folios of fail-to-migrate large + * folios left in split_folios list. Move them back to migration * list so that they could be put back to the right list by - * the caller otherwise the page refcnt will be leaked. + * the caller otherwise the folio refcnt will be leaked. */ - list_splice_init(&thp_split_pages, from); + list_splice_init(&split_folios, from); /* nr_failed isn't updated for not used */ + nr_large_failed += large_retry; nr_thp_failed += thp_retry; goto out; case -EAGAIN: - if (is_thp) - thp_retry++; - else if (!no_subpage_counting) + if (is_large) { + large_retry++; + thp_retry += is_thp; + } else if (!no_split_folio_counting) { retry++; - nr_retry_pages += nr_subpages; + } + nr_retry_pages += nr_pages; break; case MIGRATEPAGE_SUCCESS: - nr_succeeded += nr_subpages; - if (is_thp) - nr_thp_succeeded++; + nr_succeeded += nr_pages; + nr_thp_succeeded += is_thp; break; default: /* * Permanent failure (-EBUSY, etc.): - * unlike -EAGAIN case, the failed page is - * removed from migration page list and not + * unlike -EAGAIN case, the failed folio is + * removed from migration folio list and not * retried in the next outer loop. */ - if (is_thp) - nr_thp_failed++; - else if (!no_subpage_counting) + if (is_large) { + nr_large_failed++; + nr_thp_failed += is_thp; + } else if (!no_split_folio_counting) { nr_failed++; + } - nr_failed_pages += nr_subpages; + nr_failed_pages += nr_pages; break; } } } nr_failed += retry; + nr_large_failed += large_retry; nr_thp_failed += thp_retry; nr_failed_pages += nr_retry_pages; /* - * Try to migrate subpages of fail-to-migrate THPs, no nr_failed - * counting in this round, since all subpages of a THP is counted - * as 1 failure in the first round. + * Try to migrate split folios of fail-to-migrate large folios, no + * nr_failed counting in this round, since all split folios of a + * large folio is counted as 1 failure in the first round. */ - if (!list_empty(&thp_split_pages)) { + if (!list_empty(&split_folios)) { /* - * Move non-migrated pages (after 10 retries) to ret_pages + * Move non-migrated folios (after 10 retries) to ret_folios * to avoid migrating them again. */ - list_splice_init(from, &ret_pages); - list_splice_init(&thp_split_pages, from); - no_subpage_counting = true; + list_splice_init(from, &ret_folios); + list_splice_init(&split_folios, from); + no_split_folio_counting = true; retry = 1; - goto thp_subpage_migration; + goto split_folio_migration; } - rc = nr_failed + nr_thp_failed; + rc = nr_failed + nr_large_failed; out: /* - * Put the permanent failure page back to migration list, they + * Put the permanent failure folio back to migration list, they * will be put back to the right list by the caller. */ - list_splice(&ret_pages, from); + list_splice(&ret_folios, from); /* - * Return 0 in case all subpages of fail-to-migrate THPs are - * migrated successfully. + * Return 0 in case all split folios of fail-to-migrate large folios + * are migrated successfully. */ if (list_empty(from)) rc = 0;