From patchwork Mon Feb 6 06:33:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 53011 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2081776wrn; Sun, 5 Feb 2023 22:34:53 -0800 (PST) X-Google-Smtp-Source: AK7set98TT2ilEILchXsooo0OBlA8GsNQnqRCyjcOoR64GqyudghPkDUTYiq4EANmV17keQW6gbd X-Received: by 2002:a05:6a00:1790:b0:590:753d:78f5 with SMTP id s16-20020a056a00179000b00590753d78f5mr21953236pfg.20.1675665293027; Sun, 05 Feb 2023 22:34:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675665293; cv=none; d=google.com; s=arc-20160816; b=xFSgkiw4Xnj9KVntM9+9Lu4UNwLUuYAVF0M9payXdQO0zq4gQheDuk4ZUv9lmFKHBE csvvxZvLtkRA3bQgAmYiYZ4EpPyaZzE4llZK2m3LZ7MvkjOQfECJbnZWs0I8Z2tix0RJ p8l42D0ybtqFecd6XN96Wtw1+0QOHdHS4PK5X/TIehp67furvYYv9wcTiZ+JQbyCu3z9 xkSKbTacNCnpkq9Q8EcFcELhP6OCMwbEAIpy22v7AZbeHxvrcY/iofdjj5nX0MBWhf1L cmGXERk9OaAaTAY0WK/J0ewjWKlLGK/+nKDXnc0i0kkSI/MTpinjd78zhYRL1zoGhK5J XCdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=6K6EPnmkM3a5UOufl46/EB5E1x0OF9PgE2F47wj3TE0=; b=INbGM1HShBliB2z5khoRjj0NY3OZHspkA7qTOjDi5FGQyeCleF8NRLDbafxNrbA4Gg Eaoijw6ba8TdqFRXHBOLuMyASQNo+k7w0qL6JDF3uER4l6tCUi2iiADskrE0hUumIFKN YgrpZJa/0GjuafQd9wZXGyPNWkVXXYk6HANe0v5d7YmhU9H5Zo0qKIw7HUGHYNnf8G5X 1+VQSndRwgvUjGKBUV15BRWdIqrgBKjJfB4DgUOFZ4BXPO7oK6PW0w4WcQWhVBHFjY/x VNtL0JA+eU13d2A3CFyY9tI0Z65Wz7ife97+d5WWfDOiAKWvBlmr2gToBRHFBneFO/Q5 VBEg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="YbuUDnb/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n8-20020a056a00212800b005933ac113cfsi10939393pfj.98.2023.02.05.22.34.41; Sun, 05 Feb 2023 22:34:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="YbuUDnb/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229605AbjBFGdq (ORCPT + 99 others); Mon, 6 Feb 2023 01:33:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229609AbjBFGdj (ORCPT ); Mon, 6 Feb 2023 01:33:39 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9254218B3C for ; Sun, 5 Feb 2023 22:33:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675665215; x=1707201215; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fpqcNpPynLsFmyeQ5s3gi4KQwt0xUU5cyFiW/AFXtXc=; b=YbuUDnb/ZRL2RDRfSnQ9Gd5Sk3S62DUXQIF4d+g1XN7YD3CpB4lmUmmO LkuPq1EV2sZ0zSVLvHbOt8ok1Bv2xkRv/IQGOiUOSpoGMjEOElU17KJnX iP31WrRS7Dbmg/RVzBsVcrpiQOw4m3xfx6rFy+cyyT0Y8+ntVIiTyTG2Z jD69kutRI7uICJgZ7KYZh5J2DjnADevOcPaCYs2v8noJVF9t4nDEwLUMK konpjWiNvDeQrDEfubtirxt4cHkCFb0B74X9a7Ma+AeND9P2mrwyOde93 t2mjiqxtH/sTfJphMP9xeWop8ybf41QgKq8nebZAlmS75wsyI1LbcCbYd w==; X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="330432623" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="330432623" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:33:33 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="659744625" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="659744625" Received: from baoyumen-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.smartont.net) ([10.255.30.227]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:33:28 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Alistair Popple , Zi Yan , Baolin Wang , Yang Shi , Oscar Salvador , Matthew Wilcox , Bharata B Rao , haoxin , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH -v4 1/9] migrate_pages: organize stats with struct migrate_pages_stats Date: Mon, 6 Feb 2023 14:33:05 +0800 Message-Id: <20230206063313.635011-2-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230206063313.635011-1-ying.huang@intel.com> References: <20230206063313.635011-1-ying.huang@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757062410552285708?= X-GMAIL-MSGID: =?utf-8?q?1757062410552285708?= Define struct migrate_pages_stats to organize the various statistics in migrate_pages(). This makes it easier to collect and consume the statistics in multiple functions. This will be needed in the following patches in the series. Signed-off-by: "Huang, Ying" Reviewed-by: Alistair Popple Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Cc: Yang Shi Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: haoxin Cc: Minchan Kim Cc: Mike Kravetz Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Xin Hao --- mm/migrate.c | 60 +++++++++++++++++++++++++++++----------------------- 1 file changed, 34 insertions(+), 26 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index a4d3fc65085f..ef388a9e4747 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1396,6 +1396,16 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f return rc; } +struct migrate_pages_stats { + int nr_succeeded; /* Normal and large folios migrated successfully, in + units of base pages */ + int nr_failed_pages; /* Normal and large folios failed to be migrated, in + units of base pages. Untried folios aren't counted */ + int nr_thp_succeeded; /* THP migrated successfully */ + int nr_thp_failed; /* THP failed to be migrated */ + int nr_thp_split; /* THP split before migrating */ +}; + /* * migrate_pages - migrate the folios specified in a list, to the free folios * supplied as the target for the page migration @@ -1430,13 +1440,8 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, int large_retry = 1; int thp_retry = 1; int nr_failed = 0; - int nr_failed_pages = 0; int nr_retry_pages = 0; - int nr_succeeded = 0; - int nr_thp_succeeded = 0; int nr_large_failed = 0; - int nr_thp_failed = 0; - int nr_thp_split = 0; int pass = 0; bool is_large = false; bool is_thp = false; @@ -1446,9 +1451,11 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, LIST_HEAD(split_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); bool no_split_folio_counting = false; + struct migrate_pages_stats stats; trace_mm_migrate_pages_start(mode, reason); + memset(&stats, 0, sizeof(stats)); split_folio_migration: for (pass = 0; pass < 10 && (retry || large_retry); pass++) { retry = 0; @@ -1502,9 +1509,9 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, /* Large folio migration is unsupported */ if (is_large) { nr_large_failed++; - nr_thp_failed += is_thp; + stats.nr_thp_failed += is_thp; if (!try_split_folio(folio, &split_folios)) { - nr_thp_split += is_thp; + stats.nr_thp_split += is_thp; break; } /* Hugetlb migration is unsupported */ @@ -1512,7 +1519,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_failed++; } - nr_failed_pages += nr_pages; + stats.nr_failed_pages += nr_pages; list_move_tail(&folio->lru, &ret_folios); break; case -ENOMEM: @@ -1522,13 +1529,13 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (is_large) { nr_large_failed++; - nr_thp_failed += is_thp; + stats.nr_thp_failed += is_thp; /* Large folio NUMA faulting doesn't split to retry. */ if (!nosplit) { int ret = try_split_folio(folio, &split_folios); if (!ret) { - nr_thp_split += is_thp; + stats.nr_thp_split += is_thp; break; } else if (reason == MR_LONGTERM_PIN && ret == -EAGAIN) { @@ -1546,7 +1553,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_failed++; } - nr_failed_pages += nr_pages + nr_retry_pages; + stats.nr_failed_pages += nr_pages + nr_retry_pages; /* * There might be some split folios of fail-to-migrate large * folios left in split_folios list. Move them back to migration @@ -1556,7 +1563,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, list_splice_init(&split_folios, from); /* nr_failed isn't updated for not used */ nr_large_failed += large_retry; - nr_thp_failed += thp_retry; + stats.nr_thp_failed += thp_retry; goto out; case -EAGAIN: if (is_large) { @@ -1568,8 +1575,8 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_retry_pages += nr_pages; break; case MIGRATEPAGE_SUCCESS: - nr_succeeded += nr_pages; - nr_thp_succeeded += is_thp; + stats.nr_succeeded += nr_pages; + stats.nr_thp_succeeded += is_thp; break; default: /* @@ -1580,20 +1587,20 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (is_large) { nr_large_failed++; - nr_thp_failed += is_thp; + stats.nr_thp_failed += is_thp; } else if (!no_split_folio_counting) { nr_failed++; } - nr_failed_pages += nr_pages; + stats.nr_failed_pages += nr_pages; break; } } } nr_failed += retry; nr_large_failed += large_retry; - nr_thp_failed += thp_retry; - nr_failed_pages += nr_retry_pages; + stats.nr_thp_failed += thp_retry; + stats.nr_failed_pages += nr_retry_pages; /* * Try to migrate split folios of fail-to-migrate large folios, no * nr_failed counting in this round, since all split folios of a @@ -1626,16 +1633,17 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, if (list_empty(from)) rc = 0; - count_vm_events(PGMIGRATE_SUCCESS, nr_succeeded); - count_vm_events(PGMIGRATE_FAIL, nr_failed_pages); - count_vm_events(THP_MIGRATION_SUCCESS, nr_thp_succeeded); - count_vm_events(THP_MIGRATION_FAIL, nr_thp_failed); - count_vm_events(THP_MIGRATION_SPLIT, nr_thp_split); - trace_mm_migrate_pages(nr_succeeded, nr_failed_pages, nr_thp_succeeded, - nr_thp_failed, nr_thp_split, mode, reason); + count_vm_events(PGMIGRATE_SUCCESS, stats.nr_succeeded); + count_vm_events(PGMIGRATE_FAIL, stats.nr_failed_pages); + count_vm_events(THP_MIGRATION_SUCCESS, stats.nr_thp_succeeded); + count_vm_events(THP_MIGRATION_FAIL, stats.nr_thp_failed); + count_vm_events(THP_MIGRATION_SPLIT, stats.nr_thp_split); + trace_mm_migrate_pages(stats.nr_succeeded, stats.nr_failed_pages, + stats.nr_thp_succeeded, stats.nr_thp_failed, + stats.nr_thp_split, mode, reason); if (ret_succeeded) - *ret_succeeded = nr_succeeded; + *ret_succeeded = stats.nr_succeeded; return rc; } From patchwork Mon Feb 6 06:33:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 53017 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2082352wrn; Sun, 5 Feb 2023 22:37:10 -0800 (PST) X-Google-Smtp-Source: AK7set+BQQSXLD/Q0uto2X1X7uVnTqL1mVFN7JxvKzo4OhkLClOO7ur2roV4f1eSGWI2N1dOe3fO X-Received: by 2002:a17:907:7244:b0:7c0:f719:838d with SMTP id ds4-20020a170907724400b007c0f719838dmr11940954ejc.36.1675665430252; Sun, 05 Feb 2023 22:37:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675665430; cv=none; d=google.com; s=arc-20160816; b=nEVNejRHQGiAsdpd49fXEsYRxPgxE2zQlOOpq9qRWzyhh2cME9sZ325Yr0QzmcoIJ1 we4HzMDlxr5tCDimcV2Ahja5basz1XniAOZWZCw+xE/eLfoSGUZldOyL+e2KZTjelEH/ WvXyLP/5Pb+z7GPveT0pt4YT5i0tnrur+Lkgz6jqeQqIO8cdN2DZonEDcgN0SXprKPxL 8xLNofT81j6TghorkjgbIlq6BY7Skz1xzrAihmtPFNEnhBAcIULu2BEk1z3Reumr43/C 2UHKmNtzKLIu/tSSchcdApTaXSiMRu0lyLeOpT0VNWKE5DpQNI6rlAFhh146Y58g2hmJ EAeQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=02JKGbEaGFAGY6s9QDlNdYBEm6eKLQIiyyUvNtu5xDY=; b=OW/NiKmXN71Tak97qY/RvgOAyRA3g0pEb+znHmRmfZtHjmPIJcIJjH16olDn8yLhR6 iwpJ1uEH8YOHVYY6pyfY9Nph4oAlGRGCAHsvoyo/yv/mQu4cr5x3KTnXrLDxy7ziUrpo hb0dIuTucqG6YmV8RQrJNjp7srczYECP5S7B6xer7fUiuyM//0qd0CVex+jn7a4nqP7V UgZn1ZJQaS6MLTpyqxri8gCTGOJ3XQckNT/uyXCwyI1t04kwr3cRFa3n4ZxYrdoL8OBh JzIEwhA+t8DVuGI3B3dR9fwP8nHCGrwljDdC4tRpTR8VItB3+zowjFHk818RvXpAGYfo Kp5g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=klqjbF7D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x10-20020a1709060a4a00b0088b79ec31edsi13439746ejf.572.2023.02.05.22.36.46; Sun, 05 Feb 2023 22:37:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=klqjbF7D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229582AbjBFGds (ORCPT + 99 others); Mon, 6 Feb 2023 01:33:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36532 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229479AbjBFGdl (ORCPT ); Mon, 6 Feb 2023 01:33:41 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0796E144B1 for ; Sun, 5 Feb 2023 22:33:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675665219; x=1707201219; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=w1FXftS5jBdzB2pFVikf3bBSxICJXTUbCvcdqa4yc7Q=; b=klqjbF7D31Zz38xcjeUGQ56HVQ/4RebHebcfyTwAw00MAy1prOglUAxE mHSeTlPnv4bKla+N1Ea+g7CejpAbSUtTH75/DfaT2trXs8j5++R0wFLLj 804xSpkaWghB8hN2a9m7VHeZ5y3NY1Rc8jEW9ykLuEOF+4x2Y7klSqE8O ueIQazSPGANTrYpNEuecMppr80v/IBSF9ePJcoqZezJ+Ai6Au63Ei/wOT UqOg4K1vEI8MNTsmMQ8oEpJ8xG4kHIa3mULjLuvZGjFt725rc83BqPIDi FYcXjj7lj5lLDGg9RmEc/DaAdM299hnMOpUMpPemYQ/y6w3OPbsr1fIHq Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="330432640" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="330432640" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:33:38 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="659744640" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="659744640" Received: from baoyumen-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.smartont.net) ([10.255.30.227]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:33:33 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Baolin Wang , Zi Yan , Yang Shi , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH -v4 2/9] migrate_pages: separate hugetlb folios migration Date: Mon, 6 Feb 2023 14:33:06 +0800 Message-Id: <20230206063313.635011-3-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230206063313.635011-1-ying.huang@intel.com> References: <20230206063313.635011-1-ying.huang@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757062554193121673?= X-GMAIL-MSGID: =?utf-8?q?1757062554193121673?= This is a preparation patch to batch the folio unmapping and moving for the non-hugetlb folios. Based on that we can batch the TLB shootdown during the folio migration and make it possible to use some hardware accelerator for the folio copying. In this patch the hugetlb folios and non-hugetlb folios migration is separated in migrate_pages() to make it easy to change the non-hugetlb folios migration implementation. Signed-off-by: "Huang, Ying" Reviewed-by: Baolin Wang Cc: Zi Yan Cc: Yang Shi Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin Cc: Minchan Kim Cc: Mike Kravetz Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Xin Hao --- mm/migrate.c | 141 +++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 119 insertions(+), 22 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index ef388a9e4747..be7f37523463 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1396,6 +1396,8 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f return rc; } +#define NR_MAX_MIGRATE_PAGES_RETRY 10 + struct migrate_pages_stats { int nr_succeeded; /* Normal and large folios migrated successfully, in units of base pages */ @@ -1406,6 +1408,95 @@ struct migrate_pages_stats { int nr_thp_split; /* THP split before migrating */ }; +/* + * Returns the number of hugetlb folios that were not migrated, or an error code + * after NR_MAX_MIGRATE_PAGES_RETRY attempts or if no hugetlb folios are movable + * any more because the list has become empty or no retryable hugetlb folios + * exist any more. It is caller's responsibility to call putback_movable_pages() + * only if ret != 0. + */ +static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, + free_page_t put_new_page, unsigned long private, + enum migrate_mode mode, int reason, + struct migrate_pages_stats *stats, + struct list_head *ret_folios) +{ + int retry = 1; + int nr_failed = 0; + int nr_retry_pages = 0; + int pass = 0; + struct folio *folio, *folio2; + int rc, nr_pages; + + for (pass = 0; pass < NR_MAX_MIGRATE_PAGES_RETRY && retry; pass++) { + retry = 0; + nr_retry_pages = 0; + + list_for_each_entry_safe(folio, folio2, from, lru) { + if (!folio_test_hugetlb(folio)) + continue; + + nr_pages = folio_nr_pages(folio); + + cond_resched(); + + rc = unmap_and_move_huge_page(get_new_page, + put_new_page, private, + &folio->page, pass > 2, mode, + reason, ret_folios); + /* + * The rules are: + * Success: hugetlb folio will be put back + * -EAGAIN: stay on the from list + * -ENOMEM: stay on the from list + * -ENOSYS: stay on the from list + * Other errno: put on ret_folios list + */ + switch(rc) { + case -ENOSYS: + /* Hugetlb migration is unsupported */ + nr_failed++; + stats->nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, ret_folios); + break; + case -ENOMEM: + /* + * When memory is low, don't bother to try to migrate + * other folios, just exit. + */ + stats->nr_failed_pages += nr_pages + nr_retry_pages; + return -ENOMEM; + case -EAGAIN: + retry++; + nr_retry_pages += nr_pages; + break; + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + break; + default: + /* + * Permanent failure (-EBUSY, etc.): + * unlike -EAGAIN case, the failed folio is + * removed from migration folio list and not + * retried in the next outer loop. + */ + nr_failed++; + stats->nr_failed_pages += nr_pages; + break; + } + } + } + /* + * nr_failed is number of hugetlb folios failed to be migrated. After + * NR_MAX_MIGRATE_PAGES_RETRY attempts, give up and count retried hugetlb + * folios as failed. + */ + nr_failed += retry; + stats->nr_failed_pages += nr_retry_pages; + + return nr_failed; +} + /* * migrate_pages - migrate the folios specified in a list, to the free folios * supplied as the target for the page migration @@ -1422,10 +1513,10 @@ struct migrate_pages_stats { * @ret_succeeded: Set to the number of folios migrated successfully if * the caller passes a non-NULL pointer. * - * The function returns after 10 attempts or if no folios are movable any more - * because the list has become empty or no retryable folios exist any more. - * It is caller's responsibility to call putback_movable_pages() to return folios - * to the LRU or free list only if ret != 0. + * The function returns after NR_MAX_MIGRATE_PAGES_RETRY attempts or if no folios + * are movable any more because the list has become empty or no retryable folios + * exist any more. It is caller's responsibility to call putback_movable_pages() + * only if ret != 0. * * Returns the number of {normal folio, large folio, hugetlb} that were not * migrated, or an error code. The number of large folio splits will be @@ -1439,7 +1530,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, int retry = 1; int large_retry = 1; int thp_retry = 1; - int nr_failed = 0; + int nr_failed; int nr_retry_pages = 0; int nr_large_failed = 0; int pass = 0; @@ -1456,38 +1547,45 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, trace_mm_migrate_pages_start(mode, reason); memset(&stats, 0, sizeof(stats)); + rc = migrate_hugetlbs(from, get_new_page, put_new_page, private, mode, reason, + &stats, &ret_folios); + if (rc < 0) + goto out; + nr_failed = rc; + split_folio_migration: - for (pass = 0; pass < 10 && (retry || large_retry); pass++) { + for (pass = 0; + pass < NR_MAX_MIGRATE_PAGES_RETRY && (retry || large_retry); + pass++) { retry = 0; large_retry = 0; thp_retry = 0; nr_retry_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { + /* Retried hugetlb folios will be kept in list */ + if (folio_test_hugetlb(folio)) { + list_move_tail(&folio->lru, &ret_folios); + continue; + } + /* * Large folio statistics is based on the source large * folio. Capture required information that might get * lost during migration. */ - is_large = folio_test_large(folio) && !folio_test_hugetlb(folio); + is_large = folio_test_large(folio); is_thp = is_large && folio_test_pmd_mappable(folio); nr_pages = folio_nr_pages(folio); + cond_resched(); - if (folio_test_hugetlb(folio)) - rc = unmap_and_move_huge_page(get_new_page, - put_new_page, private, - &folio->page, pass > 2, mode, - reason, - &ret_folios); - else - rc = unmap_and_move(get_new_page, put_new_page, - private, folio, pass > 2, mode, - reason, &ret_folios); + rc = unmap_and_move(get_new_page, put_new_page, + private, folio, pass > 2, mode, + reason, &ret_folios); /* * The rules are: - * Success: non hugetlb folio will be freed, hugetlb - * folio will be put back + * Success: folio will be freed * -EAGAIN: stay on the from list * -ENOMEM: stay on the from list * -ENOSYS: stay on the from list @@ -1514,7 +1612,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, stats.nr_thp_split += is_thp; break; } - /* Hugetlb migration is unsupported */ } else if (!no_split_folio_counting) { nr_failed++; } @@ -1608,8 +1705,8 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (!list_empty(&split_folios)) { /* - * Move non-migrated folios (after 10 retries) to ret_folios - * to avoid migrating them again. + * Move non-migrated folios (after NR_MAX_MIGRATE_PAGES_RETRY + * retries) to ret_folios to avoid migrating them again. */ list_splice_init(from, &ret_folios); list_splice_init(&split_folios, from); From patchwork Mon Feb 6 06:33:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 53015 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2082049wrn; Sun, 5 Feb 2023 22:35:47 -0800 (PST) X-Google-Smtp-Source: AK7set8rzVdraq9Rip3ds9yK+zqTwuX+bD2+BfsEQQlmzZuuZiIS+raF5AmUKkr+E3CMQA9TFwcu X-Received: by 2002:a17:90b:3013:b0:22c:926c:bd0e with SMTP id hg19-20020a17090b301300b0022c926cbd0emr20421270pjb.24.1675665346901; Sun, 05 Feb 2023 22:35:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675665346; cv=none; d=google.com; s=arc-20160816; b=UTYMWJ8IIzbPujKgPfcs9dIbBMxEzrKqFacgCSJa5eZjo9NZYgCHDKPBUp6fycZnrv r6WTHsAyMKN3bcKXUCJrrucOEznRi8lziHzeiojplSJv5L9TWx5qQdLl/7aKSvqOlx/y z7Rd5ju1u8+bmQX7bCRQHufYXVDOU6HCUHHcJuZQLeya1RehpeLw+Ri6aJRQg6q/XLZK Wc4QPg55RJQ+l2tn2JACG6Pasm2agXdoMUAx6yurQRWjb6UMRUDhRggvhd4Ae+vx87gT 0lVnFIC5+DiyiCoyUDvR0YesHTeewGzy47+hKaZ4d7B8sJdE9eZthCmGkyzdROeYl0xF 3pMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=IHHfelv6rK82PGXo2GlZw0b94js0SQCXvYQmoU8xQ1c=; b=BYPpAjQndrkW25ipF5iSGzz4OAugtnrfTNey4OJSNgVUbdnps/DD0zA0bKQiK0JZL6 D01mZNPehczezBT2NIyoIRiExvMp4g+2VfVfnd6dPOXIRoAYW0UX/Q4UiYMrDqYwigWr 0PiLBmzSAj45signiYcZsz5wIQ7YC1pcBlqqDXP12B5DCGNb56y7hTVunk7HHjoplwVP DPu+OOcqrIjGOcsDPfWgi4UFBBoWu/Yku2nQQSu1iXxZ43egOvxbTfSCS7bxuohIhRWG HjhjOA37VqYHkAGJ2AeqNT3C5SPijuc+c2QR4d1l6JQ96Nf0EBXAI9bsgwt8DdNzV/kL t7aw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=OmygnWGC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t12-20020a17090a6a0c00b0022c4e0720b6si1973011pjj.115.2023.02.05.22.35.34; Sun, 05 Feb 2023 22:35:46 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=OmygnWGC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229638AbjBFGdv (ORCPT + 99 others); Mon, 6 Feb 2023 01:33:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36668 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229614AbjBFGdr (ORCPT ); Mon, 6 Feb 2023 01:33:47 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 867B9193F6 for ; Sun, 5 Feb 2023 22:33:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675665224; x=1707201224; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=E+ufcmCoz9FarLVwteZ2cUHMggV+b5Nv4b96FC4hzk8=; b=OmygnWGCHz2iRjFJp66fUvGEJ3x5LguCuMSlLk+CNiCs4/a4Dgalo0OB T/BVT078FsGdyRN+a/hLIh1FqLTNJBGwSyJF3D9MDsNG45jCcFfGeCQ9Q ukW+GumWSMVT7VmyuWx1jkdRsN+Le9h4k0RfAKF7dKMFoQqAU71Ui75F5 Jtr2NtQD/cGlTOk8gqoDntpTlARVokGpQshiPvWZQFANPbTlfZ0XJwGaV JeaJSIUigV2aPcnRep0lZA9OW8xT5NUI3/P/yI4wS8HJJWFbkKV0Ey2Kx iH87Z8XxHshBcM12yQnmSU/evK5HnV77fnLKdY9NLQDu9topcRuyCH7u3 w==; X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="330432662" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="330432662" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:33:43 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="659744666" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="659744666" Received: from baoyumen-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.smartont.net) ([10.255.30.227]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:33:38 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Baolin Wang , Zi Yan , Yang Shi , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH -v4 3/9] migrate_pages: restrict number of pages to migrate in batch Date: Mon, 6 Feb 2023 14:33:07 +0800 Message-Id: <20230206063313.635011-4-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230206063313.635011-1-ying.huang@intel.com> References: <20230206063313.635011-1-ying.huang@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757062466897924836?= X-GMAIL-MSGID: =?utf-8?q?1757062466897924836?= This is a preparation patch to batch the folio unmapping and moving for non-hugetlb folios. If we had batched the folio unmapping, all folios to be migrated would be unmapped before copying the contents and flags of the folios. If the folios that were passed to migrate_pages() were too many in unit of pages, the execution of the processes would be stopped for too long time, thus too long latency. For example, migrate_pages() syscall will call migrate_pages() with all folios of a process. To avoid this possible issue, in this patch, we restrict the number of pages to be migrated to be no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration. Signed-off-by: "Huang, Ying" Reviewed-by: Baolin Wang Cc: Zi Yan Cc: Yang Shi Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin Cc: Minchan Kim Cc: Mike Kravetz Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Xin Hao --- mm/migrate.c | 174 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 106 insertions(+), 68 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index be7f37523463..9a667039c34c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1396,6 +1396,11 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f return rc; } +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#define NR_MAX_BATCHED_MIGRATION HPAGE_PMD_NR +#else +#define NR_MAX_BATCHED_MIGRATION 512 +#endif #define NR_MAX_MIGRATE_PAGES_RETRY 10 struct migrate_pages_stats { @@ -1497,40 +1502,15 @@ static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, return nr_failed; } -/* - * migrate_pages - migrate the folios specified in a list, to the free folios - * supplied as the target for the page migration - * - * @from: The list of folios to be migrated. - * @get_new_page: The function used to allocate free folios to be used - * as the target of the folio migration. - * @put_new_page: The function used to free target folios if migration - * fails, or NULL if no special handling is necessary. - * @private: Private data to be passed on to get_new_page() - * @mode: The migration mode that specifies the constraints for - * folio migration, if any. - * @reason: The reason for folio migration. - * @ret_succeeded: Set to the number of folios migrated successfully if - * the caller passes a non-NULL pointer. - * - * The function returns after NR_MAX_MIGRATE_PAGES_RETRY attempts or if no folios - * are movable any more because the list has become empty or no retryable folios - * exist any more. It is caller's responsibility to call putback_movable_pages() - * only if ret != 0. - * - * Returns the number of {normal folio, large folio, hugetlb} that were not - * migrated, or an error code. The number of large folio splits will be - * considered as the number of non-migrated large folio, no matter how many - * split folios of the large folio are migrated successfully. - */ -int migrate_pages(struct list_head *from, new_page_t get_new_page, +static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, free_page_t put_new_page, unsigned long private, - enum migrate_mode mode, int reason, unsigned int *ret_succeeded) + enum migrate_mode mode, int reason, struct list_head *ret_folios, + struct migrate_pages_stats *stats) { int retry = 1; int large_retry = 1; int thp_retry = 1; - int nr_failed; + int nr_failed = 0; int nr_retry_pages = 0; int nr_large_failed = 0; int pass = 0; @@ -1538,20 +1518,9 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, bool is_thp = false; struct folio *folio, *folio2; int rc, nr_pages; - LIST_HEAD(ret_folios); LIST_HEAD(split_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); bool no_split_folio_counting = false; - struct migrate_pages_stats stats; - - trace_mm_migrate_pages_start(mode, reason); - - memset(&stats, 0, sizeof(stats)); - rc = migrate_hugetlbs(from, get_new_page, put_new_page, private, mode, reason, - &stats, &ret_folios); - if (rc < 0) - goto out; - nr_failed = rc; split_folio_migration: for (pass = 0; @@ -1563,12 +1532,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_retry_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { - /* Retried hugetlb folios will be kept in list */ - if (folio_test_hugetlb(folio)) { - list_move_tail(&folio->lru, &ret_folios); - continue; - } - /* * Large folio statistics is based on the source large * folio. Capture required information that might get @@ -1582,15 +1545,14 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, rc = unmap_and_move(get_new_page, put_new_page, private, folio, pass > 2, mode, - reason, &ret_folios); + reason, ret_folios); /* * The rules are: * Success: folio will be freed * -EAGAIN: stay on the from list * -ENOMEM: stay on the from list * -ENOSYS: stay on the from list - * Other errno: put on ret_folios list then splice to - * from list + * Other errno: put on ret_folios list */ switch(rc) { /* @@ -1607,17 +1569,17 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, /* Large folio migration is unsupported */ if (is_large) { nr_large_failed++; - stats.nr_thp_failed += is_thp; + stats->nr_thp_failed += is_thp; if (!try_split_folio(folio, &split_folios)) { - stats.nr_thp_split += is_thp; + stats->nr_thp_split += is_thp; break; } } else if (!no_split_folio_counting) { nr_failed++; } - stats.nr_failed_pages += nr_pages; - list_move_tail(&folio->lru, &ret_folios); + stats->nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, ret_folios); break; case -ENOMEM: /* @@ -1626,13 +1588,13 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (is_large) { nr_large_failed++; - stats.nr_thp_failed += is_thp; + stats->nr_thp_failed += is_thp; /* Large folio NUMA faulting doesn't split to retry. */ if (!nosplit) { int ret = try_split_folio(folio, &split_folios); if (!ret) { - stats.nr_thp_split += is_thp; + stats->nr_thp_split += is_thp; break; } else if (reason == MR_LONGTERM_PIN && ret == -EAGAIN) { @@ -1650,17 +1612,17 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_failed++; } - stats.nr_failed_pages += nr_pages + nr_retry_pages; + stats->nr_failed_pages += nr_pages + nr_retry_pages; /* * There might be some split folios of fail-to-migrate large - * folios left in split_folios list. Move them back to migration + * folios left in split_folios list. Move them to ret_folios * list so that they could be put back to the right list by * the caller otherwise the folio refcnt will be leaked. */ - list_splice_init(&split_folios, from); + list_splice_init(&split_folios, ret_folios); /* nr_failed isn't updated for not used */ nr_large_failed += large_retry; - stats.nr_thp_failed += thp_retry; + stats->nr_thp_failed += thp_retry; goto out; case -EAGAIN: if (is_large) { @@ -1672,8 +1634,8 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_retry_pages += nr_pages; break; case MIGRATEPAGE_SUCCESS: - stats.nr_succeeded += nr_pages; - stats.nr_thp_succeeded += is_thp; + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; break; default: /* @@ -1684,20 +1646,20 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (is_large) { nr_large_failed++; - stats.nr_thp_failed += is_thp; + stats->nr_thp_failed += is_thp; } else if (!no_split_folio_counting) { nr_failed++; } - stats.nr_failed_pages += nr_pages; + stats->nr_failed_pages += nr_pages; break; } } } nr_failed += retry; nr_large_failed += large_retry; - stats.nr_thp_failed += thp_retry; - stats.nr_failed_pages += nr_retry_pages; + stats->nr_thp_failed += thp_retry; + stats->nr_failed_pages += nr_retry_pages; /* * Try to migrate split folios of fail-to-migrate large folios, no * nr_failed counting in this round, since all split folios of a @@ -1708,7 +1670,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, * Move non-migrated folios (after NR_MAX_MIGRATE_PAGES_RETRY * retries) to ret_folios to avoid migrating them again. */ - list_splice_init(from, &ret_folios); + list_splice_init(from, ret_folios); list_splice_init(&split_folios, from); no_split_folio_counting = true; retry = 1; @@ -1716,6 +1678,82 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, } rc = nr_failed + nr_large_failed; +out: + return rc; +} + +/* + * migrate_pages - migrate the folios specified in a list, to the free folios + * supplied as the target for the page migration + * + * @from: The list of folios to be migrated. + * @get_new_page: The function used to allocate free folios to be used + * as the target of the folio migration. + * @put_new_page: The function used to free target folios if migration + * fails, or NULL if no special handling is necessary. + * @private: Private data to be passed on to get_new_page() + * @mode: The migration mode that specifies the constraints for + * folio migration, if any. + * @reason: The reason for folio migration. + * @ret_succeeded: Set to the number of folios migrated successfully if + * the caller passes a non-NULL pointer. + * + * The function returns after NR_MAX_MIGRATE_PAGES_RETRY attempts or if no folios + * are movable any more because the list has become empty or no retryable folios + * exist any more. It is caller's responsibility to call putback_movable_pages() + * only if ret != 0. + * + * Returns the number of {normal folio, large folio, hugetlb} that were not + * migrated, or an error code. The number of large folio splits will be + * considered as the number of non-migrated large folio, no matter how many + * split folios of the large folio are migrated successfully. + */ +int migrate_pages(struct list_head *from, new_page_t get_new_page, + free_page_t put_new_page, unsigned long private, + enum migrate_mode mode, int reason, unsigned int *ret_succeeded) +{ + int rc, rc_gather; + int nr_pages; + struct folio *folio, *folio2; + LIST_HEAD(folios); + LIST_HEAD(ret_folios); + struct migrate_pages_stats stats; + + trace_mm_migrate_pages_start(mode, reason); + + memset(&stats, 0, sizeof(stats)); + + rc_gather = migrate_hugetlbs(from, get_new_page, put_new_page, private, + mode, reason, &stats, &ret_folios); + if (rc_gather < 0) + goto out; +again: + nr_pages = 0; + list_for_each_entry_safe(folio, folio2, from, lru) { + /* Retried hugetlb folios will be kept in list */ + if (folio_test_hugetlb(folio)) { + list_move_tail(&folio->lru, &ret_folios); + continue; + } + + nr_pages += folio_nr_pages(folio); + if (nr_pages > NR_MAX_BATCHED_MIGRATION) + break; + } + if (nr_pages > NR_MAX_BATCHED_MIGRATION) + list_cut_before(&folios, from, &folio->lru); + else + list_splice_init(from, &folios); + rc = migrate_pages_batch(&folios, get_new_page, put_new_page, private, + mode, reason, &ret_folios, &stats); + list_splice_tail_init(&folios, &ret_folios); + if (rc < 0) { + rc_gather = rc; + goto out; + } + rc_gather += rc; + if (!list_empty(from)) + goto again; out: /* * Put the permanent failure folio back to migration list, they @@ -1728,7 +1766,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, * are migrated successfully. */ if (list_empty(from)) - rc = 0; + rc_gather = 0; count_vm_events(PGMIGRATE_SUCCESS, stats.nr_succeeded); count_vm_events(PGMIGRATE_FAIL, stats.nr_failed_pages); @@ -1742,7 +1780,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, if (ret_succeeded) *ret_succeeded = stats.nr_succeeded; - return rc; + return rc_gather; } struct page *alloc_migration_target(struct page *page, unsigned long private) From patchwork Mon Feb 6 06:33:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 53013 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2081901wrn; Sun, 5 Feb 2023 22:35:14 -0800 (PST) X-Google-Smtp-Source: AK7set/zk7Dv4qVyIcwlitVMoAn6AcRaaiNcBSTdGnFHvYIJKHzwS7dwgBWKtp5xlYLvQB25DqfP X-Received: by 2002:a17:90a:19d1:b0:230:a1c5:dc6e with SMTP id 17-20020a17090a19d100b00230a1c5dc6emr5334684pjj.13.1675665314436; Sun, 05 Feb 2023 22:35:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675665314; cv=none; d=google.com; s=arc-20160816; b=Q+PJVdSZtp74+ysWogO/FFeQ4OdgHyG7uDrqBaqay05u1l2clUzs2rF0aNgacRtXFl 4fwYPPfOILA87iSKZ7spZ7JJjC9c5JjLyckjE1eUxRmQV7O6yhTqEBuhIb7RTHa7OPrs IBtmSCnXI0RVBfE50rD3l8ksaqEXV+A8k8eHSBX3bw2IlIc/4lOWmo5/yzWqLedB2a3c ymvUYYZlk2N1z0sobP+jXg2RZ8SvEFcX+v004RAJD6mpXt5xeD50bKVR/qNtjAEi6fj1 Q8CJSZzbpKZG5lRlYH+AmUGNIxWmtxwMR0P1d9TrkgO6Gqiie4sGdbdZcFZvp6H4P0pY lRhQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=6zVVJXnpq/vW46SiufXNLdBJlTQFDfAZrTjpW/WPTas=; b=EBPJgbXKMvqDSmpIHVngAEwYiBnjjMfZBbXoLBiNWbE7BUROA7IR/jKY5+Mb7DBDfC ASxiqAZIp8tYncpj2psPzU3btHFq61FofUtaw4PXFJPIhZtSYIDUhN5LB1OuEkrKZ1V6 qM5TfzwzBICK5OFe9TOytrdc9jP+ORMjLxHwh8ZrCNjWB1c4TEULnvScAGSrShQ/Fm2O hyUuiYn03wWg5g0qwbdMGgenvWWtOgSmfvQYex5vuiyElkVFm428a0qhUrnu6Vo5nvH3 iZuifbkRh2Rd3L5s1vUCdw9edxOrLn+OXgqC+iglyrf1HGoICdvL/rr6X35CBGCDl9gD tNlg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="c/eRc4e5"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x7-20020a17090a8a8700b00230d21f40b9si249796pjn.72.2023.02.05.22.35.02; Sun, 05 Feb 2023 22:35:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="c/eRc4e5"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229628AbjBFGeF (ORCPT + 99 others); Mon, 6 Feb 2023 01:34:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36818 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229646AbjBFGdw (ORCPT ); Mon, 6 Feb 2023 01:33:52 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5148193F2 for ; Sun, 5 Feb 2023 22:33:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675665229; x=1707201229; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=G5ZfaUuXgSxehPG875BTf4HRHOPcGYUMTs8SwIYJqho=; b=c/eRc4e5U0Hp/1Rj7FSmCbHUEVuByAQufYHuCKuKcCqtvsN2uuTSS64D xt1FAiOdaQGOiLD67hceC/WBOzJJQdroOqpONukjLNxsq/QutPYhhZMuN ZCbiCBuzT04MKkcZk+rHZrKrjQQUOwfdGesy61sGKanLzxi9l9mbKn5tS K1dBeNcsCRzlPZKav+nD3ODXbb5vPAQ4fo2d5nnngK5hy6NLuYws8dNxP uzFlmsptSE4ESzMTyHT8EpgyM1pCBrRPsVkXA5I64MvNa1OlHQg7zezO/ ZdZ9UR9Kspst3M+VSxYRAfEmvRDI6zEXNPgDi48GvQRs+vdSjwEMZ9awE A==; X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="330432682" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="330432682" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:33:48 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="659744706" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="659744706" Received: from baoyumen-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.smartont.net) ([10.255.30.227]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:33:43 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Baolin Wang , Zi Yan , Yang Shi , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH -v4 4/9] migrate_pages: split unmap_and_move() to _unmap() and _move() Date: Mon, 6 Feb 2023 14:33:08 +0800 Message-Id: <20230206063313.635011-5-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230206063313.635011-1-ying.huang@intel.com> References: <20230206063313.635011-1-ying.huang@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757062432974010760?= X-GMAIL-MSGID: =?utf-8?q?1757062432974010760?= This is a preparation patch to batch the folio unmapping and moving. In this patch, unmap_and_move() is split to migrate_folio_unmap() and migrate_folio_move(). So, we can batch _unmap() and _move() in different loops later. To pass some information between unmap and move, the original unused dst->mapping and dst->private are used. Signed-off-by: "Huang, Ying" Reviewed-by: Baolin Wang Cc: Zi Yan Cc: Yang Shi Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin Cc: Minchan Kim Cc: Mike Kravetz Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Xin Hao --- include/linux/migrate.h | 1 + mm/migrate.c | 170 ++++++++++++++++++++++++++++++---------- 2 files changed, 130 insertions(+), 41 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 3ef77f52a4f0..7376074f2e1e 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -18,6 +18,7 @@ struct migration_target_control; * - zero on page migration success; */ #define MIGRATEPAGE_SUCCESS 0 +#define MIGRATEPAGE_UNMAP 1 /** * struct movable_operations - Driver page migration diff --git a/mm/migrate.c b/mm/migrate.c index 9a667039c34c..0428449149f4 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1009,11 +1009,53 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, return rc; } -static int __unmap_and_move(struct folio *src, struct folio *dst, +/* + * To record some information during migration, we uses some unused + * fields (mapping and private) of struct folio of the newly allocated + * destination folio. This is safe because nobody is using them + * except us. + */ +static void __migrate_folio_record(struct folio *dst, + unsigned long page_was_mapped, + struct anon_vma *anon_vma) +{ + dst->mapping = (void *)anon_vma; + dst->private = (void *)page_was_mapped; +} + +static void __migrate_folio_extract(struct folio *dst, + int *page_was_mappedp, + struct anon_vma **anon_vmap) +{ + *anon_vmap = (void *)dst->mapping; + *page_was_mappedp = (unsigned long)dst->private; + dst->mapping = NULL; + dst->private = NULL; +} + +/* Cleanup src folio upon migration success */ +static void migrate_folio_done(struct folio *src, + enum migrate_reason reason) +{ + /* + * Compaction can migrate also non-LRU pages which are + * not accounted to NR_ISOLATED_*. They can be recognized + * as __PageMovable + */ + if (likely(!__folio_test_movable(src))) + mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON + + folio_is_file_lru(src), -folio_nr_pages(src)); + + if (reason != MR_MEMORY_FAILURE) + /* We release the page in page_handle_poison. */ + folio_put(src); +} + +static int __migrate_folio_unmap(struct folio *src, struct folio *dst, int force, enum migrate_mode mode) { int rc = -EAGAIN; - bool page_was_mapped = false; + int page_was_mapped = 0; struct anon_vma *anon_vma = NULL; bool is_lru = !__PageMovable(&src->page); @@ -1089,8 +1131,8 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, goto out_unlock; if (unlikely(!is_lru)) { - rc = move_to_new_folio(dst, src, mode); - goto out_unlock_both; + __migrate_folio_record(dst, page_was_mapped, anon_vma); + return MIGRATEPAGE_UNMAP; } /* @@ -1115,11 +1157,42 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, VM_BUG_ON_FOLIO(folio_test_anon(src) && !folio_test_ksm(src) && !anon_vma, src); try_to_migrate(src, 0); - page_was_mapped = true; + page_was_mapped = 1; } - if (!folio_mapped(src)) - rc = move_to_new_folio(dst, src, mode); + if (!folio_mapped(src)) { + __migrate_folio_record(dst, page_was_mapped, anon_vma); + return MIGRATEPAGE_UNMAP; + } + + if (page_was_mapped) + remove_migration_ptes(src, src, false); + +out_unlock_both: + folio_unlock(dst); +out_unlock: + /* Drop an anon_vma reference if we took one */ + if (anon_vma) + put_anon_vma(anon_vma); + folio_unlock(src); +out: + + return rc; +} + +static int __migrate_folio_move(struct folio *src, struct folio *dst, + enum migrate_mode mode) +{ + int rc; + int page_was_mapped = 0; + struct anon_vma *anon_vma = NULL; + bool is_lru = !__PageMovable(&src->page); + + __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); + + rc = move_to_new_folio(dst, src, mode); + if (unlikely(!is_lru)) + goto out_unlock_both; /* * When successful, push dst to LRU immediately: so that if it @@ -1142,12 +1215,10 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, out_unlock_both: folio_unlock(dst); -out_unlock: /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); folio_unlock(src); -out: /* * If migration is successful, decrease refcount of dst, * which will not free the page because new page owner increased @@ -1159,19 +1230,15 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, return rc; } -/* - * Obtain the lock on folio, remove all ptes and migrate the folio - * to the newly allocated folio in dst. - */ -static int unmap_and_move(new_page_t get_new_page, - free_page_t put_new_page, - unsigned long private, struct folio *src, - int force, enum migrate_mode mode, - enum migrate_reason reason, - struct list_head *ret) +/* Obtain the lock on page, remove all ptes. */ +static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, + unsigned long private, struct folio *src, + struct folio **dstp, int force, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) { struct folio *dst; - int rc = MIGRATEPAGE_SUCCESS; + int rc = MIGRATEPAGE_UNMAP; struct page *newpage = NULL; if (!thp_migration_supported() && folio_test_transhuge(src)) @@ -1182,20 +1249,50 @@ static int unmap_and_move(new_page_t get_new_page, folio_clear_active(src); folio_clear_unevictable(src); /* free_pages_prepare() will clear PG_isolated. */ - goto out; + list_del(&src->lru); + migrate_folio_done(src, reason); + return MIGRATEPAGE_SUCCESS; } newpage = get_new_page(&src->page, private); if (!newpage) return -ENOMEM; dst = page_folio(newpage); + *dstp = dst; dst->private = NULL; - rc = __unmap_and_move(src, dst, force, mode); + rc = __migrate_folio_unmap(src, dst, force, mode); + if (rc == MIGRATEPAGE_UNMAP) + return rc; + + /* + * A page that has not been migrated will have kept its + * references and be restored. + */ + /* restore the folio to right list. */ + if (rc != -EAGAIN) + list_move_tail(&src->lru, ret); + + if (put_new_page) + put_new_page(&dst->page, private); + else + folio_put(dst); + + return rc; +} + +/* Migrate the folio to the newly allocated folio in dst. */ +static int migrate_folio_move(free_page_t put_new_page, unsigned long private, + struct folio *src, struct folio *dst, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) +{ + int rc; + + rc = __migrate_folio_move(src, dst, mode); if (rc == MIGRATEPAGE_SUCCESS) set_page_owner_migrate_reason(&dst->page, reason); -out: if (rc != -EAGAIN) { /* * A folio that has been migrated has all references @@ -1211,20 +1308,7 @@ static int unmap_and_move(new_page_t get_new_page, * we want to retry. */ if (rc == MIGRATEPAGE_SUCCESS) { - /* - * Compaction can migrate also non-LRU folios which are - * not accounted to NR_ISOLATED_*. They can be recognized - * as __folio_test_movable - */ - if (likely(!__folio_test_movable(src))) - mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON + - folio_is_file_lru(src), -folio_nr_pages(src)); - - if (reason != MR_MEMORY_FAILURE) - /* - * We release the folio in page_handle_poison. - */ - folio_put(src); + migrate_folio_done(src, reason); } else { if (rc != -EAGAIN) list_add_tail(&src->lru, ret); @@ -1516,7 +1600,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, int pass = 0; bool is_large = false; bool is_thp = false; - struct folio *folio, *folio2; + struct folio *folio, *folio2, *dst = NULL; int rc, nr_pages; LIST_HEAD(split_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); @@ -1543,9 +1627,13 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, cond_resched(); - rc = unmap_and_move(get_new_page, put_new_page, - private, folio, pass > 2, mode, - reason, ret_folios); + rc = migrate_folio_unmap(get_new_page, put_new_page, private, + folio, &dst, pass > 2, mode, + reason, ret_folios); + if (rc == MIGRATEPAGE_UNMAP) + rc = migrate_folio_move(put_new_page, private, + folio, dst, mode, + reason, ret_folios); /* * The rules are: * Success: folio will be freed From patchwork Mon Feb 6 06:33:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 53016 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2082115wrn; Sun, 5 Feb 2023 22:36:04 -0800 (PST) X-Google-Smtp-Source: AK7set/NtKI9w48zQ6QWyVqsUJ6OH7Mm7krbcC+ijpQaAvzVjm5coaQDMn2adg7GSEgkd62fZ2vL X-Received: by 2002:a05:6a20:8e23:b0:be:c7a3:2b91 with SMTP id y35-20020a056a208e2300b000bec7a32b91mr21981214pzj.0.1675665364574; Sun, 05 Feb 2023 22:36:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675665364; cv=none; d=google.com; s=arc-20160816; b=LxpT2k5ePkTbYMgSIVyPf8O1dOU4uwOV5d34t5eW+kRvq8rQYul4NLLUOBXoLqZ8LS sYXL2sfkBhqv33fBO+Gp+hTIrhakMbNW/EpaHL+usP/K46dUOb48iUxQ10QJmjzOq/kj Z0n2UvUkS5X6zyMrJlaHofS0UjApqsdo2BmzOd0y4kmHkC8ZOmrP+/IAdjxznq/8svCn SUEnCpX04ZjvYeuMXg/tG6wDHpc65bSrB6xcrpyzx4+w+HL6XNBRjQnsPFccLyg0IllL oHT+/4Qtn7WZQaLXLRmqvrFPKuimjNuj4sCxocNS6RHOT7XM26fitnTSoaNND+gxf6wt zPPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=mbmQLJ1/0WDHTi7CDhgdIm+aQmLddU6deyPoGnL37qI=; b=fnk64PnYMtSwWMkAIOn8gzWN4ccWeRHTjTfWB8bKPUCuxpMGbdR1t9Sg3/r87seuin YWW7FWBXDs4bupE8+8FnxPt/dAYnB4Hv7Nacq8ifscjKYcvvfqneMh+PXX2I+z3hg2Eq tW44mf9IVFqMFLjUIUGsYtmyuH72OYj6yfpmNdOmi0PP/ClCJx8m0K045EmqtJF5QSxL +LxoN1BCAYBaB1bC9bJFwkvLNjmIb7zQfiH6E0/kkI250tY6vreoWCgWytcRssx53tM9 YJ3VVrYbve4z5Ajld9eGkdDki5FS3Ezr5FyiUYxTu6lOCF5/Rzw/WllpFNFf3Yn42BZB W5QA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=HPz3qUw1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c10-20020a63724a000000b004f2a43099d9si10241401pgn.817.2023.02.05.22.35.52; Sun, 05 Feb 2023 22:36:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=HPz3qUw1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229728AbjBFGeS (ORCPT + 99 others); Mon, 6 Feb 2023 01:34:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37226 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229494AbjBFGeO (ORCPT ); Mon, 6 Feb 2023 01:34:14 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C11D1C334 for ; Sun, 5 Feb 2023 22:33:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675665235; x=1707201235; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=5DhuqTDUtxiYMMe7r09ZIVnr3TsjyvBnwnKzcD7MQTk=; b=HPz3qUw1xklF4nwZjlo2lMXvYAFpcyCbhwz7yO34m5XglCsnDIdzk9CW YLFIrEZACzVxQh/TrLw1wjfMX7n+hn+yK3cxoF8EmvPC47W2uOglt4iuW Sz9i8KoiymjVu8FdC4EzX3DPMoOhe8/0ylro2HgH+yI/+kTEDaDve0jo4 pD+0QE+4/9NyZQlz/fUbNYgL3nK9AafePs7UaTIZFjrRKyJ9ToyKhKd5s tvLNuo298mlnGOtuqnfYG3jE5qrnJFB8SAIe1VUfWOJ0sogE9FXmExl6R qgf3xSqiHfT5IxJ/4kVr1tsN12/vyPYjEsDpKt2hpgEL9gfPE5Kh3x56v w==; X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="330432697" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="330432697" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:33:54 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="659744733" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="659744733" Received: from baoyumen-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.smartont.net) ([10.255.30.227]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:33:48 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin , Minchan Kim , Mike Kravetz Subject: [PATCH -v4 5/9] migrate_pages: batch _unmap and _move Date: Mon, 6 Feb 2023 14:33:09 +0800 Message-Id: <20230206063313.635011-6-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230206063313.635011-1-ying.huang@intel.com> References: <20230206063313.635011-1-ying.huang@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757062485603557922?= X-GMAIL-MSGID: =?utf-8?q?1757062485603557922?= In this patch the _unmap and _move stage of the folio migration is batched. That for, previously, it is, for each folio _unmap() _move() Now, it is, for each folio _unmap() for each folio _move() Based on this, we can batch the TLB flushing and use some hardware accelerator to copy folios between batched _unmap and batched _move stages. Signed-off-by: "Huang, Ying" Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin Cc: Minchan Kim Cc: Mike Kravetz --- mm/migrate.c | 208 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 184 insertions(+), 24 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 0428449149f4..fa7212330cb6 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1033,6 +1033,33 @@ static void __migrate_folio_extract(struct folio *dst, dst->private = NULL; } +/* Restore the source folio to the original state upon failure */ +static void migrate_folio_undo_src(struct folio *src, + int page_was_mapped, + struct anon_vma *anon_vma, + struct list_head *ret) +{ + if (page_was_mapped) + remove_migration_ptes(src, src, false); + /* Drop an anon_vma reference if we took one */ + if (anon_vma) + put_anon_vma(anon_vma); + folio_unlock(src); + list_move_tail(&src->lru, ret); +} + +/* Restore the destination folio to the original state upon failure */ +static void migrate_folio_undo_dst(struct folio *dst, + free_page_t put_new_page, + unsigned long private) +{ + folio_unlock(dst); + if (put_new_page) + put_new_page(&dst->page, private); + else + folio_put(dst); +} + /* Cleanup src folio upon migration success */ static void migrate_folio_done(struct folio *src, enum migrate_reason reason) @@ -1052,7 +1079,7 @@ static void migrate_folio_done(struct folio *src, } static int __migrate_folio_unmap(struct folio *src, struct folio *dst, - int force, enum migrate_mode mode) + int force, bool force_lock, enum migrate_mode mode) { int rc = -EAGAIN; int page_was_mapped = 0; @@ -1079,6 +1106,17 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, if (current->flags & PF_MEMALLOC) goto out; + /* + * We have locked some folios, to avoid deadlock, we cannot + * lock the folio synchronously. Go out to process (and + * unlock) all the locked folios. Then we can lock the folio + * synchronously. + */ + if (!force_lock) { + rc = -EDEADLOCK; + goto out; + } + folio_lock(src); } @@ -1187,10 +1225,20 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, int page_was_mapped = 0; struct anon_vma *anon_vma = NULL; bool is_lru = !__PageMovable(&src->page); + struct list_head *prev; __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); + prev = dst->lru.prev; + list_del(&dst->lru); rc = move_to_new_folio(dst, src, mode); + + if (rc == -EAGAIN) { + list_add(&dst->lru, prev); + __migrate_folio_record(dst, page_was_mapped, anon_vma); + return rc; + } + if (unlikely(!is_lru)) goto out_unlock_both; @@ -1233,7 +1281,7 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, /* Obtain the lock on page, remove all ptes. */ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, unsigned long private, struct folio *src, - struct folio **dstp, int force, + struct folio **dstp, int force, bool force_lock, enum migrate_mode mode, enum migrate_reason reason, struct list_head *ret) { @@ -1261,7 +1309,7 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page *dstp = dst; dst->private = NULL; - rc = __migrate_folio_unmap(src, dst, force, mode); + rc = __migrate_folio_unmap(src, dst, force, force_lock, mode); if (rc == MIGRATEPAGE_UNMAP) return rc; @@ -1270,7 +1318,7 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page * references and be restored. */ /* restore the folio to right list. */ - if (rc != -EAGAIN) + if (rc != -EAGAIN && rc != -EDEADLOCK) list_move_tail(&src->lru, ret); if (put_new_page) @@ -1309,9 +1357,8 @@ static int migrate_folio_move(free_page_t put_new_page, unsigned long private, */ if (rc == MIGRATEPAGE_SUCCESS) { migrate_folio_done(src, reason); - } else { - if (rc != -EAGAIN) - list_add_tail(&src->lru, ret); + } else if (rc != -EAGAIN) { + list_add_tail(&src->lru, ret); if (put_new_page) put_new_page(&dst->page, private); @@ -1591,7 +1638,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, enum migrate_mode mode, int reason, struct list_head *ret_folios, struct migrate_pages_stats *stats) { - int retry = 1; + int retry; int large_retry = 1; int thp_retry = 1; int nr_failed = 0; @@ -1600,13 +1647,19 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, int pass = 0; bool is_large = false; bool is_thp = false; - struct folio *folio, *folio2, *dst = NULL; - int rc, nr_pages; + struct folio *folio, *folio2, *dst = NULL, *dst2; + int rc, rc_saved, nr_pages; LIST_HEAD(split_folios); + LIST_HEAD(unmap_folios); + LIST_HEAD(dst_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); bool no_split_folio_counting = false; + bool force_lock; -split_folio_migration: +retry: + rc_saved = 0; + force_lock = true; + retry = 1; for (pass = 0; pass < NR_MAX_MIGRATE_PAGES_RETRY && (retry || large_retry); pass++) { @@ -1628,16 +1681,15 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, cond_resched(); rc = migrate_folio_unmap(get_new_page, put_new_page, private, - folio, &dst, pass > 2, mode, - reason, ret_folios); - if (rc == MIGRATEPAGE_UNMAP) - rc = migrate_folio_move(put_new_page, private, - folio, dst, mode, - reason, ret_folios); + folio, &dst, pass > 2, force_lock, + mode, reason, ret_folios); /* * The rules are: * Success: folio will be freed + * Unmap: folio will be put on unmap_folios list, + * dst folio put on dst_folios list * -EAGAIN: stay on the from list + * -EDEADLOCK: stay on the from list * -ENOMEM: stay on the from list * -ENOSYS: stay on the from list * Other errno: put on ret_folios list @@ -1672,7 +1724,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, case -ENOMEM: /* * When memory is low, don't bother to try to migrate - * other folios, just exit. + * other folios, move unmapped folios, then exit. */ if (is_large) { nr_large_failed++; @@ -1711,7 +1763,19 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, /* nr_failed isn't updated for not used */ nr_large_failed += large_retry; stats->nr_thp_failed += thp_retry; - goto out; + rc_saved = rc; + if (list_empty(&unmap_folios)) + goto out; + else + goto move; + case -EDEADLOCK: + /* + * The folio cannot be locked for potential deadlock. + * Go move (and unlock) all locked folios. Then we can + * try again. + */ + rc_saved = rc; + goto move; case -EAGAIN: if (is_large) { large_retry++; @@ -1725,6 +1789,15 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, stats->nr_succeeded += nr_pages; stats->nr_thp_succeeded += is_thp; break; + case MIGRATEPAGE_UNMAP: + /* + * We have locked some folios, don't force lock + * to avoid deadlock. + */ + force_lock = false; + list_move_tail(&folio->lru, &unmap_folios); + list_add_tail(&dst->lru, &dst_folios); + break; default: /* * Permanent failure (-EBUSY, etc.): @@ -1748,12 +1821,95 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, nr_large_failed += large_retry; stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; +move: + retry = 1; + for (pass = 0; + pass < NR_MAX_MIGRATE_PAGES_RETRY && (retry || large_retry); + pass++) { + retry = 0; + large_retry = 0; + thp_retry = 0; + nr_retry_pages = 0; + + dst = list_first_entry(&dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { + is_large = folio_test_large(folio); + is_thp = is_large && folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + + cond_resched(); + + rc = migrate_folio_move(put_new_page, private, + folio, dst, mode, + reason, ret_folios); + /* + * The rules are: + * Success: folio will be freed + * -EAGAIN: stay on the unmap_folios list + * Other errno: put on ret_folios list + */ + switch(rc) { + case -EAGAIN: + if (is_large) { + large_retry++; + thp_retry += is_thp; + } else if (!no_split_folio_counting) { + retry++; + } + nr_retry_pages += nr_pages; + break; + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + break; + default: + if (is_large) { + nr_large_failed++; + stats->nr_thp_failed += is_thp; + } else if (!no_split_folio_counting) { + nr_failed++; + } + + stats->nr_failed_pages += nr_pages; + break; + } + dst = dst2; + dst2 = list_next_entry(dst, lru); + } + } + nr_failed += retry; + nr_large_failed += large_retry; + stats->nr_thp_failed += thp_retry; + stats->nr_failed_pages += nr_retry_pages; + + if (rc_saved) + rc = rc_saved; + else + rc = nr_failed + nr_large_failed; +out: + /* Cleanup remaining folios */ + dst = list_first_entry(&dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { + int page_was_mapped = 0; + struct anon_vma *anon_vma = NULL; + + __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); + migrate_folio_undo_src(folio, page_was_mapped, anon_vma, + ret_folios); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, put_new_page, private); + dst = dst2; + dst2 = list_next_entry(dst, lru); + } + /* * Try to migrate split folios of fail-to-migrate large folios, no * nr_failed counting in this round, since all split folios of a * large folio is counted as 1 failure in the first round. */ - if (!list_empty(&split_folios)) { + if (rc >= 0 && !list_empty(&split_folios)) { /* * Move non-migrated folios (after NR_MAX_MIGRATE_PAGES_RETRY * retries) to ret_folios to avoid migrating them again. @@ -1761,12 +1917,16 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, list_splice_init(from, ret_folios); list_splice_init(&split_folios, from); no_split_folio_counting = true; - retry = 1; - goto split_folio_migration; + goto retry; } - rc = nr_failed + nr_large_failed; -out: + /* + * We have unlocked all locked folios, so we can force lock now, let's + * try again. + */ + if (rc == -EDEADLOCK) + goto retry; + return rc; } From patchwork Mon Feb 6 06:33:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 53018 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2082359wrn; Sun, 5 Feb 2023 22:37:11 -0800 (PST) X-Google-Smtp-Source: AK7set+Sbw6vuuLLbp1pPfdhsjwM8vU84aUnJuVqXv+z7NWNr7d86fFZZk0pDI0S92Ph0srCDz8r X-Received: by 2002:a17:903:246:b0:198:a715:d26d with SMTP id j6-20020a170903024600b00198a715d26dmr22056156plh.8.1675665431324; Sun, 05 Feb 2023 22:37:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675665431; cv=none; d=google.com; s=arc-20160816; b=R7dFp9StRNl5H1tr7ZVnrLtn0Q7wwItHZ4x+asJcjbTAyXz9Jjsf0/gNZ/vjFyHx+o V1IpXsMLmo9lN/LTBU+AyDxBX7ipVmWMhTJBQdY4MbdfzcXrngP4I1K0cXMTJGc8G9We 22qy33FM3MBgLTBnT5hXcNuB/c8o0sRFm8zIt2PUviwU4u/JbqslngST0/W6K2i7D+xs fTS6pAkiaHZMI5rq6Sr74TYn1kA1bzx0hJu1oylNMTAGoMF9u4rJJxp+4v7wOuk45/k8 tPuFMxNvDglcvWXa/tl/538uTwIAxaM5K2rwGes+Ehs/EaqPu6Kxa59TYhJo1OZ6KpIx YPQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2CRGX/4cvOKQOeDObEC2cncEGX4sAo+G97rWZFVQ56Q=; b=PCwk3qmYiuwp912/oiOHlGHoN3JTLKJc52TCHdCjdDnUo5PB1zQdhVe5w6my1cIoLb jGxZzl2xCe0OS5k8iwZOyP611vlK2t+DG1KetzlOViRN4eh9wHNmlziaHEhnnHECy8Sj dp7y9w25WPC+dtGOuToJEH27SQSKw04wdU54rrU41dBmWtK6mgammkZ0Tk3Q1qz5U4Ul 0vWixGAT7SRGICZx/uWxSb333mj3o9qm+46VmddwJ1djbogIk2d4FzU14M9cyZ2MJogn q1zYrlOW+OAflT+kumDGlHyiR6SSsOzPH4NPjcx+hzeT8q2UexRQSsgobAArxSgdkDFA xVdA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="jeG//PnL"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f4-20020a17090274c400b00192721d6a97si10333561plt.499.2023.02.05.22.36.59; Sun, 05 Feb 2023 22:37:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="jeG//PnL"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229572AbjBFGeh (ORCPT + 99 others); Mon, 6 Feb 2023 01:34:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229657AbjBFGeW (ORCPT ); Mon, 6 Feb 2023 01:34:22 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B77C14E8B for ; Sun, 5 Feb 2023 22:34:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675665248; x=1707201248; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=vDGXAlWxo4PDOnmZT5tL3p2VWp/8kRlkez6JFp7l83o=; b=jeG//PnLJN2lKUdBmym3SCW7SwEQ3xi2MjMSvF3GF9eFZSEFVqbjqc8I jFK1tXzj7tpsCLn0J8Z5+d57twpAFKl8wdWPPGQF5z1AumL2DvAv0KNV/ n2bj26LTwW79dkiIbgHqFT/nBG+N28FTnvzbOOf1I3tB2qSSjovReqWt6 TqawFqpjIr98Cdjyg4iGOHTU7wIm1+dvTQ7pObTrRA8u7a6nO9vLYBrTh Gmn4l5DCwVPTRUk1BFEC8IYnBNuDpVMu/4ItbOBcTARqmCYrOYuqorpqP CA+ahCB+VFo1B8vm2OqGKEiORwjtEr25ud6w5Mjku2RUhN9MqXyEkDCFw A==; X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="330432725" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="330432725" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:34:00 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="659744756" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="659744756" Received: from baoyumen-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.smartont.net) ([10.255.30.227]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:33:54 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH -v4 6/9] migrate_pages: move migrate_folio_unmap() Date: Mon, 6 Feb 2023 14:33:10 +0800 Message-Id: <20230206063313.635011-7-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230206063313.635011-1-ying.huang@intel.com> References: <20230206063313.635011-1-ying.huang@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757062555527590819?= X-GMAIL-MSGID: =?utf-8?q?1757062555527590819?= Just move the position of the functions. There's no any functionality change. This is to make it easier to review the next patch via putting code near its position in the next patch. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin Cc: Minchan Kim Cc: Mike Kravetz Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/migrate.c | 102 +++++++++++++++++++++++++-------------------------- 1 file changed, 51 insertions(+), 51 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index fa7212330cb6..23eb01cfae4c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1218,6 +1218,57 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, return rc; } +/* Obtain the lock on page, remove all ptes. */ +static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, + unsigned long private, struct folio *src, + struct folio **dstp, int force, bool force_lock, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) +{ + struct folio *dst; + int rc = MIGRATEPAGE_UNMAP; + struct page *newpage = NULL; + + if (!thp_migration_supported() && folio_test_transhuge(src)) + return -ENOSYS; + + if (folio_ref_count(src) == 1) { + /* Folio was freed from under us. So we are done. */ + folio_clear_active(src); + folio_clear_unevictable(src); + /* free_pages_prepare() will clear PG_isolated. */ + list_del(&src->lru); + migrate_folio_done(src, reason); + return MIGRATEPAGE_SUCCESS; + } + + newpage = get_new_page(&src->page, private); + if (!newpage) + return -ENOMEM; + dst = page_folio(newpage); + *dstp = dst; + + dst->private = NULL; + rc = __migrate_folio_unmap(src, dst, force, force_lock, mode); + if (rc == MIGRATEPAGE_UNMAP) + return rc; + + /* + * A page that has not been migrated will have kept its + * references and be restored. + */ + /* restore the folio to right list. */ + if (rc != -EAGAIN && rc != -EDEADLOCK) + list_move_tail(&src->lru, ret); + + if (put_new_page) + put_new_page(&dst->page, private); + else + folio_put(dst); + + return rc; +} + static int __migrate_folio_move(struct folio *src, struct folio *dst, enum migrate_mode mode) { @@ -1278,57 +1329,6 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, return rc; } -/* Obtain the lock on page, remove all ptes. */ -static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, - unsigned long private, struct folio *src, - struct folio **dstp, int force, bool force_lock, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) -{ - struct folio *dst; - int rc = MIGRATEPAGE_UNMAP; - struct page *newpage = NULL; - - if (!thp_migration_supported() && folio_test_transhuge(src)) - return -ENOSYS; - - if (folio_ref_count(src) == 1) { - /* Folio was freed from under us. So we are done. */ - folio_clear_active(src); - folio_clear_unevictable(src); - /* free_pages_prepare() will clear PG_isolated. */ - list_del(&src->lru); - migrate_folio_done(src, reason); - return MIGRATEPAGE_SUCCESS; - } - - newpage = get_new_page(&src->page, private); - if (!newpage) - return -ENOMEM; - dst = page_folio(newpage); - *dstp = dst; - - dst->private = NULL; - rc = __migrate_folio_unmap(src, dst, force, force_lock, mode); - if (rc == MIGRATEPAGE_UNMAP) - return rc; - - /* - * A page that has not been migrated will have kept its - * references and be restored. - */ - /* restore the folio to right list. */ - if (rc != -EAGAIN && rc != -EDEADLOCK) - list_move_tail(&src->lru, ret); - - if (put_new_page) - put_new_page(&dst->page, private); - else - folio_put(dst); - - return rc; -} - /* Migrate the folio to the newly allocated folio in dst. */ static int migrate_folio_move(free_page_t put_new_page, unsigned long private, struct folio *src, struct folio *dst, From patchwork Mon Feb 6 06:33:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 53019 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2082462wrn; Sun, 5 Feb 2023 22:37:34 -0800 (PST) X-Google-Smtp-Source: AK7set9DY3Hqn1dYUmQUVAtF+dMORZMFCopz7y0A8IAMZou8DO3RZsOCjO2iuEs0pgx9FwhXFHHm X-Received: by 2002:a17:90a:316:b0:22c:387a:6842 with SMTP id 22-20020a17090a031600b0022c387a6842mr20295510pje.26.1675665454363; Sun, 05 Feb 2023 22:37:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675665454; cv=none; d=google.com; s=arc-20160816; b=kBXJx4sQQayN6OR9Khlz4mp00FpXNgAYtLiXu3zJz+piqmb63F2eGWIEL9QcJDWXn6 U2l5bSRmyAZw8LY9r5oFSRYnYT8QVdhIpfC0xzopxKHtaNJ+gsUMNczxfquwP+xll/oQ 1A5oxOsw4C9q8KEguL7PcGMJjeadtZ3iTcSYFeZxi2IbYIXO7bfk/x0XeX8rW5dxuHtr xsMKy4GUF4Nn7CCWNSbONstJLpQSNP2Yi27+B2ZF/a53kTQUtBwl422p+6mQfgzzUnEB AJr+ZQEewdQBf0QAgFVuA+7UITPQ/QfpuZZ5DYTOO+87cFcjZkO0fBz3e4Pdwh8YJDoa yh4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CfOZxZhQh8vQmdTck0nZFv1+sFy7RYIre0T7zyrtQIw=; b=E1/qTCs9tALAZRbQiktkVHgWEj+yZ4SJOdFOA7a4V+XVf7LupaVy9OP6KIGAVQq3ek geKksHXAsvw6V6yGk63p4uvXMVWtrh1pJ6IOkEchhjo6e7hD+ra+90EDcCUAHv9100UE l1Pf/ckx+fnltCApXg0PiIFrblMG8VE6Awh3moiVhW8mmViWD0zA1RuI/UmoCbwpGiwn HbpcwJwuW40oJ0zYL1M2n9obk9HoZGXQDA8nLGR05MV9TieJoj7DVtMws7/F2t5YuyMM pMlIpq1JWObuGMJqu+NJSzajV+FXzrEtjCDMHnc9vfCK1Ueq5yJ9AWYi4/Coup0AYo+t R/KQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=eaGea8UD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v3-20020a17090a960300b0022c19e2e734si15266526pjo.151.2023.02.05.22.37.22; Sun, 05 Feb 2023 22:37:34 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=eaGea8UD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229685AbjBFGek (ORCPT + 99 others); Mon, 6 Feb 2023 01:34:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229682AbjBFGeX (ORCPT ); Mon, 6 Feb 2023 01:34:23 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B103B1C31B for ; Sun, 5 Feb 2023 22:34:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675665249; x=1707201249; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xnyyIG4eBpTGks53Pu5vU49x/7nhh4tAaQ/5pqQLi1A=; b=eaGea8UD/imGA9+cDGJ87YK5UQnpoEQVXaQ8mTS67o/r7B2vuLYE544w YcHDLPPbEtI406QpcRSmolOYiBrXFqeQU1PCo3qDbJ2fCYKXwFT5HN6wc /HlpOHR1Vk1Lnddo5KUbXWXUHz4Q/2pAn+G5uYcfCSfUhLnrcKRBbPYOj uuG91Y0RP5JRh9kcYPEVL+kF6DAgRXPzYYtz+g792IsIS6JnAiq0wblOn xJwGzQ8GRs5xy5Z1MfXeFbiAJpDNl+D7TgmCnK8ZcxSpZvmivHTd4M1Sz 5dmP10LfXpqQF/rYieuLiPZhVKo5x9hUJ2QuS3U0x8oi5S9zRJLcdRb8s Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="330432747" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="330432747" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:34:05 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="659744775" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="659744775" Received: from baoyumen-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.smartont.net) ([10.255.30.227]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:34:00 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH -v4 7/9] migrate_pages: share more code between _unmap and _move Date: Mon, 6 Feb 2023 14:33:11 +0800 Message-Id: <20230206063313.635011-8-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230206063313.635011-1-ying.huang@intel.com> References: <20230206063313.635011-1-ying.huang@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757062579599074271?= X-GMAIL-MSGID: =?utf-8?q?1757062579599074271?= This is a code cleanup patch to reduce the duplicated code between the _unmap and _move stages of migrate_pages(). No functionality change is expected. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin Cc: Minchan Kim Cc: Mike Kravetz Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Reviewed-by: Zi Yan --- mm/migrate.c | 203 ++++++++++++++++++++------------------------------- 1 file changed, 81 insertions(+), 122 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 23eb01cfae4c..9378fa2ad4a5 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1037,6 +1037,7 @@ static void __migrate_folio_extract(struct folio *dst, static void migrate_folio_undo_src(struct folio *src, int page_was_mapped, struct anon_vma *anon_vma, + bool locked, struct list_head *ret) { if (page_was_mapped) @@ -1044,16 +1045,20 @@ static void migrate_folio_undo_src(struct folio *src, /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); - folio_unlock(src); - list_move_tail(&src->lru, ret); + if (locked) + folio_unlock(src); + if (ret) + list_move_tail(&src->lru, ret); } /* Restore the destination folio to the original state upon failure */ static void migrate_folio_undo_dst(struct folio *dst, + bool locked, free_page_t put_new_page, unsigned long private) { - folio_unlock(dst); + if (locked) + folio_unlock(dst); if (put_new_page) put_new_page(&dst->page, private); else @@ -1078,13 +1083,42 @@ static void migrate_folio_done(struct folio *src, folio_put(src); } -static int __migrate_folio_unmap(struct folio *src, struct folio *dst, - int force, bool force_lock, enum migrate_mode mode) +/* Obtain the lock on page, remove all ptes. */ +static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, + unsigned long private, struct folio *src, + struct folio **dstp, int force, bool force_lock, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) { + struct folio *dst; int rc = -EAGAIN; + struct page *newpage = NULL; int page_was_mapped = 0; struct anon_vma *anon_vma = NULL; bool is_lru = !__PageMovable(&src->page); + bool locked = false; + bool dst_locked = false; + + if (!thp_migration_supported() && folio_test_transhuge(src)) + return -ENOSYS; + + if (folio_ref_count(src) == 1) { + /* Folio was freed from under us. So we are done. */ + folio_clear_active(src); + folio_clear_unevictable(src); + /* free_pages_prepare() will clear PG_isolated. */ + list_del(&src->lru); + migrate_folio_done(src, reason); + return MIGRATEPAGE_SUCCESS; + } + + newpage = get_new_page(&src->page, private); + if (!newpage) + return -ENOMEM; + dst = page_folio(newpage); + *dstp = dst; + + dst->private = NULL; if (!folio_trylock(src)) { if (!force || mode == MIGRATE_ASYNC) @@ -1119,6 +1153,7 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, folio_lock(src); } + locked = true; if (folio_test_writeback(src)) { /* @@ -1133,10 +1168,10 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, break; default: rc = -EBUSY; - goto out_unlock; + goto out; } if (!force) - goto out_unlock; + goto out; folio_wait_writeback(src); } @@ -1166,7 +1201,8 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, * This is much like races on refcount of oldpage: just don't BUG(). */ if (unlikely(!folio_trylock(dst))) - goto out_unlock; + goto out; + dst_locked = true; if (unlikely(!is_lru)) { __migrate_folio_record(dst, page_was_mapped, anon_vma); @@ -1188,7 +1224,7 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, if (!src->mapping) { if (folio_test_private(src)) { try_to_free_buffers(src); - goto out_unlock_both; + goto out; } } else if (folio_mapped(src)) { /* Establish migration ptes */ @@ -1203,74 +1239,26 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, return MIGRATEPAGE_UNMAP; } - if (page_was_mapped) - remove_migration_ptes(src, src, false); - -out_unlock_both: - folio_unlock(dst); -out_unlock: - /* Drop an anon_vma reference if we took one */ - if (anon_vma) - put_anon_vma(anon_vma); - folio_unlock(src); out: - - return rc; -} - -/* Obtain the lock on page, remove all ptes. */ -static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, - unsigned long private, struct folio *src, - struct folio **dstp, int force, bool force_lock, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) -{ - struct folio *dst; - int rc = MIGRATEPAGE_UNMAP; - struct page *newpage = NULL; - - if (!thp_migration_supported() && folio_test_transhuge(src)) - return -ENOSYS; - - if (folio_ref_count(src) == 1) { - /* Folio was freed from under us. So we are done. */ - folio_clear_active(src); - folio_clear_unevictable(src); - /* free_pages_prepare() will clear PG_isolated. */ - list_del(&src->lru); - migrate_folio_done(src, reason); - return MIGRATEPAGE_SUCCESS; - } - - newpage = get_new_page(&src->page, private); - if (!newpage) - return -ENOMEM; - dst = page_folio(newpage); - *dstp = dst; - - dst->private = NULL; - rc = __migrate_folio_unmap(src, dst, force, force_lock, mode); - if (rc == MIGRATEPAGE_UNMAP) - return rc; - /* * A page that has not been migrated will have kept its * references and be restored. */ /* restore the folio to right list. */ - if (rc != -EAGAIN && rc != -EDEADLOCK) - list_move_tail(&src->lru, ret); + if (rc == -EAGAIN || rc == -EDEADLOCK) + ret = NULL; - if (put_new_page) - put_new_page(&dst->page, private); - else - folio_put(dst); + migrate_folio_undo_src(src, page_was_mapped, anon_vma, locked, ret); + migrate_folio_undo_dst(dst, dst_locked, put_new_page, private); return rc; } -static int __migrate_folio_move(struct folio *src, struct folio *dst, - enum migrate_mode mode) +/* Migrate the folio to the newly allocated folio in dst. */ +static int migrate_folio_move(free_page_t put_new_page, unsigned long private, + struct folio *src, struct folio *dst, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) { int rc; int page_was_mapped = 0; @@ -1283,12 +1271,8 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, list_del(&dst->lru); rc = move_to_new_folio(dst, src, mode); - - if (rc == -EAGAIN) { - list_add(&dst->lru, prev); - __migrate_folio_record(dst, page_was_mapped, anon_vma); - return rc; - } + if (rc) + goto out; if (unlikely(!is_lru)) goto out_unlock_both; @@ -1302,70 +1286,45 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, * unsuccessful, and other cases when a page has been temporarily * isolated from the unevictable LRU: but this case is the easiest. */ - if (rc == MIGRATEPAGE_SUCCESS) { - folio_add_lru(dst); - if (page_was_mapped) - lru_add_drain(); - } + folio_add_lru(dst); + if (page_was_mapped) + lru_add_drain(); if (page_was_mapped) - remove_migration_ptes(src, - rc == MIGRATEPAGE_SUCCESS ? dst : src, false); + remove_migration_ptes(src, dst, false); out_unlock_both: folio_unlock(dst); - /* Drop an anon_vma reference if we took one */ - if (anon_vma) - put_anon_vma(anon_vma); - folio_unlock(src); + set_page_owner_migrate_reason(&dst->page, reason); /* * If migration is successful, decrease refcount of dst, * which will not free the page because new page owner increased * refcounter. */ - if (rc == MIGRATEPAGE_SUCCESS) - folio_put(dst); - - return rc; -} - -/* Migrate the folio to the newly allocated folio in dst. */ -static int migrate_folio_move(free_page_t put_new_page, unsigned long private, - struct folio *src, struct folio *dst, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) -{ - int rc; - - rc = __migrate_folio_move(src, dst, mode); - if (rc == MIGRATEPAGE_SUCCESS) - set_page_owner_migrate_reason(&dst->page, reason); - - if (rc != -EAGAIN) { - /* - * A folio that has been migrated has all references - * removed and will be freed. A folio that has not been - * migrated will have kept its references and be restored. - */ - list_del(&src->lru); - } + folio_put(dst); /* - * If migration is successful, releases reference grabbed during - * isolation. Otherwise, restore the folio to right list unless - * we want to retry. + * A page that has been migrated has all references removed + * and will be freed. */ - if (rc == MIGRATEPAGE_SUCCESS) { - migrate_folio_done(src, reason); - } else if (rc != -EAGAIN) { - list_add_tail(&src->lru, ret); + list_del(&src->lru); + /* Drop an anon_vma reference if we took one */ + if (anon_vma) + put_anon_vma(anon_vma); + folio_unlock(src); + migrate_folio_done(src, reason); - if (put_new_page) - put_new_page(&dst->page, private); - else - folio_put(dst); + return rc; +out: + if (rc == -EAGAIN) { + list_add(&dst->lru, prev); + __migrate_folio_record(dst, page_was_mapped, anon_vma); + return rc; } + migrate_folio_undo_src(src, page_was_mapped, anon_vma, true, ret); + migrate_folio_undo_dst(dst, true, put_new_page, private); + return rc; } @@ -1897,9 +1856,9 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); migrate_folio_undo_src(folio, page_was_mapped, anon_vma, - ret_folios); + true, ret_folios); list_del(&dst->lru); - migrate_folio_undo_dst(dst, put_new_page, private); + migrate_folio_undo_dst(dst, true, put_new_page, private); dst = dst2; dst2 = list_next_entry(dst, lru); } From patchwork Mon Feb 6 06:33:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 53020 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2082467wrn; Sun, 5 Feb 2023 22:37:36 -0800 (PST) X-Google-Smtp-Source: AK7set+4TvBUj7DWDEYMgu8PpfKKhOEQG+yfI+xReMfKo6QGYLkPIyBjXCw/jAVwxITgUJtR6k78 X-Received: by 2002:a05:6a20:3944:b0:bf:bcfb:1fc6 with SMTP id r4-20020a056a20394400b000bfbcfb1fc6mr16696744pzg.60.1675665455788; Sun, 05 Feb 2023 22:37:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675665455; cv=none; d=google.com; s=arc-20160816; b=pzFVkZEOE7Qg1t7WeSx4s6bMH4VS77IVJTaGYZBgu+/UiSlUjSpgH/BgQp5hmtNcsq H0edS7dKyWWeDxhTVMvzTWkCrfm+ME3jEQGEOtLCFAFj6TYYYc/DqaS5SNO07bGVKAWh q00pxEjyqcB/oRRYzZM1qm0rH/wfGRuIx0vVYOGxAvfLUlHi2Nf03lYxtErQbKpyUKLp cNRpOI2e+vOOu4+CnSu6HTOyc0mRSaW/oiiEL2xvD9FsYNORJIsBAgOA5yTaeE7VZOHP KhjmL6E4ilSwGPThrybXn9Tg8Yb5JrORW6gtiB6tiI+d29FN1ISlbIUaKCb895RVp+GH /c5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=UH9aUpK4brvbUaQ61RpzxwNgcTTe1J3JqNYarO9pDJg=; b=zAh2WnOIWoZUmYduUsMHEbeE/BTeO1dpZxnPJ2v2I251Uwssba3pAD9n8kmDoVOgs3 BOF0+8iKHVCxnzAMXr1rRox24+6B71pqbYGo7DRJA8qtvBcYvFuRY6Upeg0BoIglfuDl UEuxebNcxgwahfglg6R5kfFSBI9rC8hAq1xTLmR2mALFzeJ1KKeHFdGmv0AYeMlWvv5G MRsff0iMPfm82QqMzpy2OQCzmTAKOH81m/7ydGVc5uB/GpioGE4YDa+fdJgt42V2NKCJ ZntkZ6zm9QDJvQy3TWrbmdtTfL2SRVSaD4GHA7byz2R+Nf3wYNi52xa6LhUlV2QoyFWq Uc/A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="LPuxEb2/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v185-20020a6389c2000000b004f1a86fee0esi12134798pgd.399.2023.02.05.22.37.23; Sun, 05 Feb 2023 22:37:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="LPuxEb2/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229735AbjBFGes (ORCPT + 99 others); Mon, 6 Feb 2023 01:34:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37912 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229692AbjBFGeb (ORCPT ); Mon, 6 Feb 2023 01:34:31 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12FCA1CACC for ; Sun, 5 Feb 2023 22:34:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675665252; x=1707201252; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=b/06ZIkmccm+2w2sDb3OOn+m3OxExo5G5C0UlpBP/5Y=; b=LPuxEb2/IY2BiidQxbkk/x4Y0W8CYm3ypm/2UV3W8dXpokTLZN8PMdGq MndPJ3YTH/NpT/3X2D9VhJA1h6Uc1cQKiCoz7tMk6f32TLm88fw1Fp4vD cqSk0tbyKJW9LpYK8EaTHNNeSaP96FUO2NghTCN5na1UBadnWMcEvwxsH wRpfZATDxUQf3kbFDB+1bKl515tI/qOVHVz9/8mlp0DDKflwjULz4E0ES GvKO815td/4gZSREtRmh3cbOile7utL8NcJu3fvg18q2b7sTydT4tssC3 /URX1pXtduvfsCj+EMK/bC7f2Uiu+bMNR6ZQ3ch8B4ZFnBrTRKVaIxvlI w==; X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="330432771" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="330432771" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:34:11 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="659744793" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="659744793" Received: from baoyumen-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.smartont.net) ([10.255.30.227]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:34:05 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH -v4 8/9] migrate_pages: batch flushing TLB Date: Mon, 6 Feb 2023 14:33:12 +0800 Message-Id: <20230206063313.635011-9-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230206063313.635011-1-ying.huang@intel.com> References: <20230206063313.635011-1-ying.huang@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757062581134144863?= X-GMAIL-MSGID: =?utf-8?q?1757062581134144863?= The TLB flushing will cost quite some CPU cycles during the folio migration in some situations. For example, when migrate a folio of a process with multiple active threads that run on multiple CPUs. After batching the _unmap and _move in migrate_pages(), the TLB flushing can be batched easily with the existing TLB flush batching mechanism. This patch implements that. We use the following test case to test the patch. On a 2-socket Intel server, - Run pmbench memory accessing benchmark - Run `migratepages` to migrate pages of pmbench between node 0 and node 1 back and forth. With the patch, the TLB flushing IPI reduces 99.1% during the test and the number of pages migrated successfully per second increases 291.7%. NOTE: TLB flushing is batched only for normal folios, not for THP folios. Because the overhead of TLB flushing for THP folios is much lower than that for normal folios (about 1/512 on x86 platform). Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin Cc: Minchan Kim Cc: Mike Kravetz Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/migrate.c | 4 +++- mm/rmap.c | 20 +++++++++++++++++--- 2 files changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 9378fa2ad4a5..ca6e2ff02a09 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1230,7 +1230,7 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page /* Establish migration ptes */ VM_BUG_ON_FOLIO(folio_test_anon(src) && !folio_test_ksm(src) && !anon_vma, src); - try_to_migrate(src, 0); + try_to_migrate(src, TTU_BATCH_FLUSH); page_was_mapped = 1; } @@ -1781,6 +1781,8 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; move: + try_to_unmap_flush(); + retry = 1; for (pass = 0; pass < NR_MAX_MIGRATE_PAGES_RETRY && (retry || large_retry); diff --git a/mm/rmap.c b/mm/rmap.c index b616870a09be..2e125f3e462e 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1976,7 +1976,21 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, } else { flush_cache_page(vma, address, pte_pfn(*pvmw.pte)); /* Nuke the page table entry. */ - pteval = ptep_clear_flush(vma, address, pvmw.pte); + if (should_defer_flush(mm, flags)) { + /* + * We clear the PTE but do not flush so potentially + * a remote CPU could still be writing to the folio. + * If the entry was previously clean then the + * architecture must guarantee that a clear->dirty + * transition on a cached TLB entry is written through + * and traps if the PTE is unmapped. + */ + pteval = ptep_get_and_clear(mm, address, pvmw.pte); + + set_tlb_ubc_flush_pending(mm, pte_dirty(pteval)); + } else { + pteval = ptep_clear_flush(vma, address, pvmw.pte); + } } /* Set the dirty flag on the folio now the pte is gone. */ @@ -2148,10 +2162,10 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags) /* * Migration always ignores mlock and only supports TTU_RMAP_LOCKED and - * TTU_SPLIT_HUGE_PMD and TTU_SYNC flags. + * TTU_SPLIT_HUGE_PMD, TTU_SYNC, and TTU_BATCH_FLUSH flags. */ if (WARN_ON_ONCE(flags & ~(TTU_RMAP_LOCKED | TTU_SPLIT_HUGE_PMD | - TTU_SYNC))) + TTU_SYNC | TTU_BATCH_FLUSH))) return; if (folio_is_zone_device(folio) && From patchwork Mon Feb 6 06:33:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 53021 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2082742wrn; Sun, 5 Feb 2023 22:38:39 -0800 (PST) X-Google-Smtp-Source: AK7set/zTCA1VGuvDRodGy7K8z1IaBum1K77AredXVfBnf0R1PZ3zDwgBPlyudNNXfO/NOilKxyV X-Received: by 2002:a17:902:e888:b0:196:2949:3ca7 with SMTP id w8-20020a170902e88800b0019629493ca7mr24760700plg.7.1675665519176; Sun, 05 Feb 2023 22:38:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675665519; cv=none; d=google.com; s=arc-20160816; b=WDni+F3R/48UdKQm49+n4Y7PO7Rdicj0k79GEZIE7aNuloIwzlEJn2eqxtfJOETAwW BwO9A5tSOk2ybGze6kG/g6Ie9d0BExG2Hau/lGwF/2SOnnCsT4FA6GW8EH9O9lXttoWK /z5HLuS01TY4s3cdu747iOHxd21yMMld1beO/35HBGHk36/7X9hJ0ZbMS8EmOt67gslb MmjrCsS+z8sChclUzw3zh3jfNKdn78ePj9Amdq29uHEyckU2N8Ne5wFKreXuOugbQPw2 GwwdDNVRN4atCOMq9wQ7fjehOfMn/jsTUD76zbGE/WIkB9IUFdtlOsw53T1Ki3S6Ye16 FEnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=W1e2A+EMS/OuzA6fH+/0VR0gikiUbJ6gqtwKjant+Ik=; b=NiWv/IEGiN0LzCcKH/jAjR9pzaCedpjbEJAwYU1NMLv/yw8Mxux8bnrJPzCYA3h4HQ HPu3mFiId8SWJv8grw3Rkdat5RN5MUd0GtTdL+KHRiXaODlIPer3ovk2tQstz8R4uxOs iYdYwAuTh+9VVuYbfoxncAM79vjPTxfNt1MY9esnnXJbCyPANE1hO2QHoMsAT/bcKtRq QlCV31USvv9Wikol7f4Vo57BTY+W0wE/5rCM814xfemce6sA6laGVPlY9M/eyLIwf6fk VPLwVlyJQNP77jBXIAutJh0udpkXdD3qDhFx43MFEr+0DQjELK/JMztvEEmWMdQ41sf1 ACLw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="J746sPy/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z18-20020a170903019200b00188fead2329si12064488plg.135.2023.02.05.22.38.26; Sun, 05 Feb 2023 22:38:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="J746sPy/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229486AbjBFGfB (ORCPT + 99 others); Mon, 6 Feb 2023 01:35:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37900 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229656AbjBFGe7 (ORCPT ); Mon, 6 Feb 2023 01:34:59 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 043E61BAE9 for ; Sun, 5 Feb 2023 22:34:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675665263; x=1707201263; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+KR6MAogWT7k7pvOt7gIOQt6fuAqtcKUncYjRIY0PcU=; b=J746sPy/vPNFUe0DCLAcACfKWfiirnBjMJuDRvcbPlgMo8AECy5OlZv/ ZF2Fy4ETvKKIH93uoXoTjLTlb66CnsCVltSzAirV0YpfHhajSmjD/i8Nv fIYKjRujbhdIm2zbHdtB2F1MorY8rnIYAAR2Fd5vAqw81FfyMk2QNZvye 1wqU9VxJZZwQ11KZ9VPRRHYUQ1QlpNqPJuGZ/6Iw/BC/cQSaiB1tL7JnT W1s/wQ4TWX3yPDSu8zGXlaEitJIsz7hE5JgihOmMeIT82Wu5lhNrx2iju 8V8EBPskKQhel6O5zdLeg8tK1WSx6JM5dmpkUZVjv7HEOTs4O3EU61U/6 A==; X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="330432795" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="330432795" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:34:16 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="659744803" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="659744803" Received: from baoyumen-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.smartont.net) ([10.255.30.227]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2023 22:34:10 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Alistair Popple , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , haoxin , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH -v4 9/9] migrate_pages: move THP/hugetlb migration support check to simplify code Date: Mon, 6 Feb 2023 14:33:13 +0800 Message-Id: <20230206063313.635011-10-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230206063313.635011-1-ying.huang@intel.com> References: <20230206063313.635011-1-ying.huang@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757062647453303828?= X-GMAIL-MSGID: =?utf-8?q?1757062647453303828?= This is a code cleanup patch, no functionality change is expected. After the change, the line number reduces especially in the long migrate_pages_batch(). Signed-off-by: "Huang, Ying" Suggested-by: Alistair Popple Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: haoxin Cc: Minchan Kim Cc: Mike Kravetz Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/migrate.c | 83 +++++++++++++++++++++++----------------------------- 1 file changed, 36 insertions(+), 47 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index ca6e2ff02a09..83d7ec8dfa66 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1099,9 +1099,6 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page bool locked = false; bool dst_locked = false; - if (!thp_migration_supported() && folio_test_transhuge(src)) - return -ENOSYS; - if (folio_ref_count(src) == 1) { /* Folio was freed from under us. So we are done. */ folio_clear_active(src); @@ -1359,16 +1356,6 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, struct anon_vma *anon_vma = NULL; struct address_space *mapping = NULL; - /* - * Migratability of hugepages depends on architectures and their size. - * This check is necessary because some callers of hugepage migration - * like soft offline and memory hotremove don't walk through page - * tables or check whether the hugepage is pmd-based or not before - * kicking migration. - */ - if (!hugepage_migration_supported(page_hstate(hpage))) - return -ENOSYS; - if (folio_ref_count(src) == 1) { /* page was freed from under us. So we are done. */ putback_active_hugepage(hpage); @@ -1535,6 +1522,20 @@ static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, cond_resched(); + /* + * Migratability of hugepages depends on architectures and + * their size. This check is necessary because some callers + * of hugepage migration like soft offline and memory + * hotremove don't walk through page tables or check whether + * the hugepage is pmd-based or not before kicking migration. + */ + if (!hugepage_migration_supported(folio_hstate(folio))) { + nr_failed++; + stats->nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, ret_folios); + continue; + } + rc = unmap_and_move_huge_page(get_new_page, put_new_page, private, &folio->page, pass > 2, mode, @@ -1544,16 +1545,9 @@ static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, * Success: hugetlb folio will be put back * -EAGAIN: stay on the from list * -ENOMEM: stay on the from list - * -ENOSYS: stay on the from list * Other errno: put on ret_folios list */ switch(rc) { - case -ENOSYS: - /* Hugetlb migration is unsupported */ - nr_failed++; - stats->nr_failed_pages += nr_pages; - list_move_tail(&folio->lru, ret_folios); - break; case -ENOMEM: /* * When memory is low, don't bother to try to migrate @@ -1639,6 +1633,28 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, cond_resched(); + /* + * Large folio migration might be unsupported or + * the allocation might be failed so we should retry + * on the same folio with the large folio split + * to normal folios. + * + * Split folios are put in split_folios, and + * we will migrate them after the rest of the + * list is processed. + */ + if (!thp_migration_supported() && is_thp) { + nr_large_failed++; + stats->nr_thp_failed++; + if (!try_split_folio(folio, &split_folios)) { + stats->nr_thp_split++; + continue; + } + stats->nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, ret_folios); + continue; + } + rc = migrate_folio_unmap(get_new_page, put_new_page, private, folio, &dst, pass > 2, force_lock, mode, reason, ret_folios); @@ -1650,36 +1666,9 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, * -EAGAIN: stay on the from list * -EDEADLOCK: stay on the from list * -ENOMEM: stay on the from list - * -ENOSYS: stay on the from list * Other errno: put on ret_folios list */ switch(rc) { - /* - * Large folio migration might be unsupported or - * the allocation could've failed so we should retry - * on the same folio with the large folio split - * to normal folios. - * - * Split folios are put in split_folios, and - * we will migrate them after the rest of the - * list is processed. - */ - case -ENOSYS: - /* Large folio migration is unsupported */ - if (is_large) { - nr_large_failed++; - stats->nr_thp_failed += is_thp; - if (!try_split_folio(folio, &split_folios)) { - stats->nr_thp_split += is_thp; - break; - } - } else if (!no_split_folio_counting) { - nr_failed++; - } - - stats->nr_failed_pages += nr_pages; - list_move_tail(&folio->lru, ret_folios); - break; case -ENOMEM: /* * When memory is low, don't bother to try to migrate