From patchwork Fri Feb 24 14:11:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 61334 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp931049wrd; Fri, 24 Feb 2023 06:14:00 -0800 (PST) X-Google-Smtp-Source: AK7set/ses9XUfU2pKZZJnzzPuwXQXzDCDbuQ8fTKUQOGrl3AyG4OjYe1WamqtjHLee9B4p/YjZK X-Received: by 2002:a17:906:ef8c:b0:8b1:806b:7dbb with SMTP id ze12-20020a170906ef8c00b008b1806b7dbbmr25401945ejb.51.1677248040705; Fri, 24 Feb 2023 06:14:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677248040; cv=none; d=google.com; s=arc-20160816; b=0eim4rUqi6101Uj9tT4wiw1f3suj0mOXyOzrZGaAJVlN4c0EJ7sQWQnQkZtHZiANW+ HoMv1z0VUs3uy7hUcLX3oTssd0EoTGoth0QCDowbP4ooHzsKu0Yu+6GJtxQ++2CM8JVX Dt/isga7CmBe6n0enA9z7xeyMX+UYYYsy2OzFW6vFtBSj8IiJwS9oJrReiS9fLr5gRCV DXL/+acTcQUgzNLRt+ry66n9EOptrvGUe6MyTEDAz3OP4VVRPkSxMCDlpGI5pMv46Ae6 SDl76WL+6rafcnej3oMco+kaWuJlq2AUqpK6Lup5OLxSlV94UZaXB5qHzrEsd/dz3EQV zQ/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=uyB7RmWwqLANhWNVG+Avd86xs6xec7Preg0oko4nHoU=; b=AS7Dz0VS9RkgHzZb+cJbwChHAAo9uApFPFpAUYkrA84lrzyUwjBTO/zAsPrAdvs0+s +glhVad3tqIpvGU52mbhjE49pswcn3S9APpgJYq19Em0hXIMAR6K98jKZSj264Z1fQa+ DOg6tgartwJQkZd3uNmfxcwZx9mbe4Cg8J9q1YjYT2kXyloht1Kfl/nDwjiQ2aqxRSpF VsIIgD9btD94nNLyg7uMerkDI3JcGVC1VX0oUevoq89WgLf878kAkIrVx1EQSKvfcF9r mgdCQuOSEEvfnESSUzUJF0IkCrB6qeruiW+HZTV/MaMRhGrewjTcau+g8YBBWWLsl/ra rpxw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=XKarCqaj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 26-20020a170906029a00b008d17b125784si20961437ejf.30.2023.02.24.06.13.37; Fri, 24 Feb 2023 06:14:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=XKarCqaj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229847AbjBXOMU (ORCPT + 99 others); Fri, 24 Feb 2023 09:12:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33240 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229462AbjBXOMR (ORCPT ); Fri, 24 Feb 2023 09:12:17 -0500 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C35AB10AA4 for ; Fri, 24 Feb 2023 06:12:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677247935; x=1708783935; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=X6tTKRKr9Xy2b+0gbUDPmROHq6FpaNkIJooU65G3dW4=; b=XKarCqajUK6Y+PmjJlHEJIbv8T6VcY3QqAw3L55ytRgE0OLXGHYTnM9D sm37KIx+5KBO5R0n17MWYPGvCzLucg3SZPY4V3m99q6BRBFXJM6ClmKAF Tks6vQFRkWNpYuBjdd0qjG0qHvcJEo31i7KXxXLJjwjNyiipjxFFx3Vug lEGyVSl/Ia3RTH0R66JnMkkPpLo4W8b7XMs7vmVNIduECpNUUDB3Bp6Oj 2O2eb9hIlII9dhyvAVndqUbifhOtjHBTgKqUBQI8lpC1JY/zt/FrJVcQH c31eBpxzICEY6RMKzBLaFKyGi9M/hENK6/MYK5Y6kTLXUZae4F5JzkKlf w==; X-IronPort-AV: E=McAfee;i="6500,9779,10630"; a="332167685" X-IronPort-AV: E=Sophos;i="5.97,324,1669104000"; d="scan'208";a="332167685" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2023 06:12:15 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10630"; a="741684649" X-IronPort-AV: E=Sophos;i="5.97,324,1669104000"; d="scan'208";a="741684649" Received: from bingqili-mobl2.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.255.28.19]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2023 06:12:11 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Hugh Dickins , "Xu, Pengfei" , Christoph Hellwig , Stefan Roesch , Tejun Heo , Xin Hao , Zi Yan , Yang Shi , Baolin Wang , Matthew Wilcox , Mike Kravetz Subject: [PATCH 1/3] migrate_pages: fix deadlock in batched migration Date: Fri, 24 Feb 2023 22:11:43 +0800 Message-Id: <20230224141145.96814-2-ying.huang@intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230224141145.96814-1-ying.huang@intel.com> References: <20230224141145.96814-1-ying.huang@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758722041476195089?= X-GMAIL-MSGID: =?utf-8?q?1758722041476195089?= Two deadlock bugs were reported for the migrate_pages() batching series. Thanks Hugh and Pengfei! For example, in the following deadlock trace snippet, INFO: task kworker/u4:0:9 blocked for more than 147 seconds. Not tainted 6.2.0-rc4-kvm+ #1314 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:kworker/u4:0 state:D stack:0 pid:9 ppid:2 flags:0x00004000 Workqueue: loop4 loop_rootcg_workfn Call Trace: __schedule+0x43b/0xd00 schedule+0x6a/0xf0 io_schedule+0x4a/0x80 folio_wait_bit_common+0x1b5/0x4e0 ? __pfx_wake_page_function+0x10/0x10 __filemap_get_folio+0x73d/0x770 shmem_get_folio_gfp+0x1fd/0xc80 shmem_write_begin+0x91/0x220 generic_perform_write+0x10e/0x2e0 __generic_file_write_iter+0x17e/0x290 ? generic_write_checks+0x12b/0x1a0 generic_file_write_iter+0x97/0x180 ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 do_iter_readv_writev+0x13c/0x210 ? __sanitizer_cov_trace_const_cmp4+0x1a/0x20 do_iter_write+0xf6/0x330 vfs_iter_write+0x46/0x70 loop_process_work+0x723/0xfe0 loop_rootcg_workfn+0x28/0x40 process_one_work+0x3cc/0x8d0 worker_thread+0x66/0x630 ? __pfx_worker_thread+0x10/0x10 kthread+0x153/0x190 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x29/0x50 INFO: task repro:1023 blocked for more than 147 seconds. Not tainted 6.2.0-rc4-kvm+ #1314 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:repro state:D stack:0 pid:1023 ppid:360 flags:0x00004004 Call Trace: __schedule+0x43b/0xd00 schedule+0x6a/0xf0 io_schedule+0x4a/0x80 folio_wait_bit_common+0x1b5/0x4e0 ? compaction_alloc+0x77/0x1150 ? __pfx_wake_page_function+0x10/0x10 folio_wait_bit+0x30/0x40 folio_wait_writeback+0x2e/0x1e0 migrate_pages_batch+0x555/0x1ac0 ? __pfx_compaction_alloc+0x10/0x10 ? __pfx_compaction_free+0x10/0x10 ? __this_cpu_preempt_check+0x17/0x20 ? lock_is_held_type+0xe6/0x140 migrate_pages+0x100e/0x1180 ? __pfx_compaction_free+0x10/0x10 ? __pfx_compaction_alloc+0x10/0x10 compact_zone+0xe10/0x1b50 ? lock_is_held_type+0xe6/0x140 ? check_preemption_disabled+0x80/0xf0 compact_node+0xa3/0x100 ? __sanitizer_cov_trace_const_cmp8+0x1c/0x30 ? _find_first_bit+0x7b/0x90 sysctl_compaction_handler+0x5d/0xb0 proc_sys_call_handler+0x29d/0x420 proc_sys_write+0x2b/0x40 vfs_write+0x3a3/0x780 ksys_write+0xb7/0x180 __x64_sys_write+0x26/0x30 do_syscall_64+0x3b/0x90 entry_SYSCALL_64_after_hwframe+0x72/0xdc RIP: 0033:0x7f3a2471f59d RSP: 002b:00007ffe567f7288 EFLAGS: 00000217 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f3a2471f59d RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000005 RBP: 00007ffe567f72a0 R08: 0000000000000010 R09: 0000000000000010 R10: 0000000000000010 R11: 0000000000000217 R12: 00000000004012e0 R13: 00007ffe567f73e0 R14: 0000000000000000 R15: 0000000000000000 The page migration task has held the lock of the shmem folio A, and is waiting the writeback of the folio B of the file system on the loop block device to complete. While the loop worker task which writes back the folio B is waiting to lock the shmem folio A, because the folio A backs the folio B in the loop device. Thus deadlock is triggered. In general, if we have locked some other folios except the one we are migrating, it's not safe to wait synchronously, for example, to wait the writeback to complete or wait to lock the buffer head. To fix the deadlock, in this patch, we avoid to batch the page migration except for MIGRATE_ASYNC mode. In MIGRATE_ASYNC mode, synchronous waiting is avoided. The fix can be improved further. We will do that as soon as possible. Link: https://lore.kernel.org/linux-mm/87a6c8c-c5c1-67dc-1e32-eb30831d6e3d@google.com/ Link: https://lore.kernel.org/linux-mm/874jrg7kke.fsf@yhuang6-desk2.ccr.corp.intel.com/ Signed-off-by: "Huang, Ying" Reported-by: Hugh Dickins Reported-by: "Xu, Pengfei" Cc: Christoph Hellwig Cc: Stefan Roesch Cc: Tejun Heo Cc: Xin Hao Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Matthew Wilcox Cc: Mike Kravetz --- mm/migrate.c | 62 ++++++++++++++++------------------------------------ 1 file changed, 19 insertions(+), 43 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 37865f85df6d..7ac37dbbf307 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1106,7 +1106,7 @@ static void migrate_folio_done(struct folio *src, /* Obtain the lock on page, remove all ptes. */ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, unsigned long private, struct folio *src, - struct folio **dstp, int force, bool avoid_force_lock, + struct folio **dstp, int force, enum migrate_mode mode, enum migrate_reason reason, struct list_head *ret) { @@ -1157,17 +1157,6 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page if (current->flags & PF_MEMALLOC) goto out; - /* - * We have locked some folios and are going to wait to lock - * this folio. To avoid a potential deadlock, let's bail - * out and not do that. The locked folios will be moved and - * unlocked, then we can wait to lock this folio. - */ - if (avoid_force_lock) { - rc = -EDEADLOCK; - goto out; - } - folio_lock(src); } locked = true; @@ -1247,7 +1236,7 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page /* Establish migration ptes */ VM_BUG_ON_FOLIO(folio_test_anon(src) && !folio_test_ksm(src) && !anon_vma, src); - try_to_migrate(src, TTU_BATCH_FLUSH); + try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0); page_was_mapped = 1; } @@ -1261,7 +1250,7 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page * A folio that has not been unmapped will be restored to * right list unless we want to retry. */ - if (rc == -EAGAIN || rc == -EDEADLOCK) + if (rc == -EAGAIN) ret = NULL; migrate_folio_undo_src(src, page_was_mapped, anon_vma, locked, ret); @@ -1634,11 +1623,9 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, LIST_HEAD(dst_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); bool no_split_folio_counting = false; - bool avoid_force_lock; retry: rc_saved = 0; - avoid_force_lock = false; retry = 1; for (pass = 0; pass < NR_MAX_MIGRATE_PAGES_RETRY && (retry || large_retry); @@ -1683,15 +1670,14 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, } rc = migrate_folio_unmap(get_new_page, put_new_page, private, - folio, &dst, pass > 2, avoid_force_lock, - mode, reason, ret_folios); + folio, &dst, pass > 2, mode, + reason, ret_folios); /* * The rules are: * Success: folio will be freed * Unmap: folio will be put on unmap_folios list, * dst folio put on dst_folios list * -EAGAIN: stay on the from list - * -EDEADLOCK: stay on the from list * -ENOMEM: stay on the from list * Other errno: put on ret_folios list */ @@ -1743,14 +1729,6 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, goto out; else goto move; - case -EDEADLOCK: - /* - * The folio cannot be locked for potential deadlock. - * Go move (and unlock) all locked folios. Then we can - * try again. - */ - rc_saved = rc; - goto move; case -EAGAIN: if (is_large) { large_retry++; @@ -1765,11 +1743,6 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, stats->nr_thp_succeeded += is_thp; break; case MIGRATEPAGE_UNMAP: - /* - * We have locked some folios, don't force lock - * to avoid deadlock. - */ - avoid_force_lock = true; list_move_tail(&folio->lru, &unmap_folios); list_add_tail(&dst->lru, &dst_folios); break; @@ -1894,17 +1867,15 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, */ list_splice_init(from, ret_folios); list_splice_init(&split_folios, from); + /* + * Force async mode to avoid to wait lock or bit when we have + * locked more than one folios. + */ + mode = MIGRATE_ASYNC; no_split_folio_counting = true; goto retry; } - /* - * We have unlocked all locked folios, so we can force lock now, let's - * try again. - */ - if (rc == -EDEADLOCK) - goto retry; - return rc; } @@ -1939,7 +1910,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, enum migrate_mode mode, int reason, unsigned int *ret_succeeded) { int rc, rc_gather; - int nr_pages; + int nr_pages, batch; struct folio *folio, *folio2; LIST_HEAD(folios); LIST_HEAD(ret_folios); @@ -1953,6 +1924,11 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, mode, reason, &stats, &ret_folios); if (rc_gather < 0) goto out; + + if (mode == MIGRATE_ASYNC) + batch = NR_MAX_BATCHED_MIGRATION; + else + batch = 1; again: nr_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { @@ -1963,11 +1939,11 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, } nr_pages += folio_nr_pages(folio); - if (nr_pages > NR_MAX_BATCHED_MIGRATION) + if (nr_pages >= batch) break; } - if (nr_pages > NR_MAX_BATCHED_MIGRATION) - list_cut_before(&folios, from, &folio->lru); + if (nr_pages >= batch) + list_cut_before(&folios, from, &folio2->lru); else list_splice_init(from, &folios); rc = migrate_pages_batch(&folios, get_new_page, put_new_page, private, From patchwork Fri Feb 24 14:11:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 61335 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp931202wrd; Fri, 24 Feb 2023 06:14:15 -0800 (PST) X-Google-Smtp-Source: AK7set84V9lUWDsHRFKSC2k8ODLTtTaz9gt61h2jew/9jQdQ1U5U6a/3wq7BHhhg2q6xq4dGkMmk X-Received: by 2002:a05:6402:1604:b0:4ac:d973:bb2c with SMTP id f4-20020a056402160400b004acd973bb2cmr18192319edv.28.1677248055677; Fri, 24 Feb 2023 06:14:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677248055; cv=none; d=google.com; s=arc-20160816; b=E/8uXwmQDnOxCd3XQHDZpQpjMpb5HzixHpdshhBfj+TO3FPWG8YeyWWj+PCEU+e7Iv Bf8Rac6PClhIVI4Lbnnc3JQVYog3OvPMu1b8+Tjal4oIZrcyx3tjhuG4BMrD+q8Bw/CU r8TX6Ej5V6B5VQedbaQZr5Myg6tufnqFW5USd+ZAzuK5d8wxP6zB7WHNr03UOqQrdfjT kBi0poZf6EL3o0vMz0IiZQnuNCbQ9B9oJoWtd2WNEmfb/AGYFdR9usqSXyBLGLoTklJe Hxg0wniT9wLyJtNgkKDSipN8EkQpTSS9/nyenrlaR2TI1qFavRX88K+Ez54Wm76TE0Ek /AOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QSS91WOSv9o+IhWkWMmJOZWF++6zljuzAGCtpIGB2BY=; b=igM7Eo/vQvHnIOjG0ueVRAO+yU0hG1VdYvFXSwW2MA6twPQkEmaeqR68QmgvjTeEj5 PX5YtPxgXS31Z7PwrKeyEOjNjWEUPclUEd0wEVfBWu5JTQV73mnoSIUTEQiuhEHDGGms GzYbi0iN4jBiNpl7W0X8lnTs+r1dmPx6V08s6zECNCwm14Sr5r6hO+vFQPyP+C37Ci4d bc1a8egrNMVqZ3dXWpmMjIURM3lHzypiDnvh4hSYgorI3NE+KbCn5Q1bkRa12CdbZB09 zrMzg46oE0L3YwtLl/m0GlkeAaGhnCj++deMBTiv75YrPqQXOItG1yUye1miIc4TRpuC xqUQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Oz0JiF4m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id mq48-20020a170907833000b008c9eb8758dfsi17876679ejc.219.2023.02.24.06.13.50; Fri, 24 Feb 2023 06:14:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Oz0JiF4m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229844AbjBXOMW (ORCPT + 99 others); Fri, 24 Feb 2023 09:12:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33320 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229854AbjBXOMU (ORCPT ); Fri, 24 Feb 2023 09:12:20 -0500 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 654C6C64F for ; Fri, 24 Feb 2023 06:12:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677247939; x=1708783939; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=CZw3zTaszsFTuJAllktsjeXaRcVC/hbLzU4WTsAVGdY=; b=Oz0JiF4m1K01jpcNo4vIjiohFTTzVk/ZFFJKMRfnsoSnsha6nXuUKxaQ HGSGdiCYI0V4r46mokMj0rwDzOsbgXnYeSec33ZzChMorcnJBZ6JFRe0N IiygUdHwmQqUqikD/40+WZVdzTx+7k3w+kS+ZqDb4VbwCoR7SWaTlNm3p fLIdf/06mrRVHb77TKNEE9POjoP79on9pB5fMuzUtm4dwanFZkpl/N7Jc 9JrzMwoH5x+qCJaEli8qavSJsm5s6QjZYk5idFFocH5P4RHn8/dJYBOHK m4Cd/RTQgaYmYoxt/xrz52+xj0iH0YDr59aiqZMU2uIyflU8ChONBSgK+ Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10630"; a="332167711" X-IronPort-AV: E=Sophos;i="5.97,324,1669104000"; d="scan'208";a="332167711" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2023 06:12:19 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10630"; a="741684659" X-IronPort-AV: E=Sophos;i="5.97,324,1669104000"; d="scan'208";a="741684659" Received: from bingqili-mobl2.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.255.28.19]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2023 06:12:15 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Hugh Dickins , "Xu, Pengfei" , Christoph Hellwig , Stefan Roesch , Tejun Heo , Xin Hao , Zi Yan , Yang Shi , Baolin Wang , Matthew Wilcox , Mike Kravetz Subject: [PATCH 2/3] migrate_pages: move split folios processing out of migrate_pages_batch() Date: Fri, 24 Feb 2023 22:11:44 +0800 Message-Id: <20230224141145.96814-3-ying.huang@intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230224141145.96814-1-ying.huang@intel.com> References: <20230224141145.96814-1-ying.huang@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758722057101250149?= X-GMAIL-MSGID: =?utf-8?q?1758722057101250149?= To simplify the code logic and reduce the line number. Signed-off-by: "Huang, Ying" Cc: Hugh Dickins Cc: "Xu, Pengfei" Cc: Christoph Hellwig Cc: Stefan Roesch Cc: Tejun Heo Cc: Xin Hao Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Matthew Wilcox Cc: Mike Kravetz Reviewed-by: Baolin Wang --- mm/migrate.c | 76 ++++++++++++++++++---------------------------------- 1 file changed, 26 insertions(+), 50 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 7ac37dbbf307..91198b487e49 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1605,9 +1605,10 @@ static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, free_page_t put_new_page, unsigned long private, enum migrate_mode mode, int reason, struct list_head *ret_folios, - struct migrate_pages_stats *stats) + struct list_head *split_folios, struct migrate_pages_stats *stats, + int nr_pass) { - int retry; + int retry = 1; int large_retry = 1; int thp_retry = 1; int nr_failed = 0; @@ -1617,19 +1618,12 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, bool is_large = false; bool is_thp = false; struct folio *folio, *folio2, *dst = NULL, *dst2; - int rc, rc_saved, nr_pages; - LIST_HEAD(split_folios); + int rc, rc_saved = 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); - bool no_split_folio_counting = false; -retry: - rc_saved = 0; - retry = 1; - for (pass = 0; - pass < NR_MAX_MIGRATE_PAGES_RETRY && (retry || large_retry); - pass++) { + for (pass = 0; pass < nr_pass && (retry || large_retry); pass++) { retry = 0; large_retry = 0; thp_retry = 0; @@ -1660,7 +1654,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, if (!thp_migration_supported() && is_thp) { nr_large_failed++; stats->nr_thp_failed++; - if (!try_split_folio(folio, &split_folios)) { + if (!try_split_folio(folio, split_folios)) { stats->nr_thp_split++; continue; } @@ -1692,7 +1686,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, stats->nr_thp_failed += is_thp; /* Large folio NUMA faulting doesn't split to retry. */ if (!nosplit) { - int ret = try_split_folio(folio, &split_folios); + int ret = try_split_folio(folio, split_folios); if (!ret) { stats->nr_thp_split += is_thp; @@ -1709,18 +1703,11 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, break; } } - } else if (!no_split_folio_counting) { + } else { nr_failed++; } stats->nr_failed_pages += nr_pages + nr_retry_pages; - /* - * There might be some split folios of fail-to-migrate large - * folios left in split_folios list. Move them to ret_folios - * list so that they could be put back to the right list by - * the caller otherwise the folio refcnt will be leaked. - */ - list_splice_init(&split_folios, ret_folios); /* nr_failed isn't updated for not used */ nr_large_failed += large_retry; stats->nr_thp_failed += thp_retry; @@ -1733,7 +1720,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, if (is_large) { large_retry++; thp_retry += is_thp; - } else if (!no_split_folio_counting) { + } else { retry++; } nr_retry_pages += nr_pages; @@ -1756,7 +1743,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, if (is_large) { nr_large_failed++; stats->nr_thp_failed += is_thp; - } else if (!no_split_folio_counting) { + } else { nr_failed++; } @@ -1774,9 +1761,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, try_to_unmap_flush(); retry = 1; - for (pass = 0; - pass < NR_MAX_MIGRATE_PAGES_RETRY && (retry || large_retry); - pass++) { + for (pass = 0; pass < nr_pass && (retry || large_retry); pass++) { retry = 0; large_retry = 0; thp_retry = 0; @@ -1805,7 +1790,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, if (is_large) { large_retry++; thp_retry += is_thp; - } else if (!no_split_folio_counting) { + } else { retry++; } nr_retry_pages += nr_pages; @@ -1818,7 +1803,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, if (is_large) { nr_large_failed++; stats->nr_thp_failed += is_thp; - } else if (!no_split_folio_counting) { + } else { nr_failed++; } @@ -1855,27 +1840,6 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, dst2 = list_next_entry(dst, lru); } - /* - * Try to migrate split folios of fail-to-migrate large folios, no - * nr_failed counting in this round, since all split folios of a - * large folio is counted as 1 failure in the first round. - */ - if (rc >= 0 && !list_empty(&split_folios)) { - /* - * Move non-migrated folios (after NR_MAX_MIGRATE_PAGES_RETRY - * retries) to ret_folios to avoid migrating them again. - */ - list_splice_init(from, ret_folios); - list_splice_init(&split_folios, from); - /* - * Force async mode to avoid to wait lock or bit when we have - * locked more than one folios. - */ - mode = MIGRATE_ASYNC; - no_split_folio_counting = true; - goto retry; - } - return rc; } @@ -1914,6 +1878,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, struct folio *folio, *folio2; LIST_HEAD(folios); LIST_HEAD(ret_folios); + LIST_HEAD(split_folios); struct migrate_pages_stats stats; trace_mm_migrate_pages_start(mode, reason); @@ -1947,12 +1912,23 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, else list_splice_init(from, &folios); rc = migrate_pages_batch(&folios, get_new_page, put_new_page, private, - mode, reason, &ret_folios, &stats); + mode, reason, &ret_folios, &split_folios, &stats, + NR_MAX_MIGRATE_PAGES_RETRY); list_splice_tail_init(&folios, &ret_folios); if (rc < 0) { rc_gather = rc; + list_splice_tail(&split_folios, &ret_folios); goto out; } + if (!list_empty(&split_folios)) { + /* + * Failure isn't counted since all split folios of a large folio + * is counted as 1 failure already. + */ + migrate_pages_batch(&split_folios, get_new_page, put_new_page, private, + MIGRATE_ASYNC, reason, &ret_folios, NULL, &stats, 1); + list_splice_tail_init(&split_folios, &ret_folios); + } rc_gather += rc; if (!list_empty(from)) goto again; From patchwork Fri Feb 24 14:11:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 61343 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp944703wrd; Fri, 24 Feb 2023 06:38:08 -0800 (PST) X-Google-Smtp-Source: AK7set/sm28iJnhMkcJ6c7+dFbjB9icIuiMAQiXeh3t4kzg4hszBXbXPfINYmNNP7hiGaVnm4wWX X-Received: by 2002:a17:906:2c12:b0:881:d1ad:1640 with SMTP id e18-20020a1709062c1200b00881d1ad1640mr25636669ejh.57.1677249487960; Fri, 24 Feb 2023 06:38:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677249487; cv=none; d=google.com; s=arc-20160816; b=NFOcO5XdN/qpypH3bZ4LG+fW35l/xaE30mq0vfNhhK6gWmUD3s2pSOcZnKqKxHTcqC z4djOjiQg1RYx5j47VRg8W21QuD+atuo/pGvrJVn5ApMUOTZPUQljn95L+cGaIYPUx62 raYpu8Tu3ZwNWPRaroiuEJQ+cFcSPQFyMpJqkP2cZ2axmRQnVilVOQUKMPr/UOsxHxPH zkuOoEjaV8mYDn4yP/ygYX8VNpCPhG4tvd9XYHSr4j8QpqCKWUydk2gFlyx5AW/Rv9UP rrn5+ttyRHpzRPnQ7NiL7sDW5+gHS0YkXzI2UwR0NlvUR0vrWJ10CcQpjMsa/hOJoDZV xp4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jtIb4ZWHwjQeYauy02UTZy5dAdRH53XVqaRUoqPshEM=; b=GZK8D5aRastuZIPTTYjIuNFqa6j3wzRsnRh7yRNIYpzNPouD08eabGlJJcU5hjC2vC X/m2d/VvWpiYu5ymtwsS9er4knyXuIRS+pWufyTlz1sIjaDBQHPrE/g1/oDwBMXaKje9 0iMH4HYDg9DcW1hAPwtLuNGxJXl+qmOJHwEzZE+XxdZcEYHJohJ9SDeStL53fsx7DNcs MTvYcXzyemJvQe8pFqhXuFFgxeUG9kS0lslhhT6EdcPkqy1FnmKgYzHMi5ulJ66Nx3+T PdE4uhHALlh97nNHrg8K0nZzw2AwvkKl2iFSnVnvOiTSZi4wD/fyeD/4znjP309dNGmf RLyw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=FVryWDEe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ge7-20020a170907908700b008b23d0dda74si23341696ejb.693.2023.02.24.06.37.44; Fri, 24 Feb 2023 06:38:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=FVryWDEe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229891AbjBXOMh (ORCPT + 99 others); Fri, 24 Feb 2023 09:12:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33704 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229888AbjBXOM2 (ORCPT ); Fri, 24 Feb 2023 09:12:28 -0500 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C87169AF6 for ; Fri, 24 Feb 2023 06:12:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677247943; x=1708783943; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Dxby7yNPzwalm19ztka4+4UYeHwqmfCE4JJYZx6x6X0=; b=FVryWDEezEQUTH30h6wK5CQXpb+8MAttxCZ+JGNizUmjiL8DxHqOj+WP VBjCfKNgok2ybcUXSCGhJZXAXaUKm+ifcki8ZlyXpWRDnmnlTyl46ljXv TTCSq6zXrpkJ22vs/D74lnPAAFHpa7cnrPl8EoG3Ob1w3BOBBdILPbOXW +3xX6swV7hmPWYJl2JR0TZ3PP851aCjtez6HN7kHidufiiYcmckHngEFc UFWPHH6coAqGwLGmezIvatptI+2Jk50CWvrSnGwo5oyn8fWUUIlZvuFOl Mc1vuGdATYGNA0h5gOGvEQn9ByuwQam/2O7ogPRnZ2MGTsQhFXWveq5nt A==; X-IronPort-AV: E=McAfee;i="6500,9779,10630"; a="332167736" X-IronPort-AV: E=Sophos;i="5.97,324,1669104000"; d="scan'208";a="332167736" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2023 06:12:22 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10630"; a="741684670" X-IronPort-AV: E=Sophos;i="5.97,324,1669104000"; d="scan'208";a="741684670" Received: from bingqili-mobl2.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.255.28.19]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Feb 2023 06:12:19 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Hugh Dickins , "Xu, Pengfei" , Christoph Hellwig , Stefan Roesch , Tejun Heo , Xin Hao , Zi Yan , Yang Shi , Baolin Wang , Matthew Wilcox , Mike Kravetz Subject: [PATCH 3/3] migrate_pages: try migrate in batch asynchronously firstly Date: Fri, 24 Feb 2023 22:11:45 +0800 Message-Id: <20230224141145.96814-4-ying.huang@intel.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230224141145.96814-1-ying.huang@intel.com> References: <20230224141145.96814-1-ying.huang@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758723558999793054?= X-GMAIL-MSGID: =?utf-8?q?1758723558999793054?= When we have locked more than one folios, we cannot wait the lock or bit (e.g., page lock, buffer head lock, writeback bit) synchronously. Otherwise deadlock may be triggered. This make it hard to batch the synchronous migration directly. This patch re-enables batching synchronous migration via trying to migrate in batch asynchronously firstly. And any folios that are failed to be migrated asynchronously will be migrated synchronously one by one. Test shows that this can restore the TLB flushing batching performance for synchronous migration effectively. Signed-off-by: "Huang, Ying" Cc: Hugh Dickins Cc: "Xu, Pengfei" Cc: Christoph Hellwig Cc: Stefan Roesch Cc: Tejun Heo Cc: Xin Hao Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Matthew Wilcox Cc: Mike Kravetz Tested-by: Hugh Dickins Reviewed-by: Baolin Wang --- mm/migrate.c | 65 ++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 55 insertions(+), 10 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 91198b487e49..c17ce5ee8d92 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1843,6 +1843,51 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, return rc; } +static int migrate_pages_sync(struct list_head *from, new_page_t get_new_page, + free_page_t put_new_page, unsigned long private, + enum migrate_mode mode, int reason, struct list_head *ret_folios, + struct list_head *split_folios, struct migrate_pages_stats *stats) +{ + int rc, nr_failed = 0; + LIST_HEAD(folios); + struct migrate_pages_stats astats; + + memset(&astats, 0, sizeof(astats)); + /* Try to migrate in batch with MIGRATE_ASYNC mode firstly */ + rc = migrate_pages_batch(from, get_new_page, put_new_page, private, MIGRATE_ASYNC, + reason, &folios, split_folios, &astats, + NR_MAX_MIGRATE_PAGES_RETRY); + stats->nr_succeeded += astats.nr_succeeded; + stats->nr_thp_succeeded += astats.nr_thp_succeeded; + stats->nr_thp_split += astats.nr_thp_split; + if (rc < 0) { + stats->nr_failed_pages += astats.nr_failed_pages; + stats->nr_thp_failed += astats.nr_thp_failed; + list_splice_tail(&folios, ret_folios); + return rc; + } + stats->nr_thp_failed += astats.nr_thp_split; + nr_failed += astats.nr_thp_split; + /* + * Fall back to migrate all failed folios one by one synchronously. All + * failed folios except split THPs will be retried, so their failure + * isn't counted + */ + list_splice_tail_init(&folios, from); + while (!list_empty(from)) { + list_move(from->next, &folios); + rc = migrate_pages_batch(&folios, get_new_page, put_new_page, + private, mode, reason, ret_folios, + split_folios, stats, NR_MAX_MIGRATE_PAGES_RETRY); + list_splice_tail_init(&folios, ret_folios); + if (rc < 0) + return rc; + nr_failed += rc; + } + + return nr_failed; +} + /* * migrate_pages - migrate the folios specified in a list, to the free folios * supplied as the target for the page migration @@ -1874,7 +1919,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, enum migrate_mode mode, int reason, unsigned int *ret_succeeded) { int rc, rc_gather; - int nr_pages, batch; + int nr_pages; struct folio *folio, *folio2; LIST_HEAD(folios); LIST_HEAD(ret_folios); @@ -1890,10 +1935,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, if (rc_gather < 0) goto out; - if (mode == MIGRATE_ASYNC) - batch = NR_MAX_BATCHED_MIGRATION; - else - batch = 1; again: nr_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { @@ -1904,16 +1945,20 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, } nr_pages += folio_nr_pages(folio); - if (nr_pages >= batch) + if (nr_pages >= NR_MAX_BATCHED_MIGRATION) break; } - if (nr_pages >= batch) + if (nr_pages >= NR_MAX_BATCHED_MIGRATION) list_cut_before(&folios, from, &folio2->lru); else list_splice_init(from, &folios); - rc = migrate_pages_batch(&folios, get_new_page, put_new_page, private, - mode, reason, &ret_folios, &split_folios, &stats, - NR_MAX_MIGRATE_PAGES_RETRY); + if (mode == MIGRATE_ASYNC) + rc = migrate_pages_batch(&folios, get_new_page, put_new_page, private, + mode, reason, &ret_folios, &split_folios, &stats, + NR_MAX_MIGRATE_PAGES_RETRY); + else + rc = migrate_pages_sync(&folios, get_new_page, put_new_page, private, + mode, reason, &ret_folios, &split_folios, &stats); list_splice_tail_init(&folios, &ret_folios); if (rc < 0) { rc_gather = rc;