From patchwork Tue Mar 7 05:20:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Stevens X-Patchwork-Id: 65276 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2262491wrd; Mon, 6 Mar 2023 21:57:20 -0800 (PST) X-Google-Smtp-Source: AK7set/GgpRdapQlIDQuXblPKnDe1XClkmZoq3K6LMdQZvnZXlJD3YBsjvuJN8Yc5azSLtnK+Sr5 X-Received: by 2002:aa7:d29a:0:b0:4af:69b8:52ba with SMTP id w26-20020aa7d29a000000b004af69b852bamr12679519edq.25.1678168640121; Mon, 06 Mar 2023 21:57:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678168640; cv=none; d=google.com; s=arc-20160816; b=jaD5xHdIE2Bbd+VOFpqjrFGCMWCACw9GKrF+QaQmi1QvBvSYvdMnaOF706bdGgWDIx ZK70cob+HtOh0KrNTTC/S8I2BaLDQpfah62yt2au+2/eQ0UVEqMgwR3ZRY/fwQCEFV7M Gtycg7Dj2zrboWtY2v+bpLhMSAmO24PBwvkWnJMbKay2QZXxzNVpqx7oCR6TB4EgSf7g Dba6/gVV/ZZ0w71pZOgR7ydbRY96LPqHWWL+sdCMhRwvy6+dPEzpeLg2TKYNj5vdxrUC YNuhjWxNMcotKHO+STpCLPthfvHn7P9/sXcyv3GgxyD3CYt51PixF4UdytiK1FkeHCyk UcgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=xQ8JLh1t+SQa8MFtRCjvqWkhYIqZz1dASMHutsA+ZIA=; b=Zb2oku224NIGtH9ya5LElV78zwpV26C3pmiRLGQNHchk6LVDD44DFWZy9pgWr8ETcl VH0KiMO59W9XhayqCzAr0bpCYoO5rgpx0EvhRQpL1Lg/OPCMQxQ8/HHUTbsCPBGE0i7M TNNjsiUabodFMMY8eRCuqFsyvcw2nPG3zW8CJ5yW+LhFzIfBDyxkeR9AchYoQtamYBwI +Fnc/UiTxe/drXE54/yJQ8md4oOoOTuP2QzW2pPo+VKLCPwjfwFTx9qz0Cx1lugbZa+C YkDDnG8PPLpsYSk88sE8RPUY/W6Wt4U+SZ+XxUPvi2vGBD1xJwtIxTmC5kwXuc6LgkI5 TgeA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=CMO72NJk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a21-20020aa7cf15000000b004acb5d76c54si1867895edy.643.2023.03.06.21.56.56; Mon, 06 Mar 2023 21:57:20 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=CMO72NJk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230328AbjCGFU5 (ORCPT + 99 others); Tue, 7 Mar 2023 00:20:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32900 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230135AbjCGFUy (ORCPT ); Tue, 7 Mar 2023 00:20:54 -0500 Received: from mail-pg1-x532.google.com (mail-pg1-x532.google.com [IPv6:2607:f8b0:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A74453D81 for ; Mon, 6 Mar 2023 21:20:53 -0800 (PST) Received: by mail-pg1-x532.google.com with SMTP id p6so6967879pga.0 for ; Mon, 06 Mar 2023 21:20:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1678166453; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xQ8JLh1t+SQa8MFtRCjvqWkhYIqZz1dASMHutsA+ZIA=; b=CMO72NJkW0Gjo9VKTtSKmbPwn1x80TIoUa7QDwO/WtPJ3SB4S9uE4uELHTnI6cLvRr 3/+EAcw807ziz5ELFf3PmVbEwRg4I0QT0bKP84kgt9Zew2qiUWL34pwhrz3umv9if16g EGPNFHXrFVzRkYWfFwOHqdM1K875EV6Hcx8KQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678166453; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xQ8JLh1t+SQa8MFtRCjvqWkhYIqZz1dASMHutsA+ZIA=; b=tZOOYQSK77+6rlEO5fKx4MEjwGmQe7T5W983TDJ8gGMKEmR99h70EGCBlGlRnUZUFH pazoRBtslr6HRpWfPJ8bqvuKuuGUYwS02ddbw03XFD8q2rTzxWApZ54Ie1pPbbKHYx6z pP+uBOZ/t8D+sUHsJyIb3hBGUIi89gv/6IjzjzUkasBTsSBVP9Yat1oIshSt+0DCgB2G tc4/uroYjC5LgyInBFwCNXAEQWrwA9SZAXU2Q90L+goeGlXynM70QxiUAyokLj2djdfn WPPFPTeMdx9mb3/o4MDFoI0y2c0VAKsYcgKOgzKIK3iBY39c92c2vSJvORXJhwtSCenG 3WLA== X-Gm-Message-State: AO0yUKUyP6WRR24nGggUXl+PkaRbtJODp9XChoZE+R/+aNHmYefjx3ie 95zCOPwUv0ZDcucjfS1UmoOobQ== X-Received: by 2002:a62:1ad4:0:b0:5a8:d407:60f9 with SMTP id a203-20020a621ad4000000b005a8d40760f9mr10448268pfa.29.1678166453057; Mon, 06 Mar 2023 21:20:53 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1f73:9034:ce28:4421]) by smtp.gmail.com with UTF8SMTPSA id a4-20020aa78644000000b005a8ba70315bsm7096096pfo.6.2023.03.06.21.20.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 06 Mar 2023 21:20:52 -0800 (PST) From: David Stevens X-Google-Original-From: David Stevens To: linux-mm@kvack.org, Andrew Morton Cc: Peter Xu , Matthew Wilcox , "Kirill A . Shutemov" , Yang Shi , David Hildenbrand , Hugh Dickins , Jiaqi Yan , linux-kernel@vger.kernel.org, David Stevens Subject: [PATCH v5 1/3] mm/khugepaged: refactor collapse_file control flow Date: Tue, 7 Mar 2023 14:20:34 +0900 Message-Id: <20230307052036.1520708-2-stevensd@google.com> X-Mailer: git-send-email 2.40.0.rc0.216.gc4246ad0f0-goog In-Reply-To: <20230307052036.1520708-1-stevensd@google.com> References: <20230307052036.1520708-1-stevensd@google.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759687359987998201?= X-GMAIL-MSGID: =?utf-8?q?1759687359987998201?= From: David Stevens Add a rollback label to deal with failure, instead of continuously checking for RESULT_SUCCESS, to make it easier to add more failure cases. The refactoring also allows the collapse_file tracepoint to include hpage on success (instead of NULL). Signed-off-by: David Stevens Acked-by: Peter Xu Reviewed-by: Yang Shi Acked-by: Hugh Dickins --- mm/khugepaged.c | 219 ++++++++++++++++++++++++------------------------ 1 file changed, 108 insertions(+), 111 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 3ea2aa55c2c5..b954e3c685e7 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1907,6 +1907,12 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, if (result != SCAN_SUCCEED) goto out; + __SetPageLocked(hpage); + if (is_shmem) + __SetPageSwapBacked(hpage); + hpage->index = start; + hpage->mapping = mapping; + /* * Ensure we have slots for all the pages in the range. This is * almost certainly a no-op because most of the pages must be present @@ -1919,16 +1925,10 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, xas_unlock_irq(&xas); if (!xas_nomem(&xas, GFP_KERNEL)) { result = SCAN_FAIL; - goto out; + goto rollback; } } while (1); - __SetPageLocked(hpage); - if (is_shmem) - __SetPageSwapBacked(hpage); - hpage->index = start; - hpage->mapping = mapping; - /* * At this point the hpage is locked and not up-to-date. * It's safe to insert it into the page cache, because nobody would @@ -2145,129 +2145,126 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, */ try_to_unmap_flush(); - if (result == SCAN_SUCCEED) { - /* - * Replacing old pages with new one has succeeded, now we - * attempt to copy the contents. - */ - index = start; - list_for_each_entry(page, &pagelist, lru) { - while (index < page->index) { - clear_highpage(hpage + (index % HPAGE_PMD_NR)); - index++; - } - if (copy_mc_highpage(hpage + (page->index % HPAGE_PMD_NR), - page) > 0) { - result = SCAN_COPY_MC; - break; - } - index++; - } - while (result == SCAN_SUCCEED && index < end) { + if (result != SCAN_SUCCEED) + goto rollback; + + /* + * Replacing old pages with new one has succeeded, now we + * attempt to copy the contents. + */ + index = start; + list_for_each_entry(page, &pagelist, lru) { + while (index < page->index) { clear_highpage(hpage + (index % HPAGE_PMD_NR)); index++; } + if (copy_mc_highpage(hpage + (page->index % HPAGE_PMD_NR), page) > 0) { + result = SCAN_COPY_MC; + goto rollback; + } + index++; + } + while (index < end) { + clear_highpage(hpage + (index % HPAGE_PMD_NR)); + index++; } - if (result == SCAN_SUCCEED) { - /* - * Copying old pages to huge one has succeeded, now we - * need to free the old pages. - */ - list_for_each_entry_safe(page, tmp, &pagelist, lru) { - list_del(&page->lru); - page->mapping = NULL; - page_ref_unfreeze(page, 1); - ClearPageActive(page); - ClearPageUnevictable(page); - unlock_page(page); - put_page(page); - } + /* + * Copying old pages to huge one has succeeded, now we + * need to free the old pages. + */ + list_for_each_entry_safe(page, tmp, &pagelist, lru) { + list_del(&page->lru); + page->mapping = NULL; + page_ref_unfreeze(page, 1); + ClearPageActive(page); + ClearPageUnevictable(page); + unlock_page(page); + put_page(page); + } - xas_lock_irq(&xas); - if (is_shmem) - __mod_lruvec_page_state(hpage, NR_SHMEM_THPS, nr); - else - __mod_lruvec_page_state(hpage, NR_FILE_THPS, nr); + xas_lock_irq(&xas); + if (is_shmem) + __mod_lruvec_page_state(hpage, NR_SHMEM_THPS, nr); + else + __mod_lruvec_page_state(hpage, NR_FILE_THPS, nr); - if (nr_none) { - __mod_lruvec_page_state(hpage, NR_FILE_PAGES, nr_none); - /* nr_none is always 0 for non-shmem. */ - __mod_lruvec_page_state(hpage, NR_SHMEM, nr_none); - } - /* Join all the small entries into a single multi-index entry. */ - xas_set_order(&xas, start, HPAGE_PMD_ORDER); - xas_store(&xas, hpage); - xas_unlock_irq(&xas); + if (nr_none) { + __mod_lruvec_page_state(hpage, NR_FILE_PAGES, nr_none); + /* nr_none is always 0 for non-shmem. */ + __mod_lruvec_page_state(hpage, NR_SHMEM, nr_none); + } + /* Join all the small entries into a single multi-index entry. */ + xas_set_order(&xas, start, HPAGE_PMD_ORDER); + xas_store(&xas, hpage); + xas_unlock_irq(&xas); - folio = page_folio(hpage); - folio_mark_uptodate(folio); - folio_ref_add(folio, HPAGE_PMD_NR - 1); + folio = page_folio(hpage); + folio_mark_uptodate(folio); + folio_ref_add(folio, HPAGE_PMD_NR - 1); - if (is_shmem) - folio_mark_dirty(folio); - folio_add_lru(folio); + if (is_shmem) + folio_mark_dirty(folio); + folio_add_lru(folio); - /* - * Remove pte page tables, so we can re-fault the page as huge. - */ - result = retract_page_tables(mapping, start, mm, addr, hpage, - cc); - unlock_page(hpage); - hpage = NULL; - } else { - /* Something went wrong: roll back page cache changes */ - xas_lock_irq(&xas); - if (nr_none) { - mapping->nrpages -= nr_none; - shmem_uncharge(mapping->host, nr_none); + /* + * Remove pte page tables, so we can re-fault the page as huge. + */ + result = retract_page_tables(mapping, start, mm, addr, hpage, + cc); + unlock_page(hpage); + goto out; + +rollback: + /* Something went wrong: roll back page cache changes */ + xas_lock_irq(&xas); + if (nr_none) { + mapping->nrpages -= nr_none; + shmem_uncharge(mapping->host, nr_none); + } + + xas_set(&xas, start); + xas_for_each(&xas, page, end - 1) { + page = list_first_entry_or_null(&pagelist, + struct page, lru); + if (!page || xas.xa_index < page->index) { + if (!nr_none) + break; + nr_none--; + /* Put holes back where they were */ + xas_store(&xas, NULL); + continue; } - xas_set(&xas, start); - xas_for_each(&xas, page, end - 1) { - page = list_first_entry_or_null(&pagelist, - struct page, lru); - if (!page || xas.xa_index < page->index) { - if (!nr_none) - break; - nr_none--; - /* Put holes back where they were */ - xas_store(&xas, NULL); - continue; - } + VM_BUG_ON_PAGE(page->index != xas.xa_index, page); - VM_BUG_ON_PAGE(page->index != xas.xa_index, page); + /* Unfreeze the page. */ + list_del(&page->lru); + page_ref_unfreeze(page, 2); + xas_store(&xas, page); + xas_pause(&xas); + xas_unlock_irq(&xas); + unlock_page(page); + putback_lru_page(page); + xas_lock_irq(&xas); + } + VM_BUG_ON(nr_none); + /* + * Undo the updates of filemap_nr_thps_inc for non-SHMEM file only. + * This undo is not needed unless failure is due to SCAN_COPY_MC. + */ + if (!is_shmem && result == SCAN_COPY_MC) + filemap_nr_thps_dec(mapping); - /* Unfreeze the page. */ - list_del(&page->lru); - page_ref_unfreeze(page, 2); - xas_store(&xas, page); - xas_pause(&xas); - xas_unlock_irq(&xas); - unlock_page(page); - putback_lru_page(page); - xas_lock_irq(&xas); - } - VM_BUG_ON(nr_none); - /* - * Undo the updates of filemap_nr_thps_inc for non-SHMEM file only. - * This undo is not needed unless failure is due to SCAN_COPY_MC. - */ - if (!is_shmem && result == SCAN_COPY_MC) - filemap_nr_thps_dec(mapping); + xas_unlock_irq(&xas); - xas_unlock_irq(&xas); + hpage->mapping = NULL; - hpage->mapping = NULL; - } + unlock_page(hpage); + put_page(hpage); - if (hpage) - unlock_page(hpage); out: VM_BUG_ON(!list_empty(&pagelist)); - if (hpage) - put_page(hpage); - trace_mm_khugepaged_collapse_file(mm, hpage, index, is_shmem, addr, file, nr, result); return result; } From patchwork Tue Mar 7 05:20:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Stevens X-Patchwork-Id: 65274 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2257171wrd; Mon, 6 Mar 2023 21:38:39 -0800 (PST) X-Google-Smtp-Source: AK7set+9xz5+Ih2lC3mAI7uVYvSl8djd9QlYzxI2idoZB8whFmlSlQ1HWZallgSe3wbOH69iQHk/ X-Received: by 2002:a17:906:eecc:b0:90b:167e:3050 with SMTP id wu12-20020a170906eecc00b0090b167e3050mr17160717ejb.36.1678167519571; Mon, 06 Mar 2023 21:38:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678167519; cv=none; d=google.com; s=arc-20160816; b=AQ4jEW1vgZcXwXHp7VcRaR0g/kV5u8/joWfaURm7ojkYq3L6dSKt1lH+j5nfVZ6v9z OPXBWLRYk9YBFQ1V0ayv+l7THCvA1pw1KrKXvhBai06JYLvK55Q3SSGCkNrc6GGitLYS EV1OpCGnv5mDzQJNE8T7MbtVGKAcHqPlnioRqcb/jh4dQsU/i19psBRNcRj4BtQrmWvZ X1FK10hoWzJSeW7qNLRevWSBQgMX2n4W6Vc5bcGkRJekjiSdJN2Vy7RWZ1/IDNF/3qvn IFk3i1cFfGtfen6iwVHWVESwoe9RwMVUy+SsWamqNcSrhUwYFdRLhBBuhkcjpm2+oprP rZ/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=AjnWe8/2NQ2Rb5ldPsFVe45g3piuxL8Z1VW03Vfk8mw=; b=fgfhxgq4fj6F7dY3yL4T11/J9idMRx9fWkgDwEMZLcpAKaRu+dDeejoR5jA+2i94Dx nPcCfLVf5TEB68efIW0kgIGbHBhvYzLE8TN8KQfFLpQLuipxQdnKNliJr5iaQ+CUSxB4 0AUSpi1IkMqi/Z6Q66pDUZJDFQgEZE743Z2NJXYYKzEKIgxMRVjW3BcDxMn89xinU5ip aePCRK2riL/t/09rENDlHW+7XjQ5nMVgDbVXXT937U0N2xBMYGFCRejuCRbsMmGpRioa sgeoKiM6qwmieY/FygOD79LrfTsGaz8LDlj2J1fDyFNUGjlKAlth8SHIRqbXsX0Ggfcg +Hfw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=kBImBfHt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s2-20020a170906a18200b008b178b41ec1si12516874ejy.284.2023.03.06.21.38.15; Mon, 06 Mar 2023 21:38:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=kBImBfHt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230360AbjCGFVJ (ORCPT + 99 others); Tue, 7 Mar 2023 00:21:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33134 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230189AbjCGFVB (ORCPT ); Tue, 7 Mar 2023 00:21:01 -0500 Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D96157084 for ; Mon, 6 Mar 2023 21:20:59 -0800 (PST) Received: by mail-pj1-x1030.google.com with SMTP id kb15so12137929pjb.1 for ; Mon, 06 Mar 2023 21:20:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1678166458; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AjnWe8/2NQ2Rb5ldPsFVe45g3piuxL8Z1VW03Vfk8mw=; b=kBImBfHto06kX3ajX3HrtcGpZ7bMp4syG8e7s8aLfxkmY6tZ6j07eeRjASTNxfhmYd 5CAlb6Bf9CgkXOLw28wBiNF/uvdTiwsdCe3DejJlellQEjmK1gXdf4EBh+7zGW5odEZf Wyo/lri1y95D4ABIhnanN+wvOuBP8knvIntKo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678166458; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AjnWe8/2NQ2Rb5ldPsFVe45g3piuxL8Z1VW03Vfk8mw=; b=AgwFbVSd22MGyBoNzRvh9mZIIDBZwN5h/UgIYyEj9iQIy/pKQzuBOTAD2Dhm+FmuDe awL5XAMxgCONthxScWcSDlqA/7t5Bvct5O9RxEQfT9XcBKjvl/FSbr4IDQQJgheqsGGs DlefViwt510sUQV+jy+3Y7xM6oJoERCLkOqsOWVocze7aMzId5+4npzaeca9KlHuBSSZ J6ICfvRvlWbToe0irwApMjaIdhCMWwd1I1ADz6IAo+Ce/H6B1PZRVf5pByOgw4hRPDQS tmCCiLFH9CRQef89ACr/h3jqUoQtRWu8NtheR8QxLkpTK0B0bYw1IXymotyk1G3LAVRb WBQw== X-Gm-Message-State: AO0yUKWXzHfnJ+dp1poB8cblawm1bIlB0cDngIkP0UBuLwDvJTjAPO1r oLliPH3Pf+JVUu9TGWnPVTexDQ== X-Received: by 2002:a05:6a20:3558:b0:bd:17a4:c35f with SMTP id f24-20020a056a20355800b000bd17a4c35fmr11288488pze.23.1678166458635; Mon, 06 Mar 2023 21:20:58 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1f73:9034:ce28:4421]) by smtp.gmail.com with UTF8SMTPSA id n13-20020aa7904d000000b0058b927b9653sm7293130pfo.92.2023.03.06.21.20.56 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 06 Mar 2023 21:20:58 -0800 (PST) From: David Stevens X-Google-Original-From: David Stevens To: linux-mm@kvack.org, Andrew Morton Cc: Peter Xu , Matthew Wilcox , "Kirill A . Shutemov" , Yang Shi , David Hildenbrand , Hugh Dickins , Jiaqi Yan , linux-kernel@vger.kernel.org, David Stevens Subject: [PATCH v5 2/3] mm/khugepaged: skip shmem with userfaultfd Date: Tue, 7 Mar 2023 14:20:35 +0900 Message-Id: <20230307052036.1520708-3-stevensd@google.com> X-Mailer: git-send-email 2.40.0.rc0.216.gc4246ad0f0-goog In-Reply-To: <20230307052036.1520708-1-stevensd@google.com> References: <20230307052036.1520708-1-stevensd@google.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759686184873610056?= X-GMAIL-MSGID: =?utf-8?q?1759686184873610056?= From: David Stevens Make sure that collapse_file respects any userfaultfds registered with MODE_MISSING. If userspace has any such userfaultfds registered, then for any page which it knows to be missing, it may expect a UFFD_EVENT_PAGEFAULT. This means collapse_file needs to be careful when collapsing a shmem range would result in replacing an empty page with a THP, to avoid breaking userfaultfd. Synchronization when checking for userfaultfds in collapse_file is tricky because the mmap locks can't be used to prevent races with the registration of new userfaultfds. Instead, we provide synchronization by ensuring that userspace cannot observe the fact that pages are missing before we check for userfaultfds. Although this allows registration of a userfaultfd to race with collapse_file, it ensures that userspace cannot observe any pages transition from missing to present after such a race occurs. This makes such a race indistinguishable to the collapse occurring immediately before the userfaultfd registration. The first step to provide this synchronization is to stop filling gaps during the loop iterating over the target range, since the page cache lock can be dropped during that loop. The second step is to fill the gaps with XA_RETRY_ENTRY after the page cache lock is acquired the final time, to avoid races with accesses to the page cache that only take the RCU read lock. The fact that we don't fill holes during the initial iteration means that collapse_file now has to handle faults occurring during the collapse. This is done by re-validating the number of missing pages after acquiring the page cache lock for the final time. This fix is targeted at khugepaged, but the change also applies to MADV_COLLAPSE. MADV_COLLAPSE on a range with a userfaultfd will now return EBUSY if there are any missing pages (instead of succeeding on shmem and returning EINVAL on anonymous memory). There is also now a window during MADV_COLLAPSE where a fault on a missing page will cause the syscall to fail with EAGAIN. The fact that intermediate page cache state can no longer be observed before the rollback of a failed collapse is also technically a userspace-visible change (via at least SEEK_DATA and SEEK_END), but it is exceedingly unlikely that anything relies on being able to observe that transient state. Signed-off-by: David Stevens Acked-by: Peter Xu --- include/trace/events/huge_memory.h | 3 +- mm/khugepaged.c | 92 +++++++++++++++++++++++------- 2 files changed, 73 insertions(+), 22 deletions(-) diff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge_memory.h index 46cce509957b..7ee85fff89a3 100644 --- a/include/trace/events/huge_memory.h +++ b/include/trace/events/huge_memory.h @@ -37,7 +37,8 @@ EM( SCAN_CGROUP_CHARGE_FAIL, "ccgroup_charge_failed") \ EM( SCAN_TRUNCATED, "truncated") \ EM( SCAN_PAGE_HAS_PRIVATE, "page_has_private") \ - EMe(SCAN_COPY_MC, "copy_poisoned_page") \ + EM( SCAN_COPY_MC, "copy_poisoned_page") \ + EMe(SCAN_PAGE_FILLED, "page_filled") \ #undef EM #undef EMe diff --git a/mm/khugepaged.c b/mm/khugepaged.c index b954e3c685e7..51ae399f2035 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -57,6 +57,7 @@ enum scan_result { SCAN_TRUNCATED, SCAN_PAGE_HAS_PRIVATE, SCAN_COPY_MC, + SCAN_PAGE_FILLED, }; #define CREATE_TRACE_POINTS @@ -1873,8 +1874,8 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, * - allocate and lock a new huge page; * - scan page cache replacing old pages with the new one * + swap/gup in pages if necessary; - * + fill in gaps; * + keep old pages around in case rollback is required; + * - finalize updates to the page cache; * - if replacing succeeds: * + copy data over; * + free old pages; @@ -1952,13 +1953,12 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, result = SCAN_TRUNCATED; goto xa_locked; } - xas_set(&xas, index); + xas_set(&xas, index + 1); } if (!shmem_charge(mapping->host, 1)) { result = SCAN_FAIL; goto xa_locked; } - xas_store(&xas, hpage); nr_none++; continue; } @@ -2169,21 +2169,57 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, index++; } - /* - * Copying old pages to huge one has succeeded, now we - * need to free the old pages. - */ - list_for_each_entry_safe(page, tmp, &pagelist, lru) { - list_del(&page->lru); - page->mapping = NULL; - page_ref_unfreeze(page, 1); - ClearPageActive(page); - ClearPageUnevictable(page); - unlock_page(page); - put_page(page); + if (nr_none) { + struct vm_area_struct *vma; + int nr_none_check = 0; + + i_mmap_lock_read(mapping); + xas_lock_irq(&xas); + + xas_set(&xas, start); + for (index = start; index < end; index++) { + if (!xas_next(&xas)) { + xas_store(&xas, XA_RETRY_ENTRY); + nr_none_check++; + } + } + + if (nr_none != nr_none_check) { + result = SCAN_PAGE_FILLED; + goto immap_locked; + } + + /* + * If userspace observed a missing page in a VMA with an armed + * userfaultfd, then it might expect a UFFD_EVENT_PAGEFAULT for + * that page, so we need to roll back to avoid suppressing such + * an event. Any userfaultfds armed after this point will not be + * able to observe any missing pages due to the previously + * inserted retry entries. + */ + vma_interval_tree_foreach(vma, &mapping->i_mmap, start, start) { + if (userfaultfd_missing(vma)) { + result = SCAN_EXCEED_NONE_PTE; + goto immap_locked; + } + } + +immap_locked: + i_mmap_unlock_read(mapping); + if (result != SCAN_SUCCEED) { + xas_set(&xas, start); + for (index = start; index < end; index++) { + if (xas_next(&xas) == XA_RETRY_ENTRY) + xas_store(&xas, NULL); + } + + xas_unlock_irq(&xas); + goto rollback; + } + } else { + xas_lock_irq(&xas); } - xas_lock_irq(&xas); if (is_shmem) __mod_lruvec_page_state(hpage, NR_SHMEM_THPS, nr); else @@ -2213,6 +2249,20 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, result = retract_page_tables(mapping, start, mm, addr, hpage, cc); unlock_page(hpage); + + /* + * The collapse has succeeded, so free the old pages. + */ + list_for_each_entry_safe(page, tmp, &pagelist, lru) { + list_del(&page->lru); + page->mapping = NULL; + page_ref_unfreeze(page, 1); + ClearPageActive(page); + ClearPageUnevictable(page); + unlock_page(page); + put_page(page); + } + goto out; rollback: @@ -2224,15 +2274,13 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, } xas_set(&xas, start); - xas_for_each(&xas, page, end - 1) { + end = index; + for (index = start; index < end; index++) { + xas_next(&xas); page = list_first_entry_or_null(&pagelist, struct page, lru); if (!page || xas.xa_index < page->index) { - if (!nr_none) - break; nr_none--; - /* Put holes back where they were */ - xas_store(&xas, NULL); continue; } @@ -2750,12 +2798,14 @@ static int madvise_collapse_errno(enum scan_result r) case SCAN_ALLOC_HUGE_PAGE_FAIL: return -ENOMEM; case SCAN_CGROUP_CHARGE_FAIL: + case SCAN_EXCEED_NONE_PTE: return -EBUSY; /* Resource temporary unavailable - trying again might succeed */ case SCAN_PAGE_COUNT: case SCAN_PAGE_LOCK: case SCAN_PAGE_LRU: case SCAN_DEL_PAGE_LRU: + case SCAN_PAGE_FILLED: return -EAGAIN; /* * Other: Trying again likely not to succeed / error intrinsic to From patchwork Tue Mar 7 05:20:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Stevens X-Patchwork-Id: 65277 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2262716wrd; Mon, 6 Mar 2023 21:58:12 -0800 (PST) X-Google-Smtp-Source: AK7set/t4gZhdhkDi3+DgLDyFZ4eRf19Fqc+qCYnm7XttvsTyvdFGCKG9R65q/fwXl02aLnvMKOn X-Received: by 2002:a17:907:7ea5:b0:8a5:3d1e:6302 with SMTP id qb37-20020a1709077ea500b008a53d1e6302mr15594190ejc.56.1678168692342; Mon, 06 Mar 2023 21:58:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678168692; cv=none; d=google.com; s=arc-20160816; b=DnBTTlTVw0DQDBHFJgnL86fi43ohQqP3LGM2v1ijbhHH4LlDeJvz7/GQJHgdEabPoz nVZ+MHtM8DgGjxrroaip6saiVlL9neWja4ve6levB/BGodGjXYXD0AiHztI/ntF/ceNq lYN5BUBwLg4JlHvc6fySUF/Ny7nabzb3zGNZWq9plDfWAwkgrB6yw4X0AwBCXTKyLe2+ ECy937tXx38aCytpkpajnpJcyFoNo6QrEApvJC7wpY8BGr46vw6iVPfB7JSsODLVre5W XQMvOKZaXnad0+QdkD9/nNQ1VTPKdBrd9hynoKFlImWXCFwHBfg+I1Flsbocbx+P/w61 X+MQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eFAeWPY2kcSCdT3O6CTUVlduCnpUJhZej+FbmoWYNkg=; b=iyLXn6u+s1C2GdT2tRnjDIR3dFwVB4NNdKz419GB6VMH9nx+26gTHKAicf/mrXLOJA yL008kt4Bm3FFcvRpp/3FfnU4+rj6Dk0vmFuxclZh+Qkj9lX/DsvhEIHyCDsCydWmNOb +kq7nfEzl+SuaudUKLZQMEBI6Ffe1o9RwEsB1feWf4L/14kxteseFebmQBUxU87zNPq1 M05FQWdUjYLx58Q7Dvu29ky5gs1cp7/JdII+wryY/SNgkvYmxz8fsIi8Z0nqXeitEffQ ir/bi6lHimOBdkXwuvLEVMQfcRxYE4Sj4KWWtXq3KsK46Fy6bHhCjpYiwTkZwQbA6XSx P+hw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=Q9sGR9Jp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u13-20020a170906408d00b008d0dbf15be3si9837936ejj.676.2023.03.06.21.57.49; Mon, 06 Mar 2023 21:58:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@chromium.org header.s=google header.b=Q9sGR9Jp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=chromium.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230135AbjCGFV2 (ORCPT + 99 others); Tue, 7 Mar 2023 00:21:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33252 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230336AbjCGFVN (ORCPT ); Tue, 7 Mar 2023 00:21:13 -0500 Received: from mail-pf1-x42a.google.com (mail-pf1-x42a.google.com [IPv6:2607:f8b0:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5212662318 for ; Mon, 6 Mar 2023 21:21:05 -0800 (PST) Received: by mail-pf1-x42a.google.com with SMTP id fa28so7298030pfb.12 for ; Mon, 06 Mar 2023 21:21:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1678166464; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eFAeWPY2kcSCdT3O6CTUVlduCnpUJhZej+FbmoWYNkg=; b=Q9sGR9JplEaeNqsSC0pRNCxNi/C7wmGhoCkA56L/loqSr4HfTSKbd24kgSyhErzSqH 0I6wjvacPrWo2mFsQrRLwwJXlg3UaeAhxOZ0HtAJPSslGOMFmzE06GPMU9FLNZYobvT/ 3o1S3aGwC8Ks5mnW1V4r0yFDwjDaYhzpIxfho= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1678166464; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eFAeWPY2kcSCdT3O6CTUVlduCnpUJhZej+FbmoWYNkg=; b=fQpuaLLZp5uFy/G1KFCXyF+9k/Libhtx7fTkFudRveCKob2qEpITpCrUgEjjL1RH4P s71xOT2B5I+XIBTSwn8OZFGOsDX4j+fx4i17gMDR5d/s7gO8yirPBe9t1wqY9mWBAmYY T2niQjAD50ev+nftsBNiMOUXj1yczVVosEFhwUi6U5W1LkT3vOcbfF3TbFSbGYMMMEQW V5konlkTnq7WwPb0bK9U/JP3Ovdq2neYXyTgKuf0PsleNL7DYdnT0OiR3dj7679eZKLJ aw9UH2lImX+2GcdkeAlOk0P/xZTdctuGrZH043PZSPyh7oIZIf1k2pRrv5INq+0M1t5Y jMhw== X-Gm-Message-State: AO0yUKUTDfnFNvXqod1vxPycCsanxVDzXz7KkuvQux8PWkLdj9WOwHpa 7M0WACEfzStgpe2WBJnte9OosQ== X-Received: by 2002:a62:6542:0:b0:5a8:ad9d:83f with SMTP id z63-20020a626542000000b005a8ad9d083fmr10412674pfb.24.1678166464453; Mon, 06 Mar 2023 21:21:04 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:1f73:9034:ce28:4421]) by smtp.gmail.com with UTF8SMTPSA id c26-20020aa78c1a000000b005a8f1187112sm7117378pfd.58.2023.03.06.21.21.01 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 06 Mar 2023 21:21:04 -0800 (PST) From: David Stevens X-Google-Original-From: David Stevens To: linux-mm@kvack.org, Andrew Morton Cc: Peter Xu , Matthew Wilcox , "Kirill A . Shutemov" , Yang Shi , David Hildenbrand , Hugh Dickins , Jiaqi Yan , linux-kernel@vger.kernel.org, David Stevens Subject: [PATCH v5 3/3] mm/khugepaged: maintain page cache uptodate flag Date: Tue, 7 Mar 2023 14:20:36 +0900 Message-Id: <20230307052036.1520708-4-stevensd@google.com> X-Mailer: git-send-email 2.40.0.rc0.216.gc4246ad0f0-goog In-Reply-To: <20230307052036.1520708-1-stevensd@google.com> References: <20230307052036.1520708-1-stevensd@google.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759687414815222670?= X-GMAIL-MSGID: =?utf-8?q?1759687414815222670?= From: David Stevens Make sure that collapse_file doesn't interfere with checking the uptodate flag in the page cache by only inserting hpage into the page cache after it has been updated and marked uptodate. This is achieved by simply not replacing present pages with hpage when iterating over the target range. The present pages are already locked, so replacing the with the locked hpage before the collapse is finalized is unnecessary. This fixes a race where folio_seek_hole_data would mistake hpage for an fallocated but unwritten page. This race is visible to userspace via data temporarily disappearing from SEEK_DATA/SEEK_HOLE. Fixes: f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages") Signed-off-by: David Stevens Acked-by: Peter Xu --- mm/khugepaged.c | 50 ++++++++++++------------------------------------- 1 file changed, 12 insertions(+), 38 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 51ae399f2035..bdde0a02811b 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1930,12 +1930,6 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, } } while (1); - /* - * At this point the hpage is locked and not up-to-date. - * It's safe to insert it into the page cache, because nobody would - * be able to map it or use it in another way until we unlock it. - */ - xas_set(&xas, start); for (index = start; index < end; index++) { page = xas_next(&xas); @@ -2104,13 +2098,9 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, } /* - * Add the page to the list to be able to undo the collapse if - * something go wrong. + * Accumulate the pages that are being collapsed. */ list_add_tail(&page->lru, &pagelist); - - /* Finally, replace with the new page. */ - xas_store(&xas, hpage); continue; out_unlock: unlock_page(page); @@ -2149,8 +2139,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, goto rollback; /* - * Replacing old pages with new one has succeeded, now we - * attempt to copy the contents. + * The old pages are locked, so they won't change anymore. */ index = start; list_for_each_entry(page, &pagelist, lru) { @@ -2230,11 +2219,11 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, /* nr_none is always 0 for non-shmem. */ __mod_lruvec_page_state(hpage, NR_SHMEM, nr_none); } - /* Join all the small entries into a single multi-index entry. */ - xas_set_order(&xas, start, HPAGE_PMD_ORDER); - xas_store(&xas, hpage); - xas_unlock_irq(&xas); + /* + * Mark hpage as uptodate before inserting it into the page cache so + * that it isn't mistaken for an fallocated but unwritten page. + */ folio = page_folio(hpage); folio_mark_uptodate(folio); folio_ref_add(folio, HPAGE_PMD_NR - 1); @@ -2243,6 +2232,11 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, folio_mark_dirty(folio); folio_add_lru(folio); + /* Join all the small entries into a single multi-index entry. */ + xas_set_order(&xas, start, HPAGE_PMD_ORDER); + xas_store(&xas, hpage); + xas_unlock_irq(&xas); + /* * Remove pte page tables, so we can re-fault the page as huge. */ @@ -2267,36 +2261,18 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, rollback: /* Something went wrong: roll back page cache changes */ - xas_lock_irq(&xas); if (nr_none) { mapping->nrpages -= nr_none; shmem_uncharge(mapping->host, nr_none); } - xas_set(&xas, start); - end = index; - for (index = start; index < end; index++) { - xas_next(&xas); - page = list_first_entry_or_null(&pagelist, - struct page, lru); - if (!page || xas.xa_index < page->index) { - nr_none--; - continue; - } - - VM_BUG_ON_PAGE(page->index != xas.xa_index, page); - + list_for_each_entry_safe(page, tmp, &pagelist, lru) { /* Unfreeze the page. */ list_del(&page->lru); page_ref_unfreeze(page, 2); - xas_store(&xas, page); - xas_pause(&xas); - xas_unlock_irq(&xas); unlock_page(page); putback_lru_page(page); - xas_lock_irq(&xas); } - VM_BUG_ON(nr_none); /* * Undo the updates of filemap_nr_thps_inc for non-SHMEM file only. * This undo is not needed unless failure is due to SCAN_COPY_MC. @@ -2304,8 +2280,6 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, if (!is_shmem && result == SCAN_COPY_MC) filemap_nr_thps_dec(mapping); - xas_unlock_irq(&xas); - hpage->mapping = NULL; unlock_page(hpage);