From patchwork Wed Jul 19 13:54:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 122641 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c923:0:b0:3e4:2afc:c1 with SMTP id j3csp2471471vqt; Wed, 19 Jul 2023 07:17:12 -0700 (PDT) X-Google-Smtp-Source: APBJJlFzTWq8Xaw2IU5d7If1CnzRfce9c24dPchbVe/iC8WGMk9FH9cNIYus72QSYdcoN3q15lhP X-Received: by 2002:a17:90b:3901:b0:261:1141:b70d with SMTP id ob1-20020a17090b390100b002611141b70dmr4073317pjb.45.1689776232291; Wed, 19 Jul 2023 07:17:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689776232; cv=none; d=google.com; s=arc-20160816; b=GMMprt995ourpXzBzRb+5JnxPsYkRmonhMaSJiD8XOo5FQwbeshYEqnvteQVsFXbu1 VCsrkgqmcv1jBT/1YhxEBppqGp7uu7xAo/dv2c4mnQFGvBLNlMwwMXZyfwljEajCfJdq 0rGdNwA4yXnSBArxYoCZbIhOEeKTNKFRjSg7/+yQ/xsMMscHyGAyS6LEPfOzOzs5h9zB RB14yOrJFWmtMEU3a428wb0h5b9DVLiAg9zOAupKZw5BZiB5LuJw086+I1nibF/tzUT7 gh1gR8ssGwMGbjsmW7cKbEf18IH8g0krc7U1IsYrDG84rnbq8X6Nk8bB2rWo3ZX8ASgA xwKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=2dqIz3C526jfk09wr8sX5z1Ztv6RWuu5WBWCEVsOmTA=; fh=THi7eKCjaWcVibUnyL3CTjC3PwRMYqk9JChFkSe1vn0=; b=lmOh19PKxa2+pF7XSCfq/rni+nBfFWHlm2CGDloxrlOXOphJJgDvl0Bc520AtNEQ/r F9irzvwWP/0Y6JOPf/H5Ze8wL0wJEwS8e1+3hNfzWL9O1XR1objpouAZaLw0RW2PI7TM Y+G8LY2I/syA60QLJJb6eY077ncekcNFU+Gr436hYBALo9wAwUwvANwON2LbKMpe/rTL 9jxTZ/NWmeEDY0zxhlBGJ+sU83K8vtnNNTcBv5i+sv/dgg1yg6rSpIpk+TyzetKqbVe7 tbGYa+u3K/Jw4hz2rk75xtssDwRw7VGhMUngNny52Oi7j9ghOrhS28mLkvS5YGIPolqi sJ/A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id rm2-20020a17090b3ec200b0025653dc2881si1592032pjb.23.2023.07.19.07.16.58; Wed, 19 Jul 2023 07:17:12 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231216AbjGSNzN (ORCPT + 99 others); Wed, 19 Jul 2023 09:55:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37016 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229539AbjGSNzJ (ORCPT ); Wed, 19 Jul 2023 09:55:09 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 3F0B112C for ; Wed, 19 Jul 2023 06:55:08 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4238512FC; Wed, 19 Jul 2023 06:55:51 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 672B43F6C4; Wed, 19 Jul 2023 06:55:06 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Yang Shi , "Huang, Ying" , Zi Yan Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 2/3] mm: Implement folio_remove_rmap_range() Date: Wed, 19 Jul 2023 14:54:49 +0100 Message-Id: <20230719135450.545227-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230719135450.545227-1-ryan.roberts@arm.com> References: <20230719135450.545227-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771858802254952565 X-GMAIL-MSGID: 1771858802254952565 Like page_remove_rmap() but batch-removes the rmap for a range of pages belonging to a folio. This can provide a small speedup due to less manipuation of the various counters. But more crucially, if removing the rmap for all pages of a folio in a batch, there is no need to (spuriously) add it to the deferred split list, which saves significant cost when there is contention for the split queue lock. All contained pages are accounted using the order-0 folio (or base page) scheme. Reviewed-by: Yin Fengwei Reviewed-by: Zi Yan Signed-off-by: Ryan Roberts --- include/linux/rmap.h | 2 ++ mm/rmap.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index b87d01660412..f578975c12c0 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -200,6 +200,8 @@ void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); +void folio_remove_rmap_range(struct folio *folio, struct page *page, + int nr, struct vm_area_struct *vma); void hugepage_add_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); diff --git a/mm/rmap.c b/mm/rmap.c index eb0bb00dae34..4022a3ab73a8 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1359,6 +1359,78 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, mlock_vma_folio(folio, vma, compound); } +/** + * folio_remove_rmap_range - Take down PTE mappings from a range of pages. + * @folio: Folio containing all pages in range. + * @page: First page in range to unmap. + * @nr: Number of pages to unmap. + * @vma: The VM area containing the range. + * + * All pages in the range must belong to the same VMA & folio. They + * must be mapped with PTEs, not a PMD. + * + * Context: Caller holds the pte lock. + */ +void folio_remove_rmap_range(struct folio *folio, struct page *page, + int nr, struct vm_area_struct *vma) +{ + atomic_t *mapped = &folio->_nr_pages_mapped; + int nr_unmapped = 0; + int nr_mapped; + bool last; + enum node_stat_item idx; + + if (unlikely(folio_test_hugetlb(folio))) { + VM_WARN_ON_FOLIO(1, folio); + return; + } + + VM_WARN_ON_ONCE(page < &folio->page || + page + nr > (&folio->page + folio_nr_pages(folio))); + + if (!folio_test_large(folio)) { + /* Is this the page's last map to be removed? */ + last = atomic_add_negative(-1, &page->_mapcount); + nr_unmapped = last; + } else { + for (; nr != 0; nr--, page++) { + /* Is this the page's last map to be removed? */ + last = atomic_add_negative(-1, &page->_mapcount); + if (last) + nr_unmapped++; + } + + /* Pages still mapped if folio mapped entirely */ + nr_mapped = atomic_sub_return_relaxed(nr_unmapped, mapped); + if (nr_mapped >= COMPOUND_MAPPED) + nr_unmapped = 0; + } + + if (nr_unmapped) { + idx = folio_test_anon(folio) ? NR_ANON_MAPPED : NR_FILE_MAPPED; + __lruvec_stat_mod_folio(folio, idx, -nr_unmapped); + + /* + * Queue anon large folio for deferred split if at least one + * page of the folio is unmapped and at least one page is still + * mapped. + */ + if (folio_test_large(folio) && + folio_test_anon(folio) && nr_mapped) + deferred_split_folio(folio); + } + + /* + * It would be tidy to reset folio_test_anon mapping when fully + * unmapped, but that might overwrite a racing page_add_anon_rmap + * which increments mapcount after us but sets mapping before us: + * so leave the reset to free_pages_prepare, and remember that + * it's only reliable while mapped. + */ + + munlock_vma_folio(folio, vma, false); +} + /** * page_remove_rmap - take down pte mapping from a page * @page: page to remove mapping from