From patchwork Thu Jul 27 14:18:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 12700 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a985:0:b0:3e4:2afc:c1 with SMTP id t5csp1144955vqo; Thu, 27 Jul 2023 07:36:16 -0700 (PDT) X-Google-Smtp-Source: APBJJlGKW4nw/xJVYMcNqvK+BNepkZQRIYKKWgYVGGPUn313bwlVOiWoGNVYtmzlMCOeSrN7UI55 X-Received: by 2002:a17:906:cc4f:b0:99b:4210:cc75 with SMTP id mm15-20020a170906cc4f00b0099b4210cc75mr2156655ejb.40.1690468575868; Thu, 27 Jul 2023 07:36:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690468575; cv=none; d=google.com; s=arc-20160816; b=t2nNDgXzFTIyOWQbnYqnltL7e+wLUJd4kdBdHgZNUQOUFMjMnpY2abaqzXcHodQSNj ZmQSTdQEz+DYagVRaf2wp5j88vQKCGiFh1rpdfa/7blwSwF5imyA02+fiVLFEyu5owc1 cbEGQhLWuCRgbo7MiHl2f4jIxpwRKkIficQQDwR1epugFE7yItwF5rBqazn27bLbUIe5 e7Wc4FOeg8b1APhTcY5eNJVVhcqcCvBHcUGCR4J4gq4Ul2kDfHnSnBq/wfVVWBLtlmwN OiWYG9tAUEPMKiGh57LICxpL2ITyvp+cXP5w15jpsyWvwKh01eFr6yu2kuSQMsISc4Vg GuPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from; bh=yVrovolLhdoIDypqzxVEphfZDq1TAN7hgj47sjdxDGs=; fh=rUOdZ1rV05MjXD37NRPw+6Wt+asmOiidzfpNzlU6HPE=; b=giL7Zov1E9nEtfyAhwyzrL9k0mw6WlUJ6BIAn+eSMTGrSDoH2oxDWC+R4rVPatlKlY REQPaMVuOjCrf6qumIfiGviI3480aUN35+0YmbYwEHp4slXLmIGo6lvdSuVn0S2cMpBX g3It+/381b4sKKa7wDGCx+6odd4dELa665ZgH3g2mmEyTV0ZUq0kgCPOS+WDscDRGPSr ignRpAOTKd9Q3tj+5BYvM5k9pPSWKyVfyEkgbmpML3ew1XtPCMjKjCssWUnPTdT69O29 SjB1jv96cqXncW91IyyWjHcGLFOe/Tf9yAn8GRqojZrhdyAZKCusck+vjS/0QaFLovy6 rhwQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b22-20020a170906491600b0098804fbcff0si1096282ejq.709.2023.07.27.07.35.51; Thu, 27 Jul 2023 07:36:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233360AbjG0OSw (ORCPT + 99 others); Thu, 27 Jul 2023 10:18:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232895AbjG0OSv (ORCPT ); Thu, 27 Jul 2023 10:18:51 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id A7BF019B6 for ; Thu, 27 Jul 2023 07:18:49 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6415AD75; Thu, 27 Jul 2023 07:19:32 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 5D6A73F6C4; Thu, 27 Jul 2023 07:18:47 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Yang Shi , "Huang, Ying" , Zi Yan , Nathan Chancellor , Alexander Gordeev , Gerald Schaefer Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v4 0/3] Optimize large folio interaction with deferred split Date: Thu, 27 Jul 2023 15:18:34 +0100 Message-Id: <20230727141837.3386072-1-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772584777280690310 X-GMAIL-MSGID: 1772584777280690310 Hi All, This is v4 of a small series in support of my work to enable the use of large folios for anonymous memory (known as "FLEXIBLE_THP" or "LARGE_ANON_FOLIO") [5]. It first makes it possible to add large, non-pmd-mappable folios to the deferred split queue. Then it modifies zap_pte_range() to batch-remove spans of physically contiguous pages from the rmap, which means that in the common case, we elide the need to ever put the folio on the deferred split queue, thus reducing lock contention and improving performance. This becomes more visible once we have lots of large anonymous folios in the system, and Huang Ying has suggested solving this needs to be a prerequisit for merging the main FLEXIBLE_THP/LARGE_ANON_FOLIO work. The series applies on top of v6.5-rc3 and a branch is available at [4]. NOTE: v3 is currently in mm-unstable and has a bug that affects s390, which this version fixes. Changes since v3 [3] -------------------- - Fixed bug reported on s390 [6] - Since s390 enables MMU_GATHER_NO_GATHER, __tlb_remove_page() causes a ref to be dropped on the page, but we were still using the page after that function call. - Fix by using folio_get()/folio_put() to guarrantee lifetime of page - Thanks to Nathan Chancellor for the bug report and helping me get set up with s390! - Don't use batch path if folio is not large Changes since v2 [2] -------------------- - patch 2: Reworked at Yu Zhou's request to reduce duplicated code. - page_remove_rmap() now forwards to folio_remove_rmap_range() for the !compound (PMD mapped) case. - Both page_remove_rmap() and folio_remove_rmap_range() share common epilogue via new helper function __remove_rmap_finish(). - As a result of the changes, I've removed the previous Reviewed-bys. - other 2 patches are unchanged. Changes since v1 [1] -------------------- - patch 2: Modified doc comment for folio_remove_rmap_range() - patch 2: Hoisted _nr_pages_mapped manipulation out of page loop so its now modified once per folio_remove_rmap_range() call. - patch 2: Added check that page range is fully contained by folio in folio_remove_rmap_range() - patch 2: Fixed some nits raised by Huang, Ying for folio_remove_rmap_range() - patch 3: Support batch-zap of all anon pages, not just those in anon vmas - patch 3: Renamed various functions to make their use clear - patch 3: Various minor refactoring/cleanups - Added Reviewed-By tags - thanks! [1] https://lore.kernel.org/linux-mm/20230717143110.260162-1-ryan.roberts@arm.com/ [2] https://lore.kernel.org/linux-mm/20230719135450.545227-1-ryan.roberts@arm.com/ [3] https://lore.kernel.org/linux-mm/20230720112955.643283-1-ryan.roberts@arm.com/ [4] https://gitlab.arm.com/linux-arm/linux-rr/-/tree/features/granule_perf/deferredsplit-lkml_v4 [5] https://lore.kernel.org/linux-mm/20230714160407.4142030-1-ryan.roberts@arm.com/ [6] https://lore.kernel.org/linux-mm/20230726161942.GA1123863@dev-arch.thelio-3990X/ Thanks, Ryan Ryan Roberts (3): mm: Allow deferred splitting of arbitrary large anon folios mm: Implement folio_remove_rmap_range() mm: Batch-zap large anonymous folio PTE mappings include/linux/rmap.h | 2 + mm/memory.c | 132 +++++++++++++++++++++++++++++++++++++++++++ mm/rmap.c | 125 ++++++++++++++++++++++++++++++---------- 3 files changed, 229 insertions(+), 30 deletions(-) --- 2.25.1