From patchwork Thu Jul 20 11:29:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 12336 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c923:0:b0:3e4:2afc:c1 with SMTP id j3csp3056595vqt; Thu, 20 Jul 2023 04:46:04 -0700 (PDT) X-Google-Smtp-Source: APBJJlH8d6vJXy17l6b0FMdR7P+sf5F+vaHa18p6OrozBoJPb6r9IyO1Zhc3PCys24CuAiw7Ddrs X-Received: by 2002:a05:6a20:8e10:b0:132:87da:5b57 with SMTP id y16-20020a056a208e1000b0013287da5b57mr26734612pzj.59.1689853564549; Thu, 20 Jul 2023 04:46:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689853564; cv=none; d=google.com; s=arc-20160816; b=lruqdqDg7CYWfrJXxxrPIdDsUbGVodZ26gTFVvZcNk20PmHcVjtpTOu1WbHYtTCc3/ Zt9YGhlvrCbAvQ5ebraJZnEYBR/ntVqVUAyyuxbvaH+HrzXq3n06CmhpZbOmO9cda3CY 0Si4CvKe2KHBmCfeoFDlmB6FfLvxT+q5N+ov+gIkGfpe94UpRVladxkkLIuJp8V6DjAk d4k5LSsaZYNHeE5uCJF+7IrieIIpRqYFYa1vrTt+GpN8ON5WiMvs3g2kTkas90m+T48P N0/X2CSsvnpdZBap7Z4ytwmnhE1ys35H41WiGESWwTC5eccFeQ5t86a1aTf7SCo4zIYg q3/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from; bh=VaQ0evAW+iMJ3X3MzxAmvzcl1mMF2PLKTnaX2OtDpBM=; fh=THi7eKCjaWcVibUnyL3CTjC3PwRMYqk9JChFkSe1vn0=; b=wdLu3fESykrAo7hyYGyxKcpB3yTD8QVyannpKV3CjY+Z2GVqmPPi8jR4M67DShZADL 6Pq3jau8vNF5IXxgCaOed/v7Q5hZLeZkDdZ/Y7DZumRpgpQeLJRjvRh1XcyOQcDSsBZl UO6/lLdBbospZHbFD0MqhbkmC2YcRspU5jLnVSIJb5kpW2YA69rtTDnQrRXQIPBRcev1 mItVVRLvfTlte1Ec7BDycO3Pjw/Gnc2oxv0T7rx+8VUWNHyOYvcU0ml06928FpT/yd9A lCS4xiShl4warx7PeV0L2DTpC0RapbNKBbmYpNTsHq8mSGtGkkxiDpBjh1s3swxSgJup gzSA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k17-20020a056a00169100b00659af029c39si858764pfc.90.2023.07.20.04.45.51; Thu, 20 Jul 2023 04:46:04 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231337AbjGTLaL (ORCPT + 99 others); Thu, 20 Jul 2023 07:30:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37046 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229709AbjGTLaJ (ORCPT ); Thu, 20 Jul 2023 07:30:09 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id B2622B7 for ; Thu, 20 Jul 2023 04:30:08 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BA0301FB; Thu, 20 Jul 2023 04:30:51 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D5E803F6C4; Thu, 20 Jul 2023 04:30:06 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , Yin Fengwei , David Hildenbrand , Yu Zhao , Yang Shi , "Huang, Ying" , Zi Yan Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 0/3] Optimize large folio interaction with deferred split Date: Thu, 20 Jul 2023 12:29:52 +0100 Message-Id: <20230720112955.643283-1-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771939891656462673 X-GMAIL-MSGID: 1771939891656462673 Hi All, [Sending v3 to replace yesterday's v2 after Yu Zhou's feedback] This is v3 of a small series in support of my work to enable the use of large folios for anonymous memory (known as "FLEXIBLE_THP" or "LARGE_ANON_FOLIO") [1]. It first makes it possible to add large, non-pmd-mappable folios to the deferred split queue. Then it modifies zap_pte_range() to batch-remove spans of physically contiguous pages from the rmap, which means that in the common case, we elide the need to ever put the folio on the deferred split queue, thus reducing lock contention and improving performance. This becomes more visible once we have lots of large anonymous folios in the system, and Huang Ying has suggested solving this needs to be a prerequisit for merging the main FLEXIBLE_THP/LARGE_ANON_FOLIO work. The series applies on top of v6.5-rc2 and a branch is available at [2]. I don't have a full test run with the latest versions of all the patches on top of the latest baseline, so not posting results formally. I can get these if people feel they are neccessary though. But anecdotally, for the kernel compilation workload, this series reduces kernel time by ~4% and reduces real-time by ~0.4%, compared with [1]. Changes since v2 [3] -------------------- - patch 2: - Reworked at Yu Zhou's request to reduce duplicated code. - page_remove_rmap() now forwards to folio_remove_rmap_range() for the !compound (PMD mapped) case. - Both page_remove_rmap() and folio_remove_rmap_range() share common epilogue via new helper function __remove_rmap_finish(). - As a result of the changes, I've removed the previous Reviewed-bys. - other 2 patches are unchanged. Changes since v1 [4] -------------------- - patch 2: Modified doc comment for folio_remove_rmap_range() - patch 2: Hoisted _nr_pages_mapped manipulation out of page loop so its now modified once per folio_remove_rmap_range() call. - patch 2: Added check that page range is fully contained by folio in folio_remove_rmap_range() - patch 2: Fixed some nits raised by Huang, Ying for folio_remove_rmap_range() - patch 3: Support batch-zap of all anon pages, not just those in anon vmas - patch 3: Renamed various functions to make their use clear - patch 3: Various minor refactoring/cleanups - Added Reviewed-By tags - thanks! [1] https://lore.kernel.org/linux-mm/20230714160407.4142030-1-ryan.roberts@arm.com/ [2] https://gitlab.arm.com/linux-arm/linux-rr/-/tree/features/granule_perf/deferredsplit-lkml_v3 [3] https://lore.kernel.org/linux-mm/20230719135450.545227-1-ryan.roberts@arm.com/ [4] https://lore.kernel.org/linux-mm/20230717143110.260162-1-ryan.roberts@arm.com/ Thanks, Ryan Ryan Roberts (3): mm: Allow deferred splitting of arbitrary large anon folios mm: Implement folio_remove_rmap_range() mm: Batch-zap large anonymous folio PTE mappings include/linux/rmap.h | 2 + mm/memory.c | 120 +++++++++++++++++++++++++++++++++++++++++ mm/rmap.c | 125 ++++++++++++++++++++++++++++++++----------- 3 files changed, 217 insertions(+), 30 deletions(-) --- 2.25.1