From patchwork Mon Jul 3 13:53:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 115384 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp555897vqx; Mon, 3 Jul 2023 07:14:42 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5bf1g6WVcuD4H/j4OL+rQqaBOh5htV4a87vRCcW519WHVCEbJY6B5k0mevbxsZ+IMqYn5C X-Received: by 2002:a05:6a20:4413:b0:11b:3e33:d2ce with SMTP id ce19-20020a056a20441300b0011b3e33d2cemr16234664pzb.1.1688393682231; Mon, 03 Jul 2023 07:14:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688393682; cv=none; d=google.com; s=arc-20160816; b=TddTwgsBGBnFC8i6HDlb3gO27qLvYLocLqB2pThk5bPVKKH37nLEobhT1Nf0mKwJIF fGMkRm4JngTSUJANnj9NgqeVWFtt4V71vq6FC6IFrIUTfsLJOagFrR/5FdRCY6PQAu6m wm5NuZ/w+6Jtoi6ACQMYyILz+5Vxx4gnnqjxVwVUEB0y1qOS/1wO//SdE4dRP2w+iR+P CYH8dLj3+RuteIvCIVK2btr4d5/igLBBZkeFgiBOQLXmVUwKeeuNOn1r75wFvcoTDc5Y 2rnnjgjyJzn7l1o+mx/HSdAy/T99u4AM819q+F8mCJZ3ma+KfkK8DX4SW/dUllEdVsq+ NdSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=oKmEDpPRks/ilB7GL5ZE6RM4bSXQ4CYbCYmtoGtRwVc=; fh=W2gTym8Tes+4XZMJyNdq9yGC4n1yJ+vTSxJnRC1NAwA=; b=xFOTIhZl2qOEsCHe6GDljlCTR+0dWxHzfCK3DgSgCvJMNrBb68GZv7MrZ9J9vNkg4y RelLSIgX2uBxE/ahEdqzwts3OBs7hfrFfCdIT3c2Vp0QRJMbTy0qlv5VKrtGUXlbjGad wJ72ab6cbx0i6impXa7wVD+jH69n7os2Fp3rf1tdtG7CWjG2ns5mJfoHcM3tsmmUeTwY dk5c2yN0GQWiyc9a32Eg5oaRo2XG5aPau3mCbNOcvhzvOwt0KlPz6oqf5kK6aUI77gye yNwS/9LBte5sqKf0sNTsUOkdwhr2NHJDmhIEYlxcyDwF18i+dXk0rd97Ldytxo+mLW2M giTA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c194-20020a621ccb000000b0062565210347si6746800pfc.275.2023.07.03.07.14.29; Mon, 03 Jul 2023 07:14:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231232AbjGCNx4 (ORCPT + 99 others); Mon, 3 Jul 2023 09:53:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230252AbjGCNxw (ORCPT ); Mon, 3 Jul 2023 09:53:52 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 0716010C8 for ; Mon, 3 Jul 2023 06:53:45 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 71661C14; Mon, 3 Jul 2023 06:54:27 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EDAF83F73F; Mon, 3 Jul 2023 06:53:42 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 1/5] mm: Non-pmd-mappable, large folios for folio_add_new_anon_rmap() Date: Mon, 3 Jul 2023 14:53:26 +0100 Message-Id: <20230703135330.1865927-2-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230703135330.1865927-1-ryan.roberts@arm.com> References: <20230703135330.1865927-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770409093444780390?= X-GMAIL-MSGID: =?utf-8?q?1770409093444780390?= In preparation for FLEXIBLE_THP support, improve folio_add_new_anon_rmap() to allow a non-pmd-mappable, large folio to be passed to it. In this case, all contained pages are accounted using the "small" pages scheme. Signed-off-by: Ryan Roberts Reviewed-by: Yu Zhao Reviewed-by: Yin, Fengwei --- mm/rmap.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 1d8369549424..82ef5ba363d1 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1278,31 +1278,43 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, * This means the inc-and-test can be bypassed. * The folio does not have to be locked. * - * If the folio is large, it is accounted as a THP. As the folio + * If the folio is pmd-mappable, it is accounted as a THP. As the folio * is new, it's assumed to be mapped exclusively by a single process. */ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address) { - int nr; + int nr = folio_nr_pages(folio); + int i; + struct page *page; - VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma); + VM_BUG_ON_VMA(address < vma->vm_start || + address + (nr << PAGE_SHIFT) > vma->vm_end, vma); __folio_set_swapbacked(folio); - if (likely(!folio_test_pmd_mappable(folio))) { + if (!folio_test_large(folio)) { /* increment count (starts at -1) */ atomic_set(&folio->_mapcount, 0); - nr = 1; + __page_set_anon_rmap(folio, &folio->page, vma, address, 1); + } else if (!folio_test_pmd_mappable(folio)) { + /* increment count (starts at 0) */ + atomic_set(&folio->_nr_pages_mapped, nr); + + page = &folio->page; + for (i = 0; i < nr; i++, page++, address += PAGE_SIZE) { + /* increment count (starts at -1) */ + atomic_set(&page->_mapcount, 0); + __page_set_anon_rmap(folio, page, vma, address, 1); + } } else { /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); atomic_set(&folio->_nr_pages_mapped, COMPOUND_MAPPED); - nr = folio_nr_pages(folio); __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr); + __page_set_anon_rmap(folio, &folio->page, vma, address, 1); } __lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr); - __page_set_anon_rmap(folio, &folio->page, vma, address, 1); } /** From patchwork Mon Jul 3 13:53:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 115376 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp543809vqx; Mon, 3 Jul 2023 07:00:18 -0700 (PDT) X-Google-Smtp-Source: APBJJlF8ELrabh8t8I/tCn1tD6JiuzrlGgwZwybRL1oG5yJYFYJwx3yOnFaHf8HqkNgLG66S7cVv X-Received: by 2002:a05:6a00:280a:b0:676:2a5c:7bc5 with SMTP id bl10-20020a056a00280a00b006762a5c7bc5mr11850014pfb.1.1688392817617; Mon, 03 Jul 2023 07:00:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688392817; cv=none; d=google.com; s=arc-20160816; b=by+nOyG0fuyubRXYI2uB3TnULuFN3Z4rmQwohlzidr4fTafnpBGn+PtCY+nnZHzl1x WwK5OzSQUmkjKyea/udFNHgOccqLr2GQzocQ0VVy4Z2O0cH7HiIES0pmnsMjWgF7LFsg 2N7n/f2pVO9BVa8aV/rNJehTOZiWQaqhZv8QbFLqqF2NO6pm/1M6oveW6Faqbf+t3yy4 q2RaNGDBj6AkwK7JpLno/gkgcBc1u0WvmMWWVlLeqr9Jc1Q3LhAp4lcbf/SLtRCezYI8 gACG2dV46QMe6h9yhJJt0ZVAM7oPREb4kDGM4c619cadEWIoAt6v03TAkJvMijRTFkEX nILg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=qDlU2rGsKBPfwZSw0bLPzWmjupT8xRy+8rDXmR7/VPM=; fh=W2gTym8Tes+4XZMJyNdq9yGC4n1yJ+vTSxJnRC1NAwA=; b=xxXd3AkQciKtwCxEPzEAvtm4lK0B8f0bRUuVC+vRUFzpiYZ4pdMeSmT8C3XrDKOm2U oA5gBNsLOzFBJStTUs5y6e345dWUA+kQnfn7az0fqlXHoNSBpPKbarNb2zMy3AVEwcte qh7fcdcrYstw/7EnPF1dfjR+1l1A38D5HK46P/yB3hHE7RpCMEaq1Xmu+E/Al9BSxWby 2tiZV2zjArOw0PtAS0zstHha/cYV8oBiXw/jE/s/Aqqol+oeY/p0ngFF5vyCvAzxfDsq URDVoHauTILt6W+lbEG13pAn17Rxws7xnU2pe8l20WGPy5Z2zj9KD2r+wvBfTfi/J19w SFsA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b21-20020a056a000cd500b0067c2b7b23e3si13719076pfv.345.2023.07.03.06.59.51; Mon, 03 Jul 2023 07:00:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231319AbjGCNx7 (ORCPT + 99 others); Mon, 3 Jul 2023 09:53:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229849AbjGCNx4 (ORCPT ); Mon, 3 Jul 2023 09:53:56 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id EEBF310D9 for ; Mon, 3 Jul 2023 06:53:46 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6A42A1424; Mon, 3 Jul 2023 06:54:29 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E6A393F73F; Mon, 3 Jul 2023 06:53:44 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 2/5] mm: Allow deferred splitting of arbitrary large anon folios Date: Mon, 3 Jul 2023 14:53:27 +0100 Message-Id: <20230703135330.1865927-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230703135330.1865927-1-ryan.roberts@arm.com> References: <20230703135330.1865927-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770408187148080429?= X-GMAIL-MSGID: =?utf-8?q?1770408187148080429?= With the introduction of large folios for anonymous memory, we would like to be able to split them when they have unmapped subpages, in order to free those unused pages under memory pressure. So remove the artificial requirement that the large folio needed to be at least PMD-sized. Signed-off-by: Ryan Roberts Reviewed-by: Yu Zhao Reviewed-by: Yin Fengwei --- mm/rmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/rmap.c b/mm/rmap.c index 82ef5ba363d1..bbcb2308a1c5 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1474,7 +1474,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, * page of the folio is unmapped and at least one page * is still mapped. */ - if (folio_test_pmd_mappable(folio) && folio_test_anon(folio)) + if (folio_test_large(folio) && folio_test_anon(folio)) if (!compound || nr < nr_pmdmapped) deferred_split_folio(folio); } From patchwork Mon Jul 3 13:53:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 115377 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp545160vqx; Mon, 3 Jul 2023 07:01:40 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5J3jUjZ8Hvvk1KINRC8t/Izr3VRpWjyCleUoKlamNWfKXmarkOmb8gAwz5ZPemXmGD1dWN X-Received: by 2002:a05:6a20:8e19:b0:11a:efaa:eb88 with SMTP id y25-20020a056a208e1900b0011aefaaeb88mr14204642pzj.3.1688392900232; Mon, 03 Jul 2023 07:01:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688392900; cv=none; d=google.com; s=arc-20160816; b=VFPLkcDB5a9e38W/cpFDArnWnpPGhdIKasi+NEEntdZ0kbsGxFZ2LSawc7GiotRXFE wi2V4UmpjpeSk5a9kCRBbvDvLpW+m6sRArW6Fn/vZCJpgitdVLdZnuV2/wRv/D1TD5TI e31swDiyHQBIoR/yWOlttmll3wN+ZsyTP2LVrV2oBPDfxg9oEohKTlE3NoGJaonpQ8IB ayxwgOwlj7gh/MocYQy1QBdo2xjBD/DnA/NNP/7sA/pAx2ijHoImP1XltQZLx0y6In6S DoZ49Jx/dXItwLFdIvIbJqyaFbuDHxFR9jpESalPDJJDkvX4igu6m2hE0U3cTa3Jl6LC Aqqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=USdbSWQAhhNgz/kbRbFMcj5UNwofuLjMnykBwRvmpy4=; fh=W2gTym8Tes+4XZMJyNdq9yGC4n1yJ+vTSxJnRC1NAwA=; b=XbCpRYA2PlB5D6lVTCmyMFwXW691VYUGsNKdxMK6lB0cryGnP0C+NpusPsRpUWV808 uOFMLngundKPu0JQByGrMPGejnwOD4evFWZp1WOf7RPOs2cuNjX0Ti8sCp5R3uq9Z1xE LSN5AevFkSTTQuv7WxwBCvZ9LEtlxb1TvQzV3ol71YT0887vzYgxNv6ZZUDZnYJ5Dlc5 QCFkZpxOiDD2NloIvD8DDUENFHvgBJZVNRXF+TY2KXUeYcjE5SOo00++lFdmUFFc6Gh8 8AW/HUdAYqlxbq9BpjRECOsUzHXH7MUdtKQ2s0F/DZ/lpQcyulGq1Y5SZSG3wlTAf3+r LlHQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b21-20020a056a000cd500b0067c2b7b23e3si13719076pfv.345.2023.07.03.07.01.21; Mon, 03 Jul 2023 07:01:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231139AbjGCNyI (ORCPT + 99 others); Mon, 3 Jul 2023 09:54:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230037AbjGCNyF (ORCPT ); Mon, 3 Jul 2023 09:54:05 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 7B94910FA for ; Mon, 3 Jul 2023 06:53:49 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6399A11FB; Mon, 3 Jul 2023 06:54:31 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DF28C3F73F; Mon, 3 Jul 2023 06:53:46 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 3/5] mm: Default implementation of arch_wants_pte_order() Date: Mon, 3 Jul 2023 14:53:28 +0100 Message-Id: <20230703135330.1865927-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230703135330.1865927-1-ryan.roberts@arm.com> References: <20230703135330.1865927-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770408273875915329?= X-GMAIL-MSGID: =?utf-8?q?1770408273875915329?= arch_wants_pte_order() can be overridden by the arch to return the preferred folio order for pte-mapped memory. This is useful as some architectures (e.g. arm64) can coalesce TLB entries when the physical memory is suitably contiguous. The first user for this hint will be FLEXIBLE_THP, which aims to allocate large folios for anonymous memory to reduce page faults and other per-page operation costs. Here we add the default implementation of the function, used when the architecture does not define it, which returns the order corresponding to 64K. Signed-off-by: Ryan Roberts Reviewed-by: Yu Zhao --- include/linux/pgtable.h | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index a661a17173fa..f7e38598f20b 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -13,6 +13,7 @@ #include #include #include +#include #if 5 - defined(__PAGETABLE_P4D_FOLDED) - defined(__PAGETABLE_PUD_FOLDED) - \ defined(__PAGETABLE_PMD_FOLDED) != CONFIG_PGTABLE_LEVELS @@ -336,6 +337,18 @@ static inline bool arch_has_hw_pte_young(void) } #endif +#ifndef arch_wants_pte_order +/* + * Returns preferred folio order for pte-mapped memory. Must be in range [0, + * PMD_SHIFT-PAGE_SHIFT) and must not be order-1 since THP requires large folios + * to be at least order-2. + */ +static inline int arch_wants_pte_order(struct vm_area_struct *vma) +{ + return ilog2(SZ_64K >> PAGE_SHIFT); +} +#endif + #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address, From patchwork Mon Jul 3 13:53:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 115378 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp545770vqx; Mon, 3 Jul 2023 07:02:16 -0700 (PDT) X-Google-Smtp-Source: APBJJlGm1gGYePV/OSJ4F4lMSKoQq0M2pSk77BC6VQMo1xHSthjcUNCJSmx3vmTbQdCAMe42PYJs X-Received: by 2002:a17:902:c9c5:b0:1b8:50a9:6874 with SMTP id q5-20020a170902c9c500b001b850a96874mr10298867pld.5.1688392936465; Mon, 03 Jul 2023 07:02:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688392936; cv=none; d=google.com; s=arc-20160816; b=mZg9gwWGF1olqJOEPoeZ4oMpMhsCkP5JfV9Is1ZRQTA5eKOLdmIEOME9FdOmaM69rr EsmpYRkHVrYg5KkcfYvk+atoqLMgs5mEjlG7lSaop2fKByb4ve9MG4B56RXDZRTM9cjU rnj3KMTbKJf3DISuLS9qNj3HYax8FcFb0S/7NisF7L5J9FjUPvOJ9IS8M2+qGFrRVUKn 9bRpmQFBNuu2ApY2A3VumEkrTvHve1N9H7IqEtQmy8eu7UTKhy+7N/mznVGaMANkSGa/ VrXFBozxO6dZF3WHgcFzsd1axLZbIq7yR2vECYXx2hb8QKITZnenIXkY1xFFjvPV8UkJ vY6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=M42aKf0Dh3v26rYqL7fJB4T37PH00qdaAFo246Anczo=; fh=W2gTym8Tes+4XZMJyNdq9yGC4n1yJ+vTSxJnRC1NAwA=; b=KJYvMHYvtmwNSOIstm1YfTuUVCgL1khqVmA8yFQ6f3eF0/dXiIcCQO0Qc+/eWdOknG 40k9YMqLAo7OmoIgBUiarJgmizLEWfmbQ4uVjwtLjMyJYv+q6KSiVtLH1oCh3okn1zXF 6XwVzPShFsTwLIM8NN5oYIp6Cymt/6RTotC8ZcCf9PUc+RmJB/8N2egpP/BQEL7R1uMX yiAMlf+KKz4XmGg3oYoiPPcjq6njdZ/S4gSoIwk4GUpmxf+frMF9nubFs+QAb/E44wO6 ehnpOba1K+eASUef8IClRWA/nVsjdkkbaLOcjdKOJKXZHZPt34xZ3jv1olAQuJ9xQ/QI 4xLQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t2-20020a1709027fc200b001b261eba754si9255541plb.165.2023.07.03.07.01.53; Mon, 03 Jul 2023 07:02:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230037AbjGCNyN (ORCPT + 99 others); Mon, 3 Jul 2023 09:54:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230262AbjGCNyL (ORCPT ); Mon, 3 Jul 2023 09:54:11 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 5971E1706 for ; Mon, 3 Jul 2023 06:53:51 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 76F33143D; Mon, 3 Jul 2023 06:54:33 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D88B03F73F; Mon, 3 Jul 2023 06:53:48 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 4/5] mm: FLEXIBLE_THP for improved performance Date: Mon, 3 Jul 2023 14:53:29 +0100 Message-Id: <20230703135330.1865927-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230703135330.1865927-1-ryan.roberts@arm.com> References: <20230703135330.1865927-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770408311193829091?= X-GMAIL-MSGID: =?utf-8?q?1770408311193829091?= Introduce FLEXIBLE_THP feature, which allows anonymous memory to be allocated in large folios of a specified order. All pages of the large folio are pte-mapped during the same page fault, significantly reducing the number of page faults. The number of per-page operations (e.g. ref counting, rmap management lru list management) are also significantly reduced since those ops now become per-folio. The new behaviour is hidden behind the new FLEXIBLE_THP Kconfig, which defaults to disabled for now; there is a long list of todos to make FLEXIBLE_THP robust with existing features (e.g. compaction, mlock, some madvise ops, etc). These items will be tackled in subsequent patches. When enabled, the preferred folio order is as returned by arch_wants_pte_order(), which may be overridden by the arch as it sees fit. Some architectures (e.g. arm64) can coalsece TLB entries if a contiguous set of ptes map physically contigious, naturally aligned memory, so this mechanism allows the architecture to optimize as required. If the preferred order can't be used (e.g. because the folio would breach the bounds of the vma, or because ptes in the region are already mapped) then we fall back to a suitable lower order. Signed-off-by: Ryan Roberts --- mm/Kconfig | 10 ++++ mm/memory.c | 168 ++++++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 165 insertions(+), 13 deletions(-) diff --git a/mm/Kconfig b/mm/Kconfig index 7672a22647b4..1c06b2c0a24e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -822,6 +822,16 @@ config READ_ONLY_THP_FOR_FS support of file THPs will be developed in the next few release cycles. +config FLEXIBLE_THP + bool "Flexible order THP" + depends on TRANSPARENT_HUGEPAGE + default n + help + Use large (bigger than order-0) folios to back anonymous memory where + possible, even if the order of the folio is smaller than the PMD + order. This reduces the number of page faults, as well as other + per-page overheads to improve performance for many workloads. + endif # TRANSPARENT_HUGEPAGE # diff --git a/mm/memory.c b/mm/memory.c index fb30f7523550..abe2ea94f3f5 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3001,6 +3001,116 @@ static vm_fault_t fault_dirty_shared_page(struct vm_fault *vmf) return 0; } +#ifdef CONFIG_FLEXIBLE_THP +/* + * Allocates, zeros and returns a folio of the requested order for use as + * anonymous memory. + */ +static struct folio *alloc_anon_folio(struct vm_area_struct *vma, + unsigned long addr, int order) +{ + gfp_t gfp; + struct folio *folio; + + if (order == 0) + return vma_alloc_zeroed_movable_folio(vma, addr); + + gfp = vma_thp_gfp_mask(vma); + folio = vma_alloc_folio(gfp, order, vma, addr, true); + if (folio) + clear_huge_page(&folio->page, addr, folio_nr_pages(folio)); + + return folio; +} + +/* + * Preferred folio order to allocate for anonymous memory. + */ +#define max_anon_folio_order(vma) arch_wants_pte_order(vma) +#else +#define alloc_anon_folio(vma, addr, order) \ + vma_alloc_zeroed_movable_folio(vma, addr) +#define max_anon_folio_order(vma) 0 +#endif + +/* + * Returns index of first pte that is not none, or nr if all are none. + */ +static inline int check_ptes_none(pte_t *pte, int nr) +{ + int i; + + for (i = 0; i < nr; i++) { + if (!pte_none(ptep_get(pte++))) + return i; + } + + return nr; +} + +static int calc_anon_folio_order_alloc(struct vm_fault *vmf, int order) +{ + /* + * The aim here is to determine what size of folio we should allocate + * for this fault. Factors include: + * - Order must not be higher than `order` upon entry + * - Folio must be naturally aligned within VA space + * - Folio must be fully contained inside one pmd entry + * - Folio must not breach boundaries of vma + * - Folio must not overlap any non-none ptes + * + * Additionally, we do not allow order-1 since this breaks assumptions + * elsewhere in the mm; THP pages must be at least order-2 (since they + * store state up to the 3rd struct page subpage), and these pages must + * be THP in order to correctly use pre-existing THP infrastructure such + * as folio_split(). + * + * Note that the caller may or may not choose to lock the pte. If + * unlocked, the result is racy and the user must re-check any overlap + * with non-none ptes under the lock. + */ + + struct vm_area_struct *vma = vmf->vma; + int nr; + unsigned long addr; + pte_t *pte; + pte_t *first_set = NULL; + int ret; + + order = min(order, PMD_SHIFT - PAGE_SHIFT); + + for (; order > 1; order--) { + nr = 1 << order; + addr = ALIGN_DOWN(vmf->address, nr << PAGE_SHIFT); + pte = vmf->pte - ((vmf->address - addr) >> PAGE_SHIFT); + + /* Check vma bounds. */ + if (addr < vma->vm_start || + addr + (nr << PAGE_SHIFT) > vma->vm_end) + continue; + + /* Ptes covered by order already known to be none. */ + if (pte + nr <= first_set) + break; + + /* Already found set pte in range covered by order. */ + if (pte <= first_set) + continue; + + /* Need to check if all the ptes are none. */ + ret = check_ptes_none(pte, nr); + if (ret == nr) + break; + + first_set = pte + ret; + } + + if (order == 1) + order = 0; + + return order; +} + /* * Handle write page faults for pages that can be reused in the current vma * @@ -3073,7 +3183,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) goto oom; if (is_zero_pfn(pte_pfn(vmf->orig_pte))) { - new_folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); + new_folio = alloc_anon_folio(vma, vmf->address, 0); if (!new_folio) goto oom; } else { @@ -4040,6 +4150,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) struct folio *folio; vm_fault_t ret = 0; pte_t entry; + int order; + int pgcount; + unsigned long addr; /* File mapping without ->vm_ops ? */ if (vma->vm_flags & VM_SHARED) @@ -4081,24 +4194,51 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); return handle_userfault(vmf, VM_UFFD_MISSING); } - goto setpte; + if (uffd_wp) + entry = pte_mkuffd_wp(entry); + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); + + /* No need to invalidate - it was non-present before */ + update_mmu_cache(vma, vmf->address, vmf->pte); + goto unlock; + } + + /* + * If allocating a large folio, determine the biggest suitable order for + * the VMA (e.g. it must not exceed the VMA's bounds, it must not + * overlap with any populated PTEs, etc). We are not under the ptl here + * so we will need to re-check that we are not overlapping any populated + * PTEs once we have the lock. + */ + order = uffd_wp ? 0 : max_anon_folio_order(vma); + if (order > 0) { + vmf->pte = pte_offset_map(vmf->pmd, vmf->address); + order = calc_anon_folio_order_alloc(vmf, order); + pte_unmap(vmf->pte); } - /* Allocate our own private page. */ + /* Allocate our own private folio. */ if (unlikely(anon_vma_prepare(vma))) goto oom; - folio = vma_alloc_zeroed_movable_folio(vma, vmf->address); + folio = alloc_anon_folio(vma, vmf->address, order); + if (!folio && order > 0) { + order = 0; + folio = alloc_anon_folio(vma, vmf->address, order); + } if (!folio) goto oom; + pgcount = 1 << order; + addr = ALIGN_DOWN(vmf->address, pgcount << PAGE_SHIFT); + if (mem_cgroup_charge(folio, vma->vm_mm, GFP_KERNEL)) goto oom_free_page; folio_throttle_swaprate(folio, GFP_KERNEL); /* * The memory barrier inside __folio_mark_uptodate makes sure that - * preceding stores to the page contents become visible before - * the set_pte_at() write. + * preceding stores to the folio contents become visible before + * the set_ptes() write. */ __folio_mark_uptodate(folio); @@ -4107,11 +4247,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) if (vma->vm_flags & VM_WRITE) entry = pte_mkwrite(pte_mkdirty(entry)); - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl); if (vmf_pte_changed(vmf)) { update_mmu_tlb(vma, vmf->address, vmf->pte); goto release; + } else if (order > 0 && check_ptes_none(vmf->pte, pgcount) != pgcount) { + goto release; } ret = check_stable_address_space(vma->vm_mm); @@ -4125,16 +4266,17 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) return handle_userfault(vmf, VM_UFFD_MISSING); } - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_ref_add(folio, pgcount - 1); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, pgcount); + folio_add_new_anon_rmap(folio, vma, addr); folio_add_lru_vma(folio, vma); -setpte: + if (uffd_wp) entry = pte_mkuffd_wp(entry); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry); + set_ptes(vma->vm_mm, addr, vmf->pte, entry, pgcount); /* No need to invalidate - it was non-present before */ - update_mmu_cache(vma, vmf->address, vmf->pte); + update_mmu_cache_range(vma, addr, vmf->pte, pgcount); unlock: pte_unmap_unlock(vmf->pte, vmf->ptl); return ret; From patchwork Mon Jul 3 13:53:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 115379 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp550654vqx; Mon, 3 Jul 2023 07:07:42 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ66l6DgNuD2VFWpmAtkUJl283wWIOGrLhdcqXcp+9au0kew9LP3nS/KHDBUzSj/XpPTvCDP X-Received: by 2002:a05:6830:913:b0:6b8:7653:dd66 with SMTP id v19-20020a056830091300b006b87653dd66mr13066501ott.0.1688393262256; Mon, 03 Jul 2023 07:07:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688393262; cv=none; d=google.com; s=arc-20160816; b=KUOI6JBq2TkZxISnrH3OTlYKjhQUIcGEW3uYE28TrlnjpeugUnvV3YIjNsPcT1DK1P PYZbAOuRF1Z9hWVTHfbhFwEfMwdWS2K/edBOyqm1WL8Ns0n72ihfnKOhx3ojRdUJDfFn jBzc02Ibj861saZO04quCg1Ljr7aeA80hoAk81RqPw6b2qCO/rmlJuLnfUHoECW83x3s 3tS/2j1qAzRwvTN5yTrsNO33b57ISyVTsQ6COw2WYqpceUbPhgXt0x7Kr0RLBQEh1DH2 kwev9xgdfZ5XaET8FS6TFy+xJUvkWJOHjTOVoOiRfN6TRC2SQOSz5VeE0CG9C74c4SyC C1Cg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=0Xs0/r8JxC0IODzuzWxhN6nl3kLwEjLKvoCF58989ko=; fh=W2gTym8Tes+4XZMJyNdq9yGC4n1yJ+vTSxJnRC1NAwA=; b=AFcCFBweN8yFAl+/BXA2qtEX/Dyt7bkFIvVS7LwFFAdNN8l6b9TUyyTv4aWlLtTkqv CmA4mh1bws+/dz0I/ChbImDv0lObyk/HbVanHUjvF90CsiopP5lsKTYM3+VrboKptp4s NReWExSt71i9zOyzjRc0Lvhpgo2e2IMSGrBDVoNK8Aj1TFhOXcGp0lOxE7LMEq8AQYtc /eKwY+IAa8YbXCRTlRTMQ4McPp4gKZ3X8lZTek8GY/w+LY2x6bYyFdHR5RcrNovOACe7 lLp1rDAQtgVV8hh2NTa6Fid/VnRxfPaBINqxExbiRxWxe8qP3IScxaNOEu80WdeGrW5H ka+Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k25-20020a635a59000000b00553813c2df0si18068673pgm.513.2023.07.03.07.07.22; Mon, 03 Jul 2023 07:07:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229973AbjGCNyY (ORCPT + 99 others); Mon, 3 Jul 2023 09:54:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35888 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230094AbjGCNyQ (ORCPT ); Mon, 3 Jul 2023 09:54:16 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 38C171716 for ; Mon, 3 Jul 2023 06:53:53 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6FD552F4; Mon, 3 Jul 2023 06:54:35 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EBD643F73F; Mon, 3 Jul 2023 06:53:50 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Matthew Wilcox , "Kirill A. Shutemov" , Yin Fengwei , David Hildenbrand , Yu Zhao , Catalin Marinas , Will Deacon , Anshuman Khandual , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 5/5] arm64: mm: Override arch_wants_pte_order() Date: Mon, 3 Jul 2023 14:53:30 +0100 Message-Id: <20230703135330.1865927-6-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230703135330.1865927-1-ryan.roberts@arm.com> References: <20230703135330.1865927-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770408653687393009?= X-GMAIL-MSGID: =?utf-8?q?1770408653687393009?= Define an arch-specific override of arch_wants_pte_order() so that when FLEXIBLE_THP is enabled, large folios will be allocated for anonymous memory with an order that is compatible with arm64's contpte mappings. arch_wants_pte_order() returns an order according to the following policy: For the unhinted case, when THP is not requested for the vma, don't allow anything bigger than 64K. This means we don't waste too much memory. Additionally, for 4K pages this is the contpte size, and for 16K, this is (usually) the HPA size when the uarch feature is implemented. For the hinted case, when THP is requested for the vma, allow the contpte size for all page size configurations; 64K for 4K, 2M for 16K and 2M for 64K. Additionally, the THP and NOTHP order constants are defined using Kconfig so it is possible to override them at build time. Signed-off-by: Ryan Roberts --- arch/arm64/Kconfig | 12 ++++++++++++ arch/arm64/include/asm/pgtable.h | 4 ++++ arch/arm64/mm/mmu.c | 8 ++++++++ 3 files changed, 24 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 343e1e1cae10..689c5bf13dc1 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -281,6 +281,18 @@ config ARM64_CONT_PMD_SHIFT default 5 if ARM64_16K_PAGES default 4 +config ARM64_PTE_ORDER_NOTHP + int + default 0 if ARM64_64K_PAGES # 64K (1 page) + default 2 if ARM64_16K_PAGES # 64K (4 pages; benefits from HPA where HW supports it) + default 4 if ARM64_4K_PAGES # 64K (16 pages; eligible for contpte-mapping) + +config ARM64_PTE_ORDER_THP + int + default 5 if ARM64_64K_PAGES # 2M (32 pages; eligible for contpte-mapping) + default 7 if ARM64_16K_PAGES # 2M (128 pages; eligible for contpte-mapping) + default 4 if ARM64_4K_PAGES # 64K (16 pages; eligible for contpte-mapping) + config ARCH_MMAP_RND_BITS_MIN default 14 if ARM64_64K_PAGES default 16 if ARM64_16K_PAGES diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 6fd012663a01..8463d5f9f307 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1117,6 +1117,10 @@ extern pte_t ptep_modify_prot_start(struct vm_area_struct *vma, extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); + +#define arch_wants_pte_order arch_wants_pte_order +extern int arch_wants_pte_order(struct vm_area_struct *vma); + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index af6bc8403ee4..8556c4a9b507 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1481,3 +1481,11 @@ void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte { set_pte_at(vma->vm_mm, addr, ptep, pte); } + +int arch_wants_pte_order(struct vm_area_struct *vma) +{ + if (hugepage_vma_check(vma, vma->vm_flags, false, true, true)) + return CONFIG_ARM64_PTE_ORDER_THP; + else + return CONFIG_ARM64_PTE_ORDER_NOTHP; +}