From patchwork Mon Dec 18 10:50:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 180261 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:24d3:b0:fb:cd0c:d3e with SMTP id r19csp1154295dyi; Mon, 18 Dec 2023 02:51:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IFRPj73IKFs66Nx3aKAiJBJYKw28WIqHg+EkBHqQ5fax8+A3zol4p4pJ4KAUHr+S22VNxdS X-Received: by 2002:a17:902:f543:b0:1d0:700b:3f88 with SMTP id h3-20020a170902f54300b001d0700b3f88mr22490745plf.66.1702896708446; Mon, 18 Dec 2023 02:51:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702896708; cv=none; d=google.com; s=arc-20160816; b=BTtMXHdhta0G+UkDDZqn+BigZPudLp3YMSW7mv2+Jf1Tu0RN3vtYLIjCWjKjxt/Uu4 FTnzc2X1pwD7upW+svNwYrJ2Xeh1gPS3er8CLxI7evZ8Wrs1Q+MY2xpyThVruvpqWY24 DaCffiAAz399QtTCDHcHUy69i5sUykB9FxdPyDw7JxuyDLTEIZpXTZ+w1rJmt7EZ1JXv 4x2VdTY13gblxIF+O+5dp1NUTiMZO3hYzyDqs2W38F0XunW8c2ZdDuYgG61QJFVadMbZ aJ6TV0ooctf2rso0Qp8lJkNdetzBCrhmZ1IRlIl12o/j8UFWzA+45n4K/Iidz6dHidOI yxKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=nHYd6pL2au0ylISSnRYYRDrN6i/JAMSJRS7tvhKFDUA=; fh=a3gZESXuO1ptbCtzvzEnZL5xUl+WC4Vy5hoF2yaOXSs=; b=xJwXaQwOOV4g1Xw3Y+41TlS+e/4dFg5QHHAhQBaZK1jsqUh69SbGtqasj45eNdYG/i vTf3L30d80sFf5MBcGPJllPVfOdQ+rY3pxDiETIDU8AeG3yxbDOdxMn/nikelr8CQv3h in5z5dCWR7mm3jVlm/TIuYZdTqgLN67cSklrJEYAQP9B77begNCYRqBjsB40N/YIgzzy 4bf+2+d2ULJX1AT8YdpnCjD5KCK+ADIvfn8SvfqRuB9QsqymNKOrh/QrO8uEQgVTulh2 rDHawRV5483Gd4KICPolDRhPPCcGSKLdsq6QRLWu0OrXiSfWS6sgaZ7Mx73DYzW100ST 35SA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-3370-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-3370-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id by4-20020a056a02058400b005cd8aaec136si3048107pgb.307.2023.12.18.02.51.48 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Dec 2023 02:51:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-3370-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-3370-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-3370-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 3A10E283FDE for ; Mon, 18 Dec 2023 10:51:48 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9D5CA1A284; Mon, 18 Dec 2023 10:51:25 +0000 (UTC) X-Original-To: linux-kernel@vger.kernel.org Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 909BB199CC for ; Mon, 18 Dec 2023 10:51:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 754E92F4; Mon, 18 Dec 2023 02:52:06 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9BE873F738; Mon, 18 Dec 2023 02:51:18 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 01/16] mm: thp: Batch-collapse PMD with set_ptes() Date: Mon, 18 Dec 2023 10:50:45 +0000 Message-Id: <20231218105100.172635-2-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231218105100.172635-1-ryan.roberts@arm.com> References: <20231218105100.172635-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785616619009231823 X-GMAIL-MSGID: 1785616619009231823 Refactor __split_huge_pmd_locked() so that a present PMD can be collapsed to PTEs in a single batch using set_ptes(). It also provides a future opportunity to batch-add the folio to the rmap using David's new batched rmap APIs. This should improve performance a little bit, but the real motivation is to remove the need for the arm64 backend to have to fold the contpte entries. Instead, since the ptes are set as a batch, the contpte blocks can be initially set up pre-folded (once the arm64 contpte support is added in the next few patches). This leads to noticeable performance improvement during split. Signed-off-by: Ryan Roberts Acked-by: David Hildenbrand --- mm/huge_memory.c | 59 ++++++++++++++++++++++++++++-------------------- 1 file changed, 34 insertions(+), 25 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 6be1a380a298..fbf7e95ea983 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2535,15 +2535,16 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, pte = pte_offset_map(&_pmd, haddr); VM_BUG_ON(!pte); - for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { - pte_t entry; - /* - * Note that NUMA hinting access restrictions are not - * transferred to avoid any possibility of altering - * permissions across VMAs. - */ - if (freeze || pmd_migration) { + + /* + * Note that NUMA hinting access restrictions are not transferred to + * avoid any possibility of altering permissions across VMAs. + */ + if (freeze || pmd_migration) { + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t entry; swp_entry_t swp_entry; + if (write) swp_entry = make_writable_migration_entry( page_to_pfn(page + i)); @@ -2562,28 +2563,36 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = pte_swp_mksoft_dirty(entry); if (uffd_wp) entry = pte_swp_mkuffd_wp(entry); - } else { - entry = mk_pte(page + i, READ_ONCE(vma->vm_page_prot)); - if (write) - entry = pte_mkwrite(entry, vma); + + VM_WARN_ON(!pte_none(ptep_get(pte + i))); + set_pte_at(mm, addr, pte + i, entry); + } + } else { + pte_t entry; + + entry = mk_pte(page, READ_ONCE(vma->vm_page_prot)); + if (write) + entry = pte_mkwrite(entry, vma); + if (!young) + entry = pte_mkold(entry); + /* NOTE: this may set soft-dirty too on some archs */ + if (dirty) + entry = pte_mkdirty(entry); + if (soft_dirty) + entry = pte_mksoft_dirty(entry); + if (uffd_wp) + entry = pte_mkuffd_wp(entry); + + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { if (anon_exclusive) SetPageAnonExclusive(page + i); - if (!young) - entry = pte_mkold(entry); - /* NOTE: this may set soft-dirty too on some archs */ - if (dirty) - entry = pte_mkdirty(entry); - if (soft_dirty) - entry = pte_mksoft_dirty(entry); - if (uffd_wp) - entry = pte_mkuffd_wp(entry); page_add_anon_rmap(page + i, vma, addr, RMAP_NONE); + VM_WARN_ON(!pte_none(ptep_get(pte + i))); } - VM_BUG_ON(!pte_none(ptep_get(pte))); - set_pte_at(mm, addr, pte, entry); - pte++; + + set_ptes(mm, haddr, pte, entry, HPAGE_PMD_NR); } - pte_unmap(pte - 1); + pte_unmap(pte); if (!pmd_migration) page_remove_rmap(page, vma, true);