From patchwork Fri Feb 2 08:07:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 195676 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:9bc1:b0:106:209c:c626 with SMTP id op1csp278011dyc; Fri, 2 Feb 2024 00:08:51 -0800 (PST) X-Google-Smtp-Source: AGHT+IFvxz93C8OqOjjyGR+CIg5JEcBnT2xG6Myzb/uqFCcBwT2Xl/m52ymHaGCpT+WeXub6xXnC X-Received: by 2002:aa7:d055:0:b0:55f:8ddc:6c8b with SMTP id n21-20020aa7d055000000b0055f8ddc6c8bmr688772edo.10.1706861331761; Fri, 02 Feb 2024 00:08:51 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706861331; cv=pass; d=google.com; s=arc-20160816; b=vrSlpxP1BFh246ctHyosqos7zj2TVqMqVN6UyA4Q4ibg7SoJpCZ200HPmDhM2fvUTM 3RGWf57PrZNvSNHalUH5CAG3uAZnbt3Jwp33svUMVaJ+mUGZlO2uj/9Iv9NLEenxmKgZ +QisN447ivxGe/FaTYcFf8pO9MvNk2h3PbUOumSvjXp46Q2xnsJ7w9veiCWVfoRI1Pz4 tRivVV+MFIMPaJeWD9ateN7GdfnI7JlCL+OVd7uTKhKrM0KHeXYrlF+9z4iZwthvvL6j gEDoAP4YpyhJkON82hA9DtvdZE7Rrs+/MLptX0wrQ+t5aPFNIh6icSo+k1pUtY9GXfTa 7vgw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=dr+JmlJli2R+rlUtzp9TTxxnomA8MeE7u4eqm10nX+k=; fh=7FtWjP76BxXFIKi6miwOgA52mr8RrNJwv1jl8gj1aco=; b=DpILZ/vjo+aIDGAUoR6LWN2RFgfRQlMvtCc9Lzp6twcWjpnzMFkHmtwiqG5w8hiOBC oOeXrSJdyKUQLOXZKG2yZgu07eLA/Ysorhn74Ev6QFOg4xlzoJXViD29oJEZu+XOsFYP 6vqX2kU8f/Hw3cHeVhu2fPzuWYkE/4/od+StpyH4IzwpAhHg4NqqsFSAE6KJAs3YOAM7 SfBcdbm2/oyAKQiDmTO8aktZmA2fkepDdzQbe9GJg/G7Gmc6LbjdeeABTW3TS3xritRT GRl9JBbHsvBrjbEh1wfB7oCM0HJGVq6LT6nAT35zuqlAnzm8Nb2+g6T7blGzw6Cnt2cD F0fQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49412-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49412-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com X-Forwarded-Encrypted: i=1; AJvYcCXSGWNXPdtgETMGUZHb/WCa09Rvjpj5wylN5is6N+NDYSl5unPL3WrlLeBlX90suNrooSEvea5WAoLDgkdUWh6ZOMqKAg== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id dh16-20020a0564021d3000b0055fa7f96b6bsi598756edb.313.2024.02.02.00.08.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 00:08:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-49412-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49412-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49412-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 541591F2A857 for ; Fri, 2 Feb 2024 08:08:51 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6069B182C3; Fri, 2 Feb 2024 08:08:24 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D613E17C6E for ; Fri, 2 Feb 2024 08:08:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861302; cv=none; b=eQBX1I+lOmIxRH6jHp8O4/ewl7uHM2v/HkFO9j9JCyn3pz1snHcdWFPZ6J4joLU77TA9lyAYcxTTN+36HdBJIOBTw+3dC7TkYVUHaQ87hQ3m2X2oyniP4e8ZSFHmtqnv/Z9fhlIm1koYIJ0qrD2qXAZ0SqtKMpejXK1NKdwybls= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861302; c=relaxed/simple; bh=crs7YcgUTYuqfO1G2FpVn6hqQ5vqbIBn8D+pZnE5y4s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=AXmWba+vhhMii0ZcRBxasEalutri35btSnONuW2iJCOymFQDcmeBA3yWxhMH1u9Gkkm9dK3slQwOwrGbkUVIZzpQLo6udQ8PMTsVEJxxFyv1u49CwEIk1H8XCQ3frExLPY4sx9tpJd6tZzy+Stq1C+uzto/KTzbpvdrdNdzcvUY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 85221FEC; Fri, 2 Feb 2024 00:09:02 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 99B143F5A1; Fri, 2 Feb 2024 00:08:16 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 01/25] mm: Clarify the spec for set_ptes() Date: Fri, 2 Feb 2024 08:07:32 +0000 Message-Id: <20240202080756.1453939-2-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789773827882459492 X-GMAIL-MSGID: 1789773827882459492 set_ptes() spec implies that it can only be used to set a present pte because it interprets the PFN field to increment it. However, set_pte_at() has been implemented on top of set_ptes() since set_ptes() was introduced, and set_pte_at() allows setting a pte to a not-present state. So clarify the spec to state that when nr==1, new state of pte may be present or not present. When nr>1, new state of all ptes must be present. While we are at it, tighten the spec to set requirements around the initial state of ptes; when nr==1 it may be either present or not-present. But when nr>1 all ptes must initially be not-present. All set_ptes() callsites already conform to this requirement. Stating it explicitly is useful because it allows for a simplification to the upcoming arm64 contpte implementation. Signed-off-by: Ryan Roberts Acked-by: David Hildenbrand --- include/linux/pgtable.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index f0feae7f89fb..5e7eaf8f2b97 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -229,6 +229,10 @@ static inline pte_t pte_next_pfn(pte_t pte) * @pte: Page table entry for the first page. * @nr: Number of pages to map. * + * When nr==1, initial state of pte may be present or not present, and new state + * may be present or not present. When nr>1, initial state of all ptes must be + * not present, and new state must be present. + * * May be overridden by the architecture, or the architecture can define * set_pte() and PFN_PTE_SHIFT. * From patchwork Fri Feb 2 08:07:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 195677 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:9bc1:b0:106:209c:c626 with SMTP id op1csp278280dyc; Fri, 2 Feb 2024 00:09:30 -0800 (PST) X-Google-Smtp-Source: AGHT+IEfJk/sCK6qcWTPdVzMpnUv3zn4TvLaKJ2PR7ltPEdlvxel68ijDa8E8sRyc68Bq6sxfFqW X-Received: by 2002:aa7:c2d4:0:b0:55e:f52d:1825 with SMTP id m20-20020aa7c2d4000000b0055ef52d1825mr759936edp.35.1706861370335; Fri, 02 Feb 2024 00:09:30 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706861370; cv=pass; d=google.com; s=arc-20160816; b=lpLTKqPGmfv7FAPbHyvZgyAuvqBwX34lSl8/LgTYeUkM9sDncLc66ICzn2QuHyInOb M1RN8qcVjn7JKHzYxROHM+bVGNRu6a0fdWKd5sjUKMKNfkFrTJUiuLYFsP0Rrjfx6+Mp aio1kDDz9zZ6nwxo20lJM6ysDVK9MMyxuu3HugCyH4ozvS+D39l1B3aLlL1i49QUmBr9 nrNOC1kLUDo/3SF16wyCQ33IgQOj6xESW1b8AVgQxCvgCX2Q2/Y5RtMzCu0xQ0XP/DqU eAKq1dtaEFs1vIdZhs3T1gbqQMzh1V39o87L3fARpv2PntWv9Us2x2hzDkHLxJZOo4ks ywMQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=iYGxY8gKE5MXdRT9yHHw6Vo1X8bbi/EsI1w4KtEcUbM=; fh=VEeEo0tarD8dzzw2P2Ced0azk7+v1CEIBSx1oED+PBs=; b=UvLoV6wQLAF64XRN75oNJAyS/kXfVrqvGf7vmGCSLRDRMXfg20G2jq5NG1kDLfElDn WifaSjOvsBs5zwLT+0xcIeMp/owIwyQD6SWFUCM1fypqLBEAQ7oXZmYhyK1OoySh+Kym 6z81mixvir2ut5vUi92vefB2zdtWlnyiQGLmc1tfunJ4Rd/gxKoRKpvTY/pRVO9pmioZ Hn6zxZ3HxIYKJGUwV4sk68r60SFqrWIwK7tjbpQOMREWDEqGP6PMFjaxQlbx5Trs74t3 s0S5y0SWv/4IA33oPNgJ/a2Wd4t9Rcz86tJGeHHJ0h8wTLRZF/xUpM1TN1Dzm7nrbtwq zvKQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49414-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49414-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com X-Forwarded-Encrypted: i=1; AJvYcCU7g0e7m+gOuSQrSRvGyhk6Ucsp0mmm4G/wEcI/SPAqLNHCOdykhnWjn12Af8K2MMVnF3JTOi2sv8Q/E/totABna6i7qg== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id j26-20020aa7c35a000000b0055faf2587f1si582930edr.359.2024.02.02.00.09.30 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 00:09:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-49414-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49414-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49414-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id E93F21F2AD9A for ; Fri, 2 Feb 2024 08:09:29 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 504AD18E14; Fri, 2 Feb 2024 08:08:28 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B151C182D8 for ; Fri, 2 Feb 2024 08:08:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861306; cv=none; b=nGQ/0+T+mKudxhpM+kmv0ItYn3mDlVKYTCKGTUlScpuxXR409HyXTC73ahushmcivvrwmyMFWNwcl6P5NH9x04XH0a/4opL+awB4ZzfSsId/2fOQo2dfenVpFoVrwJmpDXMSN6Yiu/DPFm2DplX0aTImLz/7UnSSXqUlybUocMg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861306; c=relaxed/simple; bh=2hkmQn4Ed8vkkGt4I+ydMlyJoDxhv4t3yUe1PaYNzQE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=S+pnDck4Dlei8lLoHY/66Ai7TmIw8BOmwDHuPEQA1/+WYU3/gerv6dLGdiez91jVBxHYoq4W9RYrOCTsPOmdhwTBQf+HORFqz1hp8sQM6lBJe/T0HJ0plGWeGH22EhZzWWSiAc7QbtvWe0BrHBDX9wYiL0GFa29F0NpiGr1tN8s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 568E21042; Fri, 2 Feb 2024 00:09:06 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6AED23F5A1; Fri, 2 Feb 2024 00:08:20 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 02/25] mm: thp: Batch-collapse PMD with set_ptes() Date: Fri, 2 Feb 2024 08:07:33 +0000 Message-Id: <20240202080756.1453939-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789773868222273050 X-GMAIL-MSGID: 1789773868222273050 Refactor __split_huge_pmd_locked() so that a present PMD can be collapsed to PTEs in a single batch using set_ptes(). This should improve performance a little bit, but the real motivation is to remove the need for the arm64 backend to have to fold the contpte entries. Instead, since the ptes are set as a batch, the contpte blocks can be initially set up pre-folded (once the arm64 contpte support is added in the next few patches). This leads to noticeable performance improvement during split. Acked-by: David Hildenbrand Signed-off-by: Ryan Roberts --- mm/huge_memory.c | 58 +++++++++++++++++++++++++++--------------------- 1 file changed, 33 insertions(+), 25 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 016e20bd813e..14888b15121e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2579,15 +2579,16 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, pte = pte_offset_map(&_pmd, haddr); VM_BUG_ON(!pte); - for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { - pte_t entry; - /* - * Note that NUMA hinting access restrictions are not - * transferred to avoid any possibility of altering - * permissions across VMAs. - */ - if (freeze || pmd_migration) { + + /* + * Note that NUMA hinting access restrictions are not transferred to + * avoid any possibility of altering permissions across VMAs. + */ + if (freeze || pmd_migration) { + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t entry; swp_entry_t swp_entry; + if (write) swp_entry = make_writable_migration_entry( page_to_pfn(page + i)); @@ -2606,25 +2607,32 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = pte_swp_mksoft_dirty(entry); if (uffd_wp) entry = pte_swp_mkuffd_wp(entry); - } else { - entry = mk_pte(page + i, READ_ONCE(vma->vm_page_prot)); - if (write) - entry = pte_mkwrite(entry, vma); - if (!young) - entry = pte_mkold(entry); - /* NOTE: this may set soft-dirty too on some archs */ - if (dirty) - entry = pte_mkdirty(entry); - if (soft_dirty) - entry = pte_mksoft_dirty(entry); - if (uffd_wp) - entry = pte_mkuffd_wp(entry); + + VM_WARN_ON(!pte_none(ptep_get(pte + i))); + set_pte_at(mm, addr, pte + i, entry); } - VM_BUG_ON(!pte_none(ptep_get(pte))); - set_pte_at(mm, addr, pte, entry); - pte++; + } else { + pte_t entry; + + entry = mk_pte(page, READ_ONCE(vma->vm_page_prot)); + if (write) + entry = pte_mkwrite(entry, vma); + if (!young) + entry = pte_mkold(entry); + /* NOTE: this may set soft-dirty too on some archs */ + if (dirty) + entry = pte_mkdirty(entry); + if (soft_dirty) + entry = pte_mksoft_dirty(entry); + if (uffd_wp) + entry = pte_mkuffd_wp(entry); + + for (i = 0; i < HPAGE_PMD_NR; i++) + VM_WARN_ON(!pte_none(ptep_get(pte + i))); + + set_ptes(mm, haddr, pte, entry, HPAGE_PMD_NR); } - pte_unmap(pte - 1); + pte_unmap(pte); if (!pmd_migration) folio_remove_rmap_pmd(folio, page, vma); From patchwork Fri Feb 2 08:07:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 195678 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:9bc1:b0:106:209c:c626 with SMTP id op1csp278359dyc; Fri, 2 Feb 2024 00:09:44 -0800 (PST) X-Google-Smtp-Source: AGHT+IHq73RsJG3M41kmLTvOWX2fqOjhxeFF/msqrQ8YtZXqSY8kSnX1tCv2sD6BaNJ+KAsIHZtf X-Received: by 2002:a17:906:1c15:b0:a35:7132:26d5 with SMTP id k21-20020a1709061c1500b00a35713226d5mr4932932ejg.63.1706861384637; Fri, 02 Feb 2024 00:09:44 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706861384; cv=pass; d=google.com; s=arc-20160816; b=J/RupANBTUXfPQUgOTr5cRnmZLKWLbNxLb+qeYHRWrA3qv+qhWQ4MARZNXwDuSVh+S xPHz5RNoeJtaIacg7lWaBUgr6W6EndKpCMjwc1sqWRuR9XYrX/2y6Q454uhg5/UkNH6L QAoYVZFxQyESwg1Edn29qDp3s9YwgQswkokMcKRlvAh2Y1F6F/wGXKonEt8w6eVRFlEm 4qa/XZC3lSfvWPaTc3MA5UaRcXqRPBh6ArZu/AbgmlI98MyHgJoFbfHBXlbUsO5UDMQ/ YKejnkFvtGON5yFCVUdardtMIcdVoH4zM0654J+m/gz2T1gmsBmL5qJDRpAd6aQdiOSl ojeg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=MiB2yK22g/POYHUygPxWvSb+30lGFDNWNkV8pOKxEt0=; fh=ST8yVJUKnvLlYqXlTKbEAx0V3QHrCt22tS85TPv5VSI=; b=CPp7NOpsamEsw42E4Paswr+971sqCjWM9c5bLvjkmiHPTFZiW8HH45eXydFVdvIgmt u1kDIBCkDvS+IOb7jP7ix0VVkVyjDrlLjIH4gsofB1ErWy9LXVUH1sY0QoIPUo0i+1Y6 /G9qcdmT4QU49w31zwv7dLvYgAa8lLORXGZ9eta9Owd/yMxar/yZJFjeKKNOxADPEGIv TBwhCZz/3tGYXnJ73I0nLSjIpPTTdkfSY/6M5fnLmK4X99QIpR7DZ7oZyR/sU1N2PcUX YvzGajNq0GHuCyyCF6nvaRlePXJuNLmJBx0UyGcN7JobLI+3g0dRWPafJKs0bhONzkFy P7QA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49415-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49415-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com X-Forwarded-Encrypted: i=1; AJvYcCXXkk7FXV+u25iyeICOLypGLQGXjXqVwyNmsrQqlSeXN4A9R87Z/NpgfYU8H2KNyKPy6JrJboQHppZvulovBIIRbJtRoQ== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id k15-20020a170906a38f00b00a36fb0c6424si475772ejz.61.2024.02.02.00.09.44 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 00:09:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-49415-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49415-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49415-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 410151F2A724 for ; Fri, 2 Feb 2024 08:09:44 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C4D5A199BA; Fri, 2 Feb 2024 08:08:31 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6361218E1A for ; Fri, 2 Feb 2024 08:08:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861310; cv=none; b=c881Vo5p4UE9Iq9fK7pUNnw8KNTcH0bDe9Gj4ftMmrXnZl2tu7byklPwDROvPiSGGwKn2kW+bnT3WTZz3CuGZ8Oxx2gk/1q9MMCdKLpeKL3mdG5xJEQWZPTtSnFMxG5xclvrPC+wfXFy+ZowC7g+Ojf5Mb0Xp5UvjU19awlBzvY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861310; c=relaxed/simple; bh=ilkctyHCFTKph7SguchDk3Oqzwpxdv9pwaYrbYycKgQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=khWin5suZO0qkx6sPjk7iohPp7VxrDncpmKVhK1kOxJgFlB0jy2kLoS16BkZulkAbJlBcPT91N9MLKHC2jWeqKVPY6/gxQewynHclxgUJ0THYiKVE02Y/qtckUuOVORWbkJTvCdyKgsLyi6X/0eORCULwphVzmeF5hVeIsBw4VI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2865B169E; Fri, 2 Feb 2024 00:09:10 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3C7A53F5A1; Fri, 2 Feb 2024 00:08:24 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 03/25] mm: Make pte_next_pfn() a wrapper around pte_advance_pfn() Date: Fri, 2 Feb 2024 08:07:34 +0000 Message-Id: <20240202080756.1453939-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789773883196876574 X-GMAIL-MSGID: 1789773883196876574 The goal is to be able to advance a PTE by an arbitrary number of PFNs. So introduce a new API that takes a nr param. We are going to remove pte_next_pfn() and replace it with pte_advance_pfn(). As a first step, implement pte_next_pfn() as a wrapper around pte_advance_pfn() so that we can incrementally switch the architectures over. Once all arches are moved over, we will change all the core-mm callers to call pte_advance_pfn() directly and remove the wrapper. Signed-off-by: Ryan Roberts --- include/linux/pgtable.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 5e7eaf8f2b97..815d92dcb96b 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -214,9 +214,15 @@ static inline int pmd_dirty(pmd_t pmd) #ifndef pte_next_pfn +#ifndef pte_advance_pfn +static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) +{ + return __pte(pte_val(pte) + (nr << PFN_PTE_SHIFT)); +} +#endif static inline pte_t pte_next_pfn(pte_t pte) { - return __pte(pte_val(pte) + (1UL << PFN_PTE_SHIFT)); + return pte_advance_pfn(pte, 1); } #endif From patchwork Fri Feb 2 08:07:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 195679 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:9bc1:b0:106:209c:c626 with SMTP id op1csp278444dyc; Fri, 2 Feb 2024 00:09:59 -0800 (PST) X-Google-Smtp-Source: AGHT+IHYyZiZhuwAGvVkGiUlLumJsdk7aqA1A+N4S9TCG1o4xTSmVvdP6cy97nHCAL0KzDVGOZ9d X-Received: by 2002:a17:907:4d0:b0:a36:d2e8:df1a with SMTP id vz16-20020a17090704d000b00a36d2e8df1amr859674ejb.2.1706861399520; Fri, 02 Feb 2024 00:09:59 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706861399; cv=pass; d=google.com; s=arc-20160816; b=hxem2Sw1ca/VQkRHWUf8x70zyuUEa9zmQ+iPnA0sQCiMs+RrmDmONRU+7030KZ1n91 vLRIbdjaHij1P405bwB41xFlgMRJC0V5mkN7/23Y8bCHtK0O3BzP4/iGx5N9YykTU+bv DGHwjf6nE4Suj4fQb+36h/aiHU2sRcqzt1v8Dc3JFAy4TbUIWGU2Og9kc3cUoXwJR7Kg 9kO4zqZyj7zYU5VkFrzguT56uicwGKSKaaqlSj/mUPvSC4viwAu2b0LTdnsa7x2hm3vV JUwuAab5+oytQHFT+CG5B5uDtYZrCNn5OhhHxm3wxQPtgwpZAi32Kxe/TJKxLEnJ0OHO hFDg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=4B3QQ0hn6W/OOMuAWwXMBnWBKPLPbfhbIhE+WOA0G6Q=; fh=BM0sVI4iJRQncmzMjfuP72Cx13uibYbEX3g7VxHYklw=; b=dOihhEMToDPvFgk+ZbeXSxVCoVl7yHtf77dPAs7dZsn5oTtHZ5Mw2VZeEBlNqMqo5G UfSmKrIhGajueVGZGV3Vdjw+7LhOHxdtl0ZUcmGR+9ITooAcKcqZFR6axlfz+yKVmDqA 4iPu+NnBZQdTinLGOxlrlviRSXBIt4e/rHM8xJJdH04M2LCKVSjSAlIVf5qDJGQRmwlv DJMmYja044QZwz+j0JQtisznxk3XpQTECvSAtxk1mATeJhXrUin5QNF7QHIOMko23aaI i+HxidTSK6LBQqBEZNfdFYbhSzX1CyWjs6irLEoa6NXbTpXkqkOvECA9teJEC50NIUn/ Q18Q==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49416-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49416-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com X-Forwarded-Encrypted: i=1; AJvYcCXmxDRF75psiX5F300mg/0uW6tsDAM6GY5LNRRZ+VnHK58AnfiMaT1kb6jbpU7k3ox222GiyZme8MD4m33URGd/wXpOJQ== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id xo14-20020a170907bb8e00b00a36fa4909c0si521896ejc.198.2024.02.02.00.09.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 00:09:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-49416-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49416-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49416-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 119141F2A7C9 for ; Fri, 2 Feb 2024 08:09:59 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 82A961B298; Fri, 2 Feb 2024 08:08:35 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 318FC1A27D for ; Fri, 2 Feb 2024 08:08:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861313; cv=none; b=fGLTUyEPGLD6aEMGwpmmwolvc3Q60Vv/z8dgv9v0RMHrHrg14gnO7IueDgw8MYG8Yt+evBMWclVhUQKEx+dKvIlPb4YofIWVOTG8jbxdWDysuuucRUZzfnSpX9jzbeSctbBy6PoIF7JXlOSHkXah8+ABCMGUXTK/r/XVwly1mQU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861313; c=relaxed/simple; bh=FKzPIlVxJLftEA3Ucgm1Tjm8StZy0DFAb3I7Ol8B7VQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=t03/781d0A+0y2QaH7venAnNA6vVEEUsHZV5k6IgiW+VoqgEXhS1UuOdekUZjaQCH7a93s7bCsa/nxCgsoYw3Oc9Kpu6u/BFo7M5/4cJh3VKic/SbuTqvGdJIs25MMhFgkTtFbfp+iUBJb7gH1DXKU1fV60fUfrP5p6zbbAEk7o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EDFB71763; Fri, 2 Feb 2024 00:09:13 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0E4093F5A1; Fri, 2 Feb 2024 00:08:27 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 04/25] arm/mm: Convert pte_next_pfn() to pte_advance_pfn() Date: Fri, 2 Feb 2024 08:07:35 +0000 Message-Id: <20240202080756.1453939-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789773898586363231 X-GMAIL-MSGID: 1789773898586363231 Core-mm needs to be able to advance the pfn by an arbitrary amount, so improve the API to do so and change the name. Signed-off-by: Ryan Roberts --- arch/arm/mm/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c index c24e29c0b9a4..137711c68f2f 100644 --- a/arch/arm/mm/mmu.c +++ b/arch/arm/mm/mmu.c @@ -1814,6 +1814,6 @@ void set_ptes(struct mm_struct *mm, unsigned long addr, if (--nr == 0) break; ptep++; - pteval = pte_next_pfn(pteval); + pteval = pte_advance_pfn(pteval, 1); } } From patchwork Fri Feb 2 08:07:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 195681 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:9bc1:b0:106:209c:c626 with SMTP id op1csp279305dyc; Fri, 2 Feb 2024 00:12:11 -0800 (PST) X-Google-Smtp-Source: AGHT+IE9KToonR4GbAZtu0QCybBhoDBuh757TgNdhIZfLkTOYKsZgXlYKT5M2btrVp0BgvGCeF9d X-Received: by 2002:ac8:5c48:0:b0:42a:b06e:84db with SMTP id j8-20020ac85c48000000b0042ab06e84dbmr3054299qtj.12.1706861531644; Fri, 02 Feb 2024 00:12:11 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706861531; cv=pass; d=google.com; s=arc-20160816; b=K+v7WpoEcypSwd06TcZLLQ8yFuK9uCy5Drraso5rbARa1Ry9nIfJr0OuYr/HzUgwtj ab4/Vot9Ez1LLyeiVUt47O6JdityGsKoGWFS9B3vW5a5WFHDGmcde2zVSErcIVWxZFEZ Ct6xKccIL9O4M2Oe5Y9QvywSC73R1UqnhLpRKDKyXMfX9Nz6vzhK+w4Wm93wGYn5BR0U byXeRkUg3yG2yReFe4DeWgWFtQljbs0iTcCdf9576N9YuEUoQ+DqN284lQI7jAjce1gr jpMHVXW/4mlEKRGQpeovf1fqLr9xLQizDdl0SCoNMWEYCDITXKAauk+TabqrtBvk3D0Z gbvw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=AwvDF9Q3mfZgddP7u26CnNByjWvMvI5KLwLV5SoUP/M=; fh=7b07W0pEXalPFmTHUGp1zCCMMkKZ1IHaIeMGvU2kMoI=; b=WxU3IeAIh6Fh/ccDEyvyJl594bnB58ngr9eMsJtj/RrnIfPdYEd3aDllDGDXt/Wh8t qiRiY4XEIESq2aiV5tqtPKjOAi/V2QPWGQJdtCcp+djVHy9oH/MJcuChNpJDJHzF3Op0 KQ0Hm67oPQc5ObykPs0llLKaOM3WubmdwnUdkhE67vLzwmdmZ36Fplo6kaAPW+Ss/uzH N+fBMJJ/uy6JYSHNOEylsyPsCSOvgcbP6Mj2biAGBkPMnXa/cfMqgTt0RLKEqlC/gDw1 I8tcbQnuEhdB0v88e5IhgzmP/OGrA+V+kS4vYNqgmtT9n1cw2vH2jznoTbWzKDlZNhIX vxcA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49417-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49417-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com X-Forwarded-Encrypted: i=1; AJvYcCV+OHKvUhGl0y1FIUGVxubf0S6N/FKD273WOoxGhjlqXeZ0WvRUun95OF8078TEv9KIpRF0+Kt8MAWr5qKKm3sppYngQw== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id s20-20020ac85cd4000000b00429b91ea4e5si1433564qta.458.2024.02.02.00.12.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 00:12:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-49417-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49417-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49417-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 752FA1C29D91 for ; Fri, 2 Feb 2024 08:10:12 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id AF14C33082; Fri, 2 Feb 2024 08:08:39 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id F260117C67 for ; Fri, 2 Feb 2024 08:08:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861318; cv=none; b=jk4ZG+G4l5Db4daKidxwuwlraxrCermAPzNfCDmR7+zfmqRZOdpGJ4zm9VUp+1ZSVFEcN+q1SenxNfrPSi0Sob20rfiBS/vFCCkTK6yUbyLWO0uyw24Bk2jF55GsazgPMmWP7yO6Em3tzM3+BLC/289CYBTr3sH3cEmaVSNQfto= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861318; c=relaxed/simple; bh=ANhmHka+h9ajCDCxtSgwGVd7uqj0IoMiS8o8hWXf4cM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QCzeWXJvoaZn9HVSCeaZFSAez6HKZZ90ID7P+Pon8r7pHCxAR8Kcu3pC/R4kBjgvg20PvztbnHdA1IDSnWRDa03LJk52f8bqkUFV7UJV7S1PIFfeZ/fJn5OXd+kNqtQXc5ufT2XOhtFKRNpSdRe+Zdo4YZPIiQIaaNwCAJa9lX8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BF9921764; Fri, 2 Feb 2024 00:09:17 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D3F633F5A1; Fri, 2 Feb 2024 00:08:31 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 05/25] arm64/mm: Convert pte_next_pfn() to pte_advance_pfn() Date: Fri, 2 Feb 2024 08:07:36 +0000 Message-Id: <20240202080756.1453939-6-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789774037073898893 X-GMAIL-MSGID: 1789774037073898893 Core-mm needs to be able to advance the pfn by an arbitrary amount, so improve the API to do so and change the name. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 9428801c1040..6a6cc78cf879 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -351,10 +351,10 @@ static inline pgprot_t pte_pgprot(pte_t pte) return __pgprot(pte_val(pfn_pte(pfn, __pgprot(0))) ^ pte_val(pte)); } -#define pte_next_pfn pte_next_pfn -static inline pte_t pte_next_pfn(pte_t pte) +#define pte_advance_pfn pte_advance_pfn +static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) { - return pfn_pte(pte_pfn(pte) + 1, pte_pgprot(pte)); + return pfn_pte(pte_pfn(pte) + nr, pte_pgprot(pte)); } static inline void set_ptes(struct mm_struct *mm, @@ -370,7 +370,7 @@ static inline void set_ptes(struct mm_struct *mm, if (--nr == 0) break; ptep++; - pte = pte_next_pfn(pte); + pte = pte_advance_pfn(pte, 1); } } #define set_ptes set_ptes From patchwork Fri Feb 2 08:07:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 195680 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:9bc1:b0:106:209c:c626 with SMTP id op1csp278650dyc; Fri, 2 Feb 2024 00:10:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IFRVr+w2GRGHeSYS1v+uGMj0TeVTkv/AcSA/mlVoSC5BnmNbppxuAwGb842wuznoHF+KmjN X-Received: by 2002:a50:ed87:0:b0:560:2b:9782 with SMTP id h7-20020a50ed87000000b00560002b9782mr152486edr.41.1706861428689; Fri, 02 Feb 2024 00:10:28 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706861428; cv=pass; d=google.com; s=arc-20160816; b=IhwbYdpzYmpW2uCMJZ50LUXUxH9HDM9t1w/lSlSiW69+m11KauBuElzF/BmKAuoYIl qjrJphevMio0YIcE2eabEigUaewO3GXfCxoR/BrnQUVQi/PaUaIUt4IYbI/E3rEyO8FP YbOy66mfcsKNh2XoixGCwK9UBqiHBRNaVWxS1Lc8lu65Pp7bRbQlYjP9m0UxIkZ7TK5d 1HopWe4bj9yOLBt76Ohs+VwzALTCKDYgcUknxe4FdgQu8aDI4MR66Jddg2v9CWp+NNFe VuBosdGA6YYgRqyBpDWobOuC6xsgQ9+Gb4O63sqdriK8VS2WhAfzCmw2I4e9eVYxFMLa yWag== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=52iYv8dj+hH79XZPEMODfoNkUJF+rthPXe736QRckVE=; fh=/UC+BlXW5Nvte9j1zSLYUEiQdcioGi73/fpdhLdfZ68=; b=gkCh9pimtN1b6CrZ4IKDc6G5dYqmf0+EYB2iBYavj3imZ/cLqvmXCuWlMGXVJPQB7V O1rZhVb34fflAYCOeW2G57k5SGtwirDXukX3BTXD7V9NipNrhvmxM4xQimyKGimPq5wX hmKlZYxCJUruUKSEC/P78wpBUu0pgnit/e2Y2GLYySaGKqWKdgnOf7qToAhVUZcTxCHe 4IQTYPNw3QjQmg9tn2phC6LMN80L5B5yWj5GmRmWB7I6ZJKjY5k9Z2kvNEyiDf/pXiNz hp4pFQugKq/Xrj8j2Yv4lpXlQJAqqkF8rCnWbPjBFo0PlKSrWdPAgAmtjp6Dp4Y97q7Q su0A==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49418-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49418-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com X-Forwarded-Encrypted: i=1; AJvYcCXoIGp8nC2GXwTXXXCuQFzxD3jucVaN+/gSxqoBDuu3o5YnPHeAJG7iT7rpSjqKYHyapQDsjL3ItH2HFyjdVO76xonoIg== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id u21-20020aa7db95000000b0055f3140c552si566353edt.424.2024.02.02.00.10.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 00:10:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-49418-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49418-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49418-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 222231F2AD0E for ; Fri, 2 Feb 2024 08:10:28 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 349B93F8DA; Fri, 2 Feb 2024 08:08:43 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E07A017C6F for ; Fri, 2 Feb 2024 08:08:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861321; cv=none; b=gvubq1Iqu4A6cvR9lCO/ax6KwVNmBL/ui9NBeYlaGRibf5j/C2skHKmsjtddraj2OfMvPkGGSRcZjIICRLAgSUGRpD+Pf48f+gLqNBauYndd0q5OsB0EMPInKqGI8h1gu1MZcKaZU15nAl6+XujclV/0JvW8UldcwV1dbI6T9t4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861321; c=relaxed/simple; bh=G4QEC3AJOHW95nVtaRI4iF4ZvUTMkYQdtSKQEs0NfH0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=R3u4ST4ur5bPiA3kFq3EBFIzWO9ib0loz3yL6H87NcUSPkNZqFmdAVjvtrsFTd4CTP1XL0wyo2MoeQ4dW7jqOW/dOW/MVY5aZYxx85f0AGGLoVW8+8KGSiseT/a0uxSHLAOlvE8HDJ4NeWr9pKLi8qvs7e7/7n/RoxZzP/5hqNs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 911E2176A; Fri, 2 Feb 2024 00:09:21 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A536F3F5A1; Fri, 2 Feb 2024 00:08:35 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 06/25] powerpc/mm: Convert pte_next_pfn() to pte_advance_pfn() Date: Fri, 2 Feb 2024 08:07:37 +0000 Message-Id: <20240202080756.1453939-7-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789773929100658412 X-GMAIL-MSGID: 1789773929100658412 Core-mm needs to be able to advance the pfn by an arbitrary amount, so improve the API to do so and change the name. Signed-off-by: Ryan Roberts --- arch/powerpc/mm/pgtable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 549a440ed7f6..6853cdb1290d 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -220,7 +220,7 @@ void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, break; ptep++; addr += PAGE_SIZE; - pte = pte_next_pfn(pte); + pte = pte_advance_pfn(pte, 1); } } From patchwork Fri Feb 2 08:07:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 195682 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:9bc1:b0:106:209c:c626 with SMTP id op1csp279937dyc; Fri, 2 Feb 2024 00:14:02 -0800 (PST) X-Google-Smtp-Source: AGHT+IHeGbu34DnT2jiY44m+QGmmOm9VRMCr7q4F1cQMLvQgMKgcYvYl0m1fkpT19AUr6gusuooG X-Received: by 2002:a50:f603:0:b0:55f:d256:b745 with SMTP id c3-20020a50f603000000b0055fd256b745mr1773649edn.30.1706861641974; Fri, 02 Feb 2024 00:14:01 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706861641; cv=pass; d=google.com; s=arc-20160816; b=S60N1ut8wPB2s0JVw64rse10Lay0cZavE56lpB1/UnvsxCkavWsDClcgaiygqW0EXH HUd34DbnTnDPr8Mkh2zVEPUBrcBMB4ZOkqWRzUsPOiUqDEJW9MzaGWtGaPffNgv/FlM3 uXzh7+8Uk2PBK65vrftLoWhLg+9AYH2Xq1BakI/ukoZ3QW0LHSeLzehAyK7UgH+p3HcP P/a4WMvHuwa2ODKRrkio2+H4KgU3h1D1dcPcCWv1KcqKljXIAhUX2vicXpDkk1XFJMDd 30pWpJaeROivu9oNv0npH72KVWnuAo40P/iQYw9mEO6KbkCdenMCkYd5YqMwIse82Q4E PDIA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=nA5bUrRt6blUgSGh1rucsmmbrNp8un7UcFTvjqqTNjA=; fh=FRrTlzE3p8cC7d5km9h1DtIXkpKyllJxB78QWhEHEs0=; b=vZeEIymbFgeGqa7T3HE0ij0tn5+6sGOVd6kw4jF0tT/zEgg/zrWPiFOJ8iLs3ULoKF asWO4jKd4itSlsGylIHJkhQ+N6eLghBmhuvZhXetSum9mDTJV3s8OARXkOXXXbxtKQUj gr1RX86t6WW1YE5OvPn3sHWjfzlz023KrdqbLui8NBdbuYN71JdutZ5Hfns9wWk7BXRM bee7emNhyh7Wn4YXqQuPLsfHQLRYv6EwBypANTiTU9CTZaPn5BtI/0tpzGJ8c8t1OUYs ZXjU5fUSHRSuE9XGN6kny1Wla5IQOPEQjrt2GfBE7EFiWSIm8a1s1569JXfJ8b3dHqVB +4Cg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49432-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49432-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com X-Forwarded-Encrypted: i=1; AJvYcCU6n87T3h2xjZjZRcmqul9h4nrlQX7LbryQk4kcuvNFk/3R9H0Wn6BsFf4nYdN0ShbRecala0BNLHKHOoRzKbIjwf+j8A== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id ee1-20020a056402290100b0054cd3ad8197si556073edb.306.2024.02.02.00.14.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 00:14:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-49432-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49432-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49432-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 93EAF1F23EFA for ; Fri, 2 Feb 2024 08:14:01 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5095B4D9EB; Fri, 2 Feb 2024 08:09:34 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id AD90C4CB45 for ; Fri, 2 Feb 2024 08:09:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861372; cv=none; b=Fl0zMyfdTb/V4Xr4ls9BL/1igJg2ylqmc2cUgHzxadStRZK1JrtNTmRlGSCPM1TW/fc4gkOVbD1Eg5yzGhrMeW7LGstM0yU7JODvSgxObwEP3rHlsyX1UZ45fCTTG/z8P6tddiNwWFdnbHJTrnCuVGC7xSuPXIlQbvkG7sCcaHE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861372; c=relaxed/simple; bh=6v3bAuhb3ybuKr98CSTFl92HE/KFoLGWUOZA4ZM01Yg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QhEUei6q2Xwor6fCTXD1nfbPYidPa7hCfuIo2zZEkx9v6gb8tqPHCDZTuDDEdpV79EjyS6Cm2cAYnO0bteYT4GN40lj0XHIWYoOiYO5II672dU3vMGwtjr9sE8ghXLS+/FiiPlz6wFg/W8Q9XHYgxn6QWDqiciqux/chunGFXsQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 704231BC0; Fri, 2 Feb 2024 00:10:12 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 82DE43F5A1; Fri, 2 Feb 2024 00:09:21 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 18/25] arm64/mm: Split __flush_tlb_range() to elide trailing DSB Date: Fri, 2 Feb 2024 08:07:49 +0000 Message-Id: <20240202080756.1453939-19-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789774153113575546 X-GMAIL-MSGID: 1789774153113575546 Split __flush_tlb_range() into __flush_tlb_range_nosync() + __flush_tlb_range(), in the same way as the existing flush_tlb_page() arrangement. This allows calling __flush_tlb_range_nosync() to elide the trailing DSB. Forthcoming "contpte" code will take advantage of this when clearing the young bit from a contiguous range of ptes. Tested-by: John Hubbard Signed-off-by: Ryan Roberts Reviewed-by: David Hildenbrand --- arch/arm64/include/asm/tlbflush.h | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 79e932a1bdf8..50a765917327 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -422,7 +422,7 @@ do { \ #define __flush_s2_tlb_range_op(op, start, pages, stride, tlb_level) \ __flush_tlb_range_op(op, start, pages, stride, 0, tlb_level, false, kvm_lpa2_is_enabled()); -static inline void __flush_tlb_range(struct vm_area_struct *vma, +static inline void __flush_tlb_range_nosync(struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long stride, bool last_level, int tlb_level) @@ -456,10 +456,19 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, __flush_tlb_range_op(vae1is, start, pages, stride, asid, tlb_level, true, lpa2_is_enabled()); - dsb(ish); mmu_notifier_arch_invalidate_secondary_tlbs(vma->vm_mm, start, end); } +static inline void __flush_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end, + unsigned long stride, bool last_level, + int tlb_level) +{ + __flush_tlb_range_nosync(vma, start, end, stride, + last_level, tlb_level); + dsb(ish); +} + static inline void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { From patchwork Fri Feb 2 08:07:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 195683 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:9bc1:b0:106:209c:c626 with SMTP id op1csp280133dyc; Fri, 2 Feb 2024 00:14:29 -0800 (PST) X-Google-Smtp-Source: AGHT+IEYv2RF0nDPFOK7IScgHpHtksNR+mREPilfSwDzAnEVbSjFoDL0LAu1tjRCDZeKhkSsf3h0 X-Received: by 2002:a17:906:5399:b0:a36:fc15:c6b2 with SMTP id g25-20020a170906539900b00a36fc15c6b2mr701245ejo.35.1706861668919; Fri, 02 Feb 2024 00:14:28 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706861668; cv=pass; d=google.com; s=arc-20160816; b=skLVJdbq3SBKUgX8JmF2cN81lrFK9SGHphGTDep4MQX761ybQvYLQtHMC12OUhUO0t J6+rX00XCYGMVrVibGCLnvoEyZcEwP3R2JkLEUZWlnwStGW2ytGJsedKJ/9AqMENZxb5 F+es5HrES9hN6Y6ChDm1U8GBq9i7M2W6DhD3pu1Wjyi0Q3ObeXGqvbPq/D83RSj7hx97 fB9oYG7fLLOAFunxbyL3YvSEx+lWop8MlUHuIzBnfq18nxS5nWtMlsN9erysrFwM/JVS ycUnomhhi8N61OHDCwrFNmEU0LTiDrVSU5RRQ6zTl94eQS1nQMtxWY/DK9LEqkMqu3Ga fmgw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=c/4EEJu2gQomaE2sg7y+ahoqZodMpQ+lo82yJLdFja4=; fh=016d0CnEATWPqigiexpNH7efL/V7tSO96UXotiDQNtQ=; b=OBxa3wK/hwCMxkL2sv9iejyyPU+OTD5F7jJ6cr9XSZZwRrKej3NzB7UwzT5Ca97tda /I7bJlZJw/awAJ0pDz9Cfx3FXZIIxz14b/fx7Hw0hCP/vxYEFtGTvN7VAkWKLFi/JN4V 7zvUWndIHcSjemKTQ4JY51nDYLvWyyVXNAeajckJEJbrvMLKGkqK0BFvnvAVlZ+AgCdS xGwMWBNCVC0TTplBpOJlZU9OwOIo0Cw17ihHmbfcCFqPWr9t7LMfM3C5fYfSpOGZkwDC OdBJrjgJsFISm5ADKy746xocuQYnVL7tN94cfVvgB1v9N9aMZwxgdybvZJWP15TBqWai N72w==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49433-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49433-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com X-Forwarded-Encrypted: i=1; AJvYcCVMDH6tIZIlJf497PYv4/ir6vrtLVFX9w+ZH2yfPOob4y561RNeggMOycfU38btfSvnM0DoGZ3qLzi6d5MqHYEu7NvmFA== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id c14-20020a17090603ce00b00a310709a5c4si555538eja.305.2024.02.02.00.14.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 00:14:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-49433-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49433-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49433-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 5BCC31F24F44 for ; Fri, 2 Feb 2024 08:14:28 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D041B4F1E9; Fri, 2 Feb 2024 08:09:37 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1F7AE4CE17 for ; Fri, 2 Feb 2024 08:09:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861375; cv=none; b=q0EiaCRtw1sKSms2KU92Cu4OtUySBlxHDtWxMCEgp96QObc6T3yB4DlEf9jWbOGR3aTUF/jXM0FqWxtkPHzPcj+Smj2DwWlOYQEUN8CEUYqFYsJJThjOHOny6n4X4yQXDphHy7JlIu92McBqaLG2Xi9AyCFUKhDm4VDdNnGO95A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861375; c=relaxed/simple; bh=SB2ag2cDhodBguRy9U5hL+cE2sLjwUFAOdPQskyP6S0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=eHpBSAIKrQiAAJ09VFg+ad4zycOsFthK+WGlKyPPB/L/ME4xK77wf+1aV0yu/T5CbdgdQmVd9wvwnDAl6X8namteixDQDgV6tUX+zMvWDgt5pLbCvDBBK2bLYmdzoiUuEshafU39I85bgOyNxvU7SiimME9hkymSPkHQa1lwsZw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1EEE91BCB; Fri, 2 Feb 2024 00:10:15 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 29FC73F8A4; Fri, 2 Feb 2024 00:09:29 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 20/25] arm64/mm: Implement new wrprotect_ptes() batch API Date: Fri, 2 Feb 2024 08:07:51 +0000 Message-Id: <20240202080756.1453939-21-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789774181640944521 X-GMAIL-MSGID: 1789774181640944521 Optimize the contpte implementation to fix some of the fork performance regression introduced by the initial contpte commit. Subsequent patches will solve it entirely. During fork(), any private memory in the parent must be write-protected. Previously this was done 1 PTE at a time. But the core-mm supports batched wrprotect via the new wrprotect_ptes() API. So let's implement that API and for fully covered contpte mappings, we no longer need to unfold the contpte. This has 2 benefits: - reduced unfolding, reduces the number of tlbis that must be issued. - The memory remains contpte-mapped ("folded") in the parent, so it continues to benefit from the more efficient use of the TLB after the fork. The optimization to wrprotect a whole contpte block without unfolding is possible thanks to the tightening of the Arm ARM in respect to the definition and behaviour when 'Misprogramming the Contiguous bit'. See section D21194 at https://developer.arm.com/documentation/102105/latest/ Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 61 ++++++++++++++++++++++++++------ arch/arm64/mm/contpte.c | 35 ++++++++++++++++++ 2 files changed, 86 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 34892a95403d..c07f0d563733 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -978,16 +978,12 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -/* - * __ptep_set_wrprotect - mark read-only while trasferring potential hardware - * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. - */ -static inline void __ptep_set_wrprotect(struct mm_struct *mm, - unsigned long address, pte_t *ptep) +static inline void ___ptep_set_wrprotect(struct mm_struct *mm, + unsigned long address, pte_t *ptep, + pte_t pte) { - pte_t old_pte, pte; + pte_t old_pte; - pte = __ptep_get(ptep); do { old_pte = pte; pte = pte_wrprotect(pte); @@ -996,6 +992,25 @@ static inline void __ptep_set_wrprotect(struct mm_struct *mm, } while (pte_val(pte) != pte_val(old_pte)); } +/* + * __ptep_set_wrprotect - mark read-only while trasferring potential hardware + * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. + */ +static inline void __ptep_set_wrprotect(struct mm_struct *mm, + unsigned long address, pte_t *ptep) +{ + ___ptep_set_wrprotect(mm, address, ptep, __ptep_get(ptep)); +} + +static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, + pte_t *ptep, unsigned int nr) +{ + unsigned int i; + + for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) + __ptep_set_wrprotect(mm, address, ptep); +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_PMDP_SET_WRPROTECT static inline void pmdp_set_wrprotect(struct mm_struct *mm, @@ -1156,6 +1171,8 @@ extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr); extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); @@ -1269,12 +1286,35 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, return contpte_ptep_clear_flush_young(vma, addr, ptep); } +#define wrprotect_ptes wrprotect_ptes +static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + if (likely(nr == 1)) { + /* + * Optimization: wrprotect_ptes() can only be called for present + * ptes so we only need to check contig bit as condition for + * unfold, and we can remove the contig bit from the pte we read + * to avoid re-reading. This speeds up fork() which is sensitive + * for order-0 folios. Equivalent to contpte_try_unfold(). + */ + pte_t orig_pte = __ptep_get(ptep); + + if (unlikely(pte_cont(orig_pte))) { + __contpte_try_unfold(mm, addr, ptep, orig_pte); + orig_pte = pte_mknoncont(orig_pte); + } + ___ptep_set_wrprotect(mm, addr, ptep, orig_pte); + } else { + contpte_wrprotect_ptes(mm, addr, ptep, nr); + } +} + #define __HAVE_ARCH_PTEP_SET_WRPROTECT static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - __ptep_set_wrprotect(mm, addr, ptep); + wrprotect_ptes(mm, addr, ptep, 1); } #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS @@ -1306,6 +1346,7 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, #define ptep_clear_flush_young __ptep_clear_flush_young #define __HAVE_ARCH_PTEP_SET_WRPROTECT #define ptep_set_wrprotect __ptep_set_wrprotect +#define wrprotect_ptes __wrprotect_ptes #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index bfb50e6b44c7..c85e64baf03b 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -23,6 +23,23 @@ static inline pte_t *contpte_align_down(pte_t *ptep) return (pte_t *)(ALIGN_DOWN((unsigned long)ptep >> 3, CONT_PTES) << 3); } +static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + /* + * Unfold any partially covered contpte block at the beginning and end + * of the range. + */ + + if (ptep != contpte_align_down(ptep) || nr < CONT_PTES) + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + + if (ptep + nr != contpte_align_down(ptep + nr)) + contpte_try_unfold(mm, addr + PAGE_SIZE * (nr - 1), + ptep + nr - 1, + __ptep_get(ptep + nr - 1)); +} + static void contpte_convert(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) { @@ -236,6 +253,24 @@ int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, } EXPORT_SYMBOL(contpte_ptep_clear_flush_young); +void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + /* + * If wrprotecting an entire contig range, we can avoid unfolding. Just + * set wrprotect and wait for the later mmu_gather flush to invalidate + * the tlb. Until the flush, the page may or may not be wrprotected. + * After the flush, it is guarranteed wrprotected. If its a partial + * range though, we must unfold, because we can't have a case where + * CONT_PTE is set but wrprotect applies to a subset of the PTEs; this + * would cause it to continue to be unpredictable after the flush. + */ + + contpte_try_unfold_partial(mm, addr, ptep, nr); + __wrprotect_ptes(mm, addr, ptep, nr); +} +EXPORT_SYMBOL(contpte_wrprotect_ptes); + int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty) From patchwork Fri Feb 2 08:07:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 195684 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:9bc1:b0:106:209c:c626 with SMTP id op1csp280248dyc; Fri, 2 Feb 2024 00:14:47 -0800 (PST) X-Google-Smtp-Source: AGHT+IGU/5pwT+PMq0tqZIw/b8vyGl/BGuQqeq/Ty8qhOwdVW1I2W0tc3T0HtX12UjH+yLdxj9x5 X-Received: by 2002:a17:907:77d3:b0:a37:1906:1173 with SMTP id kz19-20020a17090777d300b00a3719061173mr267214ejc.34.1706861687352; Fri, 02 Feb 2024 00:14:47 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706861687; cv=pass; d=google.com; s=arc-20160816; b=ulxhjOzqYNA6PCIlbXCASj4z8xQzRKGIjB0PrEUCfTJ7a3wOfSihLDA1wLE9KuAtui 3gyazB+ZDqKDOJnbbH1D1mbQQ+FuTcVjQ5aq1Rmjn/id6681tCiCi7fNo5DQnYDnBNqO b0dYndgl/eNzSIS9NzYjzfzLxDomZ06nxxuvLYO7IWeyadwYFE/8PgPfzJVoQGYFZrUb 3zVU2/sfYjidQedpvnxeKPz7JaIxHC1vNsvgz1DOaM8J6uuRXS3V/2vM0zGYMHSbzJcR 2PNtJI2aoZo/Rzwrx+GClPQ0NFToiC2G/J/KuO+jc7Amr5LLPqjW+u0LIKuILs82eOz2 QorQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=exgJk3Iib5rVvkUc8EQu+oEfH2dafVkQZy8I6MW4orI=; fh=5ozxMA9HP8hFeTA4G1/ci5sobVWNXetZjfdTZ9Iv8Jw=; b=DobwREk7PTMRkELrcLan8l9s9b6S26QRUMmJk3JXZpCvbxL9/VFWH9eywuBC+90mwd AY6D3BP7auaN0glTe4iJcb3AHQGf4MheorpWQgvQLoTApPA3Iy1I7He4IkDATBKmbYGq ZXQBK+ZiiUrhSzGSlTEOn6MJSnZtsXkoREEhkGfphiSfW4obHoEy40JrjSA7MrA3HLG2 TsruN6aZz9Dd8dinLgsWgukcUz1enotuMmhnc8FEOZgnFCgM02Fz7IXGfNPhJI7XR7Hf 9fQGKxKKf9p94D6vngbLpPp5Yslzj1fgLjJatKal0zGzZcj86/yzudygK7iaePiaWTaM Jb5g==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49434-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49434-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com X-Forwarded-Encrypted: i=1; AJvYcCXOmKERifCxusHUvEJbi0+0QSPAWZQeGLitgHwK7JOMIV0imVK0IiPnLds25TYTP6R39X8sXCMCkNBTy1imgnuHkICIxQ== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id v27-20020a1709060b5b00b00a2c342cba71si608478ejg.201.2024.02.02.00.14.47 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 00:14:47 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-49434-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49434-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49434-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id F01451F264CB for ; Fri, 2 Feb 2024 08:14:46 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5B9744F897; Fri, 2 Feb 2024 08:09:41 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 36C744EB46 for ; Fri, 2 Feb 2024 08:09:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861379; cv=none; b=SprjE4oRAvppvc2dvsQfQJghpsB92Zs0MAdHF8alpq+qhXpacoBTCVF+/5o6mV0iU21huMUBZTxQOWo4SdjjLaSe3Zs3zTEjVZSVrOE9pW7cBKQ6/5lyy+VdxdcLA68sgTmpeTFmo3GetFqQo5YK5RQli7chVwV5cbHV3wBm4jA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861379; c=relaxed/simple; bh=yjkoAf1qDptjCgLnqYeHIk4dRuTHdxms+EZ617tp34M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=YxQGX6R/xQqt3zsEqszAVXMp+Qq6zD1P4W5eqE+4Y+R8aFvw/QBZdMZY+nm79vyuTgs5ptjWiUmWdNu+pnBaUMf36V2NC/c57pi4vPndFz284ORwdffqQgHODJixcGSGY+22Ci9nhmnVW9v5qjgEc4gRF3g0fsFqMRi8+SR56QU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E4B631BCA; Fri, 2 Feb 2024 00:10:18 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 047993F5A1; Fri, 2 Feb 2024 00:09:32 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 21/25] arm64/mm: Implement new [get_and_]clear_full_ptes() batch APIs Date: Fri, 2 Feb 2024 08:07:52 +0000 Message-Id: <20240202080756.1453939-22-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789774201100281267 X-GMAIL-MSGID: 1789774201100281267 Optimize the contpte implementation to fix some of the exit/munmap/dontneed performance regression introduced by the initial contpte commit. Subsequent patches will solve it entirely. During exit(), munmap() or madvise(MADV_DONTNEED), mappings must be cleared. Previously this was done 1 PTE at a time. But the core-mm supports batched clear via the new [get_and_]clear_full_ptes() APIs. So let's implement those APIs and for fully covered contpte mappings, we no longer need to unfold the contpte. This significantly reduces unfolding operations, reducing the number of tlbis that must be issued. Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 67 ++++++++++++++++++++++++++++++++ arch/arm64/mm/contpte.c | 17 ++++++++ 2 files changed, 84 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index c07f0d563733..ad04adb7b87f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -965,6 +965,37 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, return pte; } +static inline void __clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + for (;;) { + __ptep_get_and_clear(mm, addr, ptep); + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + } +} + +static inline pte_t __get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + pte_t pte, tmp_pte; + + pte = __ptep_get_and_clear(mm, addr, ptep); + while (--nr) { + ptep++; + addr += PAGE_SIZE; + tmp_pte = __ptep_get_and_clear(mm, addr, ptep); + if (pte_dirty(tmp_pte)) + pte = pte_mkdirty(pte); + if (pte_young(tmp_pte)) + pte = pte_mkyoung(pte); + } + return pte; +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, @@ -1167,6 +1198,11 @@ extern pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); extern void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); +extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full); +extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full); extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, @@ -1254,6 +1290,35 @@ static inline void pte_clear(struct mm_struct *mm, __pte_clear(mm, addr, ptep); } +#define clear_full_ptes clear_full_ptes +static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + if (likely(nr == 1)) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __clear_full_ptes(mm, addr, ptep, nr, full); + } else { + contpte_clear_full_ptes(mm, addr, ptep, nr, full); + } +} + +#define get_and_clear_full_ptes get_and_clear_full_ptes +static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + pte_t pte; + + if (likely(nr == 1)) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + pte = __get_and_clear_full_ptes(mm, addr, ptep, nr, full); + } else { + pte = contpte_get_and_clear_full_ptes(mm, addr, ptep, nr, full); + } + + return pte; +} + #define __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -1338,6 +1403,8 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, #define set_pte __set_pte #define set_ptes __set_ptes #define pte_clear __pte_clear +#define clear_full_ptes __clear_full_ptes +#define get_and_clear_full_ptes __get_and_clear_full_ptes #define __HAVE_ARCH_PTEP_GET_AND_CLEAR #define ptep_get_and_clear __ptep_get_and_clear #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index c85e64baf03b..80346108450b 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -207,6 +207,23 @@ void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, } EXPORT_SYMBOL(contpte_set_ptes); +void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + contpte_try_unfold_partial(mm, addr, ptep, nr); + __clear_full_ptes(mm, addr, ptep, nr, full); +} +EXPORT_SYMBOL(contpte_clear_full_ptes); + +pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + contpte_try_unfold_partial(mm, addr, ptep, nr); + return __get_and_clear_full_ptes(mm, addr, ptep, nr, full); +} +EXPORT_SYMBOL(contpte_get_and_clear_full_ptes); + int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) { From patchwork Fri Feb 2 08:07:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 195691 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:9bc1:b0:106:209c:c626 with SMTP id op1csp284928dyc; Fri, 2 Feb 2024 00:26:59 -0800 (PST) X-Google-Smtp-Source: AGHT+IF6whH+UrDCocw05fkFnejyYCL7uRQlshz6uDWLkyvJgk3Omjk7NanBHHCDUdDUbWWisJ5/ X-Received: by 2002:a05:6a00:2717:b0:6dd:8211:9ff0 with SMTP id x23-20020a056a00271700b006dd82119ff0mr1624422pfv.18.1706862419034; Fri, 02 Feb 2024 00:26:59 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706862419; cv=pass; d=google.com; s=arc-20160816; b=L2FWo1SgLWQtS30XO73UwaHbHKxG6G8DjKCRKmZp9/G9XWaM3Lndnf21teAW5xjbQR 0qV5vv+p+SDGI7+VWqhiP9Kt+t0nYlU/07jPnSV9Ouy7AbSKA30DopRErc/A6S/iaG46 59k5C7fwOMtQ60UR/w/tXaxM3LHanfCMlMjq8VMydHjD6F5fpoSBMrBazd7t+kPPaXxB U1o86OQLEj6db8uVLH0Se7PnGdfROihv5g2cIIChRHsxDvEe3oMi/qz5JyL3W8m0lK8u WbV1SXKDee3gRAFQnsTkpg9t15fcypHn6uJC3n9FBqRdq9OuNDmA31gxUHr/rvC2yrZZ N0Bg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=qS1uNmviXYMG7rzOytsrcMROwkSHH753rn9wtVwEfUs=; fh=dhmdXyqriuiU2T6RfiGZjpwknI7c0+raQMN1AAztpsE=; b=q0CaHNjAJc3joqn16HaI9LzWHK5NZ4y9IbZ+/oQW2Ge43XPQHoF0832EKsch7xFjjx EmXce79JiLKgJ0VL932OdSvdwETajeORaGMNieW1yp/mZU+yEUzb0ywujtr6itsKKOU/ GX0w30Fz5V6z87sTQC/aA8+lw1cJNVxDw7DNM3FUxcjjiz+lzuu4nOMvBr8XMSLMrdTq UnA0j93PseuIgC0MV11fEKMe4srDxk20fVmJC2Os9l6DmysI49/MuTf7Nes5COGL9HCw 87+xkBoS2HTXvM2GvOdQiiBfa1rUXx6CJeTqDmb4W1cq8xfcTigdfzns4NxHiQ7I8h3o 90Jw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49438-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49438-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com X-Forwarded-Encrypted: i=1; AJvYcCXe+3yd8PbgzR553w05RF9SnyXdgcoYs5lYV8gKj/HOi/uUYv/C4sD2uEQ1iJt75xntC6Yd9zLf5cZ1//UjaLFImdurbw== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id m21-20020a63ed55000000b005d8e0d44d77si1255954pgk.226.2024.02.02.00.26.58 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 00:26:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-49438-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-49438-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-49438-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 8005BB22145 for ; Fri, 2 Feb 2024 08:15:53 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6A1A052F7B; Fri, 2 Feb 2024 08:09:56 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2BF20524A5 for ; Fri, 2 Feb 2024 08:09:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861394; cv=none; b=s6lURxytMAkT4FCVSeYqq2DcMgEtQVfkHotJt+PUrEax6SDflava2bgU8PKPFsaAcdiixy3wc9f+B001YGyFCOgkqkypEKavFEBJH3dM9qE1ptqIAKo1K0PCzEjxAAAhRr7H3ZuoSfGADxwwh248YWyyQMecntuShIXkDVbUBr0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706861394; c=relaxed/simple; bh=ijMp3DkZ/wdpVXZzkuK32MFSQt5qBP1fjMRpyIyA//8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QR73NoktLrfAEe97Isf4lmbgWbzmAgeNIs5+vDqTPBggoOmLe3A55Rf3UZhMypyubJpAOUTLfqYifGNqqMzB1GI3Wwh4aKUCjIUTN+cEsrIS9yrtcUBrpI4Zv8MeCcM/NhyOmSwU1Wbz1nIp9p5Ho739fagopZ678rgOllRhVKw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2B01B1C00; Fri, 2 Feb 2024 00:10:34 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3F6B93F5A1; Fri, 2 Feb 2024 00:09:48 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 25/25] arm64/mm: Automatically fold contpte mappings Date: Fri, 2 Feb 2024 08:07:56 +0000 Message-Id: <20240202080756.1453939-26-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789774968069472367 X-GMAIL-MSGID: 1789774968069472367 There are situations where a change to a single PTE could cause the contpte block in which it resides to become foldable (i.e. could be repainted with the contiguous bit). Such situations arise, for example, when user space temporarily changes protections, via mprotect, for individual pages, such can be the case for certain garbage collectors. We would like to detect when such a PTE change occurs. However this can be expensive due to the amount of checking required. Therefore only perform the checks when an indiviual PTE is modified via mprotect (ptep_modify_prot_commit() -> set_pte_at() -> set_ptes(nr=1)) and only when we are setting the final PTE in a contpte-aligned block. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 26 +++++++++++++ arch/arm64/mm/contpte.c | 64 ++++++++++++++++++++++++++++++++ 2 files changed, 90 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index cdc310880a3b..d3357fe4eb89 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1192,6 +1192,8 @@ void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte); * where it is possible and makes sense to do so. The PTE_CONT bit is considered * a private implementation detail of the public ptep API (see below). */ +extern void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); extern void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte); extern pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); @@ -1213,6 +1215,29 @@ extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); +static __always_inline void contpte_try_fold(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, pte_t pte) +{ + /* + * Only bother trying if both the virtual and physical addresses are + * aligned and correspond to the last entry in a contig range. The core + * code mostly modifies ranges from low to high, so this is the likely + * the last modification in the contig range, so a good time to fold. + * We can't fold special mappings, because there is no associated folio. + */ + + const unsigned long contmask = CONT_PTES - 1; + bool valign = ((addr >> PAGE_SHIFT) & contmask) == contmask; + + if (unlikely(valign)) { + bool palign = (pte_pfn(pte) & contmask) == contmask; + + if (unlikely(palign && + pte_valid(pte) && !pte_cont(pte) && !pte_special(pte))) + __contpte_try_fold(mm, addr, ptep, pte); + } +} + static __always_inline void contpte_try_unfold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) { @@ -1287,6 +1312,7 @@ static __always_inline void set_ptes(struct mm_struct *mm, unsigned long addr, if (likely(nr == 1)) { contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); __set_ptes(mm, addr, ptep, pte, 1); + contpte_try_fold(mm, addr, ptep, pte); } else { contpte_set_ptes(mm, addr, ptep, pte, nr); } diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 80346108450b..2c7dafd0552a 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -67,6 +67,70 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr, __set_ptes(mm, start_addr, start_ptep, pte, CONT_PTES); } +void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + /* + * We have already checked that the virtual and pysical addresses are + * correctly aligned for a contpte mapping in contpte_try_fold() so the + * remaining checks are to ensure that the contpte range is fully + * covered by a single folio, and ensure that all the ptes are valid + * with contiguous PFNs and matching prots. We ignore the state of the + * access and dirty bits for the purpose of deciding if its a contiguous + * range; the folding process will generate a single contpte entry which + * has a single access and dirty bit. Those 2 bits are the logical OR of + * their respective bits in the constituent pte entries. In order to + * ensure the contpte range is covered by a single folio, we must + * recover the folio from the pfn, but special mappings don't have a + * folio backing them. Fortunately contpte_try_fold() already checked + * that the pte is not special - we never try to fold special mappings. + * Note we can't use vm_normal_page() for this since we don't have the + * vma. + */ + + unsigned long folio_saddr, folio_eaddr; + unsigned long cont_saddr, cont_eaddr; + pte_t expected_pte, subpte; + struct folio *folio; + struct page *page; + unsigned long pfn; + pte_t *orig_ptep; + pgprot_t prot; + + int i; + + if (!mm_is_user(mm)) + return; + + page = pte_page(pte); + folio = page_folio(page); + folio_saddr = addr - (page - &folio->page) * PAGE_SIZE; + folio_eaddr = folio_saddr + folio_nr_pages(folio) * PAGE_SIZE; + cont_saddr = ALIGN_DOWN(addr, CONT_PTE_SIZE); + cont_eaddr = cont_saddr + CONT_PTE_SIZE; + + if (folio_saddr > cont_saddr || folio_eaddr < cont_eaddr) + return; + + pfn = pte_pfn(pte) - ((addr - cont_saddr) >> PAGE_SHIFT); + prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); + expected_pte = pfn_pte(pfn, prot); + orig_ptep = ptep; + ptep = contpte_align_down(ptep); + + for (i = 0; i < CONT_PTES; i++) { + subpte = pte_mkold(pte_mkclean(__ptep_get(ptep))); + if (!pte_same(subpte, expected_pte)) + return; + expected_pte = pte_advance_pfn(expected_pte, 1); + ptep++; + } + + pte = pte_mkcont(pte); + contpte_convert(mm, addr, orig_ptep, pte); +} +EXPORT_SYMBOL(__contpte_try_fold); + void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) {