From patchwork Thu Feb 15 10:31:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 201378 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:b825:b0:106:860b:bbdd with SMTP id da37csp319632dyb; Thu, 15 Feb 2024 03:08:17 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVcnHjJYCplfjqDUwSDSaJM0LmEdn8f+BkVKp83CGxfxpm0Oes7YaXu/x0YEczjis/HZx9R+dDx4D3PZK7SgylupSD3Pw== X-Google-Smtp-Source: AGHT+IFbEettBy5rv0hN/DRKaOwCIrwNkuBBIpFErdnrCQ9iXr8/UfMTN9tqK5j+5IhgzijNDktD X-Received: by 2002:a05:6a20:d043:b0:199:7628:286d with SMTP id hv3-20020a056a20d04300b001997628286dmr5726330pzb.30.1707995297069; Thu, 15 Feb 2024 03:08:17 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707995297; cv=pass; d=google.com; s=arc-20160816; b=trYYvyWvkXnouB6hl4ONfFSXknyoJxhLVUZf1I6X3ILcWe6rsnCP8Og7OTO0Fh7PXL laG+k6DTqHHe5whLcwE9ZxwHyQqz5HjXXnGYomvtdm5+5wJ2sgGBXXGqsjRl7LkACPQ2 LH/yao7VXyDhKpdNMYrJ6OUggyJVXNzafj/4vJnhwBmj4lJha0MKeBMDmbBI5s37WuoJ S+7r+1UoL8aGdzEC1Th9dGdSQZQtouFuvHBTO/gH68k/jqcMW2he/TYaIp3aduqAEpIJ lEAkIMmC9LnZ7a1lurIlj9UXqo2pIxE9O/lA3h0T06RCfl8lKqK6Tbb2BWi3rL+91LZv CCJw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=f6gb/nfiijie/R90uJlYvW3OmZJqXxR5XmD97JtGnxc=; fh=Heagowc1O8zVDXTyU76nJjOwDjz9r3Gi+0zmhF4ZELQ=; b=TVwwbVuiFUMM0m5r8zLbaJd0OoLg6BoINQBSjhR2SbsDYF5oxBO8s55Vkj2/A/fyTa nepfhyz3OT3DW842kPQ/mgaAE6zYb0ciE6jHT/ZqxRJUHFYnNsfILGXu9Y8g2NwpRCCZ zJbeUIU2WEY2+FcX1P3HyZuealmF5DRhYo4DThfLJiUscW7fBBsdV7LzdomL81l+SrfC NnkAamSi/l1OWHuzz7PeLVf9By0hsfWWzg2RJ4vAJsma05T5HcdICcbjvn0g6fQNrRtI gieJ1aJzzP7FcKa4E0iB5scMn7Z4Ma/rfHhp4oGC41sH4OvdtWQghmlqgARAo6s4jy2u rl8A==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66635-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66635-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id b11-20020a6567cb000000b005cdf9923679si906084pgs.727.2024.02.15.03.08.16 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 03:08:17 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-66635-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66635-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66635-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 4FD51B25536 for ; Thu, 15 Feb 2024 10:33:55 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id BF6E712AAFF; Thu, 15 Feb 2024 10:32:28 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 3808A129A75 for ; Thu, 15 Feb 2024 10:32:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993147; cv=none; b=l8EbcQD1oWPHXMUPIM4YLMV0QjuxrF/Ep8zmzpyu7wc1NJ+Y3RdcXu23+yHiEBZZwt6ea8oeADtkFQ8rChZtTyVhskn5Nqnc+3U/9Q+HSnVaPZLi2BY4/ZB8qFzbIoA4TBrxO4k5gs8Y1bsZvInXySKUVAZCvXLNI23i5iE2zfM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993147; c=relaxed/simple; bh=s0t/hAahHKineLmcpNEMFwM7j20fOnG/CtX3vPG0uZE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=MIsPXkSGwx5e3XbPRpAfCzskVkiuUbR75zlONVKBriroigK9RITjzk4KJiatJ03nLr7xUDXNHHxi7jJ5SaegVXYny1ACF1N4gSlLWeNSMRQ3kc6YJaZYTy1i2JqB9pr8M3yQ9C5P3MqLYsPXiCXm0/BVUtQeIRHeaEN3H7aZlMg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 40DFDDA7; Thu, 15 Feb 2024 02:33:04 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3722C3F7B4; Thu, 15 Feb 2024 02:32:20 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 01/18] mm: Clarify the spec for set_ptes() Date: Thu, 15 Feb 2024 10:31:48 +0000 Message-Id: <20240215103205.2607016-2-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790962876740109626 X-GMAIL-MSGID: 1790962876740109626 set_ptes() spec implies that it can only be used to set a present pte because it interprets the PFN field to increment it. However, set_pte_at() has been implemented on top of set_ptes() since set_ptes() was introduced, and set_pte_at() allows setting a pte to a not-present state. So clarify the spec to state that when nr==1, new state of pte may be present or not present. When nr>1, new state of all ptes must be present. While we are at it, tighten the spec to set requirements around the initial state of ptes; when nr==1 it may be either present or not-present. But when nr>1 all ptes must initially be not-present. All set_ptes() callsites already conform to this requirement. Stating it explicitly is useful because it allows for a simplification to the upcoming arm64 contpte implementation. Acked-by: David Hildenbrand Signed-off-by: Ryan Roberts --- include/linux/pgtable.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 49ab1f73b5c2..231370e1b80f 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -229,6 +229,10 @@ static inline pte_t pte_next_pfn(pte_t pte) * @pte: Page table entry for the first page. * @nr: Number of pages to map. * + * When nr==1, initial state of pte may be present or not present, and new state + * may be present or not present. When nr>1, initial state of all ptes must be + * not present, and new state must be present. + * * May be overridden by the architecture, or the architecture can define * set_pte() and PFN_PTE_SHIFT. * From patchwork Thu Feb 15 10:31:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 201379 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:b825:b0:106:860b:bbdd with SMTP id da37csp320063dyb; Thu, 15 Feb 2024 03:09:04 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCV2A1mRowtJA/WQpQHY1oV5WGbqfpC8XCezSSF1kDlIC6Etr1SrVs6EvTOyZuwgSog8ZfShF/KF+eGoU3e0wMt4RFFuvg== X-Google-Smtp-Source: AGHT+IHgSdd2puWaNAag8ivazFl8ndVOmoU3bv07HfzHFkkrdpZ0TxptBNQmg1pQSbkJeILhxz9v X-Received: by 2002:a05:6358:d090:b0:178:7556:11e6 with SMTP id jc16-20020a056358d09000b00178755611e6mr1288867rwb.27.1707995344089; Thu, 15 Feb 2024 03:09:04 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707995344; cv=pass; d=google.com; s=arc-20160816; b=Y+i3QnfeGM6VWmxZdbcPWCjKUsg/3xkRcxxs1+0zpL1v4yfQXBJzc9jjyHZl2imvLH oVZHE7U/HRUqjZdgnN1s088J8V5DY3ilMoJQVmJnM13ExtQDzfiQBUx3UURU/wt6MVuk vfsvowjZ3bD4EF9Oj6MCa7/tocWL2CcrsiuzqzPyzyAxijWTnzkV/TouKHGOGPrzD7Ot 5ciB0sUOnlVxtg77YK3ij+jJzcq+mC3qDVIV4VPCMbMOMmPilUKowrjwcVhfEjfaCx2U WCxd4RJuhHE0TsuqBvWLDWagmhLnlGG/6ZGE2AK32Ln8GKorD22jbMmngBK3RjwaxOTP fK2Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=iYGxY8gKE5MXdRT9yHHw6Vo1X8bbi/EsI1w4KtEcUbM=; fh=Heagowc1O8zVDXTyU76nJjOwDjz9r3Gi+0zmhF4ZELQ=; b=NBL96PaL/oTGOUZZZxHy7X8RvsQ0lqRFRQlgfLfHuBLOTMVsZ9ssGmoOekKgP+0/dA XpJm3ipr1zMjlQaMuI4/ntphNqTTcXm52rO98k5HjIyxJOFq/59Hi86EYApPqcyvZ9MF 8ndwOCEoK9dwbujf44rVG90MmCWrDShUtLJ5FacfgANidY+RWL+zC/XPaqVgOPvMEBJj nyZf9jrcE4wud+NDGOpIZN0Nf3vF+t7LS4B+tdHeun1n8NePT8d1bqi9akI83KYCBblG S5J9opcQarqeGnTWiJJMC2eq2iRI7XPBumBJ7tYL96tMSzF1PisQwui2Dahy6Gc9CCOQ 3PFA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66636-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66636-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id lb6-20020a056a004f0600b006e0a347f6ebsi982109pfb.90.2024.02.15.03.09.03 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 03:09:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-66636-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66636-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66636-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 94312B311A7 for ; Thu, 15 Feb 2024 10:34:10 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id CA1B112AAF6; Thu, 15 Feb 2024 10:32:30 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 720D712AAD7 for ; Thu, 15 Feb 2024 10:32:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993149; cv=none; b=S/0OhcLO6WiEC8pB7M7/k7tE6Qm3AoM+F6HGbsGNfDXXLZi30dGzXkhOKpmGMS6YBmHlrZ6H3UrqYe0xR3nVcFBOMpY/3xEI60klmP8vT34utrYnhGDWrudmAaBsI+7dZLjIwxQ6yfUnY+7CP1NhVWI/0MFXCANNpDqZhEEs1qY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993149; c=relaxed/simple; bh=2hkmQn4Ed8vkkGt4I+ydMlyJoDxhv4t3yUe1PaYNzQE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=laiq4LB+frTw8Lgb7diwRXB7f/oV3rwWKCxi9NElGoWluLRoRpRDr3Z4JAnF2ylsuQK64bEQCUrdZluOt8RI24l3noqCksBelGjMw5ZrUwYx1xhG9C7vKHMHpcLI//NfEuZxtIjuHpiuhNJD+mvhul40pLoemKZtQw5FWX14HCQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A52231480; Thu, 15 Feb 2024 02:33:07 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9BFF83F7B4; Thu, 15 Feb 2024 02:32:23 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 02/18] mm: thp: Batch-collapse PMD with set_ptes() Date: Thu, 15 Feb 2024 10:31:49 +0000 Message-Id: <20240215103205.2607016-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790962926105624074 X-GMAIL-MSGID: 1790962926105624074 Refactor __split_huge_pmd_locked() so that a present PMD can be collapsed to PTEs in a single batch using set_ptes(). This should improve performance a little bit, but the real motivation is to remove the need for the arm64 backend to have to fold the contpte entries. Instead, since the ptes are set as a batch, the contpte blocks can be initially set up pre-folded (once the arm64 contpte support is added in the next few patches). This leads to noticeable performance improvement during split. Acked-by: David Hildenbrand Signed-off-by: Ryan Roberts --- mm/huge_memory.c | 58 +++++++++++++++++++++++++++--------------------- 1 file changed, 33 insertions(+), 25 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 016e20bd813e..14888b15121e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2579,15 +2579,16 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, pte = pte_offset_map(&_pmd, haddr); VM_BUG_ON(!pte); - for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { - pte_t entry; - /* - * Note that NUMA hinting access restrictions are not - * transferred to avoid any possibility of altering - * permissions across VMAs. - */ - if (freeze || pmd_migration) { + + /* + * Note that NUMA hinting access restrictions are not transferred to + * avoid any possibility of altering permissions across VMAs. + */ + if (freeze || pmd_migration) { + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t entry; swp_entry_t swp_entry; + if (write) swp_entry = make_writable_migration_entry( page_to_pfn(page + i)); @@ -2606,25 +2607,32 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = pte_swp_mksoft_dirty(entry); if (uffd_wp) entry = pte_swp_mkuffd_wp(entry); - } else { - entry = mk_pte(page + i, READ_ONCE(vma->vm_page_prot)); - if (write) - entry = pte_mkwrite(entry, vma); - if (!young) - entry = pte_mkold(entry); - /* NOTE: this may set soft-dirty too on some archs */ - if (dirty) - entry = pte_mkdirty(entry); - if (soft_dirty) - entry = pte_mksoft_dirty(entry); - if (uffd_wp) - entry = pte_mkuffd_wp(entry); + + VM_WARN_ON(!pte_none(ptep_get(pte + i))); + set_pte_at(mm, addr, pte + i, entry); } - VM_BUG_ON(!pte_none(ptep_get(pte))); - set_pte_at(mm, addr, pte, entry); - pte++; + } else { + pte_t entry; + + entry = mk_pte(page, READ_ONCE(vma->vm_page_prot)); + if (write) + entry = pte_mkwrite(entry, vma); + if (!young) + entry = pte_mkold(entry); + /* NOTE: this may set soft-dirty too on some archs */ + if (dirty) + entry = pte_mkdirty(entry); + if (soft_dirty) + entry = pte_mksoft_dirty(entry); + if (uffd_wp) + entry = pte_mkuffd_wp(entry); + + for (i = 0; i < HPAGE_PMD_NR; i++) + VM_WARN_ON(!pte_none(ptep_get(pte + i))); + + set_ptes(mm, haddr, pte, entry, HPAGE_PMD_NR); } - pte_unmap(pte - 1); + pte_unmap(pte); if (!pmd_migration) folio_remove_rmap_pmd(folio, page, vma); From patchwork Thu Feb 15 10:31:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 201388 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:b825:b0:106:860b:bbdd with SMTP id da37csp324453dyb; Thu, 15 Feb 2024 03:17:41 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUQ6Vu89/khi1773S2/fYcNOtgYNdNS7mXLoaDykqBqFwtbyM5ZeHp/xMX8pE3I0YMRxFv3CM56vV8BQl5fygrkMYetTA== X-Google-Smtp-Source: AGHT+IEaCcaHpN/Y1vcsy7WfvY4YHw3GWON2iaw7Q3cgW6NGlGAsIJpAQpMZ3to40K5KHvURCSq8 X-Received: by 2002:a17:90a:468f:b0:296:454f:1e72 with SMTP id z15-20020a17090a468f00b00296454f1e72mr7232037pjf.4.1707995861544; Thu, 15 Feb 2024 03:17:41 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707995861; cv=pass; d=google.com; s=arc-20160816; b=gEEjBYgOaW86Pe+QI1rH/gKR7krfcGZUHnPy68sxpLAk34vyi+xbk2uaofPLWuSUos qBt5gdeAiPlZup3TXrBTKUd8OgdToHoqFJWlG300mO+Ja3l/XUHXBNKknn+cTW5L1Y6G prVdhIs70imbshirEFxbkhcLvpdsaOmK8HbtpSoERt/FVFWjuhGKQ4AMz7HRdCeoZZUT v8vvFmD3klUq2fmbJu41g3ZXlXmigAhq10SLGdWRGTzwNs/rjXalVroOrUF27yKeSCjP /pGedKr0usyKhx8INUnh/1p/Bj74hiD/2gPvpSHlYqRsMapre5k0ThVumvVZuZwOiQvg cgZw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=AKllLFpkdBdD4+23+umGEBd3K5r9BpGIsgRkHDi52Cw=; fh=Heagowc1O8zVDXTyU76nJjOwDjz9r3Gi+0zmhF4ZELQ=; b=m9lwhzVrGmKLOoKzmIgrp/jRXeERDHxg8evFAln4b8ErND3PAnVub3HYsCYLiU0jIM YcOTbtxqm1y2UwGegHMhFMy7NGn0mAHyWDrSkp+7FTM2yQjiRzMSg6QLrUdAqRLBdlPF ePbESe5T5OyliTbJXPWbqYAaw58hadkpQOdFMR7UzJthQlWpkgzvvRoWJ/GUtgn2TwKN TtxHK8EISBADCqC6D2uDzJXhuO5VqX+NAZIYacGy4ndNEIcf8sCyjnkizJQm/L9SlrML tF+Npc5IUHlHLJ4o9DNGM+edd8fK29zVQRGZxUo69vFwXu/8536qGudPgABw07fSTYc2 iIyg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66638-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66638-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id o8-20020a17090aac0800b00296f14f50b8si2891130pjq.50.2024.02.15.03.17.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 03:17:41 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-66638-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66638-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66638-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 0D6C6B31746 for ; Thu, 15 Feb 2024 10:34:53 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5A10B12BF29; Thu, 15 Feb 2024 10:32:34 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 71C0F12AAD7 for ; Thu, 15 Feb 2024 10:32:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993152; cv=none; b=P1kDGAlHTh8gAGGpwJt/seqShl2H7Q76bHqMe323OGEndj5dk1dZy8JJl2ptTBFoJVYo/iEXnlWQPC9DLyfE9fO8jU4J6OV7FOZn4dcdyGXj7zw1l4lHWZqL9JIvqouahe8C5E2gm5cORkHz+IJRxcjqLMzWO+A/4nr3EMIhvnA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993152; c=relaxed/simple; bh=f9J310tQ1bNpyfSQ928lGyLybpb8KOoR/8xzLYJEgEk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=TqCyAeHphWpLmoofEzi8E9V1mTmwwG545uZZpH1wEabXDB/FsaIpFsyAY1GUqpWKIU088UjSqKY1gz0GBucC1wkib/rXXhgQ2sn/sE4PVL9J7r6qVQ+9g8jWlP/Jg+lm8gEmANIihaq242LtGsYsygrS3iWLlyuAmaiApZ9z4mM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 149FD14BF; Thu, 15 Feb 2024 02:33:11 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0B7CA3F7B4; Thu, 15 Feb 2024 02:32:26 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 03/18] mm: Introduce pte_advance_pfn() and use for pte_next_pfn() Date: Thu, 15 Feb 2024 10:31:50 +0000 Message-Id: <20240215103205.2607016-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790962106240322004 X-GMAIL-MSGID: 1790963468492282768 The goal is to be able to advance a PTE by an arbitrary number of PFNs. So introduce a new API that takes a nr param. Define the default implementation here and allow for architectures to override. pte_next_pfn() becomes a wrapper around pte_advance_pfn(). Follow up commits will convert each overriding architecture's pte_next_pfn() to pte_advance_pfn(). Signed-off-by: Ryan Roberts --- include/linux/pgtable.h | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 231370e1b80f..b7ac8358f2aa 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -212,14 +212,17 @@ static inline int pmd_dirty(pmd_t pmd) #define arch_flush_lazy_mmu_mode() do {} while (0) #endif - #ifndef pte_next_pfn -static inline pte_t pte_next_pfn(pte_t pte) +#ifndef pte_advance_pfn +static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) { - return __pte(pte_val(pte) + (1UL << PFN_PTE_SHIFT)); + return __pte(pte_val(pte) + (nr << PFN_PTE_SHIFT)); } #endif +#define pte_next_pfn(pte) pte_advance_pfn(pte, 1) +#endif + #ifndef set_ptes /** * set_ptes - Map consecutive pages to a contiguous range of addresses. From patchwork Thu Feb 15 10:31:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 201342 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:b825:b0:106:860b:bbdd with SMTP id da37csp302863dyb; Thu, 15 Feb 2024 02:35:12 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUmhGcrO5U4KEvXzJDhD+jc6sAeZNplccnKPYh2TS1oeQ6n1wjzj6b/zaoKB8WhsXy58DdAboPDE4bFLg+mbXyAvP1Xbg== X-Google-Smtp-Source: AGHT+IFAZNKoqLmuDSBUGfMar5VS6hl6MU35cESnsh/ZAb8TvzGdVm7PaRkFhNFf+weK02djBe8W X-Received: by 2002:a0c:da84:0:b0:68c:82a3:a440 with SMTP id z4-20020a0cda84000000b0068c82a3a440mr1372321qvj.35.1707993312472; Thu, 15 Feb 2024 02:35:12 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707993312; cv=pass; d=google.com; s=arc-20160816; b=bYvsqw36NJL68oEwD/UTWfcslkYUuP1NDUFDzxPOdwTCR+2tUokUFVJTZL+qJuiVkd y7VjP3Kep183Bw8nC2xxNQIYbh/VGW7WYZC15oLUC/1DolSiFNsrib7FCEIvfZTU2SFW FvjQfEXww996+kC6MljkKVPW8DcFAysc0ZYNO5D8m4Nci3UnRDDTJNC378wbCW9EEIFT wTFiGnb6LPZat5YEWnmC8eSM8xFDTi6RkqBSuzHLyptLG0O5OvyFzdl5Cbw7GgizwLe8 b90VwCif7EriMnmUvm2EHZZd+9IWs84Tqv3CypnVwRnMWdE7raWjN90IKsFHeeEmKE/Z KS8w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=rWO7ZiASv+iNEWh1oRqCx438LeAl19u3CVwsECquxEQ=; fh=Heagowc1O8zVDXTyU76nJjOwDjz9r3Gi+0zmhF4ZELQ=; b=g8lLm7Jo2fLHh5dhGLtkfdY4d5S5nOhqopQXyq6lUoQS/nZBwERafGwf9+H3iY2k0I RAPL8mIUdbqrW8tYGgEuD+12n9jqoePHQWJNAy3De3Z3NuM0K+1Vag34FSsVyvd+2aiu 8JZdOBh8dQLMe/JriEO5aelgIzZHWK44Asv8RdUkTaRqtF9eCVGOiRbakD2/kEvTisFr lT11rhm9ts0tRTLGp8OzZ+OqV3NVhzr67aP0HI3jsNGFI36If+fG5sVYD3vmF2M/FFln BZbbPDCDX6T9yMMdI/qiOZLNI8CL2soBIVGeSJzrUdTD0pGgI+gDVhWpFoHKySzGMfZ2 4SeQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66639-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66639-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id c12-20020a056214224c00b0068caf864c2esi1141507qvc.312.2024.02.15.02.35.12 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 02:35:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-66639-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66639-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66639-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 7E84E1C21998 for ; Thu, 15 Feb 2024 10:35:07 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1C8F212C54B; Thu, 15 Feb 2024 10:32:37 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 3B11012BF37 for ; Thu, 15 Feb 2024 10:32:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993155; cv=none; b=YM1ZppHyi9dyiWholRuAW24bXfy0xLxZlHkq4BtbZT+f6xsoAgnqy8jORKsQHoDdJRQj0SHjW1ie2+aUIKOcwSscznPquQ5zUPP34WmTJwzNObIRo7Wa6/28k2EwZYbUoS0j/ZJla0XTxAFSV9o6/7YUsDrwcL0BV2dprajZDEI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993155; c=relaxed/simple; bh=jaRLqPxfoaS5xa20g0zrcIskSnUTkXjKILoHiMbUoFA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=l8KTc1yIIsnxmOG0yWX3ARHSAbhIH9ZJ1QnmFpPo6qOx9ijQSNp1XhJInXKWYMoUGSBZL94PN7AzEHz9pJOdjMiyPdt+3R2Y/Vja0j8tZ1746nljiVr7W7hIBFSc+gGzaywM8+/ghIcN9FezvqV38OpYgkANLtdtl/fz7wTfSJo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7862D150C; Thu, 15 Feb 2024 02:33:14 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6F72D3F7B4; Thu, 15 Feb 2024 02:32:30 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 04/18] arm64/mm: Convert pte_next_pfn() to pte_advance_pfn() Date: Thu, 15 Feb 2024 10:31:51 +0000 Message-Id: <20240215103205.2607016-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790960795469819647 X-GMAIL-MSGID: 1790960795469819647 Core-mm needs to be able to advance the pfn by an arbitrary amount, so override the new pte_advance_pfn() API to do so. Signed-off-by: Ryan Roberts Acked-by: David Hildenbrand Acked-by: Mark Rutland Acked-by: Catalin Marinas --- arch/arm64/include/asm/pgtable.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 52d0b0a763f1..b6d3e9e0a946 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -351,10 +351,10 @@ static inline pgprot_t pte_pgprot(pte_t pte) return __pgprot(pte_val(pfn_pte(pfn, __pgprot(0))) ^ pte_val(pte)); } -#define pte_next_pfn pte_next_pfn -static inline pte_t pte_next_pfn(pte_t pte) +#define pte_advance_pfn pte_advance_pfn +static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) { - return pfn_pte(pte_pfn(pte) + 1, pte_pgprot(pte)); + return pfn_pte(pte_pfn(pte) + nr, pte_pgprot(pte)); } static inline void set_ptes(struct mm_struct *mm, @@ -370,7 +370,7 @@ static inline void set_ptes(struct mm_struct *mm, if (--nr == 0) break; ptep++; - pte = pte_next_pfn(pte); + pte = pte_advance_pfn(pte, 1); } } #define set_ptes set_ptes From patchwork Thu Feb 15 10:31:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 201404 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:b825:b0:106:860b:bbdd with SMTP id da37csp331844dyb; Thu, 15 Feb 2024 03:33:06 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVPy0PVrtb5jGrDFnZe2kPhrTgR9tsgre0HWooCO9ji2h8ObwcxWDQoV72BgheVPoSLqRHtrEScPLT7BPuOKiubJmjVmw== X-Google-Smtp-Source: AGHT+IEiHjpPnkqKwlkxKCsJRmOOLG5xmb/NQB6vLwEINKf/YNsHu9EnKIWk9oOXM8QO+qO2TNKl X-Received: by 2002:a17:903:22cf:b0:1d9:411f:4303 with SMTP id y15-20020a17090322cf00b001d9411f4303mr1599665plg.9.1707996786080; Thu, 15 Feb 2024 03:33:06 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707996786; cv=pass; d=google.com; s=arc-20160816; b=sswlNsR2Q9sIhN2zRRM87Xdy0OzYMa+jt7wpu945oqPUZOEp0GAUqF8RwbZ5NcV+1X vwypYW01djvOy/EhdL+5aHPd2jh7mh14li+kxIGumimY/jI8KPf7I73/OAPA8ghYOyrk EgNJtefZjE4ltX+7QihxbWpYQEN233Rz7l2TSExL6QuEFLfRCXu5/kSzJtij4YAKTD+C vx/n1KIIu8KITU3Z92BzQ/pmoBuH7KQvp5rEZgD0b53m3tYFdzdWgBIlHXswmTlJ6j8W FGObZ5AP0Z58fCYM1yWqRRSo5OjWIo6A2wW2aggKyLAeqUyEapvLPgi+7C1GRUbO5Cp3 pJDw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=Vt7HWCdphupCFZjtUl00kIkT38SYH1HEWXlJCnuRk3c=; fh=Heagowc1O8zVDXTyU76nJjOwDjz9r3Gi+0zmhF4ZELQ=; b=W7CyPzDwLaRpEHkjF/c5W6OLK+jAxooyeekiBRVjGQbEJ4ddAvbGIAHNcbXwKWFguN TreOXVovS7cIfV3lxvU0YE5pdjQQDFH07ZHvXhfTuhbDG8BUbCHiJ7Mi0DkkdPCqdzlV LEJTGyjHHwCOpg2nP8zE9y/05GOA/bFPQgEQiwbHE1OeDi2hIg1jgxm1HlgulL/VWgIC oAC8dRvysqrRpOzWVpjporsVmORcCvJe1FvUhip4I8z002Cv8cTcEIczrrBrdWbqgwLP Zg+OBeKhDaZMilFNIlEm7xgB/0oGNqzxUgoRMQ8791pXj1utkJNyUeWcEmb0FssEoe4z Z1fw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66640-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66640-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id k3-20020a170902c40300b001db730d3f1bsi1002030plk.610.2024.02.15.03.33.05 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 03:33:06 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-66640-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66640-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66640-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id B07E3B31EFA for ; Thu, 15 Feb 2024 10:35:41 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6FF9812CD8D; Thu, 15 Feb 2024 10:32:40 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6193512BF16 for ; Thu, 15 Feb 2024 10:32:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993158; cv=none; b=WdZTKndtZgBn/CYJQ2hlzzKcWVqS6bbXZUqnO3Bq2/YLUIKa0s0KQKavPS+G7C7UXw0kqhkD98ixwqMlWRP2U6BJmNU9K826xv4kjNa4Flco12h7t+PlWCGufcGKXrJwSSr8P8xXPgBAkZTFipkEjqU9OZY1bTJpmim10CVC4Po= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993158; c=relaxed/simple; bh=QKKxOEWZPFqFjzK7t69u40gkIRbUK4JElui7wOCLF0I=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=M7j9B71u0jj5vFiV4OQU94zrXqtCYzoTsB4Q0cwu2jfuVO1bbdTEhNaVIQMtFUhresRbRK4yEoDGsJWqtnY+HkfPCm+eSovvwKgXSIYb/g9Z7vebOJD4C9soewive9RA6+evZLOfS9lG68PVXm4Cn99v8AMIyMxL/ugZeMh6Ejc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DDA7C1515; Thu, 15 Feb 2024 02:33:17 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D34703F7B4; Thu, 15 Feb 2024 02:32:33 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 05/18] x86/mm: Convert pte_next_pfn() to pte_advance_pfn() Date: Thu, 15 Feb 2024 10:31:52 +0000 Message-Id: <20240215103205.2607016-6-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790962212279053053 X-GMAIL-MSGID: 1790964437691539427 Core-mm needs to be able to advance the pfn by an arbitrary amount, so override the new pte_advance_pfn() API to do so. Signed-off-by: Ryan Roberts --- arch/x86/include/asm/pgtable.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index b50b2ef63672..69ed0ea0641b 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -955,13 +955,13 @@ static inline int pte_same(pte_t a, pte_t b) return a.pte == b.pte; } -static inline pte_t pte_next_pfn(pte_t pte) +static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) { if (__pte_needs_invert(pte_val(pte))) - return __pte(pte_val(pte) - (1UL << PFN_PTE_SHIFT)); - return __pte(pte_val(pte) + (1UL << PFN_PTE_SHIFT)); + return __pte(pte_val(pte) - (nr << PFN_PTE_SHIFT)); + return __pte(pte_val(pte) + (nr << PFN_PTE_SHIFT)); } -#define pte_next_pfn pte_next_pfn +#define pte_advance_pfn pte_advance_pfn static inline int pte_present(pte_t a) { From patchwork Thu Feb 15 10:31:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 201343 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:b825:b0:106:860b:bbdd with SMTP id da37csp303382dyb; Thu, 15 Feb 2024 02:36:24 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWSf5cuQgWcmiGnJvZDxhZdmmL6JSJ1jLn2WwZEKeQTZER0851HIYMx/1ORSg6VRskBk+/oxevBOhhMPMrY0W/04Rr/hw== X-Google-Smtp-Source: AGHT+IEUV4fFPo+LWRxzrg1PnJi/OlnlB+AwIZ4RdcKHcODmbFRRO4WD40HLqyyAPJ0Dii98fCH6 X-Received: by 2002:a05:690c:d81:b0:607:d244:5707 with SMTP id da1-20020a05690c0d8100b00607d2445707mr1238107ywb.13.1707993384619; Thu, 15 Feb 2024 02:36:24 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707993384; cv=pass; d=google.com; s=arc-20160816; b=USyeMBS6X2ld4D64fiPYFmY1WvY/uvZFVC3NeaVUAMVHlyTg38RG9XgN2R7B5pG6l8 7KKJvvmCr5e50H9eawRW8qk3BwIh4bRUfh6kEBfwi90YBNmFNG9SQ23av2er7ybO6T3M +kPsbtzRmIyQrqYFDpBwa9+dAbBGFiptfAnThJQBgrRPq4+/qDQzC8BtaDmOFYysSf18 cqIRizrUmsZO0s8ENB9Jd+EpelUKDf7Y+3f5ba3GQxtXnb3UHkvvUBdY3+5Uvu1LOylc 5gJGf0ztC+5D5mVkupWP+nvQsF9cRYdACUUsYdjmBpE8xkRFKJZv2iwxicBJqdxpOrks u43w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=L5R0Hb3vr2XhKlRaGDOJDdSoNMHQdu7Je9L2b+7crXk=; fh=Heagowc1O8zVDXTyU76nJjOwDjz9r3Gi+0zmhF4ZELQ=; b=gCTp+jy1R8UKP2Fy9Ga795Srwg/UBxs7zEHJROpj9QMVnnJwkt3aTqYhEFltldIfEF /gyZd84K0+0fYYh9idq3JuF6ieOp3eUuwcxyQoF4BxUeZo84bDtRX25JUKFBWi91ZkZM fb5fk6/OAOskVLs3ouQ7pJng0+sCjPvHjJOWj8lnFcB8V9NAWRfbdVlgP5C2tRCeUAkb cx5zfFl/DCZnm+whcVMsreKrpK7Ubiy+dzvep8zFc0W6M5PNfkJ5qWXINf6Mk/38quzr 9kEM2xMCc/C8mcfvqmKXSjKG3JZVKOu6Ov4eMFoqBc9coywUnLYepA98CzaVDYNPL6Rv r+WQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66642-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66642-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id d16-20020a05622a15d000b0042dc882fc00si1157865qty.455.2024.02.15.02.36.24 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 02:36:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-66642-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66642-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66642-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 394D21C231D1 for ; Thu, 15 Feb 2024 10:36:09 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id DFB3312D756; Thu, 15 Feb 2024 10:32:43 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 21CE912CDA3 for ; Thu, 15 Feb 2024 10:32:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993162; cv=none; b=LfiGzzo91RQE1toUsCYuKuqKSXSTLrjBE/zuq9/85RK1wQCE4nzkCjXUb13HFT9Js7hNDiuxdvF2ywnKmXJlKUFYuYaJGlzme8eJtbXWHO8sEL6QC22+v01KKLPsbmNWyi0zmYewawb0OxZkSrTPko4SVSAPylKcEgvmr245B+c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993162; c=relaxed/simple; bh=2pMOLH70RyGBPdRH3ynfTXiJWugM9Ml+vZUSCzri78g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=e1mkN/fmVArKwmDAmR3uK5rBZy2N3hR3Mm2ddLD5yZgIAMyCjDeZq0uF0+UHQ/rGI32hI03dObzYZG59iIWJc81UtGhA7X5eZU/BMh37bo4K7es5QsS9Xh1gDC0clu730PPbRG3HGPXjXWd3397Tj9H3rHURgnH4IxvWS4EA7zA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4DA741516; Thu, 15 Feb 2024 02:33:21 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 442213F7B4; Thu, 15 Feb 2024 02:32:37 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 06/18] mm: Tidy up pte_next_pfn() definition Date: Thu, 15 Feb 2024 10:31:53 +0000 Message-Id: <20240215103205.2607016-7-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790960871120951468 X-GMAIL-MSGID: 1790960871120951468 Now that the all architecture overrides of pte_next_pfn() have been replaced with pte_advance_pfn(), we can simplify the definition of the generic pte_next_pfn() macro so that it is unconditionally defined. Signed-off-by: Ryan Roberts Acked-by: David Hildenbrand --- include/linux/pgtable.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index b7ac8358f2aa..bc005d84f764 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -212,7 +212,6 @@ static inline int pmd_dirty(pmd_t pmd) #define arch_flush_lazy_mmu_mode() do {} while (0) #endif -#ifndef pte_next_pfn #ifndef pte_advance_pfn static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) { @@ -221,7 +220,6 @@ static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) #endif #define pte_next_pfn(pte) pte_advance_pfn(pte, 1) -#endif #ifndef set_ptes /** From patchwork Thu Feb 15 10:31:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 201344 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:b825:b0:106:860b:bbdd with SMTP id da37csp303694dyb; Thu, 15 Feb 2024 02:36:55 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCW3gICq/COJNrJECqS5hB6ggxgvQgR/g49AtSbI4uLQfJKTdt+fUc9xyJU0Ie4td7gHpepysNtW56x0oV7GIu2zb0WzmQ== X-Google-Smtp-Source: AGHT+IFT84f9L15GlCX1G3L8VT0yAdTSsTvcRoe0R74GH/B09L/ASal2cwZOKEumagtF1uJ+P/NS X-Received: by 2002:a17:906:6617:b0:a38:99ae:da8b with SMTP id b23-20020a170906661700b00a3899aeda8bmr942304ejp.56.1707993415671; Thu, 15 Feb 2024 02:36:55 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707993415; cv=pass; d=google.com; s=arc-20160816; b=jnRPPMpdIsSWlKR3Q/lNyVRVtZ+q3cluvBAF6+JN6cqVPVEvKmPJMFYHn6auWQWg6x egxBH9bhtTx8OSwx0R9TnuWtuNEaJutdID5aixlX+asuj+sLjCSoKhp1Xwob+B2zEvsz wx3/3YPlKhG/uiL3HxJIhtTPTih3xAg46a9CyplBTOSgbf4l5AdvUgS/rtOBn6m1poKl KK+MkogM4fv44FNDhtXTh8gDebGEqG698DDkVThiBWyOPr2e7i8uUALEqqCQ1H/OFCvd COhMXCgMddxjw+tmosxuP9yL+VBCwcQtrpD1Z2IfaJa8HH8HUbc8dttvsDNOneKardeY SusQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=4BAlAPgOnjDFN2Hap4t525VVNCrDp7iVSl0knImXGnI=; fh=Heagowc1O8zVDXTyU76nJjOwDjz9r3Gi+0zmhF4ZELQ=; b=AWLg24okUK0ywxNp9ZCqCsGxkqVPyvWtmrYGEnHk3cK5j8NwapLEdUhLescvOIsiJv xiH5JwN8/QT8lAkVHrYCGcOHDiSlSM92yYqF0xeJcabhCdBfZNoMem0KdtiT0JbP6rYQ 9aqwxDQxmyB0+3nXNZVGvuFtEoiDLwEh9+BHAMtUrCYXCHiRBga1YDAXJaA9bE+94V8+ VhAaGVxUqXiZQQTcHsjNHvL+9T9HIIQfLdItpRS1pLTDzbmXI04aCrNPESd00Ljyt48n MUttT+lqkFZUulpltn2d/GNi5FWzlcEQdnOhKm3vuX33bBl2tCFrozjIKjM3JkeUe7zb Ezbw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66643-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66643-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id a4-20020a1709062b0400b00a3d6cc63d45si532457ejg.135.2024.02.15.02.36.55 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 02:36:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-66643-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66643-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66643-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 1D65D1F23E3C for ; Thu, 15 Feb 2024 10:36:27 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 77C1612DDA2; Thu, 15 Feb 2024 10:32:48 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6C97312D76B for ; Thu, 15 Feb 2024 10:32:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993166; cv=none; b=pQmcD8WL4/o7NgC9XonQByaYisgSmsSDQRge4kGsMDq9t/qfvhEbJHBXR4FGcacpN0QZniMtrKYn+tfQ/QF422NsqTLy7g823FxLwYVOdK8WF3TaW4C+A1uLSLeiwwb7E2EPcPKpYUK5aw5vIW8UxPADxKp4iePJT79avjgOqfQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993166; c=relaxed/simple; bh=+OkHaAXcApv5i6T7H/FIyOrqmVoEbs4XCu7oHPJUMwI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=TqKcAPRl2a3EnWrjMRE2XHVX2G+KtMYAG4lyXT6acZmU4vyf/AZQqiY/xwH4kFf8YBwBBDfwbeqc9e4pGJ1uC8FzhIzN4/1sRw7eVjjM2fUCGpFbJHuYGliSWrHUVVNHzf9LWA0Rp6kFX/506kMPVHBEFlTD2/riG8M2On+saY4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B2453152B; Thu, 15 Feb 2024 02:33:24 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A90E53F7B4; Thu, 15 Feb 2024 02:32:40 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 07/18] arm64/mm: Convert READ_ONCE(*ptep) to ptep_get(ptep) Date: Thu, 15 Feb 2024 10:31:54 +0000 Message-Id: <20240215103205.2607016-8-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790960903775305285 X-GMAIL-MSGID: 1790960903775305285 There are a number of places in the arch code that read a pte by using the READ_ONCE() macro. Refactor these call sites to instead use the ptep_get() helper, which itself is a READ_ONCE(). Generated code should be the same. This will benefit us when we shortly introduce the transparent contpte support. In this case, ptep_get() will become more complex so we now have all the code abstracted through it. Tested-by: John Hubbard Signed-off-by: Ryan Roberts Acked-by: Mark Rutland Acked-by: Catalin Marinas --- arch/arm64/include/asm/pgtable.h | 12 +++++++++--- arch/arm64/kernel/efi.c | 2 +- arch/arm64/mm/fault.c | 4 ++-- arch/arm64/mm/hugetlbpage.c | 6 +++--- arch/arm64/mm/kasan_init.c | 2 +- arch/arm64/mm/mmu.c | 12 ++++++------ arch/arm64/mm/pageattr.c | 4 ++-- arch/arm64/mm/trans_pgd.c | 2 +- 8 files changed, 25 insertions(+), 19 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index b6d3e9e0a946..de034ca40bad 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -275,6 +275,12 @@ static inline void set_pte(pte_t *ptep, pte_t pte) } } +#define ptep_get ptep_get +static inline pte_t ptep_get(pte_t *ptep) +{ + return READ_ONCE(*ptep); +} + extern void __sync_icache_dcache(pte_t pteval); bool pgattr_change_is_safe(u64 old, u64 new); @@ -302,7 +308,7 @@ static inline void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, if (!IS_ENABLED(CONFIG_DEBUG_VM)) return; - old_pte = READ_ONCE(*ptep); + old_pte = ptep_get(ptep); if (!pte_valid(old_pte) || !pte_valid(pte)) return; @@ -904,7 +910,7 @@ static inline int __ptep_test_and_clear_young(pte_t *ptep) { pte_t old_pte, pte; - pte = READ_ONCE(*ptep); + pte = ptep_get(ptep); do { old_pte = pte; pte = pte_mkold(pte); @@ -986,7 +992,7 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres { pte_t old_pte, pte; - pte = READ_ONCE(*ptep); + pte = ptep_get(ptep); do { old_pte = pte; pte = pte_wrprotect(pte); diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index 0228001347be..d0e08e93b246 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -103,7 +103,7 @@ static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data) { struct set_perm_data *spd = data; const efi_memory_desc_t *md = spd->md; - pte_t pte = READ_ONCE(*ptep); + pte_t pte = ptep_get(ptep); if (md->attribute & EFI_MEMORY_RO) pte = set_pte_bit(pte, __pgprot(PTE_RDONLY)); diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 55f6455a8284..a254761fa1bd 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -191,7 +191,7 @@ static void show_pte(unsigned long addr) if (!ptep) break; - pte = READ_ONCE(*ptep); + pte = ptep_get(ptep); pr_cont(", pte=%016llx", pte_val(pte)); pte_unmap(ptep); } while(0); @@ -214,7 +214,7 @@ int ptep_set_access_flags(struct vm_area_struct *vma, pte_t entry, int dirty) { pteval_t old_pteval, pteval; - pte_t pte = READ_ONCE(*ptep); + pte_t pte = ptep_get(ptep); if (pte_same(pte, entry)) return 0; diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 6720ec8d50e7..2892f925ed66 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -485,7 +485,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, size_t pgsize; pte_t pte; - if (!pte_cont(READ_ONCE(*ptep))) { + if (!pte_cont(ptep_get(ptep))) { ptep_set_wrprotect(mm, addr, ptep); return; } @@ -510,7 +510,7 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, size_t pgsize; int ncontig; - if (!pte_cont(READ_ONCE(*ptep))) + if (!pte_cont(ptep_get(ptep))) return ptep_clear_flush(vma, addr, ptep); ncontig = find_num_contig(mm, addr, ptep, &pgsize); @@ -543,7 +543,7 @@ pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr * when the permission changes from executable to non-executable * in cases where cpu is affected with errata #2645198. */ - if (pte_user_exec(READ_ONCE(*ptep))) + if (pte_user_exec(ptep_get(ptep))) return huge_ptep_clear_flush(vma, addr, ptep); } return huge_ptep_get_and_clear(vma->vm_mm, addr, ptep); diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 4c7ad574b946..c2a9f4f6c7dd 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -113,7 +113,7 @@ static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, memset(__va(page_phys), KASAN_SHADOW_INIT, PAGE_SIZE); next = addr + PAGE_SIZE; set_pte(ptep, pfn_pte(__phys_to_pfn(page_phys), PAGE_KERNEL)); - } while (ptep++, addr = next, addr != end && pte_none(READ_ONCE(*ptep))); + } while (ptep++, addr = next, addr != end && pte_none(ptep_get(ptep))); } static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr, diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 3a27d887f7dd..343629a17042 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -173,7 +173,7 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, ptep = pte_set_fixmap_offset(pmdp, addr); do { - pte_t old_pte = READ_ONCE(*ptep); + pte_t old_pte = ptep_get(ptep); set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); @@ -182,7 +182,7 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, * only allow updates to the permission attributes. */ BUG_ON(!pgattr_change_is_safe(pte_val(old_pte), - READ_ONCE(pte_val(*ptep)))); + pte_val(ptep_get(ptep)))); phys += PAGE_SIZE; } while (ptep++, addr += PAGE_SIZE, addr != end); @@ -852,7 +852,7 @@ static void unmap_hotplug_pte_range(pmd_t *pmdp, unsigned long addr, do { ptep = pte_offset_kernel(pmdp, addr); - pte = READ_ONCE(*ptep); + pte = ptep_get(ptep); if (pte_none(pte)) continue; @@ -985,7 +985,7 @@ static void free_empty_pte_table(pmd_t *pmdp, unsigned long addr, do { ptep = pte_offset_kernel(pmdp, addr); - pte = READ_ONCE(*ptep); + pte = ptep_get(ptep); /* * This is just a sanity check here which verifies that @@ -1004,7 +1004,7 @@ static void free_empty_pte_table(pmd_t *pmdp, unsigned long addr, */ ptep = pte_offset_kernel(pmdp, 0UL); for (i = 0; i < PTRS_PER_PTE; i++) { - if (!pte_none(READ_ONCE(ptep[i]))) + if (!pte_none(ptep_get(&ptep[i]))) return; } @@ -1473,7 +1473,7 @@ pte_t ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr, pte * when the permission changes from executable to non-executable * in cases where cpu is affected with errata #2645198. */ - if (pte_user_exec(READ_ONCE(*ptep))) + if (pte_user_exec(ptep_get(ptep))) return ptep_clear_flush(vma, addr, ptep); } return ptep_get_and_clear(vma->vm_mm, addr, ptep); diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 924843f1f661..73a5e8f82586 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -36,7 +36,7 @@ bool can_set_direct_map(void) static int change_page_range(pte_t *ptep, unsigned long addr, void *data) { struct page_change_data *cdata = data; - pte_t pte = READ_ONCE(*ptep); + pte_t pte = ptep_get(ptep); pte = clear_pte_bit(pte, cdata->clear_mask); pte = set_pte_bit(pte, cdata->set_mask); @@ -245,5 +245,5 @@ bool kernel_page_present(struct page *page) return true; ptep = pte_offset_kernel(pmdp, addr); - return pte_valid(READ_ONCE(*ptep)); + return pte_valid(ptep_get(ptep)); } diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index 7b14df3c6477..f71ab4704cce 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -33,7 +33,7 @@ static void *trans_alloc(struct trans_pgd_info *info) static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) { - pte_t pte = READ_ONCE(*src_ptep); + pte_t pte = ptep_get(src_ptep); if (pte_valid(pte)) { /* From patchwork Thu Feb 15 10:31:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 201345 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:b825:b0:106:860b:bbdd with SMTP id da37csp303742dyb; Thu, 15 Feb 2024 02:37:02 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWWu7ocz4XTRSphm7e88+BnVtNHSlhP0u+RrbgoidFjyRWu+Dzt9GrcPupOoN548JZubLaC0L+bU7nk5vCWorv+EliS8w== X-Google-Smtp-Source: AGHT+IF4dhvkd13HnZAWjLQQ1C98sFWoeRSVjvDcT8Iq5Py2DWSN6Yia1+KznOiHeXlCeQE4kDD3 X-Received: by 2002:a17:907:2d29:b0:a3d:7dca:4c81 with SMTP id gs41-20020a1709072d2900b00a3d7dca4c81mr2077549ejc.18.1707993422041; Thu, 15 Feb 2024 02:37:02 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707993422; cv=pass; d=google.com; s=arc-20160816; b=tDwmqlE53srYDkr+td2eMkLaOOiLv0RBs0Bk7GtdJoZaZ0kqZBYNQGhgHUi+0DPg8k EuQJvtrJayMLGrfrpcZeo8Q+5DTCSkhcIm2TSyXj8Ur7D+lh/XQDgQFLzUu87ZXrHl+3 jDXOB6O/EOwbVYaixck7olJd1mxz6DOfvbQc5KZuF+bqbhJfgZB+g4B65wqbQSk3lex4 MLeSg1QMdHZwHC9Vrj5ZAzWNWw+K4YA2DqiwbXLlWY5RBF4qM/kMEjwj63VGoHp1BQ1W ShtdE7vHKDU4kC6rLaOk+o/wk1URM4Y6IMaRGi7J017sLR8u1bgP4y1JY8cXBGcYsIxJ 8yMg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=ZeD0LRgPYyPlqU6eixwfCpwvc0jLGbSMmuxIvFt/HH8=; fh=Heagowc1O8zVDXTyU76nJjOwDjz9r3Gi+0zmhF4ZELQ=; b=SwHWGmdS6CkQJyMgnNU5E1ef+42GjsVLp44A+wbSwIq6OyPAubw2FW5lmn3iDWdKOc vfeTTcijugJ9hGpz22FNXhhQOIPYprhCEh5AqisuS38IetEu2/ohz6DrxAIkATskTESB ejDzmxGNecu/l1Tk3x9RLrQcZ4sz2jv1pt3G09uRtQpx7bFJ74x2xrEHja0zOYilDAT4 RAMPqbr6s0iC+uUqXkICwEPnrDg6vKJAIie6Sc0YrlGv3ShVEEJ8XrfGhFw9L7UU2VT8 tvpaAEiPfZASj8vEKh75fghMW3OUJc9N70DCbw+AOvbqzdX1RyhqTOlbuXOG7NVqkVn/ i7cw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66644-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66644-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id k25-20020a17090646d900b00a3d9033c4b7si480236ejs.989.2024.02.15.02.37.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 02:37:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-66644-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66644-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66644-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 0FB311F248A3 for ; Thu, 15 Feb 2024 10:36:45 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 453BB12E1DA; Thu, 15 Feb 2024 10:32:51 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D372812D75D for ; Thu, 15 Feb 2024 10:32:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993169; cv=none; b=fKMbWTNNmx35Bzf5FdO6a0DdCqIyDLzYq2l9gcgmgu86HHm3tnRWBbt3UjEdoskgdRhr7NOfWJDEVLXfqT76HbzfPIUcLb7FLM1XCCGAeTGvc/M825+f/INRyBgfohovWlOtbB+Wv7nORCslVdLeAIvBswVQytTgRy3ZUNAhIbg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993169; c=relaxed/simple; bh=OAHXy2gdGDDW0ebIo3BAq4OM/BcS09YL/BhRhFAnEFs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qO8hpuGlqHc+1/9YKDeZGhryc3gCldqG5QxsJ6JpqbbXR5J7UZ0/JBVVEwbC6wB8NKH3H6bdVBHduW0nRi1b7yEJgmGonUuNe/ybSEZGONkuUn1WXZHkK5k8AMISqufdqwlITluUqiC+iVMLdA4b5pdEV3+J0PY2a/bHUL5/4yY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 222B7153B; Thu, 15 Feb 2024 02:33:28 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 18FC43F7B4; Thu, 15 Feb 2024 02:32:43 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 08/18] arm64/mm: Convert set_pte_at() to set_ptes(..., 1) Date: Thu, 15 Feb 2024 10:31:55 +0000 Message-Id: <20240215103205.2607016-9-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790960910649614426 X-GMAIL-MSGID: 1790960910649614426 Since set_ptes() was introduced, set_pte_at() has been implemented as a generic macro around set_ptes(..., 1). So this change should continue to generate the same code. However, making this change prepares us for the transparent contpte support. It means we can reroute set_ptes() to __set_ptes(). Since set_pte_at() is a generic macro, there will be no equivalent __set_pte_at() to reroute to. Note that a couple of calls to set_pte_at() remain in the arch code. This is intentional, since those call sites are acting on behalf of core-mm and should continue to call into the public set_ptes() rather than the arch-private __set_ptes(). Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 2 +- arch/arm64/kernel/mte.c | 2 +- arch/arm64/kvm/guest.c | 2 +- arch/arm64/mm/fault.c | 2 +- arch/arm64/mm/hugetlbpage.c | 10 +++++----- 5 files changed, 9 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index de034ca40bad..9a2df85eb493 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1084,7 +1084,7 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) #endif /* CONFIG_ARM64_MTE */ /* - * On AArch64, the cache coherency is handled via the set_pte_at() function. + * On AArch64, the cache coherency is handled via the set_ptes() function. */ static inline void update_mmu_cache_range(struct vm_fault *vmf, struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index a41ef3213e1e..59bfe2e96f8f 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -67,7 +67,7 @@ int memcmp_pages(struct page *page1, struct page *page2) /* * If the page content is identical but at least one of the pages is * tagged, return non-zero to avoid KSM merging. If only one of the - * pages is tagged, set_pte_at() may zero or change the tags of the + * pages is tagged, set_ptes() may zero or change the tags of the * other page via mte_sync_tags(). */ if (page_mte_tagged(page1) || page_mte_tagged(page2)) diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index aaf1d4939739..6e0df623c8e9 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -1072,7 +1072,7 @@ int kvm_vm_ioctl_mte_copy_tags(struct kvm *kvm, } else { /* * Only locking to serialise with a concurrent - * set_pte_at() in the VMM but still overriding the + * set_ptes() in the VMM but still overriding the * tags, hence ignoring the return value. */ try_page_mte_tagging(page); diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index a254761fa1bd..3235e23309ec 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -205,7 +205,7 @@ static void show_pte(unsigned long addr) * * It needs to cope with hardware update of the accessed/dirty state by other * agents in the system and can safely skip the __sync_icache_dcache() call as, - * like set_pte_at(), the PTE is never changed from no-exec to exec here. + * like set_ptes(), the PTE is never changed from no-exec to exec here. * * Returns whether or not the PTE actually changed. */ diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 2892f925ed66..27f6160890d1 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -247,12 +247,12 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, if (!pte_present(pte)) { for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) - set_pte_at(mm, addr, ptep, pte); + set_ptes(mm, addr, ptep, pte, 1); return; } if (!pte_cont(pte)) { - set_pte_at(mm, addr, ptep, pte); + set_ptes(mm, addr, ptep, pte, 1); return; } @@ -263,7 +263,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, clear_flush(mm, addr, ptep, pgsize, ncontig); for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); + set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); } pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, @@ -471,7 +471,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, hugeprot = pte_pgprot(pte); for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); + set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); return 1; } @@ -500,7 +500,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pfn = pte_pfn(pte); for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); + set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); } pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, From patchwork Thu Feb 15 10:31:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 201346 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:b825:b0:106:860b:bbdd with SMTP id da37csp303814dyb; Thu, 15 Feb 2024 02:37:10 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUMNoaYEMSEpBPOW+LN02ME9L79BnVadGQMD7wzD7ve2ABJr9BYT+gklnqwgULFiyKZwLFFOmf9XpucMQpeMLTq9iQrJA== X-Google-Smtp-Source: AGHT+IGt0jWGIFrWKzLjcAqX+HR0AlcW7QqiETBJHLiR8rgl4VxltKCfVW3GziU9Mrdd5M/v1EX1 X-Received: by 2002:aa7:c715:0:b0:563:7db2:4fe2 with SMTP id i21-20020aa7c715000000b005637db24fe2mr932785edq.41.1707993430398; Thu, 15 Feb 2024 02:37:10 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707993430; cv=pass; d=google.com; s=arc-20160816; b=aOeS7iSlZQR9xk0LnMcU7jC8dk3FDGtmOfGS8rftD4T6zcXxjsgCQyIF0TvkU2en82 GZxfRBaBZFyQrqIHyShI7vlwSTh8oqvdvxZ6cNHyoEE33xv8PMNWCjpkV6Pwp9rFSL79 QoCiMukqIhdzQdeQD/8Ua4SVnn/waTadfvQSc1FPfVfYzbcSMAhyCVSAAPSeDRp9p/DL mjbO+iwe8aUvtA+uhvcP6Jv+i1+dCuKSx6UiD8gTGEH1S268rhVk+X395se2cGWVSyqW uLNKNDYcoWeoUi2P8kkVD3lWCy07pcC3s4UsBY/YEgLYpwPr63nxqNo7sT+7Q8xUJmg0 2Q/A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=NNqJIU65wxbHkraYUk5NQY8zX49rVnFHXTzrcPivxfA=; fh=Heagowc1O8zVDXTyU76nJjOwDjz9r3Gi+0zmhF4ZELQ=; b=rvx1njnylM1nviP5pvQ4isEU22mBqexT7/8RSLqAondXkdowu9SbRdCsqld3I1lWWU iriYMlvaB8L0HDzMVce8/rIKJS3QcXmzygemUrKOL1IIW8OAvvQ95/Ue2eJfSZVGg12y gI1nb1bfklpfWuUcfSv93XMFtOSJqjNm5iFq47AfH+o+eHWvoiFUfFXitx4YkX7ek9lc 4VR9MmJNqA2Kf22rr2UC+EayoOy0AVIt79Iu0iGkhe4Cp/EgTfgXox4iAL8ynflv88Te mjIGqIPsT3kh4bJGB2bJXFL3mNLK8gqDKoRFm3pkXUE9164mAGxEAh1HlqN3lYAJKWj9 tgzw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66645-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66645-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id e22-20020a50d4d6000000b005638d1c43c5si538694edj.205.2024.02.15.02.37.10 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 02:37:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-66645-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66645-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66645-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 242FF1F24539 for ; Thu, 15 Feb 2024 10:37:02 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 27C6F12EBCD; Thu, 15 Feb 2024 10:32:54 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 3BAE612DD8B for ; Thu, 15 Feb 2024 10:32:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993172; cv=none; b=UVytpcxV/zGAWnWqKakhUUTr1eieJEBpS/jQThIYb63ib8cyf5zBYASWCDnt4MrjbWGzDdtVWXFUmPMT7NMZdm2B1Yb7gkOIv5hjMTZlQQzCeJzQu1sNWMlWSVgc6tApaTAkkl+vmLurkUMEj41oqUlZ/y1M2hgNMVuX9hJ8gds= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993172; c=relaxed/simple; bh=7L0V2A6x2ZTDwqOnJsQvbevz7Qvgf2Zqp4ANa3PXVOU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gzBp/uaKWx7M+2u/EJiJYLaN8Bmh5VGQp45aJrgxaYMkpnzRpgwSD9DMdddRWXnsIn5Wtyda+3zxSGYSJ2uJdyHxfuwMu2c6W4AqVucdvonL4YfWZA0dcUvNidSDOTBp3+NYAjfjNphfC6u2WIYOZNdXEhzYLvCORtMkES3o7m0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 86F1E1570; Thu, 15 Feb 2024 02:33:31 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 7D7603F7B4; Thu, 15 Feb 2024 02:32:47 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 09/18] arm64/mm: Convert ptep_clear() to ptep_get_and_clear() Date: Thu, 15 Feb 2024 10:31:56 +0000 Message-Id: <20240215103205.2607016-10-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790960919374282278 X-GMAIL-MSGID: 1790960919374282278 ptep_clear() is a generic wrapper around the arch-implemented ptep_get_and_clear(). We are about to convert ptep_get_and_clear() into a public version and private version (__ptep_get_and_clear()) to support the transparent contpte work. We won't have a private version of ptep_clear() so let's convert it to directly call ptep_get_and_clear(). Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/mm/hugetlbpage.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 27f6160890d1..48e8b429879d 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -229,7 +229,7 @@ static void clear_flush(struct mm_struct *mm, unsigned long i, saddr = addr; for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - ptep_clear(mm, addr, ptep); + ptep_get_and_clear(mm, addr, ptep); flush_tlb_range(&vma, saddr, addr); } From patchwork Thu Feb 15 10:31:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 201348 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:b825:b0:106:860b:bbdd with SMTP id da37csp304294dyb; Thu, 15 Feb 2024 02:38:04 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXheYSb0yWFoe+xwiNkVAV54zMEj6Zn5UAQZdkyRFaApH6KskJe3L/Fpwq9B2cCtXlgnaaob6e89zNu+HAdxi8Mq/ISpA== X-Google-Smtp-Source: AGHT+IGsnEz5uLqBlCxFPoY6L9iWS94ZILC2RCu+JVF1+k2WOZQykLUiYfCejrCILk44q1UsOyaB X-Received: by 2002:a05:6808:180a:b0:3c0:31e5:35f1 with SMTP id bh10-20020a056808180a00b003c031e535f1mr1672344oib.36.1707993484494; Thu, 15 Feb 2024 02:38:04 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707993484; cv=pass; d=google.com; s=arc-20160816; b=BcjW9paPq/KTaIBAPqrcrVXikOc/+iBFQOlxM9larDgjfsgHMbGZvNNRa6uGfa/Vsl irUcqg6lOfODF2O88xofIiOM5bm6AIzS25+GK474zbH41eC0yPJvwE/Xvw6mEyL9lxEz niCEuWBSyHlkrCncyY7q9StBL5V4dlAQkvPq0YMBreObhReXgUvVBAESEHElIMQJ0K8v BQfyhSGRMfe/jdP7lXchQl72wxmJqo769fY8CpIGpPkkVSleSWp8IJx8mlyEUE1v+Q0v VdPMcuoG/WbDrBHDzjxzH39fw+2kF5VAXWaICfQKA1tIWKgxZ9pDXyy7FOjgmohnomXu goUg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=i0bpuL9kL8NtPuhBKTPcA0JZgz3FoRCYzdhThhi+DB8=; fh=Heagowc1O8zVDXTyU76nJjOwDjz9r3Gi+0zmhF4ZELQ=; b=S4nn+8a2Mr5tqFZdq85Axc0kmgOcqspKdS+ODaReYiCPe2K9r4+3RPK3Sw2x1MqNCG uLrfFA/z+yV1JiOGfLv+9JO/5Sf5Isr2xP+WnXeD2gwmPkQ3islqjyjdy0akeMCLPYUQ zedF6oR8tLYf8llKVaNjnn6lkowc/mgUpsQWGFeGMV1hkPHCzZ32vkGutRWE6wCYCPWB 5Xh0fXF8NszIaUfaWrPFHEjSRpAsUzq2JTfPJx3vVc2FS+56Mp4trIsR8xYEz0Qt48UY ioQjULNQHve3uxxdk6VtBmph+R61jxDiYXiHMb0Z0cjxSavj9ry8hexSN9c9II/oMt7q OvGw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66646-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66646-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id 6-20020a05621420a600b0068f23abe876si354373qvd.308.2024.02.15.02.38.04 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 02:38:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-66646-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66646-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66646-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 1433D1C2042F for ; Thu, 15 Feb 2024 10:38:04 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 01A3E12F586; Thu, 15 Feb 2024 10:33:03 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id BAA0912E1E5 for ; Thu, 15 Feb 2024 10:32:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993176; cv=none; b=s4NPcaGtjMz43/PIz1nZWrChTu9I6jjfccVw8RtoM9huYrdDucEkzqZSrNHDEpB77y0TIM4yUXGyXV7epR7E3XrFbM0VXc6A+nR4sDxZYIFDwCK/3Mp70iUxvtK44VxHqMobfpwgyGhoZvPhJvfaNqGc/7TvAalZQxp8NcB3iiQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993176; c=relaxed/simple; bh=RYkqb28gwxlnPin/PxJUb/rGxN4F5LPgypH5LINXtU0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=LX2hgJLzdoKI+IOzo3jgtUfg34cXQ68P8tMqVSCI0Vqwm3UevbeobZ2QsOOA0jeepkSUaTpSuGbDFzGGb4fDo91eBOuRh+KnEgHDzo4i6yP+VlZ//4HHQlXWWz0Y+if++3DqB/1D4+ztFRPHlO21oonoOp7Hc+YOaOgNsoBlU5o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1344C1576; Thu, 15 Feb 2024 02:33:35 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E30303F7B4; Thu, 15 Feb 2024 02:32:50 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 10/18] arm64/mm: New ptep layer to manage contig bit Date: Thu, 15 Feb 2024 10:31:57 +0000 Message-Id: <20240215103205.2607016-11-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790960975964155636 X-GMAIL-MSGID: 1790960975964155636 Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. The following APIs are treated this way: - ptep_get - set_pte - set_ptes - pte_clear - ptep_get_and_clear - ptep_test_and_clear_young - ptep_clear_flush_young - ptep_set_wrprotect - ptep_set_access_flags Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 83 +++++++++++++++++--------------- arch/arm64/kernel/efi.c | 4 +- arch/arm64/kernel/mte.c | 2 +- arch/arm64/kvm/guest.c | 2 +- arch/arm64/mm/fault.c | 12 ++--- arch/arm64/mm/fixmap.c | 4 +- arch/arm64/mm/hugetlbpage.c | 40 +++++++-------- arch/arm64/mm/kasan_init.c | 6 +-- arch/arm64/mm/mmu.c | 14 +++--- arch/arm64/mm/pageattr.c | 6 +-- arch/arm64/mm/trans_pgd.c | 6 +-- 11 files changed, 93 insertions(+), 86 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 9a2df85eb493..7336d40a893a 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -93,7 +93,8 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) #define pte_none(pte) (!pte_val(pte)) -#define pte_clear(mm,addr,ptep) set_pte(ptep, __pte(0)) +#define __pte_clear(mm, addr, ptep) \ + __set_pte(ptep, __pte(0)) #define pte_page(pte) (pfn_to_page(pte_pfn(pte))) /* @@ -137,7 +138,7 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) * so that we don't erroneously return false for pages that have been * remapped as PROT_NONE but are yet to be flushed from the TLB. * Note that we can't make any assumptions based on the state of the access - * flag, since ptep_clear_flush_young() elides a DSB when invalidating the + * flag, since __ptep_clear_flush_young() elides a DSB when invalidating the * TLB. */ #define pte_accessible(mm, pte) \ @@ -261,7 +262,7 @@ static inline pte_t pte_mkdevmap(pte_t pte) return set_pte_bit(pte, __pgprot(PTE_DEVMAP | PTE_SPECIAL)); } -static inline void set_pte(pte_t *ptep, pte_t pte) +static inline void __set_pte(pte_t *ptep, pte_t pte) { WRITE_ONCE(*ptep, pte); @@ -275,8 +276,7 @@ static inline void set_pte(pte_t *ptep, pte_t pte) } } -#define ptep_get ptep_get -static inline pte_t ptep_get(pte_t *ptep) +static inline pte_t __ptep_get(pte_t *ptep) { return READ_ONCE(*ptep); } @@ -308,7 +308,7 @@ static inline void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, if (!IS_ENABLED(CONFIG_DEBUG_VM)) return; - old_pte = ptep_get(ptep); + old_pte = __ptep_get(ptep); if (!pte_valid(old_pte) || !pte_valid(pte)) return; @@ -317,7 +317,7 @@ static inline void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, /* * Check for potential race with hardware updates of the pte - * (ptep_set_access_flags safely changes valid ptes without going + * (__ptep_set_access_flags safely changes valid ptes without going * through an invalid entry). */ VM_WARN_ONCE(!pte_young(pte), @@ -363,23 +363,22 @@ static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) return pfn_pte(pte_pfn(pte) + nr, pte_pgprot(pte)); } -static inline void set_ptes(struct mm_struct *mm, - unsigned long __always_unused addr, - pte_t *ptep, pte_t pte, unsigned int nr) +static inline void __set_ptes(struct mm_struct *mm, + unsigned long __always_unused addr, + pte_t *ptep, pte_t pte, unsigned int nr) { page_table_check_ptes_set(mm, ptep, pte, nr); __sync_cache_and_tags(pte, nr); for (;;) { __check_safe_pte_update(mm, ptep, pte); - set_pte(ptep, pte); + __set_pte(ptep, pte); if (--nr == 0) break; ptep++; pte = pte_advance_pfn(pte, 1); } } -#define set_ptes set_ptes /* * Huge pte definitions. @@ -546,7 +545,7 @@ static inline void __set_pte_at(struct mm_struct *mm, { __sync_cache_and_tags(pte, nr); __check_safe_pte_update(mm, ptep, pte); - set_pte(ptep, pte); + __set_pte(ptep, pte); } static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr, @@ -860,8 +859,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) return pte_pmd(pte_modify(pmd_pte(pmd), newprot)); } -#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS -extern int ptep_set_access_flags(struct vm_area_struct *vma, +extern int __ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty); @@ -871,7 +869,8 @@ static inline int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t entry, int dirty) { - return ptep_set_access_flags(vma, address, (pte_t *)pmdp, pmd_pte(entry), dirty); + return __ptep_set_access_flags(vma, address, (pte_t *)pmdp, + pmd_pte(entry), dirty); } static inline int pud_devmap(pud_t pud) @@ -905,12 +904,13 @@ static inline bool pud_user_accessible_page(pud_t pud) /* * Atomic pte/pmd modifications. */ -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG -static inline int __ptep_test_and_clear_young(pte_t *ptep) +static inline int __ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, + pte_t *ptep) { pte_t old_pte, pte; - pte = ptep_get(ptep); + pte = __ptep_get(ptep); do { old_pte = pte; pte = pte_mkold(pte); @@ -921,18 +921,10 @@ static inline int __ptep_test_and_clear_young(pte_t *ptep) return pte_young(pte); } -static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, - unsigned long address, - pte_t *ptep) -{ - return __ptep_test_and_clear_young(ptep); -} - -#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH -static inline int ptep_clear_flush_young(struct vm_area_struct *vma, +static inline int __ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { - int young = ptep_test_and_clear_young(vma, address, ptep); + int young = __ptep_test_and_clear_young(vma, address, ptep); if (young) { /* @@ -955,12 +947,11 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { - return ptep_test_and_clear_young(vma, address, (pte_t *)pmdp); + return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp); } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -#define __HAVE_ARCH_PTEP_GET_AND_CLEAR -static inline pte_t ptep_get_and_clear(struct mm_struct *mm, +static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, unsigned long address, pte_t *ptep) { pte_t pte = __pte(xchg_relaxed(&pte_val(*ptep), 0)); @@ -984,15 +975,15 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ /* - * ptep_set_wrprotect - mark read-only while trasferring potential hardware + * __ptep_set_wrprotect - mark read-only while trasferring potential hardware * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. */ -#define __HAVE_ARCH_PTEP_SET_WRPROTECT -static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep) +static inline void __ptep_set_wrprotect(struct mm_struct *mm, + unsigned long address, pte_t *ptep) { pte_t old_pte, pte; - pte = ptep_get(ptep); + pte = __ptep_get(ptep); do { old_pte = pte; pte = pte_wrprotect(pte); @@ -1006,7 +997,7 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long address, pmd_t *pmdp) { - ptep_set_wrprotect(mm, address, (pte_t *)pmdp); + __ptep_set_wrprotect(mm, address, (pte_t *)pmdp); } #define pmdp_establish pmdp_establish @@ -1084,7 +1075,7 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) #endif /* CONFIG_ARM64_MTE */ /* - * On AArch64, the cache coherency is handled via the set_ptes() function. + * On AArch64, the cache coherency is handled via the __set_ptes() function. */ static inline void update_mmu_cache_range(struct vm_fault *vmf, struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, @@ -1136,6 +1127,22 @@ extern pte_t ptep_modify_prot_start(struct vm_area_struct *vma, extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); + +#define ptep_get __ptep_get +#define set_pte __set_pte +#define set_ptes __set_ptes +#define pte_clear __pte_clear +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR +#define ptep_get_and_clear __ptep_get_and_clear +#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG +#define ptep_test_and_clear_young __ptep_test_and_clear_young +#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH +#define ptep_clear_flush_young __ptep_clear_flush_young +#define __HAVE_ARCH_PTEP_SET_WRPROTECT +#define ptep_set_wrprotect __ptep_set_wrprotect +#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS +#define ptep_set_access_flags __ptep_set_access_flags + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index d0e08e93b246..9afcc690fe73 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -103,7 +103,7 @@ static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data) { struct set_perm_data *spd = data; const efi_memory_desc_t *md = spd->md; - pte_t pte = ptep_get(ptep); + pte_t pte = __ptep_get(ptep); if (md->attribute & EFI_MEMORY_RO) pte = set_pte_bit(pte, __pgprot(PTE_RDONLY)); @@ -111,7 +111,7 @@ static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data) pte = set_pte_bit(pte, __pgprot(PTE_PXN)); else if (system_supports_bti_kernel() && spd->has_bti) pte = set_pte_bit(pte, __pgprot(PTE_GP)); - set_pte(ptep, pte); + __set_pte(ptep, pte); return 0; } diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index 59bfe2e96f8f..dcdcccd40891 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -67,7 +67,7 @@ int memcmp_pages(struct page *page1, struct page *page2) /* * If the page content is identical but at least one of the pages is * tagged, return non-zero to avoid KSM merging. If only one of the - * pages is tagged, set_ptes() may zero or change the tags of the + * pages is tagged, __set_ptes() may zero or change the tags of the * other page via mte_sync_tags(). */ if (page_mte_tagged(page1) || page_mte_tagged(page2)) diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index 6e0df623c8e9..629145fd3161 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -1072,7 +1072,7 @@ int kvm_vm_ioctl_mte_copy_tags(struct kvm *kvm, } else { /* * Only locking to serialise with a concurrent - * set_ptes() in the VMM but still overriding the + * __set_ptes() in the VMM but still overriding the * tags, hence ignoring the return value. */ try_page_mte_tagging(page); diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 3235e23309ec..9a1c66183d16 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -191,7 +191,7 @@ static void show_pte(unsigned long addr) if (!ptep) break; - pte = ptep_get(ptep); + pte = __ptep_get(ptep); pr_cont(", pte=%016llx", pte_val(pte)); pte_unmap(ptep); } while(0); @@ -205,16 +205,16 @@ static void show_pte(unsigned long addr) * * It needs to cope with hardware update of the accessed/dirty state by other * agents in the system and can safely skip the __sync_icache_dcache() call as, - * like set_ptes(), the PTE is never changed from no-exec to exec here. + * like __set_ptes(), the PTE is never changed from no-exec to exec here. * * Returns whether or not the PTE actually changed. */ -int ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep, - pte_t entry, int dirty) +int __ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep, + pte_t entry, int dirty) { pteval_t old_pteval, pteval; - pte_t pte = ptep_get(ptep); + pte_t pte = __ptep_get(ptep); if (pte_same(pte, entry)) return 0; diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index c0a3301203bd..bfc02568805a 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -121,9 +121,9 @@ void __set_fixmap(enum fixed_addresses idx, ptep = fixmap_pte(addr); if (pgprot_val(flags)) { - set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, flags)); + __set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, flags)); } else { - pte_clear(&init_mm, addr, ptep); + __pte_clear(&init_mm, addr, ptep); flush_tlb_kernel_range(addr, addr+PAGE_SIZE); } } diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 48e8b429879d..0f0e10bb0a95 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -145,14 +145,14 @@ pte_t huge_ptep_get(pte_t *ptep) { int ncontig, i; size_t pgsize; - pte_t orig_pte = ptep_get(ptep); + pte_t orig_pte = __ptep_get(ptep); if (!pte_present(orig_pte) || !pte_cont(orig_pte)) return orig_pte; ncontig = num_contig_ptes(page_size(pte_page(orig_pte)), &pgsize); for (i = 0; i < ncontig; i++, ptep++) { - pte_t pte = ptep_get(ptep); + pte_t pte = __ptep_get(ptep); if (pte_dirty(pte)) orig_pte = pte_mkdirty(orig_pte); @@ -177,11 +177,11 @@ static pte_t get_clear_contig(struct mm_struct *mm, unsigned long pgsize, unsigned long ncontig) { - pte_t orig_pte = ptep_get(ptep); + pte_t orig_pte = __ptep_get(ptep); unsigned long i; for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) { - pte_t pte = ptep_get_and_clear(mm, addr, ptep); + pte_t pte = __ptep_get_and_clear(mm, addr, ptep); /* * If HW_AFDBM is enabled, then the HW could turn on @@ -229,7 +229,7 @@ static void clear_flush(struct mm_struct *mm, unsigned long i, saddr = addr; for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - ptep_get_and_clear(mm, addr, ptep); + __ptep_get_and_clear(mm, addr, ptep); flush_tlb_range(&vma, saddr, addr); } @@ -247,12 +247,12 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, if (!pte_present(pte)) { for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) - set_ptes(mm, addr, ptep, pte, 1); + __set_ptes(mm, addr, ptep, pte, 1); return; } if (!pte_cont(pte)) { - set_ptes(mm, addr, ptep, pte, 1); + __set_ptes(mm, addr, ptep, pte, 1); return; } @@ -263,7 +263,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, clear_flush(mm, addr, ptep, pgsize, ncontig); for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); + __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); } pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, @@ -393,7 +393,7 @@ void huge_pte_clear(struct mm_struct *mm, unsigned long addr, ncontig = num_contig_ptes(sz, &pgsize); for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - pte_clear(mm, addr, ptep); + __pte_clear(mm, addr, ptep); } pte_t huge_ptep_get_and_clear(struct mm_struct *mm, @@ -401,10 +401,10 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, { int ncontig; size_t pgsize; - pte_t orig_pte = ptep_get(ptep); + pte_t orig_pte = __ptep_get(ptep); if (!pte_cont(orig_pte)) - return ptep_get_and_clear(mm, addr, ptep); + return __ptep_get_and_clear(mm, addr, ptep); ncontig = find_num_contig(mm, addr, ptep, &pgsize); @@ -424,11 +424,11 @@ static int __cont_access_flags_changed(pte_t *ptep, pte_t pte, int ncontig) { int i; - if (pte_write(pte) != pte_write(ptep_get(ptep))) + if (pte_write(pte) != pte_write(__ptep_get(ptep))) return 1; for (i = 0; i < ncontig; i++) { - pte_t orig_pte = ptep_get(ptep + i); + pte_t orig_pte = __ptep_get(ptep + i); if (pte_dirty(pte) != pte_dirty(orig_pte)) return 1; @@ -452,7 +452,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, pte_t orig_pte; if (!pte_cont(pte)) - return ptep_set_access_flags(vma, addr, ptep, pte, dirty); + return __ptep_set_access_flags(vma, addr, ptep, pte, dirty); ncontig = find_num_contig(mm, addr, ptep, &pgsize); dpfn = pgsize >> PAGE_SHIFT; @@ -471,7 +471,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, hugeprot = pte_pgprot(pte); for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); + __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); return 1; } @@ -485,8 +485,8 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, size_t pgsize; pte_t pte; - if (!pte_cont(ptep_get(ptep))) { - ptep_set_wrprotect(mm, addr, ptep); + if (!pte_cont(__ptep_get(ptep))) { + __ptep_set_wrprotect(mm, addr, ptep); return; } @@ -500,7 +500,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pfn = pte_pfn(pte); for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); + __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); } pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, @@ -510,7 +510,7 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, size_t pgsize; int ncontig; - if (!pte_cont(ptep_get(ptep))) + if (!pte_cont(__ptep_get(ptep))) return ptep_clear_flush(vma, addr, ptep); ncontig = find_num_contig(mm, addr, ptep, &pgsize); @@ -543,7 +543,7 @@ pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr * when the permission changes from executable to non-executable * in cases where cpu is affected with errata #2645198. */ - if (pte_user_exec(ptep_get(ptep))) + if (pte_user_exec(__ptep_get(ptep))) return huge_ptep_clear_flush(vma, addr, ptep); } return huge_ptep_get_and_clear(vma->vm_mm, addr, ptep); diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index c2a9f4f6c7dd..9ee16cfce587 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -112,8 +112,8 @@ static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, if (!early) memset(__va(page_phys), KASAN_SHADOW_INIT, PAGE_SIZE); next = addr + PAGE_SIZE; - set_pte(ptep, pfn_pte(__phys_to_pfn(page_phys), PAGE_KERNEL)); - } while (ptep++, addr = next, addr != end && pte_none(ptep_get(ptep))); + __set_pte(ptep, pfn_pte(__phys_to_pfn(page_phys), PAGE_KERNEL)); + } while (ptep++, addr = next, addr != end && pte_none(__ptep_get(ptep))); } static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr, @@ -271,7 +271,7 @@ static void __init kasan_init_shadow(void) * so we should make sure that it maps the zero page read-only. */ for (i = 0; i < PTRS_PER_PTE; i++) - set_pte(&kasan_early_shadow_pte[i], + __set_pte(&kasan_early_shadow_pte[i], pfn_pte(sym_to_pfn(kasan_early_shadow_page), PAGE_KERNEL_RO)); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 343629a17042..6208c7541f87 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -173,16 +173,16 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, ptep = pte_set_fixmap_offset(pmdp, addr); do { - pte_t old_pte = ptep_get(ptep); + pte_t old_pte = __ptep_get(ptep); - set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); + __set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); /* * After the PTE entry has been populated once, we * only allow updates to the permission attributes. */ BUG_ON(!pgattr_change_is_safe(pte_val(old_pte), - pte_val(ptep_get(ptep)))); + pte_val(__ptep_get(ptep)))); phys += PAGE_SIZE; } while (ptep++, addr += PAGE_SIZE, addr != end); @@ -852,12 +852,12 @@ static void unmap_hotplug_pte_range(pmd_t *pmdp, unsigned long addr, do { ptep = pte_offset_kernel(pmdp, addr); - pte = ptep_get(ptep); + pte = __ptep_get(ptep); if (pte_none(pte)) continue; WARN_ON(!pte_present(pte)); - pte_clear(&init_mm, addr, ptep); + __pte_clear(&init_mm, addr, ptep); flush_tlb_kernel_range(addr, addr + PAGE_SIZE); if (free_mapped) free_hotplug_page_range(pte_page(pte), @@ -985,7 +985,7 @@ static void free_empty_pte_table(pmd_t *pmdp, unsigned long addr, do { ptep = pte_offset_kernel(pmdp, addr); - pte = ptep_get(ptep); + pte = __ptep_get(ptep); /* * This is just a sanity check here which verifies that @@ -1004,7 +1004,7 @@ static void free_empty_pte_table(pmd_t *pmdp, unsigned long addr, */ ptep = pte_offset_kernel(pmdp, 0UL); for (i = 0; i < PTRS_PER_PTE; i++) { - if (!pte_none(ptep_get(&ptep[i]))) + if (!pte_none(__ptep_get(&ptep[i]))) return; } diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 73a5e8f82586..0c4e3ecf989d 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -36,12 +36,12 @@ bool can_set_direct_map(void) static int change_page_range(pte_t *ptep, unsigned long addr, void *data) { struct page_change_data *cdata = data; - pte_t pte = ptep_get(ptep); + pte_t pte = __ptep_get(ptep); pte = clear_pte_bit(pte, cdata->clear_mask); pte = set_pte_bit(pte, cdata->set_mask); - set_pte(ptep, pte); + __set_pte(ptep, pte); return 0; } @@ -245,5 +245,5 @@ bool kernel_page_present(struct page *page) return true; ptep = pte_offset_kernel(pmdp, addr); - return pte_valid(ptep_get(ptep)); + return pte_valid(__ptep_get(ptep)); } diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index f71ab4704cce..5139a28130c0 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -33,7 +33,7 @@ static void *trans_alloc(struct trans_pgd_info *info) static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) { - pte_t pte = ptep_get(src_ptep); + pte_t pte = __ptep_get(src_ptep); if (pte_valid(pte)) { /* @@ -41,7 +41,7 @@ static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) * read only (code, rodata). Clear the RDONLY bit from * the temporary mappings we use during restore. */ - set_pte(dst_ptep, pte_mkwrite_novma(pte)); + __set_pte(dst_ptep, pte_mkwrite_novma(pte)); } else if ((debug_pagealloc_enabled() || is_kfence_address((void *)addr)) && !pte_none(pte)) { /* @@ -55,7 +55,7 @@ static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) */ BUG_ON(!pfn_valid(pte_pfn(pte))); - set_pte(dst_ptep, pte_mkpresent(pte_mkwrite_novma(pte))); + __set_pte(dst_ptep, pte_mkpresent(pte_mkwrite_novma(pte))); } } From patchwork Thu Feb 15 10:31:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 201347 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:b825:b0:106:860b:bbdd with SMTP id da37csp304250dyb; Thu, 15 Feb 2024 02:37:59 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWO7153WXP+Rni+Wtdj0UtulgVxzToUZ2limVBAGGNsjv3vele8LpHL/SdyLKDWkPlEA/OTIG7+E1pCAHkVjE1Cb4ZeDQ== X-Google-Smtp-Source: AGHT+IFs7gUktw4pBlIoSqcb22aGa2k3QfbccdecXp6vG3U2YoanZovUsujiMVEvpIgzSiYn+1Uo X-Received: by 2002:a19:9143:0:b0:511:8763:bb63 with SMTP id y3-20020a199143000000b005118763bb63mr1004839lfj.41.1707993479675; Thu, 15 Feb 2024 02:37:59 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707993479; cv=pass; d=google.com; s=arc-20160816; b=sMt/9pVkC//N4QMfdmbnD0aqoi4YRD5uev80crSzoZh1fqwSYCNfpylhJpOjSMn7hZ VnlRf+0poZTxq+pRHnK/FbafkhRlVYeoi0TOHYRRuagiy/iiO2FmdzG6c2dz4em+gCDo t4ms+p328JtX8dfSm0gywc5nq1WUB9GG62DqbQh/rR1W7625+7dX9KLF1aUrYA0Asbcf 6MO6Ee/8iik7kNwk3hKSWWnNnD93mA1DXdTV9hTZzio+/YOzoxGejCXcAoTJKxSFZ3d6 oyrmmnuFWUlvPUfSBGxRtZUbH87HUt7cUzaIWwrfe7vvgzqXF1P79U9GTmOd4+d+aFPr iELQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=4J1dTrrIzJ+DpallLluT2b35txLH0YV5OvrN+DtNofk=; fh=Heagowc1O8zVDXTyU76nJjOwDjz9r3Gi+0zmhF4ZELQ=; b=vXOE26ITSmS+jPAlzGI7ucc21FgoX7Nw6/W3CHFvgQvurWvJGbiHboSRYlVGEMLu0b /4F1zuiLteS0b/KJEbF3InU7yEc4KCdgmK7vHAFx0RLxMQRkc5fpJkOt9RX6MAiyp5A9 QKwqXxEK4UibtJ7vqI19vU1ItJBmhhwknbm+chDgQKp+BtQbNcIr94ygwyDi75HRcpH/ rSULuY+j5WXrdJjv0L6z5GiaRV0ch876Ivhawj3g/ifstP/hzMfg4Zq1myd4Z1+cWA0m o4uFxzLm4T72bRFBQOAFhmTspZNbKsaqQzjrSsksd1zsV6sGPJrVvMulsTyEhZbXMXLp Fawg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66648-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66648-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id a4-20020a1709062b0400b00a3d6cc63d45si532457ejg.135.2024.02.15.02.37.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 02:37:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-66648-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66648-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66648-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 472061F207C1 for ; Thu, 15 Feb 2024 10:37:59 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1CA0D12FB04; Thu, 15 Feb 2024 10:33:02 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id E50F012F38D for ; Thu, 15 Feb 2024 10:32:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993180; cv=none; b=MKTveGHENdRDJAb3iZsXJrfwGTAKvxLIm7uiXMK4ZkzQH9hk47NdVOfpqHbcscktIaMECtNU4lfyWBfULH+N3B+mzBFSMA8Mf4EWMRDWveTeX0FO0g7ujsTUAjLhJ9ZYMhjw07Vt60TgQq7IKMFCLp0Fp0/Gis02IzgByFgtF7g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993180; c=relaxed/simple; bh=7hm98Uy4uzJ8moWfBkXOKwgSvNFZT6zUX8a/wpE/3D8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FhZCIDTh/YoMke6UJr7oV0TobnPA9odORjdckqVvniytUCwJCtlT9Rux8bTm8G3gLmP5X9IqYocQInQtErFkOw7HnAhsmm4OJ6KRT89J9UWKshYPsHTmtU9VAWNF5cPcJhE13Kq6cjJ6we2DkYtYErqJYSIZ82gf8P+rXu15Ug0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 79677DA7; Thu, 15 Feb 2024 02:33:38 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6E5073F7B4; Thu, 15 Feb 2024 02:32:54 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 11/18] arm64/mm: Split __flush_tlb_range() to elide trailing DSB Date: Thu, 15 Feb 2024 10:31:58 +0000 Message-Id: <20240215103205.2607016-12-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790960970714181459 X-GMAIL-MSGID: 1790960970714181459 Split __flush_tlb_range() into __flush_tlb_range_nosync() + __flush_tlb_range(), in the same way as the existing flush_tlb_page() arrangement. This allows calling __flush_tlb_range_nosync() to elide the trailing DSB. Forthcoming "contpte" code will take advantage of this when clearing the young bit from a contiguous range of ptes. Ordering between dsb and mmu_notifier_arch_invalidate_secondary_tlbs() has changed, but now aligns with the ordering of __flush_tlb_page(). It has been discussed that __flush_tlb_page() may be wrong though. Regardless, both will be resolved separately if needed. Reviewed-by: David Hildenbrand Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/tlbflush.h | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 1deb5d789c2e..3b0e8248e1a4 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -422,7 +422,7 @@ do { \ #define __flush_s2_tlb_range_op(op, start, pages, stride, tlb_level) \ __flush_tlb_range_op(op, start, pages, stride, 0, tlb_level, false, kvm_lpa2_is_enabled()); -static inline void __flush_tlb_range(struct vm_area_struct *vma, +static inline void __flush_tlb_range_nosync(struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long stride, bool last_level, int tlb_level) @@ -456,10 +456,19 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, __flush_tlb_range_op(vae1is, start, pages, stride, asid, tlb_level, true, lpa2_is_enabled()); - dsb(ish); mmu_notifier_arch_invalidate_secondary_tlbs(vma->vm_mm, start, end); } +static inline void __flush_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end, + unsigned long stride, bool last_level, + int tlb_level) +{ + __flush_tlb_range_nosync(vma, start, end, stride, + last_level, tlb_level); + dsb(ish); +} + static inline void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { From patchwork Thu Feb 15 10:32:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 201359 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:b825:b0:106:860b:bbdd with SMTP id da37csp307624dyb; Thu, 15 Feb 2024 02:45:59 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXEPVqaSYAZAPQcS2WqTXgQXEC7oVyk9en/c76wmnY2QIWUJL0PHa3OEA1fw7augpuv9jz2X2SkBxfs5rF604A1wEmt3w== X-Google-Smtp-Source: AGHT+IHTVrbgFyuVB/JNo80qE94ZkIdN0vzM+Bnmn9Mx2vT897GlX3E4QXK90HENF4oBzU8h8ADw X-Received: by 2002:a05:6871:58a8:b0:214:a08c:5220 with SMTP id ok40-20020a05687158a800b00214a08c5220mr1269688oac.20.1707993959592; Thu, 15 Feb 2024 02:45:59 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707993959; cv=pass; d=google.com; s=arc-20160816; b=MxBf2Ad1fonSVi94NQbsRNzNIqGsAfFDX1ZH8zR+KTwIamkWfVJe+hLEPvSbM87nVr 1e9LUTdKpz1DiouBWDtVA7chtBTfSsfYgA+2uFZWVe/c1m+DcuIZyPwgzRIF77noHIML N4Av8hiR9/UNXcJdnWOvUrpApGP1mXsDleAzh30cKrJanVJcuajDJ05E81F3/ExjxOmq E76TvXy1VwRM3lJTTe+lQk9jr35GwhIJlb+o8aFWPexk03sjupxkvADI4meF31eu5MFa We5p0eHZgAs8MrZ3o55IsNnQ7nGjnOT5RxS18NN34tZuyWes4ldgi0F3bMtp80384cD8 bbAA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=CrpX24hJ21lNTKd4YCqgFb/8kYDx8R2PLZ109arTQ9Q=; fh=Heagowc1O8zVDXTyU76nJjOwDjz9r3Gi+0zmhF4ZELQ=; b=JaQ5TmHlH/7cfLJVdgHMcs3pR2RdbEN5YwlbfJ4KIDbF6m0WWl5tfpqmyFBnS42MRk iuy4XIP4UJsd45f5bJAsz/4ji0pkXUzDZaCfxOGQhIROT3We9j19xaTO1JK9nKFpFFMv e0EA+aRPb30Z6q6cDcpBzaGigr/mwHzr0arIetm3IU1JlEKNMV7bzVkYkDbEawZM5FUJ UfedishE8oY1TVItYc64oSCwGY6cL3oo1cWMn7+mgHdskWc+O2zcIp27Pe6J8tNUpw+o 0O1ioTvhrefAEEMecsh1G07lrqfasOoD8SB7iqrbulu9RCOE6USq4HNum0/XgYnrp48T YBPA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66651-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66651-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id l192-20020a6391c9000000b005dc42755289si898114pge.488.2024.02.15.02.45.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 02:45:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-66651-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66651-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66651-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id B88A5290486 for ; Thu, 15 Feb 2024 10:39:01 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id F3494130AF2; Thu, 15 Feb 2024 10:33:08 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1307412FF87 for ; Thu, 15 Feb 2024 10:33:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993186; cv=none; b=T2CPcUosqDn1nhRRZ+KjryazCJeOe0VX0Cr3SvqPc6KzS+Mi+Ee8Ztq/UGBzJ8JfkUtwyOXSAe7zL4oFdAFqESzUZZAjxEgnpV5PinfYcIMK0TWWERgvvi+izgebEmUU2zKnm2JBmBVmFFtFH9wjgmffXRcZcpVm3oj4/YrV9Yc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993186; c=relaxed/simple; bh=s3rVu8Pm88GImVMzq9AF6tvBQ4XPr1Ygmvq1to5V4ys=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dAbkt/7B/W4LV8CHthX3WbiifimbI0A01mTg9uP05xEadFVQp6m6W++zAg7Bzu32/UJdLm/GAxjLGru0PGGq8SQYhkR/jZLoylN2g3DdbK2DSyM41/p106Jsx5AAARJ476AXIicNPNhcEa7hQsESlnCmEJPzlNwHC+hUy68zwbo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4D79D1595; Thu, 15 Feb 2024 02:33:45 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 44B7C3F7B4; Thu, 15 Feb 2024 02:33:01 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 13/18] arm64/mm: Implement new wrprotect_ptes() batch API Date: Thu, 15 Feb 2024 10:32:00 +0000 Message-Id: <20240215103205.2607016-14-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790961474530257372 X-GMAIL-MSGID: 1790961474530257372 Optimize the contpte implementation to fix some of the fork performance regression introduced by the initial contpte commit. Subsequent patches will solve it entirely. During fork(), any private memory in the parent must be write-protected. Previously this was done 1 PTE at a time. But the core-mm supports batched wrprotect via the new wrprotect_ptes() API. So let's implement that API and for fully covered contpte mappings, we no longer need to unfold the contpte. This has 2 benefits: - reduced unfolding, reduces the number of tlbis that must be issued. - The memory remains contpte-mapped ("folded") in the parent, so it continues to benefit from the more efficient use of the TLB after the fork. The optimization to wrprotect a whole contpte block without unfolding is possible thanks to the tightening of the Arm ARM in respect to the definition and behaviour when 'Misprogramming the Contiguous bit'. See section D21194 at https://developer.arm.com/documentation/102105/ja-07/ Tested-by: John Hubbard Signed-off-by: Ryan Roberts Acked-by: Mark Rutland Acked-by: Catalin Marinas --- arch/arm64/include/asm/pgtable.h | 61 ++++++++++++++++++++++++++------ arch/arm64/mm/contpte.c | 38 ++++++++++++++++++++ 2 files changed, 89 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 831099cfc96b..8643227c318b 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -978,16 +978,12 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -/* - * __ptep_set_wrprotect - mark read-only while trasferring potential hardware - * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. - */ -static inline void __ptep_set_wrprotect(struct mm_struct *mm, - unsigned long address, pte_t *ptep) +static inline void ___ptep_set_wrprotect(struct mm_struct *mm, + unsigned long address, pte_t *ptep, + pte_t pte) { - pte_t old_pte, pte; + pte_t old_pte; - pte = __ptep_get(ptep); do { old_pte = pte; pte = pte_wrprotect(pte); @@ -996,6 +992,25 @@ static inline void __ptep_set_wrprotect(struct mm_struct *mm, } while (pte_val(pte) != pte_val(old_pte)); } +/* + * __ptep_set_wrprotect - mark read-only while trasferring potential hardware + * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. + */ +static inline void __ptep_set_wrprotect(struct mm_struct *mm, + unsigned long address, pte_t *ptep) +{ + ___ptep_set_wrprotect(mm, address, ptep, __ptep_get(ptep)); +} + +static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, + pte_t *ptep, unsigned int nr) +{ + unsigned int i; + + for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) + __ptep_set_wrprotect(mm, address, ptep); +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_PMDP_SET_WRPROTECT static inline void pmdp_set_wrprotect(struct mm_struct *mm, @@ -1149,6 +1164,8 @@ extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr); extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); @@ -1268,12 +1285,35 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, return contpte_ptep_clear_flush_young(vma, addr, ptep); } +#define wrprotect_ptes wrprotect_ptes +static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + if (likely(nr == 1)) { + /* + * Optimization: wrprotect_ptes() can only be called for present + * ptes so we only need to check contig bit as condition for + * unfold, and we can remove the contig bit from the pte we read + * to avoid re-reading. This speeds up fork() which is sensitive + * for order-0 folios. Equivalent to contpte_try_unfold(). + */ + pte_t orig_pte = __ptep_get(ptep); + + if (unlikely(pte_cont(orig_pte))) { + __contpte_try_unfold(mm, addr, ptep, orig_pte); + orig_pte = pte_mknoncont(orig_pte); + } + ___ptep_set_wrprotect(mm, addr, ptep, orig_pte); + } else { + contpte_wrprotect_ptes(mm, addr, ptep, nr); + } +} + #define __HAVE_ARCH_PTEP_SET_WRPROTECT static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - __ptep_set_wrprotect(mm, addr, ptep); + wrprotect_ptes(mm, addr, ptep, 1); } #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS @@ -1305,6 +1345,7 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, #define ptep_clear_flush_young __ptep_clear_flush_young #define __HAVE_ARCH_PTEP_SET_WRPROTECT #define ptep_set_wrprotect __ptep_set_wrprotect +#define wrprotect_ptes __wrprotect_ptes #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 6d7f40667fa2..bedb58524535 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -26,6 +26,26 @@ static inline pte_t *contpte_align_down(pte_t *ptep) return PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * CONT_PTES); } +static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + /* + * Unfold any partially covered contpte block at the beginning and end + * of the range. + */ + + if (ptep != contpte_align_down(ptep) || nr < CONT_PTES) + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + + if (ptep + nr != contpte_align_down(ptep + nr)) { + unsigned long last_addr = addr + PAGE_SIZE * (nr - 1); + pte_t *last_ptep = ptep + nr - 1; + + contpte_try_unfold(mm, last_addr, last_ptep, + __ptep_get(last_ptep)); + } +} + static void contpte_convert(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) { @@ -238,6 +258,24 @@ int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, } EXPORT_SYMBOL(contpte_ptep_clear_flush_young); +void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + /* + * If wrprotecting an entire contig range, we can avoid unfolding. Just + * set wrprotect and wait for the later mmu_gather flush to invalidate + * the tlb. Until the flush, the page may or may not be wrprotected. + * After the flush, it is guaranteed wrprotected. If it's a partial + * range though, we must unfold, because we can't have a case where + * CONT_PTE is set but wrprotect applies to a subset of the PTEs; this + * would cause it to continue to be unpredictable after the flush. + */ + + contpte_try_unfold_partial(mm, addr, ptep, nr); + __wrprotect_ptes(mm, addr, ptep, nr); +} +EXPORT_SYMBOL(contpte_wrprotect_ptes); + int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty) From patchwork Thu Feb 15 10:32:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 201350 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:b825:b0:106:860b:bbdd with SMTP id da37csp304840dyb; Thu, 15 Feb 2024 02:39:24 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUjFawCO2sphb/913wGi1LzhnTXmqOgy/jycIRmUDz1GVFUWx4jCoNh6hSiZwnqXemE/F5VXi98HQ0uwlBHcw8wsQp+nA== X-Google-Smtp-Source: AGHT+IHDiZbFZtCT2lqt7JEdJFgMXOPiNV7AQbWRITWCINwyk0Qk1bGNEe5hPvO8oZ1ISzMQLE/5 X-Received: by 2002:a17:906:b354:b0:a3d:8fb3:3ffc with SMTP id cd20-20020a170906b35400b00a3d8fb33ffcmr996031ejb.15.1707993564609; Thu, 15 Feb 2024 02:39:24 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707993564; cv=pass; d=google.com; s=arc-20160816; b=ULjNtwkCpmxTpv8YlvZQEgEHzJZI/mkV3zUZFP6/6q4Hd52Ty3hk7hbJd2zm9O1OB9 47EdmaS8uZEBV8POZ/OQIkcc/usa4JpqQf+P48A1/pqXT55K9g1CuCeHEjGVp0EDQZAN K8joas4HXUCFYjazOgsBr3uIi2fqs6pWFOvGRhuGpWpdUkHYDDKpCLfZRHOqzvLczTpy t/xuSZfio7qYNvLeH39FCoiqvF2DyiPZFwImYlXgzU3XXgKsCs70qRVEpap/grA1XVRG K2GrnmqacL+ncqIGq/t1tQ9fxyXnfohoT/9dg9ELemgkJgS01jjJD8ILii6jeGvlzO+z pLsA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=BCAtce4Ja9jmZhUdX/dylO55pAFHXc3eVtMKGJQHUkc=; fh=Heagowc1O8zVDXTyU76nJjOwDjz9r3Gi+0zmhF4ZELQ=; b=jpkBcyKEJJ1ADKDrDPIdHFcPnG8bmRM7yxkxMxthGA3ydD0vPAwdftOH8x3zunM0j1 O8H1Cb2Ct0POeR+DMeo/2QF240wDCYn/uzwxXXhUAYpiGca/qqJElICEc0pkKUbSNgyT cJcA2IODJVkH79LsrG2tMe725k9eAfghkHJimcvPPztYPk5+tvkx7lgLPlme+2V1vvIQ BCwNon+5jG7U8/rhvsm2ASOzqUDAZ7mBUa81FEebURvFM8ZaOBGf8lmKxYjOdbnwbdCk CTdE81nCcQk2zy1/N28S/9TOPDKZEyubfgXoQI2b6lQGcRisbPjIVXoU2793LIVh7RUd DI7w==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66652-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66652-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id a13-20020a170906468d00b00a3d448a4bc8si493220ejr.625.2024.02.15.02.39.24 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 02:39:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-66652-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66652-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66652-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 055E21F2225A for ; Thu, 15 Feb 2024 10:39:24 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1A60B130AF9; Thu, 15 Feb 2024 10:33:12 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6DBAB130AE8 for ; Thu, 15 Feb 2024 10:33:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993189; cv=none; b=V2nbnBGCt7KLhC/3a2RVe3MKivukTudLFPDUe0NIvwjmjvFayfRgOhNGzEikf1qCgZdqG7xt2B38vrxHYvNIgd20W9rMcHunKoViYxeZWu4vMIeIU5b0A2zBOyPIU6W7xvYnpfvJixlgFO4bFGw/J8rD3/aYH3vkGhmE+cBqshI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993189; c=relaxed/simple; bh=7isDLJO/kisDwKqfva9312wvU+xupRc6iMB919Nr0JM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fFPYkodWR3stX7dgi9NbMv1BUi8pXIGE6Bfa50ekYPKiSmNtp76KcVxcvoFFEAOegMZ3i5/ZvPvs6MSUcABa6MNAEKppT+VVSrh4Kp7JzNW+OfLAcnt3FaGgXQfcaAB9oFZPMMhvh0HM3s6O+2pviuq2+4hoKuwnGmYxg/tXRzE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B0CA71596; Thu, 15 Feb 2024 02:33:48 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A85623F7B4; Thu, 15 Feb 2024 02:33:04 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 14/18] arm64/mm: Implement new [get_and_]clear_full_ptes() batch APIs Date: Thu, 15 Feb 2024 10:32:01 +0000 Message-Id: <20240215103205.2607016-15-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790961059654620677 X-GMAIL-MSGID: 1790961059654620677 Optimize the contpte implementation to fix some of the exit/munmap/dontneed performance regression introduced by the initial contpte commit. Subsequent patches will solve it entirely. During exit(), munmap() or madvise(MADV_DONTNEED), mappings must be cleared. Previously this was done 1 PTE at a time. But the core-mm supports batched clear via the new [get_and_]clear_full_ptes() APIs. So let's implement those APIs and for fully covered contpte mappings, we no longer need to unfold the contpte. This significantly reduces unfolding operations, reducing the number of tlbis that must be issued. Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 67 ++++++++++++++++++++++++++++++++ arch/arm64/mm/contpte.c | 17 ++++++++ 2 files changed, 84 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 8643227c318b..a8f1a35e3086 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -965,6 +965,37 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, return pte; } +static inline void __clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + for (;;) { + __ptep_get_and_clear(mm, addr, ptep); + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + } +} + +static inline pte_t __get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + pte_t pte, tmp_pte; + + pte = __ptep_get_and_clear(mm, addr, ptep); + while (--nr) { + ptep++; + addr += PAGE_SIZE; + tmp_pte = __ptep_get_and_clear(mm, addr, ptep); + if (pte_dirty(tmp_pte)) + pte = pte_mkdirty(pte); + if (pte_young(tmp_pte)) + pte = pte_mkyoung(pte); + } + return pte; +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, @@ -1160,6 +1191,11 @@ extern pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); extern void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); +extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full); +extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full); extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, @@ -1253,6 +1289,35 @@ static inline void pte_clear(struct mm_struct *mm, __pte_clear(mm, addr, ptep); } +#define clear_full_ptes clear_full_ptes +static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + if (likely(nr == 1)) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __clear_full_ptes(mm, addr, ptep, nr, full); + } else { + contpte_clear_full_ptes(mm, addr, ptep, nr, full); + } +} + +#define get_and_clear_full_ptes get_and_clear_full_ptes +static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + pte_t pte; + + if (likely(nr == 1)) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + pte = __get_and_clear_full_ptes(mm, addr, ptep, nr, full); + } else { + pte = contpte_get_and_clear_full_ptes(mm, addr, ptep, nr, full); + } + + return pte; +} + #define __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -1337,6 +1402,8 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, #define set_pte __set_pte #define set_ptes __set_ptes #define pte_clear __pte_clear +#define clear_full_ptes __clear_full_ptes +#define get_and_clear_full_ptes __get_and_clear_full_ptes #define __HAVE_ARCH_PTEP_GET_AND_CLEAR #define ptep_get_and_clear __ptep_get_and_clear #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index bedb58524535..50e0173dc5ee 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -212,6 +212,23 @@ void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, } EXPORT_SYMBOL(contpte_set_ptes); +void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + contpte_try_unfold_partial(mm, addr, ptep, nr); + __clear_full_ptes(mm, addr, ptep, nr, full); +} +EXPORT_SYMBOL(contpte_clear_full_ptes); + +pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + contpte_try_unfold_partial(mm, addr, ptep, nr); + return __get_and_clear_full_ptes(mm, addr, ptep, nr, full); +} +EXPORT_SYMBOL(contpte_get_and_clear_full_ptes); + int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) { From patchwork Thu Feb 15 10:32:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 201365 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:b825:b0:106:860b:bbdd with SMTP id da37csp310288dyb; Thu, 15 Feb 2024 02:52:29 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCUbwz+B3dyhq17qOvz0/8jOjw70LkwnimoBPleR/+DKE9PHs3Wp0JmbvIVh4EBDW8jg+W4+5iq9KWaqyue1zOwIM2vrLg== X-Google-Smtp-Source: AGHT+IElzEQRCsLAljCl02VFTUM52GyfFSoWFFfg3AyI7F9U5WjduhFWfK7yDx8k7tHEHLxHf2WE X-Received: by 2002:a05:6a00:4fc8:b0:6e1:1455:be72 with SMTP id le8-20020a056a004fc800b006e11455be72mr5161580pfb.11.1707994349358; Thu, 15 Feb 2024 02:52:29 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707994349; cv=pass; d=google.com; s=arc-20160816; b=TWNJPmEtG0TForzU4p9+venUk1nFR+gVudWBCjBewI9QEJweZ+wL48nCUhW+dn6/FU Dr0Pmz2rEMHnK6RXijnXdhe4Crz8e6NNsIlIACRTxYMiVTfkaJB4ccTWFeETWCf0fRIi 9+aACwc7oOBhEY17Ho0wK455laD8xkP6b+icTqmVQb0qw37GItlMxLtmx/ZGnaSCMbYd 85BZ/5JO++VmLXSZZKnLsgNvz6WZL4WryILn5EG5i1OCNazi2y4RgpXy4FOk41udpAmk 66RO0FMadujjflAxLYwYQV1OlQIvdS3tq7JjXUOA+Od+sn4zWHvREzoF3nq4J0U+cmOZ uCJw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=BF6nKnbz4CAqHZQZG0uUn3HcEcmnuX1IgW4gkLn6dJk=; fh=Heagowc1O8zVDXTyU76nJjOwDjz9r3Gi+0zmhF4ZELQ=; b=ET9iYlCrE7xfgYKJZianasxUI5fYn6tbcAFgWGt//J28U0IU4876j37qnJ17ZQj74l kyrN0/VIpUDRdQdd6zc9pSOKZNMTHX1/LoshRl6gpRBSQV6v40LhjvTHQB3f7UfgMxsw FkjPB5koCepel+sVLj46LzaD6vh9na/ES6HJgIw9KSAmeC2JA4VGY7dEvNaybr1mMEsw SkojXu0ztKJCAGYIfDeIOiQDg9XIKKs9FQMSw5sKCYRGDQx2LKQy/IL13wPdJHbgOnZH odFOagOUmq4KMQCcQg3winqYImYbvOKYL2rjO70cydzkJgPON50Z/ktkV3t1iyyHMTWU HDVg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66653-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66653-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id m7-20020a056a00080700b006e04f294365si963138pfk.193.2024.02.15.02.52.29 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 02:52:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-66653-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66653-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66653-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id BBDD5290D30 for ; Thu, 15 Feb 2024 10:39:39 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 96DFE130E41; Thu, 15 Feb 2024 10:33:15 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A3057130E58 for ; Thu, 15 Feb 2024 10:33:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993194; cv=none; b=OKTpyrZpZlOAO99qF2GRCEAzP7VNeiF6knbstS0deaD50Pc+rCWbDwLH5CuNq6t9GZDOSnbcJxi2Sbm/LkISdYDdQSzwpGN1kHcXtPcuLocaUtunUfzf8tMYAmJ43oYo0eVe3R7wcfkWY6ZgHFC1xFroVjv/B++Xf5eD4FRrCUc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993194; c=relaxed/simple; bh=4dt5LCeatOElsPUkFZ0pGfvF4+jB8UBy2mT55eRRhoI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WqH799jRnjXZcrsRXPrcyUOnJ1ZNo3nRt5KMdZdbceEV4Cvm3byEHy3BhJ5zk5t9G6xMuiGIAcddtXivCkrggSZ01dUJtcLICLhb1iTXsMfHBbQdRENtizFHSICDMt6JTnF3PxN4vutGCK3BtolfK6nYKeKi6f/ZF7PksPv0i3A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 20A7A15A1; Thu, 15 Feb 2024 02:33:52 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 17B8B3F7B4; Thu, 15 Feb 2024 02:33:07 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 15/18] mm: Add pte_batch_hint() to reduce scanning in folio_pte_batch() Date: Thu, 15 Feb 2024 10:32:02 +0000 Message-Id: <20240215103205.2607016-16-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790961882652171108 X-GMAIL-MSGID: 1790961882652171108 Some architectures (e.g. arm64) can tell from looking at a pte, if some follow-on ptes also map contiguous physical memory with the same pgprot. (for arm64, these are contpte mappings). Take advantage of this knowledge to optimize folio_pte_batch() so that it can skip these ptes when scanning to create a batch. By default, if an arch does not opt-in, folio_pte_batch() returns a compile-time 1, so the changes are optimized out and the behaviour is as before. arm64 will opt-in to providing this hint in the next patch, which will greatly reduce the cost of ptep_get() when scanning a range of contptes. Acked-by: David Hildenbrand Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- include/linux/pgtable.h | 21 +++++++++++++++++++++ mm/memory.c | 19 ++++++++++++------- 2 files changed, 33 insertions(+), 7 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index bc005d84f764..a36cf4e124b0 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -212,6 +212,27 @@ static inline int pmd_dirty(pmd_t pmd) #define arch_flush_lazy_mmu_mode() do {} while (0) #endif +#ifndef pte_batch_hint +/** + * pte_batch_hint - Number of pages that can be added to batch without scanning. + * @ptep: Page table pointer for the entry. + * @pte: Page table entry. + * + * Some architectures know that a set of contiguous ptes all map the same + * contiguous memory with the same permissions. In this case, it can provide a + * hint to aid pte batching without the core code needing to scan every pte. + * + * An architecture implementation may ignore the PTE accessed state. Further, + * the dirty state must apply atomically to all the PTEs described by the hint. + * + * May be overridden by the architecture, else pte_batch_hint is always 1. + */ +static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) +{ + return 1; +} +#endif + #ifndef pte_advance_pfn static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) { diff --git a/mm/memory.c b/mm/memory.c index 3b8e56eb08a3..4dd8e35b593a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -988,16 +988,20 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, { unsigned long folio_end_pfn = folio_pfn(folio) + folio_nr_pages(folio); const pte_t *end_ptep = start_ptep + max_nr; - pte_t expected_pte = __pte_batch_clear_ignored(pte_next_pfn(pte), flags); - pte_t *ptep = start_ptep + 1; + pte_t expected_pte, *ptep; bool writable; + int nr; if (any_writable) *any_writable = false; VM_WARN_ON_FOLIO(!pte_present(pte), folio); - while (ptep != end_ptep) { + nr = pte_batch_hint(start_ptep, pte); + expected_pte = __pte_batch_clear_ignored(pte_advance_pfn(pte, nr), flags); + ptep = start_ptep + nr; + + while (ptep < end_ptep) { pte = ptep_get(ptep); if (any_writable) writable = !!pte_write(pte); @@ -1011,17 +1015,18 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, * corner cases the next PFN might fall into a different * folio. */ - if (pte_pfn(pte) == folio_end_pfn) + if (pte_pfn(pte) >= folio_end_pfn) break; if (any_writable) *any_writable |= writable; - expected_pte = pte_next_pfn(expected_pte); - ptep++; + nr = pte_batch_hint(ptep, pte); + expected_pte = pte_advance_pfn(expected_pte, nr); + ptep += nr; } - return ptep - start_ptep; + return min(ptep - start_ptep, max_nr); } /* From patchwork Thu Feb 15 10:32:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 201351 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:b825:b0:106:860b:bbdd with SMTP id da37csp305077dyb; Thu, 15 Feb 2024 02:39:56 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCW5zOwCW/qWGdpFhJDAHd/0ADIPRnJk+bc+JtamhnVzctcae0WQtdOXwlpo4SXDjD31k2LKjFncYC68NaG5Ed3A4rZPPA== X-Google-Smtp-Source: AGHT+IFNik2kU8NgURw70OrtVajb114dT4Z8Cqa99k8cqMzMa0b0j61wEHhx6T+KbDYdwgWXXd6z X-Received: by 2002:ac2:5197:0:b0:512:851d:82c5 with SMTP id u23-20020ac25197000000b00512851d82c5mr1156278lfi.0.1707993596347; Thu, 15 Feb 2024 02:39:56 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707993596; cv=pass; d=google.com; s=arc-20160816; b=OQYyyl0M0upL6VoF2UUFzfDx3UCMuQTukindHLpq5CzcBnlCEZXM0kHvSAWcIB8jpb zNJ5WgzG6y3QqaesiQ1ljQ8CMtXEUmlsmYSdvJ0FS+0sB4H15n3T2/5v/N4C1r1kbpsF YV1kDbVtgoIC7WE2LShkw7jzesDWWyTtRfQbVDbeDt1RUbbVYNvVgHAEPuEh/lAaxdhm 1jjz5sdzRE29DqGc8XjeAx9xRbxAaN4qX9Y4ugNWMoDPCGDQ9jlf0gq0E8M7r8lDElk3 /n3DUmR25n7zrSVmoIpIITq6HYfKNvyzeBI1EbEsVKZkAXT894HuoJN4iRW9Rens3z58 xMDg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=RBDSb1ynAOyzexiyHHWXWWE26cJncrKBIu2RWSUrpHs=; fh=Heagowc1O8zVDXTyU76nJjOwDjz9r3Gi+0zmhF4ZELQ=; b=jLo3KFUHD+6ma7Ofs7ekSU3/kfDiFI7YpmsL7J/RZ5yi9pdpKMmTIkYpYp/EDUQpBg 8yE1mOc5hTsGEO755jGhT3iyq1gP3KtQK72M81mtVZUiWdImCGDJH0pCk4VLTI7gn/BA OsAIFpLjyxGRE//3vMPoN0mPfLYzGcpFXZ4zSI7QLa9hm0Gi2HjoptXVFHCiib+z6ZjJ tlQ1c1APwGLTs9Zik3WeKtr6cl8+si9PAEQBSh0IIwifY0ZzQrdlyZCqIa0mc6ZtFCjU y+XxCaCmBARLPe1HFbc6bVyHSGFHbWuZdTfA3YCtSOSCX0vUOMbLsYp6H6gmtQ0ZQymM cnGA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66654-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66654-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id gf14-20020a170906e20e00b00a3da6c862e9si192970ejb.928.2024.02.15.02.39.56 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 02:39:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-66654-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66654-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66654-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id DFC9D1F21CF1 for ; Thu, 15 Feb 2024 10:39:55 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C3DBE131747; Thu, 15 Feb 2024 10:33:17 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 074AC130E50 for ; Thu, 15 Feb 2024 10:33:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993196; cv=none; b=No4i58qOCFk/hFPf4CRdE290bQF1VdRMEZ+GmtTKs2qcFGm+4e9hCKMqf0Ksfk33HFnJhDn15ycm1a8QNoJOyPFmtGrV4QIq2OCWGYD8odFFnXatJlRUb6lge9oJacWcFHv0dp2deXjLGtX00s8Rk1/dmUZ82+8s5lUzvrtoJgc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993196; c=relaxed/simple; bh=UQ5MbKtdXY2+2WI/a5CIOKIBnSOSlIvamDyhPiGi7rw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=J/LqXb3rhk4udjzf3/GVZRPY2jHlSlcHqTECQ1rS3mWMQi0+zjabyxjOir9jFJo6dsFGCiyIUFu/fNdfQD3Afh2H35kvSVP2PK8wXIExYsgsliiL68ck/KB+/pfs+wwEmDCYlSbLPGG1LfJ+QIzzi/ZYPtmm/lsStuDbt3koN5o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 871BF15BF; Thu, 15 Feb 2024 02:33:55 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 7B9B93F7B4; Thu, 15 Feb 2024 02:33:11 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 16/18] arm64/mm: Implement pte_batch_hint() Date: Thu, 15 Feb 2024 10:32:03 +0000 Message-Id: <20240215103205.2607016-17-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790961093059339574 X-GMAIL-MSGID: 1790961093059339574 When core code iterates over a range of ptes and calls ptep_get() for each of them, if the range happens to cover contpte mappings, the number of pte reads becomes amplified by a factor of the number of PTEs in a contpte block. This is because for each call to ptep_get(), the implementation must read all of the ptes in the contpte block to which it belongs to gather the access and dirty bits. This causes a hotspot for fork(), as well as operations that unmap memory such as munmap(), exit and madvise(MADV_DONTNEED). Fortunately we can fix this by implementing pte_batch_hint() which allows their iterators to skip getting the contpte tail ptes when gathering the batch of ptes to operate on. This results in the number of PTE reads returning to 1 per pte. Acked-by: Mark Rutland Reviewed-by: David Hildenbrand Tested-by: John Hubbard Signed-off-by: Ryan Roberts Acked-by: Catalin Marinas --- arch/arm64/include/asm/pgtable.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index a8f1a35e3086..d759a20d2929 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1213,6 +1213,15 @@ static inline void contpte_try_unfold(struct mm_struct *mm, unsigned long addr, __contpte_try_unfold(mm, addr, ptep, pte); } +#define pte_batch_hint pte_batch_hint +static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) +{ + if (!pte_valid_cont(pte)) + return 1; + + return CONT_PTES - (((unsigned long)ptep >> 3) & (CONT_PTES - 1)); +} + /* * The below functions constitute the public API that arm64 presents to the * core-mm to manipulate PTE entries within their page tables (or at least this From patchwork Thu Feb 15 10:32:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 201352 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:b825:b0:106:860b:bbdd with SMTP id da37csp305350dyb; Thu, 15 Feb 2024 02:40:32 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWuXyORo9R9OuGe8DDRQx4V2gGDfmJ+4vfqAkOO5/Q0FTv1gIwotllYXJ3WWYhWRyS2t3J/CnyjfwOKOVZzKwGATk3ezQ== X-Google-Smtp-Source: AGHT+IHuC6o5fPzZU2cbwnjjOMWsBXdyap/01wRGIpDg0C1E/dKQh1c1c1nr9y27Vn007WBEZPSS X-Received: by 2002:ac8:1403:0:b0:42d:d026:f1b2 with SMTP id k3-20020ac81403000000b0042dd026f1b2mr746749qtj.14.1707993631982; Thu, 15 Feb 2024 02:40:31 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707993631; cv=pass; d=google.com; s=arc-20160816; b=ldI1YlwSxNP6TfyOKxnzzjpNmX/z404+usxzrPTGypQy78sauUH//wjWn7kDOPJmnX 1Nhmg5vE3H9V981rwSQcdQiU8GYp1nK9nugSty6vrsbcY+mfqmVZJsIrZZXcHvU8odh9 Jk2Ttg+zp450DLHZQ4RG+SUOPU3UwmwmX60gmHyf12hMRyd7B7oAHAZ+jab3YShgNs+l OubuNgXdLwsMWjj0S3IewqZcAJ5kyDuzzP6RVaWBmczoZaicRzH8qA4IH452LoqCzMaP efagPOMtMxLGR5jIb1O8R8Yps5OMGdu7nvlhcIot3BuZI2MCR4eIKEd/cevYZcKUAQuf H9xg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=PAVJHyGMAUnF7AjViDJ+K3AaQu9PJj0R+gebBFLO5LA=; fh=Heagowc1O8zVDXTyU76nJjOwDjz9r3Gi+0zmhF4ZELQ=; b=F2o8mHslSsJq1ahA6UUZVIY9yMDguQEC1ElpA5+PQLJt3u3dsL1ZfJwXD13HDgIi/Z KbTuKb3DO2b8v3gqAiOsUqdtCh09rZxL3exhQ0u8eVbuuU5iwRPifIVB6ehV/EF4tUE8 XK6eWbS7uMFqPUU4orauMF+e/32BmxAs+Vz5oJZbkZlPTLd3ZMlGmKcmL0IIsKaFZEvY 4EHbVocH0L8Ihi9W2kZgMEZ23N4QjLhP8HQRizrM4n3loIC5JmDEOUpNfHSyu4rC/Daa /gFz+w26j3AYBBEm+KHCtFCpQ7IFpXEDzQ0b83ngRbHCpSQ32rWb5p6yWPgtqXDIWCpA 3ZCw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66655-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66655-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id r13-20020ac85c8d000000b0042cee77dd50si266724qta.238.2024.02.15.02.40.31 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 02:40:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-66655-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66655-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66655-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 29E771C21BAE for ; Thu, 15 Feb 2024 10:40:09 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4292A130E50; Thu, 15 Feb 2024 10:33:21 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6FF6384FC0 for ; Thu, 15 Feb 2024 10:33:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993199; cv=none; b=fRouSnQaQBuWEi1EAz0GniIXYifCbss4HCkEAqolYa5j4awccZhG6kil8TuqxlpdlH9uBT0SegX9wzGul4H4ohiVLoXASLbajEVEADBjG2Q7RpLzk1V9SYyhb2AJtpxjSul+50vZL6VOSbZFSWh8r5qJjjIKLm9DhK7YBukA0jE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993199; c=relaxed/simple; bh=BKVaPuYqBaCIkAqRyGQu29xHpMe0/fa3OcqaDOFtyTk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=VkazlLNyb6J5MgMuYGlhwSPYTA1s5KdSNlGPdYRVBKGSyAIIk4spvGECd9rMwx65eiFK1/8QLx0aow7k8YeEVH+ZpeT/BEOaIya5QdsLH5g/5rbTzOAHnLAj4GmCxNCDa+3nG3jtAcxfWf+ihl0BIvkxSjKk478a/jh9P6XmNEM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EBA3215DB; Thu, 15 Feb 2024 02:33:58 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E26CB3F7B4; Thu, 15 Feb 2024 02:33:14 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 17/18] arm64/mm: __always_inline to improve fork() perf Date: Thu, 15 Feb 2024 10:32:04 +0000 Message-Id: <20240215103205.2607016-18-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790961130586373498 X-GMAIL-MSGID: 1790961130586373498 As set_ptes() and wrprotect_ptes() become a bit more complex, the compiler may choose not to inline them. But this is critical for fork() performance. So mark the functions, along with contpte_try_unfold() which is called by them, as __always_inline. This is worth ~1% on the fork() microbenchmark with order-0 folios (the common case). Acked-by: Mark Rutland Signed-off-by: Ryan Roberts Acked-by: Catalin Marinas --- arch/arm64/include/asm/pgtable.h | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index d759a20d2929..8310875133ff 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1206,8 +1206,8 @@ extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); -static inline void contpte_try_unfold(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte) +static __always_inline void contpte_try_unfold(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, pte_t pte) { if (unlikely(pte_valid_cont(pte))) __contpte_try_unfold(mm, addr, ptep, pte); @@ -1278,7 +1278,7 @@ static inline void set_pte(pte_t *ptep, pte_t pte) } #define set_ptes set_ptes -static inline void set_ptes(struct mm_struct *mm, unsigned long addr, +static __always_inline void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr) { pte = pte_mknoncont(pte); @@ -1360,8 +1360,8 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, } #define wrprotect_ptes wrprotect_ptes -static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr) +static __always_inline void wrprotect_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, unsigned int nr) { if (likely(nr == 1)) { /* From patchwork Thu Feb 15 10:32:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 201366 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:b825:b0:106:860b:bbdd with SMTP id da37csp310778dyb; Thu, 15 Feb 2024 02:53:47 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVR7JbUknxnZuQrQtofhh+y1sGsTAtDHc92JM+DUy0QSKskOvqk8mQyPSh7MsiA/zFWmmtRFZIX5tii9pTHeZYQonlKCQ== X-Google-Smtp-Source: AGHT+IFO+Abi5XSBKyI661pk4WO7yOXrTPiIUSPDLjvvyen95GR9esp1QQilGRX0mx2WUS0qwrFz X-Received: by 2002:a17:902:f542:b0:1d5:dac4:5015 with SMTP id h2-20020a170902f54200b001d5dac45015mr1407474plf.11.1707994427203; Thu, 15 Feb 2024 02:53:47 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707994427; cv=pass; d=google.com; s=arc-20160816; b=FB3GqrvXimCY0D9KGlWhhjSrQ4wCZr7n33rtdLG9ZLoEA98c9S6edysouB8SmphlXD K2A8pf9GWrCzFn1VH+3s7gqGs5vRik1GdDxPgBUhW9PowJXbtXBXdALbEJZLQdosYo9w UgGX2tiBlY8eEYhLTqKmgbtcbh5ABiSxP0TxEwh/ltBZ0PBs7VYfXvLIgD1+Tkow/Yky C4ZstOkrRT1wSVEBa9bruy8t/jdKvI7+NjWjB1LPRX0TbSFKq45HwmjUC6wd2InrKa0u 7aKvvuTdef8cMLFYgYwoZobK8EIPBxfJGCIDuS2A05//Maj75gHeXdZ6BxPMbxkjVdlq 5oeQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=Zz5h8gFoLcQ5F5jny0TZcnql6XjfJNjA5U+ixiuRaqk=; fh=Heagowc1O8zVDXTyU76nJjOwDjz9r3Gi+0zmhF4ZELQ=; b=qGR0CbHaWPZeB6dxaiUn6+DKbLq26U/FT9Gh1MHRF6aOziVx+bR5JxKuonsgcf9Gm3 2d4lTXhiLF12To7iX5yG9WUwn4+z3fbSOtGPvgFBnOomXmsPxP1T+Ok4qGNQFV2GQKqT IZSFQVzPsCkc9Hfl4ymZhrA1YUaxw21KeILWOCGVLxtj6U9nknoQhgDyuzZbUNiPGbZo FtfLs2Dz+szwr5L+bskkY3qyhR/HU0UqDeguEhwxERebT/E/6nwW3YMYLzzSpxDzp2rb l2O9ltzzygOxtJXBFmLntHVTS3BDXj0GJo6+yQvj5EiNjl4hMLIbfkbmcaXYqXmZew/h SAjg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66656-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66656-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id me4-20020a170902fc4400b001db40a7519dsi952748plb.590.2024.02.15.02.53.47 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 02:53:47 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-66656-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-66656-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-66656-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 823A929196C for ; Thu, 15 Feb 2024 10:40:24 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 47A84131E34; Thu, 15 Feb 2024 10:33:25 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D443D84FC0 for ; Thu, 15 Feb 2024 10:33:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993203; cv=none; b=NAxXpINkFE8XLkh0A9rkl3MulRwap+POYjGaqRUtXzcVuIpYoKnKDF9w4XFxI8WQqCxTXkdzhon5objtRpCVX+2hPrmNRa/dOyWbGQpZS7DLCHk6WBjemlnGLZCzRblDu6cQH3MosGGzp5vvSKUUOIfhoIJfnNdtQuEqtAODbpg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707993203; c=relaxed/simple; bh=MT4/Y1PHTAJ8ayKJxBrsN+v9y+8DZ+FHAYEo1ocwO1k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=d7mZSN8D0MaADYiZaSMOvm5axAS8YWjwR1E8qrfUmqE/okVwRluRhT51L7eSHtsOf41E/0Md7grONFswchyLYWyXnOy2UATcyDiKB5OyMJw8D2rWAf3MheN4447xPpmDDxDAZjpByRhfsRJOerOBe897P3oXR4TPAtsL0byOtQQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5B7551650; Thu, 15 Feb 2024 02:34:02 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 527B93F7B4; Thu, 15 Feb 2024 02:33:18 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 18/18] arm64/mm: Automatically fold contpte mappings Date: Thu, 15 Feb 2024 10:32:05 +0000 Message-Id: <20240215103205.2607016-19-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790961964670271905 X-GMAIL-MSGID: 1790961964670271905 There are situations where a change to a single PTE could cause the contpte block in which it resides to become foldable (i.e. could be repainted with the contiguous bit). Such situations arise, for example, when user space temporarily changes protections, via mprotect, for individual pages, such can be the case for certain garbage collectors. We would like to detect when such a PTE change occurs. However this can be expensive due to the amount of checking required. Therefore only perform the checks when an indiviual PTE is modified via mprotect (ptep_modify_prot_commit() -> set_pte_at() -> set_ptes(nr=1)) and only when we are setting the final PTE in a contpte-aligned block. Signed-off-by: Ryan Roberts Acked-by: Mark Rutland Acked-by: Catalin Marinas --- arch/arm64/include/asm/pgtable.h | 26 +++++++++++++ arch/arm64/mm/contpte.c | 64 ++++++++++++++++++++++++++++++++ 2 files changed, 90 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 8310875133ff..401087e8a43d 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1185,6 +1185,8 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, * where it is possible and makes sense to do so. The PTE_CONT bit is considered * a private implementation detail of the public ptep API (see below). */ +extern void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); extern void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte); extern pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); @@ -1206,6 +1208,29 @@ extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); +static __always_inline void contpte_try_fold(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, pte_t pte) +{ + /* + * Only bother trying if both the virtual and physical addresses are + * aligned and correspond to the last entry in a contig range. The core + * code mostly modifies ranges from low to high, so this is the likely + * the last modification in the contig range, so a good time to fold. + * We can't fold special mappings, because there is no associated folio. + */ + + const unsigned long contmask = CONT_PTES - 1; + bool valign = ((addr >> PAGE_SHIFT) & contmask) == contmask; + + if (unlikely(valign)) { + bool palign = (pte_pfn(pte) & contmask) == contmask; + + if (unlikely(palign && + pte_valid(pte) && !pte_cont(pte) && !pte_special(pte))) + __contpte_try_fold(mm, addr, ptep, pte); + } +} + static __always_inline void contpte_try_unfold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) { @@ -1286,6 +1311,7 @@ static __always_inline void set_ptes(struct mm_struct *mm, unsigned long addr, if (likely(nr == 1)) { contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); __set_ptes(mm, addr, ptep, pte, 1); + contpte_try_fold(mm, addr, ptep, pte); } else { contpte_set_ptes(mm, addr, ptep, pte, nr); } diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 50e0173dc5ee..16788f07716d 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -73,6 +73,70 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr, __set_ptes(mm, start_addr, start_ptep, pte, CONT_PTES); } +void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + /* + * We have already checked that the virtual and pysical addresses are + * correctly aligned for a contpte mapping in contpte_try_fold() so the + * remaining checks are to ensure that the contpte range is fully + * covered by a single folio, and ensure that all the ptes are valid + * with contiguous PFNs and matching prots. We ignore the state of the + * access and dirty bits for the purpose of deciding if its a contiguous + * range; the folding process will generate a single contpte entry which + * has a single access and dirty bit. Those 2 bits are the logical OR of + * their respective bits in the constituent pte entries. In order to + * ensure the contpte range is covered by a single folio, we must + * recover the folio from the pfn, but special mappings don't have a + * folio backing them. Fortunately contpte_try_fold() already checked + * that the pte is not special - we never try to fold special mappings. + * Note we can't use vm_normal_page() for this since we don't have the + * vma. + */ + + unsigned long folio_start, folio_end; + unsigned long cont_start, cont_end; + pte_t expected_pte, subpte; + struct folio *folio; + struct page *page; + unsigned long pfn; + pte_t *orig_ptep; + pgprot_t prot; + + int i; + + if (!mm_is_user(mm)) + return; + + page = pte_page(pte); + folio = page_folio(page); + folio_start = addr - (page - &folio->page) * PAGE_SIZE; + folio_end = folio_start + folio_nr_pages(folio) * PAGE_SIZE; + cont_start = ALIGN_DOWN(addr, CONT_PTE_SIZE); + cont_end = cont_start + CONT_PTE_SIZE; + + if (folio_start > cont_start || folio_end < cont_end) + return; + + pfn = ALIGN_DOWN(pte_pfn(pte), CONT_PTES); + prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); + expected_pte = pfn_pte(pfn, prot); + orig_ptep = ptep; + ptep = contpte_align_down(ptep); + + for (i = 0; i < CONT_PTES; i++) { + subpte = pte_mkold(pte_mkclean(__ptep_get(ptep))); + if (!pte_same(subpte, expected_pte)) + return; + expected_pte = pte_advance_pfn(expected_pte, 1); + ptep++; + } + + pte = pte_mkcont(pte); + contpte_convert(mm, addr, orig_ptep, pte); +} +EXPORT_SYMBOL(__contpte_try_fold); + void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) {