From patchwork Mon Jan 22 19:41:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 19279 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2bc4:b0:101:a8e8:374 with SMTP id hx4csp2795622dyb; Mon, 22 Jan 2024 11:42:36 -0800 (PST) X-Google-Smtp-Source: AGHT+IF5KQUxCL3n/nmLRrCMJNsQpykGSbUpzxdo7HON+FoKZ+RKqT8z5uQ04vVDV2KhB0JsQONR X-Received: by 2002:a05:600c:2198:b0:40e:5534:f6fb with SMTP id e24-20020a05600c219800b0040e5534f6fbmr2983392wme.165.1705952555984; Mon, 22 Jan 2024 11:42:35 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1705952555; cv=pass; d=google.com; s=arc-20160816; b=NrszD4mp807UQckOPXtWNCy7LZ9RPRCMCZ913TZZvwewXpqRaeh3um6naU5Sf1m9Kd T01zbgYRlTdC/aNKkXWmuIXfD6ou1G8nN1o600+p+zJI1391SXL+jYSE0ROhLEDGqfdk FUWg6QGSKSdjtsJvmuQw9ZKaa7zQtwU68ZeMVQOl1wIccRrl7x6d+IUhdmwEmNdFc9Ce JlKQ18PXAAAAnpu/GSPZeNut6ZwqoZK1HNX3F4p1RTiEKr3t9FU+g6r5sg+obQNvA60Q q9AUi7YMgb7xi+pXiXD2w6ItniXmD4WOMv0SOmT3augSonSlS1tI3dQYrv21CsKr81BW VVnw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:message-id:date:subject:cc:to :from:dkim-signature; bh=qd9zQc8SwlMxJsbzyQie1UqPu3p8Wk+b2U2cDU4I5Gw=; fh=lZsZCQsH4ia1jFJuQK1Gi3cXLvdnrj+ub8bSs7lwlcw=; b=Tk5ixiIUO1LIOFTqxDLJxgeg1nPxpXuNy7qm4D6md0lUwsh+9/QRqYNR95fR3vjE8u RNd7v/QSIviduNE1UZbuTpYyHOPYro/2KrQkBzCB/SmSy6CuF9n65afO8Kito5jSLxUq P6qvRVNscUzAQCDr6ONWTkjN+9Fr3TcZZBXOuIGPbAbvexLtP5ZsqvVpZfFu/yZ6V/5C nyBS5N3Vig/xTcHgfwoZ5QGdQs2kqwN89xPxkCQ+H9eQJUTDFtypPFF7rF2jkjEuo9K8 ADgsbCdw1Gxr4NQpvBsQielTNzBukFYJSpTe/T9vvQ21Vt5HPitOEHUyx0rIWMn/IiWU NR6Q== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=FuxN9bdY; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-34066-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-34066-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id g27-20020a170906395b00b00a2c0f6b1eeesi11113229eje.449.2024.01.22.11.42.35 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jan 2024 11:42:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-34066-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=FuxN9bdY; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-34066-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-34066-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 94CFE1F24087 for ; Mon, 22 Jan 2024 19:42:35 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B0BCC3FB2A; Mon, 22 Jan 2024 19:42:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="FuxN9bdY" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC2763B790 for ; Mon, 22 Jan 2024 19:42:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705952537; cv=none; b=kN25tlh4xeKsgndFwgKf8k2AJ1raOYbhhJc8+gpomCCQEtKazoacXzdmx8d6Cud18KkH7VoAKRulxFrp52TmpWdK1Cf04tfY6n1pWsfvj4IFJzbmNQ8TWT6oFR54XXmj4RphM+sTIfMT7Pd/vUlu7ik0xec2dYsZgtvlHjpybeU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1705952537; c=relaxed/simple; bh=i8mCWaJqHMbNcbnMf1JJRWyQrq3ljAKp8/8osEqiNeM=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=axR0PuBA+c++U/nhVpyXxDAm5ta7fhw8GodSZGGJehJmJog5pXN2BIYZuyBZnkmWjZsJPsxzFBrOtu816f0OTSX4TeBLZzJG33/jx3HyF008+OcnRpR8qHnHgKfa4FAT6b+DNyL2HmULkzmkQos1iSDgVQ+SRiTiWYKbwnzf4GE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=FuxN9bdY; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1705952534; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=qd9zQc8SwlMxJsbzyQie1UqPu3p8Wk+b2U2cDU4I5Gw=; b=FuxN9bdY2jWl/vRlSJxDNLN9z0eT/KorGOJZDHCXSY6uYAIYoDVoSOyFDlmh9Wr+aMBvfm oRegn1VPCnEyW5/e/sz5ajpWpaLjPJqH1Duhe+Eo8QvRLYacJ3WEB5mlh1zdcLNPMDV++3 XGgHpebp74W9jZDnzTpMXtlzdkOkQWI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-351-x69pe3DCPsqpnwDc2t8HtA-1; Mon, 22 Jan 2024 14:42:11 -0500 X-MC-Unique: x69pe3DCPsqpnwDc2t8HtA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AEEA788D016; Mon, 22 Jan 2024 19:42:09 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.195.114]) by smtp.corp.redhat.com (Postfix) with ESMTP id 267043C2E; Mon, 22 Jan 2024 19:42:02 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Russell King , Catalin Marinas , Will Deacon , Dinh Nguyen , Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , "David S. Miller" , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org Subject: [PATCH v1 00/11] mm/memory: optimize fork() with PTE-mapped THP Date: Mon, 22 Jan 2024 20:41:49 +0100 Message-ID: <20240122194200.381241-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1788820906930040458 X-GMAIL-MSGID: 1788820906930040458 Now that the rmap overhaul[1] is upstream that provides a clean interface for rmap batching, let's implement PTE batching during fork when processing PTE-mapped THPs. This series is partially based on Ryan's previous work[2] to implement cont-pte support on arm64, but its a complete rewrite based on [1] to optimize all architectures independent of any such PTE bits, and to use the new rmap batching functions that simplify the code and prepare for further rmap accounting changes. We collect consecutive PTEs that map consecutive pages of the same large folio, making sure that the other PTE bits are compatible, and (a) adjust the refcount only once per batch, (b) call rmap handling functions only once per batch and (c) perform batch PTE setting/updates. While this series should be beneficial for adding cont-pte support on ARM64[2], it's one of the requirements for maintaining a total mapcount[3] for large folios with minimal added overhead and further changes[4] that build up on top of the total mapcount. Independent of all that, this series results in a speedup during fork with PTE-mapped THP, which is the default with THPs that are smaller than a PMD (for example, 16KiB to 1024KiB mTHPs for anonymous memory[5]). On an Intel Xeon Silver 4210R CPU, fork'ing with 1GiB of PTE-mapped folios of the same size (stddev < 1%) results in the following runtimes for fork() (shorter is better): Folio Size | v6.8-rc1 | New | Change ------------------------------------------ 4KiB | 0.014328 | 0.014265 | 0% 16KiB | 0.014263 | 0.013293 | - 7% 32KiB | 0.014334 | 0.012355 | -14% 64KiB | 0.014046 | 0.011837 | -16% 128KiB | 0.014011 | 0.011536 | -18% 256KiB | 0.013993 | 0.01134 | -19% 512KiB | 0.013983 | 0.011311 | -19% 1024KiB | 0.013986 | 0.011282 | -19% 2048KiB | 0.014305 | 0.011496 | -20% Next up is PTE batching when unmapping, that I'll probably send out based on this series this/next week. Only tested on x86-64. Compile-tested on most other architectures. Will do more testing and double-check the arch changes while this is getting some review. [1] https://lkml.kernel.org/r/20231220224504.646757-1-david@redhat.com [2] https://lkml.kernel.org/r/20231218105100.172635-1-ryan.roberts@arm.com [3] https://lkml.kernel.org/r/20230809083256.699513-1-david@redhat.com [4] https://lkml.kernel.org/r/20231124132626.235350-1-david@redhat.com [5] https://lkml.kernel.org/r/20231207161211.2374093-1-ryan.roberts@arm.com Cc: Andrew Morton Cc: Matthew Wilcox (Oracle) Cc: Ryan Roberts Cc: Russell King Cc: Catalin Marinas Cc: Will Deacon Cc: Dinh Nguyen Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: "Aneesh Kumar K.V" Cc: "Naveen N. Rao" Cc: Paul Walmsley Cc: Palmer Dabbelt Cc: Albert Ou Cc: Alexander Gordeev Cc: Gerald Schaefer Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Christian Borntraeger Cc: Sven Schnelle Cc: "David S. Miller" Cc: linux-arm-kernel@lists.infradead.org Cc: linuxppc-dev@lists.ozlabs.org Cc: linux-riscv@lists.infradead.org Cc: linux-s390@vger.kernel.org Cc: sparclinux@vger.kernel.org David Hildenbrand (11): arm/pgtable: define PFN_PTE_SHIFT on arm and arm64 nios2/pgtable: define PFN_PTE_SHIFT powerpc/pgtable: define PFN_PTE_SHIFT risc: pgtable: define PFN_PTE_SHIFT s390/pgtable: define PFN_PTE_SHIFT sparc/pgtable: define PFN_PTE_SHIFT mm/memory: factor out copying the actual PTE in copy_present_pte() mm/memory: pass PTE to copy_present_pte() mm/memory: optimize fork() with PTE-mapped THP mm/memory: ignore dirty/accessed/soft-dirty bits in folio_pte_batch() mm/memory: ignore writable bit in folio_pte_batch() arch/arm/include/asm/pgtable.h | 2 + arch/arm64/include/asm/pgtable.h | 2 + arch/nios2/include/asm/pgtable.h | 2 + arch/powerpc/include/asm/pgtable.h | 2 + arch/riscv/include/asm/pgtable.h | 2 + arch/s390/include/asm/pgtable.h | 2 + arch/sparc/include/asm/pgtable_64.h | 2 + include/linux/pgtable.h | 17 ++- mm/memory.c | 188 +++++++++++++++++++++------- 9 files changed, 173 insertions(+), 46 deletions(-) base-commit: 6613476e225e090cc9aad49be7fa504e290dd33d