From patchwork Wed Jan 3 09:14:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 18669 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp4910812dyb; Wed, 3 Jan 2024 01:15:22 -0800 (PST) X-Google-Smtp-Source: AGHT+IGpyY9qJkYZXJUDNbt8yT0BZ82/gMvlVEp3NaysIG7B6nxv6eo6CkoyYHRR4CMFKtMYjqpy X-Received: by 2002:a17:90a:6341:b0:28c:77c9:db96 with SMTP id v1-20020a17090a634100b0028c77c9db96mr4627223pjs.37.1704273321729; Wed, 03 Jan 2024 01:15:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704273321; cv=none; d=google.com; s=arc-20160816; b=uvBEjGiDvDcrLyFxOHm72BXScmLVH2eaeWTAMpQKVsHPoN9as6lW3iytIEzyglJ4I+ K4RcdxJ247ksnL5tK6P/t83aYkmrBqlarA4J8e+UUAROXw/tKRiGaadCuySCYlCqQNYs BhwvLRtf6Q7V3BSqqLd/AAXEW9ZfnU6R+4tvucBOKyRWUy9lF5JDJ1JWP8mMtVnKluy3 negasUv8B1USPtYO8/gEGteUGjChVSJGvWD0HqzHwvpJxb5wSr3sez6VXcxcnsKNHMGP yN34CTQHhuqQjOfabZBbeDQtBjF6rMkaofk1H2MF184hmjDW9MvwseXQVDYci+JWvCQP Ph8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:message-id:date:subject:cc:to :from:dkim-signature; bh=ECdT86ko6CPDQ7PwUTvJTmsh3hF4cTDBKdjOYZE7Ef8=; fh=JvQ3nGflNTIwPBfhSW2OJAIjHOHR+R1SiFkwzYoQoWY=; b=iNo3Aggy3MG2HA7U5H0Uh+z1UtGhqTJGSledC0qtoHDO7V092oGkR/FlihFkIXzdlb DvNYrnZ5+XbapCFKZLNbcMyXjiz4pnE0bQxQGiWDDJ+oarPvmscDvR6fKP2F1K+p6Uyn CqDabUutsYgPyl11gneMD6uLY8mwUz8csaSUHu5UXXXwCeKdPJLfAlFC6YjJyRZ+AW0I g1vZOEwSv2D5W36ouBxHLrOzQVK0diARkT2ohaYCXlnzt4gooF6gcXP/bhc2fSK/SvXp DNyDBk3/S98qR0YZjXqnaMn8FVQ/zbpzIykzC40ZGUVsUlDFU9qJ9OaQxLw1018Q9Dtc 8HGQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=P+yOHYGe; spf=pass (google.com: domain of linux-kernel+bounces-15313-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-15313-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id x2-20020a17090ab00200b0028afcdb8e9bsi879185pjq.162.2024.01.03.01.15.21 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Jan 2024 01:15:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-15313-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=P+yOHYGe; spf=pass (google.com: domain of linux-kernel+bounces-15313-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-15313-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 5BD99285429 for ; Wed, 3 Jan 2024 09:15:14 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id AEA8718AE1; Wed, 3 Jan 2024 09:14:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="P+yOHYGe" X-Original-To: linux-kernel@vger.kernel.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B2CD18628 for ; Wed, 3 Jan 2024 09:14:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1704273286; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ECdT86ko6CPDQ7PwUTvJTmsh3hF4cTDBKdjOYZE7Ef8=; b=P+yOHYGe62HGUf9HPLjbG5PQ9+bPn+xxLg74lXHLowGJOIyRE6RVfsjUX5vcHlUJYz+otX Sdwv8q6CJwQv19KGXEL3P7pEVIdvglCZgJzcxUm1BaqcQ3oDbN3PT/q1asgJTFVBcMiiMx EBsZutiOmJ+0khpKe/kQU1Ynmpl21AE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-656-YJyX-2z5NWeDsYfFUhlklw-1; Wed, 03 Jan 2024 04:14:40 -0500 X-MC-Unique: YJyX-2z5NWeDsYfFUhlklw-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6FE0C1019DE1; Wed, 3 Jan 2024 09:14:39 +0000 (UTC) Received: from x1n.redhat.com (unknown [10.72.116.69]) by smtp.corp.redhat.com (Postfix) with ESMTP id 16526492BF0; Wed, 3 Jan 2024 09:14:26 +0000 (UTC) From: peterx@redhat.com To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: James Houghton , David Hildenbrand , "Kirill A . Shutemov" , Yang Shi , peterx@redhat.com, linux-riscv@lists.infradead.org, Andrew Morton , "Aneesh Kumar K . V" , Rik van Riel , Andrea Arcangeli , Axel Rasmussen , Mike Rapoport , John Hubbard , Vlastimil Babka , Michael Ellerman , Christophe Leroy , Andrew Jones , linuxppc-dev@lists.ozlabs.org, Mike Kravetz , Muchun Song , linux-arm-kernel@lists.infradead.org, Jason Gunthorpe , Christoph Hellwig , Lorenzo Stoakes , Matthew Wilcox Subject: [PATCH v2 00/13] mm/gup: Unify hugetlb, part 2 Date: Wed, 3 Jan 2024 17:14:10 +0800 Message-ID: <20240103091423.400294-1-peterx@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.10 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787060102341803052 X-GMAIL-MSGID: 1787060102341803052 From: Peter Xu v2: - Collect acks - Patch 9: - Use READ_ONCE() to fetch pud entry [James] rfc: https://lore.kernel.org/r/20231116012908.392077-1-peterx@redhat.com v1: https://lore.kernel.org/r/20231219075538.414708-1-peterx@redhat.com This is v2 of the series, based on latest mm-unstalbe (856325d361df). The series removes the hugetlb slow gup path after a previous refactor work [1], so that slow gup now uses the exact same path to process all kinds of memory including hugetlb. For the long term, we may want to remove most, if not all, call sites of huge_pte_offset(). It'll be ideal if that API can be completely dropped from arch hugetlb API. This series is one small step towards merging hugetlb specific codes into generic mm paths. From that POV, this series removes one reference to huge_pte_offset() out of many others. One goal of such a route is that we can reconsider merging hugetlb features like High Granularity Mapping (HGM). It was not accepted in the past because it may add lots of hugetlb specific codes and make the mm code even harder to maintain. With a merged codeset, features like HGM can hopefully share some code with THP, legacy (PMD+) or modern (continuous PTEs). To make it work, the generic slow gup code will need to at least understand hugepd, which is already done like so in fast-gup. Fortunately it seems that's the only major thing I need to teach slow GUP to share the common path for now besides normal huge PxD entries. Non-gup can be more challenging, but that's a question for later. There's one major difference for slow-gup on cont_pte / cont_pmd handling, currently supported on three architectures (aarch64, riscv, ppc). Before the series, slow gup will be able to recognize e.g. cont_pte entries with the help of huge_pte_offset() when hstate is around. Now it's gone but still working, by looking up pgtable entries one by one. It's not ideal, but hopefully this change should not affect yet on major workloads. There's some more information in the commit message of the last patch. If this would be a concern, we can consider teaching slow gup to recognize cont pte/pmd entries, and that should recover the lost performance. But I doubt its necessity for now, so I kept it as simple as it can be. Test Done ========= This v1 went through the normal GUP smoke tests over different memory types on archs (using VM instances): x86_64, aarch64, ppc64le. For aarch64, tested over 64KB cont_pte huge pages. For ppc64le, tested over 16MB hugepd entries (Power8 hash MMU on 4K base page size). Patch layout ============= Patch 1-7: Preparation works, or cleanups in relevant code paths Patch 8-12: Teach slow gup with all kinds of huge entries (pXd, hugepd) Patch 13: Drop hugetlb_follow_page_mask() More information can be found in the commit messages of each patch. Any comment will be welcomed. Thanks. [1] https://lore.kernel.org/all/20230628215310.73782-1-peterx@redhat.com Peter Xu (13): mm/Kconfig: CONFIG_PGTABLE_HAS_HUGE_LEAVES mm/hugetlb: Declare hugetlbfs_pagecache_present() non-static mm: Provide generic pmd_thp_or_huge() mm: Make HPAGE_PXD_* macros even if !THP mm: Introduce vma_pgtable_walk_{begin|end}() mm/gup: Drop folio_fast_pin_allowed() in hugepd processing mm/gup: Refactor record_subpages() to find 1st small page mm/gup: Handle hugetlb for no_page_table() mm/gup: Cache *pudp in follow_pud_mask() mm/gup: Handle huge pud for follow_pud_mask() mm/gup: Handle huge pmd for follow_pmd_mask() mm/gup: Handle hugepd for follow_page() mm/gup: Handle hugetlb in the generic follow_page_mask code include/linux/huge_mm.h | 25 +-- include/linux/hugetlb.h | 16 +- include/linux/mm.h | 3 + include/linux/pgtable.h | 4 + mm/Kconfig | 3 + mm/gup.c | 362 ++++++++++++++++++++++++++++++++-------- mm/huge_memory.c | 133 +-------------- mm/hugetlb.c | 75 +-------- mm/internal.h | 7 +- mm/memory.c | 12 ++ 10 files changed, 342 insertions(+), 298 deletions(-)