From patchwork Fri Feb 9 22:15:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 199148 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:50ea:b0:106:860b:bbdd with SMTP id r10csp1170092dyd; Fri, 9 Feb 2024 14:38:05 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVwO3GAVlh0gTyPRz2ae7NGUClNMQJWtBtgGoBbiJxYKacppVSWmFLoSDHQVevlON+31gn4XRLuA0ZN4rXYQjC1l+nRGg== X-Google-Smtp-Source: AGHT+IGUYZXP5Kfab6u+vg9U/HWJZThSOc18SwV44c3TGSxEa55CZ4AxMFCmyoiIPt6RhO94xHh1 X-Received: by 2002:a05:6830:1d75:b0:6e0:d6e9:debe with SMTP id l21-20020a0568301d7500b006e0d6e9debemr549233oti.32.1707518285018; Fri, 09 Feb 2024 14:38:05 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707518284; cv=pass; d=google.com; s=arc-20160816; b=s2igbGHCgguN+N6vc+QM8Prny5dhGwZ0kJs6VgO2hXmAjxtP8dIv3Pv7SYAbMPIVH2 1K8+hqN5cVPToLMJWwPCal0igkNrMIhGS55ErzH4TmL1qHp1zNeMUREuxsXrSK4dzi7u /SkShdzy3DoUnEO5NHJlsA9TtQ0yCDh5eiCxWP2ar6Y+qMBBk6mz2cjfPJSm1pqIIDRO N3OSx2RWGYOvvpVzbYsH+B0MRGZCsFGUF9ndbyAWD1dOGLHhMY8kRvrE7XdMKUwWe1Wz Ux3dWYpm1QaOUX3zvm+xA+xK5H2nvgomxJEFEgmfA2O0zyowKPuNbJtjiySOFonXvjOV sWbQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=GUXSwV0Ba8onVJaRwxmEJ/5t7CtvdY0dXq3jtt3HQ88=; fh=qPzgHq9Q4OEAfoYSA2J4nJ2SE8tGGHj+rknwfMc4rRQ=; b=ITF7CPuKzPwynH/tF6BXeUQyNTLbrUMSuwLcF4XrCHq//PXPT1/Dm4rlNVDLE8zBrb dQd1l91hxQ6NCvYiMF5LSsj1PV8LaMmDija/nqAnCdxIs8Kj4YpcNHuMhG9QEjcUttXI tH0BpKeEGHLvRV8m8pzVhHyAgVqIwoQdgS6yniPujhUJaVP1jcsrm22XwLR3ZpdcfbR1 RYoZ4ykAJZTm1KspjGoWLaUVyGydlkS2LtBpu5slF4p0mdB6UrCUADkXGhs44xmHABaR bg8uJicV9WFfUD0LXQfNPVylG++5e7QUVzSfzi5tgHmjBHTROEFunUJ32HFBQlOeMzv3 1w2Q==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ERcXIKFB; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60073-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60073-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Forwarded-Encrypted: i=2; AJvYcCXx61O1hziteq+gXvRpmZMps+WS4bufkRhQmjFdqylfYxDx1x3qVSsDLGr3FS2UZyWrj0L7M3Zs7N6qPXW/Q/cZ7ISjLg== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id k125-20020a632483000000b005dc4f115f2csi2431002pgk.496.2024.02.09.14.38.04 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 14:38:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-60073-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ERcXIKFB; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60073-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60073-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id E5C6CB2561A for ; Fri, 9 Feb 2024 22:15:59 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0EB65374F1; Fri, 9 Feb 2024 22:15:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ERcXIKFB" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 701273613E for ; Fri, 9 Feb 2024 22:15:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516929; cv=none; b=XCwIHGBJiE4M/2+fuC0JGXdiqwdCcGuDbRFCbmQ0tkfYESUCihcIHhDxsLaIAwJs4vBIWIjMlM+B/ucZJLk3/rUbU/x8yg+kKbh0IKr0vYpf6Zyw6oE6c6JNzA+eby9tazpjH8jk9nP0nCWilRWHrFOOoO2SJKn/jLpt9cVz8HY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516929; c=relaxed/simple; bh=TnUCOm2JT+0JTJyQU5SkjDNAWwULzDd20SY3FqhSdfo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mUQpbGCtKbfcbM5GHG+YYgW2R6wnIpNyMe6gBBRqtZAKTse+YWptqonKqYJX4z98bD4pKqVEGCvd4iKTk6k0mkSNNuhdMXwpZuqEwuVRsuhNser54BVZ86MumGzJmDa9WTxrypGdzu+gQsIxs/bRhhImNqBCArWpBOErG9Pvx2o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=ERcXIKFB; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1707516926; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GUXSwV0Ba8onVJaRwxmEJ/5t7CtvdY0dXq3jtt3HQ88=; b=ERcXIKFB2dbeOOCA8ZvA+Kv2LwAlx6Qh74YqwX+mAKOnUARoHIPz9TSrRinP0odavoOAIt +TB5wKYyf81UIBpo/i/AO+HSAycTqfuS+FI9uHuRhDnsZkzHOASCEBnqhbVG4eYFo45IP7 At/Wz4iMGXVomtAxX14cB4hId/MbxW0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-42-CTYzAu7yOQSNMpmHOZXTyA-1; Fri, 09 Feb 2024 17:15:21 -0500 X-MC-Unique: CTYzAu7yOQSNMpmHOZXTyA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 85BC185A589; Fri, 9 Feb 2024 22:15:20 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.194.59]) by smtp.corp.redhat.com (Postfix) with ESMTP id 74DCF1C14B04; Fri, 9 Feb 2024 22:15:16 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Yin Fengwei , Michal Hocko , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v2 01/10] mm/memory: factor out zapping of present pte into zap_present_pte() Date: Fri, 9 Feb 2024 23:15:00 +0100 Message-ID: <20240209221509.585251-2-david@redhat.com> In-Reply-To: <20240209221509.585251-1-david@redhat.com> References: <20240209221509.585251-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790462693072376814 X-GMAIL-MSGID: 1790462693072376814 Let's prepare for further changes by factoring out processing of present PTEs. Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- mm/memory.c | 94 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 53 insertions(+), 41 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 7c3ca41a7610..5b0dc33133a6 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1532,13 +1532,61 @@ zap_install_uffd_wp_if_needed(struct vm_area_struct *vma, pte_install_uffd_wp_if_needed(vma, addr, pte, pteval); } +static inline void zap_present_pte(struct mmu_gather *tlb, + struct vm_area_struct *vma, pte_t *pte, pte_t ptent, + unsigned long addr, struct zap_details *details, + int *rss, bool *force_flush, bool *force_break) +{ + struct mm_struct *mm = tlb->mm; + struct folio *folio = NULL; + bool delay_rmap = false; + struct page *page; + + page = vm_normal_page(vma, addr, ptent); + if (page) + folio = page_folio(page); + + if (unlikely(!should_zap_folio(details, folio))) + return; + ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + arch_check_zapped_pte(vma, ptent); + tlb_remove_tlb_entry(tlb, pte, addr); + zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); + if (unlikely(!page)) { + ksm_might_unmap_zero_page(mm, ptent); + return; + } + + if (!folio_test_anon(folio)) { + if (pte_dirty(ptent)) { + folio_mark_dirty(folio); + if (tlb_delay_rmap(tlb)) { + delay_rmap = true; + *force_flush = true; + } + } + if (pte_young(ptent) && likely(vma_has_recency(vma))) + folio_mark_accessed(folio); + } + rss[mm_counter(folio)]--; + if (!delay_rmap) { + folio_remove_rmap_pte(folio, page, vma); + if (unlikely(page_mapcount(page) < 0)) + print_bad_pte(vma, addr, ptent, page); + } + if (unlikely(__tlb_remove_page(tlb, page, delay_rmap))) { + *force_flush = true; + *force_break = true; + } +} + static unsigned long zap_pte_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, struct zap_details *details) { + bool force_flush = false, force_break = false; struct mm_struct *mm = tlb->mm; - int force_flush = 0; int rss[NR_MM_COUNTERS]; spinlock_t *ptl; pte_t *start_pte; @@ -1555,7 +1603,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent = ptep_get(pte); - struct folio *folio = NULL; + struct folio *folio; struct page *page; if (pte_none(ptent)) @@ -1565,45 +1613,9 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, break; if (pte_present(ptent)) { - unsigned int delay_rmap; - - page = vm_normal_page(vma, addr, ptent); - if (page) - folio = page_folio(page); - - if (unlikely(!should_zap_folio(details, folio))) - continue; - ptent = ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm); - arch_check_zapped_pte(vma, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); - zap_install_uffd_wp_if_needed(vma, addr, pte, details, - ptent); - if (unlikely(!page)) { - ksm_might_unmap_zero_page(mm, ptent); - continue; - } - - delay_rmap = 0; - if (!folio_test_anon(folio)) { - if (pte_dirty(ptent)) { - folio_mark_dirty(folio); - if (tlb_delay_rmap(tlb)) { - delay_rmap = 1; - force_flush = 1; - } - } - if (pte_young(ptent) && likely(vma_has_recency(vma))) - folio_mark_accessed(folio); - } - rss[mm_counter(folio)]--; - if (!delay_rmap) { - folio_remove_rmap_pte(folio, page, vma); - if (unlikely(page_mapcount(page) < 0)) - print_bad_pte(vma, addr, ptent, page); - } - if (unlikely(__tlb_remove_page(tlb, page, delay_rmap))) { - force_flush = 1; + zap_present_pte(tlb, vma, pte, ptent, addr, details, + rss, &force_flush, &force_break); + if (unlikely(force_break)) { addr += PAGE_SIZE; break; } From patchwork Fri Feb 9 22:15:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 199127 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:50ea:b0:106:860b:bbdd with SMTP id r10csp1161673dyd; Fri, 9 Feb 2024 14:16:53 -0800 (PST) X-Google-Smtp-Source: AGHT+IEt9z7oyzyTKApWY9SBx+MdvgGsfW0AdyJ/fZolbDkdaMN8YYvKJPLuyUDCSqUNb6MdEZqg X-Received: by 2002:a05:6a20:3554:b0:19e:b95f:123c with SMTP id f20-20020a056a20355400b0019eb95f123cmr402011pze.49.1707517013235; Fri, 09 Feb 2024 14:16:53 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707517013; cv=pass; d=google.com; s=arc-20160816; b=UnJPxcoyOuNTZXs+paQLOyZwFlB/8Z7vsACAhqRtP+QhrNhRzAo/Y9hely8rJh1smX o0ykDNntAZ3x+qNDrXiIhCBzDqKGXLjfjlr2CUcO80KeCLYkVZXmZXfK+b4wQuOtOoHj s7Y8SKyCTvOSyPVanmP6fR3g0cz8Xey+STMSqFzoshZem2tSqw4CAA0JvgXyX+AteC4P md9iKdn9TLzvvPJZeQZz8lCX2G0QPiFS3kd2LhukJTDWSCeTNKTPjblCP/ra+xUI14mB B6s3110Mnv/92i3OBN8tpf03yneZjeXyL9zXspOH0dbOP27PjRymRw3Ws5tOGUaht/mg X+EA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=fqQsnr9EFuS72Lg9egurC5IFPM5MYMM8dNEJpcWm/gk=; fh=YS+ziU04MLyM+43Oq135+BLTx/Ri8LQlGHYEXskxsX4=; b=O4QhLI/yBWe7PRL1u52YFbyrSJFCY++qT+4WfIEniAR239+AdstY6vplsgHj16tXqp HDBsg7NE1lxMCNvdhbcb1t5A5ItX/jABbb5RaMe1ispA0JIVVY/HwZbROh1hfH4/oJUX zeBhT4IXPsyJL0FgWg6wt5y3iRyAzJGU7kPO/MdSp7skP6UmSIdWDfM+w+yZ9oteNUdZ wQZWqvK04oWcpqyjGYs4MpmRCd2qIyo9wbvSZYrdHdWIDOMpbvp9ERgmUWDF3Qidmqen c3UcQ0M6a8nTEQLNeeZ11dVUampJ1yuO0qN+22y7EZbm8qYByc5GUp4Kk+uYxHDcCTFe IeEg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZZFnRkDF; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60074-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60074-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Forwarded-Encrypted: i=2; AJvYcCWD+IT6+Cpxbo3PkqqDNqallotIwuVmGlgYVLypGANQaTRHwEkwKyDbBnly6tfTrSOg9jdMuOrfVA+rVqNif8g3OCrO8g== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id r201-20020a632bd2000000b005dc0d6a780bsi2372435pgr.402.2024.02.09.14.16.53 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 14:16:53 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-60074-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZZFnRkDF; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60074-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60074-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id A483928962C for ; Fri, 9 Feb 2024 22:16:14 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2832B21353; Fri, 9 Feb 2024 22:15:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ZZFnRkDF" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0197E18053 for ; Fri, 9 Feb 2024 22:15:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516936; cv=none; b=YZY+fvGh04Q/yf0dBlXMYBmvuq3ijbPvsnnG0eFafYrJcmXn03wOEoVZt9S2OJXZzISKFM3U6Ue2YKGzj7Yrxa0LTF7R70zIlC4WEkfIZhc7+h1UUcWjutyyNIHpyeRGFW3I3vPr8BXI2tTHjfoq2IKH3zfK7XDdS91wsYZtPFE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516936; c=relaxed/simple; bh=iKdLPd0l+BlZy4NkWR4PBv4pdjA5DJYrKJe9pxJlmq0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EkqSKKUTX2f+0DjRQcd9nCetGOO90QHZZ7Nh6AG/MeVKT4+27L5MzSH2iZT5m7Q2055+fQBhzUqabtnRQh9+325Za0hs8TycLiSOMUoy1JFkGY1w8eHSx8M+4YwQF43CZ67TwHtP++hMeBRmpJMKJmGzRmSRp4AksDT6AdfKdyU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=ZZFnRkDF; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1707516931; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fqQsnr9EFuS72Lg9egurC5IFPM5MYMM8dNEJpcWm/gk=; b=ZZFnRkDFls4A5W0/ouqSf2KaDCggt3g0BpyfEFkFhAGJCr3yQlPzbHQg6zjJzZepueJ1vc WEbK55uP+TzbH8mTXKBTVi/yMVu7Vb7GjHxlva8fkLotitLPAvMwgsRKANqnqD8FHODuu/ lssuvjtWqLIjMY3mC5DCJhmgzDZ71II= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-659-vWqlBEopN_iaGbQkhty9OQ-1; Fri, 09 Feb 2024 17:15:28 -0500 X-MC-Unique: vWqlBEopN_iaGbQkhty9OQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E6404811E81; Fri, 9 Feb 2024 22:15:26 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.194.59]) by smtp.corp.redhat.com (Postfix) with ESMTP id F06291C14B04; Fri, 9 Feb 2024 22:15:20 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Yin Fengwei , Michal Hocko , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v2 02/10] mm/memory: handle !page case in zap_present_pte() separately Date: Fri, 9 Feb 2024 23:15:01 +0100 Message-ID: <20240209221509.585251-3-david@redhat.com> In-Reply-To: <20240209221509.585251-1-david@redhat.com> References: <20240209221509.585251-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790461359545648567 X-GMAIL-MSGID: 1790461359545648567 We don't need uptodate accessed/dirty bits, so in theory we could replace ptep_get_and_clear_full() by an optimized ptep_clear_full() function. Let's rely on the provided pte. Further, there is no scenario where we would have to insert uffd-wp markers when zapping something that is not a normal page (i.e., zeropage). Add a sanity check to make sure this remains true. should_zap_folio() no longer has to handle NULL pointers. This change replaces 2/3 "!page/!folio" checks by a single "!page" one. Note that arch_check_zapped_pte() on x86-64 checks the HW-dirty bit to detect shadow stack entries. But for shadow stack entries, the HW dirty bit (in combination with non-writable PTEs) is set by software. So for the arch_check_zapped_pte() check, we don't have to sync against HW setting the HW dirty bit concurrently, it is always set. Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand --- mm/memory.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 5b0dc33133a6..4da6923709b2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1497,10 +1497,6 @@ static inline bool should_zap_folio(struct zap_details *details, if (should_zap_cows(details)) return true; - /* E.g. the caller passes NULL for the case of a zero folio */ - if (!folio) - return true; - /* Otherwise we should only zap non-anon folios */ return !folio_test_anon(folio); } @@ -1538,24 +1534,28 @@ static inline void zap_present_pte(struct mmu_gather *tlb, int *rss, bool *force_flush, bool *force_break) { struct mm_struct *mm = tlb->mm; - struct folio *folio = NULL; bool delay_rmap = false; + struct folio *folio; struct page *page; page = vm_normal_page(vma, addr, ptent); - if (page) - folio = page_folio(page); + if (!page) { + /* We don't need up-to-date accessed/dirty bits. */ + ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + arch_check_zapped_pte(vma, ptent); + tlb_remove_tlb_entry(tlb, pte, addr); + VM_WARN_ON_ONCE(userfaultfd_wp(vma)); + ksm_might_unmap_zero_page(mm, ptent); + return; + } + folio = page_folio(page); if (unlikely(!should_zap_folio(details, folio))) return; ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); arch_check_zapped_pte(vma, ptent); tlb_remove_tlb_entry(tlb, pte, addr); zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); - if (unlikely(!page)) { - ksm_might_unmap_zero_page(mm, ptent); - return; - } if (!folio_test_anon(folio)) { if (pte_dirty(ptent)) { From patchwork Fri Feb 9 22:15:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 199129 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:50ea:b0:106:860b:bbdd with SMTP id r10csp1161731dyd; Fri, 9 Feb 2024 14:17:02 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWB0tnFn1DZIIx0zhy6T1YrLfylJ1qLsN47eGeEUO07/Qhoc1JbT+JaTeZiXE9v3He9VuygCDciCGHRsYlfrakl0wefRw== X-Google-Smtp-Source: AGHT+IEiIeaqBMWXH6Q+5jtwqogUUKVETOVSvOF7+gDRgdmjzL0D0wRRSOVdlvZkcJCkN7t7K4EK X-Received: by 2002:a92:c60c:0:b0:363:d7c0:43f7 with SMTP id p12-20020a92c60c000000b00363d7c043f7mr722061ilm.5.1707517021914; Fri, 09 Feb 2024 14:17:01 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707517021; cv=pass; d=google.com; s=arc-20160816; b=dgCFZaVqCva4CA6vcqzZvFtwSWBp6LxRekzn6y5Mhe/ixEDPKimVGC6z4xe3X1tY9i 3/0O5IlHTbToYD1p6phIV9LSwHlH5LWx+fNuayiKX46/KHz+K/KYgbJZvNX36eBgzuoe Sv6oAQwtNp4tJymOAGbvXzgUSg2ack5IEbrBw6WbxCyFzoaBR2xQoOfn89ZCUtHcfLXJ /v2MJhZhnovJ6Ek35JsGnPW1UMKaqNgF2OZ9klAhSu3ymN0tyhqPinsIXmbsMLwajRuL uaVMEyPdkYuThbE1EBlVvypFC3kh8Kk3PpjKh4P4cX0Bo30jk1X6ViVUTh+el6amzSm0 ptBA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=M/Ux9HRgPwmdR3zuVPtEfBlFP5wVpYCc/8Whwmc2o7c=; fh=PqKNvp2m5wSZOdtokSwj6j13ywxn00TYevIpug0r6W0=; b=sVcqLL4ruwkqdcBPrt+qogo6jDjLLOXF3IKoob5+6+/YpqVvySvyAaCeUbZL0dgTb5 wZ36Q2/5qklmgb42mlpFom+7iChyJm5mIOao2Sli81AVTB34hQSLeW2uJS2QpWikMS8B P3Si1P+cH+13+Tkkw+Apxg+eL7vaUItLcZsca/gO/1ntBJ+zkIaiIKarMHgbeBCz7gHx hIRFrmMTy28rMkEErQy3fF3J/iOrIIy4CASoUXsQ305rnkSr3OVJsdHQD13avYRbDcxw oi0H5Z5WRjO/RTRuc96ndEUGhOylI4NMNOLqJyW7Fh9wXVD5t4r+L1uqDrHXj2MrMhgB ZjyA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OqWnVdxW; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60075-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60075-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Forwarded-Encrypted: i=2; AJvYcCV0jQn9ls6RHIGUGRS3+RHLV03kutpXIWsvPGXBgkIpTPT2RwSQ3M027I/fIMOSKhnocnM1U9vsJ8Pe+mw3ZhK+S8ii+g== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id r201-20020a632bd2000000b005dc0d6a780bsi2372435pgr.402.2024.02.09.14.17.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 14:17:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-60075-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OqWnVdxW; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60075-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60075-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 315DC28564A for ; Fri, 9 Feb 2024 22:16:30 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9AE372E63C; Fri, 9 Feb 2024 22:15:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="OqWnVdxW" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 548EF381CF for ; Fri, 9 Feb 2024 22:15:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516939; cv=none; b=avXIo2tQoDZgaDrE8Coss+K+zxoCg+d9FONBId+YAHvTeysvEM/TA/iJLZ9uZBtVI8X3gFrR/W8bW6Y6GPz9BLL9BWSD8zcTPAukfN952ZQ4efx/qhHsswErRgoIhP4FiNzqw57kjlvQGTi3C4owAxbD4pug8PZxKVOU+dQzhxI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516939; c=relaxed/simple; bh=gJjdcDN9SIZ57EMSsnxvz4s6wGh9ngGev4G0JhDn5Lw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UiEODUYFR5E2Yh+27ZswlcVMZazAn97r6iWwB2nPhhGwxgHr1dN+jdk1yf0IzKZXexQ73TPSYkeleP7ATQzkpL9GDNEMLGhAb4xg9KRhdMDgVkBsSWJIfHF3Eje3firyUYVmww8rmNR7n2xxgH4C8b0V7uN0SXvC8QJv3BqKYm8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=OqWnVdxW; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1707516937; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=M/Ux9HRgPwmdR3zuVPtEfBlFP5wVpYCc/8Whwmc2o7c=; b=OqWnVdxWb+fU4UNVKI13tV0Q3mtH+bxe6CsHFZlepgkzWo5oph2o3FZb3lx17JXwoL0qTZ DNctaMHmzjd4k2ig/ncnSfsyK0/4f/cAIp3xXnCekLSua/v9wSarciEa7ChOE7mVUrY0Y9 +pU7vKPuD4OCFZFnhtwfOe34XPXApRA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-639-ojxWH6gHP7CoaMSJUSv5fA-1; Fri, 09 Feb 2024 17:15:32 -0500 X-MC-Unique: ojxWH6gHP7CoaMSJUSv5fA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 85BE9185A780; Fri, 9 Feb 2024 22:15:31 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.194.59]) by smtp.corp.redhat.com (Postfix) with ESMTP id 512C21C14B04; Fri, 9 Feb 2024 22:15:27 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Yin Fengwei , Michal Hocko , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v2 03/10] mm/memory: further separate anon and pagecache folio handling in zap_present_pte() Date: Fri, 9 Feb 2024 23:15:02 +0100 Message-ID: <20240209221509.585251-4-david@redhat.com> In-Reply-To: <20240209221509.585251-1-david@redhat.com> References: <20240209221509.585251-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790461368812943825 X-GMAIL-MSGID: 1790461368812943825 We don't need up-to-date accessed-dirty information for anon folios and can simply work with the ptent we already have. Also, we know the RSS counter we want to update. We can safely move arch_check_zapped_pte() + tlb_remove_tlb_entry() + zap_install_uffd_wp_if_needed() after updating the folio and RSS. While at it, only call zap_install_uffd_wp_if_needed() if there is even any chance that pte_install_uffd_wp_if_needed() would do *something*. That is, just don't bother if uffd-wp does not apply. Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand --- mm/memory.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 4da6923709b2..7a3ebb6e5909 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1552,12 +1552,9 @@ static inline void zap_present_pte(struct mmu_gather *tlb, folio = page_folio(page); if (unlikely(!should_zap_folio(details, folio))) return; - ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); - arch_check_zapped_pte(vma, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); - zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); if (!folio_test_anon(folio)) { + ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); if (pte_dirty(ptent)) { folio_mark_dirty(folio); if (tlb_delay_rmap(tlb)) { @@ -1567,8 +1564,17 @@ static inline void zap_present_pte(struct mmu_gather *tlb, } if (pte_young(ptent) && likely(vma_has_recency(vma))) folio_mark_accessed(folio); + rss[mm_counter(folio)]--; + } else { + /* We don't need up-to-date accessed/dirty bits. */ + ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + rss[MM_ANONPAGES]--; } - rss[mm_counter(folio)]--; + arch_check_zapped_pte(vma, ptent); + tlb_remove_tlb_entry(tlb, pte, addr); + if (unlikely(userfaultfd_pte_wp(vma, ptent))) + zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); + if (!delay_rmap) { folio_remove_rmap_pte(folio, page, vma); if (unlikely(page_mapcount(page) < 0)) From patchwork Fri Feb 9 22:15:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 199128 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:50ea:b0:106:860b:bbdd with SMTP id r10csp1161694dyd; Fri, 9 Feb 2024 14:16:55 -0800 (PST) X-Google-Smtp-Source: AGHT+IGV0b7Pt3xbZ7iShQxlTMOCvHdEe0+7o1a24ue+3xKRm/jDSIdyq7thSnKHtIspHZMV10Id X-Received: by 2002:a9d:7646:0:b0:6e2:ce81:7b7c with SMTP id o6-20020a9d7646000000b006e2ce817b7cmr386283otl.22.1707517015057; Fri, 09 Feb 2024 14:16:55 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707517015; cv=pass; d=google.com; s=arc-20160816; b=aJg/xpNoNtk0+lEVC/FB4R5ePESajaqNdwW0eo9rvNEs4nS/Cjz8fGRCXwIRU+cFpt 6sV5cOjS4PWlRkgwgNafcBNazJ35QJvTDks5ZJRUzATF9+eGiFTXeh9Ps3XYH/xJNGbu OzOtbtYG5ZzH74kLgIYJXFReHgEMJ44R7RodNKP4t/oMtEETKuXM96qVqEY6/gGTMsEB pl7ZD71z4JhOl8mEp51lUfzWV2f0yutDyq38IWwsxz/AIbiW6NXgGfIlpqObYqImXJyS RHzlnBs2DiIZovq5bjPv54wlVBh4P3HBm76CaHLN/jmyIjbYm89AIFCI9QRo8HgJhUU+ ec6g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=SyWghWuBP6GkqnhxcyLXYq2ssX4b7HML4Pf9katCZ5k=; fh=rpQJlzjMIw3y3iEOz+eaJqKq1F01j6S+GWeTLa00+9A=; b=O3X033k8jH1RW8kwIrA4i7fig/WrNCMxOJKWXSq0r3MPrutXzV+Swi11N1bIvIltP5 1I5pR1bpwze6kafL74W5ZD5VgLW+KkGuNPrp2fEIGUeCW57KZ2mxqo56oUVfpSvvWkgx EA0SkApb8yehdAAyHttkC9+DOjrG4bYoe57++qulROwy5K3B9GjPAlqaP80AGwqqDk1X 92v3RfIMFnkiSXFrpUU93/soarnXX7dohdd/c+DgSXdo4QZGRMnEBgUqBmVp9xGwX9un gOhB0LX2MUcfYUjhqwpURzduQpcxEVtHGbSsFtMq9QzkvjVnYk3/aONmDNd+tE/fphOR No7g==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Nuog7vqS; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60076-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60076-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Forwarded-Encrypted: i=2; AJvYcCWXGT1HHP3uVi+l+K6kGJSQGLVUhnOA1ej2PPGdw1JN8tQTEnnT03pb/qzxnBFGcAIzh3gnic4QjFwTgYPr1RJbCDqUsg== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id f10-20020ac87f0a000000b0042c30fc6ac5si2941508qtk.222.2024.02.09.14.16.54 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 14:16:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-60076-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Nuog7vqS; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60076-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60076-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id BE6001C212D5 for ; Fri, 9 Feb 2024 22:16:54 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0E5B538FAD; Fri, 9 Feb 2024 22:15:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Nuog7vqS" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 89B8633CD5 for ; Fri, 9 Feb 2024 22:15:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516947; cv=none; b=hrjOZkPz34sjgmisnFCWPH2sn74W1XnIzYb6AEiPYKS1083XnvCAMiQFEvDomXshT+QTvZlX1MdtNH6Z8Ilubkzm2wcNKkBf6O/q3NnMA1B4GgmjgJ5HoQ3m1ATjtGiOuaYYaQRfagIesHmumftstbspXu4gzqR6G152WMmdCFw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516947; c=relaxed/simple; bh=X688LQ+8vv1x+HYbDq06JeZc4jgkCPC+TN2xr2DyFa8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FPMpkbeW52yC4rhv+sNd0aPrWrv9Lmza7dgn5nX1QrMG3g7l6IPyqVJ4aACd0+ctskv7Qh3X2BhEpyF46Nbhy0Y+vpJGowSFR2lOosxo8mw2kVXUMuyR0cgOKey77Oxrcp/VuQzx00L73j2Dj6XCzidJkr5M5DTRmZGpPOjhY+8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Nuog7vqS; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1707516944; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SyWghWuBP6GkqnhxcyLXYq2ssX4b7HML4Pf9katCZ5k=; b=Nuog7vqSEHMKunx6ebi6mA/fs8mk32+yst6bx6FGWBgDx8dkS/MPWE5NBogsU9RsFMK7nl 554AWkyLBKp3A1/O9s/1a04d/l41S6fzvmYJkvE4xQupnZFR5w5izc0nlZbKFnR7JEpUGt gIj8/HRkcam/JB5RtFIOkIbn48u1L4E= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-689-umYgGBJnMqW48D3blg8DZg-1; Fri, 09 Feb 2024 17:15:38 -0500 X-MC-Unique: umYgGBJnMqW48D3blg8DZg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id ED4963812023; Fri, 9 Feb 2024 22:15:36 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.194.59]) by smtp.corp.redhat.com (Postfix) with ESMTP id E569E1C14B04; Fri, 9 Feb 2024 22:15:31 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Yin Fengwei , Michal Hocko , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v2 04/10] mm/memory: factor out zapping folio pte into zap_present_folio_pte() Date: Fri, 9 Feb 2024 23:15:03 +0100 Message-ID: <20240209221509.585251-5-david@redhat.com> In-Reply-To: <20240209221509.585251-1-david@redhat.com> References: <20240209221509.585251-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790461361197558614 X-GMAIL-MSGID: 1790461361197558614 Let's prepare for further changes by factoring it out into a separate function. Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand --- mm/memory.c | 53 ++++++++++++++++++++++++++++++++--------------------- 1 file changed, 32 insertions(+), 21 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 7a3ebb6e5909..a3efc4da258a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1528,30 +1528,14 @@ zap_install_uffd_wp_if_needed(struct vm_area_struct *vma, pte_install_uffd_wp_if_needed(vma, addr, pte, pteval); } -static inline void zap_present_pte(struct mmu_gather *tlb, - struct vm_area_struct *vma, pte_t *pte, pte_t ptent, - unsigned long addr, struct zap_details *details, - int *rss, bool *force_flush, bool *force_break) +static inline void zap_present_folio_pte(struct mmu_gather *tlb, + struct vm_area_struct *vma, struct folio *folio, + struct page *page, pte_t *pte, pte_t ptent, unsigned long addr, + struct zap_details *details, int *rss, bool *force_flush, + bool *force_break) { struct mm_struct *mm = tlb->mm; bool delay_rmap = false; - struct folio *folio; - struct page *page; - - page = vm_normal_page(vma, addr, ptent); - if (!page) { - /* We don't need up-to-date accessed/dirty bits. */ - ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); - arch_check_zapped_pte(vma, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); - VM_WARN_ON_ONCE(userfaultfd_wp(vma)); - ksm_might_unmap_zero_page(mm, ptent); - return; - } - - folio = page_folio(page); - if (unlikely(!should_zap_folio(details, folio))) - return; if (!folio_test_anon(folio)) { ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); @@ -1586,6 +1570,33 @@ static inline void zap_present_pte(struct mmu_gather *tlb, } } +static inline void zap_present_pte(struct mmu_gather *tlb, + struct vm_area_struct *vma, pte_t *pte, pte_t ptent, + unsigned long addr, struct zap_details *details, + int *rss, bool *force_flush, bool *force_break) +{ + struct mm_struct *mm = tlb->mm; + struct folio *folio; + struct page *page; + + page = vm_normal_page(vma, addr, ptent); + if (!page) { + /* We don't need up-to-date accessed/dirty bits. */ + ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + arch_check_zapped_pte(vma, ptent); + tlb_remove_tlb_entry(tlb, pte, addr); + VM_WARN_ON_ONCE(userfaultfd_wp(vma)); + ksm_might_unmap_zero_page(mm, ptent); + return; + } + + folio = page_folio(page); + if (unlikely(!should_zap_folio(details, folio))) + return; + zap_present_folio_pte(tlb, vma, folio, page, pte, ptent, addr, details, + rss, force_flush, force_break); +} + static unsigned long zap_pte_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, From patchwork Fri Feb 9 22:15:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 199149 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:50ea:b0:106:860b:bbdd with SMTP id r10csp1170787dyd; Fri, 9 Feb 2024 14:39:51 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWBKmWdujY1LP57r3jSavQPhQq5xVNc+0gtSGOhNPnZF33gunQYGH5gWny+ax8pI2HZhxqBjUIloh5YI5Ouy6+e6ie30g== X-Google-Smtp-Source: AGHT+IHZDMrKu/TgTbYTUtqnbf+Xl8x52Jl36x/e9ZYvratuaDuL77g4j4XV4wuFiTvbT5GsEros X-Received: by 2002:a17:903:2583:b0:1d9:119e:af45 with SMTP id jb3-20020a170903258300b001d9119eaf45mr652966plb.41.1707518391004; Fri, 09 Feb 2024 14:39:51 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707518390; cv=pass; d=google.com; s=arc-20160816; b=J70Uxj3vl2DIt9jl+QVjrMcoreT6FclCL3KAzRAeahTc/bXW4zZCYoluLYMajfzPBy dPK2P8bBv2LeWiAX2PCE0kVM08elltYe0wAsDpYMVwTnm160M8xyk4M1UO5dI6Ybja6T vfcir8vPROjlW4ejT10gl43qyYBNINBLs+wpRRT5SSzsURdBVtgAz7mbEGZf3v3hzVpr fX0VIP+Eh7XDB7DgYoaCr//butOKsb7wgooog3o3auFNSsWoIojIR/SLB4zp4LXiWedI TeCZguV5n6T/EzPtp87CVkxXEPBZdOT59uVBy+jEGde2IfoWXlNecMlRK3maWUqQzsy/ CT8g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=4Q1ph86PH1Crd3Eqr5OMYEFX/2u3faALVbtCzM66/MM=; fh=rfuVwQingbIohDTYu35Grnd8XRU4NVvzvmtuckx5KIM=; b=0b7N+ewsN1I/8B3HxLUSVJMeq3qniionxWZPHLgap8jnvLLbc3FV4pIvHivXUFOXg/ nT+pCJWCfVhOEG/r4I/GW+LOs1rBFSWErvQmyDcyfQZyxJ3Y4A5TRfIq4I6IU0P7E1D5 XP9fq8muCvswyjxAbXGbi7wd8DyVr1dUsIIozsw5gQKSVziLNXm2DWcvgUbfXSpxI5jY qqT0SutgRlrPF7J2gWwfcyLfbipEi8xbmkwsBQhjtP6DoVlLuqElZM8Hm1ZUt80Ihh3d efy4KZ7TX8ymeRgzADIRt8ioOFGSzHDIcqGfs+wDhBkSBeGaAD2qGNOFWyJxwRQvvw96 7PBg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bmBBJUSR; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60077-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60077-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Forwarded-Encrypted: i=2; AJvYcCXnj3+NyWuCd9dDCBnR5zT99CAxq+K3BBqx9ThaEZntrGSRYgPXBzwyhuW7dIbG7TdtkUO8beQgGSaV5/YnVCOkvEe+2w== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id mc6-20020a1709032b0600b001d8e4de7a1dsi2320330plb.581.2024.02.09.14.39.50 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 14:39:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-60077-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bmBBJUSR; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60077-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60077-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 5698DB28CCE for ; Fri, 9 Feb 2024 22:17:04 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9FDE339875; Fri, 9 Feb 2024 22:15:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="bmBBJUSR" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A571E3613D for ; Fri, 9 Feb 2024 22:15:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516948; cv=none; b=uwBTZvj0svFwgsLNvYxWtlmhGiro45V6TIc76vlTUaZR+luhlYfmUA8sX19fdD0GriHK5qOMt8r19MQCKREyfuVTe2kzGG0Hg2hYcj0UWrdGu7XTQTw9sBo8RDo2t7w64gpl4o46KI50ovmtxh6V6KbNP5Mel13g7fEVJY16SPU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516948; c=relaxed/simple; bh=klZ3/8gE/hLexWRn9Rfgs4Jd0fYkSsj1v6sMZOE0W70=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NfMRQJ+bm9Fi3dKeCEaAH72JIYP9Wg9h8hL+CUd7rR/C2e2EPH0yPaynVAIqAx7OvTgweT0BAr1s7a0UCf4Nfy74AZNmKTBVVszpR83aa4nVKuXUMVkrkZsmEJ5ymriCBmt/A1Zonq2bnaBI9D3Ea3FCzA2vk3wK008ojI2Uik8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=bmBBJUSR; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1707516945; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4Q1ph86PH1Crd3Eqr5OMYEFX/2u3faALVbtCzM66/MM=; b=bmBBJUSR42eo6IRGd1YNaXuTaMenNxiDpILc/41eecvVOkarLm9SJetPhorGDueZbkCc5m Cm0DfdKAEpJ78i2ty2nz1CJUpzXkQVLiiIRA5CgDVOCVbwRGlqv69kJ6BcIV7cBLxJB2Ig zCiKIv00rW0c2uHTxNO91/U4KJYLM8I= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-271-Q3fpNq9gN4WqYc2YHXjPew-1; Fri, 09 Feb 2024 17:15:42 -0500 X-MC-Unique: Q3fpNq9gN4WqYc2YHXjPew-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8D4AE1025623; Fri, 9 Feb 2024 22:15:41 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.194.59]) by smtp.corp.redhat.com (Postfix) with ESMTP id 59E001C14B04; Fri, 9 Feb 2024 22:15:37 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Yin Fengwei , Michal Hocko , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v2 05/10] mm/mmu_gather: pass "delay_rmap" instead of encoded page to __tlb_remove_page_size() Date: Fri, 9 Feb 2024 23:15:04 +0100 Message-ID: <20240209221509.585251-6-david@redhat.com> In-Reply-To: <20240209221509.585251-1-david@redhat.com> References: <20240209221509.585251-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790462804274634832 X-GMAIL-MSGID: 1790462804274634832 We have two bits available in the encoded page pointer to store additional information. Currently, we use one bit to request delay of the rmap removal until after a TLB flush. We want to make use of the remaining bit internally for batching of multiple pages of the same folio, specifying that the next encoded page pointer in an array is actually "nr_pages". So pass page + delay_rmap flag instead of an encoded page, to handle the encoding internally. Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand --- arch/s390/include/asm/tlb.h | 13 ++++++------- include/asm-generic/tlb.h | 12 ++++++------ mm/mmu_gather.c | 7 ++++--- 3 files changed, 16 insertions(+), 16 deletions(-) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index d1455a601adc..48df896d5b79 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -25,8 +25,7 @@ void __tlb_remove_table(void *_table); static inline void tlb_flush(struct mmu_gather *tlb); static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, - struct encoded_page *page, - int page_size); + struct page *page, bool delay_rmap, int page_size); #define tlb_flush tlb_flush #define pte_free_tlb pte_free_tlb @@ -42,14 +41,14 @@ static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, * tlb_ptep_clear_flush. In both flush modes the tlb for a page cache page * has already been freed, so just do free_page_and_swap_cache. * - * s390 doesn't delay rmap removal, so there is nothing encoded in - * the page pointer. + * s390 doesn't delay rmap removal. */ static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, - struct encoded_page *page, - int page_size) + struct page *page, bool delay_rmap, int page_size) { - free_page_and_swap_cache(encoded_page_ptr(page)); + VM_WARN_ON_ONCE(delay_rmap); + + free_page_and_swap_cache(page); return false; } diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 129a3a759976..2eb7b0d4f5d2 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -260,9 +260,8 @@ struct mmu_gather_batch { */ #define MAX_GATHER_BATCH_COUNT (10000UL/MAX_GATHER_BATCH) -extern bool __tlb_remove_page_size(struct mmu_gather *tlb, - struct encoded_page *page, - int page_size); +extern bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, + bool delay_rmap, int page_size); #ifdef CONFIG_SMP /* @@ -462,13 +461,14 @@ static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) static inline void tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_size) { - if (__tlb_remove_page_size(tlb, encode_page(page, 0), page_size)) + if (__tlb_remove_page_size(tlb, page, false, page_size)) tlb_flush_mmu(tlb); } -static __always_inline bool __tlb_remove_page(struct mmu_gather *tlb, struct page *page, unsigned int flags) +static __always_inline bool __tlb_remove_page(struct mmu_gather *tlb, + struct page *page, bool delay_rmap) { - return __tlb_remove_page_size(tlb, encode_page(page, flags), PAGE_SIZE); + return __tlb_remove_page_size(tlb, page, delay_rmap, PAGE_SIZE); } /* tlb_remove_page diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 604ddf08affe..ac733d81b112 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -116,7 +116,8 @@ static void tlb_batch_list_free(struct mmu_gather *tlb) tlb->local.next = NULL; } -bool __tlb_remove_page_size(struct mmu_gather *tlb, struct encoded_page *page, int page_size) +bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, + bool delay_rmap, int page_size) { struct mmu_gather_batch *batch; @@ -131,13 +132,13 @@ bool __tlb_remove_page_size(struct mmu_gather *tlb, struct encoded_page *page, i * Add the page and check if we are full. If so * force a flush. */ - batch->encoded_pages[batch->nr++] = page; + batch->encoded_pages[batch->nr++] = encode_page(page, delay_rmap); if (batch->nr == batch->max) { if (!tlb_next_batch(tlb)) return true; batch = tlb->active; } - VM_BUG_ON_PAGE(batch->nr > batch->max, encoded_page_ptr(page)); + VM_BUG_ON_PAGE(batch->nr > batch->max, page); return false; } From patchwork Fri Feb 9 22:15:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 199130 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:50ea:b0:106:860b:bbdd with SMTP id r10csp1161882dyd; Fri, 9 Feb 2024 14:17:21 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCV+1sxod8ifBDzrOPfLVwc3qB1ZwO06gYheRbt9TFTUIgYu9EwGmo4mLDxL82nmsFiu9RcfE+ctfc7fDlnjTkRdoFAtLA== X-Google-Smtp-Source: AGHT+IFVPMpWaB457wdlhe+JbzRbDw90T/MLyDXqYdVfU7bzXXsCGyDHK6e34cNRPBELTdkfQE1Z X-Received: by 2002:a50:ee93:0:b0:560:c364:2ce2 with SMTP id f19-20020a50ee93000000b00560c3642ce2mr165415edr.41.1707517041176; Fri, 09 Feb 2024 14:17:21 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707517041; cv=pass; d=google.com; s=arc-20160816; b=DTtS2Hxq/5+ONn533pXqlAFIQHMTLj0zYV7zTCb2yxwVZfXMRVWFHp+NwiOy06as3U FSaIXUZTPAToRqDLh3AUIijWl3Wd0qGOgD9ua87b5HttL64V2+/m974GZKQ3KBN9QVpl SkAK/MyVIewt5IatjZRM7Zc2btuWsxezMYiLBAvYmKqfJPkSleIDHA75ynJfAdBpPhyQ EeZLNK8BgzHs8ZhBaJJh0/5H9Ww1ce460hVRXWv3wcBAb5qLHFWT0BjWdUy8Ktf0ewJ5 l395cXtJPACjzBCXRPa2t/5edc2U0sNwuOr1yfbNxnm7uKcap1hWpV5YizIlU3REtXJc m5Rg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=BfteBvWtn9YL0u+ZWWzLkAZQ9TqgPhMIROF4I2AA+84=; fh=YyfDa6qOTRzg3K3ADqjIvKXeprwxnS/f3IaevNS9mig=; b=owi745zPJ2/5A24ai+9hcxuaAOSnXwUd1f0C/uOkdQM070yjSKQjhHJtPSJ1R4XpY2 VC7cN0E+pdVyPVk7EzwFGnTW6MSgVJ5q8c92ExgvpM64ek3DDmHGn4lJyCu12JHccjrE tpEiIz9oP7hCt+QdsbiR7CO6uwzn8K8w50pqNHPK7tcmdUJtFbjikiuoV6jaYkDrdEDC 4CpxluHmCEPWcLkz7Ab2HlMTT2X8Vx7Q9FKQp+E1tyVyYnqHkWfcVtpE7Q1WjYGDIPHv jGwY8l3AQHJqeqNiLXsWnvY/vwEmCb/Goxg8kmdkM5/5tQ30cV38bwjdd2FHbqgoDnlC GdLw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="eCAnamW/"; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60078-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60078-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Forwarded-Encrypted: i=2; AJvYcCVNVkKtO7lo3VqfjsvIKLjspmASx8bsUuNec1Vw9QQxUi04csdWPXEPr25zZxtWXJtIMWgGcVWywcEafGNp3EJjF2UrSw== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id q5-20020a50c345000000b00560c673ccc2si163714edb.252.2024.02.09.14.17.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 14:17:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-60078-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="eCAnamW/"; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60078-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60078-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id C49CF1F215D5 for ; Fri, 9 Feb 2024 22:17:20 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 6D86939AC4; Fri, 9 Feb 2024 22:15:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="eCAnamW/" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9DD173987C for ; Fri, 9 Feb 2024 22:15:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516955; cv=none; b=eWZTCvHkygRtQxzUU+egQjaiTijUoJTdeHVgCRxo5Vi8umIDctS2IL0kxSAp0weCvCMpF87CMUxRFTXorMF6c7LnL08+i5DSbRu5D4p+bBpGGU1Kw2ZsjE1Z5of9vd2ePPL01K1i1D30Zh/+htzzZZeKUwvWO/zZAI6b47iffYw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516955; c=relaxed/simple; bh=Ivx6k9RHC7xtfNym2WBe4tYwW7cQHPIIcIfbRJkpcmw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XaswefBYypoQuTgO+Oxrw32WyT/SBaWQpGciwMqEugFrWX+MVO6nALJSZwl3/IxRjV1ASN9DKlltlnD8R0sbck9aaXK5W1+exiI9Q+7dCml5O7zjYw00xo2O484vzZKi5AYhwZ62i87/aLut9TkwrutfQVBn1f2KcXuFWumJs2A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=eCAnamW/; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1707516952; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BfteBvWtn9YL0u+ZWWzLkAZQ9TqgPhMIROF4I2AA+84=; b=eCAnamW/s1b6VFD7zf93NUdcjz2Vvt1v+wFcq731PmQU17MdFP5TebVIJcHqY8k2nNqoI4 QMltrtGV9y5hsUOVG5G/8czhHJRsMcnJDEsyN4DpBvfRWlsckP1XnFohluXLV0I9+1umP2 c3ETWBwgZRWXRo/my3xMa13gGYz9zOU= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-441-DUvgpdnlM0OO1UlRjbDzjg-1; Fri, 09 Feb 2024 17:15:48 -0500 X-MC-Unique: DUvgpdnlM0OO1UlRjbDzjg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 025DF2823807; Fri, 9 Feb 2024 22:15:47 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.194.59]) by smtp.corp.redhat.com (Postfix) with ESMTP id D5A541C14B04; Fri, 9 Feb 2024 22:15:41 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Yin Fengwei , Michal Hocko , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v2 06/10] mm/mmu_gather: define ENCODED_PAGE_FLAG_DELAY_RMAP Date: Fri, 9 Feb 2024 23:15:05 +0100 Message-ID: <20240209221509.585251-7-david@redhat.com> In-Reply-To: <20240209221509.585251-1-david@redhat.com> References: <20240209221509.585251-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790461389122818101 X-GMAIL-MSGID: 1790461389122818101 Nowadays, encoded pages are only used in mmu_gather handling. Let's update the documentation, and define ENCODED_PAGE_BIT_DELAY_RMAP. While at it, rename ENCODE_PAGE_BITS to ENCODED_PAGE_BITS. If encoded page pointers would ever be used in other context again, we'd likely want to change the defines to reflect their context (e.g., ENCODED_PAGE_FLAG_MMU_GATHER_DELAY_RMAP). For now, let's keep it simple. This is a preparation for using the remaining spare bit to indicate that the next item in an array of encoded pages is a "nr_pages" argument and not an encoded page. Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand --- include/linux/mm_types.h | 17 +++++++++++------ mm/mmu_gather.c | 5 +++-- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8b611e13153e..1b89eec0d6df 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -210,8 +210,8 @@ struct page { * * An 'encoded_page' pointer is a pointer to a regular 'struct page', but * with the low bits of the pointer indicating extra context-dependent - * information. Not super-common, but happens in mmu_gather and mlock - * handling, and this acts as a type system check on that use. + * information. Only used in mmu_gather handling, and this acts as a type + * system check on that use. * * We only really have two guaranteed bits in general, although you could * play with 'struct page' alignment (see CONFIG_HAVE_ALIGNED_STRUCT_PAGE) @@ -220,21 +220,26 @@ struct page { * Use the supplied helper functions to endcode/decode the pointer and bits. */ struct encoded_page; -#define ENCODE_PAGE_BITS 3ul + +#define ENCODED_PAGE_BITS 3ul + +/* Perform rmap removal after we have flushed the TLB. */ +#define ENCODED_PAGE_BIT_DELAY_RMAP 1ul + static __always_inline struct encoded_page *encode_page(struct page *page, unsigned long flags) { - BUILD_BUG_ON(flags > ENCODE_PAGE_BITS); + BUILD_BUG_ON(flags > ENCODED_PAGE_BITS); return (struct encoded_page *)(flags | (unsigned long)page); } static inline unsigned long encoded_page_flags(struct encoded_page *page) { - return ENCODE_PAGE_BITS & (unsigned long)page; + return ENCODED_PAGE_BITS & (unsigned long)page; } static inline struct page *encoded_page_ptr(struct encoded_page *page) { - return (struct page *)(~ENCODE_PAGE_BITS & (unsigned long)page); + return (struct page *)(~ENCODED_PAGE_BITS & (unsigned long)page); } /* diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index ac733d81b112..6540c99c6758 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -53,7 +53,7 @@ static void tlb_flush_rmap_batch(struct mmu_gather_batch *batch, struct vm_area_ for (int i = 0; i < batch->nr; i++) { struct encoded_page *enc = batch->encoded_pages[i]; - if (encoded_page_flags(enc)) { + if (encoded_page_flags(enc) & ENCODED_PAGE_BIT_DELAY_RMAP) { struct page *page = encoded_page_ptr(enc); folio_remove_rmap_pte(page_folio(page), page, vma); } @@ -119,6 +119,7 @@ static void tlb_batch_list_free(struct mmu_gather *tlb) bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, bool delay_rmap, int page_size) { + int flags = delay_rmap ? ENCODED_PAGE_BIT_DELAY_RMAP : 0; struct mmu_gather_batch *batch; VM_BUG_ON(!tlb->end); @@ -132,7 +133,7 @@ bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, * Add the page and check if we are full. If so * force a flush. */ - batch->encoded_pages[batch->nr++] = encode_page(page, delay_rmap); + batch->encoded_pages[batch->nr++] = encode_page(page, flags); if (batch->nr == batch->max) { if (!tlb_next_batch(tlb)) return true; From patchwork Fri Feb 9 22:15:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 199131 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:50ea:b0:106:860b:bbdd with SMTP id r10csp1161997dyd; Fri, 9 Feb 2024 14:17:37 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCX58V/VDSXDcJlUjCeP/taRoWXdoEcGqm2WVQaH71FQyxLWVxCIA0ZdlxA5BhrXrxLlOP04lRq6FT/AedUi+FFsAD4DXA== X-Google-Smtp-Source: AGHT+IHVSqYPVetjYdTsJ904MoU+utlgOW7hBI7WMXlDTE+DhWQ8UrahLCbW58hi1bSpHEf9ySDF X-Received: by 2002:a05:6a21:680b:b0:19c:a3de:647d with SMTP id wr11-20020a056a21680b00b0019ca3de647dmr594783pzb.19.1707517057241; Fri, 09 Feb 2024 14:17:37 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707517057; cv=pass; d=google.com; s=arc-20160816; b=USfJ6u/SnlyS5/QdT6K2bHU6h93qbuzllYfP5C8tW8ksiRn8bCCv5cE/iOGmf97p5K Cb/L8D3zAyY68f5MLYK22x79K4XFdYgK0num+2SYvvcIEHI3vaj77OgjGH0scvjMwJ83 OffT0Qh+PXc3UnKDeHcCzshvo6pZNdolmtySqwS4Mw9pVO4X6xaCIg9QLa1cUUyTUmFW /c9keq78tRfNpO8r0LpH4Q4Q2VSSDo1bZwzs0iNZS7TgPz7W7c6vdtUN/6uwky8V78se XHAg9mTYrx3OAPTuqcZ20D2xjxJtTWoBIXZ8IvbrnQBo307DFcuZMFdUZTqVaCNOkEbG foFA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=X4BN9bDZrrapc7Xd1FDfmQmuEZxXG5tNILRPdL9R2lg=; fh=y4VstOrZYRUB9RkGghFcn/3lTEEFm1Sq4kAycYylzns=; b=BO9GUL0JUI93G7rw3se3d+LuiYc0/Sq6uBIxRkRNrKUO/sOAZSjdcDshaRwKusCMH1 bE6SneFRug2/6Y5InSww4LTQRxSU+JSXVN+D3uE00MveLU04gLIWgivxmkwIGSLPJ6Ll w16KoV++M2ObWK4cHMWlaKyvi0qiexoMpZwB97g0AaFy2TXnoQYDgbbPFT2MQv57ah9N GKGdLxYP0QtAqoGS6N77YjJBFETZx1vB1qomDZzZqm6mqiDIz9hya0QIFRIRadJjLqd0 jOg0NRazJM064UrPru5G4F1OJXukgyMwz3shUiUFkImVYtxFuc4NosXc7kjfAaGN5xdK 7d7Q==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=PNAyHPtf; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60079-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60079-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Forwarded-Encrypted: i=2; AJvYcCUhJ1Wapev1Qmbf0FMK6fy1s/xGZdzxLsQL2BUKLnIwFptL/fyEuoE+7w9CN/uocGHlt6rmkYDm5oMo2FHigWwf3+zsTQ== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id ca33-20020a056a0206a100b005dc359329ffsi2595522pgb.123.2024.02.09.14.17.37 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 14:17:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-60079-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=PNAyHPtf; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60079-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60079-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 9FE52284C44 for ; Fri, 9 Feb 2024 22:17:34 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 515CA39AE9; Fri, 9 Feb 2024 22:16:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="PNAyHPtf" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1ED1336AEF for ; Fri, 9 Feb 2024 22:15:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516960; cv=none; b=bqYUfTV/LXzO3IhLsRFyXEpaaMDjgmMF9McgVkmx5Q+MIq8Vep92Mzljw0UsnH9XCC2vJ02dcIMp/U2bduIuVEDfrDuoNIsoLfbANyK0DSg1WQJDSpj7yWJl3cLA56X7JXXBQ480YrK0GIk5xk17+u1BqooG8RrXAX5IaXIy19E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516960; c=relaxed/simple; bh=3PQfbTkS/vd68uh/N1+tvV3UXJ8B0ItANJG44cGwxXU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nDP4z+jRGlhpq+mVrj6S3JwWIJ4/H8XvPD955KFEH/aTM9VnQ37ZME2EOMviZNNyJ4yschoCR4o3qwIbzWF1yGEcHj/pjJ0WVmlUrlOvVi8SJeoThrcVeax5iYTvY4Nv76mbOfp3zCnwSbIeG6TuRqQmQDm06nu0VP9+ibMx5yc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=PNAyHPtf; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1707516958; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X4BN9bDZrrapc7Xd1FDfmQmuEZxXG5tNILRPdL9R2lg=; b=PNAyHPtfJPw9kdskRJq+fXgxFH3vYP1viGXv3Re4GGsn4DPnwOb5hrKK87xksgv5GSLFWY DF81JeQVfNXAZ7oGt19XbXtC3GPSwv9DM9ksDneFguqkXUHcxEqlnpgwSeK7eZ2I9gdgZ3 yyeO7epHcuG7S1SIRO84+IFOT7kei9Q= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-373-AMAEvJNUPyeeH3DHaeR2Bw-1; Fri, 09 Feb 2024 17:15:52 -0500 X-MC-Unique: AMAEvJNUPyeeH3DHaeR2Bw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9B6CF83722E; Fri, 9 Feb 2024 22:15:51 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.194.59]) by smtp.corp.redhat.com (Postfix) with ESMTP id 666531C14B04; Fri, 9 Feb 2024 22:15:47 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Yin Fengwei , Michal Hocko , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v2 07/10] mm/mmu_gather: add tlb_remove_tlb_entries() Date: Fri, 9 Feb 2024 23:15:06 +0100 Message-ID: <20240209221509.585251-8-david@redhat.com> In-Reply-To: <20240209221509.585251-1-david@redhat.com> References: <20240209221509.585251-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790461405970774246 X-GMAIL-MSGID: 1790461405970774246 Let's add a helper that lets us batch-process multiple consecutive PTEs. Note that the loop will get optimized out on all architectures except on powerpc. We have to add an early define of __tlb_remove_tlb_entry() on ppc to make the compiler happy (and avoid making tlb_remove_tlb_entries() a macro). Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand --- arch/powerpc/include/asm/tlb.h | 2 ++ include/asm-generic/tlb.h | 20 ++++++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index b3de6102a907..1ca7d4c4b90d 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -19,6 +19,8 @@ #include +static inline void __tlb_remove_tlb_entry(struct mmu_gather *tlb, pte_t *ptep, + unsigned long address); #define __tlb_remove_tlb_entry __tlb_remove_tlb_entry #define tlb_flush tlb_flush diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 2eb7b0d4f5d2..95d60a4f468a 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -608,6 +608,26 @@ static inline void tlb_flush_p4d_range(struct mmu_gather *tlb, __tlb_remove_tlb_entry(tlb, ptep, address); \ } while (0) +/** + * tlb_remove_tlb_entries - remember unmapping of multiple consecutive ptes for + * later tlb invalidation. + * + * Similar to tlb_remove_tlb_entry(), but remember unmapping of multiple + * consecutive ptes instead of only a single one. + */ +static inline void tlb_remove_tlb_entries(struct mmu_gather *tlb, + pte_t *ptep, unsigned int nr, unsigned long address) +{ + tlb_flush_pte_range(tlb, address, PAGE_SIZE * nr); + for (;;) { + __tlb_remove_tlb_entry(tlb, ptep, address); + if (--nr == 0) + break; + ptep++; + address += PAGE_SIZE; + } +} + #define tlb_remove_huge_tlb_entry(h, tlb, ptep, address) \ do { \ unsigned long _sz = huge_page_size(h); \ From patchwork Fri Feb 9 22:15:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 199132 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:50ea:b0:106:860b:bbdd with SMTP id r10csp1162098dyd; Fri, 9 Feb 2024 14:17:51 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVJvoXwm+fcVzlqeRt7pOa6g2SLHw4fd0shrF8G1a5gdUpRp8fW+CAZOkzXxSE+VFGt78L+k/kBaVfqdJY1tHGMFe8mRQ== X-Google-Smtp-Source: AGHT+IF22Z0Y23MwAWJ+pjZvjVek/Fdjl9Akv+kLNRaDU5OJhsl9suHLB40+jOASOW2/PPDjA8cd X-Received: by 2002:a92:cd45:0:b0:363:b5bc:e8ee with SMTP id v5-20020a92cd45000000b00363b5bce8eemr613112ilq.7.1707517071504; Fri, 09 Feb 2024 14:17:51 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707517071; cv=pass; d=google.com; s=arc-20160816; b=qjeW2W+DHTMIDusDAn4DwJtlYRv2c0qblbOh+kCgzlfhtbgY2HQKf0tDdnU8/OwRQ9 Ad6XRJpqrwSnd1v3Pr1hWHyJnwKd+Ox2U6jwR23oAbHFapLGpiR8E365SS/7Kk+Zuo0r rgSkbroXeOZ39yMFzKK8tmPzqkk+qlyK72dVjH2HnWJd8ktrvoHAKQiCPwJKldiveP6K mrCQZFowN5fHhF/N6o4CWS6qTFiVS6D3T7oZpjlfY7mWWXUgWQCroh/+iUgUKfh0FSJP JBSSqT3q+14ZTSMDt5HZO1LA0oES99Q9+3yKUrrCQBur/RYaswnCKS2F4PQDOy3U5lO2 /5Qw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=1nEVDzHw7aC5MZOjj7IJGnO0raht3IChKPRkoJ/iPcQ=; fh=op2fV//NItX8bY82XyEhvOdDT5X8Own4Ggf6AQVQvzs=; b=V5oQLjROZXc0euRn2fGo0KjVtyh6oHIzzvWIO3EdIr3LHApLK1sds1IH0gxf4kr7qa CQjlzfujOVePMAgdbc8nIuccmvEKf2y0ooMMOd9oJCElIB1eY5GvS+zmRYY2reuXblw1 BBG+KZvs3AT6zvbsYGZRw+IMXUx144G0ziwfw5W82aClKCZsxNgV1dQ7GDyMT4gNXF87 mkCgH+vW6Ad7ZmB7FLNkQgpKjWhFCSkf5J2zQgknyxEU8ZOIdhYO8lV/U64Ow014nuK3 XzqiCLrDN6xNLRn8YZ36hdfM9hZBF1Hxnuj1RR2hojAUTa3luVbs43O4jOXIr7IqFvK0 TOiQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Hl9B0eLK; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60080-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60080-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Forwarded-Encrypted: i=2; AJvYcCXQe28Y0FyFlHkwRk2yCLZ4dZTV52Gq33oF8g5VSk9okDaRxo9bUmJsGYUcJyQgrMD/ql/3itczaymepN9dBFUiCKK45Q== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id bn10-20020a056a02030a00b005d8b6894062si2558358pgb.653.2024.02.09.14.17.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 14:17:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-60080-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Hl9B0eLK; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60080-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60080-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 409A22825CB for ; Fri, 9 Feb 2024 22:17:51 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id DC42339FD0; Fri, 9 Feb 2024 22:16:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Hl9B0eLK" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2670639AD8 for ; Fri, 9 Feb 2024 22:16:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516964; cv=none; b=kQMny38zvgexGCT8HIJFJIRCB60VZh1wAxM4iHe5I/HNon8ZJfBhhGnSiUttoWSYTCC5a/g2BmWrGDxvEY1I2Q6AcXeSPWC7mXqfbxx6ZzEGC1CmjoxiZO5FkgNCD6++VjwbJ/WQATqYFu3hfLyZTrDnHKQJHwQTBv3b//g1K/c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516964; c=relaxed/simple; bh=gBscCblwqfSI0RhWxyhYqNwZV4QTB2Qwi4YHtLPurK0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sNTABMXEZrnc+bGBgAzB8LzUqRa66ApoKKgGVnpnaWJCiz/4HJY3S+WVB0epvqgwaG2eDmZHkwHgW9edzumnWnwGWxNoCgLhB5znV2hZ2z2NwRyECJpU5kvyTMNvyupWw2TAgoeR70mal79wvMoG5ZgPgdSWM4FVf6kaQhyQ1qs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Hl9B0eLK; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1707516961; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1nEVDzHw7aC5MZOjj7IJGnO0raht3IChKPRkoJ/iPcQ=; b=Hl9B0eLKpidRd++7Af7Kx0nmhKl+x8tKJ7DBeNIuGhQsED5PS+vN7/S3A/SAvXrAvYBkJC NUO4qgx17j6nac4uGioQ9eXuO1NckdeCh4Pyp2qykK8MYUFbP7Nr252agxbTDGQPC+szwv XGFwKpBDtM/s/V8Fpis87TML+vPWe8Y= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-483-GjH1TBS8OeKCQERcPxqveA-1; Fri, 09 Feb 2024 17:15:57 -0500 X-MC-Unique: GjH1TBS8OeKCQERcPxqveA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6AF29185A780; Fri, 9 Feb 2024 22:15:56 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.194.59]) by smtp.corp.redhat.com (Postfix) with ESMTP id DF5C61C14B04; Fri, 9 Feb 2024 22:15:51 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Yin Fengwei , Michal Hocko , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v2 08/10] mm/mmu_gather: add __tlb_remove_folio_pages() Date: Fri, 9 Feb 2024 23:15:07 +0100 Message-ID: <20240209221509.585251-9-david@redhat.com> In-Reply-To: <20240209221509.585251-1-david@redhat.com> References: <20240209221509.585251-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790461420879329497 X-GMAIL-MSGID: 1790461420879329497 Add __tlb_remove_folio_pages(), which will remove multiple consecutive pages that belong to the same large folio, instead of only a single page. We'll be using this function when optimizing unmapping/zapping of large folios that are mapped by PTEs. We're using the remaining spare bit in an encoded_page to indicate that the next enoced page in an array contains actually shifted "nr_pages". Teach swap/freeing code about putting multiple folio references, and delayed rmap handling to remove page ranges of a folio. This extension allows for still gathering almost as many small folios as we used to (-1, because we have to prepare for a possibly bigger next entry), but still allows for gathering consecutive pages that belong to the same large folio. Note that we don't pass the folio pointer, because it is not required for now. Further, we don't support page_size != PAGE_SIZE, it won't be required for simple PTE batching. We have to provide a separate s390 implementation, but it's fairly straight forward. Another, more invasive and likely more expensive, approach would be to use folio+range or a PFN range instead of page+nr_pages. But, we should do that consistently for the whole mmu_gather. For now, let's keep it simple and add "nr_pages" only. Note that it is now possible to gather significantly more pages: In the past, we were able to gather ~10000 pages, now we can gather also gather ~5000 folio fragments that span multiple pages. A folio fragement on x86-64 can be up to 512 pages (2 MiB THP) and on arm64 with 64k in theory 8192 pages (512 MiB THP). Gathering more memory is not considered something we should worry about, especially because these are already corner cases. While we can gather more total memory, we won't free more folio fragments. As long as page freeing time primarily only depends on the number of involved folios, there is no effective change for !preempt configurations. However, we'll adjust tlb_batch_pages_flush() separately to handle corner cases where page freeing time grows proportionally with the actual memory size. Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- arch/s390/include/asm/tlb.h | 17 +++++++++++ include/asm-generic/tlb.h | 8 +++++ include/linux/mm_types.h | 20 ++++++++++++ mm/mmu_gather.c | 61 +++++++++++++++++++++++++++++++------ mm/swap.c | 12 ++++++-- mm/swap_state.c | 15 +++++++-- 6 files changed, 119 insertions(+), 14 deletions(-) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 48df896d5b79..e95b2c8081eb 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -26,6 +26,8 @@ void __tlb_remove_table(void *_table); static inline void tlb_flush(struct mmu_gather *tlb); static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, bool delay_rmap, int page_size); +static inline bool __tlb_remove_folio_pages(struct mmu_gather *tlb, + struct page *page, unsigned int nr_pages, bool delay_rmap); #define tlb_flush tlb_flush #define pte_free_tlb pte_free_tlb @@ -52,6 +54,21 @@ static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, return false; } +static inline bool __tlb_remove_folio_pages(struct mmu_gather *tlb, + struct page *page, unsigned int nr_pages, bool delay_rmap) +{ + struct encoded_page *encoded_pages[] = { + encode_page(page, ENCODED_PAGE_BIT_NR_PAGES_NEXT), + encode_nr_pages(nr_pages), + }; + + VM_WARN_ON_ONCE(delay_rmap); + VM_WARN_ON_ONCE(page_folio(page) != page_folio(page + nr_pages - 1)); + + free_pages_and_swap_cache(encoded_pages, ARRAY_SIZE(encoded_pages)); + return false; +} + static inline void tlb_flush(struct mmu_gather *tlb) { __tlb_flush_mm_lazy(tlb->mm); diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 95d60a4f468a..bd00dd238b79 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -69,6 +69,7 @@ * * - tlb_remove_page() / __tlb_remove_page() * - tlb_remove_page_size() / __tlb_remove_page_size() + * - __tlb_remove_folio_pages() * * __tlb_remove_page_size() is the basic primitive that queues a page for * freeing. __tlb_remove_page() assumes PAGE_SIZE. Both will return a @@ -78,6 +79,11 @@ * tlb_remove_page() and tlb_remove_page_size() imply the call to * tlb_flush_mmu() when required and has no return value. * + * __tlb_remove_folio_pages() is similar to __tlb_remove_page(), however, + * instead of removing a single page, remove the given number of consecutive + * pages that are all part of the same (large) folio: just like calling + * __tlb_remove_page() on each page individually. + * * - tlb_change_page_size() * * call before __tlb_remove_page*() to set the current page-size; implies a @@ -262,6 +268,8 @@ struct mmu_gather_batch { extern bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, bool delay_rmap, int page_size); +bool __tlb_remove_folio_pages(struct mmu_gather *tlb, struct page *page, + unsigned int nr_pages, bool delay_rmap); #ifdef CONFIG_SMP /* diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 1b89eec0d6df..a7223ba3ea1e 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -226,6 +226,15 @@ struct encoded_page; /* Perform rmap removal after we have flushed the TLB. */ #define ENCODED_PAGE_BIT_DELAY_RMAP 1ul +/* + * The next item in an encoded_page array is the "nr_pages" argument, specifying + * the number of consecutive pages starting from this page, that all belong to + * the same folio. For example, "nr_pages" corresponds to the number of folio + * references that must be dropped. If this bit is not set, "nr_pages" is + * implicitly 1. + */ +#define ENCODED_PAGE_BIT_NR_PAGES_NEXT 2ul + static __always_inline struct encoded_page *encode_page(struct page *page, unsigned long flags) { BUILD_BUG_ON(flags > ENCODED_PAGE_BITS); @@ -242,6 +251,17 @@ static inline struct page *encoded_page_ptr(struct encoded_page *page) return (struct page *)(~ENCODED_PAGE_BITS & (unsigned long)page); } +static __always_inline struct encoded_page *encode_nr_pages(unsigned long nr) +{ + VM_WARN_ON_ONCE((nr << 2) >> 2 != nr); + return (struct encoded_page *)(nr << 2); +} + +static __always_inline unsigned long encoded_nr_pages(struct encoded_page *page) +{ + return ((unsigned long)page) >> 2; +} + /* * A swap entry has to fit into a "unsigned long", as the entry is hidden * in the "index" field of the swapper address space. diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 6540c99c6758..d175c0f1e2c8 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -50,12 +50,21 @@ static bool tlb_next_batch(struct mmu_gather *tlb) #ifdef CONFIG_SMP static void tlb_flush_rmap_batch(struct mmu_gather_batch *batch, struct vm_area_struct *vma) { + struct encoded_page **pages = batch->encoded_pages; + for (int i = 0; i < batch->nr; i++) { - struct encoded_page *enc = batch->encoded_pages[i]; + struct encoded_page *enc = pages[i]; if (encoded_page_flags(enc) & ENCODED_PAGE_BIT_DELAY_RMAP) { struct page *page = encoded_page_ptr(enc); - folio_remove_rmap_pte(page_folio(page), page, vma); + unsigned int nr_pages = 1; + + if (unlikely(encoded_page_flags(enc) & + ENCODED_PAGE_BIT_NR_PAGES_NEXT)) + nr_pages = encoded_nr_pages(pages[++i]); + + folio_remove_rmap_ptes(page_folio(page), page, nr_pages, + vma); } } } @@ -89,18 +98,26 @@ static void tlb_batch_pages_flush(struct mmu_gather *tlb) for (batch = &tlb->local; batch && batch->nr; batch = batch->next) { struct encoded_page **pages = batch->encoded_pages; - do { + while (batch->nr) { /* * limit free batch count when PAGE_SIZE > 4K */ unsigned int nr = min(512U, batch->nr); + /* + * Make sure we cover page + nr_pages, and don't leave + * nr_pages behind when capping the number of entries. + */ + if (unlikely(encoded_page_flags(pages[nr - 1]) & + ENCODED_PAGE_BIT_NR_PAGES_NEXT)) + nr++; + free_pages_and_swap_cache(pages, nr); pages += nr; batch->nr -= nr; cond_resched(); - } while (batch->nr); + } } tlb->active = &tlb->local; } @@ -116,8 +133,9 @@ static void tlb_batch_list_free(struct mmu_gather *tlb) tlb->local.next = NULL; } -bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, - bool delay_rmap, int page_size) +static bool __tlb_remove_folio_pages_size(struct mmu_gather *tlb, + struct page *page, unsigned int nr_pages, bool delay_rmap, + int page_size) { int flags = delay_rmap ? ENCODED_PAGE_BIT_DELAY_RMAP : 0; struct mmu_gather_batch *batch; @@ -126,6 +144,8 @@ bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, #ifdef CONFIG_MMU_GATHER_PAGE_SIZE VM_WARN_ON(tlb->page_size != page_size); + VM_WARN_ON_ONCE(nr_pages != 1 && page_size != PAGE_SIZE); + VM_WARN_ON_ONCE(page_folio(page) != page_folio(page + nr_pages - 1)); #endif batch = tlb->active; @@ -133,17 +153,40 @@ bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, * Add the page and check if we are full. If so * force a flush. */ - batch->encoded_pages[batch->nr++] = encode_page(page, flags); - if (batch->nr == batch->max) { + if (likely(nr_pages == 1)) { + batch->encoded_pages[batch->nr++] = encode_page(page, flags); + } else { + flags |= ENCODED_PAGE_BIT_NR_PAGES_NEXT; + batch->encoded_pages[batch->nr++] = encode_page(page, flags); + batch->encoded_pages[batch->nr++] = encode_nr_pages(nr_pages); + } + /* + * Make sure that we can always add another "page" + "nr_pages", + * requiring two entries instead of only a single one. + */ + if (batch->nr >= batch->max - 1) { if (!tlb_next_batch(tlb)) return true; batch = tlb->active; } - VM_BUG_ON_PAGE(batch->nr > batch->max, page); + VM_BUG_ON_PAGE(batch->nr > batch->max - 1, page); return false; } +bool __tlb_remove_folio_pages(struct mmu_gather *tlb, struct page *page, + unsigned int nr_pages, bool delay_rmap) +{ + return __tlb_remove_folio_pages_size(tlb, page, nr_pages, delay_rmap, + PAGE_SIZE); +} + +bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, + bool delay_rmap, int page_size) +{ + return __tlb_remove_folio_pages_size(tlb, page, 1, delay_rmap, page_size); +} + #endif /* MMU_GATHER_NO_GATHER */ #ifdef CONFIG_MMU_GATHER_TABLE_FREE diff --git a/mm/swap.c b/mm/swap.c index cd8f0150ba3a..e5380d732c0d 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -967,11 +967,17 @@ void release_pages(release_pages_arg arg, int nr) unsigned int lock_batch; for (i = 0; i < nr; i++) { + unsigned int nr_refs = 1; struct folio *folio; /* Turn any of the argument types into a folio */ folio = page_folio(encoded_page_ptr(encoded[i])); + /* Is our next entry actually "nr_pages" -> "nr_refs" ? */ + if (unlikely(encoded_page_flags(encoded[i]) & + ENCODED_PAGE_BIT_NR_PAGES_NEXT)) + nr_refs = encoded_nr_pages(encoded[++i]); + /* * Make sure the IRQ-safe lock-holding time does not get * excessive with a continuous string of pages from the @@ -990,14 +996,14 @@ void release_pages(release_pages_arg arg, int nr) unlock_page_lruvec_irqrestore(lruvec, flags); lruvec = NULL; } - if (put_devmap_managed_page(&folio->page)) + if (put_devmap_managed_page_refs(&folio->page, nr_refs)) continue; - if (folio_put_testzero(folio)) + if (folio_ref_sub_and_test(folio, nr_refs)) free_zone_device_page(&folio->page); continue; } - if (!folio_put_testzero(folio)) + if (!folio_ref_sub_and_test(folio, nr_refs)) continue; if (folio_test_large(folio)) { diff --git a/mm/swap_state.c b/mm/swap_state.c index 7255c01a1e4e..2f540748f7c0 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -311,8 +311,19 @@ void free_page_and_swap_cache(struct page *page) void free_pages_and_swap_cache(struct encoded_page **pages, int nr) { lru_add_drain(); - for (int i = 0; i < nr; i++) - free_swap_cache(encoded_page_ptr(pages[i])); + for (int i = 0; i < nr; i++) { + struct page *page = encoded_page_ptr(pages[i]); + + /* + * Skip over the "nr_pages" entry. It's sufficient to call + * free_swap_cache() only once per folio. + */ + if (unlikely(encoded_page_flags(pages[i]) & + ENCODED_PAGE_BIT_NR_PAGES_NEXT)) + i++; + + free_swap_cache(page); + } release_pages(pages, nr); } From patchwork Fri Feb 9 22:15:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 199133 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:50ea:b0:106:860b:bbdd with SMTP id r10csp1162251dyd; Fri, 9 Feb 2024 14:18:09 -0800 (PST) X-Google-Smtp-Source: AGHT+IERt2klLePoNZ3riQD0WqIxXs9BITmD/RQGAlKqKYo4lstM8aRPDrD4XeKVe6LBxWv7V4Ix X-Received: by 2002:aa7:da51:0:b0:55f:da79:9126 with SMTP id w17-20020aa7da51000000b0055fda799126mr151314eds.34.1707517089335; Fri, 09 Feb 2024 14:18:09 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707517089; cv=pass; d=google.com; s=arc-20160816; b=LaiHZ6E4RVHxlaIBZAikDYlZ9Nn7EHjp5URb9U5lNyE9Tg8RC4SCd7SXg0o8Xf5lJj ur90LkqRJHEz1uV/ljDz7hzURjs0vhBhsCWwB4DbZKOAhC3AcDQRI4pW2AfeShzEvVNA hSkchtjcb8bxvLMbUsGCtN+8fyAlBotqT8QXDDQtQ4HNfTgql3EPvocD5F+B+Z1kk3JT AuvSOklNQdOXiynHTgEVJITqyNFMAgp/1meYnHiJdrWLdr615qakJ/NUmEY/4r5KGm1H QEArOGZ+S6vudJTohAxCq4fWHL/7KXpIQEup1ha4nxoMYfexPouib4zGwFFqVUclsbqE RegA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=m8W53GW4ZEWC3PprgXE/ukTwzuRzol5bFe9xhF3bvjg=; fh=9tvc7SVs9sBrnZk6qwA2/HxAT6T9KzL2iLcdlqeQt/A=; b=W9ZYh0R+egP+ooJN3IKHUTW7TKr4nhjHcFLURU/WWtOgfs5OVPHUFD1pKSgH04sY8o fGYt6o1payBI1oWWWMMDKcq5ataJAn7mTc73MU1KCACa/BlhbHsn/HdKcjL4igqpL7WZ nfAGWIH/LyPuLYVE05+bJ428cK0jk/Kmj2Ntix3Il/IPZRRKsDXYHoQJiweR5rIvqt+w VAyv5HqCVG8AGHBinrNqxzDnPESWpbm9za/JUxaQGmWzDzo3Rr/dylC9+YuYyZ2Vt2v/ T9dlzPa4tPGog0Dh8yPCpEGUXEjBhFUSbLRgKY9L8tRXekiLsJSJd/QHm1kW47I8DisK VOdg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KWGv6VNe; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60081-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60081-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Forwarded-Encrypted: i=2; AJvYcCUx1iWOm+sgNxlptGJ2QlOyQFHnYnKfZvtn8YzhD6hdg1tNdKwtrRkcF9usY6zDIWutf4+Rgp/X6xdorDV57yCTG2Jfkg== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id h19-20020a056402281300b0056148579524si174252ede.327.2024.02.09.14.18.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 14:18:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-60081-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KWGv6VNe; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60081-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60081-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id C24391F21501 for ; Fri, 9 Feb 2024 22:18:08 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 64D3739FFB; Fri, 9 Feb 2024 22:16:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="KWGv6VNe" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6F4239FCF for ; Fri, 9 Feb 2024 22:16:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516970; cv=none; b=iMh34OmW1Lz7AuoxwugLUjPiGfAQJ88MeD9QY2VNgkp/xLLzV68vspwhu84fxNH972ztMWZxdMfOuiJfQC9J2q8sSqe+ydl602Y71D+hswuaTz9FqyosFXfDpdnGukDOj1DLSk9l7T8p1GUJ46Qtr/tL6Rgni8+QrfQXM9qW/OY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516970; c=relaxed/simple; bh=/5mbH+9wsrYjqsJ0BTPMr7p8fqUtmFZU4UxswK+eIVc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WgyIcb1oKN8hraQ7+PFIJddBLZlw5RMnUqi9C8mnGvvFNHWDYBOu3H2JSM/AVnPFk62rqwE9Z1rZsBRWtAood1q+A2lVgx27BF7GimVCFuSXZbd1xdC72p/DwrmYEKtPnZOeNLA8uxvo8WGkCd6yHmCIPDLAmzkGDsqf07daixI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=KWGv6VNe; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1707516967; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=m8W53GW4ZEWC3PprgXE/ukTwzuRzol5bFe9xhF3bvjg=; b=KWGv6VNe1qwKYZH4ouyIx8BCMeZLXRyOykz6QqhP0wfZQMc+viANG4NiaffJ4rzf89cIuZ H+329RWgRMBLtGTaLUfoyJBO32j8tsO+dhtUhHHFJ0/+yvBaaiZWB4zazEw3M7FPOrFMY5 rkEzMCT+e3U4H9eR3RYO0RDaCrf1R7w= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-60-t2V2UsG2PJSAXQJ2V6pbnw-1; Fri, 09 Feb 2024 17:16:04 -0500 X-MC-Unique: t2V2UsG2PJSAXQJ2V6pbnw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BE704837230; Fri, 9 Feb 2024 22:16:02 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.194.59]) by smtp.corp.redhat.com (Postfix) with ESMTP id E26B01C14B04; Fri, 9 Feb 2024 22:15:56 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Yin Fengwei , Michal Hocko , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v2 09/10] mm/mmu_gather: improve cond_resched() handling with large folios and expensive page freeing Date: Fri, 9 Feb 2024 23:15:08 +0100 Message-ID: <20240209221509.585251-10-david@redhat.com> In-Reply-To: <20240209221509.585251-1-david@redhat.com> References: <20240209221509.585251-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790461439713639513 X-GMAIL-MSGID: 1790461439713639513 It's a pain that we have to handle cond_resched() in tlb_batch_pages_flush() manually and cannot simply handle it in release_pages() -- release_pages() can be called from atomic context. Well, in a perfect world we wouldn't have to make our code more at all. With page poisoning and init_on_free, we might now run into soft lockups when we free a lot of rather large folio fragments, because page freeing time then depends on the actual memory size we are freeing instead of on the number of folios that are involved. In the absolute (unlikely) worst case, on arm64 with 64k we will be able to free up to 256 folio fragments that each span 512 MiB: zeroing out 128 GiB does sound like it might take a while. But instead of ignoring this unlikely case, let's just handle it. So, let's teach tlb_batch_pages_flush() that there are some configurations where page freeing is horribly slow, and let's reschedule more frequently -- similarly like we did for now before we had large folio fragments in there. Note that we might end up freeing only a single folio fragment at a time that might exceed the old 512 pages limit: but if we cannot even free a single MAX_ORDER page on a system without running into soft lockups, something else is already completely bogus. In the future, we might want to detect if handling cond_resched() is required at all, and just not do any of that with full preemption enabled. Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand --- mm/mmu_gather.c | 50 ++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 41 insertions(+), 9 deletions(-) diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index d175c0f1e2c8..2774044b5790 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -91,18 +91,19 @@ void tlb_flush_rmaps(struct mmu_gather *tlb, struct vm_area_struct *vma) } #endif -static void tlb_batch_pages_flush(struct mmu_gather *tlb) +static void __tlb_batch_free_encoded_pages(struct mmu_gather_batch *batch) { - struct mmu_gather_batch *batch; - - for (batch = &tlb->local; batch && batch->nr; batch = batch->next) { - struct encoded_page **pages = batch->encoded_pages; + struct encoded_page **pages = batch->encoded_pages; + unsigned int nr, nr_pages; + /* + * We might end up freeing a lot of pages. Reschedule on a regular + * basis to avoid soft lockups in configurations without full + * preemption enabled. The magic number of 512 folios seems to work. + */ + if (!page_poisoning_enabled_static() && !want_init_on_free()) { while (batch->nr) { - /* - * limit free batch count when PAGE_SIZE > 4K - */ - unsigned int nr = min(512U, batch->nr); + nr = min(512, batch->nr); /* * Make sure we cover page + nr_pages, and don't leave @@ -119,6 +120,37 @@ static void tlb_batch_pages_flush(struct mmu_gather *tlb) cond_resched(); } } + + /* + * With page poisoning and init_on_free, the time it takes to free + * memory grows proportionally with the actual memory size. Therefore, + * limit based on the actual memory size and not the number of involved + * folios. + */ + while (batch->nr) { + for (nr = 0, nr_pages = 0; + nr < batch->nr && nr_pages < 512; nr++) { + if (unlikely(encoded_page_flags(pages[nr]) & + ENCODED_PAGE_BIT_NR_PAGES_NEXT)) + nr_pages += encoded_nr_pages(pages[++nr]); + else + nr_pages++; + } + + free_pages_and_swap_cache(pages, nr); + pages += nr; + batch->nr -= nr; + + cond_resched(); + } +} + +static void tlb_batch_pages_flush(struct mmu_gather *tlb) +{ + struct mmu_gather_batch *batch; + + for (batch = &tlb->local; batch && batch->nr; batch = batch->next) + __tlb_batch_free_encoded_pages(batch); tlb->active = &tlb->local; } From patchwork Fri Feb 9 22:15:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 199134 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:50ea:b0:106:860b:bbdd with SMTP id r10csp1162447dyd; Fri, 9 Feb 2024 14:18:39 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCU4shL09lxiS7CCMytG3ZmUkIXR8Byu4K7HvHXrT87gaVVAfDVXcp18TLpG8vLFMXDCsKxSiJ9YTZcF5VWCjUbaqaLc5Q== X-Google-Smtp-Source: AGHT+IF22MlrST05JTzBWI1yeeTiP2Az/WZKW8NLMS8Xg9xxES8SEFH1voq1WesdJV2yhpbNSnAf X-Received: by 2002:a0c:dc8d:0:b0:68c:9109:b5fc with SMTP id n13-20020a0cdc8d000000b0068c9109b5fcmr405751qvk.53.1707517119536; Fri, 09 Feb 2024 14:18:39 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707517119; cv=pass; d=google.com; s=arc-20160816; b=GDRky9OC+Q9+ItwZ+79kb/Qln4MVq/0JqRh4Lg84YF6CWpSdnK8Lea854Dhbm90tPN AzwPmB8vEFHMwmn2ywjb6dXX/H3YFX36+UpPB79e7ENk0iYEwQlPl2SKrEPaPAP5os+/ JPgqaI3AgW6MwsF35T7y7iQwL97oO4AUB0i+21KzvvnqY4QOJTLClcf0ej8/5oevqxO9 V6tUHGpLt6nBHLaKpCokg2gxeN727E+uS7JYmgGjpJjIRtTMi7AIfK1yFSKGqV/0zJyO kdPHCsMQIrtc4LD2Bc0iJQb+W4XAumdpjAKeOKIELx4sOCiuSgJRDwuYHzSA4zRV4IYc lr/w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=KrYYgTjhMX4rJEhHUeQn0YNRSL+kwvv2YOYlgfS0XZc=; fh=bT56Y6fIv38BEZgxMb9B/6d4L0WH8LHbatY+DtAQItY=; b=b3r5ZrPcCQMR9acLNvpcMSqeacHTxWrSalk+OZ8N0hM+dCMU7QzKkHpFOhpdYIM/+2 ju35YFSO04cpgN7uEZoSXQ9CUjSbWdda/rR7vQ6iNLciE7zZJL5SwyAgiCQbpdUIWZG4 lMphngfy3MwK2o/ng14SlQnfQLS9Ez9d2a45DVBvT3okmIgid2LhcRG4XfPgFDy9sZZr cHAUvgM2am+NYEk4UWJ+voOiuvTQ759GTSrcQvedEuGiNh96vF759b2MIUwnV1DjOVMw tdBBmkBZG9+G2fFSBTh4yPFIhaYQGxARrEHW5Mo3MgDTqL8wbL8gqlOezO9yHQsUGQ1F FvSg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=WAks4gjd; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60083-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60083-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Forwarded-Encrypted: i=2; AJvYcCXM5YM86bL7TFAF/d15fh5MNOHXK9sksvrH8PNsiheH7ZlrFsHAE9pgRMSP8xT9I48pymYhcyX4N2Fi5MSxpJqHc+Jy0A== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id gc11-20020a056214230b00b0068ccc5108b1si3145792qvb.405.2024.02.09.14.18.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 14:18:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-60083-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=WAks4gjd; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-60083-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60083-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 426BB1C21F4B for ; Fri, 9 Feb 2024 22:18:39 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2BB8D24A18; Fri, 9 Feb 2024 22:16:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="WAks4gjd" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D77C3A1C8 for ; Fri, 9 Feb 2024 22:16:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516977; cv=none; b=WTt8HEgY/fdYPxT+wPTAXLPiouvNdJVKJjk5G2XX9r2E33nVvFLjiXsay/XUbmI9lBW3bfqGmzRtKAgz0djfg3OfmGOcnSG/Voy0n6s1sR7cdyMb1TSMcd01ip9bZysuWsU0uJOGWKouPjZvmrlH28cmrdMDEdHRVk92jL0yxR0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707516977; c=relaxed/simple; bh=KDzmcfIFT7UKzeU/BDiDhtxfHD9pMkoW5n8aWMJ5xkQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GytXtKXoVq1HzISbwVfxNDp89jXe1MkVokHG/CrKD8BBNFtnjUfFk6L8vplSN8VgVV7I/nwa/yTd+pOafzqxprnKEtOq6ZdE4sIEP303WagrU7HRLDpvsx+7n+ibASw84tynoW2HyMs0s+lOEbxqbg/m0wFmu5ydqSk+Eukviqk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=WAks4gjd; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1707516974; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KrYYgTjhMX4rJEhHUeQn0YNRSL+kwvv2YOYlgfS0XZc=; b=WAks4gjdg1fAT07rApsoRN1vp9MyJ1XP98mgqFutnwAuv1uv/M6u57f97SGUl53n51F+yH Jn/2oyvSImVnNc0Fc/rSD6Nx97bgOh6fHGFmhUiHm+4bHRcm2igNQKNwlTRNrH4KGsaDtz eTejwACP7chAaFdK3mFckfWot2PjoLU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-508-uldKMFUyNS60AI0e5y5IZw-1; Fri, 09 Feb 2024 17:16:08 -0500 X-MC-Unique: uldKMFUyNS60AI0e5y5IZw-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 786F71025620; Fri, 9 Feb 2024 22:16:07 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.194.59]) by smtp.corp.redhat.com (Postfix) with ESMTP id 218D01C14B0D; Fri, 9 Feb 2024 22:16:03 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Yin Fengwei , Michal Hocko , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v2 10/10] mm/memory: optimize unmap/zap with PTE-mapped THP Date: Fri, 9 Feb 2024 23:15:09 +0100 Message-ID: <20240209221509.585251-11-david@redhat.com> In-Reply-To: <20240209221509.585251-1-david@redhat.com> References: <20240209221509.585251-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790461286877889712 X-GMAIL-MSGID: 1790461471298334189 Similar to how we optimized fork(), let's implement PTE batching when consecutive (present) PTEs map consecutive pages of the same large folio. Most infrastructure we need for batching (mmu gather, rmap) is already there. We only have to add get_and_clear_full_ptes() and clear_full_ptes(). Similarly, extend zap_install_uffd_wp_if_needed() to process a PTE range. We won't bother sanity-checking the mapcount of all subpages, but only check the mapcount of the first subpage we process. If there is a real problem hiding somewhere, we can trigger it simply by using small folios, or when we zap single pages of a large folio. Ideally, we had that check in rmap code (including for delayed rmap), but then we cannot print the PTE. Let's keep it simple for now. If we ever have a cheap folio_mapcount(), we might just want to check for underflows there. To keep small folios as fast as possible force inlining of a specialized variant using __always_inline with nr=1. Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- include/linux/pgtable.h | 70 +++++++++++++++++++++++++++++++ mm/memory.c | 92 +++++++++++++++++++++++++++++------------ 2 files changed, 136 insertions(+), 26 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index aab227e12493..49ab1f73b5c2 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -580,6 +580,76 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, } #endif +#ifndef get_and_clear_full_ptes +/** + * get_and_clear_full_ptes - Clear present PTEs that map consecutive pages of + * the same folio, collecting dirty/accessed bits. + * @mm: Address space the pages are mapped into. + * @addr: Address the first page is mapped at. + * @ptep: Page table pointer for the first entry. + * @nr: Number of entries to clear. + * @full: Whether we are clearing a full mm. + * + * May be overridden by the architecture; otherwise, implemented as a simple + * loop over ptep_get_and_clear_full(), merging dirty/accessed bits into the + * returned PTE. + * + * Note that PTE bits in the PTE range besides the PFN can differ. For example, + * some PTEs might be write-protected. + * + * Context: The caller holds the page table lock. The PTEs map consecutive + * pages that belong to the same folio. The PTEs are all in the same PMD. + */ +static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, unsigned int nr, int full) +{ + pte_t pte, tmp_pte; + + pte = ptep_get_and_clear_full(mm, addr, ptep, full); + while (--nr) { + ptep++; + addr += PAGE_SIZE; + tmp_pte = ptep_get_and_clear_full(mm, addr, ptep, full); + if (pte_dirty(tmp_pte)) + pte = pte_mkdirty(pte); + if (pte_young(tmp_pte)) + pte = pte_mkyoung(pte); + } + return pte; +} +#endif + +#ifndef clear_full_ptes +/** + * clear_full_ptes - Clear present PTEs that map consecutive pages of the same + * folio. + * @mm: Address space the pages are mapped into. + * @addr: Address the first page is mapped at. + * @ptep: Page table pointer for the first entry. + * @nr: Number of entries to clear. + * @full: Whether we are clearing a full mm. + * + * May be overridden by the architecture; otherwise, implemented as a simple + * loop over ptep_get_and_clear_full(). + * + * Note that PTE bits in the PTE range besides the PFN can differ. For example, + * some PTEs might be write-protected. + * + * Context: The caller holds the page table lock. The PTEs map consecutive + * pages that belong to the same folio. The PTEs are all in the same PMD. + */ +static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + for (;;) { + ptep_get_and_clear_full(mm, addr, ptep, full); + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + } +} +#endif /* * If two threads concurrently fault at the same page, the thread that diff --git a/mm/memory.c b/mm/memory.c index a3efc4da258a..3b8e56eb08a3 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1515,7 +1515,7 @@ static inline bool zap_drop_file_uffd_wp(struct zap_details *details) */ static inline void zap_install_uffd_wp_if_needed(struct vm_area_struct *vma, - unsigned long addr, pte_t *pte, + unsigned long addr, pte_t *pte, int nr, struct zap_details *details, pte_t pteval) { /* Zap on anonymous always means dropping everything */ @@ -1525,20 +1525,27 @@ zap_install_uffd_wp_if_needed(struct vm_area_struct *vma, if (zap_drop_file_uffd_wp(details)) return; - pte_install_uffd_wp_if_needed(vma, addr, pte, pteval); + for (;;) { + /* the PFN in the PTE is irrelevant. */ + pte_install_uffd_wp_if_needed(vma, addr, pte, pteval); + if (--nr == 0) + break; + pte++; + addr += PAGE_SIZE; + } } -static inline void zap_present_folio_pte(struct mmu_gather *tlb, +static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb, struct vm_area_struct *vma, struct folio *folio, - struct page *page, pte_t *pte, pte_t ptent, unsigned long addr, - struct zap_details *details, int *rss, bool *force_flush, - bool *force_break) + struct page *page, pte_t *pte, pte_t ptent, unsigned int nr, + unsigned long addr, struct zap_details *details, int *rss, + bool *force_flush, bool *force_break) { struct mm_struct *mm = tlb->mm; bool delay_rmap = false; if (!folio_test_anon(folio)) { - ptent = ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); + ptent = get_and_clear_full_ptes(mm, addr, pte, nr, tlb->fullmm); if (pte_dirty(ptent)) { folio_mark_dirty(folio); if (tlb_delay_rmap(tlb)) { @@ -1548,36 +1555,49 @@ static inline void zap_present_folio_pte(struct mmu_gather *tlb, } if (pte_young(ptent) && likely(vma_has_recency(vma))) folio_mark_accessed(folio); - rss[mm_counter(folio)]--; + rss[mm_counter(folio)] -= nr; } else { /* We don't need up-to-date accessed/dirty bits. */ - ptep_get_and_clear_full(mm, addr, pte, tlb->fullmm); - rss[MM_ANONPAGES]--; + clear_full_ptes(mm, addr, pte, nr, tlb->fullmm); + rss[MM_ANONPAGES] -= nr; } + /* Checking a single PTE in a batch is sufficient. */ arch_check_zapped_pte(vma, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); + tlb_remove_tlb_entries(tlb, pte, nr, addr); if (unlikely(userfaultfd_pte_wp(vma, ptent))) - zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); + zap_install_uffd_wp_if_needed(vma, addr, pte, nr, details, + ptent); if (!delay_rmap) { - folio_remove_rmap_pte(folio, page, vma); + folio_remove_rmap_ptes(folio, page, nr, vma); + + /* Only sanity-check the first page in a batch. */ if (unlikely(page_mapcount(page) < 0)) print_bad_pte(vma, addr, ptent, page); } - if (unlikely(__tlb_remove_page(tlb, page, delay_rmap))) { + if (unlikely(__tlb_remove_folio_pages(tlb, page, nr, delay_rmap))) { *force_flush = true; *force_break = true; } } -static inline void zap_present_pte(struct mmu_gather *tlb, +/* + * Zap or skip at least one present PTE, trying to batch-process subsequent + * PTEs that map consecutive pages of the same folio. + * + * Returns the number of processed (skipped or zapped) PTEs (at least 1). + */ +static inline int zap_present_ptes(struct mmu_gather *tlb, struct vm_area_struct *vma, pte_t *pte, pte_t ptent, - unsigned long addr, struct zap_details *details, - int *rss, bool *force_flush, bool *force_break) + unsigned int max_nr, unsigned long addr, + struct zap_details *details, int *rss, bool *force_flush, + bool *force_break) { + const fpb_t fpb_flags = FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; struct mm_struct *mm = tlb->mm; struct folio *folio; struct page *page; + int nr; page = vm_normal_page(vma, addr, ptent); if (!page) { @@ -1587,14 +1607,29 @@ static inline void zap_present_pte(struct mmu_gather *tlb, tlb_remove_tlb_entry(tlb, pte, addr); VM_WARN_ON_ONCE(userfaultfd_wp(vma)); ksm_might_unmap_zero_page(mm, ptent); - return; + return 1; } folio = page_folio(page); if (unlikely(!should_zap_folio(details, folio))) - return; - zap_present_folio_pte(tlb, vma, folio, page, pte, ptent, addr, details, - rss, force_flush, force_break); + return 1; + + /* + * Make sure that the common "small folio" case is as fast as possible + * by keeping the batching logic separate. + */ + if (unlikely(folio_test_large(folio) && max_nr != 1)) { + nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, + NULL); + + zap_present_folio_ptes(tlb, vma, folio, page, pte, ptent, nr, + addr, details, rss, force_flush, + force_break); + return nr; + } + zap_present_folio_ptes(tlb, vma, folio, page, pte, ptent, 1, addr, + details, rss, force_flush, force_break); + return 1; } static unsigned long zap_pte_range(struct mmu_gather *tlb, @@ -1609,6 +1644,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, pte_t *start_pte; pte_t *pte; swp_entry_t entry; + int nr; tlb_change_page_size(tlb, PAGE_SIZE); init_rss_vec(rss); @@ -1622,7 +1658,9 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, pte_t ptent = ptep_get(pte); struct folio *folio; struct page *page; + int max_nr; + nr = 1; if (pte_none(ptent)) continue; @@ -1630,10 +1668,12 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, break; if (pte_present(ptent)) { - zap_present_pte(tlb, vma, pte, ptent, addr, details, - rss, &force_flush, &force_break); + max_nr = (end - addr) / PAGE_SIZE; + nr = zap_present_ptes(tlb, vma, pte, ptent, max_nr, + addr, details, rss, &force_flush, + &force_break); if (unlikely(force_break)) { - addr += PAGE_SIZE; + addr += nr * PAGE_SIZE; break; } continue; @@ -1687,8 +1727,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, WARN_ON_ONCE(1); } pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); - zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); - } while (pte++, addr += PAGE_SIZE, addr != end); + zap_install_uffd_wp_if_needed(vma, addr, pte, 1, details, ptent); + } while (pte += nr, addr += PAGE_SIZE * nr, addr != end); add_mm_rss_vec(mm, rss); arch_leave_lazy_mmu_mode();