From patchwork Fri Jun 9 01:17:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 105207 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp650752vqr; Thu, 8 Jun 2023 18:35:36 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7pD4HVKYyv5F4gVZKl1F+D0uQLwrXQkdXv87qpu7YorqE8VmLCHdEFkBJO97mt7aTVnrrU X-Received: by 2002:a05:6358:e803:b0:129:d3c0:7339 with SMTP id gi3-20020a056358e80300b00129d3c07339mr130730rwb.0.1686274535843; Thu, 08 Jun 2023 18:35:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1686274535; cv=none; d=google.com; s=arc-20160816; b=T/dKMzglUskjiwmHblXuceyD/rKq98je1BKj6auZohNpxMNaXPkLSBEkDCEefQ/oPb yyludluB6PzVC3R6kvQwyRSjKvuyxzgr5wWVZf8T+3aBnmB1ghqiWI1RlzX4idFrZgH8 MvnIqxNJ5CXFKUnsKwoohES0ybrmm4FY6JSFrcNJeZiWt8xyKf9qWoxYJ36y1oX7TBr9 ZRlxFJhOyx3rU3WADEusmZpou3W9t0IUFtm9JhIpI3NIVSwF3H0aTl6EvFJq7WpDHg90 C62PbOqI7p8foDhmjL5I1c7K5g+hYnJr2tDGcEaRYVqvrSM+6EpU4IXcBN2xyHh3LzO5 DYCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:message-id:in-reply-to :subject:cc:to:from:date:dkim-signature; bh=YGxXR2sylNgXOWjswYon8LmdFOYpbwThTVM1QZPxixo=; b=txL2MnTSFOOFkvTxJAfn+LKEG7GR3l9yHTtEweYLRlSGcvqMccX891MJhu7P6ZRG0V 5ZImiLPojjnWnld35ObcNLmcesQz50wWly46vCklCoIwzaWmEE6fdVSaFbSBvKqLk4uk ZlPqGS+csez46BKvUcawsLEqy+YwKpw82+bRi/pFKPyBE6+XW10vLV18yC4Gdq2f5fnA 4Rum2pUXGmsXfNfFy1gi7SlI7pJIR9k7lm5zUO2Xfxmj8wQpp6Gg17E6/mvum33Ie1+0 5ADK2/SOJnEekrRk3dcueUdHE6MeqVy54pjlz1eGaIxRuLPKbxbNdZKPlaJxseOywW4V 0FZQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b="hfFZS/To"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b2-20020a639302000000b00543c1b2a326si1849026pge.177.2023.06.08.18.35.23; Thu, 08 Jun 2023 18:35:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b="hfFZS/To"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237455AbjFIBRe (ORCPT + 99 others); Thu, 8 Jun 2023 21:17:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36490 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229458AbjFIBRc (ORCPT ); Thu, 8 Jun 2023 21:17:32 -0400 Received: from mail-yw1-x1133.google.com (mail-yw1-x1133.google.com [IPv6:2607:f8b0:4864:20::1133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B2B9419AC for ; Thu, 8 Jun 2023 18:17:31 -0700 (PDT) Received: by mail-yw1-x1133.google.com with SMTP id 00721157ae682-568900c331aso12056237b3.3 for ; Thu, 08 Jun 2023 18:17:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686273451; x=1688865451; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=YGxXR2sylNgXOWjswYon8LmdFOYpbwThTVM1QZPxixo=; b=hfFZS/To38y/kHK62NcQe3NeUtPQadIbTbMGv9CwCwnaOm3+ubft1ovMBDYwgIcJHN maH31bmCKIXTcoH2mw+7AlwbT//z013fnJricqAgo1Y8wR2cLWz5jidfA/WWWdNn++DD eExfA1ytyJkqY/La00m6AnHJyexbZrQSpOFK5OQ6AFOk6cyYk4BIHtO7IYbubUELYE5/ t2x0w817NAACm7JQwyi+uQozGB37VzsQ0dIb5qTadGN0KMsSpWs2GUyHK1Uhm4UV4EVa KWOR3EiG1vfrWrdRdz5JYnJcT4eEEx83Gbf/GYzP39cr1/xlfqJ/0ZZkApmwpKDFSRuJ KEIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686273451; x=1688865451; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YGxXR2sylNgXOWjswYon8LmdFOYpbwThTVM1QZPxixo=; b=OyCBFfXBk68SoX5I76/UqS6rEIeaOMqYCHFTsOdXQYKgCOqNXubtrWq05rM8GIprOI ykgnwMb6EppVLL7vD/6vo9RmzgkIictS3JZ8V3gfnICd3RWuLfC1TKuOixFkJldVBq7w /i65uzq4/bZ7aT3FvPk8KxzlcYkCT9DJXBweORS9eKKfEuRhcpuHIIEjxsDAg50efl1c VC/dj1+BibrLQPS+E11WN0dfDRN5eM9Y+PyDxJDsF1R+S1x2d27asYdBhWKkm3O0AXNF RB9YAnTNZ4/+PPgjHdojYDJro/OkhF7+/eVCRNk123YieAvYdhGr1GtR/qHkvljiqyg2 Nctg== X-Gm-Message-State: AC+VfDwcsYPuDjWvIho1pnyTHqnStFCi4tXaUKmPfAo/UzeqdeKlauav eQQxLhNHMzIVynlm7gSMOyMrfA== X-Received: by 2002:a81:6cd5:0:b0:564:c4db:631e with SMTP id h204-20020a816cd5000000b00564c4db631emr1257858ywc.5.1686273450756; Thu, 08 Jun 2023 18:17:30 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id p131-20020a817489000000b00560beb1c97bsm287394ywc.97.2023.06.08.18.17.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Jun 2023 18:17:30 -0700 (PDT) Date: Thu, 8 Jun 2023 18:17:26 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Lorenzo Stoakes , Huang Ying , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v2 09/32] mm/pagewalkers: ACTION_AGAIN if pte_offset_map_lock() fails In-Reply-To: Message-ID: References: MIME-Version: 1.0 X-Spam-Status: No, score=-17.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1768187007291077482?= X-GMAIL-MSGID: =?utf-8?q?1768187007291077482?= Simple walk_page_range() users should set ACTION_AGAIN to retry when pte_offset_map_lock() fails. No need to check pmd_trans_unstable(): that was precisely to avoid the possiblity of calling pte_offset_map() on a racily removed or inserted THP entry, but such cases are now safely handled inside it. Likewise there is no need to check pmd_none() or pmd_bad() before calling it. Signed-off-by: Hugh Dickins Reviewed-by: SeongJae Park for mm/damon part --- fs/proc/task_mmu.c | 32 ++++++++++++++++---------------- mm/damon/vaddr.c | 12 ++++++++---- mm/mempolicy.c | 7 ++++--- mm/mincore.c | 9 ++++----- mm/mlock.c | 4 ++++ 5 files changed, 36 insertions(+), 28 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 420510f6a545..dba5052ce09b 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -631,14 +631,11 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, goto out; } - if (pmd_trans_unstable(pmd)) - goto out; - /* - * The mmap_lock held all the way back in m_start() is what - * keeps khugepaged out of here and from collapsing things - * in here. - */ pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; pte++, addr += PAGE_SIZE) smaps_pte_entry(pte, addr, walk); pte_unmap_unlock(pte - 1, ptl); @@ -1191,10 +1188,11 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr, return 0; } - if (pmd_trans_unstable(pmd)) - return 0; - pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; pte++, addr += PAGE_SIZE) { ptent = *pte; @@ -1538,9 +1536,6 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, spin_unlock(ptl); return err; } - - if (pmd_trans_unstable(pmdp)) - return 0; #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ /* @@ -1548,6 +1543,10 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, * goes beyond vma->vm_end. */ orig_pte = pte = pte_offset_map_lock(walk->mm, pmdp, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return err; + } for (; addr < end; pte++, addr += PAGE_SIZE) { pagemap_entry_t pme; @@ -1887,11 +1886,12 @@ static int gather_pte_stats(pmd_t *pmd, unsigned long addr, spin_unlock(ptl); return 0; } - - if (pmd_trans_unstable(pmd)) - return 0; #endif orig_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } do { struct page *page = can_gather_numa_stats(*pte, vma, addr); if (!page) diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 1fec16d7263e..b8762ff15c3c 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -318,9 +318,11 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr, spin_unlock(ptl); } - if (pmd_none(*pmd) || unlikely(pmd_bad(*pmd))) - return 0; pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } if (!pte_present(*pte)) goto out; damon_ptep_mkold(pte, walk->mm, addr); @@ -464,9 +466,11 @@ static int damon_young_pmd_entry(pmd_t *pmd, unsigned long addr, regular_page: #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ - if (pmd_none(*pmd) || unlikely(pmd_bad(*pmd))) - return -EINVAL; pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } if (!pte_present(*pte)) goto out; folio = damon_get_folio(pte_pfn(*pte)); diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 1756389a0609..4d0bcf6f0d52 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -514,10 +514,11 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, if (ptl) return queue_folios_pmd(pmd, ptl, addr, end, walk); - if (pmd_trans_unstable(pmd)) - return 0; - mapped_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; pte++, addr += PAGE_SIZE) { if (!pte_present(*pte)) continue; diff --git a/mm/mincore.c b/mm/mincore.c index 2d5be013a25a..f33f6a0b1ded 100644 --- a/mm/mincore.c +++ b/mm/mincore.c @@ -113,12 +113,11 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, goto out; } - if (pmd_trans_unstable(pmd)) { - __mincore_unmapped_range(addr, end, vma, vec); - goto out; - } - ptep = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!ptep) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; ptep++, addr += PAGE_SIZE) { pte_t pte = *ptep; diff --git a/mm/mlock.c b/mm/mlock.c index 40b43f8740df..9f2b1173b1b1 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -329,6 +329,10 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, } start_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!start_pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (pte = start_pte; addr != end; pte++, addr += PAGE_SIZE) { if (!pte_present(*pte)) continue;