From patchwork Tue Aug 1 22:07:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 129530 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f41:0:b0:3e4:2afc:c1 with SMTP id v1csp100611vqx; Tue, 1 Aug 2023 17:20:59 -0700 (PDT) X-Google-Smtp-Source: APBJJlGYZ2RqoLKE5ESjTI1Qf6hexOnt4scljPWeWo0aaF7Jwum6G+T6+JM6DKsCYEaqlSE88v8M X-Received: by 2002:a9d:6c97:0:b0:6b8:7e53:e7c3 with SMTP id c23-20020a9d6c97000000b006b87e53e7c3mr13635139otr.31.1690935659385; Tue, 01 Aug 2023 17:20:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690935659; cv=none; d=google.com; s=arc-20160816; b=q0zMRW5op/Urhkh1hoGb7P1sxTzKprmOLEU39bb2DF2rWGZ5aTu4Wk8IzdQQzzWskG wOJUjy7Z0vnmAiE9GH3levSODgylh5tT0p6I/QLX87o5R60VfCHN3dF6f7uNXxuYsDj8 didvXKQXkNn7+cKZs53SBDTEf93N7djkkk5mudNR0M3ZykDJcAxRKrDMfjJSTpS7iWoV 8nvLSuxMBKE9IHEBaQRAJOENsuV51uZjbIWzZ/dAqIMZQpwhqyUEsYATO3mg5PmD8QdG dhDwI6+uQj1XEj0zYfHsDVhCepW8GrKfJrFOWuptOPfsIJKaBt5HmS1DAEdM0MQzruMy +/Zw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=71RLrK6Mb0aadLJnf8lNJkVJrOaKPnZ5THHnlI7+5Ec=; fh=Dw2MP27nuXcy0JixWzXZPVDqCPGddu/JRhZXxi83XxI=; b=NCPlhtIhUmcaVhCNBn4rEwJqFlHw8rb22c2ihlS/vnVjdNR0D/Ohqb9wp/OnSZizPJ bURAzqhynNfbqnVzB+YjGS7zIM3/sP6sdrlAvyX8QkH8vv/ZVKdrQXTQS85RL5tkqLPi C6NE5W9h7YSByu340wqVnq59UJYtn3OhpKB3dpRzUxLxf1uv3J+JEVFaRHfSQ6BOHAeg lDSL7ZUD2FbETzV9Lcf37EWLuTIAXXMXl4goniuziM1qmUDJIV3v+avwi0DkIrb0x3TJ bvmuaEiX5Q67OvxrkC5lLs71eEFqANm6Xxj5zk+nX8LkwSU3MBt6ZAVB+259hABdKM5V reUA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=We6ajMOj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j192-20020a638bc9000000b0053f3b62c207si9700358pge.767.2023.08.01.17.20.27; Tue, 01 Aug 2023 17:20:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=We6ajMOj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231466AbjHAWHr (ORCPT + 99 others); Tue, 1 Aug 2023 18:07:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33952 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230256AbjHAWHl (ORCPT ); Tue, 1 Aug 2023 18:07:41 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6ECB119BE for ; Tue, 1 Aug 2023 15:07:39 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5840614b107so75692987b3.1 for ; Tue, 01 Aug 2023 15:07:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690927658; x=1691532458; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=71RLrK6Mb0aadLJnf8lNJkVJrOaKPnZ5THHnlI7+5Ec=; b=We6ajMOjm3XiLm7Fm134AiRtQwnSaKR7S3ytCgA240I8V4Cqb5DZkxb3c4VMaCqc0w V9+bLiaYnBwtT7C9b9kFVxy1bS64OrsWrAPLxZe0GmEClT2rmBC2Z2xa+eQmQ5FWLxHC yKOrz9bPWJ1QsP/sCqkL5hEzAEJ4p0Oq7uGE6QZf2POo/0NInGTznuALO13yDaRSnc2p oZZzhAMRwlN0xh4Oa0sVS9pWPrSw8B9WQU8aZavQe2FOuTA14a9EsXtVQo3/CWEI9VAF I/xuYpBEFF/+CKYPkLJOsJbz86ql23LK4Gz+qkaslMFUpxQhmC5GMs/A1z5mZyNEbgLj 4MtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690927658; x=1691532458; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=71RLrK6Mb0aadLJnf8lNJkVJrOaKPnZ5THHnlI7+5Ec=; b=LmvxBdwTId05A/zECdcatTJmA5K4tdCX2K2adGva/RiJFvbMoJ+DHUQPz1I8A/wGEZ NhAlvjy9UR/rCrY4EvTw82w/znIUepCrHoQh1LQiKKkiqR2kEfZu04WKuwwgzgN6HAzg YsVnhXSKmN+l3f/iA53gzYOinRjZYB1jidNh3H5tS0fsfwsfmgNUwbHCVAnPHwNGcF4/ /pVaIRux9Rua1J3ELn+/ka/Wpo9oYqmA2plxk9zhG0VFMx63s74HJ/svejabQOCiyv3e +z4Y7dV42IXTbP1eE67Eo+enf/vstNqsMmRmrISQBJKT0TjuYPfPL64YuTx9zp4I6dbs XqOw== X-Gm-Message-State: ABy/qLYJIRW9EplhNNzhsyP68LNHkW7Hf6os+DQ6JkZst5mI4UdEewje uOCK3HQPdXRRAp7YxZupV5VKRrWXmN8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:211c:a2ff:f17b:c5e9]) (user=surenb job=sendgmr) by 2002:a25:f89:0:b0:d0b:c67:de3b with SMTP id 131-20020a250f89000000b00d0b0c67de3bmr92009ybp.13.1690927658714; Tue, 01 Aug 2023 15:07:38 -0700 (PDT) Date: Tue, 1 Aug 2023 15:07:27 -0700 In-Reply-To: <20230801220733.1987762-1-surenb@google.com> Mime-Version: 1.0 References: <20230801220733.1987762-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.585.gd2178a4bd4-goog Message-ID: <20230801220733.1987762-2-surenb@google.com> Subject: [PATCH v2 1/6] mm: enable page walking API to lock vmas during the walk From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, Suren Baghdasaryan , Linus Torvalds X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773074549588197216 X-GMAIL-MSGID: 1773074549588197216 walk_page_range() and friends often operate under write-locked mmap_lock. With introduction of vma locks, the vmas have to be locked as well during such walks to prevent concurrent page faults in these areas. Add an additional member to mm_walk_ops to indicate locking requirements for the walk. Cc: stable@vger.kernel.org # 6.4.x Suggested-by: Linus Torvalds Suggested-by: Jann Horn Signed-off-by: Suren Baghdasaryan --- arch/powerpc/mm/book3s64/subpage_prot.c | 1 + arch/riscv/mm/pageattr.c | 1 + arch/s390/mm/gmap.c | 5 ++++ fs/proc/task_mmu.c | 5 ++++ include/linux/pagewalk.h | 11 ++++++++ mm/damon/vaddr.c | 2 ++ mm/hmm.c | 1 + mm/ksm.c | 25 ++++++++++------- mm/madvise.c | 3 +++ mm/memcontrol.c | 2 ++ mm/memory-failure.c | 1 + mm/mempolicy.c | 22 +++++++++------ mm/migrate_device.c | 1 + mm/mincore.c | 1 + mm/mlock.c | 1 + mm/mprotect.c | 1 + mm/pagewalk.c | 36 ++++++++++++++++++++++--- mm/vmscan.c | 1 + 18 files changed, 100 insertions(+), 20 deletions(-) diff --git a/arch/powerpc/mm/book3s64/subpage_prot.c b/arch/powerpc/mm/book3s64/subpage_prot.c index 0dc85556dec5..ec98e526167e 100644 --- a/arch/powerpc/mm/book3s64/subpage_prot.c +++ b/arch/powerpc/mm/book3s64/subpage_prot.c @@ -145,6 +145,7 @@ static int subpage_walk_pmd_entry(pmd_t *pmd, unsigned long addr, static const struct mm_walk_ops subpage_walk_ops = { .pmd_entry = subpage_walk_pmd_entry, + .walk_lock = PGWALK_WRLOCK_VERIFY, }; static void subpage_mark_vma_nohuge(struct mm_struct *mm, unsigned long addr, diff --git a/arch/riscv/mm/pageattr.c b/arch/riscv/mm/pageattr.c index ea3d61de065b..161d0b34c2cb 100644 --- a/arch/riscv/mm/pageattr.c +++ b/arch/riscv/mm/pageattr.c @@ -102,6 +102,7 @@ static const struct mm_walk_ops pageattr_ops = { .pmd_entry = pageattr_pmd_entry, .pte_entry = pageattr_pte_entry, .pte_hole = pageattr_pte_hole, + .walk_lock = PGWALK_RDLOCK, }; static int __set_memory(unsigned long addr, int numpages, pgprot_t set_mask, diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index 9c8af31be970..906a7bfc2a78 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -2514,6 +2514,7 @@ static int thp_split_walk_pmd_entry(pmd_t *pmd, unsigned long addr, static const struct mm_walk_ops thp_split_walk_ops = { .pmd_entry = thp_split_walk_pmd_entry, + .walk_lock = PGWALK_WRLOCK_VERIFY, }; static inline void thp_split_mm(struct mm_struct *mm) @@ -2565,6 +2566,7 @@ static int __zap_zero_pages(pmd_t *pmd, unsigned long start, static const struct mm_walk_ops zap_zero_walk_ops = { .pmd_entry = __zap_zero_pages, + .walk_lock = PGWALK_WRLOCK, }; /* @@ -2655,6 +2657,7 @@ static const struct mm_walk_ops enable_skey_walk_ops = { .hugetlb_entry = __s390_enable_skey_hugetlb, .pte_entry = __s390_enable_skey_pte, .pmd_entry = __s390_enable_skey_pmd, + .walk_lock = PGWALK_WRLOCK, }; int s390_enable_skey(void) @@ -2692,6 +2695,7 @@ static int __s390_reset_cmma(pte_t *pte, unsigned long addr, static const struct mm_walk_ops reset_cmma_walk_ops = { .pte_entry = __s390_reset_cmma, + .walk_lock = PGWALK_WRLOCK, }; void s390_reset_cmma(struct mm_struct *mm) @@ -2728,6 +2732,7 @@ static int s390_gather_pages(pte_t *ptep, unsigned long addr, static const struct mm_walk_ops gather_pages_ops = { .pte_entry = s390_gather_pages, + .walk_lock = PGWALK_RDLOCK, }; /* diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 507cd4e59d07..ef6ee330e3be 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -758,12 +758,14 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, static const struct mm_walk_ops smaps_walk_ops = { .pmd_entry = smaps_pte_range, .hugetlb_entry = smaps_hugetlb_range, + .walk_lock = PGWALK_RDLOCK, }; static const struct mm_walk_ops smaps_shmem_walk_ops = { .pmd_entry = smaps_pte_range, .hugetlb_entry = smaps_hugetlb_range, .pte_hole = smaps_pte_hole, + .walk_lock = PGWALK_RDLOCK, }; /* @@ -1245,6 +1247,7 @@ static int clear_refs_test_walk(unsigned long start, unsigned long end, static const struct mm_walk_ops clear_refs_walk_ops = { .pmd_entry = clear_refs_pte_range, .test_walk = clear_refs_test_walk, + .walk_lock = PGWALK_WRLOCK, }; static ssize_t clear_refs_write(struct file *file, const char __user *buf, @@ -1622,6 +1625,7 @@ static const struct mm_walk_ops pagemap_ops = { .pmd_entry = pagemap_pmd_range, .pte_hole = pagemap_pte_hole, .hugetlb_entry = pagemap_hugetlb_range, + .walk_lock = PGWALK_RDLOCK, }; /* @@ -1935,6 +1939,7 @@ static int gather_hugetlb_stats(pte_t *pte, unsigned long hmask, static const struct mm_walk_ops show_numa_ops = { .hugetlb_entry = gather_hugetlb_stats, .pmd_entry = gather_pte_stats, + .walk_lock = PGWALK_RDLOCK, }; /* diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h index 27a6df448ee5..27cd1e59ccf7 100644 --- a/include/linux/pagewalk.h +++ b/include/linux/pagewalk.h @@ -6,6 +6,16 @@ struct mm_walk; +/* Locking requirement during a page walk. */ +enum page_walk_lock { + /* mmap_lock should be locked for read to stabilize the vma tree */ + PGWALK_RDLOCK = 0, + /* vma will be write-locked during the walk */ + PGWALK_WRLOCK = 1, + /* vma is expected to be already write-locked during the walk */ + PGWALK_WRLOCK_VERIFY = 2, +}; + /** * struct mm_walk_ops - callbacks for walk_page_range * @pgd_entry: if set, called for each non-empty PGD (top-level) entry @@ -66,6 +76,7 @@ struct mm_walk_ops { int (*pre_vma)(unsigned long start, unsigned long end, struct mm_walk *walk); void (*post_vma)(struct mm_walk *walk); + enum page_walk_lock walk_lock; }; /* diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 2fcc9731528a..e0e59d420fca 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -386,6 +386,7 @@ static int damon_mkold_hugetlb_entry(pte_t *pte, unsigned long hmask, static const struct mm_walk_ops damon_mkold_ops = { .pmd_entry = damon_mkold_pmd_entry, .hugetlb_entry = damon_mkold_hugetlb_entry, + .walk_lock = PGWALK_RDLOCK, }; static void damon_va_mkold(struct mm_struct *mm, unsigned long addr) @@ -525,6 +526,7 @@ static int damon_young_hugetlb_entry(pte_t *pte, unsigned long hmask, static const struct mm_walk_ops damon_young_ops = { .pmd_entry = damon_young_pmd_entry, .hugetlb_entry = damon_young_hugetlb_entry, + .walk_lock = PGWALK_RDLOCK, }; static bool damon_va_young(struct mm_struct *mm, unsigned long addr, diff --git a/mm/hmm.c b/mm/hmm.c index 855e25e59d8f..277ddcab4947 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -562,6 +562,7 @@ static const struct mm_walk_ops hmm_walk_ops = { .pte_hole = hmm_vma_walk_hole, .hugetlb_entry = hmm_vma_walk_hugetlb_entry, .test_walk = hmm_vma_walk_test, + .walk_lock = PGWALK_RDLOCK, }; /** diff --git a/mm/ksm.c b/mm/ksm.c index ba266359da55..00c21fb4d94e 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -455,6 +455,12 @@ static int break_ksm_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long nex static const struct mm_walk_ops break_ksm_ops = { .pmd_entry = break_ksm_pmd_entry, + .walk_lock = PGWALK_RDLOCK, +}; + +static const struct mm_walk_ops break_ksm_lock_vma_ops = { + .pmd_entry = break_ksm_pmd_entry, + .walk_lock = PGWALK_WRLOCK, }; /* @@ -470,16 +476,17 @@ static const struct mm_walk_ops break_ksm_ops = { * of the process that owns 'vma'. We also do not want to enforce * protection keys here anyway. */ -static int break_ksm(struct vm_area_struct *vma, unsigned long addr) +static int break_ksm(struct vm_area_struct *vma, unsigned long addr, bool lock_vma) { vm_fault_t ret = 0; + const struct mm_walk_ops *ops = lock_vma ? + &break_ksm_lock_vma_ops : &break_ksm_ops; do { int ksm_page; cond_resched(); - ksm_page = walk_page_range_vma(vma, addr, addr + 1, - &break_ksm_ops, NULL); + ksm_page = walk_page_range_vma(vma, addr, addr + 1, ops, NULL); if (WARN_ON_ONCE(ksm_page < 0)) return ksm_page; if (!ksm_page) @@ -565,7 +572,7 @@ static void break_cow(struct ksm_rmap_item *rmap_item) mmap_read_lock(mm); vma = find_mergeable_vma(mm, addr); if (vma) - break_ksm(vma, addr); + break_ksm(vma, addr, false); mmap_read_unlock(mm); } @@ -871,7 +878,7 @@ static void remove_trailing_rmap_items(struct ksm_rmap_item **rmap_list) * in cmp_and_merge_page on one of the rmap_items we would be removing. */ static int unmerge_ksm_pages(struct vm_area_struct *vma, - unsigned long start, unsigned long end) + unsigned long start, unsigned long end, bool lock_vma) { unsigned long addr; int err = 0; @@ -882,7 +889,7 @@ static int unmerge_ksm_pages(struct vm_area_struct *vma, if (signal_pending(current)) err = -ERESTARTSYS; else - err = break_ksm(vma, addr); + err = break_ksm(vma, addr, lock_vma); } return err; } @@ -1029,7 +1036,7 @@ static int unmerge_and_remove_all_rmap_items(void) if (!(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma) continue; err = unmerge_ksm_pages(vma, - vma->vm_start, vma->vm_end); + vma->vm_start, vma->vm_end, false); if (err) goto error; } @@ -2530,7 +2537,7 @@ static int __ksm_del_vma(struct vm_area_struct *vma) return 0; if (vma->anon_vma) { - err = unmerge_ksm_pages(vma, vma->vm_start, vma->vm_end); + err = unmerge_ksm_pages(vma, vma->vm_start, vma->vm_end, true); if (err) return err; } @@ -2668,7 +2675,7 @@ int ksm_madvise(struct vm_area_struct *vma, unsigned long start, return 0; /* just ignore the advice */ if (vma->anon_vma) { - err = unmerge_ksm_pages(vma, start, end); + err = unmerge_ksm_pages(vma, start, end, true); if (err) return err; } diff --git a/mm/madvise.c b/mm/madvise.c index 886f06066622..bfe0e06427bd 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -233,6 +233,7 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start, static const struct mm_walk_ops swapin_walk_ops = { .pmd_entry = swapin_walk_pmd_entry, + .walk_lock = PGWALK_RDLOCK, }; static void shmem_swapin_range(struct vm_area_struct *vma, @@ -534,6 +535,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, static const struct mm_walk_ops cold_walk_ops = { .pmd_entry = madvise_cold_or_pageout_pte_range, + .walk_lock = PGWALK_RDLOCK, }; static void madvise_cold_page_range(struct mmu_gather *tlb, @@ -757,6 +759,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, static const struct mm_walk_ops madvise_free_walk_ops = { .pmd_entry = madvise_free_pte_range, + .walk_lock = PGWALK_RDLOCK, }; static int madvise_free_single_vma(struct vm_area_struct *vma, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e8ca4bdcb03c..315fd5f45e3c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6024,6 +6024,7 @@ static int mem_cgroup_count_precharge_pte_range(pmd_t *pmd, static const struct mm_walk_ops precharge_walk_ops = { .pmd_entry = mem_cgroup_count_precharge_pte_range, + .walk_lock = PGWALK_RDLOCK, }; static unsigned long mem_cgroup_count_precharge(struct mm_struct *mm) @@ -6303,6 +6304,7 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd, static const struct mm_walk_ops charge_walk_ops = { .pmd_entry = mem_cgroup_move_charge_pte_range, + .walk_lock = PGWALK_RDLOCK, }; static void mem_cgroup_move_charge(void) diff --git a/mm/memory-failure.c b/mm/memory-failure.c index ece5d481b5ff..6bfb762facab 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -831,6 +831,7 @@ static int hwpoison_hugetlb_range(pte_t *ptep, unsigned long hmask, static const struct mm_walk_ops hwp_walk_ops = { .pmd_entry = hwpoison_pte_range, .hugetlb_entry = hwpoison_hugetlb_range, + .walk_lock = PGWALK_RDLOCK, }; /* diff --git a/mm/mempolicy.c b/mm/mempolicy.c index c53f8beeb507..ec2eaceffd74 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -718,6 +718,14 @@ static const struct mm_walk_ops queue_pages_walk_ops = { .hugetlb_entry = queue_folios_hugetlb, .pmd_entry = queue_folios_pte_range, .test_walk = queue_pages_test_walk, + .walk_lock = PGWALK_RDLOCK, +}; + +static const struct mm_walk_ops queue_pages_lock_vma_walk_ops = { + .hugetlb_entry = queue_folios_hugetlb, + .pmd_entry = queue_folios_pte_range, + .test_walk = queue_pages_test_walk, + .walk_lock = PGWALK_WRLOCK, }; /* @@ -738,7 +746,7 @@ static const struct mm_walk_ops queue_pages_walk_ops = { static int queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, nodemask_t *nodes, unsigned long flags, - struct list_head *pagelist) + struct list_head *pagelist, bool lock_vma) { int err; struct queue_pages qp = { @@ -749,8 +757,10 @@ queue_pages_range(struct mm_struct *mm, unsigned long start, unsigned long end, .end = end, .first = NULL, }; + const struct mm_walk_ops *ops = lock_vma ? + &queue_pages_lock_vma_walk_ops : &queue_pages_walk_ops; - err = walk_page_range(mm, start, end, &queue_pages_walk_ops, &qp); + err = walk_page_range(mm, start, end, ops, &qp); if (!qp.first) /* whole range in hole */ @@ -1078,7 +1088,7 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest, vma = find_vma(mm, 0); VM_BUG_ON(!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))); queue_pages_range(mm, vma->vm_start, mm->task_size, &nmask, - flags | MPOL_MF_DISCONTIG_OK, &pagelist); + flags | MPOL_MF_DISCONTIG_OK, &pagelist, false); if (!list_empty(&pagelist)) { err = migrate_pages(&pagelist, alloc_migration_target, NULL, @@ -1321,12 +1331,8 @@ static long do_mbind(unsigned long start, unsigned long len, * Lock the VMAs before scanning for pages to migrate, to ensure we don't * miss a concurrently inserted page. */ - vma_iter_init(&vmi, mm, start); - for_each_vma_range(vmi, vma, end) - vma_start_write(vma); - ret = queue_pages_range(mm, start, end, nmask, - flags | MPOL_MF_INVERT, &pagelist); + flags | MPOL_MF_INVERT, &pagelist, true); if (ret < 0) { err = ret; diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 8365158460ed..d5f492356e3e 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -279,6 +279,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, static const struct mm_walk_ops migrate_vma_walk_ops = { .pmd_entry = migrate_vma_collect_pmd, .pte_hole = migrate_vma_collect_hole, + .walk_lock = PGWALK_RDLOCK, }; /* diff --git a/mm/mincore.c b/mm/mincore.c index b7f7a516b26c..dad3622cc963 100644 --- a/mm/mincore.c +++ b/mm/mincore.c @@ -176,6 +176,7 @@ static const struct mm_walk_ops mincore_walk_ops = { .pmd_entry = mincore_pte_range, .pte_hole = mincore_unmapped_range, .hugetlb_entry = mincore_hugetlb, + .walk_lock = PGWALK_RDLOCK, }; /* diff --git a/mm/mlock.c b/mm/mlock.c index 0a0c996c5c21..479e09d0994c 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -371,6 +371,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma, { static const struct mm_walk_ops mlock_walk_ops = { .pmd_entry = mlock_pte_range, + .walk_lock = PGWALK_WRLOCK_VERIFY, }; /* diff --git a/mm/mprotect.c b/mm/mprotect.c index 6f658d483704..3aef1340533a 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -568,6 +568,7 @@ static const struct mm_walk_ops prot_none_walk_ops = { .pte_entry = prot_none_pte_entry, .hugetlb_entry = prot_none_hugetlb_entry, .test_walk = prot_none_test, + .walk_lock = PGWALK_WRLOCK, }; int diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 2022333805d3..9b2d23fbf4d3 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -400,6 +400,33 @@ static int __walk_page_range(unsigned long start, unsigned long end, return err; } +static inline void process_mm_walk_lock(struct mm_struct *mm, + enum page_walk_lock walk_lock) +{ + if (walk_lock == PGWALK_RDLOCK) + mmap_assert_locked(mm); + else + mmap_assert_write_locked(mm); +} + +static inline void process_vma_walk_lock(struct vm_area_struct *vma, + enum page_walk_lock walk_lock) +{ +#ifdef CONFIG_PER_VMA_LOCK + switch (walk_lock) { + case PGWALK_WRLOCK: + vma_start_write(vma); + break; + case PGWALK_WRLOCK_VERIFY: + vma_assert_write_locked(vma); + break; + case PGWALK_RDLOCK: + /* PGWALK_RDLOCK is handled by process_mm_walk_lock */ + break; + } +#endif +} + /** * walk_page_range - walk page table with caller specific callbacks * @mm: mm_struct representing the target process of page table walk @@ -459,7 +486,7 @@ int walk_page_range(struct mm_struct *mm, unsigned long start, if (!walk.mm) return -EINVAL; - mmap_assert_locked(walk.mm); + process_mm_walk_lock(walk.mm, ops->walk_lock); vma = find_vma(walk.mm, start); do { @@ -474,6 +501,7 @@ int walk_page_range(struct mm_struct *mm, unsigned long start, if (ops->pte_hole) err = ops->pte_hole(start, next, -1, &walk); } else { /* inside vma */ + process_vma_walk_lock(vma, ops->walk_lock); walk.vma = vma; next = min(end, vma->vm_end); vma = find_vma(mm, vma->vm_end); @@ -549,7 +577,8 @@ int walk_page_range_vma(struct vm_area_struct *vma, unsigned long start, if (start < vma->vm_start || end > vma->vm_end) return -EINVAL; - mmap_assert_locked(walk.mm); + process_mm_walk_lock(walk.mm, ops->walk_lock); + process_vma_walk_lock(vma, ops->walk_lock); return __walk_page_range(start, end, &walk); } @@ -566,7 +595,8 @@ int walk_page_vma(struct vm_area_struct *vma, const struct mm_walk_ops *ops, if (!walk.mm) return -EINVAL; - mmap_assert_locked(walk.mm); + process_mm_walk_lock(walk.mm, ops->walk_lock); + process_vma_walk_lock(vma, ops->walk_lock); return __walk_page_range(vma->vm_start, vma->vm_end, &walk); } diff --git a/mm/vmscan.c b/mm/vmscan.c index 1080209a568b..3555927df9b5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4284,6 +4284,7 @@ static void walk_mm(struct lruvec *lruvec, struct mm_struct *mm, struct lru_gen_ static const struct mm_walk_ops mm_walk_ops = { .test_walk = should_skip_vma, .p4d_entry = walk_pud_range, + .walk_lock = PGWALK_RDLOCK, }; int err; From patchwork Tue Aug 1 22:07:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 129507 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f41:0:b0:3e4:2afc:c1 with SMTP id v1csp49415vqx; Tue, 1 Aug 2023 15:23:07 -0700 (PDT) X-Google-Smtp-Source: APBJJlGvqj7AD+dUoS6jFr9AcJBTcdpwc7fW+RQ6es3Sxtqxc1n+uv5kFnDn7ZOtzWRSwpRtbPLA X-Received: by 2002:a2e:83d0:0:b0:2b6:e96c:5414 with SMTP id s16-20020a2e83d0000000b002b6e96c5414mr3192605ljh.52.1690928587237; Tue, 01 Aug 2023 15:23:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690928587; cv=none; d=google.com; s=arc-20160816; b=LCu0n9Acad3BLgpuISKDmqIReHCr/sKybTvX83dpR/0lpHFSTwY4riSsPC/KEXfRsA R1YODW1TLEv5whpSuMkeQy3kE34nWz6vBww1xD/sa4q6GqB0MvMLAazXnb8Q7S2mEcLU 7hBGumKfARfvCVPyZt2hxOwndOhHYbNnk1hxFR8v4+kHLCzrS0yqkitTlhzIYrNfrtNd plfC7w5DgZr6AqAvCsuScrcT7wJE7DPfZr+cUL8Um6/1PxKZsZsijqVQZX0hJmqXpvJe JO7mreB0NjPCd6DloHXO24kugODq1cSCCdgt8m5nWLMRH1oFNu269zlNieMmOpr0iBU+ +A3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=i6HEpkZDiMCJl1HNBxUq/pPnwhYuhlhYWrvFQybpJWY=; fh=GuUSihYN9cg5LmZCV2sCu3xVOah+EC5CWq+l9Dksn6w=; b=ITZNNhcpMx3VnXzy8FhXwzDn9FP3+5kDatGGbZFSIshNm7szeMCD6CkWWq3zhJDUpJ of9UOAa5Q0EFlPd6XhliIw6v/g+FrL3JxyxhuoXQM+n03rINKUQ+XANOrT/NtgKkoBEV 0Uc1QpSjSjjCgsvERIDJSwFqBgu8f2WARx5JAaKKC8McJX5ApILur5vtjWbOq5PXbu/H lJL4aoAZcR2dF03g2vKN4neGG97ICY2IbPAnVgQ+ZKMFUvMbrbcVU61tzxl+ruTUqbo7 e6ar80gQNGq1xh+2Z/uHWJwpWkT8ZGgk/hkvcFecKGUPbzXSQiEd+0ZwH1NkcDZlAJGC 12VQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=V9cpwSEL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a4-20020a1709065f8400b0099bc0489b51si9358991eju.28.2023.08.01.15.22.43; Tue, 01 Aug 2023 15:23:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=V9cpwSEL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231812AbjHAWHt (ORCPT + 99 others); Tue, 1 Aug 2023 18:07:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33990 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231396AbjHAWHm (ORCPT ); Tue, 1 Aug 2023 18:07:42 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5F481BCF for ; Tue, 1 Aug 2023 15:07:41 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-585f04ffa3eso55637997b3.0 for ; Tue, 01 Aug 2023 15:07:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690927661; x=1691532461; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=i6HEpkZDiMCJl1HNBxUq/pPnwhYuhlhYWrvFQybpJWY=; b=V9cpwSELJrkQrszXU5HyumPij806LjFOgAKqGQckrRIsTl9WyJGm7y2tP2Z2qpZNBZ 8OnG+G2+pRoWY7Gyh/uRKHckpG/wW2klLNffgsUon5TtBq05uulzu7VlS14UeHcVePc5 6rcXGltEDaVNS9+SeI1i+AP3HDVebAv3PFT1lfSUDf6ep/jTCQ/IUFd9ttrSEA9t9DPO iGN48DWEsdFNb4WBgDh2gwPyzp4LH+hHwffYSWHoJfq3HFOTa5CTWqxb2Bx96ZnBccWK 7hzPD483pGlx4dh0dSN4wMfxSLMa6kthAGUj3r6PC/2LNnNJKhb7gl7lRbdc8DmYXSji 2fsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690927661; x=1691532461; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=i6HEpkZDiMCJl1HNBxUq/pPnwhYuhlhYWrvFQybpJWY=; b=XXSWSvP5ziJfaSlz90qFce5wgBdqINvM9xD5K3oHHKJYkJQX93hJbaeDbTAF+LMffv p/UdlFwZe06D80EZtAK8/A7HRPF67M+IdGzmEaVOH6/orMm6SGXCVbcbSKBTQVAgzeIy zONhjn7ZRqUWRhGF2dUh3VMOUi9SLGUSym959KO/BDP9xXNVDi7uWWpNCoyr50Gk4val tnP1/VAPgyTpzhV7abSlWbblzPaJN8gbUE4Ex8m1rjheygDpEIhlFNcEFfFznG8/lqbc 8R1giQkVR5K+XnhHQaO75Rx2u9I7WwVRKCS6DWIxpZqyagQ/A0ZAmLY5lKhzp9MSthhf wDiw== X-Gm-Message-State: ABy/qLa4eGuTdqZhV49AgX+MVB6vvJ0gyXDHHEj6Yy+aAr662ZiNWxW+ notOCRIG3+KUQlYVgYIaYXpWLjBjiDA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:211c:a2ff:f17b:c5e9]) (user=surenb job=sendgmr) by 2002:a81:b703:0:b0:584:41a6:6cd8 with SMTP id v3-20020a81b703000000b0058441a66cd8mr114995ywh.8.1690927661105; Tue, 01 Aug 2023 15:07:41 -0700 (PDT) Date: Tue, 1 Aug 2023 15:07:28 -0700 In-Reply-To: <20230801220733.1987762-1-surenb@google.com> Mime-Version: 1.0 References: <20230801220733.1987762-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.585.gd2178a4bd4-goog Message-ID: <20230801220733.1987762-3-surenb@google.com> Subject: [PATCH v2 2/6] mm: for !CONFIG_PER_VMA_LOCK equate write lock assertion for vma and mmap From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773067134204205895 X-GMAIL-MSGID: 1773067134204205895 When CONFIG_PER_VMA_LOCK=n, vma_assert_write_locked() should be equivalent to mmap_assert_write_locked(). Suggested-by: Jann Horn Signed-off-by: Suren Baghdasaryan Reviewed-by: Liam R. Howlett --- include/linux/mm.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 406ab9ea818f..262b5f44101d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -750,7 +750,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {} static inline void vma_start_write(struct vm_area_struct *vma) {} static inline bool vma_try_start_write(struct vm_area_struct *vma) { return true; } -static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked(struct vm_area_struct *vma) + { mmap_assert_write_locked(vma->vm_mm); } static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached) {} From patchwork Tue Aug 1 22:07:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 129519 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f41:0:b0:3e4:2afc:c1 with SMTP id v1csp69334vqx; Tue, 1 Aug 2023 16:07:47 -0700 (PDT) X-Google-Smtp-Source: APBJJlF9AMM4/g/xjC+PLFYFkWveZ+btBuMIkyXNhFQslsFd63tVJ1bz434z7HSm1CTnoqWgHAIq X-Received: by 2002:a17:902:988e:b0:1b6:72af:623d with SMTP id s14-20020a170902988e00b001b672af623dmr13191691plp.55.1690931266992; Tue, 01 Aug 2023 16:07:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690931266; cv=none; d=google.com; s=arc-20160816; b=v4f7Mv7RKvAxskvoDGsjmS09jdLjyZ0RT475d+ljoEZaDM8O8IKUuXKfEh+SVOuuO/ Ihx5dGATut0LzbRJY/g5EKc33WE2GJfMApRxEjwACQocROSMBhleYlQNlhubam99zjtK es8uhmWtDaLtCbjo/4FW7kVoJqK6LxX5IMmK+so/dXjMVi5lhavxTcFym1gt+KGuW1fO nFespe2sARvGEPXkNxIOckf3SB9KPCtWZldWaD45RZ6YKx746pNMBTJIZd4jaJiVvTrn 02ESLTmfN9BKILytOYkQRkZ+wDyq9rCMSnyV7hpIGDZKyhYb26yOm6DyLOTISgz5T77z E6kQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=4DFBzP3PjGN+0KSVb5wBxy1Zq6atXukCZSiVwP2lCA4=; fh=GuUSihYN9cg5LmZCV2sCu3xVOah+EC5CWq+l9Dksn6w=; b=G2vfrU69F7tpmCul59v3YPT/dHG1SqHDjOaZ+L7ibtm8HL93lvVvJCXzID6m9c8ChK aQDMFgNC+xYq7cFURNt4Zq/W7LJbxBgWxGMO0tInj9jMCdzpBDAUUJ1R/NqCfEDgsyt7 zZ81Amu11E031hk6Fn9IOtAcBaggg2nOukgbCmcJGShOrFUP/GCnp2OgXaQ8eFxvgwkM fPm0wdYW4nbaJQ+CLqLr1W+KUfTaJ62pep1xNfwVvAadiL2rQw9A85rZiTHNtgP9qs7D smc7OK2Mml6L7s6nB6YitgGDvA6olpH0ht7ztGQZsYUoXrK79kYd4rUtqKC+YBmt1U+o VXJw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=yfrnQPSj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i2-20020a170902e48200b001b80ecdcb88si9411192ple.473.2023.08.01.16.07.32; Tue, 01 Aug 2023 16:07:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=yfrnQPSj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231477AbjHAWHx (ORCPT + 99 others); Tue, 1 Aug 2023 18:07:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231478AbjHAWHq (ORCPT ); Tue, 1 Aug 2023 18:07:46 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4297D2128 for ; Tue, 1 Aug 2023 15:07:44 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-58419550c3aso56296927b3.0 for ; Tue, 01 Aug 2023 15:07:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690927663; x=1691532463; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4DFBzP3PjGN+0KSVb5wBxy1Zq6atXukCZSiVwP2lCA4=; b=yfrnQPSjeIcEe1QiQGFJbsVHjwcAuLH5y1rpwsS9p+vhZ0UwR6PwXsX+lyRO8LZqne i0Gs4sb/UQu9JzPwvqM6DFSny+pA/wf1YWY/9cDzKMMSVuqNYknBZN4RDzqEJ+3tgA1o 9xqYOQ9YfNuN7rSMd/QL10WuKbuUb09YlooNQnxevr0KyYMiTseErJlZMTiswhp4rbGs 2EwfGVOHZ8kOe74pKMJDxDRfvODcH40dPPxzmE/4CBhsbsPSTt0MJcXPagBusQWf3TL+ 2RYrhvVai3uiHNprCu8Jx7kGArbzCMBXdmuc0K6o1xx5D3e1kBTjmNATzd4hCjxzlGtv 8JyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690927663; x=1691532463; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4DFBzP3PjGN+0KSVb5wBxy1Zq6atXukCZSiVwP2lCA4=; b=M64IMiXYbBhF8a9MsC2yj+rSSMRKB50CDX35PnwqUG4wzG2iEV+YyDwWkC8Qy/nwDI VPUCrx+IyVpKeu8FcsBPY4eigvYOmVxPW4qit0ulKfbnY9GUlHha2qzOvHTuC2pPTYSH 2wWlPwd/5akKdUo0spQc8Wi8GrCKMnAkmP915F4AD45UzOcLULCBcmxd0DrU+vRpUrM1 Z9V/hyDlk3DYYGUEzTHmmKbToYJfDGIZoodIt1ZjwDeRR/WN0oNnkiq8O3H6I2NYdtim rXDgaVoybLRZDsQ6MjrZPXIGN7dC9jK5YZ3h7d6vBmfIuxWhnkNT0CrBGCpc/ZEDMXS2 CBTg== X-Gm-Message-State: ABy/qLYjJuWOJeRJJElDJhOhdK7EgWaTM6PC6oWAgnR0L5/TS2TRk10C taPln3d3LLzACIykRdIOwEZeiH3m0EM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:211c:a2ff:f17b:c5e9]) (user=surenb job=sendgmr) by 2002:a25:d246:0:b0:c61:7151:6727 with SMTP id j67-20020a25d246000000b00c6171516727mr103919ybg.10.1690927663525; Tue, 01 Aug 2023 15:07:43 -0700 (PDT) Date: Tue, 1 Aug 2023 15:07:29 -0700 In-Reply-To: <20230801220733.1987762-1-surenb@google.com> Mime-Version: 1.0 References: <20230801220733.1987762-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.585.gd2178a4bd4-goog Message-ID: <20230801220733.1987762-4-surenb@google.com> Subject: [PATCH v2 3/6] mm: replace mmap with vma write lock assertions when operating on a vma From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773069943893883333 X-GMAIL-MSGID: 1773069943893883333 Vma write lock assertion always includes mmap write lock assertion and additional vma lock checks when per-VMA locks are enabled. Replace weaker mmap_assert_write_locked() assertions with stronger vma_assert_write_locked() ones when we are operating on a vma which is expected to be locked. Suggested-by: Jann Horn Signed-off-by: Suren Baghdasaryan Reviewed-by: Liam R. Howlett --- mm/hugetlb.c | 2 +- mm/khugepaged.c | 5 +++-- mm/memory.c | 2 +- 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 64a3239b6407..1d871a1167d8 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5028,7 +5028,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, src_vma->vm_start, src_vma->vm_end); mmu_notifier_invalidate_range_start(&range); - mmap_assert_write_locked(src); + vma_assert_write_locked(src_vma); raw_write_seqcount_begin(&src->write_protect_seq); } else { /* diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 78c8d5d8b628..1e43a56fba31 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1495,7 +1495,7 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr, }; VM_BUG_ON(!PageTransHuge(hpage)); - mmap_assert_write_locked(vma->vm_mm); + vma_assert_write_locked(vma); if (do_set_pmd(&vmf, hpage)) return SCAN_FAIL; @@ -1525,7 +1525,7 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v pmd_t pmd; struct mmu_notifier_range range; - mmap_assert_write_locked(mm); + vma_assert_write_locked(vma); if (vma->vm_file) lockdep_assert_held_write(&vma->vm_file->f_mapping->i_mmap_rwsem); /* @@ -1570,6 +1570,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, int count = 0, result = SCAN_FAIL; int i; + /* Ensure vma can't change, it will be locked below after checks */ mmap_assert_write_locked(mm); /* Fast check before locking page if already PMD-mapped */ diff --git a/mm/memory.c b/mm/memory.c index 603b2f419948..652d99b9858a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1312,7 +1312,7 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) * Use the raw variant of the seqcount_t write API to avoid * lockdep complaining about preemptibility. */ - mmap_assert_write_locked(src_mm); + vma_assert_write_locked(src_vma); raw_write_seqcount_begin(&src_mm->write_protect_seq); } From patchwork Tue Aug 1 22:07:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 129526 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f41:0:b0:3e4:2afc:c1 with SMTP id v1csp81733vqx; Tue, 1 Aug 2023 16:37:37 -0700 (PDT) X-Google-Smtp-Source: APBJJlG5SDdL5Uc30FsnuuzPY3thZvLXz7Elx5TEruHbjaJ04+hB6QE/aoa1vtcNeQHO7qKczduO X-Received: by 2002:a05:6808:209a:b0:3a4:1983:78f5 with SMTP id s26-20020a056808209a00b003a4198378f5mr16055043oiw.3.1690933057064; Tue, 01 Aug 2023 16:37:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690933057; cv=none; d=google.com; s=arc-20160816; b=KiuEsUlVPDHyZbVnV+ntI7G3Ae+s2h1/TH/A2DIVirBvQCD0lTqW7QhqzXE7OBTUUB eZbHThe8Wp5Cvw+bqmseXHiVVPejdtCrc7JwLltrICiZYtuaBIHnrZhMIjBl05PJFCfh nf+ot6c6u4O2Jn66APCGCoe1grmtxe2l+urKXHF/gPXGfERJ/5UH7fMTze5bedfRQv7e 5zXZlh3xU0fHTbIa7Hf17OGe6xv0r8Sa0sf3jJu1g409VxguCp84Yw1MttrXzrFpfBUN GQReq+umRYygnm7z0C1vECS6o4FajVp+JOAng4LxRLB69C8dmToMnCpPOlOUQJ0/XqxR hWEA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=RKZo+1hfRepjKtnKDtyG+lpMw+Id83I1mtUQG8twV1g=; fh=Dw2MP27nuXcy0JixWzXZPVDqCPGddu/JRhZXxi83XxI=; b=W2+aq/CQIBgODy3i6koevrMEpkMfQAw0E7YhZ5yaiepsDvMzx7RUZz4hj5fVV1IVsv TYYdj21KcQqqbId5+yz7hkiaJHriG0qLSfCAMXEz+VJWSGvCefZiU9TRpjLLqiTmamW9 1tZBhIkGcieZ10JoP4HyNgHxltEGYMFIZwUXVfwhITtn8+atepKqHXIf6Y2K94dTI6t7 P9nH1AH3YHWXUFpQW32lg4jYaGlDAdM10dNjHC53Ci7zvsPhfD30rqbNimRi/qtEOsIQ KQapWSZWCwx5Qt6dISm5m+DqO7dXHUHieB/gSSSkGM2syWSdgNuIf1HQGW+iVqtG/0a6 LnDQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=ZFUINCXp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s19-20020a63af53000000b0055afc8a301asi1133084pgo.177.2023.08.01.16.37.23; Tue, 01 Aug 2023 16:37:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=ZFUINCXp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230216AbjHAWH4 (ORCPT + 99 others); Tue, 1 Aug 2023 18:07:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34024 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231596AbjHAWHs (ORCPT ); Tue, 1 Aug 2023 18:07:48 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F2ADA1BCF for ; Tue, 1 Aug 2023 15:07:46 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-585f04ffa3eso55638497b3.0 for ; Tue, 01 Aug 2023 15:07:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690927666; x=1691532466; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=RKZo+1hfRepjKtnKDtyG+lpMw+Id83I1mtUQG8twV1g=; b=ZFUINCXp69DsytCWVUi4i3FblzMVDA9BptWaaF+sKX0xKEjgkgju+DXhX3HccVVQYK XfNP/c2GWyfqQJHV5buiUdSz9eeuWGGs7bmeQuEchsb8OOL3n9wvyCurtr7tkXCQi0aN yP++75VtrNJXUbKenp0F1ykufx1TsQbFC4T42Fu0rToUR5fYyKHd1/8JmARUNH7JC7wS Lj5lgymUX89V5GFOqGp4x1br6gngtFB+xvWjpzgbM1kSIUiCB0dwywo6Jf8riCeEpuU+ fuTx3ZM3NBd+NW9LDhz7+Fp9WswDCH8sARSCiBtGMaTHvckO5JPymvcy35BZOGdWDtwz dWHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690927666; x=1691532466; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RKZo+1hfRepjKtnKDtyG+lpMw+Id83I1mtUQG8twV1g=; b=BKDhC/bd25f1jorevOrANrKmuVA2c3KRfTim2Mzq5Hd90gCf2AjFRHr12M2IBTYTP8 hHd+EmRP6trCj1GDEMSLXkKg85h+FqBRvRvnDOeOD5RcdBtbp2ry4b+Iv34B5ODXKc/L zmMEsp4ljT5cmWoh15NPGJm5uPpbJFWLc/TpjG8w1sAu3GKlglKob7JdKnT/NTKtG6xT gl2oTXsSwYjXvb/ANGsjFjAC/bdnVaZsQlQ/SuLmYCcdW2J5yxXJVbiay08VJ7/+U+0Y uIxcy5RcNVWuPT/vznUxB2oN1s8XSLjij/HD/U4hLCiM3aNtU11SUCAbHFHfS0RiUoyB J0ug== X-Gm-Message-State: ABy/qLZ4lKuH6l7iHm0PMsDhtkypc+oFa23K4jnbetk6pou410w+1oys VwOZOZJ6UrWu79jJYM1cLT3eZLcHy9Q= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:211c:a2ff:f17b:c5e9]) (user=surenb job=sendgmr) by 2002:a81:b104:0:b0:583:9913:f293 with SMTP id p4-20020a81b104000000b005839913f293mr116951ywh.1.1690927666209; Tue, 01 Aug 2023 15:07:46 -0700 (PDT) Date: Tue, 1 Aug 2023 15:07:30 -0700 In-Reply-To: <20230801220733.1987762-1-surenb@google.com> Mime-Version: 1.0 References: <20230801220733.1987762-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.585.gd2178a4bd4-goog Message-ID: <20230801220733.1987762-5-surenb@google.com> Subject: [PATCH v2 4/6] mm: lock vma explicitly before doing vm_flags_reset and vm_flags_reset_once From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, Suren Baghdasaryan , Linus Torvalds X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773071821218038627 X-GMAIL-MSGID: 1773071821218038627 Implicit vma locking inside vm_flags_reset() and vm_flags_reset_once() is not obvious and makes it hard to understand where vma locking is happening. Also in some cases (like in dup_userfaultfd()) vma should be locked earlier than vma_flags modification. To make locking more visible, change these functions to assert that the vma write lock is taken and explicitly lock the vma beforehand. Fix userfaultfd functions which should lock the vma earlier. Suggested-by: Linus Torvalds Signed-off-by: Suren Baghdasaryan Reviewed-by: Liam R. Howlett --- arch/powerpc/kvm/book3s_hv_uvmem.c | 1 + drivers/infiniband/hw/hfi1/file_ops.c | 1 + fs/userfaultfd.c | 6 ++++++ include/linux/mm.h | 10 +++++++--- mm/madvise.c | 5 ++--- mm/mlock.c | 3 ++- mm/mprotect.c | 1 + 7 files changed, 20 insertions(+), 7 deletions(-) diff --git a/arch/powerpc/kvm/book3s_hv_uvmem.c b/arch/powerpc/kvm/book3s_hv_uvmem.c index 709ebd578394..e2d6f9327f77 100644 --- a/arch/powerpc/kvm/book3s_hv_uvmem.c +++ b/arch/powerpc/kvm/book3s_hv_uvmem.c @@ -410,6 +410,7 @@ static int kvmppc_memslot_page_merge(struct kvm *kvm, ret = H_STATE; break; } + vma_start_write(vma); /* Copy vm_flags to avoid partial modifications in ksm_madvise */ vm_flags = vma->vm_flags; ret = ksm_madvise(vma, vma->vm_start, vma->vm_end, diff --git a/drivers/infiniband/hw/hfi1/file_ops.c b/drivers/infiniband/hw/hfi1/file_ops.c index a5ab22cedd41..5920bfc1e1c5 100644 --- a/drivers/infiniband/hw/hfi1/file_ops.c +++ b/drivers/infiniband/hw/hfi1/file_ops.c @@ -344,6 +344,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) goto done; } + vma_start_write(vma); /* * vm_pgoff is used as a buffer selector cookie. Always mmap from * the beginning. diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 7cecd49e078b..6cde95533dcd 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -667,6 +667,7 @@ static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx, mmap_write_lock(mm); for_each_vma(vmi, vma) { if (vma->vm_userfaultfd_ctx.ctx == release_new_ctx) { + vma_start_write(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; userfaultfd_set_vm_flags(vma, vma->vm_flags & ~__VM_UFFD_FLAGS); @@ -702,6 +703,7 @@ int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs) octx = vma->vm_userfaultfd_ctx.ctx; if (!octx || !(octx->features & UFFD_FEATURE_EVENT_FORK)) { + vma_start_write(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; userfaultfd_set_vm_flags(vma, vma->vm_flags & ~__VM_UFFD_FLAGS); return 0; @@ -783,6 +785,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma, atomic_inc(&ctx->mmap_changing); } else { /* Drop uffd context if remap feature not enabled */ + vma_start_write(vma); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; userfaultfd_set_vm_flags(vma, vma->vm_flags & ~__VM_UFFD_FLAGS); } @@ -940,6 +943,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file) prev = vma; } + vma_start_write(vma); userfaultfd_set_vm_flags(vma, new_flags); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; } @@ -1502,6 +1506,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ + vma_start_write(vma); userfaultfd_set_vm_flags(vma, new_flags); vma->vm_userfaultfd_ctx.ctx = ctx; @@ -1685,6 +1690,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ + vma_start_write(vma); userfaultfd_set_vm_flags(vma, new_flags); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; diff --git a/include/linux/mm.h b/include/linux/mm.h index 262b5f44101d..2c720c9bb1ae 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -780,18 +780,22 @@ static inline void vm_flags_init(struct vm_area_struct *vma, ACCESS_PRIVATE(vma, __vm_flags) = flags; } -/* Use when VMA is part of the VMA tree and modifications need coordination */ +/* + * Use when VMA is part of the VMA tree and modifications need coordination + * Note: vm_flags_reset and vm_flags_reset_once do not lock the vma and + * it should be locked explicitly beforehand. + */ static inline void vm_flags_reset(struct vm_area_struct *vma, vm_flags_t flags) { - vma_start_write(vma); + vma_assert_write_locked(vma); vm_flags_init(vma, flags); } static inline void vm_flags_reset_once(struct vm_area_struct *vma, vm_flags_t flags) { - vma_start_write(vma); + vma_assert_write_locked(vma); WRITE_ONCE(ACCESS_PRIVATE(vma, __vm_flags), flags); } diff --git a/mm/madvise.c b/mm/madvise.c index bfe0e06427bd..507b1d299fec 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -173,9 +173,8 @@ static int madvise_update_vma(struct vm_area_struct *vma, } success: - /* - * vm_flags is protected by the mmap_lock held in write mode. - */ + /* vm_flags is protected by the mmap_lock held in write mode. */ + vma_start_write(vma); vm_flags_reset(vma, new_flags); if (!vma->vm_file || vma_is_anon_shmem(vma)) { error = replace_anon_vma_name(vma, anon_name); diff --git a/mm/mlock.c b/mm/mlock.c index 479e09d0994c..06bdfab83b58 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -387,6 +387,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma, */ if (newflags & VM_LOCKED) newflags |= VM_IO; + vma_start_write(vma); vm_flags_reset_once(vma, newflags); lru_add_drain(); @@ -461,9 +462,9 @@ static int mlock_fixup(struct vma_iterator *vmi, struct vm_area_struct *vma, * It's okay if try_to_unmap_one unmaps a page just after we * set VM_LOCKED, populate_vma_page_range will bring it back. */ - if ((newflags & VM_LOCKED) && (oldflags & VM_LOCKED)) { /* No work to do, and mlocking twice would be wrong */ + vma_start_write(vma); vm_flags_reset(vma, newflags); } else { mlock_vma_pages_range(vma, start, end, newflags); diff --git a/mm/mprotect.c b/mm/mprotect.c index 3aef1340533a..362e190a8f81 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -657,6 +657,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, * vm_flags and vm_page_prot are protected by the mmap_lock * held in write mode. */ + vma_start_write(vma); vm_flags_reset(vma, newflags); if (vma_wants_manual_pte_write_upgrade(vma)) mm_cp_flags |= MM_CP_TRY_CHANGE_WRITABLE; From patchwork Tue Aug 1 22:07:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 129529 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f41:0:b0:3e4:2afc:c1 with SMTP id v1csp94693vqx; Tue, 1 Aug 2023 17:08:17 -0700 (PDT) X-Google-Smtp-Source: APBJJlGh4l5N/duBjEeeqQzHa3s0VD/sOgaJ+0Gq5GZlsnCaCgMFFH+2CHiCDE0pHQq8IbFJUfCQ X-Received: by 2002:a17:90a:ab92:b0:268:4c01:eb56 with SMTP id n18-20020a17090aab9200b002684c01eb56mr12085996pjq.2.1690934897386; Tue, 01 Aug 2023 17:08:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690934897; cv=none; d=google.com; s=arc-20160816; b=CYU3TTc48s4i7ZXgQRg0EFu46iH23TSrGUXWB4XV5u98y/gbcqMdPE698DzsLLSBgi MlvVn2o8bXe3H2/Dcqne9nRP6NEap/tc/EWKHa6t+VTauiyuPcyjpuq80f21a5y5ek6l 77+v0E9+xb1PvSv4ymyUi9buu3VurhHKX9onB1XBPnLXE69Q7lEzN5PSiMx4SRi3kM5c 1dTaNtnvWHW5BAh1m8IxfSl93IPBnGXTUj5WumZOXbT0GY8FX0purS0iK4S6UpvDltlS U/So2ER9XYt8TMhVANHKLXfNvTSoiePPw+ue9lGozN/y/81xupVbmMJJtDwQjL53F9E1 gpVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=QxBrE/UCHWlQTdGOylRc3uDNTY7RNXyaur9L23gJy4g=; fh=GuUSihYN9cg5LmZCV2sCu3xVOah+EC5CWq+l9Dksn6w=; b=KPPh/iQWBZzP1g2wPJWukgfnTiom1lvjM0+iD4XAmevsPmK9JnzilBXoIZF0queNqq fI3SgCNz0CEJRpIxrdyScIAKNn6lnVvRH/dP+EZICQtu3SaRugtzSo7Q/lgrxApWCW5y B89uDGybLn9J9FkHB4POncq4iwWS6JWGC3wIth9wiTje/FNFYFWD7m4eKlJQ/jpFtK8A cRyPqKH6tsl9crBNkc9KF6xC4Bkhzz+T6YlhAoQ76xS+1Nzgcwhnn2HibtzAnVcSWtvg kELmU2x6BvTfmwl9zDRgfNTnIV3ThlMPXf7gCwk6gvPbe1CtxuAhU4se5iydvk4RD48s xdUg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=LGbOmtH0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u1-20020a17090a410100b002465d98f18csi172596pjf.170.2023.08.01.17.08.04; Tue, 01 Aug 2023 17:08:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=LGbOmtH0; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230132AbjHAWH7 (ORCPT + 99 others); Tue, 1 Aug 2023 18:07:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34114 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231926AbjHAWHu (ORCPT ); Tue, 1 Aug 2023 18:07:50 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3AFF61BC3 for ; Tue, 1 Aug 2023 15:07:49 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5840614b107so75693947b3.1 for ; Tue, 01 Aug 2023 15:07:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690927668; x=1691532468; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QxBrE/UCHWlQTdGOylRc3uDNTY7RNXyaur9L23gJy4g=; b=LGbOmtH0lMhdDYO3+2ECExEYuxVvpb5dUlLecAiJAGrPI9PAJfTN7UrlhG2D2+sMgI A0HPsE7Dh7snipQv/ppKfZkrtywqFIDj94GgYYg1RmHEKQD/JrUFvPL4mXaNgmgR43fS rzNN/7cPcSEEUD6MphOEKlUIyUYRI9RNEW1ibPgtDMhMmPz+MaEiw+C/wPcvGj3gdm+s wmBOfyKZ9Eq/c9nMUYmgRF+ijS83GqRJwTrxNUo/kn28ipzSQ7VAbYmtkG9fCdYkXi0z Hhzh6iMenp89gYpIovNtlhgbKlD0/7Qoap5BixiG9npmNeurITeJhKlGDSUEA6RAa5bC 2mng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690927668; x=1691532468; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QxBrE/UCHWlQTdGOylRc3uDNTY7RNXyaur9L23gJy4g=; b=Ap1x9hm3PT8GMK937q1zhkKCDQy2c843axWqM+u6Ui0pCmvUssMZWA0JFT8JbdISkD 8XN8GHcOMzKCClzm1MUAuT6F3ELlg+8Ub/62ciezlFqjdLDR4KDdBVf1Wh9tMZfyFWdB I1JSLiY9lMuNE0SD1mI7PmnX/0am/HAT4MGdMXBe4E6PblSNbUyhMciNclFik4xWzip/ d5XUkyDXQ8JzRUWzUr2NflFth9obtnI/+T0yf/wNnp8rz0FAcAyPBj16mj256X7EXw2X 0acOJAfH6GVo05s2do9Beb0tzEgqhGR17U7BF09B1rycGScC5nqmjgN0LL4EacMOc+FS Nl+g== X-Gm-Message-State: ABy/qLbendCnW0g3QhKc4siJGqrfh2S1YxE4xFWLChquogd/9wgzVXSg LnBknTbQ1YHUbRVFTyaVv0+gYNsXUmo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:211c:a2ff:f17b:c5e9]) (user=surenb job=sendgmr) by 2002:a81:ae02:0:b0:576:b244:5a4e with SMTP id m2-20020a81ae02000000b00576b2445a4emr129366ywh.10.1690927668528; Tue, 01 Aug 2023 15:07:48 -0700 (PDT) Date: Tue, 1 Aug 2023 15:07:31 -0700 In-Reply-To: <20230801220733.1987762-1-surenb@google.com> Mime-Version: 1.0 References: <20230801220733.1987762-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.585.gd2178a4bd4-goog Message-ID: <20230801220733.1987762-6-surenb@google.com> Subject: [PATCH v2 5/6] mm: always lock new vma before inserting into vma tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, Suren Baghdasaryan X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773073750999976257 X-GMAIL-MSGID: 1773073750999976257 While it's not strictly necessary to lock a newly created vma before adding it into the vma tree (as long as no further changes are performed to it), it seems like a good policy to lock it and prevent accidental changes after it becomes visible to the page faults. Lock the vma before adding it into the vma tree. Suggested-by: Jann Horn Signed-off-by: Suren Baghdasaryan Reviewed-by: Liam R. Howlett --- mm/mmap.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 3937479d0e07..850a39dee075 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -412,6 +412,8 @@ static int vma_link(struct mm_struct *mm, struct vm_area_struct *vma) if (vma_iter_prealloc(&vmi)) return -ENOMEM; + vma_start_write(vma); + if (vma->vm_file) { mapping = vma->vm_file->f_mapping; i_mmap_lock_write(mapping); @@ -477,7 +479,8 @@ static inline void vma_prepare(struct vma_prepare *vp) vma_start_write(vp->vma); if (vp->adj_next) vma_start_write(vp->adj_next); - /* vp->insert is always a newly created VMA, no need for locking */ + if (vp->insert) + vma_start_write(vp->insert); if (vp->remove) vma_start_write(vp->remove); if (vp->remove2) @@ -3098,6 +3101,7 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, vma->vm_pgoff = addr >> PAGE_SHIFT; vm_flags_init(vma, flags); vma->vm_page_prot = vm_get_page_prot(flags); + vma_start_write(vma); if (vma_iter_store_gfp(vmi, vma, GFP_KERNEL)) goto mas_store_fail; @@ -3345,7 +3349,6 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, get_file(new_vma->vm_file); if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open(new_vma); - vma_start_write(new_vma); if (vma_link(mm, new_vma)) goto out_vma_link; *need_rmap_locks = false; From patchwork Tue Aug 1 22:07:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 129523 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f41:0:b0:3e4:2afc:c1 with SMTP id v1csp79258vqx; Tue, 1 Aug 2023 16:31:45 -0700 (PDT) X-Google-Smtp-Source: APBJJlGNtSIk7Fd8tKGYjYk6sg4MpVgUP+57lFrYB+KMOho0eqhQ8Ue7B6HKhZGmyXAo5bT9bjvI X-Received: by 2002:a05:6808:1449:b0:3a7:4fff:6485 with SMTP id x9-20020a056808144900b003a74fff6485mr3349596oiv.28.1690932705490; Tue, 01 Aug 2023 16:31:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690932705; cv=none; d=google.com; s=arc-20160816; b=GE7GR6n+UvDOHfbv9ZWEooA9aLYGKW0kSDPnjceLVliyyXb7LkoaGgKkW7ah4y7qVC E/LDp58XcxdeJxujlOc27l5BNDgpfkt98svLFIipPZMxRlSc5W263DhUao1pxW7GtHId nKeQOUbVLmMnPbWK3287zsL8mMywxTsDsQxQHjAwYI4iMqMV5rumPKdfNDLHYMwguzpy jtPzZs0iO1oGrPSph9JVRGwAFq5OfLHF7jU6urAD46a1fu60N1d4aoKRJOgVZcjTmWNZ iy2fXMje5BcJ4ntiCE1l4b1tPDtT5NzXXRYzNoPZhxfPGSkRHY5UXmvCSGN7DpQ+naF6 d69w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=FgBjF0ZO5ynPdNrndPmrtDmv0LneRZKTmqqgx6E5wg8=; fh=GbQyr9lzqgxZObh2rJgMXodl+n6ByLlEwKn1/cEHnLw=; b=m66NEtz3zOoK3kYKB4NzN5P+iZXWSRhqxe1n2cuCY9M8eXqJJD7PlzYMZudpKSCtXu 9/ioPKB1rx5EE3Sqatf/kZOiu6q1Xz2gmCekeRtPILVlpe6aQLN7cfwu97dUQvV68B3B wWi+6dlkKDGLTCQ4I6vkwlKG0lw0UM4rIrkSNqGHO8wRJxlrjblz2IIQojU0SpxwHtW5 lr7l+rYdU6Vuj16llNcWU93ehUuBMunszSH+KyusW/kELTx1Wgd4+k7/WHygq6GTCGhs j1552zB5/aVbx37PDx3KED3VL6jVUp6cQgAvEjeu3qoL+LgTyhWJpGNTHXGVGEDZp+My zGNA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=D1O6U13I; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k189-20020a6384c6000000b00542ad648fbasi9580367pgd.188.2023.08.01.16.31.31; Tue, 01 Aug 2023 16:31:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=D1O6U13I; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232747AbjHAWIO (ORCPT + 99 others); Tue, 1 Aug 2023 18:08:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34210 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231478AbjHAWIJ (ORCPT ); Tue, 1 Aug 2023 18:08:09 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 098F1268F for ; Tue, 1 Aug 2023 15:07:51 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-57320c10635so71725447b3.3 for ; Tue, 01 Aug 2023 15:07:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690927671; x=1691532471; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FgBjF0ZO5ynPdNrndPmrtDmv0LneRZKTmqqgx6E5wg8=; b=D1O6U13IyQtW6sRKIgsBESsntgiHZD2NBKEqY9x4VzN/LKKhnlNfxDelhOtdMckxRn FHM+qzWy+1vIGVvjuSnC3HOPacbJwiF+FU/44s3FUSl9rg9/MlksilEUDdYILducKN5m QBspm1K1gtu0F9WcdyEIrEuwnPMiQ8okMkqevL/Euzqha1FW9VFmXK9Kd09SaJL8SIMi 4XcGgId5NKrdkKLAeWoz/iYiTpXzRmouek8VJ8B5ZERcnA/Lkgi6E3b5jwXkBi9h6gTm 7wI1MoX2O3/0bFhkSpJ1WB6U2kt7rsAJK91S/XGtkVxCLlTHZukUWI5CAAc5iREvhiOQ j9Gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690927671; x=1691532471; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FgBjF0ZO5ynPdNrndPmrtDmv0LneRZKTmqqgx6E5wg8=; b=cArRiACyB0A3L0L1ol/Btjapdf4CDB3q3hT7vH7U8IEwosQuxFvVil1r989yRcZFey H+gw6MDe9AGqynrRtXxivkA5AOqSTouYOHSM8i8yzrujkIItjRfAAoaGM55kRS0YeIFa ybKVLRfaYDhrQOr6Il7ACAfkwLn79ipVrslMFqOL4hETzx1v6AGUiwVJvJ1c1VTgbv0R hRHrYely1e57d5aSFwWKsqayL0Weldc35+4KGtBaLlOPdVK3ck6n/vSg7o/lCXNwmyU2 2bkdsyf1wAzrsxxquKjPHuj5FF1KPkAlEK+uZA2WQ12LjFghfoDc28fH5Q726jK1NqRa M9eQ== X-Gm-Message-State: ABy/qLYXhTz+ydCVMzcJyZWqLFw/A1XyCs1uyt0P7quWzmZ8pOsLGyQP Ux69pAur2+31HNEeoysz0Oli1btywdA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:211c:a2ff:f17b:c5e9]) (user=surenb job=sendgmr) by 2002:a81:ae13:0:b0:583:a866:3450 with SMTP id m19-20020a81ae13000000b00583a8663450mr126859ywh.7.1690927671038; Tue, 01 Aug 2023 15:07:51 -0700 (PDT) Date: Tue, 1 Aug 2023 15:07:32 -0700 In-Reply-To: <20230801220733.1987762-1-surenb@google.com> Mime-Version: 1.0 References: <20230801220733.1987762-1-surenb@google.com> X-Mailer: git-send-email 2.41.0.585.gd2178a4bd4-goog Message-ID: <20230801220733.1987762-7-surenb@google.com> Subject: [PATCH v2 6/6] mm: move vma locking out of vma_prepare From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: torvalds@linux-foundation.org, jannh@google.com, willy@infradead.org, liam.howlett@oracle.com, david@redhat.com, peterx@redhat.com, ldufour@linux.ibm.com, vbabka@suse.cz, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, hannes@cmpxchg.org, dave@stgolabs.net, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, Suren Baghdasaryan , Linus Torvalds , "Liam R . Howlett" X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773071452401789421 X-GMAIL-MSGID: 1773071452401789421 vma_prepare() is currently the central place where vmas are being locked before vma_complete() applies changes to them. While this is convenient, it also obscures vma locking and makes it hard to follow the locking rules. Move vma locking out of vma_prepare() and take vma locks explicitly at the locations where vmas are being modified. Suggested-by: Linus Torvalds Signed-off-by: Suren Baghdasaryan Reviewed-by: Liam R. Howlett --- mm/mmap.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 850a39dee075..e59d83cb1d7a 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -476,16 +476,6 @@ static inline void init_vma_prep(struct vma_prepare *vp, */ static inline void vma_prepare(struct vma_prepare *vp) { - vma_start_write(vp->vma); - if (vp->adj_next) - vma_start_write(vp->adj_next); - if (vp->insert) - vma_start_write(vp->insert); - if (vp->remove) - vma_start_write(vp->remove); - if (vp->remove2) - vma_start_write(vp->remove2); - if (vp->file) { uprobe_munmap(vp->vma, vp->vma->vm_start, vp->vma->vm_end); @@ -650,6 +640,7 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, bool remove_next = false; struct vma_prepare vp; + vma_start_write(vma); if (next && (vma != next) && (end == next->vm_end)) { int ret; @@ -657,6 +648,7 @@ int vma_expand(struct vma_iterator *vmi, struct vm_area_struct *vma, ret = dup_anon_vma(vma, next); if (ret) return ret; + vma_start_write(next); } init_multi_vma_prep(&vp, vma, NULL, remove_next ? next : NULL, NULL); @@ -708,6 +700,8 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) return -ENOMEM; + vma_start_write(vma); + init_vma_prep(&vp, vma); vma_prepare(&vp); vma_adjust_trans_huge(vma, start, end, 0); @@ -946,10 +940,12 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, /* Can we merge both the predecessor and the successor? */ if (merge_prev && merge_next && is_mergeable_anon_vma(prev->anon_vma, next->anon_vma, NULL)) { + vma_start_write(next); remove = next; /* case 1 */ vma_end = next->vm_end; err = dup_anon_vma(prev, next); if (curr) { /* case 6 */ + vma_start_write(curr); remove = curr; remove2 = next; if (!next->anon_vma) @@ -958,6 +954,7 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, } else if (merge_prev) { /* case 2 */ if (curr) { err = dup_anon_vma(prev, curr); + vma_start_write(curr); if (end == curr->vm_end) { /* case 7 */ remove = curr; } else { /* case 5 */ @@ -969,6 +966,7 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, res = next; if (prev && addr < prev->vm_end) { /* case 4 */ vma_end = addr; + vma_start_write(next); adjust = next; adj_start = -(prev->vm_end - addr); err = dup_anon_vma(next, prev); @@ -983,6 +981,7 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, vma_pgoff = next->vm_pgoff - pglen; if (curr) { /* case 8 */ vma_pgoff = curr->vm_pgoff; + vma_start_write(curr); remove = curr; err = dup_anon_vma(next, curr); } @@ -996,6 +995,8 @@ struct vm_area_struct *vma_merge(struct vma_iterator *vmi, struct mm_struct *mm, if (vma_iter_prealloc(vmi)) return NULL; + vma_start_write(vma); + init_multi_vma_prep(&vp, vma, adjust, remove, remove2); VM_WARN_ON(vp.anon_vma && adjust && adjust->anon_vma && vp.anon_vma != adjust->anon_vma); @@ -2373,6 +2374,9 @@ int __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, if (new->vm_ops && new->vm_ops->open) new->vm_ops->open(new); + vma_start_write(vma); + vma_start_write(new); + init_vma_prep(&vp, vma); vp.insert = new; vma_prepare(&vp); @@ -3078,6 +3082,8 @@ static int do_brk_flags(struct vma_iterator *vmi, struct vm_area_struct *vma, if (vma_iter_prealloc(vmi)) goto unacct_fail; + vma_start_write(vma); + init_vma_prep(&vp, vma); vma_prepare(&vp); vma_adjust_trans_huge(vma, vma->vm_start, addr + len, 0);