From patchwork Sun Dec 4 16:14:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhongkun He X-Patchwork-Id: 29422 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1852532wrr; Sun, 4 Dec 2022 08:15:36 -0800 (PST) X-Google-Smtp-Source: AA0mqf7Jxa3rAANR+DVJGJpulQGD2G3bTjYYNXuacv/9FWlwik+ZP2avb1zi9W4dsEmlQncwN1r6 X-Received: by 2002:aa7:dbd8:0:b0:467:60fa:b629 with SMTP id v24-20020aa7dbd8000000b0046760fab629mr69373867edt.281.1670170536147; Sun, 04 Dec 2022 08:15:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670170536; cv=none; d=google.com; s=arc-20160816; b=WySgUJEdcM9s3xh9+kvWYvYJhfQ1xV+Q5zWaMJ7/Ly95HnU5QSM+zUGUO16FYTu0Ts WZ/1k6qgE2NuhNMARWNoT/hl5gEWX+QoArgaSdHlJf0vLwciHWRv5aUYCA3NQrwkfVi9 mG7bibOf/8bG+L7z50uyZ3Ltb56U6NtJQGteXxdu9ioNqNytgDXn7hdjkbYDQgXIHoIJ P0nRQXVzgIPv6k/HtdlN6JGqdwA4D6F2M59p9rxBL3RpjKXPKo/jNytKMU0ORkGUlU3o J7D20/dc65EidhTyBD3cJNJXkpIEulf5KInCy6bW70vzTWY4Xn2lzI/UreO9XCjVruOh l5vA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=kCicEi7EuklgtEvCZ5T+k9chhbYeMI1fhQwXDoR5e2w=; b=kbnMyUJiaTGryjwpBbxIY6iC7cNcVK/RzV4tT/UZMS+z4MeIbnbCpBlbaQZ5AJmkK+ HaIhkbtIGusfGZcxABITySd1e8DkMuL6ARM2olj8btsoSQ2VwnZkjFZcfxcODf6ezcR9 FeeYUE2Qh8b9eRxt49MDQMEo0FT3GLQTjJWpynwTU5FTOf0mn07a/t3fEzNsOPxHbWWB CuQoo/srLHIvQxl1SvkXOs+zV3UHgUDbh6cTrd5tYfzP+9p0JuGVVvGs4FM0X/bORz8+ WLn0jeoDLlHthJgqgsIF56TPnLDtEE/UH8j1LbBQfbm+Qj+GuQ7DkmWwtnTA+oXmiphP N6dw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=DN9268v2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id sa27-20020a1709076d1b00b007c0daaa5060si2725255ejc.699.2022.12.04.08.15.12; Sun, 04 Dec 2022 08:15:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=DN9268v2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229983AbiLDQOv (ORCPT + 99 others); Sun, 4 Dec 2022 11:14:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40732 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229753AbiLDQOt (ORCPT ); Sun, 4 Dec 2022 11:14:49 -0500 Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5ADE13FB3 for ; Sun, 4 Dec 2022 08:14:47 -0800 (PST) Received: by mail-pl1-x633.google.com with SMTP id g10so8758469plo.11 for ; Sun, 04 Dec 2022 08:14:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kCicEi7EuklgtEvCZ5T+k9chhbYeMI1fhQwXDoR5e2w=; b=DN9268v2i3+YlCBpBr/u+qAFDKpY0ZPHZx2wfNv72EIZ3pz9YkMHTJLvBSDMkuUCaC D+jRP0AHeKOdXlpQN/h24Li3REEESHkiPS8mBrCAxV3Le447Tm9PqSTcS+jrW1ub8EOV g9yOr4RebeIRNyICOezYtiJ+JsT+RqSL3GcPYZ4FSxpxdtWPij2bM4mlYuUmLjYJACBj 1KlAp0/qH4VcdAFwx0TOYD5ylJalVr84fLlz2dTC/izxghpZN5xf6LAVWQwiQuZ0Iua5 k5Mpnxz4xpGYOWQPUnNBeOCGtw0LtOsRB8icvfzA3oeDlGHX9QgRL8xgmvPvsLhY/3H1 niVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kCicEi7EuklgtEvCZ5T+k9chhbYeMI1fhQwXDoR5e2w=; b=JZB+x5+Q2tDe3x+mk71ezBxn70ehiiiy6fC0b78XPbqCsJ8ngtGP1dO5kuBvD6rAES 52vOXcufgmWsRLLHSeJveQHLY2zwEL/sbeC8EVgne4HcpCsNDPA/wa0vcshICsR47vJy wYLtsZuIfkKPIpzC4PsyDR/U7LGp1TdgzWo/V4int1YhOfbK5R7F1arYM6SEnboPIXsm JPqgbXYTQCsC2byxsYHn1F7RFMGnt6qx5LjCi8Wz9sT8zEzOZJc8z8iWd3410wBgODfK hZ2PBXxgNM4eukbw50t5bZjfb0OFMexqNkeVbAIi12S/2KuI1UpmbVo5pHCB6gPj/AuK IbOg== X-Gm-Message-State: ANoB5pkokXBL9lE30wDiRJVwm+uLtBAGuDxSpXs0POWz2a886mQdtfuf B4RSNP6+kxb66gkuesG6mpQ3bg== X-Received: by 2002:a17:90a:ae01:b0:213:e8b5:2d50 with SMTP id t1-20020a17090aae0100b00213e8b52d50mr85553925pjq.211.1670170487204; Sun, 04 Dec 2022 08:14:47 -0800 (PST) Received: from Tower.bytedance.net ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id n16-20020a170903111000b0016cf3f124e1sm9000323plh.234.2022.12.04.08.14.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 04 Dec 2022 08:14:46 -0800 (PST) From: Zhongkun He To: mhocko@suse.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, wuyun.abel@bytedance.com, Zhongkun He Subject: [PATCH 1/3] mm: replace mpol_put() with mpol_kill() initiated the destruction of mpol. Date: Mon, 5 Dec 2022 00:14:30 +0800 Message-Id: <20221204161432.2149375-2-hezhongkun.hzk@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221204161432.2149375-1-hezhongkun.hzk@bytedance.com> References: <20221204161432.2149375-1-hezhongkun.hzk@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751300740232542549?= X-GMAIL-MSGID: =?utf-8?q?1751300740232542549?= mpol_kill() is used to initiate destruction of mempolicy. so it is called to free the mempolicy. The pol_put() just decrement the local counter. Suggested-by: Michal Hocko Signed-off-by: Zhongkun He --- fs/hugetlbfs/inode.c | 2 +- fs/proc/task_mmu.c | 3 +-- kernel/fork.c | 4 ++-- mm/hugetlb.c | 6 +++--- mm/mempolicy.c | 42 +++++++++++++++++++++--------------------- mm/mmap.c | 10 +++++----- mm/shmem.c | 10 +++++----- 7 files changed, 38 insertions(+), 39 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index dd54f67e47fd..bad1b07f8653 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -93,7 +93,7 @@ static inline void hugetlb_set_vma_policy(struct vm_area_struct *vma, static inline void hugetlb_drop_vma_policy(struct vm_area_struct *vma) { - mpol_cond_put(vma->vm_policy); + mpol_put(vma->vm_policy); } #else static inline void hugetlb_set_vma_policy(struct vm_area_struct *vma, diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 8a74cdcc9af0..24aac42428b3 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -107,7 +107,6 @@ static void hold_task_mempolicy(struct proc_maps_private *priv) task_lock(task); priv->task_mempolicy = get_task_policy(task); - mpol_get(priv->task_mempolicy); task_unlock(task); } static void release_task_mempolicy(struct proc_maps_private *priv) @@ -1949,7 +1948,7 @@ static int show_numa_map(struct seq_file *m, void *v) pol = __get_vma_policy(vma, vma->vm_start); if (pol) { mpol_to_str(buffer, sizeof(buffer), pol); - mpol_cond_put(pol); + mpol_put(pol); } else { mpol_to_str(buffer, sizeof(buffer), proc_priv->task_mempolicy); } diff --git a/kernel/fork.c b/kernel/fork.c index 08969f5aa38d..97ba127a1b89 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -712,7 +712,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, fail_nomem_mas_store: unlink_anon_vmas(tmp); fail_nomem_anon_vma_fork: - mpol_put(vma_policy(tmp)); + mpol_kill(vma_policy(tmp)); fail_nomem_policy: vm_area_free(tmp); fail_nomem: @@ -2537,7 +2537,7 @@ static __latent_entropy struct task_struct *copy_process( bad_fork_cleanup_policy: lockdep_free_task(p); #ifdef CONFIG_NUMA - mpol_put(p->mempolicy); + mpol_kill(p->mempolicy); #endif bad_fork_cleanup_delayacct: delayacct_tsk_free(p); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 546df97c31e4..277330f40818 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1246,7 +1246,7 @@ static struct page *dequeue_huge_page_vma(struct hstate *h, h->resv_huge_pages--; } - mpol_cond_put(mpol); + mpol_put(mpol); return page; err: @@ -2315,7 +2315,7 @@ struct page *alloc_buddy_huge_page_with_mpol(struct hstate *h, if (!page) page = alloc_surplus_huge_page(h, gfp_mask, nid, nodemask); - mpol_cond_put(mpol); + mpol_put(mpol); return page; } @@ -2351,7 +2351,7 @@ struct page *alloc_huge_page_vma(struct hstate *h, struct vm_area_struct *vma, gfp_mask = htlb_alloc_mask(h); node = huge_node(vma, address, gfp_mask, &mpol, &nodemask); page = alloc_huge_page_nodemask(h, node, nodemask, gfp_mask); - mpol_cond_put(mpol); + mpol_put(mpol); return page; } diff --git a/mm/mempolicy.c b/mm/mempolicy.c index ee3e2ed5ef07..f1857ebded46 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -795,11 +795,11 @@ static int vma_replace_policy(struct vm_area_struct *vma, old = vma->vm_policy; vma->vm_policy = new; /* protected by mmap_lock */ - mpol_put(old); + mpol_kill(old); return 0; err_out: - mpol_put(new); + mpol_kill(new); return err; } @@ -890,7 +890,7 @@ static long do_set_mempolicy(unsigned short mode, unsigned short flags, ret = mpol_set_nodemask(new, nodes, scratch); if (ret) { task_unlock(current); - mpol_put(new); + mpol_kill(new); goto out; } @@ -899,7 +899,7 @@ static long do_set_mempolicy(unsigned short mode, unsigned short flags, if (new && new->mode == MPOL_INTERLEAVE) current->il_prev = MAX_NUMNODES-1; task_unlock(current); - mpol_put(old); + mpol_kill(old); ret = 0; out: NODEMASK_SCRATCH_FREE(scratch); @@ -925,8 +925,7 @@ static void get_policy_nodemask(struct mempolicy *p, nodemask_t *nodes) *nodes = p->nodes; break; case MPOL_LOCAL: - /* return empty node mask for local allocation */ - break; + /* return empty node mask for local allocation */killbreak; default: BUG(); } @@ -1370,7 +1369,7 @@ static long do_mbind(unsigned long start, unsigned long len, mmap_write_unlock(mm); mpol_out: - mpol_put(new); + mpol_kill(new); if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) lru_cache_enable(); return err; @@ -1566,7 +1565,7 @@ SYSCALL_DEFINE4(set_mempolicy_home_node, unsigned long, start, unsigned long, le new->home_node = home_node; err = mbind_range(mm, vmstart, vmend, new); - mpol_put(new); + mpol_kill(new); if (err) break; } @@ -1813,14 +1812,13 @@ static struct mempolicy *get_vma_policy(struct vm_area_struct *vma, bool vma_policy_mof(struct vm_area_struct *vma) { struct mempolicy *pol; + bool ret = false; if (vma->vm_ops && vma->vm_ops->get_policy) { - bool ret = false; - pol = vma->vm_ops->get_policy(vma, vma->vm_start); if (pol && (pol->flags & MPOL_F_MOF)) ret = true; - mpol_cond_put(pol); + mpol_put(pol); return ret; } @@ -1828,8 +1826,9 @@ bool vma_policy_mof(struct vm_area_struct *vma) pol = vma->vm_policy; if (!pol) pol = get_task_policy(current); + mpol_put(pol); - return pol->flags & MPOL_F_MOF; + return ret; } bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone) @@ -2193,7 +2192,7 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, unsigned nid; nid = interleave_nid(pol, vma, addr, PAGE_SHIFT + order); - mpol_cond_put(pol); + mpol_put(pol); gfp |= __GFP_COMP; page = alloc_page_interleave(gfp, order, nid); if (page && order > 1) @@ -2208,7 +2207,7 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, node = policy_node(gfp, pol, node); gfp |= __GFP_COMP; page = alloc_pages_preferred_many(gfp, order, node, pol); - mpol_cond_put(pol); + mpol_put(pol); if (page && order > 1) prep_transhuge_page(page); folio = (struct folio *)page; @@ -2233,7 +2232,7 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, nmask = policy_nodemask(gfp, pol); if (!nmask || node_isset(hpage_node, *nmask)) { - mpol_cond_put(pol); + mpol_put(pol); /* * First, try to allocate THP only on local node, but * don't reclaim unnecessarily, just compact. @@ -2258,7 +2257,7 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, nmask = policy_nodemask(gfp, pol); preferred_nid = policy_node(gfp, pol, node); folio = __folio_alloc(gfp, order, preferred_nid, nmask); - mpol_cond_put(pol); + mpol_put(pol); out: return folio; } @@ -2300,6 +2299,7 @@ struct page *alloc_pages(gfp_t gfp, unsigned order) policy_node(gfp, pol, numa_node_id()), policy_nodemask(gfp, pol)); + mpol_put(pol); return page; } EXPORT_SYMBOL(alloc_pages); @@ -2566,7 +2566,7 @@ mpol_shared_policy_lookup(struct shared_policy *sp, unsigned long idx) static void sp_free(struct sp_node *n) { - mpol_put(n->policy); + mpol_kill(n->policy); kmem_cache_free(sn_cache, n); } @@ -2655,7 +2655,7 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long if (curnid != polnid) ret = polnid; out: - mpol_cond_put(pol); + mpol_put(pol); return ret; } @@ -2674,7 +2674,7 @@ void mpol_put_task_policy(struct task_struct *task) pol = task->mempolicy; task->mempolicy = NULL; task_unlock(task); - mpol_put(pol); + mpol_kill(pol); } static void sp_delete(struct shared_policy *sp, struct sp_node *n) @@ -2763,7 +2763,7 @@ static int shared_policy_replace(struct shared_policy *sp, unsigned long start, err_out: if (mpol_new) - mpol_put(mpol_new); + mpol_kill(mpol_new); if (n_new) kmem_cache_free(sn_cache, n_new); @@ -2823,7 +2823,7 @@ void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol) mpol_set_shared_policy(sp, &pvma, new); /* adds ref */ put_new: - mpol_put(new); /* drop initial ref */ + mpol_kill(new); /* drop initial ref */ free_scratch: NODEMASK_SCRATCH_FREE(scratch); put_mpol: diff --git a/mm/mmap.c b/mm/mmap.c index 2def55555e05..7bf785463499 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -140,7 +140,7 @@ static void remove_vma(struct vm_area_struct *vma) vma->vm_ops->close(vma); if (vma->vm_file) fput(vma->vm_file); - mpol_put(vma_policy(vma)); + mpol_kill(vma_policy(vma)); vm_area_free(vma); } @@ -595,7 +595,7 @@ inline int vma_expand(struct ma_state *mas, struct vm_area_struct *vma, if (next->anon_vma) anon_vma_merge(vma, next); mm->map_count--; - mpol_put(vma_policy(next)); + mpol_kill(vma_policy(next)); vm_area_free(next); } @@ -836,7 +836,7 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, if (next->anon_vma) anon_vma_merge(vma, next); mm->map_count--; - mpol_put(vma_policy(next)); + mpol_kill(vma_policy(next)); if (remove_next != 2) BUG_ON(vma->vm_end < next->vm_end); vm_area_free(next); @@ -2253,7 +2253,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma, fput(new->vm_file); unlink_anon_vmas(new); out_free_mpol: - mpol_put(vma_policy(new)); + mpol_kill(vma_policy(new)); out_free_vma: vm_area_free(new); validate_mm_mt(mm); @@ -3246,7 +3246,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, unlink_anon_vmas(new_vma); out_free_mempol: - mpol_put(vma_policy(new_vma)); + mpol_kill(vma_policy(new_vma)); out_free_vma: vm_area_free(new_vma); out: diff --git a/mm/shmem.c b/mm/shmem.c index c1d8b8a1aa3b..11e57d79c104 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1485,7 +1485,7 @@ static void shmem_pseudo_vma_init(struct vm_area_struct *vma, static void shmem_pseudo_vma_destroy(struct vm_area_struct *vma) { /* Drop reference taken by mpol_shared_policy_lookup() */ - mpol_cond_put(vma->vm_policy); + mpol_put(vma->vm_policy); } static struct folio *shmem_swapin(swp_entry_t swap, gfp_t gfp, @@ -3528,7 +3528,7 @@ static int shmem_parse_one(struct fs_context *fc, struct fs_parameter *param) break; case Opt_mpol: if (IS_ENABLED(CONFIG_NUMA)) { - mpol_put(ctx->mpol); + mpol_kill(ctx->mpol); ctx->mpol = NULL; if (mpol_parse_str(param->string, &ctx->mpol)) goto bad_value; @@ -3666,7 +3666,7 @@ static int shmem_reconfigure(struct fs_context *fc) ctx->mpol = NULL; } raw_spin_unlock(&sbinfo->stat_lock); - mpol_put(mpol); + mpol_kill(mpol); return 0; out: raw_spin_unlock(&sbinfo->stat_lock); @@ -3730,7 +3730,7 @@ static void shmem_put_super(struct super_block *sb) free_percpu(sbinfo->ino_batch); percpu_counter_destroy(&sbinfo->used_blocks); - mpol_put(sbinfo->mpol); + mpol_kill(sbinfo->mpol); kfree(sbinfo); sb->s_fs_info = NULL; } @@ -3830,7 +3830,7 @@ static void shmem_free_fc(struct fs_context *fc) struct shmem_options *ctx = fc->fs_private; if (ctx) { - mpol_put(ctx->mpol); + mpol_kill(ctx->mpol); kfree(ctx); } } From patchwork Sun Dec 4 16:14:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhongkun He X-Patchwork-Id: 29423 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1852867wrr; Sun, 4 Dec 2022 08:16:22 -0800 (PST) X-Google-Smtp-Source: AA0mqf4zUuZzdgehC3vMXkJYCUMq2fmDe8mW4Q17C5kqcVWsB1XCiNRouKs6hxQxweQwX4vhH2oo X-Received: by 2002:a17:906:9f11:b0:7b4:7758:f1e1 with SMTP id fy17-20020a1709069f1100b007b47758f1e1mr58755799ejc.145.1670170582318; Sun, 04 Dec 2022 08:16:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670170582; cv=none; d=google.com; s=arc-20160816; b=S3rHwB40sfH4orimu+UjTkVWW9LAZugn3v1shhK6FHJRgW482JgeH9cJ+gCdTfcuOw V23XNupdrw4MU8UgStDUFdVd2R62gbPoZsRQMc7ehVwWX0qspwHKPqinPS4zSdCEmaoT 8Epyb4unN7ebbGHE6MDxqfwxV6cSqWGoBOjlnnYktai1A5fnqj5WcaTNSYlz/ZaRa8Uu tlGq2En7pT6MIOP+Wx8jYhF7U6LHeJpr+Yql+plTbl99XJDb8y4SjWcT2Yesz2cwUkfv dHjnyC+lbsOVeoVNERuerfKg7k++1BA1Q+g5WyycS1fmYh+fXrnqVVm0LZdvcGZAlTlj m+uA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=qRQapFdwTnMNn2b4Tmu2pfTwu3sgopbJ/w2CVO4ajW0=; b=uBhANgufwNbK/Hphe+H52QrAnaj8nHy/7nboxGIykJDMOzImSUQSW5XtXBTpsWYjql 5o2X8S5IpIeD7Wdo8cm38bt12vVzsNNEdByouUNLJBX+frY6YP+81fR8jCmbC3B0WTBP /nAu82gakNlpAQCwqHfr+0fsA6OnTnX+3dr2/vrfXb5HTtvqHeXSpqOrDuFDMlm5c0om +xe91WXZ1WZem3Sv7lfLozUYpNK6uL49eg41paCBqmVH9Cue/GWc4OceuQ64uEqDab6q z6za4pMQdT1NRvquv0MIRvowPU151VUU2wSpyPkEl0px/k/BmUsqq08aWrGy82RhBJiF lPew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=D9HiGoEw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q5-20020a056402248500b0045cfb63a033si10122501eda.551.2022.12.04.08.15.59; Sun, 04 Dec 2022 08:16:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=D9HiGoEw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230117AbiLDQPL (ORCPT + 99 others); Sun, 4 Dec 2022 11:15:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230071AbiLDQPJ (ORCPT ); Sun, 4 Dec 2022 11:15:09 -0500 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B700014005 for ; Sun, 4 Dec 2022 08:15:08 -0800 (PST) Received: by mail-pj1-x1029.google.com with SMTP id b11so9119480pjp.2 for ; Sun, 04 Dec 2022 08:15:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qRQapFdwTnMNn2b4Tmu2pfTwu3sgopbJ/w2CVO4ajW0=; b=D9HiGoEwhwlbDf046jFmOoT756+CfK+J0b38Sr1aPx3LWGaU3r2Spfkj7pxwdkOXmT vgzdlF/ruF1wjnLwFfVTBfI0RaGI/rJ66k9cuNBLzwVMTrlAbZg4GGRD0hkcq+TyUVOw /x9ashaKx4dwa0/EqZH3RjK1GYm6ogmjAXX2tnIRx57KNpmkSGJRx4qzN0P3IuMJ6MNi 5UqnA99cznD1QP3g6CSDnGSbsFaxQxJ8s5GH8lnFiv0fB45EzuenHpzYY7zg56XNR91/ ClkwTE9zWUXRn7uFZu3s4sxCnPnmTexxlnjlZ83A5000Tw+4jbEjDo6pG5hrKmBOtyyG Uvaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qRQapFdwTnMNn2b4Tmu2pfTwu3sgopbJ/w2CVO4ajW0=; b=YUGmkF6sCOLGX5o7OjjANrGxMXSJdBVz2ayL/2O2YLHNH+hOeRZg2qozrEEAKgC1N5 J5vR133+g2XMd1L09IaHuHMKEYvD9MfSz0P+Bq7BKgJs8EM1IDYHmKoy4i9zEKIrfkiJ VxL6qI08rz++NxkyyEYm4DTRf3nQdjRkS6RdQBXg7BFCaemFB5MjKsIfxmQu3mT79PDT HnzSL6i9NRctlua6+W907W6DGlgIx1wZ2IYL9wMGkaGXP/aeLx0j68LmFahiamlGAtp6 Xx61S/rYa/iDuSse/Khmn3NMQesfWZF2m+Oszlmi54gxiJnrbshp9gOfWIsnROPGyNZT X1ow== X-Gm-Message-State: ANoB5plFcf8kA+JobWaW5XjKHB3gQ+c/KozdcJMnAWO3qY4vpE0JlTbr ih1pM2CnLKeQaGFljL3HZamruw== X-Received: by 2002:a17:902:7d94:b0:188:f0e1:ef42 with SMTP id a20-20020a1709027d9400b00188f0e1ef42mr64237242plm.166.1670170508210; Sun, 04 Dec 2022 08:15:08 -0800 (PST) Received: from Tower.bytedance.net ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id n16-20020a170903111000b0016cf3f124e1sm9000323plh.234.2022.12.04.08.15.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 04 Dec 2022 08:15:07 -0800 (PST) From: Zhongkun He To: mhocko@suse.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, wuyun.abel@bytedance.com, Zhongkun He Subject: [PATCH 2/3] mm: fix the reference of mempolicy in some functions. Date: Mon, 5 Dec 2022 00:14:31 +0800 Message-Id: <20221204161432.2149375-3-hezhongkun.hzk@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221204161432.2149375-1-hezhongkun.hzk@bytedance.com> References: <20221204161432.2149375-1-hezhongkun.hzk@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751300788243449946?= X-GMAIL-MSGID: =?utf-8?q?1751300788243449946?= There are some functions that use mempolicy in process context, but don't reference it. Let's fix it to have a clear life time model. Suggested-by: Michal Hocko Signed-off-by: Zhongkun He --- mm/hugetlb.c | 16 ++++++----- mm/mempolicy.c | 78 +++++++++++++++++++++++++------------------------- 2 files changed, 48 insertions(+), 46 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 277330f40818..0c2b5233e0c9 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4353,19 +4353,19 @@ static int __init default_hugepagesz_setup(char *s) } __setup("default_hugepagesz=", default_hugepagesz_setup); -static nodemask_t *policy_mbind_nodemask(gfp_t gfp) +static nodemask_t *policy_mbind_nodemask(gfp_t gfp, struct mempolicy **mpol) { #ifdef CONFIG_NUMA - struct mempolicy *mpol = get_task_policy(current); + *mpol = get_task_policy(current); /* * Only enforce MPOL_BIND policy which overlaps with cpuset policy * (from policy_nodemask) specifically for hugetlb case */ - if (mpol->mode == MPOL_BIND && - (apply_policy_zone(mpol, gfp_zone(gfp)) && - cpuset_nodemask_valid_mems_allowed(&mpol->nodes))) - return &mpol->nodes; + if ((*mpol)->mode == MPOL_BIND && + (apply_policy_zone(*mpol, gfp_zone(gfp)) && + cpuset_nodemask_valid_mems_allowed(&(*mpol)->nodes))) + return &(*mpol)->nodes; #endif return NULL; } @@ -4375,14 +4375,16 @@ static unsigned int allowed_mems_nr(struct hstate *h) int node; unsigned int nr = 0; nodemask_t *mbind_nodemask; + struct mempolicy *mpol = NULL; unsigned int *array = h->free_huge_pages_node; gfp_t gfp_mask = htlb_alloc_mask(h); - mbind_nodemask = policy_mbind_nodemask(gfp_mask); + mbind_nodemask = policy_mbind_nodemask(gfp_mask, &mpol); for_each_node_mask(node, cpuset_current_mems_allowed) { if (!mbind_nodemask || node_isset(node, *mbind_nodemask)) nr += array[node]; } + mpol_put(mpol); return nr; } diff --git a/mm/mempolicy.c b/mm/mempolicy.c index f1857ebded46..0feffb7ff01e 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -159,7 +159,7 @@ int numa_map_to_online_node(int node) EXPORT_SYMBOL_GPL(numa_map_to_online_node); /* Obtain a reference on the specified task mempolicy */ -static mempolicy *get_task_mpol(struct task_struct *p) +static struct mempolicy *get_task_mpol(struct task_struct *p) { struct mempolicy *pol; @@ -925,7 +925,8 @@ static void get_policy_nodemask(struct mempolicy *p, nodemask_t *nodes) *nodes = p->nodes; break; case MPOL_LOCAL: - /* return empty node mask for local allocation */killbreak; + /* return empty node mask for local allocation */ + break; default: BUG(); } @@ -951,7 +952,7 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, int err; struct mm_struct *mm = current->mm; struct vm_area_struct *vma = NULL; - struct mempolicy *pol = current->mempolicy, *pol_refcount = NULL; + struct mempolicy *pol; if (flags & ~(unsigned long)(MPOL_F_NODE|MPOL_F_ADDR|MPOL_F_MEMS_ALLOWED)) @@ -966,8 +967,10 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, task_unlock(current); return 0; } + pol = get_task_mpol(current); if (flags & MPOL_F_ADDR) { + mpol_put(pol); /* put the refcount of task mpol */ /* * Do NOT fall back to task policy if the * vma/shared policy at addr is NULL. We @@ -979,27 +982,19 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, mmap_read_unlock(mm); return -EFAULT; } - if (vma->vm_ops && vma->vm_ops->get_policy) - pol = vma->vm_ops->get_policy(vma, addr); - else - pol = vma->vm_policy; - } else if (addr) - return -EINVAL; + /* obtain a reference to vma mpol. */ + pol = __get_vma_policy(vma, addr); + mmap_read_unlock(mm); + } else if (addr) { + err = -EINVAL; + goto out; + } if (!pol) pol = &default_policy; /* indicates default behavior */ if (flags & MPOL_F_NODE) { if (flags & MPOL_F_ADDR) { - /* - * Take a refcount on the mpol, because we are about to - * drop the mmap_lock, after which only "pol" remains - * valid, "vma" is stale. - */ - pol_refcount = pol; - vma = NULL; - mpol_get(pol); - mmap_read_unlock(mm); err = lookup_node(mm, addr); if (err < 0) goto out; @@ -1023,21 +1018,19 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, err = 0; if (nmask) { - if (mpol_store_user_nodemask(pol)) { + /* + * There is no need for a lock, since we get + * a reference to mpol. + */ + if (mpol_store_user_nodemask(pol)) *nmask = pol->w.user_nodemask; - } else { - task_lock(current); + else get_policy_nodemask(pol, nmask); - task_unlock(current); - } } out: - mpol_cond_put(pol); - if (vma) - mmap_read_unlock(mm); - if (pol_refcount) - mpol_put(pol_refcount); + if (pol != &default_policy) + mpol_put(pol); return err; } @@ -1923,16 +1916,18 @@ unsigned int mempolicy_slab_node(void) if (!in_task()) return node; - policy = current->mempolicy; + policy = get_task_mpol(current); if (!policy) return node; switch (policy->mode) { case MPOL_PREFERRED: - return first_node(policy->nodes); + node = first_node(policy->nodes); + break; case MPOL_INTERLEAVE: - return interleave_nodes(policy); + node = interleave_nodes(policy); + break; case MPOL_BIND: case MPOL_PREFERRED_MANY: @@ -1948,14 +1943,17 @@ unsigned int mempolicy_slab_node(void) zonelist = &NODE_DATA(node)->node_zonelists[ZONELIST_FALLBACK]; z = first_zones_zonelist(zonelist, highest_zoneidx, &policy->nodes); - return z->zone ? zone_to_nid(z->zone) : node; + node = z->zone ? zone_to_nid(z->zone) : node; + break; } case MPOL_LOCAL: - return node; + break; default: BUG(); } + mpol_put(policy); + return node; } /* @@ -2379,21 +2377,23 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, unsigned long nr_pages, struct page **page_array) { struct mempolicy *pol = &default_policy; + unsigned long pages; if (!in_interrupt() && !(gfp & __GFP_THISNODE)) pol = get_task_policy(current); if (pol->mode == MPOL_INTERLEAVE) - return alloc_pages_bulk_array_interleave(gfp, pol, + pages = alloc_pages_bulk_array_interleave(gfp, pol, nr_pages, page_array); - - if (pol->mode == MPOL_PREFERRED_MANY) - return alloc_pages_bulk_array_preferred_many(gfp, + else if (pol->mode == MPOL_PREFERRED_MANY) + pages = alloc_pages_bulk_array_preferred_many(gfp, numa_node_id(), pol, nr_pages, page_array); - - return __alloc_pages_bulk(gfp, policy_node(gfp, pol, numa_node_id()), + else + pages = __alloc_pages_bulk(gfp, policy_node(gfp, pol, numa_node_id()), policy_nodemask(gfp, pol), nr_pages, NULL, page_array); + mpol_put(pol); + return pages; } int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst) From patchwork Sun Dec 4 16:14:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhongkun He X-Patchwork-Id: 29424 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1855195wrr; Sun, 4 Dec 2022 08:23:45 -0800 (PST) X-Google-Smtp-Source: AA0mqf67OV8XMG+4A9eSSJNVv/z76P5w7lstmUs/KA69xwsrKq+x6YT0wQP0DWhb9rJccR119Xoi X-Received: by 2002:a17:90a:1bc2:b0:218:8bdb:de3f with SMTP id r2-20020a17090a1bc200b002188bdbde3fmr76649441pjr.225.1670171025713; Sun, 04 Dec 2022 08:23:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1670171025; cv=none; d=google.com; s=arc-20160816; b=q3KiPu5wHBknUP9KELEMdixnN96QPtezdxd56yicx8v+YG/BpPxaCavUa/lgXV7BNe gBB9YRznrovyARPeBXGOtAocyiWTcOk43C36c9rZo+kvq2iA+TlZXt5SUu1DWdzMfoXb K8gFAy5RXsZO/n/Dy+FGKfE1W/KNxFvKTB6NaguK4tYevSt4NW4Afai+cFooiD+eGeoY zByuODrelIGZcKvf2mkcZRkiDMx4oKmNoAzQIxmC8cJ0huXn2k/9idMpCpqus2NtuMH7 Z7G9/LMWuCnS4n17Ty224L2v/QZtUy6R1q/gBNb4dhq6yLGQRq6PxdtpOt8IK3lW9sMz sZPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=svRUdELzl130u78ZHoVrQ8kLS3/B/K+fs5QOXxsPpJs=; b=mbzUGy3WnPsu70yxT8XIwdHIZcMH7nPf3JXUIN9ndqvrjCr3U7/KIYibQoCyy2Ma3A X06xzKLFNwyz7LG2fHbgIPnI3/Bz6A3U/cXKjhU6xzUWb9w1iTjp9Ev3+DVcXHZe2Qw7 ZjPdjwtWqcCpNxNd8/pp/wG/DD9jX37ybCVVcWdtJsfg+OZlOTxwLNmSNvIuUtneniy+ JkzUxpsuIpAYfcniTAWJV9uQ1Kwny95OFRyFBgeZinipFgjjwsYHVNCKnO1S05DmkBxA 0oLrHCchw+L8+BbBxXmCahLT9RXofq0GJaEI2lxOzcyMVlLlcOQtZNGSq/jkkbwkYVzt wekA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=CrJxh2mL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 78-20020a630051000000b00476d18b0d24si12880345pga.742.2022.12.04.08.23.29; Sun, 04 Dec 2022 08:23:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=CrJxh2mL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230133AbiLDQPW (ORCPT + 99 others); Sun, 4 Dec 2022 11:15:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41046 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230151AbiLDQPS (ORCPT ); Sun, 4 Dec 2022 11:15:18 -0500 Received: from mail-pf1-x42f.google.com (mail-pf1-x42f.google.com [IPv6:2607:f8b0:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 234A214085 for ; Sun, 4 Dec 2022 08:15:17 -0800 (PST) Received: by mail-pf1-x42f.google.com with SMTP id c13so2090866pfp.5 for ; Sun, 04 Dec 2022 08:15:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=svRUdELzl130u78ZHoVrQ8kLS3/B/K+fs5QOXxsPpJs=; b=CrJxh2mLnr2Kc1c43TsZ5hHQRYXrbNQuNpWgBlRAvtB/ZBHABi5itLHDIbZd9ExahR 3YhQ/UvkZtGBvY2umOJvuIjTbtfwpJVuACQgmYsiIFvBCm8OMQ9nQcISiO/c1kTwDUFb 5YK2bw4i6KPFIMryLYNRkzut5BKoOOW7855jORF73gHk67NU1Lb+PjeQX16PBo8F6vrJ EqCEfGdMdTesLZ0vAlCC9I/lmpri57kmdp1Jv/+R1qNOJaOEdq+bACUBpqwWHURkDW/+ pV0OhPAxrXggkfQmCHVVuyH1zHQ+E2N0SbOEgaTovcU1BNRjQgD27KP7d1RX6X5q+7Y0 Iwjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=svRUdELzl130u78ZHoVrQ8kLS3/B/K+fs5QOXxsPpJs=; b=K3spOxDDkbXffvk4FGRiVtI92Zgq1blhETK0QZu2DCNymFaIsiiKxrE6df58Vb9cFd L93skmAYrzof+pZzSZX5bAlq35wwJ/P06pkK3hgEc/4WclYvXUaEVLXECG+WwY/Khrk3 suivZ5yHyNLSBUbjaOlEYH5T9mRgsWSM8MoliODdlOG2HkFT5Tuc2xYyjHBNo8a2Kn80 sCEU7lQ/S56lhVjbhv8s9gTYp0wpJIVX41tWIb9mzE/FrCfDgwtsw2yCRHUYEuWMYWEG dOfmlxmnUkBiTL+tdf1ZbsaDleHGJ3hneYwoHhBLGw/GLy6sesdmgw9CpzyF1vSOLoHZ lGKQ== X-Gm-Message-State: ANoB5plnDhIorPcwlSjXzNt08qOoh4tF3Skr2rwRVtHqqkIJ8FzAq3XB DnqATscC+1WZl5Szhy19vQW6Yg== X-Received: by 2002:a63:1f63:0:b0:460:ec46:3645 with SMTP id q35-20020a631f63000000b00460ec463645mr73298033pgm.92.1670170516616; Sun, 04 Dec 2022 08:15:16 -0800 (PST) Received: from Tower.bytedance.net ([139.177.225.248]) by smtp.gmail.com with ESMTPSA id n16-20020a170903111000b0016cf3f124e1sm9000323plh.234.2022.12.04.08.15.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 04 Dec 2022 08:15:16 -0800 (PST) From: Zhongkun He To: mhocko@suse.com, akpm@linux-foundation.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, wuyun.abel@bytedance.com, Zhongkun He Subject: [PATCH 3/3] mm: add __rcu symbol for task->mempolicy Date: Mon, 5 Dec 2022 00:14:32 +0800 Message-Id: <20221204161432.2149375-4-hezhongkun.hzk@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221204161432.2149375-1-hezhongkun.hzk@bytedance.com> References: <20221204161432.2149375-1-hezhongkun.hzk@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751301253540135018?= X-GMAIL-MSGID: =?utf-8?q?1751301253540135018?= The task->mempolicy is protected by task_lock in slow path, but there is no locking and reference in hot path for performance. It will be difficult if other processes want to adjust it. It is for these reasons to add __rcu symbol for task mempolicy. There is no need to add RCU protection to vma mempolicy, which is protected by mmap_lock. Suggested-by: Michal Hocko Signed-off-by: Zhongkun He --- include/linux/sched.h | 2 +- mm/mempolicy.c | 9 ++++----- mm/slab.c | 5 +++-- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index ffb6eb55cd13..c8a297ca61ab 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1252,7 +1252,7 @@ struct task_struct { #endif #ifdef CONFIG_NUMA /* Protected by alloc_lock: */ - struct mempolicy *mempolicy; + struct mempolicy __rcu *mempolicy; short il_prev; short pref_node_fork; #endif diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 0feffb7ff01e..837083fff9c8 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -894,8 +894,7 @@ static long do_set_mempolicy(unsigned short mode, unsigned short flags, goto out; } - old = current->mempolicy; - current->mempolicy = new; + old = rcu_replace_pointer(current->mempolicy, new, true); if (new && new->mode == MPOL_INTERLEAVE) current->il_prev = MAX_NUMNODES-1; task_unlock(current); @@ -999,7 +998,7 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, if (err < 0) goto out; *policy = err; - } else if (pol == current->mempolicy && + } else if (pol == rcu_access_pointer(current->mempolicy) && pol->mode == MPOL_INTERLEAVE) { *policy = next_node_in(current->il_prev, pol->nodes); } else { @@ -2065,7 +2064,7 @@ bool init_nodemask_of_mempolicy(nodemask_t *mask) { struct mempolicy *mempolicy; - if (!(mask && current->mempolicy)) + if (!(mask && rcu_access_pointer(current->mempolicy))) return false; task_lock(current); @@ -2426,7 +2425,7 @@ struct mempolicy *__mpol_dup(struct mempolicy *old) return ERR_PTR(-ENOMEM); /* task's mempolicy is protected by alloc_lock */ - if (old == current->mempolicy) { + if (old == rcu_access_pointer(current->mempolicy)) { task_lock(current); *new = *old; task_unlock(current); diff --git a/mm/slab.c b/mm/slab.c index 59c8e28f7b6a..f205869d6c36 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -3052,7 +3052,7 @@ static void *alternate_node_alloc(struct kmem_cache *cachep, gfp_t flags) nid_alloc = nid_here = numa_mem_id(); if (cpuset_do_slab_mem_spread() && (cachep->flags & SLAB_MEM_SPREAD)) nid_alloc = cpuset_slab_spread_node(); - else if (current->mempolicy) + else if (rcu_access_pointer(current->mempolicy)) nid_alloc = mempolicy_slab_node(); if (nid_alloc != nid_here) return ____cache_alloc_node(cachep, flags, nid_alloc); @@ -3188,7 +3188,8 @@ __do_cache_alloc(struct kmem_cache *cachep, gfp_t flags, int nodeid) int slab_node = numa_mem_id(); if (nodeid == NUMA_NO_NODE) { - if (current->mempolicy || cpuset_do_slab_mem_spread()) { + if (rcu_access_pointer(current->mempolicy) || + cpuset_do_slab_mem_spread()) { objp = alternate_node_alloc(cachep, flags); if (objp) goto out;