From patchwork Fri Jan 27 19:41:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 49649 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp1013030wrn; Fri, 27 Jan 2023 11:49:40 -0800 (PST) X-Google-Smtp-Source: AMrXdXtBqiNKNMb3QaMM/4uoCXx5fpSEqwefBRVacoo+pjLEEFOd983QCwKmI5EVp3MQOWY7jHWC X-Received: by 2002:a17:907:91d3:b0:86e:3e0c:ae50 with SMTP id h19-20020a17090791d300b0086e3e0cae50mr33997210ejz.14.1674848980405; Fri, 27 Jan 2023 11:49:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674848980; cv=none; d=google.com; s=arc-20160816; b=UDq4kKPFC8vR8Y5dFatorKA7XAFoWvXeIcY4D3C9qZ5SzqriJm6uN+vXks/4zH0Flw 1nu6Ryr70CzX+PuKQpHND0y+PJSSeIYReDYFAdNiCtCkYXM8ubMr7N6KyCerYgr62khs dhYxA7cf9Vov2nU1oHarH2lVtUd2ZY1OXgiJuJJSJzi6h7SfeZTYZszvth7H1x5E2Ueg htdVxXdu3yETrS6ZOZuDhg3qsLY5yZ+FazUgZdAvNaORgOv3Jw4Of6Ms7qzAooBtW+W/ I3Z1Xk7DJ1cRbKH+dCqD+9lBSTSzn6QruBnBC4L6KM+dMyl0OflvJvFcEvX+tTQHNmf/ WCfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=y3NZtkliWQhGwQQ6J+V08tBPQFOX7N0W14jS8GdMVSo=; b=ilC8tfLoP0dqljkp5ULgjd3DbF3gmjw6KF0ePlLS8PD0ZdcY9ZNZUZnDJBmEAo+iJV DFDyBBGqvqzK+cIy2Y9FHYY2nPqESVICRJggqBKhOCnYGwrv9BWBlXAegsFbMMuZ+5kz A7+ZUBq9Q0RuTZlaW7f9s8HXhosRVyXCBCrWQjv4L7ux9gXkpTTEpzgbsjA17UrxUcit 2NDFuFYGPY63h1nATAwTa7IwKgqpZxRqdCtdQr0lRwkraxv+iMBhKvkzpFO4Ednlxhyd GjKxnJSZD/Bme4cvxlxBAZlijcMp957UNU0jgZLXYHjG6vPQubpflrJfCwjzoWPeeNko VLaA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=R8vm5sGe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f22-20020a1709063f5600b0086ea1c7ae27si6653408ejj.241.2023.01.27.11.49.17; Fri, 27 Jan 2023 11:49:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=R8vm5sGe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232484AbjA0Tp6 (ORCPT + 99 others); Fri, 27 Jan 2023 14:45:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232148AbjA0ToV (ORCPT ); Fri, 27 Jan 2023 14:44:21 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1C1923859 for ; Fri, 27 Jan 2023 11:43:14 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id o1-20020a252801000000b0080b8600bdc9so6363168ybo.3 for ; Fri, 27 Jan 2023 11:43:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=y3NZtkliWQhGwQQ6J+V08tBPQFOX7N0W14jS8GdMVSo=; b=R8vm5sGePmLU1Pe4mKnKd7BNiKDqaMQRSE2Uxaf9jHVe/UX+8wZaZnli/NAPrvHh5D Gp2YVdh1UhOUz23uxWg1VsM9bUD5cup3/ZbXShp0DX2DQS4OYSwXrDDdrwmKhIAFH0Hw 74ENSW8vtZuV6jUaLsmkcaEy+yDT+z+95V7siTwW55EWIBuns5aniQb/TRczPRcRq3g4 VhtuB5yMK58BxiemGZ2CqsONqT2FuthDA4119d3zQhl7jpXryXulujWcH7o4rSRWH8Tl w610sqrsMHevwvr509uliYrTdDC7Uzh2VKUBma4LjNdxi7kycc6HPZnQZbUV0crFM3wd Zbgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=y3NZtkliWQhGwQQ6J+V08tBPQFOX7N0W14jS8GdMVSo=; b=U2RH4n3XmSgimihjQh1g+eLe9eLxm8x90wjyM8hU4EiCena1tnn50vrVzQxcLIq1Tf CvuAkAy7nNUkhix2kX0rTB7Aaf0dPW6qMFWGEewhH4ZKPvFt+cjzs7s8Elb+p32/Jfyl wtvKAjgF9+2RfNM0B1If+6qgLnVudCtyvw1Xxs99HSTBkq+BEQ7nDO4LWhrwtlqqSCkD H8ENE4a9cMpSInHOX01ZdDGg2tjDjJnAu4uwdsX2dmE6Q4Sv5QFFs9qm/WRMEDamnfxr ujwnlLxPjg5LFAvD1HpA1bebMGs/X4lKv+RlMuafUyv6jH3nO5LhW7hmEcUsLlhDBijq UeCg== X-Gm-Message-State: AO0yUKXBZn2bXYeyQbUK3ENULzq+MLWv/UFV4b4wSktOqoeY8kmA9e11 xfnOTApstaZKaIrO8fz6frgVoIBvYBw= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:4e19:be9:c5d0:8483]) (user=surenb job=sendgmr) by 2002:a25:db07:0:b0:80b:8dd0:7b35 with SMTP id g7-20020a25db07000000b0080b8dd07b35mr1614607ybf.322.1674848549978; Fri, 27 Jan 2023 11:42:29 -0800 (PST) Date: Fri, 27 Jan 2023 11:41:09 -0800 In-Reply-To: <20230127194110.533103-1-surenb@google.com> Mime-Version: 1.0 References: <20230127194110.533103-1-surenb@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog Message-ID: <20230127194110.533103-33-surenb@google.com> Subject: [PATCH v2 32/33] mm/mmap: free vm_area_struct without call_rcu in exit_mmap From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756206444109975271?= X-GMAIL-MSGID: =?utf-8?q?1756206444109975271?= call_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. Because exit_mmap() is called only after the last mm user drops its refcount, the page fault handlers can't be racing with it. Any other possible user like oom-reaper or process_mrelease are already synchronized using mmap_lock. Therefore exit_mmap() can free VMAs directly, without the use of call_rcu(). Expose __vm_area_free() and use it from exit_mmap() to avoid possible call_rcu() floods and performance regressions caused by it. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 2 ++ kernel/fork.c | 2 +- mm/mmap.c | 11 +++++++---- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a6de58bb40c7..1c4ddcd6fd84 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -250,6 +250,8 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); +/* Use only if VMA has no other users */ +void __vm_area_free(struct vm_area_struct *vma); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/kernel/fork.c b/kernel/fork.c index a08cc0e2bfde..d0999de82f94 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -480,7 +480,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } -static void __vm_area_free(struct vm_area_struct *vma) +void __vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); diff --git a/mm/mmap.c b/mm/mmap.c index 3d0cfbc92745..1028fe131bb7 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -133,7 +133,7 @@ void unlink_file_vma(struct vm_area_struct *vma) /* * Close a vm structure and free it. */ -static void remove_vma(struct vm_area_struct *vma) +static void remove_vma(struct vm_area_struct *vma, bool unreachable) { might_sleep(); if (vma->vm_ops && vma->vm_ops->close) @@ -141,7 +141,10 @@ static void remove_vma(struct vm_area_struct *vma) if (vma->vm_file) fput(vma->vm_file); mpol_put(vma_policy(vma)); - vm_area_free(vma); + if (unreachable) + __vm_area_free(vma); + else + vm_area_free(vma); } static inline struct vm_area_struct *vma_prev_limit(struct vma_iterator *vmi, @@ -2134,7 +2137,7 @@ static inline void remove_mt(struct mm_struct *mm, struct ma_state *mas) if (vma->vm_flags & VM_ACCOUNT) nr_accounted += nrpages; vm_stat_account(mm, vma->vm_flags, -nrpages); - remove_vma(vma); + remove_vma(vma, false); } vm_unacct_memory(nr_accounted); validate_mm(mm); @@ -3083,7 +3086,7 @@ void exit_mmap(struct mm_struct *mm) do { if (vma->vm_flags & VM_ACCOUNT) nr_accounted += vma_pages(vma); - remove_vma(vma); + remove_vma(vma, true); count++; cond_resched(); } while ((vma = mas_find(&mas, ULONG_MAX)) != NULL);